AccessDeniedException on _state

Dear forum,

I have just installed a CrateDB 4.6.4 cluster of 3 nodes and it started successful, after a restart I have this error:

[2021-10-21T09:25:31,173][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [crate01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.bootstrap.StartupExceptionProxy.<init>(StartupExceptionProxy.java:30) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:162) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.execute(CrateDB.java:138) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[crate-server.jar:?]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[crate-cli.jar:?]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[crate-cli.jar:?]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:91) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.main(CrateDB.java:84) ~[crate-app.jar:4.6.4]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:651) ~[crate-server.jar:?]
        at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.6.4]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /mnt/disk01/nodes/0/_state
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:439) ~[?:?]
        at java.nio.file.Files.newDirectoryStream(Files.java:476) ~[?:?]
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:214) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:233) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:700) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
        at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:266) ~[crate-server.jar:?]
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:416) ~[crate-server.jar:?]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:329) ~[crate-server.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:270) ~[crate-server.jar:?]
        at io.crate.node.CrateNode.<init>(CrateNode.java:53) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy$1.<init>(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.setup(BootstrapProxy.java:192) ~[crate-app.jar:4.6.4]
        at org.elasticsearch.bootstrap.BootstrapProxy.init(BootstrapProxy.java:266) ~[crate-app.jar:4.6.4]
        at io.crate.bootstrap.CrateDB.init(CrateDB.java:158) ~[crate-app.jar:4.6.4]
        ... 6 more

This already happen two times, the first time I thought I did something wrong and I have deleted the whole database disks, but this time I need to understand how to recover the problem without loose the data.

What happen here?

Regards,
Stefano

How did you install CrateDB?
Does the user that runs CrateDB have sufficient privileges on the directory?

Hi,

I have installed the DB via RPM. I am on a Centos 7.
It seems to me that everything is fine:

[root@crate01 ~]# ls -al  /mnt/disk01/nodes/0/_state
total 24
d-wxrw--wt. 2 crate crate  121 Oct 21 09:11 .
d-wxrw--wt. 3 crate crate   37 Oct 21 09:25 ..
--wxrw--wt. 1 crate crate  278 Oct 21 09:11 _7.cfe
--wxrw--wt. 1 crate crate 1776 Oct 21 09:11 _7.cfs
--wxrw--wt. 1 crate crate  379 Oct 21 09:11 _7.si
--wxrw--wt. 1 crate crate  109 Oct 20 11:11 manifest-0.st
--wxrw--wt. 1 crate crate   89 Oct 20 11:11 node-0.st
--wxrw--wt. 1 crate crate  246 Oct 21 09:11 segments_r
--wxrw--wt. 1 crate crate    0 Oct 20 11:11 write.lock

Or not?

Regards,
S.

I got it.

The permissions of the directories are wrong, this is due to the Ansible script I have used to configure the DB.

Thanks,
S.

1 Like