I’ve checked your logs and there is nothing which could tell us what is wrongs. When ROR fails to start there should be error logged - like this:
[2020-08-13T17:41:19,993][INFO ][t.b.r.b.Ror ] [n1_it] Loading ReadonlyREST settings from index failed: cannot find index
[2020-08-13T17:41:19,994][INFO ][t.b.r.b.Ror ] [n1_it] Loading ReadonlyREST settings from file: /etc/elasticsearch/readonlyrest.yml
[2020-08-13T17:41:20,006][ERROR][t.b.r.b.Ror ] [n1_it] Loading ReadonlyREST from file failed: Cannot find settings file: /etc/elasticsearch/readonlyrest.yml
[2020-08-13T17:41:20,008][ERROR][t.b.r.e.IndexLevelActionFilter] [n1_it] ROR starting failure:
You said that 2 nodes are working great with the same config, so it should be sth wrong with your node configuration I suppose. And maybe the error logs were logged to stderr and that’s why we see nothing here? Maybe you should also try to run the 3rd node in docker container and check if it starts? It’s really hard to help you, because I don’t think it’s ROR issue (at least not directly).
I installed ROR 1.21 on my 2 working nodes, it is working fine and I see there are more ROR traces at startup.
I have asked my failing node VM to be fully rebuild and I will retry installing everything the ROR 1.21 as soon as the new VM is ready.
I will keep you posted.
Thanks
I have reinstall everything.
The new ROR version is tracinbg much more and an error appears.
Is is saying :
access denied (“java.io.FilePermission” “/etc/elasticsearch/elasticsearch.yml” “read”)
but my file is perfectly readable it has the same rights that on the other nodes
[root@p-elastic-rec ~]# ls -al /etc/elasticsearch/
total 60
drwxr-s— 2 root elasticsearch 4096 Aug 3 15:39 .
drwxr-xr-x. 87 root root 12288 Jul 21 16:41 …
-rw-rw---- 1 root elasticsearch 3124 Aug 3 15:08 elasticsearch.yml
-rw-rw---- 1 root elasticsearch 131 Nov 22 2019 readonlyrest.yml
full error trace :
[2020-08-14T14:51:44,706][ERROR][t.b.r.e.IndexLevelActionFilter] [Elastic-octane-rec-3] ROR starting failure:
tech.beshu.ror.exceptions$StartingFailureException: Cannot start ReadonlyREST
at tech.beshu.ror.exceptions$StartingFailureException$.from(exceptions.scala:39) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$startRorInstance$3(IndexLevelActionFilter.scala:152) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$startRorInstance$3$adapted(IndexLevelActionFilter.scala:143) ~[?:?]
at monix.execution.Callback$$anon$2.tryApply(Callback.scala:296) ~[?:3.0.0]
at monix.execution.Callback$$anon$2.apply(Callback.scala:289) ~[?:3.0.0]
at monix.execution.Callback$$anon$2.onError(Callback.scala:286) ~[?:3.0.0]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:108) ~[?:3.0.0]
at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101) ~[?:3.0.0]
at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74) ~[?:3.0.0]
at monix.execution.Callback.apply(Callback.scala:116) ~[?:3.0.0]
at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1(TaskFromFuture.scala:112) ~[?:3.0.0]
at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1$adapted(TaskFromFuture.scala:111) ~[?:3.0.0]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) [?:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:1.8.0_262]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_262]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_262]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_262]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) [?:1.8.0_262]
Caused by: java.security.AccessControlException: access denied (“java.io.FilePermission” “/etc/elasticsearch/elasticsearch.yml” “read”)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_262]
at java.security.AccessController.checkPermission(AccessController.java:886) ~[?:1.8.0_262]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_262]
at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_262]
at java.io.FileInputStream.(FileInputStream.java:127) ~[?:1.8.0_262]
at java.io.FileReader.(FileReader.java:72) ~[?:1.8.0_262]
at better.files.File.newFileReader(File.scala:429) ~[?:3.8.0]
at better.files.File.fileReader(File.scala:432) ~[?:3.8.0]
at tech.beshu.ror.configuration.EsConfigFileLoader.loadConfigFromFile(EsConfigFileLoader.scala:33) ~[?:?]
at tech.beshu.ror.configuration.RorIndexNameConfiguration$.$anonfun$load$1(RorIndexNameConfiguration.scala:42) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81) ~[?:3.0.0]
… 11 more
The only difference I see is that the java version is slitly different :
working nodes : OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
on failing node : OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)
maybe the new java requires some specific rights greater than the one given to ROR during the install ?
I have uninstall the latest java version and install the previous one on the failing node and … it works !!!
so apparently it means that ROR for elastic 6.8.1 is not compatible with the latest java OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode.
Maybe you can build a new ROR version adding the missing java rights ?
It would be great if you could do that
Thanks
Hi, I have reproduced and checked this issue with build 25.262-b10 and couldn’t find any solution yet. However I’ve found that there is newer version of openjdk 25.265-b01, available for example here Release jdk8u265-b01 · AdoptOpenJDK/openjdk8-binaries · GitHub. The issue doesn’t show up on 25.265-b01 on my machine and ROR works without any problems. Can you check this version and possibly migrate to it? I don’t know what’s causing permission problem with 25.262-b10, but it doesn’t work even with this policy setting permission java.security.AllPermission;
Hi,
My OS is RedHat 7.8 and the last openjdk version proposed my yum is 262.b10.
265.b01 is not available.
I am not authorized to install a non yum officially available version.
And I won’t be able to keep forever on 262.b09 old java version.
This is thus blocking.
Do you know if openjdk 265.b01 will become officially availble through yum for redhat 7 ?
Thanks
Hi, I tried to add modifyThreadGroup permission to ROR plugin, but it haven’t fixed anything. I will dig it further, but I have found workaround if you are loading your config only from file. Please add this line readonlyrest.force_load_from_file: true to your elasticsearch.yml. Let me know if it worked.