Readonlyrest for elastic won't start (1.18.8_es6.8.1)

I’ve checked your logs and there is nothing which could tell us what is wrongs. When ROR fails to start there should be error logged - like this:

[2020-08-13T17:41:19,993][INFO ][t.b.r.b.Ror              ] [n1_it] Loading ReadonlyREST settings from index failed: cannot find index
[2020-08-13T17:41:19,994][INFO ][t.b.r.b.Ror              ] [n1_it] Loading ReadonlyREST settings from file: /etc/elasticsearch/readonlyrest.yml
[2020-08-13T17:41:20,006][ERROR][t.b.r.b.Ror              ] [n1_it] Loading ReadonlyREST from file failed: Cannot find settings file: /etc/elasticsearch/readonlyrest.yml
[2020-08-13T17:41:20,008][ERROR][t.b.r.e.IndexLevelActionFilter] [n1_it] ROR starting failure:

You said that 2 nodes are working great with the same config, so it should be sth wrong with your node configuration I suppose. And maybe the error logs were logged to stderr and that’s why we see nothing here? Maybe you should also try to run the 3rd node in docker container and check if it starts? It’s really hard to help you, because I don’t think it’s ROR issue (at least not directly).

And please use the newest ROR.

I installed ROR 1.21 on my 2 working nodes, it is working fine and I see there are more ROR traces at startup.
I have asked my failing node VM to be fully rebuild and I will retry installing everything the ROR 1.21 as soon as the new VM is ready.
I will keep you posted.
Thanks

I have reinstall everything.
The new ROR version is tracinbg much more and an error appears.
Is is saying :
access denied (“java.io.FilePermission” “/etc/elasticsearch/elasticsearch.yml” “read”)

but my file is perfectly readable it has the same rights that on the other nodes
[[email protected] ~]# ls -al /etc/elasticsearch/
total 60
drwxr-s— 2 root elasticsearch 4096 Aug 3 15:39 .
drwxr-xr-x. 87 root root 12288 Jul 21 16:41 …
-rw-rw---- 1 root elasticsearch 3124 Aug 3 15:08 elasticsearch.yml
-rw-rw---- 1 root elasticsearch 131 Nov 22 2019 readonlyrest.yml

full error trace :
[2020-08-14T14:51:44,706][ERROR][t.b.r.e.IndexLevelActionFilter] [Elastic-octane-rec-3] ROR starting failure:
tech.beshu.ror.exceptions$StartingFailureException: Cannot start ReadonlyREST
at tech.beshu.ror.exceptions$StartingFailureException$.from(exceptions.scala:39) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$startRorInstance$3(IndexLevelActionFilter.scala:152) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$startRorInstance$3$adapted(IndexLevelActionFilter.scala:143) ~[?:?]
at monix.execution.Callback$$anon$2.tryApply(Callback.scala:296) ~[?:3.0.0]
at monix.execution.Callback$$anon$2.apply(Callback.scala:289) ~[?:3.0.0]
at monix.execution.Callback$$anon$2.onError(Callback.scala:286) ~[?:3.0.0]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:108) ~[?:3.0.0]
at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101) ~[?:3.0.0]
at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74) ~[?:3.0.0]
at monix.execution.Callback.apply(Callback.scala:116) ~[?:3.0.0]
at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1(TaskFromFuture.scala:112) ~[?:3.0.0]
at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1$adapted(TaskFromFuture.scala:111) ~[?:3.0.0]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) [?:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:1.8.0_262]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_262]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_262]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_262]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) [?:1.8.0_262]
Caused by: java.security.AccessControlException: access denied (“java.io.FilePermission” “/etc/elasticsearch/elasticsearch.yml” “read”)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_262]
at java.security.AccessController.checkPermission(AccessController.java:886) ~[?:1.8.0_262]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_262]
at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_262]
at java.io.FileInputStream.(FileInputStream.java:127) ~[?:1.8.0_262]
at java.io.FileReader.(FileReader.java:72) ~[?:1.8.0_262]
at better.files.File.newFileReader(File.scala:429) ~[?:3.8.0]
at better.files.File.fileReader(File.scala:432) ~[?:3.8.0]
at tech.beshu.ror.configuration.EsConfigFileLoader.loadConfigFromFile(EsConfigFileLoader.scala:33) ~[?:?]
at tech.beshu.ror.configuration.RorIndexNameConfiguration$.$anonfun$load$1(RorIndexNameConfiguration.scala:42) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81) ~[?:3.0.0]
… 11 more

The only difference I see is that the java version is slitly different :
working nodes : OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
on failing node : OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)
maybe the new java requires some specific rights greater than the one given to ROR during the install ?

I have uninstall the latest java version and install the previous one on the failing node and … it works !!!
so apparently it means that ROR for elastic 6.8.1 is not compatible with the latest java OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode.
Maybe you can build a new ROR version adding the missing java rights ?
It would be great if you could do that
Thanks

I have to check it …

@lucette great analysis, thanks for reporting back about this. It’s very weird indeed and worth investigating.

I saw that you released a new 1.22 ROR version but I don’t see hints of a fix for the startup problem, did you manage to fix it ?
Thanks

@lucette sorry not yet. But the task is in a current sprint backlog

we are waiting on this fix, please let us know when this will be avaialble!

Thanks

Hi, I have reproduced and checked this issue with build 25.262-b10 and couldn’t find any solution yet. However I’ve found that there is newer version of openjdk 25.265-b01, available for example here https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u265-b01. The issue doesn’t show up on 25.265-b01 on my machine and ROR works without any problems. Can you check this version and possibly migrate to it? I don’t know what’s causing permission problem with 25.262-b10, but it doesn’t work even with this policy setting permission java.security.AllPermission;

Hi,
My OS is RedHat 7.8 and the last openjdk version proposed my yum is 262.b10.
265.b01 is not available.
I am not authorized to install a non yum officially available version.
And I won’t be able to keep forever on 262.b09 old java version.
This is thus blocking.
Do you know if openjdk 265.b01 will become officially availble through yum for redhat 7 ?
Thanks

@lucette here a prebuild for you:

https://readonlyrest-data.s3.amazonaws.com/build/1.23.0-pre6/readonlyrest-1.23.0-pre6_es6.8.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20200910/eu-west-1/s3/aws4_request&X-Amz-Date=20200910T133851Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=05eec8d279408dce9805287d4b74caccfbb290a78e42366adc43841523ae5bb1

Let us know if it works for you

@Vinod what ES version do you use?

@coutoPL we are on elastic search 7.8.1

so this one is for you:

https://readonlyrest-data.s3.amazonaws.com/build/1.23.0-pre6/readonlyrest-1.23.0-pre6_es7.8.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20200910/eu-west-1/s3/aws4_request&X-Amz-Date=20200910T141758Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a0d6deddec453d92957fd8cff478e0008c06748c0226a07c9f2fd597dee94fac

thanks will try and let you know.

Seeing this error now
[2020-09-11T06:06:11,821][INFO ][t.b.r.b.EsInitListener ] [localhost.domain] ES fully initiated. ReadonlyREST can continue …
[2020-09-11T06:06:11,998][INFO ][t.b.r.c.RorSsl$ ] [localhost.domain] Cannot find SSL configuration in elasticsearch.yml, trying: /etc/elasticsearch/bbdevqa/readonlyrest.yml
[2020-09-11T06:06:12,034][INFO ][t.b.r.b.Ror ] [localhost.domain] [CLUSTERWIDE SETTINGS] Loading ReadonlyREST settings from index …
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] java.security.AccessControlException: access denied (“java.lang.RuntimePermission” “modifyThreadGroup”)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at java.security.AccessController.checkPermission(AccessController.java:886)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at org.elasticsearch.secure_sm.SecureSM.checkThreadGroupAccess(SecureSM.java:192)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:147)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.Thread.init(Thread.java:394)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.Thread.init(Thread.java:349)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.Thread.(Thread.java:464)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at monix.execution.schedulers.ThreadFactoryBuilder$$anon$1.newThread(ThreadFactoryBuilder.scala:34)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.execution.internal.ScheduledExecutors$.scheduleOnce(ScheduledExecutors.scala:34)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.execution.schedulers.AsyncScheduler.scheduleOnce(AsyncScheduler.scala:44)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskSleep$Register.apply(TaskSleep.scala:49)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskSleep$Register.apply(TaskSleep.scala:39)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRestartCallback.start(TaskRestartCallback.scala:58)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRunLoop$.executeAsyncTask(TaskRunLoop.scala:592)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:120)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at monix.execution.Callback.apply(Callback.scala:116)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1(TaskFromFuture.scala:112)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1$adapted(TaskFromFuture.scala:111)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)

Hi, I tried to add modifyThreadGroup permission to ROR plugin, but it haven’t fixed anything. I will dig it further, but I have found workaround if you are loading your config only from file. Please add this line readonlyrest.force_load_from_file: true to your elasticsearch.yml. Let me know if it worked.