I have to check it …
@lucette great analysis, thanks for reporting back about this. It’s very weird indeed and worth investigating.
I saw that you released a new 1.22 ROR version but I don’t see hints of a fix for the startup problem, did you manage to fix it ?
Thanks
we are waiting on this fix, please let us know when this will be avaialble!
Thanks
Hi, I have reproduced and checked this issue with build 25.262-b10 and couldn’t find any solution yet. However I’ve found that there is newer version of openjdk 25.265-b01, available for example here Release jdk8u265-b01 · AdoptOpenJDK/openjdk8-binaries · GitHub. The issue doesn’t show up on 25.265-b01 on my machine and ROR works without any problems. Can you check this version and possibly migrate to it? I don’t know what’s causing permission problem with 25.262-b10, but it doesn’t work even with this policy setting permission java.security.AllPermission;
Hi,
My OS is RedHat 7.8 and the last openjdk version proposed my yum is 262.b10.
265.b01 is not available.
I am not authorized to install a non yum officially available version.
And I won’t be able to keep forever on 262.b09 old java version.
This is thus blocking.
Do you know if openjdk 265.b01 will become officially availble through yum for redhat 7 ?
Thanks
thanks will try and let you know.
Seeing this error now
[2020-09-11T06:06:11,821][INFO ][t.b.r.b.EsInitListener ] [localhost.domain] ES fully initiated. ReadonlyREST can continue …
[2020-09-11T06:06:11,998][INFO ][t.b.r.c.RorSsl$ ] [localhost.domain] Cannot find SSL configuration in elasticsearch.yml, trying: /etc/elasticsearch/bbdevqa/readonlyrest.yml
[2020-09-11T06:06:12,034][INFO ][t.b.r.b.Ror ] [localhost.domain] [CLUSTERWIDE SETTINGS] Loading ReadonlyREST settings from index …
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] java.security.AccessControlException: access denied (“java.lang.RuntimePermission” “modifyThreadGroup”)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at java.security.AccessController.checkPermission(AccessController.java:886)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[2020-09-11T06:06:12,048][WARN ][stderr ] [localhost.domain] at org.elasticsearch.secure_sm.SecureSM.checkThreadGroupAccess(SecureSM.java:192)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:147)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.Thread.init(Thread.java:394)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.Thread.init(Thread.java:349)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.lang.Thread.(Thread.java:464)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at monix.execution.schedulers.ThreadFactoryBuilder$$anon$1.newThread(ThreadFactoryBuilder.scala:34)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
[2020-09-11T06:06:12,049][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.execution.internal.ScheduledExecutors$.scheduleOnce(ScheduledExecutors.scala:34)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.execution.schedulers.AsyncScheduler.scheduleOnce(AsyncScheduler.scala:44)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskSleep$Register.apply(TaskSleep.scala:49)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskSleep$Register.apply(TaskSleep.scala:39)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRestartCallback.start(TaskRestartCallback.scala:58)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRunLoop$.executeAsyncTask(TaskRunLoop.scala:592)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:120)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
[2020-09-11T06:06:12,050][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at monix.execution.Callback.apply(Callback.scala:116)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1(TaskFromFuture.scala:112)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at monix.eval.internal.TaskFromFuture$.$anonfun$startSimple$1$adapted(TaskFromFuture.scala:111)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[2020-09-11T06:06:12,051][WARN ][stderr ] [localhost.domain] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
Hi, I tried to add modifyThreadGroup permission to ROR plugin, but it haven’t fixed anything. I will dig it further, but I have found workaround if you are loading your config only from file. Please add this line readonlyrest.force_load_from_file: true to your elasticsearch.yml. Let me know if it worked.
Hi,
Official 1.23.0 works like a charm.
Thank you very much !
2 Likes
Hi,
I’m replying on behalf of Vinod, we tried readonlyrest-1.23.0-pre9_es7.8.1.zip, now i’m not getting any errors in the logs while started ES, but while logging in the UI getting the below error.
Could not login
[exception] java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@4da020c4 rejected from java.util.concurrent.ThreadPoolExecutor@3260667b[Running, pool size = 4, active threads = 4, queued tasks = 0, completed tasks = 1] :: {“path”:"/_readonlyrest/metadata/current_user",“query”:{},“statusCode”:500,“response”:"{“error”:{“root_cause”:[{“type”:“exception”,“reason”:“java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@4da020c4 rejected from java.util.concurrent.ThreadPoolExecutor@3260667b[Running, pool size = 4, active threads = 4, queued tasks = 0, completed tasks = 1]”}],“type”:“exception”,“reason”:“java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@4da020c4 rejected from java.util.concurrent.ThreadPoolExecutor@3260667b[Running, pool size = 4, active threads = 4, queued tasks = 0, completed tasks = 1]”,“caused_by”:{“type”:“rejected_execution_exception”,“reason”:“Task monix.execution.internal.InterceptRunnable@4da020c4 rejected from java.util.concurrent.ThreadPoolExecutor@3260667b[Running, pool size = 4, active threads = 4, queued tasks = 0, completed tasks = 1]”}},“status”:500}"}
If I revert the java version back to 1.8.0_252 it will work
Even I tried the official 1.24.0 version looks like that is also throwing error
I see this in the logs, log says invalid credentials but if I revert java to 252 version then the same credential is working
[2020-10-22T17:51:33,877][ERROR][t.b.r.a.b.d.l.i.UnboundidLdapAuthenticationService] [xxxxxxxxxxxxx.com] LDAP authenticate operation failed - cause [invalid credentials]
[2020-10-22T17:51:33,873][WARN ][r.suppressed ] [xxxxxxxxxxxxx.com] path: /_readonlyrest/metadata/current_user, params: {}
java.lang.Exception: java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@1ac9ca59 rejected from java.util.concurrent.ThreadPoolExecutor@72e6973[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 11]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$handleRequest$1(IndexLevelActionFilter.scala:129) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$handleRequest$1$adapted(IndexLevelActionFilter.scala:127) ~[?:?]
at monix.execution.Callback$$anon$2.tryApply(Callback.scala:296) ~[?:?]
at monix.execution.Callback$$anon$2.apply(Callback.scala:289) ~[?:?]
at monix.execution.Callback$$anon$2.onError(Callback.scala:286) ~[?:?]
Seeing similar kind of error in kibana.log
{“type”:“log”,“@timestamp”:“2020-10-22T17:57:36Z”,“tags”:[“info”,“readonlyrest_kbn:extractIdentity”],“pid”:10265,“message”:“asked for group undefined”}
{“type”:“log”,“@timestamp”:“2020-10-22T17:57:36Z”,“tags”:[“error”,“readonlyrest_kbn:enrichFromEs”],“pid”:10265,“message”:“test authenticated call to ES failed: error[exception] java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@47eec2ca rejected from java.util.concurrent.ThreadPoolExecutor@5c4be121[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 23] :: {"path":"/_readonlyrest/metadata/current_user","query":{},"statusCode":500,"response":"{\"error\":{\"root_cause\":[{\"type\":\"exception\",\"reason\":\"java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@47eec2ca rejected from java.util.concurrent.ThreadPoolExecutor@5c4be121[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 23]\"}],\"type\":\"exception\",\"reason\":\"java.util.concurrent.RejectedExecutionException: Task monix.execution.internal.InterceptRunnable@47eec2ca rejected from java.util.concurrent.ThreadPoolExecutor@5c4be121[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 23]\",\"caused_by\":{\"type\":\"rejected_execution_exception\",\"reason\":\"Task monix.execution.internal.InterceptRunnable@47eec2ca rejected from java.util.concurrent.ThreadPoolExecutor@5c4be121[Running, pool size = 4, active threads = 2, queued tasks = 0, completed tasks = 23]\"}},\"status\":500}"} - for GET /_readonlyrest/metadata/current_user”}
{“type”:“log”,“@timestamp”:“2020-10-22T17:57:36Z”,“tags”:[“info”,“readonlyrest_kbn:loginHandler”],“pid”:10265,“message”:“identity error: undefined”}
could you confirm the java version it doesn’t work with, please?
In this version it doesn’t work openjdk version “1.8.0_262”