Updating elasticsearch 8.2.3 with 1.41.0_es8.2.3
to 8.6.2 with 1.47.0_es8.6.2
I get this error (no changes has been made to the yaml files)
[ERROR][t.b.r.a.b.Block ] [ommited] ::NETWORK::: ldap_auth rule matching got an error Rejected because the CircuitBreaker is in the Open state, attempting to close in 72 millis
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the Open state, attempting to close in 72 millis
at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$2(CircuitBreaker.scala:413) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:170) ~[?:?]
at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101) ~[?:?]
at monix.eval.internal.TaskRestartCallback$$anon$1.run(TaskRestartCallback.scala:118) ~[?:?]
at monix.execution.internal.Trampoline.monix$execution$internal$Trampoline$$immediateLoop(Trampoline.scala:66) ~[?:?]
at monix.execution.internal.Trampoline.startLoop(Trampoline.scala:32) ~[?:?]
at monix.execution.schedulers.TrampolineExecutionContext$JVMOptimalTrampoline.startLoop(TrampolineExecutionContext.scala:143) ~[?:?]
at monix.execution.internal.Trampoline.execute(Trampoline.scala:40) ~[?:?]
at monix.execution.schedulers.TrampolineExecutionContext.execute(TrampolineExecutionContext.scala:64) ~[?:?]
at monix.execution.schedulers.BatchingScheduler.execute(BatchingScheduler.scala:50) ~[?:?]
at monix.execution.schedulers.BatchingScheduler.execute$(BatchingScheduler.scala:47) ~[?:?]
at monix.execution.schedulers.AsyncScheduler.execute(AsyncScheduler.scala:31) ~[?:?]
at monix.eval.internal.TaskRestartCallback.start(TaskRestartCallback.scala:56) ~[?:?]
at monix.eval.internal.TaskRunLoop$.executeAsyncTask(TaskRunLoop.scala:592) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:120) ~[?:?]
at monix.eval.internal.TaskRunLoop$.$anonfun$restartAsync$1(TaskRunLoop.scala:222) ~[?:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]
my ror setup (working before upgrading)
ssl:
enable: true
keystore_file: “file.jks”
[omitted]
allowed_protocols: [TLSv1.2]
[omitted]
ldaps:
[ERROR][t.b.r.a.b.Block ] [ommited] ::NETWORK::: ldap_auth rule matching got an error Rejected because the CircuitBreaker is in the Open state, attempting to close in 72 millis
means that the circuit breaker (CB) for LDAP connector has been opened (because of some repetitive failures) and now rejects the requests to not overwhelm the LDAP server. The next attempt to close the CB will be after 72 millis.
This is expected and correct behavior. And IMO it has nothing in common with the update because the CB mechanism was introduced in ROR 1.30.0. Please check our documentation too:
Ok. I have found there is a previous error before. As I said no changes has been made to configuration and SSL
[2023-03-13T07:57:47,702][ERROR][t.b.r.a.b.d.l.i.UnboundidLdapConnectionPoolProvider$] [ommited] LDAP binding exception
com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to connect to server :636: IOException(LDAPException(resultCode=91 (connect error), errorMessage=‘An error occurred while attempting to establish a connection to server /:636: SSLHandshakeException(Received fatal alert: handshake_failure), ldapSDKVersion=6.0.6, revision=b8c6c463def55758ed8ec0d914c84268c944251c’))
at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:943) ~[?:?]
at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:829) ~[?:?]
at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:767) ~[?:?]
at com.unboundid.ldap.sdk.LDAPConnection.(LDAPConnection.java:587) ~[?:?]
at com.unboundid.ldap.sdk.SingleServerSet.getConnection(SingleServerSet.java:329) ~[?:?]
at com.unboundid.ldap.sdk.SingleServerSet.getConnection(SingleServerSet.java:315) ~[?:?]
at tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.$anonfun$testBindingForAllHosts$1(UnboundidLdapConnectionPoolProvider.scala:82) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81) ~[?:?]
at monix.eval.Task$.unsafeStartNow(Task.scala:4582) ~[?:?]
at monix.eval.internal.TaskBracket$BaseStart.apply(TaskBracket.scala:174) ~[?:?]
at monix.eval.internal.TaskBracket$BaseStart.apply(TaskBracket.scala:164) ~[?:?]
at monix.eval.internal.TaskRestartCallback.run(TaskRestartCallback.scala:65) ~[?:?]
at monix.execution.internal.Trampoline.monix$execution$internal$Trampoline$$immediateLoop(Trampoline.scala:66) ~[?:?]
at monix.execution.internal.Trampoline.startLoop(Trampoline.scala:32) ~[?:?]
at monix.execution.schedulers.TrampolineExecutionContext$JVMOptimalTrampoline.startLoop(TrampolineExecutionContext.scala:143) ~[?:?]
at monix.execution.internal.Trampoline.execute(Trampoline.scala:40) ~[?:?]
at monix.execution.schedulers.TrampolineExecutionContext.execute(TrampolineExecutionContext.scala:64) ~[?:?]
at monix.execution.schedulers.BatchingScheduler.execute(BatchingScheduler.scala:50) ~[?:?]
at monix.execution.schedulers.BatchingScheduler.execute$(BatchingScheduler.scala:47) ~[?:?]
at monix.execution.schedulers.ExecutorScheduler.execute(ExecutorScheduler.scala:35) ~[?:?]
at monix.eval.internal.TaskRestartCallback.start(TaskRestartCallback.scala:56) ~[?:?]
at monix.eval.internal.TaskRunLoop$.executeAsyncTask(TaskRunLoop.scala:592) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:120) ~[?:?]
at monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101) ~[?:?]
at monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74) ~[?:?]
at monix.eval.internal.TaskSleep$SleepRunnable.run(TaskSleep.scala:66) ~[?:?]
at monix.execution.internal.InterceptRunnable.run(InterceptRunnable.scala:27) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Mmm
Maybe the problem is related with the elasticsearch java release
The update in elasticsearch change from 18.0.1.1 to 19.0.2
I am trying with a custom one without success yet
More info
I have revert to my original release 8.2.3 with 1.41.0_es8.2.3
Also I have import my CA in the linux system and deleted the option ssl_trust_all_certs: true in ldaps
With the bundled java (18.0.1.1) provided by elasticsearch it fails with this “unable to find valid certification path to requested target”
With an rpm system installed release (java-17-openjdk-17.0.6.0), it works
ES_JAVA_HOME=/usr/lib/jvm/jre-17-openjdk
So that calls done by the plugin to the ldaps server are working with openjdk 17 but they fail with both included 18 (elastic 8.2.3)
Maybe i will try to update to latest release 8.6.2 with 1.47.0_es8.6.2 using openjdk 17
we will try to test LDAP with SSL with mentioned JDK (in our integration tests we take advantage of official ES images, so we use JDK provided in the images)
It is a custom CA
I am trying with this and the bundled jdk
keytool -importcert -keystore /usr/share/elasticsearch/jdk/lib/security/cacerts -file /etc/pki/ca-trust/source/anchors/mycustomca.crt -noprompt -alias ca1