Hello,
can anyone help me to explain this behaving:
Behaving
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-07-20 16:27:38 CEST; 1min 47s ago
Docs: https://www.elastic.co
Main PID: 14892 (java)
CGroup: /system.slice/elasticsearch.service
├─14892 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Dname=elasticsearch -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negat...
└─15093 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
You can see that Elasticsearch uses java with location /usr/share/elasticsearch/jdk/bin/java
On system I have installed openjdk 1.8.0 and openjdk 11.
If I have set 1.8.0 ES, install ROR plugins, Elasticsearch boots normaly.
If I set system java to openjdk-11 and restart Elasticsearch, Elasticsearch boots normally.
If I set system java to openjdk-11, remove ROR plugins, Elasticsearch doesn’t boot because because SSL/TLS (internode SSL) between nodes could not be inicialized.
Do you have any tips?
Used software
I’m using Elasticstack 7.8.0 and ROR ES, ROR KIBANA of last released version.
[2020-07-20T15:37:11,390][WARN ][o.e.t.TcpTransport ] [es-master-as2.cdis.cz] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,3,0), [Netty4TcpChannel{localAddress=/10.28.12.215:9300, remoteAddress=/10.28.12.221:53100}], closing connection
[2020-07-20T15:37:11,400][WARN ][o.e.t.TcpTransport ] [es-master-as2.cdis.cz] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,3,0), [Netty4TcpChannel{localAddress=/10.28.12.215:9300, remoteAddress=/10.28.12.221:53098}], closing connection
[2020-07-20T15:37:11,404][WARN ][o.e.t.TcpTransport ] [es-master-as2.cdis.cz] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,3,0), [Netty4TcpChannel{localAddress=/10.28.12.215:9300, remoteAddress=/10.28.12.221:53104}], closing connection
[2020-07-20T15:37:11,402][INFO ][o.e.c.c.JoinHelper ] [es-master-as2.cdis.cz] failed to join {es-master-as1.cdis.cz}{-kGzPTImR2iKRBgaENAw3Q}{wBVEdQ6kQGu7u1cSNyER2Q}{10.28.12.221}{10.28.12.221:9300}{m}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={es-master-as2.cdis.cz}{PuuDyOjET2CBJMyR4k40hg}{CDBX4hvDQzCWO7313jSzow}{10.28.12.215}{10.28.12.215:9300}{m}{xpack.installed=true, transform.node=false}, minimumTerm=594, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [es-master-as1.cdis.cz][10.28.12.221:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [es-master-as2.cdis.cz][10.28.12.215:9300] general node connection failure
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:956) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleLocalException(TransportHandshaker.java:150) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener.lambda$wrap$0(ActionListener.java:132) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:196) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:39) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.complete(CompletableContext.java:61) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:61) ~[transport-netty4-client-7.8.0.jar:7.8.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1158) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:760) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:736) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.transport.TransportException: handshake failed because connection reset
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener.lambda$wrap$0(ActionListener.java:132) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:196) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:39) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
at org.elasticsearch.common.concurrent.CompletableContext.complete(CompletableContext.java:61) ~[elasticsearch-core-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:61) ~[transport-netty4-client-7.8.0.jar:7.8.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1158) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:760) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:736) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:832) ~[?:?]
Hello,
I found out where problem was. Problem is depends on which java is used for creating keystore.
With openjdk 1.8.0 it works well, but with openjdk 11 doesn’t.
With openjdk 11 ROR reports this error:
[2020-07-24T13:14:12,653][ERROR][t.b.r.u.SSLCertParser$ ] [log-management-test-as1] ROR SSL: Failed to load SSL certs and keys from JKS Keystore! UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:457) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:90) ~[?:?]
at java.security.KeyStore.getKey(KeyStore.java:1050) ~[?:?]
at tech.beshu.ror.utils.SSLCertParser$.extractPrivateKey(SSLCertParser.scala:110) ~[core-1.21.0-pre6.jar:?]
at tech.beshu.ror.utils.SSLCertParser$.$anonfun$loadKeyAndCertificate$2(SSLCertParser.scala:78) ~[core-1.21.0-pre6.jar:?]
at cats.effect.Resource.$anonfun$use$1(Resource.scala:121) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOBracket$BracketStart.liftedTree1$1(IOBracket.scala:79) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOBracket$BracketStart.run(IOBracket.scala:79) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:93) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:93) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.9.jar:?]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) ~[scala-library-2.12.9.jar:?]
at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:93) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOBracket$BracketStart.apply(IOBracket.scala:72) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOBracket$BracketStart.apply(IOBracket.scala:52) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:136) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IORunLoop$.start(IORunLoop.scala:34) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOBracket$.$anonfun$apply$1(IOBracket.scala:44) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOBracket$.$anonfun$apply$1$adapted(IOBracket.scala:34) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IORunLoop$RestartCallback.start(IORunLoop.scala:341) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:119) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IORunLoop$.start(IORunLoop.scala:34) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.IO.unsafeRunAsync(IO.scala:257) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.internals.IOPlatform$.unsafeResync(IOPlatform.scala:38) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.IO.unsafeRunTimed(IO.scala:324) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at cats.effect.IO.unsafeRunSync(IO.scala:239) ~[cats-effect_2.12-2.0.0.jar:2.0.0]
at tech.beshu.ror.utils.SSLCertParser$.$anonfun$run$1(SSLCertParser.scala:41) ~[core-1.21.0-pre6.jar:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.9.jar:?]
at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.9.jar:?]
at tech.beshu.ror.utils.SSLCertParser$.run(SSLCertParser.scala:42) [core-1.21.0-pre6.jar:?]
at tech.beshu.ror.es.ssl.SSLNetty4HttpServerTransport$SSLHandler.$anonfun$new$1(SSLNetty4HttpServerTransport.scala:64) [readonlyrest-1.21.0-pre6_es7.8.0.jar:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [scala-library-2.12.9.jar:?]
at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:25) [core-1.21.0-pre6.jar:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:24) [core-1.21.0-pre6.jar:?]
at tech.beshu.ror.es.ssl.SSLNetty4HttpServerTransport$SSLHandler.<init>(SSLNetty4HttpServerTransport.scala:64) [readonlyrest-1.21.0-pre6_es7.8.0.jar:?]
at tech.beshu.ror.es.ssl.SSLNetty4HttpServerTransport.configureServerChannelHandler(SSLNetty4HttpServerTransport.scala:56) [readonlyrest-1.21.0-pre6_es7.8.0.jar:?]
at tech.beshu.ror.es.ssl.SSLNetty4HttpServerTransport.configureServerChannelHandler(SSLNetty4HttpServerTransport.scala:38) [readonlyrest-1.21.0-pre6_es7.8.0.jar:?]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.doStart(Netty4HttpServerTransport.java:195) [transport-netty4-client-7.8.0.jar:7.8.0]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.node.Node.start(Node.java:812) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.8.0.jar:7.8.0]
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
at com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:977) ~[?:?]
at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1058) ~[?:?]
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:855) ~[?:?]
at com.sun.crypto.provider.PKCS12PBECipherCore.implDoFinal(PKCS12PBECipherCore.java:408) ~[?:?]
at com.sun.crypto.provider.PKCS12PBECipherCore$PBEWithSHA1AndDESede.engineDoFinal(PKCS12PBECipherCore.java:440) ~[?:?]
at javax.crypto.Cipher.doFinal(Cipher.java:2207) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:401) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:291) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:395) ~[?:?]
... 53 more