Logstash cannot retrieve license information from server

Nadine · July 20, 2022, 1:45pm

Hello!

@sscarduzio
There is more than a week that we are having issue with our ELK 7.15.1. Before that, everything were working well since months.
Here is the problem: when logstash is stopped, the elasticsearch cluster is healthy, kibana is accessible. But once logstash is started, here is its outpout:

[2022-07-20T09:03:41,867][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>50, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>100, "pipeline.sources"=>["/etc/logstash/pipeline.global.conf"], :thread=>"#<Thread:0x71385f3a@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2022-07-20T09:03:43,539][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.67}
[2022-07-20T09:03:43,669][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-07-20T09:03:43,776][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,783][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-07-20T09:03:43,790][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-07-20T09:03:43,797][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,799][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,825][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,826][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,836][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,837][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:44,088][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2022-07-20T09:04:05,392][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://user:xxxxxx@master1.gestion.mrqgest:9200/_xpack'"}
[2022-07-20T09:04:35,390][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://user:xxxxxx@master1.gestion.mrqgest:9200/_xpack'"}
[2022-07-20T09:04:44,198][WARN ][logstash.outputs.elasticsearch][main] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://user:xxxxxx@master2.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://user:xxxxxx@master2.gestion.mrqgest:9200/, :error_message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master2.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2022-07-20T09:04:44,201][ERROR][logstash.outputs.elasticsearch][main] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master2.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2022-07-20T09:04:44,201][WARN ][logstash.outputs.elasticsearch][main] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://user:xxxxxx@master1.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://user:xxxxxx@master1.gestion.mrqgest:9200/, :error_message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master1.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2022-07-20T09:04:44,206][ERROR][logstash.outputs.elasticsearch][main] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master1.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2022-07-20T09:04:45,686][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://user:xxxxxx@master1.gestion.mrqgest:9200/"}

Then, here is the elasticsearch output when logstash is started

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:56) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:36) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) [elasticsearch-7.15.1.jar:7.15.1]
        at tech.beshu.ror.es.handler.RegularRequestHandler.proceed(RegularRequestHandler.scala:224) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.onAllow(RegularRequestHandler.scala:99) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$commitResult$1(RegularRequestHandler.scala:70) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [scala-library-2.12.10.jar:?]
        at scala.util.Try$.apply(Try.scala:213) [scala-library-2.12.10.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.commitResult(RegularRequestHandler.scala:68) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$2(RegularRequestHandler.scala:60) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$2$adapted(RegularRequestHandler.scala:59) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.utils.ScalaOps$AutoCloseableOps$.bracket$extension(ScalaOps.scala:168) [core-1.36.0.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$1(RegularRequestHandler.scala:59) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$1$adapted(RegularRequestHandler.scala:58) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at monix.eval.Task$Map.apply(Task.scala:4514) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task$Map.apply(Task.scala:4510) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.internal.TaskRunLoop$.startLight(TaskRunLoop.scala:331) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task.runAsyncOptF(Task.scala:811) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task.runAsyncOpt(Task.scala:709) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task.runAsync(Task.scala:659) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.scala:136) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.proceedByRorEngine(IndexLevelActionFilter.scala:122) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$apply$1(IndexLevelActionFilter.scala:107) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [scala-library-2.12.10.jar:?]
        at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:25) [core-1.36.0.jar:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) [?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:24) [core-1.36.0.jar:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.scala:93) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:90) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:70) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$doPrepareRequest$0(RestMonitoringBulkAction.java:102) [x-pack-monitoring-7.15.1.jar:7.15.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:106) [elasticsearch-7.15.1.jar:7.15.1]
        at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$getRestHandlerWrapper$2(ReadonlyRestPlugin.scala:238) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:275) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:343) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:196) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:348) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:413) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:330) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:17) [transport-netty4-client-7.15.1.jar:7.15.1]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:47) [transport-netty4-client-7.15.1.jar:7.15.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.69.Final.jar:4.1.69.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]

@sscarduzio I noticed that somewhere you talked about cluster:monitor/xpack/license/get but I don’t know how it should be set.

Any idea about this issue?

sscarduzio · July 20, 2022, 2:47pm

Hi @Nadine,

It would be useful if you show the YAML of the ROR ACL, and the logstash configuration.

Nadine · July 20, 2022, 3:52pm

Here is the logstash pipeline config:

input { beats { port => 5044 }

 }
output {
        if [client] {
                if [client] == "iis" {
                                        elasticsearch {
                                                hosts => ["https://master1:9200", "https://master2:9200"]
                                                ssl_certificate_verification => false
                                                ssl => true
                                                index=> "rq-sdf-cit-%{[indexname]}-%{+YYYY.ww}"
                                                cacert => "/etc/pki/ca-trust/source/anchors/ca-elasticsearch.pem"
                                                 user=> "user"
                                                 password=> "pass"
                                                                                                }
                        }
                }
				}

Readonlyrest.yml:

readonlyrest:
 audit_collector: true
 users:
 - username: xxxx
   auth_key: xxxx:xxxx
   groups:
   - admin_tenant_template
   - admin_tenant_srv
   - admin_tenant_publique
   - admin_tenant_expl

 access_control_rules:

# Tenant template
 - name: "Tenant super admin template"
   type: allow
   groups: ["admin_tenant_template"]
   verbosity: error
   kibana_access: admin
   kibana_index: ".kibana_template"
 - name: "Actions super admin template"
   type: allow
   actions: ["*"]
   groups: ["admin_tenant_template"]
   verbosity: error


# Tenant publique
 - name: "Tenant super admin publique"
   type: allow
   groups: ["admin_tenant_publique"]
   verbosity: error
   kibana_access: admin
   kibana_index: ".kibana_publique"
 - name: "Actions super admin publique"
   type: allow
   actions: ["*"]
   groups: ["admin_tenant_publique"]
   verbosity: error


 - name: Utilisateurs lecture publique
   type: allow
   ldap_auth:
     name: "ldap_elasticstack"
     groups:
     - "gOutil-Surveillance-Application-ElasticStack-L"
     - "gRQ-DeveloppeursEE"
     - "gSCSM-Analystes"
     - "gRQ-Developpeurs-Impact"
   kibana_index: ".kibana_publique"
   kibana_access: ro
   indices:
     - "rq-iis*"
     - "rq-winlog*"
     - "rq-enrich-iis"
   kibana_hide_apps: ["Analytics|Canvas","Analytics|Maps","Analytics|Visualize Library", "Security", "Enterprise Search","Observability","Management", "ROR Manage Kibana","ROR Security Settings"]


# Tenant expl
 - name: "Tenant super admin expl"
   type: allow
   groups: ["admin_tenant_expl"]
   verbosity: error
   kibana_access: admin
   kibana_index: ".kibana_expl"
 - name: "Actions super admin expl"
   type: allow
   actions: ["*"]
   groups: ["admin_tenant_expl"]
   verbosity: error


 - name: Utilisateurs lecture expl
   type: allow
   ldap_auth:
     name: "ldap_elasticstack"
     groups:
     - "gOutil-Surveillance-Application-APM-Expl-Admin"
   kibana_index: ".kibana_expl"
   kibana_access: ro
   indices:
     - "rq-*"
   kibana_hide_apps: ["Analytics|Canvas","Analytics|Maps","Analytics|Visualize Library", "Security", "Enterprise Search","Observability","Management", "ROR Manage Kibana","ROR Security Settings"]


 - name: "::Kibana-Lecture::"
   type: allow
   ldap_authentication: "ldap_elasticstack"
   indices: ["rq-ial*", "rq-ia*","rq-strurl*","rq-syslog*", "rq-ctm*", "rq-sdf*","rq-iis*", "rq-winlog*","rq-redis-*","rq-puppet*","rq-log-puppet*","rq-m2*",".kibana*","rq-wso2-unitaire*","rq-wso2-formation*", "rq-wso2-acceptation*","rq-wso2-simulation*","rq-autocell*","rq-classeformation*","rq-outilsuivi*","rq-msys*","rq-mx*"]
   kibana_index: ".kibana_publique"
   kibana_access: ro
   kibana_hide_apps: ["Analytics|Canvas","Analytics|Maps","Analytics|Visualize Library", "Security", "Enterprise Search","Observability","Management", "ROR Manage Kibana","ROR Security Settings"]

 ldaps:

 - name: ldap_elasticstack
   host: "palp-xxxxx"
   port: 636
   bind_dn: "CN=GestionCas ElasticSearch Auth Prd,OU=Gestion Cas,OU=MRQ,OU=Utilisateurs,DC=PROD,DC=MRQ"
   bind_password: "xxxxxx"
   search_user_base_DN: "OU=Utilisateurs,DC=PROD,DC=MRQ"
   search_groups_base_DN: "OU=MRQ,OU=Utilisateurs,DC=PROD,DC=MRQ"
   connection_pool_size: 10
   connection_timeout_in_sec: 10
   request_timeout_in_sec: 10
   cache_ttl_in_sec: 60
   user_id_attribute: "sAMAccountName"
   unique_member_attribute: "Member"
   ssl_enabled: true
   ssl_trust_all_certs: true
   group_search_filter: "(objectClass=group)"
   group_name_attribute: "name"

 ssl:
   keystore_file: "elasticstack.jks"
   keystore_pass: Surv31ll@nc3RQ.
   key_pass: xxxx

logstash.yml:

---
path.data: "/var/lib/logstash/data"
path.config: "/etc/logstash/pipeline.global.conf"
path.logs: "/var/log/logstash"
node.name: xxxx
config.reload.automatic: true
queue.type: persisted
queue.max_bytes: 1024mb
pipeline.batch.size: 50
xpack.monitoring.enabled: true

sscarduzio · July 21, 2022, 10:57am

Thank you @Nadine!

I see Logstash is configured to use these HTTP Basic credentials:

However, I don’t find in the ACL any reference to them. Does this mean you expect these to be LDAP based credentials?

Anyways, What I advise is to have a dedicated, unrestricted ACL block for Logstash. Something along the lines of:

 - name: "::LOGSTASH::"
   auth_key: "user:pass"
   verbosity: error

And by the way, I would recommend using the hashed versions of auth_key rule, like auth_key_sha512

Nadine · July 29, 2022, 1:09pm

Hi @sscarduzio

Thanks for your advices.

We fixed it by deleting some indices. Really weird, but it is okay now.

Thanks,