Unable to access monitoring in Kibana

1

I have a 8 node cluster and hosted ROR separately on other with ssl enable everything was working fine except unable to access kibana

34%20PM
Screen Shot 2019-01-24 at 8.47.34 PM.png2284×450 42.3 KB

I’m using free version of ReadonlyREST(readonlyrest-1.16.31_es6.4.2) and Elastic Stack version 6.4.2.
Config Files below

readonlyrest.yml
readonlyrest:

    ssl:
      enable: true
      keystore_file: "keystore.jks"
      keystore_pass: changeit
      key_pass: changeit
      key_alias: selfsigned

    audit_collector: true

    access_control_rules:

    # MACHINES ##################
    - name: "::LOGSTASH::"
      auth_key: logstash:logstash
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["logstash-*"]

    - name: "::KIBANA-SRV::"
      auth_key: kibana:kibana
      verbosity: error

    # GROUPS (LOCAL MULTITENANCY) #####

    - name: "::ADMIN_GRP::"
      groups: ["ROR (admin)"]
      kibana_index: .kibana

    - name: "::Infosec::"
      groups: ["Infosec"]
      kibana_access: rw
      kibana_hide_apps: ["readonlyrest_kbn", "timelion"]
      kibana_index: ".kibana_infosec"

     # GROUPS-Admins (LDAP) ##############
    - name: "LDAP group1 users"
      ldap_authentication: "ldap1"
      ldap_authorization:
        name: "ldap1"
        groups: ["testgroup1"]
      indices: [".kibana*","test*"]

    # GROUPS-Users (LDAP) ##############
    - name: "LDAP group2 users"
      ldap_authentication: "ldap1"
      ldap_auth:
        name: "ldap1"
        groups: ["testgroup2"]
      indices: [".kibana*","test*"]
      kibana_access: ro

    # USERS TO GROUPS (LOCAL MULTITENANCY) #######
    users:
    - username: admin
      auth_key: admin:admin
      groups: ["ROR (admin)", "Infosec"]

    ldaps:
    ######### LDAP1 SERVER CONFIGURATION ########################
    # group1: cartman, bong
    # group2: morgan
    # group3: morgan, cartman, bong
    #############################################################
    - name: ldap1
      rest of ldap config.....

elasticsearch.yml

xpack.security.enabled: false
http.type: ssl_netty4

I tried refering topics related to unable to access kibana monitoring tab, but unable to figure out where i’m going wrong. Please help.

2

Hello @rohith1570, this should be fixed in the latest ror. We are now at version 1.16.33

3

@sscarduzio Thanks for the response. I upgraded ROR from 1.16.31 to 1.16.33, still i getting monitoring tab Access Denied. I’m using free version readonlyrest-1.16.33_es6.4.2.zip

Using Same config as posted above.

kibana.yml
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false

4

@sscarduzio
Elasticsearch debug log

    FORBIDDEN by default req={ ID:673310904-109567359#48569, TYP:SearchRequest, CGR:N/A, USR:[no
basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:192.168.192.11,
DA:192.168.192.11, IDX:*:.monitoring-es-6-*,.monitoring-es-6-*,*:.monitoring-es-2-*,.monitoring-es-2-*,
MET:POST, PTH:/*%3A.monitoring-es-2-*%2C*%3A.monitoring-es-6-*%2C.monitoring-es-2-*%2C.monitoring
es-6-*/_search
size=10000&ignore_unavailable=true&filter_path=hits.hits._index%2Chits.hits._source.cluster_uuid%2Chits.hi
s._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license.status%2Chits.hits._source
license.type%2Chits.hits._source.license.issue_date%2Chits.hits._source.license.expiry_date%2Chits.hits._s
urce.license.expiry_date_in_millis%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state, CNT
{"query":{"bool":{"filter":[{"bool":{"should":[{"term":{"_type":"cluster_stats"}},{"term":{"type":"cluster_stats"}}]}}
{"range":{"timestamp":{"format":"epoch_millis","gte":1548426840965,"lte":1548430440965}}}]}},"collapse"
{"field":"cluster_uuid"},"sort":{"timestamp":{"order":"desc"}}}, HDR:{Connection=keep-alive, Content
Length=286, content-type=application/json, Host=192.168.192.11:9200}, HIS:[::LOGSTASH::->[auth_key
>false]], [::KIBANA-SRV::->[auth_key->false]], [::Monitoring::->[auth_key->false]], [::ADMIN_GRP::->[groups
>false]], [Testgroup access->[groups->false]], [LDAP Engineering users->[ldap_authentication->false]] }
       
[2019-01-25T15:34:00,984][DEBUG][r.suppressed             ] path: /*%3A.monitoring-es-2
*%2C*%3A.monitoring-es-6-*%2C.monitoring-es-2-*%2C.monitoring-es-6-*/_search, params: {size=10000,
ignore_unavailable=true, index=*:.monitoring-es-2-*,*:.monitoring-es-6-*,.monitoring-es-2-*,.monitoring-es-6-*,
filter_path=hits.hits._index,hits.hits._source.cluster_uuid,hits.hits._source.cluster_name,hits.hits._source.vers
on,hits.hits._source.license.status,hits.hits._source.license.type,hits.hits._source.license.issue_date,hits.hits.
source.license.expiry_date,hits.hits._source.license.expiry_date_in_millis,hits.hits._source.cluster_stats,hits.hi
s._source.cluster_state}
        tech.beshu.ror.es.IndexLevelActionFilter$1$1: forbidden
        	at tech.beshu.ror.es.IndexLevelActionFilter$1.onForbidden(IndexLevelActionFilter.java:163) ~[?:?]
        	at tech.beshu.ror.acl.ACL.lambda$check$4(ACL.java:208) ~[?:?]
        	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_191]
        	at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_191]
        	at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_191]
        	at tech.beshu.ror.acl.ACL.check(ACL.java:203) ~[?:?]
        	at tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:158) ~[?:?]
        	at tech.beshu.ror.es.IndexLevelActionFilter.lambda$apply$1(IndexLevelActionFilter.java:134) ~[?:?]
        	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
        	at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:130) ~[?:?]
        	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:93) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97) ~[elasticsearch-6.4.2.jar:6.4.2]
        	at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$5(ReadonlyRestPlugin.java:197) ~[?:?]
        	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:239) [elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:335) [elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:173) [elasticsearch-6.4.2.jar:6.4.2]
        	at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:538) [transport-netty4-client-6.4.2.jar:6.4.2]
        	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137) [transport-netty4-client-6.4.2.jar:6.4.2]
        	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.4.2.jar:6.4.2]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
5

Can you make sure you also have the latest Kibana plugin as well, not only the elasticsearch one.
Also make sure that your users have access to the “monitoring*” indices in the ACL.

6

I don’t have ROR Kibana Plugin installed. I believe there is no free version of kibana plugin right?

Updated ACL with “monitoring*” indices. Still seeing access denied.

7

Ok mystery solved.
Our plugin contains a workaround for a bug of Kibana Monitoring. This bug has the effect that the user credentials are not forwarded to Elasticsearch when you use the Monitoring app.

Unfortunately we don’t have a free version of the Kibana plugin, just the 14 days free trials. Would you be interested? I can short cut the process and hand you over one.