Hosts_local question

I suspect you are not using the ROR Kibana plugin (PRO/Enterprise) and you are facing that old (but still valid) Kibana issue where some request don’t carry the Authorization header.

If this is the case, I provided explanation and a workaround in this thread:

It worked. Thanks @sscarduzio.

Hi @sscarduzio, is it possible to skip authentication screen for kibana RO access? Thanks

Not really, ROR PRO/Enterprise Kibana plugin always require an identity for the current user. The only way to skip the login is to have the (default) identity injected via x-forwarded-for header, or JWT. Read about this in the docs

Hi @sscarduzio,

We have seen client response time increased after installing ROR plugin in elastic search 6.2.1. It’s 3x increase. I installed it on all 3 nodes in our 3 node dev cluster. All of them are master eligible, data and client nodes. Generally what is the recommendation for plugin and ACL rules? Does it need to be only on master? Please help.

Thanks,
Sai

Hello @sairamvla!

  1. it’s not normal to have significant a performance hit with ROR. Are you calling external auth systems? How long is your ACL?

  2. You should install ROR as a stateless security “filter” only in the ES nodes that receive HTTP connections, which - unless your deployment is really simplistic - are not the same node which have data and are master eligible.

Thanks @sscarduzio for your response. No, I am not calling external auth systems. I configured 9 ACL rules on all 3 nodes…

Can we see your settings?

Here is yml:

readonlyrest:
    enable: true
    response_if_req_forbidden: Sorry, your request is forbidden.

    access_control_rules:

#    - name: "Accept all requests from localhost"
#      hosts_local: [172.21.32.159]
#      actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:admin/template/*"]
    - name: "::LOGSTASH::"
      auth_key: xxx:xxx
      actions: ["cluster:monitor/main","indices:ad  min/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["log_index-*"]
    - name: "::DOCSHOUND::"
      auth_key: xxx:xxx
      actions: ["cluster:monitor/health","cluster:monitor/main","cluster:monitor/state","cluster:monitor/stats","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["docshound*"]
    - name: "::ES READONLY::"
      auth_key: xxx:xxx
      actions: ["indices:data/read/*","indices:monitor/*","cluster:monitor/health","cluster:monitor/main","cluster:monitor/state","cluster:monitor/stats"]
      indices: ["log_index-*",".kibana"]
    - name: "::ES SNAPSHOTRESTORE::"
      auth_key: xxx:xxx
      actions: ["cluster:admin/repository/*","cluster:admin/snapshot/*"]
      indices: ["log_index-*","docshound*",".kibana"]
    - name: "::ES ADMIN::"
      auth_key: xxx:xxx
      actions: ["cluster:monitor/*","cluster:admin/*","indices:admin/*","indices:data/*","indices:monitor/*"]
      indices: ["log_index-*","docshound*",".kibana"]
    - name: "::KIBANA-SRV::"
      auth_key: kibana:kibana
      verbosity: error
    - name: "::KIBANA RW DEVELOPER::"
      auth_key: xxx:xxx
      kibana_access: rw
      kibana_index: ".kibana"
      indices: ["log_index-*","docshound*",".kibana"]
    - name: "::KIBANA RO DEVELOPER::"
      auth_key: xxx:xxx
      kibana_access: ro
      indices: ["log_index-*","docshound*",".kibana"]
    - name: "workaround"
      actions: [ "indices:data/read/field_caps*", "indices:data/read/msearch", "indices:data/read/search" ]

Is it always this slow, or does the slowness build up after it’s been running for a while?

After 1-2 hours, it starts slowing down and continuous since then as requests coming in

is the memory allocation growing during that time?

No, memory usage is under control @16%…

Hi @sscarduzio, any recommendations? thank you.

@sairamvla could you connect a profiler like JVisualVM? In the past this has been the most useful tool to detect bottlenecks.

  1. Install JVisualVM
  2. Connect it to the Elasticsearch JVM instance when it’s slow
  3. Fire up the CPU “Sampler” (not the profiler, as it’s super slow)
  4. Send some traffic if it’s not there already, for 20-30 seconds
  5. Save the result, and send it here or at info AT readonlyrest DOT com
  6. Also, take a thread dump (there is a button in JVisualVM) and send that across too.