No kibana index from headers / User's kibana index issue

Travis · October 2, 2018, 8:46pm

We keep having a strange situation occur with Kibana which we need help with.

The scenario:

A privileged user will login, authenticate with Active Directory and get
the .kibana index as expected. In comparison a non-privileged user will
login and get the .kibana-nonpriv index. We have this based on Active Directory
OUs and groups.

After a short period of time privileged users will, while working, in Kibana
seem to switch, rather default, to the .kibana-nonpriv index

In an attempt to resolve the issue the users when asked to logout and back in
again will not get their expected index(.kibana) only the default non-privileged
index(.kibana-nonpriv).

Our only remedy at this point is to restart Kibana.

We’ve been trying to trap the condition to provide additional details however the
only indication we’ve been able find is from Kibana’s logs:

Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":"try extract credentials from JSON"}
Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":"groupCurrent not found in cookie, that's ok."}
Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":"try extract credentials from JSON"}
Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":"groupCurrent not found in cookie, that's ok."}
Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":" received identity payload: {\"x-ror-current-group\":null,\"x-ror-username\":\"user5\",\"x-ror-kibana-hidden-apps\":[\"kibana:dev_tools\",\"kibana:management\",\"readonlyrest_kbn\"],\"x-ror-kibana_access\":\"rw\",\"x-ror-available-groups\":[]}"}
Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":"ON_IDENTITY no kibana index from headers, setting kibana index to default configured .kibana"}
Oct  2 16:46:02 kibana1 kibana: {"type":"response","@timestamp":"2018-10-02T16:46:02Z","tags":[],"pid":13854,"method":"post","statusCode":302,"req":{"url":"/login","method":"post","headers":{"host":"localhost:5601","connection":"close","content-length":"39","origin":"https://kibana1.example.com","kbn-xsrf":"5.6.1","kbn-version":"5.6.1","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36","content-type":"application/x-www-form-urlencoded; charset=UTF-8","accept":"application/json, text/javascript, */*; q=0.01","x-requested-with":"XMLHttpRequest","referer":"https://kibana1.example.com/login","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana1.example.com/login"},"res":{"statusCode":302,"responseTime":108,"contentLength":9},"message":"POST /login 302 108ms - 9.0B"}
Oct  2 16:46:02 kibana1 kibana: {"type":"response","@timestamp":"2018-10-02T16:46:02Z","tags":[],"pid":13854,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"host":"localhost:5601","connection":"close","kbn-xsrf":"5.6.1","kbn-version":"5.6.1","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36","accept":"application/json, text/javascript, */*; q=0.01","x-requested-with":"XMLHttpRequest","referer":"https://kibana1.example.com/login","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana1.example.com/login"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"GET / 200 8ms - 9.0B"}
Oct  2 16:46:02 kibana1 kibana: {"type":"log","@timestamp":"2018-10-02T16:46:02Z","tags":["debug","readonlyrest_kbn"],"pid":13854,"message":"ON_INJECT kibana index not found, defaulting to: .kibana"}

Additional notes:

The condition is seen to occur on both our primary and secondary clusters and with local users and AD users.
curl to ES appears to continue to work as expected.

Cluster details:

Primary cluster is running Elasticsearch 5.6.1 with the 1.16.27 RoR on masters, clients and Kibana
Secondary cluster is running Elasticsearch 5.6.11 with the 1.16.27 RoR on masters, clients, data nodes and Kibana

No other logs from clients or masters seem to have anything relevant.

We can provide a sanitized readonlyrest config if needed.

Ideas? Thoughts? Comments?

Thanks

Travis

sscarduzio · October 4, 2018, 7:56pm

Hi Travis,
Was this bug observed in previous releases of ROR? Since when is it happening?
I suspect this might be an ACL block ordering issue, and the ideal way to confirm it would be to:

Have an ES+Kibana environment run alone without concurrent users (not even ingestors or scripts)
Use Kibana and reproduce the bug.
Analyse in ES logs: find the audit log of at least one ES requests made by Kibana that has been matched by one of the block that contain: kibana_index: '.kibana-nonpriv' (“Local auth for nonpriv admin”, and “Non-privileged users” as of the provided readonlyrest.yml.

This class of problems can happen when either the conditions for matching the blocks are not stringent enough, or simply the order of the blocks is not ideal as a logical sequence. Mistakes can be very difficult to spot, so narrowing the circle with the above analysis helps a lot.

Travis · October 4, 2018, 8:40pm

Hi Simone,

Thanks for getting back with us. To answer your questions:

We did not see this bug in our previous version of Kibana RoR.
It started to manifest intermittently after we applied our Kibana RoR update(1.16.27)
We suspect there may be an issue with the order of items in the config and can try to isolate it our Secondary cluster.

Quick question did anything stand out from the attached config file?

Thanks

Travis

sscarduzio · October 4, 2018, 10:24pm

Not at the first look, but it’s quite late. Will review tomorrow morning one more time.

AsaZM · October 5, 2018, 4:44pm

I am working with Travis on the above issue.

Here are the ror blocks. (current logs messages showing same behavior, old log messages are just the entries I found first)
Part of what I don’t understand is the different behavior before and after restarting kibana if this is an ACL issue.
The below seem to show that the system is changing what it thinks the default index is.

USE CASE 1
ldap admin + priv user login getting .kibana-nonpriv incorrectly before bounce.
[Localhost->[hosts->false]], [Admins->[ldap_authorization->true, ldap_authentication->true, kibana_access->true]] .kibana-nonpriv userid /.kibana-nonpriv/_search?_source=index-pattern.title%2Ctype%2Ctitle&size=10000&from=0 ALLOWED indices:data/read/search

ldap admin + priv user login > getting .kibana index correctly after bounce same user.
[Localhost->[hosts->false]], [Admins->[ldap_authorization->true, ldap_authentication->true, kibana_access->true]] .kibana userid /.kibana/_search?size=10000&from=0 ALLOWED indices:data/read/search

USE CASE 2
local admin (non ldap) + priv getting .kibana-nonpriv incorrectly before bounce
[Localhost->[hosts->false]], [Admins->[ldap_authentication->false]], priv users->[ldap_authentication->false]], [Non-priv users->[ldap_authentication->false]], [Kibana Admin->[auth_key->false]], [Lander Account->[auth_key->false]], [Rover Account->[auth_key->false]], [Logstash User->[auth_key->false]], [Local Auth for admin->[kibana_access->true, auth_key->true]] .kibana-nonpriv localadmin /.kibana-nonpriv/_search?_source=index-pattern.title%2Ctype%2Ctitle&size=10000&from=0 ALLOWED indices:data/read/search

local admin (non ldap) + priv getting .kibana after bounce correctly same user
[Localhost->[hosts->false]], [Admins->[ldap_authentication->false]], priv users->[ldap_authentication->false]], [Non-priv users->[ldap_authentication->false]], [Kibana Admin->[auth_key->false]], [Lander Account->[auth_key->false]], [Rover Account->[auth_key->false]], [Logstash User->[auth_key->false]], [Local Auth for admin->[kibana_access->true, auth_key->true]] .kibana localadmin /.kibana/_search ALLOWED indices:data/read/search

USE CASE 3 (BEHAVIOR AS EXPECTED)
non admin Non priv users have been fine the whole time and getting correctly non-priv index
[Localhost->[hosts->false]], [Admins->[ldap_authentication->false]], [priv users->[ldap_authentication->false]], [Non-priv users->[ldap_authentication->true, kibana_access->true, indices->true, kibana_hide_apps->true, kibana_index->true]] .kibana-nonpriv nonadminnonprivusername /.kibana-nonpriv/_search?size=10000&from=0 ALLOWED indices:data/read/search

This looks to me like kibana is getting the username. but it is not getting to read only rest.
Here is what seems to be happening with non admin priv users, this is what kibana shots.

USE CASE 4
Before a bounce I show log entries like this in the kibana logs. (while issue is occurring),
non admin priv user getting .kibana-nonpriv incorrectly in the kibana log
594645040:Sep 28 17:44:22 kibanurl kibana: {“type”:“log”,"@timestamp":“2018-09-28T17:44:22Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:6884,“message”:" received identity payload: {“x-ror-current-group”:null,“x-ror-username”:“privusername”,“x-ror-kibana-hidden-apps”:[“kibana:dev_tools”,“kibana:management”,“readonlyrest_kbn”],“x-ror-kibana_access”:“rw”,“x-ror-available-groups”:[]}"}
594645435-Sep 28 17:44:22 kibanaurl kibana: {“type”:“log”,"@timestamp":“2018-09-28T17:44:22Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:6884,“message”:“ON_IDENTITY no kibana index from headers, setting kibana index to default configured .kibana-nonpriv”}

After kibana is bounced, i show log entries below, user getting the correct index .kibana same user
non admin priv user getting .kibana correctly in the kibana log
598092614-Sep 28 19:45:13 kibanaurl kibana: {“type”:“log”,"@timestamp":“2018-09-28T19:45:13Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:5383,“message”:“groupCurrent not found in cookie, that’s ok.”}
598092807:Sep 28 19:45:13 kibanaurl kibana: {“type”:“log”,"@timestamp":“2018-09-28T19:45:13Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:5383,“message”:" received identity payload: {“x-ror-current-group”:null,“x-ror-username”:“privusername”,“x-ror-kibana-hidden-apps”:[“kibana:dev_tools”,“kibana:management”,“readonlyrest_kbn”],“x-ror-kibana_access”:“rw”,“x-ror-available-groups”:[]}"}
598093202-Sep 28 19:45:13 kibanaurl kibana: {“type”:“log”,"@timestamp":“2018-09-28T19:45:13Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:5383,“message”:“ON_IDENTITY no kibana index from headers, setting kibana index to default configured .kibana”}

This is what I show readonlyrest_audit log. Looks to me like readonlyrest is not getting usernames at all and is throwing them to the nonpriv index. but also forbidding all index access. Which is why there users just get a white screen.

[Localhost->[hosts->false]], [Admins->[ldap_authorization->false, ldap_authentication->true, kibana_access->true]], [priv users->[ldap_authentication->true, kibana_access->true, indices->false]], [Non-priv users->[ldap_authentication->false]], [Kibana Admin->[auth_key->false]], [Lander Account->[auth_key->false]], [Rover Account->[auth_key->false]], [Logstash User->[auth_key->false]], [Local Auth for admin->[groups->false]], [Local Auth for nonpriv admin->[groups->false]], [DevHosts Kibana->[kibana_access->true, hosts->false]], [DevHosts->[hosts->false]], [Local Auth for admin 2->[groups->false]], [Local Auth for priv admin->[groups->false]], [Script User Auth->[groups->false]] .kibana-nonpriv - /.kibana-nonpriv/_search FORBIDDEN indices

sscarduzio · October 5, 2018, 5:11pm

Thanks for the data @AsaZM, I have an idea on which I can work now.

AsaZM · October 5, 2018, 5:14pm

Found another data point.
This is the graph of where ror is showing no user (and being forbidden)

No user ror audit info.

[Localhost->[hosts->false]], [Admins->[ldap_authorization->false, ldap_authentication->true, kibana_access->true]], [priv users->[ldap_authentication->true, kibana_access->true, indices->false]], [Non-priv users->[ldap_authentication->false]], [Kibana Admin->[auth_key->false]],[Local Auth for admin->[groups->false]], [Local Auth for nonpriv admin->[groups->false]], [DevHosts Kibana->[kibana_access->true, hosts->false]], [DevHosts->[hosts->false]], [Local Auth for admin 2->[groups->false]], [Local Auth for nonpriv admin->[groups->false]], [Script User Auth->[groups->false]] .kibana-nonpriv - /.kibana-nonpriv/_search?size=10000&from=0 FORBIDDEN indices:data/read/search

kibana getting non-priv info block
[Localhost->[hosts->false]], [Admins->[ldap_authentication->false]], [priv users->[ldap_authentication->false]], [Non-priv users->[ldap_authentication->false]], [Kibana Admin->[auth_key->true]] .kibana-nonpriv kibana /_mget ALLOWED indices:data/read/mget { name: ‘Kibana Admin’, policy: ALLOW, rules: [auth_k

AsaZM · October 5, 2018, 5:15pm

This is the graph of when ror is saying kibana is being given .kibana-nonpriv

sscarduzio · October 6, 2018, 11:15am

I analysed your YAML more closely, there is a few things that are incorrect/undesirable and add noise to the problem.

As a reference, here is the original file (after formatting)

readonlyrest:
  prompt_for_basic_auth: 'false'
  audit_collector: true

  ldaps:
    - name: nonpriv
      bind_dn: 'CN=account1,OU=Community,DC=company,DC=ad,DC=example'
      search_user_base_DN: 'OU=Users,OU=Community,DC=company,DC=ad,DC=example'
      bind_password: '123456789'
      user_id_attribute: sAMAccountName
      port: '389'
      search_groups_base_DN: 'OU=Data Security Groups,DC=company,DC=ad,DC=example'
      ssl_enabled: 'false'
      host: server1
      unique_member_attribute: member
      ssl_trust_all_certs: 'true'
      connection_timeout_in_sec: 10
      request_timeout_in_sec: 10
      cache_ttl_in_sec: 60

    - name: privileged
      bind_dn: 'CN=account1,OU=Community,DC=company,DC=ad,DC=example'
      search_user_base_DN: 'OU=Privileged Users,OU=Community,DC=company,DC=ad,DC=example'
      bind_password: '123456789'
      user_id_attribute: sAMAccountName
      port: '389'
      search_groups_base_DN: 'OU=Data Security Groups,DC=company,DC=ad,DC=example'
      ssl_enabled: 'false'
      host: server1
      unique_member_attribute: member
      ssl_trust_all_certs: 'true'
      connection_timeout_in_sec: 10
      request_timeout_in_sec: 10
      cache_ttl_in_sec: 60

  access_control_rules:
    - name: Localhost
      hosts: [127.0.0.1]

    - name: Admins
      kibana_access: admin
      ldap_authentication: privileged
      ldap_authorization: 
        name: Privileged # <-- why capitalised?
        groups: ['App-Kibana-Team']
    
    - name: 'Privileged users'
      ldap_authentication: privileged
      kibana_access: rw
      indices: ['logstash-*', .kibana, 'current_*']
      kibana_hide_apps: [readonlyrest_kbn, 'kibana:dev_tools', 'kibana:management']

    - name: 'Non-privileged users'
      ldap_authentication: nonpriv
      kibana_access: rw
      kibana_index: '.kibana-nonpriv'
      indices: ['logstash-prod-wl-*', 'logstash-dev-wl-*', '.kibana-nonpriv']
      kibana_hide_apps: [readonlyrest_kbn, 'kibana:dev_tools', 'kibana:management']

    - name: 'Kibana Admin'
      auth_key: 'kibana:123456789'

    - name: 'Logstash User'
      actions: ['cluster:monitor/main', 'indices:admin/types/exists', 'indices:data/read/*', 'indices:data/write/*', 'indices:admin/template/*', 'indices:admin/create']
      auth_key: 'logstash:123456789'

    - name: 'Local auth for admin'
      groups: [Admins]
      kibana_access: admin
      type: allow

    - name: 'Local auth for nonpriv admin'
      groups: [Nonpriv Admins]
      kibana_access: admin
      kibana_index: '.kibana-nonpriv'
      type: allow
      indices: ['logstash-prod-wl-*', 'logstash-dev-wl-*', '.kibana-nonpriv']
    
    - name: DevHosts Kibana
      hosts: [10.10.10.16,10.10.11.19]
      kibana_access: admin

    - name: DevHosts
      hosts: [10.10.10.16,10.10.11.19]

    - name: 'Local auth for admin 2'
      groups: [Admins]
      type: allow

    - name: 'Local auth for nonpriv admin'
      groups: [Nonpriv Admins]
      kibana_index: '.kibana-nonpriv'
      type: allow
      indices: ['logstash-prod-wl-*', 'logstash-dev-wl-*', '.kibana-nonpriv']

    - name: 'Script User Auth'
      groups: [Script]
      type: allow

  users:
    - username: es_admin
      groups: [Admins]
      auth_key: 'es_admin:1234567'

    - username: es_admin_nonpriv
      groups: [Nonpriv Admins]
      auth_key: 'es_admin_nonpriv:1234567'

Let’s put on top all the rules that do not describe actual Kibana users with actual identities (i.e. machines, API clients)
It’s not clear to me what “DevHosts Kibana” block does, because it has “kibana_access” macro which is normally used for human users with an actual Kibana identity.

3.The "Local auth for nonpriv admin" block is written twice (we recently added a check to avoid this error, will be available in ROR for ES 1.16.28)

“Script User Auth” block is bound to a group “Script” that has no users.
If you write “Local auth for admin 2” why do you need “Local auth for admin” at all? Please choose either.
The name reference to a ldap connector was capitalized “Privileged” as opposed to “privileged”

So now here is a cleaned up version, made just to narrow down the issue.

readonlyrest:
  prompt_for_basic_auth: 'false'
  audit_collector: true

  ldaps:
    - name: nonpriv
      bind_dn: 'CN=account1,OU=Community,DC=company,DC=ad,DC=example'
      search_user_base_DN: 'OU=Users,OU=Community,DC=company,DC=ad,DC=example'
      bind_password: '123456789'
      user_id_attribute: sAMAccountName
      port: '389'
      search_groups_base_DN: 'OU=Data Security Groups,DC=company,DC=ad,DC=example'
      ssl_enabled: 'false'
      host: server1
      unique_member_attribute: member
      ssl_trust_all_certs: 'true'
      connection_timeout_in_sec: 10
      request_timeout_in_sec: 10
      cache_ttl_in_sec: 60

    - name: privileged
      bind_dn: 'CN=account1,OU=Community,DC=company,DC=ad,DC=example'
      search_user_base_DN: 'OU=Privileged Users,OU=Community,DC=company,DC=ad,DC=example'
      bind_password: '123456789'
      user_id_attribute: sAMAccountName
      port: '389'
      search_groups_base_DN: 'OU=Data Security Groups,DC=company,DC=ad,DC=example'
      ssl_enabled: 'false'
      host: server1
      unique_member_attribute: member
      ssl_trust_all_certs: 'true'
      connection_timeout_in_sec: 10
      request_timeout_in_sec: 10
      cache_ttl_in_sec: 60

  access_control_rules:
    
    # MACHINES FIRST
    - name: Localhost
      hosts: [127.0.0.1]

    - name: DevHosts Kibana
      hosts: [10.10.10.16,10.10.11.19]
      kibana_access: admin # <---- needs clarification!

    - name: DevHosts
      hosts: [10.10.10.16,10.10.11.19]

    - name: 'Kibana Admin'
      auth_key: 'kibana:123456789'

    - name: 'Logstash User'
      actions: ['cluster:monitor/main', 'indices:admin/types/exists', 'indices:data/read/*', 'indices:data/write/*', 'indices:admin/template/*', 'indices:admin/create']
      auth_key: 'logstash:123456789'


   # KIBANA USERS - LOCAL NEXT 
    - name: 'Local auth for admin'
      groups: [Admins]
      kibana_access: admin

    - name: 'Local auth for nonpriv admin'
      groups: [Nonpriv Admins]
      kibana_access: admin
      kibana_index: '.kibana-nonpriv'
      indices: ['logstash-prod-wl-*', 'logstash-dev-wl-*', '.kibana-nonpriv']

   # KIBANA USERS - REMOTE NEXT (leave remote connections for last, as they are more expensive than local)
    - name: Admins
      kibana_access: admin
      ldap_authentication: privileged
      ldap_authorization: 
        name: privileged 
        groups: ['App-Kibana-Team']
    
    - name: 'Privileged users'
      ldap_authentication: privileged
      kibana_access: rw
      indices: ['logstash-*', .kibana, 'current_*']
      kibana_hide_apps: [readonlyrest_kbn, 'kibana:dev_tools', 'kibana:management']

    - name: 'Non-privileged users'
      ldap_authentication: nonpriv
      kibana_access: rw
      kibana_index: '.kibana-nonpriv'
      indices: ['logstash-prod-wl-*', 'logstash-dev-wl-*', '.kibana-nonpriv']
      kibana_hide_apps: [readonlyrest_kbn, 'kibana:dev_tools', 'kibana:management']

# LOCAL USERS DEFINITION
  users:
    - username: es_admin
      groups: [Admins]
      auth_key: 'es_admin:1234567'

    - username: es_admin_nonpriv
      groups: [Nonpriv Admins]
      auth_key: 'es_admin_nonpriv:1234567'

sscarduzio · October 6, 2018, 11:16am

To recap:

@AsaZM Can you confirm if this issue happens with the cleaned up conf?
@AsaZM Can you confirm if this happens for the local priv/nonpriv users as well regardless of the LDAP one?

AsaZM · October 7, 2018, 12:49am

Yes. The local priv admin user is effected as well.

sscarduzio · October 8, 2018, 12:15pm

UPDATE: Finally managed to reproduce this!
UPDATE2: FIXED! Will release a pre for you to validate, and then publish the new version for everyone.

AsaZM · October 8, 2018, 2:29pm

Awesome, was it ACL related? Or something else going on after all?

AsaZM · October 8, 2018, 2:32pm

We need for 5.6.1 and 5.6.11 in order to test.

sscarduzio · October 8, 2018, 2:50pm

Basically there was a case when no kibana_index was specified, we didn’t change it at all instead of explicitly changing it to the default kibana index. This caused the stateful behaviour.

AsaZM · October 8, 2018, 7:16pm

I installed to our cluster. I am not getting a login screen, and just loading the the white screen. Does the elasticsearch version need to match?

AsaZM · October 8, 2018, 7:49pm

Changed the version in the plugin code > re-installed. Still getting no login screen, just loads me into kibana with menu and white screen where should see things on test instance.

sscarduzio · October 8, 2018, 8:05pm

@AsaZM this is an old version of Kibana, did you remember to give the “touch” command?

$ touch optimize/bundles/readonlyrest_kbn.style.css

Also, do you have any errors in JS console?

EDIT: this is solely a kibana side fix, and the target ES plugin version is 1.16.27
EDIT2: I literally just downloaded the zip file from the link I gave you and it works on my local copy of Kibana 5.6.1

AsaZM · October 8, 2018, 8:06pm

Yes I did the touch.
Looking

AsaZM · October 8, 2018, 8:47pm

I have not tried 5.6.1 yet, as that is our active cluster, trying it in the backup cluster.