Multi-tenacy changed in 1.16.30?

I upgraded my kibana to 6.5.1 and upgraded the ror enterprise plugin as well. Everything booted back up fine and the plugin appears to work but I no longer have a drop-down on the left hand side to switch tenancy. Nothing in regards to LDAP users/groups have changed since upgrading this morning so I’m assuming it’s something between ROR and Kibana with the new version. Any ideas of where to look/troubleshoot?

Hi @mmorrison,

We had a lot of fixes during the latest 3 or 4 iterations in how ROR handles current groups and available groups, especially in LDAP.

Can you share what ROR version you are coming from? And also your YAML settings, so maybe we can debug this together.

Absolutely.

Coming from ROR 1.16.29 for both Elasticsearch and Kibana. ES version was 6.5.0 and now is 6.5.1, Kibana was 6.4.3 and now is 6.5.1

Here is my readonlyrest.yml

readonlyrest:
    response_if_req_forbidden: Access is not authorized. Contact DevOps if you believe this is an error.
    access_control_rules:
    - name: "::LOGSTASH::"
      auth_key_sha265: xxxxx
      actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","cluster:monitor/xpack/info","cluster:admin/xpack/monitoring/bulk"]
      verbosity: error
    - name: "::KIBANA-SRV::"
      auth_key_sha256: xxxxxx
      verbosity: error
    - name: Allow Elastic Admins full priv
      ldap_auth:
          name: "myLDAP"
          groups: ["Elastic Admins"]
      verbosity: error
      kibana_access: admin
      kibana_index: ".kibana"

    - name: Allow Elastic Power Users enhanced privs
      ldap_auth:
          name: "myLDAP"
          groups: ["Elastic Power Users"]
      verbosity: error
      kibana_access: rw
      kibana_index: ".kibana_pwrusr"
      indices: [Array of Indexes]
      kibana_hide_apps: ["kibana:dev_tools", "readonlyrest_kbn"]

    - name: Allow Users basic access
      ldap_auth:
          name: "myLDAP"
          groups: ["Elastic Users"]
      verbosity: error
      kibana_access: rw
      kibana_index: ".kibana_esusers"
      indices: [ARRAY of index]
      kibana_hide_apps: ["readonlyrest_kbn", "kibana:dev_tools", "kibana:monitoring"]
    ldaps:
    - name: myLDAP
      host: "LDAP_URI"
      port: 636                                                     # optional, default 389
      ssl_enabled: true                                             # optional, default true
      ssl_trust_all_certs: false                                    # optional, default false
      search_user_base_DN: "ou=People,dc=SITE,dc=SITE"
      user_id_attribute: "uid"                                      # optional, default "uid"
      search_groups_base_DN: "ou=Roles,dc=SITE,dc=SITE"
      unique_member_attribute: "roleOccupant"                       # optional, default "uniqueMember"
      connection_pool_size: 10                                      # optional, default 30
      connection_timeout_in_sec: 10                                 # optional, default 1
      request_timeout_in_sec: 10                                    # optional, default 1
      cache_ttl_in_sec: 60                                          # optional, default 0 - cache disabled
      group_search_filter: "(objectClass=organizationalRole)"       # optional, default (cn)    
      group_name_attribute: "cn"

Thank you Matt, do you have a user you know the credentials of, which belongs to multiple of those groups?
If you put ES in debug mode, then you can then prepare a curl command like this:

curl -k -vvv -u myuser:passwd 'https://es-host:9200/_readonlyrest/metadata/current_user' 

You should see in the logs now if LDAP connector resolves all the groups, and the output of the curl should list the available groups (calculated as the intersection between the groups cited in your ACL, and the list of groups the user belongs to).

So, restarting ES and Kibana seemed to have brought back the drop down box to switch from Group_A to Group_B. When I click to switch tenancy into Group_B, it drops me to the login page now and says “Your request could not be authenticated, please log in again.” I log in again, get put into Group_A’s tenancy and the same thing happens when I change to Group_B again. Here is the logs I’m seeing in ES.

[2018-12-06T12:46:35,834][INFO ][t.b.r.a.ACL ] [ind-logs-client1] FORBIDDEN by default req={ ID:1710731052-1392627170#703810, TYP:GetRequest, CGR:Elastic Users, USR:mmorrison, BRS:false, KDX:null, ACT:indices:data/read/get, OA:127.0.0.1, DA:127.0.0.1, IDX:.kibana, MET:GET, PTH:/.kibana/doc/space%3Adefault, CNT:<N/A>, HDR:{authorization=, Connection=keep-alive, x-ror-current-group=Elastic Users, Host=localhost:9200, Content-Length=0}, HIS:[::LOGSTASH::->[auth_key->false]], [::KIBANA-SRV::->[auth_key->false]], [Allow Elastic Admins full priv->[ldap_authorization->false]], [Allow Elastic Power Users enhanced privs->[ldap_authorization->false]], [Allow Users basic access->[ldap_authorization->true, kibana_access->true, indices->false]] }

[2018-12-06T12:46:35,855][INFO ][t.b.r.a.ACL ] [ind-logs-client1] FORBIDDEN by default req={ ID:2038481742–954893958#703811, TYP:SearchRequest, CGR:Elastic Users, USR:mmorrison, BRS:false, KDX:null, ACT:indices:data/read/search, OA:127.0.0.1, DA:127.0.0.1, IDX:.kibana, MET:POST, PTH:/.kibana/_search?size=1000&from=0, CNT:<OMITTED, LENGTH=245>, HDR:{authorization=, x-ror-current-group=Elastic Users, Connection=keep-alive, content-type=application/json, Host=localhost:9200, Content-Length=245}, HIS:[::LOGSTASH::->[auth_key->false]], [::KIBANA-SRV::->[auth_key->false]], [Allow Elastic Admins full priv->[ldap_authorization->false]], [Allow Elastic Power Users enhanced privs->[ldap_authorization->false]], [Allow Users basic access->[ldap_authorization->true, kibana_access->true, indices->false]] }

I’ve confirmed via the command you provided and ldapsearch on my LDAP host that the user is showing as being in both groups.

@sscarduzio any ideas of where to look? My multi-tenancy is completely broken at the moment.

For some reason, during the tenancy switch, on the Kibana side, the information about the kibana index is not being changed.

See that the current group “CGR” is Elastic Users, which is supposed to have “KDX” kibana index as .kibana_esusers, but the client is requesting a search in the “.kibana” index, which they have no permission for.

Therefore the instant logout.

Strange because this is one of the first things I tested, and worked in my classic multi tenancy configuration. Will try with your configuration.

Any updates @sscarduzio? I removed the user (aka me) from the “Elastic Admins” group and left in just the “Elastic Users” and now I can’t login at all because it tries to access that .kibana index still.

still working on this, Matt. Sorry for the delay.

Any update? I updated to the latest version of ROR and I’ll still getting the same problem.

When I tried to reproduce this, I could not. Then I went on with fixing things like reporting, and kibana indices management in 6.5.x so a lot of the code that does this is now new. Can you please try now with the new version?