Issues with multitenancy in a concurrent access context

Hello,

We had experienced issues using the multi tenancy.

Our setup consists of:

  • Two tenants (which correspond to two of our customers). Let’s call them Foo and Bar.
  • An admin user for each.
  • Authentication is done through a nginx proxy.

Here are our readonlyrest.yml config file:

 readonlyrest:
   prompt_for_basic_auth: false
   access_control_rules:
   
   - auth_key: kibana:kibana
     name: '::KIBANA-SRV::'
     verbosity: error
     
   - name: 'Foo'
     indices: ['.kibana_foo', 'foo_data']
     kibana_access: admin
     kibana_index: .kibana_foo
     proxy_auth: ['foo']
     actions: ["*"]
 
   - name: 'Bar'
     indices: ['.kibana_bar', 'bar_data']
     kibana_access: admin
     kibana_index: .kibana_bar
     proxy_auth: ['bar']
     actions: ["*"]

Now here are the steps.

  1. foo connects to its Foo Kibana instance and list the visualizations.

In the log file, i get the following

[2018-02-02T14:24:44,264][INFO ][t.b.r.a.ACL ] ALLOWED by { name: ‘Foo’, policy: ALLOW} req={ ID:1250055474-1470883861#5202, TYP:SearchRequest, CGR:N/A, USR:foo, BRS:false, ACT:indices:data/read/search, OA:172.17.0.5, IDX:.kibana_foo, MET:POST, PTH:/.kibana_foo/_search?size=1000&from=0, CNT:{“version”:true,“query”:{“bool”:{“must”:[{“match_all”:{}}],“filter”:[{“bool”:{“should”:[{“term”:{“_type”:“visualization”}},{“term”:{“type”:“visualization”}}]}}]}}}, HDR:{authorization=Basic aW56c2Uub4B0aZ6kYXRhOm51bGw=, Connection=keep-alive, Content-Length=163, content-type=application/json, Host=dummy.host:9200, x-forwarded-user=foo}, HIS:[Bar->[proxy_auth->false]], [Foo->[kibana_access->true, proxy_auth->true, indices->true, kibana_index->true]], [::KIBANA-SRV::->[auth_key->false]] }

  1. Then bar connects to its Bar Kibana instance and tries to list visualization as well but i have an error.

In my browser i see:
commons.bundle.js?v=15601:37 GET https://dummy.host/api/saved_objects/?type=visualization&per_page=1000&page=1&search_fields=title^3&search_fields=description 400 (Bad Request)

In my logs:

[2018-02-02T14:24:44,264][INFO ][t.b.r.a.ACL ] FORBIDDEN by default req={ ID:1250055474-1470883861#5203, TYP:SearchRequest, CGR:N/A, USR:bar, BRS:false, ACT:indices:data/read/search, OA:172.17.0.5, IDX:.kibana_foo, MET:POST, PTH:/.kibana_foo/_search?size=1000&from=0, CNT:{“version”:true,“query”:{“bool”:{“must”:[{“match_all”:{}}],“filter”:[{“bool”:{“should”:[{“term”:{“_type”:“visualization”}},{“term”:{“type”:“visualization”}}]}}]}}}, HDR:{authorization=Basic c2lkZWwub3B0aW1kYXRhOm51bGw=, Connection=keep-alive, Content-Length=163, content-type=application/json, Host=dummy.host:9200, x-forwarded-user=bar}, HIS:[Foo->[proxy_auth->false]], [Bar->[kibana_access->true, proxy_auth->true, indices->false, kibana_index->true]], [::KIBANA-SRV::->[auth_key->false]] }

From my understanding, the issue is that the ACL engine considers that for the bar user, the IDX is .kibana_foo.

I activated the debug log and saw:

[2018-02-02T14:24:44,263][DEBUG][t.b.r.e.RequestInfo ] Discovered indices: .kibana_foo

If i look at the source code, it comes from this line. If i look at the code above, the TYP:SearchRequest does not seem to be handled (or by extractStringArrayFromPrivateMethod which i don’t really understand what it does).

If bar reload the page, it works, and i see in the logs:

[2018-02-02T14:24:44,264][INFO ][t.b.r.a.ACL ] ALLOWED by { name: ‘Bar’, policy: ALLOW} req={ ID:1250055474-1470883861#5203, TYP:SearchRequest, CGR:N/A, USR:bar, BRS:false, ACT:indices:data/read/search, OA:172.17.0.5, IDX:.kibana_bar, MET:POST, PTH:/.kibana_bar/_search?size=1000&from=0, CNT:{“version”:true,“query”:{“bool”:{“must”:[{“match_all”:{}}],“filter”:[{“bool”:{“should”:[{“term”:{“_type”:“visualization”}},{“term”:{“type”:“visualization”}}]}}]}}}, HDR:{authorization=Basic c2lkZWwub3B0aW1kYXRhOm51bGw=, Connection=keep-alive, Content-Length=163, content-type=application/json, Host=dummy.host:9200, x-forwarded-user=bar}, HIS:[Foo->[proxy_auth->false]], [Bar->[kibana_access->true, proxy_auth->true, indices->false, kibana_index->true]], [::KIBANA-SRV::->[auth_key->false]] }

In the end, if bar & foo are working at the same time, it’s totally messed up. I suspect this is related to some caching issues.

Here are version:

  • Elasticsearch: 5.6.5
  • Kibana: 5.6.5
  • readonlyrest-1.16.15_es5.6.5.zip
  • readonlyrest_kbn_enterprise-1.16.15_es5.6.5.zip

Do you have an idea what’s going wrong?

Hello @gtnx, just reproduced this in 6.1.2. It needs a fix indeed :slight_smile:

Ok thanks

  • When do you think the patch version will be ready?
  • I’m curious, our usecase seems to be very classic. How is it possible that nobody saw that before? What’s the specificity of our usecase?

Hi @gtnx,

I don’t have a precise answer to your questions right now, but this definitely is top priority for us this week. Will keep updated this thread.

@gtnx we have a fix for this issue. The fix has been introduced in 1.16.16-pre2.
I will send you a link to the build via email.