_cat/indices not working again

1

Hello all,

It looks like that there is again the bug with _cat/indices

there is my readonlyrest.yml

readonlyrest:
  access_control_rules:
  - name: Super user access
    verbosity: error
    type: allow
    groups: ["superuser"]
    indices: ["*"]
    actions: ["*"]

  - name: All ro access
    type: allow
    groups: ["allro"]
    indices: ["*"]
    actions: ["indices:data/read/*", "cluster:monitor/state", "indices:admin/get", "indices:admin/mappings/fields/get", "indices:admin/mappings/get", "indices:admin/aliases/get", "indices:admin/template/get"]

  - name: Kibana rw access
    type: allow
    groups: ["kibanarw"]
    indices: [".kibana*"]
    actions: ["indices:data/read/*","indices:data/write/*"]
  
  - name: Kibana ro access
    type: allow
    groups: ["kibanaro"]
    indices: [".kibana*"]
    actions: ["indices:data/read/*"]

  - name: Logs all access
    verbosity: error
    type: allow
    groups: ["logs"]
    indices: ["logs*"]
    actions: ["*"]

  - name: Logs ro access
    type: allow
    groups: ["logsro"]
    indices: ["logs*",".monitoring*"]
    actions: ["indices:data/read/*", "cluster:monitor/state", "indices:admin/get", "indices:admin/mappings/fields/get", "indices:admin/mappings/get", "indices:admin/aliases/get", "indices:admin/template/get"]

  - name: Alibaba all access
    type: allow
    groups: ["alibaba"]
    indices: ["alibaba*"]
    actions: ["*"]

users:
- username: elastic
  auth_key_unix: ${ELASTIC_USER}
  groups: ["superuser"]

- username: kibana
  auth_key_unix: ${KIBANA_USER}
  groups: ["superuser"]

- username: logstash
  auth_key_unix: ${LOGSTASH_USER}
  groups: ["logs"]

- username: alibaba
  auth_key_unix: ${ALIBABA_USER}
  groups: ["alibaba", "kibanarw", "logsro"]

obfuscated_headers: ["x-auth-key", "authorization"]

ssl:
keystore_file: "keystore.jks"
keystore_pass: "${PKCS12_PASSWORD}"
key_pass: "${PKCS12_PASSWORD}"
truststore_file: "truststore.jks"
truststore_pass: "${PKCS12_PASSWORD}"

ssl_internode:
keystore_file: "keystore.jks"
keystore_pass: "${PKCS12_PASSWORD}"
key_pass: "${PKCS12_PASSWORD}"
truststore_file: "truststore.jks"
truststore_pass: "${PKCS12_PASSWORD}"
certificate_verification: true
client_authentication: false

Problem is if the alibaba user will execute cat/indices from kibana, he will only see the alibaba* indices, and he should see also the logs* and .monitoring indices.

In past it was working great, but somehow its now bugged, user is able to see those indices when he tried to _search or he can see it from dashboard from index pattern, but not in _cat/indices.

I’m using the 7.7.1 elasticsearch with 1.20.0 ror, but this issue was also in es 7.7.0 ror 1.19.5, i started to develop cluster on that version, it was working well from start and after a while the issue appeared.

Can you please investigate ?

Thanks !

Regards,
Denis.

2

could you please share the ES log which shows which block was matched by _cat/indices request?

3

Here it is:

[2020-07-13T16:06:12,578][INFO ][tech.beshu.ror.accesscontrol.logging.AccessControlLoggingDecorator] [alibaba-ire-all-01] [36mALLOWED by { name: 'Alibaba all access', policy: ALLOW, rules: [groups,actions,indices] req={ ID:327866964--2068283520#70031, TYP:GetSettingsRequest, CGR:N/A, USR:alibaba, BRS:false, KDX:null, ACT:indices:monitor/settings/get, OA:XXXX, XFF:x-forwarded-for=XXXX, DA:XXXX, IDX:*, MET:GET, PTH:/_cat/indices, CNT:<N/A>, HDR:Authorization=<OMITTED>, Connection=close, content-length=0, content-type=application/json, host=XXXX, x-forwarded-for=XXXX, x-forwarded-host=XXXX:5601, x-forwarded-port=60452, x-forwarded-proto=https, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/console/proxy, HIS:[Super user access-> RULES:[groups->false], RESOLVED:[indices=*]], [All ro access-> RULES:[groups->false], RESOLVED:[indices=*]], [Kibana rw access-> RULES:[groups->true, actions->false], RESOLVED:[user=alibaba;group=kibanarw;av_groups=kibanarw;indices=*]], [Kibana ro access-> RULES:[groups->false], RESOLVED:[indices=*]], [Logs all access-> RULES:[groups->false], RESOLVED:[indices=*]], [Logs ro access-> RULES:[groups->true, actions->false], RESOLVED:[user=alibaba;group=logsro;av_groups=logsro;indices=*]], [Alibaba all access-> RULES:[groups->true, actions->true, indices->true], RESOLVED:[user=alibaba;group=alibaba;av_groups=alibaba;indices=alibaba_backend_demo]] }[0m

4

could you also show logs for the request which works well according to your first post?

5

unfortunately not, i didnt have logging implemented that time, so its not stored :confused:

6

I thought about logging the _search or index pattern request. This is weird for me that these two work as expected but _cat/indices not.

7

No idea why this being resolved with kibanarw group, when i used this

GET logs-2020-07-13/_search

There is access log:

[2020-07-13T18:33:51,487][INFO ][tech.beshu.ror.accesscontrol.logging.AccessControlLoggingDecorator] [alibaba-ire-all-01] [36mALLOWED by { name: 'Kibana rw access', policy: ALLOW, rules: [groups,actions,indices] req={ ID:1600902828--1037864470#140443, TYP:SubmitAsyncSearchRequest, CGR:N/A, USR:alibaba, BRS:false, KDX:null, ACT:indices:data/read/async_search/submit, OA:XXXX, XFF:null, DA:XXXX, IDX:<N/A>, MET:POST, PTH:/logs*/_async_search, CNT:<OMITTED, LENGTH=708.0 B> , HDR:Authorization=<OMITTED>, Connection=keep-alive, Content-Length=708, Host=XXXX, content-type=application/json, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/internal/search/es, HIS:[Super user access-> RULES:[groups->false]], [All ro access-> RULES:[groups->false]], [Kibana rw access-> RULES:[groups->true, actions->true, indices->true], RESOLVED:[user=alibaba;group=kibanarw;av_groups=kibanarw]] }[0m

I have verbosity:error on that logsro, because it was generating too much logs, let me turn this on and the log is:

[2020-07-13T18:42:11,210][INFO ][tech.beshu.ror.accesscontrol.logging.AccessControlLoggingDecorator] [alibaba-ire-all-01] [36mALLOWED by { name: 'Logs ro access', policy: ALLOW, rules: [groups,actions,indices] req={ ID:586214041-1455909269#444, TYP:SearchRequest, CGR:N/A, USR:alibaba, BRS:false, KDX:null, ACT:indices:data/read/search, OA:XXXX, XFF:x-forwarded-for=XXXX, DA:XXXX, IDX:logs-2020.07.13, MET:GET, PTH:/logs-2020.07.13/_search, CNT:<N/A>, HDR:Authorization=<OMITTED>, Connection=close, content-length=0, content-type=application/json, host=XXXX, x-forwarded-for=XXXX, x-forwarded-host=XXXX, x-forwarded-port=49442, x-forwarded-proto=https, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/console/proxy, HIS:[Super user access-> RULES:[groups->false], RESOLVED:[indices=logs-2020.07.13]], [All ro access-> RULES:[groups->false], RESOLVED:[indices=logs-2020.07.13]], [Kibana rw access-> RULES:[groups->true, actions->true, indices->false], RESOLVED:[user=alibaba;group=kibanarw;av_groups=kibanarw;indices=logs-2020.07.13]], [Kibana ro access-> RULES:[groups->false], RESOLVED:[indices=logs-2020.07.13]], [Logs all access-> RULES:[groups->false], RESOLVED:[indices=logs-2020.07.13]], [Logs ro access-> RULES:[groups->true, actions->true, indices->true], RESOLVED:[user=alibaba;group=logsro;av_groups=logsro;indices=logs-2020.07.13]] }[0m

8

ok, this is strange. I’ll have to check it. But I’m not sure is I was able to do it this week.

9

sure, let me know if you find something

10

Hello there, is there any update regarding to this ?

11

@coutoPL what is a current state of this ?

12

This task in on the top of current sprint backlog. It’ll be taken soon

13

@Sinedko I’ve checked the configuration from first post and this is my analysis:

Here is my indices:

> curl -k -u elastic:test  "http://127.0.0.1:9200/_cat/indices"
yellow open index1        DC8Oy5P0QaCql0wWLqq6NA 1 1 1 0 2.9kb 2.9kb
yellow open logs_20200202 3O2VThwHQ02Mw_VoIa6RXw 1 1 1 0 2.8kb 2.8kb
yellow open alibaba1      EUpbcPz-S_GaF2ZnMNqORw 1 1 1 0 2.8kb 2.8kb
yellow open logs_20200201 daoSFCFeQBCulOCSQixfBA 1 1 1 0 2.8kb 2.8kb
yellow open index2        tC8dHVLrQwiPEZzMpINHYA 1 1 1 0 2.9kb 2.9kb

Now, I use “alibaba” user to cat indices:

> curl -k -u alibaba:test  "http://127.0.0.1:9200/_cat/indices"
yellow open alibaba1 EUpbcPz-S_GaF2ZnMNqORw 1 1 1 0 2.9kb 2.9kb

So, this is the thing you have reported. Alibaba sees only “alibaba*” indices. Let’s check why:

[2020-09-19T17:24:07,481][DEBUG][t.b.r.a.b.r.AuthKeyRule  ] [n1_it] Attempting Login as: alibaba rc: 1665959920--708848282#246
[2020-09-19T17:24:07,482][DEBUG][t.b.r.a.b.r.AuthKeyRule  ] [n1_it] Attempting Login as: alibaba rc: 1665959920--708848282#246
[2020-09-19T17:24:07,483][DEBUG][t.b.r.a.b.Block          ] [n1_it] [Super user access] the request matches no rules in this block: {  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=<OMITTED>, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[Super user access-> RULES:[groups->false], RESOLVED:[indices=*]]  } 
[2020-09-19T17:24:07,484][DEBUG][t.b.r.a.b.Block          ] [n1_it] [All ro access] the request matches no rules in this block: {  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=<OMITTED>, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[All ro access-> RULES:[groups->false], RESOLVED:[indices=*]]  } 
[2020-09-19T17:24:07,484][DEBUG][t.b.r.a.b.r.AuthKeyRule  ] [n1_it] Attempting Login as: alibaba rc: 1665959920--708848282#246
[2020-09-19T17:24:07,484][DEBUG][t.b.r.a.b.r.ActionsRule  ] [n1_it] This request uses the action 'indices:monitor/settings/get' and none of them is on the list.
[2020-09-19T17:24:07,485][DEBUG][t.b.r.a.b.Block          ] [n1_it] [Kibana rw access] the request matches no rules in this block: {  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=<OMITTED>, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[Kibana rw access-> RULES:[groups->true, actions->false], RESOLVED:[user=alibaba;group=kibanarw;av_groups=kibanarw;indices=*]]  } 
[2020-09-19T17:24:07,486][DEBUG][t.b.r.a.b.Block          ] [n1_it] [Kibana ro access] the request matches no rules in this block: {  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=<OMITTED>, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[Kibana ro access-> RULES:[groups->false], RESOLVED:[indices=*]]  } 
[2020-09-19T17:24:07,486][DEBUG][t.b.r.a.b.r.AuthKeyRule  ] [n1_it] Attempting Login as: alibaba rc: 1665959920--708848282#246
[2020-09-19T17:24:07,486][DEBUG][t.b.r.a.b.Block          ] [n1_it] [Logs all access] the request matches no rules in this block: {  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=<OMITTED>, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[Logs all access-> RULES:[groups->false], RESOLVED:[indices=*]]  } 
[2020-09-19T17:24:07,487][DEBUG][t.b.r.a.b.r.AuthKeyRule  ] [n1_it] Attempting Login as: alibaba rc: 1665959920--708848282#246
[2020-09-19T17:24:07,487][DEBUG][t.b.r.a.b.r.ActionsRule  ] [n1_it] This request uses the action 'indices:monitor/settings/get' and none of them is on the list.
[2020-09-19T17:24:07,487][DEBUG][t.b.r.a.b.Block          ] [n1_it] [Logs ro access] the request matches no rules in this block: {  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=<OMITTED>, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[Logs ro access-> RULES:[groups->true, actions->false], RESOLVED:[user=alibaba;group=logsro;av_groups=logsro;indices=*]]  } 
[2020-09-19T17:24:07,488][DEBUG][t.b.r.a.b.r.AuthKeyRule  ] [n1_it] Attempting Login as: alibaba rc: 1665959920--708848282#246
[2020-09-19T17:24:07,488][DEBUG][t.b.r.a.b.r.IndicesRule  ] [n1_it] [1665959920--708848282#246] Checking - none or all indices ...
[2020-09-19T17:24:07,489][DEBUG][t.b.r.a.b.Block          ] [n1_it] matched { name: 'Alibaba all access', policy: ALLOW, rules: [groups,actions,indices] { found: user=alibaba;group=alibaba;av_groups=alibaba;indices=alibaba1 }
[2020-09-19T17:24:07,490][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [n1_it] ALLOWED by { name: 'Alibaba all access', policy: ALLOW, rules: [groups,actions,indices] req={  ID:1665959920--708848282#246,  TYP:GetSettingsRequest,  CGR:N/A,  USR:alibaba,  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=Basic YWxpYmFiYTp0ZXN0, Host=127.0.0.1:9200, User-Agent=curl/7.64.1, content-length=0,  HIS:[Super user access-> RULES:[groups->false], RESOLVED:[indices=*]], [All ro access-> RULES:[groups->false], RESOLVED:[indices=*]], [Kibana rw access-> RULES:[groups->true, actions->false], RESOLVED:[user=alibaba;group=kibanarw;av_groups=kibanarw;indices=*]], [Kibana ro access-> RULES:[groups->false], RESOLVED:[indices=*]], [Logs all access-> RULES:[groups->false], RESOLVED:[indices=*]], [Logs ro access-> RULES:[groups->true, actions->false], RESOLVED:[user=alibaba;group=logsro;av_groups=logsro;indices=*]], [Alibaba all access-> RULES:[groups->true, actions->true, indices->true], RESOLVED:[user=alibaba;group=alibaba;av_groups=alibaba;indices=alibaba1]]  }

Logs show that “Logs ro access” block was rejected because actions rule was not matched, because action “indices:monitor/settings/get” was not on the list.

Next block “Alibaba all access” was matched and indices list was narrowed to alibaba* indices only.

When we modify the ROR configuration like that:

    - name: Logs ro access
      type: allow
      groups: ["logsro"]
      indices: ["logs*",".monitoring*"]
      actions: ["indices:monitor/settings/get", "indices:data/read/*", "cluster:monitor/state", "indices:admin/get", "indices:admin/mappings/fields/get", "indices:admin/mappings/get", "indices:admin/aliases/get", "indices:admin/template/get"]

we will see:

> curl -k -u alibaba:test  "http://127.0.0.1:9200/_cat/indices"
yellow open logs_20200202 3O2VThwHQ02Mw_VoIa6RXw 1 1 1 0 2.9kb 2.9kb
yellow open logs_20200201 daoSFCFeQBCulOCSQixfBA 1 1 1 0 2.9kb 2.9kb

But AFAIU the result is not also the one you expect. If you would like to see logs*, monitoring* and alibaba* indices, you have to put them inside the same block. ROR stops at the first matched block.

Or maybe I missed sth?

14

hmmm, you are right, i just tested it, but what if i need to have access to two indices but for example the first index will be readonly ?

In my case i need to have access to log indices but only in readonly state and also have ability to read and write to alibaba indices.

So i cannot use two access block ? What is the reason with groups then ? Or this is only true on cat indices ? Because for sure i can read the logs indices and write to alibaba, but i cannot write to logs as alibaba user, that why i expected that it will print all the indices that user have access to.

15

also the aliases is solved by this, because of the first match of the block.

But my opinion is that is should print all the indices in all blocks, because if you have like one readonly and the second write/read then you can see only readonly index.

I will change the block regarding to the alibaba to be first, because most relevant indices is alibaba indices, but i will except to see also logs and monitoring indices in cat response, because anyway i can brwoser them in discover tab.

Also you can see all the indices in monitoring tab, under the indices, and the user can see all the indices there, even those that dont have access to, but probably this is different case, because its monitoring.

16

to solve the problem with set of indices which are readonly and other set with write access, you can create two blocks. In each of them you can also use actions or uri_re rules to support only certain kinds of requests.

The problem is _cat API, where you’d like to see all indices. But to overcome this problem, you can create third block with indices from first and second block and action “indices:monitor/settings/get”.

Then alibaba user will call /_cat/indices, the third block will be matched and he will see all the indices defined in this block.

As you noticed, the same applies to _cat/aliases.

Our ACL works like this (for ES request, after first matched block, further blocks processing is interrupted and data collected in matched blocks are used to modify ES request) from the begging of ROR’s life.

Obviously we are open to introduce new rules (maybe indices rule with write options or sth). We rather don’t want to change current behaviour of existing rules (exception: wrong beheviour because of a bug), because backward compatibility is very important for us and we don’t want to break existing configurations during updates or force our users to reconfigure them because of our change. If you feel, I’d be nice to have sth new, you can create new topic with “new feature ideas” category and we will discuss it and potentially implement.

At the moment you can use the solution described above.
Do you think we can consider this topic as solved?

17

Yes, you are right, its understandable you dont want to change whole codebase for this.

Anyway, thank you very much for the support and explanations, we can close this case now, i will try to create those blocks or i will let the main block to be first, to see the needful indices.

Later, if i dont forget i will submit the new feature ideas.

1 Like