ROR returns 401 rather than 404 if access to an index is allowed but the index does not exist

Hi, Simone,
one of our users found a nice bug in ROR. He’s using python-elasticsearch and does this:
es.indices.exists(‘intentp1_some-nice-index-he_is_allowed_to_access’)
If the index exists everything is fine. If it does not exists he gets a 401 instead of a 404 which results in a python error, and this blocks him.

Reproducer:

We are using ror 1.16.8. The block which is matched looks like this

  • accept_x-forwarded-for_header: true
    groups:
    • intentp1_rw
      hosts:
    • es-intentp1
      indices:
    • intentp1*
    • ".kibana_intentp1"
      name: es-intentp1 writer
      type: allow

When the indices: rule is removed from this block the correct 404 answer is returned.

Would it be possible to fix this ?

Hello @schwicke!

Yes this bug is quite a hidden and was discovered a few weeks back (in crazy circumstances) by @ld57. I fixed this in 1.16.9 which is about to be released together with a nice pack of fixes :slight_smile:

That sounds very good. Do you already have an ETA for 1.16.9?
Thanks a lot!

Yes, the ETA is now -2.5h :smile:

Thanks a lot for the fix! I’ve packaged the new version and it fixes the issue.

Unfortunately, we have to roll back because it introduces a new issue :
Whenever there is a closed index matching a pattern in kibana, the refresh of the index pattern no longer works. Even worse, Kibana breaks if an index pattern matches a closed index because it cannot access the types. This screws up all clusters which have closed indices.

Looking at the debugging output of ROR the access is not blocked. It seems rather as if a wrong list of indices is being returned.

Can I have a small test case to work on? Will take care of this ASAP.

We reproduced it on our test cluster. Steps:

  • create a kibana user which has restricted access to indices in the cluster. In our case the rule which is matched for him is
  - accept_x-forwarded-for_header: true
    api_keys:
    - kibana_intentp1_rw
    hosts:
    - es-intentp1
    indices:
    - intentp1-*
    - intentp1_*
    - ".kibana"
    - ".kibana_intentp1"
    kibana_access: rw
    kibana_index: ".kibana_intentp1"
    name: es-intentp1 kibana_writer
    type: allow
  • create a couple of indices with documents matching this pattern, intentp1_index1 and intentp1_index2
  • create a new index pattern in kibana for intentp1_*
  • refresh the index pattern -> that works
  • close one of the indices
  • refresh the index pattern -> all types are removed.
  • reopen the matched index
  • refresh the index patter -> works again.

let me know if you can reproduce it this way.

1 Like

@schwicke thanks for the amazing test script. I managed to reproduce and I have committed a fix!

You can see the fix (now only for 5.5.x and 5.4.x) in ROR 1.16.10-pre1

Hi, Simone,
many thanks for the fast fix! I have packaged the new version and deployed it on our test cluster. Unfortunately, I can still reproduce the issue.

Strange, just checked again and it’s working on my env. What version are you building for?

We’re on 5.4.0. The patch should be in there.
I’ll try to manually replace our binaries with yours. Could you give me a pointer to your binary version of readonlyrest-1.16.10-pre1_es5.4.0.jar ? By mail is fine.

Sure, in the meanwhile I write an integration test just for this so we can reason about it.

https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.16.10-pre1/readonlyrest-1.16.10-pre1_es5.4.0.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ/20170807/eu-west-1/s3/aws4_request&X-Amz-Date=20170807T112315Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=659549c858bf25139eeb277d835ce81c50b75df8f22c882b63f81650df04de07

It seems the link has already expired. I get ‘AccessDenied’.
Having an integration test on this may help. Keep in touch.

Sorry, now I have fixed the link!

Thanks, I managed to download it now! Same issue. I’ll try to get some debugging information to see what is going wrong. More news asap.

1 Like

Unfortunately, it is not fixed. I managed to strip down further, to have a better reproducer. I made this test with your binaries, and I get the same if I compile them myself.

I have 2 indices:

          close intentp1_licmon-licences2     LFVvISGISvWIlKkjH7eqPA                                 
green open  intentp1_licmon-licences      WC92f-p6TuGFNApe77S_fQ 5 1        2  0   19.1kb   9.5kb

and this block:

  - accept_x-forwarded-for_header: true
    groups:
    - intentp1_rw
    hosts:
    - es-intentp1
    indices:
    - intentp1-*
    - intentp1_*
    - ".kibana_intentp1"
    name: es-intentp1 writer

and

  - groups:
    - intentp1_rw
    proxy_auth: intentp1
    username: intentp1

I get:

# curl -D - -H 'X-Forwarded-For: es-intentp1' -H 'X-Forwarded-User: intentp1' 'http://127.0.0.1:9200/intentp1_licmon*/_mapping/field/*' --compressedHTTP/1.1 404 Not Found
content-type: application/json; charset=UTF-8
content-encoding: gzip
transfer-encoding: chunked

{}

ROR info mode logs:

[2017-08-07T17:49:04,941][INFO ][org.elasticsearch.plugin.readonlyrest.acl.ACL] ALLOWED by '{ block=es-intentp1 writer, match=true }' req={ ID:1062469736-670358775#8643, TYP:GetFieldMappingsRequest, USR:intentp1, BRS:true, ACT:indices:admin/mappings/fields/get, OA:127.0.0.1, IDX:intentp1_licmon-licences2,intentp1_licmon-licences, MET:GET, PTH:/intentp1_licmon*/_mapping/field/*, CNT:<OMITTED, LENGTH=0>, HDR:Accept,Accept-Encoding,content-length,Host,User-Agent,X-Forwarded-For,X-Forwarded-User, HIS:[es-internal kibana_short_url->[hosts->true, uri_re->false]], [es-intentp1 perfmon_monitor->[groups->false]], [es-intentp1 perfmon_writer->[groups->false]], [es-internal kibana_writer->[kibana_access->true, hosts->true, api_keys->false]], [es-intentp1 reader->[groups->false]], [es-intentp1 writer->[indices->true, proxy_auth->true, hosts->true]], [es-internal reader->[groups->false]], [es-internal kibana_reader->[kibana_access->true, hosts->true, api_keys->false]], [es-internal perfmon_writer->[groups->false]], [es-internal perfmon_monitor->[groups->false]], [es-internal writer->[groups->false]] } 
[2017-08-07T17:49:04,942][INFO ][org.elasticsearch.plugin.readonlyrest.acl.ACL] ALLOWED by '{ block=es-intentp1 writer, match=true }' req={ ID:1062469736-1105794931#8644, TYP:GetFieldMappingsIndexRequest, USR:intentp1, BRS:true, ACT:indices:admin/mappings/fields/get[index], OA:127.0.0.1, IDX:intentp1_licmon-licences2,intentp1_licmon-licences, MET:GET, PTH:/intentp1_licmon*/_mapping/field/*, CNT:<OMITTED, LENGTH=0>, HDR:Accept,Accept-Encoding,content-length,Host,User-Agent,X-Forwarded-For,X-Forwarded-User, HIS:[es-internal kibana_short_url->[hosts->true, uri_re->false]], [es-internal kibana_reader->[kibana_access->true, hosts->true, api_keys->false]], [es-intentp1 writer->[indices->true, proxy_auth->true, hosts->true]], [es-internal perfmon_monitor->[groups->false]], [es-internal writer->[groups->false]], [es-intentp1 perfmon_writer->[groups->false]], [es-intentp1 reader->[groups->false]], [es-internal kibana_writer->[kibana_access->true, hosts->true, api_keys->false]], [es-internal reader->[groups->false]], [es-intentp1 perfmon_monitor->[groups->false]], [es-internal perfmon_writer->[groups->false]] } 
[2017-08-07T17:49:04,943][INFO ][org.elasticsearch.plugin.readonlyrest.acl.ACL] ALLOWED by '{ block=es-intentp1 writer, match=true }' req={ ID:1062469736-276570509#8645, TYP:GetFieldMappingsIndexRequest, USR:intentp1, BRS:true, ACT:indices:admin/mappings/fields/get[index], OA:127.0.0.1, IDX:intentp1_licmon-licences2,intentp1_licmon-licences, MET:GET, PTH:/intentp1_licmon*/_mapping/field/*, CNT:<OMITTED, LENGTH=0>, HDR:Accept,Accept-Encoding,content-length,Host,User-Agent,X-Forwarded-For,X-Forwarded-User, HIS:[es-internal perfmon_writer->[groups->false]], [es-intentp1 perfmon_monitor->[groups->false]], [es-internal kibana_writer->[kibana_access->true, hosts->true, api_keys->false]], [es-intentp1 writer->[indices->true, proxy_auth->true, hosts->true]], [es-intentp1 perfmon_writer->[groups->false]], [es-internal perfmon_monitor->[groups->false]], [es-internal kibana_reader->[kibana_access->true, hosts->true, api_keys->false]], [es-intentp1 reader->[groups->false]], [es-internal reader->[groups->false]], [es-internal kibana_short_url->[hosts->true, uri_re->false]], [es-internal writer->[groups->false]] } 

After opening the closed index:

curl -s -XPOST localhost:9200/intentp1_licmon-licences2/_open
{"acknowledged":true}

I get the expected result

# curl -D - -H 'X-Forwarded-For: es-intentp1' -H 'X-Forwarded-User: intentp1' 'http://127.0.0.1:9200/intentp1_licmon*/_mapping/field/*' --compressed
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
transfer-encoding: chunked

{"intentp1_licmon-licences2":{"mappings":{"mytype":{"_routing":{"full_name":"_routing","mapping":{}},"_ttl":{"full_name":"_ttl","mapping":{}},"something.keyword":{"full_name":"something.keyword","mapping":{"keyword":{"type":"keyword","ignore_above":256}}},"_index":{"full_name":"_index","mapping":{}},"_type":{"full_name":"_type","mapping":{}},"_all":{"full_name":"_all","mapping":{}},"something":{"full_name":"something","mapping":{"something":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"_parent":{"full_name":"_parent","mapping":{}},"_field_names":{"full_name":"_field_names","mapping":{}},"_source":{"full_name":"_source","mapping":{}},"_id":{"full_name":"_id","mapping":{}},"_timestamp":{"full_name":"_timestamp","mapping":{}},"_uid":{"full_name":"_uid","mapping":{}},"_version":{"full_name":"_version","mapping":{}}}}}}

Any ideas ? Thanks a lot!

Almost done with the integration test (it takes long, sorry). For now I’m testing with search, but I will add a test case similar to your curl.

Hey @schwicke!
The test helped out, I found a better fix for this issue and I fixed also two other minor bugs. The new reference build is 1.16.10-pre3

Hi, Simone!
Great! This issue is fixed. There is something odd though. The new version no longer shows closed indices. Look at this:

# curl -D - -H 'X-Forwarded-For: es-intentp1' -H 'X-Forwarded-User: intentp1' 'http://127.0.0.1:9200/_cat/indices' --compressed
HTTP/1.1 200 OK
content-type: text/plain; charset=UTF-8
content-encoding: gzip
transfer-encoding: chunked

green open intentp1_licmon-licences  WC92f-p6TuGFNApe77S_fQ 5 1  2 0 19.1kb 9.5kb
green open intentp1-test             rPbYge41QWSXWBI01Bf-mQ 5 1  1 0  7.7kb 3.8kb
green open .kibana_intentp1          67x-xwDFQDWgXNzYcN9iMw 5 1 21 3  178kb  89kb
green open intentp1_licmon-licences2 LFVvISGISvWIlKkjH7eqPA 5 1  1 0  7.9kb 3.9kb

# curl -s -XPOST localhost:9200/intentp1_licmon-licences2/_close
{"acknowledged":true}

# curl -D - -H 'X-Forwarded-For: es-intentp1' -H 'X-Forwarded-User: intentp1' 'http://127.0.0.1:9200/_cat/indices' --compressed
HTTP/1.1 200 OK
content-type: text/plain; charset=UTF-8
content-encoding: gzip
transfer-encoding: chunked

green open intentp1_licmon-licences WC92f-p6TuGFNApe77S_fQ 5 1  2 0 19.1kb 9.5kb
green open .kibana_intentp1         67x-xwDFQDWgXNzYcN9iMw 5 1 21 3  178kb  89kb
green open intentp1-test            rPbYge41QWSXWBI01Bf-mQ 5 1  1 0  7.7kb 3.8kb

Checking back the previous iterations:
1.16.9 was OK
1.16.10-pre1 was OK
1.16.10-pre3 NOT OK

so this is a new feature of pre3 indeed.

Simone, please let me know if you prefer to have a new ticket for the index list issue in pre3.