Free ROR 1.24.0 snapshots/get results in 401 authorization error

Attempts to get list of snapshots for repository fail on version 1.24.0 with authorization block defined below. Previous known working version is 1.18.10. Authorization block of person with full admin access can retrieve list of snapshots (i.e. not using any wildcard actions list).

- name: "Test producer"
  type: allow
  ldap_auth:
    name: "ldap2"
    groups: ["ldap_group_a"]
  actions: ["indices:data/*","indices:admin/*","indices:monitor/*","cluster:monitor/*","cluster:admin/snapshot/*"]
  indices: ["infratest*"]
  repositories: ["infratest"]

What ES version do you use? We’ve fixed snapshot API handling in current, not released yet sprint, so I can send you a pre-build to test

We’re using ES 7.9.1. That’d be great, we’d be happy to test fix.

@rodaj please check this one:

https://readonlyrest-data.s3.amazonaws.com/build/1.25.0-pre9/readonlyrest-1.25.0-pre9_es7.9.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20201119/eu-west-1/s3/aws4_request&X-Amz-Date=20201119T143828Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=49dc74f7a8084a56cb0d12308800b051beb6364371a857755aadafe157552a40

patch downloaded, deployment to start shortly, should be able to post results by end of day.

1 Like

No Joy.
Ansible snipet to drive test:

  • name: Get snapshots (success)
    uri:
    url: https://{{ cluster_url }}:{{ cluster_port }}/_snapshot/{{ test_repository }}/_all
    method: GET
    user: “{{ infra_producer_id }}”
    password: “{{ infra_producer_pwd }}”
    force_basic_auth: yes
    validate_certs: no

ROR authorization block using specifically defined actions:
- name: “Test producer”
type: allow
ldap_auth:
name: “ldap2”
groups: [“es_producer”]
actions: [“indices:data/","indices:admin/”,“indices:admin/aliases”,“indices:admin/aliases/exists”,“indices:admin/aliases/get”,“indices:monitor/","cluster:monitor/”,“cluster:admin/snapshot/create”,“cluster:admin/snapshot/delete”,“cluster:admin/snapshot/get”,“cluster:admin/snapshot/restore”,“cluster:admin/snapshot/status”]
indices: [“infratest*”]
repositories: [“infratest”]

Test response:
TASK [Get snapshots (success)] *************************************************************************************************************************************************************************************
fatal: [tester.]: FAILED! => {“changed”: false, “connection”: “close”, “content”: “{“error”:{“root_cause”:[{“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”]}],“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”],“status”:401}}”, “content_length”: “205”, “content_type”: “application/json; charset=UTF-8”, “elapsed”: 0, “json”: {“error”: {“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”, “root_cause”: [{“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”}], “status”: 401}}, “msg”: “Status code was 401 and not [200]: HTTP Error 401: Unauthorized”, “redirected”: false, “status”: 401, “strict_transport_security”: “max-age=16000000”, “url”: “https://:/_snapshot/infratest/_all”, “www_authenticate”: “Basic”}

ROR authorization block using wildcard actions:
- name: “Test producer”
type: allow
ldap_auth:
name: “ldap2”
groups: [“es_producer”]
actions: [“indices:data/","indices:admin/”,“indices:monitor/","cluster:monitor/”,“cluster:admin/snapshot/"]
indices: ["infratest
”]
repositories: [“infratest”]
Test response:
fatal: [tester.]: FAILED! => {“changed”: false, “connection”: “close”, “content”: “{“error”:{“root_cause”:[{“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”]}],“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”],“status”:401}}”, “content_length”: “205”, “content_type”: “application/json; charset=UTF-8”, “elapsed”: 0, “json”: {“error”: {“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”, “root_cause”: [{“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”}], “status”: 401}}, “msg”: “Status code was 401 and not [200]: HTTP Error 401: Unauthorized”, “redirected”: false, “status”: 401, “strict_transport_security”: “max-age=16000000”, “url”: “https://:/_snapshot/infratest/_all”, “www_authenticate”: “Basic”}

are you able to show us “FORBIDDEN” log produced by ROR?

Here is couple of log entries with ip/host/port/userid redacted so it loses some of its “value”.

[2020-11-19T13:11:19,466][DEBUG][t.b.r.a.b.Block ] []^[[33m[Audit consumer] the request matches no rules in this block: { ID:1354758091-4610850#133421, TYP:GetRepositoriesRequest, CGR:N/A, USR: (attempted), BRS:true, KDX:null, ACT:cluster:admin/repository/get, OA:/32, XFF:null, DA:/32, IDX:<N/A>, MET:GET, PTH:/_snapshot, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]] } ^[[0m
[2020-11-19T13:11:19,466][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] []^[[35mFORBIDDEN by default req={ ID:1354758091-4610850#133421, TYP:GetRepositoriesRequest, CGR:N/A, USR: (attempted), BRS:true, KDX:null, ACT:cluster:admin/repository/get, OA:/32, XFF:null, DA:/32, IDX:<N/A>, MET:GET, PTH:/_snapshot, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[DataDog-> RULES:[groups->false], RESOLVED:[repositories=_all]], [cao_etm-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [cognet_consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [cognet_restore_prod_only-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [CEDP_Support-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [CEDP_Admin-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [ICI Producer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [ICI Consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Core Ingest-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Core readers-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Test producer-> RULES:[ldap_auth->true, repositories->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=infratest]], [Test consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Audit producer-> RULES:[ldap_auth->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=_all]], [Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]] }^[[0m

Function should match on this block, as the request specifically provide repository name so problem seems to be ROR thinking repository is _all, when in fact it should be for repository “infratest” only:

[Test producer-> RULES:[ldap_auth->true, repositories->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=infratest]]

@roday if the “Test producer” should be matched, let’s analyse it:

[Test producer-> RULES:[ldap_auth->true, repositories->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=infratest]]

rules:

  • ldap_auth - was matched (user has been authenticated)
  • repositories - was matched (the rule narrowed request for _all repositories, to “infratest” repository, because according to this rule, the authenticated user has access only to this one)
  • actions - this was MISMATCHED - as you can see action of this request is cluster:admin/repository/get (see log ACT:cluster:admin/repository/get), but your rule allows only ["indices:data/*","indices:admin/*","indices:monitor/*","cluster:monitor/*","cluster:admin/snapshot/*"]

So, IMO everything works fine, but you have to add cluster:admin/repository/* to actions array.

As the rest api requests was _snapshots there is nothing obvious about “repository” rule. That is going to be a source of confusion forever. I am off this week, so I won’t be able to investigate and determine exactly the rules we’ll want with cluster:admin/repository to retain our desired security until next week. Thanks for the help.

yeah, but AFAIK /_snapshot REST endpoint fetches all snapshot repositories, not all snapshots (see: Register a snapshot repository | Elasticsearch Guide [7.10] | Elastic). To get all snapshots you should use this one: Get snapshot API | Elasticsearch Guide [8.6] | Elastic

your response does not make sense. The request is using the get-snapshot-api.

I have added to authorization block “cluster:admin/repository/get” and now the error I see in logs suggests issues with indices.

[2020-11-30T14:44:31,232][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] []^[[35mINDEX NOT FOUND req={ ID:547342397-813283343#573, TYP:GetSnapshotsRequest, CGR:N/A, USR: (attempted), BRS:true, KDX:null, ACT:cluster:admin/snapshot/get, OA:/32, XFF:null, DA:/32, IDX:, MET:GET, PTH:/_snapshot/infratest/_all, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[DataDog-> RULES:[groups->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cao_etm-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_restore_prod_only-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Support-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Admin-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Producer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core Ingest-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core readers-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Test producer-> RULES:[ldap_auth->true, repositories->true, actions->true, indices->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Test consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Audit producer-> RULES:[ldap_auth->true, actions->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=*;repositories=infratest;snapshots=_all]] }^[[0m

hey, dude, wait a sec. In the previous log you showed I see:

PTH:/_snapshot,

now,

PTH:/_snapshot/infratest/_all

(so now, you are showing us different test scenario).

In the doc (linked in my previous comment) I see:

and

For me it makes sense, doesn’t it?

and also please check out the newest ROR 1.25.0. There was a fix related to snapshots _all (see: Download (UNIVERSAL) - ReadonlyREST)

I have not changed the test, it has been the exact same request each time.

The ONLY change to the ENTIRE testcase was for my mistake with aliases not having proper prefix for name.

This is the version of ansible/python/OS that ansible playbook testcase is run from:

ansible --version
ansible 2.9.6
config file = /home/rodaj_us_ibm_com/.ansible.cfg
configured module search path = [’/home/rodaj_us_ibm_com/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Dec 5 2019, 15:45:45) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

As per my previous comment (the ansible for the specific test…all indentation lost):

@rodaj can you confirm that ROR 1.25.0 behaves the same?

Error message in log slightly different in that it clearly states problems related to index. For whatever reason the authorization logic is getting fixated on index and while there is index information in snapshot the request returned should just withhold that data…not block the request entirely…otherwise everyone would have to provide an authorization block with * for indices just for repository/snapshot activity…and honestly that don’t fit our authorization model all that well. I will experiment tomorrow with some different authorization block definitions, as I suspect changing the contents of actions changes the authorization logic…i.e. the reason for the earlier confusion related to snapshot vs snapshot rest api…

[2020-12-01T23:26:06,115][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] []^[[35mINDEX NOT FOUND req={ ID:1931640505-573558696#40277, TYP:GetSnapshotsRequest, CGR:N/A, USR:(attempted), BRS:true, KDX:null, ACT:cluster:admin/snapshot/get, OA:/32, XFF:null, DA:/32, IDX:, MET:GET, PTH:/_snapshot/infratest/_all, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[DataDog-> RULES:[groups->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cao_etm-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_restore_prod_only-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [CEDP_Support-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [CEDP_Admin-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Producer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core Ingest-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core readers-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Test producer-> RULES:[ldap_auth->true, repositories->true, actions->true, indices->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Test consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [CEDP Infra Audit producer-> RULES:[ldap_auth->true, actions->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=*;repositories=infratest;snapshots=_all]] }^[[0m

Confirmed 2 things today. First, action “cluster:admin/repository/get” was never required and in fact causes a problem with security, as it results in complete by-pass of the repositories restriction on authorization block allowing users to list contents of repositories that they should not be allowed to.
Second, confirmed the bug (which I might add has been a frequent re-occurrence in ROR over the years for numerous actions…weakness in test coverage in my humble opinion) is with the indices filter…it will only allow the action to get list of all snapshots for a specific repository if and only if the indices filter is “*”.

I can “temporarily” work around the problem by creating a separate authorization block similar to following:

# temporary ROR bug work around
    - name: "Test get snapshots only"
      type: allow
      ldap_auth:
        name: "ldap2"
        groups: ["TEST_es_producer"]
      actions: ["cluster:admin/snapshot/get"]
      indices: ["*"]
      repositories: ["infratest"]