Free ROR 1.24.0 snapshots/get results in 401 authorization error

rodaj · November 18, 2020, 8:18pm

Attempts to get list of snapshots for repository fail on version 1.24.0 with authorization block defined below. Previous known working version is 1.18.10. Authorization block of person with full admin access can retrieve list of snapshots (i.e. not using any wildcard actions list).

- name: "Test producer"
  type: allow
  ldap_auth:
    name: "ldap2"
    groups: ["ldap_group_a"]
  actions: ["indices:data/*","indices:admin/*","indices:monitor/*","cluster:monitor/*","cluster:admin/snapshot/*"]
  indices: ["infratest*"]
  repositories: ["infratest"]

coutoPL · November 18, 2020, 10:40pm

What ES version do you use? We’ve fixed snapshot API handling in current, not released yet sprint, so I can send you a pre-build to test

rodaj · November 19, 2020, 12:40am

We’re using ES 7.9.1. That’d be great, we’d be happy to test fix.

coutoPL · November 19, 2020, 2:38pm

@rodaj please check this one:

https://readonlyrest-data.s3.amazonaws.com/build/1.25.0-pre9/readonlyrest-1.25.0-pre9_es7.9.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20201119/eu-west-1/s3/aws4_request&X-Amz-Date=20201119T143828Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=49dc74f7a8084a56cb0d12308800b051beb6364371a857755aadafe157552a40

rodaj · November 19, 2020, 5:21pm

patch downloaded, deployment to start shortly, should be able to post results by end of day.

rodaj · November 19, 2020, 8:20pm

No Joy.
Ansible snipet to drive test:

name: Get snapshots (success)
uri:
url: https://{{ cluster_url }}:{{ cluster_port }}/_snapshot/{{ test_repository }}/_all
method: GET
user: “{{ infra_producer_id }}”
password: “{{ infra_producer_pwd }}”
force_basic_auth: yes
validate_certs: no

ROR authorization block using specifically defined actions:
- name: “Test producer”
type: allow
ldap_auth:
name: “ldap2”
groups: [“es_producer”]
actions: [“indices:data/","indices:admin/”,“indices:admin/aliases”,“indices:admin/aliases/exists”,“indices:admin/aliases/get”,“indices:monitor/","cluster:monitor/”,“cluster:admin/snapshot/create”,“cluster:admin/snapshot/delete”,“cluster:admin/snapshot/get”,“cluster:admin/snapshot/restore”,“cluster:admin/snapshot/status”]
indices: [“infratest*”]
repositories: [“infratest”]

Test response:
TASK [Get snapshots (success)] *************************************************************************************************************************************************************************************
fatal: [tester.]: FAILED! => {“changed”: false, “connection”: “close”, “content”: “{“error”:{“root_cause”:[{“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”]}],“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”],“status”:401}}”, “content_length”: “205”, “content_type”: “application/json; charset=UTF-8”, “elapsed”: 0, “json”: {“error”: {“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”, “root_cause”: [{“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”}], “status”: 401}}, “msg”: “Status code was 401 and not [200]: HTTP Error 401: Unauthorized”, “redirected”: false, “status”: 401, “strict_transport_security”: “max-age=16000000”, “url”: “https://:/_snapshot/infratest/_all”, “www_authenticate”: “Basic”}

ROR authorization block using wildcard actions:
- name: “Test producer”
type: allow
ldap_auth:
name: “ldap2”
groups: [“es_producer”]
actions: [“indices:data/","indices:admin/”,“indices:monitor/","cluster:monitor/”,“cluster:admin/snapshot/"]
indices: ["infratest”]
repositories: [“infratest”]
Test response:
fatal: [tester.]: FAILED! => {“changed”: false, “connection”: “close”, “content”: “{“error”:{“root_cause”:[{“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”]}],“reason”:“Forbidden by ReadonlyREST ES plugin”,“due_to”:[“OPERATION_NOT_ALLOWED”],“status”:401}}”, “content_length”: “205”, “content_type”: “application/json; charset=UTF-8”, “elapsed”: 0, “json”: {“error”: {“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”, “root_cause”: [{“due_to”: [“OPERATION_NOT_ALLOWED”], “reason”: “Forbidden by ReadonlyREST ES plugin”}], “status”: 401}}, “msg”: “Status code was 401 and not [200]: HTTP Error 401: Unauthorized”, “redirected”: false, “status”: 401, “strict_transport_security”: “max-age=16000000”, “url”: “https://:/_snapshot/infratest/_all”, “www_authenticate”: “Basic”}

coutoPL · November 20, 2020, 1:28pm

are you able to show us “FORBIDDEN” log produced by ROR?

rodaj · November 20, 2020, 2:46pm

Here is couple of log entries with ip/host/port/userid redacted so it loses some of its “value”.

[2020-11-19T13:11:19,466][DEBUG][t.b.r.a.b.Block ] []^[[33m[Audit consumer] the request matches no rules in this block: { ID:1354758091-4610850#133421, TYP:GetRepositoriesRequest, CGR:N/A, USR: (attempted), BRS:true, KDX:null, ACT:cluster:admin/repository/get, OA:/32, XFF:null, DA:/32, IDX:<N/A>, MET:GET, PTH:/_snapshot, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]] } ^[[0m
[2020-11-19T13:11:19,466][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] []^[[35mFORBIDDEN by default req={ ID:1354758091-4610850#133421, TYP:GetRepositoriesRequest, CGR:N/A, USR: (attempted), BRS:true, KDX:null, ACT:cluster:admin/repository/get, OA:/32, XFF:null, DA:/32, IDX:<N/A>, MET:GET, PTH:/_snapshot, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[DataDog-> RULES:[groups->false], RESOLVED:[repositories=_all]], [cao_etm-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [cognet_consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [cognet_restore_prod_only-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [CEDP_Support-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [CEDP_Admin-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [ICI Producer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [ICI Consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Core Ingest-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Core readers-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Test producer-> RULES:[ldap_auth->true, repositories->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=infratest]], [Test consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]], [Audit producer-> RULES:[ldap_auth->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=_all]], [Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[repositories=_all]] }^[[0m

Function should match on this block, as the request specifically provide repository name so problem seems to be ROR thinking repository is _all, when in fact it should be for repository “infratest” only:

[Test producer-> RULES:[ldap_auth->true, repositories->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=infratest]]

coutoPL · November 23, 2020, 10:30am

@roday if the “Test producer” should be matched, let’s analyse it:

[Test producer-> RULES:[ldap_auth->true, repositories->true, actions->false], RESOLVED:[user=;group=es_producer;av_groups=es_producer;repositories=infratest]]

rules:

ldap_auth - was matched (user has been authenticated)
repositories - was matched (the rule narrowed request for _all repositories, to “infratest” repository, because according to this rule, the authenticated user has access only to this one)
actions - this was MISMATCHED - as you can see action of this request is cluster:admin/repository/get (see log ACT:cluster:admin/repository/get), but your rule allows only ["indices:data/*","indices:admin/*","indices:monitor/*","cluster:monitor/*","cluster:admin/snapshot/*"]

So, IMO everything works fine, but you have to add cluster:admin/repository/* to actions array.

rodaj · November 23, 2020, 2:53pm

As the rest api requests was _snapshots there is nothing obvious about “repository” rule. That is going to be a source of confusion forever. I am off this week, so I won’t be able to investigate and determine exactly the rules we’ll want with cluster:admin/repository to retain our desired security until next week. Thanks for the help.

coutoPL · November 23, 2020, 7:36pm

yeah, but AFAIK /_snapshot REST endpoint fetches all snapshot repositories, not all snapshots (see: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/snapshots-register-repository.html#snapshots-repository-plugins). To get all snapshots you should use this one: https://www.elastic.co/guide/en/elasticsearch/reference/current/get-snapshot-api.html

rodaj · November 30, 2020, 3:38pm

your response does not make sense. The request is using the get-snapshot-api.

I have added to authorization block “cluster:admin/repository/get” and now the error I see in logs suggests issues with indices.

[2020-11-30T14:44:31,232][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] []^[[35mINDEX NOT FOUND req={ ID:547342397-813283343#573, TYP:GetSnapshotsRequest, CGR:N/A, USR: (attempted), BRS:true, KDX:null, ACT:cluster:admin/snapshot/get, OA:/32, XFF:null, DA:/32, IDX:, MET:GET, PTH:/_snapshot/infratest/_all, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[DataDog-> RULES:[groups->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cao_etm-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_restore_prod_only-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Support-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Admin-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Producer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core Ingest-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core readers-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Test producer-> RULES:[ldap_auth->true, repositories->true, actions->true, indices->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Test consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Audit producer-> RULES:[ldap_auth->true, actions->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=*;repositories=infratest;snapshots=_all]] }^[[0m

coutoPL · November 30, 2020, 9:01pm

hey, dude, wait a sec. In the previous log you showed I see:

PTH:/_snapshot,

now,

PTH:/_snapshot/infratest/_all

(so now, you are showing us different test scenario).

In the doc (linked in my previous comment) I see:

and

For me it makes sense, doesn’t it?

coutoPL · November 30, 2020, 9:29pm

and also please check out the newest ROR 1.25.0. There was a fix related to snapshots _all (see: https://readonlyrest.com/download/)

rodaj · November 30, 2020, 9:31pm

I have not changed the test, it has been the exact same request each time.

The ONLY change to the ENTIRE testcase was for my mistake with aliases not having proper prefix for name.

rodaj · November 30, 2020, 9:35pm

This is the version of ansible/python/OS that ansible playbook testcase is run from:

ansible --version
ansible 2.9.6
config file = /home/rodaj_us_ibm_com/.ansible.cfg
configured module search path = [’/home/rodaj_us_ibm_com/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Dec 5 2019, 15:45:45) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]

rodaj · November 30, 2020, 9:43pm

As per my previous comment (the ansible for the specific test…all indentation lost):

coutoPL · December 1, 2020, 2:48pm

@rodaj can you confirm that ROR 1.25.0 behaves the same?

rodaj · December 1, 2020, 11:38pm

Error message in log slightly different in that it clearly states problems related to index. For whatever reason the authorization logic is getting fixated on index and while there is index information in snapshot the request returned should just withhold that data…not block the request entirely…otherwise everyone would have to provide an authorization block with * for indices just for repository/snapshot activity…and honestly that don’t fit our authorization model all that well. I will experiment tomorrow with some different authorization block definitions, as I suspect changing the contents of actions changes the authorization logic…i.e. the reason for the earlier confusion related to snapshot vs snapshot rest api…

[2020-12-01T23:26:06,115][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] []^[[35mINDEX NOT FOUND req={ ID:1931640505-573558696#40277, TYP:GetSnapshotsRequest, CGR:N/A, USR:(attempted), BRS:true, KDX:null, ACT:cluster:admin/snapshot/get, OA:/32, XFF:null, DA:/32, IDX:, MET:GET, PTH:/_snapshot/infratest/_all, CNT:<N/A>, HDR:Accept-Encoding=identity, Authorization=, Connection=close, Host=:, User-Agent=ansible-httpget, content-length=0, HIS:[DataDog-> RULES:[groups->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cao_etm-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [cognet_restore_prod_only-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [CEDP_Support-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [CEDP_Admin-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Producer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [ICI Consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core Ingest-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Core readers-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [Test producer-> RULES:[ldap_auth->true, repositories->true, actions->true, indices->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Test consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=;repositories=infratest;snapshots=_all]], [CEDP Infra Audit producer-> RULES:[ldap_auth->true, actions->false], RESOLVED:[user=;group=TEST_es_producer;av_groups=TEST_es_producer;indices=;repositories=infratest;snapshots=_all]], [Audit consumer-> RULES:[ldap_auth->false], RESOLVED:[indices=*;repositories=infratest;snapshots=_all]] }^[[0m

rodaj · December 2, 2020, 2:20pm

Confirmed 2 things today. First, action “cluster:admin/repository/get” was never required and in fact causes a problem with security, as it results in complete by-pass of the repositories restriction on authorization block allowing users to list contents of repositories that they should not be allowed to.
Second, confirmed the bug (which I might add has been a frequent re-occurrence in ROR over the years for numerous actions…weakness in test coverage in my humble opinion) is with the indices filter…it will only allow the action to get list of all snapshots for a specific repository if and only if the indices filter is “*”.

I can “temporarily” work around the problem by creating a separate authorization block similar to following:

# temporary ROR bug work around
    - name: "Test get snapshots only"
      type: allow
      ldap_auth:
        name: "ldap2"
        groups: ["TEST_es_producer"]
      actions: ["cluster:admin/snapshot/get"]
      indices: ["*"]
      repositories: ["infratest"]