Authorization failed on ES-7.8.0 with ROR-1.20.0

Users Feedback
1

Environment: RHEL 5.11, JDK 1.8.0_65

I setup new ES-7.8 cluster with 2 hosts, 2 master and 2 data nodes on each host
I also installed ROR 1.20.0 (readonlyrest-1.20.0_es7.8.0.jar) on one master node

My readonlyrest.yaml is as follows:

    readonlyrest:
        enable: true
        response_if_req_forbidden: Access denied by ROR
        access_control_rules:
        - name: "Allow read/write all indexes for the vadim user only"
          type: allow
          groups: ["team1"]
          actions: ["cluster:monitor/*","indices:data/read*","indices:data/write*","indices:admin/create"]
          indices: ["*"]
        users:
        - username: vadim
          auth_key_sha256: ab...yz
          groups: ["team1"]

For the pair vadim:secpwd I generated the values like abc…xyz for auth_key_sha256 and MN…RS for Base64
Restarted the the master node with ROR successfully.
Checked cluster health with request to not ROR protected node - showed 4 nodes joined and health=GREEN

Requested ROR protected node on Linux command line
curl -XGET http://host:port/_cat/indices?v -H ‘Authorization: Basic MN…RS’
Output
{“error”:{“root cause” … “reason”:“Access denied by ROR” , node…, cluster… “due_to” [“OPERATION_NOT_ALLOWED”]… “status”:401}

ES log file:
…
…xyz_node started
…Loading ReadonlyREST settings from index failed: cannot find index
…Loading ReadonlyREST from file: /path_to_file/readonlyrest.yml
…xyz_node ADDING BLOCK: {name: ‘Allow read/write all indexes for the vadim user only’, policy: ALLOW, rules: [groups,actions,indices]
…xyz_node Readonly REST plugin core was loaded …
…xyz_node FPORBIDDEN by default req={ ID:…, TYP:GetSettingsRequest, CGR:N/A, USR:vadim (attempted), BRS:true, KDX:null, ACT:indices:monitor/settings/get, OA:…, XFF:null, DA:…, IDX:, MET:GET, PTH: /_cut/indices, CNT:<N/A>, HDR:Accept=/, Authorization=, Host=some_host/some_port, User-Agent=curl…OpenSSL/…, content-length=0, HIS:[Allow read/write all indexes for the vadim user only-> RULES:[groups->false], RESOLVED:[indices=]]}

The same readonlyrest.yml configuration worked successfully without issues on the older ES-5.6.3 with ROR 1.16.15 (readonlyrest-1.16.15_es5.6.30.jar)

Any help on this issue would be appreciated

Thanks

2

Your log line mentions an action name that is not included in your list of allowed actions

3

Thanks a lot Simone for your response

Followed your advise I tried adding to list of allowed actions in my readonlyrest.yaml the “indices:monitor/settings/get” and "indices:monitor/"
Restared ROR protected node and
curl -XGET http://host:port/_cat/indices?v -H ‘Authorization: Basic MN…RS’
gave me the same response
{“error”:{“root cause” … “reason”:“Access denied by ROR” , node…, cluster… “due_to” [“OPERATION_NOT_ALLOWED”]… “status”:401}
The ES log showed the same information as before changes
…xyz_node FORBIDDEN by default req={ ID:…, TYP:GetSettingsRequest, CGR:N/A, USR:vadim (attempted), BRS:true, KDX:null, ACT:indices:monitor/settings/get, OA:…, XFF:null, DA:…, IDX:, MET:GET, PTH: /_cut/indices, CNT:<N/A>, HDR:Accept=/, Authorization=, Host=some_host/some_port, User-Agent=curl…OpenSSL/…, content-length=0, HIS:[Allow read/write all indexes for the vadim user only-> RULES:[groups->false], RESOLVED:[indices=*]]}

My concerns are about the "FORBIDDEN by default" and “Authorization=OMITTED” in the ES log and I have a filling may be some default value(s) need to be overriden somehow/somewhere, may be in the readonlyrest.yml or elasticsearch.yml (?)
Making url REST requests to Not ROR protected node, like list indices, list nodes, cluster health, return good responses.
The same requests to ROR protected node with -H ‘Authorization: Basic MN…RS’ gaves response “Access denied by ROR” and the …xyz_node FORBIDDEN by default req={… in the ES log

I did some reserach and not found any specifics/bugs/issues in regards to current ES-7.8 release along with readonlyrest-1.20.0_es7.8.0.jar

Any other ideas, hints, prompts what else might be checked on, changed, tested or whatever will be appreciated

I posted this topic on ROR forum and github ROR plugin because I was not sure which one is more appropriate, was not sure I’m experiencing the bug issue or ROR misconfiguration, sorry for the inconvenience. If you let me know about it then I’ll remove my post where it is not relevant to the forum content

Thanks in advance

4

I think I know what is a problem. Try this:

 readonlyrest:
        enable: true
        response_if_req_forbidden: Access denied by ROR
        access_control_rules:
        - name: "Allow read/write all indexes for the vadim user only"
          type: allow
          groups: ["team1"]
          actions: ["cluster:monitor/*","indices:data/read*","indices:data/write*","indices:admin/create"]
          indices: ["*"]
        users:
        - username: vadim
          auth_key_sha256: vadim:ab...yz
          groups: ["team1"]

auth_key_sha256 in users section is the same as auth_key_sha256 rule. It requires value in following format: user_name:sha256hash.

It’s weird that ROR is able to start (it should figure out that config is malformed)

5

Thank you for your efforts and desire to help on this issue Mateusz.
I tried auth_key_sha256: vadim:ab…yz and it didn’t help, getting the same ES log. I’m not sure why the ES log contains the "FORBIDDEN by default " and “Authorization=OMITTED” and what does this mean ?

6

@dzyubanv I’ll check your case in our integration tests this week. Will let you know

7

Thanks a lot Mateusz!
Whatever I continued to experiment so far didn’t help me

8

@dzyubanv

I’ve checked your case.

ES 7.8.0, ROR 1.21.0-pre5

readonlyrest.yml

readonlyrest:
  access_control_rules:

    - name: "Allow read/write all indexes for the vadim user only"
      type: allow
      groups: ["team1"]
      actions: ["indices:monitor/*","cluster:monitor/*","indices:data/read*","indices:data/write*","indices:admin/create"]
      indices: ["*"]

  users:
    - username: vadim
      # vadim:test
      auth_key_sha256: vadim:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 
      groups: ["team1"]

TEST 1 (success case):

curl -u vadim:test -H "Content-Type: application/json" "http://127.0.0.1:9200/_cat/indices"                                                                                         18:47:18
yellow open app-tomcat cUGXywOtQ9Whjr9I-Jc7vQ 1 1 2 0  7.5kb  7.5kb
green  open .kibana_1  Jyb1bCRUTwWtlPjY6AQoow 1 0 9 0 34.5kb 34.5kb
yellow open app-cste   qHX1_VOiTpu9OF3TqhbzFQ 1 1 2 0  7.4kb  7.4kb
yellow open app-new    aj_ncl9hQ7K8WtIjqRSE4g 1 1 1 0  4.1kb  4.1kb

logs:

[2020-07-20T18:46:30,270][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [n1_it] ALLOWED by { name: 'Allow read/write all indexes for the vadim user only', policy: ALLOW, rules: [groups,actions,indices] req={  ID:379970762--919457646#223,  TYP:GetSettingsRequest,  CGR:N/A,  USR:vadim,  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=Basic dmFkaW06dGVzdA==, Content-Type=application/json, Host=127.0.0.1:9200, User-Agent=curl/7.54.0, content-length=0,  HIS:[Allow read/write all indexes for the vadim user only-> RULES:[groups->true, actions->true, indices->true], RESOLVED:[user=vadim;group=team1;av_groups=team1;indices=*]]  }

TEST 2 (wrong password):

curl -u vadim:test1 -H "Content-Type: application/json" "http://127.0.0.1:9200/_cat/indices"                                                                                        
{"error":{"root_cause":[{"reason":"forbidden","due_to":["OPERATION_NOT_ALLOWED"]}],"reason":"forbidden","due_to":["OPERATION_NOT_ALLOWED"],"status":401}}

logs:

[2020-07-20T18:47:18,802][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [n1_it] FORBIDDEN by default req={  ID:1429885149--919457646#240,  TYP:GetSettingsRequest,  CGR:N/A,  USR:vadim (attempted),  BRS:true,  KDX:null,  ACT:indices:monitor/settings/get,  OA:127.0.0.1/32,  XFF:null,  DA:127.0.0.1/32,  IDX:*,  MET:GET,  PTH:/_cat/indices,  CNT:<N/A>,  HDR:Accept=*/*, Authorization=Basic dmFkaW06dGVzdDE=, Content-Type=application/json, Host=127.0.0.1:9200, User-Agent=curl/7.54.0, content-length=0,  HIS:[Allow read/write all indexes for the vadim user only-> RULES:[groups->false], RESOLVED:[indices=*]]  }

The only difference is that, I’ve added one action pattern: indices:monitor/*

As you can see it works as expected.
Could you please try the scenario above in your environment?

9

Thanks for the update Mateusz,

A few questions please
[1] Is readonlyrest-1.21.0-pre5_es7.8.0.zip (you tested successfully) different from readonlyrest-1.20.0_es7.8.0.zip (I tested and failed)

[2] Should I use readonlyrest-1.21.0-pre5_es7.8.0.zip ?

[3] Did you test ROR along with elasticsearch-7.8.0-linux-x86_64.tar.gz (which I tested) ?

[4] In your test cases (TEST 1 and TEST 2) curl -u vadim:test -H "Content-Type … is “test” equals 64Base_encoded_value(vadim:test) ?

[5] Is testing ROR-1.20.0/ES-7.8.0 with curl -XGET http://host:port/_cat/indices?v -H ‘Authorization: Basic 64Base_encoded_value(vadim:test)’ not correct ?

Unfortunately I still can’t make ROR work as it should, tried readonlyrest-1.20.0_es7.8.0.zip and readonlyrest-1.21.0-pre5_es7.8.0.zip with elasticsearch-7.8.0-linux-x86_64.tar.gz

Thanks so much for your help and patience

10
  1. Yes, this is the newest version of ROR - not released yet, currently developed during 1.21.0 sprint. There was no changes according to the above, but it was easier for me to test this one (that’s why I’ve attached it here)
  2. if you want to be sure you have the same setup as I had - yes
  3. no
  4. as you can see I used -u curl option. But passing header Authorization: Basic <64Base_encoded_value(vadim:test)> should work the same.
  5. see 4)

IMO you should try exactly the same setup as I used (ROR, ES, readonlyrest.yml, curl). Then if you get different result, you should copy ROR logs here. If the result is the same, it’ll be a proof, that ROR works as expected and we will try to find problem in other place (maybe request or OS?)

11

I did the following:
-deployed readonlyrest-1.21.0-pre5_es7.8.0.zip
-changed my readonlyrest.yml to make it as yours:
–changes 4 char indents for 2 indents for “access_control_rules” and “users” (should it be 4 or 2 char indents ? I used 4 in my previous tests)
–changed “indices:data/read/" and "indices:data/write/” for “indices:data/read*” and “indices:data/write*” (latest forward slash removed)
-restarted the node with ROR
-cluster health=green, all 4 nodes joined the cluster
-Request
curl -u vadim:64BaseEncodedValue(user:pwd) -H “Content-Type: application/json” http://host:port/_cat/indices
-Response
{“error”:{“root_cause”:[{“reason”:“forbidden”,“due_to”:[“OPERATION_NOT_ALLOWED”]}],“reason”:“forbidden”,“due_to”:[“OPERATION_NOT_ALLOWED”],“status”:401}}
-ES LOG
[data time][INFO ][t.b.r.a.l.AccesControlLoggingDecorator] node_name FORBIDDEN by default req={ ID:…, TYP:GetSettingsRequest, CGR:N/A, USR:vadim (attempted), BRS:true, KDX:null, ACT:indices:monitor/settings/get, OA:…, XFF:null, DA:…, IDX:, MET:GET, PTH: /_cut/indices, CNT:<N/A>, HDR:Accept=/, Authorization=, Host=some_host/some_port, User-Agent=curl/7.15.5 (x86_64-redhat-linux-gnu)…OpenSSL/…, content-length=0, HIS:[Allow read/write all indexes for the vadim user only-> RULES:[groups->false], RESOLVED:[indices=]]}

I’m using ES-7.8.0 elasticsearch-7.8.0-linux-x86_64.tar.gz
My elasticsearch.yml contains also xpack.ml.enabled: false and xpack.security.enabled: false (because I read the doc says ROR is not working with enabled xpack.security correct ?)

As I mentioned I was able to setup and run successfully the earlier ROR version with ES-5.6.3 some time back
I’m thinking what else might be related to the issue, why Authorization is OMITTED ?

Thanks Mateusz

12

why do you call it like this?

what about a following curl:

curl -u vadim:test -H "Content-Type: application/json" "http://127.0.0.1:9200/_cat/indices"

?

13

I assumed for the pair of vadim:test where the test is clear password I (1)encoded vadim:test with sha256 and placed it in readonlyrest.yml (2)encode vadim:test in Base64 and use it in the request as
curl -u vadim:shdt4…thg -H "Content-Type…
So the “test” is real value of the password of user vadim and we exposing it clearly in the request ?
I tried to request as you advised and the request rejected again

14

please upload your readonlyrest.yml file and raw zipped logs

15

and maybe you should test it also using our docker image (GitHub - sscarduzio/ror-docker-demo: All in one ELK + ReadonlyREST Free + ReadonlyREST Enterprise docker container for demo purposes)

16

Mateusz, my readonlyread.yml content is at the very beginning of this post.
I poc with ES/ROR on the firm’s hardware/software and communicating with you from my personal laptop … had to type in manually.
Unfortunately I can’t expose and upload raw logs from the firm’s environment.
Thanks for the patience and support

17

sorry, it’s difficult for me to figure out what is wrong when you edit logs and when I’m not sure if you mirrored my example. I understand that it’s difficult for you because of technical reasons.

I’ve tested your case and it shows that it works. I suppose there is sth wrong with your curl (auth - because groups rule is not matched). This is all I can do it for you at the moment.

18

A few more things Mateusz.
To my understanding you tested ROR in DEV mode on a single node ES-7.8.0 cluster.
Did you test on master only dedicated node (node.master: true, node.data: false) ? Have you ever tested the ROR-1.20.0 with ES-7.8.0 in PROD mode on multi-node cluster ?

I’m testing multi-node cluster in PROD mode deployed on 2 hosts and deployed and configured ROR only on one master dedicated node. Is it ok ? Or ROR must be deployed and configured on all nodes of ES-7.8.0 ?

When I was working with ROR on ES-5.6.3 it needed to deploy RORs on each and every node of the cluster and enable them only on the dedicated master nodes to work them properly.

Thanks in advance

19

yes, I was testing on single node. Currently (1.20.x and 1.21.0-pre5), ROR doesn’t have to be installed on each node when:

  1. you don’t use fields rule(it will change soon)
  2. when you access your cluster only though one, dedicated node

Your problem (local groups) seems to be not related to the fact that you didn’t install ROR on each node.

20

Thanks for the feedback Mateusz