ES7.11.2 1.30.0 Enterprise two contexts rw/ro issue

Hi,
ES7.11.2
Kibana 7.11.2
RoR 1.30.0

  - name: "Allow RO read access to relevant indices"
    indices: [".kibana", ".kibana_reporting", "readonlyrest_audit-*"]
    kibana_index: ".kibana"
    kibana_access: ro
    kibana_hide_apps: ["Analytics|Overview", "Analytics|Canvas", "Analytics|Maps", "Enterprise Search", "Observability", "Security", "Management", "readonlyrest_kbn"]
    groups: ["RO_master"]

  - name: "Read-Write access with RoR custom kibana index"
    indices: [".kibana_ror_custom", ".reporting.kibana_ror_custom-*", "logstash*", "readonlyrest_audit-*"]
    kibana_index: ".kibana_ror_custom"
    kibana_access: rw
    kibana_hide_apps: ["Analytics|Overview", "Analytics|Canvas", "Analytics|Maps", "Enterprise Search", "Observability", "Security", "Management", "readonlyrest_kbn"]
    groups: ["RW_ror_custom"]    

  - username: testuser_ro_master_rw_custom
    auth_key: testuser_ro_master_rw_custom:XXX
    groups: ["RO_master", "RW_ror_custom"]

When users logs in to Kibana he joins the RO_master group in the default Kibana.
User then switches context to the RW_ror_custom group and gets RW in the .kibana_ror_custom kibana.

The user can do a add dashboard, but when saving this dashboard he gets a FORBIDDEN.
I can see:

[Read-Write access with RoR custom kibana index-> RULES:[groups->true, kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]]

We tested the same in 1.29.0 and there it worked, so this is something that broke in 1.30.0 release from our perspective.

Please let me know if you require any other information.

1 Like

@coutoPL what do you think?

It can be related to:

… but on the other hand I see kibana_access->false

@ronald.vanboven

It’d be nice if you could:

  • show us your users section
  • Enable debug logs and show us the whole forbidden log (and maybe other logs which are related to the request)
  • but the best you can do for us is to show us FORBIDDEN log from ROR 1.30.0 and ALLOWED log from ROR 1.29.0
2 Likes

@coutoPL as a representative of @ronald.vanboven I will see if I can get you the information requested.

/Arjen

1 Like

the users section for these contexts consists of:

  users:
  - username: testuser_ro_master
    auth_key: testuser_ro_master:XXXX
    groups: ["RO_master"]
  - username: testuser_ro_master_rw_custom
    auth_key: testuser_ro_master_rw_custom:XXXX
    groups: ["RO_master", "RW_ror_custom"]

With the same readonlyrest config, this worked on RoR v1.29.0 with kibana/es v7.11.2

/Arjen

Kibana 7.11.2
RoR 1.29.0
Same config, you get ACL history:

 [Read-Write access with RoR custom kibana index-> RULES:[groups->true kibana_access->true indices->true kibana_hide_apps->true kibana_index->true] 
 RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom;kibana_idx=.kibana_ror_custom]]

And it is allowed.

And debug logs with 1.30.0:

{"type": "server", "timestamp": "2021-05-19T12:06:06,900+02:00", "level": "DEBUG", "component": "t.b.r.a.b.r.AuthKeyRule", "cluster.name": "x.x.x.x_elk-log", "node.name": "x.x.x.x", "message": "Attempting Login as: testuser_ro_master_rw_custom rc: 79030582-126338483#11115", "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg", "node.id": "o6WNXxsIRzGTtAoLj9HmzA"  }
{"type": "server", "timestamp": "2021-05-19T12:06:06,900+02:00", "level": "DEBUG", "component": "t.b.r.a.b.r.KibanaAccessRule", "cluster.name": "x.x.x.x_elk-log", "node.name": "x.x.x.x", "message": "KIBANA ACCESS DENIED 79030582-126338483#11115", "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg", "node.id": "o6WNXxsIRzGTtAoLj9HmzA"  }
{
    "type": "server",
    "timestamp": "2021-05-19T12:06:06,901+02:00",
    "level": "DEBUG",
    "component": "t.b.r.a.b.Block",
    "cluster.name": "x.x.x.x-elk-log",
    "node.name": "x.x.x.x",
    "message": "\u001B[33m[Read-Write access with RoR custom kibana index] the request matches no rules in this block: 
{ ID: 79030582-126338483#11115
 TYP:IndexRequest
 CGR:RW_ror_custom
 USR:testuser_ro_master_rw_custom (attempted)
 BRS: true
 KDX: null
 ACT:indices:data/write/index
 OA: x.x.x.x/32
 XFF:x.x.x.x: 8001
 DA: x.x.x.x/32
 IDX:.kibana_ror_custom
 MET:PUT
 PTH:/.kibana_ror_custom/_doc/dashboard:d3d40550-b889-11eb-a1e1-914af9365d47
 CNT: {\"dashboard\":{\"title\":\"AwesomeError2\"
\"hits\":0
\"description\":\"\"
\"panelsJSON\":\"[]\"
\"optionsJSON\":\"{\\\"useMargins\\\":true
\\\"hidePanelTitles\\\":false}\"
\"version\":1
\"timeRestore\":false
\"kibanaSavedObjectMeta\":{\"searchSourceJSON\":\"{\\\"query\\\":{\\\"query\\\":\\\"\\\"
\\\"language\\\":\\\"kuery\\\"}
\\\"filter\\\":[]}\"}}
\"type\":\"dashboard\"
\"references\":[]
\"migrationVersion\":{\"dashboard\":\"7.11.0\"}
\"updated_at\":\"2021-05-19T10:06:06.885Z\"}
 HDR:Accept-Charset=utf-8
 Authorization=<OMITTED>
 Content-Length=409
 Host=x.x.x.x:9200
 connection=close
 content-type=application/json
 user-agent=elasticsearch-js/7.11.0-rc.1 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.16.0)
 x-elastic-client-meta=es=7.11.0-rc.1
js=14.16.0
t=7.11.0-rc.1
hc=14.16.0
 x-elastic-product-origin=kibana
 x-forwarded-for=x.x.x.x:8001
 x-opaque-id=663f9661-09de-46ee-9fe6-9ff63f4d388a
 x-ror-current-group=RW_ror_custom
 x-ror-kibana-request-method=post
 x-ror-kibana-request-path=/api/saved_objects/dashboard
 HIS:[Read-Write access with RoR custom kibana index-> RULES:[groups->true
 kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]]
 } \u001B[0m"

        "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg"

        "node.id": "o6WNXxsIRzGTtAoLj9HmzA"
    }

{
    "type": "server",
    "timestamp": "2021-05-19T12:06:06,906+02:00",
    "level": "INFO",
    "component": "t.b.r.a.l.AccessControlLoggingDecorator",
    "cluster.name": "x.x.x.x_elk-log",
    "node.name": "x.x.x.x",
    "message": "\u001B[35mFORBIDDEN by default req={ ID:79030582-126338483#11115, TYP:IndexRequest, CGR:RW_ror_custom, USR:testuser_ro_master_rw_custom (attempted), BRS:true, KDX:null, ACT:indices:data/write/index, OA:x.x.x.x/32, XFF:x.x.x.x:8001, DA:x.x.x.x/32, IDX:.kibana_ror_custom, MET:PUT, PTH:/.kibana_ror_custom/_doc/dashboard:d3d40550-b889-11eb-a1e1-914af9365d47, CNT:{\"dashboard\":{\"title\":\"AwesomeError2\",\"hits\":0,\"description\":\"\",\"panelsJSON\":\"[]\",\"optionsJSON\":\"{\\\"useMargins\\\":true,\\\"hidePanelTitles\\\":false}\",\"version\":1,\"timeRestore\":false,\"kibanaSavedObjectMeta\":{\"searchSourceJSON\":\"{\\\"query\\\":{\\\"query\\\":\\\"\\\",\\\"language\\\":\\\"kuery\\\"},\\\"filter\\\":[]}\"}},\"type\":\"dashboard\",\"references\":[],\"migrationVersion\":{\"dashboard\":\"7.11.0\"},\"updated_at\":\"2021-05-19T10:06:06.885Z\"}, HDR:Accept-Charset=utf-8, Authorization=Basic XXX=, Content-Length=409, Host=x.x.x.x:9200, connection=close, content-type=application/json, user-agent=elasticsearch-js/7.11.0-rc.1 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.16.0), x-elastic-client-meta=es=7.11.0-rc.1,js=14.16.0,t=7.11.0-rc.1,hc=14.16.0, x-elastic-product-origin=kibana, x-forwarded-for=x.x.x.x:8001, x-opaque-id=663f9661-09de-46ee-9fe6-9ff63f4d388a, x-ror-current-group=RW_ror_custom, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/saved_objects/dashboard, HIS:[[Allow RO read access to relevant indices-> RULES:[groups->false] RESOLVED:[group=RW_ror_custom;indices=.kibana_ror_custom]], [Read-Write access with RoR custom kibana index-> RULES:[groups->true, kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]] ], }\u001B[0m",
    "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg",
    "node.id": "o6WNXxsIRzGTtAoLj9HmzA"
}

I hope these logs are sufficient (I have redacted stuff that should not be relevant).
Unfortunately we don’t have a completely isolated test environment in which we can test just the pure basic config :frowning:

thanks guys. I’ll try to reproduce it with our automated tests. Will let you know when I found sth.

1 Like

good news guys. I found the issue. It’s a regression. I’ve fixed it and added missing tests to avoid such issues in the future. Will send you a prebuild to test tomorrow. When you confirm it works, we’ll release ROR 1.30.1 with the fix included.

2 Likes

@ronald.vanboven @abuising please test this pre-build:

https://readonlyrest-data.s3.amazonaws.com/build/1.31.0-pre2/readonlyrest-1.31.0-pre2_es7.11.2.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20210524/eu-west-1/s3/aws4_request&X-Amz-Date=20210524T175005Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=26bf66d23f20599f72a29c338df224b82de71f31d0b12286ea95f83b91a2f8da

1 Like

@coutoPL Tested, didn’t work :frowning:

{
    "cluster.name": "x.x.x.x"

    "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg"

    "component": "t.b.r.a.l.AccessControlLoggingDecorator"

    "level": "INFO"

    "message": "\u001b[35mFORBIDDEN by default req={ ID:758841610-111586337#95153
 TYP:IndexRequest
 CGR:RW_ror_custom
 USR:testuser_ro_master_rw_custom (attempted)
 BRS:true
 KDX:null
 ACT:indices:data/write/index
 OA:x.x.x.x/32
 XFF:x.x.x.x:8001
 DA:x.x.x.x/32
 IDX:.kibana_ror_custom
 MET:PUT
 PTH:/.kibana_ror_custom/_doc/dashboard:1d4aab50-7342-11ea-972a-93e003498cbf
 CNT:<OMITTED
 LENGTH=1560.0 B> 
 HDR:Accept-Charset=utf-8
 Authorization=<OMITTED>
 Content-Length=1560
 Host=x.x.x.x:9200
 connection=close
 content-type=application/json
 user-agent=elasticsearch-js/7.11.0-rc.1 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.16.0)
 x-elastic-client-meta=es=7.11.0-rc.1
js=14.16.0
t=7.11.0-rc.1
hc=14.16.0
 x-elastic-product-origin=kibana
 x-forwarded-for=x.x.x.x:8001
 x-opaque-id=c417e9f2-db95-4f03-9198-b58918ecda63
 x-ror-current-group=RW_ror_custom
 x-ror-kibana-request-method=post
 x-ror-kibana-request-path=/api/saved_objects/dashboard/1d4aab50-7342-11ea-972a-93e003498cbf
 HIS:[::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[group=RW_ror_custom;indices=.kibana_ror_custom]]
 [Allow RO read access to relevant indices-> RULES:[groups->false] RESOLVED:[group=RW_ror_custom;indices=.kibana_ror_custom]]
 [Read-Write access with RoR custom kibana index-> RULES:[groups->true kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]]
 }\u001b[0m"

    "node.id": "o6WNXxsIRzGTtAoLj9HmzA"

    "node.name": "x.x.x.x"

    "timestamp": "2021-05-25T10:52:02
512+02:00"

    "type": "server"
}

@ronald.vanboven what about this one? https://readonlyrest-data.s3.amazonaws.com/build/1.30.1-pre1/readonlyrest-1.30.1-pre1_es7.11.2.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20210526/eu-west-1/s3/aws4_request&X-Amz-Date=20210526T075553Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=11f76d227afc61d0b0563647db6a3273fc719d0a60f55a637224488a77cfd04b

1 Like

Thanks! We’ll try!

/Arjen

Hi, Thanks! Yes, it works again!

/Arjen

1 Like

Thanks for the info. It’s released with ROR 1.30.1. Download (UNIVERSAL) - ReadonlyREST

1 Like

Thank you, it works!

/Arjen

1 Like

Hello @coutoPL,
I just upgraded my Kibana + RoR to Kibana +Elasticsearch 7.12.1 with Ror 1.31.0 and I more or less have the same issue as here, except it appears when using the import feature of saved_object api. (which use the indices:data/write/bulk action instead of indices:data/write/write.) .

I’ll add the related logs in a follow-up (I need few minutes more to gather them)

1 Like

Here are the related logs:
kibana :

{
    "type": "response",
    "@timestamp": "2021-07-05T13:50:30+02:00",
    "tags": [],
    "pid": 9,
    "method": "post",
    "statusCode": 403,
    "req": {
        "url": "/api/saved_objects/_import?overwrite=true",
        "method": "post",
        "headers": {
            "user-agent": "python-requests/2.25.1",
            "accept-encoding": "gzip, deflate",
            "accept": "*/*",
            "kbn-xsrf": "true",
            "x-kibana-index": "kibana_vgfg7119",
            "content-type": "multipart/form-data; boundary=dd4b6a7c451e8a3d0ce1e39a43a95229",
            "host": "<redacted>",
            "x-forwarded-host": "<redacted>",
            "x-forwarded-port": "443",
            "x-forwarded-proto": "https",
            "forwarded": "for=<REDACTED>;host=<REDACTED>;proto=https",
            "x-forwarded-for": "1<REDACTED>",
            "connection": "close",
            "x-ror-pkp-kibana-token": "5mztptcrljg5opoxvg4zs40rd2suwx",
            "transfer-encoding": "chunked"
        },
        "remoteAddress": "127.0.0.1",
        "userAgent": "python-requests/2.25.1"
    },
    "res": {
        "statusCode": 403,
        "responseTime": 2524,
        "contentLength": 76
    },
    "message": "POST /api/saved_objects/_import?overwrite=true 403 2524ms - 76.0B"
}

Elasticsearch :

{
    "type": "server",
    "timestamp": "2021-07-05T13:50:32,146+02:00",
    "level": "INFO",
    "component": "t.b.r.a.l.AccessControlLoggingDecorator",
    "cluster.name": "elastic-cluster",
    "node.name": "elasticsearch-22-m289f",
    "message": "\u001B[36mALLOWED by { name: '::Import::Export tool ::', policy: ALLOW, rules: [auth_key_sha512,kibana_index,kibana_access,indices] req={ ID:1635327142-1236962386#55880, TYP:MultiGetRequest, CGR:N/A, USR:import, BRS:true, KDX:.kibana_mylogin, ACT:indices:data/read/mget, OA:<REDACTED>/32, XFF:<REDACTED>, DA:<REDACTED>/32, IDX:.kibana_7.12.1, MET:POST, PTH:/_mget, CNT:<OMITTED, LENGTH=19764.0 B> , HDR:Authorization=<OMITTED>, Connection=keep-alive, Host=<redacted>.svc:9200, content-length=19764, content-type=application/json, cookie=rorCookie=<redacted>, user-agent=elasticsearch-js/7.12.0-canary.7 (linux 3.10.0-957.12.2.el7.x86_64-x64; Node.js v14.16.1), x-elastic-client-meta=es=7.12.0p,js=14.16.1,t=7.12.0p,hc=14.16.1, x-elastic-product-origin=kibana, x-forwarded-for=<redacted>, x-kibana-index=kibana_mylogin, x-opaque-id=a6477b39-0db2-4a39-8a37-68ef0534949c, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/saved_objects/_import, HIS:[Allow HealthCheck-> RULES:[methods->false] RESOLVED:[indices=.kibana_7.12.1]], [::ADMIN::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::KIBANA::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::Import::Export tool ::-> RULES:[auth_key_sha512->true, kibana_index->true, kibana_access->true, indices->true] RESOLVED:[user=import;indices=.kibana_7.12.1;kibana_idx=.kibana_mylogin]], }\u001B[0m",
    "cluster.uuid": "jS0Y755BQwSgcPyWEQ1OdQ",
    "node.id": "GVxz7Mt7QQaW2X6PfKy59Q"
}{
    "type": "server",
    "timestamp": "2021-07-05T13:50:32,490+02:00",
    "level": "INFO",
    "component": "t.b.r.a.l.AccessControlLoggingDecorator",
    "cluster.name": "elastic-cluster",
    "node.name": "elasticsearch-22-m289f",
    "message": "\u001B[35mFORBIDDEN by default req={ ID:14444974-832283029#55883, TYP:BulkRequest, CGR:N/A, USR:import (attempted), BRS:true, KDX:null, ACT:indices:data/write/bulk, OA:<REDACTED>/32, XFF:<REDACTED>, DA:<REDACTED>/32, IDX:.kibana_7.12.1, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=5826902.0 B> , HDR:Authorization=<OMITTED>, Connection=keep-alive, Host=<redacted>:9200, content-length=5826902, content-type=application/x-ndjson, cookie=rorCookie=REDACTED, user-agent=elasticsearch-js/7.12.0-canary.7 (linux 3.10.0-957.12.2.el7.x86_64-x64; Node.js v14.16.1), x-elastic-client-meta=es=7.12.0p,js=14.16.1,t=7.12.0p,hc=14.16.1, x-elastic-product-origin=kibana, x-forwarded-for=<REDACTED>, x-kibana-index=kibana_mylogin, x-opaque-id=a6477b39-0db2-4a39-8a37-68ef0534949c, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/saved_objects/_import, HIS:[Allow HealthCheck-> RULES:[methods->false] RESOLVED:[indices=.kibana_7.12.1]], [::ADMIN::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::KIBANA::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::Import::Export tool ::-> RULES:[auth_key_sha512->true, kibana_index->true, kibana_access->false] RESOLVED:[user=import;indices=.kibana_7.12.1;kibana_idx=.kibana_mylogin]], [::Allow access to create short urls::-> RULES:[ror_kbn_auth->false] RESOLVED:[indices=.kibana_7.12.1]], }\u001B[0m",
    "cluster.uuid": "jS0Y755BQwSgcPyWEQ1OdQ",
    "node.id": "GVxz7Mt7QQaW2X6PfKy59Q"
}

Related configuration :

- name: "::Import::Export tool ::"
  kibana_access: "rw"
  verbosity: info
  auth_key_sha512: "import:<REDACTED>"
  indices: [".kibana*"]
  kibana_index: ".@{header:X-Kibana-Index}"

Can you give us the ES log line that forbids this operation? The provided logs are coming from Kibana.
Sorry didnt see the other message!!

OK I can see how the block is not matching because of kibana_access rule mismatch:

[::Import::Export tool ::-> RULES:[auth_key_sha512->true, kibana_index->true, kibana_access->false]

When the action “indices:data/write/bulk”. @coutoPL can we whitelist this action towards kibana index?

Guys, this is the interesting part:

[::Import::Export tool ::-> RULES:[
auth_key_sha512->true, 
kibana_index->true, 
kibana_access->false
] 
RESOLVED:[
user=import;
indices=.kibana_7.12.1;
kibana_idx=.kibana_mylogin
]]

We see that kibana index was resolved by kibana_index rule as .kibana_mylogin. On the other hand the request are related to index: .kibana_7.12.1. This is why the kibana_access rule rejected the request.

IMO this behaviour is correct. We could ask: why the request index was .kibana_7.12.1, but not .kibana_mylogin?

Yep, I overlooked this. @jan have a look, you resolved a similar bug some time ago, didn’t you?