It can be related to:
Enhancement (ES) Username with wildcard support in users section and groups mapping
… but on the other hand I see kibana_access->false
It’d be nice if you could:
users section@coutoPL as a representative of @ronald.vanboven I will see if I can get you the information requested.
/Arjen
the users section for these contexts consists of:
users:
- username: testuser_ro_master
auth_key: testuser_ro_master:XXXX
groups: ["RO_master"]
- username: testuser_ro_master_rw_custom
auth_key: testuser_ro_master_rw_custom:XXXX
groups: ["RO_master", "RW_ror_custom"]
With the same readonlyrest config, this worked on RoR v1.29.0 with kibana/es v7.11.2
/Arjen
Kibana 7.11.2
RoR 1.29.0
Same config, you get ACL history:
[Read-Write access with RoR custom kibana index-> RULES:[groups->true kibana_access->true indices->true kibana_hide_apps->true kibana_index->true]
RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom;kibana_idx=.kibana_ror_custom]]
And it is allowed.
And debug logs with 1.30.0:
{"type": "server", "timestamp": "2021-05-19T12:06:06,900+02:00", "level": "DEBUG", "component": "t.b.r.a.b.r.AuthKeyRule", "cluster.name": "x.x.x.x_elk-log", "node.name": "x.x.x.x", "message": "Attempting Login as: testuser_ro_master_rw_custom rc: 79030582-126338483#11115", "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg", "node.id": "o6WNXxsIRzGTtAoLj9HmzA" }
{"type": "server", "timestamp": "2021-05-19T12:06:06,900+02:00", "level": "DEBUG", "component": "t.b.r.a.b.r.KibanaAccessRule", "cluster.name": "x.x.x.x_elk-log", "node.name": "x.x.x.x", "message": "KIBANA ACCESS DENIED 79030582-126338483#11115", "cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg", "node.id": "o6WNXxsIRzGTtAoLj9HmzA" }
{
"type": "server",
"timestamp": "2021-05-19T12:06:06,901+02:00",
"level": "DEBUG",
"component": "t.b.r.a.b.Block",
"cluster.name": "x.x.x.x-elk-log",
"node.name": "x.x.x.x",
"message": "\u001B[33m[Read-Write access with RoR custom kibana index] the request matches no rules in this block:
{ ID: 79030582-126338483#11115
TYP:IndexRequest
CGR:RW_ror_custom
USR:testuser_ro_master_rw_custom (attempted)
BRS: true
KDX: null
ACT:indices:data/write/index
OA: x.x.x.x/32
XFF:x.x.x.x: 8001
DA: x.x.x.x/32
IDX:.kibana_ror_custom
MET:PUT
PTH:/.kibana_ror_custom/_doc/dashboard:d3d40550-b889-11eb-a1e1-914af9365d47
CNT: {\"dashboard\":{\"title\":\"AwesomeError2\"
\"hits\":0
\"description\":\"\"
\"panelsJSON\":\"[]\"
\"optionsJSON\":\"{\\\"useMargins\\\":true
\\\"hidePanelTitles\\\":false}\"
\"version\":1
\"timeRestore\":false
\"kibanaSavedObjectMeta\":{\"searchSourceJSON\":\"{\\\"query\\\":{\\\"query\\\":\\\"\\\"
\\\"language\\\":\\\"kuery\\\"}
\\\"filter\\\":[]}\"}}
\"type\":\"dashboard\"
\"references\":[]
\"migrationVersion\":{\"dashboard\":\"7.11.0\"}
\"updated_at\":\"2021-05-19T10:06:06.885Z\"}
HDR:Accept-Charset=utf-8
Authorization=<OMITTED>
Content-Length=409
Host=x.x.x.x:9200
connection=close
content-type=application/json
user-agent=elasticsearch-js/7.11.0-rc.1 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.16.0)
x-elastic-client-meta=es=7.11.0-rc.1
js=14.16.0
t=7.11.0-rc.1
hc=14.16.0
x-elastic-product-origin=kibana
x-forwarded-for=x.x.x.x:8001
x-opaque-id=663f9661-09de-46ee-9fe6-9ff63f4d388a
x-ror-current-group=RW_ror_custom
x-ror-kibana-request-method=post
x-ror-kibana-request-path=/api/saved_objects/dashboard
HIS:[Read-Write access with RoR custom kibana index-> RULES:[groups->true
kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]]
} \u001B[0m"
"cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg"
"node.id": "o6WNXxsIRzGTtAoLj9HmzA"
}
{
"type": "server",
"timestamp": "2021-05-19T12:06:06,906+02:00",
"level": "INFO",
"component": "t.b.r.a.l.AccessControlLoggingDecorator",
"cluster.name": "x.x.x.x_elk-log",
"node.name": "x.x.x.x",
"message": "\u001B[35mFORBIDDEN by default req={ ID:79030582-126338483#11115, TYP:IndexRequest, CGR:RW_ror_custom, USR:testuser_ro_master_rw_custom (attempted), BRS:true, KDX:null, ACT:indices:data/write/index, OA:x.x.x.x/32, XFF:x.x.x.x:8001, DA:x.x.x.x/32, IDX:.kibana_ror_custom, MET:PUT, PTH:/.kibana_ror_custom/_doc/dashboard:d3d40550-b889-11eb-a1e1-914af9365d47, CNT:{\"dashboard\":{\"title\":\"AwesomeError2\",\"hits\":0,\"description\":\"\",\"panelsJSON\":\"[]\",\"optionsJSON\":\"{\\\"useMargins\\\":true,\\\"hidePanelTitles\\\":false}\",\"version\":1,\"timeRestore\":false,\"kibanaSavedObjectMeta\":{\"searchSourceJSON\":\"{\\\"query\\\":{\\\"query\\\":\\\"\\\",\\\"language\\\":\\\"kuery\\\"},\\\"filter\\\":[]}\"}},\"type\":\"dashboard\",\"references\":[],\"migrationVersion\":{\"dashboard\":\"7.11.0\"},\"updated_at\":\"2021-05-19T10:06:06.885Z\"}, HDR:Accept-Charset=utf-8, Authorization=Basic XXX=, Content-Length=409, Host=x.x.x.x:9200, connection=close, content-type=application/json, user-agent=elasticsearch-js/7.11.0-rc.1 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.16.0), x-elastic-client-meta=es=7.11.0-rc.1,js=14.16.0,t=7.11.0-rc.1,hc=14.16.0, x-elastic-product-origin=kibana, x-forwarded-for=x.x.x.x:8001, x-opaque-id=663f9661-09de-46ee-9fe6-9ff63f4d388a, x-ror-current-group=RW_ror_custom, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/saved_objects/dashboard, HIS:[[Allow RO read access to relevant indices-> RULES:[groups->false] RESOLVED:[group=RW_ror_custom;indices=.kibana_ror_custom]], [Read-Write access with RoR custom kibana index-> RULES:[groups->true, kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]] ], }\u001B[0m",
"cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg",
"node.id": "o6WNXxsIRzGTtAoLj9HmzA"
}
I hope these logs are sufficient (I have redacted stuff that should not be relevant).
Unfortunately we don’t have a completely isolated test environment in which we can test just the pure basic config 
thanks guys. I’ll try to reproduce it with our automated tests. Will let you know when I found sth.
good news guys. I found the issue. It’s a regression. I’ve fixed it and added missing tests to avoid such issues in the future. Will send you a prebuild to test tomorrow. When you confirm it works, we’ll release ROR 1.30.1 with the fix included.
@coutoPL Tested, didn’t work
{
"cluster.name": "x.x.x.x"
"cluster.uuid": "XMFAmE8rSOydA8jmOkoHYg"
"component": "t.b.r.a.l.AccessControlLoggingDecorator"
"level": "INFO"
"message": "\u001b[35mFORBIDDEN by default req={ ID:758841610-111586337#95153
TYP:IndexRequest
CGR:RW_ror_custom
USR:testuser_ro_master_rw_custom (attempted)
BRS:true
KDX:null
ACT:indices:data/write/index
OA:x.x.x.x/32
XFF:x.x.x.x:8001
DA:x.x.x.x/32
IDX:.kibana_ror_custom
MET:PUT
PTH:/.kibana_ror_custom/_doc/dashboard:1d4aab50-7342-11ea-972a-93e003498cbf
CNT:<OMITTED
LENGTH=1560.0 B>
HDR:Accept-Charset=utf-8
Authorization=<OMITTED>
Content-Length=1560
Host=x.x.x.x:9200
connection=close
content-type=application/json
user-agent=elasticsearch-js/7.11.0-rc.1 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.16.0)
x-elastic-client-meta=es=7.11.0-rc.1
js=14.16.0
t=7.11.0-rc.1
hc=14.16.0
x-elastic-product-origin=kibana
x-forwarded-for=x.x.x.x:8001
x-opaque-id=c417e9f2-db95-4f03-9198-b58918ecda63
x-ror-current-group=RW_ror_custom
x-ror-kibana-request-method=post
x-ror-kibana-request-path=/api/saved_objects/dashboard/1d4aab50-7342-11ea-972a-93e003498cbf
HIS:[::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[group=RW_ror_custom;indices=.kibana_ror_custom]]
[Allow RO read access to relevant indices-> RULES:[groups->false] RESOLVED:[group=RW_ror_custom;indices=.kibana_ror_custom]]
[Read-Write access with RoR custom kibana index-> RULES:[groups->true kibana_access->false] RESOLVED:[user=testuser_ro_master_rw_custom;group=RW_ror_custom;av_groups=RW_ror_custom;indices=.kibana_ror_custom]]
}\u001b[0m"
"node.id": "o6WNXxsIRzGTtAoLj9HmzA"
"node.name": "x.x.x.x"
"timestamp": "2021-05-25T10:52:02
512+02:00"
"type": "server"
}
@ronald.vanboven what about this one? https://readonlyrest-data.s3.amazonaws.com/build/1.30.1-pre1/readonlyrest-1.30.1-pre1_es7.11.2.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20210526/eu-west-1/s3/aws4_request&X-Amz-Date=20210526T075553Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=11f76d227afc61d0b0563647db6a3273fc719d0a60f55a637224488a77cfd04b
Thanks! We’ll try!
/Arjen
Hi, Thanks! Yes, it works again!
/Arjen
Hello @coutoPL,
I just upgraded my Kibana + RoR to Kibana +Elasticsearch 7.12.1 with Ror 1.31.0 and I more or less have the same issue as here, except it appears when using the import feature of saved_object api. (which use the indices:data/write/bulk action instead of indices:data/write/write.) .
I’ll add the related logs in a follow-up (I need few minutes more to gather them)
Here are the related logs:
kibana :
{
"type": "response",
"@timestamp": "2021-07-05T13:50:30+02:00",
"tags": [],
"pid": 9,
"method": "post",
"statusCode": 403,
"req": {
"url": "/api/saved_objects/_import?overwrite=true",
"method": "post",
"headers": {
"user-agent": "python-requests/2.25.1",
"accept-encoding": "gzip, deflate",
"accept": "*/*",
"kbn-xsrf": "true",
"x-kibana-index": "kibana_vgfg7119",
"content-type": "multipart/form-data; boundary=dd4b6a7c451e8a3d0ce1e39a43a95229",
"host": "<redacted>",
"x-forwarded-host": "<redacted>",
"x-forwarded-port": "443",
"x-forwarded-proto": "https",
"forwarded": "for=<REDACTED>;host=<REDACTED>;proto=https",
"x-forwarded-for": "1<REDACTED>",
"connection": "close",
"x-ror-pkp-kibana-token": "5mztptcrljg5opoxvg4zs40rd2suwx",
"transfer-encoding": "chunked"
},
"remoteAddress": "127.0.0.1",
"userAgent": "python-requests/2.25.1"
},
"res": {
"statusCode": 403,
"responseTime": 2524,
"contentLength": 76
},
"message": "POST /api/saved_objects/_import?overwrite=true 403 2524ms - 76.0B"
}
Elasticsearch :
{
"type": "server",
"timestamp": "2021-07-05T13:50:32,146+02:00",
"level": "INFO",
"component": "t.b.r.a.l.AccessControlLoggingDecorator",
"cluster.name": "elastic-cluster",
"node.name": "elasticsearch-22-m289f",
"message": "\u001B[36mALLOWED by { name: '::Import::Export tool ::', policy: ALLOW, rules: [auth_key_sha512,kibana_index,kibana_access,indices] req={ ID:1635327142-1236962386#55880, TYP:MultiGetRequest, CGR:N/A, USR:import, BRS:true, KDX:.kibana_mylogin, ACT:indices:data/read/mget, OA:<REDACTED>/32, XFF:<REDACTED>, DA:<REDACTED>/32, IDX:.kibana_7.12.1, MET:POST, PTH:/_mget, CNT:<OMITTED, LENGTH=19764.0 B> , HDR:Authorization=<OMITTED>, Connection=keep-alive, Host=<redacted>.svc:9200, content-length=19764, content-type=application/json, cookie=rorCookie=<redacted>, user-agent=elasticsearch-js/7.12.0-canary.7 (linux 3.10.0-957.12.2.el7.x86_64-x64; Node.js v14.16.1), x-elastic-client-meta=es=7.12.0p,js=14.16.1,t=7.12.0p,hc=14.16.1, x-elastic-product-origin=kibana, x-forwarded-for=<redacted>, x-kibana-index=kibana_mylogin, x-opaque-id=a6477b39-0db2-4a39-8a37-68ef0534949c, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/saved_objects/_import, HIS:[Allow HealthCheck-> RULES:[methods->false] RESOLVED:[indices=.kibana_7.12.1]], [::ADMIN::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::KIBANA::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::Import::Export tool ::-> RULES:[auth_key_sha512->true, kibana_index->true, kibana_access->true, indices->true] RESOLVED:[user=import;indices=.kibana_7.12.1;kibana_idx=.kibana_mylogin]], }\u001B[0m",
"cluster.uuid": "jS0Y755BQwSgcPyWEQ1OdQ",
"node.id": "GVxz7Mt7QQaW2X6PfKy59Q"
}{
"type": "server",
"timestamp": "2021-07-05T13:50:32,490+02:00",
"level": "INFO",
"component": "t.b.r.a.l.AccessControlLoggingDecorator",
"cluster.name": "elastic-cluster",
"node.name": "elasticsearch-22-m289f",
"message": "\u001B[35mFORBIDDEN by default req={ ID:14444974-832283029#55883, TYP:BulkRequest, CGR:N/A, USR:import (attempted), BRS:true, KDX:null, ACT:indices:data/write/bulk, OA:<REDACTED>/32, XFF:<REDACTED>, DA:<REDACTED>/32, IDX:.kibana_7.12.1, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=5826902.0 B> , HDR:Authorization=<OMITTED>, Connection=keep-alive, Host=<redacted>:9200, content-length=5826902, content-type=application/x-ndjson, cookie=rorCookie=REDACTED, user-agent=elasticsearch-js/7.12.0-canary.7 (linux 3.10.0-957.12.2.el7.x86_64-x64; Node.js v14.16.1), x-elastic-client-meta=es=7.12.0p,js=14.16.1,t=7.12.0p,hc=14.16.1, x-elastic-product-origin=kibana, x-forwarded-for=<REDACTED>, x-kibana-index=kibana_mylogin, x-opaque-id=a6477b39-0db2-4a39-8a37-68ef0534949c, x-ror-kibana-request-method=post, x-ror-kibana-request-path=/api/saved_objects/_import, HIS:[Allow HealthCheck-> RULES:[methods->false] RESOLVED:[indices=.kibana_7.12.1]], [::ADMIN::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::KIBANA::-> RULES:[auth_key_sha512->false] RESOLVED:[indices=.kibana_7.12.1]], [::Import::Export tool ::-> RULES:[auth_key_sha512->true, kibana_index->true, kibana_access->false] RESOLVED:[user=import;indices=.kibana_7.12.1;kibana_idx=.kibana_mylogin]], [::Allow access to create short urls::-> RULES:[ror_kbn_auth->false] RESOLVED:[indices=.kibana_7.12.1]], }\u001B[0m",
"cluster.uuid": "jS0Y755BQwSgcPyWEQ1OdQ",
"node.id": "GVxz7Mt7QQaW2X6PfKy59Q"
}
Related configuration :
- name: "::Import::Export tool ::"
kibana_access: "rw"
verbosity: info
auth_key_sha512: "import:<REDACTED>"
indices: [".kibana*"]
kibana_index: ".@{header:X-Kibana-Index}"Can you give us the ES log line that forbids this operation? The provided logs are coming from Kibana.
Sorry didnt see the other message!!
OK I can see how the block is not matching because of kibana_access rule mismatch:
[::Import::Export tool ::-> RULES:[auth_key_sha512->true, kibana_index->true, kibana_access->false]
When the action “indices:data/write/bulk”. @coutoPL can we whitelist this action towards kibana index?
Guys, this is the interesting part:
[::Import::Export tool ::-> RULES:[
auth_key_sha512->true,
kibana_index->true,
kibana_access->false
]
RESOLVED:[
user=import;
indices=.kibana_7.12.1;
kibana_idx=.kibana_mylogin
]]
We see that kibana index was resolved by kibana_index rule as .kibana_mylogin. On the other hand the request are related to index: .kibana_7.12.1. This is why the kibana_access rule rejected the request.
IMO this behaviour is correct. We could ask: why the request index was .kibana_7.12.1, but not .kibana_mylogin?
Hello !
Any updates/ideas about this ?
For the future googler: this was an Elasticsearch plugin issue, we resolved it in 1.31.0