Errors after upgrade Kibana 7.17.29 to 8.19.7

Hello,

After upgrading Elastic nodes to 8.19.7 I tried to upgrade one instance of kibana with ROR 1.67.2 ‘readonlyrest_kbn_universal-1.67.2_es8.19.7. But, after succesfully starting proces and first log in, I have some errors in kibana.log when accessing ‘/s/default/app/home#/‘

[2025-12-05T13:16:18.760+01:00][ERROR][plugins.security.authentication] License is not available or does not support security features, re-authenticatio
n is not possible (available: true, enabled: false, unavailable reason: undefined).
[2025-12-05T13:16:18.761+01:00][ERROR][plugins.streams] ResponseError: forbidden_response
        Root causes:
                forbidden_response: Sorry, wrong password or you don't have permission to this index.

or

[ERROR][plugins.security.authentication] License is not available or does not support security features, re-authentication is not possible (available: true, enabled: false, unavailable reason: undefined).
[ERROR][plugins.dataViews.dataView.hasEsData] ResponseError: forbidden_response
        Root causes:
                forbidden_response: Sorry, wrong password or you don't have permission to this index.

with ldap user for example:

- name: "::X LDAP::"
      ldap_auth:
        name: "ldap"
        groups_any_of: ["xxx"]
      indices: [".kibana*",".reporting-*", ".ds-.kibana-*", ".kibana-reporting-*", "xxx-*"]
      verbosity: error
      kibana:
        access: rw
        index: ".kibana-xxx"
      type: allow

*(from readonlyrest index config)

Additionally in Stack management → url to “/s/default/app/management“ I see that clicking in some links I get logged out when in 7.17.29 I got message “Sorry, wrong password or you don’t have permission to this index.“ which I always understood as the proper one :slight_smile:

If it may help, I do have some specific ‘Forbid’ rules from 7.17.29 that worked with ROR 1.67.2 (for Dev Tools app restrictions)

 uri_re: ["^/_(?:alias|nodes|cat|cluster|ml|ilm|license|mapping|settings|sql|analyze|autoscaling|ccr|component_template|dangling|data_stream|flush|
enrich|eql|ilm|ingest|index_template|migration|mtermvectors|processor|recovery|refresh|rank_eval|remote|rollup|script_context|script_language|scripts|se
arch_shards|searcheable_snapshots|security|segments|shard_stores|slm|snapshot|ssl|stats|tasks|template|transform|validate|watcher|xpack)"]
      verbosity: error
      type: forbid

could these restrictions be the reason of my problems?

Regards,
Michał

Hi @mikeIT

I suspect that these “forbidden_responses” you see in KBN logs are related to the block with the uri_re rule. Probably it blocks some KBN internal calls.

You can put here the ES logs with FORBIDDEN entries, and we will help to analyse them.

Additionally in Stack management → url to “/s/default/app/management“ I see that clicking in some links I get logged out when in 7.17.29 I got message “Sorry, wrong password or you don’t have permission to this index.“ which I always understood as the proper one :slight_smile:

Yeah, it would be great to know what links exactly, and look for the corresponding FORBIDDEN log.

Hi, Thank you for response.

I forgot to mention that I’m using “xpack.security.enabled: false“

But refering to previous topic, every link from these

+

here, I can see only indices in Indices(permitted from rules, in KBN 7.17.29 I cannot) others don’t work.

+ image

gets me being logged out

ES LOG (separate for every “Stack mangement app“) below

* ”Forbid API calls for some specific roles” is my rule with ‘uri_re’ to restrict permissions for standard/nonAdmin ldap users in “Dev Tools app“

[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetPipelineRequest, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/ingest/pipeline/get, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_ingest/pipeline,...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetDataStreamAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:indices:admin/data_stream/get...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetSnapshotLifecycleAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/slm/get, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_slm/policy,...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetLifecycleAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/ilm/get,...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API callsfor some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetRepositoriesRequest, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/repository/get, OA:test_IP/32, XFF:test_IP, DA:test_IP/32,IDX:<N/A>, MET:GET, PTH:/_snapshot/_all,...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetRollupJobsAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:monitor/xpack/rollup/get, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_rollup/job/_all...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API callsfor some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetFeatureUpgradeStatusRequest, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/migration/get_system_feature, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_migration/system_features,..
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:ClusterGetSettingsAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:monitor/settings, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_cluster/settings...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetEnrichPolicyAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/xpack/enrich/get, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_enrich/policy...
---------------
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetComponentTemplateAction$Request, CGR:XXX_XX, USR:XtestUSER, BRS:true, KDX:null, ACT:cluster:admin/component_template/get, OA:1 test_IP/32, XFF:test_IP, DA:test_IP/32 , IDX:<N/A>, MET:GET, PTH:/_component_template...

Rule restictions still work okay in Dev Tools app

e.g. “‘GET _ilm/policy‘ results in

{
  "error": {
    "root_cause": [
      {
        "type": "forbidden_response",
        "reason": "Sorry, wrong password or you don't have permission to this index.",
        "due_to": "FORBIDDEN_BY_BLOCK",
        "header": {
          "WWW-Authenticate": "Basic"
        }
      }
    ],
    "type": "forbidden_response",
    "reason": "Sorry, wrong password or you don't have permission to this index.",
    "due_to": "FORBIDDEN_BY_BLOCK",
    "header": {
      "WWW-Authenticate": "Basic"
    }
  },
  "status": 401
}

Additionally, I’m still using the same ROR config in kibana 7.17.29+ror 1.67.2 with ES8.19.7 without any errors/problems :wink:

Your logs are truncated. The most important part is not there

the example call your showed, clearly says: “it was forbiden by one of ACL’s block” (FORBIDDEN_BY_BLOCK).

I guess the block with uri_re forbids the call.

Full (edited unnecessary details) log here

[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetComponentTemplateAction$Request, CGR:XXX_XXX, USR:test_USER, BRS:true, KDX:null, ACT:cluster:admin/component_template/get, OA:test_IP/32, XFF:test_IP, DA:test_IP/32, IDX:<N/A>, MET:GET, PTH:/_component_template, CNT:<N/A>, HDR:Host=test_IP:9200, x-ror-correlation-id=XXX, x-opaque-id=unknownId, content-length=0, user-agent=Kibana/8.19.7, x-ror-kibana-request-method=get, x-ror-kibana-request-path=/s/default/api/console/autocomplete_entities, x-ror-kibana-index=.kibana-main, accept=application/vnd.elasticsearch+json; compatible-with=8,text/plain, x-elastic-product-origin=kibana, tracestate=es=s:0, x-forwarded-for=test_IP, traceparent=XXX, x-elastic-client-meta=es=8.19.1,js=22.17.1,t=8.9.6,hc=22.17.1, keep-alive=timeout=10, max=1000, connection=keep-alive, Accept-Charset=utf-8, cookie=x-csrf-token-XXX-session_id=XX_id; x-csrf-token-XX=XX.XX, Authorization=<OMITTED>, HIS:[Accept requests from XXX-> RULES:[hosts->false] RESOLVED:[template=GET(NonEmptyList(*))]], [::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[template=GET(NonEmptyList(*))]], [::KIBANA-SYSTEM::-> RULES:[auth_key->false] RESOLVED:[template=GET(NonEmptyList(*))]], [::LOGSTASH-SYSTEM::-> RULES:[auth_key->false] RESOLVED:[template=GET(NonEmptyList(*))]], [::LOGSTASH-USER::-> RULES:[auth_key->false] RESOLVED:[template=GET(NonEmptyList(*))]], [::Forbid API calls for some specific roles::-> RULES:[ldap_auth->true, uri_re->true] RESOLVED:[user=test_USER;group=XXX;av_groups=user_rules ;template=GET(NonEmptyList(*))]], }ESC[0m

OKay, but why do I get logged out instead of getting message about insufficient permissions e.g. “forbidden“ like in KBN 7.X.

Yeah, the log confirms what I said.

OKay, but why do I get logged out instead of getting message about insufficient permissions

I guess it’s how the 8.x Kibana handles this. AFAIR, we don’t change it.
@Dzuming Could you please confirm that?

In two previous “long time ago” ROR versions there were these improvements :slight_smile:

(2023-10-09) What's new in ROR 1.52.0  
🐞Fix (KBN) Logout after opening Stack management Upgrading assistant
(2024-01-29) What's new in ROR 1.55.0
🐞Fix (ES) Logout when a user with restricted kibana.access tried to see a restoration status of snapshots in Kibana

That seem to be quite similar. Maybe only for 7.X it worked as I thought it will still do in 8.X :upside_down_face:

Dawid checked the KBN code and it seems it behaves like I said by default when 401 is returned.

So, it seems you can remove prompt_for_basic_auth setting from your ROR ES settings and ES should start responding with 403 instead of 401. And there should not be logouts any more when forbidden response is returned.

Yes, I can confirm that after removing prompt_for_basic_auth setting from ROR ES setting, now I get the 403 with my custom message without logouts, I will test it. Thank you very much :wink:

May I have one question not directly related to this topic? Is there any option to restrict ldap RW users from changing “global“ kibana “Advanced settings“?

It’s great that it works!

May I have one question not directly related to this topic? Is there any option to restrict ldap RW users from changing “global“ kibana “Advanced settings“?

Sure. Sadly, there is no reliable way to do that. In the near feature we are planning to work on the kibana.access rule to extend the available access levels, so we will consider this use case too.

But, I see some workarounds (which don’t forbid the update though API, but your Kibana users won’t do that):

  1. kibana.hide_apps
  2. forbidding the update call

the call looks like this:

[2025-12-10T07:35:19,333][INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [es-ror-single] ALLOWED by { name: 'End users', policy: ALLOW, rules: [groups_any_of, kibana, indices] req={ ID:4dc6c0fe-5995-4eb5-bc82-ea96618cd5e2-1206620593#576049, TYP:UpdateRequest, CGR:EndUsers, USR:user1, BRS:true, KDX:.kibana_end_user1, ACT:indices:data/write/update, OA:172.19.0.4/32, XFF:localhost:15601, DA:172.19.0.2/32, IDX:.kibana_end_user1, MET:POST, PTH:/.kibana_end_user1/_update/config:8.7.1, CNT:<OMITTED, LENGTH=91.0 B> , HDR:Host=es-ror:9200, user-agent=Kibana/8.7.1, x-opaque-id=unknownId, x-elastic-client-meta=es=8.6.0p,js=16.19.1,t=8.3.1,hc=16.19.1, cookie=__Host-ror.x-csrf-token-MC4wLjAuMDo1NjAx-session_id=36332f5a6b868b74b1b1f4f8097eb1d4; __Host-ror.x-csrf-token-MC4wLjAuMDo1NjAx=635a8a2d7a545a947371b85e5f2addbe0d732d8a924943a43f068f15f305630b.d51493a43262413c38c4b9460072c9c75923ecd719b1d5ba4ab91d6337d1df4747f4901a7e718b1977ce59705ebb06e1f265ae4c9653c4d4398d3c3d5df427af, elastic-apm-traceparent=00-600a572c9e2672983cf8ae5cfe4ea82f-ee26b85bffd67082-00, x-elastic-product-origin=kibana, tracestate=es=s:0, x-ror-current-group=EndUsers, x-ror-kibana-request-method=post, Authorization=<OMITTED>, accept=application/vnd.elasticsearch+json; compatible-with=8, content-type=application/vnd.elasticsearch+json; compatible-with=8, x-ror-kibana-index=.kibana_end_user1, x-ror-correlation-id=4dc6c0fe-5995-4eb5-bc82-ea96618cd5e2, traceparent=00-600a572c9e2672983cf8ae5cfe4ea82f-ee26b85bffd67082-00, keep-alive=timeout=10, max=1000, connection=keep-alive, Accept-Charset=utf-8, x-ror-kibana-request-path=/s/default/api/kibana/settings, Content-Length=91, x-forwarded-for=localhost:15601, HIS:[KIBANA-> RULES:[auth_key->false] RESOLVED:[group=EndUsers;indices=.kibana_end_user1]], [Admins-> RULES:[groups_any_of->false] RESOLVED:[group=EndUsers;indices=.kibana_end_user1]], [End users-> RULES:[groups_any_of->true, kibana->true, indices->true] RESOLVED:[user=user1;group=EndUsers;av_groups=EndUsers;indices=.kibana_end_user1;kibana_idx=.kibana_end_user1]], }

You can use different rules (eg. uri_re or actions) to identify the type of the request, and narrow its usage to the kibana settings (e.g. by headers rule and this header: x-ror-kibana-request-path=/s/default/api/kibana/settings). And block this call.

Okay, thank you.

But what about this error seen in kibana.log

[ERROR][plugins.dataViews.dataView.hasEsData] ResponseError: forbidden_response
        Root causes:
                forbidden_response: my_custom_message

when opening Home and accesing data_views(<v. 8.X index_patterns) data?

for “/s/default/app/home#/“ i see in Es Log

[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX#XX, TYP:GetDataStreamAction$Request, CGR:ldap_role, USR:test_user, BRS:true, KDX:null, ACT:indices:admin/data_stream/get, OA:node_ip/32, XFF:node_ip, DA:node_ip/32, IDX:<N/A>, MET:GET, PTH:/_data_stream/logs, CNT:<N/A>, HDR:Host=node_ip:9200, x-opaque-id=unknownId, traceparent=XXX, content-length=0, x-ror-kibana-request-method=get, x-ror-kibana-index=.kibana-X, accept=application/vnd.elasticsearch+json; compatible-with=8,text/plain, x-ror-correlation-id=XXX, x-elastic-product-origin=kibana, tracestate=es=s:0, x-elastic-client-meta=es=8.19.1,js=22.17.1,t=8.9.6,hc=22.17.1, Authorization=<OMITTED>, x-ror-kibana-request-path=/s/default/api/streams/_status, user-agent=Kibana/8.19.7, keep-alive=timeout=10, max=1000, connection=keep-alive, Accept-Charset=utf-8, x-forwarded-for=node_ip, cookie=XX; x-csrf-token-XXX-session_id=XXX; x-csrf-token-XXX, HIS:[Accept all requests from localhost-> RULES:[hosts->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::KIBANA-SYSTEM::-> RULES:[auth_key->false]], [::LOGSTASH-SYSTEM::-> RULES:[auth_key->false]], [::LOGSTASH-USER::-> RULES:[auth_key->false]], [::Forbid API calls for some specific roles::-> RULES:[ldap_auth->true, uri_re->true] RESOLVED:[user=test_user;group=ldap_role;av_groups=ldap_roles]], }ESC[0m

and for ‘/s/default/app/discover#/‘ two logs

[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by default req={ ID:XXX#XXX, TYP:ResolveClusterActionRequest, CGR:<N/A>, USR:test_user (attempted), BRS:true, KDX:null, ACT:indices:admin/resolve/cluster, OA:nodeIP/32, XFF:nodeIP, DA:nodeIP/32, IDX:*,.*,logs-enterprise_search.api-default,logs-enterprise_search.audit-default, MET:GET, PTH:/_resolve/cluster/*,-.*,-logs-enterprise_search.api-default,-logs-enterprise_search.audit-default, CNT:<N/A>, HDR:Host=nodeIP:9200, traceparent=XXX, x-opaque-id=unknownId, content-length=0, user-agent=Kibana/8.19.7, x-ror-kibana-request-method=get, x-ror-kibana-index=.kibana-X, accept=application/vnd.elasticsearch+json; compatible-with=8,text/plain, x-ror-correlation-id=XXX, x-elastic-product-origin=kibana, tracestate=es=s:0, x-elastic-client-meta=es=8.19.1,js=22.17.1,t=8.9.6,hc=22.17.1, Authorization=<OMITTED>, keep-alive=timeout=10, max=1000, connection=keep-alive, Accept-Charset=utf-8, x-forwarded-for=nodeIP, cookie=XXX; X_SESSION=XXX; x-csrf-token-XXX; x-csrf-token-XXX; x-csrf-token-XX-session_id=XX; x-csrf-token-XX; x-csrf-token-XX-session_id=XX; x-csrf-token-XX; 12345-X=Fe26.2**XX; x-csrf-token-XX, x-ror-kibana-request-path=/s/default/internal/data_views/has_es_data, HIS:[Accept all requests from localhost-> RULES:[hosts->false] RESOLVED:[indices=*, .*, logs-enterprise_search.api-default, logs-enterprise_search.audit-default]], [::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[user=test_user;group=XX;av_groups=XXX;indices=*, .*, logs-enterprise_search.api-default, logs-enterprise_search.audit-default]],[::KIBANA-SYSTEM::-> RULES:[auth_key->false] RESOLVED:[indices=*, .*, logs-enterprise_search.api-default, logs-enterprise_search.audit-default]], [::Forbid API calls for some specific roles::-> RULES:[ldap_auth->true, uri_re->false], [::X LDAP::-> RULES:[ldap_auth->true, kibana->true, indices->false] RESOLVED:[user=test_user;group=X;av_groups=X;indices=*, .*, logs-enterprise_search.api-default, logs-enterprise_search.audit-default;kibana_idx=.kibana-X]], }ESC[0m
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XX] ESC[35mINDEX NOT FOUND req={ ID:XX, TYP:ResolveIndexAction$Request, CGR:<N/A>, USR:test_user (attempted), BRS:true, KDX:null, ACT:indices:admin/resolve/index, OA:nodeIP/32, XFF:nodeIP, DA:nodeIP/32, IDX:*:*, MET:GET, PTH:/_resolve/index/*:*, CNT:<N/A>, HDR:Host=nodeIP:9200, x-ror-kibana-request-path=/s/default/internal/index-pattern-management/resolve_index/*:*, x-opaque-id=unknownId, traceparent=XXX, content-length=0, user-agent=Kibana/8.19.7, x-ror-kibana-request-method=get, x-ror-kibana-index=.kibana-X, accept=application/vnd.elasticsearch+json; compatible-with=8,text/plain, x-ror-correlation-id=XX, x-elastic-product-origin=kibana, tracestate=es=s:0, x-elastic-client-meta=es=8.19.1,js=22.17.1,t=8.9.6,hc=22.17.1, Authorization=<OMITTED>, keep-alive=timeout=10, max=1000, connection=keep-alive, Accept-Charset=utf-8, x-forwarded-for=nodeIP, cookie=12345-XX=XXX; x-csrf-token-XX-session_id=XX; x-csrf-token-XXX, HIS:[Accept all requests from localhost-> RULES:[hosts->false] RESOLVED:[indices=*:*]], [::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[indices=*:*]], [::KIBANA-SYSTEM::-> RULES:[auth_key->false], [::Forbid API calls for some specific roles::-> RULES:[ldap_auth->true, uri_re->false] RESOLVED:[user=test_user;group=X;av_groups=X, XXX;indices=*:*]], [::X LDAP::-> RULES:[ldap_auth->true, kibana->true, indices->false] RESOLVED: user=test_user;group=XX;av_groups=XX;indices=*:*;kibana_idx=.kibana-X]], [::X LDAP::-> RULES:[ldap_auth->true, kibana->true, indices->false] ]], }ESC[0m

What additional permissions should ldap-user have to disable that specific ERROR in kibana log? Currently, the user can see data for permitted indices via ror config without any warnings in kibana GUI(that are generated only as kibana logs)? I cannot find any resolution or example in documentation ;/

The first log - its request was forbidden by the “Forbid API calls for some specific roles” block.

To analyse the two latter logs, I would need debug logs.
You can add:

logger.ror.name=tech.beshu.ror.accesscontrol.blocks.rules.elasticsearch.indices
logger.ror.level=debug

to log4j2.properties.

And then retest it again and paste logs, which should look like these:

es-ror-1  | [2025-12-10T14:15:36,192][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] Checking local indices (allowed: [frontend_logs, kibana_sample_data_*], requested: [*logs*])
es-ror-1  | [2025-12-10T14:15:36,197][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] Checking - all requested indices relate to Kibana indices ...
es-ror-1  | [2025-12-10T14:15:36,204][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] ... not matched. Continue
es-ror-1  | [2025-12-10T14:15:36,206][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] Checking - none or all indices ...
es-ror-1  | [2025-12-10T14:15:36,226][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] ... indices, aliases and data streams: [.kibana_task_manager, .alerts-transform.health.alerts-default, .inf
erence-alias, .internal.alerts-observability.threshold.alerts-default-000001, .security, .internal.alerts-security.alerts-default-000001, .ds-ilm-history-7-2025.12.10-000001, .alerts-ml.anomaly-detection-health.alerts-default, .internal.alerts-observabi
lity.slo.alerts-default-000001, .alerts-security.alerts-default, .internal.alerts-ml.anomaly-detection.alerts-default-000001, .apm-source-map, .kibana_task_manager_8.19.7_001, business_logs, .ds-.kibana-event-log-ds-2025.12.10-000001, .kibana-siem-rule-
migrations-prebuiltrules, .kibana_task_manager_8.19.7, .alerts-observability.metrics.alerts-default, .kibana_security_solution_8.19.7_001, .internal.alerts-observability.apm.alerts-default-000001, .security-7, .kibana_alerting_cases_8.19.7, .kibana_8.19
.7, .internal.alerts-security.attack.discovery.alerts-default-000001, frontend_logs, .internal.alerts-streams.alerts-default-000001, .alerts-streams.alerts-default, .kibana_ingest_8.19.7, .kibana-event-log-ds, system_logs, .internal.alerts-default.alert
s-default-000001, .kibana_security_session_1, .kibana_alerting_cases_8.19.7_001, .alerts-default.alerts-default, .kibana_usage_counters_8.19.7, .kibana_analytics_8.19.7_001, .internal.alerts-ml.anomaly-detection-health.alerts-default-000001, .apm-custom
-link, .kibana_security_solution_8.19.7, .kibana_security_solution, .alerts-dataset.quality.alerts-default, .kibana_locks-000001, .kibana_usage_counters_8.19.7_001, .siem-signals-default, .alerts-observability.apm.alerts-default, .kibana_alerting_cases,
 .slo-observability.sli-v3.5, .alerts-security.attack.discovery.alerts-default, .alerts-observability.threshold.alerts-default, .apm-agent-configuration, .alerts-observability.slo.alerts-default, .inference, .kibana_ingest_8.19.7_001, .slo-observability
.summary-v3.5.temp, .ds-.logs-deprecation.elasticsearch-default-2025.12.10-000001, .kibana_analytics_8.19.7, .internal.alerts-observability.logs.alerts-default-000001, .slo-observability.summary-v3.5, .secrets-inference, .kibana_analytics, .internal.alerts-observability.uptime.alerts-default-000001, .internal.alerts-stack.alerts-default-000001, .alerts-ml.anomaly-detection.alerts-default, .alerts-observability.logs.alerts-default, .kibana_8.19.7_001, ilm-history-7, .internal.alerts-transform.health.alerts-default-000001, .alerts-stack.alerts-default, .logs-deprecation.elasticsearch-default, .kibana-siem-rule-migrations-integrations, .kibana_security_session, .kibana_usage_counters, .kibana_ingest, .alerts-observability.uptime.alerts-default, .internal.alerts-dataset.quality.alerts-default-000001, .internal.alerts-observability.metrics.alerts-default-000001, .kibana]
es-ror-1  | [2025-12-10T14:15:36,226][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] ... not matched. Continue
es-ror-1  | [2025-12-10T14:15:36,227][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] Checking if all indices are matched ...
es-ror-1  | [2025-12-10T14:15:36,227][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] ... not matched. Continue
es-ror-1  | [2025-12-10T14:15:36,228][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] Checking - indices & aliases & data streams...
es-ror-1  | [2025-12-10T14:15:36,244][DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [es-ror-single] [884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845] ... matched [indices: frontend_logs]. Stop
es-ror-1  | [2025-12-10T14:15:36,246][INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [es-ror-single] ESC[36mALLOWED by { name: 'End users', policy: ALLOW, rules: [groups_any_of, kibana, indices] req={ ID:884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845, TYP:SearchRequest, CGR:EndUsers, USR:user1, BRS:true, KDX:.kibana_end_user1, ACT:indices:data/read/search, OA:192.168.127.1/32, XFF:null, DA:10.89.2.3/32, IDX:*logs*, MET:GET, PTH:/*logs*/_search, CNT:<N/A>, HDR:Accept=*/*, content-length=0,
User-Agent=curl/8.7.1, Host=localhost:19200, Authorization=<OMITTED>, HIS:[KIBANA-> RULES:[auth_key->false] RESOLVED:[indices=*logs*]], [Admins-> RULES:[groups_any_of->false] RESOLVED:[indices=*logs*]], [End users-> RULES:[groups_any_of->true, kibana->t
rue, indices->true] RESOLVED:[user=user1;group=EndUsers;av_groups=EndUsers;indices=frontend_logs;kibana_idx=.kibana_end_user1]], }

With added these lines in ES log4j2 config

logger.ror.name=tech.beshu.ror.accesscontrol.blocks.rules.elasticsearch.indices
logger.ror.level=debug

I have:

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [.kibana-X])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: .kibana-X]. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [.kibana-X])

… (many similar group log lines), then

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [.kibana-X])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... not matched. Continue
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: .kibana-X]. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - none or all indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... indices, aliases and data streams: [ elastic cluster indices ]

then

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: *]. Stop
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by { name: '::Forbid API calls for some specific roles::', policy: FORBID, rules: [ldap_auth, uri_re] req={ ID:XXX, TYP:GetDataStreamAction$Request, CGR:Xrole, USR:testUser, BRS:true, KDX:null, ACT:indices:admin/data_stream/get 
...

then

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .report
ing-*], requested: [.kibana-X])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: .kibana-]. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [.kibana-X_analytics_8.19.7])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: .kibana-X_analytics_8.19.7]. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [.kibana-X]

the same logs few times

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: .kibana-X_analytics_8.19.7]. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [*, .*, logs-enterprise_search.api-default, logs-enterprise_search.audit-default])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... not matched. Continue
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - write request ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Stage 7
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Stage 8
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... not matched. Stop

and then

[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by default req={ ID:XXX, TYP:ResolveClusterActionRequest, CGR:<N/A>, USR:testUser (attempted), BRS:true, KDX:null, ACT:indices:admin/resolve/cluster, OA:nodeIp/32, XFF:nodeIp, DA:nodeIp/32, IDX:*,.*,logs-enterprise_search.api-default,logs-enterprise_search.audit-default, MET:GET, PTH:/_resolve/cluster/*,-.*,-logs-enterprise_search.api-default,-logs-enterprise_search.audit-default,
...

and

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... matched [indices: *]. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking remote indices (allowed: [], requested: [*:*])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... not matched. Continue
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking - none or all indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... indices, aliases and data streams: []
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] ... not matched. Index not found. Stop
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [XXX] [XXX] Checking remote indices (allowed: [], requested: [*:*])
[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mINDEX NOT FOUND req={ ID:XXX, TYP:ResolveIndexAction$Request, CGR:<N/A>, USR:testUser (attempted), BRS:true, KDX:null, ACT:indices:admin/resolve/index, OA:nodeIp/32, XFF:nodeIp, DA:nodeIp/32, IDX:*:*, MET:GET, PTH:/_resolve/index/*:*,
...

I should have told you that you should first identify the request you want to check. E.g. the “FORBIDDEN” log. Then, from this log, take the ID (e.g. “ID:884f13d5-4f13-4f51-b06f-523151299542-1815822926#12845”) and grep the log file with this ID. Then put all the filtered log entries here.

okay, so for one common request id that would be this

[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] Checking local indices (allowed: [.kibana-reporting-*, X-*, .ds-.kibana-*, .kibana*, .reporting-*], requested: [*, .*, logs-enterprise_search.api-default, logs-enterprise_search.audit-default])
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] Checking - all requested indices relate to Kibana indices ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] ... not matched. Continue
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] Checking - write request ...
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] Stage 7
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] Stage 8
[DEBUG][t.b.r.a.b.r.e.i.IndicesRule] [ESnode] [XXXX-XX-XX-XX-XX-XX#XX] ... not matched. Stop

*many times according to permitted indices from all ror rules for all “ldap groups”

and then that last ES log

[INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [XXX] ESC[35mFORBIDDEN by default req={ ID:XXX, TYP:ResolveClusterActionRequest, CGR:<N/A>, USR:testUser (attempted), BRS:true, KDX:null, ACT:indices:admin/resolve/cluster, OA:nodeIp/32, XFF:nodeIp, DA:nodeIp/32, IDX:*,.*,logs-enterprise_search.api-default,logs-enterprise_search.audit-default, MET:GET, PTH:/_resolve/cluster/*,-.*,-logs-enterprise_search.api-default,-logs-enterprise_search.audit-default,
...

for the timestamp before kibana logging

[XXX][ERROR][plugins.dataViews.dataView.hasEsData] ResponseError: forbidden_response
        Root causes:
                forbidden_response: my_custom_massage

ok, thanks. Now, I see the problem.
Here is a pre-build with fix: ROR 1.68.0-pre10 for 8.19.7 (you can use it with ROR KBN 1.67.x)

it would be great if you could test in on your end

I’ve tested “ROR 1.68.0-pre10 for 8.19.7“ for ES 8.19.7 and do not see neither any ERROR logs in kibana :slight_smile: nor any other problems. In ES LOG just some “Forbidden” like already sent.

The „FORBIDDEN by { name: '::Forbid API calls for some specific roles::” entries show correct behaviour. The block matched the request and forbid it (due to the forbid policy).

Do you see any other forbidden logs than that?

No, not any other logs than the ones refering to “FORBIDDEN by { name: ‘::Forbid API calls for some specific roles::’, policy: FORBID,…“