Kibana alerting not working with readonlyrest

1

My company is a ReadonlyREST Enterprise subscriber and we are planning to upgrade to Elastic Stack 7.14.1 from 7.6.2

Alerting seems to be blocked by ReadOnlyRest

In my kibana logs I can see:

log   [08:05:05.235] [error][alerting][alerting][plugins][plugins] Executing Alert "6a6322e0-1afe-11ec-a178-c5e0cc2059fa" has resulted in Error: This action is forbidden. Contact Logging and Telemetry team for access   : forbidden_response: [forbidden_response] Reason: This action is forbidden. Contact Logging and Telemetry team for access   , caused by: ""
  log   [08:05:05.235] [error][alerting][alerting][plugins][plugins] Executing Alert "6a648270-1afe-11ec-a178-c5e0cc2059fa" has resulted in Error: This action is forbidden. Contact Logging and Telemetry team for access   : forbidden_response: [forbidden_response] Reason: This action is forbidden. Contact Logging and Telemetry team for access   , caused by: ""
  log   [08:05:08.251] [error][alerting][alerting][plugins][plugins] Executing Alert "6a6a00b0-1afe-11ec-a178-c5e0cc2059fa" has resulted in Error: This action is forbidden. Contact Logging and Telemetry team for access   : forbidden_response: [forbidden_response] Reason: This action is forbidden. Contact Logging and Telemetry team for access   , caused by: ""
  log   [08:05:08.281] [error][alerting][alerting][plugins][plugins] Executing Alert "6a6d0df0-1afe-11ec-a178-c5e0cc2059fa" has resulted in Error: This action is forbidden. Contact Logging and Telemetry team for access   : forbidden_response: [forbidden_response] Reason: This action is forbidden. Contact Logging and Telemetry team for access   , caused by: ""

In elasticsearch logs:

[2021-09-23T08:03:02,169][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [client-kibana-vc2coma1552312np] FORBIDDEN by default req={ ID:1588684757-1962051641#1740859, TYP:GetRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/get, OA:10.146.90.15/32, XFF:null, DA:10.146.90.15/32, IDX:.kibana_7.14.1, MET:GET, PTH:/.kibana_7.14.1/_doc/alert:6a6d0df0-1afe-11ec-a178-c5e0cc2059fa, CNT:<N/A>, HDR:Accept-Charset=utf-8, Host=vc2coma1552312np:9200, connection=close, content-length=0, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, x-opaque-id=dfd8794f-a704-4103-ae9f-133fa87c925c, HIS:[Forbid all LDAP users to delete indices-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [For Logging and Telemtry Team-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow login for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow access for splunkUI-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Forbid below indices for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [Forbid below indices for splunkUI user-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow access to all indices for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [::splunk-UI::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [::LOGSTASH::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], }
[2021-09-23T08:03:02,169][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [client-kibana-vc2coma1552312np] FORBIDDEN by default req={ ID:1591437181-813253660#1740860, TYP:GetRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/get, OA:10.146.90.15/32, XFF:null, DA:10.146.90.15/32, IDX:.kibana_7.14.1, MET:GET, PTH:/.kibana_7.14.1/_doc/space:default, CNT:<N/A>, HDR:Accept-Charset=utf-8, Host=vc2coma1552312np:9200, connection=close, content-length=0, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, x-opaque-id=d880af55-a6b9-48d4-9380-e69cc632f366, HIS:[Forbid all LDAP users to delete indices-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [For Logging and Telemtry Team-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow login for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow access for splunkUI-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Forbid below indices for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [Forbid below indices for splunkUI user-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow access to all indices for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [::splunk-UI::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [::LOGSTASH::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], }
[2021-09-23T08:03:02,170][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [client-kibana-vc2coma1552312np] FORBIDDEN by default req={ ID:1174637734-1126744422#1740861, TYP:GetRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/get, OA:10.146.90.15/32, XFF:null, DA:10.146.90.15/32, IDX:.kibana_7.14.1, MET:GET, PTH:/.kibana_7.14.1/_doc/alert:6a6a00b0-1afe-11ec-a178-c5e0cc2059fa, CNT:<N/A>, HDR:Accept-Charset=utf-8, Host=vc2coma1552312np:9200, connection=close, content-length=0, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.15.2.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, x-opaque-id=d880af55-a6b9-48d4-9380-e69cc632f366, HIS:[Forbid all LDAP users to delete indices-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [For Logging and Telemtry Team-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow login for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow access for splunkUI-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Forbid below indices for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [Forbid below indices for splunkUI user-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Allow access to all indices for LDAP users-> RULES:[ldap_authentication->false] RESOLVED:[indices=.kibana_7.14.1]], [::splunk-UI::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [::LOGSTASH::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [::KIBANA-SRV::-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], }

my kibana.yml

elasticsearch.hosts: [ "https://vc2coma1552312np:9200" ]
elasticsearch.requestTimeout: 180000
elasticsearch.ssl.verificationMode: none

elasticsearch.username: "kibana" # Kibana server use ::KIBANA-SRV:: credentials
elasticsearch.password: "xxxxxxx"

server.port: 5601
server.host: vc2coma1552312np.fmr.com
server.ssl.enabled: true
server.ssl.key: "/apps/elk/kibana/config/ssl.key"
server.ssl.certificate: "/apps/elk/kibana/config/appcertificate.crt"
server.ssl.keyPassphrase: xxxxxxx

xpack.reporting.queue.timeout: 180000
xpack.reporting.kibanaServer.port: 80
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: vc2coma1552312np

telemetry.enabled: false
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false


xpack.encryptedSavedObjects.encryptionKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xpack.reporting.encryptionKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xpack.task_manager.monitored_stats_health_verbose_log.enabled: true


readonlyrest_kbn.cookiePass: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
readonlyrest_kbn.store_sessions_in_index: true
readonlyrest_kbn.whitelistedPaths: [".*/api/status$"]

server.publicBaseUrl: https://lt-test.fmr.com:5601
2

I was able to reproduce this using a tenancy on the default “.kibana” index. Will discuss this internally and let you know.

3

@sscarduzio any updates on this issue?

4

Hi @zeeshan yes, we made alerting work today. Although, for the time being, it works only if you are on the main tenancy (kibana_access rule is absent). Would that be helpful for you?

5

When can we expect this to work on all tenancy?

6

I will let you know when we decide the best way to achieve this.

7

The issue is fixed for multiple tenancies now. Next release will have the fix. We will send you a pre-release soon in a private message in this forum.

8

This fix is available in 1.36.0 (will be available later today).