Elastic and Kibana 7.16.3
RoR for ES 1.43.0
After upgrade from Elastic and Kibana 7.9.3 and RoR for ES 1.21.0, Kibana users can’t authentication.
Index are mismached!!!
Debug log:
[2022-09-15T10:33:00,551][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] Checking local indices (allowed: [.kibana,wazuh-alerts-*,wazuh-monitoring-*,wazuh-statistics-*], requested: [.kibana_7.16.3])
[2022-09-15T10:33:00,551][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] Checking - none or all indices ...
[2022-09-15T10:33:00,552][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] ... indices and aliases: [aml-job-2022.09.14,syslog-app2-2022.09.13,wu-connector-rails-2022.09.12,thief-aspsp_requests-2022.09.15,apex-connector-rails-2022.09.12,syslog-redis01-2022.09.15,apex-connector-errors-2022.09.12,safe-errors-2022.09.13,thief-aspsp_requests-2022.09.14,jpm-demo-errors-2022.09.12,priora-rails-2022.09.12,syslog-vault01-2022.09.11,syslog-etcd01-2022.09.02,push-service-errors-2022.09.04,wu-priora-sidekiq-2022.09.08,connector-sidekiq-2022.09.01,billing-job-2022.09.09,wu-connector-rails-2022.09.11,syslog-redis01-2022.09.05,safe-rails-2022.09.06,priora-rails-2022.09.01,aml-rails-2022.09.03,syslog-selenium3-2022.09.14,push-service-sidekiq-2022.09.09,wu-priora-job-2022.09.13,tpp_verifier-job-2022.09.02,syslog-etcd01-2022.09.13,syslog-ci1-2022.09.12,jpm-demo-job-2022.09.05,wazuh-statistics-2022.35w,syslog-app3-2022.09.06,priora-rails-2022.09.10,sca_service-rails-2022.09.05,wu-priora-sidekiq-2022.09.05,desk-job-2022.09.13,syslog-etcd01-2022.09.01,sca_service-job-2022.09.09,sca_service-rails-2022.09.11,syslog-ci2-2022.09.03,sca_service-errors-2022.09.06,connector-errors-2022.09.12,tpp-verifier-sidekiq-2022.09.07,syslog-app1-2022.09.08,wu-connector-sidekiq-2022.07.20,wazuh-alerts-4.x-2022.09.10,syslog-ci3-2022.09.13,caf-connector-job-2022.09.02,access-staging2-2022.09.14,robber-queue-2022.09.04,desk-rails-2022.09.15,safe-job-2022.09.15,stealer-sinatra-2022.09.03,tpp-verifier-rails-2022.09.08,syslog-etcd03-2022.09.09,syslog-lb-redis01-2022.09.03,connector-job-2022.09.15,sso-errors-2022.09.13,push-service-job-2022.09.09,public-site-sidekiq-2022.09.07,push-service-sidekiq-2022.09.07,wu-connector-rails-2022.09.01,localizer-sidekiq-2022.09.03,syslog-backup1-2022.09.05,finastra_connector-errors-2022.09.08,categorizer-analytics-categorization-2022.09.06,tpp-verifier-sidekiq-2022.09.14,safe-errors-2022.09.07,safe-rails-2022.09.12,syslog-ci1-2022.09.01,syslog-etcd01-2022.09.12,tpp_verifier-job-2022.09.05,robber-exception-2022.09.02,syslog-staging3-2022.09.06,connector-rails-2022.09.13,stealer-sinatra-2022.09.15,sso-sidekiq-2022.09.02,localizer-rails-2022.09.09,safe-job-2022.09.04,syslog-db3-2022.09.10,%{[@metadata][target_index]}-2022.09.13,priora-job-2022.09.01,categorizer-job-2022.09.11,icinga2-2022.09.09,.kibana_task_manager_7.16.3_001,%{[@metadata][target_index]}-2022.09.08,bucket-errors-2022.09.10,apex-connector-errors-2022.09.01,.kibana_task_manager,apex-connector-rails-2022.09.01,connector-rails-2022.09.06,thief-rails-2022.09.12,finastra_connector-job-2022.09.13,connector-errors-2022.09.01,thief-aspsp_requests-2022.09.03,syslog-app2-2022.09.02,syslog-app2-2022.09.06,billing-job-2022.09.02,bucket-sidekiq-2022.09.10,syslog-etcd03-2022.09.10,billing-rails-2022.09.11,wazuh-monitoring-2022.09.12,bucket-rails-2022.09.13,syslog-ci1-2022.09.07,sso-errors-2022.09.05,doctor-errors-2022.09.08,priora-job-2022.09.04,syslog-lb-redis02-2022.09.04,sca_service-errors-2022.09.02,syslog-staging2-2022.09.01,wu-priora-job-2022.09.07,syslog-staging2-2022.09.12,thief-errors-2022.09.04,syslog-app2-2022.09.05,connector-sidekiq-2022.09.14,aml-errors-2022.09.08,syslog-ci2-2022.09.07,thief-queue-2022.09.08,doctor-job-2022.09.10,bucket-rails-2022.09.02,finastra_connector-job-2022.09.09,icinga2-2022.09.10,wu-connector-rails-2022.09.03,sso-errors-2022.09.02,safe-sidekiq-2022.09.03,doctor-job-2022.09.13,public-site-sidekiq-2022.09.13,robber-sinatra-2022.09.04,sca_service-sidekiq-2022.09.02,landsbankinn-connector-job-2022.09.15,billing-errors-2022.09.02,syslog-selenium3-2022.09.03,localizer-sidekiq-2022.09.14,desk-errors-2022.09.10,icinga2-2022.09.15,syslog-lb-redis02-2022.09.09,syslog-backup1-2022.09.13,syslog-app3-2022.09.02,syslog-hv01-2022.09.13,aml-rails-2022.09.14,safe-job-2022.09.02,connector-job-2022.09.04,caf-connector-job-2022.09.06,categorizer-job-2022.09.05,thief-rails-2022.09.10,syslog-hv01-2022.09.10,doctor-sidekiq-2022.09.14,thief-aspsp_requests-2022.09.04,push-service-rails-2022.09.04,thief-queue-2022.09.15,billing-errors-2022.09.05,syslog-ci3-2022.09.08,categorizer-rails-2022.09.01,wu-connector-errors-2022.09.05,syslog-monitoring2-2022.09.10,fencer-sinatra-2022.09.11,access-staging2-2022.09.10,aml-rails-2022.09.07,doctor-rails-2022.09.05,syslog-hv01-2022.09.03,categorizer-analytics-categorization-2022.09.02,%{[@metadata][target_index]}-2022.09.06,thief-rails-2022.09.01,billing-rails-2022.09.03,tpp-verifier-sidekiq-2022.09.10,syslog-staging3-2022.09.10,caf-connector-job-2022.09.10,jpm-demo-rails-2022.09.04,public-site-rails-2022.09.11,syslog-backup1-2022.09.02,sso-sidekiq-2022.09.07,syslog-backup1-2022.09.09,.kibana-event-log-7.16.3,robber-exception-2022.09.06,priora-errors-2022.09.05,finastra_connector-sidekiq-2022.09.12,categorizer-analytics-categorization-2022.09.15,priora-rails-2022.09.05,billing-job-2022.09.13,robber-queue-2022.09.09,wu-connector-rails-2022.09.14,syslog-lb-redis01-2022.09.08,fencer-sinatra-2022.09.06,bucket-rails-2022.09.10,tpp_verifier-job-2022.09.13,sca_service-sidekiq-2022.09.13,syslog-staging4-2022.09.08,doctor-job-2022.09.02,robber-sinatra-2022.09.15,billing-errors-2022.09.13,desk-job-2022.09.10,aml-rails-2022.09.13,public-site-sidekiq-2022.09.02,apex-connector-errors-2022.09.08,public-site-sidekiq-2022.09.10,syslog-monitoring2-2022.09.06,push-service-errors-2022.09.11,push-service-rails-2022.09.08,syslog-selenium3-2022.09.02,sca_service-job-2022.09.02,wu-connector-rails-2022.09.07,thief-exception-2022.09.10,connector-rails-2022.09.10,push-service-job-2022.09.07,billing-job-2022.09.06,syslog-staging4-2022.09.11,doctor-job-2022.09.14,billing-rails-2022.09.15,doctor-errors-2022.09.04,bucket-job-2022.09.11,syslog-redis02-2022.09.08,.kibana_7.16.3_001,bucket-errors-2022.09.07,thief-exception-2022.09.09,thief-errors-2022.09.08,public-site-rails-2022.09.09,thief-queue-2022.09.04,categorizer-job-2022.09.15,push-service-job-2022.09.14,bucket-sidekiq-2022.09.07,categorizer-analytics-categorization-2022.09.13,public-site-job-2022.09.08,syslog-hv01-2022.09.04,access-staging2-2022.09.08,wu-priora-job-2022.09.02,syslog-ci2-2022.09.10,syslog-staging2-2022.09.07,bucket-sidekiq-2022.09.13,syslog-redis01-2022.09.04,wu-connector-sidekiq-2022.09.08,icinga2-2022.09.03,syslog-vault01-2022.09.05,localizer-sidekiq-2022.09.10,billing-job-2022.09.01,connector-job-2022.09.08,icinga2-2022.09.11,wu-priora-errors-2022.09.06,doctor-sidekiq-2022.09.03,ilm-history-2-000019,desk-job-2022.09.04,icinga2-2022.08.25,billing-job-2022.09.12,sca_service-job-2022.09.13,access-staging2-2022.09.03,syslog-lb-redis01-2022.09.15,robber-sinatra-2022.09.07,syslog-app3-2022.09.13,sso-rails-2022.09.09,robber-sinatra-2022.09.01,syslog-etcd03-2022.09.11,fencer-sinatra-2022.09.15,wu-priora-sidekiq-2022.09.02,categorizer-rails-2022.09.05,billing-errors-2022.09.09,thief-exception-2022.09.03,push-service-sidekiq-2022.09.14,desk-rails-2022.09.05,public-site-job-2022.09.12,syslog-db3-2022.09.07,syslog-ci3-2022.09.05,desk-job-2022.09.08,apex-connector-sidekiq-2022.09.05,aml-rails-2022.09.02,robber-exception-2022.09.14,thief-rails-2022.09.05,syslog-staging3-2022.09.03,categorizer-analytics-uncategorized-merchants-2022.09.01,syslog-etcd02-2022.09.13,desk-job-2022.09.03,jpm-demo-errors-2022.09.02,syslog-staging4-2022.09.09,categorizer-errors-2022.09.15,jpm-demo-rails-2022.09.08,push-service-rails-2022.09.15,aml-errors-2022.09.12,syslog-staging3-2022.09.14,connector-errors-2022.09.06,syslog-etcd02-2022.09.02,wu-connector-rails-2022.09.09,sso-rails-2022.09.06,tpp-verifier-sidekiq-2022.09.03,robber-sinatra-2022.09.13,public-site-rails-2022.09.15,finastra_connector-rails-2022.09.07,priora-sidekiq-2022.09.08,wazuh-alerts-4.x-2022.09.03,caf-connector-sidekiq-2022.09.09,robber-queue-2022.09.05,priora-job-2022.09.15,aml-rails-2022.09.08,business-intelligence-sidekiq-2022.09.06,syslog-lb-redis01-2022.09.04,desk-rails-2022.09.14,connector-sidekiq-2022.09.11,priora-rails-2022.09.09,public-site-sidekiq-2022.09.06,doctor-errors-2022.09.11,aml-errors-2022.09.06,wazuh-alerts-4.x-2022.09.06,wu-priora-errors-2022.09.13,priora-job-2022.09.12,doctor-job-2022.09.06,syslog-etcd01-2022.09.06,syslog-hv01-2022.09.08,safe-rails-2022.09.01,robber-exception-2022.09.12,jpm-demo-rails-2022.09.01,caf-connector-sidekiq-2022.09.02,wu-priora-sidekiq-2022.09.04,priora-errors-2022.09.14,syslog-redis02-2022.09.12,thief-errors-2022.09.10,billing-rails-2022.09.07,access-staging2-2022.09.02,syslog-lb-redis02-2022.09.13,categorizer-errors-2022.09.07,syslog-ci2-2022.09.04,syslog-redis02-2022.09.04,priora-errors-2022.09.03,connector-job-2022.09.12,sca_service-rails-2022.09.09,bucket-rails-2022.09.07,%{[@metadata][target_index]}-2022.09.14,sca_service-job-2022.09.15,fencer-sinatra-2022.09.04,bucket-job-2022.09.15,wu-connector-sidekiq-2022.09.12,sso-rails-2022.09.14,public-site-job-2022.09.04,categorizer-rails-2022.09.11,bucket-job-2022.09.08,desk-rails-2022.09.03,safe-errors-2022.09.03,syslog-vault01-2022.09.09,ilm-history-2-000018,landsbankinn-connector-sidekiq-2022.09.14,desk-errors-2022.09.03,finastra_connector-sidekiq-2022.09.08,push-service-rails-2022.09.10,caf-connector-job-2022.09.13,safe-job-2022.09.12,sca_service-job-2022.09.04,thief-aspsp_requests-2022.09.13,caf-connector-job-2022.09.09,syslog-etcd02-2022.09.05,public-site-job-2022.09.01,apex-connector-rails-2022.09.07,aml-job-2022.09.03,finastra_connector-sidekiq-2022.09.15,thief-queue-2022.09.09,syslog-app3-2022.09.07,wazuh-monitoring-2022.09.01,public-site-rails-2022.09.04,safe-job-2022.09.05,connector-errors-2022.09.13,wu-priora-sidekiq-2022.09.11,syslog-ci2-2022.09.15,categorizer-rails-2022.09.09,finastra_connector-job-2022.09.12,thief-rails-2022.09.11,syslog-selenium3-2022.09.08,sca_service-sidekiq-2022.09.07,syslog-lb-redis02-2022.09.15,stealer-sinatra-2022.09.07,wu-priora-errors-2022.09.02,connector-errors-2022.09.02,robber-exception-2022.09.07,priora-rails-2022.09.13,jpm-demo-rails-2022.09.12,connector-rails-2022.09.09,wazuh-monitoring-2022.09.08,landsbankinn-connector-sidekiq-2022.08.01,tpp-ver
[2022-09-15T10:33:00,553][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] ... not matched. Continue
[2022-09-15T10:33:00,553][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] Checking if all indices are matched ...
[2022-09-15T10:33:00,553][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] ... not matched. Continue
[2022-09-15T10:33:00,553][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] Checking - indices & aliases ...
[2022-09-15T10:33:00,555][DEBUG][t.b.r.a.b.r.i.IndicesRule] [host.example.com][1832519757--196322652#417774] ... not matched. Stop!
[2022-09-15T10:33:00,556][DEBUG][t.b.r.a.b.Block ] [host.example.com][wazuh-readers] the request matches no rules in this block: { ID:1832519757--196322652#417774, TYP:SearchRequest, CGR:N/A, USR:maxim.cujba (attempted), BRS:true, KDX:null, ACT:indices:data/read/search, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:.kibana_7.16.3, MET:POST, PTH:/.kibana_7.16.3/_search, CNT:{"size":1000,"seq_no_primary_term":true,"from":0,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":"space"}}],"must_not":[{"exists":{"field":"namespace"}},{"exists":{"field":"namespaces"}}]}}],"minimum_should_match":1}}]}},"sort":[{"space.name.keyword":{"unmapped_type":"keyword"}}]}, HDR:Authorization=<OMITTED>, Connection=keep-alive, Host=localhost:9200, content-length=312, content-type=application/json, user-agent=elasticsearch-js/7.16.0-canary.7 (linux 4.19.0-20-amd64-x64; Node.js v16.13.0), x-elastic-client-meta=es=7.16.0p,js=16.13.0,t=7.16.0p,hc=16.13.0, x-elastic-product-origin=kibana, x-opaque-id=8bb24fb5-df80-4196-99d6-c5465d851ac0, HIS:[wazuh-readers-> RULES:[ldap_authentication->true, ldap_authorization->true, kibana_access->true, indices->false] RESOLVED:[user=maxim.cujba;group=wazuh;av_groups=wazuh;indices=.kibana_7.16.3]], }
ES aliases:
h# curl -k -XGET -u user:pass "https://localhost:9200/_cat/aliases" | grep kibana
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 630 100 630 0 0 14318 0 --:--:-- --:--:-- --:--:-- 14651
.kibana .kibana_7.16.3_001 - - - -
.kibana_7.16.3 .kibana_7.16.3_001 - - - -
.kibana_task_manager .kibana_task_manager_7.16.3_001 - - - -
.kibana_task_manager_7.16.3 .kibana_task_manager_7.16.3_001 - - - -
.kibana-event-log-7.16.3 .kibana-event-log-7.16.3-000001 - - - true
Can you help to identify the root cause of this issue?