ROR Forbidden when upgrading Kibana from 5.5 to 6.2.X

(Roger Seth) #1

HI there,
We recently upgraded our ELK stack from 5.5 to 6.2.X. while for the most parts things worked well, we found that as part of the upgrade we had to migrate our Kibana index as indicated in this page:

What this resulted in is a creation of a new .kibana-6 index aliased to .kibana as explained in that migration step. After we bring up Kibana - ROR throws this error. IT feels like the kibana/kibana auth is no longer passing. Any ideas?

".kibana-6": {
    "aliases": {
        ".kibana": {}


[2018-06-06T14:07:58,654][INFO ][t.b.r.a.ACL ] ^[[35mFORBIDDEN by default req={ ID:104099432-366820386#21395, TYP:IndexRequest, CGR:N/A, USR:[no basic auth header], BRS:true, KDX:null, ACT:indices:data/write/index, OA:, DA:, IDX:myindex, MET:POST, PTH:/myindex/ctr, CNT:<OMITTED, LENGTH=1152>, HDR:{Accept=/, Accept-Encoding=gzip, deflate, Connection=keep-alive, Content-Length=1152, Content-Type=application/json,, User-Agent=python-requests/2.18.4, X-Forwarded-For=, X-Forwarded-Port=443, X-Forwarded-Proto=https}, HIS:[::KIBANA-SRV 1::->[auth_key->false]], [::RO::->[auth_key->false]], [::RW::->[auth_key->false]], [GlobalWrite/Admin Access->[auth_key->false]], [Route53 Access->[actions->false, x_forwarded_for->true]], [::NGINX-RESTRICTED-GROUP::->[proxy_auth->false]], [::NGINX-READONLY-GROUP::->[proxy_auth->false]], [::NGINX-ADMIN-GROUP::->[proxy_auth->false]]

This is my kibana config (hasn’t changed between 5.5 and 6.2):

elasticsearch.username: "kibana"
elasticsearch.password: "kibana"

elasticsearch.requestHeadersWhitelist: [ authorization, x-forwarded-user, x-forwarded-group ]
logging.verbose: true

ROR Config (used to work with the old .kibana index):

prompt_for_basic_auth: false

- name: "::KIBANA-SRV 1::"
  kibana_access: admin
  auth_key: kibana:kibana
  verbosity: error

- name: "Global Write/Admin Access"
  auth_key: elastic:elastic
  type: allow
  actions: ["indices:data/write/*","indices:admin/*", "cluster:admin/*", "cluster:monitor/*", "indices:monitor/*"]
  verbosity: error

- name: "Route53 Access"
  type: allow
  x_forwarded_for: [""]
  actions: ["cluster:monitor/*", "indices:admin/get", "indices:admin/aliases", "indices:admin/aliases/*", "indices:admin/analyze", "indices:monitor/*", "indices:data/read/*"]
  verbosity: error

(Simone Scarduzio) #2

Hi @Daedelus,
From the log line you showed, seems like somebody without credentials is trying to write into an index called “myindex”. It’s expected that you get a forbidden.

(Roger Seth) #3

That’s the strange part. It’s only me who has access to this server and this error is thrown when I REFRESH the Kibana home page. I get a big red bar with “FORBIDDEN” on the top of the kibana discovery page

(Simone Scarduzio) #4

is it the load balancer probe?

(Roger Seth) #5

Interesting. It could mean some change in my Route53 write request. Hadn’t thought of that… Let me see if putting this rule into R53 does the trick

(Roger Seth) #6

That did it! Thanks - Should’ve thought of R53 rule earlier. thanks for the tip!