Can't log into Kibana after upgrade

#1

After upgrading cluster from 7.9.3 to 7.10.1 (including ES, RoR for ES, Kibana), I am not able to log into Kibana anymore. I am getting

[2020-12-23T12:49:18,973][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [node3.elk-apps.nbg-dc3.example.com]FORBIDDEN by default req={  ID:1396755078--1811028680#153118,  TYP:SearchRequest,  CGR:N/A,  USR:[no info about user],  BRS:true,  KDX:null,  ACT:indices:data/read/search,  OA:192.168.11.11/32,  XFF:null,  DA:192.168.11.5/32,  IDX:.kibana,  MET:POST,  PTH:/.kibana/_search,  CNT:<OMITTED, LENGTH=291.0 B> ,  HDR:Connection=keep-alive, Host=node3.elk-apps.nbg-dc3.example.com:9200, content-length=291, content-type=application/json, user-agent=elasticsearch-js/7.10.0-rc.1 (linux 4.19.0-13-amd64-x64; Node.js v10.22.1), x-elastic-product-origin=kibana, x-opaque-id=85349e9c-e8d1-427f-8180-f9e350d5ec95,  HIS: ...skipped... ]  }

while trying to log into Kibana from new private Firefox window. I am not asked about any HTTP auth, the request is immediately declined. The following is logged in reverse proxy log:

10.0.0.104 - - [23/Dec/2020:13:28:53 +0000] "GET / HTTP/1.1" 401 105 "-" "curl/7.68.0"

Before upgrade happened everything worked as expected.

#2

@drdev we have no availability of an official ROR Kibana plugin for 7.9 and older just yet.

Have you been using Kibana without ROR plugin? With Elasticsearch protected by ROR for ES?

#3

We are using just RoR for ES.

#4

It’s not forwarding the authentication header.

Check kibana.yml for:

elasticsearch.username: xxx
elasticsearch,password: yyy

#5

Well. The same config for Kibana, RoR for ES worked perfectly for 7.9.3. I have read changelog and found nothing about changing auth scheme.

#6

of course I have es.username/password set in my Kibana configs.

#7

@sscarduzio Ok, I got minimal setup to reproduce the issue, see below:

elasticsearch.yml 
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.100.201
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: false
readonlyrest.yml 
readonlyrest:
  enable: true
  prompt_for_basic_auth: true
  response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin
  access_control_rules:

  - name: "::DEFAULT::"
    type: allow
    auth_key: kibana:kibana
kibana.yml 
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.100.201:9200"]
kibana.index: ".kibana_1"
kibana.defaultAppId: "home"
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
ES log excerpt 
[2020-12-29T09:05:08,193][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [elastic.test] 
FORBIDDEN by default req={ 
 ID:1544334323-995219539#3229,  TYP:SearchRequest,  CGR:N/A,  USR:[no info about user],  BRS:true,  KDX:null, 
 ACT:indices:data/read/search,  OA:192.168.100.199/32,  XFF:null,  DA:192.168.100.201/32,  IDX:.kibana_1,  MET:POST,  
 PTH:/.kibana_1/_search,  CNT:<OMITTED, LENGTH=291.0 B> ,  HDR:Connection=keep-alive, Host=192.168.100.201:9200, 
 content-length=291, content-type=application/json, user-agent=elasticsearch-js/7.10.0-rc.1 (linux 4.19.0-13-amd64-x64; Node.js v10.22.1), 
 x-elastic-product-origin=kibana, x-opaque-id=91670809-74e5-4ce0-8512-be9ffe9b6876,  HIS:[::DEFAULT::-> RULES:[auth_key->false], 
 RESOLVED:[indices=.kibana_1]]  
 }
ES version info 
{
  "name" : "elastic.test",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "SUe4Av11S7mr_mxcuk3XCA",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
Kibana response 
{"statusCode":401,"error":"Unauthorized","message":"Forbidden by ReadonlyREST ES plugin: Response Error"}

Hopefully this helps, I am stuck. What else am I missing?

#8

@drdev can you try adding prompt_for_basic_auth: true and see if it works? Its supposed to default to true. Not sure if newer version of ROR changed the default behavior. So worth checking. You should atleast get the browser based basic auth prompt.

Even with this set to true, there is a long drawn Kibana issue due to which it does not send auth headers on a random basis. After upgrading to 7.8.1, we were seeing this issue happening more frequently (where as on 7.2.0, it used to work most of the times). Hence we decided to start using the Kibana ROR plugin to avoid the issue.

1 Like
#9

See readonlyrest.yml above. We have prompt_for_basic_auth set to true.

#10

@drdev Kibana is not really working well this time. As per kibana.yml, credentials should be always present in every call to ES. Looks like it’s not a ROR issue.

As @askids says, you could give ROR Free a try.

#11

Any chance Kibana plugin will be upgraded to support Kibana 7.10.1 in the nearest future?

#12

We are currently aiming at 7.9.3 in our beta programme.
After all features are ported, we will make it work in 7.10.x.