Problem with Kibana plugin 0.1.3

nan008 · June 8, 2017, 1:04pm

We installed ELK 5.4.1 and put ROR 1.16.3 and KBN 0.1.3

elastic-a (Master) ROR installed
elastic-b (Data node) no plugins

NO X-PACK installed anywhere

When we are running only ROR on two ES nodes everything connects fine

The problem starts when we install KBN plugin

Kibana starts normally and I can reach the login screen but then loging in with kibana user there is no reaction in ES log on any node, the form is just cleared of the username and password.

elasticsearch.yml


 - name: "Kibana Server"
      type: allow
      auth_key: kibana:kibana

kibana.yml


elasticsearch.username: "kibana"
elasticsearch.password: "kibana"

then I try with one of the user/groups user

elasticsearch.yml

    - username: anna
      auth_key: anna:anna123
      groups: ["Dev"]

elasticsearch log

[2017-06-08T12:28:32,848][INFO ][o.e.p.r.a.ACL            ]  no block has matched, forbidding by default: { ID:705656851-1176131058, TYP:NodesInfoRequest, USR:anna(?), BRS:false, ACT:cluster:monitor/nodes/info, OA:kibana, IDX:<N/A>, MET:GET, PTH:/_nodes/_local, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,content-length,Host, HIS:[Accept requests for elastic-head->[hosts->false]], [Accept all requests for indices->[hosts->false]], [Logstash->[hosts->false]], [Dev->[auth_key->false]], [Kibana Server->[auth_key->false]] }
[2017-06-08T12:28:32,848][INFO ][o.e.p.r.e.IndexLevelActionFilter] [elastic-a] forbidden request: { ID:705656851-1176131058, TYP:NodesInfoRequest, USR:anna(?), BRS:false, ACT:cluster:monitor/nodes/info, OA:kibana, IDX:<N/A>, MET:GET, PTH:/_nodes/_local, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,content-length,Host, HIS:[Accept requests for elastic-head->[hosts->false]], [Accept all requests for indices->[hosts->false]], [Logstash->[hosts->false]], [Dev->[auth_key->false]], [Kibana Server->[auth_key->false]] } Reason: null (null)

Kibana Log

{"type":"response","@timestamp":"2017-06-08T12:52:17Z","tags":[],"pid":11217,"method":"post","statusCode":200,"req":{"url":"/login","method":"post","headers":{"host":"kibana.s.com:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:53.0) Gecko/20100101 Firefox/53.0","accept":"application/json, text/javascript, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","content-type":"application/x-www-form-urlencoded; charset=UTF-8","kbn-version":"5.4.1","kbn-xsrf":"5.4.1","x-requested-with":"XMLHttpRequest","referer":"http://kibana.s.com:5601/login?nextUrl=%2Fapp%2Fkibana","content-length":"30","connection":"keep-alive","proxy-authorization":"Basic dXNlci11dWlkLTdhYTRmNjViZjEyMTk1MWJkYTQ0ZWU3MmE1YWM0NDJlOjM4YTUwZDYzMzQ5Yg=="},"remoteAddress":"182.9.176.184","userAgent":"182.9.176.184","referer":"http://kibana.s.com:5601/login?nextUrl=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":9,"contentLength":9},"message":"POST /login 200 9ms - 9.0B"}

ES then hangs and I cannot restart, stop or start it, I need to uninstall ROR and KBN plugins, stop Kibana, kill -9 ES and delete the indices to be able to start back ES.

I tested it on the ELK 5.4.0 with ROR 1.15.1 and KBN 0.1.0 and it was working fine,

Please advise urgently

sscarduzio · June 8, 2017, 1:31pm

move the auth_key to the users section. When you use the “groups” rule in a block, do not include authentication rules in the same block, move them to the "users: " section.

nan008 · June 8, 2017, 1:44pm

I have that for the groups/users in elasticsearch.yml, where should I use the auth_key?

readonlyrest:
    enable: true

    response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin

    prompt_for_basic_auth: false

  access_control_rules:
  - name: "Dev"
      type: allow
      kibana_access: ro
      kibana_hide_apps: ["timelion","readonlyrest_kbn", "kibana:management"]
      groups: ["Dev"]
      indices: [".kibana", "default", ".kibana-devnull","anna01*"]

    users:
    - username: dan
      auth_key: dan:dan123
      groups: ["Dev"]
    - username: anna
      auth_key: anna:anna123
      groups: ["Dev"]

sscarduzio · June 8, 2017, 1:46pm

Let me try this configuration on my machine.

sscarduzio · June 8, 2017, 1:50pm

You are missing the kibana daemon rule block!

nan008 · June 8, 2017, 2:09pm

Nope I have it, sorry my mistake - I should give you the whole ROR from yml file


readonlyrest:
    enable: true

    response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin

    prompt_for_basic_auth: false

    access_control_rules:
    - name: "Accept all requests for indices"
      type: allow
      hosts: ["xx.xx.xxx.x", "xx.xxx.xx.xx"]

    - name: "Kibana Server"
      type: allow
      auth_key: kibana:kibana
      #verbosity: error

    - name: "Logstash"
      type: allow
      hosts: ["xx.xxx.xx.xx", "xx.x.xxx.xx"]

    - name: "Dev"
      type: allow
      kibana_access: ro
      kibana_hide_apps: ["timelion","readonlyrest_kbn", "kibana:management"]
      groups: ["Dev"]
      indices: [".kibana", "default", ".kibana-devnull","anna01*"]

    users:
    - username: dan
      auth_key: dan:dan123
      groups: ["Dev"]
    - username: anna
      auth_key: anna:anna123
      groups: ["Dev"]

sscarduzio · June 8, 2017, 3:10pm

@nan008 You are right I found a bug. It’s a regression that happened in the last update of ROR for ES when I introduced asynchronous evaluation for groups.

A new ES build is on the way!

sscarduzio · June 8, 2017, 3:16pm

Here is a new build Download (UNIVERSAL) - ReadonlyREST

nan008 · June 8, 2017, 4:07pm

Still not working - now when I try to login with kibana:kibana it is giving me the blank green screen

ROR 1.16.4
KBN 0.1.3

sscarduzio · June 8, 2017, 4:16pm

Is this thing triggered by the ROR Kibana 0.1.3 ? does it happen with 0.1.2? I can’t reproduce this.
Have a look at the logs, any errors?

sscarduzio · June 8, 2017, 4:29pm

OK GOT IT I reproduced it, I have somethinig to work on. Keep you posted.

sscarduzio · June 9, 2017, 7:21am

Published 0.1.4. It was a problem with handling null base paths, so the login was fine, just tried to redirect you to null after checking the credentials.

I will look into automating this kind of testing once and for all.

nan008 · June 9, 2017, 10:24am

IT IS WORKING NOW!!!

sscarduzio · June 9, 2017, 11:10am

Lovely to know! I also noticed today is your first day in the ROR PRO membership! Congrats!

nan008 · June 16, 2017, 11:59am

This problem come back with our PROD env and new testing env. THIS IS URGENT

ROR 1.16.4
KBN 0.1.5 and KBN 0.1.4
ES 5.4.1 (one master node [T1], one coordinating node [T2], kibana on T2, ROR on T1)
No x-pack

I installed brand new testing env and Kibana is back to the letting all in behavior. No reaction from elasticsearch log when I login to the kibana, with made up user

elasticsearch.yml


readonlyrest:
    enable: true

    response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin

    prompt_for_basic_auth: false

    access_control_rules:
    - name: "Accept all requests for indices"
      type: allow
      hosts: ["elastic-t1", "elastic-t2"]

    - name: "Kibana Server"
      type: allow
      auth_key: kibana:kibana

    - name: "Dev"
      type: allow
      kibana_access: ro
      kibana_hide_apps: ["timelion","readonlyrest_kbn","monitoring", "kibana:management"]
      groups: ["Dev"]
      indices: [".kibana", ".kibana-devnull"]

    users:
    - username: dan
      auth_key: dan:dan123
      groups: ["Dev"]
    - username: anna
      auth_key: anna:anna123
      groups: ["Dev"]

kibana.yml

server.host: "xx.xxx.xx.x"
server.name: "Kibana"
elasticsearch.url: "http://xx.xxx.xx.x:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
elasticsearch.ssl.verificationMode: none
logging.dest: /var/log/kibana/kibana.log

kibana.log

{"type":"log","@timestamp":"2017-06-16T11:52:59Z","tags":["status","plugin:kibana@5.4.1","info"],"pid":13406,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","plugin:elasticsearch@5.4.1","info"],"pid":13406,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","plugin:readonlyrest_kbn@0.1.4","info"],"pid":13406,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","plugin:console@5.4.1","info"],"pid":13406,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","plugin:elasticsearch@5.4.1","info"],"pid":13406,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","plugin:metrics@5.4.1","info"],"pid":13406,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","plugin:timelion@5.4.1","info"],"pid":13406,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["listening","info"],"pid":13406,"message":"Server running at http://10.0.201.238:5601"}
{"type":"log","@timestamp":"2017-06-16T11:53:00Z","tags":["status","ui settings","info"],"pid":13406,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"response","@timestamp":"2017-06-16T11:53:22Z","tags":[],"pid":13406,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"host":"54.171.107.168:5601","upgrade-insecure-requests":"1","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5","accept-language":"en-us","accept-encoding":"gzip, deflate","connection":"keep-alive"},"remoteAddress":"185.7.185.182","userAgent":"185.7.185.182"},"res":{"statusCode":302,"responseTime":15,"contentLength":9},"message":"GET /app/kibana 302 15ms - 9.0B"}
{"type":"response","@timestamp":"2017-06-16T11:53:22Z","tags":[],"pid":13406,"method":"get","statusCode":200,"req":{"url":"/login?nextUrl=%2Fapp%2Fkibana","method":"get","headers":{"host":"54.171.107.168:5601","upgrade-insecure-requests":"1","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5","accept-language":"en-us","accept-encoding":"gzip, deflate","connection":"keep-alive"},"remoteAddress":"185.7.185.182","userAgent":"185.7.185.182"},"res":{"statusCode":200,"responseTime":11,"contentLength":9},"message":"GET /login?nextUrl=%2Fapp%2Fkibana 200 11ms - 9.0B"}
{"type":"response","@timestamp":"2017-06-16T11:53:22Z","tags":[],"pid":13406,"method":"get","statusCode":304,"req":{"url":"/plugins/timelion/icon.svg","method":"get","headers":{"host":"54.171.107.168:5601","connection":"keep-alive","if-none-match":"\"cb793d5314d680b7d5ce130f0393a70b51989541-gzip\"","accept":"image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5","if-modified-since":"Mon, 29 May 2017 16:20:01 GMT","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5","accept-language":"en-us","referer":"http://54.171.107.168:5601/app/kibana","accept-encoding":"gzip, deflate"},"remoteAddress":"185.7.185.182","userAgent":"185.7.185.182","referer":"http://54.171.107.168:5601/app/kibana"},"res":{"statusCode":304,"responseTime":10,"contentLength":9},"message":"GET /plugins/timelion/icon.svg 304 10ms - 9.0B"}
{"type":"response","@timestamp":"2017-06-16T11:53:22Z","tags":[],"pid":13406,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/wrench.svg","method":"get","headers":{"host":"54.171.107.168:5601","connection":"keep-alive","if-none-match":"\"088a9a98c99e406dca2354af14f688ad84826b97-gzip\"","accept":"image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5","if-modified-since":"Mon, 29 May 2017 16:20:01 GMT","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5","accept-language":"en-us","referer":"http://54.171.107.168:5601/app/kibana","accept-encoding":"gzip, deflate"},"remoteAddress":"185.7.185.182","userAgent":"185.7.185.182","referer":"http://54.171.107.168:5601/app/kibana"},"res":{"statusCode":304,"responseTime":11,"contentLength":9},"message":"GET /plugins/kibana/assets/wrench.svg 304 11ms - 9.0B"}
{"type":"response","@timestamp":"2017-06-16T11:53:22Z","tags":[],"pid":13406,"method":"get","statusCode":304,"req":{"url":"/bundles/0cebf3d61338c454670b1c5bdf5d6d8d.svg","method":"get","headers":{"host":"54.171.107.168:5601","connection":"keep-alive","if-none-match":"\"d52234e52fd4e96d20f52f4c03c0cedb8ab5fe17-gzip\"","accept":"image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5","if-modified-since":"Fri, 16 Jun 2017 11:13:31 GMT","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/603.2.5 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.5","accept-language":"en-us","referer":"http://54.171.107.168:5601/app/kibana","accept-encoding":"gzip, deflate"},"remoteAddress":"185.7.185.182","userAgent":"185.7.185.182","referer":"http://54.171.107.168:5601/app/kibana"},"res":{"statusCode":304,"responseTime":12,"contentLength":9},"message":"GET /bundles/0cebf3d61338c454670b1c5bdf5d6d8d.svg 304 12ms - 9.0B"}

sscarduzio · June 16, 2017, 8:14pm

OK trying to reproduce it now.

sscarduzio · June 16, 2017, 8:23pm

OK I think I understood the issue, you have installed ROR on a node (T1) and not in the other (T2). And you’re pointing Kibana to the node without ROR (T2).

Also, please make sure none of these IP is where Kibana lives, otherwise it will always allow anything, as browser requests are all proxied by the Kibana server.

The symptom of “kibana can log-in even with invented credentials” has to ring these two bells:

Is my ACL too permissive? ( → look at the ES logs)
Is ROR even installed or activated in the node listed in elasticsearch.url: xyz? in kibana.yml?

nan008 · June 19, 2017, 10:19am

Ok, now I understand the problem. We connected ok to ROR but it was wide open - did not take into account that Kibana/Elastic on the same machine would be an issue, that is the config advice on ES website to locate Kibana on coordinating node (same server)

I solve the issue by giving Kibana external ES IP to connect to and it is working as designed now.

So now we have
T1 (master node) ROR on
T2 (coordinating node) Kibana on with KBN plugin.

kibana.yml has external IP of T1 as url and now the list is working with restrictions and elasticsearch.log is updated with failed login attempts.

it is going to Prod today