RoR Pro Load Balancer cookie issue

Users Feedback
1

Hi,

Setup:
ES6.1.1
ES RoR: 1.16.16
Kibana RoR: 1.16.20

2 Kibana nodes behind a load balancer and apache reverse proxy
LB:443 > Kibana nodes: 8002 (SSL handling and reverse proxy) > Localhost kibana:5601
And kibana then goes to LB:9200 > ES nodes:9200(the 2 Kibana nodes + 7 seperate data nodes)

Even with setting:
readonlyrest_kbn.cookiePass: “jx3JuDpxT5l0DewmY73iS8F2tWl7eXXX”
In kibana.yml I get white screens in Kibana and redirected back to login page.

If I disable one Kibana node it works (both 01 and 02 work) but combined they don’t work.
If I bypass apache (so LB directly to Kibana:5601) it still doesn’t work.

In the kibana logs I see messages as:

{“type”:“log”,“@timestamp”:“2018-06-15T13:30:21Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:20042,“message”:“try extract credentials from JSON”}

I am having difficulty understandig why the system is behaving like this.

From logs I have:

{"type":"log","@timestamp":"2018-06-15T13:30:17Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"Received Kibana Ops event data"}

{"type":"log","@timestamp":"2018-06-15T13:30:17Z","tags":["plugin","debug"],"pid":20042,"message":"Checking Elasticsearch version"}
{"type":"response","@timestamp":"2018-06-15T13:30:18Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/bundles/kibana.style.css?v=16350","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":52,"contentLength":9},"message":"GET /bundles/kibana.style.css?v=16350 200 52ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:18Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/ui/favicons/favicon-16x16.png","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"GET /ui/favicons/favicon-16x16.png 200 8ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:18Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/ui/favicons/favicon.ico","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /ui/favicons/favicon.ico 200 6ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:18Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/bundles/commons.bundle.js?v=16350","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"*/*","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":403,"contentLength":9},"message":"GET /bundles/commons.bundle.js?v=16350 200 403ms - 9.0B"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["plugin","debug"],"pid":20042,"message":"Checking Elasticsearch version"}
{"type":"response","@timestamp":"2018-06-15T13:30:18Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/bundles/kibana.bundle.js?v=16350","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"*/*","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":1336,"contentLength":9},"message":"GET /bundles/kibana.bundle.js?v=16350 200 1336ms - 9.0B"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["license","debug","xpack"],"pid":20042,"message":"Calling Elasticsearch _xpack API"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["license","debug","xpack"],"pid":20042,"message":"Calling Elasticsearch _xpack API"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"Fetching data from kibana collector"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"Fetching data from kibana_stats collector"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"Fetching data from kibana_settings collector"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"Fetching data from reporting_stats collector"}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"not sending [kibana_settings] monitoring document because [undefined] is null or invalid."}
{"type":"log","@timestamp":"2018-06-15T13:30:20Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":20042,"message":"Uploading bulk Kibana monitoring payload"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":302,"req":{"url":"/api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_5_0","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","accept":"application/json, text/javascript, */*; q=0.01","x-requested-with":"XMLHttpRequest","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":302,"responseTime":11,"contentLength":9},"message":"GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_5_0 302 11ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":302,"req":{"url":"/api/saved_objects/?type=index-pattern&per_page=10000","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","accept":"application/json, text/plain, */*","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":302,"responseTime":4,"contentLength":9},"message":"GET /api/saved_objects/?type=index-pattern&per_page=10000 302 4ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/plugins/kibana/assets/discover.svg","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":7,"contentLength":9},"message":"GET /plugins/kibana/assets/discover.svg 200 7ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/plugins/timelion/icon.svg","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /plugins/timelion/icon.svg 200 6ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/plugins/readonlyrest_kbn/rorSVG.svg","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /plugins/readonlyrest_kbn/rorSVG.svg 200 5ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/plugins/kibana/assets/wrench.svg","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /plugins/kibana/assets/wrench.svg 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/bundles/ae11252ad19209059498cac1cd1addd7.svg","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/bundles/commons.style.css?v=16350","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/bundles/commons.style.css?v=16350"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"GET /bundles/ae11252ad19209059498cac1cd1addd7.svg 200 3ms - 9.0B"}
{"type":"log","@timestamp":"2018-06-15T13:30:21Z","tags":["debug","readonlyrest_kbn"],"pid":20042,"message":"try extract credentials from JSON"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","accept":"application/json, text/javascript, */*; q=0.01","x-requested-with":"XMLHttpRequest","kbn-version":"6.1.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /login 200 5ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/ui/favicons/favicon-32x32.png","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"GET /ui/favicons/favicon-32x32.png 200 8ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-15T13:30:21Z","tags":[],"pid":20042,"method":"get","statusCode":200,"req":{"url":"/ui/favicons/favicon-32x32.png","method":"get","headers":{"host":"kibanasmc.domain.name(anonmized):443","connection":"keep-alive","pragma":"no-cache","cache-control":"no-cache","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,de;q=0.8,nl;q=0.7"},"remoteAddress":"10.60.71.62","userAgent":"10.60.71.62","referer":"http://kibanasmc.domain.name(anonmized):443/app/kibana"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"GET /ui/favicons/favicon-32x32.png 200 3ms - 9.0B"}
{"type":"log","@timestamp":"2018-06-15T13:30:21Z","tags":["debug","readonlyrest_kbn"],"pid":20042,"message":"try extract credentials from JSON"}

Any clue where I need to look for this?

2

Hi @ronald.vanboven,
It’s a known bug that cookiePass does not work at the moment. Please enable sticky sessions in your load balancer as a workaround for now.

3

Clear, I thought I was going crazy.
Been playing with the reverseproxy configs etc etc for the last few hours, thinking I missed something.

Good to hear it is a known bug, thanks for quick response!
Maybe good to make small remark in the kibana manual that there is a known bug for this?

1 Like
4

@sscarduzio not related, but just noticed it. Do you recommend using different versions or suggest users to use same version for both plugins?

5

@askids very well noted, it’s important that the ROR versions are aligned between the Elasticsearch and Kibana plugins!!

@ronald.vanboven please keep an eye on this!

6

I noticed it as well while typing the ticket.
I fixed it this morning, now the versions are matching.

1 Like
7

@sscarduzio is this cookie issue fixed ?

8

I had a look, but could not fix it easily. Need to get out of the door SAML and fix JWT protocol ES256 before.