[ISSUE] Unable to access monitoring in Kibana after setting up basic authentication


(Askids) #4

@sscarduzio Below is my ROR entry

- name: Kibana
  type: allow
  auth_key: kibana:kibana

And below is my Kibana.yml config. I have tried this in both Win 2008 R2 and 2012. I am able to get to discovery, dev tools, but not the monitoring tab.

server.host: "machine1.domain1.com"
server.name: "My Elasticsearch Cluster"
server.port: 10201
kibana.index: ".kibana"
elasticsearch.url: "https://esmachine2.domain1.com:10200"
elasticsearch.preserveHost: true
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
server.ssl.enabled: true
server.ssl.certificate: D:/Apps/Program Files/kibana-5.5.1-windows-x86/config/elasticappclient.crt
server.ssl.key: D:/Apps/Program Files/kibana-5.5.1-windows-x86/config/elasticappclient.key
elasticsearch.ssl.certificate: D:/Apps/Program Files/kibana-5.5.1-windows-x86/config/elasticappclient.crt
elasticsearch.ssl.key: D:/Apps/Program Files/kibana-5.5.1-windows-x86/config/elasticappclient.key
elasticsearch.ssl.certificateAuthorities: [ "D:/Apps/Program Files/kibana-5.5.1-windows-x86/config/elasticappclient.pem" ]
elasticsearch.ssl.verificationMode: full
elasticsearch.requestHeadersWhitelist: ["x-proxy-user", "authorization", "x-forwarded-for", "x-forwarded-server"]
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: false
xpack.watcher.enabled: false
xpack.ml.enabled: false
logging.dest: D:/Logs/Kibana/kibana.log 

I get the below access denied error in the monitoring page.

You are not authorized to access Monitoring. To use Monitoring, you need the privileges granted by both the `kibana_user` and `monitoring_user` roles.
If you are attempting to access a dedicated monitoring cluster, this might be because you are logged in as a user that is not configured on the monitoring cluster.

(Askids) #5

@sscarduzio I installed the Pro version on Kibana. I am able to go the ROR tab and see the proper ROR config information. However, when I go to the monitoring tab, I get redirected to the fatal error. Clearing the session is not making any difference.

Looks like something went wrong. Refreshing may do the trick.

Monitoring: Error 400 Bad Request: undefined

Version: 5.5.1
Build: 15405

Can you please help on this?


(Simone Scarduzio) #6

what does the ES log say?


(Askids) #7

I have kept the verbosity to error. Looks like there is no auth header passed on. I had logged in using the ROR pro login page using the admin login.

[2017-10-31T02:51:57,357][INFO ][t.b.r.a.ACL ] e[35mFORBIDDEN by default req={ ID:1779793052--310430366#594, TYP:SearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:indices:data/read/search, OA:171.128.210.228, IDX:.monitoring-es-2-*,.monitoring-es-6-*, MET:POST, PTH:/.monitoring-es-2-*%2C.monitoring-es-6-*/_search?filter_path=hits.hits._source.cluster_uuid%2Chits.hits._source.cluster_name%2Chits.hits._source.version%2Chits.hits._source.license%2Chits.hits._source.cluster_stats%2Chits.hits._source.cluster_state, CNT:<OMITTED, LENGTH=299>, HDR:Connection,Content-Length,content-type,Host, HIS:[::ADMIN::->[auth_key->false]], [::KIBANA-SRV::->[auth_key->false]], [Read only access to Dev Team members->[ldap_authentication->false]], [FinderService->[auth_key_sha256->false]], [LDAP via elevated admin group->[ldap_authentication->false]], [::RO::->[auth_key->false]] } e[0m

Below are my ROR entries and I was using the admin login.

- name: "::KIBANA-SRV::"
  type: allow
  auth_key: kibana:kibana      
  #auth_key_sha256: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  verbosity: error

- name: "::ADMIN::"
  auth_key: admin:dev
  # KIBANA ADMIN ACCESS NEEDED TO EDIT SECURITY SETTINGS IN ROR KIBANA APP!
  kibana_access: admin
  verbosity: error

(Askids) #8

Also, saw the following error in Kibana log file.

got an error [403] Forbidden { Error: Insufficient user permissions for monitoring data
    at handleError (D:/Apps/Program Files/kibana-5.5.1-windows-x86/plugins/x-pack/plugins/monitoring/server/lib/handle_error.js:22:17)
    at D:/Apps/Program Files/kibana-5.5.1-windows-x86/plugins/x-pack/plugins/monitoring/server/routes/api/v1/cluster/clusters.js:34:22
    at throw (native)
    at step (D:\Apps\Program Files\kibana-5.5.1-windows-x86\plugins\x-pack\plugins\monitoring\server\routes\api\v1\cluster\clusters.js:20:191)
    at D:\Apps\Program Files\kibana-5.5.1-windows-x86\plugins\x-pack\plugins\monitoring\server\routes\api\v1\cluster\clusters.js:20:402
  data: null,
  isBoom: true,
  isServer: false,
  output: 
   { statusCode: 403,
     payload: 
      { statusCode: 403,
        error: 'Forbidden',
        message: 'Insufficient user permissions for monitoring data' },
     headers: { 'kbn-name': 'kibana', 'kbn-version': '5.5.1' } },
  reformat: [Function] }

Apache httpd: no basic auth header with POST
(Askids) #9

EDIT:

ReadonlyREST plugins for Kibana (Enterprise and PRO) are finally able compensate the Kibana bug and X-Pack Monitoring works normally out of the box. No need for hacks!


:warning:️WARNING
THE SOLUTION BELOW IS A HACK AND MAY MESS UP YOUR ACL, PLEASE MIND THE ORDER OF THE ACL BLOCKS AND THINK ABOUT ALL THE RAMIFICATIONS


Finally I had a break through but via work around rule.

Based on the kibana error log, I checked their clusters.js code. This is internally calling verify_monitoring_auth.js to get the authorization info. The way they have coded it is either there is xpack security or nothing. Because of this, when we disable xpack security, they assume that there is no security. I am not sure how it works on other tabs in kibana. But atleast that is what I saw in the verify_monitoring_auth.js file.

When such a request hits ROR from monitoring page with no basic auth info, ROR will definitely reject that request. So as a work around, i put in a IP based rule and have given readonly access to different indexes used in monitoring. I might have to further refine this rule to restrict it to bare minimum requirement for monitoring. But this is what I have as of now and seems to be working fine.

Here the IP belongs to the machine running Kibana.

- name: "::Monitoring::"
  type: allow
  actions: ["cluster:monitor/*","indices:admin/types/exists","indices:data/read/*"]
  indices: [".kibana*", ".kibana-devnull", "readonlyrest-*", ".monitoring*"]
  hosts: ["123.123.123.123"]      
  verbosity: error

I got this idea based on a similar suggestion that was given in below thread.

Thanks!


Kibana Monitoring is not working
(Simone Scarduzio) #10

Good job @askids!

Yes that what I would have done too.
However, why so many indices in your workaround block? I would have thought one would just need to allow those actions in “monitoring*”.

Another thing that leaves me baffled about XPack is how they don’t send any credentials to ES for that call: there’s no Authorization header, or any other cookie/credentials related headers:


(Askids) #11

Like I had mentioned in my earlier post, I am yet to refine it. Looks like “.monitoring*” alone should be sufficient. I will try it out tomorrow.

However, I was also looking at some of the entries getting created on the readonlyrest* indexes as I had the audit collector option as true. Looks like some of the ROR audit logging calls are also coming in without the basic auth header from Kibana and getting blocked by ROR :smiley: . I will need to figure out in what cases are we getting this kind of entry. So I might need to end add one more separate rule for the the Kibana IP to allow data write operations on the readonlyrest* indices.

i will post my final ruleset in couple of days once I have done some more testing.


(Simone Scarduzio) #12

Watch out from allowing the kibana server via IP, as all calls will arrive from there. Don’t forget Kibana server operates as a proxy between the browser and ES.

If something is getting blocked we need to fix it. Share some logs when you have time.


(Askids) #13

Still testing this. But we ran into one more issue when running a multi-node cluster behind a load balancer. So list of IPs was not useful. So I had to remove the hosts rule and add the x_forwarded_for rule to allow monitoring to work properly. Will post some updates over the weekend on the other observation.


(Askids) #14

We are still testing it. The initial assessment about ROR blocking ROR was totally off. Looks like occasionally, we are getting write request on the .kibana index from the Kibana server and coming in without any basic auth header as before.

“action”: “indices:data/write/index”,
“path”: “/.kibana/index-pattern/readonlyrest*”,
“indices”: [".kibana"],

The other type of error that we see coming in is also from Kibana server IP for below action.
“action”: “indices:admin/mappings/fields/get”
“path”: “/.kibana/_mapping/*/field/_source”
“indices”: [".kibana"],

Not sure, if we should grant this write permission as we have currently just given readonly access in the rule. Any idea on what information would be coming in as part of this request?

Also, as future change, can you please consider providing this as logging option so that we can capture the incoming request as well for indexing, when such requests are forbidden. This will be useful during troubleshooting in non-prod lanes. But for production servers, this can be disabled by default.

Thanks!


(Simone Scarduzio) #15

Are we receiving no credentials in ES for write requests to .kibana index patterns? Is this happening with ROR PRO/Enterprise installed?

Forbidden requests should be logged in ROR for ES already, even with verbosity: error. What other log are you suggesting? PRO TIP: have you looked at changing Kibana’s verbosity? It would trace the incoming HTTP requests too.


(Askids) #16

We already have ROR verbosity set to error. I actually wanted to know what actually comes in the indexing request. Anyway, I will try to look at the Kibana logging. But what if the requests we’re not from Kibana but some unauthorised user application. In such a case, how do we know what was rejected? We would know the index name, path and action from ROR audit log index. So it would be nice to have an option where we can capture what came in the actual request content itself. I did not find them in ROR index. Am I missing anything?

Also since we talking about audit logs, is there any plan to add option to perform scheduled cleanup of ROR audit indices - let’s say after x days/months etc. Kibana and ES does it for monitoring indexes after 7 days. We don’t have to so stingy as it’s security audit logs :grin:, but having it as a user controlled setting could be a good feature to have.


(Simone Scarduzio) #17

This may too easily backfire. Imagine if you logged the body of every Logstash request. Performance will go to crap. One option would be to do so optionally and only for forbidden requests. Would that work?

This is a very nice idea, want to add ASAP.


(Askids) #18

Yes. That would work great. Because we would only be interested in this kind of information, when the request was forbidden. This again, I would suggest as a user controlled setting so that it’s enabled only if people see a need for it.

Thanks for being receptive to suggestions!!


(Simone Scarduzio) #19

NP thanks for coming up with interesting ideas :slight_smile:


#20

Hi Gents!
I just started using Pro version today (it is awesome :slight_smile: but/and also faced with monitoring problems.
I used snippet from docs and log in as kibana server user with full access (as I correctly understand)

- name: "::KIBANA-SRV::"
  auth_key: kibana:kibana

then proceed to monitoring tab and got an error:

Monitoring indices exists:

Nothing interesting in ES logs, but in Kibana have an:

{“type”:“request”,"@timestamp":“2017-11-14T20:17:20Z”,“tags”:[“monitoring-ui”,“error”],“pid”:1,“level”:“error”,“message”:“Authorization Exception”,“error”:{“message”:“Authorization Exception”,“name”:“Error”,“stack”:“Authorization Exception :: {“path”:”/.monitoring-es-2-%2C.monitoring-es-6-/_search",“query”:{“filter_path”:“hits.hits._source.cluster_uuid,hits.hits._source.cluster_name,hits.hits._source.version,hits.hits._source.license,hits.hits._source.cluster_stats,hits.hits._source.cluster_state”},“body”:"{\“size\”:10000,\“query\”:{\“bool\”:{\“filter\”:[{\“bool\”:{\“should\”:[{\“term\”:{\"_type\":\“cluster_stats\”}},{\“term\”:{\“type\”:\“cluster_stats\”}}]}},{\“range\”:{\“timestamp\”:{\“format\”:\“epoch_millis\”,\“gte\”:1510687040611,\“lte\”:1510690640612}}}]}},\“collapse\”:{\“field\”:\“cluster_uuid\”},\“sort\”:{\“timestamp\”:{\“order\”:\“desc\”}}}",“statusCode”:403,“response”:""}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:91:20)\n at IncomingMessage.emit (events.js:185:7)\n at endReadableNT (_stream_readable.js:974:12)\n at _combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickDomainCallback (internal/process/next_tick.js:128:9)"}}
got an error [403] Forbidden { Error: Insufficient user permissions for monitoring data
at handleError (/usr/share/kibana/plugins/x-pack/plugins/monitoring/server/lib/handle_error.js:25:17)
at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/routes/api/v1/cluster/clusters.js:34:22
at throw (native)
at step (/usr/share/kibana/plugins/x-pack/plugins/monitoring/server/routes/api/v1/cluster/clusters.js:20:191)
at /usr/share/kibana/plugins/x-pack/plugins/monitoring/server/routes/api/v1/cluster/clusters.js:20:402
data: null,
isBoom: true,
isServer: false,
output:
{ statusCode: 403,
payload:
{ statusCode: 403,
error: ‘Forbidden’,
message: ‘Insufficient user permissions for monitoring data’ },
headers: { ‘kbn-name’: ‘kibana’, ‘kbn-version’: ‘5.6.3’ } },
reformat: [Function] }
{“type”:“response”,"@timestamp":“2017-11-14T20:17:20Z”,“tags”:[],“pid”:1,“method”:“post”,“statusCode”:400,“req”:{“url”:"/api/monitoring/v1/clusters",“method”:“post”,“headers”:{“host”:“kibana.k8s.com”,“connection”:“close”,“x-real-ip”:“172.27.0.2”,“x-forwarded-for”:“172.27.0.2”,“x-forwarded-host”:“kibana.k8s.com”,“x-forwarded-port”:“80”,“x-forwarded-proto”:“http”,“x-original-uri”:"/api/monitoring/v1/clusters",“x-scheme”:“http”,“content-length”:“81”,“user-agent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:56.0) Gecko/20100101 Firefox/56.0”,“accept”:“application/json, text/plain, /”,“accept-language”:“en-US,en;q=0.5”,“accept-encoding”:“gzip, deflate”,“referer”:“http://kibana.k8s.com/app/monitoring",“content-type”:“application/json;charset=utf-8”,“kbn-version”:“5.6.3”},“remoteAddress”:“100.96.11.9”,“userAgent”:“100.96.11.9”,“referer”:“http://kibana.k8s.com/app/monitoring”},“res”:{“statusCode”:400,“responseTime”:15,“contentLength”:9},“message”:"POST /api/monitoring/v1/clusters 400 15ms - 9.0B”}

Why security goes via x-pack if its have already disabled or what does it mean at all?

ROR version - 1.16.12_es5.6.3
ROR Kibana PRO - 1.16.13-20171113_es5.6.3
ES/Kibana - 5.6.3

Thanks in advance!


(Askids) #21

In most probability, it will be Kibana not sending basic auth header. I would suggest that you set verbosity to error and turn on the audit collector to true. ROR will create its own daily index. That will be easier to query and visualize later on.

In our case, like I have mentioned in above posts, we ended up setting up read-only rule on monitoring index, with IP list and xforwarded rule.

Also check if .Kibana is on your list of allowed auto create index list. You might want to include ReadonlyREST also in that list.


#22

Thanks a lot! Just adopt my settings regarding your posts and looks like it works now.
Many thanks!


(Askids) #23

Glad to help. I already spent several weeks breaking my head for not getting monitoring to work, until we figured out this work around. I don’t want anyone else to be wasting time on this :grin: You wouldn’t believe, if I told you that the Kibana issue has been open in their repo for nearly a year now. Also just be aware that currently ROR daily indexes stays forever, until in some future release auto cleanup gets added as a feature, timelines of it, only Simone or one of the contributors will be able to comment on.

@sscarduzio, given that the Kibana fix is not going to come anytime soon, why not put it under official documentation, under troubleshooting section for setting up Kibana with monitoring.

Next on my list in coming weeks is to setup separate monitoring cluster using xpack forwarders feature. Need to see if that works on basic xpack license and ROR combination. Will post some updates, once that setup is complete or you might hear sooner, if I run into issues.