LDAP Auth failures for Kibana/RoR

Using Kibana/ES 5.6.3 and RoR 1.16.13_es5.6.3, I’ve been running into a problem where auth appears to work, then fails. That is, the first login attempt succeeds, but Kibana shows a 403 in the logs. Then all subsequent attempts fail. This appears to slightly be the double login problem people have mentioned in the past, but I’m not sure…

Successful Login: https://pastebin.com/KAumFg1U
Failure Login: https://pastebin.com/RpaWMt0y

What’s particularly strange to me is the failure run seems to be missing the Basic Auth header (which is explicitly disabled in my config…right?)

Anyone have any ideas?

Since I can only paste 2 links, here’s my RoR config:

 readonlyrest:
   enable: true
   audit_collector: true
   prompt_for_basic_auth: false
   response_if_req_forbidden: Access Forbidden by FreeIPA
   ssl:
     enable: true
     key_alias: "elasticsearch"
     keystore_file: "/usr/share/elasticsearch/config/keystore.jks"
     keystore_pass: "{{ s3.environment[domain].keypass }}"
     key_pass: "{{ s3.environment[domain].keypass }}"
   access_control_rules:
     - name: Access to Ops team indices
       type: allow
       verbosity: error
       kibana_access: ro
       ldap_authentication: "ldap1"
       ldap_authorization:
         name: "ldap1"
         groups: ["es_ops"]
       indices: ["logmash-*-ops-*", ".kibana"]
       kibana_hide_apps: ["readonlyrest_kbn", "kibana:management", "timelion"]
     - name: Access to FrontEnd team indices
       type: allow
       kibana_access: ro
       verbosity: error
       ldap_authentication: "ldap1"
       ldap_authorization:
         name: "ldap1"
         groups: ["es_fe"]
       indices: ["logmash-*-fe-*", ".kibana"]
       kibana_hide_apps: ["readonlyrest_kbn", "kibana:management", "timelion"]
     - name: Kibana Admins
       type: allow
       ldap_authentication: "ldap1"
       kibana_access: admin
       ldap_authorization:
         name: "ldap1"
         groups: ["es_admin"]
       indices: ["*"]
     - name: cli access
       type: allow
       verbosity: error
       ldap_authentication: "ldap1"
       ldap_authorization:
         name: "ldap1"
         groups: ["es_cli"]
     - name: "::LOCALHOST::"
       type: allow
       verbosity: error
       hosts:
         - localhost
         - 127.0.0.1
   ldaps:
     - name: ldap1
       host: "{{ s3.environment.env.KDC.master }}"
       port: 636
       ssl_enabled: true
       ssl_trust_all_certs: true
       bind_dn: "uid={{ s3.application[role].app.bind_user }},cn=users,cn=accounts,{{ s3.environment.env.KDC.dc }}"
       bind_password: "{{ s3.application[role].app.bind_password }}"
       search_user_base_DN: "cn=users,cn=accounts,{{ s3.environment.env.KDC.dc }}"
       user_id_attribute: "uid"
       search_groups_base_DN: "cn=groups,cn=accounts,{{ s3.environment.env.KDC.dc }}"
       unique_member_attribute: "member"
       group_name_attribute: "cn"
       group_search_filter: "(objectClass=posixGroup)(cn=es_*)"
       connection_timeout_in_sec: 10

Kibana also returns this error when this failure occurs:

got an error [403] Forbidden { Authorization Exception :: {"path":"/.kibana/_search","query":{"size":10000,"from":0},"body":"{\"version\":true,\"query\":{\"bool\":{\"must\":[{\"match_all\":{}}],\"filter\":[{\"bool\":{\"should\":[{\"term\":{\"_type\":\"index-pattern\"}},{\"term\":{\"type\":\"index-pattern\"}}]}}]}}}","statusCode":403,"response":"Access Forbidden by FreeIPA"}                                                  
    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)                                                     
    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)                                          
    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)                              
    at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)                         
    at emitNone (events.js:91:20)                                      
    at IncomingMessage.emit (events.js:185:7)                          
    at endReadableNT (_stream_readable.js:974:12)                      
    at _combinedTickCallback (internal/process/next_tick.js:80:11)     
    at process._tickDomainCallback (internal/process/next_tick.js:128:9)                                                                      
  status: 403,                                                         
  displayName: 'AuthorizationException',                               
  message: 'Authorization Exception',                                  
  path: '/.kibana/_search',                                            
  query: { size: 10000, from: 0 },                                     
  body: 'Access Forbidden by FreeIPA',                                 
  statusCode: 403,                                                     
  response: 'Access Forbidden by FreeIPA',                             
  toString: [Function],                                                
  toJSON: [Function],                                                  
  isBoom: true,                                                        
  isServer: false,                                                     
  data: null,                                                          
  output:                                                              
   { statusCode: 403,                                                  
     payload:                                                          
      { statusCode: 403,                                               
        error: 'Forbidden',                                            
        message: 'Authorization Exception' },                          
     headers: { 'kbn-name': 'kibana', 'kbn-version': '5.6.3' } },      
  reformat: [Function] }                                               
{"type":"response","@timestamp":"2017-11-10T21:28:48Z","tags":[],"pid":96,"method":"get","statusCode":400,"req":{"url":"/api/saved_objects/?type=index-pattern&per_page=10000&page=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","kbn-version":"5.6.3","referer":"https://localhost:5601/app/kibana","dnt":"1","connection":"keep-alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://localhost:5601/app/kibana"},"res":{"statusCode":400,"responseTime":17,"contentLength":9},"message":"GET /api/saved_objects/?type=index-pattern&per_page=10000&page=1 400 17ms - 9.0B"}

The Kibana plugin intercepts the requests to ES and translates the encrypted cookie coming from the browser into HTTP Basic Auth (which is the only one understood by the ES back end). So yes, you should have credentials there.

Does the request originate from the browser? Can you see a failed request in Chrome’s network tab? In that case, can you right click and “copy as cURL” and paste it here?

I do get an error in the network tab:

curl ‘https://0.0.0.0:5601/api/saved_objects/?type=index-pattern&per_page=10000’ -H ‘Accept-Encoding: gzip, deflate, br’ -H ‘Accept-Language: en-US,en;q=0.9’ -H ‘kbn-version: 5.6.3’ -H ‘User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36’ -H ‘Accept: application/json, text/plain, /’ -H ‘Referer: https://0.0.0.0:5601/app/kibana’ -H ‘Cookie: rorCookie=Fe26.20e2d895208cf2684913b658503df54f0b36e3f2a3750297273bdea7f0c59df42GI_-mVlN2NW485WRy05INwy6lYQepjR-fU4qPVuxEkGVbHBJIEBHljFNYPJezRza5PsOl8SfieT8U_-q7vqHgHDijQ1PpkOBC8BZvrTqL-XBUAG_9W1d11SeKfvtrvrigqDedvqeKiziuhwg1cKUscFTH_LnaSfmQFL5Wh8axnh5QNGsgGKOqp-TmGvNTi2hDzbY2oDanEgWReyf2aWLvNX7FNrcy_hmU56RniNSb_mwdf962f310f1369057ae4ad75d1e7b3dd18239dbe7613082efcd4752cc8db402e*o1Ukn4v8TaA_fTRiTmxCLtOSApB_I7Z7Na11ih7XQ7M; username=jseekins’ -H ‘Connection: keep-alive’ --compressed --insecure

I am using an SSH tunnel right now and only one host on the far side, so sticky sessions shouldn’t be an issue.

In the Kibana browser logs you have a valid user name in the cookie

In the ES debug logs I cannot see that, but there is a instead.

`cookie=username=;``

Might be something related to the new proxy_auth pass through feature.

Sorry. That was me being paranoid about logs.
The user is actually jseekins both times.

ohhh I see haha damn.

Fun update:

Connecting directly to Elasticsearch with:
curl -k https://jseekins:@:9200/

Results in:

[2017-11-14T19:56:53,925][DEBUG][t.b.r.a.b.r.i.LdapAuthenticationAsyncRule] Attempting Login as: jseekins rc: { ID:598889597-1832738698#1028931, TYP:MainRequest, CGR:N/A, USR:jseekins(?), BRS:true, ACT:cluster:monitor/main, OA:1.1.1.1, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=/, Authorization=Basic anNlZWtpbnM6OTI3QnViYmxlc185Mjc=, content-length=0, Host=:9200, User-Agent=curl/7.47.0}, HIS:[Access to FrontEnd team indices->[ldap_authentication->true, ldap_authorization->false, kibana_access->true, indices->true]], [Access to Ops team indices->[ldap_authorization->false, ldap_authentication->true, kibana_access->true, indices->true]] }
[2017-11-14T19:56:53,925][DEBUG][t.b.r.a.d.l.l.AuthenticationLdapClientLoggingDecorator] Trying to authenticate user [jseekins] with LDAP [ldap1]
[2017-11-14T19:56:53,925][DEBUG][t.b.r.a.d.l.l.AuthenticationLdapClientLoggingDecorator] User [jseekins] authenticated by LDAP [ldap1]
[2017-11-14T19:56:53,925][DEBUG][t.b.r.a.b.r.i.IndicesSyncRule] Stage -1
[2017-11-14T19:56:53,925][DEBUG][t.b.r.a.d.l.l.AuthenticationLdapClientLoggingDecorator] Trying to fetch user with identifier [jseekins] from LDAP [ldap1]
[2017-11-14T19:56:53,925][DEBUG][t.b.r.a.d.l.l.AuthenticationLdapClientLoggingDecorator] User with identifier [jseekins] found [dn = uid=jseekins,cn=users,cn=accounts,dc=dm,dc=lan]
[2017-11-14T19:56:53,926][DEBUG][t.b.r.a.d.l.l.GroupsProviderLdapClientLoggingDecorator] Trying to fetch user [id=jseekins, dnuid=jseekins,cn=users,cn=accounts,dc=dm,dc=lan] groups from LDAP [ldap1]
[2017-11-14T19:56:53,926][DEBUG][t.b.r.a.d.l.l.GroupsProviderLdapClientLoggingDecorator] LDAP [ldap1] returned for user [jseekins] following groups: [es_admin]
[2017-11-14T19:56:53,926][DEBUG][t.b.r.a.b.Block ] matched { name: ‘Kibana Admins’, policy: ALLOW}
[2017-11-14T19:56:53,926][INFO ][t.b.r.a.ACL ] ALLOWED by { name: ‘Kibana Admins’, policy: ALLOW} req={ ID:598889597-1832738698#1028931, TYP:MainRequest, CGR:N/A, USR:jseekins, BRS:true, ACT:cluster:monitor/main, OA:1.1.1.1, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=/, Authorization=Basic anNlZWtpbnM6OTI3QnViYmxlc185Mjc=, content-length=0, Host=:9200, User-Agent=curl/7.47.0}, HIS:[Access to FrontEnd team indices->[ldap_authentication->true, ldap_authorization->false, kibana_access->true, indices->true]], [Access to Ops team indices->[ldap_authorization->false, ldap_authentication->true, kibana_access->true, indices->true]], [Kibana Admins->[ldap_authorization->true, ldap_authentication->true, kibana_access->true, indices->true]] }

Which to me kinda indicates that the problem lies in the Kibana plugin.

Wait, so your plan was to enable passwordless login?

No. Not at all. If I use the same credentials to hit the ES REST API directly, I get a successful login…
Oh…the forum’s formatting deleted my masked password because I used < and >.

The curl command I used was:
curl -k https://jseekins:password@eshost:9200/

OK so if the problem is in the kibana plugin, we should be able to reproduce this without any LDAP.
Just have a plain readonlyrest.yaml with one user and try to use the Kibana plugin. Right?

Do you have some Kibana usage steps to reproduce the bug?

I guess…Is the referenced config above not enough for steps? What would you like to see?

I only want to point at the Kibana plugin because the same user can connect directly to ES but gets refused when their auth goes through Kibana. And I’m certainly not sure that’s the problem.

Yes and this is a great step forward, the issue is Kibana not forwarding the authentication header under certain circumstances. So I should just need to install your version of ES stack + plugins, mock around with Kibana and wait for a 403. Right?

That’s what I saw, so…yes?

Sorry I can’t give you anything beyond that. This happened immediately after upgrading to the newest Kibana plugin.

Yeah thanks John, will test this and report back.

Cannot reproduce with ROR Enterprise on Kibana 5.6.3.

  1. Do you have xpack enabled in Kibana and/or in ES? If yes set it off.
  2. Do you have readonlyrest_kbn.proxy_auth_passthrough: true? (kibana.yml) It should be off.
  3. Is this something you can reproduce with the following settings?

readonlyrest:
    prompt_for_basic_auth: false
    audit_collector: true
    ssl:
      enable: true
      keystore_file: "/me/elasticsearch-readonlyrest-plugin-5.x/integration-tests/src/test/eshome/plugins/readonlyrest/keystore.jks"
      keystore_pass: readonlyrest
      key_pass: readonlyrest

    access_control_rules:

    - name: "::KIBANA-SRV::"
      auth_key: kibana:kibana
      verbosity: error

    - name: "::ADMIN_GRP::"
      auth_key: simone:dev
      kibana_access: admin


  1. I uninstall x-pack to make sure I don’t enable it.
  2. I set readonlyrest_kbn.proxy_auth_passthrough: false in kibana.yml
  3. Yep. Replacing the auth section with those changes means I can log in okay.

Ugh. I hope this isn’t something silly like a bad config. I’m going to carefully re-create my config and keep messing with this.

I think that missing kibana setting was my downfall. Not sure why I missed it before, but everything is working correctly now.

Thanks for all the debug help, Simone!

1 Like

hey John - for my curiosity - what kibana option did you miss exactly?

At this point, that’s the only change, and things seem to be working again…So I guess that was the problem? Although it doesn’t really make sense…

Maybe we’ll never know!