NGINX (SSO) to Kibana to ElasticSearch?

Daedelus · June 26, 2018, 7:24pm

Hi Simone,
We’ve finished Proof of concept testing this in our local and everything worked perfectly. we deployed these changes to our AWS EC2 Instances (only the Kibana and Elastic). For testing NGINX config, we are trying to talk to the Remote EC2 Kibana instance from our Local NGINX (as opposed to what we tried above successfully - local NGINX to local KIBANA to local Elastic).

What happens is when we try to point local NGINX (ie call http://localhost:5100 NGINX endpoint) to the remote Kibana above, we’re not seeing any hits on ElasticSearch. Instead, we get a popup that states “Logged In, but no identity metadata could be found”. Also, running developer tools, I see this weird error in the network tab of developer console:

INFO: 2018-06-26T19:21:33Z
  Adding connection to http://localhost:5100/elasticsearch                                    vendors.bundle: 165

no username found in identity, logging out {authHeaders: {…}, hiddenApps: Array(0), sid: "a3b90eba-c5ac-4ce0-8a69-20babcf83050"}                                                                                    commons.bundle: 3

NGINX Conf:

http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] “$request” ’
sendfile on;
keepalive_timeout 65;

server {
listen 5100;
server_name localhost;
  location / {
  	rewrite ^/(.*) /$1 break;
  	proxy_ignore_client_abort on;
  	proxy_pass https://myapp-kibana.myhost.com;
  	proxy_set_header X-Real-IP  $remote_addr;
  	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  	proxy_set_header Host $http_host;
  	proxy_set_header x-forwarded-user ADM123;
  	proxy_set_header x-forwarded-group admin;
  }
}
}

sscarduzio · June 26, 2018, 7:57pm

@Daedelus that error happens when Elasticsearch returns OK 200 from the login attempt, but no user identity metadata comes back from ES in the form of HTTP response headers.

So please verify that ES returns x-rr-user header on the login response back to Kibana.

This might happen for the following reasons:

The login attempt (a banal HTTP request from ROR Kibana to ES) gets matched by a very generic block (which does not describe a user) too early in the ACL chain.

Solution for this: look at the logs in ES and look for what block gets matched with this path "/_nodes/_local". This is the login request, and should get matched by some block with auth_key_* or similar, and not by some random block that just allows some IP, or action.

There is a proxy between Kibana and ES and it does not let the response headers flow back to Kibana
You are pointing Kibana to an instance of the ES cluster which hasn’t ROR installed. Or has the wrong ROR settings.

Daedelus · June 26, 2018, 9:05pm

@sscarduzio,
I have eliminated 2 and 3. In case of 1, I tried attempting the following scenario directly into ROR enabled Kibana:

If I enter a RIGHT Password thats configured in ES - I get this successful write in the ES Logs when entering the username/password directly in Kibana

[2018-06-26T20:54:43,018][INFO ][t.b.r.a.ACL ] ALLOWED by { name: 'Global Write/Admin Access', policy: ALLOW} req={ ID:947447009-1254494189#204772, TYP:NodesInfoRequest, CGR:N/A, USR:elastic, BRS:false, KDX:null, ACT:cluster:monitor/nodes/info, OA:10.203.123.11, DA:10.203.121.146, IDX:<N/A>, MET:GET, PTH:/_nodes/_local, CNT:<N/A>, HDR:{authorization=Basic ZWxhc3RpYzokYXByMSRKUnhMMEhPeiRuZE1IQ2lyYXpEZlRaem5MeS5pY0gx, Authorization=<OMITTED>, content-length=0, Connection=keep-alive, host=myapp-elasticsearch.myhost.com, X-Forwarded-Proto=https, X-Forwarded-For=10.203.123.23, X-Forwarded-Port=443}, HIS:[::KIBANA-SRV 1::->[auth_key->false]], [::RO::->[auth_key->false]], [::RW::->[auth_key->false]], [Global Write/Admin Access->[auth_key->true, actions->true]] }

If I enter a WRONG Password thats NOT configured in ES in the username/password field- I get the exact same behaviour as calling NGINX Local>Same Kibana EC2
There’s no hit in ES at all. I’ve configured all ES ACL rules to be INFO FYI. I get the same error message in the UI as in the last post.

sscarduzio · June 26, 2018, 10:55pm

@Daedelus, how about narrowing the circle a little bit and repeating the experiment using curl directly against ES? Don’t forget to use the -vvv option to show the response headers.

i.e.

curl -vvv -u user:pass -k "https://es-host:9200/_nodes/_local"

Daedelus · June 27, 2018, 1:06pm

this comes back successfully

{“_nodes”:{“total”:1,“successful”:1,“failed”:0},“cluster_name”:“my-cluster”,“nodes”:{“Aw2Wi1v6TGOhITuWp5pg4w”…

sscarduzio · June 27, 2018, 3:41pm

yes but the response headers? Are they present?

And did you observe what ACL block gets matched?

Daedelus · June 27, 2018, 7:25pm

If I invoke the node states through curl - in this case it looked like the global route53 ACL got hit.

ALLOWED by { name: 'Route53 Access', policy: ALLOW} req={ ID:1703063950-989330959#246, TYP:NodesInfoRequest, CGR:N/A, USR:[no basic auth header], BRS:true, KDX:null, ACT:cluster:monitor/nodes/info, OA:10.203.122.223, DA:10.203.121.146, IDX:<N/A>, MET:GET, PTH:/_nodes/_local, CNT:<N/A>, HDR:{Accept=*/*, Connection=keep-alive, content-length=0, host=myapp.myhost.com, User-Agent=curl/7.29.0, X-Forwarded-For=10.203.122.167, X-Forwarded-Port=443, X-Forwarded-Proto=https}, HIS:[::KIBANA-SRV 1::->[auth_key->false]], [::RO::->[auth_key->false]], [::RW::->[auth_key->false]], [Global Write/Admin Access->[auth_key->false]], [Route53 Access->[actions->true, x_forwarded_for->true]] }

I then kept the Route53 Logging Enabled but moved the NGINX ACLs to the top and got these.

[2018-06-27T19:56:02,586][INFO ][t.b.r.a.ACL ] ALLOWED by { name: '::NGINX-ADMIN-GROUP::', policy: ALLOW} req={ ID:1541857624-1574435768#199, TYP:GetRequest, CGR:N/A, USR:ADM123, BRS:false, KDX:null, ACT:indices:data/read/get, OA:10.203.122.223, DA:10.203.121.146, IDX:.kibana, MET:GET, PTH:/.kibana/doc/config%3A6.3.0, CNT:<N/A>, HDR:{Connection=keep-alive, content-length=0, host=myapp-elasticsearch.myhost.com, X-Forwarded-For=10.203.122.167, x-forwarded-group=admin, X-Forwarded-Port=443, X-Forwarded-Proto=https, x-forwarded-user=ADM123}, HIS:[::NGINX-RESTRICTED-GROUP::->[kibana_access->true, headers->false, proxy_auth->true]], [::NGINX-READONLY-GROUP::->[headers->false, kibana_access->true, proxy_auth->true]], [::NGINX-ADMIN-GROUP::->[kibana_access->true, headers->true, indices->true, proxy_auth->true, actions->true]] }

[2018-06-27T19:56:03,533][INFO ][t.b.r.a.ACL ] ALLOWED by { name: '::NGINX-ADMIN-GROUP::', policy: ALLOW} req={ ID:1983869349-1324128166#207, TYP:SearchRequest, CGR:N/A, USR:ADM123, BRS:false, KDX:null, ACT:indices:data/read/search, OA:10.203.122.223, DA:10.203.121.146, IDX:.kibana, MET:POST, PTH:/.kibana/_search?size=10000&from=0, CNT:<OMITTED, LENGTH=80>, HDR:{Connection=keep-alive, Content-Length=80, content-type=application/json, host=myapp-elasticsearch.myhost.com, X-Forwarded-For=10.203.122.167, x-forwarded-group=admin, X-Forwarded-Port=443, X-Forwarded-Proto=https, x-forwarded-user=ADM123}, HIS:[::NGINX-RESTRICTED-GROUP::->[kibana_access->true, headers->false, proxy_auth->true]], [::NGINX-READONLY-GROUP::->[headers->false, kibana_access->true, proxy_auth->true]], [::NGINX-ADMIN-GROUP::->[kibana_access->true, headers->true, indices->true, proxy_auth->true, actions->true]] }

I can see the NGINX Rules now pass and I can see the headers - but I STILL get the same “Not Logged In” Metadata Popup Error on Kibana

x-forwarded-user=ADM123
x-forwarded-group=admin

sscarduzio · June 27, 2018, 10:41pm

@Daedelus, yes the key is to move the Kibana access blocks up, and the machines’ related blocks down.
It should work now, are you sure the new settings are getting applied?

Daedelus · June 28, 2018, 1:29pm

@sscarduzio Good point - I did double check this though - both The NGINX and KIBANA blocks are now at the top. If you compare this Config with the response data coming back in the previous post, you can see that the Rule NGINX-ADMIN-GROUP is now passing successfully. These two indices are being hit consequently. But despite these, we get the this error:

ACT:indices:data/read/get
ACT:indices:data/read/search

access_control_rules:

name: “::NGINX-RESTRICTED-GROUP::”
indices: [".kibana", “.kibana_@{user}”, “@{user}_logstash-"]
headers: [“x-forwarded-group:restricted”]
kibana_access: “ro”
kibana_hide_apps: [“readonlyrest_kbn”, “apm”, “timelion”, “kibana:dev_tools”, “kibana:management”]
proxy_auth:
proxy_auth_config: “proxy1”
users: ["”]
verbosity: info
name: “::NGINX-READONLY-GROUP::”
indices: [".kibana", “.kibana_@{user}”, “@{user}_logstash-"]
headers: [“x-forwarded-group:readonly”]
kibana_access: “ro”
proxy_auth:
proxy_auth_config: “proxy1”
users: ["”]
verbosity: info
name: “::NGINX-ADMIN-GROUP::”
indices: [".kibana", “.kibana_@{user}”, “@{user}_logstash-"]
actions: ["indices:data/read/”, “indices:data/write/" ]
headers: [“x-forwarded-group:admin”]
kibana_access: admin
proxy_auth:
proxy_auth_config: “proxy1”
users: ["”]
verbosity: info
name: “::KIBANA-SRV 1::”
kibana_access: admin
auth_key: kibana:kibana
verbosity: error

sscarduzio · June 28, 2018, 3:00pm

Remove this. Why is this here?

Daedelus · June 28, 2018, 3:07pm

It was to give the ADMIN users full read/write capability to do POST requests (say through Kibana Dev). How would this rule change if were to give such admin access to for the x-forwarded-group:admin user group?

Daedelus · June 28, 2018, 3:26pm

I suppose in a backwards way I’m asking - how would we change this specific rule to be equivalent to allow Kibana to login?

sscarduzio · June 28, 2018, 3:26pm

Oh I see what you are trying to do. But the ACL works differently: all the rules in a block are evaluated in logical AND, and all the blocks are in logical OR.

So every single rule you add to a block acts as an additional constraint, not as an extra permission.

What you need to do is to duplicate the block, keep the authentication part, and add the extra actions.

- name: “::NGINX-ADMIN-GROUP::”
  indices: [".kibana", “.kibana_@{user}”, “@{user}_logstash-"]
  actions: ["indices:data/read/”, “indices:data/write/" ]
  headers: [“x-forwarded-group:admin”]
  kibana_access: admin
  proxy_auth:
    proxy_auth_config: “proxy1”
    users: ["”]
 verbosity: info

- name: “::NGINX-ADMIN-GROUP EXTRA CAPABILITIES::”
  indices: [".kibana", “.kibana_@{user}”, “@{user}_logstash-"] # review if to keep this as is?
  actions: ["indices:data/read/”, “indices:data/write/" ] # actions + kibana_access in the same block would NOT work!
  headers: [“x-forwarded-group:admin”]
  proxy_auth:
    proxy_auth_config: “proxy1”
    users: ["”]
 verbosity: info

Daedelus · June 28, 2018, 3:35pm

Ah. Got it. Apologies if this is creating a wonk in your previous post - so if I were to WRITE my requirement as this: ALLOW x-forwarded-gorup:admin ADMIN access to Kibana AND full write access to all indexes - would ACL rules written thus suffice?
PS : I got the end to end working with this combination. Just looking for your affirmration

- name: "::NGINX-ADMIN-GROUP KIBANA ADMIN ACCESS::"
  kibana_access: admin
  headers: ["x-forwarded-group:admin"]
  proxy_auth:
    proxy_auth_config: "proxy1"
    users: ["*"]
  verbosity: info

- name: "::NGINX-ADMIN-GROUP FULL INDEX  ADMIN ACCESS::"
  indices: [".kibana", ".kibana_@{user}", "@{user}_logstash-*", "myappindices-*"]
  actions: ["indices:data/read/*", "indices:data/write/*" ]
  headers: ["x-forwarded-group:admin"]
  proxy_auth:
    proxy_auth_config: "proxy1"
    users: ["*"]
  verbosity: info

sscarduzio · June 29, 2018, 4:03pm

Yes, well done Roger