Need to connect elasticsearch with Active Directory for authentication - Need help

@sscarduzio I even validated it with a YAML Validator

readonlyrest:
    ssl:
      enable: true
      keystore_file: "keystore.jks"
      keystore_pass: readonlyrest
      key_pass: readonlyrest

      audit_collector: true
      access_control_rules:

      - name: Accept requests from users in group team
        type: allow
        ldap_auth:
          name: "name"
          groups: "admins"
        indices: ["*"]

      - name: "::LOGSTASH::"
        auth_key: user:password
        actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
        indices: ["*"]

      ldaps:
      - name: name
        host: "172.**********"
        port: 389
        ssl_enabled: true
        ssl_trust_all_certs: true
        bind_dn: "cn=elkadmin,ou=elk,dc=domain,dc=local"
        bind_password: "********"
        search_user_base_DN: "ou=elk,dc=domain,dc=local"
        search_groups_base_DN: "ou=elk,dc=domain,dc=local"

Your yaml is valid, but wrong. The “ssl”, “audit_collector”, “access_control_rules” entries should be siblings.
In your file they are all child nodes to “ssl”.

Mystery solved I think!

@sscarduzio Thank you so much. yes this mystery is solved.

Here comes another one. :stuck_out_tongue_winking_eye:

getting forbidden error now:

[2018-11-08T00:16:56,301][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][4] state: [CREATED]
[2018-11-08T00:16:56,304][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1681703169--948241970#29, TYP:SearcRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:172.*********, DA:172.***********, IDX:.kibana, MET:POST, PTH:/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets, CNT:{"size":0,query":{"terms":{"type":["dashboard","visualization","search","index-pattern","graph-workspace","timelion-sheet"]}},"aggs":{"ypes":{"terms":{"field":"type","size":6}}}}, HDR:{Connection=keep-alive, Content-Length=180, content-type=application/json, Hst=readlogs.**********:9200}, HIS:[::LOGSTASH::->[auth_key->false]] }
[2018-11-08T00:16:56,305][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:302147631--1820071796#30, TYP:SearcRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:172.**************, DA:172.29.0.2, IDX:.reporting-*, MET:POST, PTH:/.reporting-*/_search?filter_path=hits.total%2Caggregations.jobTypes.buckets%2Caggregation.objectTypes.buckets%2Caggregations.layoutTypes.buckets%2Caggregations.statusTypes.buckets, CNT:{"size":0,"aggs":{"jobTypes":"terms":{"field":"jobtype","size":2}},"objectTypes":{"terms":{"field":"meta.objectType.keyword","size":3}},"layoutTypes":{"tems":{"field":"meta.layout.keyword","size":3}},"statusTypes":{"terms":{"field":"status","size":4}}}}, HDR:{Connection=keep-alie, Content-Length=255, content-type=application/json, Host=r*************:9200}, HIS:[::LOGSTASH::->[auth_key->false]]}
[2018-11-08T00:16:56,423][DEBUG][i.n.h.s.SslHandler       ] [id: 0x964b8c0c, L:/172.*************:9200 - R:/172.29.0.52:54346] HANDHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-11-08T00:16:56,426][DEBUG][t.b.r.a.ACL              ] checking request:45310663-895357540#31
[2018-11-08T00:16:56,439][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][4] state: [CREATED]->[REOVERING], reason [from store]
[2018-11-08T00:16:56,445][DEBUG][o.e.i.c.IndicesClusterStateService] [node-1] [readonlyrest_audit-2018-11-08][2] creating shad
[2018-11-08T00:16:56,448][DEBUG][o.e.i.IndexService       ] [node-1] [readonlyrest_audit-2018-11-08] [readonlyrest_audit-201811-08][2] loaded data path [/var/lib/elasticsearch/nodes/0/indices/m9MuZ5NeS0CqliU8VwARuQ/2], state path [/var/lib/elasticseach/nodes/0/indices/m9MuZ5NeS0CqliU8VwARuQ/2]
[2018-11-08T00:16:56,450][DEBUG][o.e.i.IndexService       ] [node-1] [readonlyrest_audit-2018-11-08] [readonlyrest_audit-201811-08][2] creating using an existing path [ShardPath{path=/var/lib/elasticsearch/nodes/0/indices/m9MuZ5NeS0CqliU8VwARuQ/2, shrd=[readonlyrest_audit-2018-11-08][2]}]
[2018-11-08T00:16:56,450][DEBUG][o.e.i.IndexService       ] [node-1] [readonlyrest_audit-2018-11-08] creating shard_id [readolyrest_audit-2018-11-08][2]
[2018-11-08T00:16:56,451][DEBUG][o.e.i.s.Store            ] [node-1] [readonlyrest_audit-2018-11-08][2] store stats are refrehed with refresh_interval [10s]
[2018-11-08T00:16:56,445][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!
[2018-11-08T00:16:56,451][DEBUG][t.b.r.e.RequestInfo      ] Discovered indices: .kibana
[2018-11-08T00:16:56,452][DEBUG][t.b.r.a.b.Block          ] [::LOGSTASH::] the request matches no rules in this block: { ID:4310663-895357540#31, TYP:GetRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/get, OA:72.************, DA:172.***********, IDX:.kibana, MET:GET, PTH:/.kibana/doc/config%3A6.4.2, CNT:<N/A>, HDR:{Connection=keep-alive, Conent-Length=0, Host=readlogs.********************:9200}, HIS:[::LOGSTASH::->[auth_key->false]] }
[2018-11-08T00:16:56,456][DEBUG][i.n.h.s.SslHandler       ] [id: 0x05de61ec, L:/172.***************:9200 - R:/172.****************:54350] HANDHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-11-08T00:16:56,450][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][4] starting recovery fro store ...
[2018-11-08T00:16:56,461][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][2] state: [CREATED]
[2018-11-08T00:16:56,468][DEBUG][t.b.r.a.ACL              ] checking request:340310748--948241970#32
[2018-11-08T00:16:56,452][DEBUG][r.suppressed             ] path: /.kibana/doc/config%3A6.4.2, params: {index=.kibana, id=conig:6.4.2, type=doc}
tech.beshu.ror.es.IndexLevelActionFilter$1$1: forbidden

You are sending requests without basic auth credentials, how can it work?

@sscarduzio, I am sorry but i still do not get it. For Logstash ACL block i am providing the auth_key and the other ACL block is for AD connectivity, i have specified bind credentials for that. Which basic auth credentials are we talking about here?

and i just noticed that I am getting this error as well:

[2018-11-08T14:34:55,617][ERROR][t.b.r.a.ACL              ] > Impossible to add block to ACL: Accept requests from users in group team Reason: [InitializationException] LDAP binding problem
tech.beshu.ror.acl.definitions.ldaps.LdapClientException$InitializationException: LDAP binding problem

OK much better, at least now the LDAP connector is being invoked. This can be a basic connectivity or authentication problem with the LDAP server.

Can you make sure the server is reachable and searchable from the ES machine, maybe using ldapsearch from command line?

LDAP server was reachable.

Looks like my fault, i was experimenting earlier with my script when it wasn’t working

ssl_enabled: true

i changed it to false and now it is adding ACL

[2018-11-08T15:38:51,862][INFO ][t.b.r.r.SerializationTool] no custom audit log serialisers found, proceeding with default.
[2018-11-08T15:38:52,535][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:       { name: 'Accept requests from users in group team', policy: ALLOW, rules: [ldap_auth, indices]}
[2018-11-08T15:38:52,536][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:       { name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key, actions, indices]}

However, still getting forbidden and below error:

[2018-11-08T16:36:35,211][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1265684447-432166845#75, TYP:GetRequest, CGR:N/A, USR:[n o basic auth header], BRS:false, KDX:null, ACT:indices:data/read/get, OA:172.******, DA:172.*********, IDX:.kibana, MET:GET, PTH:/.kibana/doc/confi g%3A6.4.2, CNT:<N/A>, HDR:{Connection=keep-alive, Content-Length=0, Host=readlogs.***********:9200}, HIS:[Accept requests from users in group tea m->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.ACL              ] checking request:1478575496-367159140#78
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.r.i.LdapAuthenticationAsyncRule] Basic auth header not present!
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.Block          ] [Accept requests from users in group team] the request matches no rules in this block:  { ID:1478575496-367159140#78, TYP:MonitoringBulkRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:cluster:admin/xpack/monito ring/bulk, OA:172.**********, DA:172.*********, IDX:<N/A>, MET:POST, PTH:/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000m s, CNT:{"index":{"_type":"kibana_stats"}}
{"kibana":{"uuid":"**********************","name":"Kibana","index":".kibana","host":"172.************","transport_address":"172.********:5 601","version":"6.4.2","snapshot":false,"status":"red"},"cloud":{"name":"aws","id":"i-07e48300424f85c1b","vm_type":"m4.large","region":"us-east-1" ,"zone":"us-east-1a","metadata":{"marketplaceProductCodes":null,"pendingTime":"2018-11-07T19:13:10Z","version":"2017-09-30","kernelId":null,"ramdi skId":null,"architecture":"x86_64","imageId":"ami-0ac019f4fcb7cb7e6"}}}
, HDR:{Connection=keep-alive, Content-Length=545, content-type=application/x-ndjson, Host=readlogs.********:9200}, HIS:[Accept requests from u sers in group team->[ldap_authentication->false]] }
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.Block          ] [::LOGSTASH::] the request matches no rules in this block: { ID:1478575496-367159140#7 8, TYP:MonitoringBulkRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:cluster:admin/xpack/monitoring/bulk, OA:172.********, D A:172., IDX:<N/A>, MET:POST, PTH:/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms, CNT:{"index":{"_type":"ki bana_stats"}}
{"kibana":{"uuid":"************************","name":"Kibana","index":".kibana","host":"172.*********","transport_address":"172.********:5 601","version":"6.4.2","snapshot":false,"status":"red"},"cloud":{"name":"aws","id":"i-07e48300424f85c1b","vm_type":"m4.large","region":"us-east-1" ,"zone":"us-east-1a","metadata":{"marketplaceProductCodes":null,"pendingTime":"2018-11-07T19:13:10Z","version":"2017-09-30","kernelId":null,"ramdi skId":null,"architecture":"x86_64","imageId":"ami-0ac019f4fcb7cb7e6"}}}
, HDR:{Connection=keep-alive, Content-Length=545, content-type=application/x-ndjson, Host=readlogs.**********:9200}, HIS:[Accept requests from u sers in group team->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-08T16:36:35,201][DEBUG][r.suppressed             ] path: /_xpack/monitoring/_bulk, params: {system_id=kibana, system_api_version=6, inter val=10000ms}
tech.beshu.ror.es.IndexLevelActionFilter$1$1: forbidden
        at tech.beshu.ror.es.IndexLevelActionFilter$1.onForbidden(IndexLevelActionFilter.java:163) ~[?:?]
[2018-11-08T16:23:41,036][DEBUG][i.n.u.NetUtil            ] Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. De         fault: 128
java.security.AccessControlException: access denied ("java.io.FilePermission" "/proc/sys/net/core/somaxconn" "read")

Hi @sim,
As you can see, you are still not sending any credentials! Look at the headers list, no trace of “Authorization”:

 HDR:{Connection=keep-alive, Content-Length=545, content-type=application/x-ndjson, Host=readlogs.**********:9200}

And the logs are also saying explicitly the same as above

[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!

Please use something like curl -u username:password -k 'https://eshost:9200/' if you want to test LDAP is working or not.

Hello @sscarduzio

I am passing the credentials and using the curl as you suggested already.

curl -vvv -u username@domain.local:password -k “https://readlogs****************:9200”

When i pass the logstash credentials it works but not when I pass AD credentials.

[2018-11-09T14:06:39,128][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:46682717-724168975#515, TYP:SearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:172.*********, DA:172.*********, IDX:.reporting-*, MET:POST, PTH:/.reporting-*/esqueue/_search?version=true, CNT:{"_source":{"excludes":["output.content"]},"query":{"constant_score":{"filter":{"bool":{"filter":{"term":{"jobtype":"csv"}},"should":[{"term":{"status":"pending"}},{"bool":{"filter":[{"term":{"status":"processing"}},{"range":{"process_expiration":{"lte":"2018-11-09T14:06:39.108Z"}}}]}}]}}}},"sort":[{"priority":{"order":"asc"}},{"created_at":{"order":"asc"}}],"size":10}, HDR:{Connection=keep-alive, Content-Length=371, content-type=application/json, Host=readlogs.**************:9200}, HIS:[Accept requests from users in group team->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-09T14:06:39,177][DEBUG][t.b.r.a.ACL              ] checking request:2028182155-1168792043#516
[2018-11-09T14:06:39,180][DEBUG][t.b.r.a.b.r.i.LdapAuthenticationAsyncRule] Basic auth header not present!
[2018-11-09T14:06:39,180][DEBUG][t.b.r.a.b.Block          ] [Accept requests from users in group team] the request matches no rules in this block: { ID:2028182155-1168792043#516, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:cluster:monitor/main, OA:172.***********, DA:172.**********, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Connection=keep-alive, Content-Length=0, Host=readlogs.**************:9200}, HIS:[Accept requests from users in group team->[ldap_authentication->false]] }
[2018-11-09T14:06:39,180][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!

When I curl with logstash credentials:

[2018-11-09T14:07:33,446][DEBUG][t.b.r.a.b.Block          ] matched { name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key, actions, indices]}
[2018-11-09T14:07:33,451][INFO ][t.b.r.a.ACL              ] ALLOWED by { name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key, actions, indices]} req={ ID:121996182-1666799991#660, TYP:MainRequest, CGR:N/A, USR:logstash, BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.**********, DA:172.************, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=*/*, Authorization=Basic bG9nc3Rhc2g6bG9nc3Rhc2g=, content-length=0, Host=readlogs.************:9200, User-Agent=curl/7.58.0}, HIS:[Accept requests from users in group team->[ldap_authentication->false]], [::LOGSTASH::->[indices->true, auth_key->true, actions->true]] }

here how i deal with curl :

curl -vvv -ntlm domainname\username:password -k “[https://readlogs](https://readlogs/)****************:9200”

but on my side, especially with https, my curl query stands as : (it s local credential)
curl.exe --cacert D:\certs\rootca_cert.pem --cert-type PEM -H "Authorization: Basic QWR64646454645vc21kcA==" -XGET "https://xxxxxx:9200/..."

then I think (but can not test at the moment)
curl.exe --cacert D:\certs\rootca_cert.pem --cert-type PEM -ntlm domainname\username:password -XGET "https://xxxxxx:9200/..."

but this is only some examples

1 Like

@Id57

I tried using the curl the way you do, still doesn’t work for me.

@sscarduzio please suggest, i am able to connect to ldap when i do ldapsearch even

my readonlyrest.yml config now:

readonlyrest:
   enable: true
   response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin

   ssl:
     enable: true
     keystore_file: "keystore.jks"
     keystore_pass: readonlyrest
     key_pass: readonlyrest

   audit_collector: true
   access_control_rules:

   - name: Accept requests from users in group team
     type: allow
     ldap_auth:
       name: "name"
       groups: ["admins"]
     kibana_access: rw
     kibana_index: '.kibana'
     indices: ["*"]

   # This is needed to allow Logstash (local service) to the ES (local service)
   - name: "::LOGSTASH::"
     auth_key: username:password
     actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
     indices: ["*"]

   ldaps:
   - name: name
     host: "172.***************"
     port: 389
     ssl_enabled: false
     ssl_trust_all_certs: true
     bind_dn: "cn=username,ou=elk,dc=domain,dc=local"
     bind_password: "password"
     search_user_base_DN: "ou=elk,dc=appdev,dc=local"
     search_groups_base_DN: ""cn=admins,ou=elk,dc=appdev,dc=local"
     group_from_user: true
     unique_member_attribute: "member"

This is nothing we can work with. Please show us:

  1. the whole curl command with credentials (minus password)
  2. the output of curl command above
  3. The elasticsearch log generated by that request.

@sscarduzio

  1. curl -vvv -u *****@appdev.local:password “https://readlogs.appdev.local:9200

  2. output of curl command:

* Rebuilt URL to: https://readlogs.appdev.local:9200/
*   Trying 172.*******...
* TCP_NODELAY set
* Connected to readlogs.appdev.local (172.*********) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=readlogs.appdev.local
*  start date: Nov  5 19:19:28 2018 GMT
*  expire date: Nov  2 19:19:28 2028 GMT
*  common name: readlogs.appdev.local (matched)
*  issuer: CN=readlogs.appdev.local
*  SSL certificate verify ok.
* Server auth using Basic with user 'elkadmin@appdev.local'
> GET / HTTP/1.1
> Host: readlogs.appdev.local:9200
> Authorization: Basic ZWxrYWRtaW5AYXBwZGV2LmxvY2FsOnNvYzIwMTgh
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic
< content-type: application/json; charset=UTF-8
< content-length: 135
<
* Connection #0 to host readlogs.appdev.local left intact
  1. ES logsgenerated by request
[2018-11-09T19:54:24,208][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1756903315-104142838#3740, TYP:MainRequest, CGR:N/A, USR:elkadmin@appdev.local(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.************, DA:172.*********, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=*/*, Authorization=Basic ZWxrYWRtaW5AYXBwZGV2LmxvY2FsOnNvYzIwMTgh, content-length=0, Host=readlogs.appdev.local:9200, User-Agent=curl/7.58.0}, HIS:[kibana server->[auth_key->false]], [Accept requests from users in group team->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-09T19:54:24,496][DEBUG][t.b.r.a.ACL              ] checking request:1690916047-1900108057#3741

mmmh,

well could you try :

curl -vvv -u elkadmin:password "https://readlogs.appdev.local:9200"

Is there anything in the debug log in ES that shows the LDAP activity? something is going wrong with the LDAP authentication, as I suppose this ACL block is expected to match:

[Accept requests from users in group team->[ldap_authentication->false]]

Right?

@sscarduzio

Below are the ES Debug logs, still not able to figure out why it is not authenticating.

[2018-11-12T14:20:32,752][DEBUG][r.suppressed             ] path: /, params: {}
tech.beshu.ror.es.IndexLevelActionFilter$1$1: Forbidden by ReadonlyREST ES plugin
        at tech.beshu.ror.es.IndexLevelActionFilter$1.onForbidden(IndexLevelActionFilter.java:163) ~[?:?]
        at tech.beshu.ror.acl.ACL.lambda$check$4(ACL.java:208) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_191]
        at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_191]
        at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_191]
        at tech.beshu.ror.acl.ACL.check(ACL.java:203) ~[?:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:158) ~[?:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.lambda$apply$1(IndexLevelActionFilter.java:134) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
        at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:130) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.action.RestMainAction.lambda$prepareRequest$0(RestMainAction.java:54) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97) ~[elasticsearch-6.4.2.jar:6.4.2]
        at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$5(ReadonlyRestPlugin.java:197) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:239) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:335) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:173) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:538) [transport-netty4-client-6.4.2.jar:6.4.2]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137) [transport-netty4-client-6.4.2.jar:6.4.2]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.4.2.jar:6.4.2]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2018-11-12T14:20:32,760][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1082775497-1127994063#2207791, TYP:MainRequest, CGR:N/A, USR:elkadmin@appdev.local(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.****, DA:172.*********, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=*/*, Authorization=Basic ZWxrYWRtaW5AYXBwZGV2LmxvY2FsOnNvYzIwMTgh, content-length=0, Host=readlogs.appdev.local:9200, User-Agent=curl/7.58.0}, HIS:[kibana server->[auth_key->false]], [::LOGSTASH::->[auth_key->false]], [Accept requests from users in group team->[ldap_authentication->false]] }
[2018-11-12T14:20:32,761][DEBUG][i.n.h.s.SslHandler       ] [id: 0xb3432730, L:/172.**********:9200 - R:/172.******:54204] Swallowing a harmless 'connection reset by peer / broken pipe' error that occurred while writing close_notify in response to the peer's close_notify
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
        at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2018-11-12T14:20:32,785][DEBUG][t.b.r.a.ACL              ] checking request:1109804068--1517726027#2207792

@sscarduzio please suggest

I have tried that, doesn’t work either

coming back in a few, testing on my infra structure.

@ld57 Thank you. Waiting!