Need to connect elasticsearch with Active Directory for authentication - Need help


#24

@sscarduzio I even validated it with a YAML Validator
http://www.yamllint.com/

readonlyrest:
    ssl:
      enable: true
      keystore_file: "keystore.jks"
      keystore_pass: readonlyrest
      key_pass: readonlyrest

      audit_collector: true
      access_control_rules:

      - name: Accept requests from users in group team
        type: allow
        ldap_auth:
          name: "name"
          groups: "admins"
        indices: ["*"]

      - name: "::LOGSTASH::"
        auth_key: user:password
        actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
        indices: ["*"]

      ldaps:
      - name: name
        host: "172.**********"
        port: 389
        ssl_enabled: true
        ssl_trust_all_certs: true
        bind_dn: "cn=elkadmin,ou=elk,dc=domain,dc=local"
        bind_password: "********"
        search_user_base_DN: "ou=elk,dc=domain,dc=local"
        search_groups_base_DN: "ou=elk,dc=domain,dc=local"

(Simone Scarduzio) #25

Your yaml is valid, but wrong. The “ssl”, “audit_collector”, “access_control_rules” entries should be siblings.
In your file they are all child nodes to “ssl”.

Mystery solved I think!


#26

@sscarduzio Thank you so much. yes this mystery is solved.

Here comes another one. :stuck_out_tongue_winking_eye:

getting forbidden error now:

[2018-11-08T00:16:56,301][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][4] state: [CREATED]
[2018-11-08T00:16:56,304][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1681703169--948241970#29, TYP:SearcRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:172.*********, DA:172.***********, IDX:.kibana, MET:POST, PTH:/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets, CNT:{"size":0,query":{"terms":{"type":["dashboard","visualization","search","index-pattern","graph-workspace","timelion-sheet"]}},"aggs":{"ypes":{"terms":{"field":"type","size":6}}}}, HDR:{Connection=keep-alive, Content-Length=180, content-type=application/json, Hst=readlogs.**********:9200}, HIS:[::LOGSTASH::->[auth_key->false]] }
[2018-11-08T00:16:56,305][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:302147631--1820071796#30, TYP:SearcRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:172.**************, DA:172.29.0.2, IDX:.reporting-*, MET:POST, PTH:/.reporting-*/_search?filter_path=hits.total%2Caggregations.jobTypes.buckets%2Caggregation.objectTypes.buckets%2Caggregations.layoutTypes.buckets%2Caggregations.statusTypes.buckets, CNT:{"size":0,"aggs":{"jobTypes":"terms":{"field":"jobtype","size":2}},"objectTypes":{"terms":{"field":"meta.objectType.keyword","size":3}},"layoutTypes":{"tems":{"field":"meta.layout.keyword","size":3}},"statusTypes":{"terms":{"field":"status","size":4}}}}, HDR:{Connection=keep-alie, Content-Length=255, content-type=application/json, Host=r*************:9200}, HIS:[::LOGSTASH::->[auth_key->false]]}
[2018-11-08T00:16:56,423][DEBUG][i.n.h.s.SslHandler       ] [id: 0x964b8c0c, L:/172.*************:9200 - R:/172.29.0.52:54346] HANDHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-11-08T00:16:56,426][DEBUG][t.b.r.a.ACL              ] checking request:45310663-895357540#31
[2018-11-08T00:16:56,439][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][4] state: [CREATED]->[REOVERING], reason [from store]
[2018-11-08T00:16:56,445][DEBUG][o.e.i.c.IndicesClusterStateService] [node-1] [readonlyrest_audit-2018-11-08][2] creating shad
[2018-11-08T00:16:56,448][DEBUG][o.e.i.IndexService       ] [node-1] [readonlyrest_audit-2018-11-08] [readonlyrest_audit-201811-08][2] loaded data path [/var/lib/elasticsearch/nodes/0/indices/m9MuZ5NeS0CqliU8VwARuQ/2], state path [/var/lib/elasticseach/nodes/0/indices/m9MuZ5NeS0CqliU8VwARuQ/2]
[2018-11-08T00:16:56,450][DEBUG][o.e.i.IndexService       ] [node-1] [readonlyrest_audit-2018-11-08] [readonlyrest_audit-201811-08][2] creating using an existing path [ShardPath{path=/var/lib/elasticsearch/nodes/0/indices/m9MuZ5NeS0CqliU8VwARuQ/2, shrd=[readonlyrest_audit-2018-11-08][2]}]
[2018-11-08T00:16:56,450][DEBUG][o.e.i.IndexService       ] [node-1] [readonlyrest_audit-2018-11-08] creating shard_id [readolyrest_audit-2018-11-08][2]
[2018-11-08T00:16:56,451][DEBUG][o.e.i.s.Store            ] [node-1] [readonlyrest_audit-2018-11-08][2] store stats are refrehed with refresh_interval [10s]
[2018-11-08T00:16:56,445][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!
[2018-11-08T00:16:56,451][DEBUG][t.b.r.e.RequestInfo      ] Discovered indices: .kibana
[2018-11-08T00:16:56,452][DEBUG][t.b.r.a.b.Block          ] [::LOGSTASH::] the request matches no rules in this block: { ID:4310663-895357540#31, TYP:GetRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/get, OA:72.************, DA:172.***********, IDX:.kibana, MET:GET, PTH:/.kibana/doc/config%3A6.4.2, CNT:<N/A>, HDR:{Connection=keep-alive, Conent-Length=0, Host=readlogs.********************:9200}, HIS:[::LOGSTASH::->[auth_key->false]] }
[2018-11-08T00:16:56,456][DEBUG][i.n.h.s.SslHandler       ] [id: 0x05de61ec, L:/172.***************:9200 - R:/172.****************:54350] HANDHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-11-08T00:16:56,450][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][4] starting recovery fro store ...
[2018-11-08T00:16:56,461][DEBUG][o.e.i.s.IndexShard       ] [node-1] [readonlyrest_audit-2018-11-08][2] state: [CREATED]
[2018-11-08T00:16:56,468][DEBUG][t.b.r.a.ACL              ] checking request:340310748--948241970#32
[2018-11-08T00:16:56,452][DEBUG][r.suppressed             ] path: /.kibana/doc/config%3A6.4.2, params: {index=.kibana, id=conig:6.4.2, type=doc}
tech.beshu.ror.es.IndexLevelActionFilter$1$1: forbidden

(Simone Scarduzio) #27

You are sending requests without basic auth credentials, how can it work?


#28

@sscarduzio, I am sorry but i still do not get it. For Logstash ACL block i am providing the auth_key and the other ACL block is for AD connectivity, i have specified bind credentials for that. Which basic auth credentials are we talking about here?

and i just noticed that I am getting this error as well:

[2018-11-08T14:34:55,617][ERROR][t.b.r.a.ACL              ] > Impossible to add block to ACL: Accept requests from users in group team Reason: [InitializationException] LDAP binding problem
tech.beshu.ror.acl.definitions.ldaps.LdapClientException$InitializationException: LDAP binding problem

(Simone Scarduzio) #29

OK much better, at least now the LDAP connector is being invoked. This can be a basic connectivity or authentication problem with the LDAP server.

Can you make sure the server is reachable and searchable from the ES machine, maybe using ldapsearch from command line?


#30

LDAP server was reachable.

Looks like my fault, i was experimenting earlier with my script when it wasn’t working

ssl_enabled: true

i changed it to false and now it is adding ACL

[2018-11-08T15:38:51,862][INFO ][t.b.r.r.SerializationTool] no custom audit log serialisers found, proceeding with default.
[2018-11-08T15:38:52,535][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:       { name: 'Accept requests from users in group team', policy: ALLOW, rules: [ldap_auth, indices]}
[2018-11-08T15:38:52,536][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:       { name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key, actions, indices]}

However, still getting forbidden and below error:

[2018-11-08T16:36:35,211][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1265684447-432166845#75, TYP:GetRequest, CGR:N/A, USR:[n o basic auth header], BRS:false, KDX:null, ACT:indices:data/read/get, OA:172.******, DA:172.*********, IDX:.kibana, MET:GET, PTH:/.kibana/doc/confi g%3A6.4.2, CNT:<N/A>, HDR:{Connection=keep-alive, Content-Length=0, Host=readlogs.***********:9200}, HIS:[Accept requests from users in group tea m->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.ACL              ] checking request:1478575496-367159140#78
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.r.i.LdapAuthenticationAsyncRule] Basic auth header not present!
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.Block          ] [Accept requests from users in group team] the request matches no rules in this block:  { ID:1478575496-367159140#78, TYP:MonitoringBulkRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:cluster:admin/xpack/monito ring/bulk, OA:172.**********, DA:172.*********, IDX:<N/A>, MET:POST, PTH:/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000m s, CNT:{"index":{"_type":"kibana_stats"}}
{"kibana":{"uuid":"**********************","name":"Kibana","index":".kibana","host":"172.************","transport_address":"172.********:5 601","version":"6.4.2","snapshot":false,"status":"red"},"cloud":{"name":"aws","id":"i-07e48300424f85c1b","vm_type":"m4.large","region":"us-east-1" ,"zone":"us-east-1a","metadata":{"marketplaceProductCodes":null,"pendingTime":"2018-11-07T19:13:10Z","version":"2017-09-30","kernelId":null,"ramdi skId":null,"architecture":"x86_64","imageId":"ami-0ac019f4fcb7cb7e6"}}}
, HDR:{Connection=keep-alive, Content-Length=545, content-type=application/x-ndjson, Host=readlogs.********:9200}, HIS:[Accept requests from u sers in group team->[ldap_authentication->false]] }
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!
[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.Block          ] [::LOGSTASH::] the request matches no rules in this block: { ID:1478575496-367159140#7 8, TYP:MonitoringBulkRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:cluster:admin/xpack/monitoring/bulk, OA:172.********, D A:172., IDX:<N/A>, MET:POST, PTH:/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms, CNT:{"index":{"_type":"ki bana_stats"}}
{"kibana":{"uuid":"************************","name":"Kibana","index":".kibana","host":"172.*********","transport_address":"172.********:5 601","version":"6.4.2","snapshot":false,"status":"red"},"cloud":{"name":"aws","id":"i-07e48300424f85c1b","vm_type":"m4.large","region":"us-east-1" ,"zone":"us-east-1a","metadata":{"marketplaceProductCodes":null,"pendingTime":"2018-11-07T19:13:10Z","version":"2017-09-30","kernelId":null,"ramdi skId":null,"architecture":"x86_64","imageId":"ami-0ac019f4fcb7cb7e6"}}}
, HDR:{Connection=keep-alive, Content-Length=545, content-type=application/x-ndjson, Host=readlogs.**********:9200}, HIS:[Accept requests from u sers in group team->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-08T16:36:35,201][DEBUG][r.suppressed             ] path: /_xpack/monitoring/_bulk, params: {system_id=kibana, system_api_version=6, inter val=10000ms}
tech.beshu.ror.es.IndexLevelActionFilter$1$1: forbidden
        at tech.beshu.ror.es.IndexLevelActionFilter$1.onForbidden(IndexLevelActionFilter.java:163) ~[?:?]
[2018-11-08T16:23:41,036][DEBUG][i.n.u.NetUtil            ] Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. De         fault: 128
java.security.AccessControlException: access denied ("java.io.FilePermission" "/proc/sys/net/core/somaxconn" "read")


(Simone Scarduzio) #31

Hi @sim,
As you can see, you are still not sending any credentials! Look at the headers list, no trace of “Authorization”:

 HDR:{Connection=keep-alive, Content-Length=545, content-type=application/x-ndjson, Host=readlogs.**********:9200}

And the logs are also saying explicitly the same as above

[2018-11-08T16:36:35,213][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!

Please use something like curl -u username:password -k 'https://eshost:9200/' if you want to test LDAP is working or not.


#32

Hello @sscarduzio

I am passing the credentials and using the curl as you suggested already.

curl -vvv -u [email protected]:password -k “https://readlogs****************:9200”

When i pass the logstash credentials it works but not when I pass AD credentials.

[2018-11-09T14:06:39,128][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:46682717-724168975#515, TYP:SearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:indices:data/read/search, OA:172.*********, DA:172.*********, IDX:.reporting-*, MET:POST, PTH:/.reporting-*/esqueue/_search?version=true, CNT:{"_source":{"excludes":["output.content"]},"query":{"constant_score":{"filter":{"bool":{"filter":{"term":{"jobtype":"csv"}},"should":[{"term":{"status":"pending"}},{"bool":{"filter":[{"term":{"status":"processing"}},{"range":{"process_expiration":{"lte":"2018-11-09T14:06:39.108Z"}}}]}}]}}}},"sort":[{"priority":{"order":"asc"}},{"created_at":{"order":"asc"}}],"size":10}, HDR:{Connection=keep-alive, Content-Length=371, content-type=application/json, Host=readlogs.**************:9200}, HIS:[Accept requests from users in group team->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-09T14:06:39,177][DEBUG][t.b.r.a.ACL              ] checking request:2028182155-1168792043#516
[2018-11-09T14:06:39,180][DEBUG][t.b.r.a.b.r.i.LdapAuthenticationAsyncRule] Basic auth header not present!
[2018-11-09T14:06:39,180][DEBUG][t.b.r.a.b.Block          ] [Accept requests from users in group team] the request matches no rules in this block: { ID:2028182155-1168792043#516, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, KDX:null, ACT:cluster:monitor/main, OA:172.***********, DA:172.**********, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Connection=keep-alive, Content-Length=0, Host=readlogs.**************:9200}, HIS:[Accept requests from users in group team->[ldap_authentication->false]] }
[2018-11-09T14:06:39,180][DEBUG][t.b.r.a.b.r.i.AuthKeySyncRule] Basic auth header or auth key not present!

When I curl with logstash credentials:

[2018-11-09T14:07:33,446][DEBUG][t.b.r.a.b.Block          ] matched { name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key, actions, indices]}
[2018-11-09T14:07:33,451][INFO ][t.b.r.a.ACL              ] ALLOWED by { name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key, actions, indices]} req={ ID:121996182-1666799991#660, TYP:MainRequest, CGR:N/A, USR:logstash, BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.**********, DA:172.************, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=*/*, Authorization=Basic bG9nc3Rhc2g6bG9nc3Rhc2g=, content-length=0, Host=readlogs.************:9200, User-Agent=curl/7.58.0}, HIS:[Accept requests from users in group team->[ldap_authentication->false]], [::LOGSTASH::->[indices->true, auth_key->true, actions->true]] }

(Ld57) #33

here how i deal with curl :

curl -vvv -ntlm domainname\username:password -k “[https://readlogs](https://readlogs/)****************:9200”

but on my side, especially with https, my curl query stands as : (it s local credential)
curl.exe --cacert D:\certs\rootca_cert.pem --cert-type PEM -H "Authorization: Basic QWR64646454645vc21kcA==" -XGET "https://xxxxxx:9200/..."

then I think (but can not test at the moment)
curl.exe --cacert D:\certs\rootca_cert.pem --cert-type PEM -ntlm domainname\username:password -XGET "https://xxxxxx:9200/..."

but this is only some examples


#34

@Id57

I tried using the curl the way you do, still doesn’t work for me.


#35

@sscarduzio please suggest, i am able to connect to ldap when i do ldapsearch even

my readonlyrest.yml config now:

readonlyrest:
   enable: true
   response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin

   ssl:
     enable: true
     keystore_file: "keystore.jks"
     keystore_pass: readonlyrest
     key_pass: readonlyrest

   audit_collector: true
   access_control_rules:

   - name: Accept requests from users in group team
     type: allow
     ldap_auth:
       name: "name"
       groups: ["admins"]
     kibana_access: rw
     kibana_index: '.kibana'
     indices: ["*"]

   # This is needed to allow Logstash (local service) to the ES (local service)
   - name: "::LOGSTASH::"
     auth_key: username:password
     actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
     indices: ["*"]

   ldaps:
   - name: name
     host: "172.***************"
     port: 389
     ssl_enabled: false
     ssl_trust_all_certs: true
     bind_dn: "cn=username,ou=elk,dc=domain,dc=local"
     bind_password: "password"
     search_user_base_DN: "ou=elk,dc=appdev,dc=local"
     search_groups_base_DN: ""cn=admins,ou=elk,dc=appdev,dc=local"
     group_from_user: true
     unique_member_attribute: "member"

(Simone Scarduzio) #36

This is nothing we can work with. Please show us:

  1. the whole curl command with credentials (minus password)
  2. the output of curl command above
  3. The elasticsearch log generated by that request.

#37

@sscarduzio

  1. curl -vvv -u [email protected]:password “https://readlogs.appdev.local:9200

  2. output of curl command:

* Rebuilt URL to: https://readlogs.appdev.local:9200/
*   Trying 172.*******...
* TCP_NODELAY set
* Connected to readlogs.appdev.local (172.*********) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=readlogs.appdev.local
*  start date: Nov  5 19:19:28 2018 GMT
*  expire date: Nov  2 19:19:28 2028 GMT
*  common name: readlogs.appdev.local (matched)
*  issuer: CN=readlogs.appdev.local
*  SSL certificate verify ok.
* Server auth using Basic with user '[email protected]'
> GET / HTTP/1.1
> Host: readlogs.appdev.local:9200
> Authorization: Basic ZWxrYWRtaW5AYXBwZGV2LmxvY2FsOnNvYzIwMTgh
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic
< content-type: application/json; charset=UTF-8
< content-length: 135
<
* Connection #0 to host readlogs.appdev.local left intact
  1. ES logsgenerated by request
[2018-11-09T19:54:24,208][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1756903315-104142838#3740, TYP:MainRequest, CGR:N/A, USR:[email protected](?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.************, DA:172.*********, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=*/*, Authorization=Basic ZWxrYWRtaW5AYXBwZGV2LmxvY2FsOnNvYzIwMTgh, content-length=0, Host=readlogs.appdev.local:9200, User-Agent=curl/7.58.0}, HIS:[kibana server->[auth_key->false]], [Accept requests from users in group team->[ldap_authentication->false]], [::LOGSTASH::->[auth_key->false]] }
[2018-11-09T19:54:24,496][DEBUG][t.b.r.a.ACL              ] checking request:1690916047-1900108057#3741


(Ld57) #38

mmmh,

well could you try :

curl -vvv -u elkadmin:password "https://readlogs.appdev.local:9200"


(Simone Scarduzio) #39

Is there anything in the debug log in ES that shows the LDAP activity? something is going wrong with the LDAP authentication, as I suppose this ACL block is expected to match:

[Accept requests from users in group team->[ldap_authentication->false]]

Right?


#40

@sscarduzio

Below are the ES Debug logs, still not able to figure out why it is not authenticating.

[2018-11-12T14:20:32,752][DEBUG][r.suppressed             ] path: /, params: {}
tech.beshu.ror.es.IndexLevelActionFilter$1$1: Forbidden by ReadonlyREST ES plugin
        at tech.beshu.ror.es.IndexLevelActionFilter$1.onForbidden(IndexLevelActionFilter.java:163) ~[?:?]
        at tech.beshu.ror.acl.ACL.lambda$check$4(ACL.java:208) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_191]
        at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_191]
        at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_191]
        at tech.beshu.ror.acl.ACL.check(ACL.java:203) ~[?:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:158) ~[?:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.lambda$apply$1(IndexLevelActionFilter.java:134) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
        at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:130) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.action.RestMainAction.lambda$prepareRequest$0(RestMainAction.java:54) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97) ~[elasticsearch-6.4.2.jar:6.4.2]
        at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$5(ReadonlyRestPlugin.java:197) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:239) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:335) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:173) [elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:538) [transport-netty4-client-6.4.2.jar:6.4.2]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137) [transport-netty4-client-6.4.2.jar:6.4.2]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.4.2.jar:6.4.2]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2018-11-12T14:20:32,760][INFO ][t.b.r.a.ACL              ] FORBIDDEN by default req={ ID:1082775497-1127994063#2207791, TYP:MainRequest, CGR:N/A, USR:[email protected](?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.****, DA:172.*********, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:{Accept=*/*, Authorization=Basic ZWxrYWRtaW5AYXBwZGV2LmxvY2FsOnNvYzIwMTgh, content-length=0, Host=readlogs.appdev.local:9200, User-Agent=curl/7.58.0}, HIS:[kibana server->[auth_key->false]], [::LOGSTASH::->[auth_key->false]], [Accept requests from users in group team->[ldap_authentication->false]] }
[2018-11-12T14:20:32,761][DEBUG][i.n.h.s.SslHandler       ] [id: 0xb3432730, L:/172.**********:9200 - R:/172.******:54204] Swallowing a harmless 'connection reset by peer / broken pipe' error that occurred while writing close_notify in response to the peer's close_notify
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
        at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2018-11-12T14:20:32,785][DEBUG][t.b.r.a.ACL              ] checking request:1109804068--1517726027#2207792

@sscarduzio please suggest


#41

I have tried that, doesn’t work either


(Ld57) #42

coming back in a few, testing on my infra structure.


#43

@ld57 Thank you. Waiting!