Metricbeat with ROR Pro-1.34.0_es7.10.2

1

I am trying to send data from metricbeat to elasticsearch ( both on 7.10.2 ) and we are using ROR for authentication via ldap for kibana.

I am seeing error as below in the elasticsearch logs , i have already added a block to allow indices from metricbeat however it seems to pickup logstash as the user , which seems odd to me.

Errors at metricbeat

metricbeat test output
elasticsearch: http://x.x.x.x:9200…
parse url… OK
connection…
parse host… OK
dns lookup… OK
addresses: x.x.x.x
dial up… OK
TLS… WARN secure connection disabled
talk to server… ERROR 401 Unauthorized: {“error”:{“root_cause”:[{“reason”:“Sorry, Forbidden Request by Elasticsearch. Please contact Infra team.”,“due_to”:[“OPERATION_NOT_ALLOWED”]}],“reason”:“Sorry, Forbidden Request by Elasticsearch. Please contact Infra team.”,“due_to”:[“OPERATION_NOT_ALLOWED”],“status”:401}}

Readonly rest plugin at ELS config

readonlyrest:
enable: true
prompt_for_basic_auth: true
response_if_req_forbidden: Sorry, Forbidden Request by Elasticsearch. Please contact Infra team.

access_control_rules:

  • name: “::METRICBEAT::”
    auth_key: xxx:xxx
    type: allow
    verbosity: error
    indices: [“metricbeat*”, “.monitoring-es*”]
    actions: [“*”]

Error in elastic log

[2021-12-01T14:26:01,075][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [xxxxx] FORBIDDEN by default req={ ID:91942345-951663213#36616, TYP:MainRequest, CGR:N/A, USR:logstash (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:xxxxx/32, XFF:null, DA:xxxxx/32, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Authorization=, Connection=Keep-Alive, Content-Type=application/json, Host=xxxxx:9200, User-Agent=Manticore 0.7.0, content-length=0, HIS:[::KIBANA-SRV::-> RULES:[auth_key->false]], [::BD::-> RULES:[auth_key->false]], [Accept requests from users with valid LDAP credentials for Infra-> RULES:[ldap_authentication->false]], [Accept requests from users with valid LDAP credentials for NW team-> RULES:[ldap_authentication->false]], }

Could you advise ?

2

In ES log ROR FORBIDDEN entry contains HIS field:

HIS:[::KIBANA-SRV::-> RULES:[auth_key->false]], [::BD::-> RULES:[auth_key->false]], [Accept requests from users with valid LDAP credentials for Infra-> RULES:[ldap_authentication->false]], [Accept requests from users with valid LDAP credentials for NW team-> RULES:[ldap_authentication->false]],

It can be used to figure out what rule in the given block failed to match. Eg.

  • block “::KIBANA-SRV::” - the first failed rule: “auth_key”
  • block “::BD::” - the first failed rule: “auth_key”
  • block “Accept requests from users with valid LDAP credentials for Infra” - the first failed rule: “ldap_authentication”
  • etc

The config you’ve shown us contains only one block “::METRICBEAT::”. As you can see the FORBIDDEN log doesn’t contain it, so it means, you’ve shown us:

  1. the wrong config
  2. the wrong log

BTW: when you’re struggling to solve a problem with LDAP connector, it’s a good idea to enable debug logs

3

I enabled the debug as suggested. For some reason i am getting a different error now

[2021-12-02T00:18:24,337][ERROR][t.b.r.e.IndexLevelActionFilter] [xxx] [1816468947-993770274#42114] Cannot handle the / request because ReadonlyREST failed to start
[2021-12-02T00:18:24,338][DEBUG][r.suppressed ] [xxx] path: /, params: {}
tech.beshu.ror.es.handler.response.ForbiddenResponse: forbidden
at tech.beshu.ror.es.handler.response.ForbiddenResponse$.createRorStartingFailureResponse(ForbiddenResponse.scala:69) ~[readonlyrest-1.36.0_es7.10.2.jar:?]
at tech.beshu.ror.es.IndexLevelActionFilter.proceedByRorEngine(IndexLevelActionFilter.scala:129) [readonlyrest-1.36.0_es7.10.2.jar:?]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$apply$1(IndexLevelActionFilter.scala:107) [readonlyrest-1.36.0_es7.10.2.jar:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [scala-library-2.12.10.jar:?]
at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:25) [core-1.36.0.jar:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:24) [core-1.36.0.jar:?]
at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.scala:93) [readonlyrest-1.36.0_es7.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.rest.action.RestMainAction.lambda$prepareRequest$0(RestMainAction.java:57) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115) [elasticsearch-7.10.2.jar:7.10.2]
at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$getRestHandlerWrapper$2(ReadonlyRestPlugin.scala:239) [readonlyrest-1.36.0_es7.10.2.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]
:

I have tried upgrading the ROR from 129 to 1.36 (no luck), Any suggestion if more debugging can be enabled or any further steps

4

The log says: Cannot handle the / request because ReadonlyREST failed to start

It means that ROR was not able to start. Please try to search for log “ROR starting failure”. There should be a cause of that failure