Curious behavior "red" vs "yellow" indices

#1

I’m seeing some curious behavior, and I’d love to understand what is going on.

When I execute the following with localhost …

curl --user elk_testuser1:Welcome1 ‘localhost:9200/_cat/indices?v’

… the .monitoring* indices are “yellow”.

yellow open   logs-2017.04.18                   n29scj6pS_i-QqeWHt2tXg   1   1      32607            0     26.3mb         26.3mb
yellow open   .monitoring-es-2-2017.05.04       lnnG5yAaSGKIntVTxmBvgw   1   1     227125         2582     83.1mb         83.1mb
yellow open   logs-2016.12.17                   S0TTzSyvQhqmjYeamal-aw   1   1      17735            0     12.5mb         12.5mb

But when I execute the following from localhost but specifying the actual IP …

curl --user elk_testuser1:Welcome1 ‘10.230.143.160:9200/_cat/indices?v’

… the .monitoring* indices are “red*”.

yellow open   logs-2017.04.18                   n29scj6pS_i-QqeWHt2tXg   1   1      32607            0     26.3mb         26.3mb
red*   open   .monitoring-es-2-2017.05.04       lnnG5yAaSGKIntVTxmBvgw             227125         2582     83.1mb         83.1mb
yellow open   logs-2016.12.17                   S0TTzSyvQhqmjYeamal-aw   1   1      17735            0     12.5mb         12.5mb

When executing the first statement the logs show the following (simplified):

[2017-05-11T09:23:17,712][INFO ] ... TYP:ClusterStateRequest ... ACT:cluster:monitor/state ... matched block: ::REST-ADMIN:: match: true}
[2017-05-11T09:23:17,715][INFO ] ... TYP:IndicesStatsRequest ... ACT:indices:monitor/stats ... matched block: ::REST-ADMIN:: match: true}

When executing the second statement the logs show the following (simplified):

[2017-05-11T13:31:40,445][INFO ] ... TYP:ClusterStateRequest ... ACT:cluster:monitor/state ... matched block: ::REST-ADMIN:: match: true}
[2017-05-11T13:31:40,446][INFO ] ... TYP:ClusterHealthRequest ... ACT:cluster:monitor/health ... matched block: ::KIBANA-USER-RW:: match: true}
[2017-05-11T13:31:40,448][INFO ] ... TYP:IndicesStatsRequest ... ACT:indices:monitor/stats ... matched block: ::REST-ADMIN:: match: true}

The ClusterHealthRequest request always shows when specifying the IP, but never when using localhost.

Notice, too, that this rule is matching ::KIBANA-USER-RW:: even though the statement is not originating from Kibana. (Maybe this is not how it works, though.)

The relevant rules are:

    - name: "::KIBANA-USER-RW::"
      ldap_auth:
        name: "ldap1"
        groups: ["elk_engineering"]
      type: allow
      kibana_access: rw
      indices: [".kibana", ".kibana-devnull", "logs-*"]

    # admin REST access via LDAP
    - name: "::REST-ADMIN::"
      ldap_auth:
        name: "ldap1"
        groups: ["elk_engineering"]
      type: allow
      indices: ["*"]

I’m curious if anyone has any thoughts on why this is occurring. Probably not a big deal, but I would like to resolve it if possible.

Thanks much,
Greg

#2

Is elk_testuser1:Welcome1 a valid login in ldap1? Or should in theory this be a failed login?

In general: blocks are evaluated one by one, the first who matches, the “type” policy is enforced (i.e allow/forbid) and the evaluation stops.

You can see the history of the evaluation of the access control chain in the “HST” field in the logs.

About the yellow/red index, with localhost looks like you’re asking the node’s opinion about an index (got the index, but couldn’t replicate), and with the other IP you get the cluster’s opinion (I know the index exists, but could not get the data replicated yet). That means, something else is wrong on the transport side or cluster discovery settings.

#3

Thanks for the reply.

So, it turns out this is related somewhat to my earlier post about ClusterHealthRequest (Curious behavior “red” vs “yellow” indices). I’d forgotten about that one.

In that post you suggested adding another ACL block, which I did:

    - name: "::ELASTICSEARCH::"
      actions: ["cluster:monitor/health"]
      hosts: ["127.0.0.1"]
      verbosity: error

Since it was for localhost only, and since I had verbosity set to error, that explains why I was not seeing a ClusterHealthRequest line in the logs when the localhost version of the curl statement was executed.

After disabling that ACL rule, I now get “red*” “.monitoring*” indexes whether I use localhost or specify an IP. At least it’s now consistent.

I’m curious why the ::KIBANA-USER-RW:: rule is matched even though I’m not using Kibana for the query. I suppose it doesn’t actually verify that the request is coming from Kibana, right?

If you have any further thoughts on how to resolve the “red*” indexes, I’d be interested.

Thanks,
Greg

#4

Oh, and to answer your question, yes, elk_testuser1 is a valid login.