Unclear error with incorrect password

Hi

Am using ElasticSearch 7.17, ROR 1.48.

We’ve noticed an odd behaviour when testing some API calls. We accidentally had a password wrong but the error that resulted was really non obvious. Instead of a response that indicated invalid password, the response instead was a 404 index does not exist, with the index name changed to include an ROR suffix. See below - the response from query against an index that does exist but with a bad password is the same as the response from an index that does not exist but with the correct password!

Is this something off in our configuration - if so any idea what it might be, I can’t see anything obviously wrong!

Or is this just a quirk in the way ROR works? If so, would be great to improve on the error messages here in a future release, as it took us a lot longer than it should have to diagnose and solve this problem!

Thanks,

Adrian

curl -s -u myuser:my_incorrect_password -XGET "http://elasticServer:9200/myindex_that_exists/_search?pretty=true" -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [myindex_that_exists_ROR_IgnPGL4ZhF]",
        "resource.type" : "index_or_alias",
        "resource.id" : "myindex_that_exists_ROR_IgnPGL4ZhF",
        "index_uuid" : "_na_",
        "index" : "myindex_that_exists_ROR_IgnPGL4ZhF"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [myindex_that_exists_ROR_IgnPGL4ZhF]",
    "resource.type" : "index_or_alias",
    "resource.id" : "myindex_that_exists_ROR_IgnPGL4ZhF",
    "index_uuid" : "_na_",
    "index" : "myindex_that_exists_ROR_IgnPGL4ZhF"
  },
  "status" : 404
}

curl -s -u myuser:my_correct_password -XGET "http://elasticServer:9200/myindex_that_does_not_exist/_search?pretty=true" -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [myindex_that_does_not_exist]",
        "resource.type" : "index_or_alias",
        "resource.id" : "myindex_that_does_not_exist",
        "index_uuid" : "_na_",
        "index" : "myindex_that_does_not_exist"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [myindex_that_does_not_exist_ROR_IgnPGL4ZhF]",
    "resource.type" : "index_or_alias",
    "resource.id" : "myindex_that_does_not_exist_ROR_IgnPGL4ZhF",
    "index_uuid" : "_na_",
    "index" : "myindex_that_does_not_exist_ROR_IgnPGL4ZhF"
  },
  "status" : 404
}

Hi Adrian,

I think I know why, but to be sure please share your ROR configuration.

Thanks

Hi @coutoPL , thanks for your response on this. Most of the ROR config file we have is sensitive data so would prefer not to share here. Is there any specific settings etc i could check for you? Or, I can try share a somewhat redacted version via the support portal?

Thx,

Adrian

ok, no problem. I will try to explain on the example:

readonlyrest:
  access_control_rules:

    - name: "KIBANA"
      type: allow
      auth_key: kibana:kibana
      verbosity: error

    - name: "Privileged users"
      auth_key: "user:test"
      indices: ["*logs"]
      kibana:
        access: ro

    - name: "Public users"
      indices: ["public_data"]
      kibana:
        access: ro

And curl tests:

curl -k -u user:test 'https://localhost:19200/business_logs/_search?pretty' -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 50,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "business_logs",
        "_id" : "C-ytuJQBLvyVr-NjTcmE",
        "_score" : 1.0,
        "_source" : {
          "message" : "User 2495 login successful",
          "level" : "WARN",
          "timestamp" : "2025-01-30T19:25:52Z",
          "user_id" : "2495"
        }
      }
      ....
}
curl -k -u user:wrong 'https://localhost:19200/business_logs/_search?pretty' -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [business_logs_ROR_jACix1eubK]",
        "resource.type" : "index_or_alias",
        "resource.id" : "business_logs_ROR_jACix1eubK",
        "index_uuid" : "_na_",
        "index" : "business_logs_ROR_jACix1eubK"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [business_logs_ROR_jACix1eubK]",
    "resource.type" : "index_or_alias",
    "resource.id" : "business_logs_ROR_jACix1eubK",
    "index_uuid" : "_na_",
    "index" : "business_logs_ROR_jACix1eubK"
  },
  "status" : 404
}

Explanation:
The first call had the correct credentials, and the call was allowed by the “Privileged users” block. The second call had incorrect credentials, so the “Privileged users” block was rejected (because the auth_key rule failed to match). But the next “Public users” was matched. This is the “Public users” block, so no credentials are required to match it. Moreover, it restricts the viewing of only the “public_data” index.

The indices rule doesn’t forbid the block (for RO requests) but filters them. In the example, we called the search for the business_logs index. As we know, the index exists, but this block doesn’t allow to give access to it.

So, we add the suffix with a random string to the called index name and let it pass the ACL. Why do we do that? The main reason is that we want users to feel like they are alone on the cluster.

Is it your case?

Yes, that’s it! We did have one block in our ROR config which allows for unauthenticated access to a very small subset of our indexes. If I temporarily remove that from our config, the response for request without correct password change is now HTTP 403 for me.

It doesn’t quite behave the way I might have initiatively expected, but it makes sense now at least, thanks for the explanation.

Also, note that the examples with my original post were actually slightly wrong - I had edited these to redact some sensitive details and in doing so I got it slightly wrong! There IS a difference in my case between the responses for an existing index with a bad password, and an index that does not exist but with a good password. That is, in the good password case, the "“no such index” reason in response does NOT include the _ROR… suffix - it just repeats the index name exactly as it was in the query. I’ll correct that now for the record!

1 Like

Great to hear that! Good job!