Reindex - index_not_found_exception

Users Feedback
1

Good day. Unfortunately, in our dev environment, we unable to execute _reindex with ReadonlyREST. However, we can perform other commands, like _search, _cat, etc. but just not _reindex.
We are using the free ReadonlyREST version readonlyrest-1.23.0_es7.5.0 plugin.
We see the error message -
status:404. "root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.

In our production environment, everything works fine.
We are using the free ReadonlyREST version readonlyrest-1.16.33_es5.6.8.zip

In both environments, we have this same configuration…

- name: "Basic R/W
  type: allow
  auth_key: userA:secret
  verbosity: error

Curious, did any behavior change in the new version that we should be aware? Do we need to add anything specific in our configuration for the new version (i.e. “indices:data/write/reindex”)?

3

@mkdavis could you please show us ES log related to the request?

4

here’s unsuccessful _reindex request. Looks like it can’t find USR properly…

image
image2350×662 697 KB

I can provide a successful _search request (if necessary)…

Here’s the _reindex command…
{
“source”: {
“remote”: {
“host”: “http://production-server.com:9200”,
“username”: “xxxx”,
“password”: “xxxx”
},
“index”: “oim-2019.06.23”,
“type”: “Offering”,
“query”: {
“bool”: {
“must”: [
{ “terms”: { “pid” : [“666004P”, “777602A”, “888603A”, “888634A”, “9995001”] }}
]
}
}
},
“dest”: {
“index”: “mark-tst-reindex-2020.10.01”
}
}

5

I think you didn’t pass Authorization header, so there are no credentials and that’s why rule auth_key in Basic R/W wasn’t match.

6

The authorization header is being passed the same whether _search or _reindex.
The _search, _indices, _aliases command(s), etc. works fine!
I believe the issue may be similar towhead was expressed here… failed-ldap-authentication-does-not-capture-user-id

7

Do you use ROR kibana plugin? The request above looks like a sent from postman.

8

You are correct. We are using postman. Don’t have Kibana installed.
Here’s the plugin we’re using… readonlyrest-1.23.0_es7.5.0.zip

9

So, if Kibana doesn’t add basic auth header to the request above, we cannot do much with this problem. But there is a Free version of ROR Kibana plugin - I think you can try it. The issue won’t be present there for sure. Download (UNIVERSAL) - ReadonlyREST

But please use 1.23.1.

10

And if you call ES API directly (using curl or postman), you have to add basic auth header.

11

@coutoPL, I really appreciate all your feedback so far, thanks!
We could certainly try to install this Kibana plugin, I believe we have added the Basic Auth header using

reindex - basic auth
reindex - basic auth2756×1704 486 KB
curl or postman.

12

There is sth suspicious:

  1. here you show postman response with 404 returned by ES (so ROR allowed the request to be passed)
  2. previously you’ve shown that ROR forbid the request, so you should have seen 401 (or 403)

Please double check if all information is correct. To help you I need:

  • curl request
  • curl response
  • ROR ES log related to the request
  • ROR config (or at least block which is supposed to be matched)
13

Index not found error is from prod server from where you are trying to source the data and not an error from dev server. You need to look at the prod server log and corresponding ACL blocks.

14

Hi, I work with Mark,

On production we have index and data, prod elasticsearch version is 5.6.8 and its compatible readonlyrest version. We are planning to upgrade it, on development server we have installed elasticsearch version 7.5 with readonlyrest-1.23.0_es7.5.0.

reindexing from prod to dev is working fine without readonlyrest but get above error what Mark mentioned when we enabled readonlyrest on development server.

15

I see that you are using 5.6.8 in production which has index and type support. Where as 7.x does not have support for type. If you look at your error message more closely, its showing URI as /offering/… without the index name because of which you are getting a 404 index not found from production. I really dont know how will this cross version reindexing work. I would suggest that you play around with the source Index/type values passed in the reindex script.

Typically, if you aren’t able to reload data completely, then upgrade path will involve upgrade to v6.8 first and then only upgrade to 7.X. Please refer to official ES upgrade guide.

16

Is this re-indexing issue happening due to as we have different readonlyrest versions on production and development servers ?

17

It has got more to do with you having different versions of ES (one with support for type in production and one without support for type in dev).

18

Initially, I too suspected that it may be related to the different versions. However, it makes me pause b/c if we turn off ROR in our DEV server, the _reindex works just fine. So, it seems Elasticsearch still allows the ability to reference index/type in the “remote” for _reindex.

19

Good point. In that case, some one on ROR team may need to check this and try to recreate the issue on their end now that at least the scenario is clear - request on prod is not receiving index name and only receiving type (refer to URI on your error screenshot).

20

I agree @askids. It does seem to point to ROR issue…

sscarduzio, Here’s what the ES v5.6 production index looks like…
It has type=Offering

{
    "oim-2019.06.23": {
        "mappings": {
            "Offering": {
                "properties": {
                }
            }
        }
    }
}

here’s the _reindex command…

{
  "source": {
    "remote": {
      "host": "http://production-server.com:9200", 
      "username": "xxxx",    
      "password": "xxxx"
    },
    "index": "oim-2019.06.23",
      "type": "Offering",
    "query": {
      "bool": {
        "must": [
 	      { "terms": { "productidentifier" : ["666004P”, “777602A”, “888603A”, “888634A”, “9995001"] }}
        ]
      }
    }
  },
  "dest": {
    "index": "mark-tst-reindex-2020.10.01"
  } 
}

here’s the error message…

{
    "error": {
        "root_cause": [
            {
                "type": "status_exception",
                "reason": "body={\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"Offering\",\"index_uuid\":\"_na_\",\"index\":\"Offering\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"Offering\",\"index_uuid\":\"_na_\",\"index\":\"Offering\"},\"status\":404}"
            }
        ],
        "type": "status_exception",
        "reason": "body={\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"Offering\",\"index_uuid\":\"_na_\",\"index\":\"Offering\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"Offering\",\"index_uuid\":\"_na_\",\"index\":\"Offering\"},\"status\":404}",
        "caused_by": {
            "type": "response_exception",
            "reason": "method [POST], host [http://production-server.com:9200], URI [/Offering/_search?scroll=5m&sort=_doc%3Aasc&size=1000&version=false], status line [HTTP/1.1 404 Not Found]\n{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"Offering\",\"index_uuid\":\"_na_\",\"index\":\"Offering\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"Offering\",\"index_uuid\":\"_na_\",\"index\":\"Offering\"},\"status\":404}"
        }
    },
    "status": 404
}
21

Hello,
we’ve fixed this remote reindex issue. It hasn’t been released as stable yet, but you can test development version. Here’s the link to the ROR plugin for ES 7.5.0 https://readonlyrest-data.s3.eu-west-1.amazonaws.com/build/1.28.0-pre2/readonlyrest-1.28.0-pre2_es7.5.0.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO5QS2CXRMG%2F20210218%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210218T165444Z&X-Amz-Expires=604800&X-Amz-Signature=921780a820f116790d76b08a551f5661619e0f9415db3be86eb93efc86f111c9&X-Amz-SignedHeaders=host
Please let me know if you need link to other version of ES.