Hi everyone,
after upgrading Elasticsearch from 9.5.1 to 9.5.3, we are seeing severe problems with bulk requests when ReadonlyREST 1.71.0 is enabled.
Environment:
- Elasticsearch: 9.5.3
- ReadonlyREST: 1.71.0
- Logstash: 8.19.x
- ReadonlyREST plugin reports Elasticsearch version: 9.5.3
- HTTPS enabled
After upgrading I see a lot of these lines in the logstash logs:
[2026-09-21T23:56:05,724][WARN ][logstash.outputs.elasticsearch][native_stderr][c2f7b54217d06f733b795c4f8e1237fe31cf6c8b165689409030576b51e86fc8] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch
Unreachable: [https://my-elastic9.my.domain:9200/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout] Read timed out {:url=>https://logstash:xxxxxx@my-elastic9.my.domain:9200/, :error_message=>"Elasticsearch Unreachable: [https://my-elastic9.my.domain:9200/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2026-09-21T23:56:05,724][ERROR][logstash.outputs.elasticsearch][native_stderr][c2f7b54217d06f733b795c4f8e1237fe31cf6c8b165689409030576b51e86fc8] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://my-elastic9.my.domain:9200/_bulk?filter_path=errors,items.*.error,items.*.status][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>64}
The problem is reproducible with a direct curl request, so it is not specific to Logstash.
A small bulk request (100 events) completes in roughly 80–90 ms sent directly via curl.
A bulk request containing 500 events (~848 KB NDJSON) frequently hangs indefinitely. For comparison: On another cluster running Elasticsearch 9.5.1, a bulk request with 5000 events completes in roughly 1 second.
While the 500-event request is hanging, ss on the Elasticsearch node shows the HTTP connection as:
ESTAB 611618 0 ...:9200 ...:<client-port>
The Recv-Q remains unchanged at ~612 KB for an extended period.
If the client (curl) is aborted, the connection changes to:
CLOSE-WAIT 611619 0 ...:9200 ...:<client-port>
and remains in CLOSE-WAIT even after waiting.
This also affects Elasticsearch shutdown: the shutdown hangs with messages such as:
still waiting on 346 client connections to close
Indexing pressure does not appear to be the cause. We checked _nodes/stats and saw zero indexing-pressure rejections, and the queues were empty.
Has anyone seen this behavior with ReadonlyREST 1.71.0 on Elasticsearch 9.5.3? Could this be related to HTTP request handling or bulk request buffering in the ROR layer?
I am currently testing with ReadonlyREST disabled to determine whether the problem disappears without ROR.
Thanks!