Rolling Upgrade Fails When Upgrading Elasticsearch (8.11 → 8.17.8) with ReadonlyREST Plugin

Users Feedback
1

Hi all,

We’re in the process of upgrading our Elasticsearch cluster from v8.11 to v8.17.8, and we’re also upgrading the ReadonlyREST (ROR) plugin to ensure compatibility with the newer ES version.

We need to perform a rolling upgrade to avoid any downtime, but we’ve run into issues with every approach we’ve tried so far. Here’s a quick summary of what we’ve attempted:

  1. Dockerfile-based installation: * We copy the ROR plugin .zip and install it during the image build. However, the rolling upgrade fails — all pods are terminated and recreated at once, instead of one at a time.
  2. InitContainers + emptyDir volume: Tried to inject the ROR plugin at runtime before ES starts. Pods never become ready. When port-forwarding, ES asks for credentials (looks like the plugin isn’t initializing correctly).
  3. ECK plugins field (official method): Used the spec.nodeSets[].podTemplate.spec.initContainers and plugins: field in the ECK manifest. Same issue — pods stay unready, and accessing the node requires unknown credentials.

This is what we have for readonlyrest.yml:

coordinator:
    readonlyrest.yml: |
      readonlyrest:
        enable: true
        prompt_for_basic_auth: false
        response_if_req_forbidden: Forbidden!
        access_control_rules:
          - name: "::HEALTH::"
            type: allow
            verbosity: error
            actions: ['cluster:monitor/*','indices:monitor/*']
          - name: "::READ::"
            type: allow
            verbosity: error
            actions: ['indices:data/read/*']
          - name: "::SYNONYMS-MANAGEMENT::"
            type: allow
            verbosity: error
            actions: ['cluster:admin/synonyms/*','cluster:admin/synonym_rules/*']
          - name: "::BASIC_AUTH::"
            type: allow
            verbosity: error
            auth_key_sha256: AUTH_KEY_TO_BE_REPLACED_AT_RUN_TIME
          - name: "::PROBE::"
            verbosity: error
            auth_key: "elastic-internal-probe:${INTERNAL_PROBE_PASS}"
          - name: "::ELASTIC-INTERNAL::"
            verbosity: error
            auth_key: "elastic-internal:${INTERNAL_USR_PASS}"

Ask:
Has anyone successfully performed a rolling upgrade of Elasticsearch (8.11+) with ROR plugin with aforementioned approaches or any other approach?
Any known workarounds or recommended best practices would be greatly appreciated!

Thanks in advance!

2

Hi @Smritibhandari91

Could you share the logs?

3

Hi @Dzuming,

There aren’t any errors in the logs, but we keep seeing this readiness warning:

Warning  Unhealthy  Readiness probe failed: Waiting for Elasticsearch cluster to become ready (request params: "wait_for_status=yellow&timeout=1s")
Cluster is not yet ready (request params: "wait_for_status=yellow&timeout=1s")
4

Thanks for the log.

Could you try our official Docker image as an installation method? Here is the description of how to use it

1 Like
5

Thank you! Would 8.17.4 image work for ES version 8.17.8? I don’t see any image for 8.17.8 in the repo.

6

Thank you! Would 8.17.4 image work for ES version 8.17.8? I don’t see any image for 8.17.8 in the repo.

The above link is for development-only Docker images

Here are Docker images for the 8.17.8 version for ReadonlyREST ES plugin and ReadonlyREST Kbn plugin.