Upgrading containerised ELK with ROR - changelog detail and upgrade order confusion

1

Hi all!

We use KBN PRO plugin (apologies, haven’t found out how to verify our membership on the forums just yet) and our looking to upgrade our dockerised ELK stack across major versions (7.17 → 8.19.15 → 9.x) . Our plan was as follows (for the first major ELK version upgrade)

  1. Do a rolling upgrade of ES, replaceing ES containers one node at a time in our cluster with a new container containing ES 8.19.15 and the corresponding ROR pluging (am I right in understanding that for a given version of ES, there’s one and only one compatible version of ROR I can install?)
    *during and after this step, there would be a window during which older ROR KBN versions are trying to interact with newer ROR ES versions

  2. Shut down all kibana instances then bring up one node at a time with new KBN and its corresponding ROR plugin

This order (ES→KBN) is the sequence prescribed by elastic.co, but when reviewing the ROR changelogs I found this concerning entry

”:warning:Warning (KBN|ES) Internal API incompatibilities (to take advantage of rolling update capabilities, upgrade ROR KBN first)”
which seems to prescribe the opposite sequence to what elastic.co documents (ES first, then KBN).

So my questions are

  1. How would a cluster behave with the older readonlyrest_kbn_universal-1.48.0_es7.17.5 trying to interact with the newer readonlyrest_kbn_universal-1.48.0_es7.17.5?
    1. I know ROR documents well that the KBN/ES plugin versions should be aligned but they can’t be upgraded simultaneously, so how does that transitional period work? One of our 3 clusters is quite large with a lot of data so could take days to weeks to be fully upgraded.
  3. Am I misremembering some ROR documentation saying that the ES version we run constrains us to a single corresponding ROR version? I may have inferred that from the download page.
  4. If I am wrong on that, can I go ahead and upgrade ROR to the new versions (1.69.1) before upgrading ES and KBN?

Cheers!
DOBs

1 Like
2

Hi @DOBs,

Thanks for raising this thread. After my response, I will review our documentation and add the missing parts.

am I right in understanding that for a given version of ES, there’s one and only one compatible version of ROR I can install?

No. For example, the newest ROR 1.70.1 supports all ES versions since ES 6.7.0, and the KBN plugin supports versions since 7.9.0.

However, note that, for example, ROR 1.68.0 does not support ES 9.4.2, because ES 9.4.2 was released when ROR 1.69.x had already been released.

”:warning:Warning (KBN|ES) Internal API incompatibilities (to take advantage of rolling update capabilities, upgrade ROR KBN first)”

Ah, there was a mistake in the changelog, which has already been fixed. It should say: “upgrade ROR ES first”.

We follow Elastic’s requirements here.

  1. How would a cluster behave with the older readonlyrest_kbn_universal-1.48.0_es7.17.5 trying to interact with the newer readonlyrest_kbn_universal-1.48.0_es7.17.5?

I guess you are asking about one of the following cases, or maybe all of them:

  1. KBN nodes have ROR {n-version}, some ES nodes have ROR {n-version}, and one ES node has been upgraded and has ROR {n+1 version}
  2. KBN nodes have ROR {n-version}, and all ES nodes have ROR {n+1 version}
  3. Some KBN nodes have ROR {n-version}, one KBN node has ROR {n+1 version}, and all ES nodes have ROR {n+1 version}

Excluding the state where all KBN and ES nodes have aligned ROR versions, any other state should be considered transitional. You should upgrade ROR ES first, then ROR KBN.

So, ROR should support rolling upgrades in the following cases:

  1. The ES cluster is partially upgraded
  2. The ES cluster is fully upgraded, but the KBN cluster has not been upgraded yet
  3. The ES cluster is fully upgraded, and the KBN cluster is partially upgraded

The first and second cases have always been supported.
The third case is supported since ROR 1.69.0.

  1. I know ROR documents well that the KBN/ES plugin versions should be aligned but they can’t be upgraded simultaneously, so how does that transitional period work? One of our 3 clusters is quite large with a lot of data so could take days to weeks to be fully upgraded.

Referring to the above, this is not a problem.

We advise our clients to keep ROR versions aligned because some of them were upgrading only one plugin while leaving the other one unchanged, for example, ROR ES 1.68.0 with a very old ROR KBN 1.60.0.

We guarantee that ROR KBN {n-version} can talk to ROR ES {n+1 version}, but we do not guarantee that it will be able to communicate with ROR ES {n+2 version}. That is why we tell our clients: “Please make sure that after your upgrade, the ROR plugin versions are aligned.”

  1. Am I misremembering some ROR documentation saying that the ES version we run constrains us to a single corresponding ROR version? I may have inferred that from the download page.

No, no :smiley: We are proud of the fact that each version of ROR supports the newest ES/KBN version and older ones too: all 9.x, all 8.x, all 7.x for ES and KBN starting from 7.9.0, and even the ancient ES 6.7.x for the ES plugin.

  1. If I am wrong on that, can I go ahead and upgrade ROR to the new versions (1.69.1) before upgrading ES and KBN?

Yes, this is a great idea. You can upgrade ROR first, make sure that everything works with the upgraded version, and then upgrade the ES/KBN cluster.

When you upgrade ES/KBN and ROR at the same time, in case of any problems, you will not know whether it is an ES/KBN issue or an ROR-related issue. So this approach is highly recommended.

3

Excellent, glad to have that cleared up :slight_smile:

Great nuance to know, cheers!

Hmmm, that makes me think we should maybe only upgrade our ES ROR plugin one version ahead of our KBN ROR plugin, playing leap frog a little bit to get all the way up to the latest and having X rounds of upgrades, where X is approximately the number of ROR versions that we’re behind. Is that sensible?

That’s a relief to hear.

Great point. We can keep it more modular that way :smiley:

4

Yes, if you really care about rolling upgrade your cluster, this is the safest approach.

5

Hi Mateusz,

We appreciate this is unsupported and we do care about safely doing a rolling upgrade, however we are concerned about investing so much time and effort to do the 22 rounds of upgrade to get from 1.48 to 1.70.

As such, we intend to try jump the Elasticsrch straight to ROR plugin v. 1.70 while KBN is still on v. 1.48? We are considering doing this in our test env frst but could you perhaps comment on whether we would be wasting time?

So say our ES 7.17 cluister is upgraded to plugin 1.70 while our KBN 7.17 is still still on plugin version 1.48, what are the specific risks in that case? I understand that may not be entirely understood, but we would be very grateful forany light you could shed on that scenario.

Thanks again!
DOBs

6

Yes, I understand that upgrading one version at a time is not very convenient when you are about ~20 versions behind.

Because ROR is a security plugin, we assumed that users would upgrade as soon as a new version is released. Since we respect our users’ privacy, we do not collect metrics from running ROR instances, so we cannot prove or disprove this assumption.

However, maybe we should consider supporting rolling upgrades across N versions. That would be a useful improvement.

Back to your case: the target should not be 1.70.x yet, but 1.69.1. I have checked ROR ES 1.69.1 with ROR KBN 1.48.0, and it seems the following upgrade plan should be possible:

  1. First round: ROR ES 1.48.0 → ROR ES 1.69.1
  2. First round: ROR KBN 1.48.0 → ROR KBN 1.69.1
  3. Second round: ROR ES 1.69.1 → ROR ES 1.70.2
  4. Second round: ROR KBN 1.69.1 → ROR KBN 1.70.2
1 Like
7

Only one pragmatic matter remains, then. The download page only allows the user to specify their ELK stack version, not the ROR version they want. Even picking our current older version I’m being served 1.70.2. How would I work around that?

Thanks as always,
DOBs

8

Yes, the download page is intentionally created like that :wink: As I said, we want to encourage our users to upgrade ROR when the new version is added. Because of security concerns.

But there is a way, obviously, to get older versions. Use our API:

  1. the example for ROR ES
  2. the example for ROR KBN

If you find it difficult, please let me know - I will help you with it.

1 Like
9

Tyvm! Got that working with the below

https://api.beshu.tech/download/es?pluginVersion=1.69.1&esVersion=7.17.5&email=your_email_address

and

https://api.beshu.tech/download/kbn?edition=kbn_universal&pluginVersion=1.69.1&esVersion=7.17.5&email=your_email_address

And just to acknowledge, yes. We’ve been in bad habits around upgrading. We may start using the docker images readonlyrest publishes so we’re always up to date :slight_smile:

1 Like
10

A few followups on this

We upgraded one of our cluster’s from 1.48 to 1.69.1 in 3 rounds. Maybe you can explain if/where I’m going wrong.
I used
A) readonlyrest-1.69.1_es7.17.5.zip
and
B) readonlyrest_kbn_universal-1.69.1_es7.17.5.zip
The 3 “rounds” were

  1. We updated a couple of nodes’ ES ROR version.

  2. We updated the rest of the nodes’ ES ROR version.

  3. We updated each of our nodes’ KBN ROR versions in turn.

After the first two rounds, we had the same issues where we could login to Kibana but got redirected to a spaces URL with this error

We thought that upgrading the Kibana nodes would fix that but now the Kibana containers are continuously restarting with the error

{“type”:“log”,“@timestamp”:“2026-06-24T15:05:11+00:00”,“tags”:[“fatal”,“root”],“pid”:8,“message”:“Error: KibanaConfigManager is not initialized\n at Function.e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/config/KibanaConfigManager.js:1:4448)\n at Function.e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/config/KibanaConfigManager.js:1:4273)\n at e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/config/KibanaConfigManager.js:1:4976)\n at Object. (/usr/share/kibana/plugins/readonlyrestkbn/server/index.js:1:452)\n at Module._compile (node:internal/modules/cjs/loader:1103:14)\n at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)\n at Module.load (node:internal/modules/cjs/loader:981:32)\n at Function.Module._load (node:internal/modules/cjs/loader:822:12)\n at Module.require (node:internal/modules/cjs/loader:1005:19)\n at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:80:39)”}

image
image1202×906 106 KB

11

Hello @DOBs

  1. Could you confirm you performed all steps of the upgrading process, including unpatching and patching Kibana?
  2. Could you send all Kibana logs privately, with readonlyrest_kbn.logLevel: ‘trace’ in the kibana.yml?
12

Hi @Dzuming ,

As we’re running ELK containerised, Kibana and ROR are effectively installed each time we bring up the containers. Since we’re starting from a clean slate like that, we don’t unpatch/uninstall (because the new container has nothing to unpatch or uninstall) but we do freshly install ROR and patch each time the container comes up per these post Kibana installation steps of our dockerfile below. I will change the stack trace level and get back to you with what we see. The only output from our docker logs for the kibana container is in my last post.

Cheers!,

DOBs

COPY readonlyrest_kbn_universal-1.69.1_es7.17.5.zip /usr/share/kibana/readonlyrest_kbn_universal-1.69.1_es7.17.5.zip

RUN bin/kibana-plugin install file:///usr/share/kibana/readonlyrest_kbn_universal-1.69.1_es7.17.5.zip

RUN node/bin/node plugins/readonlyrestkbn/ror-tools.js patch

13

Apologies for the delay getting back, I changed logging a few months back to prepare for the ELK upgrade and I hadn’t done it right. Now I can see a bit more detail and ROR seems upset with my license. Perhaps there’s something we need to change in our license validation going from 1.48 to 1.69?

“Detected an unhandled Promise rejection: Error: LicenseService is not initialised yet: empty clusterUuid\n at i.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/core/license/LicenseService.js:1:4387)\n at i.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/core/license/LicenseService.js:1:6808)\n at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/customMiddleware.js:1:1415)\n at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)\n at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)\n at Generator.prototype. [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)\n at asyncGeneratorStep (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)\n at _next (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)\n at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:32:7\n at new Promise ()\n at e. (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:21:12)\n at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/customMiddleware.js:1:1821)\n at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:8954)\n at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)\n at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)\n at Generator.prototype. [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)”}

14

Hi,

Perhaps there’s something we need to change in our license validation going from 1.48 to 1.69?

There shouldn’t be any changes with license validation

As I see, there is a problem with getting the Cluster UUID from ES

Could you find in the logs, line Trying to get cluster UUIDAnd paste what is next in the logs?
Also, do you see any problems with the endpoint in the Elasticsearch logs? The cluster UUID is retrieved from this endpoint.

15

Hi Dzuming,

I’ll get back on those questions ASAP, but for now NB I was able to start my container partially and get into a shell so I could see this stuff

[root@elkhost1-tech config]# docker run -it --entrypoint /bin/bash [REDACTED IMAGE NAME]

kibana@ff2aa5d47bb7:~$ node/bin/node plugins/readonlyrestkbn/ror-tools.js verify

[ROR COMPAT] Received command: verify
[ROR COMPAT] Verifying the presence of ROR hooks on Kibana files..
[ROR COMPAT] Found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/authentication_service.patch
[ROR COMPAT] Verifying patched state…
[ROR COMPAT] ReadonlyREST encountered problems during patching verification: Unable to retrieve the ReadonlyREST plugin version for which the file was patched. Please ensure all steps in the upgrade guide have been followed: For Kibana | ReadonlyREST .

16

Answers to your questions below but I believe it’s something to do with our patching step. As far as I understand from the “for kibana” ROR docs, the patching step is the same until ELK 8.15 (we’re still on ELK 7.17.5)

That message wasn’t in our kibna log from what I could see

I see this in our ES log suggesting the UUID is ok. No mention of “endpoint” or “end point”

[2026-06-25T09:06:29,330][INFO ][o.e.c.c.Coordinator ] [elkhost1-tech] cluster UUID [GzhtKgaXSIekx389DfQUWA]

17

Also about the ES endpoint - I am abloe to run curl commands to that ES endpoint from the machine the containers are running on

18

This error indicates that Kibana was patched with a version before 1.61.0, because it’s the first version we introduced the ReadonlyREST plugin info to the metadata. So, the new version found that the file is patched, but there is no info about the patcher version, which is why this error occurs. Are you able to go to the

node_modules/@kbn/security-plugin/server/authentication

folder in Kibana and check whether there are some additional files related to the authentication_service.js? because looks like, for some reason, there are some artifacts related to the patching process

That message wasn’t in our kibna log from what I could see

{“type”:“log”,“@timestamp”:“2026-06-24T15:05:11+00:00”,“tags”:[“fatal”,“root”],“pid”:8,“message”:“Error: KibanaConfigManager is not initialized\n at Function.e

These two logs indicate that the ReadonlyREST Kibana plugin wasn’t patched correctly

19

Hi, I’ll followup on those if necessary but I suspect this is us missing the switch “ --I_UNDERSTAND_AND_ACCEPT_KBN_PATCHING=yes ”

I put it in one of our branches but not the one I’m working on. Surprisingly, running it as a command in a dockerFile has worked no problem for us before without that switch

20

Never plugin version requires adding this argument to the patch command, like

node/bin/node plugins/readonlyrestkbn/ror-tools.js patch --I_UNDERSTAND_AND_ACCEPT_KBN_PATCHING=yes

1.48.0 plugin version didn’t need this argument