Updating ELK stack from 8.15.5 to 8.19.16 with RoR 1.70.2. Elasticsearch start up fails due to older RoR version (not the reason) or corrupted ES. In the message it states that “backup catalog is present, but there is no metadata file. However, when i check /usr/share/elasticsearch/plugin/readonlyrest/patch_backup/ there is a file called patch_metadata. I’ve been playing around with the permissions and made sure to set it to elasticsearch:elasticsearch and 640 but still getting issue. That file is populated and states what was patched.
Hi @stevepaz . Do you confirm you followed the ES upgrade procedure? For Elasticsearch | ReadonlyREST
Yes, i tested both upgrading ES + RoR, and starting from a fresh install of ES + RoR and both give me the same issue. When I verify the patch, it outputs no errors but when trying to start elasticsearch, it still gives the error I mentioned above.
Do you mean that you install fresh ES 8.19.16, then install ROR, patch it and start ES right? And you see the problem mentioned in the first post?
Please, tell us more about your OS, installation path and the ES source (zip, apt, docker image, ect)
when testing fresh I create my own docker image fresh with ES 8.19.16 (tar.gz) and RoR 1.70.2 zip. In the dockerfile I untar ES, then install RoR and run the patch. When deploying image, ES fails due to the issue. When upgrading on a different system, I was installing ES with RPMs directly (no pods) with same RoR. Both systems run RHEL 9.6. RoR installs to /usr/share/elasticsearch/plugins/ and my ES & RoR ymls are in /etc/elasticsearch/
Could you please share your Dockerfile?
I can provide a snippet:
# Extract Elasticsearch tarball
ENV ES_HOME=/usr/share/elasticsearch
RUN mkdir -p $ES_HOME /var/lib/elasticsearch /var/log/elasticsearch /etc/elasticsearch
RUN chown -R elasticsearch:elasticsearch $ES_HOME /var/lib/elasticsearch /var/log/elasticsearch
RUN chmod -R 755 $ES_HOME
ARG ES_VERSION=8.19.16
ADD elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz $ES_HOME/
RUN mv $ES_HOME/elasticsearch-${ES_VERSION}/* $ES_HOME/ && rmdir $ES_HOME/elasticsearch-${ES_VERSION}
# Install ReadOnlyRest
ARG ROR_VERSION=1.70.2
COPY readonlyrest-${ROR_VERSION}_es${ES_VERSION}.zip /readonlyrest.zip
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch file:///readonlyrest.zip
RUN /usr/share/elasticsearch/bin/elasticsearch-keystore create
# Apply elasticsearch patch
RUN /usr/share/elasticsearch/jdk/bin/java -jar /usr/share/elasticsearch/plugins/readonlyrest/ror-tools.jar patch \
--I_UNDERSTAND_AND_ACCEPT_ES_PATCHING=yes
RUN /usr/share/elasticsearch/jdk/bin/java -jar /usr/share/elasticsearch/plugins/readonlyrest/ror-tools.jar verify
RUN chmod 644 /usr/share/elasticsearch/plugins/readonlyrest/*
At the end of the dockerfile, I run a separate bash script that then runs elasticsearch as elasticsearch user
Ok, thanks. Will check it on my side tomorrow
1 Like
Thank you. As mentioned before, I get same issue when updating ES RPM and upgrading/patching RoR.
I had to fix the Dockerfile a little bit (it looks like it was only part of it):
FROM ubuntu:22.04
ARG ES_VERSION=8.19.16
ARG ROR_VERSION=1.70.2
ENV ES_HOME=/usr/share/elasticsearch
ENV ES_PATH_CONF=/etc/elasticsearch
ENV ES_JAVA_OPTS="-Xms512m -Xmx512m"
ENV ADMIN_USER_PASS=admin
# Create the elasticsearch user/group and standard directories
RUN groupadd -g 1000 elasticsearch && \
useradd -u 1000 -g 1000 -d $ES_HOME -s /bin/bash elasticsearch
RUN mkdir -p $ES_HOME /var/lib/elasticsearch /var/log/elasticsearch $ES_PATH_CONF
# Extract Elasticsearch tarball
# NOTE: this is the aarch64 (Apple Silicon) build; use the x86_64 tarball/filename on an Intel host.
ADD elasticsearch-${ES_VERSION}-linux-aarch64.tar.gz $ES_HOME/
RUN mv $ES_HOME/elasticsearch-${ES_VERSION}/* $ES_HOME/ && rmdir $ES_HOME/elasticsearch-${ES_VERSION}
# Move config to the standard /etc/elasticsearch location (ES_PATH_CONF)
RUN mv $ES_HOME/config/* $ES_PATH_CONF/ && rmdir $ES_HOME/config
# Install ReadOnlyRest
COPY readonlyrest-${ROR_VERSION}_es${ES_VERSION}.zip /readonlyrest.zip
RUN $ES_HOME/bin/elasticsearch-plugin install --batch file:///readonlyrest.zip && rm /readonlyrest.zip
RUN $ES_HOME/bin/elasticsearch-keystore create
# Patch Elasticsearch so ReadonlyREST can replace its security layer
RUN $ES_HOME/jdk/bin/java -jar $ES_HOME/plugins/readonlyrest/ror-tools.jar patch \
--es-path $ES_HOME \
--I_UNDERSTAND_AND_ACCEPT_ES_PATCHING=yes
RUN $ES_HOME/jdk/bin/java -jar $ES_HOME/plugins/readonlyrest/ror-tools.jar verify --es-path $ES_HOME
RUN chmod 644 $ES_HOME/plugins/readonlyrest/*
# Config
COPY elasticsearch.yml $ES_PATH_CONF/elasticsearch.yml
COPY readonlyrest.yml $ES_PATH_CONF/readonlyrest.yml
RUN chown -R elasticsearch:elasticsearch $ES_HOME /var/lib/elasticsearch /var/log/elasticsearch $ES_PATH_CONF && \
chmod -R 755 $ES_HOME
USER elasticsearch
WORKDIR $ES_HOME
EXPOSE 9200 9300
ENTRYPOINT ["/usr/share/elasticsearch/bin/elasticsearch"]
And used these confs:
cluster.name: ror-docker-cluster
node.name: ror-node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.type: single-node
xpack.security.enabled: false
readonlyrest:
access_control_rules:
- name: "ADMIN"
type: allow
auth_key: admin:${env:ADMIN_USER_PASS}
Build a Docker image and run it like that:
docker run -ti -p 9202:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" ror-es-8.19.16
And it works.
Could you please check on your side?
My dockerfile is very similar to yours and i still receive the error when starting ES. I don’t think it’s due to docker as I see the same issue installing ES RPM. My elasticsearch and readonlyrest ymls have more settings than what you have which is why I am leaning towards that to be the issue. One thing I have in my readonlyrest.yml is ssl settings
ES Error:
[2026-07-07T18:37:19,718][INFO ][t.b.r.b.LogPluginBuildInfoMessage$] [$host] Starting ReadonlyREST plugin v1.70.2 on Elasticsearch v8.19.16
[2026-07-07T18:37:19,864][INFO ][stdout ] [$host] Checking if Elasticsearch is patched ...
[2026-07-07T18:37:20,112][ERROR][o.e.b.Elasticsearch ] [$host] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: failed to load plugin class [tech.beshu.ror.es.ReadonlyRestPlugin]
at org.elasticsearch.server@8.19.16/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:519)
at org.elasticsearch.server@8.19.16/org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:425)
at org.elasticsearch.server@8.19.16/org.elasticsearch.plugins.PluginsService.lambda$loadPluginBundles$2(PluginsService.java:217)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:186)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:214)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:153)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:176)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:632)
at org.elasticsearch.server@8.19.16/org.elasticsearch.plugins.PluginsService.loadPluginBundles(PluginsService.java:217)
at org.elasticsearch.server@8.19.16/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:92)
at org.elasticsearch.server@8.19.16/org.elasticsearch.node.NodeServiceProvider.newPluginService(NodeServiceProvider.java:57)
at org.elasticsearch.server@8.19.16/org.elasticsearch.node.NodeConstruction.createEnvironment(NodeConstruction.java:453)
at org.elasticsearch.server@8.19.16/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:271)
at org.elasticsearch.server@8.19.16/org.elasticsearch.node.Node.<init>(Node.java:201)
at org.elasticsearch.server@8.19.16/org.elasticsearch.bootstrap.Elasticsearch$1.<init>(Elasticsearch.java:434)
at org.elasticsearch.server@8.19.16/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:434)
at org.elasticsearch.server@8.19.16/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:100)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
at org.elasticsearch.server@8.19.16/org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:510)
... 21 more
Caused by: java.lang.IllegalStateException: Elasticsearch is either patched by an older version of ROR or corrupted.
- if ES has been patched using some older ROR version, then try unpatching using that older ROR version
- otherwise the ES installation is corrupted and ES must be reinstalled
Problems:
- backup catalog is present, but there is no metadata file
- file x-pack-core-8.19.16.jar was patched by ROR 1.70.2
- file x-pack-ilm-8.19.16.jar was patched by ROR 1.70.2
- file x-pack-security-8.19.16.jar was patched by ROR 1.70.2
at tech.beshu.ror.es.utils.EsPatchVerifier$.verify$$anonfun$1(EsPatchVerifier.scala:31)
at tech.beshu.ror.es.utils.EsPatchVerifier$.verify$$anonfun$adapted$1(EsPatchVerifier.scala:33)
at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:28)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:74)
at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:29)
at tech.beshu.ror.es.utils.EsPatchVerifier$.verify(EsPatchVerifier.scala:33)
at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.scala:91)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
... 24 more
Patch_backup folder:
/usr/share/elasticsearch/plugins/readonlyrest/patch_backup]# ll
total 25432
-rw-r--r-- 1 elasticsearch elasticsearch 18268800 May 25 22:13 elasticsearch-8.19.16.jar
-rw-r--r-- 1 elasticsearch elasticsearch 679010 May 25 22:12 elasticsearch-entitlement-8.19.16.jar
-rw-r----- 1 elasticsearch elasticsearch 1177 Jul 7 18:10 patch_metadata
-rw-r--r-- 1 elasticsearch elasticsearch 5006368 May 25 22:14 x-pack-core-8.19.16.jar
-rw-r--r-- 1 elasticsearch elasticsearch 198210 May 25 22:14 x-pack-ilm-8.19.16.jar
-rw-r--r-- 1 elasticsearch elasticsearch 1872371 May 25 22:14 x-pack-security-8.19.16.jar
No it’s not related to docker for sure. But docker is a great solution for each of us to work locally and have pretty the same environment. It simplifies reproduction.
Make sure you use —es-path arg when you patch and verify. If it still doesnt work please show your full Dockerfile.
What base image do you use?
We use a custom image based off of our software internally. So no matter what, our images won’t be identical.
Could you please check some common base image, eg ubuntu? I’d like to exclude the possibility that sth from the base image causes the problem.
I rebuilt with the rhel9 standard base image (ubi9) and can confirm its the same issue.
ok, I was able to reproduce the problem using your Dockerfile.
The problem is in this line:
RUN chmod 644 /usr/share/elasticsearch/plugins/readonlyrest/*
Started to work when I changed it to:
RUN chmod 755 /usr/share/elasticsearch/plugins/readonlyrest/*
RUN chown -R elasticsearch:elasticsearch $ES_PATH_CONF
ROR patch verification worked because it was called before the chmod line.
It’s always something small…What you changed (plus setting the patch_metadata to elasticsearch:elasticsearch) made it work! Thank you for all your help.!
1 Like