Unpatch failing for ES 8.18.1 / ROR 1.64.2

abuising · May 20, 2025, 2:44pm

Hi, we get the following error when unpatching the Elasticsearch plugin (via Ansible) (from 8.15.1 / 1.59.0):

WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by tech.beshu.ror.tools.scala.runtime.LazyVals$ (file:/usr/share/elasticsearch/plugins/readonlyrest/ror-tools.jar)
WARNING: Please consider reporting this to the maintainers of class tech.beshu.ror.tools.scala.runtime.LazyVals$
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/readonlyrest/patch_backup/elasticsearch-8.18.1.jar
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystem.copy(UnixFileSystem.java:954)
    at java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:280)
    at java.base/java.nio.file.Files.copy(Files.java:1189)
    at tech.beshu.ror.tools.os.copy$.copyOne$1(FileOps.scala:200)
    at tech.beshu.ror.tools.os.copy$.apply(FileOps.scala:204)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.RorPluginDirectory.restore(RorPluginDirectory.scala:49)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.FileModifiersBasedPatch.restore(FilePatch.scala:46)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.MultiFilePatch.restore$$anonfun$1(FilePatch.scala:61)
    at tech.beshu.ror.tools.scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at tech.beshu.ror.tools.scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at tech.beshu.ror.tools.scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
    at tech.beshu.ror.tools.scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
    at tech.beshu.ror.tools.scala.collection.AbstractIterable.foreach(Iterable.scala:933)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.MultiFilePatch.restore(FilePatch.scala:61)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.base.TransportNetty4AwareEsPatch.restore(TransportNetty4AwareEsPatch.scala:67)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.EsPatchLoggingDecorator.restore$$anonfun$1(EsPatchLoggingDecorator.scala:39)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.EsPatchLoggingDecorator.$anonfun$adapted$1(EsPatchLoggingDecorator.scala:39)
    at tech.beshu.ror.tools.scala.util.Try$.apply(Try.scala:210)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.EsPatchLoggingDecorator.restore(EsPatchLoggingDecorator.scala:39)
    at tech.beshu.ror.tools.tech.beshu.ror.tools.core.actions$UnpatchAction.execute(actions.scala:39)
    at tech.beshu.ror.tools.RorToolsApp$.main$$anonfun$1$$anonfun$1(RorToolsApp.scala:42)
    at tech.beshu.ror.tools.RorToolsApp$.main$$anonfun$1$$anonfun$adapted$1(RorToolsApp.scala:47)
    at tech.beshu.ror.tools.scala.util.Try$.apply(Try.scala:210)
    at tech.beshu.ror.tools.RorToolsApp$.main$$anonfun$1(RorToolsApp.scala:47)
    at tech.beshu.ror.tools.scala.Option.foreach(Option.scala:437)
    at tech.beshu.ror.tools.RorToolsApp$.main(RorToolsApp.scala:54)
    at tech.beshu.ror.tools.RorToolsApp.main(RorToolsApp.scala)

When we tried to get the message a bit more clear (from the commandline), we get the following:

~# /usr/share/elasticsearch/jdk/bin/java -jar /usr/share/elasticsearch/plugins/readonlyrest/ror-tools.jar unpatch
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by tech.beshu.ror.tools.scala.runtime.LazyVals$ (file:/usr/share/elasticsearch/plugins/readonlyrest/ror-tools.jar)
WARNING: Please consider reporting this to the maintainers of class tech.beshu.ror.tools.scala.runtime.LazyVals$
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
Checking if Elasticsearch is patched ...
java.lang.IllegalStateException:
ES Corrupted! Something went wrong during patching/unpatching and the current state of ES installation is corrupted.
To recover from this state, please uninstall ReadonlyREST plugin and copy the corrupted files from ES binaries (https://www.elastic.co/downloads/elasticsearch):
- /usr/share/elasticsearch/lib/elasticsearch-8.18.1.jar
- /usr/share/elasticsearch/modules/x-pack-core/x-pack-core-8.18.1.jar
- /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.18.1.jar
- /usr/share/elasticsearch/modules/x-pack-ilm/x-pack-ilm-8.18.1.jar

        at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.base.TransportNetty4AwareEsPatch.isPatched(TransportNetty4AwareEsPatch.scala:50)
        at tech.beshu.ror.tools.tech.beshu.ror.tools.core.patches.internal.EsPatchLoggingDecorator.isPatched(EsPatchLoggingDecorator.scala:29)
        at tech.beshu.ror.tools.tech.beshu.ror.tools.core.actions$UnpatchAction.execute(actions.scala:38)
        at tech.beshu.ror.tools.RorToolsApp$.main$$anonfun$1$$anonfun$1(RorToolsApp.scala:42)
        at tech.beshu.ror.tools.RorToolsApp$.main$$anonfun$1$$anonfun$adapted$1(RorToolsApp.scala:47)
        at tech.beshu.ror.tools.scala.util.Try$.apply(Try.scala:210)
        at tech.beshu.ror.tools.RorToolsApp$.main$$anonfun$1(RorToolsApp.scala:47)
        at tech.beshu.ror.tools.scala.Option.foreach(Option.scala:437)
        at tech.beshu.ror.tools.RorToolsApp$.main(RorToolsApp.scala:54)
        at tech.beshu.ror.tools.RorToolsApp.main(RorToolsApp.scala)
UNEXPECTED ERROR: ()

Did something change or is something broken?

Thanks & regards,
Arjen Buising

michalg · May 20, 2025, 6:00pm

Hello, thanks for the report.

Let’s confirm the ES version that you currently use. The ror-tools detect, that the currently installed Elasticsearch version is 8.18.1, but the report says that it is 8.15.1. Which ES version is currently installed?

The likely explanation is that the patch was applied on some older version of Elasticsearch (perhaps 8.15.1). The ES was then updated to 8.18.1, but without unpatching before the upgrade, and patching again using the correct ROR version after the upgrade. If that is the case, then in order to recover please follow the instructions from the error message:

To recover from this state, please uninstall ReadonlyREST plugin and copy the corrupted files from ES binaries (https://www.elastic.co/downloads/elasticsearch):
- /usr/share/elasticsearch/lib/elasticsearch-8.18.1.jar
- /usr/share/elasticsearch/modules/x-pack-core/x-pack-core-8.18.1.jar
- /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.18.1.jar
- /usr/share/elasticsearch/modules/x-pack-ilm/x-pack-ilm-8.18.1.jar

If that is not the case, then please confirm the details about the ES version that is currently installed, and please share the list of files in the /usr/share/elasticsearch/plugins/readonlyrest/patch_backup directory

Regards
Michał

abuising · May 21, 2025, 6:49am

Hi @michalg,

We were testing the upgrade on a new cluster from 8.15.1 to 8.18.1 and it looks like the Ansible playbook upgraded Elasticsearch first; tried to restart it (since it’s not production, auto-restart was enabled), which failed since the ReadonlyREST plugin didn’t match the ES version. Then tried to unpatch ReadonlyREST from a patched version which doesn’t exist anymore…
I expect this won’t happen on production since we are not auto-restarting ES in the meantime, so it will continue running on the old binary, which matched the ReadonlyREST plugin version. Only after the playbook runs, we restart ES.

We will try above solution to see if it recovers from this situation.

Thanks & regards,
Arjen

michalg · May 21, 2025, 5:39pm

Hello,

Ok, great to hear that we found the underlying cause of the issue. Please let us know if the solution works or if there’s anything else we can help with regarding this problem.

On our end, we’ve recently improved ror-tools patch/unpatch/verify operations to better detect and more clearly inform users about this kind of situation. The improvements were introduced in the newest ROR version 1.64. After the upgrade to the newest ROR is complete, you may consider using the ror-tools verify command as part of your workflow. It will help detect problems before the ES is restarted.

Regards,
Michał

abuising · May 22, 2025, 7:51am

Hi @michalg,

We managed to do a rollback to the previous version since ES already had starting to make 8.18.1 indices which weren’t compatible anymore so we had to re-index indices to a node with the old version one by one, which worked. A secondary attempt to upgrading WITHOUT the auto-restart option, solved it.

Thanks for the hints!

We also saw that we needed the additional “I_UNDERSTAND_AND_ACCEPT_ES_PATCHING” parameter to our automated workflow.

P.S. verify was already part of our workflow and it’s awseome!

Regards, Arjen