[SUPPORT|kbn_ent] SHI International SAS (2)

Enterprise Support
1

Support request

ROR Version: 1.58.0

Kibana Version: 8.14.3

Elasticsearch Version: 8.14.3

Expected result: Elasticsearch cluster successfully deployed.

Actual Result:
After updating the ROR version from 1.56.0 (Kibana + Elasticsearch 8.7.1) to ROR version 1.58.0 (Kibana + Elasticsearch 8.14.3), I encountered the following error: container has runAsNonRoot and image has non-numeric user (elasticsearch)

I modified the Dockerfile that builds the Elasticsearch image by changing the following part:

FROM docker.elastic.co/elasticsearch/elasticsearch:8.14.3

COPY readonlyrest-1.58.0_es8.14.3.zip ror-es.zip
RUN bin/elasticsearch-plugin install file:///usr/share/elasticsearch/ror-es.zip --batch
RUN rm ror-es.zip

USER root
RUN jdk/bin/java -jar plugins/readonlyrest/ror-tools.jar patch
USER elasticsearch

To:

FROM docker.elastic.co/elasticsearch/elasticsearch:8.14.3

COPY readonlyrest-1.58.0_es8.14.3.zip ror-es.zip
RUN bin/elasticsearch-plugin install file:///usr/share/elasticsearch/ror-es.zip --batch
RUN rm ror-es.zip

USER root
RUN jdk/bin/java -jar plugins/readonlyrest/ror-tools.jar patch
USER 1000

also, in the podTemplate of elasticsearch.yaml i have added:
securityContext:
runAsUser: 1000

The error disappeared, and my Elasticsearch pods started running but then kept crashing, restarting, and so on.

The error is as follows:
[2024-07-22T11:19:37,428][ERROR][o.e.b.Elasticsearch ] [elasticsearch-dev-es-hot-1] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: failed to obtain node locks, tried [/usr/share/elasticsearch/data]; maybe these locations are not writable or multiple nodes were started on the same data path?
at org.elasticsearch.server@8.14.3/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:293)
at org.elasticsearch.server@8.14.3/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:509)
at org.elasticsearch.server@8.14.3/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:256)
at org.elasticsearch.server@8.14.3/org.elasticsearch.node.Node.(Node.java:192)
at org.elasticsearch.server@8.14.3/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:240)
at org.elasticsearch.server@8.14.3/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240)
at org.elasticsearch.server@8.14.3/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)
Caused by: java.io.IOException: failed to obtain lock on /usr/share/elasticsearch/data
at org.elasticsearch.server@8.14.3/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:238)
at org.elasticsearch.server@8.14.3/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:206)
at org.elasticsearch.server@8.14.3/org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:285)
… 6 more
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/node.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixPath.toRealPath(UnixPath.java:886)
at org.apache.lucene.core@9.10.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:94)
at org.apache.lucene.core@9.10.0/org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43)
at org.apache.lucene.core@9.10.0/org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44)
at org.elasticsearch.server@8.14.3/org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:231)
… 8 more
Suppressed: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/node.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)
at java.base/java.nio.file.Files.newByteChannel(Files.java:379)
at java.base/java.nio.file.Files.createFile(Files.java:657)
at org.apache.lucene.core@9.10.0/org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:84)
… 11 more
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch-dev.log

ERROR: Elasticsearch died while starting up, with exit code 1

THank you for your help!

{“customer_id”: “ec266e44-6350-4c72-a4e7-b0b5d05dacc7”, “subscription_id”: “d23ec4db-5b92-4128-9754-f11e52867f29”}

2

@rkaci It seems that this issue is not related to ROR per se. Now, your ES is started as a user 1000 (previously it was elasticsearch). It seems that the new user cannot access files created by the previous one. A quick solution will be changing the owner of the ES data folder to 1000.

3

Thank you for your feedback.

When I created the new Elasticsearch cluster, it was from scratch. I deleted the one created with the user elasticsearch. Here are the steps I followed:

  1. Deleted the old Elasticsearch cluster.
  2. Created a new image with user 1000.
  3. Created a new Elasticsearch cluster with this new image.
4

Hello,

The issue was about resources (CPU and Memory) which were not enough. I’ve scaled them up and now it’s working fine.

Thank you for your support.

5

thanks for the confirmation