We cannot upgrade ECK past version 2.1.0 because then ReadOnlyRest no longer works. Newer versions of ECK don’t use passwords but ServiceAccount tokens for authentication.
However, that version of ECK is outdated and cannot scale the cluster down on Kubernetes version 1.25. This is because it uses an obsolete version of PodDisruptionBudget.
We need a new version of RoR that works with newer ECK versions. I have heard this exists, but that documentation is not available. We require that documentation, or we cannot perform necessary operations on our clusters.
it doesn’t matter. But you have to restart ES after the ROR patching step.
in es.yml file we extract kibana user token to KIBANA_SERVICE_ACCOUNT_TOKEN variable and we use it in ROR initial settings. AFAIK ECK prepares elasticsearch.yml and kibana.yml for you and you just enrich them (e.g. like this). The kibana service token is already defined in the kibana.yml. You just need to extract it and use it in ROR settings.
We have enabled xpack security on both the master and data nodes. Unfortunately, when the first data node restarts, it is unable to find the master node and never rejoins the cluster. Here are some of the logs we are seeing:
Cannot get source of document [.readonlyrest ID=1]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
address [x.x.x.x:9300], node [null], requesting [false] connection failed: [][y.y.y.y:9300] general node connection failure: handshake failed because connection reset
Do you have any suggestions about what could be causing this issue?
We were able to get our cluster working with ROR 1.53.0/ECK 2.8.0 with leaving xpack disabled. We believe that enabling xpack on our existing cluster would require a full cluster restart based on the node behavior we observed earlier and the instructions from Elastic on enabling xpack: Set up minimal security for Elasticsearch | Elasticsearch Guide [7.13] | Elastic. We would prefer not to do this to avoid cluster downtime.
We would like to better understand why xpack is specifically noted as a requirement. Is there any reason we should not move forward without xpack enabled?
Yes, you are right - it would require a restart of the cluster.
TBH, as you and some other users proved, ROR can work with ECK with xpack.security.enabled: false, but at the moment we use some deprecated ES API (e.g. for networking). It looks like everything Elastic does is aimed at disallowing this option to be turned off in the future (it may happen in ES 9.x). Another downside is that, in the case of ECK it’s easier to configure the stack when the option is enabled.
All of that forced us to prepare for the future right now. That’s why we treat xpack.security.enabled: true as our default and we recommend users to follow official docs and use it in the same way. But power users can take advantage of the fact that (at least until Elastic decides otherwise) the xpack security can be disabled. In ES 8.x ROR supports it. In 9.x? We cannot say it’ll be possible (I mean we have tools to workaround any Elastic limitations, but we don’t know how many of our users will be insisting on leaving the xpack.security disabled for some reasons).
Oh, and there are many benefits of having xpack.security enabled. E.g. in the near future, ROR will support e.g. Elastic Agents thanks to that.
