[SUPPORT|kbn_pro] ECK Support

Support request

We cannot upgrade ECK past version 2.1.0 because then ReadOnlyRest no longer works. Newer versions of ECK don’t use passwords but ServiceAccount tokens for authentication.

However, that version of ECK is outdated and cannot scale the cluster down on Kubernetes version 1.25. This is because it uses an obsolete version of PodDisruptionBudget.

We need a new version of RoR that works with newer ECK versions. I have heard this exists, but that documentation is not available. We require that documentation, or we cannot perform necessary operations on our clusters.

Please help advise on what our options are.

ROR Version:

Kibana Version:

Elasticsearch Version:

Steps to reproduce the issue

Expected result:

Actual Result:

{“customer_id”: “65fc7dd8-6b38-478e-922b-2f65938c6079”, “subscription_id”: “4c690a6a-9a7b-4fea-973b-690baee6d4d5”}

Hi @kevinchevalier

As you noticed, we don’t have ROR-ECK integration described in our documentation. We are working on it and it should be done this year :wink:

Nevertheless, we have a working example that contains:

  • ECK 2.10.0 (the latest version at the moment)
  • ES & KBN 8.11.1 (the latest version at the moment)
  • ROR 1.53.0 (the latest version at the moment)

So, I think you should be able to use ROR.

The example is based on Docker and Kind. You can find it here.

If you have difficulties understanding what is in the example or porting it to your K8S cluster, you can ask here and we will help you.

Thanks Mateusz,

We had a couple things we couldn’t figure out. Maybe you know the answer.

  1. Do we need to enable xpack security on both Kibana and Elasticsearch, or just one?

  2. Should we enable xpack after running ror-tools.jar patch, or does the order not matter?

  3. In the example, KIBANA_SERVICE_ACCOUNT_TOKEN is set on es.yml (https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/blob/improvement/RORDEV-1055/eck/kind-cluster/ror/es.yml#L32) but seems to only be used on ror-initial-config.yml (https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/blob/improvement/RORDEV-1055/eck/kind-cluster/ror/ror-initial-config.yml#L10). Should it be on kbn.yml instead, or on both?

  1. ES only
  2. it doesn’t matter. But you have to restart ES after the ROR patching step.
  3. in es.yml file we extract kibana user token to KIBANA_SERVICE_ACCOUNT_TOKEN variable and we use it in ROR initial settings. AFAIK ECK prepares elasticsearch.yml and kibana.yml for you and you just enrich them (e.g. like this). The kibana service token is already defined in the kibana.yml. You just need to extract it and use it in ROR settings.

Thanks Mateusz,

We have enabled xpack security on both the master and data nodes. Unfortunately, when the first data node restarts, it is unable to find the master node and never rejoins the cluster. Here are some of the logs we are seeing:

Cannot get source of document [.readonlyrest ID=1]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

address [x.x.x.x:9300], node [null], requesting [false] connection failed: [][y.y.y.y:9300] general node connection failure: handshake failed because connection reset

Do you have any suggestions about what could be causing this issue?

Hello Mateusz,

We were able to get our cluster working with ROR 1.53.0/ECK 2.8.0 with leaving xpack disabled. We believe that enabling xpack on our existing cluster would require a full cluster restart based on the node behavior we observed earlier and the instructions from Elastic on enabling xpack: Set up minimal security for Elasticsearch | Elasticsearch Guide [7.13] | Elastic. We would prefer not to do this to avoid cluster downtime.

We would like to better understand why xpack is specifically noted as a requirement. Is there any reason we should not move forward without xpack enabled?

Thank you for all of your assistance.

Yes, you are right - it would require a restart of the cluster.

TBH, as you and some other users proved, ROR can work with ECK with xpack.security.enabled: false, but at the moment we use some deprecated ES API (e.g. for networking). It looks like everything Elastic does is aimed at disallowing this option to be turned off in the future (it may happen in ES 9.x). Another downside is that, in the case of ECK it’s easier to configure the stack when the option is enabled.

All of that forced us to prepare for the future right now. That’s why we treat xpack.security.enabled: true as our default and we recommend users to follow official docs and use it in the same way. But power users can take advantage of the fact that (at least until Elastic decides otherwise) the xpack security can be disabled. In ES 8.x ROR supports it. In 9.x? We cannot say it’ll be possible (I mean we have tools to workaround any Elastic limitations, but we don’t know how many of our users will be insisting on leaving the xpack.security disabled for some reasons).

Oh, and there are many benefits of having xpack.security enabled. E.g. in the near future, ROR will support e.g. Elastic Agents thanks to that.

Great! Thank you for the information Mateusz!

No problem. If you have a problem with ROR in the future, do not hesitate to ask here in the forum. For sure we will try to help.