Incompatibility with ECK (official elastic operator for Kubernetes deployment)

Installing the readonlyrest plugin and running a default cluster using the official elasticsearch operator (ECK) doesn’t work, as requests made by the operator are rejected by ROR.

Asking around in the elastic forums I got to know that the operator uses users from the file realm, namely “elastic-internal”, “elastic-internal-keystore”, and “elastic-internal-probe” (see here)

This is a complete blocker for us as I couldn’t find any way in ROR to allow requests from particular file realm users. (Also, we love the awesome support for JWT based auth, and would really like to continue using it)

Any help on this would be greatly appreciated

it culd be useful to see the log lines from Elasticsearch where it says “FORBIDDEN”. We can then reconstruct what ACL rule we should write in readonlyrest.yml

Hi Simone,

In my limited testing I saw a few actions being blocked:

FORBIDDEN by default req={ ID:1928268510-304095439#105189, TYP:SyncedFlushRequest, CGR:N/A, USR:[user not logged], BRS:true, KDX:null, ACT:indices:admin/synced_flush, OA:10.121.81.104/32, XFF:null, DA:10.121.90.58/32, IDX:<N/A>, MET:POST, PTH:/_flush/synced, CNT:<N/A>, HDR:Accept-Encoding=gzip, Authorization=, Content-Length=0, Content-Type=application/json; charset=utf-8, Host=es9k4e591r7-es-http.es9k4e591r7.svc:9200, User-Agent=Go-http-client/1.1, HIS:[Basic Auth credentials-> RULES:[auth_key_sha256->false], RESOLVED:[]], [Valid JWT viewer-> RULES:[jwt_auth->false], RESOLVED:[]], [Valid JWT writer-> RULES:[jwt_auth->false], RESOLVED:[]], [Liveliness and readiness probes-> RULES:[actions->false], RESOLVED:[]], [Cluster update settings-> RULES:[actions->false], RESOLVED:[]], [Get Kibana Mappings-> RULES:[actions->false], RESOLVED:[]], [Master voting-> RULES:[actions->false], RESOLVED:[]], [Internal requests-> RULES:[actions->false], RESOLVED:[]] }

The few actions from the operator that I observed were:

  1. cluster:monitor/state (this I believe is due to the Readiness Probes of the cluster pods)
  2. cluster:admin/voting_config/add_exclusions, cluster:admin/voting_config/clear_exclusions
  3. indices:admin/synced_flush

There might be many more, as the operator may be doing some other actions in other scenarios. Is creating ACL blocks by looking through the logs the correct way forward here?

In further discussions, it seems the operator creates a set of users through which it interacts with the cluster. It creates the user in a file realm using the values from two kubernetes secrets, which it creates automatically as part of the cluster formation.

Is there any possible way to import those users in ROR? Or if you have a cleaner approach in your mind then it would be even more helpful.

The “file realm” stores username and sha256 hashed passwords in the “users” file, which is a flat file with the syntax:

user1:pass1hash
user2:pass2hash

So with some glue code, you should fetch those users and add them in the readonlyrest.yml like so:

- name: "user 1"
  auth_key_sha256: "user1:hashedpass2"
- name: "user 2"
  auth_key_sha256: "user2:hashedpass2"

If I understand correctly, the “auth_key_sha256” field stores the “username:password” hashed using sha256. How do I un-hash the password in the file, as I would need the plain-text password the make the “username:password” hash, correct?

I remember I made a fix such that all the auth_key* rules, try to compare both versions with plain-text username, and if fails it tries with hashed username and password.

So if you configure, either:

auth_key_sha256: cleartextUsername:encryptedPassword
auth_key_sha256: encryptedUsernameAndPassword

Thank you very much for this! I would have to inject a script before the startup of elasticsearch to do this user management.

Though I wish there would be better integration for Elastic’s official way of deploying on kubernetes (I wish not to modify the default image of Elasticsearch, as it would mean an additional thing to maintain)

We could actually implement this, if this, let’s see if the approach actually works!

Hi Simone,

I tried doing it the way you suggested; I wrote a piece of code which would go and fetch user:pass from the “config/users” file realm and put them in readonlyrest,yml. This worked perfectly as expected.

In even further discussion, I learnt about how the operator currently generates and uses the credentials for the internal users it creates.

The way I chose to go forward with was creating the secrets: [cluster]-es-elastic-user and [cluster]-es-internal-users with my own generated passwords for the 4 users the operator was using. This was done before creating the Elasticsearch cluster itself, so when operator found that the secrets already exist, it just used the existing ones without creating them again. And then I just put those username-password combinations in readonlyrest.yml using environment variable substitution.

This way I avoided putting any glue code inside the elasticsearch startup script. So, I don’t have to maintain a version of elasticsearch image which contains the above code.

Could we have a proposal for ROR to be able to run OOTB with the official elastic operator? I would leave it to you to decide which way you would want to implement this.

2 Likes

Great job! Yeah we could load that user file when we bring up the ACL. How about ROR reads the users file after ES startup? By then, ES should have written the user file, right?

Yes, the operator prepares the filesystem before running the main ES container. So, all the users are already there in the user file at the time of startup of ES.

Any possibility of having this included in v1.18.10? (Maybe we could have a config parameter which would enable/disable populating readonlyrest.yml from the elasticsearch/config/users file)

Yes we definitely could implement this, but 1.18.10 merge window is closed now, we will come back with a pre build for you after the release.

1.18.10 is already released :slight_smile:

1 Like

I’m definitely watching this thread :heart:

1 Like

Hey @barryib, did you decide to manage your own kubernetes cluster in the end? Rather than getting the managed ES from AWS?

We’ve tested the AWS ES in production for one of our small service, but so far we aren’t happy with it. So we’re going to tests ECK with RoR in the next couple of weeks. We’ll decide after those tests which cluster we’ll shutdown.

1 Like

Any updates on when OOTB compatability with ECK might be released? Running on kubernetes without the Elastic Operator is a bit like bread without butter, and we’d really like to retain RoR as part of our stack.

This is targeted for 1.19.3, we are about to release 1.19.2. We will share with you a pre-build you can test as soon as we have it!