We are planning to upgrade elk from v 7.17.2 to v 8.6.2 and i came across some weird behavior.
First of all, on an empty created cluster, for each tenant there is a .kibana index created (e.g .kibana_admin for admin users etc). What is strange is that all these indices are created with a replica 0. I am not 100% sure this is a ROR related issue or not, but in the documentation it is stated that:
“When a tenant logs in for the first time, ReadonlyREST Enterprise will create the kibana index associated to the tenancy as per ACL.”
Second, due to the fact that Kibana doesn’t support multiple kibana indices anymore, it seems quite difficult to make this upgrade in a more automated way. I was wondering if you’re planning to include this in a feature ROR release?
Honestly, i tried re-creating the old indices with the new 8.6.2 mapping and settings, reindexing and creating the corresponding aliases, but it does not work, maybe i am missing an additional step in this process.
I have to mention that we use ROR Enterprise 1.48.0 for all these tests.
You should be able to do the old technique of changing the kibana.index to trick Kibana into migrating the old tenancy indices.
ROR indeed creates (if not exists) the kibana index of a tenancy every time a user actively logs in (or manually switches to) a tenancy. But not before.
If you are on a test env, try wiping all the tenancy indices, put ROR in debug mode, and try collecting logs of when you start kibana up.
Thank you for your fast reply.
So you mean i can add the kibana.index in kibana,yml and kibana wil start and perform the migration for that index? But i still have to add all of them individually in order to perform all the migrations, following this procedure:
For the tenancy part, yes you are correct , the behavior you described is normal. What i find an issue here is the fact that when each .kibana index is created, it is created with replica 0. Which was not the case before, it always had a replica 1.
Yes we need to come up with some procedure to automate the mass migration so customers don’t have to change kibana.index and restart kibana one tenancy at the time!
@Dzuming can you check if there is a smart way for us to invoke Kibana migration programmatically ? Maybe a mix of patching and ror tool code?
@Dzuming can you please also check if the API we invoke elasticsearch client to create a new kibana index is current with 8.x? See the replica number issue Diana mentions above.
Btw Diana, are you now doing your experiment with 8.x on ECK?
It would be great if it could be done in a programmatic way indeed! But also, if you figured out what happens behind when kibana starts to do the migration and you could share this, it would be awesome.
Related to the ECK experiment, yes, i started testing the ROR configuration, i think my colleague Octav already updated you about the progress.
@Dzuming can you check if there is a smart way for us to invoke Kibana migration programmatically ? Maybe a mix of patching and ror tool code?
Sure, I will take a look
can you please also check if the API we invoke elasticsearch client to create a new kibana index is current with 8.x? See the replica number issue Diana mentions above.
From kibana 8.6.0 we need to provide number_of_shards and number_of_replicas during tenant index creation, but since it’s a generic logic also for kibana 7.x. The number_of_replicas value is set symmetric to the kibana index, which is 0, but it broke the previous behavior because the default value was set by kibana automatically to 1 during the new index creation. I already made a pr with a fix and will provide a new pre-release build, for now as a quick fix, you can change the number of the replicas manually
Hi @Diana
We prepared a fix for the default number_of_replicas issue, so now, the initial value will be 1 again
Also, we added automatic migration of the tenant index. The migration will run when you log in to the Kibana or tenant switch. I will send you a pre-release build in a private message.
, thanks a lot for this!