Issue with tenancies following ES upgrade

Users Feedback
1

Hi Folks,

We recently upgraded our stack from ES/Kibana 6.8 to vs 7.6. It’s working fine in our main tenancy, but when we try to switch into any of the other tenancies we have defined in ROR, we get this error:

{“message”:“mapping set to strict, dynamic introduction of [references] within [doc] is not allowed: [strict_dynamic_mapping_exception] mapping set to strict, dynamic introduction of [references] within [doc] is not allowed”,“statusCode”:400,“error”:“Bad Request”}

We think that the upgrade process for Kibana migrated the main tenancy index and made some updates to it, but these updates were not applied to the other kibana indicies we have defined.

Just wondering if anyone has come across a solution for this issue, other than to recreate the tenancies? Thanks!

Regards,

Adrian

2

To make the migration between major versions, the easiest way to do this is the most ignorant one:

  1. download kibana 7.6 in your laptop,
  2. configure kibana.yml with a “kibana.index: < tenancy kibana index >”
  3. don’t forget also the elasticsearch.username and elasticsearch.password)
  4. run Kibana for a few seeconds
  5. now the migration script will have migrated that tenancy index like it did with the “.kibana”.

This is ugly, but for me it’s the safest at the moment.

3

Thanks Simone, that works perfectly for us. And because our setup is all built on Docker it’s even easier - we can just spin up a new Kibana container for a few seconds pointing at each tenancy to convert.

Am sure you are already aware, but just to point out that this process will create an <index>_1, <index>_2, etc with an alias referencing to most recent.That doesn’t seem to cause any issue for ROR right now but perhaps something we’ll need to watch out for in the future.

Cheers

1 Like
4

Glad it helped. Yes that is part of how Kibana folks implemented the migration. Probably for leaving a door open to a rollback. Not sure how people should go forward in case they are actually happy with the migration and delete the old. Need to research on Kibana forum…

5

@sscarduzio hi, second question in a day, apologies! Am replying on this old thread because noted we are hitting issue with tenancies again following upgrade to Kibana 7.17.5, same as covered in this thread. From a search on the site, I haven’t seen any other discussions of how to deal with .kibana indices from other tenancies, aside from this one! Is this still the best/recommended way to handle tenancies during upgrade?

6

Hi Adrian, no problem at all!

Unfortunately, at the moment, running upgrade migrations on tenancy kibana indices when you want to upgrade to a new major version of Kibana is very manual and requires an operator/tool to change kibana.index in kibana.yml and restart the Kibana process for each tenancy kibana index.

7

thanks, that’s fine, we’ll do the same as last time (and document it properly this time! :))

1 Like
8

Yes, you are very much right. Done!

9

Thanks @sscarduzio - I was referring to my own documentation, but good to have it in the main docs too for sure! :slight_smile:

1 Like