1.69.1_es9.4.2 Unable to create new tenancy?

1

Hi,

We are testing 1.69.1_es9.4.2 (Enterprise).

Part of our test is “Can we create a new tenancy”.

If we do user is logged out directly.

We can’t find any clear logs in either ROR audit, Kibana or Elasticsearch to indicate what could go wrong.

We see the new tenant indices being made.

But user is not coming passed login screen.

Relevant blocks of config:

  - name: "Read-Write access with RoR deleteme kibana index"
    #indices: [".kibana_ror_deleteme", ".kibana_ror_deleteme_reporting", "elasticsearch*"]
    kibana:
      index: ".kibana_ror_deleteme"
      access: rw
      #hide_apps: ["Analytics|Overview", "Analytics|Canvas", "Analytics|Maps", "Elasticsearch", "Observability", "Security", "readonlyrest_kbn", "Management", "Enterprise Search", "ROR Manage Kibana"]
    groups: ["RW_ror_deleteme"]    


  - username: testuser_rw_deleteme
    auth_key: testuser_rw_deleteme:XXX
    groups: ["RW_ror_deleteme"]

Could you please also test on your side and give us pointers on what kind of additional logs/info you require?

2

@abuising FYI

For our internal tracking purposes :slight_smile:

3

Hi @ronald.vanboven

We are about to release ROR 1.70.0. We fix one problem related to tenancies.
Could you please test the ROR 1.70.0 pre-builds?

ROR ES 1.70.0-pre10 for ES 9.4.2
ROR KBN 1.70.0-pre16 for KBN 9.4.2

4

In our test environment we also use container:

beshultd/elasticsearch-readonlyrest:9.4.2-ror-1.69.1

Do you have pre-build for that as well? Then we can test and let you know.

5

Yes, but it’s a separate DockerHub repo (see the “-dev” suffix):

6

Thanks @abuising will test and report back here with results.

1 Like
7

We now get a more clear error message.

We are running kibana 9.4.2.

Elasticsearch 9.4.2.

GET /_cat/aliases/.kibana?v&expand_wildcards=all
alias   index              filter routing.index routing.search is_write_index
.kibana .kibana_8.15.1_001 -      -             -              -

The error message we get when trying to login with the user that should create a new tenancy is:

{"@timestamp":"2026-06-03T17:28:05.651+02:00","log":{"level":"INFO","logger":"plugins.ReadonlyREST.authController"},"message":"Could not login in: unhandled server error: Kibana 8.15.1 deployment detected. Please upgrade to Kibana 8.18.0 or newer before upgrading to 9.x series.","x-ror-correlation-id":"xxx"}

Which is strange because we ran a 8.18.x version and 9.1.x before the situation we are in now.

If we look at the .kibana_8.15.1_001 we see:

GET /.kibana/_settings
{
  ".kibana_8.15.1_001": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "limit": "1500"
          }
        },
        "refresh_interval": "1s",
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".kibana_8.15.1_001",
        "creation_date": "1744884446868",
        "priority": "10",
        "number_of_replicas": "1",
        "uuid": "xuoPXonkT2Kin3qim5ylvQ",
        "version": {
          "created": "8512000"
        }
      }
    }
  }
}

Where the creation date epoch points to UTC:

Thursday, April 17, 2025 at 10:07:26.868 AM

Which indicates something has been going wrong for longer…

8

Hello @ronald.vanboven

Which is strange because we ran a 8.18.x version and 9.1.x before the situation we are in now.

Based on the error during Elasticsearch startup, there should be an upgrade to 8.19.x version first, so there must be some differences between Kibana / Es here, since the log from Kibana you showed is about 8.18.x

[2026-06-05T06:40:54,437][ERROR][o.e.b.Elasticsearch      ] [n1_it] fatal exception while booting Elasticsearchjava.lang.IllegalStateException: cannot upgrade a node from version [8.15.1] directly to version [9.4.2], upgrade to version [8.19.0] first.
	at org.elasticsearch.server@9.4.2/org.elasticsearch.env.NodeMetadata.verifyUpgradeToCurrentVersion(NodeMetadata.java:122)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:135)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:635)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:333)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:561)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:322)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.node.Node.<init>(Node.java:190)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.bootstrap.Elasticsearch$1.<init>(Elasticsearch.java:436)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:436)
	at org.elasticsearch.server@9.4.2/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:101)

2026-06-05 06:40:54,788 INFO exited: elasticsearch (exit status 1; not expected)

But based on the index version .kibana_8.15.1_001 seems like this index was never migrated.

  1. Are you able to show Kibana startup logs during Kibana index migration?

It should be something like

[2026-06-05T06:49:27.045+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2026-06-05T06:49:27.097+00:00][INFO ][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 30ms.
[2026-06-05T06:49:27.105+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 37ms.
  1. Are there any errors in the Elasticsearch or Kibana logs during startup?
  2. Could you check the indices of existing, working tenants? Are there also have _8.15.1 suffix?
9

Hi @Dzuming

Migration logs during startup from question 1. :

root@kibana:/var/log/kibana# systemctl restart kibana && tail -Fn40 /var/log/kibana/kibana.log | grep ‘savedobjects-service’

{“service”:{“version”:“9.4.2”,“type”:“kibana”,“state”:“initializing”,“node”:{“roles”:[“background_tasks”,“ui”]},“id”:“WCRKGfRjQF2-gaxkgA0bHw”},“ecs”:{“version”:“9.3.0”},“@timestamptimestamptimestamptimestamp”:“2026-06-05T09:23:57.493+02:00”,“message”:“Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations…”,“log”:{“level”:“INFO”,“logger”:“savedobjects-service”},“process”:{“pid”:2309607,“uptime”:44.256018671},“trace”:{“id”:“4954675c18f78c0899ee27923045238c”},“transaction”:{“id”:“fc0885eee5f4caa5”}}

{“service”:{“version”:“9.4.2”,“type”:“kibana”,“state”:“initializing”,“node”:{“roles”:[“background_tasks”,“ui”]},“id”:“WCRKGfRjQF2-gaxkgA0bHw”},@timestamp.3.0ecs”@timestamp{“version”:“@timestamp.3.0”},“@timestamp”:“2026-06-05T09:23:57.494+02:00”,“message”:“Starting saved objects migrations”,“log”:{“level”:“INFO”,“logger”:“savedobjects-service”},“process”:{“pid”:2309607,“uptime”:44.256562741},“trace”:{“id”:“4954675c18f78c0899ee27923045238c”},“transaction”:{“id”:“fc0885eee5f4caa5”}}

{“service”:{“version”:“9.4.2”,“type”:“kibana”,“state”:“initializing”,“node”:{“roles”:[“background_tasks”,“ui”]},“id”:“WCRKGfRjQF2-gaxkgA0bH@timestamp”},“ecs”:{“v@timestamprsion”:“9.3.0”},“@timestamp”:“2026-06-05T09:23:57.728+02:00”,“message”:“Completed all migrations in 234ms”,“log”:{“level”:“INFO”,“logger”:“savedobjects-service”},“process”:{“pid”:2309607,“uptime”:44.490534172},“trace”:{“id”:“4954675c18f78c0899ee27923045238c”},“transaction”:{“id”:“fc0885eee5f4caa5”}}
10

Hi @Dzuming,

These are the only ERRORS:

{"service":{"version":"9.4.2","type":"kibana","state":"available","node":{"roles":["background_tasks","ui"]},"id":"WCRKGfRjQF2-gaxkgA0bHw"},"ecs":{"version":"9.3.0"},"@timestamp":"2026-06-05T09:23:13.063+02:00","message":"Failed to add API keys to APM package policies","log":{"level":"ERROR","logger":"plugins.apm"},"process":{"pid":2108003,"uptime":81581.942828358},"trace":{"id":"adc977786eb8d657fa3ade9213550ee7"},"transaction":{"id":"91d53135362ccd3a"}}
{"service":{"version":"9.4.2","type":"kibana","state":"available","node":{"roles":["background_tasks","ui"]},"id":"WCRKGfRjQF2-gaxkgA0bHw"},"ecs":{"version":"9.3.0"},"@timestamp":"2026-06-05T09:23:13.063+02:00","message":"no elements in sequence","error":{"message":"no elements in sequence","type":"EmptyError","stack_trace":"Error: \n    at _super (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/createErrorClass.js:7:26)\n    at new EmptyErrorImpl (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/EmptyError.js:7:9)\n    at Object.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/lastValueFrom.js:24:28)\n    at ConsumerObserver.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:138:33)\n    at SafeSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at SafeSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:104:39\n    at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)\n    at ReplaySubject.Subject.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:98:24)\n    at Object.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:80:30)\n    at ConsumerObserver.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:138:33)\n    at SafeSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at SafeSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at OperatorSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at checkComplete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/mergeInternals.js:14:24)\n    at OperatorSubscriber.onFinalize (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/mergeInternals.js:48:21)\n    at OperatorSubscriber.unsubscribe (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:73:90)\n    at OperatorSubscriber._this._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:62:26)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:104:39\n    at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)\n    at Subject.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:98:24)\n    at Object.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/share.js:80:30)\n    at ConsumerObserver.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:138:33)\n    at SafeSubscriber.Subscriber._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:92:30)\n    at SafeSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/exhaustMap.js:28:37\n    at OperatorSubscriber._this._complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:56:21)\n    at OperatorSubscriber.Subscriber.complete (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:69:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/takeUntil.js:10:141\n    at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)\n    at OperatorSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subscriber.js:51:18)\n    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:69:34\n    at Object.errorContext (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)\n    at ReplaySubject.Subject.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/Subject.js:59:24)\n    at ReplaySubject.next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/ReplaySubject.js:45:31)\n    at LicensingPlugin.stop (/usr/share/kibana/node_modules/@kbn/licensing-plugin/server/plugin.js:116:16)\n    at PluginWrapper.stop (/usr/share/kibana/node_modules/@kbn/core-plugins-server-internal/src/plugin.js:139:218)\n    at /usr/share/kibana/node_modules/@kbn/core-plugins-server-internal/src/plugins_system.js:192:29\n    at processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Promise.allSettled (index 175)"},"log":{"level":"ERROR","logger":"plugins.apm"},"process":{"pid":2108003,"uptime":81581.943070799},"trace":{"id":"adc977786eb8d657fa3ade9213550ee7"},"transaction":{"id":"91d53135362ccd3a"}}

It doesn’t look like it’s related to this problem, but not sure.

11

HI @Dzuming,

It looks like most .kibana indices have 8.15.1 in the name, some (new), like .kibana_ror_custom_search_solution are 9.1.5.
and some custom .kibana indices, which are 8.18.1, but none are the targeted version, 9.4.2.

12

Hello @abuising

I identified a problem with the creation of tenancy-based indexes for a new tenancy, prepared a pre-release build, and sent it as a private message. The build contains logic to automatically repair failed tenancies.

The problem was with the way Kibana makes a migration for indexes in a new version, so instead of creating a Kibana index per version, they keep the old name if there were no relevant changes in an index mapping. That’s why you can see .kibana_8.15.1_001even when migration was correctly applied.

13

Hi @Dzuming ,

Tested with the newly provided pre-release build and now it works!

Thanks & regards,
Arjen

14

Excellent!
Thanks all.

For my planning/timelines what is the expected release date for this patch?

15

We will release a ROR 1.70.1 by the end of the week.

1 Like
16

@ronald.vanboven ROR 1.70.1 with this fixed is released!

1 Like