Kibana Cluster settings - `store_sessions_in_index` write is delayed

1

Hello everyone,

I am currently testing the (free) RoR plugin for Kibana and I am experiencing an issue with the JWT passthrough authentication.

Kibana version: Kibana 9.4.3
RoR version: 1.70.3_es9.4.3 :unicorn:

Kibana runs as 2 replicas in Kubernetes behind a standard Service, with round-robin load balancing between pods, no client stickiness.

Our Kibana config:

readonlyrest_kbn.store_sessions_in_index: true
readonlyrest_kbn.jwt_query_param: "jwt"
readonlyrest_kbn.sessions_refresh_after: 250
readonlyrest_kbn.cookiePass: "<redacted>"

The store_sessions_in_index: truesetting does not prevent cross-node auth failures; a pod’s write of a new established session to the sessions index is delayed and lazily-triggered upon accessing a resource. What that means is, when i go to my kibana[.]com deployment and hit a different node than the one responsible for the login, i get instantly redirected to the log out screen and the logs show an authentication failure:

[warning][plugins][ReadonlyREST][indexBasedSessionManager] Failed to decode session for SID <sid>: Session not found in index
[debug]  [plugins][ReadonlyREST][indexBasedSessionManager] SID <sid> not found in in-index session storage either: giving up.

However, when i navigate to kibana[.]com/app/kibana, this triggers a write to the sessions index and the session is discovered by all nodes.

Logs capturing this bug:

  • 12:16:50.498
    • session created (SID 3d1abf5a...) on node A (getTenancyId/
      authenticationFacade “User request identity for a direct request” logged with this
      SID).
  • 12:16:50.690, 12:17:44.362, 12:17:52.410
    • node B independently answers three separate requests carrying the same SID with “Session not found in index… giving up”, spanning over a minute after creation.
  • 12:18:21.318
    • first IndexRequest (POST /.readonlyrest_kbn_sessions/_doc/3d1abf5a...)
      for this SID appears in Elasticsearch’s own ROR access log. This is ~91 seconds after the session was created.
  • Node A’s own indexSessionRepository polling (Fetched all sessions from index, found: N) ran continuously every ~250-280ms for the entire gap and never incremented
    until this write landed — confirming the delay is real, not a logging artifact.

Is this intended behaviour, or a misconfiguration on my end? Is there a way to make the initial session write synchronous (or bounded to a known maximum delay), rather than opportunistic/request-triggered on the owning node?

Thank you so much!

2

Hello @dp123

Thanks for the detailed report — this is a real gap, not a misconfiguration.

Root cause: JWT passthrough sessions are created in-memory only on the handling node, not written to the sessions index at creation time. The session does eventually sync to the index — but only once a later request happens to land back on that same node; if a different node is hit first, that node finds nothing and forces a logout instead of waiting. sessions_refresh_after doesn’t control this.

Workaround now: enable session affinity/sticky sessions on your load balancer (e.g. sessionAffinity: ClientIP) so a user’s requests keep hitting the same pod until the session is persisted.

Fix: we’ll make JWT passthrough sessions write to the index synchronously at creation, same as normal login. Will follow up here once shipped.

3

Hello @dp123

I prepared a pre-release build readonlyrest_kbn_universal-1.72.0-pre1_es9.4.3.zip, with a fix for this issue. Are you able to verify it on your side?

4

Hello @Dzuming,

thank you so much for the quick response.
We have verified the functionality and I am happy to say that this bug is now fixed, as the writes are instantaneous.

Thank you!