Index pattern not working after upgrade to 7.8.1

askids · September 24, 2020, 5:15am

hi,

I recently upgrade to 7.8.1 standard version of ES and Kibana with 1.23.0.

Even when I am logged in as admin and try to create a new index pattern, I am not able to see any of the indices.

Similarly, when I try to see any of existing visulatization and dashboards, none of them are showing data.

I looked at the audit logs. All these queries are using index pattern with async_search behind the scene (something like abc*/_async_search). For all these, ROR is logging Index not found error.

In Dev tools, I also tried querying abc*/_search and got 0 match. But if I specify exact index name abcde_202009/_search, its returning the data. Similar behaviour is seen, with using _async_search from Dev tools. Using exact index name works, but using index pattern does not.

So I think that support for wild card pattern is broken in ROR for 7.8.x. Request you to kindly look into this on priority as this is now holding up the entire stack upgrade.

Thanks!

coutoPL · September 24, 2020, 1:34pm

we have integrated ROR with _async_search recently and we have tests for it. So, for sure this is not trivial to reproduce. Could you please share ROR ES logs and maybe ROR settings?

askids · September 24, 2020, 2:42pm

Like I mentioned, its not specific to async search. I am facing this issue even with generic abc*/_search. Even with abc*/_count, I get 0 count. But when I use it with exact index name, it works fine.

I am going to give it a try with ROR Kibana plugin installed and see if that resolves the issue.

coutoPL · September 24, 2020, 2:52pm

Nevertheless, when I have:

ROR ES logs (I can see the request, matched rule and collected data)
ROR settings

the reproduction is much more simple than trying to find the case which we didn’t cover in our integration tests.

askids · September 24, 2020, 7:22pm

I was able to partially get past it by installing the ROR Kibana plugin. I was still facing the issue with 1.23.0. So I downgraded to 1.22.1 for Kibana, but ROR ES is still on 1.23.0.

After installing the Kibana ROR plugin, I am now able to see the indices for creating index pattern and the visualizations are working as well. Still continuing to test it. Will keep you posted.

Unfortunately, they have blocked logging into Github on client network. So it will be sometime before I can get you any log details

askids · September 25, 2020, 12:48pm

hi @coutoPL,

Did some further trials. In 2 lanes(dev and sit), which are running 2 nodes servers, with ROR Kibana plugin installed, the issue seems to be resolved. When I try it without the ROR Kibana plugin, I can see from the audit logs that the request does not have any user info. So Its unable to resolve it. So looks like the old problem where kibana does not sent authorization header on some requests. So on these 2 lanes, its all working fine.

However, on a different lane, which has a 6 ES nodes and 3 Kibana nodes with load balancer URL, I am not able to get past login page. In the ES logs, when I login, I can see the message as ALLOWED and its using the proper ACL blocks to resolve the login, but I am still stuck on login page.

After I enter id/pwd, the URL gets updated to https://myurl.com/login?nextUrl=/spaces/space_selector. In the Kibana logs, I can see some 302 messages for /spaces/space_selector. But in ES logs, dont see any FORBIDDEN messages. Its only having ALLOWED entries. Not sure how to resolve this.

I tried uninstalling ROR kibana plugin, stopped kibana, deleted contents of optimize folder and reinstalled ROR kibana plugin. But I am unable to get past the login page. I tried with both 1.22.1 and 1.23.0. I am running 7.8.1 ES and Kibana.

Please let me know what additionally can be done to troubleshoot this issue.

Thanks!

coutoPL · September 25, 2020, 6:13pm

ok, so if it’s rather related to Kibana part of ROR, maybe @sscarduzio will advise sth?

askids · September 25, 2020, 6:37pm

I uninstalled ROR kbn 1.22.1, ran cleanup on optimize folder and installed 1.23.0. Now I am starting to see 500 error after entering id/pwd. I will enable debug for ROR kbn and see what exactly is happening on server where its working vs where its failing. Will keep you posted.

askids · September 26, 2020, 9:26pm

@sscarduzio in ROR kibana 1.23.0, I identified the issue as to why i was getting 500 error. In IdentityManager.js, in function buildIdentityFromPayload, variable kibanaIndex is set as constants.defaultKibanaIndex. It should be set to constants.defaultKibanaIndex(). Please get it corrected.

Coming back to my login issue. So I went back to version 1.22.1 ROR kbn. After enabling debug mode for ROR kbn, i see that requests are are bouncing between different servers and its not able to find the session id. If I try to login with a single server name, login works fine. But when I try to use the load balancer URL, i am unable to login as its not able to find the session.

I have following entries in kibana.yml on all 3 nodes where kibana is running. Please let me know what else do I need to get ror kibana working under a load balancer.

readonlyrest_kbn.whitelistedPaths: [".*/api/status$"]
readonlyrest_kbn.cookiePass: "somelongkey"
readonlyrest_kbn.store_sessions_in_index: true
readonlyrest_kbn.clearSessionOnEvents: ["never"]
readonlyrest_kbn.session_timeout_minutes: 60

Thanks!

sscarduzio · September 28, 2020, 4:59pm

Hi @askids, I’m sending you a build with the fix in a private message here in the forum.

askids · October 1, 2020, 9:51pm

hi @sscarduzio

The 500 error on login is now gone with v1.23.1. But I am still stuck on the login page because of the load balancer. If I use single machine based URL, it works fine. But when I use load balancer URL, can’t get past it. Any suggestions?

Thanks!

askids · October 7, 2020, 10:17am

@sscarduzio any pointers to resolve the load balancer issue for Kibana login?

sscarduzio · October 7, 2020, 2:03pm

Your settings look ok to me, we’d need t test this. But this task is in queue.

askids · November 1, 2020, 5:53pm

Any update on the kibana load balancer issue?

sscarduzio · November 2, 2020, 1:29pm

@askids I’m on this now. Can you show us the debug logs of the two kibana instances?

i.e. add to kibana.yml

readonlyrest_kbn.logLevel: debug

askids · November 3, 2020, 6:30pm

I will need to figure out a way to get you the logs Dropped you a PM on it.

BTW, its a 3 Kibana instance setup behind the load balancer.

sscarduzio · December 20, 2020, 4:03pm

I just took the time to test this out in this. It works on my latest ROR Enterprise 1.26.0-pre1_es7.8.1.

log [15:53:42.155] [info][readonlyrest_kbn:IndexBasedSessionManager(DBG)] Session does not exist in memory, need to use sessions fetched from ES
log [15:53:42.156] [info][readonlyrest_kbn:IndexBasedSessionManager(DBG)] Session from ES index: {“x-ror-username”:“admin”,“x-ror-current-group”:“ROR (admin)”,“x-ror-available-groups”:[“ROR (admin)”,“Infosec”],“x-ror-kibana_access”:“unrestricted”,“kibanaIndex”:“.kibana”,“username”:“admin”,“authHeaders”:“”,“hiddenApps”:,“kibanaAccess”:“unrestricted”,“groupsAvailable”:[“ROR (admin)”,“Infosec”],“groupCurrent”:“ROR (admin)”,“sid”:“635f22a0-a305-42ec-99e8-cacbe7485661”,“expiresAt”:1608738822131}

askids · December 21, 2020, 7:28am

hi @sscarduzio,

Are you trying this with a 3 node setup for KIbana (my ES cluster has a 6 node setup)? In my case, the sid is being created on 1st kibana node, where as subsequent check for sid is happening in 2nd kibana node.

What I am thinking is that since ES is only near real time and not real time, the index updates wont be available for search until refresh is completed. So when you perform the read on 2nd node, within the same sub-second, its logical to not get a matching sid from ROR sessions index. I think that you may need to add refresh=true on your indexing request when storing the session id on index.

Thanks!

sscarduzio · December 22, 2020, 2:22pm

OK this might explain. I prepared a build for you to test.