ROR 1.40.0 / 1.41.0 URL rewriting issue

Hello,

When upgrading our ElasticSearch/Kibana stack to 7.17.4 and ROR 1.41.0 we ran into a problem with how URLs are handled resulting in a non-functional Kibana.
I have isolated the problem to the ROR 1.40.0 and 1.41.0 Kibana plugins, they seem to be rewriting the Kibana URLs incorrectly.
My guess it is a side effect of the basepath handling fixes introduced in 1.40.0

What we’re seeing:

After login the user is redirected to https://<kibana>/s/default, which results in a 404 error from Kibana.
Changing the URL to https://<kibana>/home/app also results in a 404 but the Kibana access log shows a GET request to https://<kibana>/s/default/home/app.
In fact, all URLs get prepended with /s/default, resulting in a 404.

Authentication to ES succeeds, I can use Cerebro to browse the ES cluster no problem. Kibana also shows successful authentication.

I have also ruled out our Ingress setup (tested with multiple types and without) and Istio being the problem. Everything is pointing me towards these 2 specific versions of ROR.

Our setup is ES/Kibana 7.14.4 on Kubernetes using ECK operator 2.1.0, ROR Pro version 1.41.0

Relevant log lines (redacted and shortened):
ES:

{"type": "server", "timestamp": "2022-07-12T13:04:23,398Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "<REDACTED>", "node.name": "<REDACTED>", "message": "\u001B[36mALLOWED by { name: '::PUBLIC-RW::', policy: ALLOW, rules: [ldap_auth,indices] req={ ID:796915808-2042415205#131551, TYP:RRUserMetadataRequest, CGR:Some(<REDACTED>), USR:<REDACTED>, BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:127.0.0.1/32, XFF:null, DA:<REDACTED>/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Authorization=<OMITTED>, accept-encoding=gzip,deflate, accept=*/*, content-length=0, host=<REDACTED>:9200, user-agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), x-b3-sampled=1, x-b3-spanid=b9a2f5107467bfd5, x-b3-traceid=34334a4fb71b6b44b9a2f5107467bfd5, x-forwarded-proto=http, x-request-id=c57f3d00-94e0-9130-b878-5bb7cd4d4567, HIS:[::PUBLIC-RW::-> RULES:[ldap_auth->true, indices->true] RESOLVED:[user=<REDACTED>;group=<REDACTED>;av_groups=<REDACTED>]], }\u001B[0m", "cluster.uuid": "3XAXSVvURQGDZu75TrKpZw", "node.id": "-AjLxM1FSHOqFEpVc0KlQg" }

Kibana:

{"type":"response","@timestamp":"2022-07-12T13:04:23+00:00","tags":[],"pid":7,"method":"get","statusCode":302,"req":{"url":"/s/default/","method":"get","headers":{"host":"<REDACTED>"},"referer":"https://<REDACTED>/login?nextUrl=%2F"},"res":{"statusCode":302,"responseTime":12},"message":"GET /s/default/ 302 12ms"}
{"type":"response","@timestamp":"2022-07-12T13:04:23+00:00","tags":[],"pid":7,"method":"get","statusCode":404,"req":{"url":"/s/default","method":"get","headers":{"host":"<REDACTED>"},"referer":"https://<REDACTED>/login?nextUrl=%2F"},"res":{"statusCode":404,"responseTime":35,"contentLength":60},"message":"GET /s/default 404 35ms - 60.0B"}
{"type":"response","@timestamp":"2022-07-12T13:11:37+00:00","tags":[],"pid":7,"method":"get","statusCode":404,"req":{"url":"/s/default/app/home","method":"get","headers":{"host":"<REDACTED>"},"res":{"statusCode":404,"responseTime":29,"contentLength":60},"message":"GET /s/default/app/home 404 29ms - 60.0B"}
1 Like

Hi @twijt-nl

I don’t think I’m able to help you with Kibana, but … the interesting thing for me is the ECK integration. ROR at the moment doesn’t officially support it (we have a PoC and we proved that it can be done, but it was not released yet).

Could you please tell me how did you do that? Has it ever worked with no issues?

1 Like

Sure, It’s actually very simple to do.
Can’t share any of the Dockerfiles or yamls unfortunately.

We extend the official Kibana and Easticsearch images, install ROR the usual way and publish the new image to our own registry.
The ROR config is added through a secret and the whole thing is made environment specific using Kustomize.

This has worked like a charm for the past 7 months or so, up to and including Kibana/ES 7.17.3 with ROR 1.39.0. Updating to ROR 1.40.0+ breaks Kibana because of the URL/Routing issue.

But ECK manages one important setting: xpack.security.enabled (see Settings managed by ECK | Elastic Cloud on Kubernetes [2.3] | Elastic) and it doesn’t allow to change it. Currently, ROR requires it to be set to false. How did you overcome it? Is there a way to ignore xpack.security?

Regarding the Kibana issue, maybe @Dzuming could help.

Yeah, I’ll try to reproduce and debug this issue.

2 Likes

@coutoPL
We set xpack.security.enabled to false and readonlyrest.enabled to true using environment variables at the bottom of our extended Dockerfile like so:

ENV xpack.security.enabled=false
ENV readonlyrest.enable=true

The environment variables override anything the ECK operator does, they take precedence over everything.

@Dzuming Thanks, if any info is needed I’d be happy to supply it.

2 Likes

WOW! This is great news! We were not aware.

Obviously, it could be very dangerous to build an official solution based on this, but as a workaround, for now, it’s great!. Thanks for sharing this information :ok_hand:

Any updates on this issue?

Hi, sorry for no feedback. I’m trying to reproduce this issue, I tried several ROR plugin versions, but everything works as expected. Could you send me (it can be a private message) your elasticsearch and kibana pods eck config ?

I’ve sent the anonymised manifest as well as the minimal Dockerfiles as a PM.

2 Likes

Hi, thanks for the config, I can confirm that I was able to reproduce and fix the issue. It was the problem with not handling xpack.spaces.enabled: false properly. I will send the pre-release build soon, as a temporary workaround, you can try to remove this property from the kibana config.

1 Like

Sorry for the late reply, I was on vacation.

I’ve tested with the setting removed from the Kibana config and now it works like a charm.
I’ll see today if I can modify our build pipeline slightly so it can grab the pre-release to test that out, will get back on that today or monday.

1 Like

Unfortunately the pre-release doesn’t fix the issue for us.
Removing xpack.spaces.enabled: false from the Kibana config does still work as a workaround

I made sure to use a fresh image tag and verified the download URL so I’m 100% sure it’s using the version you provided in your PM.

Hello, did you have a chance to check it with the newest 1.43.0 ROR plugin version? I tested it on my machine now and it seems to work. instead of https://<kibana>/s/default the correct URL should be https://<kibana>

Unfortunately it doesn’t seem to work for us.
I’ve updated both the Kibana and Elastic plugins to 1.43.0 but I’m still seeing the same issue.

Below is the request/response immediately after submitting the login form.
As stated in my previous post, removing xpack.spaces.enabled from the config does still work as a workaround.

Request URL: https://<kibana>
Request Method: GET
Status Code: 302 
Remote Address: <address>:443
Referrer Policy: strict-origin-when-cross-origin
cache-control: private, no-cache, no-store, must-revalidate
content-length: 0
content-type: application/octet-stream
date: Fri, 26 Aug 2022 08:52:50 GMT
kbn-name: kibana
location: /s/default
referrer-policy: no-referrer-when-downgrade
strict-transport-security: max-age=31556926; includeSubDomains
:authority: <kibana>
:method: GET
:path: /
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,nl;q=0.8
cache-control: no-cache
cookie: rorcookie=<cookie>
pragma: no-cache
referer: https://<kibana>/login?nextUrl=%2F
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Google Chrome";v="104"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36

Any updates on this issue?
We could apply the workaround so we can at least update to the latest versions but I really don’t feel this is a long-term solution.

Again, if any further information is required I’d be happy to supply it.

Hi @twijt-nl this issue has a workaround, so a more urgent one took priority. This will be next in line.

1 Like