We are pretty sure it’s because of the amount of permissions these users have (we use Keycloak for login), but it is not a possible solution to remove permissions from them.
I investigated the logs (both Kibana and Elasticsearch), but I don’t see anything as a ‘Payload too large’ error there. The only logs I see are these:
[error][plugins][ReadonlyREST][samlController] Too ==large== JWT token size (size: 13295 of 8185 Bytes)! Please set "readonlyrest_kbn.logLevel: debug", look for "Created JWT for ROR with claims" in Kibana logs, and inspect for any data duplication in the JSON Object. If no data is being duplicated, this user has too many groups, or the individual group strings length are too long.
[warning][plugins][ReadonlyREST][authorizationHeaderBuilder][x-ror-correlation-id=37e73252-d350-4606-8218-c8e3e3ae81d8] Base64 encoded metadata headers length of 14415 exceeded maximum length of 8KB! Please reduce the metadata size! The sum of the following headers lengths should be less than 6000: [
'x-ror-origin: 23 is characters long',
'cookie: 232 is characters long',
'x-ror-correlation-id: 57 is characters long',
'x-ror-kibana-request-path: 54 is characters long',
'x-ror-kibana-request-method: 32 is characters long',
'x-forwarded-for: 33 is characters long',
'x-ror-kibana-index: 26 is characters long'
But they appear all the time and do not seem directly related to the users having this error. We have ROR logLevel in ‘debug’ but we have no other logs than this.
Is there any config that we can set to increase this limit for payload? I’ve checked ROR documentation but I couldn’t find any. Thanks in advance for any help you can provide.
I investigated the logs (both Kibana and Elasticsearch), but I don’t see anything as a ‘Payload too large’ error there.
This message does not originate from the ReadonlyREST plugin. we only inform the user that the HTTP header is too large and the limit is around 8 KB because that’s most browsers’ limit.
Is there any config that we can set to increase this limit for payload? I’ve checked ROR documentation but I couldn’t find any. Thanks in advance for any help you can provide.
Since we only log the information about header size, There is no way to change the limit. I’m wondering what generates the ‘Payload too large’, maybe there is some reverse proxy that rejects this request?
Did you see the Created JWT for ROR with claims log? Are there any duplicated groups here? Could you send claims as a private message? I will try to reproduce this issue on my side
OTOH: I remember fixing a bug for your company about this very issue a few years ago, @natalia.mellino. You tend to have a very large amount of roles in SAML assertions. I think @gustavo.yoshizaki will remember.
Hi! Yes, we have a workaround where users remove some of their unused permissions and they are able to login again, but now the amount of roles grew a lot in our company and we have a lot of users with many permissions complaining about this. So doing this it’s not viable anymore.
maybe there is some reverse proxy that rejects this request?
We have a nginx running in the kibana server but it does not seem to be the problem. I’ve checked its logs and the only thing I saw that could be related was this:
[error] 474139#474139: *1486070 upstream sent too big header while reading response header from upstream,
I tweaked some configs in nginx such as: large_client_header_buffers and proxy_buffers but had no success with it. Also, this log does not seem to be related with the problem because it does not appear every time users have a ‘payload too large’ error in screen, but I coment it just in case.
Did you see the Created JWT for ROR with claims log? Are there any duplicated groups here? Could you send claims as a private message?
I am pretty sure that there are not duplicated groups, can you tell me how do I get this info in order to provide it?
I would like to understand if all those roles are useful in the scope of ReadonlyREST usage, or if you would be in the position of writing a sort of regex / function to easily prune the unnecessary parts?
From what I see in the logs, when an user logs in, it brings all of the permissions attached to it, this include permissions that are not necessarily related to kibana. I’d have to check with my team, but the permissions for kibana have a prefix that we can use to filter them. What I don’t understand quite well is where would we implement this if it happens to be possible for us to do it.
Yes sorry, I found it right after I asked. I see the logs but I don’t find any of them with the usernames of the users which are having the problem. I could privately send you one of a random user so you can see how it is structured, but I don’t know if it’s useful for you.
Yes sorry, I found it right after I asked. I see the logs but I don’t find any of them with the usernames of the users which are having the problem. I could privately send you one of a random user so you can see how it is structured, but I don’t know if it’s useful for you.
That’s strange that these logs are not available, however since we know that all permissions are attached to the claims, we should be fine without needing to dig into the individual logs for the usernames, so it’s not necessary anymore.
