Issue uploading large payload

1

Hello!

We just migrated from ES,KB 7.6.2 to 7.12.1 and also ROR enterprise from 1.28.0 for 7.6.2 to 1.36.0 for 7.12.1.
We noticed that when trying to upload through kibana a larger payload, we receive a 413 Payload Too Large error which did not happen in the old version.
For example, I am trying to create an index template from kibana dev tools and the body has ~5000 lines.
Errors in log:

Mar 02 17:36:46 localhost kibana[1383]: PayloadTooLargeError: request entity too large
Mar 02 17:36:46 localhost kibana[1383]:     at readStream (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/body-parser/node_modules/raw-body/index.js:155:17)
Mar 02 17:36:46 localhost kibana[1383]:     at getRawBody (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/body-parser/node_modules/raw-body/index.js:108:12)
Mar 02 17:36:46 localhost kibana[1383]:     at read (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/body-parser/lib/read.js:77:3)
Mar 02 17:36:46 localhost kibana[1383]:     at jsonParser (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/body-parser/lib/types/json.js:135:5)
Mar 02 17:36:46 localhost kibana[1383]:     at /opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:6034
Mar 02 17:36:46 localhost kibana[1383]:     at Layer.handle [as handle_request] (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/layer.js:95:5)
Mar 02 17:36:46 localhost kibana[1383]:     at trim_prefix (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:317:13)
Mar 02 17:36:46 localhost kibana[1383]:     at /opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:284:7
Mar 02 17:36:46 localhost kibana[1383]:     at Function.process_params (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:335:12)
Mar 02 17:36:46 localhost kibana[1383]:     at next (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:275:10)
Mar 02 17:36:46 localhost kibana[1383]:     at /opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:5873
Mar 02 17:36:46 localhost kibana[1383]:     at Layer.handle [as handle_request] (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/layer.js:95:5)
Mar 02 17:36:46 localhost kibana[1383]:     at trim_prefix (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:317:13)
Mar 02 17:36:46 localhost kibana[1383]:     at /opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:284:7
Mar 02 17:36:46 localhost kibana[1383]:     at Function.process_params (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:335:12)
Mar 02 17:36:46 localhost kibana[1383]:     at next (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express/lib/router/index.js:275:10)

Increasing server.maxPayloadBytes in kibana.yml doesn’t seem to fix this issue.

Is there any option we could use in kibana.yml to fix this? Or any other solution?

Thanks!

2

Hello @Diana. For reference, how big is this payload?

3

Hello,

Related to the size, as you can see in the print-screen is not very large:

image
image776×154 7.47 KB

But, still, it’s a json of ~5000 lines.
I did not encounter this issue in the previous version and also, if i do the request directly to the ES cluster, i don’t have any issues.

Thanks,
Diana

4

Hi Diana,
Yes it’s tiny. We have a very large upper limit configured by default in our pre-kibana proxy (the one crashing). The most useful thing you could do for us in order to accelerate the investigation is to upgrade to the latest ROR version 1.38.0 and reproduce the stack trace you sent, so we can trace better the code.

5

Hello,
This does not seem to be happening in 1.38 so i guess we will do the upgrade to this version. Thanks!

1 Like
6

Hello!

Seems to be happening with this new version also, but this time is tested with larger payloads, 4.2 Mb.
As for the logs,seems to be the same message, i will add it here anyway:

Mar 11 13:55:49 localhost kibana[23735]: PayloadTooLargeError: request entity too large
Mar 11 13:55:49 localhost kibana[23735]:     at readStream (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/raw-body/index.js:155:17)
Mar 11 13:55:49 localhost kibana[23735]:     at executor (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/raw-body/index.js:112:5)
Mar 11 13:55:49 localhost kibana[23735]:     at new Promise (<anonymous>)
Mar 11 13:55:49 localhost kibana[23735]:     at getRawBody (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/raw-body/index.js:111:10)
Mar 11 13:55:49 localhost kibana[23735]:     at maybeParseBody (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express-http-proxy/lib/requestOptions.js:96:14)
Mar 11 13:55:49 localhost kibana[23735]:     at Object.bodyContent (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express-http-proxy/lib/requestOptions.js:104:12)
Mar 11 13:55:49 localhost kibana[23735]:     at buildProxyReq (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express-http-proxy/app/steps/buildProxyReq.js:12:84)
Mar 11 13:56:31 localhost kibana[23735]: PayloadTooLargeError: request entity too large
Mar 11 13:56:31 localhost kibana[23735]:     at readStream (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/raw-body/index.js:155:17)
Mar 11 13:56:31 localhost kibana[23735]:     at executor (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/raw-body/index.js:112:5)
Mar 11 13:56:31 localhost kibana[23735]:     at new Promise (<anonymous>)
Mar 11 13:56:31 localhost kibana[23735]:     at getRawBody (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/raw-body/index.js:111:10)
Mar 11 13:56:31 localhost kibana[23735]:     at maybeParseBody (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express-http-proxy/lib/requestOptions.js:96:14)
Mar 11 13:56:31 localhost kibana[23735]:     at Object.bodyContent (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express-http-proxy/lib/requestOptions.js:104:12)
Mar 11 13:56:31 localhost kibana[23735]:     at buildProxyReq (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/plugins/readonlyrestkbn/node_modules/express-http-proxy/app/steps/buildProxyReq.js:12:84)
7

@Dzuming can you have a look?

8

Sure, will take a look

1 Like
9

@Diana the fix is in place. Once the PR is reviewed we can create a pre build for you.

10

Thanks! Will this be included in a final release? We have to push this to a production env and we are not not very comfortable in deploying pre-releases.

11

Yes of course, 1.39.0 will have this and will be released later this week.