Hello,
Since we updated the elastic stack to 7.12.1 with the RoR 1.30.1 plugin, the kibana import / export module no longer works.
We systematically encounter this error below. It doesn’t matter how many objects are imported and the source version of kibana.
Only solution for us is to rollback to es: 7.8.1 ror: 1.28.2.
We tested an elastic 7.12.1 stack without RoR plugin the import / export works normally.
Yes I reproduced this, and know why it happens. Will fix next week. But yes I recommend to postpone the migration to 7.12 while we take care of a few bugs including this one. Thanks for your patience @erms77.
Hello,
I have the same issue; I’ll keep an eye on this topic so. @sscarduzio if needed, I can test on my infrastructure (K8s based).
I had a look at this over the weekend. it’s not extra obvious like I though. But we have no problem reproducing it, so I’m positive.
Will update this thread.
Hello @sscarduzio
Member of the Kibana Core team here, having the SO import/export under our team’s scope.
Would it be possible to share some detail on the cause of this problem? Having 3rd party plugins being able to break some core features is an issue, and I would like to try to understand how this was made possible.
Hi @pgayvallet, thanks for chipping in. My greetings to you and all the Kibana team 
As far as I can see at this stage, Kibana’s Hapi server is coming back with this message:
{“statusCode”:400,“error”:“Bad Request”,“message”:“Invalid multipart payload format”}
The client request looks like this instead:
POST /api/saved_objects/_import?overwrite=true HTTP/1.1
Host: localhost:5601
Connection: keep-alive
Content-Length: 475
sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36
kbn-version: 7.12.0
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBulpOiG00tj1BaLU
Accept: */*
Origin: http://localhost:5601
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:5601/app/management/kibana/objects
Accept-Encoding: gzip, deflate, br
Accept-Language: it-GB,it;q=0.9,en-GB;q=0.8,en-US;q=0.7,en;q=0.6
Cookie: rorSessionCookie=Fe26.2**2921ad09cab8ea65d278e63bc6f7a35ebcad5e39e122de61b00ab7f41e1bd353*1o0s_sRwD3JIXwfJzqouCw*qFAEWaWersH_VFCvYm9VD0V2V-jvtPHS6OVPm_ySzwPUqe5esousFbFazQh67SR7**b9f5e148728ae97451890f74cf59bea3e2d84eee7393ee2ad8185602cecb16a3*ayCfDrCtmUNpmljuClg9cCKYSGD7zGv6Fbvd2i7nAnQ; rorCookie=Fe26.2**f4f1b61759b2f3e24298627aaef60200da6765a923e42495348fec02013574fa*ee0Gd9l-ui9ud2JhLXKJ4w*LFaiWppQARgEq09mZw8e5XyKhyEQd0vhyecsy8j9rZraWVS-QNp4PiCuvKFxu4yZ**34c1c7258dd48d8e1bc28e5b9e0c182f315faf8096f18c0243f38684e78ffc2f*YxnfXcuWcsMM6HAxOXw37yl-JhQbrzHwsMVJXVZVDuI; rorCookie-SP={%22createdAtMillis%22:1624034003146%2C%22intervalMillis%22:5000}
------WebKitFormBoundaryBulpOiG00tj1BaLU
Content-Disposition: form-data; name="file"; filename="export.ndjson"
Content-Type: application/octet-stream
{"attributes":{"buildNum":39309},"coreMigrationVersion":"7.12.0","id":"7.12.0","migrationVersion":{"config":"7.12.0"},"references":[],"type":"config","updated_at":"2021-06-18T16:25:29.434Z","version":"WzEsMV0="}
{"exportedCount":1,"missingRefCount":0,"missingReferences":[]}
------WebKitFormBoundaryBulpOiG00tj1BaLU--
Hello ! I’ve done some packet capture on my pc, and I’ve seen some interesting things. I’ll reach you through private message (If my understanding is right, even a vanilla kibana may be impacted by the issue).
Pierre
Thanks to @pchesneau the issue was found and fixed. It was completely unrelated to Kibana because this plugin uses an internal proxy (responsible of the poor handling of the multipart POST request).
So @pgayvallet you don’t have to worry
As soon as my PR gets merged by ROR dev team, I will be able to create a pre release for you guys.
Please request a build directly to me in a private message or email if you are interested in trying it out. Otherwise, the fix will be in the next release.
Thanks Simone!
I plan this test as soon as possible and I will get back to you.
Hi there!
I’m back from few days off :
The import is operating in the new version, however, it seems to be affected by more or less the same issue as in ES7.11.2 1.30.0 Enterprise two contexts rw/ro issue - #15 by abuising
I’ve added some comments in the thread.
Regards
hi @sscarduzio
We are facing same issue (“invalid multipart payload format”) when importing saved objects in Kibana. We are using 7.10.2 standard version of ES and Kibana. We were earlier on 1.30.0 of ROR. But even after upgrading to 1.31.0, we are continuing to get the error during import. We are using the free version of ES ROR and Kibana ROR plugin.
Thanks!
Hi guys, I have a new fix for this whole issue with HTTP errors spanning from import to dev tools problems.
I will send some builds @askids @pchesneau
@askids try this build please. https://readonlyrest-data.s3.eu-west-1.amazonaws.com/build/1.32.0-pre3/free/readonlyrest_kbn_free-1.32.0-pre3_es7.12.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ%2F20210716%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210716T134752Z&X-Amz-Expires=604800&X-Amz-Signature=1c8f97d5a2a08e04697be65d983c51ab7dc9c124a85450bb294b22094b886c19&X-Amz-SignedHeaders=host
@sscarduzio, we are on 7.10.2. The one that you have shared is for 7.12.1. Can you please provide the build for 7.10.2.?
After I upgraded, now Kibana fails to start. I reverted back to old version for now. I see following error logged in stderr file.
Error: listen EADDRINUSE: address already in use 127.0.0.1:11221
at Server.setupListenHandle [as _listen2] (net.js:1280:14)
at listenInCluster (net.js:1328:12)
at doListen (net.js:1461:7)
at process._tickCallback (internal/process/next_tick.js:63:19)
In stdout, I am seeing these entries.
[12:39:13:140] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting config
[12:39:27:294] [error][plugins][ReadonlyREST][lazyUtils] Caught an error in executeWithInterval { FetchError: request to https://myelastic.domain.com/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET
at ClientRequest.<anonymous> (D:\Apps\PROGRA~1\kibana-7.10.2-windows-x86_64\plugins\readonlyrestkbn\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (events.js:198:13)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at TLSSocket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
message:
'request to https://myelastic.domain.com/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET',
type: 'system',
errno: 'ECONNRESET',
code: 'ECONNRESET' }
[12:39:28:109] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting config
[12:39:42:070] [error][plugins][ReadonlyREST][lazyUtils] Caught an error in executeWithInterval { FetchError: request to https://myelastic.domain.com/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET
at ClientRequest.<anonymous> (D:\Apps\PROGRA~1\kibana-7.10.2-windows-x86_64\plugins\readonlyrestkbn\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (events.js:198:13)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at TLSSocket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
message:
'request to https://myelastic.domain.com/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET',
type: 'system',
errno: 'ECONNRESET',
code: 'ECONNRESET' }
[12:39:42:818] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting configYou need to unpatch and patch again, plus remove the optimize folder content (rm -rf optimize/*).
Or better, unpack a fresh kibana and redo the patch + plugin install.
We do unpatch before uninstall and patch again after new install. Those are part of our standard upgrade scripts… But dont remove optimize folder. Let me try it again after removing optimize folder.
Also, should this be standard step going forward (i.e. removing optimize folder during every upgrade)?
Yes, it should because it happened so many times that customers ask me “why do I have an old version number in the ROR menu even if I upgraded?”. This is failure to detect changed sources is due to the Kibana plugin system and we have no power over it.
If you see the docs about upgrading, I already added the step a few weeks ago.