Reporting doesn't work correctly with 1.19.5

gulycka · June 16, 2020, 6:58am

Hi,

after upgrading ROR plugin (Kibana PRO + ES) from 1.19.3 to 1.19.5, reporting suddenly stopped working.
I previously reported the bug to Kibana community, but then after I tried to fully remove ROR plugin from Kibana and ES, it started to work again.

I am running ES version 7.5.2.

More detailed information about issue is reported here - Reporting stopped working due to version conflict - Kibana - Discuss the Elastic Stack

Thanks for any help,
Andrej.

sscarduzio · June 16, 2020, 1:23pm

@gulycka, did you upgrade Kibana as well? Or just the ror version?

gulycka · June 16, 2020, 2:26pm

Hi Simone,

Kibana and ES are running on version 7.5.2.
I upgraded ROR plugin for Kibana and also for ES, from 1.19.3 to 1.19.5.

After upgrade of ROR plugin X-Pack reporting plugin stopped working correctly - no reports are generated and I’m getting error messages with version conflict. More detailed info on Kibana forum. I didn’t want to duplicate all here.

Thanks,
Andrej.

gulycka · June 17, 2020, 9:27am

But to directly answer your question, Kibana and ES was not upgraded. I stayed on the same version (7.5.2) and I upgraded just ROR plugin.

gulycka · June 18, 2020, 9:34am

Hi @sscarduzio, any ideas? Users are pinging me about non-working reporting feature. Next week I will rollback ROR to older version if we will be stuck. Thanks

sscarduzio · June 18, 2020, 8:42pm

Hi @gulycka,
Yes apparently reporting is broken in 1.19.5. I’m now bisecting the changes to find what went wrong.

gulycka · June 19, 2020, 6:13am

Hi @sscarduzio,
thanks for confirmation. Let me know if I can help you somehow.

sscarduzio · June 19, 2020, 12:22pm

I understood what is the problem, has to do with some stuff we chaned when 7.7 introduced some restrictions when accessing the elasticsearch client. The real fix for this is a big change, and we need a bit more time than next week. For now, it’s worth to downgrade, I’m afraid.

Will report on this as soon as we have something to test.

gulycka · June 22, 2020, 7:56am

Hi Simone,

thanks for update.
Would you mind to share me some link to ROR ES plugin 1.19.3?

When I’m trying to use direct link with 1.19.3 as pluginVersion parameter, I’m getting latest version 1.19.5.
https://api.beshu.tech/download/es?email=<my_email>&edition=es&token=<my_token>&esVersion=7.5.2&pluginVersion=1.19.3

Thanks.

sscarduzio · June 22, 2020, 1:18pm

Yes, I will send the links in PM

gulycka · July 3, 2020, 7:12am

Hi Simone, any update for this issue?

Thanks

sscarduzio · July 3, 2020, 11:28am

Yes we’re finishing up. Sorry for the wait.

gulycka · July 6, 2020, 6:07am

Hi,

thanks for more information.
Looking forward to it.

Andrej.

gulycka · August 6, 2020, 9:30am

Hi Simone, any update on this?

sscarduzio · August 6, 2020, 10:09am

Yes this works now, will release this weekend!

gulycka · August 11, 2020, 10:52am

Hi Simone,

I downloaded latest release of ROR plugin (es + kibana pro) in version 1.21.0, where the issues should be mitigated. At least when I checked release notes - * Fix (KBN) multitenancy+reporting regression fix (for 7.6.x and earlier)

I deployed new version of ROR plugin, but the issues stayed the same - reporting doesn’t work. When I try to generate CSV report, job is added to the queue, nothing is visible via management/reporting and logs show the same messages:

{"type":"log","@timestamp":"2020-08-11T10:47:42Z","tags":["reporting","queue-job","info"],"pid":30761,"message":"Successfully queued job: kdptmg3u0nqhd6c3759ghrql"}
{"type":"response","@timestamp":"2020-08-11T10:47:42Z","tags":["api"],"pid":30761,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/csv?jobParams=","method":"post","headers":{"host":"127.0.0.1:5601","dnt":"1","kbn-version":"7.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"https://<domain>","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<domain>/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"<ip-address>","x-forwarded-host":"<domain>","x-forwarded-server":"<domain>","connection":"Keep-Alive","content-length":"1435"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<domain>/app/kibana"},"res":{"statusCode":200,"responseTime":99,"contentLength":9},"message":"POST /api/reporting/generate/csv?jobParams= 200 99ms - 9.0B"}
{"type":"log","@timestamp":"2020-08-11T10:47:44Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Job marked as claimed: /.reporting-prod.kibana-2020.08.09/_doc/kdptmg3u0nqhd6c3759ghrql"}
{"type":"log","@timestamp":"2020-08-11T10:47:44Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Starting job"}
{"type":"log","@timestamp":"2020-08-11T10:47:45Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Job execution completed successfully"}
{"type":"log","@timestamp":"2020-08-11T10:47:45Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Job data saved successfully: /.reporting-prod.kibana-2020.08.09/_doc/kdptmg3u0nqhd6c3759ghrql"}
{"type":"response","@timestamp":"2020-08-11T10:47:48Z","tags":[],"pid":30761,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0&ids=kdptfnee0nqhd6c375759uuv%2Ckdptmg3u0nqhd6c3759ghrql","method":"get","headers":{"host":"127.0.0.1:5601","kbn-system-api":"true","dnt":"1","kbn-version":"7.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<domain>/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"<ip-address>","x-forwarded-host":"<domain>","x-forwarded-server":"<domain>","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<domain>/app/kibana"},"res":{"statusCode":200,"responseTime":9,"contentLength":9},"message":"GET /api/reporting/jobs/list?page=0&ids=kdptfnee0nqhd6c375759uuv%2Ckdptmg3u0nqhd6c3759ghrql 200 9ms - 9.0B"}
{"type":"log","@timestamp":"2020-08-11T10:47:48Z","tags":["reporting","esqueue","queue-worker","warning"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - _claimPendingJobs encountered a version conflict on updating pending job kdptmg3u0nqhd6c3759ghrql: Error: [version_conflict_engine_exception] [kdptmg3u0nqhd6c3759ghrql]: version conflict, required seqNo [3], primary term [1]. current document has seqNo [5] and primary term [1], with { index_uuid=\"6rFVxvpuTgyS5qcMKhJLxQ\" & shard=\"0\" & index=\".reporting-prod.kibana-2020.08.09\" }\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}

Can you please confirm that 1.21.0 version should fix this issues? I’m still on Elasticsearch version 7.5.2.

Thanks,
Andrej.

gulycka · August 14, 2020, 5:43am

Hi @sscarduzio,

any ideas?

Thanks,
Andrej.

sscarduzio · August 16, 2020, 8:36pm

does the report eventually appear under the list of available downloadable csv?
does it happen reliably also when the reporting system indices are deleted?
I will try to reproduce it.

gulycka · August 17, 2020, 5:51am

No, there is empty list of reporting tasks under management.
I tried to delete all reporting indices, but it looks still the same.

Some screenshots from UI:

I generated CSV report from saved search, under user with unrestricted access, to avoid some issues with access privileges
I can see message, that report was enqueued
In management, there is no report

2020-08-17 07_46_38-Window1681×497 35.6 KB
I already sent you logs from Kibana in previous comment

gulycka · August 26, 2020, 8:33am

Hi @sscarduzio, any update please?