Reporting doesn't work correctly with 1.19.5

Hi,

after upgrading ROR plugin (Kibana PRO + ES) from 1.19.3 to 1.19.5, reporting suddenly stopped working.
I previously reported the bug to Kibana community, but then after I tried to fully remove ROR plugin from Kibana and ES, it started to work again.

I am running ES version 7.5.2.

More detailed information about issue is reported here - https://discuss.elastic.co/t/reporting-stopped-working-due-to-version-conflict/237109

Thanks for any help,
Andrej.

@gulycka, did you upgrade Kibana as well? Or just the ror version?

Hi Simone,

Kibana and ES are running on version 7.5.2.
I upgraded ROR plugin for Kibana and also for ES, from 1.19.3 to 1.19.5.

After upgrade of ROR plugin X-Pack reporting plugin stopped working correctly - no reports are generated and I’m getting error messages with version conflict. More detailed info on Kibana forum. I didn’t want to duplicate all here.

Thanks,
Andrej.

But to directly answer your question, Kibana and ES was not upgraded. I stayed on the same version (7.5.2) and I upgraded just ROR plugin.

Hi @sscarduzio, any ideas? Users are pinging me about non-working reporting feature. Next week I will rollback ROR to older version if we will be stuck. Thanks

Hi @gulycka,
Yes apparently reporting is broken in 1.19.5. I’m now bisecting the changes to find what went wrong.

Hi @sscarduzio,
thanks for confirmation. Let me know if I can help you somehow.

I understood what is the problem, has to do with some stuff we chaned when 7.7 introduced some restrictions when accessing the elasticsearch client. The real fix for this is a big change, and we need a bit more time than next week. For now, it’s worth to downgrade, I’m afraid.

Will report on this as soon as we have something to test.

Hi Simone,

thanks for update.
Would you mind to share me some link to ROR ES plugin 1.19.3?

When I’m trying to use direct link with 1.19.3 as pluginVersion parameter, I’m getting latest version 1.19.5.
https://api.beshu.tech/download/es?email=<my_email>&edition=es&token=<my_token>&esVersion=7.5.2&pluginVersion=1.19.3

Thanks.

Yes, I will send the links in PM

1 Like

Hi Simone, any update for this issue?

Thanks

Yes we’re finishing up. Sorry for the wait.

Hi,

thanks for more information.
Looking forward to it.

Andrej.

Hi Simone, any update on this? :slight_smile:

Yes this works now, will release this weekend!

1 Like

Hi Simone,

I downloaded latest release of ROR plugin (es + kibana pro) in version 1.21.0, where the issues should be mitigated. At least when I checked release notes - * :beetle:Fix (KBN) multitenancy+reporting regression fix (for 7.6.x and earlier)

I deployed new version of ROR plugin, but the issues stayed the same - reporting doesn’t work. When I try to generate CSV report, job is added to the queue, nothing is visible via management/reporting and logs show the same messages:

{"type":"log","@timestamp":"2020-08-11T10:47:42Z","tags":["reporting","queue-job","info"],"pid":30761,"message":"Successfully queued job: kdptmg3u0nqhd6c3759ghrql"}
{"type":"response","@timestamp":"2020-08-11T10:47:42Z","tags":["api"],"pid":30761,"method":"post","statusCode":200,"req":{"url":"/api/reporting/generate/csv?jobParams=","method":"post","headers":{"host":"127.0.0.1:5601","dnt":"1","kbn-version":"7.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"https://<domain>","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<domain>/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"<ip-address>","x-forwarded-host":"<domain>","x-forwarded-server":"<domain>","connection":"Keep-Alive","content-length":"1435"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<domain>/app/kibana"},"res":{"statusCode":200,"responseTime":99,"contentLength":9},"message":"POST /api/reporting/generate/csv?jobParams= 200 99ms - 9.0B"}
{"type":"log","@timestamp":"2020-08-11T10:47:44Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Job marked as claimed: /.reporting-prod.kibana-2020.08.09/_doc/kdptmg3u0nqhd6c3759ghrql"}
{"type":"log","@timestamp":"2020-08-11T10:47:44Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Starting job"}
{"type":"log","@timestamp":"2020-08-11T10:47:45Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Job execution completed successfully"}
{"type":"log","@timestamp":"2020-08-11T10:47:45Z","tags":["reporting","esqueue","queue-worker","info"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - Job data saved successfully: /.reporting-prod.kibana-2020.08.09/_doc/kdptmg3u0nqhd6c3759ghrql"}
{"type":"response","@timestamp":"2020-08-11T10:47:48Z","tags":[],"pid":30761,"method":"get","statusCode":200,"req":{"url":"/api/reporting/jobs/list?page=0&ids=kdptfnee0nqhd6c375759uuv%2Ckdptmg3u0nqhd6c3759ghrql","method":"get","headers":{"host":"127.0.0.1:5601","kbn-system-api":"true","dnt":"1","kbn-version":"7.5.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://<domain>/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"<ip-address>","x-forwarded-host":"<domain>","x-forwarded-server":"<domain>","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://<domain>/app/kibana"},"res":{"statusCode":200,"responseTime":9,"contentLength":9},"message":"GET /api/reporting/jobs/list?page=0&ids=kdptfnee0nqhd6c375759uuv%2Ckdptmg3u0nqhd6c3759ghrql 200 9ms - 9.0B"}
{"type":"log","@timestamp":"2020-08-11T10:47:48Z","tags":["reporting","esqueue","queue-worker","warning"],"pid":30761,"message":"kdpt801t0nqhd6c3759l1ihk - _claimPendingJobs encountered a version conflict on updating pending job kdptmg3u0nqhd6c3759ghrql: Error: [version_conflict_engine_exception] [kdptmg3u0nqhd6c3759ghrql]: version conflict, required seqNo [3], primary term [1]. current document has seqNo [5] and primary term [1], with { index_uuid=\"6rFVxvpuTgyS5qcMKhJLxQ\" & shard=\"0\" & index=\".reporting-prod.kibana-2020.08.09\" }\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}

Can you please confirm that 1.21.0 version should fix this issues? I’m still on Elasticsearch version 7.5.2.

Thanks,
Andrej.

Hi @sscarduzio,

any ideas?

Thanks,
Andrej.

  1. does the report eventually appear under the list of available downloadable csv?
  2. does it happen reliably also when the reporting system indices are deleted?
    I will try to reproduce it.
  1. No, there is empty list of reporting tasks under management.
  2. I tried to delete all reporting indices, but it looks still the same.

Some screenshots from UI:

  1. I generated CSV report from saved search, under user with unrestricted access, to avoid some issues with access privileges
    2020-08-17 07_46_25-Window
  2. I can see message, that report was enqueued
    2020-08-17 07_46_11-Window
  3. In management, there is no report
  4. I already sent you logs from Kibana in previous comment

Hi @sscarduzio, any update please?