I’ve been checking occasionally for when you would release a version compatible with ES 7.3.x, and today was the day! I downloaded the 1.18.5 ES and the Kibana PRO plug-ins and did the upgrade on my small, non-production cluster.The ES update went fine, so all three nodes are ES 7.3.0. I upgraded just one node of Kibana to 7.3.0, including the Kibana PRO, and when I started it up, it paused the usual couple of minutes performing the “optimizing” step. But then it crashed, and since it is controlled by systemd, it starts up and crashes repeatedly. Here is the slightly-edited logging from journalctl:
Sep 09 11:48:45 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Scheduled restart job, restart counter is at 19.
Sep 09 11:48:45 qim-elastic1-d7.qim.com systemd[1]: Stopped Kibana.
Sep 09 11:48:45 qim-elastic1-d7.qim.com systemd[1]: Started Kibana.
Sep 09 11:48:50 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:50Z”,“tags”:[“info”,“plugins-system”],“pid”:2199,“message”:“Setting up [1] plugins: [translations]”}
Sep 09 11:48:50 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:50Z”,“tags”:[“info”,“plugins”,“translations”],“pid”:2199,“message”:“Setting up plugin”}
Sep 09 11:48:50 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:50Z”,“tags”:[“info”,“plugins-system”],“pid”:2199,“message”:“Starting [1] plugins: [translations]”}
Sep 09 11:48:58 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:58Z”,“tags”:[“status”,“plugin:kibana@7.3.0”,“info”],“pid”:2199,“state”:“green”,“message”:“Status changed from uninitialized to green - Ready”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:58 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:58Z”,“tags”:[“status”,“plugin:elasticsearch@7.3.0”,“info”],“pid”:2199,“state”:“yellow”,“message”:“Status changed from uninitialized to yellow - Waiting for Elasticsearch”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:58 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:58Z”,“tags”:[“status”,“plugin:xpack_main@7.3.0”,“info”],“pid”:2199,“state”:“yellow”,“message”:“Status changed from uninitialized to yellow - Waiting for Elasticsearch”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
… (a bunch more initializing lines) …
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:59Z”,“tags”:[“status”,“plugin:actions@7.3.0”,“info”],“pid”:2199,“state”:“green”,“message”:“Status changed from uninitialized to green - Ready”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:59Z”,“tags”:[“status”,“plugin:alerting@7.3.0”,“info”],“pid”:2199,“state”:“green”,“message”:“Status changed from uninitialized to green - Ready”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: ROR log level: info
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:59Z”,“tags”:[“fatal”,“root”],“pid”:2199,“message”:“ReferenceError: reportingHelper is not defined\n at Plugin.reportingHelper [as externalInit] (/usr/share/kibana/plugins/readonlyrest_kbn/index.js:120:7)\n at Object.register (/usr/share/kibana/src/legacy/server/plugins/lib/plugin.js:96:20)\n at internals.Server.register (/usr/share/kibana/node_modules/hapi/lib/server.js:431:35)\n at Plugin.init (/usr/share/kibana/src/legacy/server/plugins/lib/plugin.js:100:28)\n at Plugin.init (/usr/share/kibana/node_modules/lodash/index.js:7411:25)\n at callPluginHook (/usr/share/kibana/src/legacy/server/plugins/lib/call_plugin_hook.js:53:25)”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:59Z”,“tags”:[“info”,“plugins-system”],“pid”:2199,“message”:“Stopping all plugins.”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,“@timestamp”:“2019-09-09T15:48:59Z”,“tags”:[“info”,“plugins”,“translations”],“pid”:2199,“message”:“Stopping plugin”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: FATAL ReferenceError: reportingHelper is not defined
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Failed with result ‘exit-code’.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Scheduled restart job, restart counter is at 20.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: Stopped Kibana.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: Started Kibana.
Sep 09 11:49:02 qim-elastic1-d7.qim.com systemd[1]: Stopping Kibana…
Sep 09 11:49:02 qim-elastic1-d7.qim.com systemd[1]: Stopped Kibana.
So it looks like a missing piece of JavaScript, like the installer didn’t fully succeed, except I can see a file that sounds as if it would supply that.
root@qim-elastic1-d7:/usr/share/kibana/plugins/readonlyrest_kbn# find . -name “report*”
./server/routes/lib/reporting_helper.js
root@qim-elastic1-d7:/usr/share/kibana/plugins/readonlyrest_kbn#
I followed my usual procedures in performing the upgrade, but I tried it again and made sure the entire kibana plug-in subdirectory disappeared between remove and reinstall. I do note that I normally run the kibana-plugin installer as “root” user, and lately I have to give a parameter “–allow-root” for that plugin command to let me do so. I tried doing a recursive “chown” after the fact, to make the entire readonlyrest directory and everything in it owned by the non-root “kibana:kibana” user and group, but it still crashes the same way on startup. Hmm.
Anyway, if you want me to try anything or want any more logging or info, say the word. Thank you.
– Jeff Saxe