New 7.3.0 Kibana PRO plug-in crashes with error: "FATAL ReferenceError: reportingHelper is not defined"

I’ve been checking occasionally for when you would release a version compatible with ES 7.3.x, and today was the day! I downloaded the 1.18.5 ES and the Kibana PRO plug-ins and did the upgrade on my small, non-production cluster.The ES update went fine, so all three nodes are ES 7.3.0. I upgraded just one node of Kibana to 7.3.0, including the Kibana PRO, and when I started it up, it paused the usual couple of minutes performing the “optimizing” step. But then it crashed, and since it is controlled by systemd, it starts up and crashes repeatedly. Here is the slightly-edited logging from journalctl:

Sep 09 11:48:45 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Scheduled restart job, restart counter is at 19.
Sep 09 11:48:45 qim-elastic1-d7.qim.com systemd[1]: Stopped Kibana.
Sep 09 11:48:45 qim-elastic1-d7.qim.com systemd[1]: Started Kibana.
Sep 09 11:48:50 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:50Z”,“tags”:[“info”,“plugins-system”],“pid”:2199,“message”:“Setting up [1] plugins: [translations]”}
Sep 09 11:48:50 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:50Z”,“tags”:[“info”,“plugins”,“translations”],“pid”:2199,“message”:“Setting up plugin”}
Sep 09 11:48:50 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:50Z”,“tags”:[“info”,“plugins-system”],“pid”:2199,“message”:“Starting [1] plugins: [translations]”}
Sep 09 11:48:58 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:58Z”,“tags”:[“status”,“plugin:[email protected]”,“info”],“pid”:2199,“state”:“green”,“message”:“Status changed from uninitialized to green - Ready”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:58 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:58Z”,“tags”:[“status”,“plugin:[email protected]”,“info”],“pid”:2199,“state”:“yellow”,“message”:“Status changed from uninitialized to yellow - Waiting for Elasticsearch”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:58 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:58Z”,“tags”:[“status”,“plugin:[email protected]”,“info”],“pid”:2199,“state”:“yellow”,“message”:“Status changed from uninitialized to yellow - Waiting for Elasticsearch”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}

… (a bunch more initializing lines) …

Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:59Z”,“tags”:[“status”,“plugin:[email protected]”,“info”],“pid”:2199,“state”:“green”,“message”:“Status changed from uninitialized to green - Ready”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:59Z”,“tags”:[“status”,“plugin:[email protected]”,“info”],“pid”:2199,“state”:“green”,“message”:“Status changed from uninitialized to green - Ready”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: ROR log level: info
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:59Z”,“tags”:[“fatal”,“root”],“pid”:2199,“message”:“ReferenceError: reportingHelper is not defined\n at Plugin.reportingHelper [as externalInit] (/usr/share/kibana/plugins/readonlyrest_kbn/index.js:120:7)\n at Object.register (/usr/share/kibana/src/legacy/server/plugins/lib/plugin.js:96:20)\n at internals.Server.register (/usr/share/kibana/node_modules/hapi/lib/server.js:431:35)\n at Plugin.init (/usr/share/kibana/src/legacy/server/plugins/lib/plugin.js:100:28)\n at Plugin.init (/usr/share/kibana/node_modules/lodash/index.js:7411:25)\n at callPluginHook (/usr/share/kibana/src/legacy/server/plugins/lib/call_plugin_hook.js:53:25)”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:59Z”,“tags”:[“info”,“plugins-system”],“pid”:2199,“message”:“Stopping all plugins.”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: {“type”:“log”,"@timestamp":“2019-09-09T15:48:59Z”,“tags”:[“info”,“plugins”,“translations”],“pid”:2199,“message”:“Stopping plugin”}
Sep 09 11:48:59 qim-elastic1-d7.qim.com kibana[2199]: FATAL ReferenceError: reportingHelper is not defined
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Failed with result ‘exit-code’.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: kibana.service: Scheduled restart job, restart counter is at 20.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: Stopped Kibana.
Sep 09 11:49:00 qim-elastic1-d7.qim.com systemd[1]: Started Kibana.
Sep 09 11:49:02 qim-elastic1-d7.qim.com systemd[1]: Stopping Kibana…
Sep 09 11:49:02 qim-elastic1-d7.qim.com systemd[1]: Stopped Kibana.

So it looks like a missing piece of JavaScript, like the installer didn’t fully succeed, except I can see a file that sounds as if it would supply that.

[email protected]:/usr/share/kibana/plugins/readonlyrest_kbn# find . -name “report*”
./server/routes/lib/reporting_helper.js
[email protected]:/usr/share/kibana/plugins/readonlyrest_kbn#

I followed my usual procedures in performing the upgrade, but I tried it again and made sure the entire kibana plug-in subdirectory disappeared between remove and reinstall. I do note that I normally run the kibana-plugin installer as “root” user, and lately I have to give a parameter “–allow-root” for that plugin command to let me do so. I tried doing a recursive “chown” after the fact, to make the entire readonlyrest directory and everything in it owned by the non-root “kibana:kibana” user and group, but it still crashes the same way on startup. Hmm.

Anyway, if you want me to try anything or want any more logging or info, say the word. Thank you.

– Jeff Saxe

I was trying to figure out how to increase the logging verbosity of the Kibana startup, and while I haven’t done that, I figured out that I can become the “kibana” service account and manually start it up in a way similar to how the Systemd service starts, but the logging output is easier to read.

[email protected]:/usr/share/kibana/plugins/readonlyrest_kbn# su - kibana -s /bin/bash
[email protected]:~$ 
[email protected]:~$ whoami
kibana
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ /usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
  log   [15:09:33.971] [info][plugins-system] Setting up [1] plugins: [translations]
  log   [15:09:33.987] [info][plugins][translations] Setting up plugin
  log   [15:09:33.989] [info][plugins-system] Starting [1] plugins: [translations]
  log   [15:10:09.048] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.061] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.065] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.089] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.092] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.116] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.124] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.140] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.146] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.242] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.245] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.278] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.283] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.286] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.300] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.332] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.366] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.369] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.373] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.378] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.385] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.398] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.424] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.433] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.437] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.440] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.472] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.475] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.481] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.485] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.511] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.577] [info][status][plugin:i[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.581] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.597] [info][siem] Plugin initializing
  log   [15:10:09.645] [info][siem] Plugin done initializing
  log   [15:10:09.646] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.653] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.655] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.681] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.706] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.709] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.712] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.716] [warning][encrypted_saved_objects] Generating a random key for xpack.encrypted_saved_objects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encrypted_saved_objects.encryptionKey in kibana.yml
  log   [15:10:09.718] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.732] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [15:10:09.754] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
  log   [15:10:09.763] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready
ROR log level:  info
  log   [15:10:09.769] [fatal][root] ReferenceError: reportingHelper is not defined
    at Plugin.reportingHelper [as externalInit] (/usr/share/kibana/plugins/readonlyrest_kbn/index.js:120:7)
    at Object.register (/usr/share/kibana/src/legacy/server/plugins/lib/plugin.js:96:20)
    at internals.Server.register (/usr/share/kibana/node_modules/hapi/lib/server.js:431:35)
    at Plugin.init (/usr/share/kibana/src/legacy/server/plugins/lib/plugin.js:100:28)
    at Plugin.init (/usr/share/kibana/node_modules/lodash/index.js:7411:25)
    at callPluginHook (/usr/share/kibana/src/legacy/server/plugins/lib/call_plugin_hook.js:53:25)
  log   [15:10:09.791] [info][plugins-system] Stopping all plugins.
  log   [15:10:09.792] [info][plugins][translations] Stopping plugin

 FATAL  ReferenceError: reportingHelper is not defined

[email protected]:~$ 
[email protected]:~$ 

Perhaps this is more helpful.

Hi Jeff, this issue is solved now as for 1.18.6-pre2. Will hand you a build soon. Some enterprise code leaked into PRO during the build phase.

Oh, good, I’m glad you already were aware of it! Thanks, Simone.

Sorry, I was out of office for a couple of days. How do I download this 1.18.6-pre2 version of the PRO plug-in? Do you just email it to our registered address ([email protected]…) as if we had requested it off the Download function?

I sent you a DM with the link!