Hi, I installed the elasticsearch and kibana plugin and now it won’t start kibana. Here is the error I see in the logs:
Sep 01 09:58:46 elkserver systemd[1]: kibana.service failed.
Sep 01 09:58:49 elkserver systemd[1]: kibana.service holdoff time over, scheduling restart.
Sep 01 09:58:49 elkserver systemd[1]: Stopped Kibana.
Sep 01 09:58:49 elkserver systemd[1]: Started Kibana.
Sep 01 09:58:53 elkserver kibana[16446]: [ROR] - serve.js - intercepting config
Sep 01 09:58:53 elkserver kibana[16446]: Configuring logger failed: Error: Cannot find module '…/…/…/plugins/readonlyrestkbn/kibana/kibanaConfigInterceptor
Sep 01 09:58:53 elkserver kibana[16446]: Require stack:
Sep 01 09:58:53 elkserver kibana[16446]: - /usr/share/kibana/src/cli/serve/serve.js
Sep 01 09:58:53 elkserver kibana[16446]: - /usr/share/kibana/src/cli/cli.js
Sep 01 09:58:53 elkserver kibana[16446]: - /usr/share/kibana/src/cli/dist.js
Sep 01 09:58:53 elkserver kibana[16446]: at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
Sep 01 09:58:53 elkserver kibana[16446]: at Module.Hook._require.Module.require (/usr/share/kibana/node_modules/require-in-the-middle/index.js:61:29)
Sep 01 09:58:53 elkserver kibana[16446]: at require (internal/modules/cjs/helpers.js:92:18)
Sep 01 09:58:53 elkserver kibana[16446]: at applyConfigOverrides (/usr/share/kibana/src/cli/serve/serve.js:167:10)
Sep 01 09:58:53 elkserver kibana[16446]: at applyConfigOverrides (/usr/share/kibana/src/cli/serve/serve.js:228:42)
Sep 01 09:58:53 elkserver kibana[16446]: at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target/raw/raw_config_service.js:30:24)
Sep 01 09:58:53 elkserver kibana[16446]: at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)
Sep 01 09:58:53 elkserver kibana[16446]: at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)
Sep 01 09:58:53 elkserver kibana[16446]: at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28)
Sep 01 09:58:53 elkserver kibana[16446]: at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25) {
Sep 01 09:58:53 elkserver kibana[16446]: code: ‘MODULE_NOT_FOUND’,
Sep 01 09:58:53 elkserver kibana[16446]: requireStack: [
Sep 01 09:58:53 elkserver kibana[16446]: ‘/usr/share/kibana/src/cli/serve/serve.js’,
Sep 01 09:58:53 elkserver kibana[16446]: ‘/usr/share/kibana/src/cli/cli.js’,
Sep 01 09:58:53 elkserver kibana[16446]: ‘/usr/share/kibana/src/cli/dist.js’
Sep 01 09:58:53 elkserver kibana[16446]: ]
Sep 01 09:58:53 elkserver kibana[16446]: }
Sep 01 09:58:53 elkserver kibana[16446]: FATAL Error: Cannot find module ‘…/…/…/plugins/readonlyrestkbn/kibana/kibanaConfigInterceptor’
Sep 01 09:58:53 elkserver kibana[16446]: Require stack:
Sep 01 09:58:53 elkserver kibana[16446]: - /usr/share/kibana/src/cli/serve/serve.js
Sep 01 09:58:53 elkserver kibana[16446]: - /usr/share/kibana/src/cli/cli.js
Sep 01 09:58:53 elkserver kibana[16446]: - /usr/share/kibana/src/cli/dist.js
What kibana plugin did you install? What’s the file name?
Kibana 7.14
readonlyrest_kbn_free-1.33.1_es7.14.0.zip
I have installed that very package within Kibana 7.14.0, and worked for me. Can you verify:
- you patched kibana
- patching Kibana went well:
$ node/bin/node plugins/readonlyrestkbn/ror-tools.js verify
[ROR COMPAT] Received command: verify
[ROR COMPAT] Verifying presence of ROR hooks on Kibana files..
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
verifying patched state...
http_server.js patched state VERIFIED.
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/serve.js.patch
verifying patched state...
serve.js patched state VERIFIED.
- verify the content of this directory:
$ ls plugins/readonlyrestkbn/kibana/
config configUtils.js kibanaConfigInterceptor.js kibanaHttpServerSecurityMiddleware.js patchers rorStore.js
Thanks for helping out.
I verified that I patched kibana and it went well.
bin/kibana-plugin install file:///root/readonlyrest_kbn_free-1.33.1_es7.14.0.zip
Attempting to transfer from file:///root/readonlyrest_kbn_free-1.33.1_es7.14.0.zip
Transferring 30769458 bytes…
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete
node/bin/node plugins/readonlyrestkbn/ror-tools.js patch
[ROR COMPAT] Received command: patch
[ROR COMPAT] Adding hooks on a few Kibana files if necessary…
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
Applied ‘http_server.js.patch’ to ‘/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/…/…/…/…/src/core/server/http/http_server.js’
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/serve.js.patch
Applied ‘serve.js.patch’ to ‘/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/…/…/…/…/src/cli/serve/serve.js’
node/bin/node plugins/readonlyrestkbn/ror-tools.js verify
[ROR COMPAT] Received command: verify
[ROR COMPAT] Verifying presence of ROR hooks on Kibana files…
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
verifying patched state…
http_server.js patched state VERIFIED.
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/serve.js.patch
verifying patched state…
serve.js patched state VERIFIED.
Directory: /usr/share/kibana/plugins/readonlyrestkbn/kibana
config configUtils.js kibanaConfigInterceptor.js kibanaHttpServerSecurityMiddleware.js patchers rorStore.js
Do you have issues with permissions? I.e. the Kibana daemon has no permission to read the files you installed as another user? Please compare “serve.js” (or other pre-existing kibana files) file owner and permissions with kibanaConfigInterceptor.js (or any other ROR source files).
I changed the permissions to match the original files and still does not work. same error message
OK weird. Are you using docker by chance? I really cannot reproduce this.
Can you maybe try installing Kibana from scratch, install the plugin, patch, all over again in a test environment? Can you reproduce this error systematically?
No docker, CentOS 7 only.
I’ll try to install from scratch to see if that works.
I found out I’m using a CIS CentoOS from Amazon and that appears to be the main difference.
I built another image using a regular CentOS version and it worked fine.
Are you aware of any CIS builds or issues?
OMG another instance of “security” getting in the way of people achieving goals 
Happy you found your way out of this. 
I haven’t found my way out yet. I have to use the CIS CentOS version due to requirements.
This is the error I get when running the kibana command manually versus starting the service:
/kibana --allow-root
[ROR] - serve.js - intercepting config
[07:51:10:978] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:5621
log [07:51:18.146] [info][plugins-service] Plugin “telemetry” is disabled.
log [07:51:18.151] [info][plugins-service] Plugin “telemetryManagementSection” has been disabled since the following direct or transitive dependencies are missing or disabled: [telemetry]
log [07:51:18.153] [info][plugins-service] Plugin “metricsEntities” is disabled.
log [07:51:18.154] [info][plugins-service] Plugin “osquery” has been disabled since the following direct or transitive dependencies are missing or disabled: [security]
log [07:51:18.154] [info][plugins-service] Plugin “security” is disabled.
[ROR] - serve.js - intercepting config
log [07:51:18.244] [warning][config][deprecation] plugins.scanDirs is deprecated and is no longer used
log [07:51:18.244] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [07:51:18.245] [warning][config][deprecation] “xpack.reporting.roles” is deprecated. Granting reporting privilege through a “reporting_user” role will not be supported starting in 8.0. Please set "xpack.reporting.roles.enabled
" to “false” and grant reporting privileges to users using Kibana application privileges Management > Security > Roles.
log [07:51:18.245] [warning][config][deprecation] Disabling the security plugin (xpack.security.enabled) will not be supported in the next major version (8.0). To turn off security features, disable them in Elasticsearch instead
.
log [07:51:18.280] [info][server][NotReady][http] http server running at http://aws-hub-elk1.myvest.com:5601
log [07:51:18.496] [info][plugins-system] Setting up [104] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,encryptedSavedObjects,code,usageCollection,xpackLegacy,telemetryCol
lectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,fileUpload,sav
edObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,readonlyrestkbn,home,searchprofiler,painlessLa
b,grokdebugger,graph,cloud,fleet,visTypeVega,management,watcher,upgradeAssistant,transform,snapshotRestore,savedObjectsTagging,licenseManagement,ingestPipelines,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleMana
gement,dataEnhanced,timelines,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,maps,lens,dataVisualizer,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,savedObjectsManagement,spaces,report
ing,canvas,lists,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,enterpriseSearch,console,apmOss,apm]
log [07:51:18.498] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: d592c96d-ead9-4fa9-8dba-b9706238e62b
log [07:51:20.140] [info][plugins][readonlyrestkbn] Setting up ReadonlyREST plugin - build info: {“versionString”:“free-1.33.1_es7.14.0”,“kibanaVersion”:“7.14.0”,“rorEdition”:“free”,“rorVersion”:“1.33.1”,“isProduction”:true,“isEnt
erprise”:false,“isPro”:false,“isFree”:true,“isBuildExpired”:false}
log [07:51:20.263] [warning][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.9.2009 OS. Automatically setting ‘xpack.reporting.capture.browser.chromi
um.disableSandbox: true’.
log [07:51:21.593] [info][plugins][ruleRegistry] Write is disabled, not installing assets
log [07:51:21.940] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations…
log [07:51:22.193] [error][savedobjects-service] Unable to retrieve version information from Elasticsearch nodes. Response Error
Looks like Kibana cannot contact Elasticsearch anymore. This could be due to some strict firewall configuration. That is, because ROR Kibana plugin (new platform) creates two localhost proxies, and:
- a) there needs to be permissions to bind to local ports
- b) traffic in random high ports on localhost needs to be permitted.
I would try drop all the iptables and see what happens