Thanks. But we are running into upgrade issue again, when tried upgrading it on additional servers. Even tried doing a fresh kibana install, even after deleting the existing .kibana* indices on those instances, we still get stuck stuck in kibana start up process and keep seeing the errors in loop.
In stderr, i see below error repeating in loop.
Error: listen EADDRINUSE: address already in use 127.0.0.1:11221
at Server.setupListenHandle [as _listen2] (net.js:1280:14)
at listenInCluster (net.js:1328:12)
at doListen (net.js:1461:7)
at process._tickCallback (internal/process/next_tick.js:63:19)
In kibana.log, I see below error repeating in loop. If I start kibana without ROR_Kibana plugin, KIbana starts up just fine. Enabling RORKBN brings it back into this looping status without starting kibana.
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"ml\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"watcher\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"auditTrail\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"graph\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"security\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["warning","config","deprecation"],"pid":8472,"message":"\"xpack.monitoring\" is deprecated and has been replaced by \"monitoring\""}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["warning","config","deprecation"],"pid":8472,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins-system"],"pid":8472,"message":"Setting up [93] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,encryptedSavedObjects,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,indexPatternManagement,fileUpload,savedObjects,readonlyrestkbn,dashboard,dashboardMode,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,snapshotRestore,maps,licenseManagement,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,dataEnhanced,visTypeMarkdown,visTypeTable,tileMap,regionMap,inputControlVis,visualize,esUiShared,bfetch,canvas,charts,lens,visTypeVislib,visTypeMetric,visTypeTimeseries,rollup,visTypeTagcloud,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,securitySolution,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins","readonlyrestkbn"],"pid":8472,"message":"Setting up ReadonlyREST plugin - build info: {\"versionString\":\"free-1.32.0_es7.10.2\",\"kibanaVersion\":\"7.10.2\",\"rorEdition\":\"free\",\"rorVersion\":\"1.32.0\",\"isProduction\":true,\"isEnterprise\":false,\"isPro\":false,\"isFree\":true,\"isBuildExpired\":false}"}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins","reporting","config"],"pid":8472,"message":"Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox."}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins","monitoring","monitoring"],"pid":8472,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-27T08:25:51Z","tags":["info","savedobjects-service"],"pid":8472,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-27T08:25:51Z","tags":["warning","environment"],"pid":8472,"message":"Detected an unhandled Promise rejection.\nFetchError: request to https://myhost2.mydomain.com:11200/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET"}
On stdout, i see below log entries.
[04:57:07:988] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting config
[04:57:25:944] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[04:57:25:959] [error][plugins][ReadonlyREST][lazyUtils] Caught an error in executeWithInterval { FetchError: request to https://myhost2.mydomain.com:11200/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET
at ClientRequest. (D:\Apps\PROGRA~1\kibana-7.10.2-windows-x86_64\plugins\readonlyrestkbn\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (events.js:198:13)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at TLSSocket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
message:
‘request to https://myhost2.mydomain.com:11200/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET’,
type: ‘system’,
errno: ‘ECONNRESET’,
code: ‘ECONNRESET’ }
[04:57:25:964] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting config
This new ROR kibana plugin platform’s upgrade experience is pretty poor compared to older versions. Even after physically restarting the kibana server, the error “address already in use” was not addressed. Please suggest how to resolve the issue.
