Import issue seems to be fixed now. I did install from scratch for Kibana. BTW, when will 1.32.0 be publicly available?
1 Like
hi @sscarduzio can you please let me know when will 1.32.0 be available for download? We have items scheduled for prod promotion in another 3 weeks. So trying to get a date by when it will be available to decide, if that will give us sufficient time to complete testing.
This weekend we go out with 1.32.0
Thanks. But we are running into upgrade issue again, when tried upgrading it on additional servers. Even tried doing a fresh kibana install, even after deleting the existing .kibana* indices on those instances, we still get stuck stuck in kibana start up process and keep seeing the errors in loop.
In stderr, i see below error repeating in loop.
Error: listen EADDRINUSE: address already in use 127.0.0.1:11221
at Server.setupListenHandle [as _listen2] (net.js:1280:14)
at listenInCluster (net.js:1328:12)
at doListen (net.js:1461:7)
at process._tickCallback (internal/process/next_tick.js:63:19)
In kibana.log, I see below error repeating in loop. If I start kibana without ROR_Kibana plugin, KIbana starts up just fine. Enabling RORKBN brings it back into this looping status without starting kibana.
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"ml\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"watcher\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"auditTrail\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"graph\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["info","plugins-service"],"pid":8472,"message":"Plugin \"security\" is disabled."}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["warning","config","deprecation"],"pid":8472,"message":"\"xpack.monitoring\" is deprecated and has been replaced by \"monitoring\""}
{"type":"log","@timestamp":"2021-07-27T08:25:49Z","tags":["warning","config","deprecation"],"pid":8472,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins-system"],"pid":8472,"message":"Setting up [93] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,encryptedSavedObjects,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,indexPatternManagement,fileUpload,savedObjects,readonlyrestkbn,dashboard,dashboardMode,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,snapshotRestore,maps,licenseManagement,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,dataEnhanced,visTypeMarkdown,visTypeTable,tileMap,regionMap,inputControlVis,visualize,esUiShared,bfetch,canvas,charts,lens,visTypeVislib,visTypeMetric,visTypeTimeseries,rollup,visTypeTagcloud,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,securitySolution,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins","readonlyrestkbn"],"pid":8472,"message":"Setting up ReadonlyREST plugin - build info: {\"versionString\":\"free-1.32.0_es7.10.2\",\"kibanaVersion\":\"7.10.2\",\"rorEdition\":\"free\",\"rorVersion\":\"1.32.0\",\"isProduction\":true,\"isEnterprise\":false,\"isPro\":false,\"isFree\":true,\"isBuildExpired\":false}"}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins","reporting","config"],"pid":8472,"message":"Chromium sandbox provides an additional layer of protection, and is supported for Win32 OS. Automatically enabling Chromium sandbox."}
{"type":"log","@timestamp":"2021-07-27T08:25:50Z","tags":["info","plugins","monitoring","monitoring"],"pid":8472,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-27T08:25:51Z","tags":["info","savedobjects-service"],"pid":8472,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-27T08:25:51Z","tags":["warning","environment"],"pid":8472,"message":"Detected an unhandled Promise rejection.\nFetchError: request to https://myhost2.mydomain.com:11200/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET"}
On stdout, i see below log entries.
[04:57:07:988] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting config
[04:57:25:944] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[04:57:25:959] [error][plugins][ReadonlyREST][lazyUtils] Caught an error in executeWithInterval { FetchError: request to https://myhost2.mydomain.com:11200/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET
at ClientRequest. (D:\Apps\PROGRA~1\kibana-7.10.2-windows-x86_64\plugins\readonlyrestkbn\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (events.js:198:13)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at TLSSocket.emit (events.js:198:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
message:
‘request to https://myhost2.mydomain.com:11200/.readonlyrest_kbn_sessions/_search failed, reason: read ECONNRESET’,
type: ‘system’,
errno: ‘ECONNRESET’,
code: ‘ECONNRESET’ }
[04:57:25:964] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:11221
[ROR] - serve.js - intercepting config
[ROR] - serve.js - intercepting config
This new ROR kibana plugin platform’s upgrade experience is pretty poor compared to older versions. Even after physically restarting the kibana server, the error “address already in use” was not addressed. Please suggest how to resolve the issue.
Man, I have never encountered this issue in quite a few happyh new platform customers now.
There must be something different here. Can we please schedule a zoom session? I’d like to take a closer look, if possible?
About the new platform: we removed 90% our code outside the ever changing Kibana plugin API, that meant we had to recreate and rewrite the functionality we previously gave for granted because Kibana offered them as a framework. I.e. ES client, SSL server and client, high availability http server and client.
This code is a couple of months young, but we are rapidly converging to stability, and the stability will accumulate, because it won’t be forced to rewrite our code every other month like we used to in the old platform due to Kibana API changes.
Have a look at the amount of fixes we fit in every release. This one will be yet another, no worries at all 
@askids can you create a new topic and show us the configuration you use? Are you still using the multiple elasticsearch.hosts ? What happens if you reduce it to one?
ok. I will create a new topic. In the servers, where this is failing, its still having only 1 ES host entry. Only in UAT and PROD, we have multiple ES and multiple kibana hosts. So haven’t yet tried it in UAT as Kibana is unable to start due to this failure in Dev and SIT lanes.
Are you using windows? How do you manage to patch? You installed bash for windows?
Yes. We are using Windows 2012 R2. We did not install bash. ROR Kibana docs simply uses node. So we just use Node.exe and followed same command as mentioned in docs. We are running the process through powershell.
Oh right, I thought about exactly this use case when I designed the tool, but I forgot about it now 
@askids when you want we can start debugging this in your system? Or do I need to install windows?
Let me give it one more shot. One additional issue that I noticed is that there is NO folder by name optimize under Kibana base folder. But I see a folder with name optimize under src.
D:\Apps\Progra~1\kibana-7.10.2-windows-x86_64\src\optimize
In 7.8.1, optimize folder was being created directly under kibana base folder. But we dont find it in 7.10.2. Am I supposed to delete the optimize under src folder?
Actually, looks like they removed the main optimize directory in newer versions… Interesting. The src/optimize is not related to plugins.
It was still there up to Kibana 7.9.3 though. It disappeared in 7.10.0, just checked.
Good, the problem is gone
Only the optimize problem is gone. Other issue is still there. I was able to rollback to 1.31.0 and get Kibana to start up again. But upgrading to 1.32.0 still takes Kibana into startup loop. I will enable debug logging tomorrow and see what is getting logged.
1 Like