Kibana 6.8.4 not starting after 1.19.1 plugin upgrade

Users Feedback
1

{“type”:“log”,"@timestamp":“2020-02-21T13:30:47Z”,“tags”:[“status”,“plugin:maps@6.8.4”,“info”],“pid”:325103,“state”:“green”,“message”:“Status changed from red to green - Ready”,“prevState”:“red”,“prevMsg”:"[undefined] Waiting for ReadonlyREST start"}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:47Z”,“tags”:[“status”,“plugin:spaces@6.8.4”,“info”],“pid”:325103,“state”:“green”,“message”:“Status changed from red to green - Ready”,“prevState”:“red”,“prevMsg”:"[undefined] Waiting for ReadonlyREST start"}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:47Z”,“tags”:[“error”,“task_manager”],“pid”:325103,“message”:“Failed to poll for work: [undefined] Waiting for ReadonlyREST start :: {“path”:”/.kibana_task_manager/_doc/_search",“query”:{“ignore_unavai
lable”:true},“body”:"{\“query\”:{\“bool\”:{\“must\”:[{\“term\”:{\“type\”:\“task\”}},{\“bool\”:{\“must\”:[{\“terms\”:{\“task.taskType\”:[\“maps_telemetry\”,\“vis_telemetry\”]}},{\“range\”:{\“task
.attempts\”:{\“lte\”:3}}},{\“range\”:{\“task.runAt\”:{\“lte\”:\“now\”}}},{\“range\”:{\“kibana.apiVersion\”:{\“lte\”:1}}}]}}]}},\“size\”:10,\“sort\”:{\“task.runAt\”:{\“order\”:\“asc\”}},\“seq_no_pr
imary_term\”:true}",“statusCode”:503,“response”:"{\“error\”:{\“root_cause\”:[{\“reason\”:\“Waiting for ReadonlyREST start\”}],\“reason\”:\“Waiting for ReadonlyREST start\”,\“status\”:503}}"}"}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:52Z”,“tags”:[“status”,“plugin:elasticsearch@6.8.4”,“info”],“pid”:325103,“state”:“green”,“message”:“Status changed from red to green - Ready”,“prevState”:“red”,“prevMsg”:"[undefined] Waiting for ReadonlyREST
start"}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:53Z”,“tags”:[“error”,“task_manager”],“pid”:325103,“message”:“Failed to poll for work: [undefined] Waiting for ReadonlyREST start :: {“path”:”/.kibana_task_manager/_doc/_search",“query”:{“ignore_unavai
lable”:true},“body”:"{\“query\”:{\“bool\”:{\“must\”:[{\“term\”:{\“type\”:\“task\”}},{\“bool\”:{\“must\”:[{\“terms\”:{\“task.taskType\”:[\“maps_telemetry\”,\“vis_telemetry\”]}},{\“range\”:{\“task
.attempts\”:{\“lte\”:3}}},{\“range\”:{\“task.runAt\”:{\“lte\”:\“now\”}}},{\“range\”:{\“kibana.apiVersion\”:{\“lte\”:1}}}]}}]}},\“size\”:10,\“sort\”:{\“task.runAt\”:{\“order\”:\“asc\”}},\“seq_no_pr
imary_term\”:true}",“statusCode”:503,“response”:"{\“error\”:{\“root_cause\”:[{\“reason\”:\“Waiting for ReadonlyREST start\”}],\“reason\”:\“Waiting for ReadonlyREST start\”,\“status\”:503}}"}"}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:xpack_main@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Re
ady”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:spaces@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Ready”
}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:searchprofiler@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”
:“Ready”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:ml@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Ready”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:tilemap@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Ready
“}
{“type”:“log”,”@timestamp”:“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:watcher@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Ready
“}
{“type”:“log”,”@timestamp”:“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:grokdebugger@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:"
Ready"}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:logstash@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Read
y”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:beats_management@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMs
g”:“Ready”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:maps@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Ready”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:index_management@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMs
g”:“Ready”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:index_lifecycle_management@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“gree
n”,“prevMsg”:“Ready”}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:rollup@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:“Ready”
}
{“type”:“log”,"@timestamp":“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:remote_clusters@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg
“:“Ready”}
{“type”:“log”,”@timestamp”:“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:cross_cluster_replication@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green
“,“prevMsg”:“Ready”}
{“type”:“log”,”@timestamp”:“2020-02-21T13:30:55Z”,“tags”:[“status”,“plugin:elasticsearch@6.8.4”,“error”],“pid”:325103,“state”:“red”,“message”:“Status changed from green to red - [undefined] Waiting for ReadonlyREST start”,“prevState”:“green”,“prevMsg”:
“Ready”}

2

Hi @skks do you have Elasticsearch logs about this?

3

hmmm, have some incredable
I haved 12 master-node, and 4 nodes doesn’t starting correctrly (not have connect to ldap)
and readonlyrest plugin for kibana not try work with other master-node, but in config haved all 12 nodes.
can’t use failover for starting process

4

Will Kibana start if you configure it with only one ES node?

5

Yes
in “bad nodes” return from any API request:
“waiting for start readonlyrest plugin”

6

OK, does the connectivity to LDAP ever come back eventually and the “bad” nodes converge to green status? Or do they stay down forever until restarted?

7

unfortunately only after restart ES node

8

OK Got it. @coutoPL how many times do we retry LDAP connections on startup? Is there a strategy in place at all?

9

5 times (hardcoded) - maybe we can expose it in LDAP configuration

10

Maybe a exponenital backoff could be the right strategy?

11

it’s a bit complicated. We cannot set high timeout, because of our synchronous reloading process invoked by API call (we need to wait for ROR restart before sending back response if the reload was correct or not).

But I don’t fully understand what is the issue. ROR starts asynchronously because during its startup there are some time-consuming tasks to do (like the LDAP connectivity check and so). We don’t want to block ES (to be honest, we cannot do that, even if we’d like to do so), so ROR says: “yes, keep going, I’m ready - only what I should do it’s a core loading”. Now, we know that ROR is enabled, but not ready yet. We don’t want to pass any ES REST API requests, because of security reasons. So, ROR is returning “Waiting for ReadonlyREST start” during the core startup. When core is fully loaded, ROR handles requests with respect of ACL rules. But if core load fails, ROR . should start to return “ReadonlyREST failed to start” and obviously still blocks all requests (also security reasons).

You are receiving “Waiting for ReadonlyREST start” - IMO it’s ok. ROR is starting. IMO you cannot say yet that it doesn’t start. If so, you should get “ReadonlyREST failed to start”.

It’d be nice if you could send some logs produced during ES+ROR launching.