Index settings not found - timeout at node startup

Hi

I had installed readonlyrest plugin ver 1.17.6_es6.7.1 on one of the nodes in cluster, configured via kibana plugin - everything worked perfectly. Now, I’m trying to install plugin on another node and getting error:

[CLUSTERWIDE SETTINGS] index settings not found. Will keep on using the local YAML file. Learn more about clusterwide settings at https://readonlyrest.com/pro.html

Index .readonlyrest is present on cluster - the other node with ror plugin has no problem reading it.

I have another cluster with ror plugin installed on all nodes, working like a charm. Both clusters have nearly identical configurations, run on comparable hardware. The only diffrence is that troubling cluster have a lot more indices, so restarting a node takes a little longer. I have tried turn on debug, but no new info related to issue appeared in logs. I have tried restart node a few times, with and without load on cluster. Still no results. The same with version updated to 1.18.0_es6.7.1. I there any timeout build in ror plugin for detecting settings index?

Below is revelant log fragment:

[2019-05-24T05:22:00,874][INFO ][t.b.r.e.IndexLevelActionFilter] [elastic-XXXXXXXXXXXX] Settings observer refreshing...
[2019-05-24T05:22:02,510][INFO ][t.b.r.a.f.CoreFactory    ] [elastic-XXXXXXXXXXXX] ADDING BLOCK:        { name: '::ADMIN::', policy: ALLOW, rules: [auth_key_sha256,kibana_access]
[2019-05-24T05:22:02,510][INFO ][t.b.r.a.f.CoreFactory    ] [elastic-XXXXXXXXXXXX] ADDING BLOCK:        { name: 'Allow all from 10.36.0.0/16 (elastic/logstash)', policy: ALLOW, rules: [hosts]
[2019-05-24T05:22:02,530][INFO ][t.b.r.e.IndexLevelActionFilter] [elastic-XXXXXXXXXXXX] Configuration reloaded - ReadonlyREST enabled
[2019-05-24T05:22:02,530][INFO ][t.b.r.e.IndexLevelActionFilter] [elastic-XXXXXXXXXXXX] Readonly REST plugin was loaded...
[2019-05-24T05:22:02,758][DEBUG][o.e.a.ActionModule       ] [elastic-XXXXXXXXXXXX] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2019-05-24T05:22:03,537][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:04,538][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:05,538][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:06,539][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:07,539][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:08,540][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:09,541][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:10,541][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:11,542][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:12,097][INFO ][o.e.d.DiscoveryModule    ] [elastic-XXXXXXXXXXXX] using discovery type [zen] and host providers [settings]
[2019-05-24T05:22:12,542][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:13,087][INFO ][o.e.n.Node               ] [elastic-XXXXXXXXXXXX] initialized
[2019-05-24T05:22:13,087][INFO ][o.e.n.Node               ] [elastic-XXXXXXXXXXXX] starting ...
[2019-05-24T05:22:13,252][INFO ][o.e.t.TransportService   ] [elastic-XXXXXXXXXXXX] publish_address {10.36.1.2:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {10.36.1.2:9300}
[2019-05-24T05:22:13,552][INFO ][t.b.r.s.SettingsPoller   ] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2019-05-24T05:22:13,779][INFO ][o.e.b.BootstrapChecks    ] [elastic-XXXXXXXXXXXX] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-05-24T05:22:14,555][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [elastic-XXXXXXXXXXXX] no known master node, scheduling a retry
[2019-05-24T05:22:17,895][INFO ][o.e.c.s.ClusterApplierService] [elastic-XXXXXXXXXXXX] detected_master {elastic-XXXXXXXXXXXX}....
[2019-05-24T05:22:18,502][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [cluster.remote.elastic-log.seeds] from [[]] to [["elastic....
[2019-05-24T05:22:18,504][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [cluster.remote.elastic-log.seeds] from [[]] to [["elastic....
[2019-05-24T05:22:18,513][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [3]
[2019-05-24T05:22:18,513][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [8]
[2019-05-24T05:22:18,513][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [cluster.routing.allocation.node_concurrent_outgoing_recoveries] from [2] to [8]
[2019-05-24T05:22:18,513][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [indices.breaker.total.limit] from [70%] to [80%]
[2019-05-24T05:22:18,513][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [indices.breaker.request.limit] from [60%] to [65%]
[2019-05-24T05:22:18,514][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [elastic-XXXXXXXXXXXX] Updated breaker settings request: [request,type=MEMORY,limit=14599877427/13.5gb,overhead=1.0]
[2019-05-24T05:22:18,514][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2019-05-24T05:22:18,530][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [cluster.remote.elastic-log.skip_unavailable] from [false] to [true]
[2019-05-24T05:22:18,531][INFO ][o.e.c.s.ClusterSettings  ] [elastic-XXXXXXXXXXXX] updating [indices.recovery.max_bytes_per_sec] from [40mb] to [200mb]
[2019-05-24T05:22:20,094][INFO ][o.e.l.LicenseService     ] [elastic-XXXXXXXXXXXX] license [7d16fea0-4680-4086-8cac-6cdb30dcbf67] mode [basic] - valid
[2019-05-24T05:22:20,135][INFO ][o.e.h.n.Netty4HttpServerTransport] [elastic-XXXXXXXXXXXX] publish_address {10.36.1.2:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.36.1.2:9200}
[2019-05-24T05:22:20,135][INFO ][o.e.n.Node               ] [elastic-XXXXXXXXXXXX] started
[2019-05-24T05:22:20,432][INFO ][t.b.r.e.SettingsObservableImpl] [elastic-XXXXXXXXXXXX] [CLUSTERWIDE SETTINGS] index settings not found. Will keep on using the local YAML file. Learn more about clusterwide settings at https://readonlyrest.com/pro.html

A refactoring to this part of code is currently in the making.
cc/ @coutoPL