RoR not getting started in cluster

Hi Team,

I have set up a ELK cluster with 3 elastricsearch nodes, 3 logstash nodes and one kibana node. I was successfully able to do the basic authentication using RoR plugin. (I am using free version of RoR)

So, as a next try i was trying to do the LDAP authentication. Below is the readonlyrest.yml at my end in all 3 elasticsearch nodes in elasticsearch directory,

Note:- I have replace actual LDAP details with example ones.

readonlyrest:
    enable: true
    access_control_rules:
     - name: "LDAP”
       type: allow
       ldap_auth:
         name: “MyLDAP"
         groups: [“ABC”,”XYZ”,”YYY”,”AAA”]
       indices: ["*"]

    ldaps:
     - name: "LDAP"
       host: “abc.def.com"
       port: 389
       ssl_enabled: false
       ssl_trust_all_certs: false
       bind_dn: "cn=ABC,cn=XYZ,dc=abc,dc=def,dc=com"
       search_user_base_DN: "ou=ABC,ou=XYZ,ou=YYY,ou=AAA,dc=def,dc=com"
       user_id_attribute: "uid"
       search_groups_base_DN: "ou=AAA,dc=abc,dc=def,dc=com"
       unique_member_attribute: "uniqueMember"
       connection_pool_size: 10
       connection_timeout_in_sec: 10
       request_timeout_in_sec: 10
       cache_ttl_in_sec: 60
       group_search_filter: "(objectClass=group)(cn=application*)"
       group_name_attribute: "cn"

But when i restart the elasticsearch service, nodes were able to start but RoR is not. Things get stuck at below, (logs from one of my elasticsearch node)

Blockquote [2020-01-28T01:08:45,870][INFO ][t.b.r.c.RorSsl$ ] [es-node-03] Cannot find SSL configuration is elasticsearch.yml, trying: /etc/elasticsearch/readonlyrest.yml
[2020-01-28T01:08:45,873][INFO ][t.b.r.b.Ror$ ] [es-node-03] [CLUSTERWIDE SETTINGS] Loading ReadonlyREST settings from index …
[2020-01-28T01:08:46,066][DEBUG][o.e.a.ActionModule ] [es-node-03] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2020-01-28T01:08:46,696][INFO ][o.e.d.DiscoveryModule ] [es-node-03] using discovery type [zen] and seed hosts providers [settings]
[2020-01-28T01:08:47,367][INFO ][o.e.n.Node ] [es-node-03] initialized
[2020-01-28T01:08:47,367][INFO ][o.e.n.Node ] [es-node-03] starting …
[2020-01-28T01:08:47,489][INFO ][o.e.t.TransportService ] [es-node-03] publish_address {10.156.0.158:9300}, bound_addresses {10.156.0.158:9300}
[2020-01-28T01:08:47,496][INFO ][o.e.b.BootstrapChecks ] [es-node-03] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-01-28T01:08:47,503][INFO ][o.e.c.c.Coordinator ] [es-node-03] cluster UUID [oakx2Z2wS_2jYK3zt2Kh9w]
[2020-01-28T01:08:48,413][INFO ][o.e.c.s.ClusterApplierService] [es-node-03] master node changed {previous [], current [{es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{tFC2cff7SSCh-gAAVtR3Xg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, ml.max_open_jobs=20, xpack.installed=true}]}, added {{es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{tFC2cff7SSCh-gAAVtR3Xg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, ml.max_open_jobs=20, xpack.installed=true},{es-node-02}{WQc55XBwRaaV2-A_3-nDcg}{RouNGoGPS5mPQR76DZR2fA}{10.156.0.157}{10.156.0.157:9300}{dim}{ml.machine_memory=33566867456, ml.max_open_jobs=20, xpack.installed=true},}, term: 104, version: 5334, reason: ApplyCommitRequest{term=104, version=5334, sourceNode={es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{tFC2cff7SSCh-gAAVtR3Xg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, ml.max_open_jobs=20, xpack.installed=true}}
[2020-01-28T01:08:48,425][INFO ][o.e.c.s.ClusterSettings ] [es-node-03] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-01-28T01:08:48,752][INFO ][o.e.l.LicenseService ] [es-node-03] license [1bcd9dd7-3a24-475a-a401-a97df98ac161] mode [basic] - valid
[2020-01-28T01:08:48,767][INFO ][o.e.h.AbstractHttpServerTransport] [es-node-03] publish_address {10.156.0.158:9200}, bound_addresses {10.156.0.158:9200}
[2020-01-28T01:08:48,768][INFO ][o.e.n.Node ] [es-node-03] started
[2020-01-28T01:09:06,912][INFO ][t.b.r.b.Ror$ ] [es-node-03] Loading ReadonlyREST settings from index failed: cannot find index
[2020-01-28T01:09:06,913][INFO ][t.b.r.b.Ror$ ] [es-node-03] Loading ReadonlyREST settings from file: /etc/elasticsearch/readonlyrest.yml

Plus Kibana logs show bel;ow,

Blockquote{“type”:“log”,"@timestamp":“2020-01-28T07:15:17Z”,“tags”:[“error”,“task_manager”],“pid”:1193,“message”:“Failed to poll for work: [undefined] Waiting for ReadonlyREST start :: {“path”:”/.kibana_task_manager/_search",“query”:{“ignore_unavailable”:true},“body”:"{\“query\”:{\“bool\”:{\“must\”:[{\“term\”:{\“type\”:\“task\”}},{\“bool\”:{\“must\”:[{\“terms\”:{\“task.taskType\”:[\“maps_telemetry\”,\“vis_telemetry\”,\“actions:.server-log\”,\“actions:.slack\”,\“actions:.email\”]}},{\“range\”:{\“task.attempts\”:{\“lte\”:3}}},{\“range\”:{\“task.runAt\”:{\“lte\”:\“now\”}}},{\“range\”:{\“kibana.apiVersion\”:{\“lte\”:1}}}]}}]}},\“size\”:10,\“sort\”:{\“task.runAt\”:{\“order\”:\“asc\”}},\“seq_no_primary_term\”:true}",“statusCode”:503,“response”:"{\“error\”:{\“root_cause\”:[{\“reason\”:\“Waiting for ReadonlyREST start\”}],\“reason\”:\“Waiting for ReadonlyREST start\”,\“status\”:503}}"}"}

Don’t know why RoR is not getting started. I am totally stuck here. No informative error logs are shown for RoR.

Please help me as soon as possible. Awaiting for a speedy reply.

Thanks,
Pankaj

Did you see the doc: https://github.com/beshu-tech/readonlyrest-docs/blob/master/kibana.md#loading-settings-order-of-precedence?

If ROR is not able to load settings from index it fallbacks to readonlyrest.yml.
This is what you see in log:

Loading ReadonlyREST settings from file: /etc/elasticsearch/readonlyrest.yml

Sadly, you didn’t show us more logs, following the one above.

After that you should find logs like this one:

[INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [ROR1_1] ADDING BLOCK: ...

and then:

[INFO ][t.b.r.b.RorInstance ] [ROR1_1] Readonly REST plugin core was loaded ...

ROR starts asynchronously, so until it starts the 503, Waiting for ReadonlyREST start response is sent