Waiting for ReadonlyREST start

Hi Team,

I have installed ROR on my Elastic cluster having 3 nodes and have below config file for readonlyrest.yml


readonlyrest:
        enable: true
        access_control_rules:
        - name: "Basic Authentication to get started with"
        indices: ["*"]
        type: allow
        auth_key: yyyy:xxxxxxxx

All 3 nodes are up and running but when i hit IP:9200 in browser getting below error,

{“message”:“Waiting for ReadonlyREST start: [undefined] Waiting for ReadonlyREST start”,“statusCode”:503,“error”:“Service Unavailable”}

Error logs,

[2019-11-22T05:34:26,880][INFO ][t.b.r.b.Ror$ ] [cs-es-node-01] [CLUSTERWIDE SETTINGS] Loading ReadonlyREST settings from index …
[2019-11-22T05:34:27,037][DEBUG][o.e.a.ActionModule ] [cs-es-node-01] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2019-11-22T05:34:27,439][INFO ][o.e.d.DiscoveryModule ] [cs-es-node-01] using discovery type [zen] and seed hosts providers [settings]
[2019-11-22T05:34:28,144][INFO ][o.e.n.Node ] [cs-es-node-01] initialized
[2019-11-22T05:34:28,144][INFO ][o.e.n.Node ] [cs-es-node-01] starting …
[2019-11-22T05:34:28,261][INFO ][o.e.t.TransportService ] [cs-es-node-01] publish_address {10.156.0.156:9300}, bound_addresses {10.156.0.156:9300}
[2019-11-22T05:34:28,268][INFO ][o.e.b.BootstrapChecks ] [cs-es-node-01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-11-22T05:34:28,274][INFO ][o.e.c.c.Coordinator ] [cs-es-node-01] cluster UUID [oakx2Z2wS_2jYK3zt2Kh9w]
[2019-11-22T05:34:28,564][INFO ][o.e.c.c.JoinHelper ] [cs-es-node-01] failed to join {cs-es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{WxpFpG3JTcW0ZWl4lrfdsg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={cs-es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{WxpFpG3JTcW0ZWl4lrfdsg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=9, lastAcceptedTerm=8, lastAcceptedVersion=1234, sourceNode={cs-es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{WxpFpG3JTcW0ZWl4lrfdsg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={cs-es-node-01}{AXJuaO_XQDCaw2LImf_A-g}{WxpFpG3JTcW0ZWl4lrfdsg}{10.156.0.156}{10.156.0.156:9300}{dim}{ml.machine_memory=33566875648, xpack.installed=true, ml.max_open_jobs=20}}]}
org.elasticsearch.transport.RemoteTransportException: [cs-es-node-01][10.156.0.156:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:477) [elasticsearch-7.3.2.jar:7.3.2]
at java.util.HashMap$Values.forEach(HashMap.java:976) [?:?]
at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:477) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:606) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:243) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-11-22T05:34:28,821][INFO ][o.e.c.s.ClusterApplierService] [cs-es-node-01] master node changed {previous [], current [{cs-es-node-02}{WQc55XBwRaaV2-A_3-nDcg}{NC1h9j5hRLe8v_pZSW4k4w}{10.156.0.157}{10.156.0.157:9300}{dim}{ml.machine_memory=33566867456, ml.max_open_jobs=20, xpack.installed=true}]}, added {{cs-es-node-02}{WQc55XBwRaaV2-A_3-nDcg}{NC1h9j5hRLe8v_pZSW4k4w}{10.156.0.157}{10.156.0.157:9300}{dim}{ml.machine_memory=33566867456, ml.max_open_jobs=20, xpack.installed=true},}, term: 10, version: 1235, reason: ApplyCommitRequest{term=10, version=1235, sourceNode={cs-es-node-02}{WQc55XBwRaaV2-A_3-nDcg}{NC1h9j5hRLe8v_pZSW4k4w}{10.156.0.157}{10.156.0.157:9300}{dim}{ml.machine_memory=33566867456, ml.max_open_jobs=20, xpack.installed=true}}
[2019-11-22T05:34:28,847][INFO ][o.e.h.AbstractHttpServerTransport] [cs-es-node-01] publish_address {10.156.0.156:9200}, bound_addresses {10.156.0.156:9200}
[2019-11-22T05:34:28,848][INFO ][o.e.n.Node ] [cs-es-node-01] started
[2019-11-22T05:34:47,913][INFO ][t.b.r.b.Ror$ ] [cs-es-node-01] Loading ReadonlyREST settings from index failed: cannot find index
[2019-11-22T05:34:47,914][INFO ][t.b.r.b.Ror$ ] [cs-es-node-01] Loading ReadonlyREST settings from file: /etc/elasticsearch/readonlyrest.yml

Please help in getting this sorted out asap.

Br
Pankaj N

Hi @pankaj can you share what version of ROR have you installed? So we can try to reproduce this.
BTW we’ll make sure we print the plugin versions in the logs in the upcoming versions.

Hi Simone,

Thanks for your reply. The ROR version i have used is 7.3.2.
BTW i was able to solve my issue and frankly saying i don’t know how :slight_smile: as i was trying so many things in parallel. One thing that i want to highlight, why RoR logo or something like that is not visible in basic authentication, as I have done? Any specific reason?

Also, can you please help me with the configuration of how to use Active Directory authentication on my cluster using RoR? I have running AD at my side and want to give authentication from the AD server.

Br
Pankaj N

Hi Pankai, glad you made your cluster work.

The ROR version should be something like “18.8.8”. The 7.3.2 looks like the Kibana/ES version.

Anyways, if you installed ROR for ES and Kibana, you should add this configuration:

in readonlyrest.yml

readonlyrest:
  enable: true
  access_control_rules:
  - name: "::KIBANA_SERVER::"
    auth_key: "kibana:kibana"

 - name: "Basic Authentication to get started with"
          auth_key: yyyy:xxxxxxxx

in config/kibana.yml

# Kibana server use ::KIBANA-SRV:: credentials
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"

Hi Simone,

RoR version readonlyrest-1.18.8 is installed in my cluster.

Also, the configuration you shared is already done at my end. I just want to know the configuration to do the Active Directory authentication, using RoR.

Awaiting for the same. Thanks!

Br
Pankaj N

ROR at startup checks if there is .readonlyrest index with settings. But it tries to fetch document from the index several times before it gives up. During that time ROR is going to return 503: Waiting for ReadonlyREST start. Than after in-index settings loading failure, ROR tries to load settings from your local file.

Hi

Thanks for your reply.
If it is so, i suggest to add some more info for the users, instead of “{“message”:“Waiting for ReadonlyREST start: [undefined] Waiting for ReadonlyREST start”,“statusCode”:503,“error”:“Service Unavailable”}”.

When i see this message on the browser, I personally felt that there is something wrong with the RoR plugin.

Br
Pankaj N

Guys also help me here, “with the configuration to do the Active Directory authentication, using RoR.”

Thanks!

thanks for your feedback. We’ll think about more descriptive and less misleading message.

According to AD authentication, you should check first following links:

Hi,

Thanks for your help.

WIll check the url shared.

Br
Pankaj N