Problem with ES plugin in the 15 nodes cluster

1

As you know, I tested ES and KBN in one cluster - one node env. But now we tried moving to production which has one cluster 15 nodes (3 masters, 3 coordinating, 3 ingest, 6 data ) we installed ROR on primary master and added all the nodes IP to allow all yet ROR failed. I went back to the testing env and created another node so now there is 2 nodes in the cluster, I installed ROR on both and give both 2 IP addresses to allow all and it is working. Now my question is on prod should we install ROR on 3 masters only or we need to install the plugin on all 15 for the cluster to work?

2

Hi @nan008!
What do you mean with “failed”? Can you define the expected result, the actual result and provide some logs and settings?

3

Consider that ROR is policing the REST interface of Elasticsearch, not the transport one. Therefore, no need to whitelist cluster nodes.

4

Failed means in our case:

PROD test:
ROR is installed on Master only
Master is starting
Other nodes are failing to join in the cluster (at that point we thought it is x-pack for monitoring so we uninstalled it) and you are getting one big java error (unfortunately I cannot give you the exact error as we are deleting the logs on the restart)

Master 1 has ROR and yml Allow blocks

    access_control_rules:
    - name: "Accept all requests for indices"
      type: allow
      hosts: ["elastic-m1", "elastic-m2", "elastic-m3", "elastic-i1", "elastic-i2", "elastic-i3", "elastic-c1", "elastic-c2", "elastic-c3", "elastic-d1", "elastic-d2", "elastic-d3", "elastic-d4", "elastic-d5", "elastic-d6"]

DEV testing:
Master starting (elastic-a)
ingest node starting (elastic-b)

On both nodes yml has this allow block

    access_control_rules:
    - name: "Accept all requests for indices"
      type: allow
      hosts: ["elastic-a", "elastic-b"]

The expected result would be all the nodes joining the cluster on ROR site. Also on kibana in Prod env, u can use any user and kibana is still allowing the login without any error and elasticsearch log is not reacting to the Kibana login.

I was expecting


[2017-06-06T13:50:24,174][INFO ][o.e.p.r.a.ACL            ]  no block has matched, forbidding by default:
 { ID:478731944-1571305713, TYP:NodesInfoRequest, USR:r32re2r2qtf2wr(?), 
BRS:false, ACT:cluster:monitor/nodes/info, OA:kibana, IDX:<N/A>, MET:GET,
 PTH:/_nodes/_local, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,content-length,Host,  
HIS:[Dev->[kibana_access->true, indices->true, groups->false]], [Accept all requests for indices->[hosts->false]], 
[Kibana Server->[auth_key->false]], [Accept requests for elastic-head->[hosts->false]] }

but the is no reaction in elastic log like ROR is not working and kibana is fully open

So I was planning to install it on the 3 masters in prod, I am hoping I do not need to be installing ROR on all 15 nodes

5

Well… That’s a problem :frowning: I have nothing to work on now!

It seems ROR can’t initialise properly. But that’s all I know at this point without the stack trace. All I can say is to double check the yaml syntax and be aware that x-pack.security is not compatible with ReadonlyREST. I.e.

xpack.security.enabled: false
6

We are giving it another go today in PROD so we might get the error you need. My question really is, do you think we have to install the ROR plugin and yml config on every node. Or only on 3 masters?

We do not want x-pack security only monitoring which is actually working for us in Testing env with 2 nodes (both have ROR installed and both have ROR config in yml), one kibana (KBN installed), one logstash and one filebeat.

7

Yes unfortunately there’s not much way around trying again :confused:

Q: In what nodes should I install ROR for ES?

A: If you use ROR in the scope of Kibana alone, install ROR only in the ES nodes that are referred in your Kibana instances’ settings.
And this suggestion descends from the fact that ROR plugin for ES only modifies the behaviour of the REST API in Elasticsearch.