Elastic 5.5.2 Startup Error after Installing ROR - *Could not find required attribute 'readonlyrest'

Getting this error while starting up my ElasticSearch (5.5.2) Instance after configuring ROR for the first time on my EC2. My ElasticSearch YML is Unchanged. . I looked at a few other forum posts (esp around YML indentation). No matter what I try, I keep getting this error. I have correct Indentation - for some reason this post is not showing the indendation correctly in readonlyrest.ym. What am I missing?
PS : I noticed my elasticsearch.yml and readonlyrest.yml have different yml configuration pattern.

elasticsearch.yml

cluster.name: mycluster
node.data: false
node.master: false
node.ingest: false
search.remote.connect: false
node.name: MyHost
path.data:
path.logs:
ā€¦

readonlyrest.yml

readonlyrest:
access_control_rules:
- name: ā€œGlobal Accessā€
auth_key: ā€œelastic:changemeā€
verbosity: ā€œerrorā€

Error
[2018-03-14T16:33:19,439][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded module [parent-join]
[2018-03-14T16:33:19,439][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded module [percolator]
[2018-03-14T16:33:19,439][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded module [reindex]
[2018-03-14T16:33:19,439][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded module [transport-netty3]
[2018-03-14T16:33:19,439][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded module [transport-netty4]
[2018-03-14T16:33:19,440][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded plugin [discovery-ec2]
[2018-03-14T16:33:19,440][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded plugin [readonlyrest]
[2018-03-14T16:33:19,440][INFO ][o.e.p.PluginsService ] [ip-10-187-25-65] loaded plugin [repository-s3]
[2018-03-14T16:33:20,551][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2018-03-14T16:33:20,843][INFO ][o.e.d.DiscoveryModule ] [ip-10-187-25-65] using discovery type [zen]
[2018-03-14T16:33:21,087][INFO ][t.b.r.e.SettingsObservableImpl] Could not find settings in /etc/elasticsearch/readonlyrest.yml (/etc/elasticsearch/readonlyrest.yml)
[2018-03-14T16:33:21,183][INFO ][t.b.r.e.SettingsObservableImpl] Could not find settings in /etc/elasticsearch/readonlyrest.yml (/etc/elasticsearch/readonlyrest.yml)
[2018-03-14T16:33:23,418][INFO ][t.b.r.e.SettingsObservableImpl] Could not find settings in /etc/elasticsearch/readonlyrest.yml (/etc/elasticsearch/readonlyrest.yml)
[2018-03-14T16:33:23,420][INFO ][t.b.r.e.SettingsObservableImpl] Could not find settings in /etc/elasticsearch/readonlyrest.yml (/etc/elasticsearch/readonlyrest.yml)
[2018-03-14T16:33:23,421][INFO ][t.b.r.e.SettingsObservableImpl] Could not find settings in /etc/elasticsearch/readonlyrest.yml (/etc/elasticsearch/readonlyrest.yml)
[2018-03-14T16:33:23,601][ERROR][o.e.b.Bootstrap ] Guice Exception: tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ā€˜readonlyrestā€™
at tech.beshu.ror.commons.settings.RawSettings.req(RawSettings.java:84)
at tech.beshu.ror.commons.settings.RawSettings.inner(RawSettings.java:191)
at tech.beshu.ror.commons.settings.BasicSettings.(BasicSettings.java:84)

Something wierd is happening.
I have the readonlyrest.yml in the location
/etc/elasticsearch

But even if I rename this file to some junk value - I get the exact same error as described above. Could this be some permission issue?

ls -al /etc/elasticsearch/
total 44
drwxr-x---.  5 root elasticsearch  231 Mar 14 18:28 .
drwxr-xr-x. 90 root root          8192 Mar 14 17:55 ..
-rw-r--r--.  1 root root           320 Mar  9 15:43 curator.yml
drwxr-x---.  2 root elasticsearch   31 Mar  9 15:43 discovery-ec2
-rw-rw----.  1 root elasticsearch  695 Mar  9 15:43 elasticsearch.yml
-rw-rw----.  1 root elasticsearch 3098 Mar  9 15:43 jvm.options
-rw-rw----.  1 root elasticsearch 4456 Aug 14  2017 log4j2.properties
-rw-r-----.  1 root root           127 Mar 14 17:20 readonlyrest.yml
drwxr-x---.  2 root elasticsearch   31 Mar  9 15:43 repository-s3
-rw-r--r--.  1 root root           616 Mar  9 15:43 rollover_indices.yml
drwxr-x---.  2 root elasticsearch    6 Aug 14  2017 scripts
-rw-r--r--.  1 root root          1126 Mar  9 15:43 snapshot_indices.yml

Another Update : By changing permissions to give full access to /etc/elasticsearch/readonlyrest.yml
Iā€™m beginning to get this message: but it runs into hundreds of thousands of lines in the log and then errors out

[2018-03-14T19:04:57,772][INFO ][t.b.r.e.SettingsObservableImpl] Loaded good settings from /etc/elasticsearch/readonlyrest.yml
[2018-03-14T19:04:57,859][INFO ][t.b.r.e.SettingsObservableImpl] Loaded good settings from /etc/elasticsearch/readonlyrest.yml
[2018-03-14T19:04:57,869][INFO ][t.b.r.e.SettingsObservableImpl] Loaded good settings from /etc/elasticsearch/readonlyrest.yml
[2018-03-14T19:04:57,875][INFO ][t.b.r.e.SettingsObservableImpl] Loaded good settings from /etc/elasticsearch/readonlyrest.yml
[2018-03-14T19:04:57,882][INFO ][t.b.r.e.SettingsObservableImpl] Loaded good settings from /etc/elasticsearch/readonlyrest.yml

ā€¦

[2018-03-14T19:05:00,320][ERROR][o.e.b.Bootstrap ] Guice Exception: tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ā€˜readonlyrestā€™
_ at tech.beshu.ror.commons.settings.RawSettings.req(RawSettings.java:84)_
_ at tech.beshu.ror.commons.settings.RawSettings.inner(RawSettings.java:191)_

Hi @titan1978 welcome to the forum!

Does it still happen with this build?

https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.16.17-pre3/readonlyrest-1.16.17-pre3_es5.5.2.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ/20180314/eu-west-1/s3/aws4_request&X-Amz-Date=20180314T220501Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=a54a8b7cddefbfec6f10f6c03cdaa102f1dedeeab4549129b609d7f2146cbadb

@sscarduzio Thanks - It looks like it resolved - I missed adding the enable: true in the elasticsearch yml.
I might not have caught on initially - from the documentation - it felt like I had to add this in the ROR yml and not elastic yml.

Might make some sense to inform a user of this? IE The developer missed adding this property in the elastic ylm instead of this cryptic error stack?

Wait, in elasticsearch.yml you should only have to add:

http.type: ssl_netty4

All the rest should go in readonlyrest.yml

When I do that - I get this error when starting up ES.If I add the readonlyrest.enable: true in elasticsearch.yml then readonlyrest.yml loads (but ive posted another issue on this forums that occurs after this step)

readonlyrest.yml

readonlyrest:
  enable: true
  response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin

  access_control_rules:
    - name: "just that action from localhost"
    type: allow
    actions: ["cluster:monitor/main"]
    hosts: ["127.0.0.1"]

    - name: "Global Access"
    auth_key: "elastic:changeme"
    verbosity: "error"

   - name: "My Index Access"
   auth_key: "elastic:writeme"
   type: allow
   actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
   indices: ["myindex-*", "<no_index>]

error

[2018-03-15T17:47:12,818][INFO ][t.b.r.e.SSLTransportNetty4] Loaded good settings from /etc/elasticsearch/readonlyrest.yml
[2018-03-15T17:47:12,944][ERROR][o.e.b.Bootstrap ] Exception
tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ā€˜readonlyrestā€™
at tech.beshu.ror.commons.settings.RawSettings.req(RawSettings.java:84) ~[?:?]
at tech.beshu.ror.commons.settings.RawSettings.inner(RawSettings.java:191) ~[?:?]
at tech.beshu.ror.commons.settings.BasicSettings.(BasicSettings.java:84) ~[?:?]

@sscarduzio Another thing I noticed - in this forum which you have replied to - the original posters have indicated they were setting readonlyrest.enable: true in their elasticsearch.yml (generated by puppet)

Iā€™ve been REALLY struggling to get the ROR going and the only way I could pull it off was by doing the exact same thing in my elasticsearch.yml (without using the nett4 configuration you mentioned). Iā€™ve spent close to 2 days on this and without changing the elasticsearch.yml to add the readonlyrest.enable: true - i KEEP getting this error. Its a blackbox to me - would really like to know between the forum answer where they too were putting the readonlyrest.enable setting in their elasticsearch.yml - if I am doing something wrong?

UPDATE: Confirmed! Even though my consoles says readonlyrest.yml is ā€œloadedā€ - none of its configurations are being picked up. But If I start modifying the elasticsearch.yml and add those settings directly in there - i can start seeing the changes slowly take effect. Why wont the readonlyrest.yml DESPITE being detected by console - its configurations being ignored?

OK this is weird, let me try this myself. Can you confirm you are using 1.16.17 with 5.5.2 now?

UPDATED!
I upgraded to 1.16.17. Still getting the same errors if I DONT add the readonlyrest.enable etc etc directly into elasticsearch.yml.

elastic.yml

http.type: ssl_netty4

readonlyrest.yml

readonlyrest:
enable: true

access_control_rules:
- name "Block 1 - Allowing anything from localhost"
  hosts: [127.0.0.1]

- name: "MyApp Index Access"
  auth_key: "elastic:writeme"
  type: allow
  actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","indices:admin/get"]
  indices: ["myindex-*"]

error

[2018-03-15T20:23:10,751][INFO ][o.e.p.PluginsService     ] [ip-10-187-16-110] loaded plugin [readonlyrest]
[2018-03-15T20:23:10,751][INFO ][o.e.p.PluginsService     ] [ip-10-187-16-110] loaded plugin [repository-s3]
[2018-03-15T20:23:11,964][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2018-03-15T20:23:12,034][INFO ][t.b.r.e.SSLTransportNetty4] Loaded good settings from /etc/elasticsearch/readonlyrest.yml
[2018-03-15T20:23:12,158][ERROR][o.e.b.Bootstrap          ] Exception
tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute 'readonlyrest' in file cluster:{name=my-cluster},bootstrap:{memory_lock=true},transport:{type={default=netty4}},network:{host=_ec2:privateIpv4_},cloud:{aws={region=us-west-2, proxy={host=myproxy.kdc.myhost.com, port=8099}}, node={auto_attributes=true}},pidfile:/var/run/elasticsearch/elasticsearch.pid,node:{name=ip-10-187-16-110, attr={aws_availability_zone=us-west-2a}, data=false, ingest=false, master=false},path:{data=[/opt/mount/var/data/myapp/elasticsearch], logs=/opt/mount/var/log/myapp/elasticsearch, home=/usr/share/elasticsearch},default:{path={data=/var/lib/elasticsearch, logs=/var/log/elasticsearch, conf=/etc/elasticsearch}},search:{remote={connect=false}},discovery:{ec2={groups=sg-abcd, any_group=false, tag={Name=myapp-es-west-asg}, host_type=private_ip}, zen={minimum_master_nodes=2, hosts_provider=ec2}},action:{destructive_requires_name=true},client:{type=node},http:{type=ssl_netty4}
        at tech.beshu.ror.commons.settings.RawSettings.req(RawSettings.java:100) ~[?:?]
        at tech.beshu.ror.commons.settings.RawSettings.inner(RawSettings.java:207) ~[?:?]
        at tech.beshu.ror.commons.settings.BasicSettings.<init>(BasicSettings.java:84) ~[?:?]

Thanks @titan1978, I will try this myself and report.

Worked well for me. I think you have a mess with the indentation. See what I have here, and how itā€™s indented:

@sscarduzio Thanks - Feel a bit foolish - there was a COLON ( : ) missing between the in the name attribute in the readonlyrest.yml. Do you think the bootstrapping can be enhanced to explicetly throw an error when this happens (ie syntax parsing exception?) - currently the console gave false postive by saying ā€œLoaded Good settingsā€¦ā€. This would immensely help newbies who have no clue whats going on - especially helpful considering silly mistakes like these can take up unnecessary lost hours.

1 Like

Your sacrifice was not useless, I just fixed the error message ā€œloaded good settingsā€ with ā€œloaded data from fileā€, as it didnā€™t yet parse the YAML at that stage, and made sure we print the error message from the YAML parser

[2018-03-16T11:49:22,933][ERROR][t.b.r.e.SettingsObservableImpl] Cannot parse YAML: ScannerException:mapping values are not allowed here
 in 'string', line 4, column 12:
        enabled: true
               ^

 
readonlyrest:
    audit_collector true
    enabled: true
....
1 Like