Elasticsearch down after upgrade from 6.6.0 to 6.8.0

Users Feedback
1

Any idea, why I cannot start my cluster after an upgrade from 6.6.0 (readonlyrest 1.16.34_es6.6.0) to 6.8.0 (readonlyrest-1.18.7_es6.8.0)?

readonly config was working with 6.6.0, now I get:

org.elasticsearch.bootstrap.StartupException: StartingFailureException[Errors:
Malformed settings]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]
Caused by: tech.beshu.ror.es.StartingFailureException: Errors:
Malformed settings
at tech.beshu.ror.es.StartingFailureException.from(StartingFailureException.java:35) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.<init>(IndexLevelActionFilter.scala:62) ~[?:?]
at tech.beshu.ror.es.ReadonlyRestPlugin.lambda$createComponents$3(ReadonlyRestPlugin.java:126) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at tech.beshu.ror.es.ReadonlyRestPlugin.createComponents(ReadonlyRestPlugin.java:123) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$11(Node.java:472) ~[elasticsearch-6.8.0.jar:6.8.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1492) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:475) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.0.jar:6.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.0.jar:6.8.0]

malformed settings and nothing more is hard to find… Is there a fundamental change between 1.16 and 1.18 readonlyrest plugin?

Rolling back to 6.6.0 and the old readonlyrest plugin is working fine with the same config.
Thanks a lot.

Olaf

2

yes, now we detect more settings errors at config loading level. You can share your current ROR setting and I’ll help you with fixing it. Do you use in-index setttings?

3

Sure, here is the config, which is working in 6.6.0:

# rbac for elasticsearch
readonlyrest:
enable: ${READONLYREST_ENABLE}
response_if_req_forbidden: <h1>Forbidden</h1>

access_control_rules:

- name: data
  actions: ["indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","cluster:monitor/*"]
  indices: ["logstash-*", "netflow", "systemd-*", "syslog-*", ".kibana*", "virtual-*"]
  auth_key: data:pw

- name: jump
  actions: ["indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","cluster:monitor/*"]
  indices: ["jump-*"]
  auth_key: jump:pw

- name: jaeger
  actions: ["indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","cluster:monitor/*"]
  indices: ["jaeger-*"]
  auth_key: jaeger:pw

- name: viewer (read only, but can create dashboards)
  kibana_access: ro
  auth_key: viewer:pw
  indices: [".kibana*"]

- name: promuser
  actions: ["indices:data/read/*"]
  indices: ["logstash-*", "netflow", "systemd-*", "syslog-*", "jump-*", "virtual-*"]
  auth_key: promuser:pw

# admin user
- name: Admin
  auth_key: admin:pw

# deny access without a proper sso cert validated from the ingress - proxy definition see below
- name: no-sso
  type: forbid
  proxy_auth:
    proxy_auth_config: "ingress"
    users: ["anonymous"]

# if we get a user in the x-remote-user header, check if it has valid ldap groups and if: allow access - ldap definition see below
- name: sso-and-ldap
  type: allow
  proxy_auth:
    proxy_auth_config: "ingress"
    users: ["*"]
  ldap_authorization:
    name: "ldap1"
    groups: ["ADMIN_DOMAIN_USERS","ADMIN_MONITORING_USERS"]

# get the user from the x-remote-user header
proxy_auth_configs:
- name: ingress
  user_id_header: "X-REMOTE-USER"

# ldap connection definition
ldaps:
- name: ldap1
  host: "ldap-qa.testing"
  port: 636
  ssl_enabled: true
  ssl_trust_all_certs: true
  bind_dn: "CN=monitoring-service,CN=Users,DC=ad,DC=testing,DC=test,DC=de"
  bind_password: "pw"
  search_user_base_DN: "OU=Identities,DC=ad,DC=testing,DC=test,DC=de"
  user_id_attribute: "sAMAccountName"
  search_groups_base_DN: "OU=ADMIN,OU=OSDomains,OU=CCloud,DC=ad,DC=testing,DC=test,DC=de"
  unique_member_attribute: "member"
  connection_pool_size: 10
  connection_timeout_in_sec: 10
  request_timeout_in_sec: 10
  cache_ttl_in_sec: 300
4

ok, I see what is the problem - you need indentation of readonlyrest section:

readonlyrest:
  enable: ${READONLYREST_ENABLE}
  response_if_req_forbidden: <h1>Forbidden</h1>

  access_control_rules:

    - name: data
      actions: ...

I’ve checked and it works in our tests. Please, check if it also works for you.

1 Like
5

Still not working… I sm jumping back and forth between malformed and sections, which are wrong. I have no idea anymore, because your github page has different formats. Any idea, what’s wrong?

Thanks a lot Olaf

readonlyrest:
  enable: ${READONLYREST_ENABLE}
  response_if_req_forbidden: <h1>Forbidden</h1>

  access_control_rules:

    - name: data
      actions: ["indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","cluster:monitor/*"]
      indices: ["logstash-*", "netflow", "systemd-*", "syslog-*", ".kibana*"]
      auth_key: data:pw

    - name: Admin
      auth_key: admin:pw

    - name: no-sso
      type: forbid
      proxy_auth:
        proxy_auth_config: "ingress"
        users: ["anonymous"]

    - name: sso-and-ldap
      type: allow
      proxy_auth:
        proxy_auth_config: "ingress"
        users: ["*"]
      ldap_authorization:
        name: "ldap1"
        groups: ["ADMIN_DOMAIN_USERS","ADMIN_MONITORING_USERS"]

    proxy_auth_configs:
    - name: ingress
      user_id_header: "X-REMOTE-USER"

    ldaps:
    - name: ldap1
      host: "ldap-qa.testing"
      port: 636
      ssl_enabled: true
      ssl_trust_all_certs: true
      bind_dn: "CN=monitoring-service,CN=Users,DC=ad,DC=testing,DC=test,DC=de"
      bind_password: "pw"
      search_user_base_DN: "OU=Identities,DC=ad,DC=testing,DC=test,DC=de"
      user_id_attribute: "sAMAccountName"
      search_groups_base_DN: "OU=ADMIN,OU=OSDomains,OU=testing,DC=ad,DC=test,DC=test,DC=de"
      unique_member_attribute: "member"
      connection_pool_size: 10
      connection_timeout_in_sec: 10
      request_timeout_in_sec: 10
      cache_ttl_in_sec: 300
6

Even with only this section, it’s failing with malformed… I am lost.

readonlyrest:
  enable: ${READONLYREST_ENABLE}
  response_if_req_forbidden: <h1>Forbidden</h1>

  access_control_rules:

    - name: data
      actions: ["indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create","cluster:monitor/*"]
      indices: ["logstash-*", "netflow", "systemd-*", "syslog-*", ".kibana*"]
      auth_key: data:pw
7

Olaf, I think you should send the actual YML file (not copypaste). Can you attach it? If not, send us an email: support AT readonlyrest dot com.

8

That entry is failing during config parse. The env setting is there, but the parse is not accepting this line.
If I hardcode it, it is working …

enable: ${READONLYREST_ENABLE}

Regards,

Olaf

9

ok, I know what is a problem. Non-string ENV variables was resolved wrongly.
The fix is ready: fixed: non-string ENV variables proper resolving by coutoPL · Pull Request #514 · sscarduzio/elasticsearch-readonlyrest-plugin · GitHub

Tomorrow will send you a build with this fix applied.

10

@Kuckkuck here is the newest build:

https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.18.8-pre4/readonlyrest-1.18.8-pre4_es6.8.0.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20191010/eu-west-1/s3/aws4_request&X-Amz-Date=20191010T153611Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f9004ef883de256b629247f39ce13866e7c92a90472eb7811b3b033d63b612eb

Please, check if this one works as expected

1 Like
11

Great - it works. Will it be available for newer elasticsearch versions as well? Because elastic 6.8 is just one step on the 7.x upgrade path.

Thanks a lot.

1 Like
12

Yes, obviously. We support still support all ES versions starting from 5.1.1. We should release 1.18.8 soon, so you can download version you need from here: Download (UNIVERSAL) - ReadonlyREST.