Running ROR alongside xpack on ES 6.0 problem

Users Feedback
#1

Problem: keep both xpack free and readonlyrest running

  • currently readonlyrest breaks elasticsearch with xpack and can’t run

List the ingredients:

  • software versions: ES 6.0.0, ROR 1.16.18_es6.0.0

instructions to repeat

  • xpack security is disabled (xpack.security.enabled : false)
  • ROR plugin installed (and configured)
  • elasticsearch service restart
  • message from elasticsearch log:
  1. Error injecting constructor, tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ‘readonlyrest’ in file pidfile:/var/run/elasticsearch/elasticsearch.pid,cluster:{name=elasticsearch},node:{attr={ml={max_open_jobs=10, enabled=true}}, name=log1tln},path:{data=[/log_data/elasticsearch/data], logs=/log_data/elasticsearch/logs, home=/usr/share/elasticsearch},thread_pool:{search={queue_size=1200}},client:{type=node},http:{type={default=netty4}},transport:{type={default=netty4}},xpack:{security={enabled=false}},network:{host=0.0.0.0}
    at tech.beshu.ror.es.SettingsObservableImpl.(Unknown Source)
    while locating tech.beshu.ror.es.SettingsObservableImpl
    for parameter 5 at tech.beshu.ror.es.IndexLevelActionFilter.(Unknown Source)
    while locating tech.beshu.ror.es.IndexLevelActionFilter
    while locating org.elasticsearch.action.support.ActionFilter annotated with @org.elasticsearch.common.inject.multibindings.Element(setName=,uniqueId=7)
    at unknown
    while locating java.util.Set<org.elasticsearch.action.support.ActionFilter>
    for parameter 0 at org.elasticsearch.action.support.ActionFilters.(Unknown Source)
    while locating org.elasticsearch.action.support.ActionFilters
    for parameter 4 at org.elasticsearch.xpack.persistent.RemovePersistentTaskAction$TransportAction.(Unknown Source)
    while locating org.elasticsearch.xpack.persistent.RemovePersistentTaskAction$TransportAction
    Caused by: tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ‘readonlyrest’ in file pidfile:/var/run/elasticsearch/elasticsearch.pid,cluster:{name=elasticsearch},node:{attr={ml={max_open_jobs=10, enabled=true}}, name=log1tln},path:{data=[/log_data/elasticsearch/data], logs=/log_data/elasticsearch/logs, home=/usr/share/elasticsearch},thread_pool:{search={queue_size=1200}},client:{type=node},http:{type={default=netty4}},transport:{type={default=netty4}},xpack:{security={enabled=false}},network:{host=0.0.0.0}
    at tech.beshu.ror.commons.settings.RawSettings.req(RawSettings.java:110)
    at tech.beshu.ror.commons.settings.RawSettings.inner(RawSettings.java:219)
    at tech.beshu.ror.commons.settings.BasicSettings.(BasicSettings.java:84)
    at tech.beshu.ror.commons.settings.BasicSettings.fromFile(BasicSettings.java:156)
    at tech.beshu.ror.es.SettingsObservableImpl.(SettingsObservableImpl.java:57)
    at sun.reflect.GeneratedConstructorAccessor30.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
    at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
    at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
    at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
    at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
    at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
    at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
    at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
    at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
    at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
    at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:58)
    at org.elasticsearch.common.inject.InjectorImpl$4$1.call(InjectorImpl.java:762)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
    at org.elasticsearch.common.inject.InjectorImpl$4.get(InjectorImpl.java:757)
    at org.elasticsearch.common.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:275)
    at org.elasticsearch.common.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:200)
    at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
    at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
    at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
    at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
    at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
    at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
    at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
    at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
    at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
    at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
    at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
    at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
    at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191)
    at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183)
    at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818)
    at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183)
    at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:176)
    at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161)
    at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96)
    at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
    at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
    at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42)
    at org.elasticsearch.node.Node.(Node.java:492)
    at org.elasticsearch.node.Node.(Node.java:245)
    at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
    at org.elasticsearch.cli.Command.main(Command.java:90)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)
#2

Xpack does not have anything to do with this error, you just didn’t create and populate readonlyrest.yml

#3

config file actually exist as is pretty simple, maybe there is something wrong:
cat /etc/elasticsearch/readonlyrest.yml

readonlyrest:

    audit_collector: true

    access_control_rules:

    - name: "Kibana read"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/read/*"]
      kibana_access: ro
      kibana_index: .kibana
      groups: ["Read Only"]
      verbosity: error

    - name: "Logstash write"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/write/*"]
      hosts_local: ["127.0.0.1", "localhost"]
      groups: ["Logstash"]
      verbosity: error

    users:

    - username: kibana
      auth_key: kibana:****
      groups: ["Read Only"]

    - username: logstash_internal
      auth_key: logstash_internal:****
      groups: ["Logstash"]
#4

Hi

here that I would change : (based on mine running on cluster, along with x-pack enabled (not security) on 6.2.1

readonlyrest:
    enable: true
    response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin
    audit_collector: true
    prompt_for_basic_auth: false
    

    access_control_rules:
      - name: Kibana Server (we trust this server side component, full access granted via HTTP authentication)
      # it is for kibana server getting access to elasticsearch only
      auth_key_sha1: f435b46456464646273028f
      type: allow
      hosts: [kibana_IP]

    - name: "x-pack monitoring"
      #credentials and config must be done in kibana.yml, elasticsearch.yml, xxxxbeats.yml, logstash.yml
      auth_key_sha1: 6fce41670a6546456544655447
      type: allow
      actions: ["cluster:monitor/*", "indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create", "cluster:admin/ingest/pipeline/*","cluster:admin/xpack/monitoring/*"]
      indices: [".monitoring-*"]
      verbosity: info

    - name: "Kibana read"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/read/*"]
      kibana_access: ro
      kibana_index: .kibana
      groups: ["Read Only"]
      verbosity: error

    - name: "Logstash write"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/write/*"]
      hosts_local: ["127.0.0.1", "localhost"]
      groups: ["Logstash"]
      verbosity: error

    users:

    - username: kibana
      auth_key: kibana:****
      groups: ["Read Only"]

    - username: logstash_internal
      auth_key: logstash_internal:****
      groups: ["Logstash"]

as I did not see the block for kibana server , i added it.
I added the rule for xpack monitoring. feel free to adapt

#5

Hi @kalev, sorry for the delayed response.

It looks like ROR is not picking up the readonlyrest.yml file, so it’s falling back to elasticsearch.yml and does not find the root of ROR settings in it (of course). Could you please make sure:

  • You have the latest ROR for ES installed
  • The file readonlyrest.yml is in the same directory with your elasticsearch.yml
  • The UNIX permissions are appropriate for the ES process to read the file
  • Grep the ES logs and find this line “Read data from”, this shows the path ROR looks for the readonlyrest.yml file.