Running ROR alongside xpack on ES 6.0 problem

kalev · April 17, 2018, 7:52am

Problem: keep both xpack free and readonlyrest running

currently readonlyrest breaks elasticsearch with xpack and can’t run

List the ingredients:

software versions: ES 6.0.0, ROR 1.16.18_es6.0.0

instructions to repeat

xpack security is disabled (xpack.security.enabled : false)
ROR plugin installed (and configured)
elasticsearch service restart
message from elasticsearch log:

Error injecting constructor, tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ‘readonlyrest’ in file pidfile:/var/run/elasticsearch/elasticsearch.pid,cluster:{name=elasticsearch},node:{attr={ml={max_open_jobs=10, enabled=true}}, name=log1tln},path:{data=[/log_data/elasticsearch/data], logs=/log_data/elasticsearch/logs, home=/usr/share/elasticsearch},thread_pool:{search={queue_size=1200}},client:{type=node},http:{type={default=netty4}},transport:{type={default=netty4}},xpack:{security={enabled=false}},network:{host=0.0.0.0}
at tech.beshu.ror.es.SettingsObservableImpl.(Unknown Source)
while locating tech.beshu.ror.es.SettingsObservableImpl
for parameter 5 at tech.beshu.ror.es.IndexLevelActionFilter.(Unknown Source)
while locating tech.beshu.ror.es.IndexLevelActionFilter
while locating org.elasticsearch.action.support.ActionFilter annotated with @org.elasticsearch.common.inject.multibindings.Element(setName=,uniqueId=7)
at unknown
while locating java.util.Set<org.elasticsearch.action.support.ActionFilter>
for parameter 0 at org.elasticsearch.action.support.ActionFilters.(Unknown Source)
while locating org.elasticsearch.action.support.ActionFilters
for parameter 4 at org.elasticsearch.xpack.persistent.RemovePersistentTaskAction$TransportAction.(Unknown Source)
while locating org.elasticsearch.xpack.persistent.RemovePersistentTaskAction$TransportAction
Caused by: tech.beshu.ror.commons.settings.SettingsMalformedException: Could not find required attribute ‘readonlyrest’ in file pidfile:/var/run/elasticsearch/elasticsearch.pid,cluster:{name=elasticsearch},node:{attr={ml={max_open_jobs=10, enabled=true}}, name=log1tln},path:{data=[/log_data/elasticsearch/data], logs=/log_data/elasticsearch/logs, home=/usr/share/elasticsearch},thread_pool:{search={queue_size=1200}},client:{type=node},http:{type={default=netty4}},transport:{type={default=netty4}},xpack:{security={enabled=false}},network:{host=0.0.0.0}
at tech.beshu.ror.commons.settings.RawSettings.req(RawSettings.java:110)
at tech.beshu.ror.commons.settings.RawSettings.inner(RawSettings.java:219)
at tech.beshu.ror.commons.settings.BasicSettings.(BasicSettings.java:84)
at tech.beshu.ror.commons.settings.BasicSettings.fromFile(BasicSettings.java:156)
at tech.beshu.ror.es.SettingsObservableImpl.(SettingsObservableImpl.java:57)
at sun.reflect.GeneratedConstructorAccessor30.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:58)
at org.elasticsearch.common.inject.InjectorImpl$4$1.call(InjectorImpl.java:762)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.InjectorImpl$4.get(InjectorImpl.java:757)
at org.elasticsearch.common.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:275)
at org.elasticsearch.common.inject.multibindings.Multibinder$RealMultibinder.get(Multibinder.java:200)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:176)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42)
at org.elasticsearch.node.Node.(Node.java:492)
at org.elasticsearch.node.Node.(Node.java:245)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85)

sscarduzio · April 17, 2018, 8:38am

Xpack does not have anything to do with this error, you just didn’t create and populate readonlyrest.yml

kalev · April 17, 2018, 10:26am

config file actually exist as is pretty simple, maybe there is something wrong:
cat /etc/elasticsearch/readonlyrest.yml

readonlyrest:

    audit_collector: true

    access_control_rules:

    - name: "Kibana read"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/read/*"]
      kibana_access: ro
      kibana_index: .kibana
      groups: ["Read Only"]
      verbosity: error

    - name: "Logstash write"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/write/*"]
      hosts_local: ["127.0.0.1", "localhost"]
      groups: ["Logstash"]
      verbosity: error

    users:

    - username: kibana
      auth_key: kibana:****
      groups: ["Read Only"]

    - username: logstash_internal
      auth_key: logstash_internal:****
      groups: ["Logstash"]

ld57 · April 18, 2018, 4:51pm

Hi

here that I would change : (based on mine running on cluster, along with x-pack enabled (not security) on 6.2.1

readonlyrest:
    enable: true
    response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin
    audit_collector: true
    prompt_for_basic_auth: false
    

    access_control_rules:
      - name: Kibana Server (we trust this server side component, full access granted via HTTP authentication)
      # it is for kibana server getting access to elasticsearch only
      auth_key_sha1: f435b46456464646273028f
      type: allow
      hosts: [kibana_IP]

    - name: "x-pack monitoring"
      #credentials and config must be done in kibana.yml, elasticsearch.yml, xxxxbeats.yml, logstash.yml
      auth_key_sha1: 6fce41670a6546456544655447
      type: allow
      actions: ["cluster:monitor/*", "indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create", "cluster:admin/ingest/pipeline/*","cluster:admin/xpack/monitoring/*"]
      indices: [".monitoring-*"]
      verbosity: info

    - name: "Kibana read"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/read/*"]
      kibana_access: ro
      kibana_index: .kibana
      groups: ["Read Only"]
      verbosity: error

    - name: "Logstash write"
      indices: ["logstash-*"] # aliases are taken in account!
      actions: ["indices:data/write/*"]
      hosts_local: ["127.0.0.1", "localhost"]
      groups: ["Logstash"]
      verbosity: error

    users:

    - username: kibana
      auth_key: kibana:****
      groups: ["Read Only"]

    - username: logstash_internal
      auth_key: logstash_internal:****
      groups: ["Logstash"]

as I did not see the block for kibana server , i added it.
I added the rule for xpack monitoring. feel free to adapt

sscarduzio · April 28, 2018, 1:57pm

Hi @kalev, sorry for the delayed response.

It looks like ROR is not picking up the readonlyrest.yml file, so it’s falling back to elasticsearch.yml and does not find the root of ROR settings in it (of course). Could you please make sure:

You have the latest ROR for ES installed
The file readonlyrest.yml is in the same directory with your elasticsearch.yml
The UNIX permissions are appropriate for the ES process to read the file
Grep the ES logs and find this line “Read data from”, this shows the path ROR looks for the readonlyrest.yml file.