Kibana elasticsearch plugin goes red

Hi All,

We are using readonlyrest Enterprise, and, sometimes, elasticsearch plugin for kibana goes to red, after several minutes goes back to green.

{“type”:“log”,“@timestamp”:“2018-06-01T08:00:11Z”,“tags”:[“status”,“ui settings”,“error”],“pid”:34485,“state”:“red”,“message”:“Status changed from green to red - Elasticsearch plugin is red”,“prevState”:“green”,“prevMsg”:“Ready”}

In ES we have this error at the same time (but I do not know if it is related with Kibana problem):

it seems that Kibana is asking for two kibana indexes at the same time, but we do not why is doing that…

[2018-06-01T08:00:11,576][WARN ][r.suppressed ] path: /.kibana_operacion_mantenimiento%2C.kibana_administracion/config/_search, params: {index=.kibana_operacion_mantenimiento,.kibana_administracion, type=config}
org.elasticsearch.action.search.SearchPhaseExecutionException:
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:271) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:92) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:623) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.ClassCastException

And here it is our readonlyrest configuration:

readonlyrest:
ssl:
keystore_file: “/etc/elasticsearch/ssl-private/keystore.jks”
keystore_pass: password
key_pass: password

prompt_for_basic_auth: false

user_groups_providers:

name: GroupsService
groups_endpoint: “http://mxulomih:8081/WSUserValidation/groups
auth_token_name: “user”
auth_token_passed_as: QUERY_PARAM # HEADER OR QUERY_PARAM
response_groups_json_path: “$…groups[?(@.name)].name” # see: GitHub - json-path/JsonPath: Java JsonPath implementation
cache_ttl_in_sec: 600
access_control_rules:

name: ‘::KIBANA-SRV::’
auth_key: ‘kibana:password’

name: ‘::GRAFANA::’
indices : [ cdrs* ]
auth_key: ‘grafana:password’

name: “ADMINISTRACION”
proxy_auth: “*”
kibana_index: .kibana_administracion
kibana_access: rw
kibana_hide_apps: [“readonlyrest_kbn”]
groups_provider_authorization:
user_groups_provider: GroupsService
groups: [“administracion”]

name: “ROBOTIZACION”
proxy_auth: “”
kibana_index: .kibana_robotizacion
kibana_access: rw
indices : [ cdrs, .kibana_robotizacion]
kibana_hide_apps: [“readonlyrest_kbn”, “timelion”, “kibana:dev_tools”, “kibana:management”]
groups_provider_authorization:
user_groups_provider: “GroupsService”
groups: [“robotizacion”]

name: “DESPLIEGUE”
proxy_auth: “”
kibana_index: .kibana_despliegue
kibana_access: rw
indices : [ cdrs, .kibana_despliegue]
kibana_hide_apps: [“readonlyrest_kbn”, “timelion”, “kibana:dev_tools”, “kibana:management”]
groups_provider_authorization:
user_groups_provider: “GroupsService”
groups: [“despliegue”]

name: “INGENIERIA”
proxy_auth: “”
kibana_index: .kibana_ingenieria
kibana_access: rw
indices : [ cdrs, .kibana_ingenieria]
kibana_hide_apps: [“readonlyrest_kbn”, “timelion”, “kibana:dev_tools”, “kibana:management”]
groups_provider_authorization:
user_groups_provider: “GroupsService”
groups: [“ingenieria”]

name: “OPERACION_MANTENIMIENTO”
proxy_auth: “”
kibana_index: .kibana_operacion_mantenimiento
kibana_access: rw
indices : [ cdrs, .kibana_operacion_mantenimiento]
kibana_hide_apps: [“readonlyrest_kbn”, “timelion”, “kibana:dev_tools”, “kibana:management”]
groups_provider_authorization:
user_groups_provider: “GroupsService”
groups: [“operacion_mantenimiento”]

name: “PLANIFICACION”
proxy_auth: “”
kibana_index: .kibana_planificacion
kibana_access: rw
indices : [ cdrs, .kibana_planificacion]
kibana_hide_apps: [“readonlyrest_kbn”, “timelion”, “kibana:dev_tools”, “kibana:management”]
groups_provider_authorization:
user_groups_provider: “GroupsService”
groups: [“planificacion”]

name: “SEGURIDAD”
proxy_auth: “”
kibana_index: .kibana_seguridad
kibana_access: rw
indices : [ cdrs, .kibana_seguridad]
kibana_hide_apps: [“readonlyrest_kbn”, “timelion”, “kibana:dev_tools”, “kibana:management”]
groups_provider_authorization:
user_groups_provider: “GroupsService”
groups: [“seguridad”]

Any ideas about what’s going wrong??

Thanks in advance

Hello @JosCar!
What ROR Enterprise version is this? Because we fixed something very similar in 1.16.20.

readonlyrest_kbn_enterprise-1.16.18

How could we get the last version??

Thanks in advance

Felipe

We have downloaded the new version: readonlyrest_kbn_enterprise-1.16.20_es5.5.1.zip
But, it is safe to use it with : readonlyrest-1.16.18_es5.5.1.zip ???

Should we upgrade elasticsearch plugin to 16.20 also???

Thanks in advance

Hi @fcerezo, please update both plugins.

Hi @sscarduzio,

We have update both plugins to version 1.16.20.
Kibana Elasticsearch does not go to red, that’s good.

But we are facing some bug when reading kibana index for some users, it seems that the plugin is mixing several users and kibana indexes. When this happens right side of the screen is white and cannot make querys to ES.

Maybe this kibana trace could be helpful, as you can see in x-ror-user there are several users and in x-ror-kibana_index there are two different indices.

{“type”:“log”,"@timestamp":“2018-06-25T11:44:10Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:15187,“message”:" headers from ES auth: {“x-kibana-hide-apps”:“readonlyrest_kbn, kibana:dev_tools,timelion,kibana:management,readonlyrest_kbn”,“x-rr-user”:“testhpe, rmartinro, abmorenom”,“x-ror-kibana_access”:“rw”,“x-ror-kibana_index”:".kibana_administracion, .internal_kibana",“content-type”:“application/json; charset=UTF-8”,“content-length”:“7061”}"}

Thanks in advance

Hi @fcerezo, this should be fixed in 1.16.21

@sscarduzio, and 1.16.21 is already released?
if not when do you think it will be?

Thanks in advance

Yes it’s been released last week, go to the download page :slight_smile:

Hi,

@sscarduzio

We have updated to last version and sometimes users fail to login, it seems some kind of mix up with permissions and kibana indexes

we have found this traces in kibana in with rr-user is composed by several users and there are several kibana indexes
{“type”:“log”,"@timestamp":“2018-07-11T07:34:10Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:39578,“message”:" headers from ES auth: {“x-kibana-hide-apps”:“kibana:dev_tools,timelion,kibana:management,readonlyrest_kbn”,“x-rr-user”:“abmorenom, rdedios, fguerrer, rsempere, jsanchmo, mafernas”,“x-ror-kibana_access”:“rw”,“x-ror-kibana_index”:".internal_kibana",“content-type”:“application/json; charset=UTF-8”,“content-length”:“7061”}"}
{“type”:“log”,"@timestamp":“2018-07-11T08:24:09Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:39578,“message”:" headers from ES auth: {“x-kibana-hide-apps”:“kibana:dev_tools,timelion,kibana:management,readonlyrest_kbn”,“x-rr-user”:“abmorenom, rdedios, jcompte, ROBOT_DNF”,“x-ror-kibana_access”:“rw”,“x-ror-kibana_index”:".internal_kibana",“content-type”:“application/json; charset=UTF-8”,“content-length”:“7061”}"}
{“type”:“log”,"@timestamp":“2018-07-11T08:26:01Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:39578,“message”:" headers from ES auth: {“x-kibana-hide-apps”:“kibana:dev_tools,timelion,kibana:management,readonlyrest_kbn”,“x-rr-user”:“arodicim, jsanchmo, fguerrer, ROBOT_DNF, rsempere, abmorenom”,“x-ror-kibana_access”:“rw”,“x-ror-kibana_index”:".internal_kibana",“content-type”:“application/json; charset=UTF-8”,“content-length”:“7061”}"}

HI @fcerezo, that’s weird. Can I see your readonlyrest.yml ?

@sscarduzio

Here it is
kibana indexes are not commented because this feature wasn’t working properly, our next step is to activate it.

Thanks in advance

Felipe

readonlyrest:
  ssl:
    keystore_file: "/etc/elasticsearch/ssl-private/keystore.jks"
    keystore_pass: password
    key_pass: password

  prompt_for_basic_auth: false

  user_groups_providers:
  - name: GroupsService
    groups_endpoint: "http://10.192.142.225:8081/WSUserValidation/groups"
    auth_token_name: "user"
    auth_token_passed_as: QUERY_PARAM                        # HEADER OR QUERY_PARAM
    response_groups_json_path: "$..groups[?(@.name)].name"   # see: https://github.com/json-path/JsonPath
    cache_ttl_in_sec: 600


  access_control_rules:
  - name: '::KIBANA-SRV::'
    auth_key: 'kibana:XXXXXXXXXX'

  - name: '::GRAFANA::'
    indices : [ cdrs* ]
    auth_key: 'grafana:XXXXXXXXX'

  - name: "ADMINISTRACION"
    proxy_auth: "*"
#    kibana_index: .kibana_administracion
    kibana_index: .internal_kibana
    kibana_access: rw
    kibana_hide_apps: ["readonlyrest_kbn"]
    groups_provider_authorization:
      user_groups_provider: GroupsService
      groups: ["administracion"]

  - name: "ROBOTIZACION"
    proxy_auth: "*"
#    kibana_index: .kibana_robotizacion
    kibana_index: .internal_kibana
    kibana_access: rw
    indices : [ cdrs*, .kibana_robotizacion, .internal_kibana ]
    kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["robotizacion"]

  - name: "DESPLIEGUE"
    proxy_auth: "*"
#    kibana_index: .kibana_despliegue
    kibana_index: .internal_kibana
    kibana_access: rw
    indices : [ cdrs*, .kibana_despliegue, .internal_kibana ]
    kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["despliegue"]

  - name: "INGENIERIA"
    proxy_auth: "*"
#    kibana_index: .kibana_ingenieria
    kibana_index: .internal_kibana
    kibana_access: rw
    indices : [ cdrs*, .kibana_ingenieria, .internal_kibana ]
    kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["ingenieria"]

  - name: "OPERACION_MANTENIMIENTO"
    proxy_auth: "*"
#    kibana_index: .kibana_operacion_mantenimiento
    kibana_index: .internal_kibana
    kibana_access: rw
    indices : [ cdrs*, .kibana_operacion_mantenimiento, .internal_kibana ]
    kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["operacion_mantenimiento"]

  - name: "PLANIFICACION"
    proxy_auth: "*"
#    kibana_index: .kibana_planificacion
    kibana_index: .internal_kibana
    kibana_access: rw
    indices : [ cdrs*, .kibana_planificacion, .internal_kibana ]
    kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["planificacion"]

  - name: "SEGURIDAD"
    proxy_auth: "*"
#    kibana_index: .kibana_seguridad
    kibana_index: .internal_kibana
    kibana_access: rw
    indices : [ cdrs*, .kibana_seguridad, .internal_kibana ]
    kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["seguridad"]

  - name: "TEST"
    proxy_auth: "*"
    kibana_index: .kibana_administracion
#    kibana_index: .internal_kibana
    kibana_access: rw
    kibana_hide_apps: ["readonlyrest_kbn"]
    groups_provider_authorization:
      user_groups_provider: "GroupsService"
      groups: ["test"]

Is your proxy adding a X-Forwarded-User header with a comma separated list of users for some reason?

@sscarduzio

I think we are not, user is only one word…

We are going to double check, but one one word users are allowed to enter

You can easily verify this analising the Elasticsearch logs, grep for “X-Forwarded-User”
BTW Also, you have a repeated YAML syntactic error (a space between indices and “:”)

@sscarduzio

All users in ES logs are ok this is what we have found for yesterday:

x-forwarded-user=arodicim
x-forwarded-user=aserrara
x-forwarded-user=csanzbal
x-forwarded-user=fguerrer
x-forwarded-user=jmartide
x-forwarded-user=jsanchmo
x-forwarded-user=lrodrigg
x-forwarded-user=mafernas
x-forwarded-user=rdedios
x-forwarded-user=rsempere
x-forwarded-user=testhpe

We are going to correct yaml sintax, thanks for the tip :slight_smile:

Felipe

1 Like

@sscarduzio
We have update to readonlyrest_kbn_enterprise-1.16.23_es5.5.1.zip and we are getting the same problems, we cannot set a different kibana index for a group of users, someting is mixing up that makes Kibana search into an index which is the combination of two indexes, as you can see in parameter x-ror-kibana_index

{"type":"log","@timestamp":"2018-08-23T11:47:25Z","tags":["debug","readonlyrest_kbn"],"pid":1628,"message":" headers from ES auth: {"x-kibana-hide-apps":"kibana:dev_tools,timelion,kibana:management,readonlyrest_kbn, readonlyrest_kbn","x-rr-user":"ROBOT_DNF, dsegovpa, mafernas, testhpe","x-ror-kibana_access":"rw","x-ror-kibana_index":".internal_kibana, .kibana_administracion","content-type":"application/json; charset=UTF-8","content-length":"7061"}"}
{"type":"log","@timestamp":"2018-08-23T11:47:25Z","tags":["debug","readonlyrest_kbn"],"pid":1628,"message":"ON_IDENTITY setting kibana index to .internal_kibana, .kibana_administracion"}

And also in this specific trace
{“type”:“log”,"@timestamp":“2018-08-23T11:47:25Z”,“tags”:[“debug”,“readonlyrest_kbn”],“pid”:1628,“message”:“ON_IDENTITY setting kibana index to .internal_kibana, .kibana_administracion”}

Thanks in advance and best regards

Felipe

Hello @fcerezo,

We are currently investigating this same issue with another customer, I think we are near to the solution. Please hold on until the new release!

Thanks !!!

Please keep me updated, when the new release is ready

Best regards

1 Like

@sscarduzio

Do you have any update about this issue??

Thanks in advance

Felipe