Kibana can't write its indices

kevin_thorpe · June 2, 2017, 1:29pm

I’m just working through the examples given and I just want a very simple install where Logstash can write and Kibana and direct users have only read access. This is more than enough for our logging use case. The problem is that Kibana cannot create its index. This ends up with a useless Kibana instance reporting “[index_not_found_exception] rewritten indices not found: .kibana, with { index_uuid=“na” & index=”.kibana" }" and indeed the .kibana index is missing. What am I doing wrong? This is on ES + Kibana 5.3.0

Config is:

readonlyrest:
    #optional
    response_if_req_forbidden: Sorry, your request is forbidden.

    access_control_rules:
    - name: Accept all requests from localhost
      hosts: [127.0.0.1]

    - name: All indices read only
      actions: ["indices:data/read/*"]
      indices: ["*"] # index aliases are taken in account!

    - name: Allow monitoring from everywhere
      actions: ["cluster:monitor/*","indices:monitor/*"]

    - name: Allow Logstash write access
      auth_key: logstash:logstash
      actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["*"]

    - name: Allow Kibana server process
      auth_key: kibana:kibana
      kibana_access: rw
      indices: [".kibana",".kibana-devnull","*"]

sscarduzio · June 2, 2017, 3:57pm

It doesn’t work like this, kibana daemon needs to have full access to its index. Remove “kibana_access”. That rule is designed for the browsers.

kevin_thorpe · June 6, 2017, 1:50pm

Thanks for the quick response but sorry, but I still can’t get this to work and I’ve tried dozens of things. Trying based on elastic’s own ES and Kibana docker images.

added this to the elasticsearch config

readonlyrest:
    #optional
    response_if_req_forbidden: Sorry, your request is forbidden.

    access_control_rules:
    - name: Accept all requests from localhost
      hosts: [127.0.0.1]

    - name: All indices read only
      actions: ["indices:data/read/*"]
      indices: ["*"] # index aliases are taken in account!

    - name: Allow monitoring from everywhere
      actions: ["cluster:monitor/*","indices:monitor/*"]

    - name: Allow Logstash write access
      auth_key: logstash:logstash
      actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["*"]

    - name: Allow Kibana server process
      auth_key: kibana:kibana
      indices: [".kibana",".kibana-devnull","*"]

    - name: Kibana read only
      auth_key: readonly:readonly
      kibana_access: ro
      indices: ["*"]

kibana config

kibana@db267594183e:~$ cat /usr/share/kibana/config/kibana.yml
# Default Kibana configuration from kibana-docker.

server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch-data:9200
elasticsearch.username: kibana
elasticsearch.password: kibana

docker-compose

elasticsearch-master:
  image: docker-test.pibenchmark.com/pi-elasticsearch:5.3.0_006
  environment:
  - cluster.name=pi_test_kev
  - node.master=true
  - node.data=false
  - node.ingest=true
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
  volumes:
  - elasticsearch-master-kev:/usr/share/elasticsearch/data
  volume_driver: convoy
  ports:
  - 9200:9200/tcp
  - 9300:9300/tcp

elasticsearch-data:
  image: docker-test.pibenchmark.com/pi-elasticsearch:5.3.0_006
  environment:
  - cluster.name=pi_test_kev
  - node.master=false
  - node.data=true
  - node.ingest=true
  - node.max_local_storage_nodes=3
  - discovery.zen.ping.unicast.hosts=elasticsearch-master
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
  volumes:
  - elasticsearch-data-kev:/usr/share/elasticsearch/data
  volume_driver: convoy
  ports:
  - 9200:9200/tcp
  - 9300:9300/tcp
  links:
  - elasticsearch-master:elasticsearch-master

kibana:
  image: docker-test.pibenchmark.com/pi-kibana:5.3.0_003
  ports:
  - 5601:5601/tcp
  links:
  - elasticsearch-data:elasticsearch-data

sscarduzio · June 6, 2017, 6:16pm

Yes but show us the logs!
Also be sure in that docker image there’s no xpack security enabled.

kevin_thorpe · June 8, 2017, 9:37am

Sorry, was away at a meetup yesterday. Nope there is no x-pack. Here’s the Dockerfile:

FROM docker.elastic.co/kibana/kibana:5.3.0
MAINTAINER Kevin Thorpe <kevin.thorpe@p-i.net>

# remove x-pack
RUN /usr/share/kibana/bin/kibana-plugin remove x-pack

ADD kibana.yml /usr/share/kibana/config/kibana.yml

and here’s the logs now I’ve simplified to one elasticsearch and one kibana

[kevin@kev-c7-test elk-kev]$ rancher-compose-0.8.6 -p elk-kev up elasticsearch-master kibana
INFO[0000] Creating stack elk-kev
INFO[0000] Creating service elasticsearch-master
INFO[0000] Creating service kibana
INFO[0000] [0/7] [elasticsearch-master]: Starting
INFO[0004] [1/7] [elasticsearch-master]: Started
INFO[0004] [1/7] [kibana]: Starting
elasticsearch-master_1 | 2017-06-08T09:32:06.709208267Z [2017-06-08T09:32:06,707][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch-master_1 | 2017-06-08T09:32:06.973972789Z [2017-06-08T09:32:06,973][INFO ][o.e.e.NodeEnvironment    ] [RxrVWH-] using [1] data paths, mounts [[/usr/share/elasticsearch/data (ld3-convoy-test:/data/elasticsearch-master-kev)]], net usable_space [137.7gb], net total_space [249.8gb], spins? [possibly], types [nfs4]
elasticsearch-master_1 | 2017-06-08T09:32:06.974322989Z [2017-06-08T09:32:06,974][INFO ][o.e.e.NodeEnvironment    ] [RxrVWH-] heap size [494.9mb], compressed ordinary object pointers [true]
elasticsearch-master_1 | 2017-06-08T09:32:07.200168776Z [2017-06-08T09:32:07,199][INFO ][o.e.n.Node               ] node name [RxrVWH-] derived from node ID [RxrVWH-HTZa_lRB-BUW6bg]; set [node.name] to override
elasticsearch-master_1 | 2017-06-08T09:32:07.200470483Z [2017-06-08T09:32:07,200][INFO ][o.e.n.Node               ] version[5.3.0], pid[1], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Linux/3.10.0-514.10.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_92-internal/25.92-b14]
INFO[0013] [2/7] [kibana]: Started
elasticsearch-master_1 | 2017-06-08T09:32:12.129153467Z [2017-06-08T09:32:12,128][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [aggs-matrix-stats]
elasticsearch-master_1 | 2017-06-08T09:32:12.129216094Z [2017-06-08T09:32:12,128][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [ingest-common]
elasticsearch-master_1 | 2017-06-08T09:32:12.129226582Z [2017-06-08T09:32:12,128][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [lang-expression]
elasticsearch-master_1 | 2017-06-08T09:32:12.129233823Z [2017-06-08T09:32:12,128][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [lang-groovy]
elasticsearch-master_1 | 2017-06-08T09:32:12.129240866Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [lang-mustache]
elasticsearch-master_1 | 2017-06-08T09:32:12.129739566Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [lang-painless]
elasticsearch-master_1 | 2017-06-08T09:32:12.129759157Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [percolator]
elasticsearch-master_1 | 2017-06-08T09:32:12.129768981Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [reindex]
elasticsearch-master_1 | 2017-06-08T09:32:12.129783923Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [transport-netty3]
elasticsearch-master_1 | 2017-06-08T09:32:12.129804371Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded module [transport-netty4]
elasticsearch-master_1 | 2017-06-08T09:32:12.130176170Z [2017-06-08T09:32:12,129][INFO ][o.e.p.PluginsService     ] [RxrVWH-] loaded plugin [readonlyrest]
kibana_1               | 2017-06-08T09:32:12.166288107Z {"type":"log","@timestamp":"2017-06-08T09:32:12Z","tags":["info","optimize"],"pid":7,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
elasticsearch-master_1 | 2017-06-08T09:32:14.790219742Z [2017-06-08T09:32:14,787][INFO ][o.e.p.r.a.ACL            ] ADDING #Accept all requests from localhost: readonlyrest Rules Block :: { name: 'Accept all requests from localhost', policy: ALLOW}
elasticsearch-master_1 | 2017-06-08T09:32:14.794733778Z [2017-06-08T09:32:14,793][INFO ][o.e.p.r.a.ACL            ] ADDING #All indices read only:      readonlyrest Rules Block :: { name: 'All indices read only', policy: ALLOW}
elasticsearch-master_1 | 2017-06-08T09:32:14.794760230Z [2017-06-08T09:32:14,793][INFO ][o.e.p.r.a.ACL            ] ADDING #Allow monitoring from everywhere:   readonlyrest Rules Block :: { name: 'Allow monitoring from everywhere', policy: ALLOW}
elasticsearch-master_1 | 2017-06-08T09:32:14.795043894Z [2017-06-08T09:32:14,794][INFO ][o.e.p.r.a.ACL            ] ADDING #Allow Logstash write access:        readonlyrest Rules Block :: { name: 'Allow Logstash write access', policy: ALLOW}
elasticsearch-master_1 | 2017-06-08T09:32:14.795342786Z [2017-06-08T09:32:14,794][INFO ][o.e.p.r.a.ACL            ] ADDING #Allow Kibana server process:        readonlyrest Rules Block :: { name: 'Allow Kibana server process', policy: ALLOW}
elasticsearch-master_1 | 2017-06-08T09:32:14.796169891Z [2017-06-08T09:32:14,795][INFO ][o.e.p.r.a.ACL            ] ADDING #Kibana read only:   readonlyrest Rules Block :: { name: 'Kibana read only', policy: ALLOW}
elasticsearch-master_1 | 2017-06-08T09:32:14.796404669Z [2017-06-08T09:32:14,796][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Configuration reloaded - ReadonlyREST enabled
elasticsearch-master_1 | 2017-06-08T09:32:14.797177365Z [2017-06-08T09:32:14,796][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Readonly REST plugin was loaded...
elasticsearch-master_1 | 2017-06-08T09:32:15.220061669Z [2017-06-08T09:32:15,219][INFO ][o.e.n.Node               ] initialized
elasticsearch-master_1 | 2017-06-08T09:32:15.220110082Z [2017-06-08T09:32:15,219][INFO ][o.e.n.Node               ] [RxrVWH-] starting ...
elasticsearch-master_1 | 2017-06-08T09:32:15.307942891Z [2017-06-08T09:32:15,307][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 27:ef:4a:5e:92:21:34:12
elasticsearch-master_1 | 2017-06-08T09:32:15.392612827Z [2017-06-08T09:32:15,392][INFO ][o.e.t.TransportService   ] [RxrVWH-] publish_address {10.42.254.50:9300}, bound_addresses {[::]:9300}
elasticsearch-master_1 | 2017-06-08T09:32:15.400627092Z [2017-06-08T09:32:15,400][INFO ][o.e.b.BootstrapChecks    ] [RxrVWH-] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch-master_1 | 2017-06-08T09:32:15.413240193Z [2017-06-08T09:32:15,412][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Cluster-wide settings found, overriding elasticsearch.yml
elasticsearch-master_1 | 2017-06-08T09:32:18.454270244Z [2017-06-08T09:32:18,452][INFO ][o.e.c.s.ClusterService   ] [RxrVWH-] new_master {RxrVWH-}{RxrVWH-HTZa_lRB-BUW6bg}{YNrcFgD8RjWlSH5_UtWjSQ}{10.42.254.50}{10.42.254.50:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch-master_1 | 2017-06-08T09:32:18.473617831Z [2017-06-08T09:32:18,473][INFO ][o.e.h.n.Netty4HttpServerTransport] [RxrVWH-] publish_address {10.42.254.50:9200}, bound_addresses {[::]:9200}
elasticsearch-master_1 | 2017-06-08T09:32:18.475451011Z [2017-06-08T09:32:18,475][INFO ][o.e.n.Node               ] [RxrVWH-] started
elasticsearch-master_1 | 2017-06-08T09:32:18.887021409Z [2017-06-08T09:32:18,886][INFO ][o.e.g.GatewayService     ] [RxrVWH-] recovered [4] indices into cluster_state
kibana_1               | 2017-06-08T09:34:25.062420394Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["info","optimize"],"pid":7,"message":"Optimization of bundles for kibana, timelion and status_page complete in 132.89 seconds"}
kibana_1               | 2017-06-08T09:34:25.139407634Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","plugin:kibana@5.3.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1               | 2017-06-08T09:34:25.238234640Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","plugin:elasticsearch@5.3.0","info"],"pid":7,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1               | 2017-06-08T09:34:25.271877493Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","plugin:console@5.3.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1               | 2017-06-08T09:34:25.486124038Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","plugin:timelion@5.3.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1               | 2017-06-08T09:34:25.491169955Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["listening","info"],"pid":7,"message":"Server running at http://0:5601"}
kibana_1               | 2017-06-08T09:34:25.492795453Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","ui settings","info"],"pid":7,"state":"yellow","message":"Status changed from uninitialized to yellow - Elasticsearch plugin is yellow","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch-master_1 | 2017-06-08T09:34:25.522454718Z [2017-06-08T09:34:25,521][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:739255995-826277272, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:25.522520813Z [2017-06-08T09:34:25,522][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 739255995-826277272  rewritten indices not found: .kibana
kibana_1               | 2017-06-08T09:34:25.532255498Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","plugin:elasticsearch@5.3.0","error"],"pid":7,"state":"red","message":"Status changed from yellow to red - [index_not_found_exception] rewritten indices not found: .kibana, with { index_uuid=\"_na_\" & index=\".kibana\" }","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1               | 2017-06-08T09:34:25.532873326Z {"type":"log","@timestamp":"2017-06-08T09:34:25Z","tags":["status","ui settings","error"],"pid":7,"state":"red","message":"Status changed from yellow to red - Elasticsearch plugin is red","prevState":"yellow","prevMsg":"Elasticsearch plugin is yellow"}
elasticsearch-master_1 | 2017-06-08T09:34:28.065969540Z [2017-06-08T09:34:28,065][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1713494078-210704249, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:28.066048219Z [2017-06-08T09:34:28,065][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1713494078-210704249  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:30.593900866Z [2017-06-08T09:34:30,593][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:234557505-1374069065, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:30.593943444Z [2017-06-08T09:34:30,593][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 234557505-1374069065  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:33.125365227Z [2017-06-08T09:34:33,125][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1142305382-1727775224, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:33.125408346Z [2017-06-08T09:34:33,125][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1142305382-1727775224  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:35.647817318Z [2017-06-08T09:34:35,647][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1622928868-54784024, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:35.647857890Z [2017-06-08T09:34:35,647][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1622928868-54784024  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:38.182287260Z [2017-06-08T09:34:38,179][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:747245464-602051318, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:38.182373690Z [2017-06-08T09:34:38,180][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 747245464-602051318  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:40.709024804Z [2017-06-08T09:34:40,708][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:790846794-789257139, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:40.709102710Z [2017-06-08T09:34:40,708][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 790846794-789257139  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:43.231085116Z [2017-06-08T09:34:43,229][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1189301876-2095947579, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:43.231147608Z [2017-06-08T09:34:43,230][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1189301876-2095947579  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:45.755629096Z [2017-06-08T09:34:45,754][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1635025205-44685593, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:45.755694829Z [2017-06-08T09:34:45,755][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1635025205-44685593  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:48.286693340Z [2017-06-08T09:34:48,286][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1773246283-50165861, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:48.286746138Z [2017-06-08T09:34:48,286][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1773246283-50165861  rewritten indices not found: .kibana
elasticsearch-master_1 | 2017-06-08T09:34:50.809370050Z [2017-06-08T09:34:50,808][INFO ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] forbidden request: { ID:1383931059-1942558384, TYP:ClusterHealthRequest, USR:kibana(?), BRS:false, ACT:cluster:monitor/health, OA:10.42.251.255, IDX:.kibana, MET:GET, PTH:/_cluster/health/.kibana, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,Content-Length,Host, HIS:[Accept all requests from localhost->[hosts->false]] } Reason: [.kibana] IndexNotFoundException[rewritten indices not found: .kibana]
elasticsearch-master_1 | 2017-06-08T09:34:50.809444700Z [2017-06-08T09:34:50,808][WARN ][o.e.p.r.e.IndexLevelActionFilter] [RxrVWH-] Resource not found! ID: 1383931059-1942558384  rewritten indices not found: .kibana

It’s probably something very simple that I’m missing. Working in Docker is also not helping

sscarduzio · June 8, 2017, 12:27pm

This rule is like not having it. Should disappear (think about it).

Same for this:

kevin_thorpe · June 8, 2017, 4:54pm

I understand about the '*' parts. That should be the prefix of the time based indices I’m going to create. Haven’t got any as yet so that was a placeholder for access to whatever I choose to add. I’ve changed it to logstash-* with the same results