Elasticsearch host unreachable in logstash

1

Hello,
I’m trying to use ROR for the first time. I encounter the following error in logstash :

[WARN ] 2019-01-07 12:02:58.982 [Ruby-0-Thread-10: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused}

I searched on forums but I couldn’t find something that works.

My logstash conf :

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "syslog-%{+YYYY.MM.dd}"
        user => ["logstash"]
        password => ["logstash"]
    }
}

My readonlyrest.yml :

readonlyrest:
    enable: true
    response_if_req_forbidden: Forbidden by ReadonlyREST ES plugin
    access_control_rules:

    - name: "Logstash can write and create its own indices"
      auth_key: logstash:logstash
      type: allow
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["logstash-*", "<no-index>"]

    - name: Kibana Server (we trust this server side component, full access granted via HTTP authentication)
      auth_key: admin:passwd3
      type: allow

    - name: Developer (reads only logstash indices, but can create new charts/dashboards)
      auth_key: dev:dev
      type: allow
      kibana_access: ro+
      indices: ["<no-index>", ".kibana*", "logstash*", "default"]

I hope you can help me please.

Kindest regards.

2

Hi,

as I see, elastcsearch was not listening on 9200.

could you share your elasticsearch.log ?

thx

3

Hi,

Actually I broke everything so I made a new clean install and I found the culprit. I read the logs and the doc and I found that this line was wrong “kibana_access: ro+”. It’s now "“kibana_access: ro”.

Now I have this different error with a 401 error in logstash.

[WARN ] 2019-01-08 08:25:08.351 [Ruby-0-Thread-12: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash:xxxxxx@localhost:9200/", error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}`

I think these are the lines related to the logstash error in elasticsearch.log because they only appear if I run logstash :

[2019-01-08T09:07:33,463][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:1341788589-2099370580#22988, TYP:MainRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1, DA:127.0.0.1, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Authorization=<OMITTED>, content-length=0, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->true, auth_key->true, actions->false]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }      
[2019-01-08T09:07:33,481][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:1613540594-271858237#22989, TYP:MainRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1, DA:127.0.0.1, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Authorization=<OMITTED>, content-length=0, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->true, auth_key->true, actions->false]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }       
[2019-01-08T09:07:33,483][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:28835601-492431185#22990, TYP:MainRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1, DA:127.0.0.1, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Authorization=<OMITTED>, content-length=0, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->true, auth_key->true, actions->false]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }         
[2019-01-08T09:07:33,486][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:2068055781-1176228242#22991, TYP:MainRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1, DA:127.0.0.1, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Authorization=<OMITTED>, content-length=0, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->true, auth_key->true, actions->false]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }      
[2019-01-08T09:07:33,488][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:513389603-1753945798#22992, TYP:MainRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1, DA:127.0.0.1, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Authorization=<OMITTED>, content-length=0, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->true, auth_key->true, actions->false]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }

Thanks for your help.

4

as you see, it is the “action” that logstash user tried too.

then you need to authorize this action in your readonlyrest.yml file
add it to the array actions of logstash block.

5

Thanks a lot ! I did it and now I have these errors elasticsearch.log.

[2019-01-08T13:37:40,896][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:1862215695-1062393652#2903, TYP:BulkRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:indices:data/write/bulk, OA:127.0.0.1, DA:127.0.0.1, IDX:sssd-2019.01.08, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=7009>, HDR:{Authorization=<OMITTED>, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Length=7009, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->false, auth_key->true]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }                                                                                                                                                        
[2019-01-08T13:37:40,900][INFO ][t.b.r.a.ACL              ] [jgeVQ7z] FORBIDDEN by default req={ ID:936062398-1023973884#2904, TYP:BulkRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:indices:data/write/bulk, OA:127.0.0.1, DA:127.0.0.1, IDX:sssd-2019.01.08, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=11279>, HDR:{Authorization=<OMITTED>, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=localhost:9200, Accept-Encoding=gzip,deflate, Content-Length=11279, Content-Type=application/json}, HIS:[Require HTTP Basic Auth->[auth_key->false]], [Logstash can write and create its own indices->[indices->false, auth_key->true]], [Kibana Server (we trust this server side component, full access granted via HTTP authentication)->[auth_key->false]], [Developer (reads only logstash indices, but can create new charts/dashboards)->[auth_key->false]] }

I tried to do the same and add “indices:data/write/bulk” like I did with “cluster:monitor/main”, even if I already had “indices:data/write/*” but this is not working. I’ll keep searching on my own too.

sssd-2019.01.08 is one of my index if this can help.

6

hi

this time it is not related to the action (your block definition is ok for that), but related to indice name

this time it concern the indices name : logstash tried to write to indice name : IDX:sssd-2019.01.08

but look at your block rule for logstash, it has no authorization to write to this such indices. ( infact, the good wording is " logstas hhas the right to write to indice pattern “logstash-*” ")

add to indices array in logstash block rule something like “sssd-*”

1 Like
7

Hello,

Everything is working perfectly now,

Thanks a lot for the help !