Configuring ELK

1

And could not get it to work, logstash can’t connect with elastichserach via https

Then I tried to go and use non SSL protocol http in the elasticsearch output plugin but it is not supported anymore

Also I dont know how to make Kibana SSL connection with elasticsearch, I only managed to do it with “auth_key:” plain user/pass

2

So here is the update, Kibana can connect to Elasticsearch using but Logstash can’t using example quoted above, here is dump from Logstash logs

 ESC[[2017-04-04T10:38:46,176][WARN ][logstash.outputs.elasticsearch] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2017-04-04T10:39:01,277][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://usertest123:xxxxxx@localhost:9200/]}}
[2017-04-04T10:39:01,279][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://usertest123:xxxxxx@localhost:9200/, :path=>"/"}
[2017-04-04T10:39:02,483][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTPS:0x73658e URL:https://usertest123:xxxxxx@localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://usertest123:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-04-04T10:39:02,531][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-04-04T10:39:02,554][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://usertest123:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>https://usertest123:xxxxxx@localhost:9200/, :error_message=>"Elasticsearch Unreachable: [https://usertest123:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-04-04T10:39:02,566][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Elasticsearch Unreachable: [https://usertest123:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"

Here is part of logstash config

  elasticsearch {
         ssl => true
         ssl_certificate_verification => false
         hosts => ["localhost:9200"]
         user => usertest123
         password => passtest123
        }

And here is part of Elasticsearch confing

########http.type: ssl_netty4
readonlyrest:
    enable: true
    ssl:
      enable: true
      keystore_file: "/elasticsearch/plugins/readonlyrest/keystore.jks"
      keystore_pass: readonlyrest
      key_pass: test123

    response_if_req_forbidden: Forbidden 

    access_control_rules:

    - name: "::LOGSTASH::"
 #     auth_key: usertest123:passtest123
      auth_key_sha256: "c1efee775e83427bc46175ebc8ae3fc965f8ec5b4f1007328da031e94a4b9967" # usertest123:passtest123
      type: allow
      hosts: [127.0.0.1, my public IP]
      actions: ["indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["logstash-*"]

    - name: "::KIBANA-SERVER::"
      auth_key: usertest123:passtest123
      type: allow
      hosts: [127.0.0.1, my public IP]
3

Looks like Logstash does not reach Elasticsearch, like, network level.

Have you tried setting the raw IP rather than localhost? Or even adding https://

I found some Logstash examples for the hosts part…

4

I changed localhost to 127.0.0.1, now I get this

12:59:35.309 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://usertest123:xxxxxx@127.0.0.1:9200/, :path=>“/”}
12:59:35.626 [[main]-pipeline-manager] ERROR logstash.pipeline - Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x5fc6dc @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0xf256f0 @metric=#<LogStash::Instrument::Metric:0x19c544e @collector=#<LogStash::Instrument::Collector:0xe3bf7f @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0xf2f3d9 @store=#<Concurrent::map:0x5d6ca1 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x10867c0, @fast_lookup=#<Concurrent::map:0x172499 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"8b310e981c3e89c521a6d4e11ce9d9a7a08ec99d-6"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0xd5310d …

5

what version of Logstash is this and what plugin did you install in it?
Always paste the conf file.
Also, verify with curl -k 'https://127.0.0.1:9200' you can see the ES host.

6

All E L K 5.3

plugins I have installed are “translate” and “email” as I remember

curl -k ‘https://127.0.0.1:9200’
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

7

Wait, there must be something wrong with your SSL cert or host’s SSL library if that basic curl won’t work…
Try to google that curl error…

8

I reinstalled git

Now it is this

after server reboot first I got this
curl -k ‘https://127.0.0.1:9200’
curl: (7) Failed to connect to 127.0.0.1 port 9200: Connection refused

then back to
curl: (35) gnutls_handshake() failed: An unexpected TLS packet was received.

ubuntu 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:29:17 UTC 2016 i686 i686 i686 GNU/Linux

Do i need http.content_type.required somewhere somehow or this http.type: ssl_netty4 in elasticsearch?

9

this is required for ssl in readonlyrest. How did you manage to have Kibana working without it?

10

Just using http for Kibana

How do you send request from logstash in order to make it work with elasticsearch if I use only “auth_key: logstash:logstash” before I used to write sth like in logstash “protocol … http” but in 5.3 it is not supported anymore

11

@mke in ES either you enable SSL or you don’t. What does it mean you’re just using http for Kibana?
In the elasticsearch.yml snippet you posted you activated SSL in the plugin, left the http.type commented. I don’t expect your SSL to work like this.

12

In ES confing I still have that http.type: ssl_netty4 with “#” if front so it is disabled as I understand SSL is off

So Logstash started working after I removed that part which I understand is not encrypted now
ssl => true
ssl_certificate_verification => false

Kibana uses default config with only that part changed:

elasticsearch.username: "usertest123"
elasticsearch.password: "passtest123"
and server.host: TO MY IP

If I enable http.type: ssl_netty4 in ES, Kibana has problem with authentication and I guess I would have to alter below part somehow, not sure how.

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

All I wanted from the beginning is SSL with self sign certificates

13

Cool, if you want self signed SSL between kibana daemon and Elasticsearch, then you need to put that to ‘none’ here above and modify the elasticsearch URL to be ‘https://…’ in kibana.yml

Don’t forget to re-enable SSL in elasticsearch.yml, including the http.type as I told you.

Once you’re done with that, go to logstash and do the equivalent you did to Kibana: tell it to connect using SSL, and disable the verification and point it to https://.

14

Once that is working OK, add a pair of credentials to Logstash too, exactly like you did for Kibana. Then it’s secure.