Securing WinLogBeat with SSL + Credentials

I am looking to secure WinLogBeat and I was wondering if you can use SSL alongside the username/password authentication. Using SSL, the username/password doesn’t seem to make a difference on WinLogBeat when communicating with the ElasticStack. The cluster is up and WinLogBeat sends logs no problem with SSL enabled. Given username/password on the WLB side and on the ROR side, doesn’t seem to take affect. Am I missing a setting?

Elastic - 6.2
WinLogBeat - 6.2
Kibana - 6.2 (Left out Kibana info because I don’t think it is relevant at this point)

A basic configuration for a 2 node elastic cluster and WinLogBeat looks like this:

Node-1-elasticsearch[.]yml
cluster.name: “Cluster-1”
node.master: true
node.data: false
node.ingest: false
node.name: “node-1”
network.host: 192.168[.]1.10
http.port: 9200
discovery.zen.ping.unicast.hosts: [“192[.]168.1.20”]
discovery.zen.minimum_master_nodes: 1

------------------------------- ReadOnlyREST ---------------------------------

http.type: ssl_netty4

Node-1-readonlyrest[.]yml
readonlyrest:
     ssl:
       enable: true
       keystore_file: "keystore.jks"
       keystore_pass: readonlyrest
       key_pass: readonlyrest
       allowed_protocols: [TLSv1.2]
access_control_rules:
    - name: "winlogbeat/kibana can write and create its own indices"
      auth_key_sha256: cc2db7731825a9cb972e9e3860285fc04b56bbd71b99a3e72ab7d8e3c46a55a2
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["winlogbeat-*", ".kibana*"]
    - name: "Allow all nodes to communicate"
      hosts: ["192.168.1[.]10","192.168[.]1.20"]
#cc2db7731825a9cb972e9e3860285fc04b56bbd71b99a3e72ab7d8e3c46a55a2 = winlogbeat:winlogbeat (As seen in winlogbeat.yml)

Node-2-elasticsearch[.]yml

cluster.name: "Cluster-1"
node.master: false 
node.data: true
node.ingest: false
node.name: "node-2"
network.host: 192.168.1.20
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192[.]168.1.10"]
discovery.zen.minimum_master_nodes: 1

------------------------------- ReadOnlyREST ---------------------------------

http.type: ssl_netty4

Node-2-readonlyrest[.]yml

readonlyrest:
     ssl:
       enable: true
       keystore_file: "keystore.jks"
       keystore_pass: readonlyrest
       key_pass: readonlyrest
       allowed_protocols: [TLSv1.2]
access_control_rules:
    - name: "winlogbeat can write and create its own indices"
      auth_key_sha256: cc2db7731825a9cb972e9e3860285fc04b56bbd71b99a3e72ab7d8e3c46a55a2
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["winlogbeat-*", ".kibana*"]
    - name: "Allow all nodes to communicate"
      hosts: ["192.168[.]1.10","192.168[.]1.20"]
#cc2db7731825a9cb972e9e3860285fc04b56bbd71b99a3e72ab7d8e3c46a55a2 = winlogbeat:winlogbeat (As seen in winlogbeat.yml)

WinLogBeat[.]yml (Used for multiple clients)

winlogbeat.event_logs:
  - name: Microsoft-Windows-Sysmon/Operational
    ignore_older: 2160h

#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "https://192.168[.]1.10:5601"

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: "my[.]domain[.]com:9200"
#my.domain.com --> 192.168.1.10
#============================== Security ===============================
  username: winlogbeat
  password: winlogbeat
  ssl.enabled: true
  protocol: https
  ssl.certificate_authorities: 'ca.crt'
  ssl.certificate: 'node2.crt'
  ssl.key: 'node2.key'
  ssl.verification_mode: none

Any help would be greatly appreciated!

Note: IP Addresses and creds changed to fake values.

Nic

Can you show the ES logs corresponding to the incoming request coming from WinLongBeat?

I am not sure how to identify logs that correspond directly to the endpoint shipping from WLB. Could you point me in the right direction to target those logs?

Here is a sanitized output from node-2 using ./elasticsearch -e:

[2018-03-15T14:45:48,357][INFO ][o.e.p.PluginsService ] [node-2] loaded module [aggs-matrix-stats]
[2018-03-15T14:45:48,388][INFO ][o.e.p.PluginsService ] [node-2] loaded module [analysis-common]
[2018-03-15T14:45:48,388][INFO ][o.e.p.PluginsService ] [node-2] loaded module [ingest-common]
[2018-03-15T14:45:48,388][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-expression]
[2018-03-15T14:45:48,388][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-mustache]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-painless]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [mapper-extras]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [parent-join]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [percolator]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [rank-eval]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [reindex]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [repository-url]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [transport-netty4]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded module [tribe]
[2018-03-15T14:45:48,404][INFO ][o.e.p.PluginsService ] [node-2] loaded plugin [readonlyrest]
[2018-03-15T14:45:50,607][INFO ][t.b.r.e.SettingsObservableImpl] Loaded good settings from C:\ProgramData\es\elasticsearch-6.2.2\config\readonlyrest.yml
[2018-03-15T14:45:50,732][INFO ][t.b.r.e.IndexLevelActionFilter] [node-2] Loaded good settings from C:\ProgramData\es\elasticsearch-6.2.2\config\readonlyrest.yml
[2018-03-15T14:45:50,748][INFO ][t.b.r.e.IndexLevelActionFilter] [node-2] Settings observer refreshing…
[2018-03-15T14:45:50,748][INFO ][t.b.r.e.IndexLevelActionFilter] [node-2] Configuration reloaded - ReadonlyREST disabled
[2018-03-15T14:45:50,748][INFO ][t.b.r.e.IndexLevelActionFilter] [node-2] Readonly REST plugin was loaded…
[2018-03-15T14:45:50,857][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2018-03-15T14:45:51,388][INFO ][t.b.r.e.SSLTransportNetty4] Loaded good settings from C:\ProgramData\es\elasticsearch-6.2.2\config\readonlyrest.yml
[2018-03-15T14:45:51,419][INFO ][t.b.r.e.SSLTransportNetty4] creating SSL transport
[2018-03-15T14:45:51,435][INFO ][o.e.d.DiscoveryModule ] [node-2] using discovery type [zen]
[2018-03-15T14:45:51,935][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:45:52,138][INFO ][o.e.n.Node ] [node-2] initialized
[2018-03-15T14:45:52,138][INFO ][o.e.n.Node ] [node-2] starting …
[2018-03-15T14:45:52,341][INFO ][o.e.t.TransportService ] [node-2] publish_address {192.168.1.20:9300}, bound_addresses {192.168.1.20:9300}
[2018-03-15T14:45:52,482][INFO ][o.e.b.BootstrapChecks ] [node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-15T14:45:52,951][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-2] no known master node, scheduling a retry
[2018-03-15T14:45:55,876][INFO ][o.e.c.s.ClusterApplierService] [node-2] detected_master {node-1}{U-KoEGeaSa-QXhqq5Wd4Eg}{LzA1akbfSqOg7sp2mWW7lA}{192.168.1.10}{192.168.1.10:9300}, added {{node-1}{U-KoEGeaSa-QXhqq5Wd4Eg}{LzA1akbfSqOg7sp2mWW7lA}{192.168.1.10}{192.168.1.10:9300},}, reason: apply cluster state (from master [master {node-1}{U-KoEGeaSa-QXhqq5Wd4Eg}{LzA1akbfSqOg7sp2mWW7lA}{192.168.1.10}{192.168.1.10:9300} committed version [1398]])
[2018-03-15T14:45:56,048][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:45:56,157][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: attempting with JKS keystore…
[2018-03-15T14:45:56,173][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: ssl.key_alias not configured, took first alias in keystore: caKey
[2018-03-15T14:45:56,423][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: Discovered key from JKS
[2018-03-15T14:45:56,485][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: Discovered cert chain from JKS
[2018-03-15T14:45:56,657][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: Using SSL provider: JDK
[2018-03-15T14:45:57,204][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:45:58,079][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: Available ciphers: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA
[2018-03-15T14:45:58,142][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: Available SSL protocols: TLSv1,TLSv1.1,TLSv1.2
[2018-03-15T14:45:58,157][INFO ][t.b.r.e.SSLTransportNetty4] ROR SSL: Restricting to SSL protocols: TLSv1.2
[2018-03-15T14:45:58,329][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:45:58,845][INFO ][t.b.r.e.SSLTransportNetty4] [node-2] publish_address {192.168.1.20:9200}, bound_addresses {192.168.1.20:9200}
[2018-03-15T14:45:58,860][INFO ][o.e.n.Node ] [node-2] started
[2018-03-15T14:45:59,392][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:00,407][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:01,423][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:02,501][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:03,689][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:04,814][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:05,861][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready…
[2018-03-15T14:46:06,877][INFO ][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] index settings not found. Will keep on using the local YAML file. Learn more about clusterwide settings at Pro - ReadonlyREST

There is no ALLOWED/FORBIDDEN trace of any request.

Are you sure there is traffic?

Yes there is traffic:

I can search the data in Kibana and is getting pulled in near real time.

WLB.exe -e Side:
PS C:\Program Files\wlb> .\winlogbeat.exe -e
2018-03-15T15:19:12.162-0500 INFO instance/beat.go:468 Home path: [C:\Program Files\wlb] Config path: [C:\Program Files\wlb] Data path: [C:\Program Files\wlb\data] Logs path: [C:\Program Files\wlb\logs]
2018-03-15T15:19:12.163-0500 INFO instance/beat.go:475 Beat UUID: baa8fabc-dab0-473b-aa1e-63e5f8160518
2018-03-15T15:19:12.164-0500 INFO instance/beat.go:213 Setup Beat: winlogbeat; Version: 6.2.2
2018-03-15T15:19:12.164-0500 WARN instance/metrics_other.go:8 Metrics not implemented for this OS.
2018-03-15T15:19:12.183-0500 INFO elasticsearch/client.go:145 Elasticsearch url: https://my.company.com:9200
2018-03-15T15:19:12.184-0500 INFO pipeline/module.go:76 Beat name: Win-10-23j3
2018-03-15T15:19:12.184-0500 INFO beater/winlogbeat.go:56 State will be read from and persisted to C:\Program Files\wlb\data.winlogbeat.yml
2018-03-15T15:19:12.185-0500 INFO instance/beat.go:301 winlogbeat start running.
2018-03-15T15:19:12.185-0500 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-03-15T15:19:13.529-0500 WARN transport/tls.go:227 SSL/TLS verifications disabled.
2018-03-15T15:19:13.628-0500 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.2.2
2018-03-15T15:19:13.642-0500 INFO template/load.go:73 Template already exists and will not be overwritten.
2018-03-15T15:19:14.531-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 100 events
2018-03-15T15:19:14.733-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:15.088-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:15.328-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:15.568-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:15.769-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:15.992-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:17.021-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 50 events
2018-03-15T15:19:17.232-0500 INFO beater/eventlogger.go:56 EventLog[Microsoft-Windows-Powershell/Operational] successfully published 49 events

I modified the rootLogger.level to debug:

[2018-03-15T15:43:26,340][DEBUG][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] checking index…
[2018-03-15T15:43:31,356][DEBUG][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster is ready!
[2018-03-15T15:43:31,356][DEBUG][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] checking index…
[2018-03-15T15:43:36,005][DEBUG][i.n.h.s.SslHandler ] [id: 0x1f545c74, L:/192.168.1.20:9200 - R:/192.168.1.77:62686] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-03-15T15:43:36,161][DEBUG][i.n.h.c.c.ZlibCodecFactory] -Dio.netty.noJdkZlibDecoder: false
[2018-03-15T15:43:36,161][DEBUG][i.n.h.c.c.ZlibCodecFactory] -Dio.netty.noJdkZlibEncoder: false
[2018-03-15T15:43:36,364][DEBUG][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster is ready!
[2018-03-15T15:43:36,364][DEBUG][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] checking index…
[2018-03-15T15:43:41,371][DEBUG][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster is ready!
[2018-03-15T15:43:41,371][DEBUG][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] checking index…
[2018-03-15T15:43:46,386][DEBUG][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster is ready!
[2018-03-15T15:43:46,386][DEBUG][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] checking index…
[2018-03-15T15:43:51,396][DEBUG][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster is ready!
[2018-03-15T15:43:51,396][DEBUG][t.b.r.e.SettingsObservableImpl] [CLUSTERWIDE SETTINGS] checking index…

Verify that the winlogbeats is pointing at the correct IP of the ES node that has ROR installed. I’m under the impression you’re submitting events to another node.

If in doubt, point your browser to the ROR node’s IP and you should get a log line per request.

Thanks for the help Simone, I greatly appreciate it!

I have the two nodes and I am actually sending logs to a domain name that resolves to node-2’s IP address rather than sending logs directly to an IP address. I can try using the IP instead to see if that will make a difference. Stay tuned.

So I removed the second node and pointed WLB to the only node alive. I removed any cluster settings and made sure that everything resolves to an IP Address. WLB still uses TLS 1.2 to communicate without any issues and the authentication is not taken into consideration. So WLB is still ignoring authentication with credentials.

As far as your tip to browse to the node’s IP, I don’t see a log line per request but rather:
(Am I missing something else besides: https://192.168.1.20:9200?)
{
“name” : “WjxP_pz”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “n0khRTJzTdm3sNauhoLh7g”,
“version” : {
“number” : “6.2.2”,
“build_hash” : “10b1edd”,
“build_date” : “2018-02-16T19:01:30.685723Z”,
“build_snapshot” : false,
“lucene_version” : “7.2.1”,
“minimum_wire_compatibility_version” : “5.6.0”,
“minimum_index_compatibility_version” : “5.0.0”
},
“tagline” : “You Know, for Search”
}

@napsta this does not make sense, nodes communicate through the transport API, which is not touched by ROR.

Another question, can you see in ES startup logs you can see lines like this?

[2018-03-16T13:34:45,364][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:	

Yeah I believe I had that in their for testing. I have taken it out since then.

I do not see that in any t.b.r.a.ACL logs in the INFOs. I do see that readonlyrest.yml loaded good settings and can verify TLS works across the stack.

Well this is not good! please make sure you have ROR 1.16.17, and try to add enable: true under readonlyrest:.

readonlyrest:
    enable: true
    ...

That definitely helped.

Thanks! This should helped with some troubleshooting!

[2018-03-16T13:52:20,000][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:1097684555-2010382356#5, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:20,000][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:221415149-2131140136#6, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:22,479][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:798526275-1084316240#7, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:24,997][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:926000286-2089655182#8, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:25,325][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:177139867–317928098#9, TYP:MultiSearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:indices:data/read/msearch, OA:192.168.1.20, IDX:winlogbeat-, MET:POST, PTH:/_msearch, CNT:<OMITTED, LENGTH=714>, HDR:Connection,content-length,content-type,Host,x-forwarded-for,x-forwarded-port,x-forwarded-proto, HIS: } ←[0m
[2018-03-16T13:52:26,287][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:1933044640-1986072089#10, TYP:BulkRequest, CGR:N/A, USR:wlb(?), BRS:true, ACT:indices:data/write/bulk, OA:192.168.1.77, IDX:winlogbeat-6.2.2-2018.03.16, MET:POST, PTH:/_bulk, CNT:<OMITTED, LENGTH=3613>, HDR:Accept,Accept-Encoding,Authorization,Content-Length,Content-Type,Host,User-Agent, HIS: } ←[0m
[2018-03-16T13:52:26,771][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:281822418–1440845402#11, TYP:MultiSearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:indices:data/read/msearch, OA:192.168.1.20, IDX:winlogbeat-
, MET:POST, PTH:/_msearch, CNT:<OMITTED, LENGTH=714>, HDR:Connection,content-length,content-type,Host,x-forwarded-for,x-forwarded-port,x-forwarded-proto, HIS: } ←[0m
[2018-03-16T13:52:27,291][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:380775009-2080337325#12, TYP:MainRequest, CGR:N/A, USR:wlb(?), BRS:true, ACT:cluster:monitor/main, OA:192.168.1.77, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept,Accept-Encoding,Authorization,content-length,Host,User-Agent, HIS: } ←[0m
[2018-03-16T13:52:27,510][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:580698470-628758134#13, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:29,311][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:635999601-1621821083#14, TYP:MainRequest, CGR:N/A, USR:wlb(?), BRS:true, ACT:cluster:monitor/main, OA:192.168.1.77, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept,Accept-Encoding,Authorization,content-length,Host,User-Agent, HIS: } ←[0m
[2018-03-16T13:52:30,018][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:912558-700113650#15, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:32,530][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:1371801414-812012306#16, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:33,321][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:146867291-1866377726#17, TYP:MainRequest, CGR:N/A, USR:wlb(?), BRS:true, ACT:cluster:monitor/main, OA:192.168.1.77, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept,Accept-Encoding,Authorization,content-length,Host,User-Agent, HIS: } ←[0m
[2018-03-16T13:52:35,036][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:459912881-1769053146#18, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:36,903][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:1217653824-1600680422#19, TYP:MultiSearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:indices:data/read/msearch, OA:192.168.1.20, IDX:winlogbeat-, MET:POST, PTH:/_msearch, CNT:<OMITTED, LENGTH=714>, HDR:Connection,content-length,content-type,Host,x-forwarded-for,x-forwarded-port,x-forwarded-proto, HIS: } ←[0m
[2018-03-16T13:52:37,552][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:483218287-481197937#20, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:40,065][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:108421830-2015739272#21, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:41,319][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:105206851-319468494#22, TYP:MainRequest, CGR:N/A, USR:wlb(?), BRS:true, ACT:cluster:monitor/main, OA:192.168.1.77, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept,Accept-Encoding,Authorization,content-length,Host,User-Agent, HIS: } ←[0m
[2018-03-16T13:52:42,580][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:177238744-651937925#23, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:45,087][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:774848515-1264951639#24, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:47,037][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:2123061974-709755878#25, TYP:MultiSearchRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:indices:data/read/msearch, OA:192.168.1.20, IDX:winlogbeat-
, MET:POST, PTH:/_msearch, CNT:<OMITTED, LENGTH=714>, HDR:Connection,content-length,content-type,Host,x-forwarded-for,x-forwarded-port,x-forwarded-proto, HIS: } ←[0m
[2018-03-16T13:52:47,592][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:1611242323-1400360493#26, TYP:MainRequest, CGR:N/A, USR:[no basic auth header], BRS:false, ACT:cluster:monitor/main, OA:192.168.1.20, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:Connection,Content-Length,Host, HIS: } ←[0m
[2018-03-16T13:52:48,663][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-2] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])

what helped? The enable: true? Or the update to the latest version?

enable: true did the trick.

I believe I have ROR 1.16.16.

I am looking into additional configuration now. I believe I have everything in both ROR.yml and wlb.yml but it is refusing connections.

WLB:
2018-03-16T15:22:53.617-0500 ERROR pipeline/output.go:74 Failed to connect: 403 Forbidden: {“error”:{“root_cause”:[{“type”:“status_exception”,“reason”:“forbidden”}],“type”:“status_exception”,“reason”:“forbidden”},“status”:403}

Elastic:
[2018-03-16T15:23:53,565][INFO ][t.b.r.a.ACL ] ←[35mFORBIDDEN by default req={ ID:1874114636-2061059071#10225, TYP:MainRequest, CGR:N/A, USR:wlb(?), BRS:true, ACT:cluster:monitor/main, OA:192.168.1.77, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept,Accept-Encoding,Authorization,content-length,Host,User-Agent, HIS: } ←[0m

Why the history (HIS:) is empty? Is your ACL empty?

I might need to revisit the documentation.

I am not sure why that shows empty. Are all of the settings I need to configure stored in the readonlyrest.yml file?

This file is stored alongside elasticsearch.yml and appears to work since I have set enable: true.

Currently, this is the readonlyrest.yml file that I am using:

readonlyrest:
enable: true
ssl:
enable: true
keystore_file: "keystore.jks"
keystore_pass: readonlyrest
key_pass: readonlyrest
allowed_protocols: [TLSv1.2]
access_control_rules:
- name: "winlogbeat can write and create its own indices"
auth_key_sha256: cc2db7731825a9cb972e9e3860285fc04b56bbd71b99a3e72ab7d8e3c46a55a2
actions: [“indices:data/read/","indices:data/write/”,“indices:admin/template/",“indices:admin/create”]
indices: ["winlogbeat-
”, “.kibana*”]
#cc2db7731825a9cb972e9e3860285fc04b56bbd71b99a3e72ab7d8e3c46a55a2 = winlogbeat:winlogbeat (As seen in winlogbeat.yml)