OpenShift, filebeat, ElasticSearch and ReadonlyREST


(Fabio Martinelli) #1

Dear Colleagues

I run a 6 nodes OpenShift cluster on which I’ve deployed filebeat 6.2.3 as an RPM on each node ( systemd service )

these 6 filebeat instances should send their /var/log/*log files to an ElasticSearch OpenShift project where I’ve also deployed ReadonlyREST ; there is no SSL encryption at this point and I’m fine with it.

I’m a ReadonlyREST newbie so I’m pretty sure that I’m missing some basic detail here, the thing is that ElasticSearch starts inside of its OpenShift project but it dumps forever ‘Cluster not ready…’ :

++ /usr/share/elasticsearch/bin/elasticsearch --version
Version: 6.2.3, Build: c59ff00/2018-03-13T10:06:29.741383Z, JVM: 1.8.0_161
++ /usr/share/elasticsearch/bin/elasticsearch --verbose
[2018-04-08T19:48:45,103][INFO ][o.e.n.Node               ] [] initializing ...
[2018-04-08T19:48:45,613][INFO ][o.e.e.NodeEnvironment    ] [KsQos_t] using [1] data paths, mounts [[/var/lib/elasticsearch (10.11.41.85:vol_0647f24efa7a20201ba4fc5f19608e18)]], net usable_space [18.8gb], net total_space [18.9gb], types [fuse.glusterfs]
[2018-04-08T19:48:45,614][INFO ][o.e.e.NodeEnvironment    ] [KsQos_t] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-04-08T19:48:45,937][INFO ][o.e.n.Node               ] node name [KsQos_t] derived from node ID [KsQos_txTzu0_EKEnwP2QA]; set [node.name] to override
[2018-04-08T19:48:45,938][INFO ][o.e.n.Node               ] version[6.2.3], pid[113], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-04-08T19:48:45,938][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.zNVERRwW, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-04-08T19:48:52,881][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [aggs-matrix-stats]
[2018-04-08T19:48:52,881][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [analysis-common]
[2018-04-08T19:48:52,881][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [ingest-common]
[2018-04-08T19:48:52,882][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [lang-expression]
[2018-04-08T19:48:52,882][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [lang-mustache]
[2018-04-08T19:48:52,882][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [lang-painless]
[2018-04-08T19:48:52,883][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [mapper-extras]
[2018-04-08T19:48:52,883][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [parent-join]
[2018-04-08T19:48:52,883][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [percolator]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [rank-eval]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [reindex]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [repository-url]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [transport-netty4]
[2018-04-08T19:48:52,885][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded module [tribe]
[2018-04-08T19:48:52,886][INFO ][o.e.p.PluginsService     ] [KsQos_t] loaded plugin [readonlyrest]
[2018-04-08T19:49:01,218][INFO ][t.b.r.e.SettingsObservableImpl] Read data from /etc/elasticsearch/readonlyrest.yml
[2018-04-08T19:49:01,740][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Read data from /etc/elasticsearch/readonlyrest.yml
[2018-04-08T19:49:01,760][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Settings observer refreshing...
[2018-04-08T19:49:02,403][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:	{ name: 'Accept all requests from localhost', policy: ALLOW}
[2018-04-08T19:49:02,404][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:	{ name: 'Kibana needs', policy: ALLOW}
[2018-04-08T19:49:02,405][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:	{ name: 'Accept all requests provided that you know the password', policy: ALLOW}
[2018-04-08T19:49:02,405][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Configuration reloaded - ReadonlyREST enabled
[2018-04-08T19:49:02,405][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Readonly REST plugin was loaded...
[2018-04-08T19:49:02,542][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2018-04-08T19:49:03,058][INFO ][o.e.d.DiscoveryModule    ] [KsQos_t] using discovery type [zen]
[2018-04-08T19:49:03,421][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2018-04-08T19:49:03,837][INFO ][o.e.n.Node               ] initialized
[2018-04-08T19:49:03,837][INFO ][o.e.n.Node               ] [KsQos_t] starting ...
[2018-04-08T19:49:04,428][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2018-04-08T19:49:04,549][INFO ][o.e.t.TransportService   ] [KsQos_t] publish_address {10.128.2.193:9300}, bound_addresses {[::]:9300}
[2018-04-08T19:49:04,647][INFO ][o.e.b.BootstrapChecks    ] [KsQos_t] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-08T19:49:05,433][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [KsQos_t] no known master node, scheduling a retry
[2018-04-08T19:49:07,754][INFO ][o.e.c.s.MasterService    ] [KsQos_t] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {KsQos_t}{KsQos_txTzu0_EKEnwP2QA}{ziK01llDSrKbitcrHg_ixA}{10.128.2.193}{10.128.2.193:9300}
[2018-04-08T19:49:07,766][INFO ][o.e.c.s.ClusterApplierService] [KsQos_t] new_master {KsQos_t}{KsQos_txTzu0_EKEnwP2QA}{ziK01llDSrKbitcrHg_ixA}{10.128.2.193}{10.128.2.193:9300}, reason: apply cluster state (from master [master {KsQos_t}{KsQos_txTzu0_EKEnwP2QA}{ziK01llDSrKbitcrHg_ixA}{10.128.2.193}{10.128.2.193:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-08T19:49:07,786][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2018-04-08T19:49:07,800][INFO ][o.e.h.n.Netty4HttpServerTransport] [KsQos_t] publish_address {10.128.2.193:9200}, bound_addresses {[::]:9200}
[2018-04-08T19:49:07,800][INFO ][o.e.n.Node               ] [KsQos_t] started
[2018-04-08T19:49:08,422][INFO ][o.e.g.GatewayService     ] [KsQos_t] recovered [2] indices into cluster_state
[2018-04-08T19:49:08,788][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...

I can provide my conf file if needed, kindly what might be wrong here ?

Many thanks in advance,
cheers
Fabio


(Simone Scarduzio) #2

Hi @fabiomartinelli,
The cluster not ready message is there because your cluster never reaches the green state. Please check the meaning of cluster states and make sure your cluster gets to the green state.

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/_cluster_health.html

Having said that, the failing settings poller is not fatal for ReadonlyREST and you can still try to send test data from the file beats and see if they get authenticated correctly.

And yes, show us your settings. We could review them if you want.