Dear Colleagues
I run a 6 nodes OpenShift cluster on which I’ve deployed filebeat 6.2.3 as an RPM on each node ( systemd service )
these 6 filebeat instances should send their /var/log/*log files to an ElasticSearch OpenShift project where I’ve also deployed ReadonlyREST ; there is no SSL encryption at this point and I’m fine with it.
I’m a ReadonlyREST newbie so I’m pretty sure that I’m missing some basic detail here, the thing is that ElasticSearch starts inside of its OpenShift project but it dumps forever ‘Cluster not ready…’ :
++ /usr/share/elasticsearch/bin/elasticsearch --version
Version: 6.2.3, Build: c59ff00/2018-03-13T10:06:29.741383Z, JVM: 1.8.0_161
++ /usr/share/elasticsearch/bin/elasticsearch --verbose
[2018-04-08T19:48:45,103][INFO ][o.e.n.Node ] [] initializing ...
[2018-04-08T19:48:45,613][INFO ][o.e.e.NodeEnvironment ] [KsQos_t] using [1] data paths, mounts [[/var/lib/elasticsearch (10.11.41.85:vol_0647f24efa7a20201ba4fc5f19608e18)]], net usable_space [18.8gb], net total_space [18.9gb], types [fuse.glusterfs]
[2018-04-08T19:48:45,614][INFO ][o.e.e.NodeEnvironment ] [KsQos_t] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-04-08T19:48:45,937][INFO ][o.e.n.Node ] node name [KsQos_t] derived from node ID [KsQos_txTzu0_EKEnwP2QA]; set [node.name] to override
[2018-04-08T19:48:45,938][INFO ][o.e.n.Node ] version[6.2.3], pid[113], build[c59ff00/2018-03-13T10:06:29.741383Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-04-08T19:48:45,938][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.zNVERRwW, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-04-08T19:48:52,881][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [aggs-matrix-stats]
[2018-04-08T19:48:52,881][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [analysis-common]
[2018-04-08T19:48:52,881][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [ingest-common]
[2018-04-08T19:48:52,882][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [lang-expression]
[2018-04-08T19:48:52,882][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [lang-mustache]
[2018-04-08T19:48:52,882][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [lang-painless]
[2018-04-08T19:48:52,883][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [mapper-extras]
[2018-04-08T19:48:52,883][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [parent-join]
[2018-04-08T19:48:52,883][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [percolator]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [rank-eval]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [reindex]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [repository-url]
[2018-04-08T19:48:52,884][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [transport-netty4]
[2018-04-08T19:48:52,885][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded module [tribe]
[2018-04-08T19:48:52,886][INFO ][o.e.p.PluginsService ] [KsQos_t] loaded plugin [readonlyrest]
[2018-04-08T19:49:01,218][INFO ][t.b.r.e.SettingsObservableImpl] Read data from /etc/elasticsearch/readonlyrest.yml
[2018-04-08T19:49:01,740][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Read data from /etc/elasticsearch/readonlyrest.yml
[2018-04-08T19:49:01,760][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Settings observer refreshing...
[2018-04-08T19:49:02,403][INFO ][t.b.r.a.ACL ] ADDING BLOCK: { name: 'Accept all requests from localhost', policy: ALLOW}
[2018-04-08T19:49:02,404][INFO ][t.b.r.a.ACL ] ADDING BLOCK: { name: 'Kibana needs', policy: ALLOW}
[2018-04-08T19:49:02,405][INFO ][t.b.r.a.ACL ] ADDING BLOCK: { name: 'Accept all requests provided that you know the password', policy: ALLOW}
[2018-04-08T19:49:02,405][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Configuration reloaded - ReadonlyREST enabled
[2018-04-08T19:49:02,405][INFO ][t.b.r.e.IndexLevelActionFilter] [KsQos_t] Readonly REST plugin was loaded...
[2018-04-08T19:49:02,542][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin
[2018-04-08T19:49:03,058][INFO ][o.e.d.DiscoveryModule ] [KsQos_t] using discovery type [zen]
[2018-04-08T19:49:03,421][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2018-04-08T19:49:03,837][INFO ][o.e.n.Node ] initialized
[2018-04-08T19:49:03,837][INFO ][o.e.n.Node ] [KsQos_t] starting ...
[2018-04-08T19:49:04,428][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2018-04-08T19:49:04,549][INFO ][o.e.t.TransportService ] [KsQos_t] publish_address {10.128.2.193:9300}, bound_addresses {[::]:9300}
[2018-04-08T19:49:04,647][INFO ][o.e.b.BootstrapChecks ] [KsQos_t] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-08T19:49:05,433][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [KsQos_t] no known master node, scheduling a retry
[2018-04-08T19:49:07,754][INFO ][o.e.c.s.MasterService ] [KsQos_t] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {KsQos_t}{KsQos_txTzu0_EKEnwP2QA}{ziK01llDSrKbitcrHg_ixA}{10.128.2.193}{10.128.2.193:9300}
[2018-04-08T19:49:07,766][INFO ][o.e.c.s.ClusterApplierService] [KsQos_t] new_master {KsQos_t}{KsQos_txTzu0_EKEnwP2QA}{ziK01llDSrKbitcrHg_ixA}{10.128.2.193}{10.128.2.193:9300}, reason: apply cluster state (from master [master {KsQos_t}{KsQos_txTzu0_EKEnwP2QA}{ziK01llDSrKbitcrHg_ixA}{10.128.2.193}{10.128.2.193:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-08T19:49:07,786][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
[2018-04-08T19:49:07,800][INFO ][o.e.h.n.Netty4HttpServerTransport] [KsQos_t] publish_address {10.128.2.193:9200}, bound_addresses {[::]:9200}
[2018-04-08T19:49:07,800][INFO ][o.e.n.Node ] [KsQos_t] started
[2018-04-08T19:49:08,422][INFO ][o.e.g.GatewayService ] [KsQos_t] recovered [2] indices into cluster_state
[2018-04-08T19:49:08,788][INFO ][t.b.r.c.s.SettingsPoller ] [CLUSTERWIDE SETTINGS] Cluster not ready...
I can provide my conf file if needed, kindly what might be wrong here ?
Many thanks in advance,
cheers
Fabio