java.lang.IllegalStateException: Unsupported http.type [ssl_netty4]

Hi.

A clean setup is failing to start with the following error message:

{"type": "deprecation.elasticsearch", "timestamp": "2022-09-23T18:46:21,448Z", "level": "CRITICAL", "component": "o.e.d.a.ActionModule", "cluster.name": "qa", "node.name": "esn-il01.xcade.net", "message": "The tech.beshu.ror.es.ReadonlyRestPlugin plugin installs a custom REST wrapper. This functionality is deprecated and will not be possible in Elasticsearch 8.0. If this plugin is intended to provide security features for Elasticsearch then you should switch to using the built-in Elasticsearch features instead.", "key": "3rd_party_rest_deprecation", "category": "plugins" }
{"type": "server", "timestamp": "2022-09-23T18:46:21,501Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "qa", "node.name": "esn-il01.xcade.net", "message": "creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]" }
{"type": "server", "timestamp": "2022-09-23T18:46:21,525Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "qa", "node.name": "esn-il01.xcade.net", "message": "uncaught exception in thread [main]", 
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unsupported http.type [ssl_netty4]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.3.jar:7.16.3]",
"Caused by: java.lang.IllegalStateException: Unsupported http.type [ssl_netty4]",
"at org.elasticsearch.common.network.NetworkModule.getHttpServerTransportSupplier(NetworkModule.java:227) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.newHttpTransport(Node.java:1587) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.<init>(Node.java:816) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.3.jar:7.16.3]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.3.jar:7.16.3]",
"... 6 more"] }
uncaught exception in thread [main]
java.lang.IllegalStateException: Unsupported http.type [ssl_netty4]
	at org.elasticsearch.common.network.NetworkModule.getHttpServerTransportSupplier(NetworkModule.java:227)
	at org.elasticsearch.node.Node.newHttpTransport(Node.java:1587)
	at org.elasticsearch.node.Node.<init>(Node.java:816)
	at org.elasticsearch.node.Node.<init>(Node.java:309)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/qa.log

The configuration file contains:

xpack.security.enabled: false
...
http.type: ssl_netty4

ROR plugin is installed, as shown in the first line of the log.

How could I solve the problem?

Thanks in advance

@coutoPL any insight on this?

@gustavo.yoshizaki please can you let us have the full elasticsearch logs from boot to crash?

Hi @sscarduzio

I can. But in private. Is that enough?

Thanks in advance

1 Like

nope. Did you use docker to run the ES?

Hi @coutoPL

Yes. It is running docker. I have 4 other ES clusters running with the same setup without issues.

Thanks in advance.

hmm, are you trying to say that there is no difference between the setup of these clusters (the same ROR version, the same ES version, the same ES docker image) and one of them doesn’t start because of the error shown above?

From the point of view of the software, there are no differences. Same ROR, ES, ES image and OS version.
From the point of view of configuration, the differences are related to node names and cluster size.
From the point of view of hardware, 2 clusters have much more capacity, 1 is very similar and the last one is even smaller.

Here is a fragment of the logs of a cluster launched today (to check if the problem could be replicated):

{"type": "server", "timestamp": "2022-09-23T18:15:32,978Z", "level": "INFO", "component": "t.b.r.c.l.ConfigLoadingInterpreter$", "cluster.name": "piab-cluster", "node.name": "node-piab", "message": "Loading ReadonlyREST settings from index failed: cannot find index", "cluster.uuid": "QTvNv2phRiaKDFHrti09Fw", "node.id": "ckEnlZFQRBiUSjm3PPl6vg"  }
{"type": "server", "timestamp": "2022-09-23T18:15:32,978Z", "level": "INFO", "component": "t.b.r.c.l.ConfigLoadingInterpreter$", "cluster.name": "piab-cluster", "node.name": "node-piab", "message": "Loading ReadonlyREST settings from file: Path(/usr/share/elasticsearch/config), because index not exist", "cluster.uuid": "QTvNv2phRiaKDFHrti09Fw", "node.id": "ckEnlZFQRBiUSjm3PPl6vg"  }
{"type": "server", "timestamp": "2022-09-23T18:15:37,253Z", "level": "INFO", "component": "t.b.r.a.f.RawRorConfigBasedCoreFactory", "cluster.name": "piab-cluster", "node.name": "node-piab", "message": "ADDING BLOCK:\t{ name: 'Read actions for kibana_system', policy: ALLOW, rules: [actions]", "cluster.uuid": "QTvNv2phRiaKDFHrti09Fw", "node.id": "ckEnlZFQRBiUSjm3PPl6vg"  }
...
{"type": "server", "timestamp": "2022-09-23T18:15:37,308Z", "level": "WARN", "component": "t.b.r.b.ReadonlyRest", "cluster.name": "piab-cluster", "node.name": "node-piab", "message": "Defined fls engine relies on lucene. To make it work well, all nodes should have ROR plugin installed.", "cluster.uuid": "QTvNv2phRiaKDFHrti09Fw", "node.id": "ckEnlZFQRBiUSjm3PPl6vg"  }
{"type": "server", "timestamp": "2022-09-23T18:15:37,321Z", "level": "INFO", "component": "t.b.r.b.RorInstance", "cluster.name": "piab-cluster", "node.name": "node-piab", "message": "ReadonlyREST core was loaded ...", "cluster.uuid": "QTvNv2phRiaKDFHrti09Fw", "node.id": "ckEnlZFQRBiUSjm3PPl6vg"  }

Yes you can send relevant private logs to support_group as a direct message.

Sorry @coutoPL
I have just double checked and 2 of the mentioned clusters have ES 7.16.1 (and the related ROR plugin). But the logs sent are from the cluster with the same ES version.

do you confirm that after removing http.type: ssl_netty4 the startup doesn’t fail and you are able to reach ES data through its HTTP API?

By removing that option, the service is not restarting every minute. However it is not available. The cluster itself fails because it is not able to contact the master nodes:

{"type": "server", "timestamp": "2022-09-25T16:11:04,752Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "qa", "node.name": "XXX", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [XXX, XXX, XXX] to bootstrap a cluster: have discovered [{XXX}{K7kFtzEoTXC94G354DMJww}{tnnTzlXdTAikYDa4ub8q1w}{XXX}{XXX}{cdfhimrstw}]; discovery will continue using [XXX,XXX] from hosts providers and [{XXX}{K7kFtzEoTXC94G354DMJww}{tnnTzlXdTAikYDa4ub8q1w}{XXX}{XXX}{cdfhimrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

And the request to the HTTP API:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "forbidden_response",
        "reason" : "forbidden",
        "due_to" : "READONLYREST_FAILED_TO_START"
      }
    ],
    "type" : "forbidden_response",
    "reason" : "forbidden",
    "due_to" : "READONLYREST_FAILED_TO_START"
  },
  "status" : 403
}

The logs related to ROR error are:

{"type": "server", "timestamp": "2022-09-25T16:08:19,916Z", "level": "INFO", "component": "t.b.r.c.l.ConfigLoadingInterpreter$", "cluster.name": "qa", "node.name": "esn-il01.xcade.net", "message": "Loading ReadonlyREST settings from index failed: cannot find index" }
{"type": "server", "timestamp": "2022-09-25T16:08:19,917Z", "level": "INFO", "component": "t.b.r.c.l.ConfigLoadingInterpreter$", "cluster.name": "qa", "node.name": "esn-il01.xcade.net", "message": "Loading ReadonlyREST settings from file: Path(/usr/share/elasticsearch/config), because index not exist" }

Tried with a full restart of docker service, just in case there was something cached. No luck.

this error means that ROR was not able to start because of some misconfiguration. Assuming that you disabled SSL (by removing http.type: ssl_netty4) it looks like it is sth wrong in your readonlyrest.yml file. There should be an error log explaining wthat goes wrong (probably with stack trace). Do you see it?

Bingo!
There was a credential breaking the YAML.

Thanks!

1 Like