Elaticsearch 8.4.1 - RoR plugin - elasticsearch does not start - Netty4Utils not found

orsius · September 19, 2022, 1:47pm

Dear All, I encounter an issue install the RoR plugin and not abale to start elasticsearch with the plugin install.

In brief:

the error message is: Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/transport/netty4/Netty4Utils
I use elaticsearch 8.4.1 (basic licence) with security enable (user:passwd + tls http and transport)
My goal is to use RoR for the LDAP feature as well as the kibana features.

Any ideas ?

Thanks in advance for your time and help.

kr,

nb: here is a bit more details:

# java -version
  openjdk version "11.0.16" 2022-07-19
  OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
  OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

# os used
  Ubuntu
  VERSION="20.04.5 LTS (Focal Fossa)"

# as elasticsearch user
export ES_HOME=/usr/share/elasticsearch && export ES_PATH_CONF=/etc/elasticsearch/monitoring
/usr/share/elasticsearch$ ./bin/elasticsearch-plugin list
/usr/share/elasticsearch$ ./bin/elasticsearch-plugin install file:///home/ubuntuadmin/downloads/readonlyrest-1.43.0_es8.4.1.zip
-> Installing file:///home/ubuntuadmin/downloads/readonlyrest-1.43.0_es8.4.1.zip
-> Downloading file:///home/ubuntuadmin/downloads/readonlyrest-1.43.0_es8.4.1.zip
[=================================================] 100%
. . .
-> Installed readonlyrest
-> Please restart Elasticsearch to activate any plugins installed
/usr/share/elasticsearch$ ./bin/elasticsearch-plugin list
readonlyrest

# sudoer
systemctl restart elasticsearch

[2022-09-19T15:09:25,657][ERROR][o.e.b.Elasticsearch      ] [lucrpdvop01-monitoring] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: failed to load plugin class [tech.beshu.ror.es.ReadonlyRestPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:552) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:463) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:274) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:150) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:579) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:399) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.1.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:543) ~[elasticsearch-8.4.1.jar:?]
        ... 9 more
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/transport/netty4/Netty4Utils
        at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$new$1(ReadonlyRestPlugin.scala:100) ~[?:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:26) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:25) ~[?:?]
        at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.scala:100) ~[?:?]
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:543) ~[elasticsearch-8.4.1.jar:?]
        ... 9 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.transport.netty4.Netty4Utils
        at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:588) ~[?:?]
        at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?]
        at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$new$1(ReadonlyRestPlugin.scala:100) ~[?:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:26) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:25) ~[?:?]
        at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.scala:100) ~[?:?]
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:543) ~[elasticsearch-8.4.1.jar:?]
        ... 9 more

sscarduzio · September 19, 2022, 2:17pm

Hi Gautier! Nice to meet you, welcome to the forum

Did you patch Elasticsearch?

orsius · September 19, 2022, 2:43pm

Damned! My bad, I didn’t performed the prerequisites! Thanks for your quick answer.

That being said, I read that we can’t use RoR and elasticsearch security togheter ?

ReadonlyREST and X-Pack security module can’t run together, so the latter needs to be disabled.

So if I’m not wrong I can NOT use only the “ldap / active directory” feature of RoR ?

Just to illustrate what I’m thinking: Within my deployment chain, I have to install an unscured elasticserach cluster than install RoR … and for patching, I need to “unsecured the cluster” by removing the plugin install, unpatched, update elastic linux packages, install new ror plugin version, patch back then start a secure instance ? Hummm …

sscarduzio · September 19, 2022, 3:30pm

The steps for each ES node would be:

Install Elasticsearch from packages
Append xpack.security.enabled: false to elasticsearch.yml
cd to $ES_HOME
install ROR plugin in Elasticsearch
patch Elasticsearch with ror-tool
Create readonlyrest.yml
run Elasticsearch as a linux service

Do you require SSL between ES nodes?

orsius · September 20, 2022, 11:46am

@sscarduzio, Thanks a lot for your time and quick respond to my topic.

To answer your question: Yes my goal is to have an end to end secured elasticsearch cluster (https for transport communication and well as for http communication)

Following your replies, I set a simple readonlyrest.yml file:

readonlyrest:
    #optional
    response_if_req_forbidden: Sorry, your request is forbidden.
    access_control_rules:
    - name: Accept all requests from localhost
      hosts: [127.0.0.1]

This first step allow me to restrein access for anyother server:

## OK localhost
 curl localhost:9200/_cluster/health?filter_path=status
{"status":"yellow"}

## NOK with server IP addr
curl ${MY_SRV_IP}:9200/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "forbidden_response",
        "reason" : "Sorry, your request is forbidden.",
        "due_to" : "OPERATION_NOT_ALLOWED"
      }
    ],
    "type" : "forbidden_response",
    "reason" : "Sorry, your request is forbidden.",
    "due_to" : "OPERATION_NOT_ALLOWED"
  },
  "status" : 403
}

My next step is to set a more secure readonly rest file;

Is there a way to “hide/encrypt” the auth_key value ? (see example below with elastic user)

What do you recommend me as best practices ? Create a simple readonlyrest.yaml file at first and then write advance rules within the readonlyrest elasticsearch index (if it’s possible) ?

readonlyrest:
  #optional
  response_if_req_forbidden: Sorry, your request is forbidden.
  access_control_rules:
  # - name: "Block 1 - Allowing anything"
  #   type: allow
  # - name: Accept all requests from localhost
  #   hosts: [127.0.0.1]
  # - name: Just certain indices, and read only
  #   actions: ['indices:data/read/*']
  #   indices: ['all_my_public_indices_start_with*'] # index aliases are taken in account!
  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key: elastic:changeit

sscarduzio · September 20, 2022, 4:21pm

Yes of course, you can either hash the basic auth passwords using sha512, and other algorithms.
i.e.

  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key_sha512: "admin:280ac6f...94bf9"

Or you can inject environment variables anywhere in the file using @{env:MY_VALUE}. I.e.

  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key: elastic:${MY_SECRET_PASSWORD}

Actually you can inject the whole content of the file with a single var replacement

$ cat readonlyrest.yml
${ROR_CONF}

Full SSL Encryption

We are covered about that: inter-node transport encryption and HTTPS are fully supported in Elasticsearch (using jks and pem certificates formats)

Our kibana plugin supports HTTPS on the client and server side as well, so you should be able to achieve full encryption from the data store to the browser (cookies included).

orsius · September 27, 2022, 11:39am

Hello, Following your input I go a bit further and I encountered another issue: Kibana does not start

I received “Error: header too long” ; (see complete error messag below); any idea ?

Here is my current configuration:

readonlyrest.yml

readonlyrest:
  #optional
  response_if_req_forbidden: Sorry, your request is forbidden.
  ssl:
    keystore_file: "keystore_new.jks" # or keystore.p12 for PKCS#12 format
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    allowed_protocols: ["TLSv1.2", "TLSv1.3"]
    #  allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
  ssl_internode:
    keystore_file: "keystore_new.jks" # or keystore.p12 for PKCS#12 format
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    allowed_protocols: ["TLSv1.2", "TLSv1.3"]
    #  allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
  access_control_rules:
  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key_sha512: "kibana:1e64d7d1.......830c49"

kibana.yaml

server.host: "myservernamehere01"
server.name: "myservernamehere01"
server.ssl.enabled: true
server.ssl.keystore.path: "/etc/kibana/keystore.jks"
server.ssl.keystore.password: "readonlyrest"
server.ssl.supportedProtocols: ["TLSv1.2", "TLSv1.3"]
xpack.security.enabled: false
elasticsearch.hosts: ["https://myservernamehere01:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "mypasswordhere"
logging:
  appenders:
    file:
      type: file
      fileName: /data/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

Launching kibana /usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml I get the following ERR msg:

[ROR] - serve.js - intercepting config
[12:45:40:376] [warning][plugins][ReadonlyREST][unsupportedFeaturesNotifier] logging.appenders config parameter declared in kibana.yml is not supported yet. It will be ignored
[12:45:40:376] [warning][plugins][ReadonlyREST][unsupportedFeaturesNotifier] logging.root config parameter declared in kibana.yml is not supported yet. It will be ignored
[12:45:40:404] [info][plugins][ReadonlyREST][kibanaYamlSessionManagerConfigParser] "readonlyrest_kbn.store_sessions_in_index" is not enabled.
If you use more than one Kibana nodes (High availability configurations), you MUST enable both "readonlyrest_kbn.store_sessions_in_index" and share the same "readonlyrest.cookiePass" in ALL Kibana nodes.
See documentation: https://docs.readonlyrest.com/kibana#session-management-with-multiple-kibana-instances
[12:45:40:407] [info][plugins][ReadonlyREST][serverCreator] ROR PKP SSL server will accept protocols: TLSv1.2,TLSv1.3 (bitmask: 67108864)
[12:45:40:419] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:44625
[2022-09-27T12:45:40.428+02:00][WARN ][environment] pid file already exists at /run/kibana/kibana.pid
[12:45:40:455] [error][plugins][ReadonlyREST][serverCreator] Error creating https server:  Error: header too long
    at configSecureContext (node:internal/tls/secure-context:278:15)
    at Object.createSecureContext (node:_tls_common:117:3)
    at Server.setSecureContext (node:_tls_wrap:1346:27)
    at Server (node:_tls_wrap:1205:8)
    at new Server (node:https:69:3)
    at Object.createServer (node:https:105:10)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:3609)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:1406)
    at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12211)
    at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)
    at Generator.prototype.<computed> [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:32:7
    at new Promise (<anonymous>)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:21:12
    at /usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12325
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error: header too long
    at configSecureContext (node:internal/tls/secure-context:278:15)
    at Object.createSecureContext (node:_tls_common:117:3)
    at Server.setSecureContext (node:_tls_wrap:1346:27)
    at Server (node:_tls_wrap:1205:8)
    at new Server (node:https:69:3)
    at Object.createServer (node:https:105:10)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:3609)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:1406)
    at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12211)
    at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)
    at Generator.prototype.<computed> [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:32:7
    at new Promise (<anonymous>)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:21:12
    at /usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12325
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

sscarduzio · September 27, 2022, 2:03pm

Kibana can’t understand JKS unfortunately. You will need the *.p12 format.

orsius · September 28, 2022, 6:44am

Geat, it’s works! Thank you.

The plugin start properly BUT as I use “self signed” certificate, I received several error messages (kibana is up and running) in the logs; Is there any parameter I can use to ignore these log lines ?

[2022-09-28T08:35:14.561+02:00][INFO ][plugins.readonlyrestkbn] Setting up ReadonlyREST plugin - build info: {"versionString":"enterprise-1.43.0_es8.4.1","kibanaVersion":"8.4.1","rorEdition":"enterprise","rorVersion":"1.43.0","isProduction":true,"isEnterprise":true,"isPro":false,"isFree":false,"isBuildExpired":false}
[2022-09-28T08:35:14.562+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Raspbian 15.3 OS. Automatically enabling Chromium sandbox.
[2022-09-28T08:35:14.590+02:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-09-28T08:35:14.630+02:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
Error: self signed certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)
[2022-09-28T08:35:15.172+02:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Internal Server Error</pre>
</body>
</html>

Error: self signed certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)
[2022-09-28T08:35:15.616+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Error: self signed certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
. . .

sscarduzio · September 28, 2022, 7:16am

OK you need to tell Kibana’s Elasticsearch client to trust that cert. There’s some configuration for that in Kibana.

Have a look at this page where it talks about elasticsearch.ssl.certificateAuthorities.

orsius · October 5, 2022, 1:50pm

Thanks for your answer, I manage to setup something valid:

I * create a bundle for each nodes:

the bundle.p12 contains:
1. ca.crt
2. server.crt
3. server.key

create a keystore.jks that contains the bundle
convert the keystore.jks to pkcs12

I use this keystore for kibana and elasticsearch instance on my server.

my readonly rest tls section looks like this:

readonlyrest:
  enable: true
  prompt_for_basic_auth: false

# ## optional
  response_if_req_forbidden: Sorry, your request is forbidden.
  audit_collector: true
  audit_index_template: "'readonlyrest_audit'-yyyy"

# ## TLS
  ssl:
    keystore_file: "ssl/keystore_pkcs12.jks" # or keystore.p12 for PKCS#12 format
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    allowed_protocols: ["TLSv1.2", "TLSv1.3"]
    #  allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
    client_authentication: true
    certificate_verification: true
  ssl_internode:
    keystore_file: "ssl/keystore_pkcs12.jks" # or keystore.p12 for PKCS#12 format
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    allowed_protocols: ["TLSv1.2", "TLSv1.3"]
    #  allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
    certificate_verification: true

# ## ACLs
  access_control_rules:
. . .

but for some reason, elasticesarch is still reachable without tls: http://localhost:9200 ; how can I be sure that I’m 100% TLS with authentication mandatory for the API usage

In kbana.yml file I have well set: elasticsearch.hosts: https://myserverhere.my.domain.lu:9200 ; and in the kibana logs we can spot the following:

Error: write EPROTO 140676695230400:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
    at WriteWrap.callbackTrampoline (node:internal/async_hooks:130:17)

and when trying to authenticate:

. . .
[15:15:20:310] [error][plugins][ReadonlyREST][esClient] request to https://myservernamehere04.my.domain.lu:9200/_readonlyrest/metadata/current_user failed, reason: write EPROTO 140695715321792:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
 FetchError: request to https://myservernamehere04.my.domain.lu:9200/_readonlyrest/metadata/current_user failed, reason: write EPROTO 140695715321792:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

    at ClientRequest.<anonymous> (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:527:28)
    at TLSSocket.socketErrorListener (node:_http_client:454:9)
    at TLSSocket.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  type: 'system',
  errno: 'EPROTO',
  code: 'EPROTO'
}
[15:15:20:314] [info][plugins][ReadonlyREST][authController] Could not login in: Wrong credentials
. . .

I’m confused… Any idea what I’ve obviously misconfigured?

sscarduzio · October 5, 2022, 2:21pm

Can you show us a curl -vvv http://localhost:9200 ?

orsius · October 6, 2022, 8:05am

curl -vvv http://localhost:9200
*   Trying 127.0.0.1:9200...
* Connected to localhost (127.0.0.1) port 9200 (#0)
> GET / HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.81.0
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

sscarduzio · October 6, 2022, 8:48am

But this is OK, right?
The server is actively closing the TCP connection because the request is recognised as not encrypted. What did you expect to see?

Maybe you tested with the browser and it automatically tried to upgrade to a SSL connection?

orsius · October 6, 2022, 9:07am

Hello, thanks for your reactivity, I manage to have a proper behaviour; I simply add a trustore as I was using self signed on my lab, and I may have did dome test without the proper parameter http.type: ssl_netty4 set in elasticsearch.yml; But it works now. Next steps, ActiveDriectory authentication and SAML.

# http does not reply anymore (it's what I want)
curl http://localhost:9200
  curl: (52) Empty reply from server

# https without the proper certificate does not work
curl https://localhost:9200
  curl: (60) SSL certificate problem: unable to get local issuer certificate
  More details here: https://curl.se/docs/sslcerts.html
  curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

# https reply (the normal behaviour of ror)
curl -k https://localhost:9200
{
  "name" : "xxx",
  "cluster_name" : "lab-elastic-ror",
  "cluster_uuid" : "xxx",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    . . .
  },
  "tagline" : "You Know, for Search"
}

# advanced APIs usage required authentication (the normal behaviour of ror)

curl -k https://localhost:9200/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "forbidden_response",
        "reason" : "Sorry, your request is forbidden.",
        "due_to" : "OPERATION_NOT_ALLOWED"
      }
    ],
    "type" : "forbidden_response",
    "reason" : "Sorry, your request is forbidden.",
    "due_to" : "OPERATION_NOT_ALLOWED"
  },
  "status" : 403
}

# if I use the proper TLS files, it works fine as well

curl --cert /etc/elasticsearch/ssl/myservername.crt --key ssl/myservername.key --cacert /etc/elasticsearch/ssl/ca/ca.crt https://$(hostname):9200/_cluster/health?pretty -u elastic
Enter host password for user ‘elastic’:
{
“cluster_name” : “lab-elastic-ror”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 3,
“number_of_data_nodes” : 3,
. . .