Elaticsearch 8.4.1 - RoR plugin - elasticsearch does not start - Netty4Utils not found

Dear All, I encounter an issue install the RoR plugin and not abale to start elasticsearch with the plugin install.

In brief:

  • the error message is: Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/transport/netty4/Netty4Utils
  • I use elaticsearch 8.4.1 (basic licence) with security enable (user:passwd + tls http and transport)
  • My goal is to use RoR for the LDAP feature as well as the kibana features.

Any ideas ?

Thanks in advance for your time and help.

kr,

nb: here is a bit more details:

# java -version
  openjdk version "11.0.16" 2022-07-19
  OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04)
  OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

# os used
  Ubuntu
  VERSION="20.04.5 LTS (Focal Fossa)"
# as elasticsearch user
export ES_HOME=/usr/share/elasticsearch && export ES_PATH_CONF=/etc/elasticsearch/monitoring
/usr/share/elasticsearch$ ./bin/elasticsearch-plugin list
/usr/share/elasticsearch$ ./bin/elasticsearch-plugin install file:///home/ubuntuadmin/downloads/readonlyrest-1.43.0_es8.4.1.zip
-> Installing file:///home/ubuntuadmin/downloads/readonlyrest-1.43.0_es8.4.1.zip
-> Downloading file:///home/ubuntuadmin/downloads/readonlyrest-1.43.0_es8.4.1.zip
[=================================================] 100%
. . .
-> Installed readonlyrest
-> Please restart Elasticsearch to activate any plugins installed
/usr/share/elasticsearch$ ./bin/elasticsearch-plugin list
readonlyrest

# sudoer
systemctl restart elasticsearch
[2022-09-19T15:09:25,657][ERROR][o.e.b.Elasticsearch      ] [lucrpdvop01-monitoring] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: failed to load plugin class [tech.beshu.ror.es.ReadonlyRestPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:552) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:463) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:274) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:150) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$14(PluginsService.java:579) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:399) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.1.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:543) ~[elasticsearch-8.4.1.jar:?]
        ... 9 more
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/transport/netty4/Netty4Utils
        at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$new$1(ReadonlyRestPlugin.scala:100) ~[?:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:26) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:25) ~[?:?]
        at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.scala:100) ~[?:?]
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:543) ~[elasticsearch-8.4.1.jar:?]
        ... 9 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.transport.netty4.Netty4Utils
        at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:588) ~[?:?]
        at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?]
        at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$new$1(ReadonlyRestPlugin.scala:100) ~[?:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:26) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:25) ~[?:?]
        at tech.beshu.ror.es.ReadonlyRestPlugin.<init>(ReadonlyRestPlugin.scala:100) ~[?:?]
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:543) ~[elasticsearch-8.4.1.jar:?]
        ... 9 more

Hi Gautier! Nice to meet you, welcome to the forum :slight_smile:

Did you patch Elasticsearch?

2 Likes

Damned! My bad, I didn’t performed the prerequisites! Thanks for your quick answer. :blush:

That being said, I read that we can’t use RoR and elasticsearch security togheter ?

ReadonlyREST and X-Pack security module can’t run together, so the latter needs to be disabled.

So if I’m not wrong I can NOT use only the “ldap / active directory” feature of RoR ?

Just to illustrate what I’m thinking: Within my deployment chain, I have to install an unscured elasticserach cluster than install RoR … and for patching, I need to “unsecured the cluster” by removing the plugin install, unpatched, update elastic linux packages, install new ror plugin version, patch back then start a secure instance ? Hummm … :confused:

The steps for each ES node would be:

  1. Install Elasticsearch from packages
  2. Append xpack.security.enabled: false to elasticsearch.yml
  3. cd to $ES_HOME
  4. install ROR plugin in Elasticsearch
  5. patch Elasticsearch with ror-tool
  6. Create readonlyrest.yml
  7. run Elasticsearch as a linux service

Do you require SSL between ES nodes?

@sscarduzio, Thanks a lot for your time and quick respond to my topic.

To answer your question: Yes my goal is to have an end to end secured elasticsearch cluster (https for transport communication and well as for http communication)

Following your replies, I set a simple readonlyrest.yml file:

readonlyrest:
    #optional
    response_if_req_forbidden: Sorry, your request is forbidden.
    access_control_rules:
    - name: Accept all requests from localhost
      hosts: [127.0.0.1]

This first step allow me to restrein access for anyother server:

## OK localhost
 curl localhost:9200/_cluster/health?filter_path=status
{"status":"yellow"}

## NOK with server IP addr
curl ${MY_SRV_IP}:9200/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "forbidden_response",
        "reason" : "Sorry, your request is forbidden.",
        "due_to" : "OPERATION_NOT_ALLOWED"
      }
    ],
    "type" : "forbidden_response",
    "reason" : "Sorry, your request is forbidden.",
    "due_to" : "OPERATION_NOT_ALLOWED"
  },
  "status" : 403
}

My next step is to set a more secure readonly rest file;

Is there a way to “hide/encrypt” the auth_key value ? (see example below with elastic user)

What do you recommend me as best practices ? Create a simple readonlyrest.yaml file at first and then write advance rules within the readonlyrest elasticsearch index (if it’s possible) ?

readonlyrest:
  #optional
  response_if_req_forbidden: Sorry, your request is forbidden.
  access_control_rules:
  # - name: "Block 1 - Allowing anything"
  #   type: allow
  # - name: Accept all requests from localhost
  #   hosts: [127.0.0.1]
  # - name: Just certain indices, and read only
  #   actions: ['indices:data/read/*']
  #   indices: ['all_my_public_indices_start_with*'] # index aliases are taken in account!
  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key: elastic:changeit

Yes of course, you can either hash the basic auth passwords using sha512, and other algorithms.
i.e.

  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key_sha512: "admin:280ac6f...94bf9"

Or you can inject environment variables anywhere in the file using @{env:MY_VALUE}. I.e.

  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key: elastic:${MY_SECRET_PASSWORD}

Actually you can inject the whole content of the file with a single var replacement

$ cat readonlyrest.yml
${ROR_CONF} 

Full SSL Encryption

We are covered about that: inter-node transport encryption and HTTPS are fully supported in Elasticsearch (using jks and pem certificates formats)

Our kibana plugin supports HTTPS on the client and server side as well, so you should be able to achieve full encryption from the data store to the browser (cookies included).

Hello, Following your input I go a bit further and I encountered another issue: Kibana does not start :confused:

I received “Error: header too long” ; (see complete error messag below); any idea ?

Here is my current configuration:

  • readonlyrest.yml
readonlyrest:
  #optional
  response_if_req_forbidden: Sorry, your request is forbidden.
  ssl:
    keystore_file: "keystore_new.jks" # or keystore.p12 for PKCS#12 format
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    allowed_protocols: ["TLSv1.2", "TLSv1.3"]
    #  allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
  ssl_internode:
    keystore_file: "keystore_new.jks" # or keystore.p12 for PKCS#12 format
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    allowed_protocols: ["TLSv1.2", "TLSv1.3"]
    #  allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
  access_control_rules:
  - name: "Require HTTP Basic Auth"
    type: allow
    auth_key_sha512: "kibana:1e64d7d1.......830c49"
  • kibana.yaml
server.host: "myservernamehere01"
server.name: "myservernamehere01"
server.ssl.enabled: true
server.ssl.keystore.path: "/etc/kibana/keystore.jks"
server.ssl.keystore.password: "readonlyrest"
server.ssl.supportedProtocols: ["TLSv1.2", "TLSv1.3"]
xpack.security.enabled: false
elasticsearch.hosts: ["https://myservernamehere01:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "mypasswordhere"
logging:
  appenders:
    file:
      type: file
      fileName: /data/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

Launching kibana /usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml I get the following ERR msg:

[ROR] - serve.js - intercepting config
[12:45:40:376] [warning][plugins][ReadonlyREST][unsupportedFeaturesNotifier] logging.appenders config parameter declared in kibana.yml is not supported yet. It will be ignored
[12:45:40:376] [warning][plugins][ReadonlyREST][unsupportedFeaturesNotifier] logging.root config parameter declared in kibana.yml is not supported yet. It will be ignored
[12:45:40:404] [info][plugins][ReadonlyREST][kibanaYamlSessionManagerConfigParser] "readonlyrest_kbn.store_sessions_in_index" is not enabled.
If you use more than one Kibana nodes (High availability configurations), you MUST enable both "readonlyrest_kbn.store_sessions_in_index" and share the same "readonlyrest.cookiePass" in ALL Kibana nodes.
See documentation: https://docs.readonlyrest.com/kibana#session-management-with-multiple-kibana-instances
[12:45:40:407] [info][plugins][ReadonlyREST][serverCreator] ROR PKP SSL server will accept protocols: TLSv1.2,TLSv1.3 (bitmask: 67108864)
[12:45:40:419] [info][plugins][ReadonlyREST][preElasticsearchProxy] Pre-Elasticsearch-proxy will listen on 127.0.0.1:44625
[2022-09-27T12:45:40.428+02:00][WARN ][environment] pid file already exists at /run/kibana/kibana.pid
[12:45:40:455] [error][plugins][ReadonlyREST][serverCreator] Error creating https server:  Error: header too long
    at configSecureContext (node:internal/tls/secure-context:278:15)
    at Object.createSecureContext (node:_tls_common:117:3)
    at Server.setSecureContext (node:_tls_wrap:1346:27)
    at Server (node:_tls_wrap:1205:8)
    at new Server (node:https:69:3)
    at Object.createServer (node:https:105:10)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:3609)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:1406)
    at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12211)
    at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)
    at Generator.prototype.<computed> [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:32:7
    at new Promise (<anonymous>)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:21:12
    at /usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12325
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error: header too long
    at configSecureContext (node:internal/tls/secure-context:278:15)
    at Object.createSecureContext (node:_tls_common:117:3)
    at Server.setSecureContext (node:_tls_wrap:1346:27)
    at Server (node:_tls_wrap:1205:8)
    at new Server (node:https:69:3)
    at Object.createServer (node:https:105:10)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:3609)
    at r.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/serverCreator.js:1:1406)
    at e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12211)
    at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)
    at Generator.prototype.<computed> [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:32:7
    at new Promise (<anonymous>)
    at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:21:12
    at /usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/preKibanaProxy.js:1:12325
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Kibana can’t understand JKS unfortunately. You will need the *.p12 format.

Geat, it’s works! Thank you.

The plugin start properly BUT as I use “self signed” certificate, I received several error messages (kibana is up and running) in the logs; Is there any parameter I can use to ignore these log lines ?

[2022-09-28T08:35:14.561+02:00][INFO ][plugins.readonlyrestkbn] Setting up ReadonlyREST plugin - build info: {"versionString":"enterprise-1.43.0_es8.4.1","kibanaVersion":"8.4.1","rorEdition":"enterprise","rorVersion":"1.43.0","isProduction":true,"isEnterprise":true,"isPro":false,"isFree":false,"isBuildExpired":false}
[2022-09-28T08:35:14.562+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Raspbian 15.3 OS. Automatically enabling Chromium sandbox.
[2022-09-28T08:35:14.590+02:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-09-28T08:35:14.630+02:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
Error: self signed certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)
[2022-09-28T08:35:15.172+02:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Internal Server Error</pre>
</body>
</html>

Error: self signed certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
    at TLSSocket.emit (node:events:527:28)
    at TLSSocket._finishInit (node:_tls_wrap:946:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:727:12)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17)
[2022-09-28T08:35:15.616+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Error: self signed certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1532:34)
. . .

OK you need to tell Kibana’s Elasticsearch client to trust that cert. There’s some configuration for that in Kibana.

Have a look at this page where it talks about elasticsearch.ssl.certificateAuthorities.