Problem when upgrading to ES7.11.2/ROR 1.29.0

Users Feedback
1

Hi,

We are a ROR Enterprise subscriber.
We where using ES7.8.1 with ROR1.28.2 and it worked great.

Now, we are getting the below error when upgrading to Elastic 7.11.2 and ReadonlyREST 1.29.2 version and we can’t even access the login page:

{“type”:“error”,"@timestamp":“2021-05-10T13:00:55-04:00”,“tags”:[“connection”,“client”,“error”],“pid”:5828,“level”:“error”,“error”:{“message”:“139718352385920:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”,“name”:“Error”,“stack”:“Error: 139718352385920:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”,“code”:“ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN”},“message”:“139718352385920:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”}
{“type”:“response”,"@timestamp":“2021-05-10T13:00:55-04:00”,“tags”:[],“pid”:5828,“method”:“get”,“statusCode”:401,“req”:{“url”:"/",“method”:“get”,“headers”:{“host”:“zolpr4855.tech-gestion.rqlabgest:5601”,“connection”:“keep-alive”,“cache-control”:“max-age=0”,“upgrade-insecure-requests”:“1”,“user-agent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36”,“accept”:“text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9”,“sec-fetch-site”:“none”,“sec-fetch-mode”:“navigate”,“sec-fetch-user”:"?1",“sec-fetch-dest”:“document”,“accept-encoding”:“gzip, deflate, br”,“accept-language”:“fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7”},“remoteAddress”:“172.31.66.84”,“userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36”},“res”:{“statusCode”:401,“responseTime”:33,“contentLength”:9},“message”:“GET / 401 33ms - 9.0B”}

Please help resolving this issue.
Thank you

2

Dear @abdamou, this might be a known bug in the current release that we already fixed and should be not an issue in the next release 1.30.0.

To temporarily work around the issue, please verify:

  • you have no spaces in the path to your SSL certificate
  • if you are specifying a certificate auhority, the value should be an array. For example: elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
1 Like
3

Thanks @sscarduzio . Now i have two questions:

  • can we have visibility on the release date of 1.30?
  • how can we download the Kibana 1.28.2 plugin for ES7.11.2?
4

We release every 2 or 3 weeks. Generally on the weekends. 1.30.0 is expected this weekend.
Because you are an Enterprise subscriber, you can serve yourself and download non-trial Enterprise builds on our download page.

Every subscriber has a set of authorized email addresses to download non-trial builds on behalf of their company.

If you try to donwload and only get trial builds, please send us an email at support AT readonlyrest.com indicating us your company name, and we will amend the list of email addresses allowed for download.

5

Thanks. how can we download the Kibana 1.28.2 plugin for ES7.11.2?

6

I can hand that over to you. But why are you interested in an older version in particular?

7

@sscarduzio Since we had some issues with Kibana 1.29.0 plugin for ES7.11.2 (see my first post). We want to check if the 1.28.2 could work for us.
Thanks

8

Unfortunately, we did not support 7.11 back then. ROR 1.29.0 introduces support for up to 7.11.2.
Did the suggested workarounds not work in 1.29.0?

9

Hi,
The download of Kibana Plugin 1.30 is not available even though it appairs in the changelog.
Can you verify?
BR,

10

@abdamou please check it now

1 Like
11

Hello, I’m from the same team as abdamou, I just installed ES 1.12.1 with ror-es 1.30.0 and ror-kbn 1.30.0, and we still have the same 401 Unauthorized access,
knowing that we have not changed anything in our current config compared to the config of the version that worked which is ES7.8.1 and ror-es1.20.0 and ror-kbn1.22.1
can you help us please:
{“statusCode”:401,“error”:“Unauthorized”,“message”:“forbidden: Response Error”}

{“type”:“error”,"@timestamp":“2021-05-20T17:01:55-04:00”,“tags”:[“connection”,“client”,“error”],“pid”:11568,“level”:“error”,“error”:{“message”:“140716648286080:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”,“name”:“Error”,“stack”:“Error: 140716648286080:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”,“code”:“ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN”},“message”:“140716648286080:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”}
{“type”:“error”,"@timestamp":“2021-05-20T17:01:55-04:00”,“tags”:[“connection”,“client”,“error”],“pid”:11568,“level”:“error”,“error”:{“message”:“140716648286080:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”,“name”:“Error”,“stack”:“Error: 140716648286080:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”,“code”:“ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN”},“message”:“140716648286080:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n”}
{“type”:“response”,"@timestamp":“2021-05-20T17:01:55-04:00”,“tags”:[],“pid”:11568,“method”:“get”,“statusCode”:401,“req”:{“url”:"/",“method”:“get”,“headers”:{“host”:“zolpr4855.tech-gestion.rqlabgest:5601”,“connection”:“keep-alive”,“cache-control”:“max-age=0”,“upgrade-insecure-requests”:“1”,“user-agent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36”,“accept”:“text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9”,“sec-fetch-site”:“none”,“sec-fetch-mode”:“navigate”,“sec-fetch-user”:"?1",“sec-fetch-dest”:“document”,“accept-encoding”:“gzip, deflate, br”,“accept-language”:“fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7”},“remoteAddress”:“172.31.66.84”,“userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36”},“res”:{“statusCode”:401,“responseTime”:34,“contentLength”:79},“message”:“GET / 401 34ms - 79.0B”}

Our config kibana.yml

elasticsearch.hosts:
- https://elastic:9200
elasticsearch.password: xxxxx
elasticsearch.requestTimeout: 120000
elasticsearch.ssl.verificationMode: none
elasticsearch.username: username
readonlyrest_kbn.cookiePass: cookiePass
readonlyrest_kbn.kibanaIndexTemplate: ".kibana_template"
readonlyrest_kbn.sessions_refresh_after: 1000
readonlyrest_kbn.store_sessions_in_index: true
server.host: serveur1.domaine
server.name: serveur1
server.port: '5601'
server.ssl.certificate: "/etc/kibana/kibana.crt"
server.ssl.enabled: true
server.ssl.key: "/etc/kibana/kibana.key"
server.ssl.supportedProtocols:
- TLSv1.2
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false
12

Hello @belotfi thanks for reaching out. As you might be aware of, moving from 7.8.1 to 7.12.1 means you just started using the rewritten version of ROR (a.k.a. ROR new-platform). This new version requires an extra step, that is patching Kibana.

Can you please confirm if you patched it? You can type in the kibana directory:

$ node/bin/node plugins/readonlyrestkbn/ror-tools.js verify

13

Thank you for your reply.
After using the command sudo ./kibana /plugins/readonlyrestkbn/ror-tools.js patch --allow-root,
I have a runtime error, Unable to revive connection: http: // localhost: 9200 /
While actually elasticSearch is not configured on localhost, but rather on a DNS https: //server.domain: 5601
In my configuraion, I did not indicate this url with localhost: http: // localhost: 9200 /
I never listed http: // localhost: 9200 / in my kibana.yml.
Thank you:
log:
log [20:40:49.518] [info][plugins-service] Plugin “osquery” is disabled.
log [20:40:49.620] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [20:40:49.903] [info][plugins-system] Setting up [101] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,visTypeXy,readonlyrestkbn,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]
log [20:40:49.905] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: afbf0ef0-d7de-4c1c-9493-c9ffa4f62510
log [20:40:50.168] [info][plugins][readonlyrestkbn] Setting up ReadonlyREST plugin - build info: {“versionString”:“enterprise-1.30.0_es7.12.1”,“kibanaVersion”:“7.12.1”,“rorEdition”:“enterprise”,“rorVersion”:“1.30.0”,“isProduction”:true,“isEnterprise”:true,“isPro”:false,“isFree”:false,“isBuildExpired”:false}
log [20:40:50.195] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [20:40:50.196] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
log [20:40:50.262] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [20:40:50.271] [warning][config][plugins][reporting] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.9 OS. Automatically setting ‘xpack.reporting.capture.browser.chromium.disableSandbox: true’.
log [20:40:50.272] [warning][encryptedSavedObjects][plugins] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [20:40:50.299] [warning][fleet][plugins] Fleet APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [20:40:50.382] [warning][actions][actions][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [20:40:50.396] [warning][alerting][alerts][plugins][plugins] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
log [20:40:50.561] [info][monitoring][monitoring][plugins] config sourced from: production cluster
log [20:40:50.872] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations…
log [20:40:50.891] [error][elasticsearch] Request error, retrying
GET http://localhost:9200/_xpack?accept_enterprise=true => connect ECONNREFUSED 127.0.0.1:9200
log [20:40:50.894] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
log [20:40:50.894] [warning][elasticsearch] No living connections
log [20:40:50.895] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Error: No Living connections error
log [20:40:50.897] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: No Living connections
log [20:40:50.936] [error][savedobjects-service] Unable to retrieve version information from Elasticsearch nodes.

14

Hi @abdamou can you please retry with the latest ROR? I cannot reproduce this behaviour in the current version. Plus, we fixed a lot of things after 1.30.0.

I will send you a link to the latest ROR Enterprise build in a private message in this forum.

15

Good afternoon, I wanted to piggyback off this topic as I am having a similar issue with my upgrade.
I am moving from ES 7.8.1 to 7.11.2 and ROR 1.29. 2 nodes - I have followed the steps in the documentation, including the extra post-install step. Nothing in the config YML files has been altered. They stayed the same as the 7.8.1 files which has been working with zero issues

When I attempt to open Kibana to login via HTTP://drtest:5601/login, I get this error

[11:53:09:330] [error][plugins][ReadonlyREST][cookieManager] Error: Bad hmac value

I don’t get this error when I log into Kibana via HTTP://localtest:5601/login on the other node.

I have Kibana running on both Nodes with ES and cerebro.

Also, I am having issues with authentication from the localtest server. I can login to Kibana but I can use any random password. It like ROR isn’t working at all. Any idea’s on what I may have missed or configured incorrectly and what yml file would I find that in.

I apologize for the two-part questions, I am green when it comes to ES and this is my first time doing an upgrade on ES,Kibana,ROR etc… and I didn’t want to start a new thread and clog the board.

Thank you for any help or ideas.

16

Hi Jason, welcome to the forum.

Please make sure you align to the latest available plugin versions 1.31.0 in both ES and Kibana, and let’s take it from there.

Are you using ROR Enterprise? Or which edition is this?

17

Hi Simone,

Thank you for the reply!

We are using ROR free and ROR Kibana pro.

I will look into getting the most current versions installed.

1 Like
18

Hello Simone,
Once the patch is applied,
I don’t know why I have an error on port 5601 and 560120, i use readonlyrest_kbn_enterprise-1.30.2-pre2_es7.12.1 :

(Serveur1) /usr/share/kibana $sudo node/bin/node plugins/readonlyrestkbn/ror-tools.js patch
[sudo] password for rbel123:
[ROR COMPAT] Received command: patch
[ROR COMPAT] Adding hooks on a few Kibana files if necessary…
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches/http_server.js.patch
Applied ‘http_server.js.patch’ to ‘/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/…/…/…/…/src/core/server/http/http_server.js’
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches/kbn_server.js.patch
Applied ‘kbn_server.js.patch’ to ‘/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/…/…/…/…/src/legacy/server/kbn_server.js’
[ROR COMPAT] found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches/read_config.js.patch
Applied ‘read_config.js.patch’ to ‘/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/…/…/…/…/node_modules/@kbn/config/target/raw/read_config.js’

(Serveur1) /usr/share/kibana/bin $sudo ./kibana --allow-root
log [09:15:41.358] [warning][environment] Detected an unhandled Promise rejection.
Error: invalid port: ‘5601’
log [09:15:41.365] [warning][environment] Detected an unhandled Promise rejection.
Error: invalid port: ‘560120’
log [09:15:47.177] [info][plugins-service] Plugin “ml” is disabled.
log [09:15:47.177] [info][plugins-service] Plugin “watcher” is disabled.
log [09:15:47.179] [info][plugins-service] Plugin “graph” is disabled.
log [09:15:47.180] [info][plugins-service] Plugin “osquery” is disabled.
log [09:15:47.180] [info][plugins-service] Plugin “security” is disabled.
log [09:15:47.310] [warning][config][deprecation] “xpack.monitoring” is deprecated and has been replaced by “monitoring”
log [09:15:47.311] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
log [09:15:47.311] [warning][config][deprecation] Disabling the security plugin (xpack.security.enabled) will not be supported in the next major version (8.0). To turn off security features, disable them in Elasticsearch instead.
log [09:15:47.356] [fatal][root] RangeError [ERR_SOCKET_BAD_PORT] [ERR_SOCKET_BAD_PORT]: options.port should be >= 0 and < 65536. Received 560110.
at validatePort (internal/validators.js:211:11)
at Server.listen (net.js:1444:5)
at /usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:338:31
at new Promise ()
at module.exports.internals.Core._listen (/usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:311:16)
at module.exports.internals.Core._start (/usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:285:24)
at async HttpService.runNotReadyServer (/usr/share/kibana/src/core/server/http/http_service.js:180:5)
at async HttpService.setup (/usr/share/kibana/src/core/server/http/http_service.js:80:7)
at async Server.setup (/usr/share/kibana/src/core/server/server.js:192:23)
at async Root.setup (/usr/share/kibana/src/core/server/root/index.js:47:14)
at async bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:5)
at async Command. (/usr/share/kibana/src/cli/serve/serve.js:169:5) {
code: ‘ERR_SOCKET_BAD_PORT’

thank you

19

wow that’s bizarre. Will check immediately.

20

I think you configured the server.port with double quotes, and it took it as a string instead of a number. I will add the validation in our code to avoid this comic error.