Errors patching Kibana RoR 1.60.0 plugin

1

Hi,

I encounter problems patching RoR version 1.60.0 for Kibana:

[ROR COMPAT] ReadonlyREST encountered problems during patching: Error: Failed to apply patch 'get_document_payload.patch' to '/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/../../../../node_modules/@kbn/reporting-plugin/server/routes/common/jobs/get_document_payload.js'
    at c.e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/filePatcher.js:1:3684)
    at /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2938
    at doWithPatcher (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:4254)
    at patchAll (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2762)
    at Object.main (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2279)
    at Object.<anonymous> (/usr/share/kibana/plugins/readonlyrestkbn/ror-tools.js:16:10)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)

Version of Kibana used: 8.15.1
It works perfectly with 1.59.0.

Can you have a look if something might be broken?

Thanks & regards,
Arjen Buising

2

Now it gets patched, but Kibana gets killed after:

Sep 19 14:09:46 kibana kibana[36959]: [2024-09-19T14:09:46.251+02:00][DEBUG][plugins-discovery] Successfully discovered plugin "readonlyrestkbn" at "/usr/share/kibana/plugins/readonlyrestkbn"
Sep 19 14:09:52 kibana systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE

Again, still works with RoR 1.59.0

Regards, Arjen

3

With additional memory debug log options in node:

Sep 19 14:21:43 kibana kibana[37766]: [2024-09-19T14:21:43.033+02:00][DEBUG][server] starting server
Sep 19 14:21:43 kibana kibana[37766]: [2024-09-19T14:21:43.057+02:00][DEBUG][plugins.taskManager] status core.status.derivedStatus now set to unavailable
Sep 19 14:21:43 kibana kibana[37766]: [2024-09-19T14:21:43.059+02:00][DEBUG][status] Recalculated core overall status
Sep 19 14:21:43 kibana systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE

Regards, Arjen

4

Where this is the output with a continuing to run Kibana (with RoR 1.59.0):

Sep 19 14:27:59 kibana kibana[38503]: [2024-09-19T14:27:59.630+02:00][DEBUG][status] Recalculated core overall status
Sep 19 14:27:59 kibana kibana[38503]: [2024-09-19T14:27:59.647+02:00][DEBUG][metrics.ops] memory: 472.2MB uptime: 0:00:18 load: [0.40,0.24,0.21]  delay histogram: { 50: 0.001; 95: 0.001; 99: 0.001 } utilization: 1.00000
Sep 19 14:28:00 kibana kibana[38503]: [2024-09-19T14:28:00.828+02:00][DEBUG][elasticsearch.query.data] 200 - 241.0B
Sep 19 14:28:00 kibana kibana[38503]: GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip
Sep 19 14:28:00 kibana kibana[38503]: [2024-09-19T14:28:00.830+02:00][INFO ][elasticsearch-service] Successfully connected to Elasticsearch after waiting for 1218 milliseconds
Sep 19 14:28:00 kibana kibana[38503]: [2024-09-19T14:28:00.856+02:00][DEBUG][status] Recalculated core overall status
Sep 19 14:28:00 kibana kibana[38503]: [2024-09-19T14:28:00.880+02:00][DEBUG][elasticsearch.query.data] 200 - 33.8KB
Sep 19 14:28:00 kibana kibana[38503]: GET /_cluster/settings?include_defaults=true&flat_settings=true
Sep 19 14:28:00 kibana kibana[38503]: [2024-09-19T14:28:00.881+02:00][DEBUG][savedobjects-service] Starting SavedObjects service
Sep 19 14:28:00 kibana kibana[38503]: [2024-09-19T14:28:00.931+02:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...

Regards, Arjen

5

Hello @abuising,

I was unable to replicate this issue. The initial patching problem raised suspicions. From what I recall, one probable cause could be the failure to unpatch Kibana before applying the patch. Can you verify whether these upgrading steps were followed? https://docs.readonlyrest.com/kibana#upgrading?

6

Hi @Dzuming,

I even tried it with an uninstalled Kibana, so the directory containing the plugins is completely gone before trying it.
The failure to patch might have been a special case where we came from Kibana 8.13.2 with RoR plugin 1.57.3 via Kibana 8.15.1 with RoR plugin 1.59.0 to RoR plugin 1.60.0.
After a clean install of Kibana the patching at least worked, but after that, Kibana doesn’t start anymore.
I will try with a real clean machine or something like that if it then still doesn’t work.

Thanks & regards,
Arjen

7

Hi @Dzuming,

No, after provisioning a clean node with Kibana 8.15.1 and RoR 1.60.0, it still fails with above error.

Regards, Arjen

8

Hi @abuising

Could you provide full Kibana and es logs and kibana.yml, readonlyrest.yml, and elasticsearch.yml configs? You can send it as a private message or via email.

9

I encountered a similar issue with the ror 1.60.0 BUT still using 8.5.2:

If it may helps you: as a workaround, I did:

rm -rf /usr/share/kibana/*
apt install --reinstall kibana=8.5.2
chown -R kibana:kibana /usr/share/kibana
./bin/kibana-plugin install file:///tmp/plugins/readonlyrest_kbn_universal-1.60.0_es8.5.2.zip
node/bin/node plugins/readonlyrestkbn/ror-tools.js verify
node/bin/node plugins/readonlyrestkbn/ror-tools.js patch
node/bin/node plugins/readonlyrestkbn/ror-tools.js verify
chown -R kibana:kibana /usr/share/kibana
systemctl start kibana
journalctl -xeu kibana --follow

Kibana is now properly running :slight_smile:

Do you encounter the same behavior ?

10

Hi @orsius,

I will try your solution, but it may have already been part of what I already tried…

Thanks & regards,
Arjen

11

Hi @orsius,

I tried your solution, but I still get:

Sep 24 11:01:09 kibana2 kibana[2984]: [2024-09-24T11:01:09.730+02:00][DEBUG][plugins-discovery] Successfully discovered plugin "readonlyrestkbn" at "/usr/share/kibana/plugins/readonlyrestkbn"
Sep 24 11:01:30 kibana2 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit kibana.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Sep 24 11:01:30 kibana2 systemd[1]: kibana.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit kibana.service has entered the 'failed' state with result 'exit-code'.

Thanks & regards,
Arjen

12

Did you properly set the permission and ownership to the directory: /usr/share/kibana something when we do things manually we set the ownership to root and not to the kibana user and therefore the service does not start properly.

What you can do to have more details on your issue is to start kibana manually in your terminal, like this:

# ## start a shell as kibana user
su -s /bin/bash kibana
# ## lauch kibana in debug mod and see what it says:
/usr/share/kibana/bin/kibana -c /etc/kibana.yml

Let us know if you get something relevant here.

13

ROR 1.61.0 with the fix is released

1 Like