Elasticsearch 8.4.1 - RoR Auth - Admin User Privileges for _cluster API

orsius · October 12, 2022, 6:58am

Dear all, I spotted an unwanted behavior today, I’ve created a simple local admin user in my cluster (see the configuration below) BUT for some reason I received an error 403 when I use some features of the cluster API. Any idea of what I have miss configured ?

Thanks in advance for your time and help.

kr,

To reproduce the error, I have the following basic auth in my readonlyrest file:

readonlyrest:
  response_if_req_forbidden: Sorry, your request is forbidden.
. . .
  access_control_rules:
. . .
  - name: "Admin local"
    type: allow
    auth_key_sha512: "elastic:xxxxxxxx"
    kibana_access: admin
. . .

A cluster health is fine BUT a cluster allocation expain is denied

# GET /_cluster/health 200 OK
{
  "cluster_name": "xxxx",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  . . . .
}
# GET /_cluster/allocation/explain 403 Forbidden
{
  "error": {
    "root_cause": [
      {
        "type": "forbidden_response",
        "reason": "Sorry, your request is forbidden.",
        "due_to": "OPERATION_NOT_ALLOWED"
      }
    ],
    "type": "forbidden_response",
    "reason": "Sorry, your request is forbidden.",
    "due_to": "OPERATION_NOT_ALLOWED"
  },
  "status": 403
}

In other words: “How can I configure a FULL admin user with all the cluster privileges ?”

sscarduzio · October 12, 2022, 7:19am

Hi @orsius!

To adapt to the increased amount of new cluster and index management UI present in the newer versions of Kibana, in ROR 1.44.0 release we expanded the permissions set for kibana_access: amin user so they can extensively interact with the kibana management panel without getting 403 permission denied issues from ROR ES plugin.

NB: permissions are still more restrictive in devtools (the whole idea is to protect the data/cluster from accidental modifications).

So, first of all, I recommend you to upgrade to the latest ROR Enterprise (now Universal builds).

If you would like a truly unrestricted super-admin access level, use kibana_access: unrestricted, or - equivalently - comment the kibana_access rule entirely.

orsius · October 13, 2022, 7:46am

Hello @sscarduzio, hope your are well; Thank you for your reply, I’m testing the new kibana plugins … but unfortunately I encounter the following error message:

[09:31:08:736] [error][plugins][ReadonlyREST][legacyRenderer] Failed to render login page: Error: LicenseService is not initialised yet: empty clusterUuid

I obviously have to do and additional action like: enter an activation key, copy a license file on the server, … Could you point me to proper documentation page ? Thanks

kr,

orsius · October 13, 2022, 7:56am

A rollback to the previous plugin version seems not possible

install older plugin OK

bin/kibana-plugin install file:///home/<myuserhere>/Downloads/readonlyrest_kbn_enterprise-1.43.0_es8.4.1.zip
Attempting to transfer from file:///home/<myuserhere>/Downloads/readonlyrest_kbn_enterprise-1.43.0_es8.4.1.zip
Transferring 46470882 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

patch older plugin NOK

/usr/share/kibana # node/bin/node plugins/readonlyrestkbn/ror-tools.js patch
[ROR COMPAT] Received command: patch
[ROR COMPAT] Modify a few Kibana files for ReadonlyREST...
[ROR COMPAT] Found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
[ROR COMPAT] ReadonlyREST encountered problems with compatibility manager script, please run manually using 'ror-tools.js'   Error: Failed to apply patch 'http_server.js.patch' to '/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/../../../.././node_modules/@kbn/core-http-server-internal/target_node/http_server.js'
    at c.e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/filePatcher.js:1:3684)
    at /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2856
    at doWithPatcher (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:3900)
    at patchAll (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2680)
    at Object.main (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2222)
    at Object.<anonymous> (/usr/share/kibana/plugins/readonlyrestkbn/ror-tools.js:16:10)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)

orsius · October 13, 2022, 9:23am

Ok my bad, I miss the proper documentation page:

Universal Builds - ReadonlyREST

I still encountered issue with the url: ReadonlyREST Customer Affiliation Form when I click send I received the following error msg:

Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Please advise.

sscarduzio · October 13, 2022, 10:12am

Thanks for reaching out @orsius, will take care of the affiliation form ASAP. In the meantime, your contract affiliation should be now visible, enabling you to access the activation key.

About the patching, things should work if you unpatch before patch. There should be instruction about upgrading in the ROR ES docs

orsius · October 13, 2022, 12:36pm

Hello, thanks a lot for your reactivity. The portal is now accessible.

On the other hand, I still have a particular issue with one of my lab machine:

I was trying to remove and reinstall an older version of kibana earlier today and encounter a fail state.
I try to unpatch remove and reinstall the new plugin (kibana universal)
Now I’m stuck with the following ERR:

## try to unpatch
myserver01:/usr/share/kibana # node/bin/node plugins/readonlyrestkbn/ror-tools.js unpatch
[ROR COMPAT] Received command: unpatch
[ROR COMPAT] Found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
[ROR COMPAT] Backup file found!
Restored /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/../../../.././node_modules/@kbn/core-http-server-internal/target_node/http_server.js
[ROR COMPAT] Found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/serve.js.patch
[ROR COMPAT] Backup file not found: assuming patch was not applied.

## try to patch
myserver01:/usr/share/kibana # node/bin/node plugins/readonlyrestkbn/ror-tools.js patch
[ROR COMPAT] Received command: patch
[ROR COMPAT] Modify a few Kibana files for ReadonlyREST...
[ROR COMPAT] Found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
[ROR COMPAT] Backup file found, assuming already patched. Will first unpatch and then re-patch with potentially newer code.
Restored /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/../../../.././node_modules/@kbn/core-http-server-internal/target_node/http_server.js
[ROR COMPAT] ReadonlyREST encountered problems with compatibility manager script, please run manually using 'ror-tools.js'   Error: Failed to apply patch 'http_server.js.patch' to '/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/../../../.././node_modules/@kbn/core-http-server-internal/target_node/http_server.js'
    at c.e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/filePatcher.js:1:3684)
    at /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2856
    at doWithPatcher (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:3900)
    at patchAll (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2680)
    at Object.main (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2222)
    at Object.<anonymous> (/usr/share/kibana/plugins/readonlyrestkbn/ror-tools.js:16:10)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)

## try the verify command
myserver01:/usr/share/kibana # node/bin/node plugins/readonlyrestkbn/ror-tools.js verify
[ROR COMPAT] Received command: verify
[ROR COMPAT] Verifying the presence of ROR hooks on Kibana files..
[ROR COMPAT] Found patch file /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/patches_for_kbn_distribution/http_server.js.patch
Verifying patched state...
[ROR COMPAT] ReadonlyREST encountered problems with compatibility manager script, please run manually using 'ror-tools.js'   Error: Cannot verify patched status, as the patch is not applicable to the original backed-up file
    at c.e (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/filePatcher.js:1:3003)
    at /usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:3002
    at doWithPatcher (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:3900)
    at isPatchedAll (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2970)
    at Object.main (/usr/share/kibana/plugins/readonlyrestkbn/kibana/patchers/kibanaPatcher.js:1:2142)
    at Object.<anonymous> (/usr/share/kibana/plugins/readonlyrestkbn/ror-tools.js:16:10)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)

I try to remove and force a reinstall of kibana BUT didn’t change a thing

zypper ref
zypper in -f kibana-8.4.1-1.x86_64

of course I can restore my VM but the goal here is to understand the behavior and be able to fix it if it happens in a production environment.

sscarduzio · October 13, 2022, 6:40pm

I think what happened is that the package manager keeps track of the exact files created by the Kibana package, and replaces only them with new Kibana package files, leaving old ROR patch files behind.

The stateful Kibana upgrade process should be:

install Kibana version X
Install ror plugin for version X
Run patch

======= Kibana Version X+1 is out, let’s upgrade!

unpatch Kibana version X
uninstall ror plugin version X
package manager upgrade to Kibana X+1
Install ror plugin for version X
Run patch