Kibana 7.14.1 - ror 1.34.0 - strange error with linux capatibilities on port 80

[aizerin]

I have installed Kibana 7.14.1 with readonlyrestkbn free 1.34.0. Everything works fine until I want to set Kibana running on port 80. I found that setting up capatibilies will break ror plugin:

setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana-plugin
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana-keystore
setcap cap_net_bind_service=+epi /usr/share/kibana/node/bin/node

After that Kibana will normally start and when I try to log in then it will crash without any trace of error in logs. Only this error screen is shown

I don’t even need to set port to 80, I can leave default 5601. If I will remove these capabilities, than it will start works normally.

Also version 1.33.1 of plugin works normally.

Thanks for help

[sscarduzio (Simone Scarduzio)]

Is this a behaviour that is introduced by ROR plugin? I.e. before installing ROR plugin, all was good in Kibana 7.14.1 on port 80?

[aizerin]

It seems so. Kibana works without the plugin and also with older version of plugin 1.33.1 with same Kibana version 7.14.1

[sscarduzio (Simone Scarduzio)]

@aizerin very strange! @Dzuming WDYT?

[Dzuming (Dawid Poliszak)]

Hi @aizerin Based on kibana error stack trace, looks like it’s happened during a fetch request. Could you confirm that there are no network errors in the browser?

Could you also check the browser console if there are any errors here?

[aizerin]

There are errors but I think that there will be may problem with /api/core/capatibilities API. Even when I run this curl command curl -u elk:elk http://192.168.32.22/api/core/anythingcurl -u elk:elk http://192.168.32.22/api/core/capabilities which returns 404 not found server will crash.

I created an isolated Vagrant environment so I can test it without real infrastructure. And I may be found something. There is a warning in logs just before termination of Kibana process.

Sep 15 11:09:59 es kibana[3671]: (node:3683) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Sep 15 11:09:59 es kibana[3671]: (Use `node --trace-deprecation ...` to show where the warning was created)
Sep 15 11:09:59 es kibana[3671]: Node.js process-warning detected:
Sep 15 11:09:59 es kibana[3671]: DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Sep 15 11:09:59 es kibana[3671]:     at showFlaggedDeprecation (buffer.js:194:11)
Sep 15 11:09:59 es kibana[3671]:     at new Buffer (buffer.js:278:3)
Sep 15 11:09:59 es kibana[3671]:     at Function.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/injection/capabilities/CapabilitiesApiFilter.js:1:1091)
Sep 15 11:09:59 es kibana[3671]:     at interceptResponse (/usr/share/kibana/plugins/readonlyrestkbn/proxy/preKibanaProxy/proxyBuilder.js:1:1921)
Sep 15 11:09:59 es kibana[3671]:     at /usr/share/kibana/plugins/readonlyrestkbn/node_modules/express-http-proxy/app/steps/decorateUserRes.js:69:14
Sep 15 11:09:59 es kibana[3671]:     at runMicrotasks (<anonymous>)
Sep 15 11:09:59 es kibana[3671]:     at processTicksAndRejections (internal/process/task_queues.js:95:5)
Sep 15 11:09:59 es kibana[3671]: Terminating process...

Which results in killing the Kibana process by Kibana itself, because of this warning handler kibana/exit_on_warning.js at v7.14.1 · elastic/kibana · GitHub

Which is strange because in Kibana start script there is --no-warning flag kibana/kibana at v7.14.1 · elastic/kibana · GitHub

So my theory is that new ROR uses that deprecated feature of Buffer and linux command setcap somehow do something with NODE_OPTS and --no-warning flag is not applied.

If you want to investigate it more, you can use my vagrantfile, just supply your links for download ror archives.

$elasticInstall = <<-SCRIPT
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.14.1-x86_64.rpm
rpm --install elasticsearch-7.14.1-x86_64.rpm
systemctl daemon-reload
systemctl enable elasticsearch.service
echo "node.name: es
cluster.initial_master_nodes: es
xpack.security.enabled: false
readonlyrest.force_load_from_file: true
" > /etc/elasticsearch/elasticsearch.yml
echo "readonlyrest:
  access_control_rules:
    - name: "Require HTTP Basic Auth"
      type: allow
      auth_key: elk:elk
      kibana_access: unrestricted
" > /etc/elasticsearch/readonlyrest.yml
mkdir /usr/share/elasticsearch/logs
chmod 777 /usr/share/elasticsearch/logs
mkdir /usr/share/elasticsearch/data
chmod 777 /usr/share/elasticsearch/data
/usr/share/elasticsearch/bin/elasticsearch-plugin install <your url> --batch
systemctl start elasticsearch.service
SCRIPT

$kibanaInstall = <<-SCRIPT
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.14.1-x86_64.rpm
rpm --install kibana-7.14.1-x86_64.rpm
systemctl daemon-reload
systemctl enable kibana.service
echo "server.port: 80
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "elk"
elasticsearch.password: "elk"
telemetry.enabled: false
xpack.security.enabled: false # this is fundamental!
xpack.encryptedSavedObjects.encryptionKey: \"McQfTjWnZr4u7xADFJaNdRgUkXp2s5v8yBEHKbPeShVmYq3t6w9zCF\"
" > /etc/kibana/kibana.yml
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana-plugin
setcap cap_net_bind_service=+epi /usr/share/kibana/bin/kibana-keystore
setcap cap_net_bind_service=+epi /usr/share/kibana/node/bin/node
/usr/share/kibana/bin/kibana-plugin install <your url>
/usr/share/kibana/node/bin/node /usr/share/kibana/plugins/readonlyrestkbn/ror-tools.js patch
systemctl start kibana.service
SCRIPT

Vagrant.configure("2") do |config|
    config.vm.define "es" do |node|
        node.vm.box = "generic/centos8"
        node.vm.hostname = "es"
        node.vm.provision "shell", inline: "systemctl stop firewalld.service && systemctl disable firewalld.service"
        node.vm.provision "shell", inline: "setenforce 0", run: "always"
        node.vm.provision "shell", inline: $elasticInstall
        node.vm.provision "shell", inline: $kibanaInstall
    end
end

It is funny that linux capabilities will break kibana capabilities API

[aizerin]

I found nodejs related issues with setcap

setcap on Node.js prevents processing of NODE_OPTIONS

src: allow CAP_NET_BIND_SERVICE in SafeGetenv

So I think that this issue will be solved someday with new version of nodejs. For now, I’ve set up nginx reverse proxy as a workaround.