Kibana 7.14.1 - server.basePath not working

Hi,

I have migrated ES & Kibana to version 7.14.1 and installed ROR 1.34.
When I try to access Kibana, I get the following message :
[forbidden: forbidden_response: [forbidden_response] Reason: forbidden]: forbidden

In Kibana logs, I have this :
{“type”:“response”,"@timestamp":“2021-09-09T11:52:22+02:00”,“tags”:[],“pid”:9326,“method”:“get”,“statusCode”:403,“req”:{“url”:"/",“method”:“get”,“headers”:{“host”:“myhostname”,“x-real-ip”:“xx.xx.xx.xx”,“x-forwarded-host”:“myhostname:80”,“x-forwarded-server”:“myhostname”,“x-forwarded-proto”:“http”,“x-forwarded-for”:“xx.xx.xx.xx, xx.xx.xx.xx”,“connection”:“close”,“x-scheme”:“https”,“user-agent”:“Go-http-client/1.1”,“accept-encoding”:“gzip”},“remoteAddress”:“127.0.0.1”,“userAgent”:“Go-http-client/1.1”},“res”:{“statusCode”:403,“responseTime”:48,“contentLength”:133},“message”:“GET / 403 48ms - 133.0B”}

H.

Hello Hassen!
Can you have a look at the elasticsearch log? Try to find the corresponding “forbidden” log line?

Also would be useful to see the configuration files you are using

Hi Simone,

Here are the logs from Elasticsearch :

[2021-09-13T09:17:12,008][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [slap-indexer]   FORBIDDEN by default req={ ID:812471764-1121914204#10928145, TYP:SearchRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/search, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:.kibana_7.14.1, MET:POST, PTH:/.kibana_7.14.1/_search, CNT:<OMITTED, LENGTH=312.0 B> , HDR:Connection=keep-alive, Host=localhost:9200, content-length=312, content-type=application/json, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.24.1.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, x-opaque-id=55e4a6a0-a618-48a3-9110-8ee2d4bc6ef2, HIS:[logstash with write and create permissions for its own indices-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Kibana Server-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [TGCD users-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Administrator-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], }
[2021-09-13T09:17:17,317][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [slap-indexer] FORBIDDEN by default req={ ID:2082594779--1159173687#10928434, TYP:SearchRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/search, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:.kibana_7.14.1, MET:POST, PTH:/.kibana_7.14.1/_search, CNT:<OMITTED, LENGTH=500.0 B> , HDR:Connection=keep-alive, Host=localhost:9200, content-length=500, content-type=application/json, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.24.1.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, HIS:[logstash with write and create permissions for its own indices-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Kibana Server-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [TGCD users-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Administrator-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], }
[2021-09-13T09:17:21,968][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [slap-indexer]  FORBIDDEN by default req={ ID:470609856-1121914204#10928695, TYP:SearchRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/search, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:.kibana_7.14.1, MET:POST, PTH:/.kibana_7.14.1/_search, CNT:<OMITTED, LENGTH=312.0 B> , HDR:Connection=keep-alive, Host=localhost:9200, content-length=312, content-type=application/json, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.24.1.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, x-opaque-id=d86610a8-6068-4e6d-9863-ce20efdb4b72, HIS:[logstash with write and create permissions for its own indices-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Kibana Server-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [TGCD users-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Administrator-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], }
[2021-09-13T09:17:43,756][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [slap-indexer] FORBIDDEN by default req={ ID:281463817-1121914204#10929210, TYP:SearchRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:indices:data/read/search, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:.kibana_7.14.1, MET:POST, PTH:/.kibana_7.14.1/_search, CNT:<OMITTED, LENGTH=312.0 B> , HDR:Connection=keep-alive, Host=localhost:9200, content-length=312, content-type=application/json, user-agent=elasticsearch-js/7.14.0-canary.7 (linux 3.10.0-1160.24.1.el7.x86_64-x64; Node.js v14.17.5), x-elastic-client-meta=es=7.14.0p,js=14.17.5,t=7.14.0p,hc=14.17.5, x-elastic-product-origin=kibana, x-opaque-id=5c859167-6bda-4838-929a-73c3f4d14e36, HIS:[logstash with write and create permissions for its own indices-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [Kibana Server-> RULES:[auth_key->false] RESOLVED:[indices=.kibana_7.14.1]], [TGCD users-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], [Administrator-> RULES:[ldap_auth->false] RESOLVED:[indices=.kibana_7.14.1]], }

The Readonlyrest yaml file :

parts of your configuration.

readonlyrest:
    force_load_from_file: true
    audit_collector: true
    prompt_for_basic_auth: false

access_control_rules:

- name: "logstash with write and create permissions for its own indices"
  auth_key: logstash:logstash
  actions: ["cluster:monitor/main","cluster:monitor/nodes/stats","cluster:monitor/xpack/info","cluster:admin/xpack/monitoring/bulk","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
  indices: ["logstash-*","error-*","monitoring-*","tomcat-*","nginx-*","squid-*"]

- name: "Kibana Server"
  auth_key: kibana:kibana
  verbosity: error

- name: "TGCD users"
  ldap_auth:
    name: "ldap-ep"
    groups: ["MUST-MBX-TGCD"]
  kibana_access: rw
  indices: [".kibana", ".kibana-devnull","tomcat-*","nginx-*","error-*","squid-*"]
  kibana_hide_apps: ["timelion", "kibana:dev_tools", "kibana:management","ml","uptime","apm","infra:home","infra:logs","maps","canvas","monitoring"]

- name: "Administrator"
  ldap_auth:
    name: "ldap-ep"
    groups: ["MUST-MBX-AMDIN"]
  kibana_access: admin

ldaps:

- name: ldap-ep
  host: "xxxx"
  port: 389
  ssl_enabled: false
  ssl_trust_all_certs: false
  bind_dn: "xxx"
  bind_password: "xxx"
  search_user_base_DN: "xxx"
  user_id_attribute: "sAMAccountName"
  search_groups_base_DN: "xxx"
  unique_member_attribute: "member"
  connection_pool_size: 10
  connection_timeout_in_sec: 10
  request_timeout_in_sec: 10
  cache_ttl_in_sec: 60

Regards
Hassen

Dear Hassen,

Thanks for the logs and conf provided. Please verify that you have patched Kibana, as from version 7.9.0 onwards we require a few Kibana files to be modified in order to operate with ROR.

Secondly, please note that the way we hide apps is now different (but easier!), please have a look at this new documentation.

Another remark is about “.kibana-devnull” which is completely deprecated, and should be removed.

Let me know if this helps.

Hi Simone,

I applied the patch. Kibana is starting but when I put my credentials I get the following error

** kibana logs **

[17:06:27:172] [error][plugins][ReadonlyREST][esClient] ES Authorization error: 403 Error: ES Authorization error: 403
at l.e (/apps/slap-dashboard/plugins/readonlyrestkbn/proxy/core/esClient.js:1:10768)
at l.e (/apps/slap-dashboard/plugins/readonlyrestkbn/proxy/core/esClient.js:1:3548)
at tryCatch (/apps/slap-dashboard/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)
at Generator.invoke [as _invoke] (/apps/slap-dashboard/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)
at Generator.prototype.<computed> [as next] (/apps/slap-dashboard/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)
at a (/apps/slap-dashboard/plugins/readonlyrestkbn/proxy/core/esClient.js:1:2302)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
[17:06:27:173] [info][plugins][ReadonlyREST][authController] Could not login in: Wrong credentials

** Elasticsearch logs **

[2021-09-13T17:03:01,410][INFO ][t.b.r.a.l.AccessControlLoggingDecorator] [slap-indexer] FORBIDDEN by default req={ ID:440429390-1969862486#17403, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=127.0.0.1:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[logstash with write and create permissions for its own indices-> RULES:[auth_key_sha256->false]], [Kibana Server-> RULES:[auth_key->false]], [TGCD users-> RULES:[ldap_auth->false]], [Administrator-> RULES:[ldap_auth->false]], }

I tried to connect with the user kibana. And I get the same error.

However, when I launch a curl command :

curl kibana:[email protected]:9200/_cat/health?pretty
1631545436 15:03:56 EP_Foundry yellow 1 1 248 248 0 0 6 0 - 97.6%

Somethings seems to be wrong between Kibana & ES.

Regards
Hassen

OK, can you share the kibana.yml as well?

Here is the kibana.yml file

server.port: "5601"
server.host: "0.0.0.0"
server.basePath: "/kibana"
server.rewriteBasePath: true
server.name: "dashboard"
elasticsearch.hosts: "http://127.0.0.1:9200"
elasticsearch.preserveHost: True
kibana.index: ".kibana"
kibana.defaultAppId: "home"
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
elasticsearch.requestTimeout: 300000
elasticsearch.shardTimeout: 300000
elasticsearch.startupTimeout: 50000
pid.file: "kibana.pid"
logging.dest: "/apps/dashboard/logs/kibana.log"
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false
telemetry.enabled: false

These are not supported yet in Kibana new platform unfortunately

@hassen server.basePath is currently being worked on. Will revert to this thread ASAP.

@sscarduzio Should I remove the both attributes :

server.basePath & server.rewriteBasePath

@sscarduzio

Everything is working now.

1 Like

Thank you @hassen for letting us know! Support for those two is coming very soon anyway :slight_smile:. Keep an eye on the changelogs in the download page (maybe in a week or 10 days).

I also changed the way to hide the apps. It works very well.
Thanks again for the support and I will check in some days.

1 Like

Yeah! We recently changed the way we handle app hiding, and especially in the most recent versions of Kibana it’s working super well.

Now “undesired” apps are not even rendered by the browser in the first place. So, more apps you hide, the faster Kibana. So yeah, I’m really happy of how this feature turned out as well. :slightly_smiling_face: