Forbidden after start autorefresh in canvas

Hi everybody, i’m with a problem after start autorefresh in my workpad on canvas.

My ROR config:

readonlyrest:

    audit_collector: true

    audit_index_template: "'auditoria'-yyy-MM"

    access_control_rules:


    - name: "::KIBANA-SRV::"

      auth_key: kibana:4eBBYeN37C9sgFD

    - name: "::ADMIN::"

      auth_key: elastic:teste

      kibana_access: unrestricted

      verbosity: error




My ES config:

cluster.name: “docker-cluster”

network.host: 0.0.0.0

xpack.license.self_generated.type: basic

xpack.security.enabled: false

xpack.monitoring.collection.enabled: true

readonlyrest:

force_load_from_file: true

My kibana config:

server.name: kibana

server.host: 0.0.0.0

elasticsearch.hosts: [ “http://elasticsearch:9200” ]

monitoring.ui.container.elasticsearch.enabled: true

elasticsearch.username: kibana

elasticsearch.password: 4eBBYeN37C9sgFD

server.publicBaseUrl: “http://sedehmlsa07:5601

xpack.monitoring.enabled: true

xpack.security.enabled: false

xpack.watcher.enabled: false

xpack.telemetry.enabled: false

xpack.encryptedSavedObjects.encryptionKey: “4FgSs8JXxqUQQywIxSkTjgHQYeWBUSiW”

kibana.autocompleteTerminateAfter: 1000000


Hello @rebertty, the error points to some 401 Unuthorized responses from Elasticsearch. Coul you please have a look at elasticsearch.log? You should see some log lines with “FORBIDDEN” string in them. Please share them here so we can reconstruct a way to allow those request through in your ACL :+1:

Hi, i’ll send my logs:

elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:21:34,394Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1410496141-1529426443#3161917, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:22:04,397Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1608643633-50166409#3163074, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:22:23,087Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1571602092-2015646397#3163730, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:22:31,871Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1996530766-1728309856#3163982, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:22:34,401Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:2094434365-394421213#3164042, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:22:39,429Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1116741251-226747544#3164227, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:22:53,114Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1778568975-1306264829#3164549, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:23:03,676Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:2005900520-776179147#3165141, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:23:04,404Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:503435632-868824448#3165157, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:23:34,408Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:79689881-1422589300#3166118, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:23:48,156Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1218597125-1312867719#3166589, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:23:57,518Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:733530069-77177151#3167049, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:24:04,412Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1726862675-1972000379#3167272, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:24:34,416Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1496038545-242955340#3168220, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:25:04,420Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:957643791-1078736626#3169321, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:25:34,423Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1719696278-1200022381#3170273, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:25:58,987Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:560041375-2145265539#3171011, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:26:04,431Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:140620292-862692135#3171454, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:26:05,293Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1768447839-691677021#3171463, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:26:34,435Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:651842707-162546661#3172427, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:27:04,438Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:562804315-1541145977#3173336, TYP:MainRequest, CGR:N/A, USR:elastic (attempted), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:172.29.0.6/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Accept=application/json, Authorization=<OMITTED>, Host=elasticsearch:9200, User-Agent=Go-http-client/1.1, content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
elasticsearch_1  | {"type": "server", "timestamp": "2021-12-03T10:27:23,094Z", "level": "INFO", "component": "t.b.r.a.l.AccessControlLoggingDecorator", "cluster.name": "docker-cluster", "node.name": "404a4d9afc8c", "message": "\u001B[35mFORBIDDEN by default req={ ID:1614342963-1375356540#3174472, TYP:RRUserMetadataRequest, CGR:N/A, USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get, OA:172.29.0.3/32, XFF:null, DA:172.29.0.2/32, IDX:<N/A>, MET:GET, PTH:/_readonlyrest/metadata/current_user, CNT:<N/A>, HDR:Accept-Encoding=gzip,deflate, Accept=*/*, Connection=close, Host=elasticsearch:9200, User-Agent=node-fetch/1.0 (+https://github.com/bitinn/node-fetch), content-length=0, HIS:[::LOGSTASH::-> RULES:[auth_key->false]], [::APM::-> RULES:[auth_key->false]], [::KIBANA-SRV::-> RULES:[auth_key->false]], [::ADMIN::-> RULES:[auth_key->false]], [::HEARTBEAT::-> RULES:[auth_key->false]], [::FILEBEAT::-> RULES:[auth_key->false]], [::IGESP::-> RULES:[auth_key->false]], [::GESTAO::-> RULES:[auth_key->false]], [::FRONTEND::-> RULES:[auth_key->false]], [::LDAP_ADMIN::-> RULES:[ldap_auth->false]], [::LDAP_IGESP::-> RULES:[ldap_auth->false]], [::LDAP_GESTAO::-> RULES:[ldap_auth->false]], [::LDAP_FRONTEND::-> RULES:[ldap_auth->false]], }\u001B[0m", "cluster.uuid": "5awwQkafQOeD60FNybj6Mw", "node.id": "lFBD-fWpQCOK9Fr2P2v8mA"  }
1 Like

Hi @rebertty, what ES/Kibana/ROR versions are you using? Did you patch your kibana source correctly?

What I can see is logs like:

USR:[no info about user], BRS:true, KDX:null, ACT:cluster:ror/user_metadata/get

Which look like login attempts from Kibana, but without credentials attached, I can see Authorization header is not present in those requests… Weird!

Can you confirm you are able to login successfully as “elastic” and use Kibana normally as that user with discover, visualize, dashboards Kibana apps?

I will try to reproduce with your exact configuration in the meantime with the latest ELK version.

Hi @sscarduzio, I’m using version 7.15.0 and the ROR 1.35.0 free version.
Atually i’m using all features without problems, only when I refresh my workpad:

this error happens

Inside of my workpads I have a lot of essql like this:

imagem

I just tried this, and canvas works well with the test data sets provided by Kibana.

One thing I noticed is that there is no correspondence between the provided ACL and the ACL history I find in the log lines. Are you sure the readonlyrest.yml currently in use is the same you attached in this ticket?

1 Like

@rebertty I’m available for a live debug call via zoom (now I’m curious! :stuck_out_tongue_winking_eye: )

Hi, sorry about the delay, I’m sending my full ACL:

readonlyrest:
    audit_collector: true
    audit_index_template: "'auditoria'-yyy-MM"
    access_control_rules:
    - name: "::LOGSTASH::"
      auth_key: logstash:3ArwvkriDwUrOEm
      verbosity: error

    - name: "::APM::"
      auth_key: apmserver:3zizO7ykFlL4Fsj
      verbosity: error

    - name: "::KIBANA-SRV::"
      auth_key: kibana:4eBBYeN37C9sgFD

    - name: "::ADMIN::"
      auth_key: elastic:teste
      kibana_access: unrestricted
      verbosity: error

    - name: "::HEARTBEAT::"
      auth_key: heartbeat:48sdfs58df7eqwe1
      verbosity: error

    - name: "::FILEBEAT::"
      auth_key: filebeat:6ZhFaMkQ5W3xTlz
      verbosity: error

    - name: "::IGESP::"
      auth_key: igesp:2jOe95zv2m
      kibana_access: ro
      indices: [ ".kibana", "*igesp*", "observability*"]

    - name: "::GESTAO::"
      auth_key: gestao:sefazgestaoteste
      kibana_access: ro
      indices: [ ".kibana", "logstash-db-transacao-fazendario*", "observability*"]

    - name: "::FRONTEND::"
      auth_key: frontend:93tulgD32h
      kibana_access: ro
      verbosity: info
      indices: [ ".kibana", "*front*", "observability*"]

    - name: "::LDAP_ADMIN::"
      ldap_auth:
        name: "ldap1"
        groups: ["GG_ApoioTecnologico"]
      kibana_access: unrestricted

    - name: "::LDAP_IGESP::"
      ldap_auth:
        name: "ldap1"
        groups: ["GG_ELK_IGESP"]
      kibana_access: ro
      indices: [ ".kibana", "*igesp*", "observability*"]

    - name: "::LDAP_GESTAO::"
      ldap_auth:
        name: "ldap1"
        groups: ["GG_ELK_GESTAO"]
      kibana_access: ro
      indices: [ ".kibana", "logstash-db-transacao-fazendario*", "observability*"]

    - name: "::LDAP_FRONTEND::"
      ldap_auth:
        name: "ldap1"
        groups: ["GG_ELK_FRONTEND"]
      kibana_access: ro
      indices: [ ".kibana", "*front*", "observability*"]


#Configuração de autenticação LDAP, retornando os grupos em que o usuário pertence.
    ldaps:

    - name: ldap1
      host: "XXXXXXXXXX"
      port: 389
      ssl_enabled: false
      ssl_trust_all_certs: true 
      ignore_ldap_connectivity_problems: false 
      bind_dn: "CN=account_schedule,OU=Services Users,OU=Suporte,DC=sefaz,DC=se,DC=gov,DC=br"
      bind_password: "XXXXXXXXX"
      search_user_base_DN: "ou=Desenvolvimento,dc=sefaz,dc=se,dc=gov,dc=br"
      search_groups_base_DN: "OU=Desenvolvimento,DC=sefaz,DC=se,DC=gov,DC=br"
      group_search_filter: "(objectClass=group)"
      unique_member_attribute: "member"
      user_id_attribute: "sAMAccountName"

      connection_pool_size: 30                                # optional, default 30
      connection_timeout_in_sec: 10                             # optional, default 1
      request_timeout_in_sec: 10                                # optional, default 1
      cache_ttl_in_sec: 10

It would be interesting, but I warn you that I don’t speak much English, I have availability around 15:00 UTC-3

Inspecting my Firefox I get this log:

Not sure if it’s the root cause, but I notice we are hitting some access restrictions to the browser’s local storage, and the session refresh fails, so most likely the session expires quickly after.

I have a related question: do you use multiple kibana servers (high availability) and have a load balancer/reverse proxy in front of the kibana instances?

You could try a few things, tell me which one (if any) solves the issue:

  • Try using a plain google chrome browser
  • Try adding these to kibana.yml, in order to sidestep session refresh and probing:
readonlyrest_kbn.sessions_probe_interval_seconds: 99999
readonlyrest_kbn.sessions_refresh_after: 99999
readonlyrest_kbn.store_sessions_in_index: true
1 Like

I’m using single-node, only a reverse proxy, but happening the same thing when I use the hostname to access.
I try use your parameters, but still the same thing.

Thank you @rebertty for the info.

OK so let’s recap:

  • the issue is on ROR Free 1.35.0 with Kibana 7.15.0
  • customer can reproduce the issue in Google Chrome, and super-long session refresh and session duration timeouts
  • As an unrestricted user, I create a canvas with some data
  • I click on refresh canvas button in Kibana
  • Elasticsearch sends a 403 FORBIDDEN, resulting in Kibana logging the user out.
  • The ES logs at the time of the test show forbidden request for “/” and “/_readonlyrest/metadata/current_user” without any Authorization header.

Can you confirm the above? In particular the last point, are the provided logs timestamps aligned to the time you ran the experiment?

I tried again to reproduce the issue with the latest development branch of ROR without success. However, in the meantime I would recommend trying with the latest available ROR 1.36.0, due to the vast number of fixes included.

I confirm all, I’m using ROR free 1.36.0 now, but nothing change in this case.

OK Thanks for confirming, I can confirm too that there’s something I must be missing in the way I reproduce this. The only thing remaining is to have a live debug session.

We can schedule, my user in telegram is: @rebertty

Hi, I finally managed to solve it, I need to add the ip of the kibana container with an Allow:

- name: Hosts
  type: allow  
  hosts: [ "192.168.16.6" ]

Looks like a workaround to me, the underlying issue might still be there. Are you willing to share your entire docker-compose files? This thing still stinks.

I send my compose to a repository in github, follow the link:

https://github.com/rebertty/ELK_ROR_3012.git

1 Like