JWT Header Auth Fails

Hi

I am using the elasticsearch and kibana enterprise plugin. I am trying to use jwt (from a header) to authenticate, but it fails and elasticsearch returns forbidden and kibana is redirected to the login page.

I have verified that the token is not expired, that the public key in readonlyrest.yml is correct, and that user and role claim is correct.

Is there any way to see in the logs why the jwt auth fails? I am not sure what fails (verifying signature, roles match, user claim) and can’t seem to figure it out. Let me know if you want me to send you a PM with a jwt.

OS: Windows Server 2016
ELK version: 7.2.0
ROR Elasticsearch: readonlyrest-1.18.4_es7.2.0
ROR Kibana: readonlyrest_kbn_enterprise-1.18.5-pre13_es7.2.0

Forbidden from elasticsearch log

[2019-08-26T08:49:29,434][INFO ][t.b.r.a.l.AclLoggingDecorator] [U-114-270-028-ROR] e[35mFORBIDDEN by default req={  ID:1032645823-168266012#1174,  TYP:RRAdminRequest,  CGR:N/A,  USR:[user not logged],  BRS:false,  KDX:null,  ACT:cluster:admin/rradmin/refreshsettings,  OA:172.16.111.87/32,  XFF:null,  DA:172.16.111.87/32,  IDX:<N/A>,  MET:GET,  PTH:/_readonlyrest/metadata/current_user,  CNT:<N/A>,  HDR:Connection=keep-alive, Content-Length=0, Host=u-114-270-028:9210, jwt=<MY JWT TOKEN>,  HIS:[::LOGSTASH::-> RULES:[auth_key_sha256->false], RESOLVED:[]], [::KIBANA-SRV::-> RULES:[auth_key_sha256->false], RESOLVED:[]], [::ADMIN_BP::-> RULES:[jwt_auth->false], RESOLVED:[]], [::RW_BP::-> RULES:[jwt_auth->false], RESOLVED:[]], [::RW_BP (personal)::-> RULES:[jwt_auth->false], RESOLVED:[]], [::RO_BP::-> RULES:[jwt_auth->false], RESOLVED:[]]  }e[0m

Kibana log

  log   [07:03:38.662] [error][readonlyrest_kbn:enrichFromEs] test authenticated call to ES failed{"msg":"[undefined] fo
rbidden, with { due_to={ 0=\"OPERATION_NOT_ALLOWED\" } }","path":"/_readonlyrest/metadata/current_user","query":{},"stat
usCode":401,"response":"{\"error\":{\"root_cause\":[{\"reason\":\"forbidden\",\"due_to\":[\"OPERATION_NOT_ALLOWED\"]}],\
"reason\":\"forbidden\",\"due_to\":[\"OPERATION_NOT_ALLOWED\"]},\"status\":401}","wwwAuthenticateDirective":"Basic"} - f
or GET /_readonlyrest/metadata/current_user  error: {"msg":"[undefined] forbidden, with { due_to={ 0=\"OPERATION_NOT_ALL
OWED\" } }","path":"/_readonlyrest/metadata/current_user","query":{},"statusCode":401,"response":"{\"error\":{\"root_cau
se\":[{\"reason\":\"forbidden\",\"due_to\":[\"OPERATION_NOT_ALLOWED\"]}],\"reason\":\"forbidden\",\"due_to\":[\"OPERATIO
N_NOT_ALLOWED\"]},\"status\":401}","wwwAuthenticateDirective":"Basic"}
error was:  Error: invalid identity: {"error":"[undefined] forbidden, with { due_to={ 0=\"OPERATION_NOT_ALLOWED\" } } -
for GET /_readonlyrest/metadata/current_user"}
    at h (E:\elastictestror\kibana-7.2.0-windows-x86_64\plugins\readonlyrest_kbn\server\routes\lib/identityManager.js:1:
2646)
    at f (E:\elastictestror\kibana-7.2.0-windows-x86_64\plugins\readonlyrest_kbn\server\routes\lib/identityManager.js:1:
4349)
    at process._tickCallback (internal/process/next_tick.js:68:7)

readonlyrest.yml

readonlyrest:

    access_control_rules:
    - name: "::LOGSTASH::"
      auth_key_sha256: <MY LOGSTASH CREDENTIALS>
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["queueitems-*", "sessions-*", "resources-*", "queues-*"]

    - name: "::KIBANA-SRV::"
      auth_key_sha256: <MY KIBANA CREDENTIALS>
      verbosity: error

    - name: "::ADMIN_BP::"
      indices: ["queueitems-*", "sessions-*", "resources-*", "queues-*", ".kibana", ".monitoring*", ".reporting*", ".tasks*", ".kibana_task*",".readonlyrest", "kibana_sample*"]
      kibana_access: admin
      jwt_auth:
        name: "webseal"
        roles: ["Role_kibana_admin_blueprism"]

    - name: "::RW_BP::"
      kibana_access: rw
      indices: ["queueitems-*", "sessions-*", "resources-*", "queues-*", ".kibana", ".monitoring*", ".reporting*", ".tasks*", ".kibana_task*", "kibana_sample*"]
      kibana_hide_apps: ["readonlyrest_kbn"]
      jwt_auth:
        name: "webseal"
        roles: ["Role_kibana_readwrite_blueprism"]

    - name: "::RW_BP (personal)::"
      kibana_access: rw
      indices: ["queueitems-*", "sessions-*", "resources-*", "queues-*", "[email protected]{user}", ".monitoring*", ".reporting*", ".tasks*", ".kibana_task*","@{user}-*", "kibana_sample*"]
      kibana_index: "[email protected]{user}"
      kibana_hide_apps: ["readonlyrest_kbn"]
      jwt_auth:
        name: "webseal"
        roles: ["Role_kibana_readwrite_blueprism", "Role_kibana_admin_blueprism"]

    - name: "::RO_BP::"
      kibana_access: ro
      indices: ["queueitems-*", "sessions-*", "resources-*", "queues-*", ".kibana", "kibana_sample*"]
      kibana_hide_apps: ["readonlyrest_kbn", "kibana:visualize", "canvas", "apps", "ml", "infra:infrastructure","infra:logs", "apm", "uptime", "siem", "kibana:dev_tools", "monitoring", "kibana:management"]
      jwt_auth:
        name: "webseal"
        roles: ["Role_kibana_readonly_blueprism"]

    jwt:
    - name: webseal
      signature_algo: RSA
      signature_key: "<MY PUBLIC KEY>"
      user_claim: sub
      roles_claim: groups
      header_name: jwt

elasticsearch.yml

cluster.name: rortest
node.name: U-114-270-028-ROR
network.host: U-114-270-028
http.port: 9210
transport.port: 9538

xpack.security.enabled: false

discovery.zen.ping.unicast.hosts: ["U-114-270-028"]
cluster.initial_master_nodes: 
   - U-114-270-028-ROR

kibana.yml

server.port: 5610

server.host: "localhost"

server.basePath: "/kibana"
server.rewriteBasePath: false

elasticsearch.hosts: "http://U-114-270-028:9210"

xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false

readonlyrest_kbn.forward_auth_header: jwt
elasticsearch.requestHeadersWhitelist: [jwt]
#readonlyrest_kbn.jwt_query_param: "jwt"

#Real username and password stored in keystore. For reference https://www.elastic.co/guide/en/kibana/current/secure-settings.html
#elasticsearch.username: "kibana"
#elasticsearch.password: "kibana"
readonlyrest_kbn.clearSessionOnEvents: ["never"]

Decoded JWT

{
  "sub": "G019496",
  "iss": "https://www.example.com",
  "aud": "urn:liberty",
  "iat": 1566801867,
  "exp": 1566805467,
  "sess": "f2cdaec0-c7cc-11e9-a8b8-005056824880",
  "groups": [
    "Role_kibana_admin_blueprism",
    "not_relevant_role_1",
    "not_relevant_roles_2",
    "not_relevant_roles_3",
    "not_relevant_roles_4",
    "not_relevant_roles_5",
    "not_relevant_roles_6",
    "not_relevant_roles_7",
    "not_relevant_roles_8",
    "not_relevant_roles_9",
    "not_relevant_roles_10",
    "not_relevant_roles_11",
    "not_relevant_roles_12"
  ]
}

The issue was that the header was not containing "Bearer " at the start.

Does not work with header jwt: TOKEN
Works with header jwt: Bearer TOKEN