Auth failure in Elasticsearch with JWT

Sireesha · July 25, 2019, 9:43am

I am using basic auth + JWT auth in readonlyrest.yml as follows:
readonlyrest:
access_control_rules:

- name: "Valid JWT token with admin role"
  type: allow
  jwt_auth:
    name: "jwt_provider_1"
    roles: ["admin"]

- name: "Require HTTP Basic Auth"
  type: allow
  auth_key: user:pwd

jwt:
- name: jwt_provider_1
  signature_algo: HMAC # can be NONE, RSA, HMAC (default), and EC
  signature_key: "my-key"
  user_claim: email
  roles_claim: roles # JSON-path style
  header_name: Authorization

I am able to login as ‘user’ through auth_key declaration. But when I pass JWT in header ‘Authorization’: "Bearer "+token, am getting 401 error.

Header and payload contents of the token:
—header—
{
“alg”: “HS256”,
“typ”: “JWT”
}
—payload—
{
“email”: “admin_user”,
“roles”: [“admin”]
}

signature_key is the secret key (“my-key” above) is base64 encoded and sent along with payload and header.
I saw in one of the old links that signature_key is initialized to the token itself. Please let me know if signature_key and claims sections are mutually exclusive if it’s the case.
Am using ES7.2 free plugin

I see messages of this type in ES logger.
[2019-07-25T09:07:28,595][INFO ][t.b.r.a.l.AclLoggingDecorator] [ip-172-31-1-64] ^[[36mALLOWED by { name: ‘Require HTTP Basic Auth’, policy: ALLOW, rules: [auth_key] req={ ID:2085141796-1885939433#150, TYP:XPackInfoRequest, CGR:N/A, USR:admin, BRS:false, KDX:null, ACT:cluster:monitor/xpack/info, OA:18.217.103.157/32, XFF:null, DA:172.31.1.64/32, IDX:<N/A>, MET:GET, PTH:/_xpack, CNT:<N/A>, HDR:Authorization=, Connection=keep-alive, Content-Length=0, Host=18.191.234.117:9200, HIS:[Valid JWT token with admin role-> RULES:[jwt_auth->false], RESOLVED:[]], [Require HTTP Basic Auth-> RULES:[auth_key->true], RESOLVED:[user=admin]] }^[[0m

Looks like the contents of the header are being ignored. Is there any other setting in readonlyrest.yml or elasticsearch.yml to be made?

sscarduzio · July 25, 2019, 3:40pm

The JWT block is not ignored, it just does not match because the JWT rule fails. A few reasons this could happen:

You use a key of length < 256 characters (while using HS256 algorithm)
Your token expiration date is passed
You configured ROR with a base64 encoded version of the actual key string.

For reference, below is a valid configuration with JWT:

readonlyrest:

  access_control_rules:
    - name: "jwt1_block"
      jwt_auth: "jwt1"

  jwt:
    - name: jwt1
      signature_key: "A22XIbz4NKBkka0ANWwwiJsTFeyQiFJdklRT70VieAdyk9khk1j9tc1Kg3XTSCHMWXYfb26R4pwkQRvUmdLgYWYbPiEN7VY4hWzUIbDlWZAhlkscG4BxLXfal1eKzxE5y1kdM7xTsz8bO2AFePttoAfJNNJukATJ7LXBRh8ja31qKZPKDmb31RaOqENhcGjj4jRUPUKoPsmHMowTk4SNsfiPeXqppshhruxY5hmd71IRgfRbNZ8KwHu6F75C2456"
      user_claim: sub
      signature_algo: HMAC
      header_name: Authorization

Sireesha · July 29, 2019, 2:58pm

Even after using your signature key and making other checks mentioned, the user is not being authenticated. Meanwhile am observing another issue where the jwt_auth key is not being checked, error is being thrown since the user is not logged.

Do we need to restart elasticsearch after making changes to readonlyrest.yml?
Can basic auth and jwt auth co-exist? If so, what is the order of specifying them. I have 4 to 5 rules for admin and less privileged users and JWT is declared as the first ACL block.

Log contents
" [INFO ][t.b.r.a.l.AclLoggingDecorator] [ip-addr] ^[[35mFORBIDDEN by default req={ ID:1573878324-1830986459#1725, TYP:IndexReque st, CGR:N/A, USR:[user not logged], BRS:true, KDX:null, ACT:indices:data/write/index, OA:, XFF:null, DA:, IDX:setsight_me tadata, MET:PUT, PTH:/setsight_metadata/_doc/period_metric_aliases, CNT:<OMITTED, LENGTH=24.0 B> , HDR:Accept-Encoding=gzip, deflate, Accept=/, Authorizati on=, Connection=keep-alive, Content-Length=24, Content-Type=application/json, Host=, User-Agent=python-requests/2.20.1, HIS:[Require HTTP Basic Auth-> RULES:[auth_key->false], RESOLVED:[]], [Setsight admin user-> RULES:[auth_key->false], RESOLVED:[]], [Setsight user-> RULES:[auth_key->false], RESOLVED:[]], [Vendor-> RULES:[auth_key->false], RESOLVED:[]], [::FIRST BLOCK::-> RULES:[hosts->false], RESOLVED:[]] }"

I can see that JWT ACL is not being considered.

sscarduzio · July 29, 2019, 4:50pm

Hi @Sireesha,

File based configuration behaves exactly like elasticsearch.yml. So yes you need to restart the ES node when you change the readonlyrest.yml file. With ROR PRO/Enterprise we have a Kibana app that saves settings on index and restart is not necessary anymore.
Multiple authentication methods can coexist in the same ACL, but not in the same block. Please share with us your current, complete YAML settings (minus passwords of course)?.

About your log extract, I can see there must be a mistake in your Python code where the “Authorization” header has a space in the middle of the word and no value.

Sireesha · July 29, 2019, 5:21pm

@sscarduzio

readonlyrest:
  access_control_rules:

    - name: "Valid JWT token with a writer role"
      jwt_auth: "jwt_provider_1"

    - name: "Require HTTP Basic Auth"
      type: allow
      auth_key: admin:<pwd>

    - name: "xyz admin user"
      type: allow
      auth_key: <usr2>:<pwd>
      indices: ["usr2_*"]
      actions: ["indices:data/read/*", "indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      kibana_access: rw

    - name: "Specific user"
      type: allow
      auth_key: specific_user:<pwd>
      indices: ["usr_metadata"]
      actions: ["indices:data/read/*"]
      kibana_access: ro

    - name: "Vendor"
      type: allow
      auth_key: vendor:<pwd>
      indices: ["vendor_*"]
      filter: '{"bool": { "must": { "match": { "myfield": "78790900" }}}}'
      actions: ["indices:data/read/get", "indices:data/read/search"]
      kibana_access: ro
      kibana_hide_apps: ["timelion", "kibana:dev_tools", "kibana:management"]

    - name: "::FIRST BLOCK::"
      hosts: ["127.0.0.1"]


  jwt:
    - name: "jwt_provider_1"
      signature_algo: HMAC # can be NONE, RSA, HMAC (default), and EC
      signature_key: "A22XIbz4NKBkka0ANWwwiJsTFeyQiFJdklRT70VieAdyk9khk1j9tc1Kg3XTSCHMWXYfb26R4pwkQRvUmdLgYWYbPiEN7VY4hWzUIbDlWZAhlkscG4BxLXfal1eKzxE5y1kdM7xTsz8bO2AFePttoAfJNNJukATJ7LXBRh8ja31qKZPKDmb31RaOqENhcGjj4jRUPUKoPsmHMowTk4SNsfiPeXqppshhruxY5hmd71IRgfRbNZ8KwHu6F75C2456"
user_claim: sub
header_name: Authorization

Also, pasting the log in this folder is omitting the data between < and >. The content is HDR:Authorization=’<’+OMITTED+’>’

sscarduzio · July 30, 2019, 9:28am

We have the code formatting button (looks like </>) in the editor. Please make sure you use it to avoid confusions.

sscarduzio · July 30, 2019, 9:34am

So to recap:

have you restarted the node after saving readonlyrest.yml with the above configuration?
did you see ES logs that testify that ALL the blocks and rules are loaded correctly? I.e. this is what I see on ES restart:

2019-07-30T10:13:30,919][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: '::Kafka::', policy: ALLOW, rules: [actions]
[2019-07-30T10:13:30,921][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: '::LOGSTASH::', policy: ALLOW, rules: [auth_key,indices,actions]
[2019-07-30T10:13:30,922][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: '::KIBANA-SRV::', policy: ALLOW, rules: [auth_key]
[2019-07-30T10:13:30,926][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: '::PERSONAL_GRP::', policy: ALLOW, rules: [groups,kibana_access,kibana_hide_apps,kibana_index]
[2019-07-30T10:13:30,929][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: '::ADMIN_GRP::', policy: ALLOW, rules: [groups,kibana_access]
[2019-07-30T10:13:30,930][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: '::Infosec::', policy: ALLOW, rules: [groups,kibana_access,kibana_hide_apps,kibana_index]
[2019-07-30T10:13:30,933][INFO ][t.b.r.a.f.RawRorConfigBasedCoreFactory] [GYcT8iq] ADDING BLOCK:	{ name: 'ReadonlyREST Enterprise instance #1', policy: ALLOW, rules: [ror_kbn_auth]
[2019-07-30T10:13:31,017][INFO ][t.b.r.b.RorInstance      ] [GYcT8iq] Readonly REST plugin core was loaded ...
[2019-07-30T10:13:31,243][DEBUG][o.e.a.ActionModule       ] [GYcT8iq] Using REST wrapper from plugin tech.beshu.ror.es.ReadonlyRestPlugin

Notice how each ACL block causes ROR to log a “ADDING BLOCK”.

Sireesha · July 30, 2019, 1:16pm

@sscarduzio Yes, I restarted ES after making changes to readonlyrest.yml.
I did see each ACL getting added (ADDING BLOCK… as shown in your log). But after some time, ACLs got reloaded automatically, this time without JWT block.
Earlier, I installed readonlyrest pro for Kibana and saved the .yml settings from Kibana UI. This didn’t contain any JWT ACLs. Since we don’t need this plugin for Kibana, I uninstalled it from Kibana. Note that it was not uninstalled from ES.
Is there a known issue where ES persists with reloading ACL settings from Kibana though the plugin in uninstalled on Kibana? Other than this, am not able to understand why the ACLs are getting reloaded in quick succession without ES restart.

sscarduzio · July 30, 2019, 1:56pm

@Sireesha ok mystery is solved then, the ES plugin is loading the settings from the “.readonlyrest” index left behind by the Kibana plugin.
You can remove the “.readonlyrest” index and restart ES, things will be ok.

Sireesha · July 30, 2019, 2:04pm

Yes, it worked after removing the .readonlyrest index. I thought it will be cleaned up during uninstallation of plugin. Thanks for being active on this thread and the quick turnaround.

sscarduzio · July 30, 2019, 2:27pm

No worries @Sireesha, happy to help. Were you currently in trial for PRO or Enterprise? Let me know if you have issues with that as well.

Sireesha · July 30, 2019, 3:35pm

I was using PRO, but we realized that it is not needed for our usecase.
Can any field declared in the payload of JWT be accessed using @field in the ACL?
For example, if I give {sub: “admin”, id: “3004”} in JWT payload, can I use @id in ‘indices’ or in 'filter?

sscarduzio · July 30, 2019, 5:10pm

yeah you can use variables from the JWT claims like this @{jwt:claim.json.path}
In your case, @{jwt:id}

Sireesha · July 30, 2019, 10:30pm

@sscarduzio
Here is the declaration I have in .yml life.

    - name: "Valid JWT token with admin role"
      jwt_auth:
         name:  "jwt_provider_1"
         user: "admin"

    - name: "Valid JWT token with setsight user"
      indices: ["@user_*"]
      jwt_auth:
         name: "jwt_provider_1"
         roles: ["admin_client"]

    - name: "Access for a vendor to his data"
      indices: ["@user_*"]
      filter: '{"bool": { "must": { "match": { "dim.vendor": "@{jwt:id}" }}}}'
      actions: ["indices:data/read/get", "indices:data/read/search"]
      jwt_auth:
         name: "jwt_provider_1"
         roles: ["vendor"]
jwt:
    - name: "jwt_provider_1"
      signature_algo: HMAC # can be NONE, RSA, HMAC (default), and EC
      signature_key: "A22XIbz4NKBkka0ANWwwiJsTFeyQiFJdklRT70VieAdyk9khk1j9tc1Kg3XTSCHMWXYfb26R4pwkQRvUmdLgYWYbPiEN7VY4hWzUIbDlWZAhlkscG4BxLXfal1eKzxE5y1kdM7xTsz8bO2AFePttoAfJNNJukATJ7LXBRh8ja31qKZPKDmb31RaOqENhcGjj4jRUPUKoPsmHMowTk4SNsfiPeXqppshhruxY5hmd71IRgfRbNZ8KwHu6F75C2456"
      user_claim: sub
      roles_claim: roles
      id_claim: id
      header_name: Authorization

The second JWT ACL is allowing the user to write into indices which start with its username. But the read is forbidden. Do we need to declare another ACL or explicitly list in the actions field?
Because the read on indices using user pattern is not successful, am not sure if am using id in the right way. I included id_claim in jwt_provider_1. Is this required at all?

Sireesha · July 31, 2019, 1:41pm

Using @{user} or @user in indices with or without roles in ACL is not having an effect through JWT tokens is not working. indices or kibana_indices is not declared in the ACL.The user is able to write to all indices but not read from any index.
@sscarduzio Not able to use the roles or any other custom field either. Please let me know if there is any known issue or an available fix for this.

sscarduzio · August 1, 2019, 10:19am

@Sireesha seriously, I need you to use the code button (icon is </> ) when you attach YAML or logs. Can you please edit your posts above?

Sireesha · August 1, 2019, 10:36am

@sscarduzio you can check now.

sscarduzio · August 1, 2019, 12:17pm

For as much as I understood, you could try:

   - name: "Valid JWT token with admin role"
      jwt_auth:
         name:  "jwt_provider_1"
         user: "admin"

    # THE READS WILL BE INTERCEPTED BY THIS BLOCK SO THEY GET FILTERED
    - name: "Access for a vendor to his data "  
      indices: ["@{user}_*"]
      filter: '{"bool": { "must": { "match": { "dim.vendor": "@{jwt:id}" }}}}'
      actions: ["indices:data/read/get", "indices:data/read/search"]
      jwt_auth:
         name: "jwt_provider_1"
         roles: ["vendor"]

   # THE WRITES WILL FAIL TO MATCH THE BLOCK ABOVE ("filter" ONLY MATCHES READS)
    - name: "Valid JWT token with setsight user"
      indices: ["@{user}_*"]
      jwt_auth:
         name: "jwt_provider_1"
         roles: ["admin_client"]

Sireesha · August 1, 2019, 2:17pm

@sscarduzio
When I create a token with this payload {sub: ‘setsight’, roles:“admin_client”}, I get write privileges through the first jwt ACL.

[2019-08-01T13:55:24,476][INFO ][t.b.r.a.l.AclLoggingDecorator] [ip] ^[[36mALLOWED by { name: 'Valid JWT token with admin role', policy: ALLOW, rules: [jwt_auth] req={ ID:530578388-1044832523#5272, TYP:IndexRequest, CGR:N/A, USR:setsight, BRS:true, KDX:null, ACT:indices:data/write/index, OA:<ip>, XFF:null, DA:<ip>, IDX:setsight_metadata, MET:PUT, PTH:/setsight_metadata/_doc/period_metric_aliases, CNT:<OMITTED, LENGTH=888.0 B> , HDR:Accept=application/json, text/plain, */*, Authorization=<OMITTED>, Connection=close, Content-Length=888, Content-Type=application/json, Host=<ip>, User-Agent=axios/0.19.0, HIS:[Valid JWT token with admin role-> RULES:[jwt_auth->true], RESOLVED:[user=setsight]]

I thought that since user is ‘admin’ in the first rule, the second rule gets applied.

Then, I removed the first rule completely, and executed the request. It was forbidden due to this.
[2019-08-01T14:13:43,454][INFO ][t.b.r.a.l.AclLoggingDecorator] [ip] ^[[35mFORBIDDEN by default req={ ID:1587389397-774997354#2583, TYP:GetRequest, CGR:N/A, USR:[user not logged], BRS:true, KDX:null, ACT:indices:data/read/get, OA:<ip>, XFF:null, DA:<ip>, IDX:setsight_metadata, MET:GET, PTH:/setsight_metadata/_doc/period_metric_aliases, CNT:<N/A>, HDR:Accept=application/json, text/plain, */*, Connection=close, Host=<ip>, User-Agent=axios/0.19.0, content-length=0, HIS:[Access for a vendor to his data -> RULES:[jwt_auth->false], RESOLVED:[]], [Valid JWT token with setsight user-> RULES:[jwt_auth->false], RESOLVED:[]], [Require HTTP Basic Auth-> RULES:[auth_key->false], RESOLVED:[]], [Setsight admin user-> RULES:[auth_key->false], RESOLVED:[]], [Setsight user-> RULES:[auth_key->false], RESOLVED:[]], [Vendor-> RULES:[auth_key->false], RESOLVED:[]], [Local host access-> RULES:[hosts->false], RESOLVED:[]] }

I expected USR:setsight instead here. Please also look at the jwt_provider declaration because the user is not being picked up from the token in some cases as shown above.

Sireesha · August 2, 2019, 9:05am

I just had to replace user with roles in the first ACL, and the rest worked as expected.