Access based on filter using terms on a list (array)

I am using a filter based access, I have user_id_list and role in the JWT token payload. I want to see all the documents whose field is from the list supplied in user_id_list.
I use the following ACL.

  • name: “Access for data of selected users”
    filter: ‘{“bool”: { “must”: { “terms”: { “user_id”: “@{jwt:user_id_list}” }}}}’
    jwt_auth:
    name: “jwt_provider_1”
    roles: [“user_access”]

user_id is of type ‘keyword’. When I give the same in query section in ES http GET request, I see the docs of all users given in the list.

I see the following error in ES logs.

[2020-04-16T12:49:00,183][DEBUG][o.e.a.s.TransportSearchAction] [ip-172-31-6-235] All shards failed for phase: [query]
org.elasticsearch.common.ParsingException: [terms] query does not support [user_id]
at org.elasticsearch.index.query.TermsQueryBuilder.fromXContent(TermsQueryBuilder.java:391) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchModule.lambda$registerQuery$12(SearchModule.java:852) ~[elasticsearch-7.2.0.jar:7.2.0]

I tried with this query too -> { “terms”: { “user_id”: “@{jwt:user_id_list}” }}
I see the same error in ES logs.

Can anyone please let me know the correct way of achieving this functionality?

@eswrkman can you make an example of how would you envision the filter expression to look like once expanded, given for example user_id_list = ["alice", "bob"]?

Select all docs with user_id=‘alice’ or user_id=‘bob’.

This is nohting that can be done with simple string substitution and array “explode”. The end result of the expansion should in theory be something like:


{
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "user_id": {
              "value": "alice"
            }
          }
        },
        {
          "term": {
            "user_id": {
              "value": "bob"
            }
          }
        }
      ]
    }
  }
}

So nothing comes to mind to me. @coutoPL WDYT?

wow, it looks like sth hard (or impossible) to do without parsing a query (I mean we need to know a context to properly build new query)

@coutoPL Would this be too ugly? LOL

:warning: WARNING PSEUDO-CODE (not actually working)

filter: '{"query":{"bool":{"should":[@{jwt:user_ids}.mkString("""{"term":{"user_id",{"value":"""", ",", """"}}}""") ]}}}'

I understand all of this should be configured in filter rule? what is this syntax? :stuck_out_tongue:

It’s a pseudocode inspired to Scala’s mkString function, hypothetically chained to a ROR ACL dynamic variable.

In pure Scala would look like this:

Seq("alice","bob").mkString("""{"term":{"user_id",{"value":"""", ",", """"}}}""")

@sscarduzio @coutoPL
As I mentioned, submitting an ES query (not in ROR) like this worked:

POST /_search
{  
  "query": {
    "terms":{
      "user_id": ["alice", "bob"]
     }
  }
}

and even this worked:

 POST /_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "user_id": ["alice","bob"]
          }
        }
      ]
    }
  }
}

I am using ES 7.2 and the documentation says that same clauses can be used in both query and filter contexts.

Will this information help?

I checked our tests and we have sth which can be helpful here:

According to this and the post above, I think sth like this below should work:

filter: ‘{“bool”: { “must”: { “terms”: { “user_id”: [@{jwt:user_id_list}] }}}}’

When user_id_list will be passed as a list in JWT payload, then it should be resolved to the first shown example.

How to test this kind of configuration though? Will the expansion be shown in logs (or debug logs?) whenever a search request with a JWT arrives?

AFAIR there is no log with expanded variables.

@coutoPL This is exactly what I tried in the ACL given in beginning of this thread. I saw an error in ES logs, and not able to get the data when the token is passed (details of both given in the first message in this thread).

please notice that you filter query from the first post is a little bit different :slight_smile: Or maybe you have already checked new one and this is also doesn’t work for you?

@coutoPL With the filter you suggested, I got this error from ES logs

Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'alice': was expecting ('true', 'false' or 'null')
 at [Source: [email protected]; line: 1, column: 66]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]

Seems to be a parsing problem here. Does ROR accept lists in token payloads?
Are we supposed to give a comma delimited string instead of a list and use the above syntax?

But did you get this error at ES starting level or at request handling level. And you know, a stacktrace is much longer - this part tells me nothing about context.

Sorry about that. Here is the full log.
[2020-04-21T17:53:29,063][WARN ][r.suppressed ] [ip-172-31-6-235] path: /index-2020-04-11/_doc/_search, params: {index=index-2020-04-11, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1223) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1197) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]

Caused by: org.elasticsearch.ElasticsearchException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'alice': was expecting ('true', 'false' or 'null')
 at [Source: [email protected]; line: 1, column: 66]
        at org.elasticsearch.ExceptionsHelper.convertToElastic(ExceptionsHelper.java:64) ~[elasticsearch-7.2.0.jar:7.2.0]
        at tech.beshu.ror.es.security.RoleIndexSearcherWrapper.wrap(RoleIndexSearcherWrapper.java:119) ~[?:?]
        at org.elasticsearch.index.shard.IndexSearcherWrapper.wrap(IndexSearcherWrapper.java:77) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1232) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1216) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:632) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:622) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:585) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Dv3358McsJacobs': was expecting ('true', 'false' or 'null')
 at [Source: [email protected]; line: 1, column: 66]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2839) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1903) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[jackson-core-2.8.11.jar:2.8.11]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.TermsQueryBuilder.parseValues(TermsQueryBuilder.java:418) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.TermsQueryBuilder.fromXContent(TermsQueryBuilder.java:376) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchModule.lambda$registerQuery$12(SearchModule.java:852) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:314) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.BoolQueryBuilder.fromXContent(BoolQueryBuilder.java:299) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchModule.lambda$registerQuery$12(SearchModule.java:852) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:314) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.QueryShardContext.parseInnerQueryBuilder(QueryShardContext.java:389) ~[elasticsearch-7.2.0.jar:7.2.0]
        at tech.beshu.ror.es.security.RoleIndexSearcherWrapper.wrap(RoleIndexSearcherWrapper.java:113) ~[?:?]
        at org.elasticsearch.index.shard.IndexSearcherWrapper.wrap(IndexSearcherWrapper.java:77) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1232) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1216) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:632) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:622) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:585) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) ~[?:?]

The error was at request handling level. ES is up and running, no problem during startup, even in my earlier trials.

Ok thanks for the stack. I’ll write a test, because at a first glance I don’t see why this happened. Will back to you when I have sth

@eswrkman I wrote a test for your case and it works like a charm :wink: Maybe you should check again your JWT payload. Notice that user_id_list should be JSON array of strings. Like below:

  "sub": "test",
  "userId": "user5",
  "user_id_list": [
    "alice",
    "bob"
  ]
}