Access based on filter using terms on a list (array)

eswrkman · April 16, 2020, 4:46pm

I am using a filter based access, I have user_id_list and role in the JWT token payload. I want to see all the documents whose field is from the list supplied in user_id_list.
I use the following ACL.

name: “Access for data of selected users”
filter: ‘{“bool”: { “must”: { “terms”: { “user_id”: “@{jwt:user_id_list}” }}}}’
jwt_auth:
name: “jwt_provider_1”
roles: [“user_access”]

user_id is of type ‘keyword’. When I give the same in query section in ES http GET request, I see the docs of all users given in the list.

I see the following error in ES logs.

[2020-04-16T12:49:00,183][DEBUG][o.e.a.s.TransportSearchAction] [ip-172-31-6-235] All shards failed for phase: [query]
org.elasticsearch.common.ParsingException: [terms] query does not support [user_id]
at org.elasticsearch.index.query.TermsQueryBuilder.fromXContent(TermsQueryBuilder.java:391) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchModule.lambda$registerQuery$12(SearchModule.java:852) ~[elasticsearch-7.2.0.jar:7.2.0]

I tried with this query too -> { “terms”: { “user_id”: “@{jwt:user_id_list}” }}
I see the same error in ES logs.

Can anyone please let me know the correct way of achieving this functionality?

sscarduzio · April 16, 2020, 6:47pm

@eswrkman can you make an example of how would you envision the filter expression to look like once expanded, given for example user_id_list = ["alice", "bob"]?

eswrkman · April 16, 2020, 7:43pm

Select all docs with user_id=‘alice’ or user_id=‘bob’.

sscarduzio · April 19, 2020, 3:52pm

This is nohting that can be done with simple string substitution and array “explode”. The end result of the expansion should in theory be something like:


{
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "user_id": {
              "value": "alice"
            }
          }
        },
        {
          "term": {
            "user_id": {
              "value": "bob"
            }
          }
        }
      ]
    }
  }
}

So nothing comes to mind to me. @coutoPL WDYT?

coutoPL · April 19, 2020, 4:22pm

wow, it looks like sth hard (or impossible) to do without parsing a query (I mean we need to know a context to properly build new query)

sscarduzio · April 19, 2020, 5:49pm

@coutoPL Would this be too ugly? LOL

WARNING PSEUDO-CODE (not actually working)

filter: '{"query":{"bool":{"should":[@{jwt:user_ids}.mkString("""{"term":{"user_id",{"value":"""", ",", """"}}}""") ]}}}'

coutoPL · April 19, 2020, 5:58pm

I understand all of this should be configured in filter rule? what is this syntax?

sscarduzio · April 19, 2020, 6:06pm

It’s a pseudocode inspired to Scala’s mkString function, hypothetically chained to a ROR ACL dynamic variable.

In pure Scala would look like this:

Seq("alice","bob").mkString("""{"term":{"user_id",{"value":"""", ",", """"}}}""")

eswrkman · April 20, 2020, 4:07am

@sscarduzio @coutoPL
As I mentioned, submitting an ES query (not in ROR) like this worked:

POST /_search
{  
  "query": {
    "terms":{
      "user_id": ["alice", "bob"]
     }
  }
}

and even this worked:

 POST /_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "user_id": ["alice","bob"]
          }
        }
      ]
    }
  }
}

I am using ES 7.2 and the documentation says that same clauses can be used in both query and filter contexts.

Will this information help?

coutoPL · April 20, 2020, 2:29pm

I checked our tests and we have sth which can be helpful here:

github.com

sscarduzio/elasticsearch-readonlyrest-plugin/blob/01b35251a3e60b3529b7b624316aa6c9989716c5/core/src/test/scala/tech/beshu/ror/unit/acl/blocks/variables/RuntimeResolvableVariablesTests.scala#L209


      
            val variable = forceCreateSingleVariable("@{jwt:tech.beshu.mainGroup}")
              .resolve(currentUserMetadataRequestBlockContextFrom(
                _.withJwtToken(JwtTokenPayload {
                  val claims = new DefaultClaims()
                  claims.put("tech", Map("beshu" -> Map("mainGroup" -> "group1").asJava).asJava)
                  claims
                })
              ))
            variable shouldBe Right("group1")
          }
          "jwt variable is used with correct JSON path to strings array value and JWT token was set" in {
            val variable = forceCreateSingleVariable("@{jwt:tech.beshu.groups}")
              .resolve(currentUserMetadataRequestBlockContextFrom(
                _.withJwtToken(
                  JwtTokenPayload {
                    val claims = new DefaultClaims()
                    claims.put("tech", Map("beshu" -> Map("groups" -> List("group1", "group2").asJava).asJava).asJava)
                    claims
                  }
                )
              ))

According to this and the post above, I think sth like this below should work:

filter: ‘{“bool”: { “must”: { “terms”: { “user_id”: [@{jwt:user_id_list}] }}}}’

When user_id_list will be passed as a list in JWT payload, then it should be resolved to the first shown example.

sscarduzio · April 20, 2020, 4:34pm

How to test this kind of configuration though? Will the expansion be shown in logs (or debug logs?) whenever a search request with a JWT arrives?

coutoPL · April 20, 2020, 4:45pm

AFAIR there is no log with expanded variables.

eswrkman · April 20, 2020, 5:34pm

@coutoPL This is exactly what I tried in the ACL given in beginning of this thread. I saw an error in ES logs, and not able to get the data when the token is passed (details of both given in the first message in this thread).

coutoPL · April 20, 2020, 5:49pm

please notice that you filter query from the first post is a little bit different Or maybe you have already checked new one and this is also doesn’t work for you?

eswrkman · April 21, 2020, 11:07am

@coutoPL With the filter you suggested, I got this error from ES logs

Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'alice': was expecting ('true', 'false' or 'null')
 at [Source: java.io.StringReader@5cbfe71; line: 1, column: 66]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]

Seems to be a parsing problem here. Does ROR accept lists in token payloads?
Are we supposed to give a comma delimited string instead of a list and use the above syntax?

coutoPL · April 21, 2020, 2:32pm

But did you get this error at ES starting level or at request handling level. And you know, a stacktrace is much longer - this part tells me nothing about context.

eswrkman · April 21, 2020, 5:59pm

Sorry about that. Here is the full log.
[2020-04-21T17:53:29,063][WARN ][r.suppressed ] [ip-172-31-6-235] path: /index-2020-04-11/_doc/_search, params: {index=index-2020-04-11, type=_doc}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:441) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1223) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1197) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]

Caused by: org.elasticsearch.ElasticsearchException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'alice': was expecting ('true', 'false' or 'null')
 at [Source: java.io.StringReader@1fb25c09; line: 1, column: 66]
        at org.elasticsearch.ExceptionsHelper.convertToElastic(ExceptionsHelper.java:64) ~[elasticsearch-7.2.0.jar:7.2.0]
        at tech.beshu.ror.es.security.RoleIndexSearcherWrapper.wrap(RoleIndexSearcherWrapper.java:119) ~[?:?]
        at org.elasticsearch.index.shard.IndexSearcherWrapper.wrap(IndexSearcherWrapper.java:77) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1232) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1216) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:632) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:622) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:585) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Dv3358McsJacobs': was expecting ('true', 'false' or 'null')
 at [Source: java.io.StringReader@1fb25c09; line: 1, column: 66]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2839) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1903) ~[jackson-core-2.8.11.jar:2.8.11]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[jackson-core-2.8.11.jar:2.8.11]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.TermsQueryBuilder.parseValues(TermsQueryBuilder.java:418) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.TermsQueryBuilder.fromXContent(TermsQueryBuilder.java:376) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchModule.lambda$registerQuery$12(SearchModule.java:852) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:314) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.BoolQueryBuilder.fromXContent(BoolQueryBuilder.java:299) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchModule.lambda$registerQuery$12(SearchModule.java:852) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:415) ~[elasticsearch-x-content-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:314) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.query.QueryShardContext.parseInnerQueryBuilder(QueryShardContext.java:389) ~[elasticsearch-7.2.0.jar:7.2.0]
        at tech.beshu.ror.es.security.RoleIndexSearcherWrapper.wrap(RoleIndexSearcherWrapper.java:113) ~[?:?]
        at org.elasticsearch.index.shard.IndexSearcherWrapper.wrap(IndexSearcherWrapper.java:77) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1232) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:1216) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:632) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:622) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:585) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) ~[?:?]

eswrkman · April 21, 2020, 6:04pm

The error was at request handling level. ES is up and running, no problem during startup, even in my earlier trials.

coutoPL · April 21, 2020, 6:12pm

Ok thanks for the stack. I’ll write a test, because at a first glance I don’t see why this happened. Will back to you when I have sth

coutoPL · April 29, 2020, 4:59pm

@eswrkman I wrote a test for your case and it works like a charm Maybe you should check again your JWT payload. Notice that user_id_list should be JSON array of strings. Like below:

  "sub": "test",
  "userId": "user5",
  "user_id_list": [
    "alice",
    "bob"
  ]
}