Field rule not working when exceeding a certain no of docs

Diana · September 12, 2023, 2:52pm

Hello,

I encountered a strange situation , i will describe it below.
Tested with both ELK version 8.6.2 /ROR 1.51 & ELK version 7.17.2/ROR 1.43.0.
ROR configuration used:

  - name: "fftest Kibana"
    groups: ["fftest"]
    indices: ["test01"]
    fields: ["~user1"]
    kibana_access: "rw"
    kibana_index: ".kibana_fftest"
  - name: "fftest Kibana 1"
    groups: ["fftest"]
    indices: [".kibana_fftest" ,"test01"]
    kibana_access: "rw"
    kibana_index: ".kibana_fftest"

I am creating an index called test01 and i want to exclude from searches the “user1” field.
At first i created this index with only a few documents(<10).
I get the expected response, without the user1 field in any search i do.

If i insert more than 10 documents (or 29 docs for elk 8.6.2), the user1 field starts appearing in the search results, but only for a match all query. I will add below used queries and responses:

GET _search?size=100
{
  "query": {
    "match_all": {}
  }
}

response:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 29,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test01",
        "_id": "UHfSiYoB2fo4isNUxF1X",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "HwnSiYoBJCN9dDN95zXQ",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "TXfSiYoB2fo4isNUFF3v",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "T3fSiYoB2fo4isNUt12h",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "UXfSiYoB2fo4isNU1l1S",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "IAnSiYoBJCN9dDN99TU9",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "0q7SiYoBkxx3SbNmqPX4",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "IQnTiYoBJCN9dDN9BTXY",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "067TiYoBkxx3SbNmFPWA",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "HgnSiYoBJCN9dDN9kzWm",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "3K7UiYoBkxx3SbNmIfUD",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "3a7UiYoBkxx3SbNmMPWp",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "VHfUiYoB2fo4isNUXl1T",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "2K7TiYoBkxx3SbNm0PVH",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "2a7TiYoBkxx3SbNm4PVb",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "2q7TiYoBkxx3SbNm7_VP",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "U3fUiYoB2fo4isNUEl1F",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "267UiYoBkxx3SbNmA_W1",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "JgnUiYoBJCN9dDN9RDUm",
        "_score": 1,
        "_source": {
          "user1": "a",
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "TnfSiYoB2fo4isNUgl03",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "IwnTiYoBJCN9dDN9VjWV",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "167TiYoBkxx3SbNmnvWo",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "UnfTiYoB2fo4isNUJF1_",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "1K7TiYoBkxx3SbNmNPW5",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "IgnTiYoBJCN9dDN9SDUM",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "JAnTiYoBJCN9dDN9jjVu",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "JQnTiYoBJCN9dDN9wDVU",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "1a7TiYoBkxx3SbNmafWC",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      },
      {
        "_index": "test01",
        "_id": "1q7TiYoBkxx3SbNmfPWP",
        "_score": 1,
        "_source": {
          "user2": "b",
          "user3": "c"
        }
      }
    ]
  }
}

GET test01/_search
{
  "query": {
    "match": {
      "user1": "a"
    }
  }
}

response:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

GET test01/_search
{
  "query": {
    "exists": {
      "field": "user1"
    }
  }
}

response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

As you can see in the match all query response, only 10 documents are filtered, and the others show all the fields, not taking the rule into account anymore.

coutoPL · September 12, 2023, 4:38pm

WOW. This is interesting. I’m taking a look!

coutoPL · September 12, 2023, 8:01pm

I was able to reproduce the issue. We’re working on fixing it

coutoPL · September 22, 2023, 7:34pm

We have a fix. It will be released with ROR 1.51.1 this weekend.

coutoPL · September 25, 2023, 7:50pm

We have a little delay with the ROR 1.51.1 release, so I’m sending the pre-build

coutoPL · September 26, 2023, 2:48pm

ROR 1.51.1 has been released.
@Diana can you confirm the issue is gone?

octav · November 22, 2023, 1:41pm

Hello @coutoPL , problem was fixed for the match_all, however there is a very strange problem happening now.

I have an index where i’ve made a rule for fields, that for some user, he cannot view the contents of that field - ~input_raw for example.
This index contains a field df_date where date is stored like 2023-02-21T11:47:22.992000

After that, i’m trying from the Dev Tools in kibana, with that user i’m executing the following code:

GET /discovery-pseudo-dataset/_search
{
  "query": {
    "match": {
      "input_raw": "12345678"
    }
  }
}

It doesn’t return anything which is ok.
However, if i’m trying the same query using the following python libraries:

elasticsearch==7.12.1
install elasticsearch-dsl==7.4.0
install elasticsearch-service
install urllib3==1.26.2

and i’m executing the following code

import elasticsearch_dsl, elasticsearch
from ssl import create_default_context, CERT_NONE
from elasticsearch_service import ElasticsearchService
context = create_default_context()
context.check_hostname = False
context.verify_mode = CERT_NONE
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

extra = {'scheme':'https','http_auth_username':'account','http_auth_password':'password', 'ssl_context': context}

es=ElasticsearchService('es-cluster.com', '443', **extra)

q = elasticsearch_dsl.Q("range", **{"df_date": {"gte": "2023-02-20", "lte": "2023-02-22"}})

returned_data = es.get_documents_with_q('dataset', query=q)

returned_data['input_raw']

returned_data

that input_raw field is displayed.

coutoPL · November 22, 2023, 4:10pm

maybe the python lib uses _msearch or _async_search instead of _search?

Could you please show us the ALLOWED log from the ES log. I’d say more.

Nevertheless, it looks like a bug.

octav · November 23, 2023, 2:11pm

PM sent with requested logs.

coutoPL · November 23, 2023, 8:17pm

Thanks.

I’ve checked your logs, and this is how I thought - the python lib doesn’t use _search, but Scroll API | Elasticsearch Guide [8.11] | Elastic.

I will try to reproduce it in our tests.

octav · November 28, 2023, 7:51am

Hello @coutoPL ,

Any news about this request?

coutoPL · November 28, 2023, 8:51pm

Sadly I cannot reproduce it using ES API only. I tried to use the python code you sent locally but it doesn’t work. Maybe you can prepare a dockerfile to run your python script? You can create a new branch in this repo and put your code there.

octav · November 29, 2023, 2:32pm

Hey @coutoPL ,

It was harsh to reproduce this but finally i was able to do so.
I’ve sent you in a PM all details on how to reproduce this problem.
Looking forward!

coutoPL · December 17, 2023, 8:20pm

ROR 1.54.0 with the fix is released.

octav · December 18, 2023, 7:20am

Thank you very much!