/_readonlyrest/metadata/current_user Data too large

Hi… After i updated Kibana to 7.8.1-1 and [email protected], users cant login. I get the following error.

error was: Server Error: 429 - for GET /_readonlyrest/metadata/current_user [parent] Data too large, data for [<http_request>] would be [6166482944/5.7gb], which is larger than the limit of [6120328396/5.6gb], real usage: [6166482944/5.7gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, accounting=0/0b]

Any ideas ?

Wow! How many (LDAP?) groups are associated to the user?
Any stack trace available in ES logs?
Does it do this for all the users? Or only some?

Not sure… The"search_user_base_DN" is pretty big. Could that be a the issue?

No, theres no errors in ES.

All users, i think.

The issue is that the AD has a pretty bad user design. Users are spread across below OU’s.

UserGroup1=OU,DC=example,DC=org
UserGroup2=OU,DC=example,DC=org
UserGroup3=OU,DC=example,DC=org

So the search_user_base_DN has to be DC=example,DC=org which is the whole AD.
Does the search_user_base_DN setting take multiple values ?

@sscarduzio Any ideas ?

are you able to show us the doc (id=1) from .readonlyrest index?

I removed the access control rules. Didn’t think they were necessary.

{
  "_index" : ".readonlyrest",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 9,
  "_seq_no" : 8,
  "_primary_term" : 5,
  "found" : true,
  "_source" : {
    "settings" : """readonlyrest:
    enable: true
    #optional
    response_if_req_forbidden: Sorry, your request is forbidden.

    access_control_rules:
    

    ## Removed rules ##


    ldaps:
    
    - name: ldap1
      host: "<ip-address>"
      port: 389                                                 # optional, default 389
      ssl_enabled: false                                        # optional, default true
      ssl_trust_all_certs: true                                 # optional, default false
      bind_dn: "cn=f_elasticsearch_ldap_sync,OU=Elastic,OU=Funktioner,OU=Standard,DC=example,DC=org"
      bind_password: "<password>"                                 # optional, skip for anonymous bind
      search_user_base_DN: "DC=example,DC=org"
      user_id_attribute: "sAMAccountName"
      unique_member_attribute: "Member"
      search_groups_base_DN: "DC=example,DC=org"
      connection_pool_size: 10                                  # optional, default 30
      connection_timeout_in_sec: 10                             # optional, default 1
      request_timeout_in_sec: 10                                # optional, default 1
      cache_ttl_in_sec: 60                                      # optional, default 0 - cache disabled
"""
  }
}

you know, this information can be helpful. I wonder how many blocks you have. Maybe your configuration is pretty big (many blocks and rules)?

At the moment I have no starting point

Sure… I serialized the data a bit… Please let me know if you need anything else.

- name: <#######> - RW ALL
  type: allow
  kibana_access: admin
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  verbosity: error # don't log successful request


- name: <#######> - RO Limited
  type: allow
  kibana_access: ro
  ldap_auth:
     name: "ldap1"
     groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  verbosity: info
  
- name: <#######> - RO Limited
  type: allow
  kibana_access: ro
  ldap_auth:
     name: "ldap1"
     groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>-*"]
  verbosity: info
  
- name: <#######> - Restricted
  type: allow
  kibana_access: ro
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>-*", "<#######>-*"]
  verbosity: info

- name: <#######> - Restricted
  type: allow
  kibana_access: ro
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>*"]
  verbosity: info
  
- name: <#######> - RW
  type: allow
  kibana_access: rw
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>*"]
  verbosity: info
  
- name: <#######> - Restricted
  type: allow
  kibana_access: ro
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>*", "<#######>*"]
  verbosity: info
  
- name: <#######> - Restricted
  type: allow
  kibana_access: ro
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>-*"]
  verbosity: info
  
- name: <#######> - Restricted
  type: allow
  kibana_access: ro
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
  indices: [".kibana", "welcome*", "<#######>-*"]
  verbosity: info

- name: "::KIBANA-SRV::"
  auth_key: kibana:<#######>
  verbosity: error # don't log successful request
  
- name: Accept all requests from localhost
  hosts: ["<#######>", "<#######>", "<#######>", "<#######>"]
  ldap_auth:
      name: "ldap1"
      groups: ["<#######>"]
  verbosity: error # don't log successful request

- name: Accept all requests from logstash and <#######>
  hosts: ["<#######>", "<#######>"]
  verbosity: error # don't log successful request

and one more thing: do you see any stacktrace in ES logs?

Not much… I found this though.

[2020-09-23T19:41:49,967][INFO ][o.e.m.j.JvmGcMonitorService] [hostname.org] [gc][old][624751][1] duration [8.1s], collections [1]/[8.7s], total [8.1s]/[8.1s], memory [5.9gb]->[5.2gb]/[6gb], all_pools {[young] [6mb]->[0b]/[0b]}{[old] [5.9gb]->[5.2gb]/[6gb]}{[survivor] [0b]->[0b]/[0b]}
[2020-09-23T19:41:49,969][WARN ][o.e.m.j.JvmGcMonitorService] [hostname.org] [gc][624751] overhead, spent [8.5s] collecting in the last [8.7s]

Maybe it’s a memory issue ? Heap is set to 6GB in ES jvm.options.

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms6g
-Xmx6g

Could you please enable debug logs (https://github.com/beshu-tech/readonlyrest-docs/blob/master/elasticsearch.md#scenario-you-cant-understand-why-your-requests-are-being-forbidden-by-readonlyrest-or-viceversa) and show me logged LDAP responses?

Sorry the long wait. The error is not persistent.
The error just occurred again, I’ve enabled debug logs, restarted ES and now it running fine again.
Now I have to wait for the error to occur again.