Hi… After i updated Kibana to 7.8.1-1 and readonlyrest_kbn@1.21.0, users cant login. I get the following error.
error was: Server Error: 429 - for GET /_readonlyrest/metadata/current_user [parent] Data too large, data for [<http_request>] would be [6166482944/5.7gb], which is larger than the limit of [6120328396/5.6gb], real usage: [6166482944/5.7gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, accounting=0/0b]
Any ideas ?
Wow! How many (LDAP?) groups are associated to the user?
Any stack trace available in ES logs?
Does it do this for all the users? Or only some?
Not sure… The"search_user_base_DN" is pretty big. Could that be a the issue?
No, theres no errors in ES.
All users, i think.
The issue is that the AD has a pretty bad user design. Users are spread across below OU’s.
UserGroup1=OU,DC=example,DC=org
UserGroup2=OU,DC=example,DC=org
UserGroup3=OU,DC=example,DC=org
So the search_user_base_DN has to be DC=example,DC=org which is the whole AD.
Does the search_user_base_DN setting take multiple values ?
are you able to show us the doc (id=1) from .readonlyrest index?
I removed the access control rules. Didn’t think they were necessary.
{
"_index" : ".readonlyrest",
"_type" : "_doc",
"_id" : "1",
"_version" : 9,
"_seq_no" : 8,
"_primary_term" : 5,
"found" : true,
"_source" : {
"settings" : """readonlyrest:
enable: true
#optional
response_if_req_forbidden: Sorry, your request is forbidden.
access_control_rules:
## Removed rules ##
ldaps:
- name: ldap1
host: "<ip-address>"
port: 389 # optional, default 389
ssl_enabled: false # optional, default true
ssl_trust_all_certs: true # optional, default false
bind_dn: "cn=f_elasticsearch_ldap_sync,OU=Elastic,OU=Funktioner,OU=Standard,DC=example,DC=org"
bind_password: "<password>" # optional, skip for anonymous bind
search_user_base_DN: "DC=example,DC=org"
user_id_attribute: "sAMAccountName"
unique_member_attribute: "Member"
search_groups_base_DN: "DC=example,DC=org"
connection_pool_size: 10 # optional, default 30
connection_timeout_in_sec: 10 # optional, default 1
request_timeout_in_sec: 10 # optional, default 1
cache_ttl_in_sec: 60 # optional, default 0 - cache disabled
"""
}
}you know, this information can be helpful. I wonder how many blocks you have. Maybe your configuration is pretty big (many blocks and rules)?
At the moment I have no starting point
Sure… I serialized the data a bit… Please let me know if you need anything else.
- name: <#######> - RW ALL
type: allow
kibana_access: admin
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
verbosity: error # don't log successful request
- name: <#######> - RO Limited
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
verbosity: info
- name: <#######> - RO Limited
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>-*"]
verbosity: info
- name: <#######> - Restricted
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>-*", "<#######>-*"]
verbosity: info
- name: <#######> - Restricted
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>*"]
verbosity: info
- name: <#######> - RW
type: allow
kibana_access: rw
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>*"]
verbosity: info
- name: <#######> - Restricted
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>*", "<#######>*"]
verbosity: info
- name: <#######> - Restricted
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>-*"]
verbosity: info
- name: <#######> - Restricted
type: allow
kibana_access: ro
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
indices: [".kibana", "welcome*", "<#######>-*"]
verbosity: info
- name: "::KIBANA-SRV::"
auth_key: kibana:<#######>
verbosity: error # don't log successful request
- name: Accept all requests from localhost
hosts: ["<#######>", "<#######>", "<#######>", "<#######>"]
ldap_auth:
name: "ldap1"
groups: ["<#######>"]
verbosity: error # don't log successful request
- name: Accept all requests from logstash and <#######>
hosts: ["<#######>", "<#######>"]
verbosity: error # don't log successful requestand one more thing: do you see any stacktrace in ES logs?
Not much… I found this though.
[2020-09-23T19:41:49,967][INFO ][o.e.m.j.JvmGcMonitorService] [hostname.org] [gc][old][624751][1] duration [8.1s], collections [1]/[8.7s], total [8.1s]/[8.1s], memory [5.9gb]->[5.2gb]/[6gb], all_pools {[young] [6mb]->[0b]/[0b]}{[old] [5.9gb]->[5.2gb]/[6gb]}{[survivor] [0b]->[0b]/[0b]}
[2020-09-23T19:41:49,969][WARN ][o.e.m.j.JvmGcMonitorService] [hostname.org] [gc][624751] overhead, spent [8.5s] collecting in the last [8.7s]
Maybe it’s a memory issue ? Heap is set to 6GB in ES jvm.options.
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms6g
-Xmx6gCould you please enable debug logs (readonlyrest-docs/elasticsearch.md at master · beshu-tech/readonlyrest-docs · GitHub) and show me logged LDAP responses?
Sorry the long wait. The error is not persistent.
The error just occurred again, I’ve enabled debug logs, restarted ES and now it running fine again.
Now I have to wait for the error to occur again.