[SUPPORT|kbn_ent] ldap and parentheses

driveirk · December 29, 2023, 4:38pm

Support request

ROR Version: Enterprise 1.53.0_es7.15.1

Kibana Version: 7.15.1

Elasticsearch Version: 7.15.1

Steps to reproduce the issue
Create an ldap group using a parentheses.
For example my-groups (test)

Actual Result:
Forbiden

[2023-12-28T18:54:12,183][ERROR][tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapNestedGroupsService] [server01] LDAP getting groups of [my-groups (test)] group returned error; Unable to parse string '(&(cn=*)(member=CN=my-groups (test),OU=aaa,OU=Groups,OU=company,DC=office,DC=company,DC=com))' as an LDAP filter because it contains an unexpected opening parenthesis at position 62.
[2023-12-28T18:54:12,183][ERROR][tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapAuthorizationService] [server01] LDAP getting user groups returned error; Unable to parse string '(&(cn=*)(member=CN=my-groups (test),OU=aaa,OU=Groups,OU=company,DC=office,DC=company,DC=com))' as an LDAP filter because it contains an unexpected opening parenthesis at position 62.

and next error

Basic Kibana and regular indices access for all Office users: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the Open state, attempting to close in 9217 millis

All users are loggouted.

Expected result:
Minimally, only the user who is in the problem group is not allowed in and there are no problems with LDAP.
It would be better if you allow the use of parentheses and other permitted characters in LDAP.

{“customer_id”: “6c4a385b-2ae8-4f02-a9cd-ef24addfb5b3”, “subscription_id”: “32d4073f-dc2f-4056-a868-842727c637cd”}

coutoPL · December 29, 2023, 7:46pm

Thanks for the report. It seems that parentheses are not escaped correctly. We will fix it before releasing ROR 1.55.0.

driveirk · January 2, 2024, 3:42pm

Another example where the group is disabled.

Allow Kibana and regular indices access for AD users in disable group: ldap_authorization rule matching got an error Unable to parse string '(&(cn=*)(member=CN=disable group,OU=Groups (Disabled),OU=company,DC=office,DC=company,DC=com))' as an LDAP filter because it contains an unexpected opening parenthesis at position 47.
com.unboundid.ldap.sdk.LDAPException: Unable to parse string '(&(cn=*)(member=CN=disable group,OU=Groups (Disabled),OU=company,DC=office,DC=company,DC=com))' as an LDAP filter because it contains an unexpected opening parenthesis at position 47

driveirk · January 4, 2024, 1:08pm

@coutoPL Please clarify that this bug also affected 1.49.1?
I downgraded to 1.49.1, but the error remained.
Any information on when 1.55.0 will be released?

driveirk · January 4, 2024, 6:36pm

I found the problem. When disable
nested_groups_depth: 3
the error disappears.

coutoPL · January 4, 2024, 10:34pm

ROR 1.55.0 release should be expected within 2 weeks.
But I will send you a pre-build with the fix earlier.

coutoPL · January 10, 2024, 7:20pm

It’s fixed. Here is a pre-build to test: ROR 1.55.0-pre2 for ES 7.15.1

driveirk · January 18, 2024, 5:32pm

The tests were successful, we are waiting for the release of a stable version.

coutoPL · January 18, 2024, 8:28pm

The release will be done within a week or so.

driveirk · September 9, 2024, 6:49pm

Previously, these users prevented everyone from logging in.
Now the problem only affects users who have “()” in ldap groups.

[2024-09-09T08:03:30,987][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server] Allow unrestricted access to ACL-APP-LDAP-GROUP: ldap_authorization rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
	at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
	at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
	at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
	at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
	at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
	at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
	at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
	at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
	at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
	at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
	at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]

After removing the user from the problematic group, the user logged in without errors.

Enterprise 1.55.0_es7.15.1
elasticsearch 7.15.1
kibana 7.15.1
example group name:
AAA-BBB-Cccccc Dddd (Eeeee)

coutoPL · September 11, 2024, 3:45pm

Hi, was it a nested group or a regular (root level) group?

driveirk · September 11, 2024, 4:43pm

the group was in root level

coutoPL · September 11, 2024, 4:52pm

ok, I will check it.
Previously, we fixed the nested group issue, so we may have missed sth with the root-level groups.

coutoPL · September 11, 2024, 6:53pm

Nope, this is not it.
Could you please show me:

ROR LDAP connector configuration
the problematic group and user LDAP entry?

driveirk · September 12, 2024, 12:15pm

ROR LDAP connector configuration

    ldaps:
    - name: "Ldap_name"
      hosts:
        - ldap://server:port
      ha: "FAILOVER"                                             # optional, default "FAILOVER"
      ssl_enabled: false                                            # optional, default true
      ssl_trust_all_certs: true                                     # optional, default false
      bind_dn: "CN=Kibana ,OU=Service Accounts,OU=Company,DC=office,DC=company,DC=com"        # optional, skip for anonymous bind
      bind_password: "pass"                                     # optional, skip for anonymous bind
      search_user_base_DN: "DC=office,DC=company,dc=com"
      search_groups_base_DN: "DC=office,DC=company,DC=com"
      user_id_attribute: "AccountName"                                      # optional, default "uid"
      unique_member_attribute: "member"                             # optional, default "uniqueMember"
      connection_pool_size: 100                                     # optional, default 30
      connection_timeout_in_sec: 30                                 # optional, default 1
      request_timeout_in_sec: 30                                    # optional, default 1
      cache_ttl_in_sec: 60                                          # optional, default 0 - cache disabled
      group_search_filter: "(cn=*)"                                 # optional, default (cn=*)
      group_name_attribute: "cn"                                    # optional, default "cn"
      nested_groups_depth: 3

the problematic group and user LDAP entry?
ldap group ACL-PRN-Armor Printer Users (Malaga)

user LDAP entry
Need login log or username? example username - Nikolay.Shin

coutoPL · September 12, 2024, 7:45pm

Thanks.
Regarding the 2nd point, it’d be great if you could show me the LDAP schema too. I’d like to reproduce it on my side

driveirk · September 13, 2024, 7:56pm

I sent you the LDAP schema via private message.

coutoPL · September 16, 2024, 7:44pm

@driveirk I’ve checked the schema you sent and I prepared a reproducer:

github.com/beshu-tech/ror-sandbox

[RORDEV-1275] LDAP with group with spaces

beshu-tech:master ← beshu-tech:reproduction/RORDEV-1275

opened 07:38PM - 16 Sep 24 UTC

coutoPL

+142 -1

1. `cd ror-demo-cluster` 2. `./run.sh`: ``` _____ _ ____ … _ _____ ______ _____ _______ | __ \ | |/ __ \ | | | __ \| ____|/ ____|__ __| | |__) |___ __ _ __| | | | |_ __ | |_ _| |__) | |__ | (___ | | | _ // _ \/ _| |/ _| | | | | '_ \| | | | | _ /| __| \___ \ | | | | \ \ __/ (_| | (_| | |__| | | | | | |_| | | \ \| |____ ____) | | | |_| \_\___|\__,_|\__,_|\____/|_| |_|_|\__, |_| \_\______|_____/ |_| __/ | Preparing Elasticsearch & Kibana with ROR environment ... ----------------- Enter Elasticsearch version (default: 8.15.1): 7.15.1 Use ES ROR: 1. From API 2. From FILE Your choice: 1 Enter ROR Elasticsearch version (default: 1.60.0): ----------------- Enter Kibana version (default: 7.15.1): Use KBN ROR: 1. From API 2. From FILE Your choice: 1 Enter ROR Kibana version (default: 1.60.0): ``` 3. go to: `http://localhost:15601` and log in as `morgan:user1` 4. Check groups: <img width="685" alt="Screenshot 2024-09-16 at 21 39 25" src="https://github.com/user-attachments/assets/b862c37d-c208-48bb-babf-7505e7d3c497">

IMO, it works as expected. Please take a look. So, the problem is not the LDAP connector IMHO.

Please, add these lines to log4j2.properties to enable LDAP debug logs and try to find log like this:

es-ror-1   | [2024-09-16T19:39:20,282][DEBUG][t.b.r.a.b.d.l.LoggableLdapAuthorizationService$WithoutGroupsFilteringDecorator] [es-ror-single] [7524e71b-c5ea-4958-8c19-16181238ac76-2057563163#3040] LDAP [ldap2] returned for user [morgan] following groups: [(id=AAA-BBB-Cccccc Dddd (Eeeee),name=AAA-BBB-Cccccc Dddd (Eeeee)),(id=group1 (nested),name=group1 (nested)),(id=group3,name=group3)]

As you see in the log, the AAA-BBB-Cccccc Dddd (Eeeee) group was successfully returned to ROR from LDAP. If you see the same line, it means that the LDAP connector works as expected.

And the problem probably lays somewhere in the ACL configuration. Could you please show us the block, that should ALLOW the request? (I want to see the ldap_auth or ldap_authorization rule configuration)

driveirk · September 17, 2024, 8:51am

The problem is in all ACLs, not just the allowed ones.
Full log:

[2024-09-10T06:17:17,960][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Access delete doc: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
[2024-09-10T06:17:17,965][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Admin users: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
[2024-09-10T06:17:17,969][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Partial access: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
[2024-09-10T06:17:17,973][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Partial access add func: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
[2024-09-10T06:17:17,977][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Partial forbid user: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
[2024-09-10T06:17:17,980][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Elevated Users: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]
[2024-09-10T06:17:17,984][ERROR][tech.beshu.ror.accesscontrol.blocks.Block] [server1] Basic Kibana: ldap_authentication rule matching got an error Rejected because the CircuitBreaker is in the HalfOpen state
monix.execution.exceptions.ExecutionRejectedException: Rejected because the CircuitBreaker is in the HalfOpen state
        at monix.execution.exceptions.ExecutionRejectedException$.apply(ExecutionRejectedException.scala:40) ~[?:?]
        at monix.catnap.CircuitBreaker.$anonfun$unsafeProtect$3(CircuitBreaker.scala:431) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at timeout @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testLdapBinding(UnboundidLdapConnectionPoolProvider.scala:227) ~[?:?]
        at map @ tech.beshu.ror.utils.TaskOps$.andThen$extension(TaskOps.scala:30) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$getFromCacheOrRunAction$2(Cachable.scala:68) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.getFromCacheOrRunAction(Cachable.scala:67) ~[?:?]
        at asyncBoundary @ tech.beshu.ror.accesscontrol.blocks.definitions.ldap.implementations.UnboundidLdapConnectionPoolProvider$.testBindingForAllHosts(UnboundidLdapConnectionPoolProvider.scala:171) ~[?:?]
        at runSyncUnsafe @ tech.beshu.ror.buildinfo.BuildInfoReader$.$anonfun$create$1(BuildInfoReader.scala:35) ~[?:?]
        at withPermit @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:60) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.$anonfun$call$1(Cachable.scala:59) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at map @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.semaphoreOf(Cachable.scala:86) ~[?:?]
        at flatMap @ tech.beshu.ror.accesscontrol.utils.CacheableActionWithKeyMapping.call(Cachable.scala:58) ~[?:?]

Here is the ROR config:

    access_control_rules:
    - name: "kibana user"
      auth_key: kibana:pass
      kibana_access: unrestricted

    - name: "Access delete doc"
      users: ["user.delete"]
      ldap_authentication:
        name: "ldap"
      actions: ["indices:data/write/delete"]

    - name: "Admin users"
      type: allow
      ldap_authentication:
        name: "ldap"
      ldap_authorization:
        name: "ldap"
        groups: ["Admin"]
      kibana_access: unrestricted

    - name: "Partial access"
      indices: ["*kibana*", "*:*-logs-aaa*", "*:*-logs-aab*", "*:*-logs-bbb*", "*:*logs-ccc*", ..., "*:*-logs-zzz*"]
      kibana_access: rw
      ldap_authentication:
        name: "ldap"
      ldap_authorization:
        name: "ldap"
        groups: ["partial"]
      kibana_hide_apps: ["timelion", "readonlyrest_kbn", "Timelion", "Canvas", "Maps", "Logs", "Uptime", "Enterprise Search", "Machine Learning", "Metrics", "User Experience", "Dev Tools", "Integrations", "Fleet", "Osquery", "Stack Management", "Observability", "ROR Manage Kibana"] 

    - name: "Partial access add func"
      indices: ["*kibana*", "*:*-logs-aaa*", "*:*-logs-aab*", "*:*-logs-bbb*", "*:*logs-ccc*", ..., "*:*-logs-zzz*"]
      kibana_access: rw
      ldap_authentication:
        name: "ldap"
      ldap_authorization:
        name: "ldap"
        groups_and: ["partial", "Elevated Users"]
      kibana_hide_apps: ["readonlyrest_kbn", "Timelion", "Canvas", "Maps", "Home", "Logs", "Uptime", "Enterprise Search", "Machine Learning", "Metrics", "User Experience", "Management"]

    - name: "Partial forbid user"
      type: forbid
      ldap_authentication:
        name: "ldap"
      ldap_authorization:
        name: "ldap"
        groups: ["partial"]

    - name: "Elevated Users"
      type: allow
      ldap_authentication:
        name: "ldap"
      ldap_authorization:
        name: "ldap"
        groups: ["Elevated Users"]
      kibana_access: rw
      kibana_hide_apps: ["readonlyrest_kbn", "Timelion", "Canvas", "Maps", "Home", "Logs", "Uptime", "Enterprise Search", "Machine Learning", "Metrics", "User Experience", "Management"]

    - name: "Basic Kibana"
      type: allow
      ldap_authentication:
        name: "ldap"
      ldap_authorization:
        name: "ldap"
        groups: ["ACL-APP-Kibana-Users"]
      kibana_access: rw
      kibana_hide_apps: ["timelion", "readonlyrest_kbn", "Timelion", "Canvas", "Maps", "Logs", "Uptime", "Enterprise Search", "Machine Learning", "Metrics", "User Experience", "Dev Tools", "Integrations", "Fleet", "Osquery", "Stack Management", "Observability", "ROR Manage Kibana"] 

    - name: "LOCALHOST-only access"
      hosts: ["127.0.0.1"]
      headers_and: ["~x-forwarded-for:*", "~X-Passed-Nginx:*"]

I found that the problem is reproduced even with a system account.
login: use.systems
has only one group: ACL-APP-Kibana-Users
There are no nested groups.

But there are successful logins under this user.

Chronology:
Successful authorization of another user (excludes problems with ldap or interference of other users)
error of user use.systems
error of another system user rcv.systems (it has 2 groups and no nested groups)
After that, there are no errors again.

There is no () in the group name.

coutoPL · September 17, 2024, 6:16pm

The errors we see in the provided logs are related to the LDAP connector circuit breaker. The circuit breaker is always enabled and the default settings are:

circuit_breaker:                                        
  max_retries: 10                                           
  reset_duration: 10s

When your LDAP server starts to fail, after 10 failed attempts, the circuit breaker opens and starts to fail fast (without hitting your server). After 10 sec, it enters the half-open state (we see this in the log) and calls the LDAP server - if it succeeds, the circuit breaker is closed again, if it fails, it opens and waits another 10sec for the next check if the LDAP server is healthy.

When you see these logs, it means your LDAP server struggles.
You can enable debug logs for the LDAP connector (I showed it above) to see how many requests go to the LDAP server and check if the caching settings are enough to handle your traffic.

If you want to see how the LDAP connector behaves “without” circuit breaker, please configure sth like this:

circuit_breaker:                                        
  max_retries: 10000000 # 10 mln requests in a row should fail to opens the circuit breaker                                           
  reset_duration: 10s

You should be able to see the real reason why it fails (the underlying error from the LDAP connector).