LDAP connection, authentication and Index wise authorisation

Hi @sscarduzio,
I will first clarify our requirements.

  1. Users should be authenticated through LDAP.
  2. For a particular group of users there should be accessibility of particular index not to all indexes.
    (Example: Suppose we have “user1” and “user2” in a group called “Technolgy” and I have two indexes “index1” and “index2” But I want to give access for only “index1”, "index2 should not be accessible after login by “user1” and “user2” .)
  3. At the end we need security for index data means after login by “user1” and “user2” they can see only “index1” data also they can not visualise data of other indexes they can visualise only “index1” data.
  4. Note: I have given you already my LDAP related data for connectivity that should be work. Because same I had used in x-pack and that was working fine. So please provide proper solution on it.

HI @ajit,

Look at this example: the following configuration will grant:

  • user1 to access Kibana, and view data from index “index_user1” only.
  • user2 to access Kibana, and view data from index “index_user2” only.

Provided that LDAP contains user1, user2 belonging to the LDAP group “Technology”.

  readonlyrest:

  access_control_rules

  - name: "Technology users"
    kibana_access: "rw"
    groups: ["Technology"]
    indices: [".kibana", "index_@{user}"]

  users:
  - username: user1
    groups: ["Technology"]
    ldap_authentication: "ldap1"

  - username: user2
    groups: ["Technology"]
    ldap_authentication: "ldap1"

  ldaps:
  - name: ldap1
    .....

Hi Simone Scarduzio,

In this configuration there is no LDAP Configuration and SSL Block.
let me explain our requirements-

  1. LDAP Authentication for users but there is no any LDAP configuration in the snippet provided by you.
    we have used our LDAP configuration but that is not working.

Below is our Configuration Code-

readonlyrest:

prompt_for_basic_auth: false

ssl:
enable: true
keystore_file: “/opt/ElasticSearchKibana/elasticsearch-6.2.4/config/keystore.jks”
keystore_pass: readonlyrest
key_pass: readonlyrest
key_alias: elk01 #This is needed only when the keystore has multiple entries

audit_collector: true

access_control_rules:

  • name: “Technology users”
    kibana_access: “rw”
    groups: [“Technology”]
    indices: [“.kibana”]

users:

  • username: c-shubhamg
    groups: [“Technology”]
    ldap_authentication: “ldap1”

  • username: c-ajitb
    groups: [“Technology”]
    ldap_authentication: “ldap1”

ldaps:

  • name: ldap1
    host: “ad.example.com
    port: 123
    ssl_enabled: false
    ssl_trust_all_certs: true
    bind_dn: “CN=c-shubhamg,OU=Technology,OU=Corporate Technology,OU=Corporate Group,OU=Mumbai Crisil House,DC=ad,DC=crisil,DC=com”
    bind_password: “Abc@2019”
    search_user_base_DN: “OU=Technology,dc=crisil,dc=com”
    search_groups_base_DN: “OU=Technology,dc=crisil,dc=com”

Note:we have already tried more than 100 scenarios related to LDAP Connectivity and all is not working.
kindly provide exact solution for LDAP Authentication and index Authorization with SSL Block.

Hi @sscarduzio,
We have used your configuration but after starting kibana and elasticsearch getting authorisation exception.
Please test solution with your LDAP credentials and groups then send me tested code with all blocks (example: SSL block, ACL blocks, Users, and ldaps.)
We have tested lots of scenario but not getting success. Please first concentrate on LDAP connectivity. We should login into kibana using LDAP user credentials. Then we will proceed.

elasticsearch.yml

Append the following line:

http.type: ssl_netty4

kibana.yml

append the following:

elasticsearch.username: "kibana"
elasticsearch.password: "kibana"

readonlyrest.yml

readonlyrest:

  prompt_for_basic_auth: false
    
  ssl:
    enable: true
    keystore_file: "keystore.jks" # put the JKS in the same dir with readonlyrest.yml
    keystore_pass: readonlyrest
    key_pass: readonlyrest
    key_alias: elk01    #This is needed only when the keystore has multiple entries

  audit_collector: true

  access_control_rules:

  - name: "kibana server"
    auth_key: kibana:kibana

  - name: "Technology users"
    kibana_access: "rw"
    groups: ["Technology"]
    #indices: [".kibana"]

  users:
  - username: c-shubhamg
    groups: ["Technology"]
    ldap_authentication: "ldap1"

  - username: c-ajitb
    groups: ["Technology"]
    ldap_authentication: "ldap1"

  ldaps:
  - name: ldap1
    host: "ad.crisil.com"
    port: 389
    ssl_enabled: false
    ssl_trust_all_certs: true
    bind_dn: "CN=c-shubhamg,OU=Technology,OU=Corporate Technology,OU=Corporate Group,OU=Mumbai Crisil House,DC=ad,DC=crisil,DC=com"
    bind_password: "May@2018"
    search_user_base_DN: "OU=Technology,dc=crisil,dc=com"
    search_groups_base_DN: "OU=Technology,dc=crisil,dc=com"

Hi @sscarduzio,

The Configuration is given by you is not working.
As we have told you,we have tested all these scenarios but did’t get success.

we have already added

"elasticsearch.username: “kibana”
"elasticsearch.password: “kibana” in kibana.yml

we have already added “http.type: ssl_netty4” in elasticsearch.yml

Below is my readonlyrest.yml

readonlyrest:
prompt_for_basic_auth: false

ssl:
  enable: true
  keystore_file: "/opt/ElasticSearchKibana/elasticsearch-6.2.4/config/keystore.jks"
  keystore_pass: readonlyrest
  key_pass: readonlyrest
  key_alias: elk01    #This is needed only when the keystore has multiple entries    

audit_collector: true

access_control_rules:

- name: "kibana server"
  auth_key: kibana:kibana

- name: "admin"
  auth_key: admin:admin

- name: "Technology users"
  kibana_access: "rw"
  groups: ["Technology"]
 #indices: [".kibana"]

users:
- username: c-shubhamg
  groups: ["Technology"]
  ldap_authentication: "ldap1"

- username: c-ajitb
  groups: ["Technology"]
  ldap_authentication: "ldap1"

- username: c-akhilesht
  groups: ["Technology"]
  ldap_authentication: "ldap1"


ldaps:

- name: ldap1
  host: "ad.example.com"
  port: 123
  ssl_enabled: false
  ssl_trust_all_certs: true
  bind_dn: "CN=c-shubhamg,OU=Technology,OU=Corporate Technology,OU=Corporate Group,OU=Mumbai Crisil House,DC=ad,DC=crisil,DC=com"
  bind_password: "Abc@2014"
  search_user_base_DN: "OU=Technology,dc=crisil,dc=com"
  search_groups_base_DN: "OU=Technology,dc=crisil,dc=com"

so kindly provide us a well tested and working configuration.

Note: We strongly concentrate on LDAP connectivity,then we will proceed.Users must be login into kibana using LDAP credentials.

Dear @Akhilesh,

I have spun up a test server for you with kibana, elasticsearch and LDAP installed. I am currently successfully using the following configuration:

elasticsearch.yml

appended line:

http.type: ssl_netty4

readonlyrest.yml

readonlyrest:

    ssl:
      enable: true
      keystore_file: "keystore.jks"
      keystore_pass: readonlyrest
      key_pass: readonlyrest

    audit_collector: true

    access_control_rules:

    # MACHINES ##################
    - name: "::LOGSTASH::"
      auth_key: logstash:logstash
      actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["logstash-*"]

    - name: "::KIBANA-SRV::"
      auth_key: kibana:kibana
      verbosity: error

    # GROUPS (LOCAL MULTITENANCY) #####

    - name: "::PERSONAL_GRP::"
      groups: ["Personal"]
      kibana_access: rw
      kibana_hide_apps: ["readonlyrest_kbn", "timelion"]
      kibana_index: ".kibana_@{user}"
  
    - name: "::ADMIN_GRP::"
      groups: ["ROR (admin)"]
      kibana_access: admin
      kibana_index: .kibana

    - name: "::Infosec::"
      groups: ["Infosec"]
      kibana_access: rw
      kibana_hide_apps: ["readonlyrest_kbn", "timelion"]
      kibana_index: ".kibana_infosec"

    - name: "Testgroup access"
      groups: ["testgroup"]
      kibana_access: rw
      kibana_hide_apps: ["readonlyrest_kbn", "timelion"]
      kibana_index: ".kibana_testgroup"
      
    # GROUPS (LDAP) ##############  
    - name: "Technology users"
      kibana_access: "admin"
      groups: ["group1"]

    # USERS TO GROUPS (LOCAL MULTITENANCY) #######
    users:
    - username: admin
      auth_key: admin:dev
      groups: ["ROR (admin)", "Infosec"]    

    - username: simone
      auth_key: simone:dev
      groups: ["ROR (admin)", "Personal", "Infosec"]

    - username: testuser
      auth_key: testuser:testuser
      groups: ["testgroup"]
      
    # USERS TO GROUPS (LDAP) ######## 
    - username: cartman
      groups: ["group1"]
      ldap_authentication: "ldap1" 
      
    - username: bong
      groups: ["group1"]
      ldap_authentication: "ldap1"
      
    
    ldaps:
    ######### LDAP1 SERVER CONFIGURATION ########################
    # group1: cartman, bong
    # group2: morgan
    # group3: morgan, cartman, bong
    #############################################################
    - name: ldap1
      host: localhost
      port: 32779                                                 # default 389
      ssl_enabled: false                                        # default true
      ssl_trust_all_certs: true                                 # default false
      bind_dn: "cn=admin,dc=example,dc=com"                     # skip for anonymous bind
      bind_password: "password"                                 # skip for anonymous bind
      search_user_base_DN: "ou=People,dc=example,dc=com"
      search_groups_base_DN: "ou=Groups,dc=example,dc=com"
      user_id_attribute: "uid"                                  # default "uid"
      unique_member_attribute: "uniqueMember"                   # default "uniqueMember"
      connection_pool_size: 10                                  # default 30
      connection_timeout_in_sec: 10                             # default 1
      request_timeout_in_sec: 10                                # default 1
      cache_ttl_in_sec: 60                                      # default 0 - cache disabled

kibana.yml

appended:

elasticsearch.url: "https://localhost:9200"
elasticsearch.ssl.verificationMode: "none"
elasticsearch.username: kibana
elasticsearch.password: kibana

I can provide the credentials to my test server.

Hi @sscarduzio,
Still we are not getting success for LDAP Authentiaction,
I have some doubts-

  1. Any Relation between ACL block (Local Multitenancy) and LDAP in the reference of Authentication?
    2.I don’t want to define groups want to directly give access to AD(LDAP) users,is it possible?
    3.Writing roles for AD is complusory?
    4.can we test by removing default forbideen policy?

Hi @Akhilesh,

  1. no
  2. this was not your requirement, you requirement was that you want LDAP users within a certain group to be able to log in.
  3. Not sure what this question means, but since you only want LDAP users belonging to a specific group to log in, then ROR needs to know what group it is.
  4. This does not make sense to me, if we can’t understand what user is logging in, we can’t create a session for them.

Can you document what exactly is going on? I mean you say “not getting success” this provides zero context to me in order to help you.

Why don’t we try to formalise this a little more:

1. What is my high level objective (always cite ES version, ROR version and product name)

-_______________

2. How did I attempt to do it (mention all the settings)

-________________

3. Expected result

-_________________

4. Actual result (a sentence, and extensive debug logs)

-__________________

Hi @sscarduzio,

1. What is my high level objective (always cite ES version, ROR version and product name)

 My high level objective is User Authentication via LDAP,then i will focus on authorization of 
 specific index   for  a specific user.

2. How did I attempt to do it (mention all the settings)

   Kibana.yml

 server.port: 5601
 elasticsearch.username: "kibana"
 elasticsearch.password: "kibana"

 elasticsearch.yml

 bootstrap.system_call_filter: false
 http.type: ssl_netty4
 cluster.name: my-application
 network.host: localhost

readonlyrest.yml

readonlyrest:

ssl:
  enable: true
  keystore_file: "/opt/ElasticSearchKibana/elasticsearch-6.2.4/config/keystore.jks"
  keystore_pass: readonlyrest
  key_pass: readonlyrest
  key_alias: elk01    #This is needed only when the keystore has multiple entries

audit_collector: true

access_control_rules:

# MACHINES ##################
- name: "::LOGSTASH::"
  auth_key: logstash:logstash
  actions: ["indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
  indices: ["logstash-*"]

- name: "::KIBANA-SRV::"
  auth_key: kibana:kibana
  verbosity: error

- name: "::admin::"
  auth_key: admin:admin
   
# GROUPS (LDAP) ##############  
- name: "Technology users"
  kibana_access: admin
  groups: ["Technology"]
  kibana_index: .kibana

  
# USERS TO GROUPS (LDAP) ######## 
users:
- username: c-shubhamg
  groups: ["Technology"]
  ldap_authentication: "ldap1" 
  
- username: c-ajitb
  groups: ["Technology"]
  ldap_authentication: "ldap1"

- username: c-akhilesht
  groups: ["Technology"]
  ldap_authentication: "ldap1"
  

ldaps:
######### LDAP1 SERVER CONFIGURATION ########################
# group1: cartman, bong
# group2: morgan
# group3: morgan, cartman, bong
#############################################################
- name: ldap1
  host: "example.com"
  port: 389                                                 # default 389
  ssl_enabled: false                                        # default true
  ssl_trust_all_certs: true                                 # default false
  bind_dn: "CN=c-ShubhamG,OU=Technology,OU=Corporate Technology,OU=Corporate Group,OU=Mumbai Crisil House,DC=ad,DC=crisil,DC=com"                     # skip for anonymous bind
  bind_password: "Abc@2017"                                 # skip for anonymous bind
  search_user_base_DN: "dc=ad,dc=crisil,dc=com"
  search_groups_base_DN: "dc=ad,dc=crisil,dc=com"
  user_id_attribute: "uid"                                  # default "uid"
  unique_member_attribute: "uniqueMember"                   # default "uniqueMember"
  connection_pool_size: 10                                  # default 30
  connection_timeout_in_sec: 10                             # default 1
  request_timeout_in_sec: 10                                # default 1
  cache_ttl_in_sec: 60

3. Expected result

      Users must be authenticate through LDAP and specific index should authorized through kIbana for a 
      specific user. 

4. Actual result (a sentence, and extensive debug logs)

   After setting all configuration when we are trying to login with our LDAP user credentials,we get an error 
   "⚠️ Could not login: Unauthorized (401)"  in the login screen.

@Akhilesh the Elasticsearch debug logs are missing, and are the most important.

Hi @sscarduzio,

here is the log for two LDAP users-

[2018-06-11T20:05:00,825][INFO ][t.b.r.a.ACL ] e[35mFORBIDDEN by default req={ ID:2007089208-377515351#368, TYP:NodesInfoRequest, CGR:N/A,
USR:c-ajitb(?), BRS:false, KDX:null, ACT:cluster:monitor/nodes/info, OA:172.21.153.176, DA:0.0.0.0, IDX:<N/A>, MET:GET, PTH:/_nodes/_local, CNT:<N/A>,
HDR:{authorization=Basic Yy1haml0YjpwYXNzQDEyMzQ=, Connection=close, Authorization=, content-length=0, Host=mumchelk01:9200}, HIS:[::LOGSTASH::->[auth_key->false]],
[::KIBANA-SRV::->[auth_key->false]], [::admin::->[auth_key->false]], [Technology users->[groups->false]] } e[0m

[2018-06-11T20:05:35,884][INFO ][t.b.r.a.ACL ] e[35mFORBIDDEN by default req={ ID:929035787-528569185#452, TYP:NodesInfoRequest, CGR:N/A,
USR:c-ShubhamG(?), BRS:false, KDX:null, ACT:cluster:monitor/nodes/info, OA:172.21.153.176, DA:0.0.0.0, IDX:<N/A>, MET:GET, PTH:/_nodes/_local, CNT:<N/A>,
HDR:{authorization=Basic Yy1TaHViaGFtRzpNYXlAMjAxOA==, Connection=close, Authorization=, content-length=0, Host=mumchelk01:9200}, HIS:[::LOGSTASH::->[auth_key->false]],
[::KIBANA-SRV::->[auth_key->false]], [::admin::->[auth_key->false]], [Technology users->[groups->false]] } e[0m

  1. Please set Elasticsearch rootLogger in debug mode from log4j2.properties
  2. Show us the DEBUG lines inherent to the LDAP query and response
  3. I can see you are mixing up capitalization: I read USR:c-ShubhamG in the logs and c-shubhamg in the settings. Which one is it in LDAP?

Hi @sscarduzio,
I am running elasticsearch in debug mode. But getting exception .kibana index not found. Is this the reason LDAP connectivity not working.?

the index not found is unrelated with LDAP

Hi @sscarduzio,

Thanks for you valuable reply of each and every query.
we will get “unauthorized exception” when we are trying to login with our LDAP users.
now my doubt is,can LDAP users login without authorization of any index?

When there is no indices rule, it means there is no limitations about indices. So, that is no problem. And once again, .kibana index not found is not an authorization issue coming from ROR.

Do you have more LDAP related debug logs after that exception?

Hi @sscarduzio,

Actually we are getting “:warning:️ Could not login: Unauthorized (401)” in the login screen and
“path: /.kibana/_mappings, params: {index=.kibana} org.elasticsearch.index.IndexNotFoundException: no such index” in the log file and console.
we are stuck in login with LDAP users,please give us an exact solution.

I have just mailed you the log file,please see for reference.

Hi @sscarduzio,

My Manager Mr. Amol Kulkarni will mail you the log File for you reference.
please see and suggest us to finish this LDAP related isssue.

Hi sscarduzio,

In elasticsearch.yml,I mentioned only few required things:

1. bootstrap.system_call_filter: false
2. http.type: ssl_netty4
3. cluster.name: my-application
4. network.host: SOMETHINGELK01

I am confirm that there is no below lines-

1. action.auto_create_index:
2. index.mapper.dynamic:

in my elasticsearch.yml