Assistance for ELK noob trying readonlyrest

1

Hi -

I’m working on deploying an ELK cluster with filebeat/logstash/elasticsearch/kibana, and have the readonlyrest plugin working on the Elasticsearch side and am evaluating the enterprise Kibana plugin, but have some questions. I’ve done a bit of searching on the forum but haven’t found anything that directly addresses my questions.

Quick overview of our current setup:

  • Filebeat (6.6.0) on clients (mix of CentOS and RedHat 6 and 7) using syslog listener and app logfile watching to pass on to Logstash nodes (working)
  • Logstash node(s) (6.6.0) configured with necessary grok and such, passing data to ES cluster over encrypted port 9200 (working)
  • Elasticsearch cluster (4 hosts in current test cluster) (6.5.4, version dictated by readonlyrest version support) using readonlyrest to enable password auth for the logstash boxes.
  • Kibana (6.5.4, version dictated by elasticsearch) - originally had been using httpd reverse proxy but had a number of odd issues with that so am currently trying to deploy RoR Kibana plugin listening directly.
  • All of the logstash/elasticsearch/kibana hosts are CentOS 7, current version and fully patched.

We use puppet enterprise to deploy the configuration, so I’m working with the appropriate elastic modules for each of the above. I looked and didn’t see any community modules for readonlyrest so have been configuring that with fairly simple puppet configurations. Can supply that info if needed/interested.

I’ve been trying to follow the instructions found in the kibana.md on github, but don’t fully follow.

Here is the kibana.yml as it stands right now:

---
elasticsearch.password: derpderpderp
elasticsearch.requestTimeout: '60000'
elasticsearch.ssl.certificateAuthorities: "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
elasticsearch.url: https://elasticsearch1-0:9200
elasticsearch.username: elasticsearch
kibana.index: ".bz1kibana"
logging.dest: "/var/log/elk/kibana.log"
server.host: derp.internal.maas360.com
server.port: '8888'
server.ssl.enabled: false
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: true
xpack.security.enabled: false
xpack.watcher.enabled: false

readonlyrest.yml - only installed on the elasticsearch hosts, but the same file is installed and configured on all four hosts.

# yamllint disable rule:line-length
readonlyrest:
  enable: true
  ssl:
    enable: true
    keystore_file: "elasticsearch.jks"
    keystore_pass: "derpderpderp"
    key_pass: "derpderpderp"
  access_control_rules:
    - name: "Allow localhost"
      hosts: [127.0.0.1]
    - name: "::ADMIN::"
      type: allow
      auth_key: "admin:derp"
      # KIBANA ADMIN ACCESS NEEDED TO EDIT SECURITY SETTINGS IN ROR KIBANA APP!
      kibana_access: admin
    - name: "::LOGSTASH::"
      type: allow
      auth_key: "elasticsearch:derpderpderp"
      verbosity: error
    - name: "readonly"
      type: allow
      auth_key: "readonly:ylnodear"
      kibana_access: ro
      indices: [".kibana", ".kibana-devnull", "logstash-*"]
      kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management"]
    - name: "ldap for the rest"
      ldap_authentication:
        name: "ldap1"
        groups: ["ipausers"]
        type: allow
        indices: ["*-*"]
        verbosity: error
  ldaps:
    - name: ldap1
      host: "ldapserver.example.com"
      port: 636
      ssl_enabled: true
      ssl_trust_all_certs: true
      search_user_base_DN: "cn=accounts,dc=derp,dc=localdomain"
      search_groups_base_DN: "cn=accounts,dc=derp,dc=localdomain"
      user_id_attribute: "uid"
      unique_member_attribute: "member"
      connection_pool_size: 10
      connection_timeout_in_sec: 30
      request_timeout_in_sec: 30
      cache_ttl_in_sec: 60
      group_search_filter: "(objectclass=top)"
      group_name_attribute: "cn"

The readonlyrest.yml is modified from the version that I inherited which is being used on our current Elasticsearch 2.x cluster which is in production. That version only specifies the logstash user/pass and the LDAP.

My questions:

  • Initial install of the plugin doesn’t go smoothly, as it hangs up on the “Optimizing and caching browser bundles…” step - have let it run for hours and it doesn’t complete. I have done this step from the installation instructions, but it doesn’t seem to help:
    $ touch optimize/bundles/readonlyrest_kbn.style.css
    Each time I’ve tried to install I wind up cancelling out of the Optimizing step and hoping the plugin was installed properly.

  • readonlyrest.yml only goes on the elasticsearch systems, correct? I don’t need a version of it on the kibana host? There’s no config specific to the kibana plugin on the kibana side beyond what goes in kibana.yml?

  • I’ve been unable to login to Kibana with anything but the logstash password.

  • Is it possible to test the kibana access with simple curl commands to the elasticsearch port 9200? I’ve gotten used to using that and found I had to add type: allow to each entry under access_control_rules in the readonlyrest.yml file to get that access to authenticate properly, but I don’t know if that translates properly. Also, using the above readonlyrest.yml on the elasticsearch boxes I can curl the _cat/indices on port 9200 using the “admin” account but not the “readonly” account, which gets a JSON “forbidden” error.

I think that’s it for now, but am quite sure there will be more questions. Thanks for the assistance.

2

Hi @cmh, welcome to the forum :slight_smile:

Straight to your questions:

  1. This “Optimizing and caching” phase getting stuck is such a pain, even for other Kibana plugins. Make sure that you have enough free memory, and bump up the node max heap:
    export NODE_OPTIONS="--max-old-space-size=4096"
    If it still gets stuck, wait the high CPU activity ceases, and hit CTRL+C.
  1. readonlyrest.yml belongs to our Elasticsearch plugin. It is a file that lives in the same directory of elasticsearch.yml. Instead, our Kibana plugins are configured exclusively via kibana.yml. I amended the kibana plugin documentation to make this extra clear. Thanks for pointing it out!
  1. We need to check what’s wrong using curl directly to ES, and then looking in your elasticsearch logs. Have a look at the troubleshooting section, it explains how to read the ACL logs, see what rule did not match, and also how to see debug messages in LDAP.
  1. The readonly account uses the “kibana_access” rule, which denies all request, except those that are expected to be sent by a typical Kibana session. That is why your request for indices is not successful. It’s explained better in the docs.
    Is there a reason why you need that account to see the list of all indices?
3

Thanks for the reply!

I hadn’t seen the troublshooting section, will read through that and hopefully that’ll answer my questions.

Regarding getting the list of indices via the readonly account, that’s just something I typically do to confirm that the API is working properly.

1 Like
4

Follow-up question. I’ve been trying to get our LDAP auto to work with admin rights granted to members of the “kibana-admin” group, and have everyone else default to non-admin and remove some apps they don’t need to see, but I just can’t get it to work. I’ve got two accounts, and no matter what I’ve tried I wasn’t able to get the accounts to act differently. Hoping I’m missing something obvious, I spoke with folks on the team who were more well versed with LDAP, we tried some things but just can’t get it to work.

Here’s the readonlyrest.yml:

---
# yamllint disable rule:line-length
readonlyrest:
  enable: true
  ssl:
    enable: true
    keystore_file: "elasticsearch.jks"
    keystore_pass: "derpderp"
    key_pass: "derpderp"
  access_control_rules:
    - name: "Allow localhost"
      hosts: [127.0.0.1]
    - name: "::RO::"
      auth_key: ro:derp
      kibana_access: ro
      kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management", "apm"]
    - name: "::ADMIN::"
      type: allow
      auth_key: "admin:derp"
      kibana_access: admin
    - name: "::LOGSTASH::"
      auth_key: "elasticsearch:derpderpderp"
      verbosity: error
    - name: "Limited Kibana access for devqa"
      kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:dev_tools", "kibana:management", "apm"]
      ldap_authentication:
        name: "ldap1"
        groups: ["devqa"]
        type: allow
        verbosity: debug
    - name: "ldap admin for kibana-admin"
      kibana_access: admin
      ldap_authentication:
        name: "ldap1"
        groups: ["kibana-admin"]
        type: allow
        indices: ["*"]
        verbosity: error
  ldaps:
    - name: ldap1
      host: "ldap.example.com"
      port: 636
      ssl_enabled: true
      ssl_trust_all_certs: true
      search_user_base_DN: "cn=users,cn=accounts,dc=dev,dc=localdomain"
      search_groups_base_DN: "cn=groups,cn=accounts,dc=dev,dc=localdomain"
      user_id_attribute: "uid"
      unique_member_attribute: "memberof"
      connection_pool_size: 10
      connection_timeout_in_sec: 30
      request_timeout_in_sec: 30
      cache_ttl_in_sec: 60
      group_search_filter: "(objectclass=top)"
      group_name_attribute: "cn"

The admin account works, the read-only account works, the logstash account works - everything non-LDAP works as expected.

As you can see, I’ve been trying to limit access for members of the “devqa” group, but what I would like to do is have anyone not in the “kibana-admin” group get these restrictions, whereas anyone in “kibana-admin” gets admin access. If I leave in just the first LDAP block, anyone gets limited access, regardless of group. If I leave in just the second, everyone gets admin access. If I leave both in, anyone gets the limited access.

I guess I’m missing something fundamental about the LDAP authentication and groups permissions. I’ve searched a couple posts which I thought would help but it didn’t clear it up.

5

Hi @cmh,

There are mainly two problems with your settings:

  1. If you want to require authentication (user should have valid credentials) and authorization (user should belong to the right group(s)), you either use “ldap_auth” rule (unified rule for authentication+authorization) or “ldap_authentication” and “ldap_authorization” rules together. See the two distinct examples in the docs.

  2. The type, indices and verbosity rules do not belong under “ldap_authentication”, but they should be indented at the same level with “name”. So therefore:

    - name: "ldap admin for kibana-admin"
      kibana_access: admin
      ldap_authentication: # <-- this is also wrong, should be "ldap_auth"
        name: "ldap1"
        groups: ["kibana-admin"] 
      type: allow
      indices: ["*"]  # <-- this is pleonastic. Restricting to "any index" does not make any sense.
      verbosity: error

One more thing: the verbosity is either info or error. Info is default.

6

Okay, I think I’ve got it sorted, thanks for the assistance!

I’ve now a new question. We use puppet to provision our stuff, and I discovered that when puppet lays down a new readonlyrest.yml, even if I restart the elasticsearch server that kibana is communicating with, the active config doesn’t change. So, I did my debugging using the admin account and the ReadonlyREST tab in Kibana, and then transferred that config to puppet. I also discovered I could load the puppet-provisioned readonlyrest.yml by clicking “Load Default” - my question is, what’s the best way to have an update to the file overwrite the config that’s stored in the .readonlyrest index? Preferably, it’d be something that could happen from puppet, so from a command line. I imagine I could delete the .readonlyrest index as it only has one document - that config - but that seems like a heavy-handed approach. Maybe overwriting the document in the .readonlyrest index?

7

You can use the same API that ROR Enterprise/PRO is using to validate and store in-index the settings coming from the YAML web editor.

POST $ES_URL/_readonlyrest/admin/config

{
  "settings": "...JSON escaped settings YAML"
}
8

Cool, I was able to write a script to push the config. I’ll probably have it query the config to compare before pushing it, but it looks like it’ll work. Thanks!

1 Like