Permission denied for Kibana admin account

More info - something must be wrong with my whole config as LDAP auth seems to be the only thing working. Logstash is now unable to log to the ELK cluster using another local account.

Here’s my (sanitized) readonlyrest.yml file:

# yamllint disable rule:line-length
# THIS FILE IS PROVISIONED BY PUPPET
# However, once it gets loaded into the .readonlyrest index,
#  you might need to use an admin account to log into Kibana
#  and choose "Load default" from the "ReadonlyREST" tab.
# Alternately, you can use the "update-ror" script in ~cmh/bin/
readonlyrest:
  enable: true
  response_if_req_forbidden: Forbidden by ReadonlyREST plugin
  ssl:
    enable: true
    keystore_file: "elasticsearch.jks"
    keystore_pass: "pass"
    key_pass: "pass"
  access_control_rules:
    # LOCAL: Kibana admin account
    - name: "local-admin"
      auth_key: "admin:pass"
      kibana_access: admin
    # LOCAL: Logstash servers inbound access
    - name: "local-logstash"
      auth_key: "logstash:pass"
      # Local accounts for routine access should have less verbisity
      #  to keep the amount of logfile noise down
      verbosity: error
    # LOCAL: Kibana server
    - name: "local-kibana"
      auth_key: "kibana:pass"
      verbosity: error
    # LOCAL: Puppet communication
    - name: "local-puppet"
      auth_key: "puppet:pass"
      verbosity: error
    # LOCAL: Elastalert
    - name: "elastalert"
      auth_key: "elastalert:pass"
      verbosity: error
    # LDAP: kibana-admin group
    - name: "ldap-admin"
      kibana_access: admin
      kibana_hide_apps: [""]
      ldap_auth:
        name: "ldap1"
        groups: ["kibana-admin"]
      type: allow
    # LDAP for everyone else
    - name: "ldap-all"
      # possibly include: "kibana:dev_tools",
      kibana_hide_apps: ["readonlyrest_kbn", "timelion", "kibana:management", "apm"]
      ldap_auth:
        name: "ldap1"
        groups: ["kibana-admin", "admins", "prod-admins", "devqa", "development", "ipausers"]
      type: allow
    # Allow localhost
    - name: "localhost"
      hosts: ["127.0.0.1"]
  # Define the LDAP connection
  ldaps:
    - name: ldap1
      host: "freeipa.example.com"
      port: 636
      bind_dn: "uid=system,cn=stuff,dc=localdomain"
      bind_password: "pass"
      ssl_enabled: true
      ssl_trust_all_certs: true
      search_user_base_DN: "cn=users,cn=accounts,dc=stuff,dc=localdomain"
      search_groups_base_DN: "cn=groups,cn=accounts,dc=stuff,dc=localdomain"
      user_id_attribute: "uid"
      unique_member_attribute: "member"
      connection_pool_size: 10
      connection_timeout_in_sec: 30
      request_timeout_in_sec: 30
      cache_ttl_in_sec: 60
      group_search_filter: "(objectclass=top)"
      group_name_attribute: "cn"

The logstash boxes have the same username/password combination found in the readonlyrest.yml file, but I’m still getting denied.

[2019-04-26T16:51:23,623][INFO ][t.b.r.a.ACL              ] [bz1elasticsearch2-1.bb.internal.maas360.com] FORBIDDEN by default req={ ID:1271538866-900160031#11003, TYP:MainRequest, CGR:N/A, USR:logstash(?), BRS:true, KDX:null, ACT:cluster:monitor/main, OA:{logstash box}, DA:{es node}, IDX:<N/A>, MET:HEAD, PTH:/, CNT:<N/A>, HDR:{Authorization=<OMITTED>, content-length=0, Connection=Keep-Alive, User-Agent=Manticore 0.6.4, Host=elasticsearch:9200, Accept-Encoding=gzip,deflate, Content-Type=application/json}, HIS:[local admin->[auth_key->false]], [kibana-admin ldap->[ldap_authentication->false]], [devqa ldap->[ldap_authentication->false]], [readonly test->[auth_key->false]], [kibana server->[auth_key->false]], [logstash->[auth_key->false]], [localhost->[hosts->false]] }

Anything obvious that’s wrong with this config? I’ve set it up according to the instructions - at least I think I have, but something’s obviously wrong here.