RoR readonlyrest ES - [8.5.2] - unable to setup beat index management

Users Feedback
1

Dear All, using elastic stack v8.5.2 and ror (1.44.0), I’m not able to find the proper “rigts and permissions” to let a beat setup the --index-management properly.

  1. I received a 401 for filebeat setup --index-management
  2. I don’t see any error in ror audit log
  3. If I deactivate the ror security, it works.

Could you please help me to fix this ? Thank you.
kr,
G.

ps: A couple of lines, just to explain my point of view; it’s not just about the index template, “beat setup” command create also:

  • Setup kibana dashboards, kibana patterns(kibana dataview), Setup Ingest pipelines, …
  • The index-management creates all components related to Elasticsearch index management, including template, ilm policy and rollover alias, …

As you may understand, I don’t want to do this manually foreach beats instances we used; I would like to use the beats command each time I change the version used to be sure to have the latest config and also to be sure to have all the required settings properly in place.

nb: Please find below my terminal output, an extract of my restonlyrest config and screen captures.

  • The communication with my servers, and my configuration are OK
# config file use: OK
root@server01:/etc/filebeat# filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat test config
Config OK
# handshake with servers: OK
root@server01:/etc/filebeat# filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat test output | tail -n8
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 8.5.2
# setup of other element: OK
root@server01:/etc/filebeat# filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
  • setup of all components related to Elasticsearch index management, including template, ilm policy and rollover alias :NOK
root@server01:/etc/filebeat# filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat setup --index-management
Exiting: error loading template: failed to put data stream: could not put data stream: 401 Unauthorized: {"error":{"root_cause":[{"type":"forbidden_response","reason":"Sorry, your request is forbidden.","due_to":"OPERATION_NOT_ALLOWED","header":{"WWW-Authenticate":"Basic"}}],"type":"forbidden_response","reason":"Sorry, your request is forbidden.","due_to":"OPERATION_NOT_ALLOWED","header":{"WWW-Authenticate":"Basic"}},"status":401}. Response body: {"error":{"root_cause":[{"type":"forbidden_response","reason":"Sorry, your request is forbidden.","due_to":"OPERATION_NOT_ALLOWED","header":{"WWW-Authenticate":"Basic"}}],"type":"forbidden_response","reason":"Sorry, your request is forbidden.","due_to":"OPERATION_NOT_ALLOWED","header":{"WWW-Authenticate":"Basic"}},"status":401}
  • Here is an extract of the readonyrest config:
# ## ACLs
  access_control_rules:
  - name: "::KIBANA-SRV::"
    ...
  - name: "::LOGSTASH::"
    ...
  - name: "x-pack monitoring - collect readonly - no indices"
    auth_key_sha256: "remote_monitoring_user:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    type: allow
    actions: ["indices:monitor/state","cluster:monitor/state","cluster:monitor/stats","indices:monitor/stats","indices:monitor/recovery","indices:monitor/settings/get","cluster:ror/user_metadata/get","indices:data/read/mget"]
    #indices: [".monitoring-*","metricbeat-*"]
    verbosity: info
  - name: "x-pack monitoring - self service"
    auth_key_sha256: "remote_monitoring_user:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    type: allow
    actions: ["indices:monitor/*","cluster:monitor/*","cluster:admin/ilm/*", "indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/index_template/*", "indices:admin/create", "cluster:admin/ingest/pipeline/*","cluster:admin/xpack/monitoring/*","indices:admin/data_stream/*"]
    indices: [".monitoring-*","metricbeat-*",".kiban*",".ds-metricbeat*",".ds-.monitoring*","filebeat*", ".ds-filebeat*"]
    verbosity: info
  . . .

  • No entry in the ror audit log :confused:
    (check in the discover section in kibana with a filter: NOT final°state: ALLOWED

  • The beats can’t send, ingest any data in elasticsearch without the index management properly
    (check in the monitoring section in kibana: I can see all my beats agent (version, allocated memory, etc)

1 Like
2

We manage to solve this issue using:

  1. a dedicated ror block with high privileges
  2. and a one time command to setup the beats. (see details below);

But for some reason the error msg ‘401 unauthorized return by the beats’ is not logged in readyonlyrest_audit index :confused:

  • readonlyrest.yml index
  - name: "beat --setup user"
    type: allow
    auth_key: beatinstall:changeme
    actions: ["cluster:*", "indices:*", "internal:*"]
    verbosity: info
  • server teminal output
# using the default credential stored in the beat yml file
root@server01:/etc/heartbeat-elastic# heartbeat test config
Config OK

root@server01:/etc/heartbeat-elastic# heartbeat test output
elasticsearch: https://server07:9200...
  parse url... OK
  connection...
    . . .
  talk to server... OK
  version: 8.5.2

# setup the template, ilm rules, kibana dashboard etc; using a superuser with env variable that overwrite the ones that are stored in the beat yml file
root@server01:/etc/heartbeat-elastic# heartbeat setup -E "output.elasticsearch.username=beatinstall" -E "output.elasticsearch.password=xxxxxxxxxx"
Index setup finished.

have a nice day,

Kr,

Gautier.

2 Likes