In my elasticsearch logs I see that readonlyrest blocks requests. What I do not understand is why those requests exist at all.
In short: It looks like logstash is requesting indices:admin/seq_no/global_checkpoint_sync
This is a log entry:
2025-11-04T13:43:14,495][INFO ][t.b.r.a.l.AccessControlListLoggingDecorator] [elastic-node] \u001b[35mFORBIDDEN by default req={ ID:33afc0ed-2312-470b-a3d7-14742f02bc30-668907550#326944661, TYP:GlobalCheckpointSyncAction$Request, CGR:<N/A>, USR:logstash (attempted), BRS:true, KDX:null, ACT:indices:admin/seq_no/global_checkpoint_sync, OA:a.b.c.d/32, XFF:null, DA:e.f.g.h/32, IDX:2025.11.04, MET:POST, PTH:/_bulk, CNT:<N/A>, HDR:Content-Type=application/json, User-Agent=Logstash/8.18.3 (OS=Linux-5.15.0-157-generic-amd64; JVM=Eclipse Adoptium-21.0.7) logstash-output-elasticsearch/11.22.12, transfer-encoding=chunked, Host=elastic-node:9200, Connection=Keep-Alive, Authorization=<OMITTED>, Accept-Encoding=gzip,deflate, x-elastic-product-origin=logstash-output-elasticsearch, HIS:[LOGSTASH-> RULES:[auth_key_sha256->true, actions->false] RESOLVED:[user=logstash;indices=mylogs]]
(I needed to change hostnames and ip adresses, also I reduced the HIS to the only rule with matching auth_key_sha256)
The rule blocking this requests exists, here it is:
- name: "LOGSTASH"
verbosity: error
auth_key_sha256: <some sha256 sum>
actions:
- "indices:data/*"
- "indices:admin/types/exists"
- "indices:admin/template/*"
- "indices:admin/index_template/get"
- "indices:admin/create"
- "indices:admin/get"
- "indices:admin/index_template/get"
- "indices:admin/index_template/put"
- "cluster:monitor/main"
- "cluster:monitor/xpack/info"
- "cluster:monitor/xpack/license/get"
- "cluster:admin/ilm/get"
The only tool accessing elasticsearch via this rule is the elasticsearch-output of a logstash pipeline. I think elasticsearch output is not sending requests containing this action, as far as I know this is a internal only action.
Also I do not have a user “logstash” inside of elasticsearch let alone a user “logstash” with the exact same password I use in the logstash pipelines.
So, can anybody help me figure out what is going on? Is logstash sending those requests? Is readonlyrest somehow confused about the actions used vs the actions it is logging? What is the consequence if these requests are blocked, I guess shards will not be synced properly? Should I allow this action for logstash?
Thanks in advance