ROR with ES 8.3.2 and Logstash 8.3.2

Hi,

Here is the error I received when I start Logstash

> [2022-07-28T11:51:59,369][ERROR][logstash.javapipeline    ][slap] Pipeline error {:pipeline_id=>"slap", :exception=>#<ArgumentError: invalid byte sequence in UTF-8>, :backtrace=>["org/jruby/RubyRegexp.java:1146:in `=~'", "org/jruby/RubyString.java:1663:in `=~'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:72:in `block in add_patterns_from_file'", "org/jruby/RubyIO.java:3329:in `each'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:70:in `add_patterns_from_file'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:471:in `block in add_patterns_from_files'", "org/jruby/RubyArray.java:1821:in `each'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:467:in `add_patterns_from_files'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:280:in `block in register'", "org/jruby/RubyArray.java:1821:in `each'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:276:in `block in register'", "org/jruby/RubyHash.java:1415:in `each'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.2/lib/logstash/filters/grok.rb:271:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/apps/slap-shipper/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/apps/slap-shipper/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/apps/slap-shipper/logstash-core/lib/logstash/java_pipeline.rb:599:in `maybe_setup_out_plugins'", "/apps/slap-shipper/logstash-core/lib/logstash/java_pipeline.rb:245:in `start_workers'", "/apps/slap-shipper/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/apps/slap-shipper/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/apps/slap-shipper/pipelines/pipeline-kafka-elasticsearch.conf"], :thread=>"#<Thread:0x41704c2c run>"}
> [2022-07-28T11:51:59,370][ERROR][logstash.outputs.elasticsearch][slap] Failed to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 'http://localhost:9200/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:397:in `exists?'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:402:in `template_exists?'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:81:in `template_install'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:154:in `block in after_successful_connection'"]}
> [2022-07-28T11:51:59,373][INFO ][logstash.javapipeline    ][slap] Pipeline terminated {"pipeline.id"=>"slap"}
> [2022-07-28T11:51:59,374][ERROR][logstash.outputs.elasticsearch][slap] Failed to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 'http://localhost:9200/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:397:in `exists?'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:402:in `template_exists?'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:81:in `template_install'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", "/apps/slap-shipper/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:154:in `block in after_successful_connection'"]}

My readonlyrest .yml file contains this configuration for Logstash

....
 - name: "logstash with write and create permissions for its own indices"
      auth_key: logstash:logstash
      actions: ["cluster:monitor/xpack/license/get","cluster:monitor/main","cluster:monitor/nodes/stats","cluster:monitor/xpack/info","cluster:admin/xpack/monitoring/bulk","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
      indices: ["logstash-*","error-*","monitoring-*","tomcatlog-*"]
....

Maybe an action is missing.

Best regards
H.

{:message=>"Got response code ‘403’ contacting Elasticsearch

Yes, and the elasticsearch.log will tell you exactly what it is, if you grep for “FORBIDDEN”.

I found that :

...
FORBIDDEN by default req={ ID:1401342291-1645870592#2410, TYP:PutComposableIndexTemplateAction$Request, CGR:<N/A>, USR:logstash (attempted), BRS:true, KDX:null, ACT:indices:admin/index_template/put
...

Then I added this to the configuration :

"indices:admin/index_template/*"

But still the same error.

Hassen, a big chunk of log line is missing. Especially the “IDX” (resolved targeted indices) and “HIS” (ACL evaluation history)