ES8.15.1 ROR 1.59.0: Audit outputs, disable log doesn't work?

1

Hi,

Could it be that there is a bug in disabling audit output to logs?
Documentation suggest:

readonlyrest:
  audit:
    enabled: true
    outputs: 
    - type: index
    - type: log
      enabled: false # by default is true

But no matter what combination I try it keeps logging towards log.

To disable logging towards log (what I want) I tried:

readonlyrest:
  enable: true
  prompt_for_basic_auth: false
  response_if_req_forbidden: "xxx"
  audit:
    enabled: true
    outputs: [index]

readonlyrest:
  enable: true
  prompt_for_basic_auth: false
  response_if_req_forbidden: "xxx"
  audit:
    enabled: true
    outputs:
      - type: index

readonlyrest:
  enable: true
  prompt_for_basic_auth: false
  response_if_req_forbidden: "xxx"
  audit:
    enabled: true
    outputs:
      - type: index
      - type: log
        enabled: false

But I always keep getting messages like:

{
    "type": "server",
    "timestamp": "2024-10-10T06:54:20,277+02:00",
    "level": "INFO",
    "component": "t.b.r.a.l.AccessControlLoggingDecorator",
    "cluster.name": "xxx",
    "node.name": "xxx",
    "message": "\u001B[36mALLOWED by { removed rules logging for obivous reasons }\u001B[0m",
    "cluster.uuid": "xxx",
    "node.id": "xxx"
}

I managed to disable logging towards index by using:

readonlyrest:
  enable: true
  prompt_for_basic_auth: false
  response_if_req_forbidden: "xxx"
  audit:
    enabled: true
    outputs:
      - type: log

So disabling logging towards index is possible. But this is the opposite of what I want :slight_smile:

For completeness the log4j properties file:

status = info

# log action execution errors for easier debugging
logger.action.name = org.elasticsearch.action
logger.action.level = debug

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker%m%n

appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
appender.rolling.layout.type = ESJsonLayout
appender.rolling.layout.type_name = server
appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true

rootLogger.level = info
rootLogger.appenderRef.console.ref = console
rootLogger.appenderRef.rolling.ref = rolling

appender.deprecation_rolling.type = RollingFile
appender.deprecation_rolling.name = deprecation_rolling
appender.deprecation_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_deprecation.log
appender.deprecation_rolling.layout.type = PatternLayout
appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker%.-10000m%n
appender.deprecation_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_deprecation-%i.log.gz
appender.deprecation_rolling.policies.type = Policies
appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.deprecation_rolling.policies.size.size = 1GB
appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
appender.deprecation_rolling.strategy.max = 4

logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = warn
logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
logger.deprecation.additivity = false

appender.index_search_slowlog_rolling.type = RollingFile
appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
appender.index_search_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog.log
appender.index_search_slowlog_rolling.layout.type = PatternLayout
appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker%.-10000m%n
appender.index_search_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_search_slowlog-%d{yyyy-MM-dd}.log
appender.index_search_slowlog_rolling.policies.type = Policies
appender.index_search_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.index_search_slowlog_rolling.policies.time.interval = 1
appender.index_search_slowlog_rolling.policies.time.modulate = true

logger.index_search_slowlog_rolling.name = index.search.slowlog
logger.index_search_slowlog_rolling.level = trace
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
logger.index_search_slowlog_rolling.additivity = false

appender.index_indexing_slowlog_rolling.type = RollingFile
appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
appender.index_indexing_slowlog_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog.log
appender.index_indexing_slowlog_rolling.layout.type = PatternLayout
appender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker%.-10000m%n
appender.index_indexing_slowlog_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_index_indexing_slowlog-%d{yyyy-MM-dd}.log
appender.index_indexing_slowlog_rolling.policies.type = Policies
appender.index_indexing_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.index_indexing_slowlog_rolling.policies.time.interval = 1
appender.index_indexing_slowlog_rolling.policies.time.modulate = true

logger.index_indexing_slowlog.name = index.indexing.slowlog.index
logger.index_indexing_slowlog.level = trace
logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
logger.index_indexing_slowlog.additivity = false

elasticsearch.yml doesn’t seem to contain anything related to logging, so skipping that.

How can remove these types of messages from the log of elasticsearch?

2

Hi @ronald.vanboven

Could you show us the log snippet you want to disable?

3

Hi,

It is part of the post above. We log in JSON so the messages we get are:

{
    "type": "server",
    "timestamp": "2024-10-10T07:46:02,791+02:00",
    "level": "INFO",
    "component": "t.b.r.a.l.AccessControlLoggingDecorator",
    "cluster.name": "xxx-prd_elk-log",
    "node.name": "xxx",
    "message": "\u001B[36mALLOWED by { name: 'Metricbeat monitoring', policy: ALLOW, rules: [auth_key_sha256] req={ ID:8d3c12a3-77db-488a-8482-9eea81572cdc-642191814#265396, TYP:MainRequest, CGR:<N/A>, USR:metricbeat, BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Authorization=<OMITTED>, Host=localhost:9200, User-Agent=Elastic-Metricbeat/8.13.2 (linux; amd64; d41b4978ea7b4d7c6020b47ffd8a3b8642531fe3; 2024-04-02 10:08:18 +0000 UTC), X-Elastic-Product-Origin=beats, content-length=0, HIS:[Metricbeat monitoring-> RULES:[auth_key_sha256->true] RESOLVED:[user=metricbeat]], }\u001B[0m",
    "cluster.uuid": "xxx",
    "node.id": "xxx"
}

or unformatted:

{"type":"server","timestamp":"2024-10-10T07:46:02,791+02:00","level":"INFO","component":"t.b.r.a.l.AccessControlLoggingDecorator","cluster.name":"xxx-prd_elk-log","node.name":"xxx","message":"\u001B[36mALLOWED by{ name:'Metricbeat monitoring', policy:ALLOW, rules:[auth_key_sha256] req={ ID:8d3c12a3-77db-488a-8482-9eea81572cdc-642191814#265396, TYP:MainRequest, CGR:<N/A>, USR:metricbeat, BRS:true, KDX:null, ACT:cluster:monitor/main, OA:127.0.0.1/32, XFF:null, DA:127.0.0.1/32, IDX:<N/A>, MET:GET, PTH:/, CNT:<N/A>, HDR:Accept-Encoding=gzip, Authorization=<OMITTED>, Host=localhost:9200, User-Agent=Elastic-Metricbeat/8.13.2 (linux; amd64; d41b4978ea7b4d7c6020b47ffd8a3b8642531fe3; 2024-04-02 10:08:18 +0000 UTC), X-Elastic-Product-Origin=beats, content-length=0, HIS:[Metricbeat monitoring-> RULES:[auth_key_sha256->true] RESOLVED:[user=metricbeat]]}\u001B[0m","cluster.uuid":"xxx","node.id":"xxx"}

Let me clean up the message a bit and send one with more info. One sec. Done, these messages give a bit more info.
@coutoPL These are the audit logging message of ROR right, we want to remove those from the logging to log file.

4

ok, so it seems I need to clarify some things.

In ROR we have the auditing feature. Info about matched/forbidden requests can be stored in the index (as documents) and in logs (as log entries … in particular, it may be a separate file, not the one with ES and ROR logs). The audit configuration can be found in the readonlyrest.audit section.

On the other hand, we have ROR plugin logs. The one you showed is this kind of log. ROR can log much useful information (depending on the log level configured in the log4j2.properties file). One of the information is a matched/forbidden block. We can control the logging using:

  1. verbosity rule - https://docs.readonlyrest.com/elasticsearch#verbosity (configured per block)
  2. log4j2.properties file configuration.
logger.ror.name=tech.beshu.ror.accesscontrol.logging
logger.ror.level=warn

By adding this you will cut off all ROR match/forbidden logs.

I hope, now it’s a little bit more clear.

1 Like
5

Yes, that makes perfect sense.
I will configure that and confirm back here for future readers of the topic.

6

Yep, this works.

Thanks!

1 Like