[DONE] Feature: Control what users to log/not log

(Daniel Mota Leite) #1

Right now logging is very spammy as it logs every attempt for all users. When we have authentication for beats and logstash, the log is huge.

A way to disable log for certain users would be very useful, as we could log only the end user or failing users and ignore the remaining users that are known to work. Those that need audit logs can always not disable any user.

(Simone Scarduzio) #2

Yeah I was also thinking logs need more attention and configurability, at least per user. BTW it’s probable that you can filter logs using regex with l4j2 in the meanwhile :slight_smile:

(Daniel Mota Leite) #3

it was hard, as there are no examples how to work with regexfilter in properties files, but i manage to configure it!
So here is the way to log to a file all except some users (logstash, beat and kibana for me)

#Plugin readonly rest logging file definition
appender.access_log_rolling.type = RollingFile
appender.access_log_rolling.name = access_log_rolling
appender.access_log_rolling.fileName = ${sys:es.logs}_access.log
appender.access_log_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n
appender.access_log_rolling.layout.type = PatternLayout
appender.access_log_rolling.filePattern = ${sys:es.logs}_access-%d{yyyy-MM-dd}.log.gz
appender.access_log_rolling.policies.type = Policies
appender.access_log_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.access_log_rolling.policies.time.interval = 1
appender.access_log_rolling.policies.time.modulate = true

logger.access_log_rolling.name = org.elasticsearch.plugin.readonlyrest.acl
logger.access_log_rolling.level = info
logger.access_log_rolling.appenderRef.access_log_rolling.ref = access_log_rolling
logger.access_log_rolling.additivity = false

# exclude kibana, beat and logstash users as they generate too much noise
logger.access_log_rolling.filter.regex.type = RegexFilter
logger.access_log_rolling.filter.regex.regex = .*USR:(kibana|beat|logstash),.*
logger.access_log_rolling.filter.regex.onMatch = DENY
logger.access_log_rolling.filter.regex.onMisMatch = ACCEPT


(Simone Scarduzio) #4

HAHA dude you’re the best :+1: :rocket::joy:

Thanks & will add to docs :writing_hand:

(Simone Scarduzio) #5

Now there’s the verbosity rule that lets you skip logging allowed requests if they match a certain block (useful for Kibana deamon, which is always polling).

(Daniel Mota Leite) #6

@sscarduzio Is that verbosity global or can we set it per rule?

(Simone Scarduzio) #7

it is per rule-block. See example in README.md

 - name: "::KIBANA-SRV::"
      auth_key: kibana:kibana
      verbosity: error # don't log successful request
      type: allow
- name "..."
 #other rules  

(Daniel Mota Leite) #8

Ohh, sorry i didn’t notice it in the README.md! That is perfect! thanks! :smiley:

ps: ES is already in version 5.3.1, so a new build is always welcome :wink:

(Simone Scarduzio) #9

Yeah I didn’t have the time to release 1.15.0, but it’s there and it works with 5.3.{1,2}


This is a build for 1.15.0 for 5.3.2, will have more time to update the website after this week.