Audit log to optionally log executed queries


(Tozuan) #1

:bulb: Audit log to optionally log executed queries

There are many cases where sensitive information is stored in ElasticSearch and there is a reason to give access to the data for some users, but still need to be able to detect malicious or unnecessary access.

Thus it would be good for audit log to also record the executed query on specific indices, so that users’ actions can be reviewed later. Also possibly, the amount of returned documents could be good to record.

There should be

  1. a new multi value setting, “audit_include_query” which species that which indices’ queries are logged. The setting should support wildcards.

  2. if there is a query to an index that matches a rule, the request body(which includes users’ query) would be included in the audit log

Considerations and possible side effects of the feature.

  • Performance impact
  • Log size
  • Information security when using the feature

:eyes: Example

Example configuration:

 readonlyrest:
 audit_collector: true
 audit_include_query: ["bla-*", "sensitive-index", "*-personal" ]

Request body - and entry which would be added to audit log

{
   "query":{
     "query_string": {
     	"default_field": "product_id",
       "query":"HFD-DE4"
     }
   }
}

References

Github conversation
Discussion about similar feature at the forum

:rocket: Let’s do this?

  • 1
  • 2
  • 3
  • 4
  • 5

0 voters


(Askids) #2

@sscarduzio is there any work going for this feature?

Also, when you add this option, it may be beneficial to log these into separate index rather than clogging actual audit index.


(Simone Scarduzio) #3

There was some work done on audit recently (custom audit index name and time granularity feature), but not this one yet.

I was thinking we could push this a bit further and create a sort of ACL for loggers. Albeit it would not stop the evaluation at the first matching block, and log a request multiple times in multiple indices if more than a block matches.

...
audit:
  enable: true
  index_loggers:
  - name: "privacy audit"
    index_template: "'trace-'YYYY-MM"
    indices: ["sensitive-*", "*-private-*"]
    log_fields: ["content", "path", "oa", "indices" ]

  - name: "redflags audit"
    index_template: "'redflags-'YYYY-MM"
    actions: ["*delete*", "*flush*"]
    log_fields: ["*"] # can omit the rule all together to mean log every field

  - name: "trace audit"
    index_template: "'trace-'YYYY-MM"
    log_fields: ["~content"] # don't log the body of less interesting the requests (i.e. bulk indexing)

what do you think?
PS: I was trying to invent a rule name and syntax to express a cleanup policy and frequency. I.e. “cleanup stuff older than 6 months, every 24 hours”. Maybe you can suggest?


(Askids) #4

Yes. This looks to be cleaner approach. Also, today, ROR already creates audit related index, when we enable audit collector. So now are we saying that this will be in addition to it or is this going to replace that existing audit index altogether? I would vote for in addition to existing feature.

Regarding your other question, how about using “expiry” or “purge_after” or “expiry_frequency” or some variant of it?

Also, once you add these audit indexes, probably having some standard dashboards which can use these indexes would also be a nice to have feature on the Kibana side :wink: