Set Readonlyrest_audit index


(Marcus Caepio) #1

Hi @sscarduzio
For Rollover of the readonlrest_audits we want to use an alias instead of the default readonlyrest_audit-YYYY-MM-DD.
Where do we set the index name?

Regards,
Marcus


(Simone Scarduzio) #2

Actually this can be a default strategy for ROR! What do others think?


(Hagen Montag) #3

a configurable index name (leading dot) was perfect for me


(Simone Scarduzio) #4

Yes but now we have automatic suffix of the index name with year-month-day. Would you throw away that feature?

The other option i was envisioning is to make ROR clean up after itself by deleting old audit logs with a default of 2 weeks or so.


(Marcus Caepio) #5

Not throw away, but make it default. And in the ROR settings, the option audit_log_index: xyz is given. I would not delete audit logs automatically as long as one can decide, how long he will keep it. Could be a case, that one will keep the logs for longer than 2 weeks.

What I wanted to do, is written here:


(Ela) #6

Yes, having the ability to configure the prefix of the audit index name would be helpful.
But if the audit_log_index=xyz, then the indices should still have YYYY-MM-DD suffix, and then we could have the xyz alias that points to all xyz-* indices (for searching purposes).
Re: cleanup - this should not be a default behaviour (as @MarcusCaepio said, one may want to / need to keep audit logs for longer)


(Marcus Caepio) #7

@elaPa exactly what I thought


(Simone Scarduzio) #8

Hello everyone, I just committed a change that permits you to specify the index name template, and with it the time-granularity of the audit logs i.e. one daily, hourly, yearly or monthly index.

Example:

readonlyrest:
  audit_collector: true
  audit_index_template: "'custom-prefix'-yyyy-MM"  # <--monthly pattern

This will be available in ROR Free for Elasticsearch 1.16.20


Roll up/Clean Up - ROR Audit indices
(Askids) #9

@sscarduzio does this also support providing the number of shards to be used per index? Since using different intervals may lead to different size of collected log, having the option to provide the number of shards can be useful.


(Simone Scarduzio) #10

@askids No, but you can achieve that with index templates, right?


(Askids) #11

Yes. It should be. But now that we support all ROR config in separate file, I was wondering, if we want to really split this configuration as its still related to ROR indexes? From my perspective, it makes sense to keep all ROR config together.


(Roger Seth) #12

EDIT : Ignore this. Looks like the SINGLE quotes are relevant (wasn’t apparent from the documentation - thought the single quotes were just a place holder. Leaving this comment as-is for others to read in case they face this). This YML seems to work:

YML:

readonlyrest:
    audit_collector: true
    audit_index_template: 'zoo-audit'-yyyy-MM

Hello - When configuring the Audit configuration - I get this error. Any ideas what is amiss? (IF I remove the template section, it seems to come up fine). Both the configurations below give the same error.
(Am Using ES 6.3.0)

YML:

readonlyrest:
    audit_collector: true
    audit_index_template: zoo-audit-yyyy-MM

YML(With double quotes):

readonlyrest:
    audit_collector: true
    audit_index_template: "zoo-audit-yyyy-MM"

ERROR:

[2018-08-21T09:01:42,236][ERROR][t.b.r.e.IndexLevelActionFilter] [9dnIe3z] Cannot configure ReadonlyREST plugin
java.lang.IllegalArgumentException: Illegal pattern character ‘o’
at java.text.SimpleDateFormat.compile(SimpleDateFormat.java:826) ~[?:1.8.0_151]


(Ld57) #13

Good to ear that !

I will also use the index pattern definition :slight_smile:


(Simone Scarduzio) #14

Yes guys, the string gets thrown as is into a Java SimpleDateFormat, all online documentation about how to use the SimpleDateFormat class describes well how this works. The docs have obviously less examples.


(Roger Seth) #15

thanks @sscarduzio. The documentation is pretty good - its just human nature to see placeholder and replace them :). A modest suggestion : Catching Java Formatter exception (which is what I suppose is thrown) and outputting a friendly “correction message” right when ROR bootstrapping fails instead of the error I posted?


(Simone Scarduzio) #16

DONE! Thanks for the feedback :slight_smile: