Roll up/Clean Up - ROR Audit indices on user configured schedule
This idea was discussed as part of another thread. I am just creating a separate thread to track this separately and not to clutter the original thread which was opened for a different issue.
Currently, ROR audit indices are created on a daily basis. These indices are not automatically cleaned up. So suggestion is to add configuration options to Clean Up and/or RollUp indices. I could think of 2 ways to make this work.
First one is where you give the frequency setting option upfront - which decides the frequency of index creation - daily, weekly, monthly, quarterly and a second setting that controls when to clean up the indexes.
Second option is continue with creating daily indexes, but give option to delete them after n days (I would suggest default to 15 days) and also give option to consolidate the indexes into weekly, monthly, quarterly. For people who are using ES purely for log analytics, storing historical security audit logs might not be a big requirement. So they might want regular clean up. But for those who are using ES for storing/searching some sensitive data, historical audit logs will still be a need. So instead of continuing with just daily indexes, its better to consolidate the indexes as part of the clean up process, which means whenever you delete the index, you also copy the data into one of the consolidated index.
Though some of these activities can be done via Curator, its better to have these part of ROR itself so that individual users don’t have to spend time and effort to perform the curator setup and testing.
Along with this, I would also suggest to add the configuration to define the number of shards to be used. Currently ROR default to 5 shard/index. But for smaller usage volume, it might be beneficial to have a limited number for shards. Similarly for rolled up indices (monthly, quarterly), more shard allocation might be needed for higher usage volume. So better to have it as configurable option.
Let’s do this?