Roll up/Clean Up - ROR Audit indices

New Features Ideas
1

:bulb: Roll up/Clean Up - ROR Audit indices on user configured schedule

This idea was discussed as part of another thread. I am just creating a separate thread to track this separately and not to clutter the original thread which was opened for a different issue.

Currently, ROR audit indices are created on a daily basis. These indices are not automatically cleaned up. So suggestion is to add configuration options to Clean Up and/or RollUp indices. I could think of 2 ways to make this work.

First one is where you give the frequency setting option upfront - which decides the frequency of index creation - daily, weekly, monthly, quarterly and a second setting that controls when to clean up the indexes.

Second option is continue with creating daily indexes, but give option to delete them after n days (I would suggest default to 15 days) and also give option to consolidate the indexes into weekly, monthly, quarterly. For people who are using ES purely for log analytics, storing historical security audit logs might not be a big requirement. So they might want regular clean up. But for those who are using ES for storing/searching some sensitive data, historical audit logs will still be a need. So instead of continuing with just daily indexes, its better to consolidate the indexes as part of the clean up process, which means whenever you delete the index, you also copy the data into one of the consolidated index.

Though some of these activities can be done via Curator, its better to have these part of ROR itself so that individual users don’t have to spend time and effort to perform the curator setup and testing.

Along with this, I would also suggest to add the configuration to define the number of shards to be used. Currently ROR default to 5 shard/index. But for smaller usage volume, it might be beneficial to have a limited number for shards. Similarly for rolled up indices (monthly, quarterly), more shard allocation might be needed for higher usage volume. So better to have it as configurable option.

:rocket: Let’s do this?

  • 1
  • 2
  • 3
  • 4
  • 5

0 voters

Show results
2

@sscarduzio did you get a chance to make any progress on this feature? We have been using ROR in production for few months now. So wanted to provide some update. Since ROR is creating daily indexes, this list has now reached some 150+ indexes which will have to be manually collated and merged now. Any alternate options that you can suggest in the intermediate?

Thanks!

3

Not yet, sorry. The development in ROR for ES is proceeding fast, but the backlog in terms of features and innovation is growing faster.

In the meanwhile you can use this curator script to consolidate daily indices into monthly/yearly indices:

4

Quick and dirty (manual) process to roll up daily indexes into a monthly index

POST _reindex
{
  "source": {
    "index": "readonlyrest_audit-2018-03-*"
  },
  "dest": {
    "index": "readonlyrest_audit-2018-03"
  }
}

Then delete the old dailys

DELETE readonlyrest_audit-2018-03-*

2 Likes
5

thanks @brian, This helps.

6

Part of the requested feature is now available in ROR for ES v1.16.20.
Custom audit index name template

1 Like