[DEPRECATED] Rewrite indices rule

New Features Ideas
1

:bulb: Index rewriting rule

When a request targets one or more index, replace one or more indices with a given replacement.

:eyes: Example


When Kibana writes a new dashboard object to “.kibana”, retarget the write to “.kibana_user1”.

indices_rewrite: ["(^\\.kibana.*|^logstash.*)", "$1_user1"]


This would unlock multi tenancy using a single instance of Kibana.
So writing to .kibana would end up writing on .kibana_user1.

EDIT: it’s not possible to do this because the Kibana daemon has its own set of credentials. Multi tenancy is not possible to be achieved with ES plugin alone. A Kibana plugin is necessary (and on the way).

HOWEVER This rewrite feature is enough for log ingestion, so, here is a better example:

indices_rewrite: ["^logstash.*", "$1_@user"]

All logstash agents that want to write to logstash* may write to logstash-*_someUser

:rocket: Let’s do this?

  • 1
  • 2
  • 3
  • 4
  • 5

0 voters

Show results
2

that would also be a very good feature! But do not forget to give a way for the admin to manage all the .kibana-* and maybe be able to update configs. i do not want to enter with 3 different users to add a new indice or dashboard :slight_smile:

One may use it for groups instead of users and the config management be less a problem, as usually groups share the same config and are usually fewer when compared with users numbers

3

Would it be OK if the admin would rewrite:

indices_rewrite: [".kibana", ".kibana.*"]

Which means that when admin’s Kibana asks for objects from .kibana index, all the objects come down?

Alternatively, the ROR Kibana plugin could add the feature of a Kibana app where to click the various available kibana indices, changing the associated kibana index on the fly via clicking links.
That would be a distinct feature though.

4

Good news! The basic version of this is in master now.

  1. This works (rewrite requested index to something known at configure-time)
indices_rewrite: ["^\\.kibana.*", ".kibana_user1"]
  1. This works (rewrite requested index to something known at configure-time + prefix/suffix)
indices_rewrite: ["(^\\.kibana.*|^logstash.*)", "$1_user1"] # => .kibana_user1, .logstash-2017-03-01_user1
  1. This does not yet work (rewrite requested index to N existing indices)
indices_rewrite: [".kibana", ".kibana.*"] # => .kibana_user1, .kibana_user2
5

Hi SImone,

This looks very interesting. I suppose that this is still in the early stages. I’ve been playing with this a little bit, setting a rule like
readonlyrest.access_control_rules:

- indices_rewrite: ["^\\.kibana", ".kibana_psaiz02"]
  indices:
  - .kibana
  name: Allow read for default
  type: allow

and I’m getting the following error:

[2017-03-21T16:20:43,413][INFO ][o.e.p.r.IndexLevelActionFilter] [q500N6i] forbidden request: { ID:866fb19ddcc04fcdb7f723afdca42bcd, TYP:CreateIndexRequest, USR:null, BRS:false, ACT:indices:admin/create, OA:127.0.0.1, IDX:.kibana, MET:PUT, PTH:/.kibana, CNT:<OMITTED, LENGTH=178>, HDR:Connection,content-type,Host,Content-Length, EFF:1, HIS:[Allow read for default->[]], [Allow read for default->[indices->true, indices_rewrite->true]] } Reason: java.lang.ClassCastException: org.elasticsearch.action.admin.indices.create.CreateIndexRequest cannot be cast to org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest

This is with es 5.2.1, ror 1.14.0

6

Hey Pablo,

Good catch! You spotted a subtle bug in RequestContext.setIndex. I fixed it and released this pre-build for you.

https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.14.1-pre1/readonlyrest-1.14.1-pre1_es5.2.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ/20170321/eu-west-1/s3/aws4_request&X-Amz-Date=20170321T235739Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=0ce04f5d72f1a7b6ce2dcb138822af78e65728009c2b9e1359ca218717c014fd

PS: Congrats for the 5 new particles :champagne: :wink:

7

Hi Simone,

Thanks for the patch!
The error message has changed slightly

[2017-03-22T12:05:09,643][INFO ][o.e.p.r.a.RequestContext ] id: dedcefdeb2894c2d84d24161896d5794 - Replacing indices. Old:[.kibana] New:[.kibana_psaiz02]
[2017-03-22T12:05:09,643][INFO ][o.e.p.r.IndexLevelActionFilter] [q500N6i] forbidden request: { ID:dedcefdeb2894c2d84d24161896d5794, TYP:IndexRequest, USR:null, BRS:false, ACT:indices:data/write/index, OA:127.0.0.1, IDX:.kibana, MET:POST, PTH:/.kibana/config/5.2.1/_create, CNT:<OMITTED, LENGTH=17>, HDR:Connection,content-type,Host,Content-Length, EFF:1, HIS:[Allow read for default->[indices->true, indices_rewrite->true]], [Allow read for default->[]] } Reason: java.lang.ClassCastException: org.elasticsearch.action.index.IndexRequest cannot be cast to org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest

So, now is IndexRequest (instead of CreateIndexRequest).

Cheers,
pablo

8

There was still something wrong. Here’s a build with a more comprehensive fix:

https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.14.1-pre3/readonlyrest-1.14.1-pre3_es5.2.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ/20170322/eu-west-1/s3/aws4_request&X-Amz-Date=20170322T214131Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=1caeee3245fcd4cf9dd977cbf260e29ca2549eacebfd8817c78cfba379ffae4d

9

Hi Simone,

Thanks for the patch. I’m still getting the same error

[2017-03-23T10:50:14,673][INFO ][o.e.p.r.a.RequestContext ] id: 6a905f7803684eb29856f2b7536356df - Replacing indices. Old:[.kibana] New:[.kibana_psaiz02]
[2017-03-23T10:50:14,673][INFO ][o.e.p.r.a.ACL            ] request: { ID:6a905f7803684eb29856f2b7536356df, TYP:SearchRequest, USR:null, BRS:false, ACT:indices:data/read/search, OA:127.0.0.1, IDX:.kibana_psaiz02, MET:POST, PTH:/.kibana/config/_search, CNT:<OMITTED, LENGTH=77>, HDR:Connection,content-type,Host,Content-Length, EFF:0, HIS:[Allow read for default->[]], [Allow read for default->[indices->true, indices_rewrite->true]] } matched block: Allow read for default match: true}
[2017-03-23T10:50:14,678][INFO ][o.e.p.r.a.RequestSideEffects] Committing 1 effects
[2017-03-23T10:50:14,679][INFO ][o.e.p.r.a.RequestContext ] id: 010733e7cc9240b3b8f8b581c60d5d68 - Replacing indices. Old:[.kibana] New:[.kibana_psaiz02]
[2017-03-23T10:50:14,679][INFO ][o.e.p.r.IndexLevelActionFilter] [q500N6i] forbidden request: { ID:010733e7cc9240b3b8f8b581c60d5d68, TYP:IndexRequest, USR:null, BRS:false, ACT:indices:data/write/index, OA:127.0.0.1, IDX:.kibana, MET:POST, PTH:/.kibana/config/5.2.1/_create, CNT:<OMITTED, LENGTH=17>, HDR:Connection,content-type,Host,Content-Length, EFF:1, HIS:[Allow read for default->[]], [Allow read for default->[indices->true, indices_rewrite->true]] } Reason: java.lang.ClassCastException: org.elasticsearch.action.index.IndexRequest cannot be cast to org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest


[root@psaiz02 plugins]# /usr/share/elasticsearch/bin/elasticsearch-plugin list -v
Plugins directory: /usr/share/elasticsearch/plugins
readonlyrest
- Plugin information:
Name: readonlyrest
Description: Safely expose Elasticsearch REST API
Version: 1.14.1-pre3_es5.2.1
 * Classname: org.elasticsearch.plugin.readonlyrest.wiring.ReadonlyRestPlugin
10

That error was because my reflection algorithm didn’t navigate the object hierarchy more than 2 levels deep looking for the indices field to be changed. Now it’s recursive. Plus, I’m changing the index name also in the search response hits.

https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.14.1-pre4/readonlyrest-1.14.1-pre4_es5.2.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ/20170327/eu-west-1/s3/aws4_request&X-Amz-Date=20170327T173342Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=fc65c29df2c665e6f45651cbf202bd0cc355e9f5cec7be49576b3cd82dbf015c

11

Hi, in this method ,If .kibana_xyz (rewritten one) index does not already exist in es , can we define to create it on runtime?
Like If it does’nt exist …create it?

12

@psaiz can you provide me with proper settings that you are using?

13

@shubhamverma27 this feature is very partial: it does not work for requests other than bulk, get/multiget and search. I don’t recommend relying on it for anything else.

14

okayy… Its Currently not of much use as We can’t even create /delete index pattern into the .kibana_xyz index as while doing any POST/GET it redirects to .kibana index which results in timeout :confused:

15

yeah forget this feature, I probably should remove it as its application field is a small set of request types…