Inherent limitations of ReadonlyREST with regard to Cross-Cluster Search?

Users Feedback
1

Hi,

Let’s assume two Elasticsearch clusters:

  • one “data” cluster with ReadonlyREST installed;
  • one “ccs” cluster without ReadonlyREST.

“data” was configured as a remote cluster to “ccs”.

As far as I understand:

  • “ccs” will connect to “data” over the Transport layer, not the HTTP layer, and this cannot be changed;
  • “ccs” will forward the Authorization header over the Transport layer (I checked by sniffing the network traffic);
  • ReadonlyREST interacts only with HTTP requests (hence the “REST” in its name) and leaves exchanges over the Transport layer absolutely untouched;
  • Consequently, ReadonlyREST does not filter the results that flow from “data” to “ccs”.

Do the statements above make sense? Have I understood correctly? Thanks in advance.

2
  1. True
  2. OK
  3. True
  4. It actually does. The cross cluster search request, as long as arrives to the HTTP endpoint of a node with ROR installed, can be normally allowed/rejected (before Elasticsearch evaluates it) based on network parameters (origin address/network, destination address/network), http parameters (method, body, uri), but also elasticsearch specific parameters like actions and indices.

For example, a rule like this is effective in the context of a cross cluster search request:

indices: [".kibana", "logstash-*", "css_cluster:remote_indices-*"]
3

Ok, it seems we agree on the technical background. Let me rephrase the various options:

Option #1: ccs cluster without ROR, data cluster with ROR: HTTP requests that arrive directly at data will get filtered responses, HTTP requests that arrive at ccs will get unfiltered responses.

Option #2: ccs cluster with ROR, data cluster with ROR: all HTTP requests will get filtered responses, but it becomes necessary to configure the ACLs twice: once in ccs, once in data.

Option #3: ccs cluster with ROR, data cluster without ROR: HTTP requests that arrive directly at ccs will get filtered responses, HTTP requests that arrive at data will get unfiltered responses.

In the end, to achieve a reasonably secure setup using ROR and CCS, it is necessary to:

  • install ROR on all clusters (data clusters + CCS cluster)
  • duplicate ACLs on the CCS cluster
  • ensure data nodes are reachable through the Transport layer only from trusted nodes.

Is this correct? Please note I do not wish to demean ROR, I just want to ensure I fully understand the objective limitations of both ROR and CCS before moving on with architectural decisions.

1 Like
4

Yeah I think you are right.

I would add that theoretically, this is the work of a simple poller that checks for changes in “cluster1:.readonlyrest” index and conditionally writes it across to “cluster2:.readonlyrest”.

And also, remember that at the moment the “filter” and “fields” rule won’t have effect in the remote cluster. BTW: can you expand on your wireshark capture where you saw the authorization header going through? I actually didn’t know it was possible, might turn up as an opportunity to transmit access control information to remote clusters (i.e. making filter and fields rules work).

5

In a simple case with one ccs cluster hitting only one data cluster, it should indeed remain a simple poller; but I expect things to get more complicated if the ccs cluster has multiple remote data clusters.

Oh, yeah, that makes sense: since “filter” and “fields” rules are applied by the ROR plugin installed on data nodes, even if the ccs cluster has ROR installed and the ACLs are properly duplicated, ROR will find itself unable to apply “filter” and “fields” rules. In the end, ROR’s filter and fields rules are simply incompatible with CCS.

I confirm I have seen the Authorization header from the initial HTTP request passed multiple times in the transport layer, reaching all data nodes in the remote cluster.
That said, I believe this is because I tested with a X-Pack-enabled CCS cluster hitting a ROR-enabled data cluster. This passing of the Authorization header is documented here:

1 Like
6

I was thinking, in case that there’s more clusters, we could have a configuration key in ROR that says which one would be the “master cluster”, and all the nodes of the “slave clusters” would just poll for what’s the current configuration in the master cluster. I.e.

readonlyrest:
   ...
   master_cluster: cluster1
   ...

So that the in-index security settings are going to be polled by all the nodes of all the cluster from “cluster1:.readonlyrest” index. Wouldn’t this be sufficient? :thinking:

7

I am not sure this master/slave design matches any real-life use case. An interesting usage of Cross-Cluster Search is what follows:

  • multiple Elasticsearch clusters, each holding various kinds of data, each with their own design, sizing, shard policies, and, of course, ReadonlyREST security rules, typically with field-level security (because it is useful and because if we get this working, we can get anything working). Truly independent clusters, except maybe they share the same source for users and groups (e.g. LDAP).
  • a dedicated, data-less Elasticsearch cluster (possibly a single-node cluster, it does not really matter), with just enough ReadonlyREST rules to require authentication, and multiple remote Elasticsearch clusters.
  • Requests arriving on the data-less cluster get forwarded to the adequate remote clusters; these clusters have their own ReadonlyREST security rules which are well tailored to the indices and documents they host, and thus take care of field-level security on their side, without the data-less cluster having to know anything about it. All filtered responses flow down to the data-less cluster which merges them and send the final response to the client.

In the end, I do not think there is an actual need to replicate ReadonlyREST rules across clusters. When I suggested duplicating the rules, it was only as a workaround to CCS/Transport requests being ignored by ReadonlyREST.

8

Sounds very complex, if I was in an architectural committee I would ask you at least 10 reasons why we can’t do this in a simpler way haha. :smiling_imp:

9

Ahah, I guess it sounds complex. But it is only as complex as the combination of the Cross-Cluster Search and Field-Level Security features. CCS is interesting when one has multiple clusters, possibly after having read Multiple Elasticsearch Clusters, or a Monster Cluster? | Elastic Blog ; the annoying part of CCS is its dependency on the Transport Layer (I assume this was done for performance reasons).
To be fair, SearchGuard / Open Distro also fail to implement proper CCS+FLS; it seems only X-Pack applies FLS filtering to CCS queries.

10

The use case you are describing is supported by Search Guard. All security rules, including DLS/FLS, are executed on each cluster independently. So you can have different security configurations on each cluster for the same user, tailored to the indices and documents the cluster holds. And then set up a coordinating / data-less cluster that handles the queries. If this is not working you can always ask on the SG forum or open an issue on GitHub. (Disclaimer: I am the Co-Founder of Search Guard)

11

Hi Jochen,

I was indeed considering reporting a bug to Search Guard. I confirm I was unable to make CCS+FLS work as expected (hitting the data-holding Elasticsearch cluster yielded filtered results, hitting the CCS-only Elasticsearch instance yielded unfiltered results). But, to be fair, I had to “time out” at some point in my experimentations with Search Guard, so maybe I just need to give it some more time…
Needless to say, it is nice to read that this use case is or should be supported :slight_smile: