Circuit breaking exception and memory leak

1

ROR team,

We are using Enterprise version of ROR plugin.
ROR version : readonlyrest_kbn_enterprise-1.19.5
ES version: 7.7.0

ES nodes crash with circuit_breaking_exception after 7-10 days. The JVM heap keep raising.
I raised a ticket against Elastic and they figured out its ROR plugin causing the leaks. Below is the final email from them. I cannot remove the plugin in the Production to experiment this. 100’s of users are using the ELK cluster.

Can ROR team please help?

Our team has found the following in their investigation of the heap dump.
 
scala.collection objects which are not internally used in elasticsearch and are allocated by non elasticsearch classtech.beshu.ror.es.request.context.types.SearchEsRequestContext#1
yaml parser related object that are used by elasticsearch but which in this case were allocated by tech.beshu.ror.utils.yaml.parser$FlatteningConstructor#1
cluster:admin/rradmin/refreshsettings that is not in elasticsearch.
The bullet points above both point to objects used by the ReadonlyREST Suite. It looks like this plugin (readonlyrest version 1.19.5) is installed on all of your cluster nodes.

Our dev team as asked that you attempt to reproduce this issue after removing the readonlyrest plugin.

Best,
Andrew

Access Your Case: https://support.elastic.co/customers/s/case/5004M00000eAiYU
2

Hi @praveenmak,

Thanks for your report. It seems that the leak you are talking about is already fixed in ROR 1.22.1 (see Download (UNIVERSAL) - ReadonlyREST), but we recommend to keep the ROR updated, so please try 1.24.0.

Please let me know if it helped

3

Can I try 1.24 on 7.7.0 ?

4

yes, sure.

(at least 20 chars per post)

5

I will try it on our DR cluster first and let you know Thanks

1 Like