Performance Issue with ROR Authentication


(meir) #1

Hi @sscarduzio

I’m working with ROR Plugin version 5.6.5 for a while now and our teams complaining that they are facing a Performance Issues while sending search/bulk query that take more then 30s - 01:30 ~.

A fast test show the simple command using curl with ROR and without ROR:

without authentication method
[email protected] : ~$ time curl -XGET 'http://cnt:9200/_cat/nodes
xxx.xxx.xxx.xxx 60 99 0 0.59 0.20 0.15 mdi - dnt02
xxx.xxx.xxx.xxx 41 99 0 0.11 0.06 0.07 mdi - dnt01
xxx.xxx.xxx.xxx 5 27 6 0.09 0.05 0.05 - - cnt
xxx.xxx.xxx.xxx 41 99 1 0.02 0.03 0.05 mdi * dnt03

real 0m0.913s
user 0m0.003s
sys 0m0.008s

with authentication method

[email protected] : ~$ time curl -u’admin:XXX’ -XGET 'https://cnt:9200/_cat/nodes
xxx.xxx.xxx.xxx 60 99 0 0.59 0.20 0.15 mdi - dnt02
xxx.xxx.xxx.xxx 41 99 0 0.11 0.06 0.07 mdi - dnt01
xxx.xxx.xxx.xxx 5 27 6 0.09 0.05 0.05 - - cnt
xxx.xxx.xxx.xxx 41 99 1 0.02 0.03 0.05 mdi * dnt03

real 0m2.434s
user 0m0.037s
sys 0m0.109s

My system is designed with with the recommendation of elasticsearch site 1 coordinator node and 3 data nodes and stunnel between all 4 nodes.

Please advice.


(Simone Scarduzio) #2

How can it be slow just listing the cluster nodes??


(Simone Scarduzio) #3

I have a few questions:

  1. what version of ROR is that you are using
  2. how many blocks do you have in the ACL
  3. does it stay constantly slow in time from the first moment or it becomes slow after a while?

(meir) #4

First Thanks for the fast reply ,

regarding your question ,

  1. our elasticsearch version is 5.6.5 and ROR “version”: “1.16.14”,
  2. I dont have a lot of block in the ACL about 10-15
  3. in the first query it’s takes ~30-~1:30 then it become little bit better because of the cache

we just want to rule out that the performance issue is came from ROR plugin


(Simone Scarduzio) #5

Can you tell if the slow down is reflected on high CPU usage? Or high memory consumption?


(meir) #6

In the beginning we had memory consumption of 100% that have been solved,
Today our elk env’ using ~10-~20% memory usage and ~30% of CPU utilization


(Simone Scarduzio) #7

OK but can you associate any increment on resources usage to the presence of ROR plugin? Can you repeat the test with/without ROR and see if the response delay is associated with a resources leak?

Another thing you should inspect is if you have external authentication connectors like LDAP. And if the authentication server is responding in a timely fashion, or introducing the delay.


(meir) #8

Hi @sscarduzio,

We want to isolate the problem by checking several ways to solve it.
I started to check if there impact on our Vcenter Datastore and if the query’s have problem in the storage side.
I will update when I will finish my analysis.

Thanks,
Meir


(Simone Scarduzio) #9

Thanks for the update @meirfi! Will stay tuned.


(meir) #10

Hi @sscarduzio ,

Just want to update about the performance issue that I had.
There isn’t a lot of articles about fine tuning for elasticsearch on virtual machine so …
I had to dig to and do system modification to get better performance and its working very well.

button line is that I didn’t found any correlation between the authentication the performance issue

Thanks,
Meir.


(Simone Scarduzio) #11

Thank you @meirfi!
Glad everything is working now, and much appreciated that you took the time to report on this. :raised_hands: