Sure, understood, i will explain this to the customer and try to convince him to purcharse Enterprise version, because of the support and cool kibana features.
Thank you.
Hi Simone.
I am heading the Information Retrieval Solutions team for a large Insurance company and as Sinedko explained to you we are currently using the Free version.
Now we are evaluating to buy the Enterprise license but before to make the decision it will help us to have more clarity on the issue Sinedko showed you, I see that you put free version under low priority but the issue is preventing us to move forward with the evaluation of the licensed one.
In order to give you more context: we are going to have different deployment: one already in place that requires LDAP and we are fine, one in progress that requires JWT and here we have the issue.
If at the end you will find the time to give us some heads up, it will be very appreciated.
Beside that, as I said we are in the process to buy the Enterprise license, for a couple of time I submitted a request on your portal to get a quote and be in contact for further questions but none replied to us.
In that respect, I would ask you if you can contact me, based on the request I already sent you or directly getting my contact from this blog.
Thank you a lot.
Kind regards,
Paolo.
Hi Paolo, sorry about the inconvenience, turns out our contact form had an issue. Just answering this thread for completeness, as we already have an email dialogue ongoing. Letās continue over there.
Hello just for info, this issues is also present on 1.22.1, also that _cat/indices and _cat/aliases, i just updated the version and the bug is still there. Its quite annonying, because i cannot check aliases or all indices that certain user have.
We are considering to buy license but if those issues are present we are not very keen to do it. Hope you understand us.
Hi @Sinedko. We have scheduled the _cat/indices fix for the next week.
Please check out this build, because I think _cat/aliases may be already solved:
i tested it now, but no changes, the _cat/aliases still respond with blank response, but the good thing is the JWT issue is solved, elastic started without any issues with JWT block
ok, i was happy, but the issue is there again, i did only one restart of es bevause of some change in readonlyrest and it appears again, so i cannot start es again when the jwt block is present
ok, noted. As I said - next week someoneās going to look at it
@Sinedko this weekend is release time for us, we will have this bumped in priority first thing next week.
Thanks for the patience.
Hi @Sinedko, Iāve been assigned to work on your issue with JWT token. I tried to reproduce this issue with provided Dockerfile and configuration files, unfortunately without any success. readonlyrest.yml had commented out JWT TEST section, but even after uncommenting it everything worked fine for me. Because some parts were still missing, like for example env file, I had to fill these gaps with some dummy data. I thought about how can we separate other factors and at the same time work on the same configuration. Because you are already using docker I created self-sufficient docker compose bundle based on what you sent. If you arenāt familiar with docker compose and you donāt have it on your system hereās page with instructions how to install it Overview | Docker Documentation. Please run ROR from this bundle and check if it still hangs on Loading ReadonlyREST settings from file:.
Here are instructions how to run it:
- Download this bundle ror-dc-bundle.zip (31.1 KB). It contains everything you sent + dummy data in env file + generated certs.
- Extract this archive in some convenient for you directory.
- Copy
readonlyrest-1.23.0-pre7_es7.7.1.zipintopluginsdirectory of extracted archive. Itās the same build coutoPL mentioned above. https://readonlyrest-data.s3.amazonaws.com/build/1.23.0-pre7/readonlyrest-1.23.0-pre7_es7.7.1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA5SJIWBO54AGBERLX/20200916/eu-west-1/s3/aws4_request&X-Amz-Date=20200916T190734Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=f38abd79039f658be552e7b042b97fd4b57a699c583e1917d80b7b0e10e7c1f5 - Open terminal and go to root directory of extracted archive, where
docker-compose.ymlfile is located. - Run
docker-compose up -d. It will build image using Dockerfile and run it in the background.
You donāt have to create data and log directory, they will be created by docker compose in the same directory where docker-compose.yml file is located. Additionally your previous data and logs wonāt be overwritten as this compose is configured to work and modify data only inside of directory where docker-compose.yml file is located.
After you run this bundle please check and let me know if ES still hangs on Loading ReadonlyREST settings from file:. Please also let me know if you have additional questions.
Hello, thank you for your reply.
Basically we are using the same config, except that i dont use docker compose so i used my docker-run script.
If i try to first time run it, then its fine, but if i do the restart of the elasticsearch with deleteing and starting the container again, then it happend, its stuck forever on loading readonlyrest configuration.
Its very confusing because sometimes its working and sometime its not, but right now i reproduced it, firstime i run the es with this config, everything fine, then i deleted container and recreate again, and the issue appears again and now after restart its still stuck, i dont understand what is happening.
After the stuck i removed the jwt acl block, then rebuilded image and run it again, its working fine, then i added the jwt block again, rebuild, run and its still stuck, i dont really understand why the first run was succesfull.
I tried to restart docker aswell, its still stuck. I tried to restart the machine, still stuck.
Docker host versions:
docker --version
Docker version 19.03.12, build 48a66213fe
uname -a
Linux ire-dev-elasticsearch-01 4.15.0-117-generic #118-Ubuntu SMP Fri Sep 4 20:02:41 UTC 2020 x86_64 x86_64 x86_64 GNU/LinuxThank you for explanation, but I still werenāt able to reproduce this issue. I will try to write script to test it on your machine. Can you tell me in the meantime if you are able to install docker compose on your system? It makes working with docker a little easier, but If itās not possible I will try to manage with just docker.
You dont need to write a script, i can install the docker compose but i dont use it, and anyway i think it will be the same way, because i can write 1:1 script to run the container as in docker compose, i already shared the script i think, but let me share again:
Feel free to edit it as you need.
# Fail if one of the commands fails
set -e
CERT_NAME=$(ls /opt/elasticsearch/certs | grep "p12" | sed 's/\.p12//g')
docker network inspect alibaba > /dev/null 2>&1 || docker network create alibaba
docker rm alibaba-elasticsearch -f 2>&1 || true
docker run -d --restart unless-stopped -p 9200:9200 -p 9300:9300 \
--net alibaba \
--env-file /opt/elasticsearch/env \
--ulimit nofile=65536:65536 --ulimit memlock=-1:-1 \
-v /opt/elasticsearch/certs/${CERT_NAME}.keystore.jks:/opt/elasticsearch/config/keystore.jks:ro \
-v /opt/elasticsearch/certs/${CERT_NAME}.truststore.jks:/opt/elasticsearch/config/truststore.jks:ro \
-v /opt/insightsre/elasticsearch:/opt/elasticsearch/data \
-v /var/log/insightsre/elasticsearch:/var/log/insightsre/elasticsearch \
--name alibaba-elasticsearch \
alibaba-elasticsearch:7.7.1
It will find the cert name on /opt/elasticsearch/cert of th p12 cert, then it will use the name for truststore and keystore cert and mount them, aswell the log dir and data dir is mounted and env file is being used.
It also removing the container if its present when you run the scirpt to ensure that container will be completly new and its creating docker network to be able to communicate with other containers.
So basically its 1:1 vs the docker compose, you need to only change the pathing in volumes in your local system.
Let me know if you cannot reproduce it still, if i follow the task i wrote then i can reproduce it everytime.
The problem with this is script is that it uses a lot of things which I donāt have access to, but could have impact on behaviour of ES. Iām aware that there could be some data that you donāt want to share. This is why I wanted to create self-sufficient bundle, to have separated, controlled environment, which could be easily shared. This is also the reason why I wanted to use docker compose as it makes it little easier to control such separation. It would be great to have script which could reproduce issue you are struggling with on such controlled environment.
Iāve created script which prepares something like sandbox using docker-compose. Please download it using git checkout git@github.com:pielas/ror-test-env-342.git and execute ./run_test.sh inside downloaded directory. It will set up sandboxed environment and check if ES started correctly.
Please share output of this script.
Hello,
Biggest issue of all issues is i cannot download anything on this company machines.
Why you wrote that you dont have access to some things in script ? Let me write the script for your case.
# Fail if one of the commands fails
set -e
docker network inspect alibaba > /dev/null 2>&1 || docker network create alibaba
docker rm alibaba-elasticsearch -f 2>&1 || true
docker run -d --restart unless-stopped -p 9200:9200 -p 9300:9300 \
--net alibaba \
--env-file ./env \
--ulimit nofile=65536:65536 --ulimit memlock=-1:-1 \
-v ./certs/keystore.jks:/opt/elasticsearch/config/keystore.jks:ro \
-v ./certs/truststore.jks:/opt/elasticsearch/config/truststore.jks:ro \
-v ./data:/opt/elasticsearch/data \
-v ./logs:/var/log/insightsre/elasticsearch \
--name alibaba-elasticsearch \
alibaba-elasticsearch:7.7.1
Please run the script in root folder of your bundle, you will have access to everything, and you are right i cannot provide you elasticsearch data, but anyway it will be the big bug then if some data will affect whole es while using ror.
And btw, your github link is not correct, i wanted to check on my different machine.
Ok, I will think about how we can work this out without need to download anything.
I meant that you are using your local keystore, local data directory and local env file which are out of my reach and they could potentially cause troubles. This is why I wanted you to run it with dummy data I provided to check if it still hangs.
Have you run this script on your machine with data Iāve sent with bundle? It fails for me on
docker: Error response from daemon: create ./certs/keystore.jks: "./certs/keystore.jks" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path. and AFAIK docker doesnāt allow to use relative paths(docker compose doesnāt have problems with that).
Ah sorry, it should be clone instead of checkout here is correct command git clone git@github.com:pielas/ror-test-env-342.git. It would be great if you could run it on your different machine and check if it works there.
Did you try to use your absolute path ? i donāt know what pathing and folder structure you are using 
Itās not hard to rewrite the path in that script, but here is script for you (please be in the project root folder when running):
#!/bin/bash
DIR=`pwd`
docker build ${DIR} -t alibaba-elasticsearch:7.7.1
docker rm alibaba-elasticsearch -f 2>&1 || true
docker run -d --restart unless-stopped -p 9200:9200 -p 9300:9300 \
--env-file ${DIR}/env \
--ulimit nofile=65536:65536 --ulimit memlock=-1:-1 \
-v ${DIR}/certs/keystore.jks:/opt/elasticsearch/config/keystore.jks:ro \
-v ${DIR}/certs/truststore.jks:/opt/elasticsearch/config/truststore.jks:ro \
-v ${DIR}/data:/opt/elasticsearch/data \
-v ${DIR}/log:/var/log/insightsre/elasticsearch \
--name alibaba-elasticsearch \
alibaba-elasticsearch:7.7.1
I can only test on macbook. I was able to clone the project with (i found your account on github)
git clone https://github.com/pielas/ror-test-env-342.git
I tested both solutions with docker compose and aswell with my script on mac and everything is fine even after 10 restarts, so i dont really understand what is happening on that ubuntu.
Do you use some internet check when loading jwt blocks? Because this machine is in china, so its probably dont have access to all internet (i know its silly, but that comes to my mind), but that is not a case i think, becasue few times its worked well.
Do you have any opinion why it can be stuck on ubuntu 18 ? I tried to restart docker, whole machine, remove all content and do it again, but still the same thing.
BTW after the testing, please remove the project from github, thank you.
I know itās not hard to rewrite that, but that wasnāt the case. I wanted to use exactly 1:1 scripts to eliminate other factors. This is why I wanted you to use my configuration as it was.
I checked running ES with ROR with internet connection disabled completely and it started correctly.
Unfortunately I donāt have any idea. I will think about it in the meantime, but If you now have working installation of ES with ROR can you check if you are able to reproduce this issue on your computer? Ideally in this sandbox environment. Maybe for example env file has some value which is causing trouble.
Ok, no problem.
There is some chance that enabling debug logging would help to find what is causing ES loading to hang. Please change rootLogger.level = info to rootLogger.level = debug in your log4j2.properties file and check if there are more entries in ROR log.
Hello,
i did the test again, i enabled jwt block after while, everything was working well, then i restarted the es and issue is back there, i enabled the debug log in root logger, but no debug log is on the log file, its ending with loading readonly rest confiuration and this is all in the log. But if i enabled the
logger.access_log_rolling.level = debug
I can see few debug logs there:
[2020-09-28T07:08:14,588][ERROR][tech.beshu.ror.es.services.EsIndexJsonContentService] [alibaba-dev-ire-all-01] Cannot get source of document [.readonlyrest ID=1]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:118) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:152) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:136) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:103) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:62) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) ~[elasticsearch-7.7.1.jar:7.7.1]
at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$apply$1(IndexLevelActionFilter.scala:84) ~[?:?]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[?:?]
at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:25) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:24) ~[?:?]
at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.scala:82) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:388) ~[elasticsearch-7.7.1.jar:7.7.1]
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:492) ~[elasticsearch-7.7.1.jar:7.7.1]
at tech.beshu.ror.es.services.EsIndexJsonContentService.$anonfun$sourceOf$1(EsIndexJsonContentService.scala:55) ~[?:?]
at monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:81) ~[?:?]
at monix.eval.internal.TaskRunLoop$.$anonfun$restartAsync$1(TaskRunLoop.scala:222) ~[?:?]
at monix.execution.internal.InterceptRunnable.run(InterceptRunnable.scala:27) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) ~[?:?]
[2020-09-28T07:08:15,994][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:08:15,997][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:08:15,997][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:08:16,029][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:08:16,035][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:08:17,534][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:08:34,617][INFO ][tech.beshu.ror.boot.Ror ] [alibaba-dev-ire-all-01] Loading ReadonlyREST settings from index failed: cannot find index
[2020-09-28T07:08:34,618][INFO ][tech.beshu.ror.boot.Ror ] [alibaba-dev-ire-all-01] Loading ReadonlyREST settings from file: /usr/share/elasticsearch/config/readonlyrest.yml
[2020-09-28T07:08:38,242][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:00,242][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:00,243][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:00,402][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:00,441][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:00,454][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:00,454][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:08,295][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:09:38,336][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:10:00,356][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:10:00,356][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:10:00,356][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:10:00,383][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:10:00,438][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
[2020-09-28T07:10:00,438][DEBUG][tech.beshu.ror.es.dlsfls.RoleIndexSearcherWrapper$] [alibaba-dev-ire-all-01] FLS: _fields not found in threadContext
One more thing we have slight differencies in env file, api is random 25 char string and the proxy_token is random 256 char string.
And i have also logstash connected to es, so there is probability that logstash will call es during the loading, can it be issue ?