Issues with JWT in 1.16.32

jbiel-myriad · January 2, 2019, 10:05pm

Hi. I’m attempting an upgrade of ES + ReadonlyREST and experiencing an problem with what seems to be JWT.

old: ES 6.4.3, RoR 1.16.29
new: ES 6.5.4, RoR 1.16.32

The issue that’s experienced is that startup hangs after the following log lines:

[2019-01-02T21:50:39,220][DEBUG][t.b.r.e.IndexLevelActionFilter] [host]Read data from /etc/elasticsearch/readonlyrest.yml
[2019-01-02T21:50:39,240][INFO ][t.b.r.e.IndexLevelActionFilter] [host]Settings observer refreshing..

readonlyrest.yml:

readonlyrest:
...
  access_control_rules:
  - name: admin
    kibana_access: rw
    jwt_auth:
      name: "jwt_provider_1"
      roles: ["writer"]

  jwt:
  - name: jwt_provider_1
    signature_key: "..."
    signature_algo: RSA
    user_claim: preferred_username

We have other config options/ACL rules in place (LDAP, basic auth, etc.) If the “admin” jwt auth section is commented out Elasticsearch/RoR starts up properly. Does anyone have any ideas how to further troubleshoot what’s going wrong here? We’ve tried a variety of changes to this config section to no avail. Prior to the upgrade the jwt_auth value was just a string with the name of the ACL (admin.) After reviewing the docs we changed it over to a hash/map. TYIA!

dominic.lear · January 5, 2019, 11:05pm

Hi, I’m also experiencing this issue. I’ve raised a github issue here.

I think the issue is unrelated to the ES version as i’m using 6.2.4 and on another computer i have an old version of RoR that works with the same config. I downloaded it around 23rd December.

sscarduzio · January 28, 2019, 11:10pm

Linking here some evolution on this case on GitHub

github.com/sscarduzio/elasticsearch-readonlyrest-plugin

JWT auth configuration in 6.2.4 hangs Elasticsearch

opened 09:35PM - 05 Jan 19 UTC

closed 02:12PM - 29 Jan 19 UTC

dominic-lear

Hi, im trying to set up JWT auth with elasticsearch with the below details but w…hen starting elasticsearch it hangs at `Settings observer refreshing`. I have this config working on another computer but that might be with slightly different versions or ROR/java. Can you provide any insight into what the issue may be? --- Java version ``` java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) ``` Elasticsearch version: 6.2.4 Readonlyrest version: https://readonlyrest-data.s3.amazonaws.com/build/1.16.32/readonlyrest-1.16.32_es6.2.4.zip Readonlyrest config ``` readonlyrest: access_control_rules: - name: "BLock 1 - Allow" type: allow hosts: ["127.0.0.1", "192.168.10.1", "192.168.10.190"] jwt_auth: name: "jwt_provider_1" roles: ["user"] - name: "Block 2" type: forbid jwt: - name: jwt_provider_1 signature_algo: HMAC # can be NONE, RSA, HMAC (default), and EC signature_key: "my-key" user_claim: user roles_claim: roles # JSON-path style header_name: Authorization ssl: keystore_file: "keystore.jks" keystore_pass: readonlyrest key_pass: readonlyrest ``` When starting Elasticsearch it hangs at ``` 2019-01-05T21:29:59,570][INFO ][o.e.n.Node ] [rate-search-local-1] initializing ... [2019-01-05T21:29:59,693][INFO ][o.e.e.NodeEnvironment ] [rate-search-local-1] using [1] data paths, mounts [[/ (/dev/mapper/vagrant--vg-root)]], net usable_space [51.1gb], net total_space [61.7gb], types [ext4] [2019-01-05T21:29:59,704][INFO ][o.e.e.NodeEnvironment ] [rate-search-local-1] heap size [247.6mb], compressed ordinary object pointers [true] [2019-01-05T21:29:59,724][INFO ][o.e.n.Node ] [rate-search-local-1] node name [rate-search-local-1], node ID [PDJznGXUTuSjmqdLDso3Yg] [2019-01-05T21:29:59,726][INFO ][o.e.n.Node ] [rate-search-local-1] version[6.2.4], pid[5601], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/4.15.0-32-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_191/25.191-b12] [2019-01-05T21:29:59,728][INFO ][o.e.n.Node ] [rate-search-local-1] JVM arguments [-Xms256m, -Xmx256m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.enforce.bootstrap.checks=true, -Des.path.home=/home/vagrant/elasticsearch, -Des.path.conf=/home/vagrant/elasticsearch/config] [2019-01-05T21:30:04,429][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [aggs-matrix-stats] [2019-01-05T21:30:04,430][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [analysis-common] [2019-01-05T21:30:04,431][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [ingest-common] [2019-01-05T21:30:04,433][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [lang-expression] [2019-01-05T21:30:04,434][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [lang-mustache] [2019-01-05T21:30:04,435][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [lang-painless] [2019-01-05T21:30:04,437][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [mapper-extras] [2019-01-05T21:30:04,437][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [parent-join] [2019-01-05T21:30:04,438][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [percolator] [2019-01-05T21:30:04,439][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [rank-eval] [2019-01-05T21:30:04,440][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [reindex] [2019-01-05T21:30:04,441][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [repository-url] [2019-01-05T21:30:04,441][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [transport-netty4] [2019-01-05T21:30:04,441][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded module [tribe] [2019-01-05T21:30:04,442][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [readonlyrest] [2019-01-05T21:30:04,447][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-core] [2019-01-05T21:30:04,448][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-deprecation] [2019-01-05T21:30:04,450][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-graph] [2019-01-05T21:30:04,450][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-logstash] [2019-01-05T21:30:04,450][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-ml] [2019-01-05T21:30:04,451][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-monitoring] [2019-01-05T21:30:04,451][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-security] [2019-01-05T21:30:04,452][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-upgrade] [2019-01-05T21:30:04,453][INFO ][o.e.p.PluginsService ] [rate-search-local-1] loaded plugin [x-pack-watcher] [2019-01-05T21:30:11,111][INFO ][t.b.r.e.IndexLevelActionFilter] [rate-search-local-1] Settings observer refreshing... ``` it never gets past `Settings observer refreshing` if i remove the lines below, elasticsearch does start correctly. ``` jwt_auth: name: "jwt_provider_1" roles: ["user"] ```