Permission denied for Kibana admin account

21

@cmh do you have any error messages?

22

Just sent to support email:

...
[2019-04-30T18:43:24,624][INFO ][o.e.p.PluginsService     ] [elastic.example.com] loaded plugin [readonlyrest]
[2019-04-30T18:43:28,935][WARN ][t.b.r.e.ReadonlyRestPlugin] [elastic.example.com][ReadonlyRestPlugin] could not check if had remote ES clusters: Failed to get setting group for [] setting prefix and setting [pidfile] because of a missing '.'
[2019-04-30T18:43:28,956][INFO ][t.b.r.e.IndexLevelActionFilter] [elastic.example.com] Settings observer refreshing...
[2019-04-30T18:43:31,746][ERROR][o.e.b.Bootstrap          ] elastic.example.com] Exception
java.lang.IllegalArgumentException: Predicate isEmpty() did not fail.
    at eu.timepit.refined.api.RefinedType.unsafeRefine(RefinedType.scala:32) ~[?:?]
    at eu.timepit.refined.api.RefinedType.unsafeRefine$(RefinedType.scala:29) ~[?:?]
    at eu.timepit.refined.api.RefinedType$$anon$1.unsafeRefine(RefinedType.scala:55) ~[?:?]
    at eu.timepit.refined.api.RefinedTypeOps.unsafeFrom(RefinedTypeOps.scala:41) ~[?:?]
...

Looks like the relevant part is:

could not check if had remote ES clusters: Failed to get setting group for [] setting prefix and setting [pidfile] because of a missing '.'

23

Ugh - just discovered one of my clusters was running a mix of 1.17.5 (one node) and 1.16.34. I guess that’s the “radically different codebase”.

1 Like
24

@cmh it’d be nice if you could show more logs (eg. error log is truncated) or configuration you’re trying to run (it’d be simpler for us to tell what is wrong, replicate error or fix error message)

25

The full readonlyrest.yml is posted above in comment #7 (Permission denied for Kibana admin account - #7 by cmh) and is still the same.

I could post more of the logfiles, but it’s troublesome to have to go through and sanitize them and hope I got everything. I looked through them, and that failed to get setting looked to be the only really relevant part, didn’t thin the full stack was necessary.

26

@cmh we should provide better error message, but I see that this is the problem:

This value cannot be empty string.

1 Like
27

So that config works in 1.17.5 but causes the cluster to not be able to start in 1.17.6?

28

@cmh we’ve improved config validation at startup time. Before 1.17.6 some validations were done at request handling level. Now, ROR does them at config loading level (before ROR starts). If ROR finds invalid configuration, it won’t start.

1 Like
29

Is there some sort of config test that can be run to verify the ROR config before trying a restart?

I’ll agree there! :smiley:

I’ve got an opinion on a valid config option having an empty value being a hard error that would completely prevent the cluster from starting, but okay. I think we might have found the issue, my testing so far is looking promising. Thanks for the help!

30

@cmh IMO all config errors should be found as soon as possible. For you it might not be “a hard error”, but for another user can.

@sscarduzio I don’t know what versioning rules are you using, but maybe this version should be 1.18.0 to avoid such doubts?

31

Thus my question about a config test option, such as this one for logstash:

logstash --path.settings /etc/logstash --config.test_and_exit -f $file

Best to be able to test for bad config before submitting the restart to reload it.

2 Likes
32

@cmh yep, we don’t have it yet, but also we have a plan to introduce it in future

2 Likes
33

At the moment, we have the YAML checked in the hot reload API: if your settings fail checks, the old one will be kept and an error is returned.

Actually makes a lot of sense, we should have bumped the 17 away when we moved to the new Scala core TBH.