Issues with letsencrypt certs from dehydrated: `cURL error 60: SSL certificate problem: unable to get local issuer certificate`

We are having issues setting up RoR for elasticsearch with nextcloud. I followed the docker related instructions provided here and went on with the encryption part here.

  1. I will give some details about my configs
  2. I will describe the problem
  3. Some open/remaining questions

1. Configs

I am getting my certs via dehydrated from letsencrypt, which gives me:

ls -lha /var/lib/dehydrated/certs/mydomain.com/

drwxr-xr-x 2 1000 root 4.0K Dec 5 15:55 .
drwx------ 3 root root 4.0K Nov 25 13:54 ..
-rwxr-xr-x 1 1000 root 554 Nov 25 12:37 cert-1764070647.csr
-rwxr-xr-x 1 1000 root 1.4K Nov 25 12:37 cert-1764070647.pem
lrwxrwxrwx 1 1000 root 19 Nov 25 12:37 cert.csr → cert-1764070647.csr
lrwxrwxrwx 1 1000 root 19 Nov 25 12:37 cert.pem → cert-1764070647.pem
-rwxr-xr-x 1 1000 root 1.6K Nov 25 12:37 chain-1764070647.pem
lrwxrwxrwx 1 1000 root 20 Nov 25 12:37 chain.pem → chain-1764070647.pem
-rwxr-xr-x 1 1000 root 2.9K Nov 25 12:37 fullchain-1764070647.pem
lrwxrwxrwx 1 1000 root 24 Nov 25 12:37 fullchain.pem → fullchain-1764070647.pem
-rw------- 1 1000 root 306 Dec 5 15:46 privkey-1764070647.pem
lrwxrwxrwx 1 1000 root 22 Nov 25 12:37 privkey.pem → privkey-1764070647.pem

I set the ownership by:

sudo chown -R 1000:0 /var/lib/dehydrated/certs/mydomain.com/

I will probably have to changes this later, but for know this is fine.

I mount the certs (and the other 2 configs) in docker-compose.yml, by:

docker-compose.yml
volumes:
  - type: bind
    source: /var/lib/dehydrated/certs/mydomain.com/
    target: /usr/share/elasticsearch/config/certs
  - type: bind
    source: config/readonlyrest.yml
    target: /usr/share/elasticsearch/config/readonlyrest.yml
  - type: bind
    source: config/elasticsearch.yml
    target: /usr/share/elasticsearch/config/elasticsearch.yml
config/readonlyrest.yml

readonlyrest:
access_control_rules:

  • name: “Require HTTP Basic Auth”
    type: allow
    auth_key: user:password

ssl:
enabled: true
server_certificate_file: certs/cert.pem
server_certificate_key_file: certs/privkey.pem

config/elasticsearch.yml

network.host: 0.0.0.0

Enable security features

xpack.security.enabled: false

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

#xpack.security.http.ssl:

enabled: false

keystore.path: cert

Enable encryption and mutual authentication between cluster nodes

#xpack.security.transport.ssl:

enabled: false

verification_mode: certificate

keystore.path: certs/transport.p12

truststore.path: certs/transport.p12

http.type: ssl_netty4

When pulling up the container, I get:

$ docker logs root-es-ror-1
"log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"042c1555b319","elasticsearch.cluster.name":"elasticsearch","error.type":"tech.beshu.ror.utils.SSLCertHelper$UnableToInitializeSslContextBuilderUsingProvidedFiles","error.message":"Unable to initialize Key Manager Factory using provided keystore.","error.stack_trace":"tech.beshu.ror.utils.SSLCertHelper$UnableToInitializeSslContextBuilderUsingProvidedFiles: Unable to initialize Key Manager Factory using provided keystore.\n\tat tech.beshu.ror.utils.SSLCertHelper$UnableToInitializeSslContextBuilderUsingProvidedFiles$.apply(SSLCertHelper.scala:367)\n\tat tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext$$anonfun$1(SSLCertHelper.scala:136)\n\tat unsafeRunSync @ tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext(SSLCertHelper.scala:112)\n\tat unsafeRunSync @ tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext(SSLCertHelper.scala:112)\n\tat unsafeRunSync @ tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext(SSLCertHelper.scala:112)\n\tat unsafeRunSync @ tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext(SSLCertHelper.scala:112)\n\tat apply @ tech.beshu.ror.utils.SSLCertHelper$.loadPrivateKey$$anonfun$2(SSLCertHelper.scala:280)\n\tat unsafeRunSync @ tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext(SSLCertHelper.scala:112)\n\tat apply @ tech.beshu.ror.utils.SSLCertHelper$.loadPrivateKey(SSLCertHelper.scala:273)\n\tat fromAutoCloseable @ tech.beshu.ror.utils.SSLCertHelper$.loadPrivateKey(SSLCertHelper.scala:273)\n\tat fromAutoCloseable @ tech.beshu.ror.utils.SSLCertHelper$.loadPrivateKey(SSLCertHelper.scala:273)\n\tat use @ tech.beshu.ror.utils.SSLCertHelper$.loadPrivateKey(SSLCertHelper.scala:281)\n\tat flatMap @ tech.beshu.ror.utils.SSLCertHelper$.getPrivateKeyAndCertificateChainFromPemFiles(SSLCertHelper.scala:266)\n\tat map @ tech.beshu.ror.utils.SSLCertHelper$.prepareSslContextBuilder(SSLCertHelper.scala:346)\n\tat map @ tech.beshu.ror.utils.SSLCertHelper$.prepareServerSSLContext(SSLCertHelper.scala:112)\nCaused by: java.lang.IllegalArgumentException: unknown object in getInstance: org.bouncycastle.asn1.ASN1ObjectIdentifier\n\tat org.bouncycastle.asn1.ASN1Sequence.getInstance(Unknown Source)\n\tat org.bouncycastle.asn1.pkcs.PrivateKeyInfo.getInstance(Unknown Source)\n\tat tech.beshu.ror.utils.SSLCertHelper$.loadPrivateKey$$anonfun$2$$anonfun$1(SSLCertHelper.scala:277)\n\t... 9 more\n"}
es-ror-1  | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch.log
es-ror-1  | 
es-ror-1  | ERROR: Elasticsearch died while starting up, with exit code 1

Asking Google I came up with converting the privkey to pks8, with:

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in certselastic/privkey-1764070647.pem -out certselastic/privkey-pkcs8.pem

Because privkey.pem is a softlink, I converted the real file.

When I now set server_certificate_key_file: to the converted privkey-pks8.pem the container starts up as intended.

I configured the Fulltextsearch App in Nextcloud with: https:// user:password@mydomain .com:9200/

2. Problem

I try to do the first indexing, with sudo -u www-data php8.3 /var/www/nextcloud/occ fulltextsearch:index and get:

cURL error 60: SSL certificate problem: unable to get local issuer certificate (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)

But when I try https://user:password@mydomain.com:9200/ in firefox, or chrome, the certificate gets accepted! They show the letsencrypt certificate and say it’s fine.

I tried the same with openssl s_client -connect ``mydomain.com:9200 and there I also get an Error:

$ openssl s_client -connect mydomain.com:9200
CONNECTED(00000003)
depth=0 CN = mydomain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mydomain.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = mydomain.com
verify return:1

while it works with all our other domains. I found suggestions to point curl on the nextcloud server to /ets/openssl/ca-certificates.crt, which I did (in all possible php configs, although I now I am using php8.3):

/etc/php/8.3/cli/php.ini:curl.cainfo = /etc/ssl/certs/ca-certificates.crt
/etc/php/8.3/fpm/php.ini:curl.cainfo = /etc/ssl/certs/ca-certificates.crt
/etc/php/8.3/phpdbg/php.ini:curl.cainfo = /etc/ssl/certs/ca-certificates.crt
/etc/php/8.4/cli/php.ini:curl.cainfo = /etc/ssl/certs/ca-certificates.crt
/etc/php/8.4/fpm/php.ini:curl.cainfo = /etc/ssl/certs/ca-certificates.crt

Questions

  • What type of .pem files are supported here? Is the problem, that I converted the original privkey.pem and now the RoR plugin accepts it, but the clients have issues with accepting it?
  • Why can’t RoR just deal with the certs provided by dehydrated?

We would really like to avoid self-signed certificates, any help would be appreciated :slight_smile:

Hello @TU-AStA
Thanks for your message. We’ll look into this further and try to reproduce the problem on our side. We’ll get back to you with more details as soon as we have them.

1 Like