SAML - Could not login in: Empty metadata

Hi.

After configuring SAML integration in Kibana, when trying to authenticate, I have the following error:

[00:23:30:843] [info][plugins][ReadonlyREST][authController] Could not login in: Empty metadata

This is the kibana configuration related to SAML:

readonlyrest_kbn.auth:
  signature_key: "key"
  saml_serv1:
    enabled: true
    type: saml
    issuer: https://vkibana.obs
    buttonName: "Keycloak"
    entryPoint: 'https://vkeycloak.obs/realms/LDAP/protocol/saml'
    kibanaExternalHost: 'vkibana.obs'
    protocol: https
    usernameParameter: 'NameID'
    groupsParameter: 'memberOf'
    logoutUrl: 'https://vkeycloak.obs/realms/LDAP/protocol/saml/resolve'
    cert: 'certificate'

And this is the elasticsearch configuration:

  - name: "ReadonlyREST Enterprise instance #1"
    ror_kbn_auth:
      name: "kbn1"

  ror_kbn:
  - name: kbn1
    signature_key: "key"

This is a sample of the SAML Response sent by the IdP:

<samlp:Response xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol"
                xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion"
                Destination="https://vkibana.obs/ror_kbn_saml_serv1/assert"
                ID="ID_6a96d1f2-a3c7-41cf-90f7-494ac80e4a36"
                InResponseTo="_f9cc74a0c7103dbbb003"
                IssueInstant="2022-09-25T00:23:30.656Z"
                Version="2.0"
                >
    <saml:Issuer>https://vkeycloak.obs/realms/LDAP</saml:Issuer>
    <dsig:Signature xmlns:dsig="http://www.w3.org/2000/09/xmldsig#">
        <dsig:SignedInfo>
            <dsig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
            <dsig:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />
            <dsig:Reference URI="#ID_6a96d1f2-a3c7-41cf-90f7-494ac80e4a36">
                <dsig:Transforms>
                    <dsig:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
                    <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                </dsig:Transforms>
                <dsig:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />
                <dsig:DigestValue>digest</dsig:DigestValue>
            </dsig:Reference>
        </dsig:SignedInfo>
        <dsig:SignatureValue>signature</dsig:SignatureValue>
        <dsig:KeyInfo>
            <dsig:KeyName>D-Iv4jANhgLQVYAv_a0OhHeEh8EQA0uViZHjGXlbdQA</dsig:KeyName>
            <dsig:X509Data>
                <dsig:X509Certificate>certificate</dsig:X509Certificate>
            </dsig:X509Data>
            <dsig:KeyValue>
                <dsig:RSAKeyValue>
                    <dsig:Modulus>key</dsig:Modulus>
                    <dsig:Exponent>AQAB</dsig:Exponent>
                </dsig:RSAKeyValue>
            </dsig:KeyValue>
        </dsig:KeyInfo>
    </dsig:Signature>
    <samlp:Status>
        <samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success" />
    </samlp:Status>
    <saml:Assertion xmlns="urn:oasis:names:tc:SAML:2.0:assertion"
                    ID="ID_cc5a8457-ac9c-4902-9c3c-69155e98bffe"
                    IssueInstant="2022-09-25T00:23:30.656Z"
                    Version="2.0"
                    >
        <saml:Issuer>https://vkeycloak.obs/realms/LDAP</saml:Issuer>
        <saml:Subject>
            <saml:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">gustavo.yoshizaki@avature.net</saml:NameID>
            <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                <saml:SubjectConfirmationData InResponseTo="_f9cc74a0c7103dbbb003"
                                              NotOnOrAfter="2022-09-25T00:28:28.656Z"
                                              Recipient="https://vkibana.obs/ror_kbn_saml_serv1/assert"
                                              />
            </saml:SubjectConfirmation>
        </saml:Subject>
        <saml:Conditions NotBefore="2022-09-25T00:23:28.656Z"
                         NotOnOrAfter="2022-09-25T00:24:28.656Z"
                         >
            <saml:AudienceRestriction>
                <saml:Audience>https://vkibana.obs</saml:Audience>
            </saml:AudienceRestriction>
        </saml:Conditions>
        <saml:AuthnStatement AuthnInstant="2022-09-25T00:23:30.657Z"
                             SessionIndex="467f57d8-53d6-493e-9925-55e76764d452::e28803c3-3242-4c36-bb0d-88f0b29afea0"
                             SessionNotOnOrAfter="2022-09-25T10:23:30.657Z"
                             >
            <saml:AuthnContext>
                <saml:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:unspecified</saml:AuthnContextClassRef>
            </saml:AuthnContext>
        </saml:AuthnStatement>
        <saml:AttributeStatement>
            <saml:Attribute Name="Role"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >manage-account-links</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="Role"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >uma_authorization</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="Role"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >view-profile</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="Role"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >manage-account</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="Role"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >offline_access</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="Role"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >default-roles-ldap</saml:AttributeValue>
            </saml:Attribute>
        </saml:AttributeStatement>
    </saml:Assertion>
</samlp:Response>

I also tried using saml:NameID as usernameParameter without success.

Thanks in advance.

“empty metadata” error message is presented in ReadonlyREST Free/PRO/Enterprise when the login request is checked by the ACL and gets accepted by an ACL block with no authentication rule in it.

It would be useful to see the whole ACL, you might have a catch all block with a hosts rule in it, or something similar?

This is the initial configuration file used on a clean environment:

---
readonlyrest:
  audit_collector: true
  audit_index_template: "'readonlyrest_audit'-yyyy"
  prompt_for_basic_auth: false
  ssl:
    enable: true
    keystore_file: keystore.jks
    keystore_pass: pass
    key_pass: pass
  response_if_req_forbidden: Access forbidden
  access_control_rules:
  - name: Logtash
    auth_key_sha256: hash
    kibana_access: admin
    verbosity: error
  - name: Kibana
    auth_key_sha256: hash
    kibana_access: rw
    verbosity: error
  - name: No access allowed to ReadonlyRest
    type: forbid
    indices:
    - ".readonlyrest"
  - name: "ReadonlyREST Enterprise instance #1"
    ror_kbn_auth:
      name: "kbn1"

  ror_kbn:
  - name: kbn1
    signature_key: "key"

With this, Kibana will not start with the following error:

  log   [16:36:55.754] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  log   [16:36:55.754] [info][savedobjects-service] Starting saved objects migrations
  log   [16:36:55.980] [info][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 29ms.
  log   [16:36:55.983] [info][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 30ms.
  log   [16:36:56.036] [error][savedobjects-service] [.kibana] Action failed with 'forbidden_response: [forbidden_response] Reason: Access forbidden'. Retrying attempt 1 in 2 seconds.
  log   [16:36:56.037] [info][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 56ms.
  log   [16:36:56.038] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'forbidden_response: [forbidden_response] Reason: Access forbidden'. Retrying attempt 1 in 2 seconds.
  log   [16:36:56.039] [info][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 55ms.
  log   [16:36:58.082] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'forbidden_response: [forbidden_response] Reason: Access forbidden'. Retrying attempt 2 in 4 seconds.
  log   [16:36:58.083] [info][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 2044ms.
  log   [16:36:58.090] [error][savedobjects-service] [.kibana] Action failed with 'forbidden_response: [forbidden_response] Reason: Access forbidden'. Retrying attempt 2 in 4 seconds.
  log   [16:36:58.090] [info][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 2054ms.
  log   [16:37:02.135] [error][savedobjects-service] [.kibana] Action failed with 'forbidden_response: [forbidden_response] Reason: Access forbidden'. Retrying attempt 3 in 8 seconds.
  log   [16:37:02.135] [info][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 4045ms.
  log   [16:37:02.137] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'forbidden_response: [forbidden_response] Reason: Access forbidden'. Retrying attempt 3 in 8 seconds.
  log   [16:37:02.137] [info][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 4055ms.
  log   [16:37:10.142] [error][savedobjects-service] [.kibana] Action failed with 'Elasticsearch host https://velasticsearch:9200 is unavailable'. Retrying attempt 4 in 16 seconds.
  log   [16:37:10.142] [info][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 8007ms.
  log   [16:37:10.144] [error][savedobjects-service] [.kibana_task_manager] Action failed with 'Elasticsearch host https://velasticsearch:9200 is unavailable'. Retrying attempt 4 in 16 seconds.
  log   [16:37:10.144] [info][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 8007ms.

After adding this ACL on the end of the list:

...
  - name: "ReadonlyREST Enterprise instance #1"
    ror_kbn_auth:
      name: "kbn1"
  - name: Kibana indexes
    type: allow
    indices:
    - ".management-beats*"
    - ".kibana*"
    - ".tasks*"
    - wazuh-*
    verbosity: error

  ror_kbn:
...

Kibana was able to create the indexes:

  log   [16:38:59.687] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  log   [16:38:59.687] [info][savedobjects-service] Starting saved objects migrations
  log   [16:38:59.952] [info][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET. took: 30ms.
  log   [16:38:59.954] [info][savedobjects-service] [.kibana] INIT -> CREATE_NEW_TARGET. took: 34ms.
  log   [16:39:00.230] [info][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 278ms.
  log   [16:39:00.270] [info][savedobjects-service] [.kibana] CREATE_NEW_TARGET -> MARK_VERSION_INDEX_READY. took: 316ms.
  log   [16:39:00.309] [info][savedobjects-service] [.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 79ms.
  log   [16:39:00.309] [info][savedobjects-service] [.kibana_task_manager] Migration completed after 387ms
  log   [16:39:00.341] [info][savedobjects-service] [.kibana] MARK_VERSION_INDEX_READY -> DONE. took: 71ms.
  log   [16:39:00.342] [info][savedobjects-service] [.kibana] Migration completed after 422ms

But when I try accessing using SSO, the Empty metadata error appears:

[16:40:22:272] [info][plugins][ReadonlyREST][authController] Could not login in: Empty metadata
[16:40:23:207] [info][plugins][ReadonlyREST][LicenseService] >> Cannot get the cluster stats. Got status: 403

And if I remove the last ACL block, the error changes to:

[16:42:21:458] [error][plugins][ReadonlyREST][esClient] ES Authorization error: 403 Error: ES Authorization error: 403
    at l.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/core/esClient.js:1:16663)
    at l.e (/usr/share/kibana/plugins/readonlyrestkbn/proxy/core/esClient.js:1:5171)
    at tryCatch (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:45:40)
    at Generator.invoke [as _invoke] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:274:22)
    at Generator.prototype.<computed> [as next] (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/regenerator-runtime/runtime.js:97:21)
    at asyncGeneratorStep (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/usr/share/kibana/plugins/readonlyrestkbn/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
[16:42:21:459] [info][plugins][ReadonlyREST][authController] Could not login in: Wrong credentials

The ACL is wrong:


---
readonlyrest:
  audit_collector: true
  audit_index_template: "'readonlyrest_audit'-yyyy"
  prompt_for_basic_auth: false
  ssl:
    enable: true
    keystore_file: keystore.jks
    keystore_pass: pass
    key_pass: pass
  response_if_req_forbidden: Access forbidden
 
  access_control_rules:
  - name: Logtash
    auth_key_sha256: hash
    kibana_access: admin
    verbosity: error
  - name: Kibana  # <- this block should be used by Kibana server and requires unrestricted access
    auth_key_sha256: hash
    kibana_access: rw # <- delete this!
    verbosity: error
  - name: No access allowed to ReadonlyRest
    type: forbid
    indices:
    - ".readonlyrest"
  - name: "ReadonlyREST Enterprise instance #1"
    ror_kbn_auth:
      name: "kbn1"

  ror_kbn:
  - name: kbn1
    signature_key: "key"

The rest looks fine.

PS: remove the block you added

  - name: Kibana indexes  # <- delete the whole block, no  need!
    type: allow
    indices:
    - ".management-beats*"
    - ".kibana*"
    - ".tasks*"
    - wazuh-*
    verbosity: error

This change:

    kibana_access: rw # <- delete this!

Solved the problem of a new environent and this block is no longer required:

  - name: Kibana indexes  # <- delete the whole block, no  need!
    type: allow
    indices:
    - ".management-beats*"
    - ".kibana*"
    - ".tasks*"
    - wazuh-*
    verbosity: error

However, it not solved the Empty metadata error.

This is the full ROR configuration:

---
readonlyrest:
  audit_collector: true
  audit_index_template: "'readonlyrest_audit'-yyyy"
  prompt_for_basic_auth: false
  ssl:
    enable: true
    keystore_file: keystore.jks
    keystore_pass: changeme
    key_pass: changeme
  response_if_req_forbidden: Access forbidden
  access_control_rules:
  - name: Logtash
    auth_key_sha256: hash
    kibana_access: admin
    verbosity: error
  - name: Kibana
    auth_key_sha256: hash
    verbosity: error
  - name: No access allowed to ReadonlyRest
    type: forbid
    indices:
    - ".readonlyrest"
  - name: "ReadonlyREST Enterprise instance #1"
    ror_kbn_auth:
      name: "kbn1"

  ror_kbn:
  - name: kbn1
    signature_key: "key"

The kibana configuration did not change at all. Neither did the IdP SAML response.

OK let me analyse this configuration a bit more:

readonlyrest.yml

readonlyrest:
 
...
  - name: Logtash
    auth_key_sha256: hash
    kibana_access: admin # <- delete this line, it only make sense for browser sessions.
    verbosity: error
...

SAML assertion

Your SAML IdP is not giving you back the user name in the SAML assertion attributes list. As you can see it’s we can see all the LDAP groups listed in a series of <saml:Attribute Name="Role" [...] >, but I’d also expect to see something like: <saml:Attribute Name="uid" [...], or even <saml:Attribute Name="mail" [...] where we can pull a unique user identifier.

Maybe talk to the IdP administrator, ask if they can add some attributes to represent the user identifier.

    kibana_access: admin # <- delete this line, it only make sense for browser sessions.

Yes. I just added to be able to access Kibana and check the logs un the ROR index.

The IdP is sending the following top level assertion:

        <saml:Subject>
            <saml:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">gustavo.yoshizaki@avature.net</saml:NameID>
            <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                <saml:SubjectConfirmationData InResponseTo="_c034875e04c597d7b588"
                                              NotOnOrAfter="2022-09-26T16:01:00.565Z"
                                              Recipient="https://vkibana.obs/ror_kbn_saml_serv1/assert"
                                              />

image

It is also sending metadata such as the roles in the Attributes section (which I plan to use for authorization inside Kibana):

image

If I understood correctly, the plugin is only able to read the attributes section so the IdP should also add the saml:NameID in that section. Is that right?

The attribute username was added to the SAML response as well as the kibana.yaml update. However, the Empty metadata error still persists.

This is the new attribute:

            <saml:Attribute FriendlyName="username"
                            Name="username"
                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"
                            >
                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                     xsi:type="xs:string"
                                     >gustavo.yoshizaki</saml:AttributeValue>
            </saml:Attribute>

image

And this the kibana configuration:

readonlyrest_kbn.auth:
  signature_key: "key"
  saml_serv1:
    enabled: true
    type: saml
    issuer: https://vkibana.obs
    buttonName: "Keycloak"
    entryPoint: 'https://vkeycloak.obs/realms/LDAP/protocol/saml'
    kibanaExternalHost: 'vkibana.obs'
    protocol: https
    usernameParameter: 'username'
    groupsParameter: 'memberOf'
    logoutUrl: 'https://vkeycloak.obs/realms/LDAP/protocol/saml/resolve'
    cert: 'cert'

With the same result:

[16:54:34:464] [info][plugins][ReadonlyREST][authController] Could not login in: Empty metadata

Oh OK great! Then let’s try use it in the kibana.yml?

readonlyrest_kbn.auth:
  signature_key: "key"
  saml_serv1:
    [...]
    usernameParameter: 'username'
    groupsParameter: 'Role'
    [...]

You should have these two debug log lines in kibana.log you can grep:

“Obtained user name from profile”
“Obtained groups from profile”

This would verify if we are intercepting the username and groups right.

No luck. Still Empty metadata message.

@Dzuming could we have an issue with the SAML connector not picking up the assertion attributes?

I checked locally, and it works as expected

I did some tests and research myself on this.

The library we are using is doing a pre-parsing from XML to Javascript object which misses out on a lot of the content of the actual XML Assertion.

For example I have an assertion

<saml:Assertion
    xmlns="urn:oasis:names:tc:SAML:2.0:assertion" ID="ID_c8863275-83b0-4cad-80bf-015d01c905c3" IssueInstant="2022-09-27T10:10:16.168Z" Version="2.0"
    xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
    <saml:Issuer>http://localhost:8080/auth/realms/ror</saml:Issuer>
    <dsig:Signature
        xmlns:dsig="http://www.w3.org/2000/09/xmldsig#">
        <dsig:SignedInfo>
            <dsig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
            <dsig:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
            <dsig:Reference URI="#ID_c8863275-83b0-4cad-80bf-015d01c905c3">
                <dsig:Transforms>
                    <dsig:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                    <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
                </dsig:Transforms>
                <dsig:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
                <dsig:DigestValue>ZdPkh8ZhAJ2C48jQqd8vuz6cDeqkW/8nu0mA2oB2nkI=</dsig:DigestValue>
            </dsig:Reference>
        </dsig:SignedInfo>
        <dsig:SignatureValue>FJ+Z1YRjEAcMdMATPFvJCoPaToW3QlU58Yq0GbL5nWpxgF/6NLtuLtXwTGsCKEKDHrDqqnm8lNBiP2mXKgIX9fsVvjT0Qq0oSMkZ5Ujgv/k1sDa/LNAtqdoqSOvpERuF3Ol1gflX2YFTlqMP8kF0VGHGgFrLfJKJ8cnqxNiO3+pwrcgeGDmttyAYssqTZtmLned57TjoQX3OwU5S/C3465WxJC8xOEJSEDssI/SA46HoY8DII6l6AKm9Jx79CZ7uy252FptK8kT1ZB04nFY4+uKcfRh1+/yEB+nO5EyOVK/qGxVECtwXmaBv5n/GI8Wg2Vwx+BVMWUFXpLlp2KgpuA==</dsig:SignatureValue>
        <dsig:KeyInfo>
            <dsig:KeyName>o_8wOaFz5O7P9DOUkFedCdbSy6RAZs5Noytayao46EE</dsig:KeyName>
            <dsig:X509Data>
                <dsig:X509Certificate>MIIClTCCAX0CBgGAkyzPIDANBgkqhkiG9w0BAQsFADAOMQwwCgYDVQQDDANyb3IwHhcNMjIwNTA1MDc0MjA5WhcNMzIwNTA1MDc0MzQ5WjAOMQwwCgYDVQQDDANyb3IwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCigkWUUB/kHBe76u6CxDK/W2kbqCfcZbWrCrMpQH5ESQC5YqG/WzObusaw5lWiGThzqJPFVXxbDnS+uh56ROw27TpxsSMwgyD2GpW42yl2G0Jrf3HYKskmFnrcy2PMTml8DbyGC3v59RRxVv2XR32cE3BeiID4WFhIcsWjrKTjB+0U+rhrocRXG+dalxiT0dyE1wgwkcEGCZu8vnKRYmC5sazw9xDWAZVcN+SKQweRMWu4ZO6TcrtIERT8NrswUlqij0c1VbmmC3qraK5Lmfn8QCE7ZDH17o8n8HyGBoETnQUHcfL9fFHF/cD/+CnjWocBAX0rjLBnl5S3IuaWT78xAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAIgtUSteWw6ZVlpfV5ATkF1o9ZPcJj4KZ+BNfcB5UIcCPCONvO3GiHskCbHTSFSFtBEaxe1mLb2MEptqG0OVDOUtZtoOqWIc2KIObzkgBtOmLmt/modDlRbTzeoNwV4oiwz36VvoY1urxJYglUzQMQVJhEYPw5sxb/Yxj/4nUSgtF9ohVsEJCa0N+WSSykT87M1Afm/kKOO8hS7iGv7+SpSHfAXKQBHXQ4cLOuLoAs6OPPO0/U+XTdvEchXIrh24VPJAG+i5Y/kPkFbQciw8ULpMmwCSWH8W5vXAUfQMSJ47UkfD3TpzAZKNeSVGtwWv6i0EUrpXoPNC/+yuPuTWoQg=</dsig:X509Certificate>
            </dsig:X509Data>
            <dsig:KeyValue>
                <dsig:RSAKeyValue>
                    <dsig:Modulus>ooJFlFAf5BwXu+rugsQyv1tpG6gn3GW1qwqzKUB+REkAuWKhv1szm7rGsOZVohk4c6iTxVV8Ww50vroeekTsNu06cbEjMIMg9hqVuNspdhtCa39x2CrJJhZ63MtjzE5pfA28hgt7+fUUcVb9l0d9nBNwXoiA+FhYSHLFo6yk4wftFPq4a6HEVxvnWpcYk9HchNcIMJHBBgmbvL5ykWJgubGs8PcQ1gGVXDfkikMHkTFruGTuk3K7SBEU/Da7MFJaoo9HNVW5pgt6q2iuS5n5/EAhO2Qx9e6PJ/B8hgaBE50FB3Hy/XxRxf3A//gp41qHAQF9K4ywZ5eUtyLmlk+/MQ==</dsig:Modulus>
                    <dsig:Exponent>AQAB</dsig:Exponent>
                </dsig:RSAKeyValue>
            </dsig:KeyValue>
        </dsig:KeyInfo>
    </dsig:Signature>
    <saml:Subject>
        <saml:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">user@example.com</saml:NameID>
        <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
            <saml:SubjectConfirmationData InResponseTo="_d1530bfc03bce150029f" NotOnOrAfter="2022-09-27T10:15:14.168Z" Recipient="https://localhost:5601/ror_kbn_saml_kc/assert"/>
        </saml:SubjectConfirmation>
    </saml:Subject>
    <saml:Conditions NotBefore="2022-09-27T10:10:14.168Z" NotOnOrAfter="2022-09-27T10:11:14.168Z">
        <saml:AudienceRestriction>
            <saml:Audience>ror</saml:Audience>
        </saml:AudienceRestriction>
    </saml:Conditions>
    <saml:AuthnStatement AuthnInstant="2022-09-27T10:10:16.169Z" SessionIndex="8616bdc3-5ac0-48aa-9071-fe67a5d40cbb::7049966a-52ba-4cbf-86f2-e9929b7d19bd" SessionNotOnOrAfter="2022-09-27T20:10:16.169Z">
        <saml:AuthnContext>
            <saml:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:unspecified</saml:AuthnContextClassRef>
        </saml:AuthnContext>
    </saml:AuthnStatement>
    <saml:AttributeStatement>
        <saml:Attribute FriendlyName="surname" Name="urn:oid:2.5.4.4" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">ojoj
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute FriendlyName="givenName" Name="urn:oid:2.5.4.42" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">ijojo
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute FriendlyName="email" Name="urn:oid:1.2.840.113549.1.9.1" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">user@example.com
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">manage-account
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">offline_access
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">ror
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">manage-account-links
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">default-roles-ror
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">view-profile
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">ror-viewer
            </saml:AttributeValue>
        </saml:Attribute>
        <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
            <saml:AttributeValue
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">uma_authorization
            </saml:AttributeValue>
        </saml:Attribute>
    </saml:AttributeStatement>
</saml:Assertion>

The library parses it as JSON:

{
    "issuer": "http://localhost:8080/auth/realms/ror",
    "inResponseTo": "_0cc40a16e53306d5a5eb",
    "sessionIndex": "cd295e7c-d97d-4cca-a814-fc465c7b2baa::7049966a-52ba-4cbf-86f2-e9929b7d19bd",
    "nameID": "user@example.com",
    "nameIDFormat": "urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress",
    "urn:oid:2.5.4.4": "ojoj",
    "urn:oid:2.5.4.42": "ijojo",
    "urn:oid:1.2.840.113549.1.9.1": "user@example.com",
    "Role": "manage-account",
    "attributes":
    {
        "urn:oid:2.5.4.4": "ojoj",
        "urn:oid:2.5.4.42": "ijojo",
        "urn:oid:1.2.840.113549.1.9.1": "user@example.com",
        "Role": "uma_authorization"
    }
}

Notice how multiple “Role” XML elements are flattened into a single "Role": "uma_authorization" literally losing all the group names except the last one!

@gustavo.yoshizaki If you want you can grep the debug log line in kibana.log “Obtained raw profile” to see what is the badly parsed object ROR is seeing and trying to extract data from.


Proposed Fix

@Dzuming I propose we introduce:

  1. the possibility to query the raw XML assertion via XPath: let the user define the query in the configuration for username and groups.

  2. Introduce a debug log line that will output the raw XML assertion, so people can see it and design the XPath query

  3. deprecate the old usernameParameter and groupsParameter with a big warning in the logs

XPath is boring, I know. But it’s also well understood in the industry, and it’s native for XML.
The only annoying thing is that SAML assertion tags are namespaced, so they include a : colon character, and queries will look something like this:

Example

usernameXpath: '//*[name()='saml:NameID']/text()'
groupsXpath: '//*[name()='saml:Attribute'][@Name="Role"]/*/text()'`

Result

username = user@example.com
groups = [manage-account, offline_access, ror, manage-account-links, default-roles-ror, view-profile, ror-viewer, uma_authorization]

We need to write [name()=XXX] instead of /XXX/ due to the presence of the colon character that can’t be escaped.

Besides the technical improvements I jotted down above, @gustavo.yoshizaki let’s see if we can make the current version of ROR working for you anyway.

Can you try and grep the debug log line in kibana.log for the string Obtained raw profile?

BTW: our SAML assertions look similar on the username part, so these are the parameters I got working:

            usernameParameter: "nameID"
            groupsParameter: "Role" 

I found a workaround for letting the library parse multiple attributes with the same name!

In Keycloak, you switch on “Single Role Attribute”, so it outputs a single Attribute with multiple values, rather than repeating the same attribute with the same names N times.

In a call with @sscarduzio we found out that the problem is in the amount of data sent from Kibana to Elasticsearch:

[16:01:28:809] [trace][plugins][ReadonlyREST][esClient] Authorization attempt returned:  {
  error: {
    root_cause: [ [Object] ],
    type: 'too_long_frame_exception',
    reason: 'HTTP header is larger than 8192 bytes.'
  },
  status: 400
}

That is related to the amount of groups the authenticated user has.

Hi @gustavo.yoshizaki! Great debug session!
In the meantime, I found this configuration for Elasticsearch that could help:

http.max_header_size: 16kb # or more?

Hi @sscarduzio

It could be used for local testing of the plugin integration. But for production, where all the nodes of ES would have to be restarted and that there is no guarantee of the amount of groups sent, I would prefer changing the request to POST and not sending the data in the headers.

I’ve been thinking about the changing request to post thing. Unfortunately it’s not simple: Kibana and Elasticsearch communicate via HTTP, and because this is a stateless protocol, the credentials should be presented in every request’s header.

The only way to avoid this would be to create an API endpoint able to receive a POST with the JWT payload, save the metadata (including the long list of groups) to a server-side session (an index?) and respond with a session ID that will be presented by Kibana at every request.

This requires substantial changes to the product, and introduces a session cache index that has to be cleaned up periodically.

There’s another option: we could keep the authentication within the headers by compressing the authorization header and reduce it by at least 1/3, and/or split the long header value indefinitely in multiple headers.

What’s the order of magnitude of groups list length you will require?

Right now, we would need to map 300 indexes to the groups in a 1:1 relationship. Maybe we could use 1 group for a few similar indexes. But they will be a few cases.

Then we are planning to use more complex rules filtering the documents and/or the fields. That would make unpredictable the amount of groups to be sent.

It seems that we will have to reconsider the option http.max_header_size. But it will require a lot of synchronization to apply it in all the Elasticsearch nodes without affecting the service availability.

1 Like