Why?

If you have an existing PKI infrastructure, mutual TLS is a convenient way to do authentication and authorization. OpenSearch supports it, although the documentation is lacking in practical examples. Let's fix that!

Configuring OpenSearch Security

Files we're touching

This post uses the OpenSearch security YAML files for configuration. While the docs recommend "creat[ing] other users, roles, mappings, action groups, and tenants using OpenSearch Dashboards or the REST API," we're going against that advice to make things a bit more clear.

In a typical Deb or RPM installation, all files listed are found in /etc/opensearch/opensearch-security.

Working Example with mTLS

Below is a working snippet of config.yml from my lab. Basic auth is enabled as well.

_meta:
  type: "config"
  config_version: "2"
config:
  dynamic:
    http:
      anonymous_auth_enabled: true
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: "4"
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: true
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop

Note username_attribute: cn above. That means we are using the client certificate's Common Name to determine role mappings. The default is to use the full Distingushed Name (DN).

In my lab, I have a vector client that needs to write to OpenSearch. Let's look at its certificate CN:

cat vector-client.service.rl.crt | certtool -i | grep "Subject:"
        Subject: CN=vector-client.service.rl,OU=clients,O=🀚🀛🀜 🀈🀉🀊 🀓🀔🀕 🀀🀀 🀄︎🀄︎ RIICHILAB 🀄︎

We also need to make sure the client cert has "TLS WWW Client" listed as a Key Purpose.

cat vector-client.service.rl.crt | certtool -i | grep -3 "Purpose"
       Extensions:
               Key Purpose (not critical):
                       TLS WWW Client.

The CN in the certificate has to match sure the user definition in roles_mapping.yml:

vector_role:
  users:
  - "vector-client.service.rl"

And the role vector_role must be defined in roles.yml:

vector_role:
  reserved: false
  hidden: false
  cluster_permissions:
    - "indices:data/write/bulk"
    - "cluster_monitor"
  index_permissions:
    - index_patterns:
        - "logs-*"
      allowed_actions:
        - "data_access"
        - "create_index"

Once all these things are in place, run the apply-security-admin.sh script to activate the config. See my earlier blog post for more details on this script and how it works.

Troubleshooting

It doesn't always work the first time. Or the second, third, etc...

Here are some ways to troubleshoot when you don't get the results you're expecting.

For the API calls below, I am invoking cURL as follows:

curl -XGET -H 'accept: application/json' --cert ${client_cert} --key ${client_key} --cacert ${ca} https://${opensearch_endpoint}:9200/${api_route}

Some API calls require super admin permissions, others should be run using the client certificate. Consult the table below for more details. (And sorry the table renders poorly, I'm working on it ;)

Permissions are a whole 'nother ball of wax...

You now have a better idea of how to configure OpenSearch security with mutual TLS. However, figuring out what permissions are needed for which use cases is a bit more of a hassle. I hope to share more details on specific clients (such as vector) in future blog posts. Stay tuned!