Configure Access to Kafka Cluster

The following information describes how to use credentials to configure access to the Kafka cluster.

Kafka mTLS authentication

Communication with your Kafka cluster is TLS secured, meaning both the client and the Kafka cluster authenticate each other. The client authenticates the server by verifying the server's certificate, and the server authenticates the client by verifying the client's certificate. As the Kafka cluster does not have publicly signed certificates, you must validate them with the cluster's certificate authority. Authentication happens via mutual TLS (mTLS). Therefore, your cluster maintains a client certificate authority to sign authenticated user certificates.

Get certificates and key

To connect and authenticate to your Kafka cluster, you must fetch the required two certificates and a key from the user's API endpoint. Below are the steps to get the required certificates and key with curl commands for a cluster created in Frankfurt (de-fra) region.

# Get the cluster's CA certificate
curl --location https://kafka.de-fra.ionos.com/clusters/${clusterId}/users/${userId}/access --header "Authorization: Bearer ${personalToken}" |  yq -r '.metadata.certificateAuthority' > ca-cert.pem
# verify
openssl x509 -in ca-cert.pem -text -noout

# Get the (admin) users client certificate
curl --location https://kafka.de-fra.ionos.com/clusters/${clusterId}/users/${userId}/access --header "Authorization: Bearer ${personalToken}" |  yq -r '.metadata.certificate' > admin-cert.pem
# verify
openssl x509 -in admin-cert.pem -text -noout

# Get the (admin) users client key
curl --location https://kafka.de-fra.ionos.com/clusters/${clusterId}/users/${userId}/access --header "Authorization: Bearer ${personalToken}" |  yq -r '.metadata.privateKey' > admin-key.pem
# verify
openssl rsa -in admin-key.pem -check

Convert certificates & key

You will need different file formats for the certificates depending on the consumer/producer's implementation. The following sections show how to create and use them with the Kafka Command-Line Interface (CLI) Tools.

PKCS#12 (.p12 / .pfx)

# Create a ca-cert.p12 (with openssl >3.2 )
openssl pkcs12 -export -nokeys -in ca-cert.pem -out ca-cert.p12 -passout "pass:changeit" -jdktrust anyExtendedKeyUsage
# Create a ca-cert.p12 (with keytool)
keytool -importcert -storetype PKCS12 -keystore ca-cert.p12 -storepass changeit -alias cluster-ca -file ca-cert.pem -noprompt
# verify
openssl pkcs12 -info -in ca-cert.p12

# Create an admin.p12
openssl pkcs12 -export -in admin-cert.pem -inkey admin-key.pem -out admin.p12 -passout "pass:admin_p12_pass"
# verify
openssl pkcs12 -info -nodes -in admin.p12

Your admin.properties files should look like this:

security.protocol=SSL
ssl.truststore.type=PKCS12
ssl.truststore.location=ca-cert.p12
ssl.truststore.password=changeit
ssl.endpoint.identification.algorithm=

ssl.keystore.type=PKCS12
ssl.keystore.location=admin.p12
ssl.keystore.password=admin_p12_pass

bin/kafka-topics.sh --list --bootstrap-server=clusterIp:Port --command-config admin.properties

Java KeyStore (JKS)

# Create a Java Truststore
keytool -import -alias cluster-ca -file ca-cert.pem -keystore truststore.jks -storepass changeit -noprompt
# verify
keytool -list -keystore truststore.jks -rfc -storepass changeit

# Create a Java Keystore
openssl pkcs12 -export -in admin-cert.pem -inkey admin-key.pem -out admin.p12 -passout "pass:admin_p12_pass"
keytool -importkeystore -srckeystore admin.p12 -srcstorepass admin_p12_pass -destkeystore admin.ks -storepass admin_jks_pass
# verify
keytool -list -keystore admin.ks -rfc -storepass admin_jks_pass
# verify including the key
keytool -importkeystore -srckeystore admin.ks -srcstorepass admin_jks_pass -deststoretype PKCS12 -destkeystore filename.p12 -storepass p12_pass; openssl pkcs12 -info -nodes -in filename.p12 -passin "pass:p12_pass"; rm -f filename.p12

Your admin.properties files should look similar to the following:

security.protocol=SSL
ssl.truststore.location=truststore.jks
ssl.truststore.password=changeit
ssl.endpoint.identification.algorithm=

ssl.keystore.location=admin.ks
ssl.keystore.password=admin_jks_pass

bin/kafka-topics.sh --list --bootstrap-server=clusterIp:Port --command-config admin.properties

PKCS#8 PEM

# No need to do anything with the ca-cert.pem it can be used without any modification
# verify
openssl x509 -in ca-cert.pem -text -noout

# Create a admin.pem containing key and cert
# as the Kafka CLI tool requires the key in PKCS#8 and to be secured with a passphrase we need to convert it first
openssl pkcs8 -in admin-key.pem -passout "pass:admin_pem_pass" -topk8 -v1 PBE-SHA1-3DES -out admin.pem
cat admin-cert.pem >> admin.pem
# verify
openssl x509 -in admin.pem -text -noout
openssl pkey -in admin.pem -check

Your admin.properties files should look similar to the following:

security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=ca-cert.pem
ssl.endpoint.identification.algorithm=

ssl.keystore.type=PEM
ssl.keystore.location=admin.pem
ssl.key.password=admin_pem_pass

bin/kafka-topics.sh --list --bootstrap-server=clusterIp:Port --command-config admin.properties