Securing a Zookeeper ensemble

In the previous post, we looked at how to build a three cluster Zookeeper ensemble. However, the ensemble was not secured in any way. This would allow unauthorised clients to query Zookeeper and to push data to znodes. It also allows unauthorised Zookeeper instances to join the ensemble and potentially even instruct the cluster to shut down.

Even in secured networks, it’s a good idea to use some of the security features available in Zookeeper. In this post we’ll look at two security mechanisms: mutual TLS (mTLS) and SASL authentication. We’ll set up these security features on the server-server communication (leader election protocols) and client-server communication (Kafta to Zookeeper).

mTLS for quorum protocol

Mutual TLS (mTLS) is a mechanism to allow communication only where both parties trust each other. If this is enabled for the leader election protocol (Zookeeper server to Zookeeper server) then only trusted servers can join the ensemble. To enable mTLS, every Zookeeper server must have a signed certificate and a trust store containing the certificates of other servers in the ensemble.

Let’s start by creating the certificates, keystores and the trust store. Assuming we have Zookeeper running on three servers called:

• zookeeper-1.europe-north1-a.c.zookeeper-12345.internal
• zookeeper-2.europe-north1-b.c.zookeeper-12345.internal
• zookeeper-3.europe-north1-c.c.zookeeper-12345.internal

Use the Java keytool to create a self-signed keystore each server and a trust store containing all three certificates:

HOSTNAME=zookeeper-1.europe-north1-a.c.zookeeper-12345.internal
PASSWORD=keypa55
keytool -genkeypair -alias $HOSTNAME -keyalg RSA -keysize 2048 -dname "cn=$HOSTNAME" -keypass $PASSWORD -keystore $HOSTNAME-zk-keystore.jks -storepass $PASSWORD -validity 3650
keytool -exportcert -alias $HOSTNAME -keystore $HOSTNAME-zk-keystore.jks -file $HOSTNAME.cer -rfc
keytool -importcert -alias $HOSTNAME -file $HOSTNAME.cer -keystore zk-truststore.jks -storepass $PASSWORD

Run the above for each HOSTNAME to produce four files:

• zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
• zookeeper-2.europe-north1-b.c.zookeeper-12345.internal-zk-keystore.jks
• zookeeper-3.europe-north1-c.c.zookeeper-12345.internal-zk-keystore.jks
• zk-truststore.jks

Copy the files to the three Zookeeper servers and then add the following to each zoo.cfg:

sslQuorum=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.quorum.keyStore.location=/usr/local/zookeeper/conf/zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
ssl.quorum.keyStore.password=keypa55
ssl.quorum.trustStore.location=/usr/local/zookeeper/conf/zk-truststore.jks
ssl.quorum.trustStore.password=keypa55

(adjust the ssl.quorum.keyStore.location property to pick the keystore for this server)

Restart the three Zookeeper instances to enable (and enforce) mTLS for server-server communication.

SASL Authentication for quorum protocol

Zookeeper supports server-server mutual authentication using Simple Authentication and Security Layer (SASL). It supports Kerberos and Digest-MD5 schemes. Kerberos is the stronger authentication scheme but requires additional infrastructure. This example uses the simpler Digest-MD5 scheme which just requires some usernames and passwords to be set up.

First, add some config to zoo.cfg to enable (and require) SASL:

quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.kerberos.servicePrincipal=servicename/_HOST
quorum.cnxn.threads.size=20

Then create a jaas.conf file containing the our username and password:

QuorumServer {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_zkquorumlearner="zkqlpa55";
};
 
QuorumLearner {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       username="zkquorumlearner"
       password="zkqlpa55";
};

In this case our our Quorum Learner has username zkquorumlearner and password zkqlpa55. The Quorum Server is configured to accept that credential.

Finally, enable the JAAS config by adding it to Zookeeper’s SERVER_JVMFLAGS. The easiest way is to add this to java.env in /usr/local/zookeeper/conf/ (create the file if it doesn’t exist):

SERVER_JVMFLAGS="-Djava.security.auth.login.config=/usr/local/zookeeper/conf/jaas.conf"

Again, restart the three Zookeeper instances for this to take effect.

mTLS for client connections

Zookeeper supports the same security mechanisms for client-server communications. In this example we want to secure communication with a Kafka client.

To enforce mTLS for client communication, add the following to the zoo.cfg on all three servers:

secureClientPort=2182
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
ssl.keyStore.location=/usr/local/zookeeper/conf/zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
ssl.keyStore.password=keypa55
ssl.trustStore.location=/usr/local/zookeeper/conf/zk-truststore.jks
ssl.trustStore.password=keypa55

Note that we’re using the same keystore / truststore files that we created earlier. Again, make sure that the ssl.keyStore.location is the keystore for this server.

Add the corresponding configuration to the Kafka server.properties

# Required to use TLS-to-ZooKeeper (default is false)
zookeeper.ssl.client.enable=true
# Required to use TLS-to-ZooKeeper
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
# Define key/trust stores to use TLS-to-ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=/usr/local/zookeeper/conf/zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
zookeeper.ssl.keystore.password=keypa55
zookeeper.ssl.truststore.location=/usr/local/zookeeper/conf/zk-truststore.jks
zookeeper.ssl.truststore.password=keypa55
# Tells broker to create ACLs on znodes
zookeeper.set.acl=true

SASL authentication for client connections

Finally, we’ve already enabled SASL authentication in Zookeeper so using it for client connections is as simple as adding a new entry to the existing jaas.conf file:

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_kafka="kafkapa55";
};

Create a corresponding kafka_jaas.conf file for Kafka, this one contains the credentials that the client will authenticate with:

Client {
	org.apache.zookeeper.server.auth.DigestLoginModule required
 	username="kafka"
	password="kafkapa55";
};

Use the KAFKA_OPTS environment variable to enable this in Kafka:

export KAFKA_OPTS="-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_jaas.conf"

Now restart all three Zookeeper instances and Kafka. The Kafka logs should show that it’s using Digest-MD5 authentication:

INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
INFO Client successfully logged in. (org.apache.zookeeper.Login)
INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
INFO Opening socket connection to server zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182. (org.apache.zookeeper.ClientCnxn)
INFO SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
INFO SSL handler added for channel: [id: 0x661af545] (org.apache.zookeeper.ClientCnxnSocketNetty)
INFO Socket connection established, initiating session, client: /10.166.0.2:42300, server: zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182 (org.apache.zookeeper.Clien$
INFO channel is connected: [id: 0x661af545, L:/10.166.0.2:42300 - R:zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182] (org.apache.zookeeper.ClientCnxnSocketNetty)
INFO Session establishment complete on server zookeeper-2.europe-north1-b.c.zookeeper-12345.internal/10.166.0.3:2182, session id = 0x2000004d3be0001, negotiated timeout = 18000 (org.apache.$
INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)