# How to Set Up Disaster Recovery (DR) for AWS MSK with MirrorMaker 2 – Step-by-Step Guide

In today's cloud-native world, ensuring high availability and resilience for streaming platforms like Apache Kafka is mission-critical. **Amazon MSK (Managed Streaming for Apache Kafka)** offers a powerful, fully managed Kafka service. However, it doesn't natively provide cross-region disaster recovery (DR). In this guide, you’ll learn how to configure **cross-region DR for AWS MSK using Apache Kafka MirrorMaker 2 (MM2)** — a robust, open-source replication tool.

This comprehensive walkthrough includes prerequisites, cluster setup, networking, and end-to-end validation to help you build a production-ready DR solution.

---

## 🔍 What is AWS MSK?

**AWS MSK (Managed Streaming for Apache Kafka)** is a fully managed service that simplifies running Apache Kafka on AWS. It eliminates the operational overhead of provisioning servers, configuring clusters, and managing availability.

**Key features of AWS MSK**:

* Fully managed Apache Kafka
    
* Secure by default with VPC, TLS, and IAM integration
    
* Scalable with automatic broker scaling and storage expansion
    
* Native support for monitoring via CloudWatch and logging integrations
    

---

## 🔄 What is MirrorMaker 2?

**MirrorMaker 2 (MM2)** is the enhanced replication utility introduced in **Apache Kafka 2.4+**. It’s designed for copying data between Kafka clusters and is built on Kafka Connect, providing modularity, scalability, and fault tolerance.

### Key capabilities:

* Real-time replication of topics and consumer offsets
    
* Support for multiple clusters
    
* Active-passive and active-active configurations
    
* Flexible replication policies and error handling
    

---

## 💡 Available Methods for Kafka DR – Why MirrorMaker 2?

Several options exist for disaster recovery in Kafka:

| Method | Real-Time | Offset Sync | Cost | Complexity | Description |
| --- | --- | --- | --- | --- | --- |
| **MirrorMaker 2 (MM2)** | ✅ | ✅ | Low–Med | Medium | Open-source Kafka-native tool ideal for AWS MSK with IAM support. |
| **Confluent Replicator** | ✅ | ✅ | High | High | Commercial-grade tool with advanced features. |
| **Custom Producers/Consumers** | ✅ | ❌ | Medium | High | Build-your-own with full control. |
| **Kafka Streams or Flink** | ✅ | ❌ | High | High | Stream processing with built-in replication logic. |
| **S3 Backup & Restore** | ❌ | ❌ | Low | Low | Periodic export-import, cold DR only. |

**Why we chose MirrorMaker 2 for this guide**:

* Seamless integration with AWS MSK and IAM authentication
    
* No additional licensing or external dependencies
    
* Good balance of simplicity, performance, and reliability
    

---

## 🧰 Prerequisites

To follow this tutorial, ensure you have:

* An AWS account with permissions for MSK, EC2, IAM, and VPC
    
* AWS CLI installed and configured
    
* Java 11+ installed on the EC2 instance
    
* Kafka client tools (Apache Kafka binaries)
    
* Two VPCs in different regions (e.g., `ap-south-1` and `us-east-1`)
    
* AWS MSK IAM Authentication JAR: `aws-msk-iam-auth.jar`
    

---

## 🏗️ Step 1: Create Primary and DR MSK Clusters

### 🔹 Create Primary Cluster (`ap-south-1`)

1. Go to **Amazon MSK > Create Cluster**
    
2. Select **Custom create**
    
3. Cluster name: `msk-primary`
    
4. Kafka version: `3.0+`
    
5. Brokers: `kafka.m5.large`, 3 brokers, 1000 GiB EBS each
    
6. Network: Select VPC and 3 subnets
    
7. Enable:
    
    * Encryption at rest (KMS)
        
    * TLS in-transit encryption
        
    * IAM authentication
        
8. Assign a security group to allow port **9198** from EC2
    
    ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXcfwe7JRLC7qCPwnDbg8f7pJvaFLL9bItw6BcM-yPwSP8OmDUd-DSMEvj4mOAChy4Um71gpsvcTjDw8O1uWi_08l7XRbybeI7Tl6kThtlDnI6wSAqSKcmuaOXqQ1IgQoiUSMtMi4g?key=kyqryDzyBOwqmUNT6iVkzw align="left")
    

### 🔹 Create DR Cluster (`us-east-1`)

Repeat the above steps in the `us-east-1` region, using the cluster name `msk-dr`. Ensure consistent configuration across clusters.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXesMYJ57wAGFVYZj4Cgh-YLT_Ccv8AwiuSboN8ooug8INyuXQOEmOasFLwS7VGeljbIFgIQGH64NqbaGSV9OSVyRr7tHUWBnpibrMOenvFoOE9jIZ2kxqbG_IgFdb4zsoAWNmJY?key=kyqryDzyBOwqmUNT6iVkzw align="left")

---

## 🔐 Step 2: Configure Network and Security

Update security group rules:

* **MSK SG**: Allow **inbound TCP 9198** from EC2 SG or IP
    
* **EC2 SG**: Allow **outbound 9198** to both MSK clusters
    
* Enable SSH (port 22) on EC2 for management access
    
    ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXexFgSA-bBAS3UVh-dTa1gKgAMhB86420zNuHKDzklcYz1EH9h910wPx7-go9X7_E_YutJmta02PYQVKe1RQqKEVYqToCCjOpxBpj9AJdz1Zgk3--wAJkU_A0JOdrGAoqFrIPpgZw?key=kyqryDzyBOwqmUNT6iVkzw align="left")
    

---

## 💻 Step 3: Launch EC2 with Kafka Client

### 🚀 Launch EC2

* Region: `ap-south-1`
    
* Type: `t3.medium` or higher
    
* Attach IAM role for MSK and Secrets Manager (if needed)
    
* Ensure internet access (NAT or public IP)
    

### 🛠️ Install Tools and Configure

```bash
sudo yum update -y
sudo yum install -y java-11-amazon-corretto
wget https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
tar -xzf kafka_2.13-3.3.1.tgz
export KAFKA_HOME=$(pwd)/kafka_2.13-3.3.1

wget https://github.com/aws/aws-msk-iam-auth/releases/latest/download/aws-msk-iam-auth.jar
export IAM_JAR=$(pwd)/aws-msk-iam-auth.jar
```

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfYQhwJhHrxk5y_JvR2XL2w_sRf-a0_P76-2nlrjNQb-58W0iz5WlRGzyRliMmgQzGuXQDoD1BdXVMnrK7YymQgvWkNtfp5vXSFt-MP_UKe27bNFxix0V9BUBSJzTi7lwZ38uVkRw?key=kyqryDzyBOwqmUNT6iVkzw align="left")

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfC3S3k1B-8zYjrsUBR25yPPGY4UDv947CuUzBHAcnAuzJfIFN1_2mXmtujmyzTrgYGe4ZMGdZzC4WXUEKfwmvb7A4lFYaaGa8y4HSuatVa1SN_heazurb89J0tPDTrTvGXvsUWig?key=kyqryDzyBOwqmUNT6iVkzw align="left")

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXcBynanupq_F3UGyPHXNcgK5SDIrdSXCeuut2ly4i-9It0oUioezYI9J_3QXcxvjF4mZO_a_yCrem2zN7H6I6hmwPxGN8vJrNieVTbTb3GCIAhuy3Z9Vn9GZgCD6NXrWy0RZH-D1A?key=kyqryDzyBOwqmUNT6iVkzw align="left")

### ✍️ Create [`client.properties`](http://client.properties)

```bash
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

ssl.truststore.location=/home/ec2-user/msk-certs/truststore.jks
ssl.truststore.password=anjali
```

> ☝️ Make sure the truststore includes AWS MSK’s CA certificate.
> 
> ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXe3ongNAZID_n1cTtMkzMCxWAs4LxGdWa3zlu53ySopTyhx6FarbLpA5DbcOGCY2CympwmQgER4Z7FReCklCsfAC7YhRY6erK6fV5SW4cYXmObvNM9S1twtzYXx5mDtLk1CN6Lo?key=kyqryDzyBOwqmUNT6iVkzw align="left")

---

## 🧵 Step 4: Create Kafka Topic on Primary Cluster

```bash
CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-topics.sh \
  --create \
  --topic test-topic \
  --partitions 3 \
  --replication-factor 3 \
  --bootstrap-server <primary-broker-list> \
  --command-config /home/ec2-user/msk-certs/client.properties
```

Replace `<primary-broker-list>` with your actual MSK bootstrap brokers.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXf8DCENnIY-dL4eptLVsRZ08XZq57lj_0NwPGZh_0od4L6buYGCmaqRaP9gVEWI1jxzQNdjUn7A6h1zEkqj1CPauBZFshdkxgurFXyYrI_GuOyrtd8J780SaV5lLcQOdiB9dkosUQ?key=kyqryDzyBOwqmUNT6iVkzw align="left")

---

## ⚙️ Step 5: Configure and Run MirrorMaker 2

### ✍️ Create [`mm2.properties`](http://mm2.properties)

```bash
clusters = primary,dr

primary.bootstrap.servers=<primary-brokers>
primary.security.protocol=SASL_SSL
primary.sasl.mechanism=AWS_MSK_IAM
primary.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
primary.ssl.truststore.location=/home/ec2-user/msk-certs/kafka.client.truststore.jks
primary.ssl.truststore.password=anjali

dr.bootstrap.servers=<dr-brokers>
dr.security.protocol=SASL_SSL
dr.sasl.mechanism=AWS_MSK_IAM
dr.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
dr.ssl.truststore.location=/home/ec2-user/msk-certs/kafka.client.truststore.jks
dr.ssl.truststore.password=anjali

tasks.max=2
topics=test-topic
groups=.*
replication.policy.class=org.apache.kafka.connect.mirror.DefaultReplicationPolicy
```

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXcXZSys_yPQIU6rGBy7VIE_gMBJLrNPsMdaGn06Y9_JuIeK-JXfhxnKDE1SxCs-tTEFrstW_E4IzJkpmNC_O3HZbmXD2qWcs5ltv7l3RmQogGmXwtBMPfIhhzunfzK4IpVrmzvukA?key=kyqryDzyBOwqmUNT6iVkzw align="left")

### ▶️ Run MM2

```bash
CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/*:$KAFKA_HOME/libs/connect-runtime-*.jar:$KAFKA_HOME/libs/connect-api-*.jar \
  $KAFKA_HOME/bin/connect-mirror-maker.sh /home/ec2-user/mm2.properties
```

---

## ✅ Step 6: Validate Replication

### 🔎 List Topics on DR

```bash
CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-topics.sh \
  --list \
  --bootstrap-server <dr-broker> \
  --command-config /home/ec2-user/msk-certs/client.properties
```

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdbmVytrDCrmVzHof-R4sG2EalG6Ug9zk-EX37LiOtDmOm2pUsZrkJEBkujwMprD5z9DbawYpJyjdpJX-Mtfucob3VUfcwOF6k3BZIuo72XlOl2Y0IOavDphtGiXtZYTpX4eVSwEw?key=kyqryDzyBOwqmUNT6iVkzw align="left")

### 📥 Consume Messages from DR

```bash
CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-console-consumer.sh \
  --topic mm2-test-topic \
  --from-beginning \
  --bootstrap-server <dr-broker> \
  --consumer.config /home/ec2-user/msk-certs/client.properties
```

---

## ✍️ Step 7: Test Message Flow

### 📨 Produce to Primary

```bash
CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-console-producer.sh \
  --topic test-topic \
  --bootstrap-server <primary-broker> \
  --producer.config /home/ec2-user/msk-certs/client.properties
```

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdaSxnzdlDhBQXHaTYQlyVPr2AvQ6yOwHQ-BAF0eKDxEQPi1dopd1m5br-f6UI18Co_gfVSeY2DARQZzQvsLYcBr_LGumLXyiH7jnxdclAMPC6b8xV5p8vQafKK79huIV3VX7vu?key=kyqryDzyBOwqmUNT6iVkzw align="left")

Type some messages and hit Enter.

### ✅ Confirm on DR

Re-run the consumer from the DR cluster to verify real-time replication.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXf2zfw9b2QDGEGZapUlBJA1yWqIo4Uc3-0R2zJ-alUkmgpzQ8Q4F9XG-wrW_syPJcyaRia3zlBmsN_WdjFu7pE03hasTHTP1RquwO3velEsxPzc_NqQ3LNgSA5TRk17ohq-i0VyCA?key=kyqryDzyBOwqmUNT6iVkzw align="left")

💬 **Have you implemented DR for Kafka in your architecture?**  
Drop your approach or challenges in the comments — I'd love to hear how others tackle cross-region resilience!

🔔 **Follow me for more AWS infrastructure and streaming data posts!**