# Kubernetes Outage Postmortem: Nodes Stuck in NotReady Due to CNI Failure

Recently, we encountered a **critical production outage** in our Kubernetes cluster. New nodes provisioned during autoscaling remained in a `NotReady` state, leading to service disruptions and failed health checks across workloads.

In this post, I’ll walk you through:

* What caused the issue
    
* How we identified and resolved it
    
* Best practices to prevent similar failures in your clusters
    

---

## 🔥 What Went Wrong

During a surge in traffic, our cluster autoscaler kicked in and added new nodes. However, these nodes failed to become **Ready**, resulting in:

* ❌ Workloads not scheduled
    
* ❌ Services unreachable
    
* ❌ Health checks failing, pods crashing
    

A quick check with:

```bash
kubectl get nodes
```

revealed multiple entries like:

```bash
ip-node-ip.eu-west-1.compute.internal   NotReady
```

To dig deeper, we inspected system logs and found this error:

```bash
container runtime network not ready: NetworkReady=false
NetworkPluginNotReady: docker: network plugin is not ready: cni config uninitialized
```

---

## ⚠️ Root Cause: CNI Plugin Failure (Calico)

These errors indicated a **CNI (Container Network Interface) misconfiguration**. Our cluster was using **Calico** as the CNI, and it wasn’t initializing properly.

The Calico pods responsible for managing the network stack were either stuck or not starting due to missing configurations.

---

## 🛠️ How We Fixed It

### 1\. Delete and Recreate Calico Pods

Force Kubernetes to restart Calico:

```bash
kubectl delete pod -n kube-system -l k8s-app=calico-node
```

This helped in recreating the Calico pods with the current (and correct) configuration.

---

### 2\. Reapply Calico Configuration

On some nodes, the CNI config was missing or corrupted:

```bash
/etc/cni/net.d/10-calico.conflist
```

We reinstalled Calico using the official manifest:

```bash
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
```

---

### 3\. Restart the Kubelet on Affected Nodes

Restarting the kubelet reinitialized the CNI network stack:

```bash
sudo systemctl restart kubelet
```

---

### ✅ Verification

After applying the fixes, we verified node status:

```bash
kubectl get nodes
```

Now showed:

```bash
ip-node-ip.eu-west-1.compute.internal   Ready
```

Services started recovering, and workloads were rescheduled.

---

## 🧠 Lessons Learned: How to Prevent This in the Future

### 🔁 1. Backup CNI Configurations

Always back up CNI configs, especially:

```bash
/etc/cni/net.d/10-calico.conflist
```

This helps with disaster recovery and rapid bootstrapping.

---

### 📈 2. Monitor Calico Health and Node Status

Set up monitoring and alerting for:

```bash
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get nodes
```

Alert if:

* Calico pods crash or restart frequently
    
* Nodes enter or stay in `NotReady`
    

---

### ⚠️ 3. Avoid Mixing CNIs

**Do not mix different CNI plugins** (e.g., Calico and AWS VPC CNI) unless you're explicitly building a hybrid setup. It introduces instability and unexpected behavior.

> In our case, we’ve since migrated to the **AWS VPC CNI**, which aligns better with EKS and provides native integration with VPC IP address management.

---

## 📌 Conclusion

Networking is the backbone of Kubernetes, and when the CNI fails, everything breaks.

This incident was a sharp reminder of the importance of:

* Validating CNI configurations
    
* Monitoring node readiness
    
* Keeping your control plane and worker nodes in sync
    

By following the steps outlined above and applying proactive monitoring, you can **prevent CNI-related outages** and ensure high availability for your workloads.