# Production Change: Migrating a StatefulSet from Large to Smaller Nodes in EKS (Without Downtime) We had a production application running on **Amazon EKS** as a StatefulSet. Each replica had its own PersistentVolumeClaim backed by **Amazon EBS**. During the initial launch phase, we deployed the workload on large instances to remove any performance uncertainty. After a few weeks of monitoring, the data was clear: * CPU utilization consistently below 35% * Memory below 40% * No disk pressure * Stable traffic and latency We were clearly over-provisioned. The goal was straightforward: Move the StatefulSet from a large node group to a smaller node group to reduce infrastructure cost — without downtime. This was a live production system. --- ## Cluster Setup We had two managed node groups: * `aws-devops-agent-eks-test-ng1` (large instances) * `migration-ng` (smaller instances) Workload characteristics: * StatefulSet with 3 replicas * Each pod had its own PVC (created via `volumeClaimTemplates`) * StorageClass backed by EBS * PodDisruptionBudget configured: ```bash maxUnavailable: 1 ``` Before touching the production workload, we tested the entire flow using a demo Nginx StatefulSet with PVCs. This allowed us to observe storage detach/attach behavior safely. --- ## What Happens When You Change nodeSelector in a StatefulSet Changing the `nodeSelector` modifies the pod template inside the StatefulSet: ```bash spec: template: spec: nodeSelector: ``` Any change under `spec.template` updates the pod template hash. That automatically triggers a rolling update. No manual restart command is required. For each pod, Kubernetes performs the following lifecycle: 1. Terminate pod on the old node 2. Detach the EBS volume 3. Schedule pod on a new node 4. Attach the same volume 5. Mount filesystem 6. Start container 7. Wait for readiness probe Because this is a StatefulSet: * Pod identity remains stable * PVC remains the same * The EBS volume remains in its original Availability Zone The primary risk during migration is not parallel restarts — StatefulSet prevents that by default. The real concern is restart pacing and storage stability between transitions. --- ## Why We Explicitly Kept OrderedReady We defined: ```bash podManagementPolicy: OrderedReady ``` StatefulSet supports two policies: * OrderedReady * Parallel With OrderedReady: * Pods are terminated in reverse ordinal order (pod-2 → pod-1 → pod-0) * The controller waits for a pod to become Ready before proceeding to the next one This guarantees serialized lifecycle transitions. Only one pod is ever moving at a time. For storage-backed workloads, predictability is more important than speed. --- ## Why We Increased minReadySeconds Originally: ```bash minReadySeconds: 30 ``` During migration, we increased it: ```bash minReadySeconds: 60 ``` This does not delay traffic. It delays rollout progression. The behavior becomes: * Pod becomes Ready * Controller waits 60 seconds * Then proceeds to the next pod That buffer provides: * Storage stabilization time * Application warm-up window * A monitoring observation period before the next restart OrderedReady ensures serialization. minReadySeconds ensures pacing. Together, they create controlled transitions. --- ## What PodDisruptionBudget Actually Protects The PodDisruptionBudget does not control rolling update speed. It protects against voluntary disruptions such as: * Node drain * Evictions * Autoscaler actions With: ```bash maxUnavailable: 1 ``` We ensured that even outside the rollout logic, no more than one pod could be voluntarily disrupted at a time. This preserved availability guarantees during infrastructure operations. --- ## Availability Zone Validation EBS volumes are Availability Zone bound. Because PVCs were already provisioned, each PersistentVolume existed in a specific AZ. Before migration, we verified: * The smaller node group spans the same AZs as the large node group * There is node capacity in those AZs If a pod’s volume resides in `ap-south-1a`, the new node must also be in `ap-south-1a`. Otherwise, the pod remains Pending due to volume node affinity constraints. This check is mandatory for StatefulSet migrations using EBS. --- ## What We Changed Only two fields were modified. Before: ```bash minReadySeconds: 30 nodeSelector: eks.amazonaws.com/nodegroup: aws-devops-agent-eks-test-ng1 ``` After: ```bash minReadySeconds: 60 nodeSelector: eks.amazonaws.com/nodegroup: migration-ng ``` Everything else remained unchanged. Small change. Controlled blast radius. --- ## Execution Steps: 1. Created the smaller node group (`migration-ng`) 2. Verified AZ alignment and resource headroom 3. Updated the StatefulSet manifest 4. Applied the updated configuration: ```bash kubectl apply -f statefulset.yaml ``` Because the pod template changed, the StatefulSet controller automatically initiated a rolling update. We first executed this full flow using the demo Nginx StatefulSet to validate behavior before applying it to the production workload. --- ## Observed Behavior For each replica: * Pod terminated on the large node * EBS detached * Pod scheduled onto a smaller node in the same AZ * Volume attached * Pod became Ready * Controller waited 60 seconds * Next pod restarted There were no overlapping transitions. No downtime. No error spike. Stable latency throughout. --- ## Final Result Before: 3 large instances After: 3 smaller instances Outcome: * Reduced compute cost * Preserved availability * Maintained performance * No operational instability --- ## What This Actually Was This was not just resizing infrastructure. It was a controlled lifecycle transition of a stateful workload. StatefulSet migrations are safe when you: * Respect controller behavior * Serialize restarts * Add rollout pacing * Validate storage topology * Test with a safe workload first That’s exactly what we did. And the migration completed without downtime.