Migrating from Amazon Linux 2 to Amazon Linux 2023

Search for a command to run...

No comments yet. Be the first to comment.
Real production migration and incident playbooks focused on safe execution, root cause analysis, and rollback-first DevOps practices. Each post documents how real production issues were handled and fixed without downtime.
A practical, real-world walkthrough of Redis OSS snapshot export, cross-account restore failures, and the production-safe migration pattern.
This migration was performed on a production workload where cost reduction was prioritized over zone-level high availability.

1. Overview What I Designed I designed a hybrid infrastructure architecture: Terraform → Foundation Layer Crossplane → Dynamic Lifecycle Layer ArgoCD → GitOps Enforcement This created a continuou

Cross-cloud VM migration is not a disk copy task. It is: An access model transformation A replication lifecycle management exercise A downtime control operation A cost boundary decision We execu

When AWS introduced AWS DevOps Agent, I was less interested in feature lists and more interested in one practical question. Can it actually reduce investigation time during real production-style failu

Migrating object storage across cloud providers is not a copy task.It is a cost, network, and security boundary problem. We migrated 10+ TB of object data from Google Cloud Storage to Amazon S3 under

Amazon Linux 2 (AL2) will reach end-of-support on June 30, 2026. After that date, AWS will no longer provide security updates, patches, or new packages.
Although AL2 continues to receive maintenance updates today, it is no longer the forward-looking platform. Amazon Linux 2023 (AL2023) is the long-term replacement, offering a predictable 5-year lifecycle per release (2 years standard + 3 years maintenance) along with modernized system components.
If you are running production workloads on AL2, migration should be planned early — not rushed near 2026.
This article explains how to safely migrate in a large production environment with:
Standalone EC2 instances
Auto Scaling Groups (ASG)
Amazon EKS worker nodes
Assume a worst-case environment of 200 AL2 instances.
There is no supported in-place upgrade path from AL2 to AL2023.
AL2023 introduces:
Updated kernel
Newer system libraries
Updated OpenSSL and crypto policies
cgroup v2 by default
Updated container runtime stack
Attempting in-place OS mutation:
Is unsupported
Is difficult to test
Has no clean rollback
Is unsafe for Kubernetes worker nodes
The correct production pattern is:
Build new → Validate → Controlled cutover → Preserve rollback → Decommission old
| Workload Type | Count |
| Standalone EC2 (stateful) | 40 |
| Auto Scaling Groups (stateless) | 120 |
| EKS worker nodes | 40 |
| Total | 200 |
Each workload type requires a different strategy.
Databases
Legacy applications
EC2 instances using Elastic IP
Applications dependent on local storage
Before any migration:
Create EBS snapshots
Confirm snapshot completion
This is your rollback baseline.
Create a new EC2 instance with:
Same VPC and subnet
Same security groups
Same IAM role
Same instance type
Do not modify the AL2 instance.
Test explicitly:
Runtime versions (Java, Python, Node)
OpenSSL behavior
Crypto policy differences
Systemd services
Hardcoded paths in custom scripts
Small OS-level changes can break production services.
Initial sync while application is running
Stop or freeze writes
Final sync with checksum validation
Example:
rsync -avh --checksum --numeric-ids /data/ new-server:/data/
For database systems, prefer logical dump/restore over raw filesystem copy to avoid corruption risk.
Start the application on AL2023 and verify:
Application logs
Health endpoints
Downstream connectivity
Resource utilization
Only after validation:
Stop service on AL2
Switch Elastic IP or DNS
Keep AL2 intact until full confidence.
Web servers
APIs
Microservices
The main risk is pushing a broken AMI to the entire fleet.
Include:
Monitoring agents
Security agents
Logging agents
Application bootstrap scripts
Test full userdata execution.
Simulate instance termination to confirm auto-recovery.
Update only:
Keep AL2 template available for rollback.
Increase desired capacity by 1.
Validate:
Load balancer health checks
Application startup
Logs and error rate
Metrics stability
Do not skip canary testing.
Use safe rollout configuration:
Configure minimum healthy percentage appropriate for fleet size (e.g., 90–100% for large fleets).
Warm-up time configured
ELB health checks enabled
Monitor closely during rollout.
If instability occurs:
Cancel refresh
Revert launch template
AL2023 introduces changes that can affect Kubernetes workloads:
cgroup v2
Updated kernel
Updated container runtime
This can impact:
DaemonSets
Monitoring agents
Security tooling
CNI plugins
Create a new managed node group (or Karpenter pool).
Do not modify AL2 nodes yet.
kubectl taint node <al2-node> os=al2:NoSchedule
Effect:
No new pods schedule on AL2
Existing pods continue running
Scale workloads or deploy new services.
Confirm:
Pods schedule on AL2023
Networking functions correctly
Metrics and logs flow normally
Check:
VPC CNI
CoreDNS
kube-proxy
Logging agents
Monitoring agents
Security tools
Ensure cluster add-ons (VPC CNI, CoreDNS, kube-proxy) versions are compatible with AL2023 node AMIs before rollout.
Also verify PodDisruptionBudgets:
kubectl get pdb -A
Ignoring PDBs can cause draining failures or partial outages.
kubectl drain <node> --ignore-daemonsets
Observe workload behavior during drain.
Delete only after full stability confirmation.
Rollback is possible only until deletion.
Attempting in-place OS upgrades
Skipping canary validation
Ignoring bootstrap script testing
Draining EKS nodes prematurely
Not snapshotting stateful systems
Removing rollback resources too early
Amazon Linux 2 reaches end-of-support on June 30, 2026. Migration to Amazon Linux 2023 should follow immutable infrastructure principles. For stateful EC2, use parallel instances with validated data sync and controlled DNS cutover. For Auto Scaling Groups, roll out a new AMI using canary and guarded instance refresh. For EKS, introduce AL2023 nodes, prevent new scheduling on AL2, validate workloads and cluster add-ons, then drain and remove AL2 nodes after stability is confirmed. Maintain rollback until the final step.
This migration is not about replacing servers.
It is about maintaining production stability while upgrading the platform.
Plan early.
Validate carefully.
Preserve rollback.
Decommission only after confidence.