This article documents a real production-style migration of a Kubernetes StatefulSet backed by Amazon EBS from Multi-AZ to Single-AZ in Amazon EKS.

The migration was executed at the storage layer using EBS snapshots during a controlled maintenance window.

Objectives

• Reduce cross-AZ replication and inter-AZ data transfer cost

• Preserve existing EBS-backed data

• Avoid provisioning a parallel Kafka cluster

• Maintain deterministic storage recovery

• Ensure logical RPO = 0 with verified clean shutdown and completed snapshot.

This was a cost-first architectural decision with acknowledged availability trade-offs.

2. Business Context

Kafka (with ZooKeeper) was deployed across:

• ap-south-1b

• ap-south-1c

Multi-AZ improved resilience, but introduced:

• Continuous cross-AZ replication traffic

• Inter-AZ data transfer billing

• Increased recurring infrastructure cost

After analyzing recurring billing, cross-AZ traffic became a major cost driver.

Cost analysis showed that inter-AZ data transfer and cross-AZ replication traffic accounted for a significant percentage of the monthly Kafka infrastructure spend.

While Multi-AZ improved resilience, the business determined that the availability gain did not justify the recurring transfer cost for this workload profile.

Business decision:

Consolidate into single AZ
Downtime acceptable
No data loss allowed

3. Technical Constraint — Why This Is Not a Simple Scheduler Change

The workload runs as a StatefulSet using volumeClaimTemplates.

Storage backend: Amazon EBS via EBS CSI driver.

Important constraints:

• EBS volumes are strictly AZ-scoped

• PVs include topology.kubernetes.io/zone nodeAffinity

• PVCs are tightly bound to PVs

• EBS cannot attach across AZ

If we restrict nodeAffinity without moving storage:

• Pod schedules successfully

• Volume attach fails

• Pod stuck in ContainerCreating

This is fundamentally a storage locality constraint.

Storage must move before pods move.

4. Architecture Before Migration

Multi-AZ StatefulSet with cross-AZ replication traffic between replicas.

Characteristics

• Replica-0 in ap-south-1b

• Replica-1 in ap-south-1c

• Independent EBS volumes per replica

• Continuous cross-AZ replication

• Higher availability

• Higher recurring cost

5. Migration Options Evaluated

Option 1 — Snapshot-Based Storage Migration (Chosen)

Flow:

1. Validate Kafka stability

2. Scale StatefulSet to 0

3. Snapshot EBS volumes

4. Restore volumes in the target AZ

5. Rebind PVCs via static PVs

6. Restrict scheduling

7. Safe staged bring-up

Properties:

• Logical RPO = 0, assuming:

  • No under-replicated partitions

  • Clean shutdown

  • Snapshot completion verified

• Planned downtime required

• No duplicate cluster

• Lowest infrastructure cost

• Requires operational precision

Option 2 — Dual Cluster + MirrorMaker2

Flow:

1. Deploy new Kafka cluster in single AZ

2. Configure MirrorMaker2

3. Replicate topics

4. Validate offsets

5. Cut traffic

6. Decommission old cluster

Properties:

• Near-zero downtime

• Higher infrastructure cost

• More operational complexity

• Easier rollback

Because downtime was acceptable and cost reduction was urgent, Option 1 was selected.

6. Multi-AZ Deployment YAML (Reproducible Lab Setup)

# multi-az-zk.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: tools
---
apiVersion: v1
kind: Service
metadata:
  name: zk-staging-headless
  namespace: tools
  labels:
    app: cp-zookeeper
    release: kafka-staging
spec:
  clusterIP: None
  selector:
    app: cp-zookeeper
    release: kafka-staging
  ports:
    - name: client
      port: 2181
      targetPort: 2181
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk-staging
  namespace: tools
spec:
  serviceName: zk-staging-headless
  replicas: 2
  selector:
    matchLabels:
      app: cp-zookeeper
      release: kafka-staging
  template:
    metadata:
      labels:
        app: cp-zookeeper
        release: kafka-staging
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - ap-south-1b
                      - ap-south-1c
      containers:
        - name: cp-zookeeper
          image: docker.io/confluentinc/cp-zookeeper:5.5.6
          volumeMounts:
            - name: datadir
              mountPath: /var/lib/zookeeper/data
            - name: datalogdir
              mountPath: /var/lib/zookeeper/log
  volumeClaimTemplates:
    - metadata:
        name: datadir
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: gp3
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: datalogdir
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: gp3
        resources:
          requests:
            storage: 10Gi

Apply:

kubectl apply -f multi-az-zk.yaml

Validate zone distribution:

kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pods -o wide -n tools
kubectl describe pv <pv-name>

7. Production Migration — Snapshot-Based Execution

Step 1 — Validate Kafka Stability

Before shutdown, ensure:

• No under-replicated partitions

• No leader elections are ongoing

• ISR stable

Example:

kubectl logs <pod-name> -n tools

Step 2 — Protect Reclaim Policy

Before deleting PVCs:

kubectl get pv

Ensure persistentVolumeReclaimPolicy: Retain.

If not:

kubectl patch pv <pv-name> \
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

This prevents accidental EBS deletion.

Step 3 — Scale Down

kubectl scale statefulset zk-staging -n tools --replicas=0
kubectl get pods -n tools

Pre-Snapshot Validation — StatefulSet Ordinal Mapping (Critical)

Before taking snapshots, validate which EBS volume belongs to which StatefulSet ordinal.

StatefulSet PVC naming convention:

<claim-name>-<statefulset-name>-<ordinal>

Example:

datadir-zk-staging-0
datadir-zk-staging-1

Validate volume mapping:

kubectl get pvc -n tools
kubectl describe pvc datadir-zk-staging-1 -n tools

Extract:

Volume: pvc-xxxx

Then map to AWS volume:

aws ec2 describe-volumes \
  --filters Name=tag:KubernetesCluster,Values=<cluster-name>

Or inspect via:

kubectl describe pv <pv-name>

Confirm:

• Correct ordinal

• Correct AZ

• Correct volume ID

If you restore the wrong ordinal volume to the wrong replica, data corruption or cluster quorum failure can occur.

Never assume volume ordering.

Validate explicitly.

Step 4 — Snapshot Volumes

aws ec2 create-snapshot \
  --volume-id vol-xxxx \
  --description "zk-migration"

Wait until the snapshot state = completed before proceeding.

Do not scale up or delete any additional resources until snapshot completion is verified via AWS CLI or console.

Step 5 — Restore With Matching Performance

Check original performance:

aws ec2 describe-volumes --volume-ids <volume-id>

Restore:

aws ec2 create-volume \
  --snapshot-id snap-xxxx \
  --availability-zone ap-south-1b \
  --volume-type gp3 \
  --iops 3000 \
  --throughput 125

Do not proceed until the restored volume state is "available" and fully initialized.

The restored volume must match the original volume type, IOPS, and throughput.

If performance parameters are reduced during restore:

• Kafka disk flush latency may increase

• Log segment recovery may slow

• Consumer lag may spike

• ZooKeeper session instability may occur

Storage migration must preserve performance characteristics, not just data.

Step 6 — Delete Replica PVCs

kubectl delete pvc datadir-zk-staging-1 -n tools
kubectl delete pvc datalogdir-zk-staging-1 -n tools

Because the reclaim policy is set to Retain, the underlying EBS volumes will not be deleted.

Step 7 — Static PV Restore YAML

# restore-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-restore-datadir-1
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-RESTORED-DATADIR
    fsType: ext4
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - ap-south-1b
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-restore-datalogdir-1
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-RESTORED-DATALOGDIR
    fsType: ext4
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - ap-south-1b

Apply:

kubectl apply -f restore-pv.yaml
kubectl get pv

Step 8 — Restore PVC YAML

# restore-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-zk-staging-1
  namespace: tools
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-restore-datadir-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datalogdir-zk-staging-1
  namespace: tools
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-restore-datalogdir-1

Apply and verify:

kubectl apply -f restore-pvc.yaml
kubectl get pv,pvc -n tools

Step 9 — Restrict Scheduling to Single AZ

Update nodeAffinity to:

ap-south-1b

Apply updated StatefulSet.

Step 10 — Safe Staged Bring-Up

kubectl scale statefulset zk-staging -n tools --replicas=1

Validate stability.

Then:

kubectl scale statefulset zk-staging -n tools --replicas=2

Migration complete.

8. Architecture After Migration

Inter-AZ data transfer cost is eliminated because both replicas now operate within ap-south-1b, but AZ-level redundancy is removed.

Capacity Validation Before Consolidation

Before consolidating both replicas into a single Availability Zone, validate:

• Worker node CPU headroom

• Available memory capacity

• EBS volume attachment limits per node

• Network bandwidth availability

• Failure of ap-south-1b will now result in a full service outage until recovery.

Single-AZ consolidation increases resource contention risk and expands the blast radius.

Cost optimization must not introduce saturation instability.

Trade-Off

Single AZ failure = Full outage.

Availability ↓

Cost ↓

Intentional architectural decision.

Rollback Strategy

If issues occur:

1. Scale down

2. Restore original snapshots in their original Availability Zones.

3. Recreate original PV bindings

4. Revert nodeAffinity

5. Scale up gradually

Final Takeaway

This migration was not a scheduler tweak.

It was a storage topology redesign.

Stateful workloads are constrained by storage locality.

Storage must move before pods move.

What I Designed

I designed a hybrid infrastructure architecture:

• Terraform → Foundation Layer

• Crossplane → Dynamic Lifecycle Layer

• ArgoCD → GitOps Enforcement

This created a continuously reconciling cloud control plane inside Kubernetes.

Why It Was Required

Our platform crossed:

• 50+ microservices

• Multiple engineering teams

• Multi-region expansion

• PR-driven infrastructure workflows

• Feature branch–based short-lived environments

Terraform workflows became operationally slow due to:

• State contention

• Long plan times

• PR bottlenecks

• Manual drift detection

Provisioning was working.
Lifecycle control was missing.

Constraint

• No full rewrite

• No infrastructure instability

• Zero data loss

• Minimal migration risk

Engineering Principle Followed

Separate provisioning from lifecycle reconciliation.

Provision once.
Reconcile continuously.

2. Problem Statement

Existing Architecture

Terraform model:

Plan → Apply → Exit

After apply:

• No continuous reconciliation

• Drift detection only on next plan

• Manual console changes remain undetected

Failure Risk

If an engineer modified:

• RDS storage encryption

• Deletion protection

• Security groups

• IAM policies

Terraform would not react until the next plan/apply cycle.

Drift became silent operational risk.

What Would Break

• Compliance posture

• Backup guarantees

• Encryption enforcement

• Network boundaries

• Incident recovery confidence

Why It Was Unacceptable

At scale:

Manual governance does not work.

Infrastructure must enforce its declared state.

3. Architecture After Implementation

Control plane flow:

Developer commits YAML
↓
ArgoCD syncs to cluster
↓
Kubernetes API stores desired state
↓
Crossplane controller watches resource
↓
Crossplane calls AWS API
↓
Cloud resource created/updated
↑
Continuous reconciliation loop

Terraform foundation layer:

Terraform
↓
VPC
Subnets
EKS Control Plane
Core Networking

Clear separation of responsibilities.

4. Design Decisions

4.1 Core Component Choice

Terraform for foundation

Why I chose it:

• Mature state handling

• Strong bootstrap ecosystem

• Clear isolation of foundational infrastructure

Multi-account governance was enforced via separate state isolation and account-factory patterns; Terraform itself does not natively provide org-level governance.

Trade-off:

• No continuous reconciliation

Risk accepted:

• Foundation changes are rare and tightly controlled

Crossplane for dynamic infrastructure

Why I chose it:

• Kubernetes-native control loop

• GitOps-friendly

• CRD-based lifecycle management

Trade-off:

• Adds API server load

• Adds controller complexity

Risk accepted:

• Infrastructure lifecycle now depends on cluster health

4.2 Failure Detection

Reconciliation loop ensures:

Actual State == Desired State

Manual console change → Crossplane reconciles.

Trade-off:

• AWS API throttling possible

• Eventual consistency delays

Risk accepted:

• Tuned provider/controller concurrency and AWS API backoff settings to mitigate throttling

4.3 Event Routing

Git Commit
→ ArgoCD
→ Kubernetes API
→ Crossplane Controller
→ AWS API

Rollback = git revert.

Trade-off:

• Git becomes critical dependency

Risk accepted:

• Strong repository governance and PR controls

4.4 Automation Logic

Platform team defined Compositions.

Example:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xpostgres
spec:
  compositeTypeRef:
    apiVersion: platform.io/v1alpha1
    kind: XPostgres
  resources:
    - name: database
      base:
        apiVersion: database.aws.crossplane.io/v1beta1
        kind: RDSInstance
        spec:
          deletionPolicy: Orphan
          forProvider:
            storageEncrypted: true
            deletionProtection: true

Application team used Claim:

apiVersion: platform.io/v1alpha1
kind: PostgresClaim
metadata:
  name: app-db
spec:
  parameters:
    storage: 20

Why I chose this:

• Central policy enforcement

• Developer abstraction

• Clear ownership boundary

Trade-off:

• Composition update blast radius

• Requires versioning discipline

Risk accepted:

• Versioned compositions per environment

4.5 DNS / Networking

DNS and core networking remained Terraform-managed.

Reason:

• High blast radius

• Low change frequency

• Complex dependency graph

Control plane expansion was phased deliberately.

5. Implementation Snippet

Install Crossplane:

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace

Install AWS Provider:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.54.2

Configure ProviderConfig (IRSA recommended):

apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws
spec:
  credentials:
    source: IRSA

ProviderConfig was configured using IRSA to avoid static credentials.

RDS Example:

apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: platform-db
spec:
  deletionPolicy: Orphan
  forProvider:
    region: us-east-1
    dbInstanceClass: db.t3.micro
    allocatedStorage: 20
    engine: postgres
    storageEncrypted: true
    deletionProtection: true
  providerConfigRef:
    name: aws

Assumes a default VPC and subnet group already exist; in hardened environments, explicit subnetGroupName and securityGroupIds must be specified.

6. Traffic / DNS Consideration

Database endpoints remained AWS-managed.

No DNS switching automated through Crossplane.

Reason:

• Database endpoints are stable

• DNS manipulation has high blast radius

• Networking remained Terraform-owned

7. Validation Process

Step 1 – Manual Drift Simulation

Modified RDS parameter in AWS Console.

Observed:

• Crossplane detected change

• Reconciliation restored desired state

Step 2 – Deletion Policy Test

Deleted CR with:

deletionPolicy: Orphan

Observed:

• Cloud resource retained

• CR removed

Step 3 – Delete Mode Test

Changed to:

deletionPolicy: Delete

Deleted CR.

Observed:

• Cloud resource removed

Lifecycle behavior verified.

Step 4 – API Throttling Simulation

Created multiple resources in parallel.

Observed:

• AWS API throttling errors

• Provider retry with exponential backoff

Validated concurrency and backoff tuning necessity.

8. Cost & Trade-offs

Infrastructure Cost

• Additional Crossplane controller pods

• Increased etcd object count

• Higher AWS API call volume

Cost impact: Moderate.

Operational Complexity

Increased:

• Controller debugging

• Composition versioning

• CRD lifecycle management

Reduced:

• Manual drift remediation

• Terraform PR bottlenecks

• Apply-time surprises

RTO

Improved.

Drift auto-corrected without manual intervention.

RPO

No direct change.

Depends on AWS-native backup policies.

Scaling Impact

Pros:

• Safe self-service for app teams

• Git-auditable infrastructure

• Continuous compliance enforcement

Cons:

• API server load increases

• AWS rate limit sensitivity

• Composition update blast radius

9. When This Design Makes Sense

✔️ 50+ services
✔️ Dedicated platform team
✔️ GitOps maturity
✔️ Kubernetes-native organization
✔️ High infrastructure churn

When NOT to Use It

• Small teams

• Low churn infrastructure

• No Kubernetes maturity

• Multi-account bootstrap phase

• Extremely complex networking requirements

10. Final Takeaway

Terraform builds infrastructure.

Crossplane manages the infrastructure lifecycle.

GitOps enforces declared intent.

This was not a tool replacement exercise.

It was an architectural shift from:

Provisioning mindset → Control plane mindset

At scale, lifecycle enforcement matters more than provisioning speed.

Hybrid architecture made lifecycle enforcement operationally viable at scale.

It is:

• An access model transformation

• A replication lifecycle management exercise

• A downtime control operation

• A cost boundary decision

We executed a production-grade migration from Google Compute Engine to Amazon EC2 using AWS Application Migration Service (MGN) under strict operational constraints:

• Target RPO ≈ 0 at cutover (achieved through application write freeze, replication validation, and successful final synchronization)

• Downtime < 30 minutes

• No SSH/RDP lockout

• Controlled replication cost

• Preserved rollback capability

This article documents a technically precise, enterprise-review-safe execution model.

1. Why We Chose AWS MGN

We evaluated:

• Manual snapshot export/import

• Image-based migration

• Backup/restore workflows

• Application rebuild

We selected AWS MGN because:

• Continuous block-level replication minimizes downtime

• Test migrations do not impact production

• Cutover is controlled and repeatable

• No application redesign required (pure rehost model)

This was lift-and-shift, not modernization.

2. What AWS MGN Actually Does

AWS MGN is an agent-based block-level replication service.

It:

• Installs a replication agent on the source VM

• Reads disk blocks at the operating system level

• Continuously replicates encrypted data to AWS

• Uses staging replication servers and EBS volumes

• Launches test and cutover EC2 instances

Critical Clarification

MGN replicates disk state.

It migrates:

• Local OS users stored on disk

• SSH configuration files stored on disk

• Application binaries

• System configuration

It does not migrate:

• GCP OS Login (IAM-based access)

• Metadata-injected SSH keys

• DNS records

• Load balancers

• Managed services (Cloud SQL, etc.)

• IAM identities

Any identity mechanism tied to GCP metadata or IAM must be replaced with persistent OS-level users or SSM-based access prior to cutover.

3. Migration Architecture

GCP VM
↓
MGN Replication Agent
↓
Encrypted TLS Transfer
↓
AWS Staging Replication Server
↓
EBS Replication Volumes
↓
Launch Template
↓
Test EC2 Instance
↓
Cutover EC2 Instance

Replication continues until cutover.

• Data encrypted in transit (TLS)

• EBS encryption configurable at rest

• Replication throughput depends on bandwidth, write churn, and staging configuration

4. MGN Lifecycle States (Operational Meaning)

Not Ready

Initial full replication in progress.

Ready for Testing

Initial synchronization complete. Continuous delta replication active.

Test in Progress

Test EC2 launched. Source VM continues running.

Ready for Cutover

Replication functioning correctly with minimal lag.

Production Cutover Gate

Proceed only if:

• Migration lifecycle = Ready for cutover

• Replication status = Healthy

• Replication lag is negligible or zero

Never execute cutover if replication is not Healthy.

Cutover in Progress

Final synchronization executing and EC2 launching.

Cutover Complete

AWS EC2 becomes the active production workload.

5. Primary Risk: Access Model Mismatch

Default GCP access model:

• OS Login (IAM-based)

• Metadata-injected SSH

• Console SSH

Default AWS access model:

• OS-level local users

• SSH key pairs

• SSM Session Manager

These models are incompatible.

If a VM relies exclusively on GCP OS Login or metadata-based SSH, administrative access will be lost after migration.

6. Mandatory Pre-Migration Access Preparation

Linux

Create a persistent administrative user:

sudo useradd awsadmin
sudo mkdir -p /home/awsadmin/.ssh
sudo chmod 700 /home/awsadmin/.ssh
sudo nano /home/awsadmin/.ssh/authorized_keys
sudo chmod 600 /home/awsadmin/.ssh/authorized_keys
sudo chown -R awsadmin:awsadmin /home/awsadmin
sudo usermod -aG sudo awsadmin

Validate SSH access before migration.

Attach IAM role:

AmazonSSMManagedInstanceCore

This ensures emergency fallback access via Session Manager.

Windows

• Enable local Administrator

• Enable RDP

• Allow TCP 3389

• Validate login before migration

7. Network Requirements

On GCP VM (Outbound Required)

No inbound firewall rules are required on the source VM for MGN.

On AWS

If staging and target instances reside in private subnets, outbound connectivity must be provided via:

• NAT Gateway

• VPC Endpoints

• Direct Connect

• VPN

NAT is required only if no internet access or VPC endpoints are configured.

8. Replication Consistency Model

AWS MGN provides crash-consistent replication.

To achieve near-zero RPO:

1. Freeze application writes

2. Confirm replication status = Healthy

3. Confirm replication lag ≈ 0

4. Execute cutover

For database workloads, application quiescing is mandatory.

MGN does not provide automatic application-consistent snapshots.

9. Replication Timing Reality

Initial synchronization depends on:

• Disk size

• Network bandwidth

• IOPS

• Write churn

Example (environment dependent):

• 50 GB → 20–40 minutes

• 200 GB → 1–2 hours

• 1 TB → Dependent on available bandwidth

Downtime is not determined by disk size.

Downtime = write freeze duration + final synchronization time.

10. Cutover Execution Model

Validated production sequence:

1. Freeze application writes

2. Confirm replication status = Healthy

3. Confirm lag negligible

4. Launch cutover instance

5. Stop GCP VM

6. Update DNS or routing

Observed downtime: 5–30 minutes.

11. Rollback Strategy

If validation fails:

• Do not delete source VM

• Restore DNS to GCP

• Maintain replication configuration

• Reattempt cutover

Never decommission the source environment until fully validated.

12. Cost and Scale Considerations

AWS Costs

• Staging replication servers

• EBS volumes

• Snapshots

• NAT (if used)

• Data transfer

GCP Cost

• Network egress (primary cost driver)

Large parallel migrations may:

• Increase staging cost

• Increase EBS consumption

• Saturate bandwidth

• Trigger API throttling

Recommendation: Execute migrations in controlled waves.

13. When This Approach Makes Sense

Suitable for:

• Lift-and-shift

• Data center exit

• Legacy workload relocation

• Time-constrained transitions

Not suitable for:

• Cloud-native redesign

• Container migrations

• Managed database modernization

• Application refactoring

Final Engineering Conclusion

AWS Application Migration Service is predictable when:

• Access models are reconciled

• OS compatibility is validated

• Replication health and lag are monitored

• Cutover is controlled

• Rollback capability is preserved

Most migration failures are not replication failures.

They are planning failures.

Cross-cloud VM migration is an operational engineering discipline — not a file transfer task.

Can it actually reduce investigation time during real production-style failures?

To answer that, I tested it using controlled failure scenarios instead of relying purely on documentation.

What This Article Covers

• What it actually does

• How it works architecturally

• What I observed during testing

• Where it helps

• Where it does not

No marketing. Only practical evaluation.

What Is AWS DevOps Agent?

AWS DevOps Agent is an AI-powered investigation capability that analyzes AWS telemetry and generates structured incident timelines with evidence-backed probable causes.

It is not:

• A chatbot

• An auto-remediation engine

• A monitoring replacement

It does not generate telemetry.

It consumes existing signals and correlates them.

High-Level Architecture

Application Workload
        ↓
AWS Services (EC2 / RDS / EKS / ALB / Lambda)
        ↓
CloudWatch Metrics + Logs + Events
        ↓
AWS DevOps Agent Correlation Engine
        ↓
Incident Timeline + Evidence + Root Cause Hypothesis

It acts as a reasoning layer on top of CloudWatch telemetry.

Monitoring detects the problem.

The DevOps Agent explains it.

It does not independently detect incidents — it analyzes them after alerts are triggered.

Where It Gets Its Data

The DevOps Agent analyzes signals primarily from:

• Amazon CloudWatch (metrics & logs)

• AWS resource configuration events

• Control plane activity

• Deployment-related changes

Common services involved during investigation include:

• Amazon EC2

• Amazon RDS

• Amazon EKS

• Elastic Load Balancing

• AWS Lambda

If metrics and logs are incomplete, investigation quality drops.

Observability maturity directly affects output quality.

Testing Scenario 1: High CPU on Burstable EC2

Setup

• Burstable EC2 instance

• Sustained workload applied

• Manual SSH session before spike

Symptoms

• High CPU alarm

• Increased latency

What the Agent Correlated

• CPUUtilization spike

• CPUCreditBalance drop

• Increased NetworkIn and NetworkOut metrics

• SSH login event

Conclusion

Sustained workload exhausted burst credits.

This was expected behavior for a burstable instance — not infrastructure failure.

Instead of manually checking multiple dashboards, the agent produced a structured investigation timeline.

Testing Scenario 2: Application Down (Nginx Configuration Error)

Setup

• Manual Nginx configuration change

• Introduced an invalid directive

• Restarted service

Symptoms

• Website inaccessible

• Instance healthy

• No CPU or memory pressure

What the Agent Correlated

• Service restart failure

• Configuration change event

• Application log error

• No correlated resource exhaustion

Conclusion

Application configuration error.

Not a scaling or capacity issue.

The agent correctly separated this from the earlier CPU incident.

Operational Impact

The biggest benefit was not detection — it was compression.

What normally requires:

• Checking multiple dashboards

• Reviewing deployment history

• Inspecting logs manually

Was presented as a structured investigation narrative.

That compression directly impacts MTTR.

In short, DevOps Agent improves explanation quality — not detection capability.

What Worked Well

Timeline Clarity

It clearly shows:

• What changed

• When it changed

• What metrics moved

• What correlated

This reduces guesswork during incidents.

Multi-Signal Correlation

It combines:

• Metrics

• Logs

• Configuration changes

• Access events

This cross-signal reasoning improves investigation speed.

Issue Isolation

Multiple issues can overlap in production.

The DevOps Agent attempts to isolate causal chains instead of merging everything into one root cause.

That improves RCA accuracy.

Limitations

No Deep Application Debugging

It cannot analyze:

• Business logic bugs

• Runtime memory leaks

• Thread-level behavior

Unless those signals are exposed through telemetry.

SSH Blind Spot

Commands executed via SSH are invisible unless command logging is enabled.

Proper logging discipline is required.

Observability Dependency

Insight quality depends on:

• Log completeness

• Metric granularity

• Tagging consistency

• Retention strategy

Weak telemetry produces weak conclusions.

Cost Considerations

Because it operates on CloudWatch telemetry, overall cost is tied to observability depth.

Cost drivers include:

• Log ingestion volume

• Metric storage

• Retention duration

The DevOps Agent itself is not the primary cost driver.

CloudWatch log ingestion, retention policies, and metric granularity determine overall observability spend.

Organizations must balance investigation visibility with cost control.

When It Makes Sense

Best suited for:

• Multi-service AWS architectures

• Teams handling frequent incidents

• Organizations aiming to reduce MTTR

• Standardizing RCA processes

Less useful when:

• Infrastructure is extremely simple

• Logging is minimal

• Systems are mostly outside AWS

Final Thoughts

AWS DevOps Agent should be positioned as:

• An investigation accelerator

• A structured reasoning layer over telemetry

• An MTTR reduction enabler

• A standardization tool for incident analysis

It does not replace engineers.

It amplifies the quality of your existing observability.

It is most effective in mature environments with structured logging and tagging standards.

Strong telemetry in → structured reasoning out.

Weak telemetry in → weak conclusions out.

We migrated 10+ TB of object data from Google Cloud Storage to Amazon S3 under strict enterprise constraints:

• Zero data loss

• No public exposure

• No long-lived credentials

• Private networking compatibility

• Full audit traceability

• Controlled and predictable cost

This document describes the architecture and execution model validated for production use.

1. The Engineering Constraint

In cross-cloud migration, the execution location determines financial and security risk.

If migration runs outside GCP:

• GCS internet egress charges apply

• Traffic traverses public endpoints

• Credential exposure surface increases

• Cost becomes unpredictable at scale

At a multi-terabyte volume, this is unacceptable.

The objective was clear:

Eliminate public egress from GCS while maintaining integrity and operational control.

2. Architectural Design

Migration was executed inside GCP using rclone on a Google Compute Engine (GCE) VM.

Data Flow

GCS Bucket
   ↓
GCE VM (rclone)
   ↓
HTTPS
   ↓
S3 Bucket

Impact of this design:

• GCS → VM traffic remains internal

• No GCS public egress billing

• AWS inbound transfer remains free

• Execution remains fully controlled

The primary cost risk was removed architecturally, not operationally.

3. Deployment Modes Validated

The design was tested under:

This ensures compatibility with both PoC and hardened enterprise environments.

4. Network & Security Model

GCP Side

• Private VM (no external IP)

• Private Google Access enabled

• Cloud NAT or VPN for outbound traffic

• No inbound exposure

• Metadata-based IAM authentication

No service account JSON keys were used.

Access to GCS was restricted to the VM-attached service account.

AWS Side

Two supported access patterns:

Public S3 (PoC only)

Standard endpoint with IAM control.

Private S3 (Production)

• S3 Gateway VPC Endpoint

• Bucket policy restricted using aws:SourceVpce

• No public S3 exposure

Example condition:

{
  "Condition": {
    "StringEquals": {
      "aws:SourceVpce": "vpce-xxxxxxxx"
    }
  }
}

Traffic remains private across environments.

5. Authentication Strategy

GCS

[gcs]
type = google cloud storage
env_auth = true

• VM-attached service account

• Metadata server authentication

• No static credential storage

AWS

• PoC: Temporary access key

• Production: IAM Role / STS

No long-lived credentials were introduced.

6. Migration Execution Model

Execution followed controlled phases.

Phase 1 — Pre-Flight Validation

• IAM verification

• DNS resolution check

• Connectivity confirmation

• Source and destination listing

Mandatory dry-run:

rclone copy gcs:<bucket> s3:<bucket> \
  --dry-run --checksum

No migration proceeded without validation.

Phase 2 — Controlled Transfer

rclone copy gcs:<bucket> s3:<bucket> \
  --checksum \
  --fast-list \
  --transfers=8 \
  --checkers=8 \
  --progress

Controls enforced:

• Checksum validation

• Resume-safe execution

• Tuned concurrency

• Encrypted HTTPS transport

Concurrency was deliberately limited to prevent throttling.

Phase 3 — Integrity Verification

rclone check gcs:<bucket> s3:<bucket>

This ensured:

• No checksum mismatches

• No partial transfers

• No silent corruption

Logs and artifacts were archived for audit compliance.

7. Cost Model (5 TB Reference)

If executed externally, GCS internet egress alone would exceed this amount.

Cost predictability was achieved through architectural control.

8. Alternatives Considered

Managed services were evaluated.

Observed trade-offs:

• Per-GB transfer charges

• Reduced retry visibility

• Limited private networking control

• Additional service dependency

For small migrations, managed services are acceptable.

For enterprise-scale workloads requiring cost governance and auditability, direct execution provides stronger guarantees.

9. Outcome

• Zero data loss

• No public exposure

• IAM-based authentication

• Private networking compatibility

• Full checksum validation

• Resume-safe execution

• Audit-ready logs

• Production approval

10. Conclusion

Cross-cloud storage migration is not about moving objects.

It is about defining the correct execution boundary.

By executing the migration inside GCP, we eliminated public egress cost, preserved private networking, reduced credential risk, and maintained deterministic control over the entire process.

When execution placement is correct, migration risk becomes controlled, cost becomes predictable, and integrity becomes measurable.

Export snapshot → Restore in another account.

In practice, it is not that straightforward.

In this case, we migrated Redis OSS from Account A to Account B in the same region (ap-south-1) using default encryption (SSE-S3).

This post documents:

• Why snapshot export initially failed

• Why cross-account restore failed

• What actually worked

• The production-safe migration pattern

No assumptions. Only what was tested and validated.

1. Architecture Overview

Source Account (Account A)
Redis OSS → Manual Snapshot → Export to S3

Target Account (Account B)
S3 (copied snapshot) → Restore Redis OSS

Key constraint:
No public bucket. No insecure workaround. Production-safe design.

2. Phase 1 — Snapshot Export to S3 (Why It Failed)

When exporting the snapshot from ElastiCache to S3, we encountered:

ElastiCache was unable to validate the authenticated user has access to the S3 bucket

At that time:

• Bucket existed

• Same region (ap-south-1)

• Block public access enabled

• SSE-S3 encryption enabled

• Bucket policy configured

Yet export failed.

Root Cause

In standard AWS regions, ElastiCache export and seed operations commonly require ACL-based grants using the service Canonical ID.
In opt-in regions, AWS also documents a bucket policy–based approach using the elasticache-snapshot service principal.

Modern S3 guidance promotes disabling ACLs, but Redis export requires:

• ACLs enabled

• ElastiCache Canonical ID added

Without ACL configuration, export validation fails.

Correct Export Configuration

Step 1 — Enable ACLs

S3 → Bucket → Permissions → Object Ownership

Change from:

Bucket owner enforced

To:

ACLs enabled (Bucket owner preferred)

Save changes.

Step 2 — Add ElastiCache Canonical ID

S3 → Bucket → Permissions → ACL → Edit

Add Canonical ID:

540804c33a284a299d2547575ce1010f2312ef3da9b3a053c8bc45bf233e4353

Grant:

• Objects → List, Write

• Bucket ACL → Read, Write

Important:

• Block Public Access remained enabled

• No “Everyone” permission added

After this configuration, the snapshot export succeeded.

For standard (non-GovCloud) AWS regions, ElastiCache uses the same Canonical ID. Only GovCloud regions use different Canonical IDs. Always verify from AWS official documentation before configuring.

3. Phase 2 — Cross-Account Restore Attempt

Snapshot successfully exported to:

s3://redis-oss-backup-bucket/redis-oss-backup-0001.rdb

The next step was restoring in Account B.

The restore operation is executed by the ElastiCache service in the target account, and cross-account restore requires exact service principal and permission configuration, and our bucket policy did not satisfy those requirements.

Attempt — Cross-Account Bucket Policy

We initially configured a bucket policy allowing the ElastiCache service principal access. However, snapshot seeding requires the correct regional elasticache-snapshot service principal and specific permissions (including s3:GetBucketAcl). Our configuration did not match the documented requirement, which contributed to restore failure.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowElastiCacheFromAccountB",
      "Effect": "Allow",
      "Principal": {
        "Service": "elasticache.amazonaws.com"
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::redis-oss-backup-bucket",
        "arn:aws:s3:::redis-oss-backup-bucket/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT_B_ID"
        }
      }
    }
  ]
}

Restore failed with:

No permission to access S3 object

Even though:

• Region matched

• Object key was correct

• SSE-S3 encryption used

• Service-linked role existed

Restore did not succeed.

Why It Failed

In our tested configuration, cross-account restore required more precise service principal permissions than our implementation provided.

The restore process expects:

• Bucket ownership alignment

• Same-account execution boundary

In our tested implementation, the cross-account restore failed with the bucket policy configuration we applied. Rather than iterating further on service-layer permission nuances, we adopted a production-safe pattern: copy the snapshot into an S3 bucket owned by the target account and restore from there.

4. Diagnostic Test — Public Object

To isolate the issue, we temporarily made the object public.

Restore succeeded immediately.

This confirmed:

• Snapshot file was valid

• Region correct

• Encryption is not the issue

• The permission boundary was the root cause

Public access is not production-safe and was used only for troubleshooting.

5. Final Production-Safe Migration Pattern

Instead of forcing cross-account restore, we redesigned the architecture.

Correct Approach

Account A
Export snapshot → S3 bucket A

↓

Secure object copy

↓

Account B
Restore from S3 bucket B

This ensures ownership alignment and eliminates the cross-account restore boundary.

6. Implementation Steps

Step 1 — Create S3 Bucket in Account B

• Region: ap-south-1

• Block Public Access: Enabled

• Object Ownership: Bucket owner enforced

• Default Encryption: SSE-S3

Step 2 — Copy Snapshot Securely

aws s3 cp s3://redis-oss-backup-bucket/redis-oss-backup-0001.rdb \
s3://redis-oss-backup-bucket-b/redis-oss-backup-0001.rdb \
--source-region ap-south-1 \
--region ap-south-1

After copy:

• Object owner becomes Account B

• No cross-account policy required

Ownership alignment resolves the restore issue.

Step 3 — Restore in Account B

Use:

redis-oss-backup-bucket-b/redis-oss-backup-0001.rdb

Restore completed successfully.

7. When to Use Replication Instead

If the requirement is disaster recovery or continuous synchronization:

Use S3 Cross-Account Replication.

For one-time migration, manual secure copy is simpler and cleaner.

8. Key Engineering Takeaways

1. In standard regions, Redis OSS export typically requires ACL configuration with the ElastiCache Canonical ID.

2. Cross-account restore requires precise configuration of the service principal and permissions.

3. Ownership alignment via secure object copy eliminates cross-account complexity and reduces operational risk.

4. SSE-S3 simplifies cross-account migration.

5. Public object works but is insecure.

6. Ownership alignment is the key requirement.

7. Secure object copy is the cleanest production solution.

Final Principle

When migrating workloads across AWS accounts:

Do not force cross-account service execution boundaries when the service is not designed to support them.

Move the data.

Align ownership.

Restore cleanly.

That is production-grade cloud engineering.

This post documents how I designed and implemented cross-region disaster recovery for a production MySQL database running on Amazon RDS.

The requirement was straightforward:

If the primary region (ap-south-1) becomes unavailable, the database must recover automatically from another region with minimal downtime and no manual intervention.

The approach relies entirely on native AWS services and focuses on controlled automation rather than introducing additional orchestration layers.

2. Problem Statement

The application was running with:

• A single primary RDS instance in ap-south-1

• No cross-region replica

• A manual disaster recovery procedure

In the event of a regional failure:

• The database endpoint would become unreachable

• The application would lose connectivity

• Recovery would depend on human response time

This created an unacceptable recovery model for a production-facing workload.

The goal was to introduce regional resilience while keeping the system operationally predictable and architecturally simple.

3. Architecture After DR Implementation

Application
↓
Primary RDS (ap-south-1)
↓
Cross-Region Read Replica (us-east-1)

Automation Flow:

CloudWatch Alarm
→ SNS Topic
→ Lambda (us-east-1)
→ Promote Read Replica

After promotion, the replica becomes a standalone primary database.

4. Design Decisions

4.1 Cross-Region Read Replica

A read replica was created in us-east-1 from the primary RDS instance.

Replication is managed natively by RDS.

Important consideration:

Cross-region replication is asynchronous. This introduces a non-zero Recovery Point Objective (RPO). A small amount of recent data may not be replicated during failover.

For this workload, that trade-off was acceptable.

4.2 Failure Detection

Failure detection was implemented using Amazon CloudWatch.

A CloudWatch alarm was configured on the primary instance. The initial metric used was:

DatabaseConnections < 1

If the instance stops accepting connections, the alarm transitions to ALARM state.

In more mature setups, detection can be improved using:

• RDS instance status

• RDS event notifications

• Composite alarms

The alarm publishes its state change to an SNS topic.

4.3 Event Routing

Amazon Simple Notification Service was introduced to decouple monitoring from execution.

The flow:

CloudWatch → SNS → Lambda

This separation provides:

• Clear event-driven architecture

• Loose coupling

• Extensibility for future automation

4.4 Automated Promotion

Promotion logic was implemented using AWS Lambda in us-east-1.

The Lambda role was granted minimal permission:

rds:PromoteReadReplica

Core logic:

import boto3

def lambda_handler(event, context):
    client = boto3.client('rds', region_name='us-east-1')

    replica_id = "your-read-replica-id"

    client.promote_read_replica(
        DBInstanceIdentifier=replica_id
    )

    return {
        "statusCode": 200,
        "message": f"Promotion initiated for {replica_id}"
    }

When triggered, Lambda initiates promotion of the replica. Promotion typically completes within a few minutes.

5. DNS Consideration

Promotion alone does not restore application traffic.

The application must connect to the new primary endpoint.

This is typically handled using Amazon Route 53 with:

• Failover routing policies

• Health checks

• Low TTL configuration

Without DNS integration, traffic would continue to target the failed endpoint.

This is a critical part of the disaster recovery strategy.

6. Validation Process

To validate the setup:

• The primary instance was stopped

• CloudWatch alarm triggered

• SNS published the event

• Lambda executed

• The read replica was promoted

Post-promotion validation included:

• Verifying instance state

• Testing database connectivity

• Executing queries

• Confirming application behavior

Failover occurred without manual intervention.

7. Cost and Trade-offs

Cross-region disaster recovery increases infrastructure cost:

• An additional RDS instance

• Cross-region replication traffic

• Additional storage

This cost is deliberate and justified by the reduction in downtime risk.

RTO: A few minutes (detection + promotion time)
RPO: A few seconds (due to asynchronous replication)

These expectations must be clearly defined with stakeholders before implementation.

8. When This Approach Makes Sense

This architecture is suitable when:

• A few seconds of data loss is acceptable

• Automated regional failover is required

• Operational simplicity is preferred

• The workload is not using Aurora Global Database

For workloads requiring near-zero RPO and faster failover, managed global database architectures may be more appropriate.

9. Final Takeaway

I implemented a cross-region disaster recovery architecture for Amazon RDS using native AWS services.

The system:

• Detects primary failure automatically

• Triggers event-driven automation

• Promotes a cross-region replica

• Reduces human dependency during outages

The design balances simplicity, cost, and resilience. It introduces regional fault tolerance without adding unnecessary operational complexity.

Amazon Linux 2 (AL2) will reach end-of-support on June 30, 2026. After that date, AWS will no longer provide security updates, patches, or new packages.

Although AL2 continues to receive maintenance updates today, it is no longer the forward-looking platform. Amazon Linux 2023 (AL2023) is the long-term replacement, offering a predictable 5-year lifecycle per release (2 years standard + 3 years maintenance) along with modernized system components.

If you are running production workloads on AL2, migration should be planned early — not rushed near 2026.

This article explains how to safely migrate in a large production environment with:

• Standalone EC2 instances

• Auto Scaling Groups (ASG)

• Amazon EKS worker nodes

Assume a worst-case environment of 200 AL2 instances.

Why In-Place Upgrade Is Not Recommended

There is no supported in-place upgrade path from AL2 to AL2023.

AL2023 introduces:

• Updated kernel

• Newer system libraries

• Updated OpenSSL and crypto policies

• cgroup v2 by default

• Updated container runtime stack

Attempting in-place OS mutation:

• Is unsupported

• Is difficult to test

• Has no clean rollback

• Is unsafe for Kubernetes worker nodes

The correct production pattern is:

Build new → Validate → Controlled cutover → Preserve rollback → Decommission old

Example Production Layout (200 Instances)

Each workload type requires a different strategy.

1. Standalone EC2 (Stateful Systems)

Typical Examples

• Databases

• Legacy applications

• EC2 instances using Elastic IP

• Applications dependent on local storage

Step 1 – Create a Safety Checkpoint

Before any migration:

• Create EBS snapshots

• Confirm snapshot completion

This is your rollback baseline.

Step 2 – Launch Parallel AL2023 Instance

Create a new EC2 instance with:

• Same VPC and subnet

• Same security groups

• Same IAM role

• Same instance type

Do not modify the AL2 instance.

Step 3 – Validate Dependency Compatibility

Test explicitly:

• Runtime versions (Java, Python, Node)

• OpenSSL behavior

• Crypto policy differences

• Systemd services

• Hardcoded paths in custom scripts

Small OS-level changes can break production services.

Step 4 – Controlled Data Migration

1. Initial sync while application is running

2. Stop or freeze writes

3. Final sync with checksum validation

Example:

rsync -avh --checksum --numeric-ids /data/ new-server:/data/

For database systems, prefer logical dump/restore over raw filesystem copy to avoid corruption risk.

Step 5 – Validate Before Cutover

Start the application on AL2023 and verify:

• Application logs

• Health endpoints

• Downstream connectivity

• Resource utilization

Only after validation:

• Stop service on AL2

• Switch Elastic IP or DNS

Keep AL2 intact until full confidence.

2. Auto Scaling Groups (Stateless Systems)

Typical Examples

• Web servers

• APIs

• Microservices

The main risk is pushing a broken AMI to the entire fleet.

Step 1 – Build an AL2023 Golden AMI

Include:

• Monitoring agents

• Security agents

• Logging agents

• Application bootstrap scripts

Test full userdata execution.
Simulate instance termination to confirm auto-recovery.

Step 2 – Create New Launch Template Version

Update only:

• AMI ID

Keep AL2 template available for rollback.

Step 3 – Canary Deployment

Increase desired capacity by 1.

Validate:

• Load balancer health checks

• Application startup

• Logs and error rate

• Metrics stability

Do not skip canary testing.

Step 4 – Controlled Instance Refresh

Use safe rollout configuration:

• Configure minimum healthy percentage appropriate for fleet size (e.g., 90–100% for large fleets).

• Warm-up time configured

• ELB health checks enabled

Monitor closely during rollout.

If instability occurs:

• Cancel refresh

• Revert launch template

3. Amazon EKS Worker Nodes (Highest Blast Radius)

AL2023 introduces changes that can affect Kubernetes workloads:

• cgroup v2

• Updated kernel

• Updated container runtime

This can impact:

• DaemonSets

• Monitoring agents

• Security tooling

• CNI plugins

Safe EKS Migration Flow

Step 1 – Add AL2023 Node Group

Create a new managed node group (or Karpenter pool).
Do not modify AL2 nodes yet.

Step 2 – Taint AL2 Nodes

kubectl taint node <al2-node> os=al2:NoSchedule

Effect:

• No new pods schedule on AL2

• Existing pods continue running

Step 3 – Validate Scheduling on AL2023

Scale workloads or deploy new services.

Confirm:

• Pods schedule on AL2023

• Networking functions correctly

• Metrics and logs flow normally

Step 4 – Validate Cluster Add-ons and DaemonSets

Check:

• VPC CNI

• CoreDNS

• kube-proxy

• Logging agents

• Monitoring agents

• Security tools

Ensure cluster add-ons (VPC CNI, CoreDNS, kube-proxy) versions are compatible with AL2023 node AMIs before rollout.

Also verify PodDisruptionBudgets:

kubectl get pdb -A

Ignoring PDBs can cause draining failures or partial outages.

Step 5 – Drain AL2 Nodes

kubectl drain <node> --ignore-daemonsets

Observe workload behavior during drain.

Step 6 – Delete AL2 Node Group

Delete only after full stability confirmation.

Rollback is possible only until deletion.

Common Production Mistakes

• Attempting in-place OS upgrades

• Skipping canary validation

• Ignoring bootstrap script testing

• Draining EKS nodes prematurely

• Not snapshotting stateful systems

• Removing rollback resources too early

Executive Summary

Amazon Linux 2 reaches end-of-support on June 30, 2026. Migration to Amazon Linux 2023 should follow immutable infrastructure principles. For stateful EC2, use parallel instances with validated data sync and controlled DNS cutover. For Auto Scaling Groups, roll out a new AMI using canary and guarded instance refresh. For EKS, introduce AL2023 nodes, prevent new scheduling on AL2, validate workloads and cluster add-ons, then drain and remove AL2 nodes after stability is confirmed. Maintain rollback until the final step.

Final Takeaway

This migration is not about replacing servers.
It is about maintaining production stability while upgrading the platform.

Plan early.
Validate carefully.
Preserve rollback.
Decommission only after confidence.

As part of readiness planning for high-demand production scenarios, we executed a large-scale rollout simulation on one of our production clusters running on
Amazon Elastic Kubernetes Service.

The cluster hosts thousands of pods, supports active CI/CD workflows, and runs with autoscaling enabled.

To validate system behavior under operational stress, we triggered an update of approximately 2000 pods in parallel.

The objective was simple: identify performance boundaries before peak traffic windows.

2. What Happened

During the rollout:

• kubectl get pods responses became noticeably slow

• Deployment progression slowed

• CI pipelines interacting with the cluster experienced delays

• API response times increased

There were:

• No worker node failures

• No pod crashes

• No autoscaling instability

Once the rollout completed, API performance returned to normal.

3. Observations

CloudWatch Container Insights showed:

• A sharp spike in API server request volume

• Increased API server latency

• Minor request drops during peak rollout

• Automatic normalization after rollout completion

The behavior was consistent during heavy parallel updates.

This indicated temporary control plane saturation under burst traffic.

4. Root Cause Analysis

Updating ~2000 pods simultaneously generated significant API traffic, including:

• Pod create and update requests

• Deployment controller reconciliation

• Watch stream updates

• kubelet status reporting

• Autoscaler interactions

• CI/CD polling

All of these operations flow through the Kubernetes API server.

By default,
Amazon Elastic Kubernetes Service
uses reactive auto-scaling for its control plane.

Reactive scaling introduces a short window where burst request volume can temporarily exceed allocated API capacity before scaling adjusts.

During that window:

• API latency increases

• kubectl commands respond slowly

• Rollout completion time extends

The system stabilizes once traffic decreases and scaling catches up.

This was a burst capacity boundary — not a failure.

5. Risk Consideration

Under normal operating conditions, temporary latency during heavy rollout may be acceptable.

However, during high-demand production windows:

• Deployment speed directly impacts mitigation time

• Autoscaling responsiveness is critical

• API stability affects operational recovery

Control plane latency becomes an operational risk during peak events.

6. Solution Evaluated

To eliminate the burst latency window, we evaluated Provisioned Control Plane in
Amazon Elastic Kubernetes Service.

Provisioned Control Plane allows selecting predefined control plane capacity tiers instead of relying entirely on reactive scaling.

This provides:

• Reserved API throughput

• Predictable control plane performance

• Reduced throttling during heavy rollouts

• Improved stability under burst conditions

Higher tiers provide greater sustained API capacity, with increased operational cost.

7. Action Taken

We decided to validate the higher control plane tier in non-production first.

Steps performed:

1. Upgraded the control plane tier.

2. Re-ran heavy rollout simulations.

3. Compared API latency and request drop metrics.

4. Evaluated stability improvement versus cost impact.

Command used:

aws eks update-cluster-config \
  --name apps-eks \
  --control-plane-scaling-config tier=tier-xl

Verification:

aws eks describe-cluster --name apps-eks

Production rollout will be based on measured improvement and cost justification.

8. Key Learnings

• Large clusters expose control plane limits during parallel rollouts.

• Reactive scaling introduces short latency windows under burst traffic.

• Deployment scale directly influences API server performance.

• Control plane capacity planning must be part of production architecture decisions.

• Provisioned Control Plane is suitable for environments with frequent heavy updates or high operational demand.

Final Outcome

The incident did not cause downtime or workload failure.

It identified a control plane burst capacity boundary during large-scale rollout testing.

By addressing it during readiness validation, we reduced operational risk before peak demand scenarios.

During the initial launch phase, we deployed the workload on large instances to remove any performance uncertainty.

After a few weeks of monitoring, the data was clear:

• CPU utilization consistently below 35%

• Memory below 40%

• No disk pressure

• Stable traffic and latency

We were clearly over-provisioned.

The goal was straightforward:

Move the StatefulSet from a large node group to a smaller node group to reduce infrastructure cost — without downtime.

This was a live production system.

Cluster Setup

We had two managed node groups:

• aws-devops-agent-eks-test-ng1 (large instances)

• migration-ng (smaller instances)

Workload characteristics:

• StatefulSet with 3 replicas

• Each pod had its own PVC (created via volumeClaimTemplates)

• StorageClass backed by EBS

• PodDisruptionBudget configured:

maxUnavailable: 1

Before touching the production workload, we tested the entire flow using a demo Nginx StatefulSet with PVCs. This allowed us to observe storage detach/attach behavior safely.

What Happens When You Change nodeSelector in a StatefulSet

Changing the nodeSelector modifies the pod template inside the StatefulSet:

spec:
  template:
    spec:
      nodeSelector:

Any change under spec.template updates the pod template hash.

That automatically triggers a rolling update.

No manual restart command is required.

For each pod, Kubernetes performs the following lifecycle:

1. Terminate pod on the old node

2. Detach the EBS volume

3. Schedule pod on a new node

4. Attach the same volume

5. Mount filesystem

6. Start container

7. Wait for readiness probe

Because this is a StatefulSet:

• Pod identity remains stable

• PVC remains the same

• The EBS volume remains in its original Availability Zone

The primary risk during migration is not parallel restarts — StatefulSet prevents that by default.
The real concern is restart pacing and storage stability between transitions.

Why We Explicitly Kept OrderedReady

We defined:

podManagementPolicy: OrderedReady

StatefulSet supports two policies:

• OrderedReady

• Parallel

With OrderedReady:

• Pods are terminated in reverse ordinal order (pod-2 → pod-1 → pod-0)

• The controller waits for a pod to become Ready before proceeding to the next one

This guarantees serialized lifecycle transitions.

Only one pod is ever moving at a time.

For storage-backed workloads, predictability is more important than speed.

Why We Increased minReadySeconds

Originally:

minReadySeconds: 30

During migration, we increased it:

minReadySeconds: 60

This does not delay traffic.

It delays rollout progression.

The behavior becomes:

• Pod becomes Ready

• Controller waits 60 seconds

• Then proceeds to the next pod

That buffer provides:

• Storage stabilization time

• Application warm-up window

• A monitoring observation period before the next restart

OrderedReady ensures serialization.
minReadySeconds ensures pacing.

Together, they create controlled transitions.

What PodDisruptionBudget Actually Protects

The PodDisruptionBudget does not control rolling update speed.

It protects against voluntary disruptions such as:

• Node drain

• Evictions

• Autoscaler actions

With:

maxUnavailable: 1

We ensured that even outside the rollout logic, no more than one pod could be voluntarily disrupted at a time.

This preserved availability guarantees during infrastructure operations.

Availability Zone Validation

EBS volumes are Availability Zone bound.

Because PVCs were already provisioned, each PersistentVolume existed in a specific AZ.

Before migration, we verified:

• The smaller node group spans the same AZs as the large node group

• There is node capacity in those AZs

If a pod’s volume resides in ap-south-1a, the new node must also be in ap-south-1a.
Otherwise, the pod remains Pending due to volume node affinity constraints.

This check is mandatory for StatefulSet migrations using EBS.

What We Changed

Only two fields were modified.

Before:

minReadySeconds: 30

nodeSelector:
  eks.amazonaws.com/nodegroup: aws-devops-agent-eks-test-ng1

After:

minReadySeconds: 60

nodeSelector:
  eks.amazonaws.com/nodegroup: migration-ng

Everything else remained unchanged.

Small change. Controlled blast radius.

Execution

Steps:

1. Created the smaller node group (migration-ng)

2. Verified AZ alignment and resource headroom

3. Updated the StatefulSet manifest

4. Applied the updated configuration:

kubectl apply -f statefulset.yaml

Because the pod template changed, the StatefulSet controller automatically initiated a rolling update.

We first executed this full flow using the demo Nginx StatefulSet to validate behavior before applying it to the production workload.

Observed Behavior

For each replica:

• Pod terminated on the large node

• EBS detached

• Pod scheduled onto a smaller node in the same AZ

• Volume attached

• Pod became Ready

• Controller waited 60 seconds

• Next pod restarted

There were no overlapping transitions.

No downtime.
No error spike.
Stable latency throughout.

Final Result

Before:

3 large instances

After:

3 smaller instances

Outcome:

• Reduced compute cost

• Preserved availability

• Maintained performance

• No operational instability

What This Actually Was

This was not just resizing infrastructure.

It was a controlled lifecycle transition of a stateful workload.

StatefulSet migrations are safe when you:

• Respect controller behavior

• Serialize restarts

• Add rollout pacing

• Validate storage topology

• Test with a safe workload first

That’s exactly what we did.

And the migration completed without downtime.

1. Overview

In this post, we walk through a real production migration of a Kubernetes workload from NGINX Ingress Controller to Kubernetes Gateway API, implemented using Envoy Gateway, on Amazon EKS.

The key objective was to:

• Migrate safely with zero downtime

• Avoid introducing unnecessary cloud-specific complexity

• Align the platform with Kubernetes’ future networking direction

This guide is written from a platform ownership perspective, not a lab or demo setup.

2. Problem Statement

The application was already running in production and exposed using NGINX Ingress Controller.

While the setup was stable, the following risks were identified:

• The NGINX Ingress Controller project has moved toward reduced long-term maintenance focus, increasing uncertainty around future support guarantees.

• No long-term guarantees for:

  • Security patches

  • CVE fixes

  • Compatibility with future Kubernetes versions

• Ingress sits at the cluster edge, making it a high-blast-radius component

Although there was no immediate outage, continuing with an edge component under reduced maintenance posed long-term operational and security risks.

3. Existing Production Architecture (Before Migration)

User
  ↓
AWS LoadBalancer (auto-created by Service)
  ↓
NGINX Ingress Controller
  ↓
Application Service (ClusterIP)
  ↓
Application Pods

Characteristics of the existing setup

• Stable and functional

• Easy to operate

• Tightly coupled to controller-specific annotations

• Limited separation between platform and application ownership

4. Why Gateway API?

Kubernetes Gateway API is positioned as the successor to Ingress, designed to solve long-standing limitations.

Key improvements over Ingress

Gateway API introduces:

• GatewayClass – defines platform capability

• Gateway – infrastructure-level entry point

• HTTPRoute – application-level routing rules

This model is more scalable, auditable, and production-safe.

5. Why Envoy Gateway in This Case?

The cluster did not have AWS Load Balancer Controller installed.

Installing it mid-migration would have required:

• IAM and IRSA setup

• Additional operational complexity

• Increased blast radius during a live migration

Instead, we chose Envoy Gateway, because it:

• Is a first-class Gateway API implementation

• Does not depend on AWS-specific controllers

• Creates and manages its own dataplane

• Is vendor-neutral and portable

• Allows parallel validation with minimal risk

This decision was intentional, not a workaround.

I intentionally avoided introducing AWS Load Balancer Controller during migration to prevent IAM, IRSA, and cloud-controller changes from increasing the migration blast radius. The goal was to change one edge component at a time.

6. Migration Strategy (Zero Downtime)

A direct replacement was not acceptable.

Chosen strategy

NGINX Ingress LoadBalancer  → continues serving production traffic
Envoy Gateway LoadBalancer → used for validation

Traffic was cut over only after successful validation was completed.

The existing Ingress resource was left untouched to prevent configuration drift and unintended side effects during migration.

This ensured:

• No user impact

• Easy rollback

• Controlled blast radius

7. Step-by-Step Implementation

Step 1: Application Deployment (Already in Place)

The application was deployed with:

• Kubernetes Deployment

• Service of type ClusterIP

No changes were required at the application level.

Step 2: NGINX Ingress (Existing Production Entry)

NGINX Ingress Controller was already installed and exposed the application via an AWS LoadBalancer.

This remained untouched during the migration.

Step 3: Install Gateway API CRDs

Gateway API resources must exist before any controller can operate.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

Step 4: Install Envoy Gateway

Envoy Gateway was installed using Helm via OCI registry.

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.7.0 \
  -n envoy-gateway-system \
  --create-namespace

The Envoy Gateway version was explicitly pinned to v1.7.0 after verifying compatibility with Gateway API v1.0.0 and the EKS cluster version.
Version pinning ensures deterministic deployments, reproducibility, and safe rollback capability in production environments.

Step 5: Create GatewayClass (Platform Ownership)

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

This explicitly defined Envoy Gateway as the cluster’s Gateway API implementation.

Step 6: Create Gateway (Infrastructure Entry Point)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: app-gateway
  namespace: default
spec:
  gatewayClassName: envoy
  listeners:
  - protocol: HTTP
    port: 80

This created a new AWS LoadBalancer, separate from the existing NGINX Ingress LB.

Step 7: Create HTTPRoute (Application Routing)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app
  namespace: default
spec:
  parentRefs:
  - name: app-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: app
      port: 8088

This replaced the Ingress routing logic using Gateway API primitives.

8. Validation

At this stage:

NGINX LB → Production users
Gateway LB → Validation traffic

Validation was performed at multiple levels:

Application Layer

• Verified HTTP 200 responses using curl

• Tested authentication flows

• Executed critical user workflows

• Confirmed session persistence behavior

Infrastructure Layer

• Checked LoadBalancer health check status

• Verified readiness and liveness probes

• Monitored pod logs for errors or unexpected restarts

• Confirmed correct backend service port mapping

• Reviewed Envoy Gateway metrics and controller logs to ensure no reconciliation errors or route attachment failures were present.

Traffic & Stability

• Compared response latency between both entry points

• Monitored 4xx and 5xx error rates

• Verified no increase in backend CPU or memory usage

Only after all validation checkpoints passed was production cutover approved.

9. Cost Considerations During Migration

Running NGINX Ingress and Envoy Gateway in parallel resulted in two active AWS LoadBalancers during the validation window, temporarily increasing infrastructure cost.

However:

• The overlap period was intentionally short.

• The additional cost was justified to eliminate downtime risk.

• The parallel approach reduced blast radius during migration.

Cost was intentionally traded for reliability and controlled risk.

10. Cutover and Cleanup

After all validation checks passed:

kubectl delete ingress app-ingress

Traffic shift was verified immediately after deletion by validating active connections on the Gateway LoadBalancer and confirming healthy backend responses.

The legacy NGINX Ingress was removed only after confirming stable traffic flow through the Gateway LoadBalancer.

Rollback plan:

• Re-apply the Ingress resource if needed

• Restore DNS if traffic switch involved domain update

The migration was reversible during the validation window.

Optionally, after a stability window:

helm uninstall ingress-nginx -n ingress-nginx

The Gateway API entry point became the sole production path.

11. Final Architecture (After Migration)

User
  ↓
AWS LoadBalancer
  ↓
Envoy Gateway (Gateway API)
  ↓
Application Service
  ↓
Application Pods

12. Key Learnings

1. Gateway without HTTPRoute does nothing — infrastructure and routing are intentionally separated

2. Gateway API enforces clearer ownership boundaries than Ingress

3. Parallel migration is the safest approach for production workloads

4. Envoy Gateway is an effective bridge when cloud-native controllers are not yet in place

13. When Would AWS Load Balancer Controller Be Used?

In a later phase, once the platform is stable on the Gateway API.

Typical evolution:

NGINX Ingress
→ Envoy Gateway (Gateway API adoption)
→ AWS Load Balancer Controller (cloud-native optimization)

14. Failure Scenarios Considered

The following risks were evaluated before migration:

• Gateway created without HTTPRoute (no traffic routing)

• Incorrect backend service port reference

• Namespace mismatch between Gateway and HTTPRoute

• LoadBalancer health check failures

• Controller crash or misconfiguration

• Gateway API CRD and controller version mismatch

• DNS TTL delays during traffic switch

By running both entry points in parallel, these risks were isolated and mitigated.

15. Final Takeaway

I designed and executed a zero-downtime migration from NGINX Ingress to Gateway API by running both entry points in parallel.

I validated routing behavior, health checks, infrastructure readiness, and traffic stability before shifting production traffic.

This approach reduced blast radius, preserved service availability, and aligned the platform with Kubernetes’ evolving networking model.

We were running a Node.js backend using PM2 on a Linux server.

Application details:

• Process manager: PM2

• Mode: fork

• User: root

• Deployment: Manual setup on VM

• No containerization

• No autoscaling

The service was running fine in steady state.

Incident Summary

• Trigger: Server reboot during OS patching

• Impact: Application unavailable for 18 minutes

• Root Cause: PM2 not registered with systemd

• Resolution: Integrated PM2 with systemd and enabled process resurrection

Incident Timeline

The server was restarted as part of routine OS patching.

After the reboot:

• The server came up successfully

• SSH access was normal

• But the backend application was down

• API health checks failed

• External traffic started returning errors

Running:

pm2 ls

It returned no running processes because the PM2 daemon was not started after reboot.

Impact

• Application downtime until manual intervention

• No auto-recovery mechanism

• Increased MTTR

• Hidden operational risk exposed

This exposed a design gap.

Detection

The issue was detected via failed API health checks after reboot.

There was no alert configured to monitor the PM2 daemon state.

Downtime lasted approximately 18 minutes until manual intervention restored the service.

Root Cause Analysis

PM2 had previously been started manually in an interactive shell session.

It was never registered with systemd.

As a result, it did not start automatically after reboot.

There was:

• No systemd integration

• No startup registration

• No process resurrection configuration

On reboot:

System Boot
  ↓
No PM2 daemon started
  ↓
No Node process started
  ↓
Application Down

This was not a runtime failure.

This was a lifecycle management design failure.

Design Correction

I redesigned the startup flow to align with production expectations.

Goal

Ensure that:

• PM2 daemon starts automatically on boot

• Saved processes are restored

• No manual intervention required

Implementation

Step 1: Register PM2 with systemd

pm2 startup

This generated a systemd unit file:

/etc/systemd/system/pm2-root.service

And enabled it:

systemctl enable pm2-root

Step 2: Freeze Running Processes

pm2 save

This created:

/root/.pm2/dump.pm2

Without this file, resurrection would not occur.

Step 3: Ensure systemd Controls PM2

Initially:

systemctl status pm2-root

Showed:

inactive (dead)

Meaning PM2 was still running from shell, not systemd.

Corrected by:

pm2 kill
systemctl start pm2-root

Now:

Active: active (running)

Final Boot Flow (Production Aligned)

System Boot
   ↓
systemd
   ↓
pm2-root.service
   ↓
pm2 resurrect
   ↓
Node Application Starts

Validation

Server reboot was performed.

Post-reboot validation:

pm2 ls
systemctl status pm2-root

Result:

• Application automatically started

• No manual intervention required

• The application started automatically without manual intervention, reducing MTTR for reboot-related events to near zero.

Rollback Strategy

If the systemd integration failed:

• Disable pm2-root service

• Manually start PM2 using pm2 start

• Validate application health endpoint

• Restore previous working state

This ensured there was a recovery path during configuration changes.

Preventive Measures

• Standardized server bootstrap process to register PM2 with systemd

• Added reboot validation checklist after OS patching

• Integrated service state checks into monitoring alerts

• Planned migration to a dedicated service user

• Documented lifecycle management requirements

Risks Identified

1. Running as Root

PM2 was configured under root.

Risk:

• Larger blast radius in case of compromise

• Principle of least privilege was violated.

Future improvement:

• Dedicated service user

2. Using NVM for Node

Systemd environment path contained:

/root/.nvm/versions/node/...

Risk:

• Node version changes may break startup

• NVM is not ideal for production servers

Better design:

• Install Node globally

• Lock version

Production Takeaway

In production systems, every long-running process must be supervised by the system init layer.

Running does not imply lifecycle management.

If the init system does not supervise your process, you do not have a resilient system.

The failure was not due to Node. Not due to PM2. Not due to application code.

It was a lifecycle management design gap.

Deploying Apache Superset on Kubernetes using the official Helm chart appears straightforward when following the documentation. In real-world environments, however, production deployments often expose issues across multiple layers — Helm dependency resolution, container image integrity, Python runtime behavior, database connectivity, and secret management.

This article walks through a real-world failure analysis, explains the root causes, and documents the production-ready deployment that supports:

• In-cluster PostgreSQL & Redis

• External PostgreSQL (e.g., AWS RDS) & External Redis

• Optional Kubernetes Secret–based credential injection

The final architecture is flexible, secure, and restart-safe.

1. Problem Statement

We attempted to deploy Apache Superset on Kubernetes using the official Helm chart.

Target Setup

• Apache Superset (Web + Celery Worker)

• PostgreSQL (metadata database)

• Redis (Celery broker and caching)

• Kubernetes

• Helm-based deployment

• Custom Superset image

• Optional external PostgreSQL (AWS RDS)

• Optional external Redis

Expected Outcome

• Superset UI accessible

• Database migrations completed successfully

• Celery workers start without errors

• Stable across restarts

• Secure credential handling

What Actually Happened

The deployment failed at multiple stages:

• Dependency image pull failures

• Python module errors inside the container

• Runtime package installation failures

• SECRET_KEY validation error

• Database connectivity issues

This was a multi-layer failure — not a single misconfiguration.

2. Issue #1: PostgreSQL and Redis Images Not Found

Observed Error

ImagePullBackOff
Failed to pull image
not found

Both PostgreSQL and Redis pods failed to start.

Root Cause

The Helm chart referenced specific image tags that were no longer available in the container registry.

Helm does not validate tag existence.
Kubernetes only detects the failure during image pull.

Until dependencies are healthy:

• Superset init job cannot complete

• Application errors remain hidden

• Debugging becomes misleading

Infrastructure must be stable before diagnosing application issues.

3. Fix #1: Diagnostic Use of latest

To confirm whether the issue was caused by deprecated image tags or application logic, dependency images were temporarily switched to latest.

postgresql:
  image:
    tag: latest

redis:
  image:
    tag: latest

This confirmed:

• The Helm chart’s default tags were deprecated.

• The infrastructure was blocking deployment.

• Superset itself was not the initial issue.

⚠ The latest tag was used only for diagnostics.
In production environments, pinned image versions are recommended for deterministic deployments.

Once dependencies were running, the real application error surfaced.

4. Issue #2: psycopg2 Module Missing

Superset failed with:

ModuleNotFoundError: No module named 'psycopg2'

This affected:

• Superset Web pod

• Superset Worker pod

• Superset Init DB job

5. Why This Breaks Superset

Superset requires a metadata database.

Dependency chain:

Superset → SQLAlchemy → psycopg2 → PostgreSQL

If psycopg2 is missing:

• Superset cannot start

• Database migrations fail

• Celery workers fail

• No fallback mode exists

6. Why Runtime Installation Failed

Attempts included:

• extraPipPackages

• bootstrapScript

• Installing packages inside running pods

• Init container installation

All failed.

Root Cause

The official Superset image runs inside a prebuilt Python virtual environment:

/app/.venv/

Key details:

• Superset executes strictly inside this environment.

• Runtime installations either failed.

• Or installed packages outside the active environment.

• Container immutability was violated.

Even when psycopg2 appeared installed, it was outside Superset’s active virtual environment — making it effectively unusable.

7. Correct Fix: Build a Custom Immutable Superset Image

Database drivers must be installed at image build time.

Dockerfile Used

FROM apachesuperset.docker.scarf.sh/apache/superset:3.0.0

USER root

RUN apt-get update && apt-get install -y libpq-dev gcc \
 && /app/.venv/bin/python -m ensurepip --upgrade \
 && /app/.venv/bin/python -m pip install --no-cache-dir psycopg2==2.9.9

USER superset

Why This Works

• Installs psycopg2 inside Superset’s active virtual environment

• Immutable and reproducible

• Restart-safe

• Production aligned

8. Flexible Credential Management

Superset supports multiple ways to provide database and Redis credentials.

Option A – Directly in Helm Values (Testing Only)

supersetNode:
  connections:
    db_type: postgresql
    db_host: my-db-endpoint
    db_port: "5432"
    db_user: superset
    db_pass: superset123
    db_name: superset

Suitable for:

• Local testing

• Temporary debugging

• Learning environments

⚠ Credentials stored in plaintext.

Option B – Kubernetes Secret Injection (Recommended)

Instead of storing credentials in Helm values, they can be injected securely.

Create Secret

kubectl create secret generic superset-backend-secret \
  --from-literal=DB_HOST=<db-endpoint> \
  --from-literal=DB_PORT=5432 \
  --from-literal=DB_USER=<db-user> \
  --from-literal=DB_PASSWORD=<db-password> \
  --from-literal=DB_NAME=<db-name> \
  --from-literal=REDIS_HOST=<redis-endpoint> \
  --from-literal=REDIS_PORT=6379

Reference Secret in Helm Values

envFromSecrets:
  - superset-backend-secret

Superset connections then use environment variables:

supersetNode:
  connections:
    db_type: postgresql
    db_host: "$(DB_HOST)"
    db_port: "$(DB_PORT)"
    db_user: "$(DB_USER)"
    db_pass: "$(DB_PASSWORD)"
    db_name: "$(DB_NAME)"
    redis_host: "$(REDIS_HOST)"
    redis_port: "$(REDIS_PORT)"

Benefits:

• No plaintext credentials in Git

• Secure runtime injection

• Easier rotation

• Environment portability

Using Kubernetes Secrets is optional but strongly recommended for production.

9. Database & Redis Architecture Options

Superset supports two architectural modes.

Option 1 – In-Cluster PostgreSQL & Redis

Enable Helm-managed dependencies:

postgresql:
  enabled: true

redis:
  enabled: true

Best for:

• Development

• Testing

• Small internal tools

Pros:

• Simple

• Self-contained

Cons:

• You manage backups

• You manage scaling

• Higher operational overhead

Option 2 – External PostgreSQL & Redis (Optional)

Disable internal services:

postgresql:
  enabled: false

redis:
  enabled: false

Best for:

• Production

• High availability needs

• Managed backups

• Reduced operational risk

Pros:

• Managed durability

• Better reliability

• Clear stateless/stateful separation

External services are optional — the deployment remains flexible.

The final production architecture is designed to support both Helm-managed in-cluster stateful services and externally managed database/cache services (such as AWS RDS and ElastiCache), ensuring operational flexibility and scalability across environments.

10. Enforcing SSL for Database Connections

import os

SQLALCHEMY_DATABASE_URI = (
    f"postgresql+psycopg2://{os.environ['DB_USER']}:{os.environ['DB_PASSWORD']}"
    f"@{os.environ['DB_HOST']}:{os.environ['DB_PORT']}/{os.environ['DB_NAME']}"
    "?sslmode=require"
)

Ensures encrypted communication with PostgreSQL.

11. Startup Readiness Handling

Init containers wait for DB and Redis:

command:
  - dockerize
  - -wait
  - tcp://$(DB_HOST):$(DB_PORT)
  - -wait
  - tcp://$(REDIS_HOST):$(REDIS_PORT)
  - -timeout
  - 120s

Prevents:

• CrashLoopBackOff

• Early DB connection failures

• Celery startup issues

12. Secure SUPERSET_SECRET_KEY

extraSecretEnv:
  SUPERSET_SECRET_KEY: <strong-random-secret>

Superset refuses to start without a secure secret key.

13. Final Deployment

helm upgrade --install superset apache/superset \
  -f values.yaml \
  --namespace superset \
  --create-namespace

14. Final Root Cause Summary

Deployment failed due to:

• Deprecated dependency image tags

• Missing psycopg2 driver in container

• Runtime package installation is incompatible with Superset’s virtual environment

• Missing secure SECRET_KEY

Resolution involved:

• Diagnosing infrastructure image failures

• Building a custom immutable Superset image

• Securely injecting credentials

• Supporting flexible DB/Redis architecture

• Enforcing SSL

• Implementing readiness checks

15. 30-Second Summary

Apache Superset initially failed due to deprecated dependency image tags and a missing PostgreSQL driver inside the container. Runtime installation failed because the official Superset image runs inside a prebuilt Python virtual environment, making post-start package installation ineffective. The issue was resolved by building a custom immutable image with psycopg2 installed at build time, securely managing credentials, and supporting both in-cluster and external database/Redis architectures. The final deployment is stable, secure, and production-ready.

Keywords:
Apache Superset Kubernetes,
Superset Helm Chart,
Superset Production Deployment,
psycopg2 error in Superset,
Kubernetes ImagePullBackOff,
Superset with AWS RDS,
Superset External PostgreSQL,
Superset Redis configuration

In this post, I’ll walk you through:

• What caused the issue

• How we identified and resolved it

• Best practices to prevent similar failures in your clusters

🔥 What Went Wrong

During a surge in traffic, our cluster autoscaler kicked in and added new nodes. However, these nodes failed to become Ready, resulting in:

• ❌ Workloads not scheduled

• ❌ Services unreachable

• ❌ Health checks failing, pods crashing

A quick check with:

kubectl get nodes

revealed multiple entries like:

ip-node-ip.eu-west-1.compute.internal   NotReady

To dig deeper, we inspected system logs and found this error:

container runtime network not ready: NetworkReady=false
NetworkPluginNotReady: docker: network plugin is not ready: cni config uninitialized

⚠️ Root Cause: CNI Plugin Failure (Calico)

These errors indicated a CNI (Container Network Interface) misconfiguration. Our cluster was using Calico as the CNI, and it wasn’t initializing properly.

The Calico pods responsible for managing the network stack were either stuck or not starting due to missing configurations.

🛠️ How We Fixed It

1. Delete and Recreate Calico Pods

Force Kubernetes to restart Calico:

kubectl delete pod -n kube-system -l k8s-app=calico-node

This helped in recreating the Calico pods with the current (and correct) configuration.

2. Reapply Calico Configuration

On some nodes, the CNI config was missing or corrupted:

/etc/cni/net.d/10-calico.conflist

We reinstalled Calico using the official manifest:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

3. Restart the Kubelet on Affected Nodes

Restarting the kubelet reinitialized the CNI network stack:

sudo systemctl restart kubelet

✅ Verification

After applying the fixes, we verified node status:

kubectl get nodes

Now showed:

ip-node-ip.eu-west-1.compute.internal   Ready

Services started recovering, and workloads were rescheduled.

🧠 Lessons Learned: How to Prevent This in the Future

🔁 1. Backup CNI Configurations

Always back up CNI configs, especially:

/etc/cni/net.d/10-calico.conflist

This helps with disaster recovery and rapid bootstrapping.

📈 2. Monitor Calico Health and Node Status

Set up monitoring and alerting for:

kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get nodes

Alert if:

• Calico pods crash or restart frequently

• Nodes enter or stay in NotReady

⚠️ 3. Avoid Mixing CNIs

Do not mix different CNI plugins (e.g., Calico and AWS VPC CNI) unless you're explicitly building a hybrid setup. It introduces instability and unexpected behavior.

In our case, we’ve since migrated to the AWS VPC CNI, which aligns better with EKS and provides native integration with VPC IP address management.

📌 Conclusion

Networking is the backbone of Kubernetes, and when the CNI fails, everything breaks.

This incident was a sharp reminder of the importance of:

• Validating CNI configurations

• Monitoring node readiness

• Keeping your control plane and worker nodes in sync

By following the steps outlined above and applying proactive monitoring, you can prevent CNI-related outages and ensure high availability for your workloads.

This comprehensive walkthrough includes prerequisites, cluster setup, networking, and end-to-end validation to help you build a production-ready DR solution.

🔍 What is AWS MSK?

AWS MSK (Managed Streaming for Apache Kafka) is a fully managed service that simplifies running Apache Kafka on AWS. It eliminates the operational overhead of provisioning servers, configuring clusters, and managing availability.

Key features of AWS MSK:

• Fully managed Apache Kafka

• Secure by default with VPC, TLS, and IAM integration

• Scalable with automatic broker scaling and storage expansion

• Native support for monitoring via CloudWatch and logging integrations

🔄 What is MirrorMaker 2?

MirrorMaker 2 (MM2) is the enhanced replication utility introduced in Apache Kafka 2.4+. It’s designed for copying data between Kafka clusters and is built on Kafka Connect, providing modularity, scalability, and fault tolerance.

Key capabilities:

• Real-time replication of topics and consumer offsets

• Support for multiple clusters

• Active-passive and active-active configurations

• Flexible replication policies and error handling

💡 Available Methods for Kafka DR – Why MirrorMaker 2?

Several options exist for disaster recovery in Kafka:

Why we chose MirrorMaker 2 for this guide:

• Seamless integration with AWS MSK and IAM authentication

• No additional licensing or external dependencies

• Good balance of simplicity, performance, and reliability

🧰 Prerequisites

To follow this tutorial, ensure you have:

• An AWS account with permissions for MSK, EC2, IAM, and VPC

• AWS CLI installed and configured

• Java 11+ installed on the EC2 instance

• Kafka client tools (Apache Kafka binaries)

• Two VPCs in different regions (e.g., ap-south-1 and us-east-1)

• AWS MSK IAM Authentication JAR: aws-msk-iam-auth.jar

🏗️ Step 1: Create Primary and DR MSK Clusters

🔹 Create Primary Cluster (ap-south-1)

1. Go to Amazon MSK > Create Cluster

2. Select Custom create

3. Cluster name: msk-primary

4. Kafka version: 3.0+

5. Brokers: kafka.m5.large, 3 brokers, 1000 GiB EBS each

6. Network: Select VPC and 3 subnets

7. Enable:

   • Encryption at rest (KMS)

   • TLS in-transit encryption

   • IAM authentication

8. Assign a security group to allow port 9198 from EC2

   

🔹 Create DR Cluster (us-east-1)

Repeat the above steps in the us-east-1 region, using the cluster name msk-dr. Ensure consistent configuration across clusters.

🔐 Step 2: Configure Network and Security

Update security group rules:

• MSK SG: Allow inbound TCP 9198 from EC2 SG or IP

• EC2 SG: Allow outbound 9198 to both MSK clusters

• Enable SSH (port 22) on EC2 for management access

  

💻 Step 3: Launch EC2 with Kafka Client

🚀 Launch EC2

• Region: ap-south-1

• Type: t3.medium or higher

• Attach IAM role for MSK and Secrets Manager (if needed)

• Ensure internet access (NAT or public IP)

🛠️ Install Tools and Configure

sudo yum update -y
sudo yum install -y java-11-amazon-corretto
wget https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
tar -xzf kafka_2.13-3.3.1.tgz
export KAFKA_HOME=$(pwd)/kafka_2.13-3.3.1

wget https://github.com/aws/aws-msk-iam-auth/releases/latest/download/aws-msk-iam-auth.jar
export IAM_JAR=$(pwd)/aws-msk-iam-auth.jar

✍️ Create client.properties

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

ssl.truststore.location=/home/ec2-user/msk-certs/truststore.jks
ssl.truststore.password=anjali

☝️ Make sure the truststore includes AWS MSK’s CA certificate.

🧵 Step 4: Create Kafka Topic on Primary Cluster

CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-topics.sh \
  --create \
  --topic test-topic \
  --partitions 3 \
  --replication-factor 3 \
  --bootstrap-server <primary-broker-list> \
  --command-config /home/ec2-user/msk-certs/client.properties

Replace <primary-broker-list> with your actual MSK bootstrap brokers.

⚙️ Step 5: Configure and Run MirrorMaker 2

✍️ Create mm2.properties

clusters = primary,dr

primary.bootstrap.servers=<primary-brokers>
primary.security.protocol=SASL_SSL
primary.sasl.mechanism=AWS_MSK_IAM
primary.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
primary.ssl.truststore.location=/home/ec2-user/msk-certs/kafka.client.truststore.jks
primary.ssl.truststore.password=anjali

dr.bootstrap.servers=<dr-brokers>
dr.security.protocol=SASL_SSL
dr.sasl.mechanism=AWS_MSK_IAM
dr.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
dr.ssl.truststore.location=/home/ec2-user/msk-certs/kafka.client.truststore.jks
dr.ssl.truststore.password=anjali

tasks.max=2
topics=test-topic
groups=.*
replication.policy.class=org.apache.kafka.connect.mirror.DefaultReplicationPolicy

▶️ Run MM2

CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/*:$KAFKA_HOME/libs/connect-runtime-*.jar:$KAFKA_HOME/libs/connect-api-*.jar \
  $KAFKA_HOME/bin/connect-mirror-maker.sh /home/ec2-user/mm2.properties

✅ Step 6: Validate Replication

🔎 List Topics on DR

CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-topics.sh \
  --list \
  --bootstrap-server <dr-broker> \
  --command-config /home/ec2-user/msk-certs/client.properties

📥 Consume Messages from DR

CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-console-consumer.sh \
  --topic mm2-test-topic \
  --from-beginning \
  --bootstrap-server <dr-broker> \
  --consumer.config /home/ec2-user/msk-certs/client.properties

✍️ Step 7: Test Message Flow

📨 Produce to Primary

CLASSPATH=$IAM_JAR:$KAFKA_HOME/libs/* $KAFKA_HOME/bin/kafka-console-producer.sh \
  --topic test-topic \
  --bootstrap-server <primary-broker> \
  --producer.config /home/ec2-user/msk-certs/client.properties

Type some messages and hit Enter.

✅ Confirm on DR

Re-run the consumer from the DR cluster to verify real-time replication.

💬 Have you implemented DR for Kafka in your architecture?
Drop your approach or challenges in the comments — I'd love to hear how others tackle cross-region resilience!

🔔 Follow me for more AWS infrastructure and streaming data posts!

In this post, we’ll walk through how to set up a cross-region disaster recovery (DR) strategy for DynamoDB using Global Tables across us-east-1 and ap-south-1, covering setup, replication, and failover validation.

🔧 Use Case

• Architecture: Active-Active (bi-directional replication)

• Primary Region: us-east-1 (US East - N. Virginia)

• Disaster Recovery Region: ap-south-1 (Asia Pacific - Mumbai)

• Objective: Set up Global Tables in DynamoDB to ensure DR-readiness with real-time replication.

  

🛠 Step-by-Step Setup

Step 1: Create a DynamoDB Table in the Primary Region

1. Log in to the AWS Management Console.

2. Switch Region to US East (N. Virginia) – us-east-1.

3. Navigate to Services > DynamoDB.

4. Click on "Create Table".

5. Enter table details:

   • Table name: test-table

   • Partition key: user-id (String)

6. Leave defaults as-is (On-demand billing recommended).

7. Click "Create Table".

8. Wait until the table status is “Active”.

   

Step 2: Add Replica Region (ap-south-1)

1. While still in us-east-1, open the test-table.

2. Go to the "Global Tables" tab.

3. Click "Add region".

4. Select Asia Pacific (Mumbai) – ap-south-1.

5. Click "Create Replica".

6. AWS will provision a replica in ap-south-1 with automatic schema synchronization.

   

Step 3: Verify Replication

1. Switch Region to ap-south-1.

2. Go to DynamoDB > Tables.

3. Confirm that test-table appears.

4. Open it and validate that the schema matches the original.

   

Step 4: Test Bi-Directional Replication

A. Insert Data in Primary (us-east-1)

1. Switch back to us-east-1.

2. Open test-table > Explore table items > Create item.

3. Add:

    {
      "UserID": "U001",
      "Name": "Alice"
    }
   

4. Click "Create item".

   

B. Verify in DR Region (ap-south-1)

1. Switch to ap-south-1.

2. Navigate to test-table > Explore table items.

3. Verify that Alice’s record is replicated.

   

C. Insert in DR Region (ap-south-1)

1. In ap-south-1, add:

    {
      "UserID": "U002",
      "Name": "Bob"
    }
   

2. Switch to us-east-1 and verify Bob's entry is available there too.

   

Step 5: Simulate Region Failure (Optional Testing)

To test failover without bringing down the actual AWS region:

1. Block the us-east-1 endpoint locally:

    sudo nano /etc/hosts
   

2. Add the following line:

    127.0.0.1 dynamodb.us-east-1.amazonaws.com
   

3. Now perform read/write operations only in ap-south-1.

4. Once done, remove or comment out the line from /etc/hosts.

   

✅ Conclusion

With DynamoDB Global Tables, building a resilient, low-latency, and globally available application becomes straightforward. This architecture is ideal for mission-critical systems that demand real-time DR and global data access. By following this guide, you’ve successfully implemented an active-active cross-region DR strategy using only the AWS Console.

📌 Further Reading

• AWS Docs: DynamoDB Global Tables

• Handling Conflicts in DynamoDB Global Tables

• DynamoDB Pricing

💬 Have you implemented cross-region DR for DynamoDB or other AWS services? Share your setup in the comments!

🧭 Architecture Overview

The DR architecture consists of:

• A primary MongoDB instance running in one AWS region.

• A replica set member (secondary) in a separate AWS region.

• Data replication between these two nodes.

📌 Diagram on page 1 shows a simple two-region EC2-based MongoDB setup with replication arrows connecting the nodes.

🔧 Implementation Guide

1. EC2 Instance Setup

• Launch two Ubuntu EC2 instances, one in each AWS region.

• Security Groups must allow:

  • Inbound traffic on TCP port 27017 (MongoDB default port) from known IPs.

2. Install MongoDB

Run these commands on both instances:

sudo apt-get install gnupg curl
curl -fsSL https://www.mongodb.org/static/pgp/server-8.0.asc | \
  sudo gpg -o /usr/share/keyrings/mongodb-server-8.0.gpg --dearmor

echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] \
https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/8.0 multiverse" | \
sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list

sudo apt-get update
sudo apt-get install -y mongodb-org
sudo systemctl start mongod
sudo systemctl enable mongod

3. Configure MongoDB

Edit the MongoDB config file on both servers:

sudo nano /etc/mongod.conf

Set the following values:

net:
  port: 27017
  bindIp: 0.0.0.0

replication:
  replSetName: "rs0"

security:
  authorization: enabled

Then restart the MongoDB service:

sudo systemctl restart mongod

4. Initialize Replica Set

Log in to the Mongo shell on the primary:

mongosh

Run this setup:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "primary-public-ip:27017", priority: 2 },
    { _id: 1, host: "secondary-public-ip:27017", priority: 1 }
  ]
})

Verify it using:

rs.status()

5. Create Admin User

On the primary:

db.createUser({
  user: "adminUser",
  pwd: "securePassword",
  roles: [{ role: "root", db: "admin" }]
})

Reconnect with auth:

mongosh --host primary-public-ip:27017 -u adminUser -p securePassword --authenticationDatabase admin

6. Test Replication

Insert on primary:

use testdb
db.testCollection.insertOne({ message: "Testing replication", timestamp: new Date() })

Read from the secondary:

mongosh --host secondary-public-ip:27017 -u adminUser -p securePassword --authenticationDatabase admin

Enable read preference:

db.getMongo().setReadPref("secondaryPreferred")
use testdb
db.testCollection.find()

7. Simulate Failover

Stop MongoDB on the primary:

sudo systemctl stop mongod

Then, on the secondary, verify promotion:

rs.status()  // This node should now be PRIMARY

Bring the primary back:

sudo systemctl start mongod

📝 8. Checklist / TL;DR

Here’s a quick reference summary to validate your setup:

✅ EC2s launched in two AWS regions
✅ MongoDB installed and configured
✅ Replica set initialized
✅ Authentication set up
✅ Replication verified
✅ Failover tested

✅ Conclusion

This guide demonstrates how to:

• Set up a MongoDB replica set across AWS regions.

• Secure it with authentication.

• Test real-time replication and failover.

While this is a basic manual setup, production deployments may benefit from:

• Private networking (e.g., VPC peering).

• Automation via Terraform or Ansible.

• Monitoring using MongoDB Ops Manager or Prometheus.

Even though containers provide abstraction, they still rely on the host’s kernel, CPU scheduler, memory handling, and I/O layers. With smart tuning:

• Network throughput can rise 📈

• Memory management can become more efficient 🔁

• Latency can be reduced for disk and CPU operations ⚡

📊 Flow of the Optimization Process

This clear 5-step flow ensures reproducibility and insight into performance improvements.

🛠️ Setup and Preliminaries

• OS: Ubuntu 22.04

• Application: Nginx inside a Docker container

• Benchmark Tools: stress-ng, iperf3, dd, htop, iotop, iftop, sysstat

Prerequisites

Ensure Docker is installed and running:

docker --version

Install benchmarking and monitoring tools:

sudo apt update
sudo add-apt-repository universe
sudo apt update
sudo apt install -y htop iotop iftop sysstat

📦 Step 1: Launch the Nginx Container

docker run -d --name nginx-test -p 8089:80 nginx

Verify by navigating to: http://localhost:8089

📉 Step 2: Baseline Performance (Before Tuning)

CPU Test

stress-ng --cpu 4 --timeout 60s --metrics-brief

Memory Test

stress-ng --vm 2 --vm-bytes 1G --timeout 60s --metrics-brief

Disk I/O Test

dd if=/dev/zero of=testfile bs=1G count=1 oflag=dsync
rm -f testfile

Network Test (Loopback)

Terminal 1:

iperf3 -s

Terminal 2:

iperf3 -c 127.0.0.1

⚙️ Step 3: Apply Linux Host Tuning

CPU Tuning

Install the cpupower tools and set the performance governor:

sudo apt install -y linux-tools-common linux-tools-generic
sudo cpupower frequency-set --governor performance

Memory Tuning

sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.dirty_ratio=20
sudo sysctl -w vm.dirty_background_ratio=10

Disk I/O Tuning

Change scheduler (check your block device name, e.g., nvme0n1):

cat /sys/block/nvme0n1/queue/scheduler
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

Network Tuning

sudo sysctl -w net.core.somaxconn=65535
sudo sysctl -w net.core.netdev_max_backlog=250000
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

Kernel Parameters for Containers

sudo tee /etc/sysctl.d/99-container-tuning.conf > /dev/null <<EOF
fs.file-max = 2097152
vm.max_map_count = 262144
net.ipv4.ip_local_port_range = 1024 65535
EOF

sudo sysctl --system

🔁 Step 4: Re-deploy and Re-test

Recreate the container:

docker rm -f nginx-test
docker run -d --name nginx-test -p 8089:80 nginx

Repeat benchmarks as earlier to observe improvements.

Step 5: Re-run the NGINX Container

docker rm -f nginx-test
docker run -d --name nginx-test -p 8089:80 nginx

🔁 Step 6: Re-test the Performance (Post-Tuning)

After applying system-level optimizations, it's time to benchmark again and compare the performance with the baseline.

✅ CPU Re-test

Use the following command to stress the CPU with 4 workers for 60 seconds:

stress-ng --cpu 4 --timeout 60s --metrics-brief

✅ Memory Re-test

Run a memory stress test with 2 workers using 1 GB each for 60 seconds:

stress-ng --vm 2 --vm-bytes 1G --timeout 60s --metrics-brief

✅ Disk I/O Re-test

Evaluate sequential disk write speed using dd:

if=/dev/zero of=testfile bs=1G count=1 oflag=dsync
rm -f testfile

✅ Network Re-test (Loopback Throughput)

Step 1: Start the iperf3 server

iperf3 -s

Step 2: In another terminal (same host), run the iperf3 client

iperf3 -c 127.0.0.1

This will test internal loopback throughput after networking optimizations.

📊 Benchmark Results: Before vs. After

📌 Analysis

• CPU & Memory: Minor decrease in throughput—likely due to more deterministic scheduling from tuning.

• Disk I/O: No change—indicating the scheduler change didn’t affect sequential write speed.

• Network: Solid improvement in throughput thanks to buffer tuning and backlog increase.

✅ Summary

Tuning a Linux host for containerized workloads isn't just about maxing out performance—it's about tailoring the system to match your application’s behavior.

Key Takeaways:

• Network stack tuning had the most measurable benefit (3.6% gain).

• Memory and CPU tuning had slight trade-offs, possibly due to overhead or governor changes.

• Always benchmark before and after to validate changes.

• Apply these techniques to any containerized app, not just Nginx.

🧩 Architecture Overview

Here’s the flow of data across our components:

1. NGINX application on an EC2 instance in a public subnet.

2. A Load Balancer is attached to the EC2 instance.

3. AWS WAF rules are applied to block access from certain sources.

4. Logs sent to Amazon Kinesis Data Firehose.

5. Firehose delivers logs to an S3 Bucket.

6. Logstash reads from S3 and parses logs.

7. Logs are indexed into OpenSearch servers.

8. Visualized through OpenSearch Dashboards.

🏗️ Step 1: Deploy NGINX on EC2

Spin up an EC2 instance in a public subnet and install NGINX:

sudo apt-get update -y
sudo apt-get install nginx -y 
sudo systemctl enable nginx
sudo systemctl start nginx
sudo systemctl status nginx

Ensure the EC2 is behind an Application Load Balancer (ALB).

Step 2: Protect with AWS WAF

Attach AWS WAF to your ALB. Create custom rules to filter traffic (e.g., block based on IP, User-Agent, geo, etc.).

Enable logging in WAF and configure Kinesis Data Firehose as the log destination.

🧾 Step-by-Step Instructions

1. Create an AWS WAF Web ACL

1. Go to the AWS WAF & Shield service in the AWS Console.

2. Click “Create web ACL”.

3. Provide a name and description (e.g., nginx-web-acl).

4. Select Region (e.g., us-east-1) and Resource type as “Regional resources”.

5. Choose Associated AWS resource type: select Application Load Balancer.

6. Choose the ALB that routes traffic to your NGINX EC2 instance.

   

2. Add WAF Rules

You can use managed rules or define custom ones. Examples:

➤ Block Local IP Range (Simulate External Access Block)

To simulate blocking local system traffic, add a custom rule to block a specific IP:

1. Under “Add rules”, click Add my own rules and rule groups.

2. Create a rule:

   • Rule name: BlockLocalIP

   • Type: IP set

   • Create a new IP set with your local public IP.

   • Set action to Block.

3. Add the rule to the Web ACL.

   

   

   3. Configure Logging to Kinesis Data Firehose

   WAF logs can be sent to Amazon Kinesis Data Firehose, which will forward them to S3.

   1. On the left nav, go to Logging and metrics under WAF.

   2. Click “Enable logging”.

   3. Select your Web ACL.

   4. Choose the Kinesis Firehose delivery stream that you created earlier (waf-logs-stream).

   5. Optionally, add filters or redactions.

   6. Save changes.

      

      

      

      4. Verify Logs Are Flowing

      1. Send traffic through your Load Balancer (via browser or curl).

      2. In the S3 bucket, JSON log files arrive via Firehose.

         • Each log event will contain request metadata like clientIp, action, ruleMatched, httpRequest, etc.

      3. These logs will be picked up later by Logstash and placed in your pipeline.

         

         

🐳 Step 3: Deploy OpenSearch and Logstash with Docker

Create a Docker network:

docker network create opensearch-net

🔍 Run OpenSearch:

docker run -d --name opensearch-node1 \
  --network opensearch-net \
  -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Redhat@123" \
  opensearchproject/opensearch:latest

📊 OpenSearch Dashboards:

docker run -d --name opensearch-dashboards \
  --network opensearch-net \
  -p 5601:5601 \
  -e OPENSEARCH_HOSTS='["https://opensearch-node1:9200"]' \
  -e OPENSEARCH_USERNAME='admin' \
  -e OPENSEARCH_PASSWORD='Redhat@123' \
  opensearchproject/opensearch-dashboards:latest

📥 Step 5: Logstash to Process WAF Logs

Use the official image with the OpenSearch plugin:

docker run -d --name logstash-01 \
  --network opensearch-net \
  -v /home/neetesh/cloudkeeper-workspace/waf-promethuses-02/logstash-pipeline:/usr/share/logstash/pipeline \
  -v /tmp/logstash-s3:/tmp/logstash-s3 \
  opensearchproject/logstash-oss-with-opensearch-output-plugin:latest

🔧 Logstash Pipeline Configuration

input {
  s3 {
    bucket => "poc-s3-bucket-00000001"
    prefix => ""                   # Adjust if you want to limit path
    region => "us-east-1"               # Replace with your bucket’s region
    codec => "json"
    sincedb_path => "/tmp/logstash-s3.sincedb"
    temporary_directory => "/tmp/logstash-s3" 
    access_key_id => "ACCESS_KEY_ID"
    secret_access_key => "SECRET_ACCESS_KEY"
  }
}

filter {
  if "_jsonparsefailure" in [tags] {
    drop { }
  }

  if [action] {
    mutate { add_field => { "action" => "%{[action]}" } }
  }

  if [terminatingRuleId] {
    mutate { add_field => { "terminating_rule" => "%{[terminatingRuleId]}" } }
  }

  if [httpRequest][clientIp] {
    mutate { add_field => { "client_ip" => "%{[httpRequest][clientIp]}" } }
  }

  if [httpRequest][country] {
    mutate { add_field => { "country" => "%{[httpRequest][country]}" } }
  }

  if [httpRequest][httpMethod] {
    mutate { add_field => { "http_method" => "%{[httpRequest][httpMethod]}" } }
  }

  if [httpRequest][uri] {
    mutate { add_field => { "uri" => "%{[httpRequest][uri]}" } }
  }

  if [httpRequest][httpVersion] {
    mutate { add_field => { "http_version" => "%{[httpRequest][httpVersion]}" } }
  }

  if [httpRequest][headers] {
    ruby {
      code => '
        begin
          headers = event.get("[httpRequest][headers]")
          headers.each do |h|
            if h["name"].downcase == "user-agent"
              event.set("user_agent", h["value"])
            elsif h["name"].downcase == "host"
              event.set("host_header", h["value"])
            end
          end
        rescue => e
          event.tag("_header_parse_failure")
        end
      '
    }
  }
}

output {
  opensearch {
    hosts => ["https://opensearch-node1:9200"]
    user => "admin"
    password => "Redhat@123"
    ssl => true
    ssl_certificate_verification => false
    index => "aws-waf-logs-test-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

📈 Step 6: Visualize Logs in OpenSearch Dashboards

Navigate to http://localhost:5601 and log in with:

• Username: admin

• Password: Redhat@123

Create an index pattern: aws-waf-logs-test-*

You can now create rich visualizations such as:

📦 Full OpenSearch Dashboard: JSON Import Snippets

You can now import a ready-made dashboard with rich visualizations into OpenSearch Dashboards using the export JSON you've created.

📁 Included Visualizations

Your exported dashboard includes:

{"attributes":{"buildNum":8430,"defaultIndex":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d"},"id":"3.0.0","migrationVersion":{"config":"7.9.0"},"references":[],"type":"config","updated_at":"2025-05-14T11:19:38.632Z","version":"WzIsMV0="}
{"attributes":{"fields":"[{\"count\":0,\"name\":\"@timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"@version\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"@version.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"@version\"}}},{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_score\",\"type\":\"number\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_type\",\"type\":\"string\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"action\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"action.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"action\"}}},{\"count\":0,\"name\":\"captchaResponse.failureReason\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"captchaResponse.failureReason.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"captchaResponse.failureReason\"}}},{\"count\":0,\"name\":\"captchaResponse.responseCode\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"client_ip\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"client_ip.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"client_ip\"}}},{\"count\":0,\"name\":\"country\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"country.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"country\"}}},{\"count\":0,\"name\":\"event.original\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"event.original.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"event.original\"}}},{\"count\":0,\"name\":\"formatVersion\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"host.name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"host.name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"host.name\"}}},{\"count\":0,\"name\":\"host_header\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"host_header.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"host_header\"}}},{\"count\":0,\"name\":\"httpRequest.clientIp\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.clientIp.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.clientIp\"}}},{\"count\":0,\"name\":\"httpRequest.country\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.country.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.country\"}}},{\"count\":0,\"name\":\"httpRequest.headers.name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.headers.name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.headers.name\"}}},{\"count\":0,\"name\":\"httpRequest.headers.value\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.headers.value.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.headers.value\"}}},{\"count\":0,\"name\":\"httpRequest.httpMethod\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.httpMethod.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.httpMethod\"}}},{\"count\":0,\"name\":\"httpRequest.httpVersion\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.httpVersion.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.httpVersion\"}}},{\"count\":0,\"name\":\"httpRequest.uri\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"httpRequest.uri.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"httpRequest.uri\"}}},{\"count\":0,\"name\":\"http_method\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"http_method.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"http_method\"}}},{\"count\":0,\"name\":\"http_version\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"http_version.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"http_version\"}}},{\"count\":0,\"name\":\"log.file.path\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"log.file.path.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"log.file.path\"}}},{\"count\":0,\"name\":\"nonTerminatingMatchingRules.action\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"nonTerminatingMatchingRules.action.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"nonTerminatingMatchingRules.action\"}}},{\"count\":0,\"name\":\"nonTerminatingMatchingRules.ruleId\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"nonTerminatingMatchingRules.ruleId.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"nonTerminatingMatchingRules.ruleId\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.customValues.key\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.customValues.key.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.customValues.key\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.customValues.name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.customValues.name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.customValues.name\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.customValues.value\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.customValues.value.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.customValues.value\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.evaluationWindowSec\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.evaluationWindowSec.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.evaluationWindowSec\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.limitKey\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.limitKey.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.limitKey\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.maxRateAllowed\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"rateBasedRuleList.rateBasedRuleId\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.rateBasedRuleId.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.rateBasedRuleId\"}}},{\"count\":0,\"name\":\"rateBasedRuleList.rateBasedRuleName\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"rateBasedRuleList.rateBasedRuleName.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"rateBasedRuleList.rateBasedRuleName\"}}},{\"count\":0,\"name\":\"terminatingRuleId\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"terminatingRuleId.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"terminatingRuleId\"}}},{\"count\":0,\"name\":\"terminatingRuleType\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"terminatingRuleType.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"terminatingRuleType\"}}},{\"count\":0,\"name\":\"terminating_rule\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"terminating_rule.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"terminating_rule\"}}},{\"count\":0,\"name\":\"timestamp\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"uri\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"uri.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"uri\"}}},{\"count\":0,\"name\":\"user_agent\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"user_agent.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"user_agent\"}}},{\"count\":0,\"name\":\"webaclId\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"webaclId.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"webaclId\"}}}]","timeFieldName":"@timestamp","title":"aws-waf-logs-*"},"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2025-05-14T11:19:34.933Z","version":"WzEsMV0="}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"WAF Actions (ALLOW vs BLOCK vs CAPTCHA)","uiStateJSON":"{\"vis\":{\"colors\":{\"BLOCK\":\"#ef9988\"}}}","version":1,"visState":"{\"title\":\"WAF Actions (ALLOW vs BLOCK vs CAPTCHA)\",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"action.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":true,\"labels\":{\"last_level\":true,\"show\":false,\"truncate\":100,\"values\":true},\"legendPosition\":\"right\",\"row\":true,\"type\":\"pie\"}}"},"id":"99127f00-30b7-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T12:15:02.158Z","version":"WzEzLDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Total HTTP Requests","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Total HTTP Requests\",\"type\":\"metric\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{\"customLabel\":\"Total HTTP Requests\"},\"schema\":\"metric\"}],\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":60}}}}"},"id":"63e0d4c0-30b8-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T11:41:35.628Z","version":"WzYsMV0="}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"key\":\"action.keyword\",\"negate\":false,\"params\":{\"query\":\"BLOCK\"},\"type\":\"phrase\",\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"match_phrase\":{\"action.keyword\":\"BLOCK\"}}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Blocked HTTP Requests","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Blocked HTTP Requests\",\"type\":\"metric\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{\"customLabel\":\"Blocked Requests\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"filters\",\"params\":{\"filters\":[{\"input\":{\"query\":\"\",\"language\":\"kuery\"},\"label\":\"\"}]},\"schema\":\"group\"}],\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":60}}}}"},"id":"7132d620-30bb-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T12:03:30.357Z","version":"WzgsMV0="}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"HTTP Versions Breakdown","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"HTTP Versions Breakdown\",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"http_version.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":false,\"values\":true,\"last_level\":true,\"truncate\":100}}}"},"id":"7119bbf0-30d3-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T14:55:14.223Z","version":"WzE4LDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"HTTP Methods","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"HTTP Methods\",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"http_method.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":false,\"values\":true,\"last_level\":true,\"truncate\":100},\"row\":true}}"},"id":"1f420b20-30d3-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T14:53:01.512Z","version":"WzE1LDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Top  Hosts","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Top  Hosts\",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"host_header.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":false,\"values\":true,\"last_level\":true,\"truncate\":100}}}"},"id":"49df5b20-30d4-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T15:01:22.321Z","version":"WzI3LDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Top Countries ","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Top Countries \",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"country.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":false,\"values\":true,\"last_level\":true,\"truncate\":100}}}"},"id":"0824e100-30d4-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T15:05:39.905Z","version":"WzMwLDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Top IP Addresses","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Top IP Addresses\",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"client_ip.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":false,\"values\":true,\"last_level\":true,\"truncate\":100}}}"},"id":"e4418fe0-30d3-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T15:05:50.998Z","version":"WzMxLDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Top User Agents","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Top User Agents\",\"type\":\"pie\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"user_agent.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":false,\"values\":true,\"last_level\":true,\"truncate\":100}}}"},"id":"26b45d80-30d4-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T15:00:35.246Z","version":"WzI1LDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Top Web ACLs","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Top Web ACLs\",\"type\":\"table\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"webaclId.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"bucket\"}],\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\"}}"},"id":"c6952af0-30d4-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T15:05:19.523Z","version":"WzI5LDFd"}
{"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"674ecf5a-ed50-411f-8178-d0c28c2f0acd\",\"w\":24,\"x\":0,\"y\":0},\"panelIndex\":\"674ecf5a-ed50-411f-8178-d0c28c2f0acd\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_0\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"9a2fb88c-a482-4adc-9ab9-1693caca9e07\",\"w\":24,\"x\":24,\"y\":0},\"panelIndex\":\"9a2fb88c-a482-4adc-9ab9-1693caca9e07\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_1\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"02404d24-4e9f-4120-bc80-5931a1e8fe7c\",\"w\":24,\"x\":24,\"y\":15},\"panelIndex\":\"02404d24-4e9f-4120-bc80-5931a1e8fe7c\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_2\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"681f6ea4-757a-4fbd-b74d-20698edf01dd\",\"w\":24,\"x\":0,\"y\":15},\"panelIndex\":\"681f6ea4-757a-4fbd-b74d-20698edf01dd\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_3\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"1c3d4763-f99f-4945-a32f-c6553518f059\",\"w\":24,\"x\":24,\"y\":30},\"panelIndex\":\"1c3d4763-f99f-4945-a32f-c6553518f059\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_4\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"9ce23390-217d-4d1d-a9df-0d9a2d858966\",\"w\":24,\"x\":0,\"y\":30},\"panelIndex\":\"9ce23390-217d-4d1d-a9df-0d9a2d858966\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_5\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"d9b7d60d-e78b-473c-9493-4ed9cdeb824f\",\"w\":24,\"x\":24,\"y\":45},\"panelIndex\":\"d9b7d60d-e78b-473c-9493-4ed9cdeb824f\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_6\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"a7e493f6-23df-4c6d-b95a-d45fe9735d57\",\"w\":24,\"x\":0,\"y\":45},\"panelIndex\":\"a7e493f6-23df-4c6d-b95a-d45fe9735d57\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_7\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"60758b57-6454-4bd0-a723-091816c4ed24\",\"w\":24,\"x\":24,\"y\":60},\"panelIndex\":\"60758b57-6454-4bd0-a723-091816c4ed24\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_8\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"26cb5b48-4840-4a02-92f5-f783e6053c98\",\"w\":24,\"x\":0,\"y\":60},\"panelIndex\":\"26cb5b48-4840-4a02-92f5-f783e6053c98\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_9\"},{\"embeddableConfig\":{},\"gridData\":{\"h\":15,\"i\":\"61169e8e-a911-47dd-8f4f-abab036fa0a7\",\"w\":24,\"x\":24,\"y\":75},\"panelIndex\":\"61169e8e-a911-47dd-8f4f-abab036fa0a7\",\"version\":\"3.0.0\",\"panelRefName\":\"panel_10\"}]","timeRestore":false,"title":"WAF-Monitorings","version":1},"id":"7d2643d0-30bc-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"dashboard":"7.9.3"},"references":[{"id":"99127f00-30b7-11f0-9eb5-8f6a0d106a1d","name":"panel_0","type":"visualization"},{"id":"63e0d4c0-30b8-11f0-9eb5-8f6a0d106a1d","name":"panel_1","type":"visualization"},{"id":"7132d620-30bb-11f0-9eb5-8f6a0d106a1d","name":"panel_2","type":"visualization"},{"id":"7119bbf0-30d3-11f0-9eb5-8f6a0d106a1d","name":"panel_3","type":"visualization"},{"id":"1f420b20-30d3-11f0-9eb5-8f6a0d106a1d","name":"panel_4","type":"visualization"},{"id":"49df5b20-30d4-11f0-9eb5-8f6a0d106a1d","name":"panel_5","type":"visualization"},{"id":"0824e100-30d4-11f0-9eb5-8f6a0d106a1d","name":"panel_6","type":"visualization"},{"id":"e4418fe0-30d3-11f0-9eb5-8f6a0d106a1d","name":"panel_7","type":"visualization"},{"id":"26b45d80-30d4-11f0-9eb5-8f6a0d106a1d","name":"panel_8","type":"visualization"},{"id":"c6952af0-30d4-11f0-9eb5-8f6a0d106a1d","name":"panel_9","type":"visualization"},{"id":"63e0d4c0-30b8-11f0-9eb5-8f6a0d106a1d","name":"panel_10","type":"visualization"}],"type":"dashboard","updated_at":"2025-05-15T06:06:12.962Z","version":"WzM2LDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Unique IP Address Count","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Unique IP Address Count\",\"type\":\"metric\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"client_ip.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":5,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"},\"schema\":\"group\"}],\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":60}}}}"},"id":"8e9c5c50-30d3-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T14:56:03.733Z","version":"WzE5LDFd"}
{"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"Number of Requests per Country","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"Number of Requests per Country\",\"type\":\"histogram\",\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"params\":{\"customLabel\":\"Total Requests\"},\"schema\":\"metric\"},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"params\":{\"field\":\"country.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":10,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"Country\"},\"schema\":\"segment\"}],\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"filter\":true,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Total Requests\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"histogram\",\"mode\":\"stacked\",\"data\":{\"label\":\"Total Requests\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"lineWidth\":2,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"labels\":{\"show\":false},\"thresholdLine\":{\"show\":false,\"value\":10,\"width\":1,\"style\":\"full\",\"color\":\"#E7664C\"}}}"},"id":"397b01f0-30d7-11f0-9eb5-8f6a0d106a1d","migrationVersion":{"visualization":"7.10.0"},"references":[{"id":"50aec450-30b5-11f0-9eb5-8f6a0d106a1d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2025-05-14T15:22:22.388Z","version":"WzM0LDFd"}
{"exportedCount":15,"missingRefCount":0,"missingReferences":[]}

🛠️ How to Import the Dashboard

1. Navigate to OpenSearch Dashboards → Dashboards Management → Saved Objects.

   

2. Click Import and upload your JSON file.

   

3. Confirm and overwrite if prompted.

4. Navigate to Dashboards → WAF-Monitorings.

This dashboard will instantly visualize live or historical WAF logs streamed from AWS into OpenSearch via your pipeline.

✅ Conclusion

This blog walked you through building a log analysis pipeline using AWS WAF, Firehose, S3, Logstash, and OpenSearch. With this setup, you gain full visibility into suspicious traffic, helping improve your application’s security posture in real-time.

Solution Overview

🧱 Components Used:

• AWS Lambda (Python): A serverless function that stops RDS instances.

• Amazon EventBridge Scheduler: Triggers the Lambda function once per day.

• IAM Role: Grants permissions for the Lambda to stop RDS instances.

  

⚙️ How It Works:

1. EventBridge Scheduler triggers the Lambda function every day.

2. Lambda function iterates over a list of RDS instances and attempts to stop each one.

3. All execution logs are stored in Amazon CloudWatch Logs for auditing and debugging.

Step-by-Step Setup Guide

1️⃣ . Create the Lambda Function

• Go to the AWS Lambda Console.

• Click “Create function” → Select Author from scratch.

• Function Name: stop-rds-lambda-function

• Runtime: Choose Python 3.13 or above.

• Permissions: Attach an IAM role with the following permissions:

  • rds:StopDBInstance

  • rds:DescribeDBInstances

  • logs:CreateLogGroup

  • logs:CreateLogStream

  • logs:PutLogEvents

  • 

Add Lambda Code

Paste the following code in the function editor:

import boto3
import logging

rds = boto3.client('rds')
logger = logging.getLogger()
logger.setLevel(logging.INFO)

DB_INSTANCES = ['testing-db-us-east-1-demo']  # Add your DB instance identifiers here

def lambda_handler(event, context):
    stopped_instances = []
    failed_instances = []

    for db_id in DB_INSTANCES:
        try:
            logger.info(f"Attempting to stop DB instance: {db_id}")
            response = rds.stop_db_instance(DBInstanceIdentifier=db_id)
            logger.info(f"Stop initiated for: {db_id}")
            stopped_instances.append(db_id)
        except Exception as e:
            logger.error(f"Failed to stop {db_id}: {str(e)}")
            failed_instances.append({'db_id': db_id, 'error': str(e)})

    return {
        'statusCode': 200,
        'stopped_instances': stopped_instances,
        'failed_instances': failed_instances
    }

3️⃣ . Create EventBridge Schedule

• Navigate to Amazon EventBridge → Scheduler.

• Click Create schedule.

• Schedule type: Choose Rate-based schedule.

• Set Rate: Every 1 day.

• Target:

  • Choose a Lambda function.

  • Select the Lambda function you created (stop-rds-lambda-function).

• Leave input blank.

• Click Next → Create schedule.

✅ Testing the Lambda

• Go to your Lambda function.

• Click Test to run it manually.

• Check the Amazon RDS Console to verify if the specified instances are now stopped.

• Review CloudWatch Logs for detailed output and error handling.

📌 Summary

By using AWS Lambda with EventBridge Scheduler, you can automate daily shutdowns of RDS instances, reducing unnecessary costs without manual intervention. This is especially helpful for non-production environments.