When it comes to Chaos Engineering, there’s a big difference between “beginner” and “expert”. While early-stage chaos experiments might involve basic pod terminations or API latency injections, advanced practitioners know that true system resilience comes from testing the edge cases — the rare, complex failures that only happen in production.
On Day 3 of our 10 Days of Christmas Chaos, we’re diving into the world of advanced fault injection with “Frosty the Fault Injection” as our guide. If you’ve mastered basic chaos, it’s time to level up with more sophisticated experiments. From CPU throttling and “noisy neighbor” tests to advanced network partitioning, these techniques will challenge even the most experienced chaos engineers.
Ready to sharpen your skills? Let’s get started.
🎄 Advanced Chaos Experiment 1: CPU Throttling and “Noisy Neighbor” Simulations
What it Tests: System performance under CPU starvation and resource contention.
The Real-World Scenario: In shared infrastructure environments (like Kubernetes or cloud platforms), “noisy neighbors” can starve your critical workloads of CPU and memory. This happens when one container, pod, or virtual machine consumes more resources than expected, leaving other workloads without enough compute power.
How to Run This Experiment:
- Use Kubernetes Resource Limits: Set a “cpu limit” on pods to create a constrained environment. Use stress tools like
stress-ngto consume as much CPU as possible within the container. - Use Chaos Tools: Tools like Gremlin or Pumba let you inject CPU throttling into running workloads.
- Track Metrics: Measure request latency, error rates, and CPU usage on downstream services during the experiment.
What You’ll Learn:
- How your autoscaling logic responds to CPU starvation.
- If your priority workloads (like user-facing services) are properly protected from noisy neighbors.
- How well rate-limiting and backpressure controls kick in.
🎉 Pro Tip: Set resource limits on all production workloads to ensure “noisy neighbors” can’t impact critical services.
🎄 Advanced Chaos Experiment 2: Consistent Chaos Injection in CI/CD Pipelines
What it Tests: Continuous resilience testing during software delivery.
The Real-World Scenario: If you’re only running chaos experiments “on-demand” during scheduled game days, you’re missing out. Advanced teams like Netflix and Google inject chaos during every CI/CD deployment, ensuring that resilience tests run every time new code ships.
How to Run This Experiment:
- Automate Chaos in CI/CD: Integrate tools like Litmus Chaos or Gremlin into your CI/CD workflows (like GitHub Actions or Jenkins pipelines).
- Trigger Experiments on Deployments: Add a chaos step as part of your “post-deployment tests.” Example: Run a “network latency test” after every new deployment.
- Build Automated Rollbacks: If chaos causes a failure, ensure rollbacks are automatic and fast.
What You’ll Learn:
- If new code changes introduce resilience regressions.
- How well your release process handles chaos. (Does the CI/CD pipeline fail gracefully?)
- Whether rollbacks happen automatically when chaos is detected.
🎉 Pro Tip: If your chaos tests break the deployment, fail the build. Don’t let “failing gracefully” become a post-release discovery.
🎄 Advanced Chaos Experiment 3: Testing Database Rollbacks, Isolation, and Replicas
What it Tests: Database resilience under failure conditions like rollback, replica sync delays, and isolation.
The Real-World Scenario: Imagine a customer places an order, but halfway through, the database fails. Can you guarantee the order isn’t processed twice? Can your services read consistent data from a read-replica while the master is down?
How to Run This Experiment:
- Simulate Failovers: Use chaos tools to stop the “master” database instance, forcing a failover to replicas.
- Simulate Transaction Rollbacks: Use SQL transaction tests to create “incomplete commits” that trigger rollbacks.
- Delay Replica Syncs: Use network partitioning tools (like Pumba) to delay syncs between primary and replica nodes.
What You’ll Learn:
- How well your system handles read-after-write consistency.
- Whether you have any “ghost records” caused by rollbacks or race conditions.
- If your data replication lag causes inconsistencies during failover.
🎉 Pro Tip: Always test “read-after-write” guarantees for user-facing data like user profiles, payments, and transactions.
🎄 Advanced Chaos Experiment 4: Advanced Network Partitioning
What it Tests: How your system responds to partial network failures.
The Real-World Scenario: What happens when half of your Kubernetes cluster can’t talk to the other half? This is known as a “network partition” or split-brain scenario. Without proper fallback logic, systems can end up with “data divergence” — where one half of the system holds a different view of the data than the other.
How to Run This Experiment:
- Use Tools Like Pumba or Istio: Create network partitions at the container, pod, or node level.
- Partition at the AZ or Region Level: Partition an entire AWS or Azure region to see if traffic fails over.
- Create Bi-Directional Partitions: Partition one half of the cluster from the other half, and watch how services behave.
What You’ll Learn:
- How well your systems handle split-brain conditions.
- If your failover paths (like multi-region traffic routing) actually work.
- Whether your distributed systems have proper conflict resolution logic.
🎉 Pro Tip: Run “network split” chaos experiments during Game Days to see how well on-call engineers can detect and resolve the issue.
🎉 Which Advanced Chaos Experiments Do You Want to See?
These experiments are just the beginning. Advanced chaos engineering requires creativity, experimentation, and boldness. What chaos experiments do you want to see broken down next?
Here’s some inspiration:
- API Chaos: What happens when API rate limits hit unexpectedly?
- TLS Expiration: Simulate expired certificates and see how apps handle it.
- 3rd Party API Failures: What happens when external providers go down?
Drop your chaos experiment ideas in the comments! We’ll feature the most creative ones in a future post.
Closing Thoughts
Advanced Chaos Engineering isn’t for the faint of heart. It requires deep system knowledge, a robust testing platform, and a strong incident response plan. But it’s worth it. The failures you find today are the outages you prevent tomorrow.
As Frosty the Fault Injection would say, “There must have been some magic in that old system log they found” — because every log, every alert, and every experiment is a chance to improve.
This Christmas, as you sip on eggnog and reflect on your system’s resilience, ask yourself: Are you ready for advanced chaos? If not, start with one of these experiments and level up your skills. 🎄🎁
