← Back to home
War Stories
Real production incidents, anonymized. No client names, no screenshots, no repo links.
Just the patterns that break systems — and the fixes that save them.
SEV1 Response Automotive Kafka / Debezium
The Kafka Consumer Group Bug That Looked Like a Network Problem
The Problem
A platform processing 50K+ events/minute experienced cascading CDC failures. Debezium connectors stalled,
consumer lag spiked to 6+ hours, and downstream services timed out. The on-call team had been chasing
"network issues" for 6 hours.
What I Did
- Identified a consumer group rebalance caused by a deployment split partitions unevenly — one consumer got 8, another got 0.
- Designed a static membership strategy to eliminate hot partitions during deployments.
- Created a Kubernetes-native failover runbook, removing dependency on drifted Terraform state.
- Hardened Terraform modules with remote state locking to prevent future drift.
The Outcome
Recovery within 4 hours. Consumer lag normalized. System stable under 2x peak load.
Team retained a 12-month advisory retainer.
Migration Assessment Enterprise WildFly / Kubernetes
From WildFly to Kubernetes: A Migration Assessment Framework
The Problem
A mid-size enterprise spent 3 months on a "lift-and-shift" Docker approach for their JBoss suite.
Result: memory leaks, 8-minute startup times, and failed health checks.
What I Did
- Built a 6-point evaluation framework: statelessness, config externalization, observability, startup behavior, health checks, resource predictability.
- Scored each service. Two ready for containerization. Four needed intermediate refactoring.
- Designed a phased plan with sidecar patterns and a Strangler Fig approach for the monolith.
The Outcome
First two services migrated in 6 weeks. Memory usage dropped 40%. Startup time: 8 minutes → 45 seconds.
MLOps / CI-CD Healthtech AWS / Terraform
The ML Pipeline That Worked Locally But Broke in Production
The Problem
A healthtech startup's inference pipeline worked in Jupyter but failed silently in AWS ECS.
Model artifacts weren't versioned. Feature stores were out of sync. No rollback strategy.
What I Did
- Audited the full pipeline: ingestion → features → training → artifacts → inference → monitoring.
- Introduced DVC for model versioning and a unified feature store contract.
- Rebuilt CI/CD with Terraform-managed ECS, parameterized model versions, and drift detection.
The Outcome
Inference failures dropped from ~15% to under 0.5%. Model rollbacks became one-click.
Data science team could deploy without DevOps.