Breaking: Cloudflare Wraps Up 'Code Orange: Fail Small' – Promises Stronger, More Resilient Network
Cloudflare announced today the completion of a sweeping internal engineering project, Code Orange: Fail Small, aimed at preventing global outages like those that struck on November 18 and December 5, 2025. The initiative, which spanned over two quarters, focused on safer configuration changes, reduced failure impact, and improved incident management.
“This work was laser-focused on the root causes of those disruptions,” a Cloudflare spokesperson said. “We’ve introduced new tools and processes that make our network far more resilient to future incidents.” The company confirmed that the measures would have prevented both outages, which affected millions of users worldwide.
Safer Configuration Changes
At the core of the changes is a new component called Snapstone, which enables health-mediated deployment for configuration updates. Previously, internal configuration changes could propagate instantly across the network, risking widespread impact if errors occurred. Now, changes are rolled out progressively with real-time health monitoring, allowing automatic rollback if problems are detected.
“Think of it as a safety net for every configuration change,” a Cloudflare engineer explained. “Snapstone lets us catch issues before they ever affect customer traffic.” High-risk configuration pipelines have been identified and new tools built to manage changes more carefully, ensuring that only safe, verified updates reach production.
Reducing the Impact of Failure
Cloudflare also revised its “break glass” procedures – emergency override mechanisms used during incidents. These are now designed to limit blast radius and prevent cascading failures. Incident management workflows were overhauled to shorten response times and improve coordination across teams.
“We’ve hardened our response playbooks,” said a Cloudflare network reliability manager. “If something does go wrong, we can now isolate and fix it much faster, with less disruption to customers.”
Preventing Drift and Regressions
To ensure improvements stick, Cloudflare introduced measures to prevent configuration drift and regressions over time. Automated checks and regular audits now enforce consistent application of new policies across the entire network. This includes stricter review processes for all changes, not just those related to previous outages.
“We’re not just patching a hole; we’re changing how we operate,” the spokesperson noted. “This is a permanent shift toward proactive reliability.”
Background
The November 18 and December 5, 2025 outages were triggered by configuration errors in Cloudflare’s global network. The November incident involved a corrupted data file, while the December outage stemmed from a faulty control flag in the global configuration system. Both caused widespread service degradation for millions of websites and applications.
Cloudflare’s internal post‑mortems revealed gaps in how configuration changes were tested and deployed. The Code Orange project was launched shortly after to address these vulnerabilities, with a mandate to “fail small” – ensuring any single failure impacts the smallest possible set of users.
What This Means
For customers, the key benefit is increased reliability. Configuration errors that once could take down large portions of the network will now be caught and rolled back automatically, often without any noticeable impact. Progressive rollouts mean that even if a mistake slips through, damage is contained.
“Our customers can expect fewer unplanned outages and faster resolution when issues do arise,” the reliability manager said. “We’re building trust through transparency and engineering rigor.” Cloudflare also committed to more transparent communication during incidents, including clearer updates on root causes and mitigation steps.
The completion of Fail Small does not mean Cloudflare is done improving. “Resilience is a journey, not a destination,” the spokesperson emphasized. “We will continue to invest in protecting our network and our customers.”