The Challenge of Circular Dependencies
Did you know that GitHub hosts its own source code on github.com? As the platform’s biggest customer, the company tests changes internally before rolling them out to users. However, this practice introduces a critical risk: if github.com goes down, engineers lose access to the very code needed to fix it. This creates a simple circular dependency: to deploy GitHub, you need GitHub. While a mirror of the code and built assets for rollback provide a safety net, more subtle circular dependencies persist.
For instance, a deployment script might inadvertently rely on an internal service or download a binary from GitHub itself. During an outage, such dependencies can halt recovery. To tackle this, GitHub designed a new host-based deployment system that leverages eBPF (extended Berkeley Packet Filter) to monitor and block problematic calls. This article explores how eBPF helps break these cycles and offers insights for writing your own eBPF programs.
Types of Circular Dependencies
Consider a hypothetical scenario: A MySQL outage prevents GitHub from serving release data from repositories. To resolve the incident, engineers need to roll out a configuration change to the affected MySQL nodes by executing a deploy script. The following circular dependencies could arise:
Direct Dependency
The MySQL deploy script attempts to pull the latest release of an open source tool from GitHub. Since GitHub cannot serve the release data due to the outage, the script cannot complete. This is a straightforward dependency on the very service that is down.
Hidden Dependencies
The deploy script uses a servicing tool already present on the machine’s disk. However, when the tool runs, it checks GitHub for an update. If it cannot contact GitHub (because of the outage), the script may fail or hang, depending on how the tool handles the error.
Transient Dependencies
The deploy script calls, via an API, another internal service (such as a migrations service). That service, in turn, attempts to fetch the latest release of an open source tool from GitHub to use the new binary. The failure propagates back to the deploy script.
How eBPF Addresses the Problem
Traditionally, each team owning stateful hosts had to manually review deployment scripts for circular dependencies. In practice, many dependencies go unnoticed. GitHub turned to eBPF, a kernel technology that allows sandboxed programs to run in the kernel without modifying source code or loading kernel modules. With eBPF, GitHub can selectively monitor and block specific system calls from deployment scripts.
Implementing eBPF for Deployment Safety
GitHub’s approach involves attaching eBPF programs to system call tracepoints (e.g., connect(), open()) to detect when a deploy script tries to access a blacklisted resource, such as an internal service or GitHub itself. The eBPF program can either log the event or block the call entirely, ensuring that deployment code cannot create new circular dependencies. This is done without altering the script or the application.
For example, during the MySQL outage scenario, an eBPF program could intercept any attempt by the deploy script to reach api.github.com and deny the call, forcing the script to use local resources. The program can also emit metrics for monitoring and debugging.
Key Takeaways
By integrating eBPF into their deployment pipeline, GitHub enhances site reliability and deployment safety. The technology provides fine-grained control over system behavior without performance overhead or code modifications. Other teams can adopt similar methods to prevent circular dependencies in their own host-based systems.
For a deeper dive, refer to the original GitHub engineering blog post on types of circular dependencies and implementation details. With eBPF, the burden of manual review is replaced by automated enforcement, making deployments safer and more resilient.