Railway's network architecture, a mesh ring connecting AWS, GCP, and Metal, incorporates high availability into its interconnects, path routing, and database. This design effectively mitigates regional outages and data plane issues within a single cloud provider. However, recent incidents highlight critical vulnerabilities within the Google Cloud control plane, challenging traditional multi-cloud resilience strategies.
Your Multi-Cloud Isn't Truly Multi-Cloud
Railway's network architecture, a mesh ring connecting AWS, GCP, and Metal, incorporates high availability into its interconnects, path routing, and database. This design effectively mitigates regional outages and data plane issues within a single cloud provider. However, the broader implications of a compromised Google Cloud control plane demand a re-evaluation of these strategies.
The May 2024 UniSuper incident highlighted a critical distinction: the problem frequently shifts from data plane issues to control plane vulnerabilities. Google's VPC, by its fundamental design, does not inherently guarantee high availability against control plane failures spanning multiple zones or regions without explicit, advanced architectural patterns. While Railway has indicated plans to add shards to Metal and AWS for VPC redundancy, this addresses only part of the issue. The UniSuper deletion stemmed from a subscription-level administrative action, rather than a VPC issue. This administrative action, likely originating from Google's internal provisioning systems, bypassed all customer-level redundancy, exposing a profound weakness in the Google Cloud control plane.
This incident exposed a global administrative blast radius, far exceeding a typical regional failure. It underscored that even robust multi-region deployments within a single cloud provider might not protect against systemic control plane errors.
The Google Cloud Control Plane Is the New Blast Radius
When Google Cloud's internal processes can trigger a cascading deletion across redundant systems, it exposes a fundamental vulnerability: the hyperscaler's own operational consistency. You can architect for multi-region failover and distribute your data, but if the underlying provider's control plane can globally delete your account or subscription, your availability is compromised, demonstrating the perils of misapplied internal consistency. This is precisely the risk posed by an uncontained Google Cloud control plane failure.
The observation that GCP experiences more global outages compared to AWS's often region-contained issues (such as us-east-1 control plane problems) is a frequently discussed pattern within the industry. This suggests a fundamental divergence in how these providers architect their control planes, particularly concerning the scope of internal consistency and the containment of blast radii. Understanding how the Google Cloud control plane operates differently is crucial for designing truly resilient systems.
Why Your Backups Need to Live Elsewhere
While the 3-2-1 backup rule offers a strong foundation, its application in the cloud requires further consideration due to the risk of subscription-level deletions. If your "3 copies" and "2 different media" reside within the same cloud account, a subscription-level deletion event can eradicate them all. This necessitates independent provider backups, completely isolated from the primary cloud's administrative domain and the potential reach of the Google Cloud control plane.
Your critical data must exist entirely outside the blast radius of your primary cloud provider's control plane. For recovery, relying on Google Cloud's soft deletes requires rapid support contact. However, the customer support experience is often reported to be challenging, characterized by difficulties in reaching effective human assistance and navigating Account Executive interactions. This directly impacts your Recovery Time Objective (RTO) and Recovery Point Objective (RPO). A slow, unresponsive support channel extends downtime, irrespective of technical redundancy, making the integrity of the Google Cloud control plane even more critical.
Railway's past incident, where deleting a production database also deleted its backups, highlights a design flaw in their API at the time. It also underscores the need for rigorously validated, independent backup strategies immune to application-level or administrative errors. This incident serves as a stark reminder that even application-level safeguards can be bypassed by underlying control plane actions or design flaws.
What True Resilience Looks Like
For platforms like Railway, building on hyperscalers inherently involves accepting certain dependencies. Nevertheless, architectural design can significantly minimize the impact of such failures. True resilience mandates genuine multi-cloud deployment, running active-active or active-passive workloads across *distinct* cloud providers, not merely different regions within a single one. Should GCP experience a global control plane failure, an AWS or Metal deployment must seamlessly assume operational control without manual intervention.
This necessitates decoupled control planes, ensuring critical data and operational state are not solely governed by a single provider's administrative domain. Strategies might include managing proprietary Kubernetes clusters across clouds or leveraging services that abstract the underlying infrastructure. Crucially, externalized backup and disaster recovery are non-negotiable. Backups must reside entirely outside the primary cloud provider's account—whether in another cloud, on-premise, or via a specialized service—forming the ultimate defense against a global deletion event originating from the Google Cloud control plane.
Furthermore, idempotent operations are fundamental for any system requiring data loss recovery. If event consumers are not idempotent, restoring from a backup and reprocessing events, especially with at-least-once delivery guarantees like Kafka's, risks duplicate charges or critical state corruption. Implementing robust idempotency ensures that even if a system needs to reprocess events due to a control plane incident, data integrity is maintained.
Finally, automated failover and recovery are paramount. Reliance on human support, particularly when it is demonstrably inadequate, inevitably extends outage durations. Failover mechanisms, including DNS propagation, which now typically completes within five minutes, must be as automated as technically feasible. This automation reduces human error and accelerates recovery, directly counteracting the potential delays introduced by a compromised Google Cloud control plane or slow support channels.
The UniSuper incident clearly demonstrated that even the largest cloud providers can make severe errors at the control plane level. Google Cloud's "one-of-a-kind" claim does not negate the architectural lesson: your resilience strategy must account for the possibility that your cloud provider might inadvertently delete your entire infrastructure. The critical consideration is not merely the inevitability of such incidents, but rather the proactive measures taken to ensure rapid recovery when the underlying infrastructure itself becomes the source of compromise, especially when dealing with the complexities of the Google Cloud control plane.
