How Google Cloud Deleted a $125 Billion Account
In an “unprecedented,” “isolated,” and “one-of-a-kind occurrence that has never occurred before,’ Google Cloud accidentally pressed the “delete” button on the cloud account of a pension fund worth $125 billion. This article explores the incident, detailing what transpired, how Google recovered, and why it’s significant for managed service providers (MSPs) and small to medium-sized business (SMB) customers.
Table of Contents
What Happened at Google Cloud?
Google’s startling incident shook the tech world in May 2024, spotlighting the leading cause of data loss: human error. Due to a combination of human oversight and systemic vulnerabilities, Google Cloud deleted the private cloud account of UniSuper, a provider of retirement savings services for employees in the Australian research and higher education field. Reliant on Google’s cloud infrastructure for operations, UniSuper suffered a widespread, week-long outage where the organization’s 620,000 account holders could not access their accounts.
The catastrophe began with a routine maintenance procedure during which an engineer misconfigured an internal tool while provisioning. This seemingly minor mistake triggered a chain reaction, ultimately canceling UniSuper’s private cloud subscription and deleting all of the enterprise’s business-critical backups. Adding to the issue was the absence of immediate safeguards to stop or reverse such an extensive deletion process quickly.
Essentially, Google Cloud’s automated systems, designed to streamline operations, paradoxically contributed to the disaster by failing to recognize the error in time. The incident underscores the fragility of even the most advanced cloud platforms, revealing data protection and recovery protocol gaps.
Why Did the Outage Last So Long?
In a joint statement from UniSuper and Google Cloud CEOs, the organizations explain how complete downtime occurred and the role of third-party backup in disaster recovery. UniSuper’s data security strategy included duplication in two geographies to provide protection against outages and data loss. The problem with “putting all your eggs in one basket,” in this case, the Google Cloud basket, is that once one location is deleted, both locations are deleted. So, when UniSuper’s Private Cloud subscription was canceled, it caused deletion across both geographies, resulting in complete downtime.
Luckily, UniSuper was reinforcing data security with an additional backup service provider. Third-party backup and disaster recovery (BDR) providers independently maintain data backups for uninterrupted business continuity and rapid disaster recovery. With proactive, secure, and protected backups for its backups, UniSuper was able to recover in less time than it would have taken relying solely on Google. The joint statement reads…
“UniSuper had backups in place with an additional service provider. These backups have minimized data loss and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.”
How Did Google Cloud Recover UniSuper’s Account?
Swift and coordinated efforts were essential to recover UniSuper’s account, limit downtime consequences, and reestablish business continuity. Immediately after the error was discovered, Google Cloud’s incident response team launched a thorough investigation. Their priority was understanding the root cause, traced back to that internal system misconfiguration. Engineers worked around the clock to reconstruct the lost data with the help of a third-party backup provider and data recovery tools.
The Google team employed sophisticated data retrieval techniques, including snapshot recovery and incremental backups, to reconstruct the most recent and complete version of the UniSuper account. Equally as critical was UniSuper’s IT department, which collaborated with Google to ensure all critical data points were accurately restored.
What’s the Takeaway for MSPs?
As an MSP, it’s necessary for you and your clients to fully understand the threat vectors coming for your data – and, again, most data loss is caused by humans. While the joint statement describes the incident as “an isolated, ‘one-of-a-kind occurrence’ that has never before occurred,” when it comes down to it, it was a Google employee who caused the error, and that happens every day.
Utilize this story to inform clients of the importance of comprehensive, third-party business continuity and disaster recovery (BCDR) for the public cloud. Explain the impact of complete downtime and highlight the recovery features and technologies necessary for restoring data. For example, Axcient x360Recover delivers cloud and appliance-based deployments to protect both environments with backup and virtualization.
Leveraging built-in, always-on security with the following x360Recover features, Axcient partners can align their services with this particular use case. In essence, show them how your MSP can restore data in a similar situation.
Utilize Chain-Free, Image-Based Backups
Axcient’s proprietary Chain-Free backups provide a modern solution to the unending pain points of chains: compliance costs, storage overages, manual reseeding, failed backups, and storage bloat. With Chain-Free, data is stored in a native virtualizable state with a pointer array algorithm, making each recovery point independent.
When corruption or malware occurs, bad data blocks are isolated and can be independently deleted without risking data integrity in your backup dataset. Previous backups can still be recovered almost instantaneously, with new incremental backups being taken. Axcient’s patented Chain-Free technology enables pooled storage at a flat fee, custom and secure retention, and near-instant recovery.
Automatically Verify Backup Integrity
AutoVerify automatically verifies the integrity of backups using daily intelligent endpoint testing on Axcient and self-hosted private vaults, ensuring backups are always available and useable. It validates the recoverability of backup snapshots and captures a screenshot of the protected system to give clients and MSPs peace of mind. This on-by-default backup protection also immediately alerts, escalates, and even self-heals when backup failures are detected to safeguard against the potentially devastating consequences of backup burn.
Establish a Last Line of Defense
AirGap is an anti-ransomware and data deletion feature that serves as your last defense after an accidental incident or malicious attack on your backup files. Using Axcient’s Chain-Free backup technology, AirGap creates a firewall gap between the actual filesystem and the recovery solution. Continuous native snapshots of the filesystem are automatically taken and stored in a safety archive that is separate from the actual filesystem.
Data deletion requests can only be fulfilled after varying time gaps have passed and multiple validations are made internally. So, even when you think the data has been deleted, it’s actually safe in the archive, waiting and ready to be restored in as little as 15 minutes.
Self-Manage Disaster Recovery and DR Testing
Virtual Office with runbooks enables business continuity, IT systems testing, and anywhere, anytime data access for both cloud and appliance-based deployments of x360Recover. This cloud failover feature lets MSPs start virtual machines in the Axcient Cloud of one or more protected devices to replace all impacted production infrastructure temporarily.
Leveraging pre-planned recovery runbooks for DR testing, MSPs can also create a Virtual Office running within the Axcient data center using matching existing server configurations. Instantly recover production servers and workstations, perform regular full-office recovery tests, and easily secure access to Virtual Office instances using VPN, site-to-site OpenVPN, and port forwarding.
Ready to Reinforce Backups and Always Recover?
If you’re not confident that your MSP can recover a client in a similar situation, you must rethink your BCDR solution and data protection strategies. Learn more about Axcient x360Recover for public cloud BCDR and x360Cloud for Google Workspace backup and recovery to see how Axcient is different.
Author
Related posts
How well could you sleep with reliable cloud-based backups and recovery?
Take a deep dive into Axcient’s proprietary, automated security features to see how we’re ensuring uninterrupted business continuity — no matter what:
