On Wednesday morning, the Solana blockchain was on the verge of a complete freeze due to a routing error at a key data center provider, which resulted in nearly 29% of the network's staked tokens going offline, according to Marinade Finance.
The blockchain ceases to finalize transactions when over a third of the staked coins are unavailable. Marinade indicated that the network was dangerously close to this threshold, within about 20 million tokens. Approximately 90 validators were impacted, collectively losing 333 SOL in rewards, a minor amount that will be compensated through "validator bonds."
Marinade warned, "If delinquency had exceeded a third, nothing would finalize for SOL holders, and there’s no bond for that. The February 2024 halt took around five hours to resolve," in an explanatory post.
Solana, which boasts $4.3 billion in assets locked within its DeFi protocols, is known for being a faster and more affordable option compared to Ethereum. However, it has experienced multiple outages previously.
The incident originated from a faulty internet route from Teraswitch's facility in Miami, which affected data centers in Europe and Asia, disrupting validators located in cities such as London, Amsterdam, Frankfurt, Singapore, and Tokyo, while North America remained unaffected. The issue was resolved in approximately 10 minutes, and normal traffic resumed by 4:16 a.m. UTC.
A single network operator, identified as AS2032, managed over a quarter of all staked tokens necessary for securing the network, surpassing the safety limit set by Solana. Almost all of these tokens went offline simultaneously. Other companies experienced an additional loss of 14 million tokens during this brief period. Many affected validators, including a major one named Helius, remained offline for the full 33 minutes since their backup systems failed to activate.
This incident serves as a stark reminder that if more than one-third of the network's tokens were to go offline simultaneously, it would lead to a total freeze of the blockchain for all SOL holders, with no straightforward solution to the resulting larger issues.