By Alex Liu, Foresight News
The decentralized storage network Arweave launched the computing layer AO, which successfully caused the reflux of AR currency price, ecology and popularity, which can be said to be a turnaround. As a general computing chain, Sui launched the decentralized storage network Walrus. What kind of waves will it set off?
Background
team
The development company behind Solana is called Solana Labs, the development company behind Aptos is called Aptos Labs, and the development company behind Sui is called Mysten Labs (it’s just that unique). Most of the founders and employees of Mysten Labs came from Diem, a blockchain project that was disbanded by Facebook (now Meta).
Walrus is the latest product classified as a "protocol and platform" by Mysten Labs. It is a decentralized storage network. Walrus means "walrus" in English. Its official website has the slogan "thriving like a walrus" and "adaptable like a walrus", which conveys the reliability and availability of the protocol as a storage system.
Contact with Sui
Walrus is built on Sui and uses Sui to coordinate the sale of storage space and metadata. However, using Walrus does not require building applications or products on Sui, and the new governance token WAL will be used as a utility token instead of SUI.
Comparison of competitive products
Decentralized storage protocols are generally divided into two categories. The first category is a fully replicated system, and the main competitors Filecoin and Arweave are typical representatives of this type of system. The main advantage of this type is the complete file availability on the storage nodes, making it easy to access and migrate files even if a storage node is offline. This setup can bring a permissionless environment because storage nodes do not need to rely on each other to recover files.
The reliability of such systems depends on the robustness of the selected storage nodes. Under the classic one-third static adversary model and the assumption of an infinite pool of candidate storage nodes, achieving "twelve nines" security (i.e., the probability of losing file access is less than 10^-12) requires storing more than 25 copies on the network. This results in a 25x storage overhead. In addition, there is the problem of possible Sybil attacks, where malicious actors can pretend to store multiple copies of a file, weakening the integrity of the system.
The second type of decentralized storage service uses Reed-Solomon (RS) coding. RS coding divides files into smaller parts, called slices, where each slice represents a portion of the original file. As long as the total size of the slices is larger than the size of the original file, the original file can be decoded. RS coding also has its disadvantages. The encoding and decoding process relies on domain operations, polynomial evaluation, and interpolation, which are computationally expensive. These operations are only practical when the size of the domain and the number of slices are relatively small, thus limiting the size of the encoded file and the number of participating storage nodes. Otherwise, the cost of encoding becomes very high, limiting the degree of decentralization. Another problem is that when a storage node goes offline and needs to be replaced by another node, unlike a fully replicated system, the data cannot simply be copied from one node to another. The RS-coded system requires all existing storage nodes to send their slices to the replacement node, which then recovers the lost slices. However, this process results in O(|blob|) data being transferred over the network. Frequent recovery operations reduce the storage savings gained by reducing replication.
Storage Challenges
Regardless of the replication protocol used, all existing decentralized storage systems face two additional challenges:
Ongoing challenges are required to ensure that storage nodes retain data and do not discard it. This is critical in an open decentralized system that provides payment for storage, but currently this practice limits the scalability of the system because each file requires a separate challenge.
Storage nodes need coordination: they need to know who is in the system, which files have been paid for storage, implement incentives for participation, and manage challenges and mechanisms to mitigate abuse. This is why each of the above systems has implemented a custom blockchain to execute transactions and introduced cryptocurrencies outside of the storage protocol.
Core Innovation
Faced with these challenges, what innovations does Walrus have that can bring different solutions to decentralized storage?
In simple terms:
By adopting the innovative technology of erasure coding, Walrus is able to quickly and robustly encode unstructured data blocks into smaller shards, which are distributed and stored in a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data blocks can be quickly reconstructed using some of the shards. This is possible while keeping the replication factor at only 4x to 5x, which is comparable to existing cloud services, with the advantages of decentralization and wider fault resilience.
Specifically:
Walrus has launched RedStuff, a new 2D encoding algorithm designed for Byzantine Fault Tolerance. RedStuff is based on fountain codes and combines the advantages of fast operation and high reliability.
RedStuff encodes data into primary and secondary slices through simple operations (mainly XOR, exclusive OR operation). These slices are distributed among storage nodes, and each node holds a unique combination. RedStuff uses different thresholds for encoding in different dimensions. The primary dimension adopts a recovery threshold of f+1, which allows asynchronous writing because only 2f+1 signatures are required to prove that the data block is available, which has formed a replication factor of 3 times.
The secondary dimension uses a recovery threshold of 2f+1, which enables asynchronous storage proofs for the first time while only introducing 1.5x additional replication, resulting in a total replication factor of less than 5x. More importantly, lost slices can be recovered based on the amount of lost data, saving bandwidth, all thanks to 2D encoding.
The advantages of RedStuff include: the use of simple XOR operations makes encoding/decoding faster than RS encoding; due to low storage overhead, the system can scale to hundreds of nodes and has high resilience and fault tolerance, ensuring data recovery even in the event of Byzantine failures.
As a permissionless protocol, Walrus is equipped with an efficient committee reconfiguration protocol to cope with the natural loss of storage nodes and ensure the continuous availability of data. When a new committee replaces the current committee between two epochs, the reconfiguration protocol ensures that all data blocks that have exceeded the point of availability (PoA) are still available. RedStuff's 2D encoding makes state migration more efficient, and even if some nodes are unavailable, other nodes can recover the lost slices.
Node 1 and Node 3 help Node 4 recover slice data
Walrus introduces an asynchronous challenge protocol to verify that nodes store data correctly. The protocol allows efficient storage proofs that ensure data availability without relying on network assumptions, and its cost scales logarithmically with the number of stored files.
Walrus' economic model is based on staking, combined with a reward and penalty mechanism. The innovative storage certification mechanism scales logarithmically with the number of stored files, reducing the cost of proving file storage.
In summary, Walrus, with the RedStuff protocol at its core, provides a scalable, resilient, and economically viable decentralized storage solution that delivers high authenticity, integrity, auditability, and availability at a reasonable cost.
All this is thanks to Sui as the control layer of Walrus. Having a scalable, programmable and secure infrastructure as a coordination layer allows it to focus on the core issues of decentralized storage.
Potential Airdrops
Walrus will launch an independent token WAL, and Utility has staking, governance, etc. How can you get WAL airdrops? Referring to the way to obtain AO, holding SUI may be one of the ways.
Walrus is expected to launch a testnet soon, and the mainnet launch time is yet to be determined. You can go to the official documentation to learn how to deploy your own website using Walrus.
Sources:
Walrus Whitepaper:
https://docs.walrus.site/walrus.pdf
Walrus: Decentralized storage and DA protocol, L2 and large storage can be built based on Sui: https://foresightnews.pro/article/detail/63040
Mysten Labs Researcher X Thread:
https://x.com/LefKok/status/1836868240666153293
