Decentralization Storage: The Long Journey from Concept to Practicality

Decentralization storage was once one of the hottest tracks in the blockchain industry. Filecoin, as the leading project of the last bull market, once had a market capitalization exceeding 10 billion USD. Arweave, with its concept of permanent storage, also reached a market cap of 3.5 billion USD. However, as the availability of cold data storage is questioned, the necessity of permanent storage has also been called into question, and whether decentralization storage can truly be implemented remains an unresolved issue.

Recently, the emergence of Walrus has brought new vitality to the long-quiet storage track. Meanwhile, the Shelby project launched by Aptos in collaboration with Jump Crypto aims to achieve breakthroughs in the field of hot data storage. So, is it possible for Decentralization storage to make a comeback and provide solutions for a wide range of application scenarios? Or is it merely another round of concept speculation? This article will analyze the development trajectories of four projects: Filecoin, Arweave, Walrus, and Shelby, explore the narrative changes of Decentralization storage, and attempt to answer this question: How far is the popularization path of Decentralization storage?

Filecoin: Storage is just a facade, mining is the essence

Filecoin is one of the early rising altcoins, and its development direction naturally revolves around Decentralization. This is a common characteristic of early altcoins - seeking the meaning of Decentralization in various traditional fields. Filecoin is no exception; it links storage with Decentralization and points out the trust issues of centralized data storage service providers. Therefore, Filecoin's goal is to shift centralized storage to Decentralized storage. However, certain aspects sacrificed in the process of achieving Decentralization have become the pain points that later projects like Arweave or Walrus attempt to address. To understand why Filecoin is essentially just a mining coin, one needs to recognize the objective limitations of its underlying technology, IPFS, which is not suitable for handling hot data.

IPFS: Transmission bottleneck of Decentralization architecture

IPFS( InterPlanetary File System) was introduced around 2015, aiming to disrupt the traditional HTTP protocol through content addressing. The biggest drawback of IPFS is its extremely slow retrieval speed. In an era where traditional data service providers can achieve millisecond-level response times, retrieving a file from IPFS still takes several seconds, making it difficult to promote in practical applications and explaining why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is mainly suitable for "cold data", which refers to static content that does not change often, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic web pages, online games, or artificial intelligence applications, the P2P protocol does not have a significant advantage over traditional CDNs.

Although IPFS itself is not a blockchain, its design concept of directed acyclic graph (DAG) is highly compatible with many public chains and Web3 protocols, making it inherently suitable as a foundational framework for blockchain. Therefore, even if it does not have practical value, it is already sufficient as a foundational framework that carries the blockchain narrative. Early copycat projects only needed a workable framework to embark on their journey into the stars and the sea. However, as Filecoin developed to a certain stage, the fundamental flaws introduced by IPFS began to hinder its progress.

The logic of mining coins under the storage cloak

The original intention of IPFS's design was to allow users to not only store data but also to be part of the storage network. However, without economic incentives, it is difficult for users to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS, but will not contribute their own storage space or store others' files. It is against this backdrop that Filecoin was born.

In the token economic model of Filecoin, there are mainly three roles: users are responsible for paying fees to store data; storage miners earn token incentives for storing user data; and retrieval miners provide data when users need it and receive incentives.

This model has potential malicious space. Storage miners may fill garbage data after providing storage space to obtain rewards. Since this garbage data will not be retrieved, even if it is lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. Filecoin's replication proof consensus can only ensure that user data is not privately deleted, but it cannot prevent miners from filling garbage data.

The operation of Filecoin largely relies on the continuous investment of miners in the token economy, rather than on the real demand for distributed storage from end users. Although the project is still iterating, at this stage, the ecological construction of Filecoin aligns more with the "mining coin logic" rather than the definition of an "application-driven" storage project.

Arweave: Founded on Long-Termism, Defeated by Long-Termism

If the design goal of Filecoin is to build an incentivized, provable Decentralization "data cloud" shell, then Arweave takes a different extreme in storage: providing the capability for permanent data storage. Arweave does not attempt to build a distributed computing platform; its entire system is based on a core assumption - important data should be stored once and remain forever on the network. This extreme long-termism makes Arweave vastly different from Filecoin in terms of mechanism, incentive models, hardware requirements, and narrative perspective.

Arweave uses Bitcoin as a learning object, attempting to continuously optimize its permanent storage network over long periods measured in years. Arweave does not care about marketing, nor does it care about competitors and market trends. It is only progressing along the path of iterating network architecture, indifferent to whether anyone is paying attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave was warmly embraced in the last bull market; and because of long-termism, even after hitting rock bottom, Arweave may still survive several rounds of bull and bear markets. The only question is whether there will be a place for Arweave in the future of Decentralization storage. The existence value of permanent storage can only be proven over time.

Since version 1.5, the Arweave mainnet has been committed to enabling a wider range of miners to participate in the network at minimal cost, despite losing market discussion heat to the recent version 2.9. It encourages miners to maximize data storage, continuously enhancing the robustness of the entire network. Arweave is well aware that it does not align with market preferences, thus adopting a conservative approach, not embracing the miner community, and the ecology has completely stagnated. It aims to upgrade the mainnet at minimal cost while continuously lowering hardware thresholds without compromising network security.

A review of the upgrade path from 1.5 to 2.9

The Arweave version 1.5 exposed a vulnerability where miners could rely on GPU stacking instead of actual storage to optimize block production chances. To curb this trend, version 1.7 introduced the RandomX algorithm, which restricts the use of specialized computing power and instead requires general-purpose CPUs to participate in mining, thereby weakening power centralization.

In version 2.0, Arweave adopts SPoA, transforming data proofs into a concise path of Merkle tree structure, and introduces format 2 transactions to reduce synchronization burdens. This architecture alleviates network bandwidth pressure, significantly enhancing node collaboration capabilities. However, some miners can still evade true data holding responsibilities through centralized high-speed storage pool strategies.

To correct this bias, version 2.4 introduced the SPoRA mechanism, which incorporates global indexing and slow hash random access, requiring miners to genuinely hold data blocks to participate in effective block generation, thereby weakening the effects of hash power stacking from a mechanical perspective. As a result, miners began to focus on storage access speed, promoting the application of SSDs and high-speed read-write devices. Version 2.6 introduced hash chain control for block generation rhythm, balancing the marginal benefits of high-performance devices and providing fair participation space for small and medium miners.

Subsequent versions further strengthen network collaboration capabilities and storage diversity: 2.7 adds collaborative mining and pool mechanisms, enhancing the competitiveness of small miners; 2.8 introduces a composite packaging mechanism, allowing large-capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in replica_2_9 format, significantly improving efficiency and reducing computational dependencies, completing the closed loop of data-driven mining models.

Overall, Arweave's upgrade path clearly presents its long-term strategy oriented towards storage: while continuously resisting the trend of computing power centralization, it also aims to lower the participation threshold to ensure the long-term viability of the protocol.

Walrus: Is Embracing Hot Data Hype or Hidden Potential?

The design philosophy of Walrus is completely different from that of Filecoin and Arweave. Filecoin's starting point is to create a decentralized and verifiable storage system, at the cost of cold data storage; Arweave's starting point is to create an on-chain Alexandria library that can permanently store data, at the cost of too few scenarios; Walrus's starting point is to optimize the storage costs of hot data storage protocols.

Magic Modification of Error Correction Codes: Cost Innovation or Old Wine in New Bottles?

In terms of storage cost design, Walrus believes that the storage expenses of Filecoin and Arweave are unreasonable, as both latter adopt a fully replicated architecture. Their main advantage lies in each node holding a complete replica, which provides strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still maintains data availability. However, this also means that the system requires multiple copies for redundancy to maintain robustness, which in turn drives up storage costs. Particularly in Arweave's design, the consensus mechanism itself encourages nodes to store redundant data to enhance data security. In contrast, Filecoin is more flexible in cost control, but at the expense that some low-cost storage may carry a higher risk of data loss. Walrus attempts to find a balance between the two, aiming to enhance availability through structured redundancy while controlling replication costs, thereby establishing a new compromise between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology for reducing node redundancy, originating from Reed-Solomon(RS) coding. RS coding is a very traditional erasure coding algorithm, and erasure coding is a technique that allows for the doubling of datasets by adding redundant fragments( erasure code) to reconstruct the original data. From CD-ROMs to satellite communications and QR codes, it is frequently used in daily life.

Erasure coding allows users to take a block, for example, 1MB in size, and then "expand" it to 2MB in size, where the additional 1MB is special data known as erasure coding. If any byte in the block is lost, users can easily recover those bytes through the code. Even if up to 1MB of the block is lost, you can recover the entire block. The same technology enables computers to read all the data on a CD-ROM, even if it has been damaged.

The most commonly used is RS coding. The implementation method starts from k information blocks, constructs the related polynomial, and evaluates it at different x coordinates to obtain the encoded blocks. Using RS erasure codes, the probability of randomly sampling large amounts of missing data is very small.

What is the most notable feature of RedStuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed and stored across a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This is made possible while maintaining a replication factor of only 4 to 5 times.

Therefore, it is reasonable to define Walrus as a lightweight redundancy and recovery protocol redesigned around a Decentralization scenario. Compared to traditional erasure codes ( such as Reed-Solomon ), RedStuff no longer pursues strict mathematical consistency, but instead makes realistic trade-offs for data distribution, storage verification, and computational costs. This model abandons the instant decoding mechanism required for centralized scheduling, instead opting to verify whether nodes hold specific data copies through on-chain Proof, thereby adapting to a more dynamic and marginalized network structure.

The core design of RedStuff is to split data into two categories: primary slices and secondary slices. The primary slices are used to recover the original data, and their generation and distribution are subject to strict constraints. The recovery threshold is f+1, and 2f+1 signatures are required as availability endorsements; secondary slices, on the other hand, are through.