Exploring the decentralized storage landscape

Zachary Trudeau

March 22, 2023

Have you ever wondered how big the internet is? It seems like a rhetorical question, as the internet endlessly expands without any known limit to its capacity. And how would we quantify its actual size?

We would have to consider the amount of data transferred, along with every website and its data being hosted across the internet. Think about the amount of data being created as you read this article—every Tweet and Google search query happening this instant is adding to the size of the internet.

In 2021, there were 79 zettabytes of data generated. You can refer to this page here for memory size units. The size of the internet has been growing exponentially each year. And if you would like to read more about these data statistics, you can do so here. But how is this important for Web3?

On-chain data and new possibilities

For starters, we need to consider the cost of data on the Ethereum network. Depending on different variables such as network loads and market prices, it can easily cost well into the tens of millions to store a single gigabyte of data—this is neither scalable nor feasible. Also, with Ethereum’s EIP-4444 proposal, data older than a year will not be served by clients on the P2P layer.

The case of Ethereum might be more extreme, but similar issues can also impact other L1 networks. The number of NFTs being minted with audio and video content is continuously increasing, along with the demand for storage. The option to offload on-chain data can significantly free up network resources and make it easier for validators to get started.

This brings us to decentralized storage. Let’s see how it will transform the storage landscape in Web3, as well as provide more opportunities and solutions for the developers out there BUIDLing amazing DApps!

Introducing decentralized storage

Up until now, when we stored data in the cloud, it has traditionally been stored on centralized servers in a data center based in a singular location. This is centralized storage. Decentralized storage, on the other hand, is data stored across a distributed network of nodes in multiple locations.

It works on a peer-to-peer (P2P) basis in which one user will rent out their free storage space to another user. The fees are paid in the native token of that particular storage protocol. This Web3 technology is revolutionizing the way data can be stored by offering benefits not achievable with traditional centralized storage systems.

Cost savings

In centralized storage, the prices are dictated by policies intended to generate the maximum profit, and there’s not much competition in the market to challenge the status quo. Decentralized storage changes this by allowing a new paradigm of storage providers into the mix. These providers rent out their space at competitive rates to win the bid for the storage contract.

Security and reliability

Because Web3 data is stored on a distributed network, there is no single point of failure that can lose, destroy, or alter the data. The data redundancy policy of having it spread across the network ensures that it is always accessible and maintains integrity. On a centralized network, this is not the case, as data can be lost or corrupted.

Enhanced privacy

Blockchain data can be sharded and distributed to many hosts across a decentralized network. This makes the file inaccessible to everyone, except to the holder of the private key for that data chunk. Data on centralized servers is often a target for hackers because it can have immeasurable value.

Easy data migration

Data can easily be moved between IPFS-based storage providers if the user wants. This kind of flexibility allows a user a higher level of control with how and where their data is stored. By contrast, centralized providers can keep users locked into contracts with premium costs and penalties that don’t favor the user.

A necessary progress

As we move further into the Web3 space, the importance and relevance of decentralized storage cannot be emphasized enough. In fact, in order for Web3 to scale it is absolutely necessary to have a decentralized solution. A case in point, the aforementioned Ethereum EIP-4444 proposal makes it all but optional to store historical on-chain data somewhere that it can be accessed.

Consider how young Web3 still is. Now consider how much data will be generated on-chain as the space continues to grow and attract more users. All the NFT’s that will be minted, all the websites that will be hosted, and other future uses of smart contract data will generate enough data to require its own storage space to keep networks running as efficiently as possible. Regardless of the protocol, decentralized storage is required for Web3 to grow and function properly.

Figure 2: The exponential growth of data; Source firstsiteguide.com

This is just taking into account the technical aspect based on how Web3 operates. There is also the developer aspect to decentralized storage. For developers, there is a new piece to the puzzle here. The ability to write smart contracts that interact with both on-chain and off-chain data is something that was not previously possible. Now developers have more power than ever to BUIDL world changing DApps with data.

The current landscape

At the moment, there are four primary L1 decentralized storage protocols serving both individual and enterprise levels of use. They are Filecoin, Storj, Arweave, and Sia, which we will break these down further to see how they compare with each other.

Filecoin

Filecoin is a sophisticated protocol offering data infrastructure solutions for multiple use cases. Its versatility also makes it one of the more misunderstood protocols. Enabled by its tokenomics, it allows users to store large quantities of data with smart contract support at incredibly affordable costs.

Although primarily used for cold storage, with CDN retrieval, computation of data, and the Filecoin Virtual Machine (FVM), hot storage solutions are also possible with this protocol. Additionally, through Filecoin’s FVM smart contracts can be built on top of permanent storage allowing developers to create powerful DApps. The data itself may be stored off-chain, but its integrity and assurance of proper storage are kept via smart contracts on-chain.

Storj

Storj offers flexibility for developers of all types, from individual to enterprise. It strikes the perfect balance between Web2 and Web3 protocols with S3 compatible edge services coupled with its own native protocol and IPFS integration. Comprehensive management tools are made available seamlessly while service level agreements guarantee high standards of quality assurance.

It offers a robust streaming capacity of up to 24 Gbps, making it an ideal choice for hot storage solutions. Moreover, its native token STORJ is an ERC20 standard and compatible with the Ethereum network, which ensures smooth usability while encrypting and coding data redundancy itself—ensuring maximum security without sacrificing performance.

Arweave

Arweave has gained significant attention in the decentralized storage space by providing innovative solutions to ensure data permanence. Notable solutions include their lazy smart contract and endowment model. This combined with strategic partnerships with major L1s have elevated Arweave as an attractive option for developers to BUIDL within its ecosystem.

Despite being well-positioned in the decentralized storage market, some important issues must be considered. Without consequences for dropped data, there is no assurance that it will remain permanent. Furthermore, incentivizing miners to upload and retrieve large volumes of information timely can become much more expensive due to bandwidth charges. Finally, AR token price fluctuations heavily impact storage costs.

Sia

With Sia, you can benefit from both decentralization and cost-efficiency. The protocol’s miners and community hold 95% of the token, making it the most decentralized option for data storage. It also excels in hot storage use cases with an impressive TTFB (time to first byte) of around 200 milliseconds.

Miners can get up and running with minimal start-up costs due to the hard drive being the only expense. The surging popularity of DApps on this platform has created a strong network within its community. However, their data model and messaging architecture are mostly proprietary and could still be used for improvement for interoperability between protocols.

The obstacles to adoption

While this technology shows high potential and is meant to solve many issues in the Web3 space, it is not met without challenges and limitations. We can break these down into two categories—technical and people challenges.

Technical

The storage service providers have distributed data centers with varying internet bandwidth capacities. However, providing a consistent Service Level Agreement (SLA) guarantee across all locations presents a challenge to overcome. Unless certain requirements are standardized, the quality of service can be affected from region to region.

The interoperability between both varying L1 protocols and Web2 to Web3 providers is another set of challenges. To be able to query across multiple L1 protocols with differing messaging standards needs a solution. Additionally, data on a centralized server may need to communicate with data on a decentralized server.

Centralized competition

With centralized storage solutions, users often get more than storage. They are usually offered an entire suite of services for analytics, computations, and front-end applications. For enterprise users, this can be a driving factor in which providers to choose. It will be a matter of time before Web3 can offer similar tools.

With data securely stored across a distributed network, centralized computing has difficulty querying it. This is an obstacle in the business-to-business adoption of decentralized storage systems. Without seamless integration with content delivery networks (CDN), this technology remains out of reach for many organizations.

There has been a lack of tools to improve the developer and user experiences. A lack of proper SDKs and UIs has made the barrier to entry steeper for those looking to migrate into decentralized storage solutions. Although, this will not likely be the case for too much longer.

People

Humans are creatures of habit. Breaking old routines and adopting new ones is a challenge from one generation to the next. At present, centralized storage is a comfort zone for many users, and it does take time to build people’s trust and confidence in a new technology without skepticism and Luddite resistance. Additionally, when you need predictable results, you already know what you’re getting with centralized providers.

On an enterprise level, costs need to be calculated to determine whether such a migration is beneficial and feasible. However, there are vast differences between the two storage mediums, and what may apply to one doesn’t apply to the other.

For example, Web2 vendors tend to lock in customers with contacts that make migration costs expensive and inconvenient to business. This problem doesn’t exist in decentralized storage, but because of this, costs could be overestimated, and the migration discouraged.

Knowledge matters

There has not been a push for public awareness or education on decentralized storage and Web3 solutions in general. This leads to misconceptions and misinformation about the technology, like the possibility of data loss or privacy and security concerns—which both are actually strengths in Web3. But also, without the proper knowledge, the mechanics and benefits are perplexing for many users still.

Lastly, some enterprises face the challenge of adhering to strict data protection regulations. If it is required to know the exact physical location of where the data is being stored, this can present legal difficulties.

As you can see there are obstacles to overcome yet. But that’s the key word here—overcome. There are more opportunities than ever for Web3 developers to innovate and shape things into an ecosystem of abundance and high-level user experience. And with the Web3 community growing every day, the people’s challenges to this are decreased.

A Web3 and Web2 comparison

While there are many differences between decentralized and centralized storage, the main technical difference is in the data transfer protocols. In Web2, this was accomplished with a client-server model, HyperText Transfer Protocol (HTTP). In Web3, it is done using a P2P model via InterPlanetary File System (IPFS).

IPFS versus HTTP

Rather than relying on one server to store files and deliver content as is typical in HTTP systems, IPFS uses nodes spread across an extensive network for file distribution and access. This innovative system allows for faster access to content with increased security and reliability compared to conventional HTTP connections with a central point.

HTTP is location-based and utilizes URLs to locate resources with an IP address. IPFS is content-based and uses a content-addressing system called content ID (CID)—this is achieved through identifying each item based off its distinct cryptographic hash.

IPFS accelerates content delivery by allowing data to be cached with multiple nodes, whereas HTTP caching is done directly on the user’s device. It offers great scalability through its caching capabilities, eliminating the need for client-side HTTP cache. This allows content to be efficiently stored, served from multiple nodes, resulting in optimal performance.

HTTP is primarily designed for delivering web-centric content, like HTML and CSS. However, IPFS has the capacity to distribute a much broader range of information, from photos and videos to large files with complex data sets.

How decentralized storage works

The mechanics of how decentralized storage operates are much simpler than the mystery surrounding Web3 solutions. Data is stored in a network of nodes which are not controlled by any single entity. And this network is geographically distributed, which makes it more resistant to failure, censorship, and data breaches.

When the data is stored on the network, it is divided into smaller chunks and encrypted before being stored on multiple nodes in the network. Because of the tamper-proof nature of the blockchain and the fact that no single node controls the entire set of data, it is incredibly difficult for hackers to access sensitive information.

When retrieving data from the network, a user’s request will pull the necessary data chunks from the various network nodes and reassemble them. This entire process of data retrieval is transparent to the user, who gets to access the data like if it was stored on centralized servers.

Figure 4: Saving and retrieving a file with decentralized storage

Security and privacy

Similar to centralized systems, security and privacy remain key priorities in decentralized storage systems. To accomplish this; encryption, consensus algorithms, and smart contracts are used in conjunction with each other to ensure that data remains secure and private within the network.

Data security

Data security is strengthened through encryption. Before being transferred and stored onto the network, confidential information is encrypted to prevent unauthorized access. When requested by users, data remains safely protected until decryption takes place—ensuring that sensitive information stays secure at all times.

Consensus mechanism

Consensus algorithms act as a cornerstone in network security and reliability. By establishing trust between nodes across the entire system, they prevent malicious actors from manipulating data or disrupting operations. In turn, this enables smooth functioning of networks with peace-of-mind assurance that concerns over tampering are addressed.

Smart contracts

Smart contracts provide an automated and secure layer of protection for data stored in decentralized systems. Through self-executing code, they can ensure that only those who have been granted permission can access valuable information stored on a decentralized storage system. This safeguards sensitive digital assets from malicious forces.

Figure 5: Risk mitigation techniques for decentralized storage

Technical components

The technical details of decentralized storage can be complex and vary from system to system. However, with encryption, chunking, P2P networks, distributed hash tables, consensus algorithms, smart contracts, and incentives, decentralized storage systems can provide a secure and powerful alternative to traditional centralized storage systems.

Figure 6: Components of decentralized storage

Use cases for decentralized storage

There is a plethora of ways that decentralized storage can be used, and each use case of it brings improvements and innovation. Here are some ways that industries can benefit from its adoption:

Increased data security for industries with sensitive or confidential data, such as healthcare, finance, and government
Improved data accessibility from multiple locations and devices for industries that require remote access, such as manufacturing and logistics
Cost savings by eliminating the need for expensive hardware and infrastructure for industries that rely heavily on data storage and processing, such as media and entertainment
Increased transparency by allowing multiple parties to access and verify data in industries that require transparency and accountability, such as supply chain management and government
Improved data backup and recovery by allowing data to be replicated across multiple nodes in industries that rely on continuous access to data, such as healthcare and emergency services

Figure 7: Use case scenarios for decentralized storage

The future of decentralized storage

As the future of decentralized storage continues to grow, from advancements in privacy protection to innovations in security solutions, it’s clear that these technologies will only become more prominent in the years ahead. With further exploration into blockchain capabilities along with a greater emphasis on data safety and reliability requirements, decentralization processes aim to forever change how we handle digital information management.

Decentralized storage is projected to increase in popularity as people and enterprises recognize its offering of a reliable and confidential way to retain information. Upholding this trajectory, the emergence of Web3 and its uses e.g., DeFi, NFTs, etc., strengthen expectations that decentralized storage uptake will continue gaining traction over time.

As decentralized storage solutions become more prevalent, interoperability is emerging as a key area of focus. The development and implementation of new protocols and standards will be integral to the success of this sector. They will allow for users increased ease in transferring their data across disparate systems—unlocking further value for participants involved.

The steps taken today

Decentralized storage systems are quickly advancing in terms of security and privacy, as developers implement cutting-edge technologies such as advanced encryption algorithms, improved consensus mechanisms, and smart contract-based solutions to protect data. With these advancements comes an increased assurance that private information is safe from malicious actors on the network.

As the need for data analytics continues to grow, so does the importance of decentralized storage systems. These secure and private solutions offer a democratized way for individuals and organizations alike to access large amounts of valuable data in order to make better informed decisions. With decentralization comes an unprecedented opportunity to innovate with trusted information at hand.

The decentralized storage market is set to become more competitive and dynamic as time progresses, with ever-increasing offerings from both new entrants and existing players. This heightened competition will result in lowered prices which should make the technology easier to access for individuals and organizations alike while simultaneously providing improved quality services.

In conclusion

To summarize, decentralized storage is a Web3 innovation that is transforming the way data is stored. With this advancement, developers are able to BUIDL world changing DApps with numerous benefits, including cost savings and enhanced privacy.

While there are challenges ahead of us, the potential of this technology gives us a lot to be excited about. We are presented with an opportunity to tap into a secure, reliable, censorship-resistant platform for data storage by leveraging Web3’s decentralization capabilities.

If used correctly and efficiently, this could have an enormous positive impact on the current technological landscape. As this technology continues to mature and evolve, you can expect to see further innovation in Web3 storage solutions in the future.

What are your thoughts on decentralized storage? Just remember that whatever you want to BUIDL on Web3, Chainstack is here to connect you to everything you need—sign up here to get started!