A New Open Source Initiative For Decentralized, Incentivized Data Storage
Abstract
Orchid is launching an open source decentralized data storage project. We believe that no third party should be in the middle of transactions between a client and a storage provider, and our design reflects this. Because our design is a truly decentralized marketplace, we minimize barriers to participation both for clients and providers while providing protection from bad actors through careful design that rewards honest participation and makes attacks costly. With Orchid’s decentralized data storage solution, clients are able to select a cohort of providers using the Orchid Directory mechanism, erasure code their data with Twin-coding, send data to providers, and perform payments. Through incentive alignment, clients are able to go offline and be assured that their data will be there when they come back. Providers advertise their storage capabilities by locking funds in a staking contract, prove that they are storing data correctly by making bonded commitments and performing cryptographically sound self-audits, and help maintain data availability by participating in repairs.
1 Overview
Entrusting your data to a single company entails the risk that it can be lost or compromised by that organization. Individual companies may disappear or become financially unstable, products go defunct (Google, notoriously, sunsets products all the time), and services that were free when you engaged them may suddenly force you to pay, or introduce intrusive advertising or API changes (viz. Reddit). Many major companies in recent memory have been the target of data breaches, or worse, have been found to be selling your data to affiliates. The alternative proposed in this paper is decentralized, incentivized data storage that removes single points of failure and orchestrates independent Providers to securely store your data.
While some companies offer a degree of safety by storing data in multiple geographic locations, it is still subject to loss if the organization itself is compromised. Moreover, clients are subjected to whatever price these companies decide to charge. Fees for data retrieval or migration can often be expensive, even prohibitively so, causing lock-in. By contrast, decentralized data storage empowers the client to price shop, is configurable in a multitude of ways, and provides an opportunity for smaller storage providers at the network edge to profit from contributing to a robust data storage system without the need for massive data centers. While there are existing products that claim to provide decentralized incentivized data storage, we believe there is significant room for improvement in the space, both in terms of more comprehensive decentralization and more robust incentive design.
With Orchid’s storage design, you distribute your data into a self-monitoring self-healing system. That means you can go offline with the assurance that your data will still be there when you come back. All of this is done in a trustless, incentive-aligned, and configurable way. Achieving this goal requires innovative protocols with rigorous incentives that accommodate Provider failure, reward Providers who participate in repairs, and make your data self-maintaining and self-healing.
1.2 The Case for Decentralized Storage
Digital assets residing on blockchain networks and decentralized Web 3.0 applications (DApps) require significantly more data storage than available on blockchains, such as Ethereum, which have very limited and costly storage capabilities. Decentralized storage networks offer a solution by providing storage for additional data, particularly metadata, while upholding decentralization advantages over centralized alternatives, ensuring resilience against single points of failures, guaranteed persistence, censorship resistance, verifiability, scalability, transferability, and community engagement.
A key application for metadata lies in the domain of non-fungible tokens (NFTs), representing various assets such as art, music, movies, tickets, and real-world assets (RWAs). The crucial metadata linked to these NFT assets, such as the image file required to view the actual art, requires secure storage. The decentralized approach ensures that metadata complements the asset without relying on a trusted third party. Currently, NFT metadata storage is often entrusted to centralized providers like Amazon Web Services (AWS). However, this poses challenges as NFT buyers can’t be assured that metadata will persist. For example, the art associated with an NFT would disappear if an AWS customer were to close the account that stored the metadata the NFT pointed to. Although protocols like IPFS are addressing this problem by providing a universal locator, they don’t guarantee metadata persistence. Decentralized storage networks can empower NFT owners by ensuring the persistent storage of metadata, with options for alterable and unalterable states as well as for proving authenticity or enabling transfer of control over the metadata to the ultimate owner of an NFT.
The potential of decentralized storage networks extends beyond NFTs to diverse Web 3.0 applications like gaming assets, metaverse property, legal documentation for real-world assets, and artificial intelligence (AI). They unlock use cases restrained by centralized solutions, such as secure and scalable infrastructure for open and collaborative AI development or decentralized social networks free from censorship. These networks play a pivotal role in achieving an increasingly decentralized Web 3.0 tech stack by addressing DApp challenges, where reliance on centralized providers like AWS poses accessibility risks. Orchid’s open-source project uniquely aims to cater to these emerging use cases, including bandwidth-intensive scenarios, with a design that minimizes costs.
2 Concepts
- Actors: Client, Provider, Blockchain
- Erasure Coding
- Audits
- Bonded Commitment
- Rate Certificate
The Blockchain neutrally evaluates the contracts, payment terms, cryptographic security, commitment reveals, etc. in order to facilitate, secure, and otherwise finalize payments.
The Client decides what data to store, how to encode it into blocks, how to choose Providers, how to map those blocks onto those Providers, and how to persist and retrieve that data through payments to those Providers.
The Provider decides which Client blocks to accept, how to store those blocks, how much to charge for that service, when to send invoices, when to claim payments, and so on.
2.3 Audits
Audits are a proxy for data retrievability. They come in two flavors: interactive and non-interactive. Interactive audits are performed by the Client, in direct communication with the Provider, in response to an invoice. A successful interactive audit precedes a nanopayment. Non-interactive audits are performed by the Provider, absent the Client, by submitting a bonded commitment to the Blockchain. Correctness of these audits permits the Blockchain to release a payment from Client to Provider.
2.4 Bonded Commitment
A Bonded Commitment is a binding commitment (in the sense of commitment schemes) that is bonded (in the sense of debt security). This is a key component of the mechanism that enables a Provider to “self-audit.”
2.5 Rate Certificate
A Rate Certificate is a credible commitment created and signed by the Client, held by the Provider, which entitles the Provider to payment according to the amount and subject to the terms described in the certificate. The certificate is presented, along with supporting evidence, to the Blockchain whenever a Provider wishes to claim payment.
3 Components
3.1 Provider Selection and Staking
Clients choose Providers from the available pool using the Orchid Directory mechanism. The Orchid Directory is an Ethereum smart contract that grants access to a trustless and permissionless marketplace. Clients use the smart contract to discover Providers via a stake-weighted random selection algorithm. Providers advertise their availability and soundness by locking funds in a staking contract, in turn receiving a proportional amount of new Client demand. Clients can further filter Providers using criteria in the form of curated lists or more advanced selection logic.
3.2 Payment Handling
3.2.1 Nanopayments
Probabilistic nanopayments, as implemented by Orchid, rely on provably fair stochastic interactions that eliminate the vast majority of on-chain data and allow for efficient exchange of arbitrarily small values with one or many parties. Nanopayments pay for bandwidth when data is sent between a Client and Provider or between Providers. For example, nanopayments are sent whenever a Block is first uploaded from Client to Provider, when transferring Blocks between Providers during repairs, or when downloading a Block from Provider to Client to reconstruct a file. For a detailed discussion of nanopayments see our previous work.
3.3 The Auditor Problem, or, Who Audits the Auditors?
Audits are a proxy for data retrievability. A successful audit serves as justification for a Provider to receive payment from the Client. Many decentralized storage projects rely on an actor, which we will generically call an “auditor”, to perform this operation. However, reliance on an auditor adds a level of complexity to the incentive design that these projects largely ignore. For example, a Provider may conspire with or secretly pose as an auditor to receive favorable treatment or sow distrust among its competitors.
3.3.1 Eliminating the auditor
We eliminate the auditor by constructing a trustless on-chain mechanism for Providers to self-audit. The Providers are incentivized to perform these audits in order to collect payment, guaranteeing correctness through a bonded commitment to maintain Client data. With these incentives and assurances, the Client can safely go offline, and the Provider can continue to receive payment in accordance with their Rate Certificate.
3.3.2 Why do self-audits work?
Given that blockchain transactions are expensive, it would be impractical to frequently submit a bonded commitment for each Client. Efficient commitment schemes such as KZG commitments allow Providers to post one commitment for multiple pieces of data. However, this approach has a couple of major problems, including the incentive for dishonest Providers to supply false reports. To overcome this, we impose a bond which acts as a strong disincentive for false reporting.
3.4 File Upload, Download, and Maintenance
The Client selects a cohort of Providers for a given dataset such that each Provider stores a redundantly encoded piece of data, called a Block. The Providers in a given cohort participate in regenerating lost Blocks and ensuring that any new Providers meet the Client’s selection criteria.
3.4.1 Erasure Coding
In a system where anyone who participates in the directory by staking can receive new Clients, some of your Providers will inevitably fail. Orchid’s protocol allows for Provider failure and optimizes the speed and cost of repair. Providers are incentivized to act according to the protocol. Erasure coding is a way for the Client to give each Provider a piece of the data so that if the Client receives pieces from any sufficiently large subset of the Providers, the Client is able to reconstruct the data in full.
Orchid’s design uses Twin-coding to circumvent costly repairs. If a Type 0 Provider fails, then it is repaired by Type 1 Providers, and vice versa, ensuring that only the necessary bits of data need to be sent to repair any missing Block, dramatically reducing bandwidth costs.
Repairs
If a Provider in a cohort stops responding to audits or posts a faulty audit, a repair process is initiated. The cohort is incentivized to cooperate to construct a new Rate Certificate, select a new Provider according to rules defined by the Client, and help the new Provider restore the missing Block. When the Client is online or offline, there are structured protocols for initiating repairs, ensuring timely responses from Providers and maintaining data integrity throughout the process.
4 Comparison To Other Projects
Orchid’s design differs fundamentally from other decentralized storage projects by prioritizing true decentralization and efficiency, using mechanisms like erasure coding instead of replication, and replacing auditors with self-audits.
5 Conclusion
Orchid’s design for a marketplace for decentralized data storage gives control and autonomy to users. It enables Providers to supply storage with very few limitations and empowers Clients to configure nearly all aspects of their data storage and be assured that their data will be there when they come back.