Filecoin: A Cryptocurrency Operated File Storage Network

1e96a1b27a6cb85df68d728cf3695b0c46dbd44d lecoin.io

July 15, 2014

Abstract

Filecoin is a distributed electronic currency similar to Bitcoin. Unlike Bitcoin’s computation-only proof-of-work, Filecoin’s proof-of-work function includes a proof-of-retrievability component, which requires nodes to prove they store a particular file. The Filecoin network forms an entirely distributed file storage system, whose nodes are incentivized to store as much of the entire network’s data as they can. The currency is awarded for storing files, and is transferred in transactions, as in Bitcoin. Files are added to the network by spending currency. This produces strong monetary incentives for individuals to join and work for the network. In the course of ordinary operation of the Filecoin network, nodes contribute useful work in the form of storage and distribution of valuable data.

1 Introduction

Many computer systems store and access data via commercial service providers. At present, a handful of large providers serve most of these markets. New market entrants are rare, as direct competition at full scale with incumbent providers is virtually impossible. This makes it hard to optimize certain inefficiencies. For example, the data transfer bottleneck today is mostly last-mile ISPs; both the internet backbone and local area networks are orders of magnitude faster. Distributed services, in which agents have individual incentives to store data and optimize local distribution, could provide vastly better solutions. In addition, one major goal of distributed storage systems is to ensure the preservation of important files. In this respect, current storage systems are brittle. First, they tend to be centrally managed by one service provider, linking the fate of the files stored to the fate of that organization. Second, in widely-used file retrieval schemes such as HTTP, a file is identified by its location rather than by its content. This makes file availability depend upon the uptime of specific hosts, rather than the existence of the file anywhere in the network. Third, widely-used schemes place the perpetual burden of serving a file on the original creator—either by hosting the file herself or hiring a provider to do so—which is often unsustainable, particularly for large files such as scientific data sets. What is needed is a globally distributed network whose individual agents can serve any file requests and are strongly incentivized to do so. Blockchain-based cryptocurrencies, a recent development, can organize and incentivize large networks of machines to perform computations as proofs-of-work. In Bitcoin, these computations tend to be useless. Others have attempted to organize more useful algorithmic work; for example, Primecoin rewards finding chains of prime numbers. However, these systems could also perform other kinds of useful work in clear immediate demand, such as storing and distributing files. In this paper, we propose a solution that incentivizes file storage using a Bitcoin-like cryptocurrency network, whose work function incorporates proofs-of-retrievability.

2 Design

Like Bitcoin, Filecoin implements a transaction ledger via a blockchain, in which each block must be accompanied by a proof-of-work based on a cryptographic hash function. The proof-of-work parameter is dynamically adjusted so that one block occurs roughly every ten minutes, as in Bitcoin. Further, as in most blockchain-based constructions, clients should only consider transactions to be committed after a sequence of several valid blocks, where the number of blocks is determined by the assumed restrictions on the power of the adversary (e.g., that no adversary can control a majority of the network’s computation power). However, Filecoin makes a number of central changes to the standard Bitcoin-style design.

2.1 The Piece Set

The first key component of Filecoin is the addition of a growing sequence of data pieces, which form the files stored by the network. A piece is an opaque segment of data of fixed size S, a tunable network parameter. Each piece introduced to the Filecoin blockchain appears in some block Bt, via a special Put transaction. The piece set P is ordered chronologically. As in Bitcoin, all nodes must keep the entire blockchain in local storage, but pieces are distributed among all Filecoin nodes. This scheme allows the Filecoin blockchain to provide storage of data orders of magnitude larger than the blockchain itself.

2.2 Put and Get Transactions

The second key addition in Filecoin is the specification of Put and Get transactions. Put transactions add files to the network’s storage: each Put transaction includes a list of piece records introducing new pieces. A piece record is a value of the form:

record_i = (H(p_i); H(i); pki; rewardi)

where H(pi) is the cryptographic hash of a piece, i is a sequence of authenticators for pi, pki is a verification key, and rewardi are reward parameters. When a Put transaction is finalized—mined as part of a block—all pieces identified by the piece records are deemed stored by the network. A Get transaction is used to surface a specific piece previously stored on demand. When a Get transaction is finalized, the pieces listed are transmitted to the transaction issuer. These two transactions extend the Bitcoin protocol with a standard way to store and retrieve files, which are made up of concatenated pieces.

2.3 Incentivizing Piece Storage via Challenges

Get transactions are not enough to incentivize miners to store all pieces, as miners cannot predict the distribution of future Get transactions and their expected reward. Filecoin’s most important departure from Bitcoin-style cryptocurrencies is a modification of the work function. In order to succeed in mining a block, on top of the usual hash-based proof-of-work, miners also must prove that they currently store a particular set of "challenge" pieces. Each piece introduces overhead in the blockchain, which every node must store.

For example, consider a blockchain storing records of pieces ending with block Bt-1. In order to add block Bt to the blockchain, a miner must prove that they are storing a particular sequence of pieces, as determined by the partially-mined block Bt-0 (which includes a Bitcoin-style proof-of-work). Specifically, the starting piece index it is determined as follows:

i_t = H(Bt_0) mod |P|

This index points to a sequence of pieces pi in P, for i ranging from it through (it+k-1), where k is a tunable difficulty parameter. This index choice samples pieces uniformly. The size of the challenge k gives the flexibility of making challenges harder or easier to match node capacity.

2.4 Dispersing Pieces in the Bundle

As discussed so far, Filecoin nodes have an incentive to hoard pieces, to retain the rarest for themselves and thereby increase their expected block-minting reward. Filecoin provides a piece dispersion mechanism to prevent monopoly or hoarding in general. When mining a new block, nodes must also present a set of pieces. This set is called the bundle and includes all the pieces pi and corresponding i referenced in the new blockchain head.

2.5 Verification via Compact Proofs-of-Retrievability

The Filecoin blockchain must be verifiable by any node in the network, including new nodes who have no prior knowledge. Filecoin proofs PieceProof(pi;t) have two components: H(Bt-0jjpi) and i. The first component, the hash value H(Bt-0jjpi), allows any node, given the original piece pi, to verify that the miner of block Bt was in possession of pi at the time of mining the block Bt. The second component, i, is a compact proof-of-retrievability.

2.6 Block Construction and Verification Procedures

A new Filecoin block Bt is mined by preparing the new transactions it will include, constructing the block, and preparing its corresponding bundle. Block Bt contains various elements including parent, txRoot, nonce, PoWt, and PoRt.

2.7 Reward Parameters and Secondary Markets

Filecoin allows Put transactions to specify piece rewards. This construction enables users to spend Filecoin currency to strengthen the incentives of storing particular pieces. Reward parameters rewardi for each piece must be specified in the value record i in the corresponding Put transaction. The reward function design space is ample, so we describe only a straightforward construction and leave others for future development.

2.8 Consensus Mechanisms

In Filecoin, block mining represents useful work: previously stored files have been proven to remain in the network, new files have been added, and new transactions have been issued. In the version of Filecoin described in this paper, the useful storage service is layered on top of Bitcoin consensus: after a proof-of-work is solved, the miner must then also provide a corresponding proof-of-retrievability.

3 Conclusion

We present a new cryptocurrency and file storage network called Filecoin. Filecoin enables outsourcing of data storage to a fluid distributed network of service providers. Individual providers are incentivized to allocate their storage resources to cover all requested data pieces, since any such piece may be the subject of a profitable future block-mining challenge. Parties may opt to join or leave the network at will, without compromising the robustness of the system. Tunable parameters trade off between churn and resiliency, replication factors, and consensus strategies.