One of the primary objectives of a distributed storage system is to reliably store large amounts of source data for long durations using a large number @math of unreliable storage nodes, each with @math bits of storage capacity. Storage nodes fail randomly over time and are replaced with nodes of equal capacity initialized to zeroes, and thus bits are erased at some rate @math . To maintain recoverability of the source data, a repairer continually reads data over a network from nodes at a rate @math , and generates and writes data to nodes based on the read data. The distributed storage source data capacity is the maximum amount of source data that can be reliably stored for long periods of time. We prove the distributed storage source data capacity asymptotically approaches @math as @math and @math grow. This equation expresses a fundamental trade-off between network traffic and storage overhead to reliably store source data.
When s @cite_11 , @cite_1 , @cite_10 with small @math and @math are used to repair objects as @math grows, the needed peak repairer read rate with respect to a grows far beyond the lower bound average of . This is because there is a good chance that multiple s occur over a small interval of time, implying that repair for s must occur in a very short interval of time. In contrast, for the repairer described in the proof of , the peak repairer read rate essentially matches the lower bound average .
@cite_10 @cite_1 @cite_11
2492240243 2119087823
This paper presents an explicit construction for an @math regenerating code over a field @math operating at the Minimum Storage Regeneration (MSR) point. The MSR code can be constructed to have rate @math as close to @math as desired, sub-packetization given by @math , for @math , field size no larger than @math and where all code symbols can be repaired with the same minimum data download. The construction modifies a prior construction by Sasidharan et. al. which required far larger field-size. A building block appearing in the construction is a scalar MDS code of block length @math . The code has a simple layered structure with coupling across layers, that allows both node repair and data recovery to be carried out by making multiple calls to a decoder for the scalar MDS code. While this work was carried out independently, there is considerable overlap with a prior construction by Ye and Barg. It is shown here that essentially the same architecture can be employed to construct MSR codes using vector binary MDS codes as building blocks in place of scalar MDS codes. The advantage here is that computations can now be carried out over a field of smaller size potentially even over the binary field as we demonstrate in an example. Further, we show how the construction can be extended to handle the case of @math under a mild restriction on the choice of helper nodes. Peer-to-peer distributed storage systems provide reliable access to data through redundancy spread over nodes across the Internet. A key goal is to minimize the amount of bandwidth used to maintain that redundancy. Storing a file using an erasure code, in fragments spread across nodes, promises to require less redundancy and hence less maintenance bandwidth than simple replication to provide the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate a new fragment in a distributed way while transferring as little data as possible across the network. In this paper, we introduce a general technique to analyze storage architectures that combine any form of coding and replication, as well as presenting two new schemes for maintaining redundancy using erasure codes. First, we show how to optimally generate MDS fragments directly from existing fragments in the system. Second, we introduce a new scheme called regenerating codes which use slightly larger fragments than MDS but have lower overall bandwidth use. We also show through simulation that in realistic environments, regenerating codes can reduce maintenance bandwidth use by 25 or more compared with the best previous design - a hybrid of replication and erasure codes - while simplifying system architecture.