One of the primary objectives of a distributed storage system is to reliably store large amounts of source data for long durations using a large number @math of unreliable storage nodes, each with @math bits of storage capacity. Storage nodes fail randomly over time and are replaced with nodes of equal capacity initialized to zeroes, and thus bits are erased at some rate @math . To maintain recoverability of the source data, a repairer continually reads data over a network from nodes at a rate @math , and generates and writes data to nodes based on the read data. The distributed storage source data capacity is the maximum amount of source data that can be reliably stored for long periods of time. We prove the distributed storage source data capacity asymptotically approaches @math as @math and @math grow. This equation expresses a fundamental trade-off between network traffic and storage overhead to reliably store source data.
When s @cite_11 , @cite_1 , @cite_10 with small @math and @math are used to repair objects as @math grows, the needed peak repairer read rate with respect to a grows far beyond the lower bound average of . This is because there is a good chance that multiple s occur over a small interval of time, implying that repair for s must occur in a very short interval of time. In contrast, for the repairer described in the proof of , the peak repairer read rate essentially matches the lower bound average .
@cite_10 @cite_1 @cite_11
2507181206 2886748290 2962858508 2963754880
Given any r and n, we present an explicit construction of high-rate maximum distance separable (MDS) array codes that can optimally repair any d failed nodes from any h helper nodes for all h, 1 ≤ h ≤ r and d, k ≤ d ≤ n − h simultaneously. These codes can be constructed over any base field F as long as |F| ≥ sn; where s = lcm(1, 2, …, r). The encoding, decoding, repair of failed nodes, and update procedures of these codes all have low complexity. Our results present a significant improvement over earlier results which can only construct explicit codes for the case of at most 3 parity nodes, and these existing constructions can only optimally repair a single node failure by accessing all the surviving nodes. In the second part of the paper we give an explicit construction of Reed-Solomon codes with asymptotically optimal repair bandwidth. Binary maximum distance separable (MDS) array codes contain @math information columns and @math parity columns in which each entry is a bit that can tolerate @math arbitrary erasures. When a column in an MDS code fails, it has been proven that we must download at least half of the content from each helper column if @math columns are selected as the helper columns. If the lower bound is achieved such that the @math helper columns can be selected from any @math surviving columns, then the repair is an optimal repair. Otherwise, if the lower bound is achieved with @math specific helper columns, the repair is a weak-optimal repair. This paper proposes a class of binary MDS array codes with @math and @math that asymptotically achieve weak-optimal repair of an information column with @math helper columns. We show that there exist many encoding matrices such that the corresponding binary MDS array codes can asymptotically achieve weak-optimal repair for repairing any information column. The repair problem for an (n, k) error-correcting code calls for recovery of an unavailable coordinate of the codeword by downloading as little information as possible from a subset of the remaining coordinates. Using the terminology motivated by coding in distributed storage, we attempt to repair a failed node by accessing information stored on d helper nodes, where k &#x227C; d &#x227C; n &#x2013; 1, and using as little repair bandwidth as possible to recover the lost information.By the so-called cut-set bound (, 2010), the repair bandwidth of an (n,k = n &#x2013; r) MDS code using d helper nodes is at least dl (d + 1 &#x2013; k), where l is the size of the node. A number of constructions of MDS array codes have been shown to meet this bound with equality. In a related but separate line of work, Guruswami and Wootters (2016) studied repair of Reed-Solomon (RS) codes, showing that it is possible to perform repair using a smaller bandwidth than under the trivial approach. At the same time, their work as well as follow-up papers stopped short of constructing RS codes (or any scalar MDS codes) that meet the cut-set bound with equality, which has been an open problem in coding theory.In this work we present a solution to this problem, constructing RS codes of length n over the field of size (ql, l = exp((1 + o(1)n log n) that meet the cut-set bound. We also prove an almost matching lower bound on l, showing that super-exponential scaling is both necessary and sufficient for achieving the cut-set bound using linear repair schemes. More precisely, we prove that for scalar MDS codes (including the RS codes) to meet this bound, the sub-packetization l must satisfy l &#x227D; exp((1 + o(1))k log k). Maximum distance separable (MDS) codes are optimal error-correcting codes in the sense that they provide the maximum failure tolerance for a given number of parity nodes. Suppose that an MDS code with @math information nodes and @math parity nodes is used to encode data in a distributed storage system. It is known that if @math out of the @math nodes are inaccessible and @math surviving (helper) nodes are used to recover the lost data, then we need to download at least @math fraction of the data stored in each of the helper nodes ( , 2010 and , 2013). If this lower bound is achieved for the repair of any @math erased nodes from any @math helper nodes, we say that the MDS code has the @math -optimal repair property. We study high-rate MDS array codes with the optimal repair property (also known as minimum storage regenerating codes, or MSR codes). Explicit constructions of such codes in the literature are only available for the cases where there are at most three parity nodes, and these existing constructions can only optimally repair a single node failure by accessing all the surviving nodes. In this paper, given any @math and @math , we present two explicit constructions of MDS array codes with the @math -optimal repair property for all @math and @math simultaneously. Codes in the first family can be constructed over any base field @math as long as @math , where @math . The encoding, decoding, repair of failed nodes, and update procedures of these codes all have low complexity. Codes in the second family have the optimal access property and can be constructed over any base field @math as long as @math . Moreover, both code families have the optimal error resilience capability when repairing failed nodes. We also construct several other related families of MDS codes with the optimal repair property.