To harness the ever growing capacity and decreasing cost of storage, providing an abstraction of dependable storage in the presence of crash-stop and Byzantine failures is compulsory. We propose a decentralized Reed Solomon coding mechanism with minimum communication overhead. Using a progressive data retrieval scheme, a data collector contacts only the necessary number of storage nodes needed to guarantee data integrity. The scheme gracefully adapts the cost of successful data retrieval to the number of storage node failures. Moreover, by leveraging the Welch-Berlekamp algorithm, it avoids unnecessary computations. Compared to the state-of-the-art decoding scheme, the implementation and evaluation results show that our progressive data retrieval scheme has up to 35 times better computation performance for low Byzantine node rates. Additionally, the communication cost in data retrieval is derived analytically and corroborated by Monte-Carlo simulation results. Our implementation is flexible in that the level of redundancy it provides is independent of the number of data generating nodes, a requirement for distributed storage systems
In the context of network storage for wireless sensor networks, randomized linear codes @cite_0 and fountain codes @cite_16 have been applied with the objective that a data collector can retrieve unit data from each of @math data sources by accessing any @math out @math storage nodes, and thus up to @math crash-stop node failures can be tolerated. However, such schemes cannot recover from data modifications in the field. Compared to erasure based solutions, the key distinctions are i) coding is done at the storage nodes rather than at the data source, and ii) each storage node only has unit capacity. Later, we provide a reference implementation of a single data collector problem using the proposed primitives. Our evaluation studies shows that our implementation outperforms the distributed storage scheme based on random linear network coding in almost all metrics.
@cite_0 @cite_16
1998021782 2105185344 2951800112 2102047288
We consider a distributed storage system employing some existing regenerate codes where the storage nodes are scattered in a wireless network. The data collector connects to the storage nodes via orthogonal channels and downloads data symbols from these nodes. In the existing data reconstruction schemes for distributed storage systems, the data collector downloads all symbols from a subset of the storage nodes. Such a full-downloading approach becomes inefficient in wireless networks since due to fading, the wireless channels may not offer sufficient bandwidths for full downloading. Moreover, full-downloading is also less power efficient than partial downloading. In this paper, given a coding scheme employed by the wireless distributed storage system, we propose a partial downloading scheme that allows downloading a portion of the symbols from any storage node. We formulate a cross-layer wireless resource allocation problem for data reconstruction in distributed storage systems employing such partial downloading. To derive the fundamental properties of partial downloading as well as to reduce the complexity of wireless resource allocation, we derive necessary and sufficient conditions for data reconstructability for partial downloading, in terms of the numbers of downloaded symbols from the storage nodes. We also propose channel and power allocation schemes for partial downloading in wireless distributed storage systems. Simulation results are provided to demonstrate the significant power savings by the proposed partial downloading scheme compared to the full-downloading methods for distributed storage. Distributed storage systems provide reliable access to data through redundancy spread over individually unreliable nodes. Application scenarios include data centers, peer-to-peer storage systems, and storage in wireless networks. Storing data using an erasure code, in fragments spread across nodes, requires less redundancy than simple replication for the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate encoded fragments in a distributed way while transferring as little data as possible across the network. For an erasure coded system, a common practice to repair from a single node failure is for a new node to reconstruct the whole encoded data object to generate just one encoded block. We show that this procedure is sub-optimal. We introduce the notion of regenerating codes, which allow a new node to communicate functions of the stored data from the surviving nodes. We show that regenerating codes can significantly reduce the repair bandwidth. Further, we show that there is a fundamental tradeoff between storage and repair bandwidth which we theoretically characterize using flow arguments on an appropriately constructed graph. By invoking constructive results in network coding, we introduce regenerating codes that can achieve any point in this optimal tradeoff. Distributed storage systems provide reliable access to data through redundancy spread over individually unreliable nodes. Application scenarios include data centers, peer-to-peer storage systems, and storage in wireless networks. Storing data using an erasure code, in fragments spread across nodes, requires less redundancy than simple replication for the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate encoded fragments in a distributed way while transferring as little data as possible across the network. For an erasure coded system, a common practice to repair from a node failure is for a new node to download subsets of data stored at a number of surviving nodes, reconstruct a lost coded block using the downloaded data, and store it at the new node. We show that this procedure is sub-optimal. We introduce the notion of regenerating codes, which allow a new node to download of the stored data from the surviving nodes. We show that regenerating codes can significantly reduce the repair bandwidth. Further, we show that there is a fundamental tradeoff between storage and repair bandwidth which we theoretically characterize using flow arguments on an appropriately constructed graph. By invoking constructive results in network coding, we introduce regenerating codes that can achieve any point in this optimal tradeoff. We consider exact repair of failed nodes in maximum distance separable (MDS) code based distributed storage systems. It is well known that an (n, k) MDS code can tolerate failure (erasure) of up to n − k storage disks, when the code is used to store k information elements over n distributed storage disks. The focus of this paper is optimal recovery, in terms of repair bandwidth - the amount of data to be downloaded to repair a failed node - for a single failed node. When a single node fails, it has been previously shown by Dimakis et. al. that the amount of repair bandwidth is at least equation units, when each storage disk stores ℒ units of data. The achievability of this lower bound of equation units, for arbitrary values of (n, k); has been shown previously using asymptotic code constructions based on asymptotic interference alignment. However, the existence of finite codes satisfying this lower bound has been shown only for specific regimes of (n, k) and their existence for arbitrary values of (n, k) remained open. In this paper, we provide the first known construction of a finite code for arbitrary (n, k), which can repair a single failed systematic node by downloading exactly equation units of data. The code that we construct is based on permutation matrices and hence termed the Permutation Code.