To harness the ever growing capacity and decreasing cost of storage, providing an abstraction of dependable storage in the presence of crash-stop and Byzantine failures is compulsory. We propose a decentralized Reed Solomon coding mechanism with minimum communication overhead. Using a progressive data retrieval scheme, a data collector contacts only the necessary number of storage nodes needed to guarantee data integrity. The scheme gracefully adapts the cost of successful data retrieval to the number of storage node failures. Moreover, by leveraging the Welch-Berlekamp algorithm, it avoids unnecessary computations. Compared to the state-of-the-art decoding scheme, the implementation and evaluation results show that our progressive data retrieval scheme has up to 35 times better computation performance for low Byzantine node rates. Additionally, the communication cost in data retrieval is derived analytically and corroborated by Monte-Carlo simulation results. Our implementation is flexible in that the level of redundancy it provides is independent of the number of data generating nodes, a requirement for distributed storage systems
In the context of network storage for wireless sensor networks, randomized linear codes @cite_0 and fountain codes @cite_16 have been applied with the objective that a data collector can retrieve unit data from each of @math data sources by accessing any @math out @math storage nodes, and thus up to @math crash-stop node failures can be tolerated. However, such schemes cannot recover from data modifications in the field. Compared to erasure based solutions, the key distinctions are i) coding is done at the storage nodes rather than at the data source, and ii) each storage node only has unit capacity. Later, we provide a reference implementation of a single data collector problem using the proposed primitives. Our evaluation studies shows that our implementation outperforms the distributed storage scheme based on random linear network coding in almost all metrics.
@cite_0 @cite_16
1966612101 2164328519
In this correspondence, we consider the problem of constructing an erasure code for storage over a network when the data sources are distributed. Specifically, we assume that there are n storage nodes with limited memory and k < n sources generating the data. We want a data collector, who can appear anywhere in the network, to query any k storage nodes and be able to retrieve the data. We introduce decentralized erasure codes, which are linear codes with a specific randomized structure inspired by network coding on random bipartite graphs. We show that decentralized erasure codes are optimally sparse, and lead to reduced communication, storage and computation cost over random linear coding. It may not be feasible for sensor networks monitoring nature and inaccessible geographical regions to include powered sinks with Internet connections. We consider the scenario where sinks are not present in large-scale sensor networks, and unreliable sensors have to collectively resort to storing sensed data over time on themselves. At a time of convenience, such cached data from a small subset of live sensors may be collected by a centralized (possibly mobile) collector. In this paper, we propose a decentralized algorithm using fountain codes to guarantee the persistence and reliability of cached data on unreliable sensors. With fountain codes, the collector is able to recover all data as long as a sufficient number of sensors are alive. We use random walks to disseminate data from a sensor to a random subset of sensors in the network. Our algorithms take advantage of the low decoding complexity of fountain codes, as well as the scalability of the dissemination process via random walks. We have proposed two algorithms based on random walks. Our theoretical analysis and simulation-based studies have shown that, the first algorithm maintains the same level of fault tolerance as the original centralized fountain code, while introducing lower overhead than naive random-walk based implementation in the dissemination process. Our second algorithm has lower level of fault tolerance than the original centralized fountain code, but consumes much lower dissemination cost.