The proliferation of sensing and monitoring applications motivates adoption of the event stream model of computation. Though sliding windows are widely used to facilitate effective event stream processing, it is greatly challenged when the event sources are distributed and asynchronous. To address this challenge, we first show that the snapshots of the asynchronous event streams within the sliding window form a convex distributive lattice (denoted by Lat-Win). Then we propose an algorithm to maintain Lat-Win at runtime. The Lat-Win maintenance algorithm is implemented and evaluated on the open-source context-aware middleware we developed. The evaluation results first show the necessity of adopting sliding windows over asynchronous event streams. Then they show the performance of detecting specified predicates within Lat-Win, even when faced with dynamic changes in the computing environment.
The lattice of global snapshots is a key notion in modeling behavior of asynchronous systems @cite_4 @cite_28 @cite_14 @cite_31 @cite_6 @cite_1 @cite_33 @cite_30 @cite_36 @cite_18 , and is widely used in areas such as distributed program debugging @cite_2 @cite_15 and fault-tolerance @cite_22 . One critical challenge is that the lattice of snapshots evolves to exponential size in the worst-case. Various schemes are used to cope with the lattice explosion problem @cite_3 @cite_18 @cite_7 @cite_29 @cite_37 . For example, in @cite_3 @cite_18 , the authors proposed the computation slice to efficiently compute all global states which satisfy a regular predicate @cite_3 . In @cite_8 , the authors discussed that certain (useless) part of the lattice can be removed at runtime to reduce the size of lattice. In this work, we make use of the observation that, in many tracking monitoring applications, it is often prohibitive and, more importantly, unnecessary to process the entire streams. Thus, we use sliding windows over distributed event streams to reduce the size of the lattice.
@cite_30 @cite_18 @cite_14 @cite_4 @cite_33 @cite_22 @cite_15 @cite_28 @cite_36 @cite_7 @cite_29 @cite_1 @cite_37 @cite_6 @cite_3 @cite_8 @cite_2 @cite_31
2095126777 1512015702 2403712541 1498481277 2151102492 2129754261 2158492447 1567346565 2028633127 2400872280 141391985 2886192190 2124317079 2035265484 2013708313 2129344187 2963866791 2264057010
The snapshot object is an important tool for constructing wait-free asynchronous algorithms. We relate the snapshot object to the lattice agreement decision problem. It is shown that any algorithm for solving lattice agreement can be transformed into an implementation of a snapshot object. The overhead cost of this transformation is only a linear number of read and write operations on atomic single-writer multi-reader registers. The transformation uses an unbounded amount of shared memory. We present a deterministic algorithm for lattice agreement that used O(log2 n) operations on 2-processor Test & Set registers, plus O(n) operations on atomic single-writer multi-reader registers. The shared objects are used by the algorithm in a dynamic mode, that is, the identity of the processors that access each of the shared objects is determined dynamically during the execution of the algorithm. By a randomized implementation of 2-processors Test & Set registers from atomic registers, this algorithm implies a randomized algorithm for lattice agreement that uses an expected number of O(n) operations on (dynamic) atomic single-writer multi-reader registers. Combined with our transformation this yields implementations of atomic snapshots with the same complexity. We address an open question, regarding whether a lattice code with lattice decoding (as opposed to maximum-likelihood (ML) decoding) can achieve the additive white Gaussian noise (AWGN) channel capacity. We first demonstrate how minimum mean-square error (MMSE) scaling along with dithering (lattice randomization) techniques can transform the power-constrained AWGN channel into a modulo-lattice additive noise channel, whose effective noise is reduced by a factor of spl radic (1+SNR SNR). For the resulting channel, a uniform input maximizes mutual information, which in the limit of large lattice dimension becomes 1 2 log (1+SNR), i.e., the full capacity of the original power constrained AWGN channel. We then show that capacity may also be achieved using nested lattice codes, the coarse lattice serving for shaping via the modulo-lattice transformation, the fine lattice for channel coding. We show that such pairs exist for any desired nesting ratio, i.e., for any signal-to-noise ratio (SNR). Furthermore, for the modulo-lattice additive noise channel lattice decoding is optimal. Finally, we show that the error exponent of the proposed scheme is lower bounded by the Poltyrev exponent. Many proposals for managing replicated data use sites running the Snapshot Isolation (SI) concurrency control mechanism, and provide 1-copy SI or something similar, as the global isolation level. This allows good scalability, since only ww-conflicts need to be managed globally. However, 1-copy SI can lead to data corruption and violation of integrity constraints [5]. 1-copy serializability is the global correctness condition that prevents data corruption. We propose a new algorithm Replicated Serializable Snapshot Isolation (RSSI) that uses SI at each site, and combines this with a certification algorithm to guarantee 1-copy serializable global execution. Management of ww-conflicts is similar to what is done in 1-copy SI. But unlike previous designs for 1-copy serializable systems, we do not need to prevent all rw-conflicts among concurrent transactions. We formalize this in a theorem that shows that many rw-conflicts are indeed false positives that do not risk non-serializable behavior. Our proposed RSSI algorithm will only abort a transaction when it detects a well-defined pattern of two consecutive rw-edges in the serialization graph. We have built a prototype that integrates our RSSI with the existing open-source PostgresR(SI) system. Our performance evaluation shows that there is a worst-case overhead of about 15 for getting full 1copy serializability as compared to 1-copy SI in a cluster of 8 nodes, with our proposed RSSI clearly outperforming the previous work [6] for update-intensive workloads. In round-by-round models of distributed computing processes run in a sequence of (synchronous or asynchronous) rounds. The advantage of the round-by-round approach is that invariants established in the first round are preserved in later rounds. An elegant asynchronous round-by-round shared memory model, is the iterated snapshots model (IS). Instead of the snapshots model where processes share an array m[ċ] that can be accessed any number of times, indexed by process ID, where Pi writes to m[i] and can take a snapshot of the entire array, we have processes share a two-dimensional array m[ċ, ċ], indexed by iteration number and by process ID, where Pi in iteration r writes once to m[r, i] and takes one snapshot of row r, m[r, ċ]. The IS model lends itself more easily to combinatorial analysis. However, to show that whenever a task is impossible in the IS model the task is impossible in the snapshots model, a simulation is needed. Such a simulation was presented by Borowsky and Gafni in PODC97; namely, it was shown how to take a wait-free protocol for the snapshots model, and transform it into a protocol for the IS model, solving the same task. In this paper we present a new simulation from the snapshots model to the IS model, and show that it can be extended to work with models stronger that wait-free. The main contribution is to show that the simulation can work with models that have access to certain communication objects, called 01-tasks. This extends the result of Gafni, Rajsbaum and Herlihy in DISC'2006 stating that renaming is strictly weaker than set agreement from the IS model to the usual non-iterated wait-free read write shared memory model. We also show that our simulation works with t-resilient models and the more general dependent process failure model of Junqueira and Marzullo. This version of the simulation extends previous results by Herlihy and Rajsbaum in PODC'2010 and DISC'2010 about the topological connectivity of a protocol complex in an iterated dependent process failure model, to the corresponding non-iterated model. Lattice agreement is a key decision problem in distributed systems. In this problem, processes start with input values from a lattice, and must learn (non-trivial) values that form a chain. Unlike consensus, which is impossible in the presence of even a single process failure, lattice agreement has been shown to be decidable in the presence of failures. In this paper, we consider lattice agreement problems in asynchronous, message passing systems. We present an algorithm for the lattice agreement problem that guarantees liveness as long as a majority of the processes are non-faulty. The algorithm has a time complexity of O(N) message delays, where N is the number of processes. We then introduce the generalized lattice agreement problem, where each process receives a (potentially unbounded) sequence of values from an infinite lattice and must learn a sequence of increasing values such that the union of all learnt sequences is a chain and every proposed value is eventually learnt. We present a wait-free algorithm for solving generalized lattice agreement. The algorithm guarantees that every value received by a correct process is learnt in O(N) message delays. We show that this algorithm can be used to implement a class of replicated state machines where (a) commands can be classified as reads and updates, and (b) all update commands commute. This algorithm can be used to realize serializable and linearizable replicated versions of commonly used data types. The percolation phase transition and the mechanism of the emergence of the giant component through the critical scaling window for random graph models, has been a topic of great interest in many dierent communities ranging from statistical physics, combinatorics, computer science, social networks and probability theory. The last few years have witnessed an explosion of models which couple random aggregation rules, that specify how one adds edges to existing congurations, and choice, wherein one selects from a ‘limited’ set of edges at random to use in the conguration. These models exhibit fascinating new phenomenon, ranging from delaying or speeding up the emergence of the giant component, to percolation", where the diameter of the scaling window is several orders of magnitude smaller than that for standard random graph models. While an intense study of such models has ensued, understanding the actual emergence of the giant component and merging dynamics in the critical scaling window has remained impenetrable to a rigorous analysis. In this work we take an important step in the analysis of such models by studying one of the standard examples of such processes, namely the Bohman-Frieze model, and provide rst results on the asymptotic dynamics, through the critical scaling window, that lead to the emergence of the giant component for such models. We identify the scaling window and show that through this window, the component sizes properly rescaled converge to the standard multiplicative coalescent. Proofs hinge on a careful analysis of certain innite-type branching processes with types taking values in the space of RCLL paths, and stochastic analytic techniques to estimate susceptibility functions of the components all the way through the scaling window where these functions explode. Previous approaches for analyzing random graphs at criticality have relied largely on classical breadth-rst search techniques that exploit asymptotic connections with Brownian excursions. For dynamic random graph models evolving via general Markovian rules, such approaches fail and we develop a quite dierent set of tools that can potentially be used for the study of critical dynamics for all bounded size rules. A new viewpoint for adopting the lattice reduction in communication over multiple-input multiple-output (MIMO) broadcast channels is introduced. Lattice basis reduction helps us to reduce the average transmitted energy by modifying the region which includes the constellation points. The new viewpoint helps us to generalize the idea of lattice-reduction-aided (LRA) preceding for the case of unequal-rate transmission, and obtain analytic results for the asymptotic behavior (signal-to-noise ratio (SNR) rarr infin) of the symbol error rate for the LRA precoding and the perturbation technique. Also, the outage probability for both cases of fixed-rate users and fixed sum rate is analyzed. It is shown that the LRA method, using the Lenstra-Lenstra-Lovasz (LLL) algorithm, achieves the optimum asymptotic slope of symbol error rate (called the precoding diversity). In the coming big data era, stress test to IT systems under extreme data volume is crucial to the adoption of computing technologies in every corner of the cyber world. Appropriately generated benchmark datasets provide the possibility for administrators to evaluate the capacity of the systems when real datasets hard obtained have not extreme cases. Traditional benchmark data generators, however, mainly target at producing relation tables of arbitrary size following fixed distributions. The output of such generators are insufficient when it is used to measure the stability of the architecture with extremely dynamic and heavy workloads, caused by complicated hiden factors in the generation mechanism of real world, e.g. dependency between stocks in the trading market and collaborative human behaviors on the social network. In this paper, we present a new framework, called Chronos, to support new demands on streaming data benchmarking, by generating and simulating realistic and fast data streams in an elastic manner. Given a small group of samples with timestamps, Chronos reproduces new data streams with similar characteristics of the samples, preserving column-wise correlations, temporal dependency and order statistics of the snapshot distributions at the same time. To achieve such realistic requirements, we propose 1) a column decomposition optimization technique to partition the original relation table into small sub-tables with minimal correlation information loss, 2) a generative and extensible model based on Latent Dirichlet Allocation to capture temporal dependency while preserving order statistics of the snapshot distribution, and 3) a new generation and assembling method to efficiently build tuples following the expected distribution on the snapshots. To fulfill the vision of elasticity, we also present a new parallel stream data generation mechanism, facilitating distributed nodes to collaboratively generate tuples with minimal synchronization overhead and excellent load balancing. Our extensive experimental studies on real world data domains confirm the efficiency and effectiveness of Chronos on stream benchmark generation and simulation. Random graph models with limited choice have been studied extensively with the goal of understanding the mechanism of the emergence of the giant component. One of the standard models are the Achlioptas random graph processes on a fixed set of (n ) vertices. Here at each step, one chooses two edges uniformly at random and then decides which one to add to the existing configuration according to some criterion. An important class of such rules are the bounded-size rules where for a fixed (K 1 ), all components of size greater than (K ) are treated equally. While a great deal of work has gone into analyzing the subcritical and supercritical regimes, the nature of the critical scaling window, the size and complexity (deviation from trees) of the components in the critical regime and nature of the merging dynamics has not been well understood. In this work we study such questions for general bounded-size rules. Our first main contribution is the construction of an extension of Aldous’s standard multiplicative coalescent process which describes the asymptotic evolution of the vector of sizes and surplus of all components. We show that this process, referred to as the standard augmented multiplicative coalescent (AMC) is ‘nearly’ Feller with a suitable topology on the state space. Our second main result proves the convergence of suitably scaled component size and surplus vector, for any bounded-size rule, to the standard AMC. This result is new even for the classical Erdős–Renyi setting. The key ingredients here are a precise analysis of the asymptotic behavior of various susceptibility functions near criticality and certain bounds from (The barely subcritical regime. Arxiv preprint, 2012) on the size of the largest component in the barely subcritical regime. Lattice codes used under the Compute-and-Forward paradigm suggest an alternative strategy for the standard Gaussian multiple-access channel (MAC): The receiver successively decodes integer linear combinations of the messages until it can invert and recover all messages. In this paper, this strategy is developed and analyzed. For the two-user MAC, it is shown that without time-sharing, the entire capacity region can be attained with a single-user decoder as soon as the signal-to-noise ratios are above @math . A partial analysis is given for more than two users. Lastly the strategy is extended to the so-called dirty MAC where two interfering signals are known non-causally to the two transmitters in a distributed fashion. Our scheme extends the previously known results and gives new achievable rate regions. The Iterated Immediate Snapshot model (IIS), due to its elegant topological representation, has become standard for applying topological reasoning to distributed computing. In this paper, we focus on relations between IIS and the more realistic (non-iterated) read-write Atomic- Snapshot memory model (AS). We grasp equivalences between subsets of runs of AS and IIS (we call them sub-IIS and sub-AS models). To establish an equivalence between a sub-AS model M and a sub-IIS model M', we need two algorithms, a forward simulation F : AS -> IIS and a backward simulation B: IIS -> AS, such that B(F(M)) is a subset of M and F(B(M')) is a subset of M'. AS and IIS are provided with such simulations and, thus, they have the same task computability power. However, the relations between proper sub-AS and sub-IIS models remained unclear until now. In this paper, we present a two-way simulation protocol that provides an equivalent sub-IIS model for any adversarial sub-AS model, i.e., for any sub-AS model specified by the sets of live processes. We achieve the result by ensuring that, under the two-way simulation, the set of live processes in an AS run coincides with the set of fast processes in the simulated IIS run, and vice versa. This paper studies the lattice agreement problem and the generalized lattice agreement problem in distributed message passing systems. In the lattice agreement problem, given input values from a lattice, processes have to non-trivially decide output values that lie on a chain. We consider the lattice agreement problem in both synchronous and asynchronous systems. For synchronous lattice agreement, we present two algorithms which run in @math and @math rounds, respectively, where @math denotes the height of the input sublattice @math , @math is the number of crash failures the system can tolerate, and @math is the number of processes in the system. These algorithms have significant better round complexity than previously known algorithms. The algorithm by attiya1995atomic takes @math synchronous rounds, and the algorithm by Mavronicolasa mavronicolasabound takes @math rounds. For asynchronous lattice agreement, we propose an algorithm which has time complexity of @math message delays which improves on the previously known time complexity of @math message delays. The generalized lattice agreement problem defined by in faleiro2012generalized is a generalization of the lattice agreement problem where it is applied for the replicated state machine. We propose an algorithm which guarantees liveness when a majority of the processes are correct in asynchronous systems. Our algorithm requires @math units of time in the worst case which is better than @math units of time required by the algorithm of faleiro2012generalized . We consider distributed compression of a pair of Gaussian sources in which the goal is to reproduce a linear function of the sources at the decoder. It has recently been noted that lattice codes can provide improved compression rates for this problem compared to conventional, unstructured codes. We show that by including an additional linear binning stage, the state-of-the-art lattice scheme can be improved, in some cases by an arbitrarily large factor. We then describe a lower bound on the optimal sum rate for the case in which the variance of the linear combination exceeds the variance of one of the sources. This lower bound shows that unstructured codes achieve within one bit of the optimal sum rate at any distortion level. We also describe an outer bound on the rate-distortion region that holds in general, which for the special case of communicating the difference of two positively correlated Gaussian sources shows that the unimproved lattice scheme is within one bit of the rate region at any distortion level. We are interested in the clusters formed by a Poisson ensemble of Markovian loops on infinite graphs. This model was introduced and studied in Le Jan (C R Math Acad Sci Paris 350(13–14):643–646, 2012, Ill J Math 57(2):525–558, 2013). It is a model with long range correlations with two parameters ( ) and ( ). The non-negative parameter ( ) measures the amount of loops, and ( ) plays the role of killing on vertices penalizing ( ( >0 )) or favoring ( ( <0 )) appearance of large loops. It was shown in Le Jan (Ill J Math 57(2):525–558, 2013) that for any fixed ( ) and large enough ( ), there exists an infinite cluster in the loop percolation on ( Z ^d ). In the present article, we show a non-trivial phase transition on the integer lattice ( Z ^d ) ( (d 3 )) for ( =0 ). More precisely, we show that there is no loop percolation for ( =0 ) and ( ) small enough. Interestingly, we observe a critical like behavior on the whole sub-critical domain of ( ), namely, for ( =0 ) and any sub-critical value of ( ), the probability of one-arm event decays at most polynomially. For (d 5 ), we prove that there exists a non-trivial threshold for the finiteness of the expected cluster size. For ( ) below this threshold, we calculate, up to a constant factor, the decay of the probability of one-arm event, two point function, and the tail distribution of the cluster size. These rates are comparable with the ones obtained from a single large loop and only depend on the dimension. For (d=3 ) or 4, we give better lower bounds on the decay of the probability of one-arm event, which show importance of small loops for long connections. In addition, we show that the one-arm exponent in dimension 3 depends on the intensity ( ). Recovery from tracking failure is essential in any simultaneous localization and tracking system. In this context, we explore an efficient keyframe-based relocalization method based on frame encoding using randomized ferns. The method enables automatic discovery of keyframes through online harvesting in tracking mode, and fast retrieval of pose candidates in the case when tracking is lost. Frame encoding is achieved by applying simple binary feature tests which are stored in the nodes of an ensemble of randomized ferns. The concatenation of small block codes generated by each fern yields a global compact representation of camera frames. Based on those representations we define the frame dissimilarity as the block-wise hamming distance (BlockHD). Dissimilarities between an incoming query frame and a large set of keyframes can be efficiently evaluated by simply traversing the nodes of the ferns and counting image co-occurrences in corresponding code tables. In tracking mode, those dissimilarities decide whether a frame pose pair is considered as a novel keyframe. For tracking recovery, poses of the most similar keyframes are retrieved and used for reinitialization of the tracking algorithm. The integration of our relocalization method into a hand-held KinectFusion system allows seamless continuation of mapping even when tracking is frequently lost. We present several results regarding the properties of a random vector, uniformly distributed over a lattice cell. This random vector is the quantization noise of a lattice quantizer at high resolution, or the noise of a dithered lattice quantizer at all distortion levels. We find that for the optimal lattice quantizers this noise is wide-sense-stationary and white. Any desirable noise spectra may be realized by an appropriate linear transformation ("shaping") of a lattice quantizer. As the dimension increases, the normalized second moment of the optimal lattice quantizer goes to 1 2 spl pi e, and consequently the quantization noise approaches a white Gaussian process in the divergence sense. In entropy-coded dithered quantization, which can be modeled accurately as passing the source through an additive noise channel, this limit behavior implies that for large lattice dimension both the error and the bit rate approach the error and the information rate of an additive white Gaussian noise (AWGN) channel. In Part II of this paper, also in this issue, we carried out a detailed mean-square-error analysis of the performance of asynchronous adaptation and learning over networks under a fairly general model for asynchronous events including random topologies, random link failures, random data arrival times, and agents turning on and off randomly. In this Part III, we compare the performance of synchronous and asynchronous networks. We also compare the performance of decentralized adaptation against centralized stochastic-gradient (batch) solutions. Two interesting conclusions stand out. First, the results establish that the performance of adaptive networks is largely immune to the effect of asynchronous events: the mean and mean-square convergence rates and the asymptotic bias values are not degraded relative to synchronous or centralized implementations. Only the steady-state mean-square-deviation suffers a degradation in the order of ν, which represents the small step-size parameters used for adaptation. Second, the results show that the adaptive distributed network matches the performance of the centralized solution. These conclusions highlight another critical benefit of cooperation by networked agents: cooperation does not only enhance performance in comparison to stand-alone single-agent processing, but it also endows the network with remarkable resilience to various forms of random failure events and is able to deliver performance that is as powerful as batch solutions. We explore clustering problems in the streaming sliding window model in both general metric spaces and Euclidean space. We present the first polylogarithmic space O(1)-approximation to the metric k-median and metric k-means problems in the sliding window model, answering the main open problem posed by Babcock, Datar, Motwani and O'Callaghan [5], which has remained unanswered for over a decade. Our algorithm uses O(k3log6W) space and poly(k, log W) update time, where W is the window size. This is an exponential improvement on the space required by the technique due to Babcock, et al We introduce a data structure that extends smooth histograms as introduced by Braverman and Ostrovsky [11] to operate on a broader class of functions. In particular, we show that using only polylogarithmic space we can maintain a summary of the current window from which we can construct an O(1)-approximate clustering solution. Merge-and-reduce is a generic method in computational geometry for adapting offline algorithms to the insertion-only streaming model. Several well-known coreset constructions are maintainable in the insertion-only streaming model using this method, including well-known coreset techniques for the k-median and k-means problems in both low-and high-dimensional Euclidean spaces [31, 15]. Previous work [27] has adapted coreset techniques to the insertion-deletion model, but translating them to the sliding window model has remained a challenge. We give the first algorithm that, given an insertion-only streaming coreset of space s (maintained using merge-and-reduce method), maintains this coreset in the sliding window model using O(s2e--2 log W) space. For clustering problems, our results constitute the first significant step towards resolving problem number 20 from the List of Open Problems in Sublinear Algorithms [39].