322407
IRLbot: Scaling to 6 billion pages and beyond
This article shares our experience in designing a Web crawler that can download billions of pages using a single-server implementation and models its performance. We first show that current crawling algorithms cannot effectively cope with the sheer volume of URLs generated in large crawls, highly branching spam, legitimate multimillion-page blog sites, and infinite loops created by server-side scripts. We then offer a set of techniques for dealing with these issues and test their performance in an implementation we call IRLbot. In our recent experiment that lasted 41 days, IRLbot running on a single server successfully crawled 6.3 billion valid HTML pages (7.6 billion connection requests) and sustained an average download rate of 319 mb/s (1,789 pages/s). Unlike our prior experiments with algorithms proposed in related work, this version of IRLbot did not experience any bottlenecks and successfully handled content from over 117 million hosts, parsed out 394 billion links, and discovered a subset of the Web graph with 41 billion unique nodes.
8744802
On static and dynamic partitioning behavior of large-scale networks
In this paper, we analyze the problem of network disconnection in the context of large-scale P2P networks and understand how both static and dynamic patterns of node failure affect the resilience of such graphs. We start by applying classical results from random graph theory to show that a large variety of deterministic and random P2P graphs almost surely (i.e., with probability 1-o(1)) remain connected under random failure if and only if they have no isolated nodes. This simple, yet powerful, result subsequently allows us to derive in closed-form the probability that a P2P network develops isolated nodes, and therefore partitions, under both types of node failure. We finish the paper by demonstrating that our models match simulations very well and that dynamic P2P systems are extremely resilient under node churn as long as the neighbor replacement delay is much smaller than the average user lifetime
47232248
Graph-theoretic analysis of structured peer-to-peer systems: routing distances and fault resilience
This paper examines graph-theoretic properties of existing peer-to-peer networks and proposes a new infrastructure based on optimal-diameter de Bruijn graphs. Since generalized de Bruijn graphs exhibit very short average distances and high resilience to node failure, they are well suited for distributed hash tables (DHTs). Using the example of Chord, CAN, and de Bruijn, we study the routing performance, graph expansion, clustering properties, and bisection width of each graph. Having confirmed that de Bruijn graphs offer the best diameter and highest connectivity among the existing peer-to-peer structures, we offer a very simple incremental building process that preserves optimal properties of de Bruijn graphs under uniform user joins/departures. We call the combined peer-to-peer architecture optimal diameter routing infrastructure.
3413304
On Node Isolation Under Churn in Unstructured P2P Networks with Heavy-Tailed Lifetimes
Previous analytical studies [12], [18] of unstructured P2P resilience have assumed exponential user lifetimes and only considered age-independent neighbor replacement. In this paper, we overcome these limitations by introducing a general node-isolation model for heavy-tailed user lifetimes and arbitrary neighbor-selection algorithms. Using this model, we analyze two age-biased neighbor-selection strategies and show that they significantly improve the residual lifetimes of chosen users, which dramatically reduces the probability of user isolation and graph partitioning compared to uniform selection of neighbors. In fact, the second strategy based on random walks on age-weighted graphs demonstrates that for lifetimes with infinite variance, the system monotonically increases its resilience as its age and size grow. Specifically, we show that the probability of isolation converges to zero as these two metrics tend to infinity. We finish the paper with simulations in finite-size graphs that demonstrate the effect of this result in practice.
1202188
Load-Balancing Performance of Consistent Hashing: Asymptotic Analysis of Random Node Join
Balancing of structured peer-to-peer graphs, including their zone sizes, has recently become an important topic of distributed hash table (DHT) research. To bring analytical understanding into the various peer-join mechanisms based on consistent hashing, we study how zone-balancing decisions made during the initial sampling of the peer space affect the resulting zone sizes and derive several asymptotic bounds for the maximum and minimum zone sizes that hold with high probability. Several of our results contradict those of prior work and shed new light on the theoretical performance limitations of consistent hashing. We use simulations to verify our models and compare the performance of the various methods using the example of recently proposed de Bruijn DHTs.
122609601
Attractor dimension estimates for two-dimensional shear flows
Abstract We study the large time behavior of boundary and pressure-gradient driven incompressible fluid flows in elongated two-dimensional channels with emphasis on estimates for their degrees of freedom, i.e., the dimension of the attractor for the solutions of the Navier-Stokes equations. For boundary driven shear flows and flux driven channel flows we present upper bounds for the degrees of freedom of the form cα Re 3 2 where c is a universal constant, α denotes the aspect ratio of the channel (length/width), and Re is the Reynolds number based on the channel width and the imposed “outer” velocity scale. For fixed pressure gradient driven channel flows we obtain an upper bound of form c'α Re2, where c' is another universal positive constant and the Reynolds number is based on a velocity defined by the infimum, over all possible trajectories, of the time averaged mass flux per unit channel width. We discuss these results in terms of physical arguments based on small length scales in turbulent flows.
14457115
General Method to Build Deterministic Small-World Networks Based on Cayley Graph
Many real-life networks, such as the WWW, biological realms, and social networks are small-world networks. Most previous researches on small-world networks have been based on probabilistic methods. In this paper, we present a general method by using Cayley graph to build deterministic small-world networks. We also analyze the cluster topology of the Cayley graph with small world properties by constructing its left Coset graph. This method has great value in constructing interconnection networks model and analyzing many other real network models.
3479727
p-Percent Coverage Schedule in Wireless Sensor Networks
We investigate the p-percent coverage problem in this paper and propose two algorithms to prolong network lifetime based on the fact that for some applications full coverage is not necessary and different subareas of the monitored area may have different coverage requirements. The first algorithm, CPCA, is a centralized algorithm which selects the least number of nodes to monitor p-percent of the monitored area. The second algorithm, DPCP, is a distributed algorithm which can determine a set of nodes in a distributed manner to cover p-percent of the monitored area. Both of the algorithms guarantee network connectivity. The simulation results show that our algorithms can remarkably prolong network lifetime, have less than 5% unrequired coverage for large networks and employ nodes fairly for most cases.
37107541
Ant Colony Optimization-Based Location-Aware Routing for Wireless Sensor Networks
The routing for Wireless Sensor Networks (WSNs) is a key and hard problem, and it is a research topic in the field of WSN applications. Based on Ant Colony Optimization (ACO), this paper proposes a novel adaptive intelligent routing scheme for WSNs. Following the proposed scheme, a high performance routing algorithm for WSNs is designed. The proposed routing scheme is very different from the existing ACO based routing schema for WSNs. On one hand, in the proposed scheme, the search range for an ant to select its next-hop node is limited to a subset of the set of the neighbors of the current node. On the other hand, by fusing the residual energy and the global and local location information of nodes, the new probability transition rules for an ant to select its next-hop node are defined. Compared with other ACO based routing algorithms for WSNs, the proposed routing algorithm has a better network performance on aspects of energy consumption, energy efficiency, and packet delivery latency.
205494087
A uniformly dissipative scheme for stationary statistical properties of the infinite Prandtl number model
The purpose of this short communication is to announce that a class of numerical schemes, uniformly dissipative approximations, which uniformly preserve the dissipativity of the continuous infinite dimensional dissipative complex (chaotic) systems possess desirable properties in terms of approximating stationary statistics properties. In particular, the stationary statistical properties of these uniformly dissipative schemes converge to those of the continuous system at vanishing mesh size. The idea is illustrated on the infinite Prandtl number model for convection and semi-discretization in time, although the general strategy works for a broad class of dissipative complex systems and fully discretized approximations. As far as we know, this is the first result on rigorous validation of numerical schemes for approximating stationary statistical properties of general infinite dimensional dissipative complex systems.