3425479
IRLbot: scaling to 6 billion pages and beyond
This paper shares our experience in designing a web crawler that can download billions of pages using a single-server implementation and models its performance. We show that with the quadratically increasing complexity of verifying URL uniqueness, BFS crawl order, and fixed per-host rate-limiting, current crawling algorithms cannot effectively cope with the sheer volume of URLs generated in large crawls, highly-branching spam, legitimate multi-million-page blog sites, and infinite loops created by server-side scripts. We offer a set of techniques for dealing with these issues and test their performance in an implementation we call IRLbot. In our recent experiment that lasted 41 days, IRLbot running on a single server successfully crawled 6.3 billion valid HTML pages ($7.6$ billion connection requests) and sustained an average download rate of 319 mb/s (1,789 pages/s). Unlike our prior experiments with algorithms proposed in related work, this version of IRLbot did not experience any bottlenecks and successfully handled content from over 117 million hosts, parsed out 394 billion links, and discovered a subset of the web graph with 41 billion unique nodes.
1669034
Modeling Heterogeneous User Churn and Local Resilience of Unstructured P2P Networks
Previous analytical results on the resilience of unstructured P2P systems have not explicitly modeled heterogeneity of user churn (i.e., difference in online behavior) or the impact of in-degree on system resilience. To overcome these limitations, we introduce a generic model of heterogeneous user churn, derive the distribution of the various metrics observed in prior experimental studies (e.g., lifetime distribution of joining users, joint distribution of session time of alive peers, and residual lifetime of a randomly selected user), derive several closed-form results on the transient behavior of in-degree, and eventually obtain the joint in/out degree isolation probability as a simple extension of the out-degree model.
3433910
Residual-Based Measurement of Peer and Link Lifetimes in Gnutella Networks
Existing methods of measuring lifetimes in P2P systems usually rely on the so-called create-based method (CBM), which divides a given observation window into two halves and samples users "created" in the first half every Delta time units until they die or the observation period ends. Despite its frequent use, this approach has no rigorous accuracy or overhead analysis in the literature. To shed more light on its performance, we flrst derive a model for CBM and show that small window size or large Delta may lead to highly inaccurate lifetime distributions. We then show that create-based sampling exhibits an inherent tradeoff between overhead and accuracy, which does not allow any fundamental improvement to the method. Instead, we propose a completely different approach for sampling user dynamics that keeps track of only residual lifetimes of peers and uses a simple renewal-process model to recover the actual lifetimes from the observed residuals. Our analysis indicates that for reasonably large systems, the proposed method can reduce bandwidth consumption by several orders of magnitude compared to prior approaches while simultaneously achieving higher accuracy. We finish the paper by implementing a two-tier Gnutella network crawler equipped with the proposed sampling method and obtain the distribution of ultrapeer lifetimes in a network of 6.4 million users and 60 million links. Our experimental results show that ultrapeer lifetimes are Pareto with shape a alpha ap 1.1; however, link lifetimes exhibit much lighter tails with alpha ap 1.9.
1202188
Load-Balancing Performance of Consistent Hashing: Asymptotic Analysis of Random Node Join
Balancing of structured peer-to-peer graphs, including their zone sizes, has recently become an important topic of distributed hash table (DHT) research. To bring analytical understanding into the various peer-join mechanisms based on consistent hashing, we study how zone-balancing decisions made during the initial sampling of the peer space affect the resulting zone sizes and derive several asymptotic bounds for the maximum and minimum zone sizes that hold with high probability. Several of our results contradict those of prior work and shed new light on the theoretical performance limitations of consistent hashing. We use simulations to verify our models and compare the performance of the various methods using the example of recently proposed de Bruijn DHTs.
1958263
On zone-balancing of peer-to-peer networks: analysis of random node join
Balancing peer-to-peer graphs, including zone-size distributions, has recently become an important topic of peer-to-peer (P2P) research [1], [2], [6], [19], [31], [36]. To bring analytical understanding into the various peer-join mechanisms, we study how zone-balancing decisions made during the initial sampling of the peer space affect the resulting zone sizes and derive several asymptotic results for the maximum and minimum zone sizes that hold with high probability.
322407
IRLbot: Scaling to 6 billion pages and beyond
This article shares our experience in designing a Web crawler that can download billions of pages using a single-server implementation and models its performance. We first show that current crawling algorithms cannot effectively cope with the sheer volume of URLs generated in large crawls, highly branching spam, legitimate multimillion-page blog sites, and infinite loops created by server-side scripts. We then offer a set of techniques for dealing with these issues and test their performance in an implementation we call IRLbot. In our recent experiment that lasted 41 days, IRLbot running on a single server successfully crawled 6.3 billion valid HTML pages (7.6 billion connection requests) and sustained an average download rate of 319 mb/s (1,789 pages/s). Unlike our prior experiments with algorithms proposed in related work, this version of IRLbot did not experience any bottlenecks and successfully handled content from over 117 million hosts, parsed out 394 billion links, and discovered a subset of the Web graph with 41 billion unique nodes.
20162487
An efficient and scalable pairwise key pre-distribution scheme for sensor networks using deployment knowledge
Pairwise key establishment is a fundamental security service for sensor networks. However, establishing pairwise key in sensor networks is a challenging problem, particularly due to the resource constraints on sensor nodes and the threat of node compromises. On the other hand, adding new nodes to a sensor network is a fundamental requirement for their continuous operation over time, too. We analyze the weaknesses of security due to node capture when adding sensor nodes using key pre-distribution schemes with ''fixed'' key pools. In this paper, we propose a new approach, which separates the nodes into groups, the nodes in a group communicate with each other using pairwise keys pre-distributed, the communications between any two neighbor groups are accomplished also through pairwise keys, which is computed based on the pre-distributed Hash chain. We show that the performance (e.g. continuous connectivity, continuous network resilience against node capture and memory usage) of sensor networks can be substantially improved by using our scheme. The scheme and its detailed performance evaluation are presented.
29379375
Boundary layers in channel flow with injection and suction
We present a rigorous result regarding the boundary layer associated with the incompressible Newtonian channel flow with injection and suction.
3479727
p-Percent Coverage Schedule in Wireless Sensor Networks
We investigate the p-percent coverage problem in this paper and propose two algorithms to prolong network lifetime based on the fact that for some applications full coverage is not necessary and different subareas of the monitored area may have different coverage requirements. The first algorithm, CPCA, is a centralized algorithm which selects the least number of nodes to monitor p-percent of the monitored area. The second algorithm, DPCP, is a distributed algorithm which can determine a set of nodes in a distributed manner to cover p-percent of the monitored area. Both of the algorithms guarantee network connectivity. The simulation results show that our algorithms can remarkably prolong network lifetime, have less than 5% unrequired coverage for large networks and employ nodes fairly for most cases.
41225271
On Internal Cardinal Direction Relations
Internal Cardinal Direction (ICD) relations can be considered as the refinement to the contains/within topological relation. It is widely used to describe the position of an object in a region. In this paper, three ICD models with varying degrees of details are presented – ICD-5, ICD-9 and ICD-13. In each of these, the notion of a “middle part” is defined using Minimum Bounding Rectangles (MBR). Then focusing on ICD-9, three representation methods are discussed. They are major portion, point set of intersections and proportions of intersections respectively. The ICD-9 model is validated by a cognitive experiment, which helped to determine the size of the middle part in ICD-9 and validate the MBR-based partition method. Based on a psychological theory about vague predicates, conceptual neighborhood and intersection of ICD relations are also discussed briefly.
37107541
Ant Colony Optimization-Based Location-Aware Routing for Wireless Sensor Networks
The routing for Wireless Sensor Networks (WSNs) is a key and hard problem, and it is a research topic in the field of WSN applications. Based on Ant Colony Optimization (ACO), this paper proposes a novel adaptive intelligent routing scheme for WSNs. Following the proposed scheme, a high performance routing algorithm for WSNs is designed. The proposed routing scheme is very different from the existing ACO based routing schema for WSNs. On one hand, in the proposed scheme, the search range for an ant to select its next-hop node is limited to a subset of the set of the neighbors of the current node. On the other hand, by fusing the residual energy and the global and local location information of nodes, the new probability transition rules for an ant to select its next-hop node are defined. Compared with other ACO based routing algorithms for WSNs, the proposed routing algorithm has a better network performance on aspects of energy consumption, energy efficiency, and packet delivery latency.