The degree distribution is one of the most fundamental properties used in the analysis of massive graphs. There is a large literature on graph sampling, where the goal is to estimate properties (especially the degree distribution) of a large graph through a small, random sample. The degree distribution estimation poses a significant challenge, due to its heavy-tailed nature and the large variance in degrees. We design a new algorithm, SADDLES, for this problem, using recent mathematical techniques from the field of sublinear algorithms. The SADDLES algorithm gives provably accurate outputs for all values of the degree distribution. For the analysis, we define two fatness measures of the degree distribution, called the h-index and the z-index. We prove that SADDLES is sublinear in the graph size when these indices are large. A corollary of this result is a provably sublinear algorithm for any degree distribution bounded below by a power law. We deploy our new algorithm on a variety of real datasets and demonstrate its excellent empirical behavior. In all instances, we get extremely accurate approximations for all values in the degree distribution by observing at most 1 of the vertices. This is a major improvement over the state-of-the-art sampling algorithms, which typically sample more than 10 of the vertices to give comparable results. We also observe that the h and z-indices of real graphs are large, validating our theoretical analysis.
The subfield of within theoretical computer science can be thought of as a formalization of graph sampling to estimate properties. Indeed, our description of the main problem follows this language. There is a very rich body of mathematical work in this area (refer to Ron's survey @cite_9 ). Practical applications of graph property testing are quite rare, and we are only aware of one previous work on applications for finding dense cores in router networks @cite_3 . The specific problem of estimating the average degree (or the total number of edges) was studied by Feige @cite_21 and Goldreich-Ron @cite_28 . and focus on the problem of estimating higher moments of the degree distribution @cite_1 @cite_2 . One of the main techniques we use of simulating edge queries was developed in sublinear algorithms results of @cite_17 @cite_15 in the context of triangle counting and degree moment estimation. We stress that all these results are purely theoretical, and their practicality is by no means obvious.
@cite_28 @cite_9 @cite_21 @cite_1 @cite_3 @cite_2 @cite_15 @cite_17
2590395400 2266714125 2124450885 2062833795
We revisit the classic problem of estimating the degree distribution moments of an undirected graph. Consider an undirected graph @math with @math vertices, and define (for @math ) @math . Our aim is to estimate @math within a multiplicative error of @math (for a given approximation parameter @math ) in sublinear time. We consider the sparse graph model that allows access to: uniform random vertices, queries for the degree of any vertex, and queries for a neighbor of any vertex. For the case of @math (the average degree), @math queries suffice for any constant @math (Feige, SICOMP 06 and Goldreich-Ron, RSA 08). Gonen-Ron-Shavitt (SIDMA 11) extended this result to all integral @math , by designing an algorithms that performs @math queries. We design a new, significantly simpler algorithm for this problem. In the worst-case, it exactly matches the bounds of Gonen-Ron-Shavitt, and has a much simpler proof. More importantly, the running time of this algorithm is connected to the degeneracy of @math . This is (essentially) the maximum density of an induced subgraph. For the family of graphs with degeneracy at most @math , it has a query complexity of @math . Thus, for the class of bounded degeneracy graphs (which includes all minor closed families and preferential attachment graphs), we can estimate the average degree in @math queries, and can estimate the variance of the degree distribution in @math queries. This is a major improvement over the previous worst-case bounds. Our key insight is in designing an estimator for @math that has low variance when @math does not have large dense subgraphs. Numerous graph mining applications rely on detecting subgraphs which are large near-cliques. Since formulations that are geared towards finding large near-cliques are hard and frequently inapproximable due to connections with the Maximum Clique problem, the poly-time solvable densest subgraph problem which maximizes the average degree over all possible subgraphs "lies at the core of large scale data mining" [10]. However, frequently the densest subgraph problem fails in detecting large near-cliques in networks. In this work, we introduce the k-clique densest subgraph problem, k ≥ 2. This generalizes the well studied densest subgraph problem which is obtained as a special case for k=2. For k=3 we obtain a novel formulation which we refer to as the triangle densest subgraph problem: given a graph G(V,E), find a subset of vertices S* such that τ(S*)=max limitsS ⊆ V t(S) |S|, where t(S) is the number of triangles induced by the set S. On the theory side, we prove that for any k constant, there exist an exact polynomial time algorithm for the k-clique densest subgraph problem . Furthermore, we propose an efficient 1 k-approximation algorithm which generalizes the greedy peeling algorithm of Asahiro and Charikar [8,18] for k=2. Finally, we show how to implement efficiently this peeling framework on MapReduce for any k ≥ 3, generalizing the work of Bahmani, Kumar and Vassilvitskii for the case k=2 [10]. On the empirical side, our two main findings are that (i) the triangle densest subgraph is consistently closer to being a large near-clique compared to the densest subgraph and (ii) the peeling approximation algorithms for both k=2 and k=3 achieve on real-world networks approximation ratios closer to 1 rather than the pessimistic 1 k guarantee. An interesting consequence of our work is that triangle counting, a well-studied computational problem in the context of social network analysis can be used to detect large near-cliques. Finally, we evaluate our proposed method on a popular graph mining application. Sampling is a standard approach in big-graph analytics; the goal is to efficiently estimate the graph properties by consulting a sample of the whole population. A perfect sample is assumed to mirror every property of the whole population. Unfortunately, such a perfect sample is hard to collect in complex populations such as graphs (e.g. web graphs, social networks), where an underlying network connects the units of the population. Therefore, a good sample will be representative in the sense that graph properties of interest can be estimated with a known degree of accuracy. While previous work focused particularly on sampling schemes to estimate certain graph properties (e.g. triangle count), much less is known for the case when we need to estimate various graph properties with the same sampling scheme. In this paper, we pro- pose a generic stream sampling framework for big-graph analytics, called Graph Sample and Hold (gSH), which samples from massive graphs sequentially in a single pass, one edge at a time, while maintaining a small state in memory. We use a Horvitz-Thompson construction in conjunction with a scheme that samples arriving edges without adjacencies to previously sampled edges with probability p and holds edges with adjacencies with probability q. Our sample and hold framework facilitates the accurate estimation of subgraph patterns by enabling the dependence of the sampling process to vary based on previous history. Within our framework, we show how to produce statistically unbiased estimators for various graph properties from the sample. Given that the graph analytics will run on a sample instead of the whole population, the runtime complexity is kept under control. Moreover, given that the estimators are unbiased, the approximation error is also kept under control. Finally, we test the performance of the proposed framework (gSH) on various types of graphs, showing that from a sample with -- 40K edges, it produces estimates with relative errors Sampling from large graphs is an area of great interest, especially since the emergence of huge structures such as Online Social Networks (OSNs) and the World Wide Web (WWW). These networks, when viewed as graphs, often contain hundreds of millions of vertices and billions of edges. The large scale properties of a network can be summarized in terms of parameters of the underlying graph, such as the total number of vertices, edges and triangles. The large size of these networks makes it computationally expensive to obtain such structural properties of the underlying graph by exhaustive search. If we can estimate these properties by taking small but representative samples from the network, then size is no longer such a problem. In this paper we present a general framework to estimate network properties using random walks. These methods work under the assumption we are able to obtain local characteristics of a vertex during each step of the random walk, for example the number and labels of the neighboring vertices of a specific vertex These assumptions are relatively reasonable in practice, but may add some additional query cost to each step of the random walk. We also present some practical methods to estimate the total number of edges links m, number of vertices nodes n and number of connected triads of vertices (triangles) t in graphs with degree distributions which follow a power-law and higher number of triangles higher than expected in random graphs. We use these graphs since they tend to better correspond to the structure of large online networks, and in fact some of the data used are taken from such a network. Additionally we present experimental estimates for n, m, t we obtained using our methods on real or manufactured networks. In order to make the methods practical, the total number of steps made by the walk was limited to at most the size n of the network. In fact the results appear to converge for a lower number of steps, indicating that our proposed methods are feasible in practice.