Twitter is a social media giant famous for the exchange of short, 140-character messages called "tweets". In the scientific community, the microblogging site is known for openness in sharing its data. It provides a glance into its millions of users and billions of tweets through a "Streaming API" which provides a sample of all tweets matching some parameters preset by the API user. The API service has been used by many researchers, companies, and governmental institutions that want to extract knowledge in accordance with a diverse array of questions pertaining to social media. The essential drawback of the Twitter API is the lack of documentation concerning what and how much data users get. This leads researchers to question whether the sampled data is a valid representation of the overall activity on Twitter. In this work we embark on answering this question by comparing data collected using Twitter's sampled API service with data collected using the full, albeit costly, Firehose stream that includes every single published tweet. We compare both datasets using common statistical metrics as well as metrics that allow us to compare topics, networks, and locations of tweets. The results of our work will help researchers and practitioners understand the implications of using the Streaming API.
From a statistical point of view, the law of large numbers'' (mean of a sample converges to the mean of the entire ) and the Glivenko-Cantelli theorem (the unknown distribution @math of an attribute in a population can be approximated with the observed distribution @math ) guarantee satisfactory results from sampled data when the randomly selected sub-sample is big enough. From network algorithmic @cite_25 perspective the question is more complicated. Previous efforts have delved into the topic of network sampling and how working with a restricted set of data can affect common network measures. The problem was studied earlier in @cite_5 , where the author proposes an algorithm to sample networks in a way that allows one to estimate basic network properties. More recently, @cite_32 and @cite_27 have studied the affect of data error on common network centrality measures by randomly deleting and adding nodes and edges. The authors discover that centrality measures are usually most resilient on dense networks. @cite_37 , the authors study global properties of simulated random graphs to better understand data error in social networks. @cite_35 proposes a strategy for sampling large graphs to preserve network measures.
@cite_35 @cite_37 @cite_32 @cite_27 @cite_5 @cite_25
2062833795 2028897509 180417844 2146008005
Sampling from large graphs is an area of great interest, especially since the emergence of huge structures such as Online Social Networks (OSNs) and the World Wide Web (WWW). These networks, when viewed as graphs, often contain hundreds of millions of vertices and billions of edges. The large scale properties of a network can be summarized in terms of parameters of the underlying graph, such as the total number of vertices, edges and triangles. The large size of these networks makes it computationally expensive to obtain such structural properties of the underlying graph by exhaustive search. If we can estimate these properties by taking small but representative samples from the network, then size is no longer such a problem. In this paper we present a general framework to estimate network properties using random walks. These methods work under the assumption we are able to obtain local characteristics of a vertex during each step of the random walk, for example the number and labels of the neighboring vertices of a specific vertex These assumptions are relatively reasonable in practice, but may add some additional query cost to each step of the random walk. We also present some practical methods to estimate the total number of edges links m, number of vertices nodes n and number of connected triads of vertices (triangles) t in graphs with degree distributions which follow a power-law and higher number of triangles higher than expected in random graphs. We use these graphs since they tend to better correspond to the structure of large online networks, and in fact some of the data used are taken from such a network. Additionally we present experimental estimates for n, m, t we obtained using our methods on real or manufactured networks. In order to make the methods practical, the total number of steps made by the walk was limited to at most the size n of the network. In fact the results appear to converge for a lower number of steps, indicating that our proposed methods are feasible in practice. We study the statistical properties of the sampled scale-free networks, deeply related to the proper identification of various real-world networks. We exploit three methods of sampling and investigate the topological properties such as degree and betweenness centrality distribution, average path length, assortativity, and clustering coefficient of sampled networks compared with those of original networks. It is found that the quantities related to those properties in sampled networks appear to be estimated quite differently for each sampling method. We explain why such a biased estimation of quantities would emerge from the sampling procedure and give appropriate criteria for each sampling method to prevent the quantities from being overestimated or underestimated. Recently, there has been a great deal of research focusing on the development of sampling algorithms for networks with small-world and or power-law structure. The peerto-peer research community (e.g., [7]) have used sampling to quickly explore and obtain a good representative sample of the network topology, as these networks are hard to explore completely and have significant amounts of churn in their topology. For collecting data from social networks, researchers often use snowball sampling (e.g., [2]) due to the lack of access to the complete graph. have developed Forest Fire Sampling, which uses a hybrid combination of snowball sampling and random-walk sampling to produce samples that match the temporal evolution of the underlying social network [5]. have developed a Metropolis algorithm which samples in a manner designed to match desired properties in the original network [3]. Although there has been a great deal of research focusing on the the development of sampling algorithms, much of this work is based on empirical study and evaluation (i.e., measuring the similarity between sampled and original network properties). There has been some work (e.g., [4, 8, 6]) that has studied the statistical properties of samples of complex networks produced by traditional sampling algorithms such as node sampling, edge sampling and random walks. However, there has been relatively little attention paid to the development of a theoretical foundation for sampling from networks—including a formal framework for sampling, an understanding of various network characteristics and their dependencies, and an analysis of their impact on the accuracy of sampling algorithms. In this paper, we reconsider the foundations of network sampling and attempt to formalize the goals, and process of, sampling, in order to frame future development and analysis of sampling algorithms. Given a huge real graph, how can we derive a representative sample? There are many known algorithms to compute interesting measures (shortest paths, centrality, betweenness, etc.), but several of them become impractical for large graphs. Thus graph sampling is essential.The natural questions to ask are (a) which sampling method to use, (b) how small can the sample size be, and (c) how to scale up the measurements of the sample (e.g., the diameter), to get estimates for the large graph. The deeper, underlying question is subtle: how do we measure success?.We answer the above questions, and test our answers by thorough experiments on several, diverse datasets, spanning thousands nodes and edges. We consider several sampling methods, propose novel methods to check the goodness of sampling, and develop a set of scaling laws that describe relations between the properties of the original and the sample.In addition to the theoretical contributions, the practical conclusions from our work are: Sampling strategies based on edge selection do not perform well; simple uniform random node selection performs surprisingly well. Overall, best performing methods are the ones based on random-walks and "forest fire"; they match very accurately both static as well as evolutionary graph patterns, with sample sizes down to about 15 of the original graph.