With the advent of the big data, graph are processed in an iterative manner, which incrementally described in the form of graph in big data applications. Most currently, graph processing methods treat the underlying map data as black boxes. However, as shown in experimental evaluation, graph structures often have diversity, different graph processing methods are very sensitive to the graph structure and show different performance for different data sets. Based on this, a graph processing method for graph structure analysis is proposed in this paper: (1) This paper calculates the vertex activity of a graph according to the in-degree and out-degree, and divide the corresponding vertices into the hot or cold partitions; (2) According to the change of graph structure caused by partial vertex convergence after iteration, this paper reclassifies the partitions, divides the lower active vertices into cold partition and reduces the frequency of calculation, which thereby reducing the cache miss rate and the I O overhead caused by active vertices as well; (3) The partition with highest vertex status degree are given a priority calculation in this paper. In detail, more pronounced and more frequent vertices have higher processing priority. In this way, the convergence speed of the graph vertices is accelerated, and the running time of the graph algorithm in the big data environment is reduced. Our experiments show that compared with the latest system, the proposed method can double the performance of different graph algorithms and data sets.
Pregel @cite_11 divide the graph by hashing the vertex id which ensure loading balancing. Yet Pregel uses message communication paradigm, messages that need to be processed will be huge when vertices has many adjacent points. Coincidentally, Pregel performs inefficiency in power-law graph and only allows global synchronization. X-Pregel @cite_25 optimizes Pregel's messaging mechanism by reducing the number of messages that needed to be delivered in every iteration. Giraph @cite_26 adds more features compare to Pregel, including master computation, out-of-core computation, etc. But the poor locality of data access limits its effective.
@cite_26 @cite_25 @cite_11
2949679549 2127653767 217817341 2064635301
Massive graphs, such as online social networks and communication networks, have become common today. To efficiently analyze such large graphs, many distributed graph computing systems have been developed. These systems employ the "think like a vertex" programming paradigm, where a program proceeds in iterations and at each iteration, vertices exchange messages with each other. However, using Pregel's simple message passing mechanism, some vertices may send receive significantly more messages than others due to either the high degree of these vertices or the logic of the algorithm used. This forms the communication bottleneck and leads to imbalanced workload among machines in the cluster. In this paper, we propose two effective message reduction techniques: (1)vertex mirroring with message combining, and (2)an additional request-respond API. These techniques not only reduce the total number of messages exchanged through the network, but also bound the number of messages sent received by any single vertex. We theoretically analyze the effectiveness of our techniques, and implement them on top of our open-source Pregel implementation called Pregel+. Our experiments on various large real graphs demonstrate that our message reduction techniques significantly improve the performance of distributed graph computation. Many practical computing problems concern large graph. Standard problems include web graph analysis and social networks analysis like Facebook, Twitter. The scale of these graph poses challenge to their efficient processing. To efficiently process large-scale graph, we create X-Pregel, a graph processing system based on Google's Computing Pregel model [1], by using the state-of-the-art PGAS programming language X10. We do not purely implement Google Pregel by using X10 language, but we also introduce two new features that do not exists in the original model to optimize the performance: (1) an optimization to reduce the number of messages which is exchanged among workers, (2) a dynamic re-partitioning scheme that effectively reassign vertices to different workers during the computation. Our performance evaluation demonstrates that our optimization method of sending messages achieves up to 200 speed up on Pagerank by reducing the network I O to 10 times in comparison with the default method of sending messages when processing SCALE20 Kronecker graph [2](vertices = 1,048,576, edges = 33,554,432). It also demonstrates that our system processes large graph faster than prior implementation of Pregel such as GPS [3](stands for graph processing system) and Giraph [4]. To meet the challenge of processing rapidly growing graph and network data created by modern applications, a number of distributed graph processing systems have emerged, such as Pregel and GraphLab. All these systems divide input graphs into partitions, and employ a "think like a vertex" programming model to support iterative graph computation. This vertex-centric model is easy to program and has been proved useful for many graph algorithms. However, this model hides the partitioning information from the users, thus prevents many algorithm-specific optimizations. This often results in longer execution time due to excessive network messages (e.g. in Pregel) or heavy scheduling overhead to ensure data consistency (e.g. in GraphLab). To address this limitation, we propose a new "think like a graph" programming paradigm. Under this graph-centric model, the partition structure is opened up to the users, and can be utilized so that communication within a partition can bypass the heavy message passing or scheduling machinery. We implemented this model in a new system, called Giraph++, based on Apache Giraph, an open source implementation of Pregel. We explore the applicability of the graph-centric model to three categories of graph algorithms, and demonstrate its flexibility and superior performance, especially on well-partitioned data. For example, on a web graph with 118 million vertices and 855 million edges, the graph-centric version of connected component detection algorithm runs 63X faster and uses 204X fewer network messages than its vertex-centric counterpart. Pregel [23] was recently introduced as a scalable graph mining system that can provide significant performance improvements over traditional MapReduce implementations. Existing implementations focus primarily on graph partitioning as a preprocessing step to balance computation across compute nodes. In this paper, we examine the runtime characteristics of a Pregel system. We show that graph partitioning alone is insufficient for minimizing end-to-end computation. Especially where data is very large or the runtime behavior of the algorithm is unknown, an adaptive approach is needed. To this end, we introduce Mizan, a Pregel system that achieves efficient load balancing to better adapt to changes in computing needs. Unlike known implementations of Pregel, Mizan does not assume any a priori knowledge of the structure of the graph or behavior of the algorithm. Instead, it monitors the runtime characteristics of the system. Mizan then performs efficient fine-grained vertex migration to balance computation and communication. We have fully implemented Mizan; using extensive evaluation we show that---especially for highly-dynamic workloads---Mizan provides up to 84 improvement over techniques leveraging static graph pre-partitioning.