Distributed graph platforms like Pregel have usedvertex-centric programming models to process the growing cor-pus of graph datasets using commodity clusters. However, theirregular structure of graphs causes load imbalances acrossmachines, and this is exacerbated for non-stationary graphalgorithms where not all parts of the graph are active at thesame time. As a result, such graph platforms do not make efficientuse of distributed resources. In this paper, we decouple graphpartitioning from placement on hosts, and introduce strategiesfor elastic placement of graph partitions on Cloud VMs to reducethe cost of execution compared to a static placement, even aswe minimize the increase in makespan. These strategies areinnovative in modeling the graph algorithm's non-stationarybehavior a priori using a metagraph sketch. We validate ourstrategies for several real-world graphs, using runtime tracesfor approximate Betweenness Centrality (BC) algorithm on oursubgraph-centric GoFFish graph platform. Our strategies areable to reduce the cost of execution by up to 54 , comparedto a static placement, while achieving a makespan that is within25 of the optimal.
Distributed graph processing systems divide the graph into a number of partitions which are placed across machines for execution. The quality of partitioning impacts the load on a machine, cost of communication between vertices, and the iterations required to converge. A variety of partitioning techniques have been tried. Giraph's default partitioner hashes vertex IDs to machines, to balance the number vertices per machine. Other approaches that balance the number edges per partition, for algorithms that are edge-bound, have been tried @cite_15 . GoFFish tries to balance the number of vertices per partition while also minimizing the edge cuts between partitions. This gives partitions with well-connected components that suits its subgraph-centric model. Multi-level partitioning schemes have also been identified to improve the CPU utilization @cite_12 . Blogel further uses special 2D partitoners for spatial graphs to improve the convergence time for reachability algorithms.
@cite_15 @cite_12
2221846841 1463623805 2516174802 2225938006
Distributed graph processing platforms have helped emerging application domains use commodity clusters and Clouds to analyze large graphs. Vertex-centric programming models like Google Pregel, and their subgraph-centric variants, specify data-parallel application logic for a single vertex or component that execute iteratively. The locality and balancing of components within partitions affects the performance of such platforms. We propose three partitioning strategies for a subgraph-centric model, and analyze their impact on CPU utilization, communication, iterations, and makespan. We analyze these using Breadth First Search and PageRank algorithms on powerlaw and spatio-planar graphs. They are validated on a commodity cluster using our GoFFish subgraph-centric platform, and compared against Apache Giraph vertex-centric platform. Our experiments show upto 8 times improvement in utilization resulting to upto 5 times improvement of overall makespan for flat and hierarchical partitioning over the default strategy due to improved machine utilization. Further, these also exhibit better horizontal scalability relative to Giraph. The availability of larger and larger graph datasets, growing exponentially over the years, has created several new algorithmic challenges to be addressed. Sequential approaches have become unfeasible, while interest on parallel and distributed algorithms has greatly increased. Appropriately partitioning the graph as a preprocessing step can improve the degree of parallelism of its analysis. A number of heuristic algorithms have been developed to solve this problem, but many of them subdivide the graph on its vertex set, thus obtaining a vertex-partitioned graph. Aim of this paper is to explore a completely different approach based on edge partitioning, in which edges, rather than vertices, are partitioned into disjoint subsets. Contribution of this paper is twofold: first, we introduce a graph processing framework based on edge partitioning, that is flexible enough to be applied to several different graph problems. Second, we show the feasibility of these ideas by presenting a distributed edge partitioning algorithm called d-fep. Our framework is thoroughly evaluated, using both simulations and an Hadoop implementation running on the Amazon EC2 cloud. The experiments show that d-fep is efficient, scalable and obtains consistently good partitions. The resulting edge-partitioned graph can be exploited to obtain more efficient implementations of graph analysis algorithms. Vertex-centric graph processing systems such as Pregel, PowerGraph, or GraphX recently gained popularity due to their superior performance of data analytics on graph-structured data. These systems exploit the graph structure to improve data access locality during computation, making use of specialized graph partitioning algorithms. Recent partitioning techniques assume a uniform and constant amount of data exchanged between graph vertices (i.e., uniform vertex traffic) and homogeneous underlying network costs. However, in real-world scenarios vertex traffic and network costs are heterogeneous. This leads to suboptimal partitioning decisions and inefficient graph processing. To this end, we designed GrapH, the first graph processing system using vertex-cut graph partitioning that considers both, diverse vertex traffic and heterogeneous network, to minimize overall communication costs. The main idea is to avoid frequent communication over expensive network links using an adaptive edge migration strategy. Our evaluations show an improvement of 60 in communication costs compared to state-of-the-art partitioning approaches. In order to cope with the ever-increasing data volume, distributed stream processing systems have been proposed. To ensure scalability most distributed systems partition the data and distribute the workload among multiple machines. This approach does, however, raise the question how the data and the workload should be partitioned and distributed. A uniform scheduling strategy--a uniform distribution of computation load among available machines--typically used by stream processing systems, disregards network-load as one of the major bottlenecks for throughput resulting in an immense load in terms of intermachine communication. In this paper we propose a graph-partitioning based approach for workload scheduling within stream processing systems. We implemented a distributed triple-stream processing engine on top of the Storm realtime computation framework and evaluate its communication behavior using two real-world datasets. We show that the application of graph partitioning algorithms can decrease inter-machine communication substantially (by 40 to 99 ) whilst maintaining an even workload distribution, even using very limited data statistics. We also find that processing RDF data as single triples at a time rather than graph fragments (containing multiple triples), may decrease throughput indicating the usefulness of semantics.