Fast, reliable, and efficient data transmission across wide-area networks is a predominant bottleneck for data-intensive cloud applications. This paper introduces OneDataShare, which is designed to eliminate the issues plaguing effective cloud-based data transfers of varying file sizes and across incompatible transfer end-points. The vision of OneDataShare is to achieve high-speed data communication, interoperability between multiple transfer protocols, and accurate estimation of delivery time for advance planning, thereby maximizing user-profit through improved and faster data analysis for business intelligence. The paper elaborates on the desirable features of OneDataShare as a cloud-hosted data transfer scheduling and optimization service, and how it is aligned with the vision of harnessing the power of the cloud and distributed computing. Experimental evaluation and comparison with existing real-life file transfer services show that the transfer throughout achieved by OneDataShare is 6.5 times greater.
Use of cloud platform to provide data transfer as a service to scientists with increased performance in terms of speed and security has been addressed in @cite_37 which identifies Globus as a probable solution. Recent developments have focused on secured data transmission using Globus with transfer optimization using static parameters @cite_32 @cite_25 . The authors showed results of data transfers using static parameters over the network using REST APIs, however Globus does not employ dynamic throughput optimization or interprotocol translation (at the time of the writing of this article).
@cite_37 @cite_25 @cite_32
2734134136 2613152794 2009900694 1904504745 2221846841 2136340918 2089227674 2295874103 1832849348 2002028225 1656271119 1502216169 2963508401 2763269482 2110514750 2102710358 2394115245 2236077559 2013770282 2055929537
Globus offers a suite of data and user management capabilities to the research community, encompassing data transfer and sharing, user identity and authorization, and data publication. Globus capabilities are accessible via both a web browser and REST APIs. Web access allows Globus to address the needs of research labs through a software-as-a-service model; the newer REST APIs address the needs of developers of research services, who can now use Globus as a platform, outsourcing complex user and data management tasks to Globus cloud-hosted services. Here we review Globus capabilities and outline how it is being applied as a platform for scientific services. Disk-to-disk wide-area file transfers involve many subsystems and tunable application parameters that pose significant challenges for bottleneck detection, system optimization, and performance prediction. Performance models can be used to address these challenges but have not proved generally usable because of a need for extensive online experiments to characterize subsystems. We show here how to overcome the need for such experiments by applying machine learning methods to historical data to estimate parameters for predictive models. Starting with log data for millions of Globus transfers involving billions of files and hundreds of petabytes, we engineer features for endpoint CPU load, network interface card load, and transfer characteristics; and we use these features in both linear and nonlinear models of transfer performance, We show that the resulting models have high explanatory power. For a representative set of 30,653 transfers over 30 heavily used source-destination pairs ("edges''),totaling 2,053 TB in 46.6 million files, we obtain median absolute percentage prediction errors (MdAPE) of 7.0 and 4.6 when using distinct linear and nonlinear models per edge, respectively; when using a single nonlinear model for all edges, we obtain an MdAPE of 7.8 . Our work broadens understanding of factors that influence file transfer rate by clarifying relationships between achieved transfer rates, transfer characteristics, and competing load. Our predictions can be used for distributed workflow scheduling and optimization, and our features can also be used for optimization and explanation. Globus Online manages fire-and-forget file transfers for big-data, high-performance scientific collaborations. Migrating computational intensive tasks from mobile devices to more resourceful cloud servers is a promising technique to increase the computational capacity of mobile devices while saving their battery energy. In this paper, we consider an MIMO multicell system where multiple mobile users (MUs) ask for computation offloading to a common cloud server. We formulate the offloading problem as the joint optimization of the radio resources—the transmit precoding matrices of the MUs—and the computational resources—the CPU cycles second assigned by the cloud to each MU—in order to minimize the overall users’ energy consumption, while meeting latency constraints. The resulting optimization problem is nonconvex (in the objective function and constraints). Nevertheless, in the single-user case, we are able to compute the global optimal solution in closed form. In the more challenging multiuser scenario, we propose an iterative algorithm, based on a novel successive convex approximation technique, converging to a local optimal solution of the original nonconvex problem. We then show that the proposed algorithmic framework naturally leads to a distributed and parallel implementation across the radio access points, requiring only a limited coordination signaling with the cloud. Numerical results show that the proposed schemes outperform disjoint optimization algorithms. Distributed graph processing platforms have helped emerging application domains use commodity clusters and Clouds to analyze large graphs. Vertex-centric programming models like Google Pregel, and their subgraph-centric variants, specify data-parallel application logic for a single vertex or component that execute iteratively. The locality and balancing of components within partitions affects the performance of such platforms. We propose three partitioning strategies for a subgraph-centric model, and analyze their impact on CPU utilization, communication, iterations, and makespan. We analyze these using Breadth First Search and PageRank algorithms on powerlaw and spatio-planar graphs. They are validated on a commodity cluster using our GoFFish subgraph-centric platform, and compared against Apache Giraph vertex-centric platform. Our experiments show upto 8 times improvement in utilization resulting to upto 5 times improvement of overall makespan for flat and hierarchical partitioning over the default strategy due to improved machine utilization. Further, these also exhibit better horizontal scalability relative to Giraph. We consider the problem of resource allocation in downlink OFDMA systems for multi service and unknown environment. Due to users' mobility and intercell interference, the base station cannot predict neither the Signal to Noise Ratio (SNR) of each user in future time slots nor their probability distribution functions. In addition, the traffic is bursty in general with unknown arrival. The probability distribution functions of the SNR, channel state and traffic arrival density are then unknown. Achieving a multi service Quality of Service (QoS) while optimizing the performance of the system (e.g. total throughput) is a hard and interesting task since it depends on the unknown future traffic and SNR values. In this paper we solve this problem by modeling the multiuser queuing system as a discrete time linear dynamic system. We develop a robust H∞ controller to regulate the queues of different users. The queues and Packet Drop Rates (PDR) are controlled by proposing a minimum data rate according to the demanded service type of each user. The data rate vector proposed by the controller is then fed as a constraint to an instantaneous resource allocation framework. This instantaneous problem is formulated as a convex optimization problem for instantaneous subcarrier and power allocation decisions. Simulation results show small delays and better fairness among users. Improving performance of a mobile application by offloading its computation onto a cloudlet has become a prevalent paradigm. Among mobile applications, the category of interactive data-streaming applications is emerging while having not yet received sufficient attention. During computation offloading, the performance of this category of applications (including response time and throughput) depends on network latency and bandwidth between the mobile device and the cloudlet. Although a single cloudlet can provide satisfactory network latency, the bandwidth is always the bottleneck of the throughput. To address this issue, we propose to use multiple cloudlets for computation offloading so as to alleviate the bandwidth bottleneck. In addition, we propose to use multiple module instances to complete a module, enabling more fine-grained computation partitioning, since data processing in many modules of data-streaming applications could be highly parallelized. Specifically, at first we apply a fine-grained data-flow model to characterize mobile interactive data-streaming applications. Then we build a unified optimization framework that achieves maximization of the overall utilities of all mobile users, and design an efficient heuristic for the optimization problem, which is able to make trade-off between throughput and energy consumption at each mobile device. At the end we verify our algorithm with extensive simulation. The results show that the overall utility achieved by our heuristic is close to the precise optimum, and our multiple-cloudlet mechanism significantly outperforms the single-cloudlet mechanism. Handoff mechanisms allow mobile users to move across multiple wireless access points while maintaining their voice and or data sessions. A traditional handoff process is concerned with smoothly transferring a mobile device session from its current access point (or cell) to a target access point (or cell). These handoff characteristics are sufficient for voice calls and background data transfers, however nowadays many mobile applications are heavily based on data and processing capabilities from the cloud. Such applications, especially those that require greater interactivity, often demand not only a smooth session transfer, but also the maintenance of quality of service requirements that impact a user's experience. In this context, the Fog Computing paradigm arises to overcome delays encountered when applications need low latency to access data or offload processing to the cloud. Fog computing introduces a distributed cloud layer, composed of cloudlets (i.e., "small clouds" with lower computational capacity), between the user and the cloud. Cloudlets allow low latency access to data or processing capabilities, which can be accomplished by offering a VM to the user. An overview of Fog computing is first providing, relating it to general concepts in Cloud-based systems, followed by a general architecture to support virtual machine migration in this emerging paradigm -- discussing both the benefits and challenges associated with such migration. Unlike traditional multihop forwarding among homogeneous static sensor nodes, use of mobile devices for data collection in wireless sensor networks has recently been gathering more attention. It is known that the use of mobility significantly reduces the energy consumption at each sensor, elongating the functional lifetime of the network, in exchange for increased data delivery latency. However, in previous work, mobility and communication capabilities are often underutilized, resulting in suboptimal solutions incurring unnecessarily large latency. In this paper, we focus on the problem of finding an optimal path of a mobile device, which we call "data mule," to achieve the smallest data delivery latency in the case of minimum energy consumption at each sensor, i.e., each sensor only sends its data directly to the data mule. We formally define the path selection problem and show the problem is @math -hard. Then we present an approximation algorithm and analyze its approximation factor. Numerical experiments demonstrate that our approximation algorithm successfully finds the paths that result in 10 -50 shorter latency compared to previously proposed methods, suggesting that controlled mobility can be exploited much more effectively. A major obstacle for the adoption of cloud services in enterprises is the potential loss of control over sensitive data. Companies often have to safeguard a subset of their data because it is crucial to their business or they are required to do so by law. In contrast, cloud service providers handle enterprise data without providing guarantees and may put confidentiality at risk. In order to maintain control over their sensitive data, companies typically block all access to a wide range of cloud services at the network level. Such restrictions significantly reduce employee productivity while offering limited practical protection in the presence of malicious employees. In this paper, we suggest a practical mechanism to ensure that an enterprise maintains control of its sensitive data while employees are allowed to use cloud services. We observe that most cloud services use HTTP as a transport protocol. Since HTTP offers well-defined methods to transfer files, inspecting HTTP messages allows the propagation of data between the enterprise and cloud services to be monitored independently of the implementation of specific cloud services. Our system, CloudFilter, intercepts file transfers to cloud services, performs logging and enforces data propagation policies. CloudFilter controls where files propagate after they have been uploaded to the cloud and ensures that only authorised users may gain access. We show that CloudFilter can be applied to control data propagation to Dropbox and GSS, describing the realistic data propagation policies that it can enforce. The need to distribute large files across multiple wide-area sites is becoming increasingly common, for instance, in support of scientific computing, configuring distributed systems, distributing software updates such as open source ISOs or Windows patches, or disseminating multimedia content. Recently a number of techniques have been proposed for simultaneously retrieving portions of a file from multiple remote sites with the twin goals of filling the client's pipe and overcoming any performance bottlenecks between the client and any individual server. While there are a number of interesting tradeoffs in locating appropriate download sites in the face of dynamically changing network conditions, to date there has been no systematic evaluation of the merits of different protocols. This paper explores the design space of file distribution protocols and conducts a detailed performance evaluation of a number of competing systems running in both controlled emulation environments and live across the Internet. Based on our experience with these systems under a variety of conditions, we propose, implement and evaluate Bullet' (Bullet prime), a mesh based high bandwidth data dissemination system that outperforms previous techniques under both static and dynamic conditions. Trends in conventional storage infrastructure motivate the development of foundational technologies for building a wide-area read-write storage repository capable of providing a single image of a distributed storage resource The overarching design goals of such an infrastructure include client performance, global resource utilization, system scalability (providing a single logical view of larger resource and user pools) and application scalability (enabling single applications with large resource requirements) Such a storage infrastructure forms the basis for second generation data-grid efforts underlying massive data handling in high-energy physics, nanosciences, and bioinformatics, among others. This paper describes some of the foundational technologies underlying such a repository, Plethora, for semi-static peer-to-peer (P2P) networks implemented on a wide-area Internet testbed In contrast to many current efforts that focus entirely on unstructured dynamic P2P environments, Plethora focuses on semi-static peers with strong network connectivity and a partially persistent network state In a semi-static P2P network, peers are likely to remain participants in the network over long periods of time (e.g., compute servers), and are capable of providing reasonably high availability and response-time guarantees The repository integrates novel concepts in locality enhancing overlay networks, transactional semantics for read-write data coupled with hierarchical versioning, and novel erasure codes for robustness While mentioning approaches taken by Plethora to other problems, this paper focuses on the problem of routing data request to blocks, while integrating caching and locality enhancing overlays into a single framework We show significant performance improvements resulting from our routing techniques. Seamless computing and data access is enabled by the emerging technology of mobile micro-clouds (MMCs). Different from traditional centralized clouds, an MMC is typically connected directly to a wireless base-station and provides services to a small group of users, which allows users to have instantaneous access to cloud services. Due to the limited coverage area of base-stations and the dynamic nature of mobile users, network background traffic, etc., the question of where to place the services to cope with these dynamics arises. In this paper, we focus on dynamic service placement for MMCs. We consider the case where there is an underlying mechanism to predict the future costs of service hosting and migration, and the prediction error is assumed to be bounded. Our goal is to find the optimal service placement sequence which minimizes the average cost over a given time. To solve this problem, we first propose a method which solves for the optimal placement sequence for a specific look-ahead time-window, based on the predicted costs in this time-window. We show that this problem is equivalent to a shortest-path problem and propose an algorithm with polynomial time-complexity to find its solution. Then, we propose a method to find the optimal look-ahead window size, which minimizes an upper bound of the average cost. Finally, we evaluate the effectiveness of the proposed approach by simulations with realworld user-mobility traces. Modern planetary-scale online services have massive data to transfer over the wide area network (WAN). Due to the tremendous cost of building WANs and the stringent timing requirement of distributed applications, it is critical for network operators to make efficient use of network resources to optimize data transfers. By leveraging software-defined networking (SDN) and reconfigurable optical devices, recent solutions design centralized systems to jointly control the network layer and the optical layer. While these solutions show it is promising to significantly reduce data transfer times by centralized cross-layer control, they do not have any theoretical guarantees on the proposed algorithms. This paper presents approximation algorithms and theoretical analysis for the online transfer scheduling problem over optical WANs. The goal of the scheduling problem is to minimize the makespan (the time to finish all transfers) or the total sum of completion times. We design and analyze various greedy, online scheduling algorithms that can achieve 3-competitive ratio for makespan, 2-competitive ratio for minimum sum completion time for jobs of unit size, and 3α-competitive ratio for jobs of arbitrary transfer size and each node having degree constraint d, where α = 1 when d = 1 and α = 1.86 when d ≥ 2. We also evaluated the performance of these algorithms and compared the performance with prior heuristics. Cluster computing applications like MapReduce and Dryad transfer massive amounts of data between their computation stages. These transfers can have a significant impact on job performance, accounting for more than 50 of job completion times. Despite this impact, there has been relatively little work on optimizing the performance of these data transfers, with networking researchers traditionally focusing on per-flow traffic management. We address this limitation by proposing a global management architecture and a set of algorithms that (1) improve the transfer times of common communication patterns, such as broadcast and shuffle, and (2) allow scheduling policies at the transfer level, such as prioritizing a transfer over other transfers. Using a prototype implementation, we show that our solution improves broadcast completion times by up to 4.5X compared to the status quo in Hadoop. We also show that transfer-level scheduling can reduce the completion time of high-priority transfers by 1.7X. The increasing popularity of cloud storage services has lead companies that handle critical data to think about using these services for their storage needs. Medical record databases, power system historical information and financial data are some examples of critical data that could be moved to the cloud. However, the reliability and security of data stored in the cloud still remain major concerns. In this paper we present DEPSKY, a system that improves the availability, integrity and confidentiality of information stored in the cloud through the encryption, encoding and replication of the data on diverse clouds that form a cloud-of-clouds. We deployed our system using four commercial clouds and used PlanetLab to run clients accessing the service from different countries. We observed that our protocols improved the perceived availability and, in most cases, the access latency when compared with cloud providers individually. Moreover, the monetary costs of using DEPSKY on this scenario is twice the cost of using a single cloud, which is optimal and seems to be a reasonable cost, given the benefits. Mobile users typically have high demand on localized and location-based information services. To always retrieve the localized data from the remote cloud, however, tends to be inefficient, which motivates fog computing. The fog computing, also known as edge computing, extends cloud computing by deploying localized computing facilities at the premise of users, which prestores cloud data and distributes to mobile users with fast-rate local connections. As such, fog computing introduces an intermediate fog layer between mobile users and cloud, and complements cloud computing toward low-latency high-rate services to mobile users. In this fundamental framework, it is important to study the interplay and cooperation between the edge (fog) and the core (cloud). In this paper, the tradeoff between power consumption and transmission delay in the fog-cloud computing system is investigated. We formulate a workload allocation problem which suggests the optimal workload allocations between fog and cloud toward the minimal power consumption with the constrained service delay. The problem is then tackled using an approximate approach by decomposing the primal problem into three subproblems of corresponding subsystems, which can be, respectively, solved. Finally, based on simulations and numerical results, we show that by sacrificing modest computation resources to save communication bandwidth and reduce transmission latency, fog computing can significantly improve the performance of cloud computing. Distributed cloud applications, including batch processing, streaming, and scale-out databases, generate a significant amount of network traffic and a considerable fraction of their runtime is due to network activity. This paper initiates the study of deterministic algorithms for collocating frequently communicating nodes in a distributed networked systems in an online fashion. In particular, we introduce the Balanced RePartitioning (BRP) problem: Given an arbitrary sequence of pairwise communication requests between n nodes, with patterns that may change over time, the objective is to dynamically partition the nodes into ( ) clusters, each of size k, at a minimum cost. Every communication request needs to be served: if the communicating nodes are located in the same cluster, the request is served locally, at cost 0; if the nodes are located in different clusters, the request is served remotely using inter-cluster communication, at cost 1. The partitioning can be updated dynamically (i.e., repartitioned), by migrating nodes between clusters at cost ( ) per node migration. The goal is to devise online algorithms which find a good trade-off between the communication and the migration cost, i.e., “rent” or “buy”, while maintaining partitions which minimize the number of inter-cluster communications. BRP features interesting connections to other well-known online problems. In particular, we show that scenarios with ( =2 ) generalize online paging, and scenarios with (k=2 ) constitute a novel online version of maximum matching. We consider settings both with and without cluster-size augmentation. Somewhat surprisingly (and unlike online paging), we prove that any deterministic online algorithm has a competitive ratio of at least k, even with augmentation. Our main technical contribution is an (O(k k ) )-competitive deterministic algorithm for the setting with (constant) augmentation. This is attractive as, in contrast to ( ), k is likely to be small in practice. For the case of matching ( (k=2 )), we present a constant competitive algorithm that does not rely on augmentation. The elasticity afforded by cloud computing allows consumers to dynamically request and relinquish computing and storage resources and pay for them on a pay-per-use basis. Cloud computing providers rely on virtualization techniques to manage the dynamic nature of their infrastructure allowing consumers to dynamically allocate and deallocate virtual machines of different capacities. Cloud providers need to optimally decide the best allocation of virtual machines to physical machines as the demand varies dynamically. When making such decisions, cloud providers can migrate VMs already allocated and or use external cloud providers. This paper considers the problem in which the cloud provider wants to maximize its revenue, subject to capacity, availability SLA, and VM migration constraints. The paper presents a heuristic solution, called Near Optimal (NOPT), to this NP-hard problem and discusses the results of its experimental evaluation in comparison with a best fit (BF) allocation strategy. The results show that NOPT provides a 45 improvement in average revenue when compared with BF for the parameters used in the experiment. Moreover, the NOPT algorithm maintained the availability close to one for all classes of users while BF exhibited a lower availability and even failed to meet the availability SLA at times. We consider a MIMO multicell system wherein several Mobile Users (MUs) ask for computation offloading to a common cloud server through their femto-access points. We formulate the computation offloading problem as a joint optimization of the radio resources??the transmit precoding matrices of the MUs??and the computational resources??the CPU cycles second assigned by the cloud to each MU??in order to minimize the overall users' energy consumption while meeting the latency constraints imposed by the applications running on the MUs. The resulting optimization problem is nonconvex (in the objective function and the constraints), and there are constraints coupling all the optimization variables. To cope with the nonconvexity, we hinge on successive convex approximation techniques and propose an iterative algorithm converging to a local optimal solution of the original nonconvex problem. The algorithm is also suitable for a parallel implementation across the access point, with limited coordination signaling with the cloud. Numerical results show that the proposed joint optimization yields significant energy savings with respect to more traditional schemes performing a separate optimization of the radio and computational resources.