Fast, reliable, and efficient data transmission across wide-area networks is a predominant bottleneck for data-intensive cloud applications. This paper introduces OneDataShare, which is designed to eliminate the issues plaguing effective cloud-based data transfers of varying file sizes and across incompatible transfer end-points. The vision of OneDataShare is to achieve high-speed data communication, interoperability between multiple transfer protocols, and accurate estimation of delivery time for advance planning, thereby maximizing user-profit through improved and faster data analysis for business intelligence. The paper elaborates on the desirable features of OneDataShare as a cloud-hosted data transfer scheduling and optimization service, and how it is aligned with the vision of harnessing the power of the cloud and distributed computing. Experimental evaluation and comparison with existing real-life file transfer services show that the transfer throughout achieved by OneDataShare is 6.5 times greater.
Server to server data transfers in industry environments have been evaluated in @cite_30 . The proponents have analyzed what benchmarks should be followed when transferring data from one server to another at an industry level. They have evaluated and emphasized the importance of logging and time-stamping the transfer activity at every stage of the transfer for security and auditing purpose. Additionally, the authors provided a mechanism for identifying nearest server when the transfer needs to be speed effective. However, besides security and speed, how to support multi-protocol data transfers and design criteria for such a distributed data delivery system were not discussed. Also the overhead of time-stamping at every stage have been ignored.
@cite_30
2265092719
In distributed storage systems, erasure codes have been increasingly deployed to tolerate server failures without loss of data. Traditional erasure codes, such as Reed-Solomon codes, suffer from a high volume of network transfer and disk I O to recover unavailable data at failed storage servers. Typically, unavailable data at different failed storage servers in a distributed storage system are fixed separately. It has been shown that it is possible to reduce the volume of network transfer significantly by reconstructing data from multiple storage servers at the same time. However, there has been no construction of erasure codes to achieve it without imposing strict constraints on system parameters. In this paper, we propose Beehive codes, designed for optimizing the volume of network transfers to fix the data on multiple failed storage servers. Beehive codes can be constructed over a wide range of system parameters at code rate no more than 0.5, while incurring slightly more storage overhead than Reed-Solomon codes. To achieve the optimal storage overhead as Reed-Solomon codes, we further extend vanilla Beehive codes to MDS Beehive codes, which incurs near-optimal volumes of network transfers during reconstruction. We implement both Beehive and MDS Beehive Codes in C++ and evaluate their performance on Amazon EC2. Our evaluation results have clearly shown that the volume of both network transfers and disk I O can be conserved by a substantial margin. MapReduce is a widely used framework for distributed computing. Data shuffling between the Map phase and Reduce phase of a job involves a large amount of data transfer across servers, which in turn accounts for increase in job completion time. Recently, Coded MapReduce has been proposed to offer savings with respect to the communication cost incurred in data shuffling. This is achieved by creating coded multicast opportunities for shuffling through repeating Map tasks at multiple servers. We consider a server-rack architecture for MapReduce and in this architecture, propose to divide the total communication cost into two: intra-rack communication cost and cross-rack communication cost. Having noted that cross-rack data transfer operates at lower speed as compared to intra-rack data transfer, we present a scheme termed as Hybrid Coded MapReduce which results in lower cross-rack communication than Coded MapReduce at the cost of increase in intra-rack communication. In addition, we pose the problem of assigning Map tasks to servers to maximize data locality in the framework of Hybrid Coded MapReduce as a constrained integer optimization problem. We show through simulations that data locality can be improved considerably by using the solution of optimization to assign Map tasks to servers. Naor, Pinkas, and Sumner introduced and implemented a sealed-bid, two-server auction system that is perhaps the most efficient and practical to date. Based on a cryptographic primitive known as oblivious transfer, their system aims to ensure privacy and correctness provided that at least one auction server behaves honestly. As observed in [19], however, the NPS system suffers from a security flaw in which one of the two servers can cheat so as to modify bids almost arbitrarily and without detection. We propose a means of repairing this flaw while preserving the attractive practical elements of the NPS protocol, including minimal round complexity for servers and minimal computation by players providing private inputs. Our proposal requires a slightly greater amount of computation and communication on the part of the two auction servers, but actually involves much less computation on the part of bidders. This latter feature makes our proposal particularly attractive for use with low-power devices. While the original proposal of NPS involved several dozen exponentiations for a typical auction, ours by contrast involves only several dozen modular multiplications. The key idea in our proposal is a form of oblivious transfer that we refer to as verifiable proxy oblimous transfer (VPOT). This paper presents a smooth way to include Network Coding in the Constrained Application Protocol (CoAP) for large resource transmissions. Devices in the Internet of Things usually communicate using short messages with little data. In some cases, for example, requesting firmware updates, bigger resources need to be transferred. CoAP's recently finalized blockwise transfer scheme can handle large resources, but is not efficient in lossy environments. Network Coding has proven to be more error resistant. This paper demonstrates the limitations of CoAP's existing blockwise transfer scheme and presents a new approach, based on Network Coding. The evaluation compares CoAP's regular blockwise transfer to the new Network Coding extension. Measurements on an implemented client-server application with simulated losses and delay, confirm the benefits of the extension, resulting in reduced transfer durations. As a recently emerging service, multiview video provides a new viewing experience with a high degree of freedom. However, due to the huge data amounts transferred, multiview video's delivery remains a daunting challenge. In this paper, we propose a multiview video-streaming system based on IP multicast. It can support a large number of users while still maintaining a high degree of interactivity and low bandwidth consumption. Based on a careful user study, we have developed two schemes: one is for automatic delivery and the other for on-demand delivery. In automatic delivery, a server periodically multicasts special effect snapshots at a certain time interval. In on-demand delivery, the server delivers the snapshots based on distribution of user requests. We conducted extensive experiments and user-experience studies to evaluate the proposed system's performance, and found that it provides satisfying multiview video service for users on a large scale. Abstract Decrypting and inspecting encrypted malicious communications may assist crime detection and prevention. Access to client or server memory enables the discovery of artefacts required for decrypting secure communications. This paper develops the MemDecrypt framework to investigate the discovery of encrypted artefacts in memory and applies the methodology to decrypting the secure communications of virtual machines. For Secure Shell, used for secure remote server management, file transfer, and tunnelling inter alia, MemDecrypt experiments rapidly yield AES-encrypted details for a live secure file transfer including remote user credentials, transmitted file name and file contents. Thus, MemDecrypt discovers cryptographic artefacts and quickly decrypts live SSH malicious communications including the detection and interception of data exfiltration of confidential data. Systems and methods are provided for managed file transfer. An enterprise server may receive a request from a sender to send a file to a recipient and may determine a location server that is closest to the location of the recipient. A server-to-server transfer can be automatically initiated to move the file to the location server that is closest to the location of the recipient. All discussions of interface design criteria emphasize the importance of consistent operating procedures both within and across applications. This paper presents a model for positive transfer and thus a theoretical definition of consistency. An experiment manipulating training orders for utility tasks was designed to evaluate the transfer model. The experimental manipulations produced large transfer effects. Quantitative predictions were derived from the Kieras and Polson (1985) theory of human-computer interaction and the transfer model and fit using regression techniques. The transfer model accounted for 88 of the variance of the 31 cell means. Many classic and emerging security attacks usually introduce illegal control flow to victim programs. This paper proposes an approach to detecting violation of control flow integrity based on hardware support for performance monitoring in modern processors. The key observation is that the abnormal control flow in security breaches can be precisely captured by performance monitoring units. Based on this observation, we design and implement a system called CFIMon, which is the first non-intrusive system that can detect and reason about a variety of attacks violating control flow integrity without any changes to applications (either source or binary code) or requiring special-purpose hardware. CFIMon combines static analysis and runtime training to collect legal control flow transfers, and leverages the branch tracing store mechanism in commodity processors to collect and analyze runtime traces on-the-fly to detect violation of control flow integrity. Security evaluation shows that CFIMon has low false positives or false negatives when detecting several realistic security attacks. Performance results show that CFIMon incurs only 6.1 performance overhead on average for a set of typical server applications. Wireless links used for mobile communications have some problems such as narrow bandwidth and low reliability. To offer high speed communication on wireless links, we have proposed SHAKE(SHAring multiple paths procedure for cluster networK Environment). In SHAKE, mobile hosts which are connected with fast local link each other use multiple wireless links owned by each host simultaneously to communicate with hosts on the internet. In this paper, we propose a fast WWW access method with SHAKE (Web SHAKE). The feature of Web SHAKE is that it does not require any special software on web servers on the internet to use SHAKE. The feature is realized by an use of HTTP Proxy Server which works on each mobile host. We present the result of performance evaluation of the Web SHAKE. We have shown that data transfer speed was improved by Web SHAKE in comparison with normal WWW access from three experiments. Kernels in the convolutional layers of deep convolutional networks are believed to act as feature extractors, progressively highlighting more domain-specific features in the upper network layers. Thus lower-level features might be suitable for transfer. We analyse this in wearable activity recognition by reusing kernels learned on a source domain on another target domain. We consider transfer between users, application domains, sensor modalities and sensor locations. We characterize the trade-offs of transferring various convolutional layers along model size, learning speed, recognition performance and training data. Through novel kernel visualisations and comparative evaluations we identify what kernels are predominantly sensitive to, amongst sensor characteristics, motion dynamics and on-body placement. Kernel transfer reduces training time by 17 without additional complexity. We derive recommendations on when transfer is most suitable. We present Slurpie: a peer-to-peer protocol for bulk data transfer. Slurpie is specifically designed to reduce client download times for large, popular files, and to reduce load on servers that serve these files. Slurpie employs a novel adaptive downloading strategy to increase client performance, and employs a randomized backoff strategy to precisely control load on the server. We describe a full implementation of the Slurpie protocol, and present results from both controlled local-area and wide-area testbeds. Our results show that Slurpie clients improve performance as the size of the network increases, and the server is completely insulated from large flash crowds entering the Slurpie network. Many real-world applications run on untrusted servers or are run on servers that are subject to strong insider attacks. Although we cannot prevent an untrusted server from modifying or deleting data, with tamper-evident data structures, we can discover when this has occurred. If an untrusted server knows that a particular reply will not be checked for correctness, it is free to lie. Auditing for correctness is thus a frequent but overlooked operation. In my thesis, I present and evaluate new efficient data structures for tamper-evident logging and tamper-evident storage of changing data on untrusted servers, focussing on the costs of the entire system. The first data structure is a new tamper-evident log design. I propose new semantics of tamper-evident logs in terms of the auditing process, required to detect misbehavior. To accomplish efficient auditing, I describe and benchmark a new tree-based data structure that can generate such proofs with logarithmic size and space, significantly improving over previous linear constructions while also offering a flexible query mechanism with authenticated results. The remaining data structures are designs for a persistent authenticated dictionary (PAD) that allows users to send lookup requests to an untrusted server and get authenticated answers, signed by a trusted author, for both the current and historical versions of the dataset. Improving on prior constructions that require logarithmic storage and time, I present new classes of efficient PAD algorithms offering constant-sized authenticated answers or constant storage per update. I implement 21 different versions of PAD algorithms and perform a comprehensive evaluation using contemporary cloud-computing prices for computing and bandwidth to determine the most monetarily cost-effective designs. Virtual machine technology and the ease with which VMs can be migrated within the LAN, has changed the scope of resource management from allocating resources on a single server to manipulating pools of resources within a data center. We expect WAN migration of virtual machines to likewise transform the scope of provisioning compute resources from a single data center to multiple data centers spread across the country or around the world. In this paper we present the CloudNet architecure as a cloud framework consisting of cloud computing platforms linked with a VPN based network infrastructure to provide seamless and secure connectivity between enterprise and cloud data center sites. To realize our vision of efficiently pooling geographically distributed data center resources, CloudNet provides optimized support for live WAN migration of virtual machines. Specifically, we present a set of optimizations that minimize the cost of transferring storage and virtual machine memory during migrations over low bandwidth and high latency Internet links. We evaluate our system on an operational cloud platform distributed across the continental US. During simultaneous migrations of four VMs between data centers in Texas and Illinois, CloudNet's optimizations reduce memory migration time by 65 and lower bandwidth consumption for the storage and memory transfer by 19GB, a 50 reduction. This paper presents a intrusion tolerant architecture for distributed services, especially COTS servers. An intrusion tolerant system assumes that attacks will happen, and some will be successful. However, a wide range of mission critical applications need to provide continuous service despite active attacks or partial compromise. The proposed architecture emphasizes on continuity of operation. It strives to mitigate the effects of both known and unknown attack. We make use techniques of fault tolerant computing, specifically redundancy, diversity, acceptance test, textitvoting—, as well as adaptive reconfiguration. Our architecture consists of five functional components that work together to extend the fault tolerance capability of COTS servers. In addition, the architecture provides mechanisms to audit the COTS servers and internal components for signs of compromise. The auditing as well as adaptive reconfiguration components evaluate the environment threats, identify potential sources of compromise and adaptively generate new configurations for the system. Auditing system logs is an important means of ensuring systems' security in situations where run-time security mechanisms are not sufficient to completely prevent potentially malicious activities. A fundamental requirement for reliable auditing is the integrity of the log entries. This paper presents an infrastructure for secure logging that is capable of detecting the tampering of logs by powerful adversaries residing on the device where logs are generated. We rely on novel features of trusted hardware TPM to ensure the continuity of the logging infrastructure across power cycles without help from a remote server. Our infrastructure also addresses practical concerns including how to handle high-frequency log updates, how to conserve disk space for storing logs, and how to efficiently verify an arbitrary subset of the log. Importantly, we formally state the tamper-proofness guarantee of our infrastructure and verify that our basic secure logging protocol provides the desired guarantee. To demonstrate that our infrastructure is practical, we implement a prototype and evaluate its performance. Cluster computing applications like MapReduce and Dryad transfer massive amounts of data between their computation stages. These transfers can have a significant impact on job performance, accounting for more than 50 of job completion times. Despite this impact, there has been relatively little work on optimizing the performance of these data transfers, with networking researchers traditionally focusing on per-flow traffic management. We address this limitation by proposing a global management architecture and a set of algorithms that (1) improve the transfer times of common communication patterns, such as broadcast and shuffle, and (2) allow scheduling policies at the transfer level, such as prioritizing a transfer over other transfers. Using a prototype implementation, we show that our solution improves broadcast completion times by up to 4.5X compared to the status quo in Hadoop. We also show that transfer-level scheduling can reduce the completion time of high-priority transfers by 1.7X. This paper describes the Network-Attached Secure Disk (NASD) storage architecture, prototype implementations oj NASD drives, array management for our architecture, and three, filesystems built on our prototype. NASD provides scalable storage bandwidth without the cost of servers used primarily, for transferring data from peripheral networks (e.g. SCSI) to client networks (e.g. ethernet). Increasing datuset sizes, new attachment technologies, the convergence of peripheral and interprocessor switched networks, and the increased availability of on-drive transistors motivate and enable this new architecture. NASD is based on four main principles: direct transfer to clients, secure interfaces via cryptographic support, asynchronous non-critical-path oversight, and variably-sized data objects. Measurements of our prototype system show that these services can be cost-effectively integrated into a next generation disk drive ASK. End-to-end measurements of our prototype drive andfilesysterns suggest that NASD cun support conventional distributed filesystems without performance degradation. More importantly, we show scaluble bandwidth for NASD-specialized filesystems. Using a parallel data mining application, NASD drives deliver u linear scaling of 6.2 MB s per clientdrive pair, tested with up to eight pairs in our lab. Wireless charging through directed radio frequency (RF) waves is an emerging technology that can be used to replenish the battery of a sensor node, albeit at the cost of data communi- cation in the network. This tradeoff between energy transfer and communication functions requires a fresh perspective on medium access control (MAC) protocol design for appropriately sharing the channel. Through an experimental study, we demonstrate how the placement, the chosen frequency, and number of the RF energy transmitters impact the sensor charging time. These studies are then used to design a MAC protocol called RF-MAC that optimizes energy delivery to sensor nodes, while minimizing disruption to data communication. In the course of the protocol design, we describe mechanisms for (i) setting the maximum energy charging threshold, (ii) selecting specific transmitters based on the collective impact on charging time, (iii) requesting and granting energy transfer requests, and (iv) evaluating the respective priorities of data communication and energy transfer. To the best of our knowl- edge, this is the first distributed MAC protocol for RF energy harvesting sensors, and through a combination of experimenta- tion and simulation studies, we observe 300 maximum network throughput improvement over the classical modified unslotted CSMA MAC protocol. The notion of self-similarity has been shown to apply to wide-area and local-area network traffic. We show evidence that the subset of network traffic that is due to World Wide Web (WWW) transfers can show characteristics that are consistent with self-similarity, and we present a hypothesized explanation for that self-similarity. Using a set of traces of actual user executions of NCSA Mosaic, we examine the dependence structure of WWW traffic. First, we show evidence that WWW traffic exhibits behavior that is consistent with self-similar traffic models. Then we show that the self-similarity in such traffic can be explained based on the underlying distributions of WWW document sizes, the effects of caching and user preference in file transfer, the effect of user "think time", and the superimposition of many such transfers in a local-area network. To do this, we rely on empirically measured distributions both from client traces and from data independently collected at WWW servers.