A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Numerous systems have been designed which use virtualization to subdivide the ample resources of a modern computer. Some require specialized hardware, or cannot support commodity operating systems. Some target 100% binary compatibility at the expense of performance. Others sacrifice security or functionality for speed. Few offer resource isolation or performance guarantees; most provide only best-effort provisioning, risking denial of service.This paper presents Xen, an x86 virtual machine monitor which allows multiple commodity operating systems to share conventional hardware in a safe and resource managed fashion, but without sacrificing either performance or functionality. This is achieved by providing an idealized virtual machine abstraction to which operating systems such as Linux, BSD and Windows XP, can be ported with minimal effort.Our design is targeted at hosting up to 100 virtual machine instances simultaneously on a modern server. The virtualization approach taken by Xen is extremely efficient: we allow operating systems such as Linux and Windows XP to be hosted simultaneously for a negligible performance overhead --- at most a few percent compared with the unvirtualized case. We considerably outperform competing commercial and freely available solutions in a range of microbenchmarks and system-wide tests. <s> BIB001 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> The physical resources of vehicles and roadside infrastructures are stringently constrained in vehicular networks. The application of mobile cloud computing technology will significantly improve the utilization of intensive physical resources. In the newly emerged paradigm of cloud-assisted vehicular networks, vehicle mobility poses a significant challenge to the continuity of cloud services. This paper proposes efficient Virtual Machine (VM) live migration mechanisms to deal with the problem. In particular, a selective dirty page transfer strategy is designed to enhance the efficiency of data transfer in VM live migration. Besides, an optimal resource reservation scheme is proposed to ensure sufficient physical resources at a target cloud site such that migration dropping is significantly reduced. Simulations are carried out to demonstrate the efficiency of the two proposed mechanisms. <s> BIB002 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Despite the broad utilization of cloud computing, some applications and services still cannot benefit from this popular computing paradigm due to inherent problems of cloud computing such as unacceptable latency, lack of mobility support and location-awareness. As a result, fog computing, has emerged as a promising infrastructure to provide elastic resources at the edge of network. In this paper, we have discussed current definitions of fog computing and similar concepts, and proposed a more comprehensive definition. We also analyzed the goals and challenges in fog computing platform, and presented platform design with several exemplar applications. We finally implemented and evaluated a prototype fog computing platform. <s> BIB003 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Mobile edge clouds (MECs) are small cloud-like infrastructures deployed in close proximity to users, allowing users to have seamless and low-latency access to cloud services. When users move across different locations, their service applications often need to be migrated to follow the user so that the benefit of MEC is maintained. In this paper, we propose a layered framework for migrating running applications that are encapsulated either in virtual machines (VMs) or containers. We evaluate the migration performance of various real applications under the proposed framework. <s> BIB004 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Fog computing, an extension of cloud computing services to the edge of the network to decrease latency and network congestion, is a relatively recent research trend. Although both cloud and fog offer similar resources and services, the latter is characterized by low latency with a wider spread and geographically distributed nodes to support mobility and real-time interaction. In this paper, we describe the fog computing architecture and review its different services and applications. We then discuss security and privacy issues in fog computing, focusing on service and resource availability. Virtualization is a vital technology in both fog and cloud computing that enables virtual machines (VMs) to coexist in a physical server (host) to share resources. These VMs could be subject to malicious attacks or the physical server hosting it could experience system failure, both of which result in unavailability of services and resources. Therefore, a conceptual smart pre-copy live migration approach is presented for VM migration. Using this approach, we can estimate the downtime after each iteration to determine whether to proceed to the stop-and-copy stage during a system failure or an attack on a fog computing node. This will minimize both the downtime and the migration time to guarantee resource and service availability to the end users of fog computing. Last, future research directions are outlined. <s> BIB005 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Edge clouds are emerging as a popular paradigm of computation. In edge clouds, computation and storage can be distributed across a large number of locations, allowing applications to be hosted at the edge of the network close to the end-users. Virtual machine live migration is a key mechanism which enables applications to be nimble and nomadic as they respond to changing user locations and workload. However, VM live migration in edge clouds poses a number of challenges. Migrating VMs between geographically separate locations over slow wide-area network links results in large migration times and high unavailability of the application. This is due to network reconfiguration delays as user traffic is redirected to the newly migrated location. In this paper, we propose the use of multi-path TCP to both improve VM migration time and network transparency of applications. We evaluate our approach in a commercial public cloud environment and an emulated lab based edge cloud testbed using a variety of network conditions and show that our approach can reduce migration times by up to 2X while virtually eliminating downtimes for most applications. <s> BIB006 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> VM handoff enables rapid and transparent placement changes to executing code in edge computing use cases where the safety and management attributes of VM encapsulation are important. This versatile primitive offers the functionality of classic live migration but is highly optimized for the edge. Over WAN bandwidths ranging from 5 to 25 Mbps, VM handoff migrates a running 8 GB VM in about a minute, with a downtime of a few tens of seconds. By dynamically adapting to varying network bandwidth and processing load, VM handoff is more than an order of magnitude faster than live migration at those bandwidths. <s> BIB007 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Mobile edge clouds (MECs) bring the benefits of the cloud closer to the user, by installing small cloud infrastructures at the network edge. This enables a new breed of real-time applications, such as instantaneous object recognition and safety assistance in intelligent transportation systems, that require very low latency. One key issue that comes with proximity is how to ensure that users always receive good performance as they move across different locations. Migrating services between MECs is seen as the means to achieve this. This article presents a layered framework for migrating active service applications that are encapsulated either in virtual machines (VMs) or containers. This layering approach allows a substantial reduction in service downtime. The framework is easy to implement using readily available technologies, and one of its key advantages is that it supports containers, which is a promising emerging technology that offers tangible benefits over VMs. The migration performance of various real applications is evaluated by experiments under the presented framework. Insights drawn from the experimentation results are discussed. <s> BIB008 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Supporting smooth movement of mobile clients is important when offloading services on an edge computing platform. Interruption-free client mobility demands seamless migration of the offloading service to nearby edge servers. However, fast migration of offloading services across edge servers in a WAN environment poses significant challenges to the handoff service design. In this paper, we present a novel service handoff system which seamlessly migrates offloading services to the nearest edge server, while the mobile client is moving. Service handoff is achieved via container migration. We identify an important performance problem during Docker container migration. Based on our systematic study of container layer management and image stacking, we propose a migration method which leverages the layered storage system to reduce file system synchronization overhead, without dependence on the distributed file system. We implement a prototype system and conduct experiments using real world product applications. Evaluation results reveal that compared to state-of-the-art service handoff systems designed for edge computing platforms, our system reduces the total duration of service handoff time by 80%(56%) with network bandwidth 5Mbps(20Mbps). <s> BIB009 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> A new level of factory automation demands processing vast amounts of data, complex orchestration of cyber-physical systems, and coordination of computation as well as communication resources in real-time. Virtualization and decentralized computation is becoming a de-facto solution for factory automation. Edge Computing (EC) is a promising approach to achieve the low latencies required by many industrial systems. It employs resource rich edge servers distributed within a factory that are placed close to end devices and assist them in executing computation intensive tasks and also in coordinating with each other. This paper discusses the requirements and challenges of EC for factory automation applications. In a distributed EC infrastructure, safe and timely operation of industrial applications require load balancing and mobility support and thus a seamless service migration between the edge servers. With the recent advances in virtualization and due to its advantages, virtual machine (VM) and container technologies are pavings its way into factory. Though containers have some distinctive advantages over VMs in EC, the service live migration has comparatively high downtime. This paper proposes a novel live migration scheme called redundancy migration that reduces the downtime by a factor of 1.8 compared to the stock migration in linux containers. <s> BIB010 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> The development of the 5G technology has been driven by the need for faster and higher-capacity networks that would be able to sustain modern, high-demanding applications and low-latency communication. For achieving such requirements, the services should be shifted towards the vicinity of the users as much as possible. In this scope, Multi-access Edge Cloud (MEC) will play a tremendous role in 5G technology by hosting various services close to the end-users. Thanks to the MEC technology, different services could be placed near the User Equipment (UEs) for enabling very low latency and high bandwidth communication, which is required by real-time applications, such as mobile broadband in vehicles that are characterized by high UEs mobility. In order to ensure the service requirement in terms of low-latency communication, those services should follow (i.e., service migration) the user mobility by placing them always at the closest MEC. Motivated by the evolution of real time applications, we propose and evaluate three different mechanisms to improve the end user experience by using container-based live migration paradigm. In these approaches, we leverage the follow-me edge concept for enabling lightweight live migration. While the two first solutions take into consideration the mobile users' paths (e.g., cars), the third one is oblivious to the users' paths. The obtained results demonstrate the efficiency of these solutions compared to prior works. <s> BIB011 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Low-latency is an advantage of edge computing. To utilize this advantage, a server-side application communicating with user equipment (UE) should be migrated from a server executing the application to a server near the UE's destination. However, the downtime caused by this migration can be a problem for applications that require ultra-low latency. To reduce the downtime, we propose a novel method for migrating only processes that contain UE-specific states, and not the entire application. The method introduces conversion of inter-process communication channels to enable communication continuity of the migrated processes. Finally, we show that the downtime for our method is shorter than that for VM migration and container migration. <s> BIB012 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Low latency and integration with edge computing facilities are two of the main enablers of the upcoming fifth-generation mobile network. The geographical distribution of virtualized compute and storage resources (e.g., NFVI PoPs) allows to deploy application/service instances/components closer to mobile subscribers, and to provide much higher performance levels in terms of latency. These new degrees of freedom should be suitably managed in order to place and to migrate application components along with subscribers’ move. Given the mass-scale of subscribers and the non-negligible overhead to move application components over resource-constrained edge facilities, this management should be performed in a highly scalable and effective way. In this respect, this paper presents the design of a comprehensive edge computing framework, including an orchestration algorithm, able to place and to move services in few milliseconds (<25 ms). The algorithm provides scalability levels able to support services instantiated on a per-user basis (i.e., personal services) also in the presence of complex network and service environments. The performance evaluation has been carried out by considering different service chains within diverse software live migration technologies. <s> BIB013 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Live migration, underpinned by virtualisation technologies, has enabled improved manageability and fault tolerance for servers. However, virtualised server infrastructures suffer from significant processing overheads, system inconsistencies, security issues and unpredictable performance which makes them unsuitable for low-power and resource-constraint computing devices that processing latency-sensitive, “Big-data”-type data. Consequently, we ask: “How do we eliminate the overhead of virtualisation whilst still retaining its benefits?” Motivated by this question, we investigate a practical approach for a bare-metal live migration scheme for ARM-based instances low-power servers and edge devices. In this paper, we position ARM-based bare-metal live migration as a technique that will underpin the efficiency on edge-computing and on Micro-datacentres. We also introduce our early work on identifying three key technical challenges and discuss their solutions. <s> BIB014 </s> A Survey on Mobility-Induced Service Migration in the Fog, Edge, and Related Computing Paradigms <s> Virtualization Technology-Related Optimizations. <s> Mobile users across edge networks require seamless migration of offloading services. Edge computing platforms must smoothly support these service transfers and keep pace with user movements around the network. However, live migration of offloading services in the wide area network poses significant service handoff challenges in the edge computing environment. In this paper, we propose an edge computing platform architecture which supports seamless migration of offloading services while also keeping the moving mobile user “in service” with its nearest edge server. We identify a critical problem in the state-of-the-art tool for Docker container migration. Based on our systematic study of the Docker container storage system, we propose to leverage the layered nature of the storage system to reduce file system synchronization overhead, without dependence on the distributed file system. In contrast to the state-of-the-art service handoff method in the edge environment, our system yields a 80 percent (56 percent) reduction in handoff time under 5 Mbps (20 Mbps) network bandwidth conditions. <s> BIB015
Contributions in this category assume the decision on "when and where to migrate" has been reached using one of the approaches discussed in Section 5.2.1. As such, they rather focus on reducing the downtime as well as the total migration time, as we detail next. VM-Related Optimizations. The authors in BIB002 propose to improve the default strategy for dirty page transfer in Xen BIB001 , where pages that have been previously sent are sent again in the following iterations. They instead propose a strategy that copies a fixed number of pages in each round. These pages are selected as the ones with the smallest dirtiest rates. The same problem is addressed in BIB005 , where a "smart precopy VM live migration framework" is proposed. The amount of dirty pages observed in previous iterations is provided as an input to a regression model that predicts the expected downtime in each iteration. Then, if the predicted downtime is greater than a predefined downtime threshold, the "stop and copy" stage can be performed. Although the authors claim that the proposed framework can ensure minimum downtime, no evaluation results have been provided. Aligned with the previous two proposals, the authors in BIB006 propose a new precopy algorithm, where "cold" pages, that is, the memory pages with lower probabilities of being written to, are sent to the destination, while the "hot" pages are skipped, hence the proposed name "hot-skip." This results in a reduced length of iterations and a shorter migration time, as shown in the conducted simulations. A different approach to reduce VM migration delays is adopted in  BIB007 BIB003 . In fact, the authors in BIB003 propose two VM migration approaches. In the first one, termed "full migration," a snapshot of the VM is taken by the old fog. After compressing the resulting data, the old fog sends it to the target fog, where it will be decompressed and resumed. An enhancement to this lies in the second approach, that is, the "incremental migration," which considers that both fogs have the base VM image and therefore, only the incremental part will be sent to the destination fog, thus resulting in shorter response times. The authors in  propose a pipeline composed of different stages, aiming to optimize the state transfer between the source and the destination cloudlets by efficiently capturing the state differences among them, while in BIB007 , precaching base VM images into every cloudlet is performed so that blocks that are already present at the target do not need to be transferred again. Container-Related Optimizations. Despite constituting a promising step toward alleviating the downtime issue, the aforementioned VM-centric approaches fail to meet the delay constraints of certain time-sensitive applications. This has led the research community to transition from VMspecific optimizations to container-specific ones. For instance, authors in BIB004 BIB008 propose a threelayer framework for migrating active service applications in an MEC environment. The proposed framework applies for containers (but also to VMs) and is based on the use of three layers, a generic "base layer" present in all MECs, an "application layer" that consists of "an idle version of the application," and an "instance layer" that contains the running state. Using this framework, only layers that have not been found at the destination will be transferred and therefore significant time savings can be achieved. Similarly, building on the layered structure of Docker containers, the authors in BIB015 BIB009 propose avoiding the transfer of redundant image layers to reduce latency. A different approach is proposed in BIB010 , where "redundancy migration" is proposed to avoid the time-consuming "stop and copy" live migration stage. This approach works as follows. As soon as the destination server (DS) is selected, the user starts buffering its packets to it. It is then up to the DS to forward these packets to the old server and retrieve the results to be sent back to the user. Then, while the application is still running in the old server, its checkpoint is created and sent to the DS. The DS uses this checkpoint to restart the container and starts replaying buffered packets from the moment of the checkpoint creation. When the buffer is empty, the destination container takes over and the source container can be shut down. Evaluation results conducted against LXD live migration show that "redundancy migration" is able to effectively reduce the latency. The authors in BIB011 address the need for a lightweight migration of service states among MECs by proposing three different mechanisms of container-based live migration. They propose an iterative approach and a diskless approach in the case where the user path can be known, whereas a shared storage-based approach is proposed for situations where the mobility pattern is random. Testbed experiments are conducted to compare the three approaches and show their performance in terms of migration time and downtime reductions. BIB002 Simulation BIB005 - BIB006 Simulation BIB003 Testbed  Implementation BIB007 Implementation BIB004 , BIB008 Emulation BIB009 , BIB015 Implementation BIB010 Implementation BIB011 Testbed BIB012 Prototype BIB013 Testbed BIB014 Process-Related Optimizations. Apart from optimizations done at the VM and container levels, process-level optimization has been proposed in BIB012 and BIB013 . Motivated by the need to minimize network resource usage in edge environments, the main idea in BIB012 consists of only transferring stateful processes that contain application states; the OS and the stateless processes do not need to be migrated. In order to enable communication continuity between the processes, the authors introduce a method to properly convert interprocess communication channels. The prototype implementation results show less downtime compared to VM and container migrations, in addition to a low overhead of the proposed channel conversion. Process-level optimization is also considered in BIB013 , where authors present multi-context processes (MCPs) as an additional degree of virtualization other than usual techniques (e.g., bare metal, hypervisor-based, container-based). An MCP can be seen as a sort of workspace that runs directly on the hardware, bypassing the hypervisor. A possible implementation of MCPs can be done using DPDK libraries  . VMs and MCPs differ in the size of their virtual (base) images. Authors also consider that these virtual images are present at the destination. Therefore, only the dynamic state needs to be transferred to reduce the overhead. Evaluation results show that the migration downtime for VMs is over two orders of magnitude greater than MCPs.