Cloud Service Providers deliver their products in form of ”as-a-Service”, which are typically categorized by the level of abstraction. This approach hides the implementation details and shows only functionality to the user. However, the problem is that it is hard to measure the performance of Cloud services, because they behave like black boxes. Especially with Function-as-a-Service it is even more difficult because it completely hides server and infrastructure management from users by design. Cloud Service Prodivers usually restrict the maximum size of code, memory and runtime of Cloud Functions. Nevertheless, users need clarification if more ressources are needed to deliver services in high quality. In this regard, we present the architectural design of a new Function-as-a-Service benchmarking tool, which allows users to evaluate the performance of Cloud Functions. Furthermore, the capabilities of the framework are tested on an isolated platform with a specific workload. The results show that users are able to get insights into Function-as-a-Service environments. This, in turn, allows users to identify factors which may slow down or speed up the performance of Cloud Functions.
@cite_7 present a performance-oriented serverless computing platform to study serverless implementation considerations. Their prototype provides important insights how functions are managed and executed in serverless environments. The authors also discuss several implementation challenges such as function scaling and container discovery in detail. Furthermore, they propose useful metric to evalute the performance of serverless platform. The presented prototype and its internal architecture, in particular, helped us to design and implement the FaaS Benchmarking Framework.
@cite_7
2735574181 2804874586 2889111823 2909714733
We present the design of a novel performance-oriented serverless computing platform implemented in. NET, deployed in Microsoft Azure, and utilizing Windows containers as function execution environments. Implementation challenges such as function scaling and container discovery, lifecycle, and reuse are discussed in detail. We propose metrics to evaluate the execution performance of serverless platforms and conduct tests on our prototype as well as AWS Lambda, Azure Functions, Google Cloud Functions, and IBM's deployment of Apache OpenWhisk. Our measurements show the prototype achieving greater throughput than other platforms at most concurrency levels, and we examine the scaling and instance expiration trends in the implementations. Additionally, we discuss the gaps and limitations in our current design, propose possible solutions, and highlight future research. Serverless computing platforms provide function(s)-as-a-Service (FaaS) to end users while promising reduced hosting costs, high availability, fault tolerance, and dynamic elasticity for hosting individual functions known as microservices. Serverless Computing environments, unlike Infrastructure-as-a-Service (IaaS) cloud platforms, abstract infrastructure management including creation of virtual machines (VMs), operating system containers, and request load balancing from users. To conserve cloud server capacity and energy, cloud providers allow hosting infrastructure to go COLD, deprovisioning containers when service demand is low freeing infrastructure to be harnessed by others. In this paper, we present results from our comprehensive investigation into the factors which influence microservice performance afforded by serverless computing. We examine hosting implications related to infrastructure elasticity, load balancing, provisioning variation, infrastructure retention, and memory reservation size. We identify four states of serverless infrastructure including: provider cold, VM cold, container cold, and warm and demonstrate how microservice performance varies up to 15x based on these states. The Function as a Service (FaaS) subtype of serverless computing provides the means for abstracting away from servers on which developed software is meant to be executed. It essentially offers an event-driven and scalable environment in which billing is based on the invocation of functions and not on the provisioning of resources. This makes it very attractive for many classes of applications with bursty workload. However, the terms under which FaaS services are structured and offered to consumers uses mechanisms like GB–seconds (that is, X GigaBytes of memory used for Y seconds of execution) that differ from the usual models for compute resources in cloud computing. Aiming to clarify these terms, in this work we develop a microbenchmark that we use to evaluate the performance and cost model of popular FaaS solutions using well known algorithmic tasks. The results of this process show a field still very much under development, and justify the need for further extensive benchmarking of these services. The number of available FaaS platforms increases with the rising popularity of a "serverless" architecture and development paradigm. As a consequence, a high demand for benchmarking FaaS platforms exists. In response to this demand, new benchmarking approaches that focus on different objectives continuously emerge. In this paper, we call for community participation to conduct a collaborative systematic literature review with the goal to establish a community-driven knowledge base.