Lightning fast enclave network services with async I/O
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Zohrevandi, Mohsen
Enclave technologies such as SGX generally have a relatively high context-switching cost. This is particularly noticeable when doing (network) I/O. In this talk we'll present the first non-LibOS implementation of an asynchronous I/O model for SGX. This gives you a language-native async I/O programming experience in Rust, outperforming any other way to build SGX network services.
 Hello everyone. My name is Mohsen and I'm going to talk to you about what we have recently been doing to support asynchronous IO in SGX Enclips. Over the past few years we have been developing a Rust platform for writing code that runs in SGX Enclips. This Enclips development platform or EDP is available as open source software and we have been using it at Fortanix as the backbone of multiple security oriented products. Producing SGX Enclips has never been easier. All you need to do is compile your Rust code for the Fortanix SGX target as you can see here. This will produce an Enclips that can be run with our ready to use Enclips runner or a custom Enclips runner if you are choosing if you need special user space extensions. EDP is purposefully designed for network services and that allows us to limit the API surface needed from untrusted user space component. We have only 16 user calls in EDP which is a very small subset of what's needed to support a general purpose application framework. For example, Linux has hundreds of system calls and there are Enclips platforms that expose all or a significant portion of those system calls to the algorithm. A significant source of performance overhead in SGX is context switching. The example shown here compares the time it takes to do a single system call in Linux versus a single user call in EDP and as you can see there is a very significant difference. So to improve performance we simply need to avoid context switching as much as possible. In a network service where there are many concurrently running threads the operating system needs to switch between the threads so they can all access the CPU. In EDP the Enclips runner can have a smaller number of threads than the Enclips itself and yet service all the Enclips threads. We have implemented this M2N threading model in EDP's Enclips runner to reduce the cost of OS context switching. But what about SGX context switches? There are some published academic work on this. The first paper listed here uses shared memory buffers to send e-calls and on-calls between the Enclips and the user space instead of SGX context switching. There is also a busy wait algorithm to optimize performance when there are many consecutive e-calls and on-calls. The second paper takes a different approach. They implement M2N threading inside the Enclips. We have implemented an approach similar to the first paper for EDP and integrated that into Rust's Async IOS system. This makes it possible to compile code written in Rust using the Async Await syntax with EDP and leverage the performance benefits of fewer context switches as well. Now let's briefly talk about Async IOS in Rust. Rust provides an interface called Future which represents any computation that will finish at some point and produce an output. In order to make progress on a future, you need to have a runtime component that pulls the future. When pulled, the future either returns pending, in which case the future will be pulled again once it can make progress, or it returns ready with an output value. This is a very powerful abstraction because it allows a single thread to pull hundreds or even thousands of futures and drive them to completion. But how would you actually implement the future trade? Well, fortunately in most cases you don't need to explicitly implement the future trade. Rust provides a syntax that allows the programmer to write code that is very similar to non-Async code. The syntax consists of Async code blocks and functions and the Await keyword, which is applied to values that implement the future trade to get their output value. This example code shows how you might write code that uses Async Await syntax. As you can see, it looks very similar to something you might normally write except for the Async Await keyword. At compile time, Rust will de-sugar this function into a type that implements the future trade. This process in itself is a very interesting topic, but it has been discussed extensively elsewhere. So I won't go into the details of how the compiler transforms your code. Note that when writing Async code, you should not use types that block the current thread. For example, the TCP stream type provided in Rust's standard library does not support Async IO. However, there are Rust libraries that provide a version of these fundamental types that can work asynchronously. One such library is Tokyo, and I'll talk more about Tokyo later on. All right, let's get back to EDP and see what it takes to be able to compile and run Async IO code in EDP enclaves. First we need to talk about how we perform blocking IO in EDP enclaves. An enclaves needs to do IO, say, read from a TCP socket. It cannot directly call the read system call because all interactions with the operating system have to go through the user space component. This is a fundamental restriction imposed by Intel SGX. So instead, the enclaves will perform an enclaves exit to switch execution context to the user space and request to perform the IO operation. This is what we call a user call. The user space then in turn calls the appropriate system call and returns the results back to the enclave by doing another context switch to enter the enclave. So there are four context switches to satisfy a single IO operation in the traditional blocking IO model. As we established earlier, these context switches are expensive, and we should try to minimize the context switches as much as possible. The solution we have implemented for EDP is to use FIFO cubes shared between the enclave and the user space to send user calls and return values. When the enclave wants to perform a user call, it sends a user call descriptor through a shared memory FIFO cube. At this point, the enclave can continue execution and do other useful work instead of blocking for the results. The enclave runner, which is the user space component in charge of handling user space and user calls sent by the enclave, will receive the user call through the cube, performs the requested user call, and then sends back the results to the enclave through a separate FIFO cube. So in addition to avoiding S check context switching, this model also enables us to perform asynchronous IO. The enclave submits an IO call without blocking for the results and is notified once the results are ready. This is similar to IO completion ports in Microsoft Windows. Now if you recall, I mentioned a Rust library called Tokyo, which provides fundamental types that can be used in asynchronous programs. Tokyo uses another library called MIO. MIO is essentially a thin abstraction layer over the EpoL API, which is available in Linux and many other similar operating systems. So in order to get Tokyo to compile in EDP, we had to port MIO to EDP. Let's see how we did that. EpoL is an API that is used to monitor multiple file descriptors to see if it's possible to perform IO operations on any of them. Once a file descriptor is ready for IO, EpoL generates a readiness event so the application can actually try to perform the IO operation. Now as you can imagine, this is different from the IO completion model we have designed for EDP. So the challenge was to bridge the gap between these models. Fortunately, this was already done when MIO was first ported to Windows and we used a similar approach. It's interesting to note that the current MIO implementation for Windows has somewhat moved away from the IO completion ports since people have found hidden Windows APIs that provide readiness events. They did this to improve performance because using IO completion ports requires copying buffers in order to be correct with respect to Rust's lifetimes and borrow checking system. However, for EDP, we need to copy buffers to user space memory in any implementation because the output memory is encrypted and cannot be accessed by user space. That's why we have opted to use this model which also alleviates the SCX context switching overhead. I thought it might be interesting to elaborate more on how we actually translate the IO completion model to IO readiness model. Let's consider the case of reading from a TCP socket. On the left side, you can see how E-Poll can be used to read from a socket. Once the socket is connected, it cannot be read until the Read Readiness event is generated by E-Poll. Once the socket is ready, the program can call Read to read bytes from the socket. Once all the available bytes have been read, the read call will return an error to indicate that there are no more bytes available immediately and the program needs to wait for another readiness event. On the right side, you can see how we implemented this using our ACNQs in ADP. Once the socket is connected, we immediately send a read user call through the FICO queue and enter the pending Read State. Once the user space sends back the results of that user call, we generate a readable readiness event and transition to readable state. At this point, the program can call Read on the socket and receive bytes from the chunk of bytes that we have received. Once all of those bytes are delivered, we issue another read user call and transition to pending Read State and repeat the whole process. Writes are a bit more complicated. As you can see here, writes are very similar to reads in E-Poll lines, but that's not the case in ADP. Once the socket is connected, we immediately generate a writable readiness event and transition to writable state. Once the program calls write on the socket, we send a write user call using our FICO queue and transition to pending Write State. At this point, we're still waiting to receive the results of that first write, but the program is free to call write on the socket again, in which case we buffer those bytes and acknowledge that they will be sent. This can go on until the user space buffer for the socket is filled, in which case we will return a wood block indicating to the program that it cannot further write to the socket at this point. Once the write user call results are received from the user space, we mark the bytes as written and check the buffer. If there are more bytes that need to be sent, we issue another write user call and continue until the buffer is empty, at which point we transition back to writable state and generate a writable readiness event so the program can continue writing. Other user calls, such as connect and accept, are also implemented in a similar fashion. Overall, this seems like a reasonable method to implement MIU's abstraction layer in ADP using our async queue. To validate the performance benefits of our approach, we ran a few tests to compare blocking IO versus async IO in both Linux and ADP. In our first test, we wrote a small program that listens for HTTP requests with an input encoded in JSON and echoes back the message to the client encoded similarly. We wrote two versions of this code, one version uses traditional blocking IO and the other version uses async IO. We compiled and ran these two programs both on Linux and also as an SCX enclave with ADP. We measured how many queries per second each combination can achieve with various degrees of concurrent client connections. As you might imagine, running the programs natively on Linux has higher performance compared to SCX since there is no SCX overhead. The blocking versus async doesn't seem to differ a lot, but that's just an artifact of the simplicity of our test program. In the SCX case, you can see that the performance of the blocking version drops noticeably for more than 2000 connections, but the async version provides a steady performance even for a higher number of connections. To make the test a little more realistic, we added another service in this mix, a hash service that listens on a TCP port and computes the hash of input messages sends back the results. Then we modified the echo program to connect to the hash service for every request and include the hash result in its response to each client. This is more like a web service that might connect to a database to fulfill every HTTP request it receives. Here are the results. As you can see, there is a noticeable difference between blocking and async versions in Linux. The async version can achieve higher throughput. There is also a drop in performance of the blocking version after 2000 connections, just as we saw before. Note that the blocking version needs to handle each client connection in a separate thread, so 5000 connections means 5000 threads competing for resources, while the async IO version only uses a few threads regardless of the number of client connections. That can explain the superior performance of the async version. The SCX numbers also tell a similar story, although the overall numbers are smaller compared to Linux because of S-share overheads. The blocking version also has a drop in performance at the 2000 connection mark. One more thing that we should mention here is that both the blocking and async versions are run with the same Onclave runner that implements the M2N threading model that we discussed before. Therefore the blocking version is also benefiting from reduced OS context switching in ADP. There are other benefits to using async IO in ADP. For one, in SCX version 1, the Onclave needs to have a fixed number of thread control structures, which is determined at Onclave build time. This limits how many Onclave threads can execute concurrently. If you build an Onclave with 100 TCSs, you can only have up to 100 concurrent threads. Traditional blocking IO network applications usually dedicate one thread per client connection. So an ADP Onclave need to be built with a huge number of TCSs to be able to handle thousands of concurrent connections. On the other hand, by using async IO in ADP, the Onclave only needs a handful of threads and TCSs to be able to handle thousands of connections. Consequently, Onclaves can initialize faster and consume less stack space. Also, there will be no limit on the number of concurrent connections that your service can handle just because of fixed number of TCSs. Well, that's all for me. Thank you for listening, and I hope you found this presentation both useful and interesting. My colleague Dr. Beekman will answer your questions. Thank you.
 Hello everyone.  My name is Mohsen and I'm going to talk to you about what we have recently been doing  to support asynchronous IO in SGX Enclips.  Over the past few years we have been developing a Rust platform for writing code that runs  in SGX Enclips.  This Enclips development platform or EDP is available as open source software and we have  been using it at Fortanix as the backbone of multiple security oriented products.  Producing SGX Enclips has never been easier.  All you need to do is compile your Rust code for the Fortanix SGX target as you can see  here.  This will produce an Enclips that can be run with our ready to use Enclips runner or a  custom Enclips runner if you are choosing if you need special user space extensions.  EDP is purposefully designed for network services and that allows us to limit the API surface  needed from untrusted user space component.  We have only 16 user calls in EDP which is a very small subset of what's needed to support  a general purpose application framework.  For example, Linux has hundreds of system calls and there are Enclips platforms that  expose all or a significant portion of those system calls to the algorithm.  A significant source of performance overhead in SGX is context switching.  The example shown here compares the time it takes to do a single system call in Linux  versus a single user call in EDP and as you can see there is a very significant difference.  So to improve performance we simply need to avoid context switching as much as possible.  In a network service where there are many concurrently running threads the operating  system needs to switch between the threads so they can all access the CPU.  In EDP the Enclips runner can have a smaller number of threads than the Enclips itself  and yet service all the Enclips threads.  We have implemented this M2N threading model in EDP's Enclips runner to reduce the cost  of OS context switching.  But what about SGX context switches?  There are some published academic work on this.  The first paper listed here uses shared memory buffers to send e-calls and on-calls between  the Enclips and the user space instead of SGX context switching.  There is also a busy wait algorithm to optimize performance when there are many consecutive  e-calls and on-calls.  The second paper takes a different approach.  They implement M2N threading inside the Enclips.  We have implemented an approach similar to the first paper for EDP and integrated that  into Rust's Async IOS system.  This makes it possible to compile code written in Rust using the Async Await syntax with  EDP and leverage the performance benefits of fewer context switches as well.  Now let's briefly talk about Async IOS in Rust.  Rust provides an interface called Future which represents any computation that will finish  at some point and produce an output.  In order to make progress on a future, you need to have a runtime component that pulls  the future.  When pulled, the future either returns pending, in which case the future will be pulled again  once it can make progress, or it returns ready with an output value.  This is a very powerful abstraction because it allows a single thread to pull hundreds  or even thousands of futures and drive them to completion.  But how would you actually implement the future trade?  Well, fortunately in most cases you don't need to explicitly implement the future trade.  Rust provides a syntax that allows the programmer to write code that is very similar to non-Async  code.  The syntax consists of Async code blocks and functions and the Await keyword, which is  applied to values that implement the future trade to get their output value.  This example code shows how you might write code that uses Async Await syntax.  As you can see, it looks very similar to something you might normally write except for the Async  Await keyword.  At compile time, Rust will de-sugar this function into a type that implements the future trade.  This process in itself is a very interesting topic, but it has been discussed extensively  elsewhere.  So I won't go into the details of how the compiler transforms your code.  Note that when writing Async code, you should not use types that block the current thread.  For example, the TCP stream type provided in Rust's standard library does not support  Async IO.  However, there are Rust libraries that provide a version of these fundamental types that  can work asynchronously.  One such library is Tokyo, and I'll talk more about Tokyo later on.  All right, let's get back to EDP and see what it takes to be able to compile and run  Async IO code in EDP enclaves.  First we need to talk about how we perform blocking IO in EDP enclaves.  An enclaves needs to do IO, say, read from a TCP socket.  It cannot directly call the read system call because all interactions with the operating  system have to go through the user space component.  This is a fundamental restriction imposed by Intel SGX.  So instead, the enclaves will perform an enclaves exit to switch execution context to the user  space and request to perform the IO operation.  This is what we call a user call.  The user space then in turn calls the appropriate system call and returns the results back to  the enclave by doing another context switch to enter the enclave.  So there are four context switches to satisfy a single IO operation in the traditional blocking  IO model.  As we established earlier, these context switches are expensive, and we should try to minimize  the context switches as much as possible.  The solution we have implemented for EDP is to use FIFO cubes shared between the enclave  and the user space to send user calls and return values.  When the enclave wants to perform a user call, it sends a user call descriptor through a shared  memory FIFO cube.  At this point, the enclave can continue execution and do other useful work instead of blocking  for the results.  The enclave runner, which is the user space component in charge of handling user space  and user calls sent by the enclave, will receive the user call through the cube, performs the  requested user call, and then sends back the results to the enclave through a separate  FIFO cube.  So in addition to avoiding S check context switching, this model also enables us to perform  asynchronous IO.  The enclave submits an IO call without blocking for the results and is notified once the results  are ready.  This is similar to IO completion ports in Microsoft Windows.  Now if you recall, I mentioned a Rust library called Tokyo, which provides fundamental types  that can be used in asynchronous programs.  Tokyo uses another library called MIO.  MIO is essentially a thin abstraction layer over the EpoL API, which is available in Linux  and many other similar operating systems.  So in order to get Tokyo to compile in EDP, we had to port MIO to EDP.  Let's see how we did that.  EpoL is an API that is used to monitor multiple file descriptors to see if it's possible to  perform IO operations on any of them.  Once a file descriptor is ready for IO, EpoL generates a readiness event so the application  can actually try to perform the IO operation.  Now as you can imagine, this is different from the IO completion model we have designed  for EDP.  So the challenge was to bridge the gap between these models.  Fortunately, this was already done when MIO was first ported to Windows and we used a  similar approach.  It's interesting to note that the current MIO implementation for Windows has somewhat  moved away from the IO completion ports since people have found hidden Windows APIs that  provide readiness events.  They did this to improve performance because using IO completion ports requires copying  buffers in order to be correct with respect to Rust's lifetimes and borrow checking system.  However, for EDP, we need to copy buffers to user space memory in any implementation  because the output memory is encrypted and cannot be accessed by user space.  That's why we have opted to use this model which also alleviates the SCX context switching  overhead.  I thought it might be interesting to elaborate more on how we actually translate the IO completion  model to IO readiness model.  Let's consider the case of reading from a TCP socket.  On the left side, you can see how E-Poll can be used to read from a socket.  Once the socket is connected, it cannot be read until the Read Readiness event is generated  by E-Poll.  Once the socket is ready, the program can call Read to read bytes from the socket.  Once all the available bytes have been read, the read call will return an error to indicate  that there are no more bytes available immediately and the program needs to wait for another  readiness event.  On the right side, you can see how we implemented this using our ACNQs in ADP.  Once the socket is connected, we immediately send a read user call through the FICO queue  and enter the pending Read State.  Once the user space sends back the results of that user call, we generate a readable  readiness event and transition to readable state.  At this point, the program can call Read on the socket and receive bytes from the chunk  of bytes that we have received.  Once all of those bytes are delivered, we issue another read user call and transition  to pending Read State and repeat the whole process.  Writes are a bit more complicated.  As you can see here, writes are very similar to reads in E-Poll lines, but that's not  the case in ADP.  Once the socket is connected, we immediately generate a writable readiness event and transition  to writable state.  Once the program calls write on the socket, we send a write user call using our FICO  queue and transition to pending Write State.  At this point, we're still waiting to receive the results of that first write, but the program  is free to call write on the socket again, in which case we buffer those bytes and acknowledge  that they will be sent.  This can go on until the user space buffer for the socket is filled, in which case we  will return a wood block indicating to the program that it cannot further write to the  socket at this point.  Once the write user call results are received from the user space, we mark the bytes as  written and check the buffer.  If there are more bytes that need to be sent, we issue another write user call and continue  until the buffer is empty, at which point we transition back to writable state and generate  a writable readiness event so the program can continue writing.  Other user calls, such as connect and accept, are also implemented in a similar fashion.  Overall, this seems like a reasonable method to implement MIU's abstraction layer in ADP  using our async queue.  To validate the performance benefits of our approach, we ran a few tests to compare blocking  IO versus async IO in both Linux and ADP.  In our first test, we wrote a small program that listens for HTTP requests with an input  encoded in JSON and echoes back the message to the client encoded similarly.  We wrote two versions of this code, one version uses traditional blocking IO and the other  version uses async IO.  We compiled and ran these two programs both on Linux and also as an SCX enclave with ADP.  We measured how many queries per second each combination can achieve with various degrees  of concurrent client connections.  As you might imagine, running the programs natively on Linux has higher performance compared  to SCX since there is no SCX overhead.  The blocking versus async doesn't seem to differ a lot, but that's just an artifact  of the simplicity of our test program.  In the SCX case, you can see that the performance of the blocking version drops noticeably for  more than 2000 connections, but the async version provides a steady performance even  for a higher number of connections.  To make the test a little more realistic, we added another service in this mix, a hash  service that listens on a TCP port and computes the hash of input messages sends back the  results.  Then we modified the echo program to connect to the hash service for every request and  include the hash result in its response to each client.  This is more like a web service that might connect to a database to fulfill every HTTP  request it receives.  Here are the results.  As you can see, there is a noticeable difference between blocking and async versions in Linux.  The async version can achieve higher throughput.  There is also a drop in performance of the blocking version after 2000 connections, just  as we saw before.  Note that the blocking version needs to handle each client connection in a separate thread,  so 5000 connections means 5000 threads competing for resources, while the async IO version  only uses a few threads regardless of the number of client connections.  That can explain the superior performance of the async version.  The SCX numbers also tell a similar story, although the overall numbers are smaller compared  to Linux because of S-share overheads.  The blocking version also has a drop in performance at the 2000 connection mark.  One more thing that we should mention here is that both the blocking and async versions  are run with the same Onclave runner that implements the M2N threading model that we  discussed before.  Therefore the blocking version is also benefiting from reduced OS context switching in ADP.  There are other benefits to using async IO in ADP.  For one, in SCX version 1, the Onclave needs to have a fixed number of thread control structures,  which is determined at Onclave build time.  This limits how many Onclave threads can execute concurrently.  If you build an Onclave with 100 TCSs, you can only have up to 100 concurrent threads.  Traditional blocking IO network applications usually dedicate one thread per client connection.  So an ADP Onclave need to be built with a huge number of TCSs to be able to handle thousands  of concurrent connections.  On the other hand, by using async IO in ADP, the Onclave only needs a handful of threads  and TCSs to be able to handle thousands of connections.  Consequently, Onclaves can initialize faster and consume less stack space.  Also, there will be no limit on the number of concurrent connections that your service  can handle just because of fixed number of TCSs.  Well, that's all for me.  Thank you for listening, and I hope you found this presentation both useful and interesting.  My colleague Dr. Beekman will answer your questions.  Thank you.
tee_async_io_0_167.png tee_async_io_168_359.png tee_async_io_360_419.png tee_async_io_420_2219.png tee_async_io_2220_2759.png tee_async_io_2760_5087.png tee_async_io_5088_6383.png tee_async_io_6384_8603.png tee_async_io_8604_9227.png tee_async_io_9228_10547.png tee_async_io_10548_12371.png tee_async_io_12372_13139.png tee_async_io_13140_15395.png tee_async_io_15396_17819.png tee_async_io_17820_20435.png tee_async_io_20436_21791.png tee_async_io_21792_22691.png tee_async_io_22692_25535.png tee_async_io_25536_25691.png tee_async_io_25692_26003.png tee_async_io_26004_26291.png tee_async_io_26292_26375.png tee_async_io_26376_26963.png tee_async_io_26964_27527.png tee_async_io_27528_27635.png tee_async_io_27636_27754.png