A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> In the past decade Graphics Processing Units (GPUs) have advanced from simple fixedfunction graphics accelerators to fully-programmable multi-core architectures capable of supporting thousand of concurrent threads. Their use has spread from the specialized field of graphics into more general processing domains ranging from biomedical imaging to stock market prediction. Despite their increased computational power and range of applications, the security implications of GPUs have not been carefully studied. It has been assumed that the use of a GPU as a coprocessor with physically separate memory space, minimal support for multi-user programming, and limited I/O capability inherently guarantees security. This research challenges this assumption by demonstrating multiple security vulnerabilities in the current GPU computing infrastructure. Specifically, it focuses on the following three areas: 1. Denial-of-Service by overwhelming the capabilities of the GPU so it is unable to provide responsiveness to the host operating system. 2. Information leakage due to the way that modern GPUs fail to randomize pointers and zero out memory. 3. The use of GPUs to assist CPU-resident malware through obfuscation and unpacking or acceleration of computational intensive tasks such as password cracking or encryption. Through the use of WebGL and CUDA, we successfully developed a proof of concept attack for the first two vulnerabilities listed above. For the third, we considered several different types of attacks and their implications. In all cases we also suggest possible security measures to fix these vulnerabilities. While the impact of each of these particular exploits is currently hardware and OS specific, current trends in GPU architecture indicate that these problems are only going to rise in importance going forward. <s> BIB001 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> Graphics processing units (GPUs) are important components of modern computing devices for not only graphics rendering, but also efficient parallel computations. However, their security problems are ignored despite their importance and popularity. In this paper, we first perform an in-depth security analysis on GPUs to detect security vulnerabilities. We observe that contemporary, widely-used GPUs, both NVIDIA's and AMD's, do not initialize newly allocated GPU memory pages which may contain sensitive user data. By exploiting such vulnerabilities, we propose attack methods for revealing a victim program's data kept in GPU memory both during its execution and right after its termination. We further show the high applicability of the proposed attacks by applying them to the Chromium and Firefox web browsers which use GPUs for accelerating webpage rendering. We detect that both browsers leave rendered webpage textures in GPU memory, so that we can infer which web pages a victim user has visited by analyzing the remaining textures. The accuracy of our advanced inference attack that uses both pixel sequence matching and RGB histogram matching is up to 95.4%. <s> BIB002 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> We describe Oclgrind, a platform designed to enable the creation of developer tools for analysis and debugging of OpenCL programs. Oclgrind simulates how OpenCL kernels execute with respect to the OpenCL standard, adhering to the execution and memory models that it defines. A simple plugin interface allows developer tools to observe the simulation and collect execution information to provide useful analysis, or catch bugs that would be otherwise difficult to spot when running the application on a real device. We give details about the implementation of the simulator, and describe how it can be extended with plugins that provide useful developer tools. We also present several example use-cases that have already been created using this platform, motivated by real-world problems that OpenCL developers face. <s> BIB003 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> Graphics processing units (GPUs) are increasingly common on desktops, servers, and embedded platforms. In this article, we report on new security issues related to CUDA, which is the most widespread platform for GPU computing. In particular, details and proofs-of-concept are provided about novel vulnerabilities to which CUDA architectures are subject. We show how such vulnerabilities can be exploited to cause severe information leakage. As a case study, we experimentally show how to exploit one of these vulnerabilities on a GPU implementation of the AES encryption algorithm. Finally, we also suggest software patches and alternative approaches to tackle the presented vulnerabilities. <s> BIB004 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> General Purpose Graphics Processing Units (GPGPUs) are present in most modern computing platforms. They are also increasingly integrated as a computational resource on clusters, data centers, and cloud infrastructure, making them possible targets for attacks. We present a first study of covert channel attacks on GPGPUs. GPGPU attacks offer a number of attractive properties relative to CPU covert channels. These channels also have characteristics different from their counterparts on CPUs. To enable the attack, we first reverse engineer the hardware block scheduler as well as the warp to warp scheduler to characterize how co-location is established. We exploit this information to manipulate the scheduling algorithms to create co-residency between the trojan and the spy. We study contention on different resources including caches, functional units and memory, and construct operational covert channels on all these resources. We also investigate approaches to increase the bandwidth of the channel including: (1) using synchronization to reduce the communication cycle and increase robustness of the channel; (2) exploiting the available parallelism on the GPU to increase the bandwidth; and (3) exploiting the scheduling algorithms to create exclusive co-location to prevent interference from other possible applications. We demonstrate operational versions of all channels on three different Nvidia GPGPUs, obtaining error-free bandwidth of over 4 Mbps, making it the fastest known microarchitectural covert channel under realistic conditions. CCS CONCEPTS • Security and privacy → Security in hardware; Hardware attacks and countermeasures; Hardware reverse engineering; • Computer systems organization → Single instruction, multiple data; <s> BIB005 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> GPUs have become an integral part of modern systems, but their implications for system security are not yet clear. This paper demonstrates both that discrete GPUs cannot be used as secure co-processors and that GPUs provide a stealthy platform for malware. First, we examine a recent proposal to use discrete GPUs as secure co-processors and show that the security guarantees of the proposed system do not hold on the GPUs we investigate. Second, we demonstrate that (under certain circumstances) it is possible to bypass IOMMU protections and create stealthy, long-lived GPU-based malware. We demonstrate a novel attack that compromises the in-kernel GPU driver and one that compromises GPU microcode to gain full access to CPU physical memory. In general, we find that the highly sophisticated, but poorly documented GPU hardware architecture, hidden behind obscure close-source device drivers and vendor-specific APIs, not only make GPUs a poor choice for applications requiring strong security, but also make GPUs into a security threat. <s> BIB006 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> Abstract According to previous reports, information could be leaked from GPU memory; however, the security implications of such a threat were mostly over-looked, because only limited information could be indirectly extracted through side-channel attacks. In this paper, we propose a novel algorithm for recovering raw data directly from the GPU memory residues of many popular applications such as Google Chrome and Adobe PDF reader. Our algorithm enables harvesting highly sensitive information including credit card numbers and email contents from GPU memory residues. Evaluation results also indicate that nearly all GPU-accelerated applications are vulnerable to such attacks, and adversaries can launch attacks without requiring any special privileges both on traditional multi-user operating systems, and emerging cloud computing scenarios. <s> BIB007 </s> A Survey of Techniques for Improving Security of GPUs <s> Challenges and tradeoffs <s> Buffer overflows are a common source of program crashes, data corruption, and security problems. In this work, we demonstrate that GPU-based workloads can also cause buffer overflows, a problem that was traditionally ignored because CPUs and GPUs had separate memory spaces. Modern GPUs share virtual, and sometimes physical, memory with CPUs, meaning that GPU-based buffer overflows are capable of producing the same program crashes, data corruption, and security problems as CPU-based overflows. While there are many tools to find buffer overflows in CPU-based applications, the shift towards GPU-enhanced programs has expanded the problem beyond their capabilities. This paper describes a tool that uses canaries to detect buffer overflows caused by GPGPU kernels. It wraps OpenCL API calls and alerts users to any kernel that writes outside of a memory buffer. We study a variety of optimizations, including using the GPU to perform the canary checks, which allow our tool to run at near application speeds. The resulting runtime overhead, which scales with the number of buffers used by the kernel, is 14% across 175 applications in 16 GPU benchmark suites. In these same suites, we found 13 buffer overflows in 7 benchmarks. <s> BIB008
Protecting GPUs, although important, presents several challenges, as we discuss below. Limitations of CPU-based security solutions: After launching the program on GPU (device), CPU (host) remains isolated and thus, it cannot monitor the execution of GPU. Hence, security mechanisms proposed on CPUs, such as a CPU taint-tracking scheme may not work for GPUs. For example, they may not detect a GPU-resident malware and thus, an attacker can use GPU as the polymorphic malware extractor whereby the host can load the compressed/encrypted code on GPU and then call a GPU kernel to quickly unpack/decrypt the code BIB001 . Also, the GPU can be used for launching a brute force attack to crack the passwords. Similarly, since a sharp increase in GPU load is likely to go undetected more easily compared to that in CPU load, a GPU malware is more stealthy. In fact, given the close interaction between GPU and the host, a compromised GPU also threatens the security of host processor. Lack of documentation and open-source tools: Existing GPU vendors take "security-through-obscurity approach" for securing GPUs. As such they do not reveal complete information about GPU microarchitecture and thus, most of this information is obtained through reverse engineering attempts only   BIB005 . Lack of official documentation allows GPU vendors to introduce architectural changes for boosting performance, even at the cost of jeopardizing GPU security. The scarcely-available documentation on GPU discusses only performance, and not security-related issues. For example, GPU binary utilizes closed-source assembly language which cannot be inspected by existing anti-virus tools. Similarly, GPU vendors do not define/document how the deallocated memory is handled BIB004 . Further, in GPUs, the separation of responsibilities between hardware/software is not clear and has been changing over GPU generations BIB006 . Due to this, reasoning about security guarantees becomes difficult. Lack of data erasure: GPU hardware/drivers do not erase their memories and thus, in ShM, LoM, GlM and registers, data persists after deallocation BIB001 BIB004 . Further, while most kinds of memories can be deleted by users, GPUs do not allow users to erase some kinds of memories which store sensitive data such as kernel codes, constant data and call-by-value arguments BIB002 . Also, since the LoM and registers are compilermanaged, erasing them is not straightforward. By exploiting this, an adversary can leak sensitive information. For example, to leak information from the register file, an adversary can write a kernel with the same occupancy and thread-block size as the victim kernel. This ensures similar, predictable partitioning of register file. Then, the malicious kernel can be coded in a way to read the target registers. Increasing reliance on GPUs: GPUs are increasingly being used for accelerating a wide variety of applications, such as browsers, document processors (e.g., Adobe reader, Microsoft Office, Libre Office), scientific computing (MATLAB) and image processing tools, etc. For instance, WebGL allows browsers to utilize GPUs for accelerating the rendering of webpages. In fact, Google Chrome provides an option for using GPU to render the entire webpage and not merely WebGL content  . However, this can be leveraged to launch DoS attack remotely by somehow making a user open a malicious website which overloads users' GPUs. Also notice that in these graphics applications, the ability to read/write GPU memory can allow an adversary to change the contents displayed on user's screen. Similarly, in mission-critical applications where GPUs store and process sensitive data, an attack on them can have huge financial and social consequences. For example, Zhou et al. BIB007 demonstrate extraction of credit card numbers and email contents from remanent data in GPU memory. Also, in companies, a malicious insider may access classified documents which were opened on a shared GPU by an authorized user. Characteristics of GPU architecture and usage model: In GPUs, the presence of multiple memories with different access rights and lifetimes complicates their security solutions and mandates individual security solutions for them. Also, since even non-privileged users can run GPU programs, a large number of users can attack or exploit GPU. Further, once the data is copied from the host memory to device memory, it is managed by the GPU driver and not the host processor/OS. However, GPU drivers may not be as rigorously evaluated from the security perspective as the existing operating systems. Vulnerability in virtualization and cloud scenarios: As the computational capabilities of GPU increase, a single GPU is increasingly being shared among multiple users. As such, major cloud services provide GPU computing platforms such as Amazon web services, Google cloud platform and IBM cloud BIB005 . However, different users in the cloud computing scenario may not trust other. For example, an adversary can rent a GPUbased VM and leak information of users using other VMs on the same system via GPU memory. Clearly, with GPU virtualization approach, the risks of information-leakage is even higher than that with native execution. Vulnerability due to buffer overflows: Conventionally, GPU and CPU memory were separated due to which corruption of CPU memory by GPU was difficult. Also, GPU applications did not typically perform tasks such as pointer-dereferencing, and hence, buffer corruption did not cause much harm BIB008 . Further, since GPU memory was sparsely allocated, buffer overflow did not lead to overwriting useful data. However, as recent GPUs share virtual and even physical memory with CPUs  , buffer overflows in GPU can also lead to security and correctness problems BIB008 . Existing tools to detect GPU buffer overflow, such as Oclgrind may incur latency overheads of up to 300 times BIB003 . Challenges in attacking GPUs: While securing GPUs is challenging as mentioned above, attacking GPUs also presents challenges. By virtue of its massively parallel architecture, GPUs can simultaneously perform multiple encryptions and hence, the timing of individual encryptions cannot be measured. In fact, the encryption with the highest latency dominates the overall latency. Further, since only one cudaContext can run on GPU at any time, a data-leakage attack can obtain only the final snapshot of the previous process BIB004 . Similarly, in a cloud environment, both the cloud and GPU architectures offer layers of obscurity which makes it difficult to launch an attack on GPUs. The techniques discussed in the rest of this article propose strategies for overcoming these challenges. Table 1 classifies the research works based on attack type and the GPU component being studied. It is evident that nearly all important GPU components are vulnerable to attacks. Table 2 organizes the works on the basis of evaluation platform and programming language. Clearly, the GPUs from all major vendors are vulnerable to attacks. 