1. Field of the Invention
The present invention relates to the protection of computer systems. More particularly, the present invention relates to a system and method of detecting and preventing buffer overflows and Return-to-LIBC attacks.
2. Description of Related Art
Buffer overflow techniques have often been used by malicious hackers and virus writers to attack computer systems. Buffers are data storage areas that typically hold a predefined finite amount of data. A buffer overflow occurs when a program attempts to store more data into a buffer than the buffer can accommodate, i.e., the program attempts to store more than the predefined finite amount of data.
One category of buffer overflow, sometimes called stack-based buffer overflow, involves overwriting stack memory, sometimes called the stack. Stack-based buffer overflow is typically caused by programs that do not verify the length of the data being copied into a buffer.
When the data exceeds the size of the buffer, the extra data can overflow into the adjacent memory locations. In this manner, it is possible to corrupt valid data and possibly change the execution flow and instructions.
In the particular case of a Return-to-LIBC attack, hereinafter also referred to as a RLIBC attack, the attacker overflows the stack in such a way that a return address will be replaced to point to a library function in a loaded library inside the process address space. Thus, when the return address is used by the overflowed process, a library function will be executed. This way the attacker runs at least one call function, such as an Application Programming Interface (API), to make operating system function calls and run a command shell on the compromised system remotely.