TECHNICAL FIELD

Embodiments described herein generally relate to techniques for detecting malware, and more particularly to techniques for detecting return oriented programming exploits.

BACKGROUND ART

Return and jump oriented programming (ROP/JOP) exploits are a growing threat for software applications. These techniques allow an attacker to execute code even if security measures such as non-executable memory and code signing are used. In ROP, an attacker gains control of the call stack and then executes carefully chosen machine instruction sequences, called “gadgets.” Each gadget typically ends in a return instruction and is code within an existing program (or library). Chained together via a sequence of carefully crafted return addresses, these gadgets allow an attacker to perform arbitrary operations. JOP attacks do not depend upon the stack for control flow, but use a dispatcher gadget to take the role of executing functional gadgets that perform primitive operations.

Detection of ROP exploits is complicated due to the nature of the attack. A number of techniques have been proposed to subvert attacks based on return-oriented programming.

The first approach is randomizing the location of program and library code, so that an attacker cannot accurately predict the location of usable gadgets. Address space layout randomization (ASLR) is an example of this approach. Unfortunately, ASLR is vulnerable to information leakage attacks and once the code location is inferred, a return-oriented programming attack can still be constructed. Randomization approach can be taken further by employing relocation at runtime. This complicates the process of finding gadgets but incurs significant overhead.

A second approach modifies the operating system with compiler support to track that return instructions actually divert control flow back to a location immediately following a call instruction. This prevents gadget chaining, but carries a heavy performance penalty. In addition, it is possible to mount JOP attacks without using return instructions at all, by using JMP instructions.

In a third approach, some Intrusion Protection Systems (IPSes) invalidate memory pages of a process except one currently executed page. Most regular jumps land within the same page. Passing control flow to a different page causes an exception that allows the IPS to check the control flow. This technique may also introduce a noticeable overhead.

Better approaches to both ROP and JOP attacks that do not incur large performance penalties would be desirable.

