<commit_before># Web Exploit Detector

![Web Exploit Detector logo](https://www.polaris64.net/i/blogs/online-security/web-exploit-detector.png?w=250&h=)


## Introduction
The Web Exploit Detector is a Node.js application used to detect possible infections, malicious code and suspicious files in web hosting environments. This application is intended to be run on web servers hosting one or more websites. Running the application will generate a list of files that are potentially infected together with a description of the infection and references to online resources relating to it.

The application is hosted here on GitHub so that others can benefit from it, as well as allowing others to contribute their own detection rules.

## Links

 - My website: [https://www.polaris64.net/](https://www.polaris64.net/)
 - My cybersecurity blog, which contains articles describing some of these exploits and how to remove them: [https://www.polaris64.net/blog/cyber-security](https://www.polaris64.net/blog/cyber-security)
 - [Contact me](https://www.polaris64.net/contact_me.php)

## Installation
I plan to implement this project as an npm module soon, but until that point it will need to be cloned and run directly as a Node.js application: -

 1. `git clone https://github.com/polaris64/web_exploit_detector`
 2. `cd web_exploit_detector`
 3. `npm install --production`

## Running
Simply call the script via node and pass the path to your webroot as follows: -

`node index.js --webroot=/var/www/html`

Other command-line options are available, simply run `node index.js` to see a help message describing them.

## Usage as a module
The `src/web-exploit-detector.js` script is a module that exports the set of rules as `rules` as well as a number of functions: -

 - `executeTests(settings)`: runs the exploit checker based on the passed `settings` object. For usage, please consult the `index.js` script.
 - `formatResult(results)`: takes the `results` returned from `executeTests()` and generates a string of results ready for output.
 - `getFileList(path)`: returns an array of files from the base `path` using `readDirRecursive()`.
 - `processRule(rule, files)`: processes a single `rule` on an array of `files` (string paths).
 - `readDirRecursive(path)`: recursive function which returns a Promise which will be resolved with an array of all files in `path` and sub-directories.

The `index.js` script is a simple command-line interface (CLI) to this module, so reading `index.js` shows one way in which this module can be used.

## Rule engine
The application operates using a collection of "rules" which are loaded when the application is run. Each rule consists of an ID, name, description, list of URLs, tags, deprecation flag and most importantly a test. The test can be one of the following: -

 - A regular expression: the simplest type of test, any files matching the regex will pass the test.
 - A Boolean callback: the callback function must return a Boolean value indicating if the passed file passes the test. The callback is free to perform any synchronous operations.
 - A Promise callback: the callback function must return a Promise which is resolved with a Boolean value indicating if the passed file passes the test. This type of callback is free to perform any asynchronous operations.

## Expanding on the rules
As web-based exploits are constantly evolving and new exploits are being created, the set of rules need to be updated too. As I host a number of websites I am constantly observing new kinds of exploits, so I will be adding to the set of rules whenever I can. I run this tool on my own servers, so of course I want it to be as functional as possible!

This brings me onto the reasons why I have made this application available as an open-source project: firstly so that you and others can benefit from it and secondly so that we can all collaborate to contribute detection rules so that the application is always up to date.

### Contributing rules
If you have discovered an exploit that is not detected by this tool then please either contact me to let me know or even better, write your own rule and add it to the third-party rule-set (rules/third-party/index.js), then send me a pull request.

Don't worry if you don't know how to write your own rules; the most important thing is that the rule gets added, so feel free to send me as much information as you can about the exploit and I will try to create my own rule for it.

Rules are categorised, but the simplest way to add your own rule is to add it to the third-party rule-set mentioned above. Rule IDs are written in the following format: "author:type:sub-type(s):rule-id". For example, one of my own rules is "P64:php:cms:wordpress:wso\_webshell". "P64" is me (the author), "php:cms:wordpress" is the grouping (a PHP-specific rule, for the Content Management System (CMS) called WordPress) and "wso\_webshell" is the specific rule ID. When writing your own rules, try to follow this format, and replace "P64" with your own GitHub username or other unique ID.

### Unit tests
The project contains a set of Jasmine tests which can be run using `npm test`.

When developing, tests can also be run whenever a source file changes by running `npm run watch:test`.

Please note that unless you already have Jasmine and/or nodemon installed, you should run `npm install` in non-production mode to ensure that the dev-dependencies are installed.
<commit_msg>README.md: fixed heading level<commit_after># Web Exploit Detector

![Web Exploit Detector logo](https://www.polaris64.net/i/blogs/online-security/web-exploit-detector.png?w=250&h=)


## Introduction
The Web Exploit Detector is a Node.js application used to detect possible infections, malicious code and suspicious files in web hosting environments. This application is intended to be run on web servers hosting one or more websites. Running the application will generate a list of files that are potentially infected together with a description of the infection and references to online resources relating to it.

The application is hosted here on GitHub so that others can benefit from it, as well as allowing others to contribute their own detection rules.

## Links

 - My website: [https://www.polaris64.net/](https://www.polaris64.net/)
 - My cybersecurity blog, which contains articles describing some of these exploits and how to remove them: [https://www.polaris64.net/blog/cyber-security](https://www.polaris64.net/blog/cyber-security)
 - [Contact me](https://www.polaris64.net/contact_me.php)

## Installation
I plan to implement this project as an npm module soon, but until that point it will need to be cloned and run directly as a Node.js application: -

 1. `git clone https://github.com/polaris64/web_exploit_detector`
 2. `cd web_exploit_detector`
 3. `npm install --production`

## Running
Simply call the script via node and pass the path to your webroot as follows: -

`node index.js --webroot=/var/www/html`

Other command-line options are available, simply run `node index.js` to see a help message describing them.

## Usage as a module
The `src/web-exploit-detector.js` script is a module that exports the set of rules as `rules` as well as a number of functions: -

 - `executeTests(settings)`: runs the exploit checker based on the passed `settings` object. For usage, please consult the `index.js` script.
 - `formatResult(results)`: takes the `results` returned from `executeTests()` and generates a string of results ready for output.
 - `getFileList(path)`: returns an array of files from the base `path` using `readDirRecursive()`.
 - `processRule(rule, files)`: processes a single `rule` on an array of `files` (string paths).
 - `readDirRecursive(path)`: recursive function which returns a Promise which will be resolved with an array of all files in `path` and sub-directories.

The `index.js` script is a simple command-line interface (CLI) to this module, so reading `index.js` shows one way in which this module can be used.

## Rule engine
The application operates using a collection of "rules" which are loaded when the application is run. Each rule consists of an ID, name, description, list of URLs, tags, deprecation flag and most importantly a test. The test can be one of the following: -

 - A regular expression: the simplest type of test, any files matching the regex will pass the test.
 - A Boolean callback: the callback function must return a Boolean value indicating if the passed file passes the test. The callback is free to perform any synchronous operations.
 - A Promise callback: the callback function must return a Promise which is resolved with a Boolean value indicating if the passed file passes the test. This type of callback is free to perform any asynchronous operations.

## Expanding on the rules
As web-based exploits are constantly evolving and new exploits are being created, the set of rules need to be updated too. As I host a number of websites I am constantly observing new kinds of exploits, so I will be adding to the set of rules whenever I can. I run this tool on my own servers, so of course I want it to be as functional as possible!

This brings me onto the reasons why I have made this application available as an open-source project: firstly so that you and others can benefit from it and secondly so that we can all collaborate to contribute detection rules so that the application is always up to date.

### Contributing rules
If you have discovered an exploit that is not detected by this tool then please either contact me to let me know or even better, write your own rule and add it to the third-party rule-set (rules/third-party/index.js), then send me a pull request.

Don't worry if you don't know how to write your own rules; the most important thing is that the rule gets added, so feel free to send me as much information as you can about the exploit and I will try to create my own rule for it.

Rules are categorised, but the simplest way to add your own rule is to add it to the third-party rule-set mentioned above. Rule IDs are written in the following format: "author:type:sub-type(s):rule-id". For example, one of my own rules is "P64:php:cms:wordpress:wso\_webshell". "P64" is me (the author), "php:cms:wordpress" is the grouping (a PHP-specific rule, for the Content Management System (CMS) called WordPress) and "wso\_webshell" is the specific rule ID. When writing your own rules, try to follow this format, and replace "P64" with your own GitHub username or other unique ID.

## Unit tests
The project contains a set of Jasmine tests which can be run using `npm test`.

When developing, tests can also be run whenever a source file changes by running `npm run watch:test`.

Please note that unless you already have Jasmine and/or nodemon installed, you should run `npm install` in non-production mode to ensure that the dev-dependencies are installed.

{'fraction_non_alphanumeric': 0.06261459479281262, 'fraction_numerical': 0.004767143381004767, 'mean_word_length': 4.365961633054599, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2782844', 'n_tokens_mistral': 2858, 'n_tokens_neox': 2695, 'n_words': 1635}