<commit_before># cti-pattern-validator

*This is an [OASIS Open Repository](https://www.oasis-open.org/resources/open-repositories/). See the [Governance](#governance) section for more information.*

The STIX 2 Pattern Validator is a software tool for checking the syntax of the Cyber Threat Intelligence (CTI) STIX Pattern expressions, which are used within STIX to express conditions (represented with the Cyber Observable data model) that indicate particular cyber threat activity. The repository contains source code, an ANTLR grammar, automated tests and associated documentation for the tool. The validator can be used as a command-line tool or as a Python library which can be included in other applications.

[![Travis-CI Build Status](https://api.travis-ci.org/oasis-open/cti-pattern-validator.svg?branch=master)](https://travis-ci.org/oasis-open/cti-pattern-validator)

## Requirements

- [Python](https://www.python.org) 2.7, 3.3, 3.4, or 3.5
- ANTLR grammar runtime (4.5.3 or newer):
    - [antlr4-python2-runtime](https://pypi.python.org/pypi/antlr4-python2-runtime) (Python 2.7)
    - [antlr4-python3-runtime](https://pypi.python.org/pypi/antlr4-python3-runtime) (Python 3)
- [enum34](https://pypi.python.org/pypi/enum34) (Python 3.3)
- [six](https://pypi.python.org/pypi/six)

## Installation

Using [pip](https://pip.pypa.io) is highly recommended:

```bash
$ pip install stix2-patterns
```

For more information about installing Python packages, see the [Python
Packaging User Guide](https://packaging.python.org/installing/).

## Usage

The STIX Pattern Validator provides an executable script (`validate-patterns`)
in addition to being an importable Python library.

The `validate-patterns` script accepts patterns from either direct user input
or a file passed as an option.

### User Input

When prompted, enter a pattern to validate and press enter. The validator will
supply whether the pattern has passed or failed. If the pattern fails the test,
the validator will supply where the first syntax error occurred. The validator
will continue to prompt for patterns until Ctrl-C is pressed. Example:

```bash
$ validate-patterns

Enter a pattern to validate: [file-object:hashes.md5 = '79054025255fb1a26e4bc422aef54eb4']

PASS: [file-object:hashes.md5 = '79054025255fb1a26e4bc422aef54eb4']
```

### File Input

```bash
$ validate-patterns -f <path_to_file>
```

Use \<path\_to\_file> to specify the path to a file containing a set of
patterns to validate. Each pattern must be on a separate line of the file so
that the validator may determine where the pattern begins and ends. The
validator will supply the PASS/FAIL result of each pattern.

## Testing

The STIX Pattern Validator's test suite can be run with
[pytest](http://pytest.org).

You can also test against the examples provided in the supplied `examples.txt`
file.

```bash
$ validate-patterns -f stix2patterns/test/spec_examples.txt
```

## Updating the Grammar

If you modify STIXGrammar.g4, you'll need to use the Java ANTLR package to generate new Python source files (the ANTLR .jar file is not needed for normal use of the validator).

1. Download antlr-4.5.3-complete.jar from http://www.antlr.org/download/
2. From the `grammars` directory, run

    ```bash
    $ java -cp "/path/to/antlr-4.5.3-complete.jar" -Xmx500M org.antlr.v4.Tool -Dlanguage=Python2 STIXPattern.g4
    ```

## Governance

This GitHub public repository ( **[https://github.com/oasis-open/cti-pattern-validator](https://github.com/oasis-open/cti-pattern-validator)** ) was [proposed](https://lists.oasis-open.org/archives/cti/201609/msg00001.html) and [approved](https://www.oasis-open.org/committees/ballot.php?id=2971) [[bis](https://issues.oasis-open.org/browse/TCADMIN-2431)] by the [OASIS Cyber Threat Intelligence (CTI) TC](https://www.oasis-open.org/committees/cti/) as an [OASIS Open Repository](https://www.oasis-open.org/resources/open-repositories/) to support development of open source resources related to Technical Committee work.

While this Open Repository remains associated with the sponsor TC, its development priorities, leadership, intellectual property terms, participation rules, and other matters of governance are [separate and distinct](https://github.com/oasis-open/cti-pattern-validator/blob/master/CONTRIBUTING.md#governance-distinct-from-oasis-tc-process) from the OASIS TC Process and related policies.<commit_msg>Update README since grammar was moved.<commit_after># cti-pattern-validator

*This is an [OASIS Open Repository](https://www.oasis-open.org/resources/open-repositories/). See the [Governance](#governance) section for more information.*

The STIX 2 Pattern Validator is a software tool for checking the syntax of the Cyber Threat Intelligence (CTI) STIX Pattern expressions, which are used within STIX to express conditions (represented with the Cyber Observable data model) that indicate particular cyber threat activity. The repository contains source code, an ANTLR grammar, automated tests and associated documentation for the tool. The validator can be used as a command-line tool or as a Python library which can be included in other applications.

[![Travis-CI Build Status](https://api.travis-ci.org/oasis-open/cti-pattern-validator.svg?branch=master)](https://travis-ci.org/oasis-open/cti-pattern-validator)

## Requirements

- [Python](https://www.python.org) 2.7, 3.3, 3.4, or 3.5
- ANTLR grammar runtime (4.5.3 or newer):
    - [antlr4-python2-runtime](https://pypi.python.org/pypi/antlr4-python2-runtime) (Python 2.7)
    - [antlr4-python3-runtime](https://pypi.python.org/pypi/antlr4-python3-runtime) (Python 3)
- [enum34](https://pypi.python.org/pypi/enum34) (Python 3.3)
- [six](https://pypi.python.org/pypi/six)

## Installation

Using [pip](https://pip.pypa.io) is highly recommended:

```bash
$ pip install stix2-patterns
```

For more information about installing Python packages, see the [Python
Packaging User Guide](https://packaging.python.org/installing/).

## Usage

The STIX Pattern Validator provides an executable script (`validate-patterns`)
in addition to being an importable Python library.

The `validate-patterns` script accepts patterns from either direct user input
or a file passed as an option.

### From Python Code

The `run_validator` function can be called on any Python string. It returns a
list of errors encountered while parsing the pattern.

```python
from stix2patterns.validator import run_validator

pattern = "[file-object:hashes.md5 = '79054025255fb1a26e4bc422aef54eb4']"
errors = run_validator(pattern)
```

### User Input

When prompted, enter a pattern to validate and press enter. The validator will
supply whether the pattern has passed or failed. If the pattern fails the test,
the validator will supply where the first syntax error occurred. The validator
will continue to prompt for patterns until Ctrl-C is pressed. Example:

```bash
$ validate-patterns

Enter a pattern to validate: [file-object:hashes.md5 = '79054025255fb1a26e4bc422aef54eb4']

PASS: [file-object:hashes.md5 = '79054025255fb1a26e4bc422aef54eb4']
```

### File Input

```bash
$ validate-patterns -f <path_to_file>
```

Use \<path\_to\_file> to specify the path to a file containing a set of
patterns to validate. Each pattern must be on a separate line of the file so
that the validator may determine where the pattern begins and ends. The
validator will supply the PASS/FAIL result of each pattern.

## Testing

The STIX Pattern Validator's test suite can be run with
[pytest](http://pytest.org).

You can also test against the examples provided in the supplied `examples.txt`
file.

```bash
$ validate-patterns -f stix2patterns/test/spec_examples.txt
```

## Updating the Grammar

The ANTLR pattern grammar is maintained in the
[stix2-json-schemas](https://github.com/oasis-open/cti-stix2-json-schemas/blob/master/pattern_grammar/STIXPattern.g4)
repository. If the grammar changes, the code in this repository should be
updated to match. To do so, use the Java ANTLR package to generate new Python
source files. (The .jar file is not needed for normal use of the validator).

1. Download antlr-4.5.3-complete.jar from http://www.antlr.org/download/
2. From inside the `grammars` directory, run

    ```bash
    $ java -cp "/path/to/antlr-4.5.3-complete.jar" -Xmx500M org.antlr.v4.Tool -Dlanguage=Python2 /path/to/STIXPattern.g4
    ```

## Governance

This GitHub public repository ( **[https://github.com/oasis-open/cti-pattern-validator](https://github.com/oasis-open/cti-pattern-validator)** ) was [proposed](https://lists.oasis-open.org/archives/cti/201609/msg00001.html) and [approved](https://www.oasis-open.org/committees/ballot.php?id=2971) [[bis](https://issues.oasis-open.org/browse/TCADMIN-2431)] by the [OASIS Cyber Threat Intelligence (CTI) TC](https://www.oasis-open.org/committees/cti/) as an [OASIS Open Repository](https://www.oasis-open.org/resources/open-repositories/) to support development of open source resources related to Technical Committee work.

While this Open Repository remains associated with the sponsor TC, its development priorities, leadership, intellectual property terms, participation rules, and other matters of governance are [separate and distinct](https://github.com/oasis-open/cti-pattern-validator/blob/master/CONTRIBUTING.md#governance-distinct-from-oasis-tc-process) from the OASIS TC Process and related policies.
{'fraction_non_alphanumeric': 0.1082030003191829, 'fraction_numerical': 0.025853814235556973, 'mean_word_length': 4.724725943970768, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 37, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7498604', 'n_tokens_mistral': 3072, 'n_tokens_neox': 2844, 'n_words': 962}