<commit_before># yara-rule-porter

yara-rule-porter is a package to transform yara rules.

It reads in yara rules from files or data bases, parses them and applies a set of normalization or transformation scripts.

After that it will export the rules to files or data bases again.

It is written in Perl and uses the great Yara Parser [Parse::YARA][http://search.cpan.org/~moofu/Parse-YARA-0.02/lib/Parse/YARA.pm] written by <NAME>

# Installation

download the source and unpack it into a directory of your choice.  There are no external dependencies (Parse::YARA is already included in the package).

# Usage
	usage: bin/dedupe.pl [options] file [dir ...]
	
	parses, im- and exports yara rules from different places
	
	options:
	--help                this help text
	--debug               show what's going on
	--verbose             even more information
	--include pattern     regular expression of filenames to include
	                      can be given multiple times
	                      default: include everything
	 --exclude pattern    regular expression of filenames to exclude
	                      can be given multiple times
	                      default: exclude nothing

# Example

## reading in a directory tree of yara rules and report duplicate rule names

    # only read *.yar files, starting recursively at ../data/test/
    $ bin/dedupe.pl --include '\.yar$' ../data/test/
    duplicate rule:magic_bytes file:../data/test/magic.yar

# TODOs
* implement data base im-/export
* implement pre-/post normalization scripts
* fix string modifier bug ```tests/01_string_modifier.yar<commit_msg>* markdown fixes<commit_after># yara-rule-porter

yara-rule-porter is a package to transform yara rules.

It reads in yara rules from files or databases, parses them and applies a set of normalization or transformation scripts.

After that it will export the rules to files or databases again.

It is written in Perl and uses the Yara Parser [Parse::YARA](http://search.cpan.org/~moofu/Parse-YARA-0.02/lib/Parse/YARA.pm) written by <NAME>.

# Installation

download the source and unpack it into a directory of your choice.  There are no external dependencies (Parse::YARA is already included in the package).

# Usage
	usage: bin/dedupe.pl [options] file [dir ...]
	
	parses, im- and exports yara rules from different places
	
	options:
	--help                this help text
	--debug               show what's going on
	--verbose             even more information
	--include pattern     regular expression of filenames to include
	                      can be given multiple times
	                      default: include everything
	 --exclude pattern    regular expression of filenames to exclude
	                      can be given multiple times
	                      default: exclude nothing

# Example

* reading in a directory tree of yara rules and report duplicate rule names

    # only read *.yar files, starting recursively at ../data/test/
    $ bin/dedupe.pl --include '\.yar$' ../data/test/
    duplicate rule:magic_bytes file:../data/test/magic.yar

# TODOs
* implement database im-/export
* implement pre-/post normalization scripts
* fix string modifier bug
{'fraction_non_alphanumeric': 0.07210031347962383, 'fraction_numerical': 0.0025078369905956114, 'mean_word_length': 2.6220204313280364, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2571831', 'n_tokens_mistral': 881, 'n_tokens_neox': 814, 'n_words': 372}