<commit_before># regex


## regex plugin

There are many plugins but we expect the Regex  Plugin to be one of the most popular. Here we create a set of expressions (terms) that we wish to search for. Here we use a small subset of the terms used in clinical trials (`consort0`):

## typical file
Here are a few from the file. Comments are inserted after each line...
```
<compoundRegex title="consort0">
```
There can be many `<regex>`es, and all are bracketed in `<compoundRegex>` ... `</compoundRegex>`. The `title0` is used to annotate the output (e.g. naming the output directory.
```
  <regex fields="stata">Stata</regex>
```
This is the simplest example. There must be an exact match with the string `Stata`. Note that this is case-sensitive by default.
```
  <regex fields="samplesize">sample size</regex>
```
This is a case-sensitive two-word phrase. At present there must be exactly one space in the target but we are working towards making it more general.
```
  <regex fields="followup">follow\-?up</regex>
```
This matches `follow-up` or `followup`. The `?` means optional. Because `-` has a special meaning, we "escape" it by a preceding backslash.
```
  <regex fields="exclude">exclud~</regex>
```
This is a special ContentMine extension. `~` is a common lexicographical device which means "any suffix", so `excludes`, `excluding`, `exclude` but not `exclusion`. [Note 1]
```
  <regex fields="withdraw">withdr[ae]w~</regex>
```
The `[xyz]` construct is common and means either x or y or z. Here it means "a" or "e" followed by "w" and then any suffix, so manages `withdrawal` `withdrew` , `withdrawing`.
```
  <regex fields="random">[Rr]andom~</regex>
```
A combination of `Random` and `random` followed by an optional suffix.
</compoundRegex>...
```
Regexes can get very complex and unreadable, and they are also modified by the system so you are well advised to start with exact words, test wherever possible and only add new constructs when necessary. If you aren't sure, put in all the optionas explicitly , e.g. :
```
  <regex fields="random">random</regex>
  <regex fields="random">randomization)</regex>
  <regex fields="random">randomisation)</regex>
  <regex fields="random">randomised)</regex>
```


## running the examples

The examples can be run from the commandline in the Debian environment 

```<commit_msg>Add missing ``` which was screwing up the rest of REGEX.md<commit_after># regex


## regex plugin

There are many plugins but we expect the Regex  Plugin to be one of the most popular. Here we create a set of expressions (terms) that we wish to search for. Here we use a small subset of the terms used in clinical trials (`consort0`):

## typical file
Here are a few from the file. Comments are inserted after each line...
```
<compoundRegex title="consort0">
```
There can be many `<regex>`es, and all are bracketed in `<compoundRegex>` ... `</compoundRegex>`. The `title0` is used to annotate the output (e.g. naming the output directory.
```
  <regex fields="stata">Stata</regex>
```
This is the simplest example. There must be an exact match with the string `Stata`. Note that this is case-sensitive by default.
```
  <regex fields="samplesize">sample size</regex>
```
This is a case-sensitive two-word phrase. At present there must be exactly one space in the target but we are working towards making it more general.
```
  <regex fields="followup">follow\-?up</regex>
```
This matches `follow-up` or `followup`. The `?` means optional. Because `-` has a special meaning, we "escape" it by a preceding backslash.
```
  <regex fields="exclude">exclud~</regex>
```
This is a special ContentMine extension. `~` is a common lexicographical device which means "any suffix", so `excludes`, `excluding`, `exclude` but not `exclusion`. [Note 1]
```
  <regex fields="withdraw">withdr[ae]w~</regex>
```
The `[xyz]` construct is common and means either x or y or z. Here it means "a" or "e" followed by "w" and then any suffix, so manages `withdrawal` `withdrew` , `withdrawing`.
```
  <regex fields="random">[Rr]andom~</regex>
```
A combination of `Random` and `random` followed by an optional suffix.
```
</compoundRegex>
```
Regexes can get very complex and unreadable, and they are also modified by the system so you are well advised to start with exact words, test wherever possible and only add new constructs when necessary. If you aren't sure, put in all the optionas explicitly , e.g. :
```
  <regex fields="random">random</regex>
  <regex fields="random">randomization)</regex>
  <regex fields="random">randomisation)</regex>
  <regex fields="random">randomised)</regex>
```


## running the examples

The examples can be run from the commandline in the Debian environment 

```
{'fraction_non_alphanumeric': 0.12208562950402713, 'fraction_numerical': 0.001695633743111488, 'mean_word_length': 4.604513064133017, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6458646', 'n_tokens_mistral': 1413, 'n_tokens_neox': 1320, 'n_words': 625}