<commit_before>## Loofah

A Javascript library that scrubs data.

## Introduction

```javascript
Scrubbers = require 'lib/loofah'
```
You often need to scrub sensitive information from data before moving it to an insecure location, or publishing it. For example, you may want to publish errors on [Sentry](https://app.getsentry.com/) and before doing this, you should ensure that you are not exposing fields like passwords, api keys or usernames. Loofah provides a number of easily extensible and configurable helper functions to remove these. For example, to get an object where the value is redacted for any keys that are either `password` or `secret` you would call.

```javascript
clean_object = Scrubbers.bad_keys(['password', 'secret'])(object)
```

## Library Functions


The loofah library provides a number of helper functions or scrubbers which can operate on strings and objects. Any other data type will be returned unchanged. To use one of these functions, call it with a list of keywords and then call the result of this on the data you want to scrub. 

```javascript
clean_object = Scrubbers.function_name([keywords])(object)
```

The keywords specify search terms and can either be regular expressions or strings. Some functions will scrub the match (bad_vals) while others expect the search term to be a key, and will scrub the value that follows. Strings will be converted to regular expressions by the scrubber. The Regex will generally be case sensitive and need to match an entire key.

### bad_keys
If passed an object, redacts all information stored in a key that matches one of the keywords.

```javascript
Scrubbers.bad_keys(['secret', 'password'])({password: { a: '<PASSWORD>', b: '<PASSWORD>'}, secrets: 'shhh'})
// {password: { a: '[REDACTED]', b: '[REDACTED]'}, secrets: 'shhh'} // secrets is not matched
```
If not passed an object, `bad_keys` returns what it was given.

### bad_vals
Redacts all substrings that match one of the keywords. Defaults to case sensitive regex.

```javascript
Scrubbers.bad_vals(['thisIsOurApiKey'])( 'Don't steal our thisIsOurApiKey')
// Don't steal our [REDACTED]
```


### url_encode
Redacts the value of a url encoded `'<key>=<value>'` pair where the key matches one of the keywords.

```javascript<commit_msg>Correct comments showed up quote errors<commit_after>## Loofah

A Javascript library that scrubs data.

## Introduction

```javascript
Scrubbers = require 'lib/loofah'
```
You often need to scrub sensitive information from data before moving it to an insecure location, or publishing it. For example, you may want to publish errors on [Sentry](https://app.getsentry.com/) and before doing this, you should ensure that you are not exposing fields like passwords, api keys or usernames. Loofah provides a number of easily extensible and configurable helper functions to remove these. For example, to get an object where the value is redacted for any keys that are either `password` or `secret` you would call.

```javascript
clean_object = Scrubbers.bad_keys(['password', 'secret'])(object)
```

## Library Functions


The loofah library provides a number of helper functions or scrubbers which can operate on strings and objects. Any other data type will be returned unchanged. To use one of these functions, call it with a list of keywords and then call the result of this on the data you want to scrub. 

```javascript
clean_object = Scrubbers.function_name([keywords])(object)
```

The keywords specify search terms and can either be regular expressions or strings. Some functions will scrub the match (bad_vals) while others expect the search term to be a key, and will scrub the value that follows. Strings will be converted to regular expressions by the scrubber. The Regex will generally be case sensitive and need to match an entire key.

### bad_keys
If passed an object, redacts all information stored in a key that matches one of the keywords.

```javascript
Scrubbers.bad_keys(['secret', 'password'])({password: { a: '<PASSWORD>', b: '<PASSWORD>'}, secrets: 'shhh'})
// {password: { a: '[REDACTED]', b: '[REDACTED]'}, secrets: 'shhh'} // secrets is not matched
```
If not passed an object, `bad_keys` returns what it was given.

### bad_vals
Redacts all substrings that match one of the keywords. Defaults to case sensitive regex.

```javascript
Scrubbers.bad_vals(['thisIsOurApiKey'])( "Don't steal our thisIsOurApiKey")
// Don't steal our [REDACTED]
```


### url_encode
Redacts the value of a url encoded `'<key>=<value>'` pair where the key matches one of the keywords.

```javascript
{'fraction_non_alphanumeric': 0.08836395450568679, 'fraction_numerical': 0.0, 'mean_word_length': 4.954427083333333, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4766010', 'n_tokens_mistral': 1233, 'n_tokens_neox': 1181, 'n_words': 632}