<commit_before># deidentify
Deidentifies LABS consortium data<commit_msg>adds usage to readme<commit_after># deidentify

Tools for deidentifying LABS consortium data.

`date_eliminator.py` eliminates date fields from a directory of CSV files according to user input. We applied this script to LABS consortium data (in particular, the ASCII subdirectories from the LABS 2 CD) to remove surgery operation dates, which generally had the label `SURGDATE`. On a first run, this script looks in each CSV for every column that has at least one date and asks the user whether that column should be eliminated. Its output is a new directory of CSVs with the selected dates removed as well as a configuration file that allows reproducing the run.

Usage:
```
python date_eliminator.py -i /path/to/input/directory -o /path/to/output/directory
```

To reproduce our run, cat the configuration file "date_eliminator.conf" from this directory into the script, as in
```
cat date_eliminator.conf | python date_eliminator.py \
    -i "/path/to/Longitudinal Assessment of Bariatric Surgery (LABS-=2) Preliminary/ASCII Database" \
    -o /path/to/output/directory
```

# License
This software is licensed under the MIT License. See LICENSE for details.
{'fraction_non_alphanumeric': 0.06158833063209076, 'fraction_numerical': 0.0016207455429497568, 'mean_word_length': 4.613636363636363, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '693295', 'n_tokens_mistral': 352, 'n_tokens_neox': 324, 'n_words': 164}