<commit_before># Setup

This script requires Python 3.5 and various packages/libraries, along with the following [Jim Kent binaries][jksrc]:

- `bedToBigBed`
- `fetchChromSizes`

[jksrc]: http://hgdownload.cse.ucsc.edu/admin/exe/

To setup the Python environment you can use `conda` or `virtualenv` + `pip`.

## Using `conda`

On most computers (particularly Macs) it is easiest to install [Anaconda](https://www.continuum.io/downloads), and then:

    $ conda create --name track_scraper --file conda-requirements.txt
    $ source activate track_scraper
    $ python get_tracks.py

## Using `virtualenv` and `pip`

1. You need Python 3.5 and libxml >2.9.x to be installed and accessible on PATH. If you're on Minerva, `source minerva-modules.sh` will load them.

2. To set up a virtual environment, run

       $ virtualenv -p \`which python3\` venv

3. Run `source bin/env/activate` to activate this environment. You should now see `(venv)` at the beginning of your prompt.

4. Run `pip install -r requirements.txt` to locally install required packages.

*Important:* Re-run steps (1) and (3) in any new shell to re-activate the virtualenv before you use this script.

# Usage

`cd` into this directory, activate the environment as necessary if you used `virtualenv`, and 

    $ python get_tracks.py

If you only want to fetch tracks for organism databases matching a prefix, e.g. `hg38`,

    $ python get_tracks.py --org_prefix=hg38<commit_msg>fix formatting<commit_after># Setup

This script requires Python 3.5 and various packages/libraries, along with the following [Jim Kent binaries][jksrc]:

- `bedToBigBed`
- `fetchChromSizes`

[jksrc]: http://hgdownload.cse.ucsc.edu/admin/exe/

To setup the Python environment you can use `conda` or `virtualenv` + `pip`.

## Using `conda`

On most computers (particularly Macs) it is easiest to install [Anaconda](https://www.continuum.io/downloads), and then:

    $ conda create --name track_scraper --file conda-requirements.txt
    $ source activate track_scraper
    $ python get_tracks.py

## Using `virtualenv` and `pip`

1. You need Python 3.5 and libxml >2.9.x to be installed and accessible on PATH. If you're on Minerva, `source minerva-modules.sh` will load them.

2. To set up a virtual environment, run

            $ virtualenv -p \`which python3\` venv

3. Run `source bin/env/activate` to activate this environment. You should now see `(venv)` at the beginning of your prompt.

4. Run `pip install -r requirements.txt` to locally install required packages.

*Important:* Re-run steps (1) and (3) in any new shell to re-activate the virtualenv before you use this script.

# Usage

`cd` into this directory, activate the environment as necessary if you used `virtualenv`, and 

    $ python get_tracks.py

If you only want to fetch tracks for organism databases matching a prefix, e.g. `hg38`,

    $ python get_tracks.py --org_prefix=hg38
{'fraction_non_alphanumeric': 0.09641873278236915, 'fraction_numerical': 0.011707988980716254, 'mean_word_length': 3.8336106489184694, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1738319', 'n_tokens_mistral': 941, 'n_tokens_neox': 883, 'n_words': 368}