<commit_before># Setup the temporary crawler

1. Setup a machine with Ubuntu installed. For example, by creating an AWS instance.

 **NOTE:** make sure the machine has sufficient disk space. For example, if using AWS, you will need to either create the instance with a large root partition or mount additional storage and use symlinks to get Postgresql to use this storage instead of the root partition.

2. **Make sure that the time zone of your machine is set to Eastern Time.** Run `date`, if the result isn't EDT or EST, run `sudo dpkg-reconfigure tzdata` and select America/New_York.

 **Be aware that AWS machines are often initially setup with UTC as the timezone instead of Eastern.**

3. Install Media Cloud on the machine.

4. cd to the Media Cloud installation directory

5. Import media and feeds from the production server by running `python python_scripts/media_import.py --api-key API-KEY` where API-KEY is an API key for mediacloud.org.

6. Start supvisisor by running `./supervisor/supervisord.sh`

7. Optional: Stop the extractor workers by running `./supervisor/supervisorctl.sh stop extract_and_vector`

8. Let the crawler on this machine run as long as desired.

# Exporting

When you're ready to export:
<commit_msg>Update the temporary crawler instructions to mention installing the Media Cloud Python API Client.<commit_after># Setup the temporary crawler

1. Setup a machine with Ubuntu installed. For example, by creating an AWS instance.

 **NOTE:** make sure the machine has sufficient disk space. For example, if using AWS, you will need to either create the instance with a large root partition or mount additional storage and use symlinks to get Postgresql to use this storage instead of the root partition.

2. **Make sure that the time zone of your machine is set to Eastern Time.** Run `date`, if the result isn't EDT or EST, run `sudo dpkg-reconfigure tzdata` and select America/New_York.

 **Be aware that AWS machines are often initially setup with UTC as the timezone instead of Eastern.**

3. Install Media Cloud on the machine.

5. Install the Media Cloud API Python client from https://github.com/c4fcm/MediaCloud-API-Client

5. cd to the Media Cloud installation directory

6. Import media and feeds from the production server by running `python python_scripts/media_import.py --api-key API-KEY` where API-KEY is an API key for mediacloud.org.

7. Start supvisisor by running `./supervisor/supervisord.sh`

8. Optional: Stop the extractor workers by running `./supervisor/supervisorctl.sh stop extract_and_vector`

9. Let the crawler on this machine run as long as desired.

# Exporting

When you're ready to export:

{'fraction_non_alphanumeric': 0.05866867243324558, 'fraction_numerical': 0.006769462203836028, 'mean_word_length': 4.2362204724409445, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '507197', 'n_tokens_mistral': 711, 'n_tokens_neox': 679, 'n_words': 395}