<commit_before># Files
"""
|-- download_scripts - folder for scripts called by download_raw_data.py
|   +-- *.py
|
|-- clean_scripts - folder for scripts called by clean_data.py
|   +-- *.py
|
|-- download_raw_data.py - script to download and update raw data  
|-- clean_data.py - script to clean raw data into interim folder
|-- preprocess_data.py - script to process the data into processed folder for visualization and analysis
"""
# Steps to run:

1. python download_raw_data.py
2. python clean_data.py
3. python preprocess_data.py

# Processed Data Structure (End goal for data scripting)
## Weekly
| location    | year   | week | dengue | malaria | MaxTempC | MeanTempC | MinTempC | DewPointC | ... etc.
| Singapore | 2012 | 1       | 1          | 0          | 32             | 30               | 28             | 20            | ... etc.

## Yearly
| location    | year   | dengue | malaria | MaxTempC | MeanTempC | MinTempC | DewPointC | ... etc.
| Singapore | 2012 | 100       | 10        | 32             | 30               | 28             | 20            | ... etc.

If data does not exist for that location (e.g. weekly Malaysia malaria), it will not be recorded as an observation in the table (e.g. weekly table).<commit_msg>put indent before folders structure<commit_after># Files
"""
  |-- download_scripts - folder for scripts called by download_raw_data.py
  |   +-- *.py
  |
  |-- clean_scripts - folder for scripts called by clean_data.py
  |   +-- *.py
  |
  |-- download_raw_data.py - script to download and update raw data  
  |-- clean_data.py - script to clean raw data into interim folder
  |-- preprocess_data.py - script to process the data into processed folder for visualization and analysis
"""
# Steps to run:

1. python download_raw_data.py
2. python clean_data.py
3. python preprocess_data.py

# Processed Data Structure (End goal for data scripting)
## Weekly
| location    | year   | week | dengue | malaria | MaxTempC | MeanTempC | MinTempC | DewPointC | ... etc.
| Singapore | 2012 | 1       | 1          | 0          | 32             | 30               | 28             | 20            | ... etc.

## Yearly
| location    | year   | dengue | malaria | MaxTempC | MeanTempC | MinTempC | DewPointC | ... etc.
| Singapore | 2012 | 100       | 10        | 32             | 30               | 28             | 20            | ... etc.

If data does not exist for that location (e.g. weekly Malaysia malaria), it will not be recorded as an observation in the table (e.g. weekly table).
{'fraction_non_alphanumeric': 0.10166799046862589, 'fraction_numerical': 0.02779984114376489, 'mean_word_length': 2.0240096038415367, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7276292', 'n_tokens_mistral': 801, 'n_tokens_neox': 703, 'n_words': 271}