<commit_before># Scraping BoM

This repo contains scripts to get information from the BoM website.

Currently we only harvest the links we are looking to get data from.

Run script to get all links using

```bash
make getLinks
```

Assumes that you have make installed on your machine.<commit_msg>updated README to document current state of the project<commit_after># Scraping BoM

This repo contains scripts to get information from the BoM website.

Currently we only harvest the links we are looking to get data from.

Run script to get all links using

```bash
make getLinks
```

**TO DO:**

1. Use the links to get the summary data on movies
2. Use links to get box office data

**Currently Working On:** scraping the summary data

* *have* written the functions to collect the data
* *have not* written a script that runs the scraper on against the csvs that contain
 the links

Assumes that you have make installed on your machine.
{'fraction_non_alphanumeric': 0.04695837780149413, 'fraction_numerical': 0.0021344717182497333, 'mean_word_length': 4.240223463687151, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1130440', 'n_tokens_mistral': 252, 'n_tokens_neox': 240, 'n_words': 152}