<commit_before># IMDB-Scrape

Gathers information about movies stored in a local directory and compiles it into a searchable and sortable HTML document

## Demo

Script output loaded with some randomly chosen sample data is available [here](http://www.jasonwillems.com/sites/mediadatabase/output/). I've tried to include several foreign titles to ensure that they're tagged correctly. Commit used for these reports is a few behind.

## Screenshots

#### List of Content
![Screenshot](http://at1as.github.io/github_repo_assets/imdb-scrape.jpg)
#### Movie Details
![Screenshot](http://at1as.github.io/github_repo_assets/imdb-scrape2.jpg)


## Usage

Download the repository here, set the appropriate permissions and launch IMDB-Scrape via:
```bash
$ git clone https://github.com/at1as/IMDB-Scrape.git
$ cd IMDB-Scrape
$ vim conf.json
$ python retriever.py
```

##### Environment Configuration

Set your environment in conf.json:
* *max_quantity* [Integer] => the maximum number of entries to retrieve from IMDB
* *asset_location* [String] => the absolute path to the folder containing the files
* *exclude_files* [Array of Strings] => Exclude these files from retrieval (even if their extensions are included)
* *include_extensions* [Array of Strings] => files will these extensions (or folders) will be retrieved from IMDB. These should all be lower case

##### Removing Entries

For items tagged incorrectly, remove them using the bundled script
```bash
$ python remove_entry.py --movie "<movie_title>"
$ python remove_entry.py --series "<movie_title>"
```<commit_msg>Restrict image width<commit_after># IMDB-Scrape

Gathers information about movies stored in a local directory and compiles it into a searchable and sortable HTML document

## Demo

Script output loaded with some randomly chosen sample data is available [here](http://www.jasonwillems.com/sites/mediadatabase/output/). I've tried to include several foreign titles to ensure that they're tagged correctly. Commit used for these reports is a few behind.

## Screenshots

#### List of Content
<img src="http://at1as.github.io/github_repo_assets/imdb-scrape.jpg" width="600px">
#### Movie Details
<img src="http://at1as.github.io/github_repo_assets/imdb-scrape2.jpg" width="600px">


## Usage

Download the repository here, set the appropriate permissions and launch IMDB-Scrape via:
```bash
$ git clone https://github.com/at1as/IMDB-Scrape.git
$ cd IMDB-Scrape
$ vim conf.json
$ python retriever.py
```

##### Environment Configuration

Set your environment in conf.json:
* *max_quantity* [Integer] => the maximum number of entries to retrieve from IMDB
* *asset_location* [String] => the absolute path to the folder containing the files
* *exclude_files* [Array of Strings] => Exclude these files from retrieval (even if their extensions are included)
* *include_extensions* [Array of Strings] => files will these extensions (or folders) will be retrieved from IMDB. These should all be lower case

##### Removing Entries

For items tagged incorrectly, remove them using the bundled script
```bash
$ python remove_entry.py --movie "<movie_title>"
$ python remove_entry.py --series "<movie_title>"
```
{'fraction_non_alphanumeric': 0.10110584518167456, 'fraction_numerical': 0.0044233807266982625, 'mean_word_length': 5.14757281553398, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6657779', 'n_tokens_mistral': 946, 'n_tokens_neox': 887, 'n_words': 352}