<commit_before># What it is

`parser.py` extracts Jeopardy! clues from the [J! Archive][1] website and dumps them into a SQLite database for use elsewhere. No particular application is intended, but a fair number of clues are skipped over when parsing. Any application requiring the complete set of clues will not find it here.

# Getting started

Clone the repo: `git clone git://github.com/whymarrh/jeopardy-parser.git`  
`cd jeopardy-parser`  
`chmod +x download.sh`  
`chmod +x parser.py`  
Run the download script and parser: `./download.sh && ./parser.py`)  

The "real" running time of the parsing script (i.e. `(time ./parser.py) 2>&1 | grep real`) should not be more than 30 minutes on a decent machine. I know even that is quite a lot, but there are **a lot** of clues to go through. With [FTS3][2] enabled, the resulting database is sized at around 70M.

# Regarding the initial download

The download script does not mirror the entire site, it simply gets the game pages. Also note that no media files are downloaded (e.g. pictures) and the parser does not account for media clues. The complete download of the pages (at the time of writing) is approx. 295M.

  [1]: http://j-archive.com/
  [2]: http://www.sqlite.org/fts3.html<commit_msg>Readme file.<commit_after># What is this

`parser.py` extracts Jeopardy! clues from the [J! Archive][1] website and dumps them into a SQLite database for use elsewhere. No particular application is intended, but a fair number of clues are skipped over when parsing. Any application requiring the complete set of clues will not find it here.

# Getting started

Clone the repo: `git clone git://github.com/whymarrh/jeopardy-parser.git`  
`cd jeopardy-parser`  
`chmod +x download.sh`  
`chmod +x parser.py`  
Run the download script and parser: `./download.sh && ./parser.py`)  

The "real" running time of the parsing script (i.e. `(time ./parser.py) 2>&1 | grep real`) should not be more than 30 minutes on a decent machine. I know even that is quite a lot, but there are **a lot** of clues to go through. With [FTS3][2] enabled, the resulting database is sized at around 70M.

# Regarding the initial download

The download script does not mirror the entire site, it simply gets the game pages. Also note that no media files are downloaded (e.g. pictures) and the parser does not account for media clues. The complete download of the pages (at the time of writing) is approx. 295M, and can take up to [6.5 hours][3] (1s per page, plus a 5s wait between downloads, 3970 times).

  [1]: http://j-archive.com/
  [2]: http://www.sqlite.org/fts3.html
  [3]:http://www.wolframalpha.com/input/?i=%281s+%2B+5s%29+*+3970
{'fraction_non_alphanumeric': 0.09572072072072071, 'fraction_numerical': 0.019144144144144143, 'mean_word_length': 3.9260628465804066, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '849061', 'n_tokens_mistral': 842, 'n_tokens_neox': 775, 'n_words': 391}