<commit_before>This document describes the architecture of Goim and some key design decisions 
for the database.


### Overview

The purpose of the main `goim` command (the `main` package) is to provide a 
command line interface for managing a Goim database. This includes searching 
and updating. It also provides some convenience commands like `rename` that use 
IMDb data to help you manage your media.

The logic for searching and querying the database is contained inside the 
`imdb` and `imdb/search` sub-packages. The logic for parsing IMDb lists and 
updating the database is all inside the `main` package. Stated differently, the 
`imdb` and `imdb/search` packages do not contain any logic for adding data to 
the Goim database. All operations that update the database are done in 
transactions so that if they fail, the original data should be preserved.

The `imdb` sub-package contains schema definitions and indices for both SQLite 
and PostgreSQL. Opening a Goim databse through the `imdb` package automatically 
makes sure the schema is up-to-date with the package version.


### Database

The design of the database is very simple. In general, there are three types of 
tables:

  - Tables for storing identifiers/names (e.g., atom, name)
  - Tables for storing entities (e.g., movie, tvshow, episode, actor)
  - Tables for storing attributes about entities (e.g., plot, goof)

All entity and attribute tables are completely wiped whenever they are updated. 
The atom and name tables are not. This can result in stale but benign rows in 
the atom and name tables, but it preserves primary keys across updates.

All rows in all tables (sans migration meta data) are immutable.


### Unique identifiers

The data from IMDb does not include any numeric unique identifiers for each 
movie, TV show, episode, actor, etc. While such identifiers clearly exist on 
their web site, this data is explicitly not included in the "alternative 
interfaces" plain text data dump.

Instead, there appears to be an undocumented guarantee that the name of an 
entity, combined with some miscellaneous attributes, will be unique. For 
example, the full unique string for The Matrix is:

    The Matrix (1999)

This string is then used in other list files to identify this movie uniquely. 
The string may include attributes like "(TV)" or "(V)" for "made for TV" and 
"made for video", respectively.

In the rare case where two movies share the same name and were released in the 
same year, a roman numeral indicates uniqueness:

    The Maury Island Incident (2014/I)
    The Maury Island Incident (2014/II)

These unique strings also apply to TV shows, episodes, actors, etc. For 
example, the full unique string for The Simpsons episode "<NAME>" 
is:
<commit_msg>silly imdb<commit_after>This document describes the architecture of Goim and some key design decisions 
for the database.


### Overview

The purpose of the main `goim` command (the `main` package) is to provide a 
command line interface for managing a Goim database. This includes searching 
and updating. It also provides some convenience commands like `rename` that use 
IMDb data to help you manage your media.

The logic for searching and querying the database is contained inside the 
`imdb` and `imdb/search` sub-packages. The logic for parsing IMDb lists and 
updating the database is all inside the `main` package. Stated differently, the 
`imdb` and `imdb/search` packages do not contain any logic for adding data to 
the Goim database. All operations that update the database are done in 
transactions so that if they fail, the original data should be preserved.

The `imdb` sub-package contains schema definitions and indices for both SQLite 
and PostgreSQL. Opening a Goim databse through the `imdb` package automatically 
makes sure the schema is up-to-date with the package version.


### Database

The design of the database is very simple. In general, there are three types of 
tables:

  - Tables for storing identifiers/names (e.g., atom, name)
  - Tables for storing entities (e.g., movie, tvshow, episode, actor)
  - Tables for storing attributes about entities (e.g., plot, goof)

All entity and attribute tables are completely wiped whenever they are updated. 
The atom and name tables are not. This can result in stale but benign rows in 
the atom and name tables, but it preserves primary keys across updates.

All rows in all tables (sans migration meta data) are immutable.


### Unique identifiers

The data from IMDb does not include any numeric unique identifiers for each 
movie, TV show, episode, actor, etc. While such identifiers clearly exist on 
their web site, this data is explicitly not included in the "alternative 
interfaces" plain text data dump. Moreover, IMDb's terms of use explicitly
forbids trying to recover this data.

Instead, there appears to be an undocumented guarantee that the name of an 
entity, combined with some miscellaneous attributes, will be unique. For 
example, the full unique string for The Matrix is:

    The Matrix (1999)

This string is then used in other list files to identify this movie uniquely. 
The string may include attributes like "(TV)" or "(V)" for "made for TV" and 
"made for video", respectively.

In the rare case where two movies share the same name and were released in the 
same year, a roman numeral indicates uniqueness:

    The Maury Island Incident (2014/I)
    The Maury Island Incident (2014/II)

These unique strings also apply to TV shows, episodes, actors, etc. For 
example, the full unique string for The Simpsons episode "<NAME>" 
is:

{'fraction_non_alphanumeric': 0.04909837529012676, 'fraction_numerical': 0.004284949116229245, 'mean_word_length': 4.191844300278035, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3030481', 'n_tokens_mistral': 1489, 'n_tokens_neox': 1407, 'n_words': 863}