<commit_before># iguana2 roadmap

A complete rewrite of [iguana](https://github.com/alecgorge/iguana) to be more abstracted and support a variety of sources. It also sets up data to be better prepared for future additions of users and statistic generation.

The primary technical goal to provide an abstraction over media sources such as [archive.org](http://archive.org), [panicstream.com](http://panicstream.com) and [phish.in](http://phish.in) and integrate metadata sources such as [setlist.fm](http://setlist.fm) or [phish.net](http://phish.net) to provide extended metadata or reviews. This is complicated by the fact that a media source may provide reliable metadata as well.

The ultimate functional goal of iguana2 is to allow me to unify the networking and playback codebases for PhishOD, Listen to the Dead and the upcoming Relisten app without losing any of the extra features [PhishOD](http://phishod.alecgorge.com) has because of much better data [phish.in](http://phish.in) and [phish.net](http://phish.net) provide.

## Definitions

Data types: boolean, date, datetime, int, float, varchar, text

All data structures have a unique, autoincrementing id and a created_at and updated_at field.

#### Data Collection Techniques

* `Phish`
  1. Import media and some metadata from [phish.in](http://phish.in)
  2. Import more metadata from [phish.net](http://phish.net)
  
* `ArchiveSetlistFm`
  1. Import setlist metadata from [setlist.fm](http://setlist.fm)
  2. Import media from [archive.org](http://archive.org) by matching the data in each source to the show metadata imported from `setlist.fm`. Show, venue and song data comes from setlist.fm, source data and track names come from archive.org but the setlist should be visible in the UI. **Report when a match is not found, with enough information to import that single item later after the algorithm has been tweaked**.
  
* `JustArchive` (when setlist.fm doesn't have good information for an artist, basically iguana 1.0)
	1. Import media and build setlist information based on [archive.org](http://archive.org). Only attempt to reuse venues when they belong specifically to that artist (setlist.fm venues are shared across artists).

* `WidespreadPanic`
	1. Import setlist data just like `ArchiveSetlistfm`.
	2. Import media from [panicstream.com](http://panicstream.com).

### Artist

Column Name | Type | Comments
:---------- | :--- | :-------
name | string
identifier | string | potentially used by the `data_source` to look up data appropriately
data_source | string | an identifier refering to one of the different data collection techniques (Phish, ArchiveSetlist, JustArchive, etc)
features | has_one **FeatureSet** | what kind of functionality can we expect the `data_source` to provide. This is mostly to guide the UI so it can be written in a general way

Relation Name | Type | Comments
:---------- | :--- | :-------
years | has_many **Year**
shows | has_many **Show**

### Year

Cached summerization data about each year of an artist.

Column Name | Type | Comments
:---------- | :--- | :-------
year | int 
show_count | int | **NOT** the number of recordings
recording_count | int
duration | The sum of the duration of each show in the year (use the longest source per show)
avg_duration | float | The average duration of each show in the year (use the longest source per show)
avg_rating | float | The average of all the average ratings for each show in the year

Relation Name | Type | Comments
:---------- | :--- | :-------
artist | belongs_to **Artist**
<commit_msg>update spc<commit_after># iguana2 roadmap

A complete rewrite of [iguana](https://github.com/alecgorge/iguana) to be more abstracted and support a variety of sources. It also sets up data to be better prepared for future additions of users and statistic generation.

The primary technical goal to provide an abstraction over media sources such as [archive.org](http://archive.org), [panicstream.com](http://panicstream.com) and [phish.in](http://phish.in) and integrate metadata sources such as [setlist.fm](http://setlist.fm) or [phish.net](http://phish.net) to provide extended metadata or reviews. This is complicated by the fact that a media source may provide reliable metadata as well.

The ultimate functional goal of iguana2 is to allow me to unify the networking and playback codebases for PhishOD, Listen to the Dead and the upcoming Relisten app without losing any of the extra features [PhishOD](http://phishod.alecgorge.com) has because of much better data [phish.in](http://phish.in) and [phish.net](http://phish.net) provide.

## Definitions

Data types: boolean, date, datetime, int, float, varchar, text

All data structures have a unique, autoincrementing id and a created_at and updated_at field.

#### Data Collection Techniques

* `Phish`
  1. Import media and some metadata from [phish.in](http://phish.in)
  2. Import more metadata from [phish.net](http://phish.net)
  
* `ArchiveSetlistFm`
  1. Import setlist metadata from [setlist.fm](http://setlist.fm)
  2. Import media from [archive.org](http://archive.org) by matching the data in each source to the show metadata imported from `setlist.fm`. Show, venue and song data comes from setlist.fm, source data and track names come from archive.org but the setlist should be visible in the UI. **Report when a match is not found, with enough information to import that single item later after the algorithm has been tweaked**.
  
* `JustArchive` (when setlist.fm doesn't have good information for an artist, basically iguana 1.0)
	1. Import media and build setlist information based on [archive.org](http://archive.org). Only attempt to reuse venues when they belong specifically to that artist (setlist.fm venues are shared across artists).
	
* `WidespreadPanic`
	1. Import setlist data just like `ArchiveSetlistfm`.
	2. Import media from [panicstream.com](http://panicstream.com).

### Artist

Column Name | Type | Comments
:---------- | :--- | :-------
name | string
identifier | string | potentially used by the `data_source` to look up data appropriately
data_source | string | an identifier refering to one of the different data collection techniques (Phish, ArchiveSetlist, JustArchive, etc)

Relation Name | Type | Comments
:---------- | :--- | :-------
features | has_one **FeatureSet** | what kind of functionality can we expect the `data_source` to provide. This is mostly to guide the UI so it can be written in a general way
years | has_many **Year**
shows | has_many **Show**

### Year

Cached summerization data about each year of an artist.

Column Name | Type | Comments
:---------- | :--- | :-------
year | int 
show_count | int | **NOT** the number of recordings
recording_count | int
duration | The sum of the duration of each show in the year (use the longest source per show)
avg_duration | float | The average duration of each show in the year (use the longest source per show)
avg_rating | float | The average of all the average ratings for each show in the year

Relation Name | Type | Comments
:---------- | :--- | :-------
artist | belongs_to **Artist**

{'fraction_non_alphanumeric': 0.10664040517726506, 'fraction_numerical': 0.0030951041080472708, 'mean_word_length': 3.96785464709993, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '390951', 'n_tokens_mistral': 1940, 'n_tokens_neox': 1896, 'n_words': 913}