<commit_before># Audio Pipeline
A set of audio processing tools used to process and prepare audio files for ingestion into asset management systems.

To Install
-----------
It is assumed that you already have Python 3.4 or greater installed, and it is in your path.

1. Checkout this repository
2. Install musicbrainzngs  `pip install musicbrainzngs`
3. Install mutagen  `pip install mutagen`
4. Install yattag  `pip install yattag`
5. Install mammoth  `pip install mommoth`
6. Install smartsheet  `pip install smartsheet-python-sdk`
7. Install yaml  `pip install pyyaml`
8. Profit!

Programs
-----------
### File Walker
5. Install mammoth  `pip install mommoth`
6. Install smartsheet  `pip install smartsheet-python-sdk`
7. Install mammoth  `pip install pyyaml`

### Programs
#### File Walker
Walks a directory structure of audio files and does the following:
 * Pulls out the MusicBrainz Release ID, Track Number, and Disc Number from the file metadata
 * Querys MusicBrainz based on the Release ID, Track Number, and Disc Number, and return the metadata for the Track and Release
 * Querys MusicBrainz based on the Artist ID of all artist's in the album's Artist Credit, and return the metadata for each Artist
 * Writes this information out to the destination directory:
   * File are written to a directory structure of the form:
     *found*: Backup copy of all succesfully processed audio files
     *not_found*: Copy of all tracks that filewalker did not process successfully
     *artist_meta*: Serialized artist metadata files
     *release_meta*: Serialized release metadata files
     *track_meta*: Serialized track metadata files
     *track*: All successfully processed audio files, renamed to match the track ItemCode
     *session_logs*: Tab separated text files of the form:
      `<Item Type>   <ItemCode>  <Item Name>`
       Current log files:
        * *filewalker_log*: all releases, tracks, artists, and labels
        * *label_log*: all labels
        * *release_log*: all releases
   * Serialized metadata files are named according to their ItemCode. Release filenames are given prefix 'r'; artist filenames are given prefix 'a'.
   * Currently files can only be written as Dalet-compatible XML
   
#####Options:

######General:

  * *--delete, -d*
    Delete source audio files after processing

  * *--generate, -g*
    Generate metadata only, without copying audio

  * *--mbhost*
    Set server to retrieve MusicBrainz data from. Default is musicbrainz.org; another server can be manually specified

  * *--no_artist*
    Do not generate artist metadata XMLs.

  * *--i, --gen_item_code*
    Generate a unique item code for all audio files

######KEXP-specific:

 * *--source, -s*  
    Passes in the source of audio files, to be recorded in XML.

    **Allowed values:**
    
    - Melly  
    - CD Library
      
 * *--category, -c*   
    Passes in the category of audio files, to be recorded in XML.
        
    **Allowed values:**

    - EXP - Experimental  
    - HIP - Hip Hop  
    - JAZ - Jazz  
    - LOC - Local  
    - REG - Reggae  
    - ROC - Rock & Pop  
    - ROO - Roots  
    - SOU - Soundtracks  
    - WOR - World  
    - ACQ - Recent Acquisitions  
    - LIV - Live on KEXP

 * *--rotation, -r*   
    Passes in the rotation status of audio files, to be recorded in XML.  
        
    **Allowed values:**
    
    - Heavy
    - Library  
    - Light  
    - Medium  
    - R/N

  * *--anchor, -a*
    Set KEXPAnchorStatus field of all tracks to '1'
      
#####To Run

1. Navigate to the top-level audio_pipeline directory
2. run `FileWalker.py source_directory destination_directory options`

IMPORTANT: Audio files must have release MBID, as well as track number and disc number associated with that MBID, for FileWalker to succeed. 

   
### TomatoBanana
A program to assist in the review and entry of metadata of ripped audio discs
 * Pulls relevent metadata from audio files
 * Displays release metadata onscreen in an easily human-readable format for review
 * Releases are displayed in the same order that they were ripped, for easy processing when returning discs to cases
 * Allows the easy entry of obscenity rating values for individual tracks
 * An interface for complete metadata entry is easily accessed from a release in TB

#####To Run
1. Navigate to the top-level audio_pipeline directory
2. Run `TomatoBanana.py`

#### Review Parser<commit_msg>Basic readme for review_parser<commit_after># Audio Pipeline
A set of audio processing tools used to process and prepare audio files for ingestion into asset management systems.

To Install
-----------
It is assumed that you already have Python 3.4 or greater installed, and it is in your path.

1. Checkout this repository
2. Install musicbrainzngs  `pip install musicbrainzngs`
3. Install mutagen  `pip install mutagen`
4. Install yattag  `pip install yattag`
5. Install mammoth  `pip install mommoth`
6. Install smartsheet  `pip install smartsheet-python-sdk`
7. Install yaml  `pip install pyyaml`
8. Profit!

### Programs
#### File Walker
Walks a directory structure of audio files and does the following:
 * Pulls out the MusicBrainz Release ID, Track Number, and Disc Number from the file metadata
 * Querys MusicBrainz based on the Release ID, Track Number, and Disc Number, and return the metadata for the Track and Release
 * Querys MusicBrainz based on the Artist ID of all artist's in the album's Artist Credit, and return the metadata for each Artist
 * Writes this information out to the destination directory:
   * File are written to a directory structure of the form:
     *found*: Backup copy of all succesfully processed audio files
     *not_found*: Copy of all tracks that filewalker did not process successfully
     *artist_meta*: Serialized artist metadata files
     *release_meta*: Serialized release metadata files
     *track_meta*: Serialized track metadata files
     *track*: All successfully processed audio files, renamed to match the track ItemCode
     *session_logs*: Tab separated text files of the form:
      `<Item Type>   <ItemCode>  <Item Name>`
       Current log files:
        * *filewalker_log*: all releases, tracks, artists, and labels
        * *label_log*: all labels
        * *release_log*: all releases
   * Serialized metadata files are named according to their ItemCode. Release filenames are given prefix 'r'; artist filenames are given prefix 'a'.
   * Currently files can only be written as Dalet-compatible XML
   
#####Options:

######General:

  * *--delete, -d*
    Delete source audio files after processing

  * *--generate, -g*
    Generate metadata only, without copying audio

  * *--mbhost*
    Set server to retrieve MusicBrainz data from. Default is musicbrainz.org; another server can be manually specified

  * *--no_artist*
    Do not generate artist metadata XMLs.

  * *--i, --gen_item_code*
    Generate a unique item code for all audio files

######KEXP-specific:

 * *--source, -s*  
    Passes in the source of audio files, to be recorded in XML.

    **Allowed values:**
    
    - Melly  
    - CD Library
      
 * *--category, -c*   
    Passes in the category of audio files, to be recorded in XML.
        
    **Allowed values:**

    - EXP - Experimental  
    - HIP - Hip Hop  
    - JAZ - Jazz  
    - LOC - Local  
    - REG - Reggae  
    - ROC - Rock & Pop  
    - ROO - Roots  
    - SOU - Soundtracks  
    - WOR - World  
    - ACQ - Recent Acquisitions  
    - LIV - Live on KEXP

 * *--rotation, -r*   
    Passes in the rotation status of audio files, to be recorded in XML.  
        
    **Allowed values:**
    
    - Heavy
    - Library  
    - Light  
    - Medium  
    - R/N

  * *--anchor, -a*
    Set KEXPAnchorStatus field of all tracks to '1'
      
#####To Run

1. Navigate to the top-level audio_pipeline directory
2. run `FileWalker.py source_directory destination_directory options`

IMPORTANT: Audio files must have release MBID, as well as track number and disc number associated with that MBID, for FileWalker to succeed. 

   
### TomatoBanana
A program to assist in the review and entry of metadata of ripped audio discs
 * Pulls relevant metadata from audio files
 * Displays release metadata onscreen in an easily human-readable format for review
 * Releases are displayed in the same order that they were ripped, for easy processing when returning discs to cases
 * Allows the easy entry of obscenity rating values for individual tracks
 * An interface for complete metadata entry is easily accessed from a release in TB

#####To Run
1. Navigate to the top-level audio_pipeline directory
2. Run `TomatoBanana.py`

### Review Parser
A program used to push the weekly album reviews into Dalet
 * Reads the reviews from a MS Word docx file
 * Matches the albums in the review sheet to either albums in a specially formatted Smartsheet or a Filewalker Release log in order to get each releases MusicBrainz ID
 * Generates a set of XML documents used by Dalet to update the releases with the reviews.

#####To Run
1. Navigate to the top-level audio_pipeline directory
2. Run `review_parser.py input_file options`

#####Options:
 * *--api_key, -k*
   Your Smartsheet API Key
  
 * *--worksheet, -w*
   Smartsheet Worksheet ID that contains the reviews associated MusicBrainz ID's
      
 * *--dalet, -d*
   Directory to put Dalet Impex files in.
{'fraction_non_alphanumeric': 0.07168764668231278, 'fraction_numerical': 0.003733731598037124, 'mean_word_length': 3.046180405697022, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4760907', 'n_tokens_mistral': 2641, 'n_tokens_neox': 2526, 'n_words': 1243}