<jupyter_start><jupyter_text>Setting up the database for training the model This short tutorial runs through the necessary steps to set up the Koster database.<jupyter_code>import pandas as pd
# Enter the path to movie files
movies_path = "/cephyr/NOBACKUP/groups/snic2021-6-9/movies"<jupyter_output><empty_output><jupyter_text>0. Video pre-processing (If your video is sufficiently compressed already and does not contain any sensitive information, you may skip this step)<jupyter_code>%cd "/your/movie/directory"
# This script will loop through the movie files you provided, compress them and blur areas 
# with potential sensitive information
!bash ingestion_scripts/add_blur.sh /your/movie/directory
# Now link this folder symbolically to the rest of the movies in the Koster Library
!ln -s /your/movie/directory/* movies_path
# Enter the correct path to the database preparation scripts
%cd "/usr/src/app/koster_data_management/db_setup"<jupyter_output>/usr/src/app/koster_data_management/db_setup<jupyter_text>1. Ensure all movie files are uploaded to the movie directory on the SNIC server<jupyter_code>!ls $movies_path
# Define a path for the db to be stored (example /cephyr/NOBACKUP/groups/snic2021-6-9/db_files)
# Note: you should have permissions to write to this location
db_path = "/cephyr/NOBACKUP/groups/snic2021-6-9/db_files/tutorial_demo.db"<jupyter_output><empty_output><jupyter_text>2. Setup the db file by using the batch script that creates the initial database from the original movies (skip to step 3 if only adding to existing database)<jupyter_code># Zooniverse credentials
# Specify username and password of a valid zooniverse account
user_zoo = ""
pass_zoo = ""
# Define suitable thresholds
# Specify Google Drive links of the csv files with information about the species choices,
# original movies and duplicated clips
sp_file_id = "https://drive.google.com/file/d/1dnueH3BjJrMK8buVjfyFbxfu0E-5dX7Z/view?usp=sharing"
mv_file_id = "https://drive.google.com/file/d/1xYcmMUjAawnYIyti9QNTs-oBf8XshJvs/view?usp=sharing"
dp_file_id = "https://drive.google.com/file/d/1z72CqTtEBtqk6936H1YNrCjc5NRopF0g/view?usp=sharing"
# Frame thresholds
obj_thresh = 0.8
eps_thresh = 0.5
# Specify the Zooniverse workflows of interest and their versions
workflow_clip = 11767
workflow_clip_version = 227
workflow_frame = 12852
workflow_frame_version = 21.85

# Specify the agreement threshold required among cit scientists
agg_user_clip = 0.5
agg_user_frames = 0.5

# Specifiy the min number of different Zooniverse users required per subject
min_users_clip = 3
min_users_frames = 5
## Run the setup script to populate the db with original movies
# Remove current db (optional)
!rm $db_path
# Initiate the db
%run -i "init.py" --db_path $db_path

# Populate the db with info from the csv files
%run -i "static.py" --species_file_id $sp_file_id --movies_file_id $mv_file_id --db_path $db_path --movies_path $movies_path

# Populate the db with info of subjects uploaded to Zooniverse
%run -i "subjects_uploaded.py" --user $user_zoo --password $pass_zoo --db_path $db_path --duplicates_file_id $dp_file_id

# Process the clips that have been classified in Zooniverse
%run -i "process_clips.py" --user $user_zoo --password $pass_zoo --db_path $db_path --duplicates_file_id $dp_file_id

# Process the frames that have been classified in Zooniverse
%run -i "process_frames.py" --user $user_zoo --password $pass_zoo --db_path $db_path --duplicates_file_id $dp_file_id
from utils import summary_utils
summary_utils.clips_summary(db_path)<jupyter_output><empty_output><jupyter_text>3. Add new species and/or movies to initial database<jupyter_code>import add
# New movies
add.add_new_movies("https://drive.google.com/file/d/1IBBm4GqZGUZvnVJ3DbIHwJ_JQQW6sCEl/view?usp=sharing", 
                   db_path,
                   movies_path)
add.add_new_movies("https://drive.google.com/file/d/19VlIQo749P__8EO-ZAL89Jk4m-1aM-s0/view?usp=sharing",
                   db_path,
                   movies_path)
add.add_new_movies("https://drive.google.com/file/d/161gRRNivEUJnkLCh7Wea7f0YXEyEJ6cv/view?usp=sharing",
                   db_path,
                   movies_path)
# New species
add.add_species("https://drive.google.com/file/d/18_5h4fzX7zau-JltIRPoFrltJm_cbxyb/view?usp=sharing",
                db_path)
conn = db_utils.create_connection(db_path)
movies_table = pd.read_sql_query("SELECT * FROM movies", conn)
movies_table.head()<jupyter_output><empty_output><jupyter_text>4. Enrich original database file with new subject sets (frames) from different species<jupyter_code># Specify workflow id and workflow versions to be incorporated. Note that any versions higher than the specified
# version will be included for a specific workflow
clips_zoo_workflow = 17719
clips_zoo_workflow_version = 0
frames_zoo_workflow_version = 31
# Run the aggregation script
run = f"python process_clips.py -u {user_zoo} -p {pass_zoo} -db {db_path} -thr {agg_user_clip} \
     -nu {min_users_clip} -zw {clips_zoo_workflow} -zwv {clips_zoo_workflow_version} \
     -du https://drive.google.com/file/d/1z72CqTtEBtqk6936H1YNrCjc5NRopF0g/view?usp=sharing"
!{run}
summary_utils.clips_summary(db_path)
# Run the aggregation script
run = f"python process_frames.py -u {user_zoo} -p {pass_zoo} -db {db_path} -obj {obj_thresh} \
     -zwv {frames_zoo_workflow_version} -eps {eps_thresh} -iua {agg_user_frames} -nu {min_users_frames} \
     -du https://drive.google.com/file/d/1z72CqTtEBtqk6936H1YNrCjc5NRopF0g/view?usp=sharing"
!{run}
conn = db_utils.create_connection(db_path)
frame_data = pd.read_sql_query("SELECT * FROM agg_annotations_frame", conn)
frame_data.species_id.value_counts()<jupyter_output><empty_output><jupyter_text>5. Move on to training_tutorial.ipynb for model training<jupyter_code># END<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08940852819807428, 'fraction_numerical': 0.021664374140302613, 'mean_word_length': 5.084728033472803, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '308788', 'license': 'MIT', 'n_tokens_mistral': 2151, 'n_tokens_neox': 2035, 'n_words': 517, 'path': 'development/notebooks/database_tutorial.ipynb', 'repo_name': 'ocean-data-factory-sweden/koster_zooniverse', 'size': '5816'}