<jupyter_start><jupyter_text>Bachelor project notebook Musical patterns for predictionBefore continuing, make sure to download and install abjad, LilyPond (needed by abjad) and pretty_midi, as well as the standard python librairies, such as numpy.  Generation for a single fileWe first need to import the files for the generation process. All three methods take a path to a midi (.mid) file, and produce a continuation in the same folder.<jupyter_code>from simple_first_order_mm import generate_prediction_with_simple_markov
from string_based import generate_prediction_with_string_based
from translation_based import generate_prediction_with_translation_based<jupyter_output><empty_output><jupyter_text>We also need to know where the output pdf files (the music scores) will be generated. A NOTE such as "The Pärt demo requires abjad-ext-tonality" can be ignored.<jupyter_code>import abjad
print(abjad.AbjadConfiguration().abjad_output_directory)<jupyter_output><empty_output><jupyter_text>Then we'll set the path to the file that will be used in the next few cells. It corresponds to the path of a midi (.mid) file, for which a continuation has to be produced.<jupyter_code>midi_filename = "MIDI_files/random_example.mid"<jupyter_output><empty_output><jupyter_text>Let's generate a continuation with the simple Markov model algorithm. Warnings from pretty_midi or abjad can be ignored.<jupyter_code>generate_prediction_with_simple_markov(midi_filename)
# Optionally, we can tweak some parameters for the generation, default is:
# generate_prediction_with_simple_markov(midi_filename, notes_to_generate = 4, with_smoothing=False, probability_known_states=0.9)<jupyter_output><empty_output><jupyter_text>Now, try to generate with the string-based approach.<jupyter_code>generate_prediction_with_string_based(midi_filename)
# Same options can be applied, default is:
# generate_prediction_with_string_based(midi_filename, patterns_to_generate = 4, with_smoothing=False, probability_known_patterns=0.9)<jupyter_output><empty_output><jupyter_text>Finally, we can generate with the translation-based algorithm.<jupyter_code>generate_prediction_with_translation_based(midi_filename)
# Default is:
# generate_prediction_with_translation_based(midi_filename, patterns_to_generate = 4, with_smoothing=False, probability_known_patterns=0.9)<jupyter_output><empty_output><jupyter_text>Generation for datasetsAll of this was for individual files. However, for the evaluation, we need to do this procedure for a whole dataset. First import the needed functions.<jupyter_code>from simple_first_order_mm import generate_prediction_with_simple_markov_for_dataset
from string_based import generate_prediction_with_string_based_for_dataset
from translation_based import generate_prediction_with_translation_based_for_dataset<jupyter_output><empty_output><jupyter_text>The functions work with csv files, and produce both csv and midi files, so that the results can be heard by using MuseScore, for example. Please follow these instructions:1. Download either one of these three monophonic datasets (the larger the longer it will take): small, medium or large.2. Unzip it somewhere adequate, like in "../Datasets" (in the parent folder as this file is located).3. For all three methods to work, you will need to create multiple folders, at the same location as the "prime_csv" folder, in the downloaded dataset:    - For the simple first order Markov model: "markov_without_prediction_midi" and "markov_without_prediction_csv".    - For the string-based approach: "markov_with_prediction_midi" and "markov_with_prediction_csv".    - For the translation-based approach: "markov_with_non_exact_prediction_midi" and "markov_with_non_exact_prediction_csv".4. Set the variable name "dataset_filepath" below to the path you chose in 2.<jupyter_code>dataset_filepath = "../Datasets/PPDD-Sep2018_sym_mono_small/"<jupyter_output><empty_output><jupyter_text>Once done, we can safely run the three functions below.<jupyter_code>generate_prediction_with_simple_markov_for_dataset(dataset_filepath)
# Default is generate_prediction_with_simple_markov_for_dataset(dataset_filepath, notes_to_generate = 30,with_smoothing=True,probability_known_states=0.9)
generate_prediction_with_string_based_for_dataset(dataset_filepath)
# Default is generate_prediction_with_string_based_for_dataset(dataset_filepath, patterns_to_generate = 20,with_smoothing=True,probability_known_states=0.9)
generate_prediction_with_translation_based_for_dataset(dataset_filepath)
# Default is generate_prediction_with_translation_based_for_dataset(dataset_filepath, patterns_to_generate = 20,with_smoothing=True,probability_known_states=0.9)<jupyter_output><empty_output><jupyter_text>Note that each method is much slower than the previous one.  EvaluationWith the three outputs generated, we can now run the evaluation code. Please download it from here and follow these instructions:1. Copy and paste the content of [config_corrected.py](config_corrected.py) into config.py (in the newly downloaded evaluation folder), and modify "OUTPUT_FOLDER", "DATASET_PATH" and "FILENAME_FRAGMENT".2. Edit cs.py to [cs_corrected.py](cs_corrected.py) (copy and paste the code).3. Execute the following cell (you might need to correct the path).<jupyter_code>%run ../Evaluation/evaluate_prediction.py<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06084359835759612, 'fraction_numerical': 0.006159014557670772, 'mean_word_length': 8.32, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '172615', 'license': 'MIT', 'n_tokens_mistral': 1607, 'n_tokens_neox': 1552, 'n_words': 469, 'path': 'Generation/Generation.ipynb', 'repo_name': 'v1ruso/BachelorProject', 'size': '5358'}