<jupyter_start><jupyter_text>CSV manipulation for initial use<jupyter_code>import pandas as pd
import os
userPath = os.environ['USERPROFILE']
basePath = userPath + "\\GitHub\\UniMolFinalExam\\Dataset\\"
driversList = ["Angelo", "Aniceto"]
driversLabel = ["A", "B"]
index = 0<jupyter_output><empty_output><jupyter_text>Open a file, appen a new column with the same entry (driver's name) in all the rows and creates a new CSV file<jupyter_code>for driver in driversList:
    driversLabel[index]
    oldFilePath = basePath + "Original_CSV\\" + driver

    print("Current path -> " + oldFilePath)

    with os.scandir(oldFilePath) as dirs:
        for entry in dirs:
            df = pd.read_csv(entry, sep=',', quotechar='"')
            df['Driver'] = driversLabel
            newFilePath = basePath + "Prepared_CSV\\" + driver + "\\" + entry.name
            print("Current file -> " + entry.name)
            df.to_csv(newFilePath, index=False)
    
    index += 1<jupyter_output>Current path -> C:\Users\angel\GitHub\UniMolFinalExam\Dataset\Original_CSV\Angelo
Current file -> trackLog-2019-dic-23_13-53-06.csv
Current file -> trackLog-2019-dic-23_16-29-22.csv
Current file -> trackLog-2019-dic-26_17-48-59.csv
Current file -> trackLog-2019-dic-30_09-13-27.csv
Current file -> trackLog-2019-dic-30_15-48-11.csv
Current file -> trackLog-2019-dic-31_12-51-24.csv
Current file -> trackLog-2019-dic-31_16-14-44.csv
Current file -> trackLog-2019-dic-31_17-06-00.csv
Current file -> trackLog-2019-nov-02_19-20-18.csv
Current file -> trackLog-2019-nov-03_12-51-02.csv
Current file -> trackLog-2019-ott-03_22-57-51.csv
Current file -> trackLog-2019-ott-06_15-20-22.csv
Current file -> trackLog-2019-ott-06_16-12-10.csv
Current file -> trackLog-2019-ott-07_08-38-01.csv
Current file -> trackLog-2019-ott-08_12-52-44.csv
Current file -> trackLog-2019-ott-08_12-59-20.csv
Current file -> trackLog-2019-ott-08_13-00-01.csv
Current file -> trackLog-2019-ott-09_10-40-26.csv
Current file -> tr[...]<jupyter_text>All files from a single driver are merged in a single CSV<jupyter_code>for driver in driversList:
    oldFilePath = basePath + "Prepared_CSV\\" + driver + "\\"

    print("Current path -> " + oldFilePath)

    os.chdir(oldFilePath)
    files = os.listdir()

    final_df = pd.concat([pd.read_csv(csv).assign(New=os.path.basename(csv)) for csv in files])
    newFilePath = basePath + "Preprocessed_CSV\\" + driver + ".csv"
    final_df.to_csv(newFilePath, index=False)<jupyter_output>Current path -> C:\Users\angel\GitHub\UniMolFinalExam\Dataset\Prepared_CSV\Angelo\
Current path -> C:\Users\angel\GitHub\UniMolFinalExam\Dataset\Prepared_CSV\Aniceto\
{'fraction_non_alphanumeric': 0.14123867069486404, 'fraction_numerical': 0.0823262839879154, 'mean_word_length': 3.705150976909414, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '609582', 'license': 'MIT', 'n_tokens_mistral': 1105, 'n_tokens_neox': 960, 'n_words': 183, 'path': 'Notebooks/Python/Pre-Processing dataset.ipynb', 'repo_name': 'AngelusGi/UniMolFinalExam', 'size': '2648'}