<jupyter_start><jupyter_text>Pre-process comparison dataPrepare the ilastik classification results and CNN classification results so that the data can be statistically compared, to evaluate the accuracy of the various approaches  Step 1: Combine all CNN and Ilastik csv filesPrevious pre-processing has interpreted classified data from Ilastik and the CNN into csv files. These files are now combined into a dataframe with an indicator of what the algorithm is, what the image size is (small, med, large), and the classification probabilities per 30 representative tile coordinate<jupyter_code>import glob
import os
import pandas as pd

# Function to read the json file and convert the dictionary object in the 'classifiedData' column
# into seperate columns
def load_prediction_file(file_name, algo_type, algo, image_size, image_id):
    data = pd.read_csv(file_name) 
    data['algorithm_type'] = algo_type
    data['algorithm'] = algo
    data['image_size'] = image_size
    data['image_id'] = image_id
    return data

all_frames = None

# 1a: Combine CNN classification CSV files into a dataframe

all_cnn_prediction_files = glob.glob('../../TestPredictions/CNN/*.csv')

for pred_file in all_cnn_prediction_files:
    head_tail = os.path.split(pred_file)
    pre, ext = os.path.splitext(head_tail[1])
    parts = pre.split('_')
    algo = parts[0]
    size = parts[1]
    image = parts[2] + '_' + parts[3]
    frame = load_prediction_file(pred_file, 'cnn', algo, size, image)
    all_frames = pd.concat([all_frames, frame])
   

# 1b: Add the Ilastik classification CSV files into a dataframe
all_ilastik_prediction_files = glob.glob('../../TestPredictions/Ilastik/*.csv')

for pred_file in all_ilastik_prediction_files:
    head_tail = os.path.split(pred_file)
    pre, ext = os.path.splitext(head_tail[1])
    parts = pre.split('_')
    # test-large_DJI_0120100_Probabilities
    size = parts[0]
    image = parts[1] + '_' + parts[2]    
    frame = load_prediction_file(pred_file, 'ilastik', 'default', size, image)
    all_frames = pd.concat([all_frames, frame])

all_frames
    
# 1c: Reformat the data into a consistent dataset

# Ilastik columns:
# A = Foliage
# B = Water
# C = Building
# D = Road

# CNN columns:
# 0 - Unknown
# 1 - Foliage
# 2 - Water
# 3 - Road
# 4 - Building

all_frames['Foliage'] = all_frames['1'].combine_first(all_frames['A'])
all_frames['Water'] = all_frames['2'].combine_first(all_frames['B'])
all_frames['Building'] = all_frames['4'].combine_first(all_frames['C'])
all_frames['Road'] = all_frames['3'].combine_first(all_frames['D'])

all_frames_save = all_frames.drop(['0', '1', '2', '3', '4', 'A', 'B', 'C', 'D'], axis = 1)<jupyter_output><empty_output><jupyter_text>Step 2: Enrich the dataAdd the test case to the dataset, as well as the video id so that the video dataset can be joined into it<jupyter_code># 2a: Calculate the video ID
all_frames_save['video_id'] = all_frames_save.image_id.str[:8]

# 2b: Find the test case (identified by the parent folder of the test image), and update the dataframe to hold it
all_frames_save['test_case'] = ''
all_images = all_frames_save.image_id.unique()

for image_id in all_images:
    matching_test_files = glob.glob('../../Texture_Repo/Donegal_Rural_Terrain_Textures/Test_Images/*/{}.jpg'.format(image_id))
    test_file = matching_test_files[0]
    head_tail = os.path.split(test_file)
    filename = head_tail[1]
    parts = test_file.split('\\')
    test_case = parts[len(parts) - 2]
    all_frames_save.loc[all_frames_save['image_id'] == image_id, 'test_case'] = test_case<jupyter_output><empty_output><jupyter_text>Step 3: Merge the ground truth data into the datasetAdding the ground truth data to the dataset will allow the efficacy of the various prediction algorithms to be compared and contrasted<jupyter_code># 3a - load the ground truth data

ground_truth_data = pd.read_pickle('../../Texture_Repo/Donegal_Rural_Terrain_Textures/classified/all_data.pkl')
# Remove 12 null rows - an artifact of the classification app
classified_data = classified_data[~classified_data['xCoord'].isnull()]
# 3b - Remove the duplicated image / tile sequences (some images were classified multiple times and they all show up)
classified_data = classified_data.drop(['classificationSourceQueue', 'classifiedOn'], axis=1)
classified_data = classified_data.drop_duplicates()
# 3c - Now join the ground truth data into the dataframe with all the predictions in
joined_data = pd.merge(all_frames_save, classified_data, left_on=['image_id','tile_x', 'tile_y'], 
                       right_on = ['sourceImageName', 'xCoord', 'yCoord'])
# Reformat the columns
joined_data = joined_data.drop(['xCoord', 'yCoord', 'sourceImageName'], axis=1)
joined_data = joined_data.rename(columns={"category": "ground_truth"})
# Replace all NaN values with zeroes
joined_data = joined_data.fillna(0)

# Inspect the data
joined_data<jupyter_output><empty_output><jupyter_text>Step 4: Merge the video information into the comparison dataframeLoad the video meta data csv file and merge it into the algorithm evaluation data<jupyter_code>video_meta = pd.read_csv('../../Texture_Repo/Donegal_Rural_Terrain_Textures/Video_Info.csv')

compare_data = pd.merge(joined_data, video_meta, left_on= 'video_id', right_on = 'File_ID')
compare_data
# Finally save the resulting dataset to a pickle file for comparison
compare_data.to_pickle('../../TestPredictions/evaluate_algo_data.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10950375389122871, 'fraction_numerical': 0.009888298846365134, 'mean_word_length': 5.192743764172335, 'pattern_counts': {'":': 1, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '584095', 'license': 'MIT', 'n_tokens_mistral': 1800, 'n_tokens_neox': 1731, 'n_words': 497, 'path': 'Evaluate_Algos/PreProcess_Comparison_Data.ipynb', 'repo_name': 'Joon/Drone_texture_workbooks', 'size': '5461'}