<jupyter_start><jupyter_text>Case 2. Create directories and datasets<NAME>   22.2.2018   Cognitive Systems for Health Technology Applications   Helsinki Metropolia University of Applied Science    IntroductionThis script is for: 1. Checking how many samples are in the original downloaded dataset2. Creating the directory structure for the datasets3. Creating the directories4. Splitting the data into train, validation, and test sets5. Coping the data into the directoriesDataset is downloaded from: https://github.com/Nomikxyz/retinopathy-dataset   1. Check the downloaded dataFirst it is needed to check that we have access to the downloaded and extracted data. This script reads the filenames in `nosymptoms` and `sympotms` subfoldes and counts how many observations are there totally.<jupyter_code>import os, shutil
# List all filenames in the master dataset and count how many samples there are
original_dir = 'C:\\Users\\Joona\\retinopathy-dataset-master'

class1 = 'nosymptoms'
original_nosymptoms_dir = os.path.join(original_dir, class1)
nosymptoms_fnames = os.listdir(original_nosymptoms_dir)

class2 = 'symptoms'
original_symptoms_dir = os.path.join(original_dir, class2)
symptoms_fnames = os.listdir(original_symptoms_dir)

len(nosymptoms_fnames), len(symptoms_fnames)<jupyter_output><empty_output><jupyter_text>2. Directory structureAfter that it is needed to create the names for the directories and save them into variables.<jupyter_code># Base directory is where the datasets will be created
base_dir = 'dataset2'

# For training set
sub_dir = 'train'
train_dir = os.path.join(base_dir, sub_dir)
train_nosymptoms_dir = os.path.join(base_dir, sub_dir, class1)
train_symptoms_dir = os.path.join(base_dir, sub_dir, class2)

# For validation set
sub_dir = 'validation'
validation_dir = os.path.join(base_dir, sub_dir)
validation_nosymptoms_dir = os.path.join(base_dir, sub_dir, class1)
validation_symptoms_dir = os.path.join(base_dir, sub_dir, class2)

# For test set
sub_dir = 'test'
test_dir = os.path.join(base_dir, sub_dir)
test_nosymptoms_dir = os.path.join(base_dir, sub_dir, class1)
test_symptoms_dir = os.path.join(base_dir, sub_dir, class2)<jupyter_output><empty_output><jupyter_text>3. Create directoriesNext script is checked if the folder already exists. If there is not folders alredy folders are created the directory structure for the train, test, and evaluation sets.<jupyter_code>if not(os.path.exists(base_dir)):
    print('Creating dataset folders to:', base_dir)
    os.mkdir(base_dir)
    os.mkdir(train_dir)
    os.mkdir(train_nosymptoms_dir)
    os.mkdir(train_symptoms_dir)
    os.mkdir(validation_dir)
    os.mkdir(validation_nosymptoms_dir)
    os.mkdir(validation_symptoms_dir)
    os.mkdir(test_dir)
    os.mkdir(test_nosymptoms_dir)
    os.mkdir(test_symptoms_dir)
else:
    print(base_dir, 'already exists!')<jupyter_output>Creating dataset folders to: dataset2<jupyter_text>4. Split the data filenames into train, validation and test setsWhen the directories are ready, it is time to split the original dataset into training, validation and test sets. For that we use scikit-learn library's train_test_split function. First we split the dataset with rule 80%-20% and then we split 80% to 60% and 20%. Finally we get:- 60% training set- 20% validation set, and - 20% test set.This needs to be repeated both for the healthy (nosymptoms) and disease (symptom) cases.<jupyter_code>from sklearn.model_selection import train_test_split
# Disease (symptom) cases split

# Take 20 % out for testing
train_symptoms_fnames, test_symptoms_fnames = train_test_split(symptoms_fnames, test_size = 0.2)

# From the remaining 80% take 0.25 (=0.8*0.25 = 20% of total) out for validation
train_symptoms_fnames, validation_symptoms_fnames = train_test_split(train_symptoms_fnames, test_size = 0.25)

len(train_symptoms_fnames), len(validation_symptoms_fnames), len(test_symptoms_fnames)
# For debugging purposes, remove the comment marks.
# print(train_symptoms_fnames)
# print(validation_symptoms_fnames)
# print(test_symptoms_fnames)
# Healthy (nosyptom) cases split

# Take 20 % out for testing
train_nosymptoms_fnames, test_nosymptoms_fnames = train_test_split(nosymptoms_fnames, test_size = 0.2)

# From the remaining 80% take 0.25 (20% of total) out for validation
train_nosymptoms_fnames, validation_nosymptoms_fnames = train_test_split(train_nosymptoms_fnames, test_size = 0.25)

len(train_nosymptoms_fnames), len(validation_nosymptoms_fnames), len(test_nosymptoms_fnames)
# For debugging purposes, remove the comment marks.
# print(train_nosymptoms_fnames)
# print(validation_nosymptoms_fnames)
# print(test_nosymptoms_fnames)<jupyter_output><empty_output><jupyter_text>5. Copy data into the directoriesFinally the original data is copied into the training, validation and test directories. As this might take some time, it is wantd to watch the time spend on it.<jupyter_code>import time
tStart = time.time()

# Copy the original files into the dataset folders

# Training set
# Disease 
for fname in train_symptoms_fnames:
    src = os.path.join(original_symptoms_dir, fname)
    dst = os.path.join(train_symptoms_dir, fname)
    shutil.copyfile(src, dst)
# Healthy 
for fname in train_nosymptoms_fnames:
    src = os.path.join(original_nosymptoms_dir, fname)
    dst = os.path.join(train_nosymptoms_dir, fname)
    shutil.copyfile(src, dst)

# Validation set
# Disease 
for fname in validation_symptoms_fnames:
    src = os.path.join(original_symptoms_dir, fname)
    dst = os.path.join(validation_symptoms_dir, fname)
    shutil.copyfile(src, dst)
# Healthy
for fname in validation_nosymptoms_fnames:
    src = os.path.join(original_nosymptoms_dir, fname)
    dst = os.path.join(validation_nosymptoms_dir, fname)
    shutil.copyfile(src, dst)

# Test set
# Disease
for fname in test_symptoms_fnames:
    src = os.path.join(original_symptoms_dir, fname)
    dst = os.path.join(test_symptoms_dir, fname)
    shutil.copyfile(src, dst)
# Healthy
for fname in test_nosymptoms_fnames:
    src = os.path.join(original_nosymptoms_dir, fname)
    dst = os.path.join(test_nosymptoms_dir, fname)
    shutil.copyfile(src, dst)

tStop = time.time()
tElapsed = tStop - tStart
print('Time elapsed: {:.2f} sec'.format(tElapsed))<jupyter_output>Time elapsed: 242.76 sec
{'fraction_non_alphanumeric': 0.08125298139608841, 'fraction_numerical': 0.013515662267451106, 'mean_word_length': 5.763440860215054, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '305645', 'license': 'MIT', 'n_tokens_mistral': 2229, 'n_tokens_neox': 2085, 'n_words': 576, 'path': 'Case 2. Creating directories.ipynb', 'repo_name': 'Joonakle/cognitive-systems-for-healthtechnology-applications', 'size': '6297'}