<jupyter_start><jupyter_text>Visualising Data**Author:** [<NAME>](https://github.com/AngusTheMack) ([<EMAIL>](mailto:<EMAIL>))**Achievement** Using the [training dataset](https://s3.eu-central-1.amazonaws.com/aicrowd-static/datasets/snake-species-identification-challenge/train.tar.gz) from the [Snake Species Identification Challenge](https://www.aicrowd.com/challenges/snake-species-identification-challenge), as well as the [Labelmap](https://aicrowd-production.s3.eu-central-1.amazonaws.com/dataset_files/challenge_3/e29091a0-37cb-4cb8-a01e-cde5e90fb8a5_class_id_maapping.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ6IZH6GWKDCCDFAQ%2F20190925%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20190925T084258Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c269210cc06417eb7bf112f125bc82422ee72a8859d39850c54cfd6c2b6b52fc) we look at the various classes, and the balance of their data.  IntroductionThis notebook is based off of the [Production Data Science Repo](https://github.com/Satalia/production-data-science) by [Satalia](https://www.satalia.com/)Despite the `snakes_starter_kit` reading in and using a basic model of the data, it doesn't give much insight into the data's nuance. Hence, this notebook will simply look at some of the different aspects of the data. Thus, in this notebook we will go over:* Data LoadingSystem details for reproducibility:<jupyter_code>%load_ext watermark
%watermark
import os
import csv
import random
import pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
def class_idxs(csv_file_name='../../data/class_idx.csv'):
    """
    Takes in a CSV file with the associated class - index mapping
    """
    with open(csv_file_name, newline='') as csv_file:
        data = list(csv.reader(csv_file))

    class_idx= {}
    for line in data[1:]:
        class_idx[line[1]] = line[0]
    return class_idx
class_idx = class_idxs()
print("Class|Latin Name")
print("*********************************")
for key in class_idx:
    print('{0: <5}|'.format(key),class_idx[key])
def get_name(class_index, class_idxs=class_idx):
    """
    Given a class_index  i.e.: 4 or class-4, return the latin name of the class
    """
    try:
        if '-' in class_index:
            return class_idxs[class_index[class_index.rfind('-')+1:]]
        else:
            return class_idxs[class_index]
    except Exception as e:
        print("Function get_name error:",e)
def get_paths(dir_name="../../data/train", class_idx=class_idx):
    """
    Takes in the directory and loads all the image paths into an array, as well as info about the classes
    """
    img_path = pd.DataFrame(columns=["image_path","class"])
    num_imgs = pd.DataFrame(columns=['class','latin_name','images'])
    for i, folder in enumerate(os.listdir(dir_name)):
        image_path = os.path.join(dir_name, folder)
        class_images = os.listdir(image_path)
        num_class_img = len(class_images)
        num_imgs.loc[len(num_imgs)] = [folder, get_name(folder),num_class_img]
        for j, name in enumerate(class_images):
            current_path = os.path.join(image_path, name)
            img_path.loc[len(img_path)] = [current_path,folder]
    return num_imgs, img_path
num_class_imgs, data = get_paths()
num_class_imgs = num_class_imgs.sort_values('images')
num_class_imgs.set_index('class')
x = data['image_path'].values
y = data['class'].values
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, shuffle=True, stratify=y)
def display_training_data(x_train, y_train, figsize=(10,10)):
    """
    Given x as image data and y as the associated class, display 25 random images from the dataset
    """
    plt.figure(figsize=figsize)
    data_size = len(x_train)
    for i in range(25):
        plt.subplot(5,5,i+1)
        plt.xticks([])
        plt.yticks([])
        plt.grid(False)
        random_index = random.randint(0, data_size)
        img_path = x_train[random_index]
        img = plt.imread(img_path)
        plt.imshow(img)
        plt.title(get_name(y_train[random_index]))
    plt.show()
display_training_data(x_train,y_train)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12910179640718564, 'fraction_numerical': 0.030179640718562873, 'mean_word_length': 4.388387096774194, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 8, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '370076', 'license': 'MIT', 'n_tokens_mistral': 1519, 'n_tokens_neox': 1430, 'n_words': 304, 'path': 'notebooks/visualising_data/visualising_data.ipynb', 'repo_name': 'AngusTheMack/snake-id', 'size': '4224'}