<jupyter_start><jupyter_text>Loading Data with PyTorch datasets and dataloaders<jupyter_code>#importing packages
import torch
from torch.utils.data import Dataset
from torchvision import datasets
from torchvision.transforms import ToTensor, Lambda
import matplotlib.pyplot as plt
%matplotlib inline
#loading training data

training_data = datasets.FashionMNIST(
    root="data",
    train=True,
    download=True,
    transform=ToTensor()
)
#loading test data

test_data = datasets.FashionMNIST(
    root="data",
    train=False,
    download=True,
    transform=ToTensor()
)
#iterating and visualizing datasets
labels_map = {
    0:"T-Shirt",
    1:"Trouser",
    2:"Pullover",
    3:"Dress",
    4:"Coat",
    5:"Sandal",
    6:"Shirt",
    7:"Sneaker",
    8:"Bag",
    9:"Ankle Boot",
}

figure = plt.figure(figsize=(8,8))
cols, rows = 3,3
for i in range(1, cols*rows + 1):
  sample_idx = torch.randint(len(training_data), size=(1,)).item()
  img, label = training_data[sample_idx]
  figure.add_subplot(rows, cols, i)
  plt.title(labels_map[label])
  plt.axis("off")
  plt.imshow(img.squeeze(), cmap="gray")
plt.show()
#buildimg a custom image dataset
import os
import pandas
import torchvision.io

class CustomImageDataset(Dataset):
  def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
    self.img_labels = pd.read_csv(annotations_file)
    self.img_dir = img_dir
    self.transform = transform
    self.target_transform = target_transform

  def __len__(self):
    return len(self.img_labels)

  def __getitem__(self, idx):
    img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
    image = tvio.read_image(img_path)
    label = self.img_labels.iloc[idx, 1]
    if self.transform:
      image = self.transform(image)
    if self.target_transform:
      label = self.target_transform(label)
    sample = {"image":image, "label":label }
    return sample<jupyter_output><empty_output><jupyter_text>The __init__ function is run when instantiating the dataset obect, we initialize the directory contaiing the images, annotations file<jupyter_code>def __init__(self, annotations_ffile, img_dir, transform=None, target_transform=None):
  self.img_labels = pd.read_csv(annotations_file)
  self.img_dir = img_dir
  self.transform = transform
  self.target_transform = target_transform<jupyter_output><empty_output><jupyter_text>The __len__ function returns the number of samples in a dateste<jupyter_code>def __len__(self):
    return len(self.img_labels)<jupyter_output><empty_output><jupyter_text>The __getitem__ function loads and returns a sample from the dataset from the given index at __idx__, it locates the images location on disk and turns it into a tensor using __read image__ receives the image's label from the __self.img_labels__ calls the transform function (if applicable) and returns the image in tensor form in a coresponding python dict<jupyter_code>def __getitem__(self, idx):
    img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
    image = read_image(img_path)
    label = self.img_labels.iloc[idx, 1]
    if self.transform:
        image = self.transform(image)
    if self.target_transform:
        label = self.target_transform(label)
    sample = {"image": image, "label": label}
    return sample<jupyter_output><empty_output><jupyter_text>Preparing your data for data loaders The __Dataset__  retrieves our datsets features and labels one instance at a time. When Trainig models we usually want to pass samples in __minibatches__, reshuffle the data at every __epoch__ to reduce model overfitting and make use of python's __multiprocessing__ top speed up data retrieval. __DataLoader__ is an iterbale that implements this complexity for us in a simple API<jupyter_code>from torch.utils.data import DataLoader
train_DataLoader = DataLoader(training_data, batch_size=64, shuffle=True)

test_DataLoader = DataLoader(test_data, batch_size=64, shuffle=True)<jupyter_output><empty_output><jupyter_text>Iterating Through Data Loaders Each iteration below returns a batch of __train features__ and __train labels__.<jupyter_code>#display image and label
train_features, train_labels = next(iter(train_DataLoader))
print(f"Labels Batche size: {train_features.size()}")
print(f"Labels Batch Size:{train_labels.size()}")

img = train_features[0].squeeze()

label = train_labels[0]

plt.imshow(img, cmap="gray")
plt.show()
print(f"Label:{label}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09636934110264456, 'fraction_numerical': 0.006051098162259076, 'mean_word_length': 5.3757142857142854, 'pattern_counts': {'":': 4, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '327162', 'license': 'MIT', 'n_tokens_mistral': 1445, 'n_tokens_neox': 1422, 'n_words': 382, 'path': 't1.ipynb', 'repo_name': 'EdemGold/microsoft-tutorial-pytorch-code', 'size': '4462'}