<jupyter_start><jupyter_text>**Creating Tensorflow Dataset form Existing Tensors**If the data already exists in the form of a tensor object, a Python list, or a NumPyarray, we can easily create a dataset using the **tf.data.Dataset.from_tensor_slices()**function. This function returns an object of class Dataset, which wecan use to iterate through the individual elements in the input dataset. Data တွေ က tensor type ဒါမှအဟုတ် list,NumPy array အနေနဲ့တည်ရှိနေပြီးသား ဆို tf.data.Dataset.from_slices() function ကိုသုံး ပြီး dataset တစ်ခု ကို ဖန်တီးနိုင်တယ်။ Dataset ရယ့် object အနေနဲ့ return ပြန်ပေးတယ်။ဒီfunction မှာ optional argument အနေနဲ့ drop_reminder ဆိုတာပါတယ်။ ဒီကောင်က Tensor ထဲမာပါဝင်တယ့်element စုစုပေါင်းကို desired batch size နဲ့စားလို့ မပြတ်တယ့်အခါမျိုးမှာ အသုံးဝင်တယ်။ "Create a dataset from list of value"<jupyter_code>a = [1.2, 3.4, 7.5, 4.1, 5.0, 1.0]
ds = tf.data.Dataset.from_tensor_slices(a)
print(ds)<jupyter_output><TensorSliceDataset shapes: (), types: tf.float32><jupyter_text>We can easily iterate through a dataset entry by entry as follows:<jupyter_code>for item in ds :
  print(item)<jupyter_output>tf.Tensor(1.2, shape=(), dtype=float32)
tf.Tensor(3.4, shape=(), dtype=float32)
tf.Tensor(7.5, shape=(), dtype=float32)
tf.Tensor(4.1, shape=(), dtype=float32)
tf.Tensor(5.0, shape=(), dtype=float32)
tf.Tensor(1.0, shape=(), dtype=float32)<jupyter_text>If we want to create batches from this dataset, with a desired batch size of 3, we can do this as follows:<jupyter_code>ds_batch = ds.batch(3)
for i, elem in enumerate(ds_batch, 1):
  print('batch {}:'.format(i), elem.numpy())<jupyter_output>batch 1: [1.2 3.4 7.5]
batch 2: [4.1 5.  1. ]<jupyter_text>**Combining two tensors into a joint dataset**  Often, we may have the data in two (or possibly more) tensors. For example, wecould have a tensor for features and a tensor for labels. In such cases, we need tobuild a dataset that combines these tensors together, which will allow us to retrievethe elements of these tensors in tuples. Tensors Data တွေက 2မျိုး ဒါမှမဟုတ် 2မျိုး ထက်ပိုရှိနိုင်တယ်။ ဥပမာ အားဖြင့် Data ရယ့် Input feature နဲ့ Labels.<jupyter_code>tf.random.set_seed(1)
t_x = tf.random.uniform([4, 3], dtype=tf.float32)  # x hold feature value
t_y = tf.range(4) # y hold class lable
print(t_x)
print(t_y)<jupyter_output>tf.Tensor(
[[0.16513085 0.9014813  0.6309742 ]
 [0.4345461  0.29193902 0.64250207]
 [0.9757855  0.43509948 0.6601019 ]
 [0.60489583 0.6366315  0.6144488 ]], shape=(4, 3), dtype=float32)
tf.Tensor([0 1 2 3], shape=(4,), dtype=int32)<jupyter_text>Now, we want to create a joint dataset from these two tensors. Note that there is a required one-to-one correspondence between the elements of these two tensors:<jupyter_code>ds_x = tf.data.Dataset.from_tensor_slices(t_x)
ds_y = tf.data.Dataset.from_tensor_slices(t_y)
ds_joint = tf.data.Dataset.zip((ds_x, ds_y))
for example in ds_joint:
  print(' x:', example[0].numpy(),
         ' y:', example[1].numpy())<jupyter_output>x: [0.16513085 0.9014813  0.6309742 ]  y: 0
 x: [0.4345461  0.29193902 0.64250207]  y: 1
 x: [0.9757855  0.43509948 0.6601019 ]  y: 2
 x: [0.60489583 0.6366315  0.6144488 ]  y: 3<jupyter_text>Alternatively, we can create the joint dataset using **tf.data.Dataset.from_tensor_slices()** as follows:<jupyter_code>ds_joint = tf.data.Dataset.from_tensor_slices((t_x, t_y))
for example in ds_joint:
  print(' x:', example[0].numpy(),
          ' y:', example[1].numpy())<jupyter_output>x: [0.16513085 0.9014813  0.6309742 ]  y: 0
 x: [0.4345461  0.29193902 0.64250207]  y: 1
 x: [0.9757855  0.43509948 0.6601019 ]  y: 2
 x: [0.60489583 0.6366315  0.6144488 ]  y: 3<jupyter_text>Feature Scaling<jupyter_code>ds_trans = ds_joint.map(lambda x, y: (x*2-1.0, y))
for example in ds_trans:
  print(' x:', example[0].numpy(),
         ' y:', example[1].numpy())<jupyter_output>x: [-0.6697383   0.80296254  0.26194835]  y: 0
 x: [-0.13090777 -0.41612196  0.28500414]  y: 1
 x: [ 0.951571   -0.12980103  0.32020378]  y: 2
 x: [0.20979166 0.27326298 0.22889757]  y: 3<jupyter_text>**Shuffle, batch, and repeat** shuffle လုပ်ခြင်း batch ခွဲ ခြင်း repeat လုပ်ခြင်းဖြင့် model train နေစဉ်မှာoverfit ဖြစ် ခြင်းမှကာကွယ်ပေးနိုင်ပါတယ်။ shuffle ()method မာ ပါဝင်တယ့် buffer_sizeဆိုတာကတော့သူက အဓိက အားဖြင့် datasets ထဲမာ ရှိတယ့် elements အရေအတွက်ကိုsuffle မလုပ်မှီ grouped လိုက်သတ်မှတ်ပေးတာပါ။ Model ကို epoch အများကြီးနဲ့ trainတယ့်အခါမှာ dataset ကိုshuffleလုပ်ခြင်း itertate လုပ်ခြင်း များလိုအပ်ပါတယ်။  ShuffleThe .shuffle() method requires an argument calledbuffer_size, which determines how many elements in the dataset are groupedtogether before shuffling.To ensure complete randomization during each epoch, we can simplychoose a buffer size that is equal to the number of the training examples, as in thepreceding code (buffer_size=len(t_x)).<jupyter_code>tf.random.set_seed(1)
ds = ds_joint.shuffle(buffer_size=len(t_x))
for example in ds:
   print(' x:', example[0].numpy(),
           ' y:', example[1].numpy())<jupyter_output>x: [0.9757855  0.43509948 0.6601019 ]  y: 2
 x: [0.4345461  0.29193902 0.64250207]  y: 1
 x: [0.16513085 0.9014813  0.6309742 ]  y: 0
 x: [0.60489583 0.6366315  0.6144488 ]  y: 3<jupyter_text>Batch<jupyter_code>ds = ds_joint.batch(batch_size=3, drop_remainder=False)
batch_x, batch_y = next(iter(ds))
print('Batch-x:\n', batch_x.numpy())


print('Batch-y: ', batch_y.numpy())<jupyter_output>Batch-y:  [0 1 2]<jupyter_text>RepeatWhen training a model for multiple epochs, we need to shuffle anditerate over the dataset by the desired number of epochs. So, let's repeat the batcheddataset twice:<jupyter_code>ds = ds_joint.batch(3).repeat(count=2)
for i,(batch_x, batch_y) in enumerate(ds):
    print(i, batch_x, batch_y.numpy())<jupyter_output>0 tf.Tensor(
[[0.16513085 0.9014813  0.6309742 ]
 [0.4345461  0.29193902 0.64250207]
 [0.9757855  0.43509948 0.6601019 ]], shape=(3, 3), dtype=float32) [0 1 2]
1 tf.Tensor([[0.60489583 0.6366315  0.6144488 ]], shape=(1, 3), dtype=float32) [3]
2 tf.Tensor(
[[0.16513085 0.9014813  0.6309742 ]
 [0.4345461  0.29193902 0.64250207]
 [0.9757855  0.43509948 0.6601019 ]], shape=(3, 3), dtype=float32) [0 1 2]
3 tf.Tensor([[0.60489583 0.6366315  0.6144488 ]], shape=(1, 3), dtype=float32) [3]<jupyter_text>If we change the order of these twooperations, that is, first batch and then repeat, the results will be different:<jupyter_code>ds = ds_joint.repeat(count=2).batch(3)
for i,(batch_x, batch_y) in enumerate(ds):
  print(i, batch_x.shape, batch_y.numpy())<jupyter_output>0 (3, 3) [0 1 2]
1 (3, 3) [3 0 1]
2 (2, 3) [2 3]<jupyter_text>Combine the operations in the following order: (1) shuffle, (2) batch, and (3) repeat:<jupyter_code>tf.random.set_seed(1)
ds = ds_joint.shuffle(4).batch(2).repeat(3)
for i,(batch_x, batch_y) in enumerate(ds):
  print(i, batch_x.shape, batch_y.numpy())<jupyter_output>0 (2, 3) [2 1]
1 (2, 3) [0 3]
2 (2, 3) [0 3]
3 (2, 3) [1 2]
4 (2, 3) [3 0]
5 (2, 3) [1 2]<jupyter_text>let's try a different order: (2) batch, (1) shuffle, and (3) repeat<jupyter_code>tf.random.set_seed(1)
ds = ds_joint.batch(2).shuffle(4).repeat(3)
for i,(batch_x, batch_y) in enumerate(ds):
  print(i, batch_x.shape, batch_y.numpy())<jupyter_output>0 (2, 3) [0 1]
1 (2, 3) [2 3]
2 (2, 3) [0 1]
3 (2, 3) [2 3]
4 (2, 3) [2 3]
5 (2, 3) [0 1]<jupyter_text>**Creating a dataset from files on your localstorage disk** We will use the pathliblibrary to generate a list of image files:<jupyter_code>import pathlib
imgdir_path = pathlib.Path("/content/drive/My Drive/cat_dog_images" )
file_list = sorted([str(path) for path in imgdir_path.glob('*.jpg')])
print(file_list)<jupyter_output>['/content/drive/My Drive/cat_dog_images/Cat1.jpg', '/content/drive/My Drive/cat_dog_images/Cat2.jpg', '/content/drive/My Drive/cat_dog_images/Cat3.jpg', '/content/drive/My Drive/cat_dog_images/Dog1.jpg', '/content/drive/My Drive/cat_dog_images/Dog2.jpg', '/content/drive/My Drive/cat_dog_images/Dog3.jpg']<jupyter_text>Next, we will visualize these image examples using Matplotlib:<jupyter_code>import matplotlib.pyplot as plt
import os
fig = plt.figure(figsize=(10, 5))
for i, file in enumerate(file_list):
  img_raw = tf.io.read_file(file)
  img = tf.image.decode_image(img_raw)
  print('Image shape: ', img.shape)
  ax = fig.add_subplot(2, 3, i+1)
  ax.set_xticks([]); ax.set_yticks([])
  ax.imshow(img)
  ax.set_title(os.path.basename(file), size=15)
plt.tight_layout()
plt.show()<jupyter_output>Image shape:  (159, 318, 3)
Image shape:  (183, 275, 3)
Image shape:  (183, 275, 3)
Image shape:  (159, 317, 3)
Image shape:  (194, 259, 3)
Image shape:  (194, 259, 3)<jupyter_text>Thelabels for these images are provided within their filenames. So, we extract theselabels from the list of filenames, assigning label 1 to dogs and label 0 to cats:<jupyter_code>labels = [1 if 'Dog' in os.path.basename(file) else 0 for file in file_list]
print(labels)<jupyter_output>[0, 0, 0, 1, 1, 1]<jupyter_text>Now, we have two lists: a list of filenames (or paths of each image) and a list of theirlabels. In the previous section, you already learned two ways of creating a jointdataset from two tensors. Here, we will use the second approach as follows:<jupyter_code>ds_files_labels = tf.data.Dataset.from_tensor_slices((file_list, labels))
for item in ds_files_labels:
  print(item[0].numpy(), item[1].numpy())<jupyter_output>b'/content/drive/My Drive/cat_dog_images/Cat1.jpg' 0
b'/content/drive/My Drive/cat_dog_images/Cat2.jpg' 0
b'/content/drive/My Drive/cat_dog_images/Cat3.jpg' 0
b'/content/drive/My Drive/cat_dog_images/Dog1.jpg' 1
b'/content/drive/My Drive/cat_dog_images/Dog2.jpg' 1
b'/content/drive/My Drive/cat_dog_images/Dog3.jpg' 1<jupyter_text>We have called this dataset ds_files_labels, since it has filenames and labels.Next, we need to apply transformations to this dataset: load the image content fromits file path, decode the raw content, and resize it to a desired size, for example,80 × 120. Previously, we saw how to apply a lambda function using the .map()method. However, since we need to apply multiple preprocessing steps this time,we are going to write a helper function instead and use it when calling the .map()method:<jupyter_code>def load_and_preprocess(path, label):
 image = tf.io.read_file(path)
 image = tf.image.decode_jpeg(image, channels=3)
 image = tf.image.resize(image, [img_height, img_width])
 image /= 255.0
 return image, label

img_width, img_height = 120, 80
ds_images_labels = ds_files_labels.map(load_and_preprocess)
fig = plt.figure(figsize=(10, 6))
for i,example in enumerate(ds_images_labels):
 ax = fig.add_subplot(2, 3, i+1)
 ax.set_xticks([]); ax.set_yticks([])
 ax.imshow(example[0])
 ax.set_title('{}'.format(example[1].numpy()),
 size=15)
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>**Fetching available datasets from the tensorflow_datasets library**<jupyter_code>pip install tensorflow-datasets<jupyter_output>Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.6/dist-packages (2.1.0)
Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (3.12.4)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.15.0)
Requirement already satisfied: dill in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (0.3.2)
Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (20.2.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (4.41.1)
Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.1.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.18.5)
Requirement already satisfied: wrapt in /usr/local/lib/python3.6/dist-packages[...]<jupyter_text>let's import this module and take a look at the list of available datasets:<jupyter_code>import tensorflow_datasets as tfds
print(len(tfds.list_builders()))

print(tfds.list_builders()[:5])<jupyter_output>155
['abstract_reasoning', 'aeslc', 'aflw2k3d', 'amazon_us_reviews', 'arc']
{'fraction_non_alphanumeric': 0.15056142939103354, 'fraction_numerical': 0.09040242603065322, 'mean_word_length': 5.378463146889702, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '500347', 'license': 'MIT', 'n_tokens_mistral': 5830, 'n_tokens_neox': 5240, 'n_words': 1248, 'path': 'coding-exercise/week2/part1/data-processing-with-tensorflow.ipynb', 'repo_name': 'NyanSwanAung/TensorFlow-Beginner', 'size': '12201'}