<jupyter_start><jupyter_text>Segmentation of Indian Traffic<jupyter_code>import math
from PIL import Image, ImageDraw
from PIL import ImagePath
import pandas as pd
import os
from os import path
from tqdm import tqdm
import json
import cv2
import numpy as np
import matplotlib.pyplot as plt
import urllib<jupyter_output><empty_output><jupyter_text>1. You can download the data from this link, and extract it2. All your data will be in the folder "data" 3. Inside the data you will be having two folders|--- data|-----| ---- images|-----| ------|----- Scene 1|-----| ------|--------| ----- Frame 1 (image 1)|-----| ------|--------| ----- Frame 2 (image 2)|-----| ------|--------| ----- ...|-----| ------|----- Scene 2|-----| ------|--------| ----- Frame 1 (image 1)|-----| ------|--------| ----- Frame 2 (image 2)|-----| ------|--------| ----- ...|-----| ------|----- .....|-----| ---- masks|-----| ------|----- Scene 1|-----| ------|--------| ----- json 1 (labeled objects in image 1)|-----| ------|--------| ----- json 2 (labeled objects in image 1)|-----| ------|--------| ----- ...|-----| ------|----- Scene 2|-----| ------|--------| ----- json 1 (labeled objects in image 1)|-----| ------|--------| ----- json 2 (labeled objects in image 1)|-----| ------|--------| ----- ...|-----| ------|----- .....  Task 1: Preprocessing  1. Get all the file name and corresponding json files<jupyter_code>def return_file_names_df(root_dir):
    # write the code that will create a dataframe with two columns ['images', 'json']
    # the column 'image' will have path to images
    # the column 'json' will have path to json files
    
    return data_df
data_df = return_file_names_df(root_dir)
data_df.head()<jupyter_output><empty_output><jupyter_text>> If you observe the dataframe, we can consider each row as single data point, where first feature is image and the second feature is corresponding json file<jupyter_code>def grader_1(data_df):
    for i in data_df.values:
        if not (path.isfile(i[0]) and path.isfile(i[1]) and i[0][12:i[0].find('_')]==i[1][10:i[1].find('_')]):
            return False
    return True
grader_1(data_df)
data_df.shape<jupyter_output><empty_output><jupyter_text>2. Structure of sample Json file  * Each File will have 3 attributes    * imgHeight: which tells the height of the image    * imgWidth: which tells the width of the image    * objects: it is a list of objects, each object will have multiple attributes,        * label: the type of the object        * polygon: a list of two element lists, representing the coordinates of the polygon  Compute the unique labels Let's see how many unique objects are there in the json file.to see how to get the object from the json file please check this blog<jupyter_code>def return_unique_labels(data_df):
    # for each file in the column json
    #       read and store all the objects present in that file
    # compute the unique objects and retrun them
    # if open any json file using any editor you will get better sense of it
    return unique_labels
unique_labels = return_unique_labels(data_df)<jupyter_output><empty_output><jupyter_text><jupyter_code>label_clr = {'road':10, 'parking':20, 'drivable fallback':20,'sidewalk':30,'non-drivable fallback':40,'rail track':40,\
                        'person':50, 'animal':50, 'rider':60, 'motorcycle':70, 'bicycle':70, 'autorickshaw':80,\
                        'car':80, 'truck':90, 'bus':90, 'vehicle fallback':90, 'trailer':90, 'caravan':90,\
                        'curb':100, 'wall':100, 'fence':110,'guard rail':110, 'billboard':120,'traffic sign':120,\
                        'traffic light':120, 'pole':130, 'polegroup':130, 'obs-str-bar-fallback':130,'building':140,\
                        'bridge':140,'tunnel':140, 'vegetation':150, 'sky':160, 'fallback background':160,'unlabeled':0,\
                        'out of roi':0, 'ego vehicle':170, 'ground':180,'rectification border':190,\
                   'train':200}
def grader_2(unique_labels):
    if (not (set(label_clr.keys())-set(unique_labels))) and len(unique_labels) == 40:
        print("True")
    else:
        print("Flase")

grader_2(unique_labels)<jupyter_output>True<jupyter_text>* here we have given a number for each of object types, if you see we are having 21 different set of objects* Note that we have multiplies each object's number with 10, that is just to make different objects look differently in the segmentation map* Before you pass it to the models, you might need to devide the image array /10.  3. Extracting the polygons from the json files<jupyter_code>def get_poly(file):
    # this function will take a file name as argument
    
    # it will process all the objects in that file and returns
    
    # label: a list of labels for all the objects label[i] will have the corresponding vertices in vertexlist[i]
    # len(label) == number of objects in the image
    
    # vertexlist: it should be list of list of vertices in tuple formate 
    # ex: [[(x11,y11), (x12,y12), (x13,y13) .. (x1n,y1n)]
    #     [(x21,y21), (x22,y12), (x23,y23) .. (x2n,y2n)]
    #      .....
    #     [(xm1,ym1), (xm2,ym2), (xm3,ym3) .. (xmn,ymn)]]
    # len(vertexlist) == number of objects in the image
    
    # * note that label[i] and vertextlist[i] are corresponds to the same object, one represents the type of the object
    # the other represents the location
    
    # width of the image
    # height of the image
    return w, h, label, vertexlist
def grader_3(file):
    w, h, labels, vertexlist = get_poly(file)
    print(len((set(labels)))==18 and len(vertexlist)==227 and w==1920 and h==1080 \
          and isinstance(vertexlist,list) and isinstance(vertexlist[0],list) and isinstance(vertexlist[0][0],tuple) )

grader_3('data/mask/201/frame0029_gtFine_polygons.json')<jupyter_output>True<jupyter_text>4. Creating Image segmentations by drawing set of polygons  Example<jupyter_code>import math 
from PIL import Image, ImageDraw 
from PIL import ImagePath  
side=8
x1 = [ ((math.cos(th) + 1) *9, (math.sin(th) + 1) * 6) for th in [i * (2 * math.pi) / side for i in range(side)] ]
x2 = [ ((math.cos(th) + 2) *9, (math.sin(th) + 3) *6) for th in [i * (2 * math.pi) / side for i in range(side)] ]

img = Image.new("RGB", (28,28))
img1 = ImageDraw.Draw(img)
# please play with the fill value
# writing the first polygon
img1.polygon(x1, fill =20)
# writing the second polygon
img1.polygon(x2, fill =30)

img=np.array(img)
# note that the filling of the values happens at the channel 1, so we are considering only the first channel here
plt.imshow(img[:,:,0])
print(img.shape)
print(img[:,:,0]//10)
im = Image.fromarray(img[:,:,0])
im.save("test_image.png")
def compute_masks(data_df):
    # after you have computed the vertexlist plot that polygone in image like this
    
    # img = Image.new("RGB", (w, h))
    # img1 = ImageDraw.Draw(img)
    # img1.polygon(vertexlist[i], fill = label_clr[label[i]])
    
    # after drawing all the polygons that we collected from json file, 
    # you need to store that image in the folder like this "data/output/scene/framenumber_gtFine_polygons.png"
    
    # after saving the image into disk, store the path in a list
    # after storing all the paths, add a column to the data_df['mask'] ex: data_df['mask']= mask_paths
    
    return data_df
data_df = compute_masks(data_df)
data_df.head()
def grader_3():
    url = "https://i.imgur.com/4XSUlHk.png"
    url_response = urllib.request.urlopen(url)
    img_array = np.array(bytearray(url_response.read()), dtype=np.uint8)
    img = cv2.imdecode(img_array, -1)
    my_img = cv2.imread('data/output/201/frame0029_gtFine_polygons.png')    
    plt.imshow(my_img)
    print((my_img[:,:,0]==img).all())
    print(np.unique(img))
    print(np.unique(my_img[:,:,0]))
    data_df.to_csv('preprocessed_data.csv', index=False)
grader_3()<jupyter_output>True
[  0  10  20  40  50  60  70  80  90 100 120 130 140 150 160]
[  0  10  20  40  50  60  70  80  90 100 120 130 140 150 160]<jupyter_text>Task 2: Applying Unet to segment the images * please check the paper: https://arxiv.org/abs/1505.04597* * As a part of this assignment we won't writingt this whole architecture, rather we will be doing transfer learning* please check the library https://github.com/qubvel/segmentation_models* You can install it like this "pip install -U segmentation-models==0.2.1", even in google colab you can install the    same with "!pip install -U segmentation-models==0.2.1" * Check the reference notebook in which we have solved one end to end case study of image forgery detection using same  unet* The number of channels in the output will depend on the number of classes in your data, since we know that we are having 21 classes, the number of channels in the output will also be 21* This is where we want you to explore, how do you featurize your created segmentation map note that the original map will be of (w, h, 1) and the output will be (w, h, 21) how will you calculate the loss, you can check the examples in segmentation github* please use the loss function that is used in the refence notebooks  Task 2.1: Dice loss * Explain the Dice loss* 1. Write the formualtion* 2. Range of the loss function* 3. Interpretation of loss function* 4. Write your understanding of the loss function, how does it helps in segmentation  Task 2.2: Training Unet * Split the data into 80:20.* Train the UNET on the given dataset and plot the train and validation loss.* As shown in the reference notebook plot 20 images from the test data along with its segmentation map, predicted map.  Task 3: Training CANet<jupyter_code>import tensorflow as tf
# tf.compat.v1.enable_eager_execution()
from tensorflow import keras
from tensorflow.keras.layers import *
from tensorflow.keras.preprocessing import image
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.layers import UpSampling2D
from tensorflow.keras.layers import MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import concatenate
from tensorflow.keras.layers import Multiply
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Input, Add, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.utils import plot_model
from tensorflow.keras.initializers import glorot_uniform
K.set_image_data_format('channels_last')
K.set_learning_phase(1)<jupyter_output>D:\installed\Anaconda3\lib\site-packages\tensorflow\python\framework\dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
D:\installed\Anaconda3\lib\site-packages\tensorflow\python\framework\dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
D:\installed\Anaconda3\lib\site-packages\tensorflow\python\framework\dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
D:\installed\Anaconda3\lib\site-packages\tensorflow\python\framework\dtypes.py:529: FutureWarning: Passing (type[...]<jupyter_text>* as a part of this assignment we will be implementing the architecture based on this paper https://arxiv.org/pdf/2002.12041.pdf* We will be using the custom layers concept that we used in seq-seq assignment* You can devide the whole architecture can be devided into two parts    1. Encoder    2. Decoder    * Encoder:    * The first step of the encoder is to create the channel maps [$C_1$, $C_2$, $C_3$, $C_4$]    * $C_1$ width and heigths are 4x times less than the original image    * $C_2$ width and heigths are 8x times less than the original image    * $C_3$ width and heigths are 8x times less than the original image    * $C_4$ width and heigths are 8x times less than the original image    * you can reduce the dimensions by using stride parameter.    * [$C_1$, $C_2$, $C_3$, $C_4$] are formed by applying a "conv block" followed by $k$ number of "identity block". i.e the $C_k$ feature map will single "conv block" followed by $k$ number of "identity blocks".                        * The conv block and identity block of $C_1$: the number filters in the covolutional layers will be $[4,4,8]$ and the number of filters in the parallel conv layer will also be $8$.    * The conv block and identity block of $C_2$: the number filters in the covolutional layers will be $[8,8,16]$ and the number of filters in the parallel conv layer will also be $16$.    * The conv block and identity block of $C_3$: the number filters in the covolutional layers will be $[16,16,32]$ and the number of filters in the parallel conv layer will also be $32$.    * The conv block and identity block of $C_4$: the number filters in the covolutional layers will be $[32,32,64]$ and the number of filters in the parallel conv layer will also be $64$.    * Here $\oplus$ represents the elementwise sum            NOTE: these filters are of your choice, you can explore more options also        * Example: if your image is of size $(512, 512, 3)$        * the output after $C_1$ will be $128*128*8$        * the output after $C_2$ will be $64*64*16$        * the output after $C_3$ will be $64*64*32$        * the output after $C_4$ will be $64*64*64$<jupyter_code>class convolutional_block(tf.keras.layers.Layer):
    def __init__(self, kernel=3,  filters=[4,4,8], stride=1, name="conv block"):
        super().__init__(name=name)
        self.F1, self.F2, self.F3 = filters
        self.kernel = kernel
        self.stride = stride
    def call(self, X):
        # write the architecutre that was mentioned above
        return X
class identity_block(tf.keras.layers.Layer):
    def __init__(self, kernel=3,  filters=[4,4,8], name="identity block"):
        super().__init__(name=name)
        self.F1, self.F2, self.F3 = filters
        self.kernel = kernel
    def call(self, X):
        # write the architecutre that was mentioned above
        return X<jupyter_output><empty_output><jupyter_text>* The output of the $C_4$ will be passed to $\text{Chained Context Aggregation Module (CAM)}$* The CAM module will have two operations names Context flow and Global flow* The Global flow:     * as shown in the above figure first we willl apply  global avg pooling which results in (, 1, 1, number_of_filters) then applying BN, RELU, $1*1 \text{ Conv}$ layer sequentially which results a matrix (, 1, 1, number_of_filters). Finally apply upsampling / conv2d transpose to make the output same as the input dimensions (, input_height, input_width, number_of_filters)    * If you use upsampling then use bilinear pooling as interpolation technique* The Context flow:     * as shown in the above figure (c) the context flow will get inputs from two modules `a. C4` `b. From the above flow`     * We will be concatinating the both inputs on the last axis.    * After the concatination we will be applying  Average pooling  which reduces the size of feature map by $N\times$ times    * In the paper it was mentioned that to apply a group convolutions, but for the assignment we will be applying the simple conv layers with kernel size $(3*3)$    * We are skipping the channel shuffling     * similarly we will be applying a simple conv layers with kernel size $(3*3)$ consider this output is X    * later we will get the Y=(X $\otimes \sigma((1\times1)conv(relu((1\times1)conv(X))))) \oplus X$, here $\oplus$ is elementwise addition and $\otimes$ is elementwise multiplication    * Finally apply upsampling / conv2d transpose to make the output same as the input dimensions (, input_height, input_width, number_of_filters)    * If you use upsampling then use bilinear pooling as interpolation techniqueNOTE: here N times reduction and N time increments makes the input and out shape same, you can explore with the N values, you can choose N = 2 or 4* Example with N=2:    * Assume the C4 is of shape (64,64,64) then the shape of GF will be (64,64,32)    * Assume the C4 is of shape (64,64,64) and the shape of GF is (64,64,32) then the shape of CF1 will be (64,64,32)    * Assume the C4 is of shape (64,64,64) and the shape of CF1 is (64,64,32) then the shape of CF2 will be (64,64,32)    * Assume the C4 is of shape (64,64,64) and the shape of CF2 is (64,64,32) then the shape of CF3 will be (64,64,32)<jupyter_code>class global_flow(tf.keras.layers.Layer):
    def __init__(self, name="global_flow"):
        super().__init__(name=name)
        
    def call(self, X):
        # implement the global flow operatiom
        return X
class context_flow(tf.keras.layers.Layer):    
    def __init__(self, name="context_flow"):
        super().__init__(name=name)
    def call(self, X):
        # here X will a list of two elements 
        INP, FLOW = X[0], X[1] 
        # implement the context flow as mentioned in the above cell
        return X<jupyter_output><empty_output><jupyter_text>* As shown in the above architecture we will be having 4 context flows* if you have implemented correctly all the shapes of Global Flow, and 3 context flows will have the same dimension* the output of these 4 modules will be added to get the same output matrix * The output of after the sum, will be sent to the Feature selection module $FSM$ * Example:    * if the shapes of GF, CF1, CF2, CF3 are (64,64,32), (64,64,32), (64,64,32), (64,64,32), (64,64,32) respectivly then after the sum we will be getting (64,64,32), which will be passed to the next module. Feature selection module:* As part of the FSM we will be applying a conv layer (3,3) with the padding="same" so that the output and input will have same shapes* Let call the output as X* Pass the X to global pooling which results the matrix (, 1, 1, number_of_channels)* Apply $1*1$ conv layer, after the pooling* the output of the $1*1$ conv layer will be passed to the Batch normalization layer, followed by Sigmoid activation function.* we will be having the output matrix of shape (, 1, 1, number_of_channels) lets call it 'Y'* we can interpret this as attention mechanisum, i.e for each channel we will having a weight* the dimension of X (, w, h, k) and output above steps Y is (, 1, 1, k) i.e we need to multiply each channel of X will be multiplied with corresponding channel of Y* After creating the weighted channel map we will be doing upsampling such that it will double the height and width.* apply upsampling with bilinear pooling as interpolation technique* Example:    * Assume the matrix shape of the input is (64,64,32) then after upsampling it will be (128,128,32)<jupyter_code>class fsm(tf.keras.layers.Layer):    
    def __init__(self, name="feature_selection"):
        super().__init__(name=name)
        
    def call(self, X):
        # implement the FSM modules based on image in the above cells
        return FSM_Conv_T<jupyter_output><empty_output><jupyter_text>* Adapted Global Convolutional Network (AGCN):            * AGCN will get the input from the output of the "conv block" of $C_1$        * In all the above layers we will be using the padding="same" and stride=(1,1)        * so that we can have the input and output matrices of same size    * Example:    * Assume the matrix shape of the input is (128,128,32) then the output it will be (128,128,32)<jupyter_code>class agcn(tf.keras.layers.Layer):    
    def __init__(self, name="global_conv_net"):
        super().__init__(name=name)
        
    def call(self, X):
        # please implement the above mentioned architecture
        return X<jupyter_output><empty_output><jupyter_text>*     * as shown in the architecture, after we get the AGCN it will get concatinated with the FSM output* If we observe the shapes both AGCN and FSM will have same height and weight* we will be concatinating both these outputs over the last axis* The concatinated output will be passed to a conv layers with filters = number of classes in our data set and the activation function = 'relu'* we will be using padding="same" which results in the same size feature map* If you observe the shape of matrix, it will be 4x times less than the original image* to make it equal to the original output shape, we will do 4x times upsampling of rows and columns* apply upsampling with bilinear pooling as interpolation technique* Finally we will be applying sigmoid activation.* Example:    * Assume the matrix shape of AGCN is (128,128,32)  and FSM is (128,128,32) the concatination will make it (128, 128, 64)    * Applying conv layer will make it (128,128,21)    * Finally applying upsampling will make it (512, 512, 21)    * Applying sigmoid will result in the same matrix (512, 512, 21)<jupyter_code>X_input = Input(shape=(128,128,3))

# Stage 1
X = Conv2D(64, (3, 3), name='conv1', padding="same", kernel_initializer=glorot_uniform(seed=0))(X_input)
X = BatchNormalization(axis=3, name='bn_conv1')(X)
X = Activation('relu')(X)
X = MaxPooling2D((2, 2), strides=(2, 2))(X)
print(X.shape)<jupyter_output><empty_output><jupyter_text>* If you observe the arcitecture we are creating a feature map with 2x time less width and height* we have written the first stage of the code above.* Write the next layers by using the custom layers we have written<jupyter_code># write the complete architecutre

model = Model(inputs = X, outputs = output)

model.summary()
tf.keras.utils.plot_model(
    model, to_file='model4.png', show_shapes=True, show_layer_names=True,
    rankdir='TB')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11334185070268298, 'fraction_numerical': 0.03590983756159883, 'mean_word_length': 3.626767996622335, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '629626', 'license': 'Apache-2.0', 'n_tokens_mistral': 7116, 'n_tokens_neox': 6551, 'n_words': 2733, 'path': 'Module_29/Segmentation_Assignment.ipynb', 'repo_name': 'VijayRameshkumar/AppliedAI', 'size': '21916'}