<jupyter_start><jupyter_text>Multi-label image classification model`predict the different genres just by looking at the poster``Multi-Label Image Classification is different from Multi-Class Image Classification``When we can classify an image into more than one class (as in the image above) or any image that has more than one class ,it is known as a multi-label image classification problem.` `We propose to achieve movie genre classification based only on movie poster images. A deep neural network is constructed to jointly describe visual appearance and object information, and classify a given movie poster image into genres. Because a movie may belong to multiple genres, this is a multi-label image classification problem. To facilitate related studies, we collect a large-scale movie poster dataset, associated with various metadata. Based on this dataset, we fine-tune a pretrained convolutional neural network to extract visual representation, and adopt a state-of-the-art framework to detect objects in posters. ` Understanding the Multi-Label Image Classification Model Architecture`Now, the pre-processing steps for a multi-label image classification task will be similar to that of a multi-class problem. The key difference is in the step where we define the model architecture.``We use a softmax activation function in the output layer for a multi-class image classification model. For each image, we want to maximize the probability for a single class. As the probability of one class increases, the probability of the other class decreases. So, we can say that the probability of each class is dependent on the other classes.``But in case of multi-label image classification, we can have more than one label for a single image. We want the probabilities to be independent of each other. Using the softmax activation function will not be appropriate. Instead, we can use the sigmoid activation function. This will predict the probability for each class independently. It will internally create n models (n here is the total number of classes), one for each class and predict the probability for each class.``Using sigmoid activation function will turn the multi-label problem to n – binary classification problems. So for each image, we will get probabilities defining whether the image belongs to class 1 or not, and so on. Since we have converted it into a n – binary classification problem, we will use the binary_crossentropy loss. Our aim is to minimize this loss in order to improve the performance of the model.` `Our aim is to predict the genre of a movie using just its poster image. Can you guess why it is a multi-label image classification problem? Think about it for a moment before you look below.A movie can belong to more than one genre, right? It doesn’t just have to belong to one category, like action or comedy. The movie can be a combination of two or more genres. Hence, multi-label image classification.` get the dataset from here https://www.cs.ccu.edu.tw/~wtchu/projects/MoviePoster/index.html<jupyter_code>!wget https://www.cs.ccu.edu.tw/~wtchu/projects/MoviePoster/Movie_Poster_Dataset.zip
!wget https://www.cs.ccu.edu.tw/~wtchu/projects/MoviePoster/Movie_Poster_Metadata.zip<jupyter_output>--2019-10-21 10:05:23--  https://www.cs.ccu.edu.tw/~wtchu/projects/MoviePoster/Movie_Poster_Metadata.zip
Resolving www.cs.ccu.edu.tw (www.cs.ccu.edu.tw)... 192.168.127.12
Connecting to www.cs.ccu.edu.tw (www.cs.ccu.edu.tw)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1529967 (1.5M) [application/zip]
Saving to: ‘Movie_Poster_Metadata.zip’

Movie_Poster_Metada 100%[===================>]   1.46M   954KB/s    in 1.6s    

2019-10-21 10:05:26 (954 KB/s) - ‘Movie_Poster_Metadata.zip’ saved [1529967/1529967]<jupyter_text>`This dataset was collected from the IMDB website. One poster image was collected from one (mostly) Hollywood movie released from 1980 to 2015. Each poster image is associated with a movie as well as some metadata like ID, genres, and box office. The ID of each image is set as its file name.`<jupyter_code>!unzip Movie_Poster_Dataset.zip
!unzip Movie_Poster_Metadata.zip
# We've taken iamges from 1980 to 2015 ---> 36 images
import os
path = '/content/Movie_Poster_Dataset/'
folders = os.listdir(path)

print('Total Number of folders of movies based on year : ' , len(folders))
image_set = []
movie_set = {}
for year in folders:
  liss = os.listdir(path + year + '/')
  for i in liss:
    image_set+=[path + year + '/' +i]
  movie_set[year] = len(liss)
  
print(len(movie_set))
print(movie_set)
#Now calculate the Total number of images 
l=0
for k,v in movie_set.items():
  l+=v
print("TOtal Number of Images in the folder from 1980 to 2015 : " , l)
print(len(image_set))
image_set[:5]<jupyter_output>8052<jupyter_text>Use Machine learning library skimage to visualize the image<jupyter_code>from skimage.io import imread, imshow
import matplotlib.pyplot as plt
%matplotlib inline
image_org = imread(image_set[0])
print(image_org.shape)
imshow(image_org)<jupyter_output>(450, 300, 3)<jupyter_text>Now we will create another folder and save all the image in same folder together for easy access<jupyter_code>!mkdir Images    
from PIL import Image  
import PIL  
#Save all the images in anothr folder 
for i in image_set:
  im1 = Image.open(i)  
  im1 = im1.save("/content/Images/" +i.split('Dataset/')[1].split('/')[0] + '_' +i.split('Dataset/')[1].split('/')[1] ) 
#Check the Image folder now 
container = os.listdir("/content/Images/")
print(len(container))
container[:5]
#Check whether all the image different or same
print(len(set(container)))
from skimage.io import imread, imshow
import matplotlib.pyplot as plt
%matplotlib inline
image_org = imread("/content/Images/" + container[0])
print(image_org.shape)
imshow(image_org)<jupyter_output>(451, 300, 3)<jupyter_text>Importing all the necessary libraries<jupyter_code>import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.utils import to_categorical
from keras.preprocessing import image
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from tqdm import tqdm
%matplotlib inline
meta_data_fold = os.listdir('/content/groundtruth/')
print(len(meta_data_fold))
meta_data_fold[0] , meta_data_fold[1]<jupyter_output>36<jupyter_text>https://stackoverflow.com/a/58405718/8950300 Create the dataset by extracting iformation from the text files<jupyter_code>column = ['id','release','rating','imdbID','imdbVotes','genre','title','writers', 'actors','types','box_money','time','lang','country','awards','meta_score', 'director','rated']  

import pandas as pd
data = pd.DataFrame(columns = column)
data

for jk in range(36):
  file = '/content/groundtruth/' +meta_data_fold[jk]
  file1 = open(file,"rb")  
  st = file1.read()
  st.decode('UTF-16')
  import re
  import ast
  strim = st.decode('UTF-16')
  strim = re.sub('\r\n}' , '}' ,strim )
  strim = re.sub('{\r\n  ' , '{' ,strim )
  strim = re.sub(',\r\n  ' , '|' ,strim )
  strim = re.sub('\r\n' , '|' ,strim )
  strim = strim.strip('\r\n')
  strim = re.sub('{' , '' ,strim )
  strim = re.sub('}' , '' ,strim )
  # print(strim)
  liss = strim.split('|')
  # print("Old length : " , len(liss))
  del liss[-1]
  # print("New length : " , len(liss))
  l = len(liss)
  # print(len(liss)//20)
  # print(liss)
  count = 0
  id = []
  release = []
  rating = []
  imdbID = []
  imdbVotes = []
  genre = []
  title = []
  writers = []
  actors = []
  types = []
  box_money = []
  time = []
  lang = []
  country = []
  awards = []
  meta_score = []
  director = []
  rated = []

  for i in range(0,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , liss[i].split(':')[1].strip())
    id.append(liss[i].split(':')[1].strip().split('ObjectId("56fbcde771a0651bcc')[1].split('")')[0])


  for i in range(1,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    if(ast.literal_eval(liss[i].split(':')[1].strip())!='N/A'):
      release.append('-'.join(ast.literal_eval(liss[i].split(':')[1].strip()).split(' ')))
    else:
      release.append('-')


  for i in range(2,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    if(ast.literal_eval(liss[i].split(':')[1].strip())!='N/A'):
      rating.append(float(ast.literal_eval(liss[i].split(':')[1].strip())))
    else:
      rating.append(0.0)



  for i in range(3,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    imdbID.append(ast.literal_eval(liss[i].split(':')[1].strip()))


  for i in range(4,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    if( ast.literal_eval(liss[i].split(':')[1].strip())!= 'N/A'):
      imdbVotes.append(ast.literal_eval(liss[i].split(':')[1].strip()))
    else:
      imdbVotes.append(0)


  for i in range(5,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    hal = [j.strip() for j in ast.literal_eval(liss[i].split(':')[1].strip()).split(',')]
    genre.append('/'.join(hal))




  for i in range(6,l , 20):
    count+=1
  #   print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    try:
      title.append(ast.literal_eval(liss[i].split(':')[1].strip()))
    except:
      title.append(liss[i].split(':')[1].strip().split('"')[1])




  for i in range(7,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    writers.append(ast.literal_eval(liss[i].split(':')[1].strip()).split(','))


  for i in range(8,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    actors.append(ast.literal_eval(liss[i].split(':')[1].strip()))


  for i in range(9,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    types.append(ast.literal_eval(liss[i].split(':')[1].strip()))



  for i in range(10,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    lang.append(ast.literal_eval(liss[i].split(':')[1].strip()).split())

  for i in range(12,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    time.append('-'.join(ast.literal_eval(liss[i].split(':')[1].strip()).split()))



  for i in range(13,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    if(liss[i].split(':')[1].strip()!='null'):
      box_money.append(int(''.join(ast.literal_eval(liss[i].split(':')[1].strip()).split(','))))
    else:
      box_money.append(0)


  for i in range(14,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    country.append('/'.join(ast.literal_eval(liss[i].split(':')[1].strip()).split(',')))


  for i in range(15,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    if(ast.literal_eval(liss[i].split(':')[1].strip())!='N/A'):
      awards.append(ast.literal_eval(liss[i].split(':')[1].strip()))
    else:
      awards.append('None')



  for i in range(16,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))
    if(ast.literal_eval(liss[i].split(':')[1].strip())!='N/A'):
      meta_score.append(ast.literal_eval(liss[i].split(':')[1].strip()))
    else:
      meta_score.append(0)



  for i in range(17,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))

    if(ast.literal_eval(liss[i].split(':')[1].strip())!='N/A'):
      director.append(ast.literal_eval(liss[i].split(':')[1].strip()))
    else:
      director.append('-')



  for i in range(18,l , 20):
    count+=1
    #print(count, ast.literal_eval((liss[i].split(':')[0].strip())) , ast.literal_eval(liss[i].split(':')[1].strip()))

    if(ast.literal_eval(liss[i].split(':')[1].strip())!='N/A'):
      rated.append(ast.literal_eval(liss[i].split(':')[1].strip()))
    else:
      rated.append('-')

  df = pd.DataFrame()
  df['id'] = id
  df['release'] = release
  df['rating'] = rating
  df['imdbID'] = imdbID
  df['imdbVotes'] = imdbVotes
  df['genre'] = genre
  df['title'] = title 
  df['writers'] = writers
  df['actors'] = actors
  df['types'] = types
  df['box_money'] = box_money
  df['time'] = time
  df['lang'] = lang
  df['country'] = country 
  df['awards'] =awards
  df['meta_score'] = meta_score 
  df['director'] = director
  df['rated'] = rated
  data = pd.concat([data, df], ignore_index=True)
  
  
  
print(data.shape)
data.head()<jupyter_output><empty_output><jupyter_text>Now we are gonna mount pre structured dataset here from the google drive<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
!unzip '/content/drive/My Drive/Multi_Label_dataset.zip'
import os
driv = os.listdir('/content/Multi_Label_dataset/')
driv   #here we have another folder inside the 'Multi_Label_dataset' folder<jupyter_output><empty_output><jupyter_text>This data set file is generated from the poster meta data<jupyter_code>import pandas as pd
train = pd.read_csv('/content/Multi_Label_dataset/'  + driv[0])    # reading the csv file
print(train.shape)
train.head()<jupyter_output>(7254, 27)<jupyter_text>`Here we have more than one genre for a movie.`<jupyter_code>import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.utils import to_categorical
from keras.preprocessing import image
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from tqdm import tqdm
%matplotlib inline
train_image = []
for i in tqdm(range(2000)):
    img = image.load_img('/content/Multi_Label_dataset/'  + driv[1] + '/'+train['Id'][i]+'.jpg',target_size=(400,400,3))
    img = image.img_to_array(img)
    img = img/255
    train_image.append(img)
X = np.array(train_image)
X.shape
print(train['Genre'][2])
plt.imshow(X[2])<jupyter_output>['Comedy']<jupyter_text>`We will remove the Id and genre columns from the train file and convert the remaining columns to an array which will be the target for our images:`<jupyter_code>y = np.array(train[:2000].drop(['Id', 'Genre'],axis=1))
y.shape
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=0.1)<jupyter_output><empty_output><jupyter_text>`The next step is to define the architecture of our model. The output layer will have 25 neurons (equal to the number of genres) and we’ll use sigmoid as the activation function.`<jupyter_code>model = Sequential()
model.add(Conv2D(filters=16, kernel_size=(5, 5), activation="relu", input_shape=(400,400,3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))


model.add(Conv2D(filters=32, kernel_size=(5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))



model.add(Conv2D(filters=64, kernel_size=(5, 5), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))



model.add(Conv2D(filters=64, kernel_size=(5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))


model.add(Flatten())    #convert multidimensional to one dimensional


model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))


model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))


model.add(Dense(25, activation='sigmoid'))  #output layer
model.summary()
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test), batch_size=64)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3005: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Train on 1800 samples, validate on 200 samples
Epoch 1/10
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.[...]<jupyter_text>check model's performance on unseen images<jupyter_code>img = image.load_img('GOT.jpg',target_size=(400,400,3))
img = image.img_to_array(img)
img = img/255<jupyter_output><empty_output><jupyter_text>`Now, we will predict the genre for these posters using our trained model. The model will tell us the probability for each genre and we will take the top 3 predictions from that.``we will make predictions for the GOT & AVENGERS posters`<jupyter_code>classes = np.array(train[:2000].columns[2:])
proba = model.predict(img.reshape(1,400,400,3))
proba
np.argsort(proba[0])
top_3 = np.argsort(proba[0])[:-4:-1]
top_3
for i in range(3):
    print("{}".format(classes[top_3[i]])+" ({:.3})".format(proba[0][top_3[i]]))
plt.imshow(img)<jupyter_output>Drama (0.503)
Comedy (0.439)
Crime (0.233)<jupyter_text>`Impressive! Our model suggests Drama, Thriller and Action genres for Game of Thrones. That classifies GoT pretty well. Let’s try our model on the Avengers poster. Preprocess the image:`<jupyter_code>img = image.load_img('avengers.jpeg',target_size=(400,400,3))
img = image.img_to_array(img)
img = img/255
classes = np.array(train.columns[2:])
proba = model.predict(img.reshape(1,400,400,3))
print(np.argsort(proba[0]))
top_3 = np.argsort(proba[0])[:-4:-1]
top_3
for i in range(3):
    print("{}".format(classes[top_3[i]])+" ({:.3})".format(proba[0][top_3[i]]))
plt.imshow(img)<jupyter_output>Drama (0.503)
Comedy (0.44)
Crime (0.229)<jupyter_text>`The genres our model comes up with are Drama, Action and Thriller. Again, these are pretty accurate results. Can the model perform equally well for Bollywood movies ? Let’s find out. We will use this Golmal 3 poster.`<jupyter_code>img = image.load_img('golmal.jpeg',target_size=(400,400,3))
img = image.img_to_array(img)
img = img/255
classes = np.array(train.columns[2:])
proba = model.predict(img.reshape(1,400,400,3))

print(np.argsort(proba[0]))
top_3 = np.argsort(proba[0])[:-4:-1]
top_3

for i in range(3):
    print("{}".format(classes[top_3[i]])+" ({:.3})".format(proba[0][top_3[i]]))
plt.imshow(img)<jupyter_output>Drama (0.504)
Comedy (0.443)
Crime (0.227)<jupyter_text>`Success ! Golmaal 3 was a comedy and our model has predicted it as the topmost genre. The other predicted genres are Drama and Romance – a relatively accurate assessment. We can see that the model is able to predict the genres just by seeing their poster.` `we should make the dataset in such a way that all the genre categories will have comparatively equal distribution. because if a certain genre is repeating in most of the training images, our model might overfit on that genre. And for every new image, the model might predict the same genre. To overcome this problem, you should try to have an equal distribution of genre categories.`<jupyter_code><jupyter_output><empty_output><jupyter_text>others how to decode byte 0xff in python` as we know this hexadecimal encoding so , utf-8 , codec and other decoders are not able to decode this byte string`     `here we will use 'UTF-16'  or 'utf-16' encoding to decode the oxff byte array`<jupyter_code>k ="this world is very beautiful"
k.encode('utf-16')
k = b'\xff\xfet\x00h\x00i\x00s\x00 \x00w\x00o\x00r\x00l\x00d\x00 \x00i\x00s\x00 \x00v\x00e\x00r\x00y\x00 \x00b\x00e\x00a\x00u\x00t\x00i\x00f\x00u\x00l\x00'
k.decode('utf-16')
k = b'\xff\xfe{\x00\r\x00\n\x00 \x00 \x00"\x00_\x00i\x00d\x00'
print(str(k, "utf-16"))
print(k.decode('UTF-16'))<jupyter_output>{
  "_id
{
  "_id
{'fraction_non_alphanumeric': 0.16408434020374318, 'fraction_numerical': 0.0335939350864724, 'mean_word_length': 4.999431495167709, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 59, 'https://': 5, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '636119', 'license': 'MIT', 'n_tokens_mistral': 7890, 'n_tokens_neox': 7385, 'n_words': 1822, 'path': 'Multilabel_classification/code.ipynb', 'repo_name': 'priyanshgupta1998/Image_Processing', 'size': '21103'}