<jupyter_start><jupyter_text>__Team 3 Final Project: Ships in San Francisco__<NAME>, <NAME>, <NAME>December 15, 2021 ________ 1.0 Problem Statement________ There is a great need for data analysis within the image analysis realm. Being able to analyze images quickly and effectively becomes essential as technology advances - from detecting diseases off of a CT scan, to finding missing persons on CCTV. There is also huge potential for the shipping industry to benefit from image analysis in many ways. In this notebook we will be training a convolutional neural network (CNN) to identify ships using commercial satellite images of San Francisco Bay. The images we will be working with are available via [Kaggle](https://www.kaggle.com/rhammell/ships-in-satellite-imagery)) and includes a classification of ship and no ship. Others have already trained a CNN using the the existing classifications with more than a 99% accuracy, however in reality the classifications for no ship is not accurate because approximately one third of those images are of partial ships. We hope to train a model to accurately recognize partial ships so they can also be considered as ships in decision making. This is most useful for private port opporators or governments so they can better identify potential clogs and better spread out import/export docking dates and times to prevent what happened recently in Long Beach (thedrive.com [link text](https://www.thedrive.com/the-war-zone/42612/satellite-images-show-massive-armada-of-idle-cargo-ships-waiting-to-dock-in-long-beach)). Such entities would be better served to leverage software that can identify the total number of ships plus half of total number of partial ships (assume 2 partials make a whole ship at the current zoom level). ________ 2.0 Dataset________ As the Kaggle website explains, the dataset consists of image chips extracted from Planet satellite imagery collected over the San Francisco Bay and San Pedro Bay areas of California and distributed as a JSON formatted text file shipsnet.json. The JSON contains:*   ***data*** - list of 19200 integers with the first 6400 entries contain the red channel values, the next 6400 the green, and the final 6400 the blue*   ***label*** - Values of 1 or 0, representing the "ship" class and "no-ship" classification*   ***locations*** - list of latitude and longitude coordinates of the image center point*   ***scene_ids*** - The unique identifier of the PlanetScope visual scene the image chip was extracted from In addition, we created our own second JSON file of partial ships data 'partial_ships_data.json'. It is a list of 436 unique latitude and longitude coordinate pairs that correspond to a partial ship images. To create this, we first manually examined all 4,000 images to isolate just those of partial ship images. We then manipulated this file list in EXCEL accordingly:we first removed the filetype from the image name, then using the RIGHT function to extract the 37 characters at the end of the filename which correspond to the longitude and latitude values of the image, then did a find and replace to change underscores and spaces for commas and brackets as needed to create the correct JSON format.  2.1 Import Libraries<jupyter_code>import pandas as pd
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt 
!pip install mgrs
import mgrs<jupyter_output>Collecting mgrs
  Downloading mgrs-1.4.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (47 kB)
[?25l
[K     |██████▉                         | 10 kB 17.0 MB/s eta 0:00:01
[K     |█████████████▊                  | 20 kB 20.5 MB/s eta 0:00:01
[K     |████████████████████▋           | 30 kB 12.1 MB/s eta 0:00:01
[K     |███████████████████████████▌    | 40 kB 9.6 MB/s eta 0:00:01
[K     |████████████████████████████████| 47 kB 2.3 MB/s 
[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from mgrs) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->mgrs) (3.0.6)
Installing collected packages: mgrs
Successfully installed mgrs-1.4.2<jupyter_text>2.2 Import JSON 2.2.1. If using local environment You will need to download two .json files to your local directory, where this notebook is saved. The two files are listed below with the link to the Google Drive where they are stored. Please contact any of our team members if you have trouble downloading these files. Also be aware that one of these files is over 300 MBshipsnet.jsonpartial_ship_data.jsonhttps://drive.google.com/drive/folders/1NY1helYQlz3Af9mpOmJSy0nPBKtRYPMY?usp=sharing<jupyter_code>#Load JSON from local root folder
##import os
##cwd = os.getcwd()
##shipdata = pd.read_json(os.path.join(cwd,"shipsnet.json"))
##partialshipdata = pd.read_json(os.path.join(cwd,"partial_ship_data.json"))<jupyter_output><empty_output><jupyter_text>2.2.2. If using Google Colab<jupyter_code>#Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')
#Save JSON data as variable
shipdata = pd.read_json('/content/drive/MyDrive/EM-0213/shipsnet/shipsnet.json')
partialshipdata = pd.read_json('/content/drive/MyDrive/EM-0213/partial_ship_data.json')<jupyter_output><empty_output><jupyter_text>2.3 Read JSON<jupyter_code># Read the shipsnet.json to inspect the objects
shipdata.head()
# Read the partialshipdata.json to inspect the objects
partialshipdata.head()<jupyter_output><empty_output><jupyter_text>________ 3.0 Data Cleaning________ Our first model identifies ships using using Tensorflow and Keras. To prepare the data for this model, we first renamed the "data" column to "pixel_data" and "labels" column to "ships" to be more specific. We then used lambda processing to convert pixel data to a NumPy array then divided by 255 to normalize the values. Finally we reshaped the data (assumes 80 pixels by 80 pixels and 3 color bands - RGB).<jupyter_code>#rename data column
shipdata=shipdata.rename(columns={"data": "pixel_data"})
#rename labels column
shipdata=shipdata.rename(columns={"labels": "ships"})
# Normalize and reshape the image data
shipdata["normalized_pixel_data"] = shipdata["pixel_data"].apply(lambda x: (np.array(x) / 255).reshape(80, 80, 3))<jupyter_output><empty_output><jupyter_text>Our second model identifies partial ships using similar methodology. For this we need a new column called partial_ship to stor classifications of 1 or 0 for 'partial ship' or 'no ship' respectively. After manually preparting our partial ships data JSON as described in 2.0 above, we use it lookup to see if a coordinate listed in shipsdata exists in partialshipsdata. If so, then convert partial_ship classification to 1. To make this easier to find a match, we first separate 'locations' data into separate 'latitude' and 'longitude' columns and convert to MSGR unit which is a single location string to work with.<jupyter_code># Create a new column of zeros to the shipdata dataframe called 'partial_ships'
shipdata['partial_ships'] = 0
# Rename data column
partialshipdata=partialshipdata.rename(columns={0: "latitude"})
# Rename data column
partialshipdata=partialshipdata.rename(columns={1: "longitude"})
# Split 'locations' column into two separate columns for 'latitude' and 'longitude'
shipdata[['latitude','longitude']] = pd.DataFrame(shipdata.locations.tolist(), index= shipdata.index)
# Convert latitude and longitude in shipdata to a float
shipdata['latitude'] = shipdata['latitude'].astype(float);
shipdata['longitude'] = shipdata['longitude'].astype(float);
# Convert latitude and longitude in partialshipdata to a float
partialshipdata['latitude'] = partialshipdata['latitude'].astype(float);
partialshipdata['longitude'] = partialshipdata['longitude'].astype(float);
# Convert latitude and longitude to MGRS for easier lookup and matching between the files; 
# This will assist us to assign partial ship classification to the correct impages in shipsdata
shipdata['mgrs'] = 0
partialshipdata['mgrs'] = 0
# Convert latitude and longitude in shipdata to MGRS
m = mgrs.MGRS()
shipdata.astype({'mgrs': 'object'}).dtypes
for x in range(len(shipdata)):
  a = m.toMGRS(shipdata.longitude[x],shipdata.latitude[x])
  shipdata.mgrs[x] = a
# Read shipdata head to confirm MGRS conversion
shipdata.head()
# Convert longitude and latitude in partialshipdata to MGRS
m = mgrs.MGRS()
partialshipdata.astype({'mgrs': 'object'}).dtypes
for x in range(len(partialshipdata)):
  a = m.toMGRS(partialshipdata.longitude[x],partialshipdata.latitude[x])
  partialshipdata.mgrs[x] = a
# Read partialshipdata head to confirm MGRS conversion
partialshipdata.head()
# Assign a partial_ship value of 1 for rows that have a longitude value in list of partial_ships_longitude 
shipdata.loc[shipdata['mgrs'].isin(partialshipdata['mgrs']), "partial_ships"] = 1
# As a test, select rows whose longitude column value equals 10SEG5707378217 to see if it has a partial ship value of 1
df2=shipdata.loc[shipdata['mgrs'] == "10SEG5707378217"]
df2.head()
# As a test, select rows whose longitude column value equals 10SEG4449172490 to see if it has a partial ship value of 0
df2=shipdata.loc[shipdata['mgrs'] == "10SEG4449172490"]
df2.head()<jupyter_output><empty_output><jupyter_text>Additional checks reveal no missing or duplicate data (NOTE: we confirmed all image file names were unique via excel).<jupyter_code># Check to see if dataframe has any missing data
np.where(pd.isnull(shipdata))<jupyter_output><empty_output><jupyter_text>________ 4.0 Visualizations________ Our RBG histogram visualization of a few images can be found below. We also have provided visuals throughout our analysis, including looking at the image inputs to the model and then looking at the saliency of the evaluated image after the model has been generated.  Histograms of Images<jupyter_code># import json data to create numpy arrays for visual analysis
# same data set as above, however reloaded for simplicity
import json
# f = open(os.path.join(cwd,"shipsnet.json"))
f = open(r'/content/drive/MyDrive/EM-0213/shipsnet/shipsnet.json')
dataset = json.load(f) # load the json file
f.close()
data = np.array(dataset['data']).astype('uint8') # image data
ifship = np.array(dataset['labels']).astype('uint8') # labels is binary ship or no ship classification
n_spectrum = 3 # color channel (RGB)
width = 80
height = 80
data_reshape = data.reshape([-1, 3, 80, 80]).transpose([0,2,3,1]) # transpose data for image processing
data_reshape = data_reshape/255 # normalize the pixel data
print(data_reshape.shape) 
# check shape is compatible for plotting image, have 4000 images 80x80 pixels and 3 color channels
pd.Series(ifship).value_counts() # check how many ships/1 or no ships/0
# create function to plot images and RBG histograms
# building off of https://www.kaggle.com/paultimothymooney/identifying-ships-in-satellite-images

def plothisto(a,b,c):
    '''a is the array we're looking at, b is the binary classification array,
    c is the index or row'''
    a = a[c] # call the image
    plt.figure(figsize=(10,5)) # establish plotting figures of image and histogram
    plt.subplot(1,2,1) # create subplots to show both image and histogram
    plt.imshow(a) # show image of ship or no ship, note this is the actual known result, not predicted
    plt.axis('off')
    plt.title('Ship' if b[c]==1 else 'Not A Ship') # check ship or no ship category to label image. ship is 1
    histo = plt.subplot(1,2,2) 
    histo.set_ylabel('Count') # histrogram axes labels
    histo.set_xlabel('Pixel Intensity')
    n_bins = 30 
    # evaluate pixels for each red, green, blue intensities
    plt.hist(a[:,:,0].flatten(), bins= n_bins, lw = 0, color='r', alpha=0.5);
    plt.hist(a[:,:,1].flatten(), bins= n_bins, lw = 0, color='g', alpha=0.5);
    plt.hist(a[:,:,2].flatten(), bins= n_bins, lw = 0, color='b', alpha=0.5);
# look at the data for four random images (chosen random ship and no ship images)
plothisto(data_reshape,ifship,500)
plothisto(data_reshape,ifship,750)
plothisto(data_reshape,ifship,2500)
plothisto(data_reshape,ifship,3500)<jupyter_output><empty_output><jupyter_text>We do not see any pattern withing the RBG histograms for ship vs no ship classification. We will look at salience after evaluating our models to see what the model sees in the image to understand how the model "looks" our images. ________ 5.0 Model 1: Identifying Ships in Satellite Imagery________ Model built upon tutorial here: https://towardsdatascience.com/classifying-ships-in-satellite-imagery-with-neural-networks-944024879651 We chose this model because we wanted to extend  a model that had a 99% accuracy for full ship. As a first step we split the data into training and testing sets and use a 80/20 split. The model will train on 3200 images and validate its results on 800 other images.  Issue with the codeAfter investigation, we found that the tutorial was processing the pixel data incorrectly, looking at a 3x3 B&W plot instead of the image as is. For our study, we want to analyze images as-is. Below you will see the transposed pixel data so image appears as-is.<jupyter_code># Define X and Y
X = shipdata["normalized_pixel_data"]
Y = shipdata["ships"]
# Split the data into training and testing sets
from sklearn.model_selection import train_test_split
(X_train, X_test, Y_train, Y_test) = train_test_split(X, Y, test_size=0.20, random_state=42)
# Transform the training and testing data into arrays for TensorFlow
X_train = np.array([x for x in X_train])
X_test = np.array([x for x in X_test])
Y_train = np.array([y for y in Y_train])
Y_test = np.array([y for y in Y_test])
print(Y_test[750]) # see if image is ship or no ship classification in test and training arrays
print(Y_train[750])

# show images from pixel data for known ship and no ship
fig, axes = plt.subplots(1,2,figsize=(14,5)) # plot two images side by side
axes[0].imshow(X_test[750])
axes[1].imshow(X_train[750])

# tutorial did not process images in the best format
# Need to transpose pixels for image to be displayed/analyzed correctly
X_test = X_test.reshape([-1, 3, 80, 80]).transpose([0,2,3,1]) # transpose shape data
X_train = X_train.reshape([-1, 3, 80, 80]).transpose([0,2,3,1])
# verify images are processed correctly
fig, axes = plt.subplots(1,2,figsize=(14,5)) # plot two images side by side
axes[0].imshow(X_test[750])
axes[1].imshow(X_train[750])

# images look as they do as a png. OK to proceed with this data for the model
from tensorflow.keras import datasets, layers, models
from tensorflow.keras.layers import Activation
# Starts the model with a sequential ANN
model = models.Sequential()
# Adds the first convulsion layer and follows up with max pooling
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(80, 80, 3)))
model.add(layers.MaxPooling2D((2, 2)))
# Add additional hidden layers
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
# Flattens the input into a 1D tensor
model.add(layers.Flatten())
# Makes the input more readable for classification
model.add(layers.Dense(64, activation='relu'))
# Classifies - ensure the input in the number of classes, indexed
# at 0
model.add(layers.Dense(1))
# Final activation function
model.add(Activation('sigmoid'))
model.summary()
# Compile the model
# Use binary_crossentropy because there are only 2 classes present
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
# Train the model
gen_model = model.fit(X_train, Y_train, epochs=10, validation_data=(X_test, Y_test))
# Evaluate the model
from sklearn.metrics import classification_report, confusion_matrix
predictions = model.predict(X_test)
print(classification_report(Y_test, predictions.round()))
cf_matrix = confusion_matrix(Y_test, predictions.round())
print(cf_matrix)
# plot heat map of correct and incorrect results for ship and no ship classifications
import seaborn as sns
sns.heatmap(cf_matrix, annot=True)<jupyter_output><empty_output><jupyter_text>Look at salience to understand what model sees<jupyter_code># Salience evaluation built upon tutorial here https://usmanr149.github.io/urmlblog/cnn/2020/05/01/Salincy-Maps.html

import keras

# creating function to show image being processed by the model and the 
# salience which the model sees of the image
def plotsal(a,b):
    '''a is the pixel array we're looking at, b is index'''
    _img = a[b] #recall an image to view saliency on

    img = keras.preprocessing.image.img_to_array(_img) # load the array
    img = img.reshape((1, *img.shape)) # match the required shape for model prediction
    y_pred_model = model.predict(img).round() # predict outcome of the model
    y_result = 'Ship' if y_pred_model==1 else 'Not A Ship' # want to show what the model predicts for the saliency image
    #print("Predicted value is " + str(y_pred_model1) + " so model predicts " + y_result + " is present .")
    images = tf.Variable(img, dtype=float) 

    with tf.GradientTape() as tape:
        pred = model(images, training=False)
        class_idxs_sorted = np.argsort(pred.numpy().flatten())[::-1]
        loss = pred[0][class_idxs_sorted[0]]
    # plot original images with saliency image and predicted result
    grads = tape.gradient(loss, images)
    dgrad_abs = tf.math.abs(grads)
    dgrad_max_ = np.max(dgrad_abs, axis=3)[0]
    arr_min, arr_max  = np.min(dgrad_max_), np.max(dgrad_max_)
    grad_eval = (dgrad_max_ - arr_min) / (arr_max - arr_min + 1e-18)
    fig, axes = plt.subplots(1,2,figsize=(14,5))
    axes[0].imshow(_img)
    plt.title('Ship Predicted' if y_pred_model==1 else 'No Ship Predicted')
    i = axes[1].imshow(grad_eval,cmap="jet",alpha=0.8)
    fig.colorbar(i)

plotsal(X_test,100)
plotsal(X_test,200)
plotsal(X_test,300)
plotsal(X_test,750)<jupyter_output><empty_output><jupyter_text>Salience Result of Model 1Looking at the ship vs. no ship gradient plots, we see that there is recognition in a line shape for ships while the no ship images have a single point or points that are no in a line. This may be how the model recognizes when there is a ship in the image ________ 6.0 Model 2: Identifying Partial Ships in Satellite Imagery________ For Model 2, we will be looking at images only for images with partial ships or no ships. This model will see how well we can predict if a partial ship is present.<jupyter_code># Define X and Y
data_10086 = shipdata.loc[shipdata['ships'] == 0] # load just the no ship images

X = data_10086["normalized_pixel_data"] # looking at the normalized data
Y = data_10086["partial_ships"] # look at the partial ship or no ship known data for each image
# Split the data into training and testing sets
from sklearn.model_selection import train_test_split
(X_train, X_test, Y_train, Y_test) = train_test_split(X, Y, test_size=0.20, random_state=42)
# Transform the training and testing data into arrays for TensorFlow
X_train = np.array([x for x in X_train])
X_test = np.array([x for x in X_test])
Y_train = np.array([y for y in Y_train])
Y_test = np.array([y for y in Y_test])
print(Y_test[200]) # see if image is ship or no ship classification in test and training arrays
print(Y_train[200])

# show images from pixel data for known ship and no ship
fig, axes = plt.subplots(1,2,figsize=(14,5)) # plot two images side by side
axes[0].imshow(X_test[200])
axes[1].imshow(X_train[200])

# tutorial did not process images in the best format
# Need to transpose pixels for image to be displayed/analyzed correctly
X_test = X_test.reshape([-1, 3, 80, 80]).transpose([0,2,3,1]) # transpose shape data
X_train = X_train.reshape([-1, 3, 80, 80]).transpose([0,2,3,1])
# verify images are processed correctly
fig, axes = plt.subplots(1,2,figsize=(14,5)) # plot two images side by side
axes[0].imshow(X_test[200])
axes[1].imshow(X_train[200])

# images look as they do as a png. OK to proceed with this data for the model
from tensorflow.keras import datasets, layers, models
from tensorflow.keras.layers import Activation
# Starts the model with a sequential ANN
model2 = models.Sequential()
# Adds the first convulsion layer and follows up with max pooling
model2.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(80, 80, 3)))
model2.add(layers.MaxPooling2D((2, 2)))
# Add additional hidden layers
model2.add(layers.Conv2D(64, (3, 3), activation='relu'))
model2.add(layers.MaxPooling2D((2, 2)))
model2.add(layers.Conv2D(64, (3, 3), activation='relu'))
# Flattens the input into a 1D tensor
model2.add(layers.Flatten())
# Makes the input more readable for classification
model2.add(layers.Dense(64, activation='relu'))
# Classifies - ensure the input in the number of classes, indexed
# at 0
model2.add(layers.Dense(1))
# Final activation function
model2.add(Activation('sigmoid'))
model2.summary()
# Compile the model
# Use binary_crossentropy because there are only 2 classes present
model2.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
# Train the model
gen_model2 = model2.fit(X_train, Y_train, epochs=10, validation_data=(X_test, Y_test))
# Evaluate the model
from sklearn.metrics import classification_report, confusion_matrix
predictions_1 = model2.predict(X_test)
print(classification_report(Y_test, predictions_1.round()))
cf_matrix_1 = confusion_matrix(Y_test, predictions_1.round())
print(cf_matrix_1)
import seaborn as sns
sns.heatmap(cf_matrix_1, annot=True)
pd.DataFrame(gen_model2.history).plot()
# looking at salience for model 2
# similar to salience function for model 1 except 1 is partial ship instead of ship

def plotsal(a,b):
    '''a is the pixel array we're looking at, b is index'''
    _img = a[b] #recall an image to view saliency on

    img = keras.preprocessing.image.img_to_array(_img) # load the array
    img = img.reshape((1, *img.shape)) # match the required shape for model prediction
    y_pred_model = model2.predict(img).round() # predict outcome of the model
    y_result = 'Ship' if y_pred_model==1 else 'Not A Ship'
    #print("Predicted value is " + str(y_pred_model1) + " so model predicts " + y_result + " is present .")
    images = tf.Variable(img, dtype=float) 

    with tf.GradientTape() as tape:
        pred = model2(images, training=False)
        class_idxs_sorted = np.argsort(pred.numpy().flatten())[::-1]
        loss = pred[0][class_idxs_sorted[0]]
    
    grads = tape.gradient(loss, images)
    dgrad_abs = tf.math.abs(grads)
    dgrad_max_ = np.max(dgrad_abs, axis=3)[0]
    arr_min, arr_max  = np.min(dgrad_max_), np.max(dgrad_max_)
    grad_eval = (dgrad_max_ - arr_min) / (arr_max - arr_min + 1e-18)
    fig, axes = plt.subplots(1,2,figsize=(14,5))
    axes[0].imshow(_img)
    plt.title('Partial Ship Predicted' if y_pred_model==1 else 'No Ship Predicted')
    i = axes[1].imshow(grad_eval,cmap="jet",alpha=0.8)
    fig.colorbar(i)

plotsal(X_test,100)
plotsal(X_test,200)
plotsal(X_test,250)
plotsal(X_test,300)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10054844606946983, 'fraction_numerical': 0.028641072516758074, 'mean_word_length': 5.136485042735043, 'pattern_counts': {'":': 2, '<': 47, '<?xml version=': 0, '>': 49, 'https://': 6, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 12, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '195763', 'license': 'Apache-2.0', 'n_tokens_mistral': 7640, 'n_tokens_neox': 7078, 'n_words': 2624, 'path': 'Shipsnet.ipynb', 'repo_name': 'ehadda02/em-0213-final-project-template', 'size': '22993'}