<jupyter_start><jupyter_text>Project 3 : Dynamic COVID-19 Spreading Control via Permissive Population Mobility Policies - Mobility Model Team NJIT Data Science ProgramCourseCS634 Data Mining - Summer 2020Instructor<NAME>, Ph.D Professor of Practice, NJIT & Adjunct NYUTeaching Assistant<NAME> (PhD student)Students:*   <NAME>*   <NAME>*   <NAME>*   <NAME>  Libraries<jupyter_code># Libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>Loading training dataset<jupyter_code># Path of training data file in Google Drive (each group will modify the path of its traininig file)
# path_csv = "/content/drive/My Drive/Colab Notebooks/AgentBasedModelling/training_data.csv"
path_csv = "/content/drive/My Drive/Notebooks/Mobility/training_data.csv"
# This function mount google colab and, read training file csv, returns a dataframe
def ds_from_googlecolab(path_csv):
  from google.colab import drive
  drive.mount('/content/drive')
  ds = pd.read_csv(path_csv)
  return ds
# Called to the function above mentioned to bring the training dataframe with the 5 first rows.
df_train = ds_from_googlecolab(path_csv)
df_train.head()
# Understanding the shape of the training dataset
print("Rows:", df_train.shape[0], "Columns:", df_train.shape[1])<jupyter_output>Rows: 3414443 Columns: 6<jupyter_text>Verifying empty values<jupyter_code># Finding empty values over the columns used in the project.
def num_of_empty(ds):
  ds_empty= ds.loc[(ds['pickup_grid_number']=='') | (ds['dropoff_grid_number']=='') | (ds['hour']=='')]
  return ds_empty.shape[0]
num_row_empty = num_of_empty(df_train)
print("Number of row with empty values: ", num_row_empty)
df_train.describe()<jupyter_output><empty_output><jupyter_text>We can observe in the table, the dataset has trips just of month 7. The feature hour looks well, the minimum is 0 and maximum is 23 hour. Minutes looks good as well, from 0 minutes to 59 minutes. Pickup has trips from zone 0 to zone 2500, clearly skewed right with a mean of zone 344. Dropoff has trips from zone 0 to 2500, clearly skewed right as well with a mean of zone 369.  Creating General Matrix Pickup vs Dropoff This matrix generates the number of trips by zones<jupyter_code>matrix = df_train.groupby(['pickup_grid_number','dropoff_grid_number']).size().unstack().fillna(0)
matrix
# Understanding shape of the matrix
matrix.shape<jupyter_output><empty_output><jupyter_text>We find 957 pickup zones by 979 dropoff zones used in the all trips  Calculating Number of trips in the same zone<jupyter_code># This function return a tuple with the trips from a zone to the same zone.
def same_zone(matriz):
  res = []
  v_cols= matriz.columns
  v_rows= matriz.index
  for i in v_cols:
    for j in v_rows:
      if (i==j):
        res.append((i,matriz[i][j]))
  return (res)
# Calling the function same_zone given the matrix.
resul = same_zone(matrix)
resul
# This function searches in the tuple the area with the most trips.
def Max_SameZone(tuple):
  return max(tuple,key=lambda zone:zone[1])
# Calling to the function Max_SameZone
print("Zone:", Max_SameZone(resul)[0], "Number of trips:", Max_SameZone(resul)[1])
# This function searches in the tuple the area with the fewest trips.
def Min_SameZone(tuple):
  return min(tuple,key=lambda zone:zone[1])
# Calling to the function Min_SameZone
print("Zone:", Min_SameZone(resul)[0], "Number of trips:", Min_SameZone(resul)[1])
# This function sort the tuple ascendent or descendent
def Sort_Tuple(tup,asc):
  if type(asc)==bool:
    if asc==True: 
      return(sorted(tup, key = lambda x: x[1])) 
    else:
      return(sorted(tup, key = lambda x: -x[1]))
  else:
    print("Parameter asc should be boolean")
 # Printing the tuple ordered
 print(Sort_Tuple(resul,True)) # False is descendent , True is ascendent
# Getting 5 high demand zones, picking and dropping in the same zone.
FiveTuple=Sort_Tuple(resul,False)[:5]
FiveTuple
# This functions gets only the zones
def Get_Zones(tuple):
  return [tuple_[0] for tuple_ in FiveTuple]
# This functions gets only the number of trips
def Get_NumTrips(tuple):
  return [tuple_[1] for tuple_ in FiveTuple]
# Retriving Zones and values of the 5 high demand zones, picking and dropping in the same zone.
Zones=Get_Zones(FiveTuple)
Values=Get_NumTrips(FiveTuple)
Zones, Values
# Function that create a barchart with the high demand in the same zone
def barchart(tuple):
  objects = Get_Zones(FiveTuple)
  y_pos = np.arange(len(objects))
  performance = Get_NumTrips(FiveTuple)

  plt.bar(y_pos, performance, align='center', alpha=0.5)
  plt.xticks(y_pos, objects)
  plt.xlabel('Zones')
  plt.ylabel('# trips')
  plt.title('High Demand in the same zone')

  plt.show()
# Tuple ordered to show it in the barchart
FiveGreatest=Sort_Tuple(resul,False)[:5]
FiveGreatest
# Calling to the function barchart to plot the 5 zones with high demand in trips, from / to same zone.
barchart(FiveGreatest)<jupyter_output><empty_output><jupyter_text>The above graphic show us that the zone 805. People use this zone to pick up and dropoff.  Total de trips given the zone<jupyter_code># All pickup trips given zone
def pickup_by_zone(matriz,zone):
  sum_zone =matriz[matriz.index==zone].sum(axis=1)
  return sum_zone.values[0]
# Example: How many pickup trips were in the zone 805
pickup_by_zone(matrix,805)<jupyter_output><empty_output><jupyter_text>This makes sense, because in the barchart showed lines above, we saw that zone 805 is the most used from / to same zone, this means that the difference between 1318 and 1102 are 216 trips, that were picked up from the zone 805 but the droppoff was in a different zone.<jupyter_code># All dropoff trips given zone
def droppoff_by_zone(matriz,zone):
  return matriz[zone].sum()
# Example: How many dropoff trips were in the zone 2500
droppoff_by_zone(matrix,2500)
# All trips from / to zone
def total_zone(matriz,zonepickup, zonedropoff):
  total = pickup_by_zone(matriz,zonepickup)+droppoff_by_zone(matriz,zonedropoff)
  return total
# Calculating all the trips that were picked up in the zone 1 and all the trips droppoff in the zone 1 
total_zone(matrix,1,1)<jupyter_output><empty_output><jupyter_text>Creating matrix pickup vs dropoff by hours<jupyter_code>matrizbyhours = df_train.groupby(['hour','pickup_grid_number','dropoff_grid_number']).size().unstack().fillna(0)
matrizbyhours<jupyter_output><empty_output><jupyter_text>Data visualization  Heatmap to visualize the zones of pickup and dropoff aggregated<jupyter_code># This function create a heatmap to visualize the zones of pickup and dropoff, we give the matrix, width and height of the heatmap.
def plot(matriz,width,height):
  fig, ax = plt.subplots(figsize=(width,height)) 
  ax = sns.heatmap(matriz, 
      vmin=0, vmax=max(matriz.max()),
      cmap=sns.diverging_palette(20, 220, n=20),
      square=False,
  )
  ax.set_xticklabels(
      ax.get_xticklabels(),
      rotation=45,
      horizontalalignment='right'
  );
plot(matrix,15,10)
# Zoom to visualize better, we used 500 records in accordance with 75% (third quartile)
plot(matrix.iloc[0:500,0:500],15,10)<jupyter_output><empty_output><jupyter_text>This heatmap show us many things interesting like: there are strong lines on the Y axis, one of them is on the pickup number 180, this makes sense because 180 is a zone with many trips according to the graph showed lines above.Clearly, we can see a diagonal line, showing that many trips were picked up and dropped off in the same zone.  Heatmap to visualize the zones of pick up and dropoff by hours<jupyter_code># Function filter matrix given the hour
def trips_by_hour(matriz,hour): # hours [0 to 23]
  if ((hour<0) | (hour>23)): # returns true if all digits otherwise false
    return print("Range of months is [0,23]")
  else:
    mf = matriz.loc[hour]
    return mf
# Plotting matrix of the hour 20, 8PM
trips_in_hour=trips_by_hour(matrizbyhours,20)
# plotting 500 first rows
plot(trips_in_hour.iloc[:500,:500],15,10)<jupyter_output><empty_output><jupyter_text>Heatmap for each hour<jupyter_code># Plotting matrix by each hour from 1 hour to 23 hour
for i in range(24):
    trips_in_hour=trips_by_hour(matrizbyhours,i)
    plot(trips_in_hour.iloc[:500,:500],3,2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09133709981167608, 'fraction_numerical': 0.02248116760828625, 'mean_word_length': 5.481311975591152, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '455314', 'license': 'Apache-2.0', 'n_tokens_mistral': 2911, 'n_tokens_neox': 2705, 'n_words': 944, 'path': 'mobility model/MobilityModel .ipynb', 'repo_name': 'pantelis/cs634-summer-2020-capstone', 'size': '8541'}