<jupyter_start><jupyter_text>Training A Graph State Prediction Network (GSPNet): Simplified versionThis notebook is a prototype on how to train a graph state prediction network using CNN and LSTM.The content is devided into __3__ parts:1. Data preprocessing2. Model building, Training and Tuning3. Prediction and per demand modifacationThe model is built with [PyTorch](https://pytorch.org/).This version is simplified and most of intermediate explainatory codes and comments are removed except those directly contribute to generating images. For duck process, see the prototype version.  Part 1: Data Preprocessing 1. Load Libraries and DataLoad in the data. Check the data integrity.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import sqlalchemy
import time
import psycopg2
import warnings
import re
import torch
from dask import dataframe as dd
from PIL import Image
from IPython.display import display
from scipy import stats
from matplotlib import pyplot as plt

%matplotlib inline
%config InlineBackend.figure_format = 'retina'<jupyter_output><empty_output><jupyter_text>Load data for test only. Use dask.<jupyter_code>table = dd.read_csv('dataset/nytaxi_yellow_2017_dec.csv')
table.head()<jupyter_output><empty_output><jupyter_text>Check table shape:<jupyter_code>table.shape<jupyter_output><empty_output><jupyter_text>Check data:Ratio of trips last longer than 30 minutes: (better smaller than 0.1)<jupyter_code>(table.loc[table['trip_time_sec'] > 1800].shape[0] / table.shape[0])<jupyter_output><empty_output><jupyter_text>2. Time Interval PreprocessingCreate time interval and process original table to generate images.Change datetime columns from str type to Timestamp.Function to create time **intervals**:<jupyter_code>def timesplit(stp: str, etp: str, freq='10min'):
    '''
    Create a DatetimeIndx interval.
    
    Params:
        stp: string, starting time point, first left bound
        etp: string, ending time point, last right bound
        freq: frequency, time interval unit of the splice operation
    The stp and etp must of pattern "yyyy-mm-dd hh:mm:ss", otherwise exception will be raised.
    
    Return:
        A list of time intervals tuples,each item is a tuple of two
        interval(i.e., pandas.core.indexes.datetimes.DatetimeIndex object)
        For example, a possible return could be [(2017-01-01 00:00:00, 2017-01-01 00:10:00),
                                                 (2017-01-01 00:10:00, 2017-01-01 00:20:00)]
    '''
    pattern = re.compile('^([0-9]{4})-([0-1][0-9])-([0-3][0-9])\s([0-1][0-9]|[2][0-3]):([0-5][0-9]):([0-5][0-9])$')
    if pattern.match(stp) and pattern.match(etp):
        time_bounds = pd.date_range(stp, etp, freq=freq)
        sub_intervals = list(zip(time_bounds[:-1], time_bounds[1:]))
        print(len(time_bounds), len(sub_intervals))
        return sub_intervals
    else:
        raise Exception('Provided time bound is of invalid format.')<jupyter_output><empty_output><jupyter_text>3. Location PreprocessingPreprocess location related information: mapping ids and locations, then generate adjacency matrices.<jupyter_code># Load zone lookup table
zones = pd.read_csv('dataset/taxi_zone_lookup.csv')
print(f'shape is: {zones.shape}')
# very important globals:
yellow_zone = zones.loc[zones['Borough'] == 'Manhattan']
print(yellow_zone.shape)
img_size = yellow_zone.shape[0]

real_id = list(map(str, list(yellow_zone.loc[:,'LocationID'])))
conv_id = [i for i in range(img_size)]
assert len(real_id) == len(conv_id)
mp = dict(zip(real_id, conv_id))
print(mp)<jupyter_output>(69, 4)
{'4': 0, '12': 1, '13': 2, '24': 3, '41': 4, '42': 5, '43': 6, '45': 7, '48': 8, '50': 9, '68': 10, '74': 11, '75': 12, '79': 13, '87': 14, '88': 15, '90': 16, '100': 17, '103': 18, '104': 19, '105': 20, '107': 21, '113': 22, '114': 23, '116': 24, '120': 25, '125': 26, '127': 27, '128': 28, '137': 29, '140': 30, '141': 31, '142': 32, '143': 33, '144': 34, '148': 35, '151': 36, '152': 37, '153': 38, '158': 39, '161': 40, '162': 41, '163': 42, '164': 43, '166': 44, '170': 45, '186': 46, '194': 47, '202': 48, '209': 49, '211': 50, '224': 51, '229': 52, '230': 53, '231': 54, '232': 55, '233': 56, '234': 57, '236': 58, '237': 59, '238': 60, '239': 61, '243': 62, '244': 63, '246': 64, '249': 65, '261': 66, '262': 67, '263': 68}<jupyter_text>4. Generate Tensor: 3 matrices (layers) of connectionGenerate images that represent traffic states.Define functions to generate desired outcomes:<jupyter_code>left = pd.to_datetime('2017-04-01 00:00:00')
print(f'Generating images on {left.year}-{left.month}-{left.day} ...')
print(left.minute, left.second)
def gen_snap_layers(table, bound):
    '''
    Generate Past layer, Now layer and Future layer for one snapshot.
    Params:
        table:
        bounds:
    Return:
        PNF layers, a list.
    '''
    # left bound and right bound of time interval
    assert type(bound) == tuple
    left = bound[0]
    right = bound[1]
    
    if left.hour == 0 and left.minute == 0 and left.second == 0:
        print(f'Generating images on {left.year}-{left.month}-{left.day}...')
    
    # no need to sort table indeed?
    projected_table = table.loc[:, ['tripid',
                                    'tpep_pickup_datetime',
                                    'tpep_dropoff_datetime',
                                    'pulocationid',
                                    'dolocationid']]

    # The condition of making snapshot should be:
    # at least one temporal end of a trip should be within the bounds:
    snap = projected_table.loc[
        ((projected_table['tpep_pickup_datetime'] >= left) &
         (projected_table['tpep_pickup_datetime'] < right)) |
        ((projected_table['tpep_dropoff_datetime'] >= left) &
         (projected_table['tpep_dropoff_datetime'] < right))]

    # temp table to generate F,P,N layers
    # keep snap intact
    temp_snap = snap.copy()

    # Use the interval to 'catch' corresponding trips.
    # future layer
    f_layer = temp_snap.loc[(temp_snap['tpep_pickup_datetime'] < right) &
                             (temp_snap['tpep_pickup_datetime'] >= left) &
                             (temp_snap['tpep_dropoff_datetime'] >= right)]
    # past layer
    p_layer = temp_snap.loc[(temp_snap['tpep_pickup_datetime'] < left) &
                             (temp_snap['tpep_dropoff_datetime'] >= left) &
                             (temp_snap['tpep_dropoff_datetime'] < right)]
    # now layer
    n_layer = temp_snap.loc[(temp_snap['tpep_pickup_datetime'] >= left) &
                             (temp_snap['tpep_dropoff_datetime'] < right)]

    # Their count should add up to total trips caught
    assert temp_snap.shape[0] == f_layer.shape[0] + p_layer.shape[0] + n_layer.shape[0]

    return p_layer, n_layer, f_layer


# Function that combines layers to an image
def gen_image(p_layer, n_layer, f_layer):
    '''
    Generate an image using given matrices.
    Params:
        p_layer: matrix of past layer
        n_layer: matrix of now layer
        f_layer: matrix of future layer
    Return:
        A PIL image.
    '''
    # create a snapshot
    snapshot = np.zeros([img_size, img_size, 3], dtype='float64')

    # unexpected zones
    left_zones = set()

    # future-Red: 0
    for _, row in f_layer.iterrows():
        try:
            snapshot[mp[str(row['pulocationid'])], mp[str(row['dolocationid'])], 0] += 1
        except Exception as e:
            left_zones.add(str(row['pulocationid']))
            left_zones.add(str(row['dolocationid']))

    # past-Green: 1
    for _, row in p_layer.iterrows():
        try:
            snapshot[mp[str(row['pulocationid'])], mp[str(row['dolocationid'])], 1] += 1
        except Exception as e:
            left_zones.add(str(row['pulocationid']))
            left_zones.add(str(row['dolocationid']))

    # now-Blue: 2
    for _, row in n_layer.iterrows():
        try:
            snapshot[mp[str(row['pulocationid'])], mp[str(row['dolocationid'])], 2] += 1
        except Exception as e:
            left_zones.add(str(row['pulocationid']))
            left_zones.add(str(row['dolocationid']))

    # normalize
    snapshot *= 255 // snapshot.max()
    snapshot = snapshot.astype('uint8')
    image = Image.fromarray(snapshot)
    return image


# generate tensor, because saving image as intermediate data is not efficient
def gen_tensor(p_layer, n_layer, f_layer):
    '''
    Generate a tensor using given matrices.
    Params:
        p_layer: matrix of past layer
        n_layer: matrix of now layer
        f_layer: matrix of future layer
    Return:
        A torch tensor.
    '''
    # create a snapshot
    snapshot = np.zeros([img_size, img_size, 3], dtype='float64')

    # unexpected zones
    left_zones = set()

    # future-Red: 0
    for _, row in f_layer.iterrows():
        try:
            snapshot[mp[str(row['pulocationid'])], mp[str(row['dolocationid'])], 0] += 1
        except Exception as e:
            left_zones.add(str(row['pulocationid']))
            left_zones.add(str(row['dolocationid']))

    # past-Green: 1
    for _, row in p_layer.iterrows():
        try:
            snapshot[mp[str(row['pulocationid'])], mp[str(row['dolocationid'])], 1] += 1
        except Exception as e:
            left_zones.add(str(row['pulocationid']))
            left_zones.add(str(row['dolocationid']))

    # now-Blue: 2
    for _, row in n_layer.iterrows():
        try:
            snapshot[mp[str(row['pulocationid'])], mp[str(row['dolocationid'])], 2] += 1
        except Exception as e:
            left_zones.add(str(row['pulocationid']))
            left_zones.add(str(row['dolocationid']))

    # normalize
    snapshot *= 255 // snapshot.max()
    snapshot = torch.from_numpy(snapshot)
    return snapshot


# Function that gets a specific layer of snapshot.
def get_channel(image, layer:str):
    '''
    Get a layer of the snapshot.
    Params:
        image: PIL image
        channel: one of R-F,G-P,B-N
    Return:
        single channel image
    '''
    assert layer in ['P', 'N', 'F']
    namedict = {'P': 'G', 'N': 'B', 'F': 'R'}
    chandict = {'R':0, 'G':1, 'B':2}
    template = np.array(image)
    chan = np.zeros([*template.shape], dtype='uint8')
    chan[:,:,chandict[namedict[layer]]] = image.getchannel(namedict[layer])
    chan = Image.fromarray(chan)
    return chan<jupyter_output><empty_output><jupyter_text>Save data path:<jupyter_code># image save directory:
visual_path = r'D:\GITHUB\GSPNet\snapshots\visualization'
data_path = r'D:\GITHUB\GSPNet\snapshots\data'<jupyter_output><empty_output><jupyter_text>Start process:<jupyter_code>months = ['04', '05', '06', '07', '08', '09', '10', '11', '12']
for m in range(len(months)-1):
    print(months[m], months[m+1])
month_dict = {
                    '01': 'jan',
                    '02': 'feb',
                    '03': 'mar',
                    '04': 'apr',
                    '05': 'may',
                    '06': 'jun',
                    '07': 'jul',
                    '08': 'aug',
                    '09': 'sep',
                    '10': 'oct',
                    '11': 'nov',
                    '12': 'dec'
                }

months = ['12']

start = time.time()
print(f'Process started at {time.ctime()}')

for m in range(len(months)):
    print(f'Generating time intervals...')
    timelist = timesplit(f'2017-{months[m]}-01 00:00:00', f'2018-01-01 00:00:00')

    # convert dtype for entire table here:
    print(f'Preparing table data...')
    table = pd.read_csv(f'dataset/nytaxi_yellow_2017_{month_dict[months[m]]}.csv')

    print(f'table shape: {table.shape}')
    table['tpep_pickup_datetime'] = pd.to_datetime(table['tpep_pickup_datetime'])
    table['tpep_dropoff_datetime'] = pd.to_datetime(table['tpep_dropoff_datetime'])

    print('start generating...')
    for i, bound in enumerate(timelist):
    
        p_layer, n_layer, f_layer = gen_snap_layers(table, bound)
        tensor = gen_tensor(p_layer, n_layer, f_layer)
        torch.save(tensor, f'{data_path}\\nytaxi_yellow_2017_{month_dict[months[m]]}_d{i}.pt')
    
        image = gen_image(p_layer, n_layer, f_layer)
        vimage = image.resize((690,690)) # multiply by factor of 100
        vimage.save(f'{visual_path}\\nytaxi_yellow_2017_{month_dict[months[m]]}_v{i}.jpg')
    
print(f'Image and tensor generation done in {time.time() - start :.2f} seconds.')<jupyter_output>Process started at Mon Dec 31 21:49:30 2018
Generating time intervals...
4465 4464
Preparing table data...
table shape: (9446596, 11)
start generating...
Generating images on 2017-12-1...
Generating images on 2017-12-2...
Generating images on 2017-12-3...
Generating images on 2017-12-4...
Generating images on 2017-12-5...
Generating images on 2017-12-6...
Generating images on 2017-12-7...
Generating images on 2017-12-8...
Generating images on 2017-12-9...
Generating images on 2017-12-10...
Generating images on 2017-12-11...
Generating images on 2017-12-12...
Generating images on 2017-12-13...
Generating images on 2017-12-14...
Generating images on 2017-12-15...
Generating images on 2017-12-16...
Generating images on 2017-12-17...
Generating images on 2017-12-18...
Generating images on 2017-12-19...
Generating images on 2017-12-20...
Generating images on 2017-12-21...
Generating images on 2017-12-22...
Generating images on 2017-12-23...
Generating images on 2017-12-24...
Generating imag[...]
{'fraction_non_alphanumeric': 0.14362256288138117, 'fraction_numerical': 0.0592350052091085, 'mean_word_length': 3.036948032442175, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '462046', 'license': 'MIT', 'n_tokens_mistral': 5032, 'n_tokens_neox': 4411, 'n_words': 1188, 'path': 'notebook/GSPNet_reconstruct.ipynb', 'repo_name': 'spencerpomme/GSPNet', 'size': '13438'}