<jupyter_start><jupyter_text>Configure the notebookThis is done in this notebook in addition, so the notebook can run without being called from the mainnotebook- Load file with secrets- Set some constants- Load libraries- Load variables from the main notebook from file, this is not needed if the notbook is running from inside the main notebook The API secrets are not pushed to github. This is handled by placing them in a file named secrets and removing version control from that file.Since the file is not in the same directory as the notebooks extra code is needed to add the path to the sys search path The secrets file containts two variables used in the Foursquare API: CLIENT_ID and CLIENT_SECRETIt contains two variables used int the travel time API: APP_ID and API_KEY<jupyter_code>import os
project_folder_path = os.path.dirname(os.getcwd())
project_folder_path
import sys
sys.path.insert(0, project_folder_path)

import secrets
print('secrets.py imported')<jupyter_output>secrets.py imported<jupyter_text>The data path contains data loaded from the net. The sanbox accounts used to load the data have limits on the number of requests. Storing the results allows for restarting the kernal without having to make new calls to the API.<jupyter_code>DATA_PATH = project_folder_path + '/data/external/'
print('Data path is :{}'.format(DATA_PATH))<jupyter_output>Data path is :/Users/danielhaugstvedt/Developer/coursera_capstone/data/external/<jupyter_text>Import the standard libraries<jupyter_code>import numpy as np # library to handle data in a vectorized manner
import pandas as pd # library for data analsysis

from geopy.geocoders import Nominatim # convert an address into latitude and longitude values

import requests # library to handle requests
import json
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors

# import k-means from clustering stage
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

import folium # map rendering library

import pickle # needed to store variables

import time # Try to not overload the traveltime API

print('Libraries imported.')
file_name = 'neighborhoods_final.p'
with open(DATA_PATH + file_name, 'rb') as infile:
    neighborhoods_final = pickle.load(infile)
file_name = 'ny_venues_final.p'
with open(DATA_PATH + file_name, 'rb') as infile:
    ny_venues_final = pickle.load(infile)<jupyter_output><empty_output><jupyter_text>Get the travel time between the different neighborhoodsUsing the traveltime API, get the travel time between the different neighborhoods **Post request example**```POST /v4/time-filter HTTP/1.1Host: api.traveltimeapp.comContent-Type: application/jsonAccept: application/jsonX-Application-Id: APP_IDX-Api-Key: API_KEY``` Convert the information in the post request example to URI and header<jupyter_code>URI_traveltime = 'HTTPS://api.traveltimeapp.com/v4/time-filter'
headers = {'Host': 'api.traveltimeapp.com',
           'Content-Type': 'application/json', 
           'Accept': 'application/json', 
           'X-Application-Id': secrets.APP_ID, # APP ID is in separate file not under version control
           'X-Api-Key': secrets.API_KEY} # API_KEY is in separate file not under version control<jupyter_output><empty_output><jupyter_text>Fuction that adds locations to the json request<jupyter_code>def add_locations(locations, latitudes, longitudes):
    
    json_request = {'locations': [], 'departure_searches':[], 'arrival_searches':[]}
    
    for loc, lat, long in zip(locations, latitudes, longitudes):
        json_request['locations'].append({
            'id': loc,
            'coords': {
                'lat': lat, 
                'lng': long
            }
        })
    
    return json_request<jupyter_output><empty_output><jupyter_text>Add a searches to the json request<jupyter_code>def add_search(json_request, from_location, travel_time, departure_time):
    
    arrival_locations = []
    for location_dic in json_request['locations']:
        arrival_locations.append(location_dic['id'])
        
    del arrival_locations[arrival_locations.index(from_location)]
    
    json_request['departure_searches'].append({
        'id': 'from {}'.format(from_location),
        'departure_location_id': from_location,
        'arrival_location_ids': arrival_locations,
        'transportation': {'type': 'public_transport'}, 
        'departure_time' : departure_time,
        'travel_time': travel_time,
        'range': {
                'enabled': True,
                'max_results': 3,
                'width': 600
            },
        'properties': ['travel_time']
    })
    
    return json_request<jupyter_output><empty_output><jupyter_text>Make three vectors- limits on API is 2000 arival locations- limits on API is 10 searches<jupyter_code>locations = neighborhoods_final.loc[:, 'Neighborhood']
latitudes = neighborhoods_final.loc[:, 'Latitude']
longitudes = neighborhoods_final.loc[:, 'Longitude']
print('We need less than 2000 locations. The number we are using is {}'.format(len(locations)))<jupyter_output>We need less than 2000 locations. The number we are using is 262<jupyter_text>Test if the file are stored, if not load the the data from the APIWhen using the API, loop through all locations and do a post request where the location is the origin of the search.<jupyter_code>locations_subset = locations[0:150]
file_name = 'traveltime_0_149.p'

try:
    print('Trying to open file: {}'.format(DATA_PATH + file_name))
    with open(DATA_PATH + file_name, 'rb') as infile:
        results = pickle.load(infile)
        print('Sucess loading from file')
except FileNotFoundError:
    print('Failed to load from local file, trying to load from web')
    results = []
    for location in locations_subset:
        json_request_setup = add_locations(locations, latitudes, longitudes)
        json_request = add_search(json_request_setup, location, 60*60*1.5, '2019-01-11T13:00:00Z')
        
        # The fallback should not be used if you are not really sure 
        #result = requests.post(URI_traveltime, headers=headers, json=json_request)
        print('{} request returned with status code: {} and reason: {}'.format(location, result.status_code, result.reason))
        if result.status_code != 200:
            break
            
        json_result = result.json()
        results.append(json_result['results'])
        #time.sleep(60) # sleep 60 s, this is used when doing many queries. The api can get overloaded.
        
    with open(DATA_PATH + file_name, 'wb') as outfile:
        pickle.dump(results, outfile)
results_0_149 = results
print(len(locations_subset))
print(len(results_0_149))
locations_subset = locations[150:193]
file_name = 'traveltime_150_192.p'

try:
    print('Trying to open file: {}'.format(DATA_PATH + file_name))
    with open(DATA_PATH + file_name, 'rb') as infile:
        results = pickle.load(infile)
        print('Sucess loading from file')
except FileNotFoundError:
    print('Failed to load from local file, trying to load from web')
    results = []
    for location in locations_subset:
        json_request_setup = add_locations(locations, latitudes, longitudes)
        json_request = add_search(json_request_setup, location, 60*60*1.5, '2019-01-11T13:00:00Z')
        
        # The fallback should not be used if you are not really sure 
        #result = requests.post(URI_traveltime, headers=headers, json=json_request)
        print('{} request returned with status code: {} and reason: {}'.format(location, result.status_code, result.reason))
        if result.status_code != 200:
            break
            
        json_result = result.json()
        results.append(json_result['results'])
        #time.sleep(60) # sleep 60 s, this is used when doing many queries. The api can get overloaded.
        
    with open(DATA_PATH + file_name, 'wb') as outfile:
        pickle.dump(results, outfile)
        
results_150_192 = results
print(len(locations_subset))
print(len(results_150_192))
locations_subset = locations[193:261]
file_name = 'traveltime_193_260.p'

try:
    print('Trying to open file: {}'.format(DATA_PATH + file_name))
    with open(DATA_PATH + file_name, 'rb') as infile:
        results = pickle.load(infile)
        print('Sucess loading from file')
except FileNotFoundError:
    print('Failed to load from local file, trying to load from web')
    results = []
    for location in locations_subset:
        json_request_setup = add_locations(locations, latitudes, longitudes)
        json_request = add_search(json_request_setup, location, 60*60*1.5, '2019-01-11T13:00:00Z')
        
        result = requests.post(URI_traveltime, headers=headers, json=json_request)
        print('{} request returned with status code: {} and reason: {}'.format(location, result.status_code, result.reason))
        if result.status_code != 200:
            break
            
        json_result = result.json()
        results.append(json_result['results'])
        time.sleep(60) # sleep 60 s, this is used when doing many queries. The api can get overloaded.
        
    with open(DATA_PATH + file_name, 'wb') as outfile:
        pickle.dump(results, outfile)
        
results_193_260 = results
print(len(locations_subset))
print(len(results_193_260))
locations_subset = locations[261:]
file_name = 'traveltime_261.p'

try:
    print('Trying to open file: {}'.format(DATA_PATH + file_name))
    with open(DATA_PATH + file_name, 'rb') as infile:
        results = pickle.load(infile)
        print('Sucess loading from file')
except FileNotFoundError:
    print('Failed to load from local file, trying to load from web')
    results = []
    for location in locations_subset:
        json_request_setup = add_locations(locations, latitudes, longitudes)
        json_request = add_search(json_request_setup, location, 60*60*1.5, '2019-01-11T13:00:00Z')
        
        result = requests.post(URI_traveltime, headers=headers, json=json_request)
        print('{} request returned with status code: {} and reason: {}'.format(location, result.status_code, result.reason))
        if result.status_code != 200:
            break
            
        json_result = result.json()
        results.append(json_result['results'])
        time.sleep(60) # sleep 60 s, this is used when doing many queries. The api can get overloaded.
        
    with open(DATA_PATH + file_name, 'wb') as outfile:
        pickle.dump(results, outfile)
        
results_261 = results
print(len(locations_subset))
len(results_0_149) + len(results_150_192) + len(results_193_260) + len(results_261)<jupyter_output><empty_output><jupyter_text>Make the travel time into json into a data frame<jupyter_code>travel_time_df = (pd.DataFrame(index=locations, columns=locations) # Data frame with NaNs
     .fillna(0))

for result in results_0_149:
    result = result[0] # get the dictionary from the array
    from_location = ' '.join(result['search_id'].split(" ")[1:])
    for to_location_dic in result['locations']:
        to_location = to_location_dic['id']
        travel_time = to_location_dic['properties'][0]['travel_time']
        travel_time_df.loc[from_location, to_location] = travel_time / (60*60) # Hours
        
for result in results_150_192:
    result = result[0] # get the dictionary from the array
    from_location = ' '.join(result['search_id'].split(" ")[1:])
    for to_location_dic in result['locations']:
        to_location = to_location_dic['id']
        travel_time = to_location_dic['properties'][0]['travel_time']
        travel_time_df.loc[from_location, to_location] = travel_time / (60*60) # Hours
        
for result in results_193_260:
    result = result[0] # get the dictionary from the array
    from_location = ' '.join(result['search_id'].split(" ")[1:])
    for to_location_dic in result['locations']:
        to_location = to_location_dic['id']
        travel_time = to_location_dic['properties'][0]['travel_time']
        travel_time_df.loc[from_location, to_location] = travel_time / (60*60) # Hours
        
for result in results_261:
    result = result[0] # get the dictionary from the array
    from_location = ' '.join(result['search_id'].split(" ")[1:])
    for to_location_dic in result['locations']:
        to_location = to_location_dic['id']
        travel_time = to_location_dic['properties'][0]['travel_time']
        travel_time_df.loc[from_location, to_location] = travel_time / (60*60) # Hours
        
sum(travel_time_df.sum(axis=0)==0)<jupyter_output><empty_output><jupyter_text>Clean up values you do not want to pass back to the main notebook<jupyter_code>del results_261
del results_193_260
del results_150_192
del results_0_149
del results
del locations_subset
del URI_traveltime
del headers
del to_location      
del to_location_dic
del travel_time 
del from_location<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09573329255237804, 'fraction_numerical': 0.022556965896926136, 'mean_word_length': 3.88569293985805, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '529538', 'license': 'MIT', 'n_tokens_mistral': 4129, 'n_tokens_neox': 3847, 'n_words': 1140, 'path': 'notebooks/travel_time.ipynb', 'repo_name': 'haugstve/Coursera_Capstone', 'size': '13078'}