<jupyter_start><jupyter_text>BiciMAD - Machine learning predictionsCopyright © 2017 <NAME>  | [@rameerez [tw]](http://twitter.com/rameerez) | [GitHub](http://github.com/rameerez)This code is Open Source, released under the MIT License.  Why/whatWe've studied the actual BiciMAD users' behavior in other notebooks – but can we predict user behavior using Artificial Intelligence?![crowd](img/crowd.jpg)[Photo credit](https://unsplash.com/photos/ABGaVhJxwDQ)  Data sourcesApart from BiciMAD's data, we're using meteo data. All credits to [AEMET OpenData](http://www.aemet.es/es/datos_abiertos/AEMET_OpenData) (Spanish Meteorological Agency).I tried to get **hourly** meteo observations for the month of March 2017, but couldn't find any source. Therefore we'll be grouping data by day.  Definitions<jupyter_code># Data describing all bike rides
BIKE_RIDES_DATASET = './data/export_20170301_20170406.json'
# Data describing all stations (used just for getting stations' coordinates & names)
STATIONS_DATASET = './data/stations.json'
# AEMET's (Spanish Meteorological Agency) observations in Madrid (Retiro) for each day in March 2017
MARCH_WEATHER_DATASET = './data/march_weather_madrid_retiro.json'
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib as plt
import math
import dateutil.parser
from pandas.io.json import json_normalize
import tflearn
df = pd.read_json(BIKE_RIDES_DATASET, lines=True)
df[:3]<jupyter_output><empty_output><jupyter_text>Converting unplug time JSON to valid DatetimeUnfortunately the `unplug_hourTime` comes in an unnecesarily descriptive JSON. Let's convert that to a valid `Datetime` so we can do stuff like `date.month`<jupyter_code>def parse_unplug_datetime(row):
    date_as_string = row['unplug_hourTime']['$date']
    return dateutil.parser.parse(date_as_string)
df['unplug_datetime'] = df.apply(parse_unplug_datetime, axis=1)<jupyter_output><empty_output><jupyter_text>Yay! Valid datetimes. Let's now group by month and day and plot the number of rides by day:<jupyter_code>times = pd.DatetimeIndex(df.unplug_datetime)
df.groupby([times.month, times.day])._id.count().plot()<jupyter_output><empty_output><jupyter_text>The dataset contains data for the entire month of March and some days of April. Let's filter and get just March data.<jupyter_code>march_df = df[times.month == 3]
march_times = pd.DatetimeIndex(march_df.unplug_datetime)
march_df.groupby([march_times.month, march_times.day])._id.count().plot(legend=False, rot=90, xticks=pd.Series(range(1,31)))<jupyter_output><empty_output><jupyter_text>Does weather affect bike usage?<jupyter_code># Caution! orient='records' is required!
march_weather_df = pd.read_json(MARCH_WEATHER_DATASET, orient='records')
march_weather_df[:1]<jupyter_output><empty_output><jupyter_text>Well, shit. Is nobody going to record their data in a freaking valid way? Those "float" numbers are NOT numbers – they are strings! The only valid float separator is a point `.`, not a comma `,`.And that date... that is a string, too. Gosh.Time to clean (again) our data:<jupyter_code>def comma_string_to_float(float_string):
    return float(float_string.replace(".","").replace(",","."))
def operations(row):
  df1 = comma_string_to_float(row['prec'])
  df2 = comma_string_to_float(row['tmed'])
  return df1, df2

march_weather_df["rain"], march_weather_df["avgtemp"] = zip(*march_weather_df.apply(lambda row: operations(row), axis=1))
def fecha_to_date(row):
    return dateutil.parser.parse(row['fecha'])

march_weather_df['datetime'] = march_weather_df.apply(fecha_to_date, axis=1)
march_weather_df[['datetime','rain','avgtemp']][:3]<jupyter_output><empty_output><jupyter_text>And that would be how you get valid float numbers for meteo observations. Let's plot rain:<jupyter_code>march_weather_df.rain.plot(rot=90, xticks=pd.Series(range(1,31)))<jupyter_output><empty_output><jupyter_text>Interesting! Rain seems to be the inverse of usage! Let's recall:<jupyter_code>march_df.groupby([march_times.month, march_times.day])._id.count().plot(legend=False, rot=90, xticks=pd.Series(range(1,31)))<jupyter_output><empty_output><jupyter_text>As we can see, the days with the most rain are the days with the least bike rides.  From daily to hourly meteo dataSo, AEMET won't provide hourly observations. Instead, they provide daily observations.Ideas:- **Temp**: mintemp & maxtemp are available, along with the time at which both ocurred. Avgtemp is available, too. Strategy idea: linear distribution from 00:00 to mintemp time (last day temp to mintemp), then from mintemp time to maxtemp time and so on.- **Pressure**: same strategy as temperature.- **Rain**: the most naive assumption would be hourly precipitation = daily/24, but we can add some precision taking into account the previous and next day's precipitation to make it an ascending, descending or normal distribution.- **Wind speed**: Only avg is provided. For simplicity we could assume the hourly wind speed to be just the average for the day.  Machine learningLet's go ahead and use Artificial Intelligence to train a neural network to predict number of bike trips per day depending on rain and average temperature for that day.  Data preparation First, let's create a dataframe with the usage per day.<jupyter_code>df_usage_by_day = march_df.set_index('unplug_datetime').groupby(pd.Grouper(freq='D'))._id.count().to_frame(name="count").reset_index()
df_usage_by_day[:3]<jupyter_output><empty_output><jupyter_text>Then, we'll need to join it with our weather data:<jupyter_code>march_weather_df[['datetime','rain','avgtemp']][:3]<jupyter_output><empty_output><jupyter_text>Well, more stuff to clean. Seems like our BiciMAD data now has `datetime.datetime` timestamps and our AEMET data has `datetime.date` timestamps. Let's remove the hour, minutes and seconds and make it timezone naive.<jupyter_code>df_usage_by_day['unplug_date'] = df_usage_by_day['unplug_datetime'].dt.date
df_usage_by_day[:1]<jupyter_output><empty_output><jupyter_text>Yay! Let's join the tables<jupyter_code>daily_frequencies_with_meteo_df = march_weather_df.set_index('datetime').join(df_usage_by_day.set_index('unplug_date')).reset_index()
daily_frequencies_with_meteo_df[['datetime','rain','avgtemp','count']][:1]<jupyter_output><empty_output><jupyter_text>Well and we kind of need to know the day of the week instead of the actual date:<jupyter_code>daily_frequencies_with_meteo_df['dayofweek'] = daily_frequencies_with_meteo_df['datetime'].dt.dayofweek
daily_frequencies_with_meteo_df[['dayofweek','rain','avgtemp','count']][:1]<jupyter_output><empty_output><jupyter_text>Note: The day of the week with Monday=0, Sunday=6  Let's use a deep neural network<jupyter_code>neural_net_data = daily_frequencies_with_meteo_df[['dayofweek','rain','avgtemp','count']].copy()<jupyter_output><empty_output><jupyter_text>Just for convenience, let's save the dataframe we'll use in a separate .csv file. This way, if the training goes wrong and we have to restart the kernel, we won't have to rerun all dataprep operations again.<jupyter_code>neural_net_data.to_csv('data/neural_net_data.csv')<jupyter_output><empty_output><jupyter_text>And let's import that just to make sure we're using the .csv data:<jupyter_code>neural_net_data = pd.read_csv('data/neural_net_data.csv', index_col=0)
neural_net_data[:1]<jupyter_output><empty_output><jupyter_text>Now over to the TFLearn part. Note that there is no need to read the .csv prior to this point, we're just doing it for illustration purposes.<jupyter_code># Load CSV file, indicate that the fifth (3) column represents labels. Ignore the first column (it just contains IDs)
from tflearn.data_utils import load_csv
data, labels = load_csv('data/neural_net_data.csv', target_column=4, columns_to_ignore=[0])<jupyter_output><empty_output><jupyter_text>This creates two variables.These, as described by the [TFLearn documentation](http://tflearn.org/models/dnn/) are the two datasets required to train our Deep Neural Network.Please note that **a neural network model can only perform operations over numbers**. We need to convert every non-numerical label/value to a numerical representation before this point. It's not the case, so we continue.The first variable, `data`, our `X_inputs` a list, array or ndarray. Data to feed to train model.The second one, `labels`, our targets (Labels) to feed to train model.---We're now going to create the Deep Neural Network. We declare its shape as `[None, 3]`, meaning that we have unknown rows (`None` dimension) and that our data has `3` features (`dayofweek`,`rain`,`avgtemp`).<jupyter_code># Build neural network
net = tflearn.input_data(shape=[None, 3])
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 1)
net = tflearn.regression(net, optimizer='sgd', loss='mean_square', learning_rate=0.1)
# Need to do this so we don't get dimension error when training the neural net
labels = np.asarray(labels)[:, np.newaxis]
# Define model
model = tflearn.DNN(net)
# Start training (apply gradient descent algorithm)
model.fit(data, labels, n_epoch=100, batch_size=16, show_metric=True, snapshot_epoch=True)<jupyter_output>Training Step: 199  | total loss: [1m[32m3195796.00000[0m[0m | time: 0.003s
| SGD | epoch: 100 | loss: 3195796.00000 - binary_acc: 0.0000 -- iter: 16/31
Training Step: 200  | total loss: [1m[32m3325802.00000[0m[0m | time: 0.006s
| SGD | epoch: 100 | loss: 3325802.00000 - binary_acc: 0.0000 -- iter: 31/31
--<jupyter_text>Neural net trained! Let's test it with an example day.Say day 10 of our data:<jupyter_code>neural_net_data.iloc[10]<jupyter_output><empty_output><jupyter_text>Weekday 5 (Saturday), rain 0.0, avgtemp 16.1. Had a total of 9402 travels.Let's test it:<jupyter_code>day = [5.0, 0.0, 16.1]
pred = model.predict([day])
print("Prediction:", pred[0][0])<jupyter_output>Prediction: 11933.9
{'fraction_non_alphanumeric': 0.11037841128132292, 'fraction_numerical': 0.02130465658922593, 'mean_word_length': 6.89271417133707, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '445519', 'license': 'MIT', 'n_tokens_mistral': 3451, 'n_tokens_neox': 3197, 'n_words': 946, 'path': 'service-usage-machine-learning.ipynb', 'repo_name': 'rameerez/bicimad-data-analysis', 'size': '9863'}