<jupyter_start><jupyter_text>We're looking at all June flights departing from ATL (Atlanta) between 2006 and 2017<jupyter_code>!gsutil cp gs://going-tfx/flight_data/atl_june.csv /tmp/data/atl_june.csv
DATA_DIR = "/tmp/data"
ALL_ATL_JUNE = os.path.join(DATA_DIR, 'atl_june.csv')
TRAIN_FILE=os.path.join(DATA_DIR, 'atl_june_train.csv')
TEST_FILE=os.path.join(DATA_DIR, 'atl_june_test.csv')
NUM_RECORDS = !cat $ALL_ATL_JUNE | wc -l
NUM_RECORDS = int(NUM_RECORDS[0]) - 1
NUM_TRAIN = 80000
NUM_TEST = 1000
"Total number of June flights: {}".format(NUM_RECORDS)<jupyter_output><empty_output><jupyter_text>Here, we split the file into train and test examples<jupyter_code>df = pd.read_csv(ALL_ATL_JUNE)
df_sample = df.sample(frac=1.0, random_state=1) # pd.sample() also shuffles the records
df_sample[:NUM_TRAIN].to_csv(TRAIN_FILE, index=False)
df_sample[NUM_TRAIN: NUM_TRAIN + NUM_TEST].to_csv(TEST_FILE, index=False)
df_train = pd.read_csv(TRAIN_FILE)
df_test = pd.read_csv(TEST_FILE)
df_train.describe()
df_test.describe()
try:
    sess.close()
except:
    print("OK. Was already closed or didn't exist. That's fine.")
    
sess = tf.InteractiveSession()<jupyter_output>OK. Was already closed or didn't exist. That's fine.<jupyter_text>--- tf.estimator input_functionAn input function is a function that takes no arguments and returns a tensor graph that produces a new batch whenever evaluated - until exhausted. Note that we're using ```tf.data```'s ```map()``` semantics to perform transformations. That's the easiest way, but may not perform so well on big data. Later, we'll use pipelines, either directly or via tf.transforms.<jupyter_code>COLUMNS = ['FL_DATE', 'FL_YEAR', 'FL_MONTH', 'FL_DOM', 'FL_DOW', 'UNIQUE_CARRIER', 'FL_NUM',
       'ORIGIN_AIRPORT_SEQ_ID', 'DEST_AIRPORT_SEQ_ID', 'ORIGIN', 'DEST',
       'CRS_DEP_TIME', 'DEP_DELAY', 'TAXI_OUT', 'WHEELS_OFF', 'WHEELS_ON',
       'TAXI_IN', 'CRS_ARR_TIME', 'ARR_DELAY', 'CANCELLED',
       'CANCELLATION_CODE', 'DIVERTED', 'DISTANCE']
SELECT_COLUMNS=[1,4, 5,10,11,12,17,18,22]
SELECT_FEATURES = ['FL_YEAR', 'FL_DOW', 'UNIQUE_CARRIER', 'DEST', 'CRS_DEP_TIME', 'DEP_DELAY', 'CRS_ARR_TIME', 'DISTANCE']
DEFAULTS = [["-"], [], [], [], [], ["-"], ["-"], ["-"], ["-"], ["-"], ["-"], [], [], [], [], [], [], [], [], [], ['NONE'], [], []]
def make_decode_csv(col_names, select_indexes, defaults, label_name) :
    selected_defaults = [defaults[i] for i in select_indexes]
    def _decode_csv(row): 
        cols = tf.decode_csv(row, select_cols=select_indexes, 
                             record_defaults=selected_defaults)
        decoded = dict(zip([col_names[i] for i in select_indexes], cols))
        label = decoded.pop(label_name)
        return decoded, label
    return _decode_csv

        
def make_input_fn(file_name, headers, col_names, label_name, indexes, defaults, transform=None, batch_size=1, repeat=1):
    
    decode_csv = make_decode_csv(col_names, indexes, defaults, label_name)
    
    def _input_fn():
        text_lines = tf.data.TextLineDataset(file_name)
        if headers:
            text_lines=text_lines.skip(1)
        decoded = text_lines.map(decode_csv)

        if transform is not None:
            decoded = decoded.map(transform)
        
        features = decoded.repeat(repeat).batch(batch_size).make_one_shot_iterator().get_next()
        return features
    
    return _input_fn<jupyter_output><empty_output><jupyter_text>------See how lines of csv are parsed into tensors<jupyter_code>defaults = [tf.constant([0]), tf.constant([0]), tf.constant(['CHF'])]
decode_csv = make_decode_csv(['a', 'b', 'c'], [0,1,2], defaults, 'c')
row = tf.constant(["10,20,Dollar", "20,30,SGD"])
feats, labels = decode_csv(row)
feats['a'].eval(), feats['b'].eval(), labels.eval()<jupyter_output><empty_output><jupyter_text>------Any additional transformation like e.g. one-hot encoding or bucketizing is done in this transform function<jupyter_code>def transform(feats, labels):
    
    dow = feats['FL_DOW']
    dow = tf.cast(dow, tf.int64)
    dow_hot = tf.one_hot(dow-1, 7, dtype=tf.int64)
    feats['FL_DOW'] = dow_hot

    dep_time = feats['CRS_DEP_TIME'] / 100.0
    dep_time = tf.cast(dep_time, tf.int64)
    dt_hot = tf.one_hot(dep_time, 24, dtype=tf.int64)
    feats['CRS_DEP_TIME'] = dt_hot
    
    arr_time = feats['CRS_ARR_TIME'] / 100.0
    arr_time = tf.cast(arr_time, tf.int64)
    at_hot = tf.one_hot(arr_time, 24, dtype=tf.int64)
    feats['CRS_ARR_TIME'] = at_hot

    dist = feats['DISTANCE'] / 1000.0
    feats['DISTANCE'] = dist
    
    dep_d = feats['DEP_DELAY']
    feats['DEP_DELAY'] = dep_d
    
    return feats, labels<jupyter_output><empty_output><jupyter_text>------The input function returns a pair consisting of a dictionary of input tensors and a label tensor.These tensors are actually datasets, that produce the respective next batch, when evaluated. That's how we avoid loading the entire dataset into memory for training.<jupyter_code>input_fn = make_input_fn(TRAIN_FILE,
    True, COLUMNS, 'ARR_DELAY', SELECT_COLUMNS, DEFAULTS, transform, 2)
input_fn()<jupyter_output><empty_output><jupyter_text>Observe that the two evaluations below produce batches of 2 records each.<jupyter_code>feats, delay = input_fn()
all_tensors = [feats[name] for name in SELECT_FEATURES]
all_tensors.append(delay)
with tf.Session() as sess:
    for _ in range(2):
        print(sess.run(all_tensors))<jupyter_output>[array([2013., 2008.], dtype=float32), array([[0, 0, 0, 0, 1, 0, 0],
       [0, 0, 0, 0, 0, 1, 0]]), array(['DL', 'DL'], dtype=object), array(['PHL', 'PNS'], dtype=object), array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
        0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
        0, 0]]), array([377.,  -1.], dtype=float32), array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
        0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
        0, 0]]), array([0.666, 0.272], dtype=float32), array([350., -16.], dtype=float32)]
[array([2006., 2007.], dtype=float32), array([[0, 0, 0, 0, 0, 0, 1],
       [0, 0, 0, 1, 0, 0, 0]]), array(['EV', 'DL'], dtype=object), array(['XNA', 'PDX'], dtype=object), array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
        0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        0, 0]]), arra[...]<jupyter_text>Let's compare that to what the equivalent pandas dataframe shows<jupyter_code>df_train[:4][['FL_YEAR', 'FL_DOW', 'UNIQUE_CARRIER', 'DEST', 'CRS_DEP_TIME', 'DEP_DELAY', 'CRS_ARR_TIME', 'ARR_DELAY', 'DISTANCE']]<jupyter_output><empty_output><jupyter_text>Cool! That seems to work! The input function produces the same values that we can see in the training data. ------First, we'll only be looking at departure delay and distance to destination to predict arrival delay<jupyter_code>fc_dep_delay = fc.numeric_column('DEP_DELAY', dtype=tf.float32)
fc_distance = fc.numeric_column('DISTANCE', dtype=tf.float32)
regressor = tf.estimator.LinearRegressor(feature_columns=[fc_dep_delay, fc_distance])
pred_input_fn = make_input_fn(TEST_FILE,
    True, COLUMNS, 'ARR_DELAY', SELECT_COLUMNS, DEFAULTS, transform, NUM_TEST, 1)
NUM_EPOCHS = 5
BATCH_SIZE = 400
train_input_fn = make_input_fn(TRAIN_FILE,
    True, COLUMNS, 'ARR_DELAY', SELECT_COLUMNS, DEFAULTS, transform, BATCH_SIZE, NUM_EPOCHS)
regressor.train(train_input_fn)<jupyter_output>INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpNWLfPG/model.ckpt.
INFO:tensorflow:loss = 644076.0, step = 1
INFO:tensorflow:global_step/sec: 27.2087
INFO:tensorflow:loss = 83461.266, step = 101 (3.680 sec)
INFO:tensorflow:global_step/sec: 21.7729
INFO:tensorflow:loss = 61313.367, step = 201 (4.591 sec)
INFO:tensorflow:global_step/sec: 23.9469
INFO:tensorflow:loss = 84446.97, step = 301 (4.176 sec)
INFO:tensorflow:global_step/sec: 25.1945
INFO:tensorflow:loss = 60820.887, step = 401 (3.972 sec)
INFO:tensorflow:global_step/sec: 22.6948
INFO:tensorflow:loss = 84637.19, step = 501 (4.406 sec)
INFO:tensorflow:global_step/sec: 25.5626
INFO:tensorflow:loss = 60778.29, step = 601 (3.912 sec)
INFO:tensorflow:global_step/sec: 25.37
INFO:tensorflow:loss = 8[...]<jupyter_text>------Let's see whether our forecast is any good<jupyter_code>def test_prediction(pred, test_file, num_test):
    """
    pred: the result of regressor.predict()
    test_file
    """
    dataframe=pd.read_csv(test_file)
    #predicted = [pred.next()['predictions'][0] for i in range (num_test)]
    predicted = [x['predictions'][0] for x in res]
    np_pred = np.array(predicted)
    lbls = list(dataframe[0:num_test]['ARR_DELAY'])
    np_lbls = np.array(lbls)
    avg_err = np.average(np.abs(np_lbls - np_pred))
    rmse = np.sqrt(np.average(np.square(np_lbls - np_pred)))
    pd.DataFrame(np_lbls - np_pred).plot.hist(bins=40)
    print
    print("Average error: {}, RMSE: {}". format(avg_err, rmse))
pred_input_fn = make_input_fn(
    TEST_FILE, True, COLUMNS, 'ARR_DELAY', 
    SELECT_COLUMNS, DEFAULTS, transform, NUM_TEST, 1)
res = regressor.predict(pred_input_fn)

test_prediction(res, TEST_FILE, NUM_TEST)<jupyter_output>INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpNWLfPG/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.

Average error: 9.43496884799, RMSE: 14.0224600357<jupyter_text>Taking more features into accountWe see that predictions are not too shabby, however we may get better by taking day of week and time of departure and arrival into account...<jupyter_code>fc_crs_dep_time = fc.categorical_column_with_identity('CRS_DEP_TIME', 24)
fc_crs_arr_time = fc.categorical_column_with_identity('CRS_ARR_TIME', 24)
fc_dow = fc.categorical_column_with_identity('FL_DOW', 7)

feature_columns=[
    fc_dep_delay, 
    fc_distance, 
    fc_crs_dep_time, 
    fc_crs_arr_time,
    fc_dow]

regressor2 = tf.estimator.LinearRegressor(feature_columns=feature_columns)
regressor2.train(train_input_fn)
res = regressor2.predict(pred_input_fn)
test_prediction(res, TEST_FILE, NUM_TEST)<jupyter_output>INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /tmp/tmpoccmjm/model.ckpt-1000
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.

Average error: 9.42191620812, RMSE: 14.0181772695
{'fraction_non_alphanumeric': 0.1468895078922934, 'fraction_numerical': 0.04818941504178273, 'mean_word_length': 4.831618841364374, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '281095', 'license': 'Apache-2.0', 'n_tokens_mistral': 4564, 'n_tokens_neox': 4067, 'n_words': 943, 'path': 'kovalevskyi/ATL_June.ipynb', 'repo_name': 'Project-Ellie/tutorials', 'size': '10770'}