<jupyter_start><jupyter_text>Resfeber gas data set exporationVery basic preliminary exploration of our primary gasoline price dataset. For future reference the data set can be accessed through api as well.[source](https://www.eia.gov/dnav/pet/pet_pri_gnd_a_epm0_pte_dpgal_w.htm)* Explore* Clean* Regression prediction* Summary and Next steps<jupyter_code>import pandas as pd

df = pd.read_excel('PET_PRI_GND_A_EPM0_PTE_DPGAL_W.xls', sheet_name = 2)

df.head()
header = df.iloc[1]
df = df[2:]
df.columns = header

df.head()
df.shape
df.describe()
# Okay lets maybe split the date up into year month and day.
df['Date'].astype('datetime64[ns]')
df['Date'] = df['Date'].astype('datetime64[ns]')
df.head()
df['Month'] = df['Date'].apply(lambda x: x.month)
df['Day'] = df['Date'].apply(lambda x: x.day)
df['Year'] = df['Date'].apply(lambda x: x.year)

df.head()
truncated_headers = ['date', 'east_coast', 'new_england', 'central_atlantic', 
                     'lower_atlantic', 'midwest', 'gulf_coast', 
                     'rocky_mountain', 'west_coast', 'west_coast_no_cal', 
                     'month', 'day', 'year']

df.columns = truncated_headers
df.head()
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Based off this, I suspect that only the first 267 entries have nans.  So checking to see if it is the case.  If so, I can evaluate the dates, and either cut all those entries out, or potentially don't use the west_coast_no_cal column.<jupyter_code>df.iloc[267:].isnull().sum()
clean = df.iloc[267:]
clean.head()
# I'm going to iterate through each region building a model testing
# and validating. Need a list of the regions for that.
y_list = clean.columns
y_list = y_list.drop(['date', 'month', 'day', 'year'])
y_list
X = clean[['month', 'day', 'year']]
X.head()<jupyter_output><empty_output><jupyter_text>PredictionsFor each region I'm going to use a mean guess as the baseline.  Then I'm going to train a linear regression model, and compare the mean absolute error of each model agains the baseline.<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error, r2_score, mean_squared_error
import numpy as np

for target in y_list:
    y = clean[target]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.24,
                                                       random_state = 42)
    baseline = DummyRegressor(strategy = 'mean')
    baseline.fit(X_train, y_train)
    y_pred_base = baseline.predict(X_test)
    mae_base = mean_absolute_error(y_test, y_pred_base)
    mse_base = mean_squared_error(y_test, y_pred_base)
    rmse_base = np.sqrt(mse_base)
    r2_base = r2_score(y_test, y_pred_base)
    
    model = LinearRegression()
    model.fit(X_train, y_train)
    
    y_pred_train = model.predict(X_train)
    mae_train = mean_absolute_error(y_train, y_pred_train)
    mse_train = mean_squared_error(y_train, y_pred_train)
    rmse_train = np.sqrt(mse_train)
    r2_train = r2_score(y_train, y_pred_train)
    
    y_pred_test = model.predict(X_test)
    mae_test = mean_absolute_error(y_test, y_pred_test)
    mse_test = mean_squared_error(y_test, y_pred_test)
    rmse_test = np.sqrt(mse_test)
    r2_test = r2_score(y_test, y_pred_test)
    
    print(f'@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@')
    print(f'Evaluating {target} Region Gas Predictions')
    print()
    print(f'@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@')
    print(f'------------------------------------------------------------------')
    print(f'                Mean Absolute Error')
    print(f'------------------------------------------------------------------')
    print(f'Baseline error: ${mae_base:.2f} per galon')
    print(f'Train error:    ${mae_train:.2f} per galon')
    print(f'Test error:     ${mae_test:.2f} per galon')
    print(f'------------------------------------------------------------------')
    print(f'                Mean Squared Error')
    print(f'------------------------------------------------------------------')
    print(f'Baseline error: ${mse_base:.2f} per galon')
    print(f'Train error:    ${mse_train:.2f} per galon')
    print(f'Test error:     ${mse_test:.2f} per galon')
    print(f'------------------------------------------------------------------')
    print(f'                Root Mean Squared Error')
    print(f'------------------------------------------------------------------')
    print(f'Baseline error: ${rmse_base:.2f} per galon')
    print(f'Train error:    ${rmse_train:.2f} per galon')
    print(f'Test error:     ${rmse_test:.2f} per galon')
    print(f'------------------------------------------------------------------')
    print(f'                R-Squared (coefficient of determination)')
    print(f'------------------------------------------------------------------')
    print(f'Baseline r2: {r2_base:.2f}')
    print(f'Train r2:    {r2_train:.2f}')
    print(f'Test r2:     {r2_test:.2f}')
    print(f'------------------------------------------------------------------')
    print()
    print()<jupyter_output>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Evaluating east_coast Region Gas Predictions

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
------------------------------------------------------------------
                Mean Absolute Error
------------------------------------------------------------------
Baseline error: $0.68 per galon
Train error:    $0.56 per galon
Test error:     $0.56 per galon
------------------------------------------------------------------
                Mean Squared Error
------------------------------------------------------------------
Baseline error: $0.66 per galon
Train error:    $0.43 per galon
Test error:     $0.43 per galon
------------------------------------------------------------------
                Root Mean Squared Error
------------------------------------------------------------------
Baseline error: $0.81 per galon
Train error:    $0.66 per galon
Test error:     $0.65 per galon
-------------------[...]
{'fraction_non_alphanumeric': 0.30061152236884453, 'fraction_numerical': 0.012069520437721274, 'mean_word_length': 1.9510921177587843, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '581440', 'license': 'MIT', 'n_tokens_mistral': 2000, 'n_tokens_neox': 1885, 'n_words': 447, 'path': 'notebooks/exploration-nb/gas_exploration.ipynb', 'repo_name': 'Lambda-School-Labs/LabsPT13-Resfeber-DS', 'size': '6214'}