<jupyter_start><jupyter_text>Regression Exercise - SolutionsCalifornia Housing DataThis data set contains information about all the block groups in California from the 1990 Census. In this sample a block group on average includes 1425.5 individuals living in a geographically compact area. The task is to aproximate the median house value of each block from the values of the rest of the variables.  It has been obtained from the LIACC repository. The original page where the data set can be found is: http://www.liaad.up.pt/~ltorgo/Regression/DataSets.html.  The Features: * housingMedianAge: continuous. * totalRooms: continuous. * totalBedrooms: continuous. * population: continuous. * households: continuous. * medianIncome: continuous. * medianHouseValue: continuous.   The Data ** Import the cal_housing.csv file with pandas. Separate it into a training (70%) and testing set(30%).**<jupyter_code>import pandas as pd
housing = pd.read_csv('cal_housing_clean.csv')
housing.head()
housing.describe().transpose()
x_data = housing.drop(['medianHouseValue'],axis=1)
y_val = housing['medianHouseValue']
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(x_data,y_val,test_size=0.3,random_state=101)<jupyter_output><empty_output><jupyter_text>Scale the Feature Data** Use sklearn preprocessing to create a MinMaxScaler for the feature data. Fit this scaler only to the training data. Then use it to transform X_test and X_train. Then use the scaled X_test and X_train along with pd.Dataframe to re-create two dataframes of scaled data.**<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
scaler.fit(X_train)
X_train = pd.DataFrame(data=scaler.transform(X_train),columns = X_train.columns,index=X_train.index)
X_test = pd.DataFrame(data=scaler.transform(X_test),columns = X_test.columns,index=X_test.index)<jupyter_output><empty_output><jupyter_text>Create Feature Columns** Create the necessary tf.feature_column objects for the estimator. They should all be trated as continuous numeric_columns. **<jupyter_code>housing.columns
import tensorflow as tf
age = tf.feature_column.numeric_column('housingMedianAge')
rooms = tf.feature_column.numeric_column('totalRooms')
bedrooms = tf.feature_column.numeric_column('totalBedrooms')
pop = tf.feature_column.numeric_column('population')
households = tf.feature_column.numeric_column('households')
income = tf.feature_column.numeric_column('medianIncome')
feat_cols = [ age,rooms,bedrooms,pop,households,income]<jupyter_output><empty_output><jupyter_text>** Create the input function for the estimator object. (play around with batch_size and num_epochs)**<jupyter_code>input_func = tf.estimator.inputs.pandas_input_fn(x=X_train,y=y_train ,batch_size=10,num_epochs=1000,
                                            shuffle=True)<jupyter_output><empty_output><jupyter_text>** Create the estimator model. Use a DNNRegressor. Play around with the hidden units! **<jupyter_code>model = tf.estimator.DNNRegressor(hidden_units=[6,6,6],feature_columns=feat_cols)<jupyter_output>INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: C:\Users\Marcial\AppData\Local\Temp\tmp244d0d44
INFO:tensorflow:Using config: {'_session_config': None, '_save_summary_steps': 100, '_keep_checkpoint_max': 5, '_model_dir': 'C:\\Users\\Marcial\\AppData\\Local\\Temp\\tmp244d0d44', '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_tf_random_seed': 1, '_save_checkpoints_secs': 600, '_save_checkpoints_steps': None}<jupyter_text>** Train the model for ~1,000 steps. (Later come back to this and train it for more and check for improvement) **<jupyter_code>model.train(input_fn=input_func,steps=25000)<jupyter_output>INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into C:\Users\Marcial\AppData\Local\Temp\tmp244d0d44\model.ckpt.
INFO:tensorflow:step = 1, loss = 5.68896e+11
INFO:tensorflow:global_step/sec: 581.528
INFO:tensorflow:step = 101, loss = 5.52951e+11 (0.172 sec)
INFO:tensorflow:global_step/sec: 611.865
INFO:tensorflow:step = 201, loss = 5.91057e+11 (0.163 sec)
INFO:tensorflow:global_step/sec: 619.467
INFO:tensorflow:step = 301, loss = 5.48503e+11 (0.162 sec)
INFO:tensorflow:global_step/sec: 683.11
INFO:tensorflow:step = 401, loss = 1.96651e+11 (0.146 sec)
INFO:tensorflow:global_step/sec: 671.59
INFO:tensorflow:step = 501, loss = 3.18206e+11 (0.149 sec)
INFO:tensorflow:global_step/sec: 680.779
INFO:tensorflow:step = 601, loss = 7.34722e+11 (0.147 sec)
INFO:tensorflow:global_step/sec: 683.111
INFO:tensorflow:step = 701, loss = 6.52219e+11 (0.146 sec)
INFO:tensorflow:global_step/sec: 641.376
INFO:tensorflow:step = 801, loss = 1.67628e+11 (0.155 sec)
INFO:te[...]<jupyter_text>** Create a prediction input function and then use the .predict method off your estimator model to create a list or predictions on your test data. **<jupyter_code>predict_input_func = tf.estimator.inputs.pandas_input_fn(
      x=X_test,
      batch_size=10,
      num_epochs=1,
      shuffle=False)
pred_gen = model.predict(predict_input_func)
predictions = list(pred_gen)<jupyter_output>INFO:tensorflow:Restoring parameters from C:\Users\Marcial\AppData\Local\Temp\tmp244d0d44\model.ckpt-25000<jupyter_text>** Calculate the RMSE. Do this manually or use [sklearn.metrics](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html) **<jupyter_code>final_preds = []
for pred in predictions:
    final_preds.append(pred['predictions'])
from sklearn.metrics import mean_squared_error
mean_squared_error(y_test,final_preds)**0.5<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1073436401967674, 'fraction_numerical': 0.04620520028109627, 'mean_word_length': 6.672506738544475, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '625864', 'license': 'MIT', 'n_tokens_mistral': 2077, 'n_tokens_neox': 1857, 'n_words': 443, 'path': '02-TensorFlow-Basics/07-Regression-Exercise-Solutions-Final.ipynb', 'repo_name': 'joemmetry/tf-dl-notes', 'size': '5692'}