<jupyter_start><jupyter_text>EmbeddingRandomForest  Random Forest with EmbeddingsSo far we have created both a random forest and a NN to do tabular modeling. One thing intresting about a NN is that it contains embeddings. Why don't we try to use these embeddings from the Neural Network in Random Forests? Will it improve the random forest? Lets find out!<jupyter_code>#hide
!pip install -Uqq fastbook kaggle waterfallcharts treeinterpreter dtreeviz
import fastbook
fastbook.setup_book()
#hide
from fastbook import *
from kaggle import api
from pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype
from fastai.tabular.all import *
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor
from dtreeviz.trees import *
from IPython.display import Image, display_svg, SVG

pd.options.display.max_rows = 20
pd.options.display.max_columns = 8<jupyter_output><empty_output><jupyter_text>Unzipping data<jupyter_code>import zipfile

z= zipfile.ZipFile('bluebook-for-bulldozers.zip') #unzip first
z.extractall() #extract<jupyter_output><empty_output><jupyter_text>Grabbing the DataSimiler to what we did in lesson 9, we will grab our data, set the ordinal var, and feature engineer the date.<jupyter_code>df_nn = pd.read_csv(Path()/'TrainAndValid.csv', low_memory=False) #Data

#Set ordinal variables using our order
sizes = 'Large','Large / Medium','Medium','Small','Mini','Compact'
df_nn['ProductSize'] = df_nn['ProductSize'].astype('category')
df_nn['ProductSize'].cat.set_categories(sizes, ordered=True, inplace=True)

dep_var = 'SalePrice'
df_nn[dep_var] = np.log(df_nn[dep_var]) #remember we need to take log of the label (Kaggle requires)

df_nn = add_datepart(df_nn, 'saledate') #Also remember that we used feature engineering on date<jupyter_output><empty_output><jupyter_text>Continous and Categorical columns<jupyter_code>cont_nn,cat_nn = cont_cat_split(df_nn, max_card=9000, dep_var=dep_var) #Max_card makes it so that any col with more than
                                                                            # 9000 lvls, it will be treated as cont
cont_nn<jupyter_output><empty_output><jupyter_text>> Notice that it's missing saleElpased from the cont_nn. We need to add this as we want this col to be treated as cont.<jupyter_code>cont_nn.append('saleElapsed')
cont_nn
cat_nn.remove('saleElapsed')
df_nn.dtypes['saleElapsed'] #must change to int as an object type will cause error
df_nn['saleElapsed'] = df_nn['saleElapsed'].astype('int')<jupyter_output><empty_output><jupyter_text>SplitWe want to split our data by date, not randomly.<jupyter_code>cond = (df_nn.saleYear<2011) | (df_nn.saleMonth<10)
train_idx = np.where( cond)[0]
valid_idx = np.where(~cond)[0]

splits = (list(train_idx),list(valid_idx))<jupyter_output><empty_output><jupyter_text>Tabular objectNow that we have everything we need, lets create our tabular object.<jupyter_code>procs_nn = [Categorify, FillMissing, Normalize]

to_nn = TabularPandas(df_nn, procs_nn, cat_nn, cont_nn, splits=splits, y_names=dep_var)
dls = to_nn.dataloaders(1024) #minibatches
y = to_nn.train.y
y.min(),y.max()<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>learn = tabular_learner(dls, y_range=(8,12), layers=[500,250],
                        n_out=1, loss_func=F.mse_loss)

learn.lr_find() #find best lr
learn.fit_one_cycle(5, 1e-2) #train
#Functions will be using to calc error
def r_mse(pred,y): return round(math.sqrt(((pred-y)**2).mean()), 6)
def m_rmse(m, xs, y): return r_mse(m.predict(xs), y)
preds,targs = learn.get_preds()
r_mse(preds,targs)<jupyter_output><empty_output><jupyter_text>> This is actually very good  Random ForestLets now create our random forest and compare it to our NN  Tabular object<jupyter_code>procs = [Categorify, FillMissing]
rf_to = TabularPandas(df_nn, procs, cat_nn, cont_nn, y_names=dep_var, splits=splits)
#Grab our x,y
xs,y = rf_to.train.xs,rf_to.train.y 
valid_xs,valid_y = rf_to.valid.xs,rf_to.valid.y<jupyter_output><empty_output><jupyter_text>Random Forest<jupyter_code>#Method below creates our random forest and fits it
def rf(xs, y, n_estimators=40, max_samples=200_000, max_features=0.5, min_samples_leaf=5, **kwargs):
    return RandomForestRegressor(n_jobs=-1, n_estimators=n_estimators,
                                max_samples=max_samples, max_features=max_features,
                                min_samples_leaf=min_samples_leaf, oob_score=True).fit(xs, y)
m = rf(xs, y) #Fitting
m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)<jupyter_output><empty_output><jupyter_text>> So it seems our random forest preformed worse in comparison to the NN. Let's improve this by adding the NN embeddings!  Adding embeddings<jupyter_code>learn.model.embeds[:5] #These are just some of the embedding within the NN<jupyter_output><empty_output><jupyter_text>**The function below extracts the embeddings from the model**<jupyter_code>def embed_features(learner, xs):
    xs = xs.copy()
    for i, feature in enumerate(learn.dls.cat_names):
        emb = learner.model.embeds[i].cpu()
        new_feat = pd.DataFrame(emb(tensor(xs[feature], dtype=torch.int64)), index=xs.index, columns=[f'{feature}_{j}' for j in range(emb.embedding_dim)])
        xs.drop(columns=feature, inplace=True)
        xs = xs.join(new_feat)
    return xs
#Extracting embeddings
embeded_xs = embed_features(learn, learn.dls.train.xs)
xs_valid = embed_features(learn, learn.dls.valid.xs)
embeded_xs.shape, xs_valid.shape<jupyter_output><empty_output><jupyter_text>Fitting embeddingsNow that we have our embeddings, lets fit it into the random forest.<jupyter_code>m = rf(embeded_xs, y) #Fitting
m_rmse(m, embeded_xs, y), m_rmse(m, xs_valid, valid_y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10565709312445605, 'fraction_numerical': 0.009399477806788513, 'mean_word_length': 5.767962308598351, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '49592', 'license': 'Apache-2.0', 'n_tokens_mistral': 2096, 'n_tokens_neox': 2006, 'n_words': 493, 'path': '_notebooks/EmbeddingRandomForest.ipynb', 'repo_name': 'usama280/PasteBlogs', 'size': '5745'}