<jupyter_start><jupyter_text>Exploring datastructures for datasetA Pandas exploration. Find the best datastructure to explore and transform the dataset (both training and test dataframes). Use case:- find all numerical features (filtering)- transform all numerical features (e.g. take square)- replace NaN values for a numerical feature- plot distribution for a column in the training dataset<jupyter_code>import sys
import os

import pandas as pd
import seaborn as sns
sys.path.insert(1, os.path.join(sys.path[0], '..'))  # add parent directory to path
import samlib<jupyter_output><empty_output><jupyter_text>Using `samlib.DataSet`Original approach used in `data_exploration_numerical_features`- class that contains 3 dataframes attributes (train, test, df, where df is the full dataframe- whenever df is updated, the train and test frames are updatedThis allows to work with the training dataset, and to update/transform the full dataset if necessary, so that the transformation is also applied to the test dataframe that will be needed for the final prediction.<jupyter_code>raw_train = pd.read_csv('../data/train_prepared_light.csv')
raw_test = pd.read_csv('../data/test_prepared_light.csv')
ds = samlib.DataSet(raw_train, raw_test)
is_num = ds.dtypes != object
dfnum = ds.df.loc[:, is_num]
dfnum.head()
ds.apply(lambda df: df.loc[:, is_num]**2, inplace=True)
ds.df.head()
ds.df.MasVnrArea.isnull().sum()
ds.df.loc[ds.df.MasVnrArea.isnull(), 'MasVnrArea'] = 0
ds.df.MasVnrArea.isnull().sum()
sns.distplot(ds.train.GrLivArea)<jupyter_output><empty_output><jupyter_text>Works but not so great because requires a new dependency (samlib) and a different way of working compared to Pandas. Need to learn the behaviour of the DataSet class, and remember to use the `apply` method otherwise the `train` and `test` sets are *not* going to be kept in sync (for example when assigning to a slice of `ds.df`)  Using an extra categorical `dataset` column<jupyter_code>traindf = raw_train.copy()
testdf = raw_test.copy()
traindf['dataset'] = 'train'
testdf['dataset'] = 'test'
df = pd.concat([traindf, testdf])<jupyter_output><empty_output><jupyter_text>Then we can filter using the value of the `dataset` column<jupyter_code>train = df['dataset'] == 'train'
test = ~train
df[train].head()
df[test].head()
is_num = df.dtypes != object
dfnum = df.loc[:, is_num]
dfnum.head()
df.loc[:, is_num] = dfnum **2
df.head()
df.MasVnrArea.isnull().sum()
df.loc[df.MasVnrArea.isnull(), 'MasVnrArea'] = 0
df.MasVnrArea.isnull().sum()
sns.distplot(df.loc[train, 'GrLivArea'])<jupyter_output><empty_output><jupyter_text>Works quite well but takes a bit of work to setup and requires to keep two boolean series (`train` and `test`) to filter the dataset whenever needed. An improvement over `samlib.DataSet` though.   Using Panel object<jupyter_code>panel = pd.Panel({'train':raw_train.copy(), 'test': raw_test.copy()})
panel.train.head()
panel.test.head()<jupyter_output><empty_output><jupyter_text>The above is very nice, but unfortunately a panel isn't a dataframe so we can't really get a view of the full data. Also we seem to have lost all the data types:<jupyter_code>is_num = panel.train.dtypes != object
any(is_num)<jupyter_output><empty_output><jupyter_text>So we must keep the raw data if we want to filter the numerical columns :-(<jupyter_code>is_num = raw_train.dtypes != object
numpanel = panel.loc[:, :, is_num]
numpanel
numpanel.train.head()<jupyter_output><empty_output><jupyter_text>Finally this raises an error!<jupyter_code>try:
    panel.loc[:, :, is_num] = panel.loc[:, :, is_num]**2
except NotImplementedError as err:
    print('raises NotImplementedError: ', err)<jupyter_output><empty_output><jupyter_text>Looked promising initially but not really workable as we can't assign an indexer with a Panel yet. We really need a dataframe object.  Using multi-index on rows<jupyter_code>traindf = raw_train.copy()
testdf = raw_test.copy()
df = pd.concat([traindf, testdf], keys=('train', 'test'))
df.head()
df.tail()<jupyter_output><empty_output><jupyter_text>The test and train datasets can be accessed by filtering the index. Nice but not quite as compact as `df[train]`, though we don't need the extra `train` (and `test`) masks.<jupyter_code>df.loc['train'].head()
is_num = df.dtypes != object
dfnum = df.loc[:, is_num]
dfnum.head()
df.loc[:, is_num] = dfnum **2
df.head()
df.MasVnrArea.isnull().sum()
df.loc[df.MasVnrArea.isnull(), 'MasVnrArea'] = 0
df.MasVnrArea.isnull().sum()
sns.distplot(df.GrLivArea.train)<jupyter_output><empty_output><jupyter_text>Another way of doing it<jupyter_code>sns.distplot(df.loc['train', 'GrLivArea'])<jupyter_output><empty_output><jupyter_text>Works very well.   Using multi-index on columns (swapped levels)Swap the levels to fix the issue with filtering on features in the column multi-index case.<jupyter_code>traindf = raw_train.copy()
testdf = raw_test.copy()
df = pd.concat([traindf, testdf], axis=1, keys=('train','test')).swaplevel(axis=1)
df.sort_index(axis=1, inplace=True)  # needed otherwise we get in trouble for slicing
df.head()
df.tail()<jupyter_output><empty_output><jupyter_text>The test and train datasets can be accessed by filtering the index. Nice but not quite as compact as `df[train]`, though we don't need the extra `train` (and `test`) masks.<jupyter_code>df.xs('train', level=1, axis=1).head()  # or use IndexSlice<jupyter_output><empty_output><jupyter_text>We must also deal with the extra index level when filtering, but it's not too bad.<jupyter_code>is_num = df.dtypes != object
dfnum = df.loc[:, is_num]
dfnum.head()
df.loc[:, is_num] = dfnum **2
df.head()<jupyter_output><empty_output><jupyter_text>Getting nulls and setting nulls (without fillna) is a little tricky. Boolean indexing is (by definition) meant to work over rows, not rows *and columns. We can use boolean arrays with DataFrame.mask though. But this is definitely something to keep in mind when using multi indexing over columns.<jupyter_code>df.MasVnrArea = df.MasVnrArea.mask(df.MasVnrArea.isnull(), 0)
df.MasVnrArea.tail()<jupyter_output><empty_output><jupyter_text>Visualizing the training dataset is pretty easy.<jupyter_code>sns.distplot(df.GrLivArea.train)<jupyter_output><empty_output><jupyter_text>Using multi-index on columnsMakes it easier to filter on dataset (train or test) and has the advantage of being a dataframe.<jupyter_code>traindf = raw_train.copy()
testdf = raw_test.copy()
df = pd.concat([traindf, testdf], axis=1, keys=('train','test'))
df.head()
df.tail()<jupyter_output><empty_output><jupyter_text>The test and train datasets can be accessed by filtering the index. Nice but not quite as compact as `df[train]`, though we don't need the extra `train` (and `test`) masks.<jupyter_code>df.train.head()<jupyter_output><empty_output><jupyter_text>We must also deal with the extra index level when filtering, but it's not too bad.<jupyter_code>is_num = df.dtypes != object
dfnum = df.loc[:, is_num]
dfnum.head()
df.loc[:, is_num] = dfnum **2
df.head()<jupyter_output><empty_output><jupyter_text>Definitely harder to slice accross columns. It's possible (unlike with panels), but hard (requires pd.IndexSlice).<jupyter_code>df.loc[:, pd.IndexSlice[:, 'MasVnrArea']].isnull().sum()<jupyter_output><empty_output><jupyter_text>You can also use a cross section to get the data more easily, but you can't use this for sssignments<jupyter_code>df.xs('MasVnrArea', axis=1, level=1).head()
df.loc[:, pd.IndexSlice[:, 'MasVnrArea']] = 0
df.loc[:, pd.IndexSlice[:, 'MasVnrArea']].isnull().sum()<jupyter_output><empty_output><jupyter_text>Visualizing the training dataset is pretty easy.<jupyter_code>sns.distplot(df.train.GrLivArea)<jupyter_output><empty_output><jupyter_text>Using dataset type as label ** Method 1: add columns then use set_index **<jupyter_code>traindf = raw_train.copy()
testdf = raw_test.copy()
traindf['Dataset'] = 'train'
testdf['Dataset'] = 'test'
df = pd.concat([traindf, testdf])
df.set_index('Dataset').head()<jupyter_output><empty_output><jupyter_text>** Method 2: use concat and droplevel **<jupyter_code>traindf = raw_train.copy()
testdf = raw_test.copy()
df = pd.concat([traindf, testdf], keys=('train', 'test'))
df.index = df.index.droplevel(1)
df.head()

df.tail()<jupyter_output><empty_output><jupyter_text>The test and train datasets can be accessed by using `loc` .<jupyter_code>df.loc['train'].head()<jupyter_output><empty_output><jupyter_text>Filtering columns is very easy<jupyter_code>is_num = df.dtypes != object
dfnum = df.loc[:, is_num]
dfnum.head()
df.loc[:, is_num] = dfnum **2
df.head()
df.MasVnrArea.isnull().sum()
df.loc[df.MasVnrArea.isnull(), 'MasVnrArea'] = 0
df.MasVnrArea.isnull().sum()
sns.distplot(df.GrLivArea.train)<jupyter_output><empty_output><jupyter_text>Another way of doing it<jupyter_code>sns.distplot(df.loc['train', 'GrLivArea'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13045439170143194, 'fraction_numerical': 0.003044311647310858, 'mean_word_length': 6.555366269165247, 'pattern_counts': {'":': 0, '<': 109, '<?xml version=': 0, '>': 109, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '8521', 'license': 'MIT', 'n_tokens_mistral': 3098, 'n_tokens_neox': 3082, 'n_words': 842, 'path': 'research/dataset_structure.ipynb', 'repo_name': 'samuelsinayoko/kaggle-housing-prices', 'size': '8869'}