<jupyter_start><jupyter_text>Load Demo Datafrom [kaggle](https://www.kaggle.com/c/house-prices-advanced-regression-techniques)<jupyter_code>df = pd.read_csv("../data/train.csv")
df.head()<jupyter_output><empty_output><jupyter_text>Nominal- and Ordinal-scale VariablesBoth nominal- and ordinal-scale variables can be OneHot encoded<jupyter_code># values are strings or some other alphanumeric mix
nom_str = [
    'MSZoning', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities', 
    'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 
    'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 
    'MasVnrType', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond', 
    'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'Heating', 'HeatingQC', 
    'CentralAir', 'Electrical', 'KitchenQual', 'Functional', 'FireplaceQu', 
    'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PavedDrive', 
    'PoolQC', 'Fence', 'MiscFeature', 'SaleType', 'SaleCondition']

# nominal values are already encoded
nom_int = ['MSSubClass', 'MoSold']

# ordinal values encoded as integers
ord_int = ['OverallQual', 'OverallCond']
df[nom_str].describe()
df[nom_int].astype(str).describe()
df[ord_int].astype(str).describe()<jupyter_output><empty_output><jupyter_text>How to encode 1 variableLet's pick `GarageType`. The variable has missing values (None, Nan, etc) and 6 distince values<jupyter_code>x = df['GarageType']<jupyter_output><empty_output><jupyter_text>Label EncodingStore the unique values in list named `mapping`<jupyter_code>mapping = [e for e in set(x) if pd.notnull(e)]
mapping = dict(enumerate(mapping))  # this will be useful lateron
mapping<jupyter_output><empty_output><jupyter_text>Encode mapped values. Everything else is encoded as `None`(The same can be done with sklearn's `LabelEncoder`)<jupyter_code>xencoded = grouplabelencode(x, mapping, nastate=False)
pd.DataFrame(np.c_[x, xencoded])[1447:1452]<jupyter_output><empty_output><jupyter_text>One-Hot EncodingFor each encoding create 1 column, or resp. 1 new dummy variable, or resp. 1 new one-hot feature.(The same can be done with `pd.get_dummies(xencoded, prefix='x')` or sklearn's `OneHotEncoder`)<jupyter_code>def onehotencode(xencoded, mapping):
    import scipy.sparse
    out = scipy.sparse.lil_matrix((len(xencoded), len(mapping)), dtype=int)
    enc = list(mapping.keys())
    for i, j in enumerate(xencoded):
        if j in enc:
            out[i, j] = 1
    return out
xonehot = onehotencode(xencoded, mapping)
xonehot.toarray()
import scipy.sparse
test = scipy.sparse.hstack([xonehot, xonehot.sum(axis=1)])
test
np.sum(xonehot.toarray(), axis=0)<jupyter_output><empty_output><jupyter_text>Make it look goodIf that's even possible ...<jupyter_code>colnam = list(mapping.keys())
pd.DataFrame(xonehot.toarray(), columns=colnam)[1447:1452]
prefix = 'cool'
sep = '_'
colnam = [prefix + sep + str(k) for k in mapping.keys()]
pd.DataFrame(xonehot.toarray(), columns=colnam)[1447:1452]
colnam = list(mapping.values())
pd.DataFrame(xonehot.toarray(), columns=colnam)[1447:1452]
prefix = 'cool'
sep = '_'
colnam = [prefix + sep + str(v) for v in mapping.values()]
pd.DataFrame(xonehot.toarray(), columns=colnam)[1447:1452]
tostr = lambda s: ''.join(str(s).split()).lower()
prefix = 'cool'
sep = '_'
colnam = [prefix + sep + tostr(k) + sep + tostr(v) for k, v in mapping.items()]
pd.DataFrame(xonehot.toarray(), columns=colnam)[1447:1452]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15369204350314825, 'fraction_numerical': 0.017744705208929592, 'mean_word_length': 6.516129032258065, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '220984', 'license': 'MIT', 'n_tokens_mistral': 1336, 'n_tokens_neox': 1261, 'n_words': 274, 'path': 'examples/one hot encoding - step by step.ipynb', 'repo_name': 'kmedian/onehot', 'size': '3494'}