<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Importing Libraries and Creating data2&nbsp;&nbsp;Preparation3&nbsp;&nbsp;Encoding This notebook has been prepared making use of the datasets in  link. But i believe, what i am presenting is simpler, more up-to-date and thus better. sitesindeki datasetler kullanılarak hazırlanmıştır. This is more up-to-date because for example we don't need LabelEncoding before OnehHotEncoding for string features any more. Simpler, because we have coded less.I don't even think about pandas' get_dummies method as it is too long to implement.  Importing Libraries and Creating data<jupyter_code>import pandas as pd
from sklearn.preprocessing import OneHotEncoder
import mlextension as ml #for the function returning onehot columns
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Let's first create the train set<jupyter_code>df_train = pd.DataFrame([["London", "car", 20],
                   ["Cambridge", "car", 10], 
                   ["Liverpool", "bus", 30]], 
                  columns=["city", "transport", "duration"])
df_train<jupyter_output><empty_output><jupyter_text>Now test set<jupyter_code>df_test = pd.DataFrame([["Manchester", "bike", 30], 
                        ["Cambridge", "car", 40], 
                        ["Liverpool", "bike", 10]], 
                       columns=["city", "transport", "duration"])
df_test<jupyter_output><empty_output><jupyter_text>Preparation<jupyter_code>#which columns to encode
cat_columns = ["city", "transport"]<jupyter_output><empty_output><jupyter_text>The main difference in my solution, **we are getting the whole set's unique items**. Because we may lack an item in both train and test set, as in this example. In train set, we don't have Manchester in city column and bike in transport column, whereas in test set we lack London in city and bus in transport.<jupyter_code>whole=pd.concat([df_train,df_test],axis=0)
whole
cats=ml.getfullitemsforOHE(whole,cat_columns) #the last parameter is True as defaul, which sorts the items.
#but it doesn't matter whether they are sorted or not,it is just a matter of preference
cats<jupyter_output><empty_output><jupyter_text>Encoding<jupyter_code>ohe=OneHotEncoder(categories=cats, sparse=False,handle_unknown="ignore")
X_trans=ohe.fit_transform(df_train[cat_columns])
#train set will be transformed as below and go into a model like this
X_trans
#let's presnet them in a dataframe
pd.DataFrame(X_trans,columns=ohe.get_feature_names(cat_columns))
#let's transform the test set as well, only transform(not fit)
x_test_trans=ohe.transform(df_test[cat_columns])
pd.DataFrame(x_test_trans,columns=ohe.get_feature_names(cat_columns))
#now, we have a new dataset that has values neither train set nor test set has, let's see if any problems crop up
# Istanbul in place of Manchester, plane in place of second bike
df_new = pd.DataFrame([["Istanbul", "bike", 30], 
                        ["Cambridge", "car", 40], 
                        ["Liverpool", "plane", 10]], 
                       columns=["city", "transport", "duration"])
df_new
#thanks to handle_unknown="ignore" parameter no problems come up. remove that parameter, run again and see the error
x_new_trans=ohe.transform(df_new[cat_columns])
pd.DataFrame(x_new_trans,columns=ohe.get_feature_names(cat_columns))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09753928253780018, 'fraction_numerical': 0.006818855618144086, 'mean_word_length': 4.888307155322862, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '650506', 'license': 'Apache-2.0', 'n_tokens_mistral': 1063, 'n_tokens_neox': 990, 'n_words': 330, 'path': 'OneHotEncoding.ipynb', 'repo_name': 'VolkiTheDreamer/PythonRocks', 'size': '3373'}