<jupyter_start><jupyter_text>Python Machine Learning - Code Examples  Data Pre-Processing Organizing Training Sets  and Test sets.  Overview - [Dealing with missing data](Dealing-with-missing-data)  - [Eliminating samples or features with missing values](Eliminating-samples-or-features-with-missing-values)- [Handling categorical data](Handling-categorical-data)  - [Mapping ordinal features](Mapping-ordinal-features)  - [Encoding class labels](Encoding-class-labels)  - [Performing one-hot encoding on nominal features](Performing-one-hot-encoding-on-nominal-features)-[Imputing missing values](Imputing-missing-values)- [Bringing features onto the same scale](Bringing-features-onto-the-same-scale)   - [Understanding the scikit-learn estimator API](Understanding-the-scikit-learn-estimator-API)   - [Partitioning a dataset in training and test sets](Partitioning-a-dataset-in-training-and-test-sets)- [Ranking Features](Ranking Features)    - [univariate methods](univariate methods)<jupyter_code>#from IPython.display import Image
%matplotlib inline
# Added version check for recent scikit-learn 0.18 checks
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version
#Disable warning
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning)<jupyter_output><empty_output><jupyter_text>Dealing with missing data<jupyter_code>import pandas as pd
from io import StringIO

csv_data = '''A,B,C,D
1.0,2.0,3.0,4.0
5.0,6.0,,8.0
10.0,11.0,12.0,'''



df = pd.read_csv(StringIO(csv_data))
df
# print the number of missing 
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Eliminating samples or features with missing values<jupyter_code>#### deleting rows of the data frame
df.dropna()
# only drop rows where all columns are NaN
df.dropna(how='all')  
# drop rows that have not at least 4 non-NaN values
df.dropna(thresh=4)
# only drop rows where NaN appear in specific columns (here: 'C')
df.dropna(subset=['C'])
# Eliminate the feature with NaN
df.dropna(axis=1)<jupyter_output><empty_output><jupyter_text>Handling categorical data<jupyter_code>import pandas as pd

df = pd.DataFrame([['green', 'M', 10.1, 'class1'],
                   ['red', 'L', 13.5, 'class2'],
                   ['blue', 'XL', 15.3, 'class1']])

df.columns = ['color', 'size', 'price', 'classlabel']
df<jupyter_output><empty_output><jupyter_text>Mapping ordinal features<jupyter_code>size_mapping = {'XL': 3,
                'L': 2,
                'M': 1}

df['size'] = df['size'].map(size_mapping)
df
inv_size_mapping = {v: k for k, v in size_mapping.items()}
df['size'].map(inv_size_mapping)<jupyter_output><empty_output><jupyter_text>Encoding class labels<jupyter_code>import numpy as np

class_mapping = {label: idx for idx, label in enumerate(np.unique(df['classlabel']))}
class_mapping
df['classlabel'] = df['classlabel'].map(class_mapping)
df
inv_class_mapping = {v: k for k, v in class_mapping.items()}
df['classlabel'] = df['classlabel'].map(inv_class_mapping)
df
from sklearn.preprocessing import LabelEncoder

class_le = LabelEncoder()
y = class_le.fit_transform(df['classlabel'].values)
print(np.shape(y),y)
y
class_le.inverse_transform(y)<jupyter_output><empty_output><jupyter_text>Performing one-hot encoding on nominal features<jupyter_code>X = df[['color', 'size', 'price']].values

color_le = LabelEncoder()
X[:, 0] = color_le.fit_transform(X[:, 0])
X
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder()
ohe.fit_transform(X).toarray()
pd.get_dummies(df[['price', 'color', 'size']])<jupyter_output><empty_output><jupyter_text>Imputing missing values<jupyter_code>import pandas as pd
from io import StringIO

csv_data = '''A,B,C,D
1.0,2.0,3.0,4.0
5.0,6.0,,8.0
10.0,11.0,12.0,'''



df = pd.read_csv(StringIO(csv_data))
df
from sklearn.impute import SimpleImputer
# type of imputation
imr = SimpleImputer(missing_values=np.nan, strategy='mean')
# Calculation
imr = imr.fit(df)
#Application
imputed_data = imr.transform(df.values)
imputed_data
df.values<jupyter_output><empty_output><jupyter_text>Remark:  Understanding the scikit-learn estimator API To evaluate correctly a methodology at least two distinct data sets are used:1.  The training set: where all parameters of the model are learned. Even the pre-processing step parameters. The "fit" of the model of the transformation is achieved with this data set.2. Afterwards  both sets ( training and test ) are  transformed.  ![title](FIG/bloco.png)  Bringing features onto the same scale<jupyter_code>ex = pd.DataFrame([0, 1, 2, 3, 4, 5])

# standardize
ex[1] = (ex[0] - ex[0].mean()) / ex[0].std(ddof=0)

# Please note that pandas uses ddof=1 (sample standard deviation) 
# by default, whereas NumPy's std method and the StandardScaler
# uses ddof=0 (population standard deviation)

# normalize
ex[2] = (ex[0] - ex[0].min()) / (ex[0].max() - ex[0].min())
ex.columns = ['input', 'standardized', 'normalized']
ex
from sklearn.preprocessing import MinMaxScaler
# Data should be in matrices
#mms = MinMaxScaler()
# Train fit and transform. Test only transform.
#X_train_norm = mms.fit_transform(X_train)
#X_test_norm = mms.transform(X_test)
from sklearn.preprocessing import StandardScaler

#stdsc = StandardScaler()
#X_train_std = stdsc.fit_transform(X_train)
#X_test_std = stdsc.transform(X_test)<jupyter_output><empty_output><jupyter_text>Ranking FeaturesIn classification problems it can be relevant to have an idea how features values are distinct according to some criteria with the classes.  Univariate Feature SelectionUnivariate feature selection selects the best features by running univariate statistical tests like **chi-squared test, F-1 test, and mutual information** methods.<jupyter_code>from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2, mutual_info_classif<jupyter_output><empty_output><jupyter_text>Partitioning a dataset in training and test setsUsually the data is stored into one file. Then after reading is needed to organize the data into two subsets.1. The basic strategy is the hold out method. 2. The percentage of members in both classes are kept in the division.<jupyter_code>if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split
    
    
#label   reading from a data frame to a 
#y=df.as_matrix(columns=[df.columns[0]])

# features
#X=df2.as_matrix(columns=df.columns[1:])

X_train,X_test,y_train,y_test = train_test_split(X,y, test_size = 0.3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1173638278663858, 'fraction_numerical': 0.015197111044237136, 'mean_word_length': 5.909563409563409, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '623199', 'license': 'MIT', 'n_tokens_mistral': 2255, 'n_tokens_neox': 2161, 'n_words': 585, 'path': 'ED/Exercises/Exercise 1/.ipynb_checkpoints/DataTransformations-checkpoint.ipynb', 'repo_name': 'lengors/ua-repository', 'size': '6646'}