<jupyter_start><jupyter_text>Crime Analysis   Data Preperation ProcessThe process for getting data ready for a machine learning algorithm can be summarized in three steps:1. Select Data - Consider what data is available, what data is missing and what data can be removed.2. Preprocess Data - Organize your selected data by formatting, cleaning and sampling from it.3. Transform Data - Transform preprocessed data ready for machine learning by engineering features using scaling, attribute decomposition and attribute aggregation.   1. Select DataWe'll use the combined data that we processed from the raw files from Toronto's Open Data.> http://data.torontopolice.on.ca/<jupyter_code>import pandas as pd
pd.set_option('display.max_columns', 500)
pd.set_option('display.max_rows', 500)


crimes = pd.read_csv('../data/interim/crimes.csv')
crimes.head(50)<jupyter_output><empty_output><jupyter_text>2.0 Preprocess DataHandling categorical data- **Label Encoder**: It is used to transform non-numerical labels to numerical labels (or nominal categorical variables). Numerical labels are always between 0 and n_classes-1.     - convertible -> 0    - hardtop -> 1    - hatchback -> 2    - sedan -> 3    - wagon -> 4- **Dummy Coding / One-Hot-Encoding**: Dummy coding is a commonly used method for converting a categorical input variable into continuous variable. ‘Dummy’, as the name suggests is a duplicate variable which represents one level of a categorical variable. Presence of a level is represent by 1 and absence is represented by 0. For every level present, one dummy variable will be created.    - `OneHotEncoder` cannot process string values directly. If your nominal features are strings, then you need to first map them into integers.    - `pandas.get_dummies` is kind of the opposite. By default, it only converts string columns into one-hot representation, unless columns are specified.<jupyter_code>crimes["MCI"].value_counts()
from sklearn.preprocessing import LabelEncoder
from sklearn.pipeline import Pipeline

class MultiColumnLabelEncoder:
    def __init__(self,columns = None):
        self.columns = columns # array of column names to encode

    def fit(self,X,y=None):
        return self # not relevant here

    def transform(self,X):
        '''
        Transforms columns of X specified in self.columns using
        LabelEncoder(). If no columns specified, transforms all
        columns in X.
        '''
        output = X.copy()
        if self.columns is not None:
            for col in self.columns:
                output[col] = LabelEncoder().fit_transform(output[col])
        else:
            for colname,col in output.iteritems():
                output[colname] = LabelEncoder().fit_transform(col)
        return output

    def fit_transform(self,X,y=None):
        return self.fit(X,y).transform(X)
crimes = MultiColumnLabelEncoder(columns = ['premisetype','offence', 'Neighbourhood', "occurrencemonth", "MCI"]).fit_transform(crimes)
# get dummies
#crimes = pd.get_dummies(crimes, columns=['MCI'])
list(crimes)
ax = crimes['MCI'].value_counts().plot(kind='bar', figsize=(10,7), fontsize=13);
ax.set_alpha(0.8)
ax.set_title("Major Crime Indicators", fontsize=18)

# create a list to collect the plt.patches data
totals = []

# find the values and append to list
for i in ax.patches:
    totals.append(i.get_height())

# set individual bar lables using above list
total = sum(totals)

# set individual bar lables using above list
for i in ax.patches:
    # get_x pulls left or right; get_height pushes up or down
    ax.text(i.get_x()-.03, i.get_height()+.5, \
            str(round((i.get_height()/total)*100, 2))+'%', fontsize=15,
                color='dimgrey')<jupyter_output><empty_output><jupyter_text>Handling  imbalanced datasets<jupyter_code># define X and y
feature_cols = ['premisetype', 'occurrenceyear', 'occurrencemonth', 
                'day_id', 'occurrencedayofyear', 'occurrencehour', 
                'Hood_ID', 'Neighbourhood', 'Lat', 'Long']
X = crimes[feature_cols]
y = crimes.MCI
 from imblearn.over_sampling import SMOTE
    
sm = SMOTE(random_state=42)
X_res, y_res = sm.fit_sample(X,y)<jupyter_output><empty_output><jupyter_text>Export to CSV<jupyter_code>crimes.to_csv('../data/processed/crimes_le.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09900301414328773, 'fraction_numerical': 0.009969858567122653, 'mean_word_length': 4.0162790697674415, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '111514', 'license': 'MIT', 'n_tokens_mistral': 1317, 'n_tokens_neox': 1242, 'n_words': 424, 'path': 'notebooks/2.0-is-data-preprocessing.ipynb', 'repo_name': 'siopow/Toronto-Crime-Analysis', 'size': '4313'}