<jupyter_start><jupyter_text>License Stuff```Copyright 2021 <NAME>Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.```  Introduction to preprocessing with pandas When I ask a pythonners to read a csv file they will often do it with the raw csv library : ``` python import csvwith open('file.csv', 'r') as file:    my_reader = csv.reader(file, delimiter=',')    for row in my_reader:        print(row)````This isn't necessarily bad but for any data preprocessing further down the line it is going to be rather complex.In Data Analysis or Data Science most people use the following library to read and process datasets<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. Released in 2008 Interesting facts: Pandas is based on NumPy ( another famous python library ) but not 100%.Examples:   * Matrix operations are based on NumPy logic (since that library implements efficient and fast Cython code for performing matrix algebra operations. )   * Data structures in Pandas are Pandas specific and not based on NumPy In practice Let's try to open the Titanic passenger dataset (name, age, price of ticket, etc.) to make some Data Analysis.<jupyter_code>df  = pd.read_csv('../Datasets/titanic.csv')
# lets preview our dataframe using df.head()
df.head()<jupyter_output><empty_output><jupyter_text>Some Data Cleaning / Preprocessing<jupyter_code># we are GDPRD / DPA conscious, thus we may want to drop that name column, we don't need to gossip on who died...
# here is how to do it
df = df.drop('Name',axis=1)
df.head()
# to select one column or multiple columns we can perform as follow:

df['Survived'].head()
# not that using loc render the same results
df.loc[:,'Survived'].head() 
df[['Survived', 'Sex']].head()
# say we are only interested with young peoples

df[df['Age']<24].head()
# say we don't like those spaces in the ticket number how we can remove that ? 

#first let's create a function which takes a string and then return the string without spaces

def trim_spaces(string):
    
    if type(string)!=str:
        raise Exception(f"Proposed varaible :  {string} is of type {type(string)} instead of str string")
    
    return string.replace(" ", "")



# second lets apply that manipulation to the column Ticket

df['Ticket'].apply(lambda x: trim_spaces(x)).head()<jupyter_output><empty_output><jupyter_text>Some Data Analysis<jupyter_code># this is an handy way to statiscally describe your dataset with one command
df.describe()
#lets say I want to get those statistics female vs male

df.groupby(['Survived','Sex'])['Age'].describe()
df.groupby('Sex').describe()
df.groupby('Pclass')['Survived'].count().plot.bar()
df.Age.hist(bins=7)<jupyter_output><empty_output><jupyter_text>More advance problems ( Memory )When you receive a message like this from one of your best friends, what do you do ? If you are going to work in a Big Data context, you may need to open large files. Several strategies can be implemented to handle that. Using python generatorsif you are not familiar with generators ``` yield syntax```  checkout : https://realpython.com/introduction-to-python-generators/> This method is useful when training machien learning models since they often rely on a concept of "Batch size" for training, thus the data can be passed to the model by chunksize.<jupyter_code>df2  = pd.read_csv('../Datasets/titanic.csv',chunksize=3)

for dataframe in df2:
    print(dataframe.head())<jupyter_output>PassengerId  Survived  Pclass  \
0            1         0       3   
1            2         1       1   
2            3         1       3   

                                                Name     Sex  Age  SibSp  \
0                            Braund, Mr. <NAME>    male   22      1   
1  Cumings, Mrs. <NAME> (<NAME> Th...  female   38      1   
2                             Heikkinen, <NAME>  female   26      0   

   Parch            Ticket     Fare Cabin Embarked  
0      0         A/5 21171   7.2500   NaN        S  
1      0          PC 17599  71.2833   C85        C  
2      0  STON/O2. 3101282   7.9250   NaN        S  
   PassengerId  Survived  Pclass  \
3            4         1       1   
4            5         0       3   
5            6         0       3   

                                           Name     Sex   Age  SibSp  Parch  \
3  Futrelle, Mrs. <NAME> (<NAME>)  female  35.0      1      0   
4                      Allen, Mr. W[...]<jupyter_text>Using DaskA Dask DataFrame is a large parallel DataFrame composed of many smaller Pandas DataFrames, split along the index. These Pandas DataFrames may live on disk for larger-than-memory computing on a single machine, or on many different machines in a cluster.https://docs.dask.org/en/<jupyter_code>!pip3 install dask
import dask
import dask.dataframe as dd
dfd = dd.read_csv('../Datasets/titanic.csv')
#everything works the same (almost) but now the dataset can be distributed
dfd.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07233834095362508, 'fraction_numerical': 0.01714565643370346, 'mean_word_length': 3.152542372881356, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '523475', 'license': 'MIT', 'n_tokens_mistral': 1738, 'n_tokens_neox': 1635, 'n_words': 768, 'path': 'Data PreProcessing/Data PreProcessing.ipynb', 'repo_name': 'ThierDev/machine_learning_introductions', 'size': '6172'}