<jupyter_start><jupyter_text>Data Wrangling & Munging in Python And, if you want to see how to clean data on Python effectively, then check out my equivalent video via Rstudio! I'll pretty much be following the same steps in my data wrangling process in Rstudio but in PythonNote that I do not include the various joining operations since that is a topic on its own...![Data](Wrangle_Pic.png) Link: https://www.youtube.com/watch?v=iLI_ymRe0w0  Datasets in use:- Abalone dataset: http://archive.ics.uci.edu/ml/datasets/Abalone<jupyter_code># Loading in packages
import pandas as pd
# Loading in the data sets. (I've just loaded the datasets from R...)
abalone = pd.read_csv("abalone.csv")
# First things first, let's see if there are any NA's 
abalone.isnull().sum() # None.
# If there were columns that had null values, you can run the following function to return those row values:
# abalone[abalone[column_name].isnull()]<jupyter_output><empty_output><jupyter_text>One Hot Encode Process<jupyter_code>abalone.head()
# Let's remove that first column
abalone = abalone.iloc[: , 1:]
abalone.head()
pd.get_dummies(abalone['Type']) # Converted the First column, a categorical feature, to one hot encoded features.
abalone_one_hot = pd.concat([pd.get_dummies(abalone['Type']), abalone.iloc[: , 1:]], axis=1)
abalone_one_hot.head()<jupyter_output><empty_output><jupyter_text>The str() & Summary() versions in Python<jupyter_code>abalone_one_hot.info()
abalone_one_hot.describe()
# Let's run through the apply functions really quick. Let's convert all the features to a character type.
'''
Difference between the applymap() and the apply() is that the applymap() is by element (in a df) whereas apply() 
works on row/columns (series and and dataframes) and map() works on element-wise on a series.
'''
# applymap is a function that runs a function through all the features that exist in the dataset.
aoh_string = abalone_one_hot.applymap(str) # str is a function; this can be changed i.e lamda x: x**2
#abalone_one_hot['F'].apply(int)
aoh_string.info()
aoh_string.head()<jupyter_output><empty_output><jupyter_text>Filtering<jupyter_code># Let's get the observations whose longestShell > 0.3 and Diameters is < 0.3. How would we go about doing this?
abalone[(abalone['LongestShell'] > 0.3) & (abalone['Diameter'] < 0.3)]
# How but if we only want Type M? ez.
#abalone[(abalone["Type"] == "M")]
#abalone['Type'].str.contains('MF') # Etc.. you can pinpoint what strings you want in your dataset (similar to grep)
#abalone[(abalone["Type"] == "M") & (abalone["Type"] == "F")] # And operator
#abalone[(abalone["Type"] == "M") | (abalone["Type"] == "F")] # Or operator
abalone[abalone["Type"] != "M"] # No M's?
# We can also create a list of objects to see if the dataframe row values exist in the list!
list_to_compare = ["Male", "M"]
mask = abalone['Type'].isin(list_to_compare) # a boolean value
print(mask.head())
abalone[mask]
# How but we group by gender and average the rest of the features to get some summary statistics?
# Grouping function....
abalone.groupby('Type').sum()
# You can also group by multiple categorical variables if you have multiple categorical features...
# Let's just select a few columns...
#abalone[["Type", "LongestShell", "Rings"]]
# We can also flip the order...
abalone[["Rings","Type", "LongestShell"]]
# Based on our results earlier, we can also sort by Rings...
abalone[["Type", "LongestShell", "Rings"]].sort_values('Rings', ascending = False) # Change to false if we want other direction.
# Let's create a column...
abalone.head()
abalone["Rings_T_ShellWeight"] = abalone['Rings'] * abalone['ShellWeight']
abalone
# Maybe you do not want any duplicates in your dataset. So Let's run the drop duplicates function
abalone.drop_duplicates # Nothing because there are no duplicate ROWS.
# Creating dummy dataframe to illustrate
columns = ['A','B', 'C']
df_ = pd.DataFrame(index=range(1,10), columns=columns)
df_ = df_.fillna(0) # with 0s rather than NaNs
df_
df_.drop_duplicates() # only maintain one row that represents the duplicated rows.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12478632478632479, 'fraction_numerical': 0.00463980463980464, 'mean_word_length': 5.470774091627172, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '408638', 'license': 'MIT', 'n_tokens_mistral': 1316, 'n_tokens_neox': 1269, 'n_words': 476, 'path': 'Data Wrangling/Wrangle_Munging.ipynb', 'repo_name': 'SpencerPao/Data_Science', 'size': '4095'}