<jupyter_start><jupyter_text>Handling Structural ErrorsYoutube linkhttps://youtu.be/0737K5c0JhIStructural errors are those that arise during measurement, data transfer, or other types of "poor housekeeping.", such as:   - Data Type Conversion  - Syntax Errors (Remove white spaces)  - Fix Typos<jupyter_code>import pandas as pd
# Import the dataset

df_titanic = pd.read_csv("https://raw.githubusercontent.com/atulpatelDS/Data_Files/master/Feature_Engineering/Structural_Error/titanic_train.csv")
df_titanic.head()
df_titanic.drop(["Name","Ticket"],inplace=True,axis=1)
df_titanic.head()<jupyter_output><empty_output><jupyter_text>Data Type Conversion<jupyter_code>df_titanic.dtypes
## Lets use some calculation on fare column
## Lets calculate the mean value of Fare
df_titanic.Fare.mean()<jupyter_output><empty_output><jupyter_text>We can see that Fare is showing the data type "object". We need to change the data type of this column.<jupyter_code>## We will replace the dtype "object" to "float"
df_titanic["Fare"] = df_titanic.Fare.str.replace("$","").astype(float)
df_titanic.dtypes
df_titanic.Fare
## Lets use some calculation on fare column
## Lets calculate the mean value of Fare
df_titanic.Fare.mean()<jupyter_output><empty_output><jupyter_text>Syntax Errors  Remove White SpacesExtra white spaces at the beginning or the end of a string should be removed.<jupyter_code>import pandas as pd
df = pd.DataFrame({
    'company_code': ['Abcd','EFGF ',' fhygy','abcd',' '],
    'date_of_sale ': ['12/05/2002','16/02/1999','25/09/1998','12/02/2022','15/09/1997'],
    'sale_amount': [12348.5, 233331.2, 22.5, 2566552.0, 23.0]})
df
print("Original DataFrame:")
print(df)
print("\nIs space is present?")
# Check if the string ends with a white space
#df['company_code_is_title'] = list(map(lambda x: x.endswith(" "), df['company_code']))
# Check if the string start with a white space
#df['company_code_is_title'] = list(map(lambda x: x.startswith(" "), df['company_code']))
# strip : Remove spaces at the beginning and at the end of the string:
df['company_code_is_title'] = list(map(lambda x: x.strip(), df['company_code']))
print(df)
df<jupyter_output><empty_output><jupyter_text>Lets apply on Titanic Dataset<jupyter_code># Check if the string ends with a white space
#df_titanic["Sex_r"] = list(map(lambda x:x.endswith(" "),df_titanic["Sex_New"]))
df_titanic["Sex_r"] = list(map(lambda x:x.endswith(" "),df_titanic["Sex"]))
# Check if the string start with a white space
#df_titanic["Sex_l"] = list(map(lambda x:x.startswith(" "),df_titanic["Sex_New"]))
df_titanic["Sex_l"] = list(map(lambda x:x.startswith(" "),df_titanic["Sex"]))
df_titanic
print(df_titanic[df_titanic["Sex_l"] == True])
print(df_titanic[df_titanic["Sex_r"] == True])
df_titanic.drop(["Sex_r","Sex_l"],inplace=True,axis=1)
df_titanic
df_titanic["Sex_New"] = list(map(lambda x: x.strip(), df_titanic['Sex']))
print(df_titanic)<jupyter_output>PassengerId  Survived  Pclass     Sex   Age  SibSp  Parch   Fare Cabin  \
0              1         0   Third    male  22.0      1      0   7.25   NaN   
1              2         1   First  female  38.0      1      0  71.28   C85   
2              3         1   Third  female  26.0      0      0   7.93   NaN   
3              4         1   First  female  35.0      1      0  53.10  C123   
4              5         0   Third    male  35.0      0      0   8.05   NaN   
..           ...       ...     ...     ...   ...    ...    ...    ...   ...   
886          887         0  Second    male  27.0      0      0  13.00   NaN   
887          888         1   First  female  19.0      0      0  30.00   B42   
888          889         0   Third  female   NaN      1      2  23.45   NaN   
889          890         1   First    male  26.0      0      0  30.00  C148   
890          891         0   Third    male  32.0      0      0   7.75   NaN   

    Embarked Sex_New  
0          S    male  
1    [...]<jupyter_text>Fix typosWhen we collect the raw data we can observe that same type of string values are stored in many different ways.We need to recognize these values in the dataset.<jupyter_code>## Lets check whether each value in the below dataset is male or female.
df_titanic
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
sns.countplot(x = "Sex_New",data = df_titanic)
df_titanic['Sex_New'].value_counts()
sns.set(style="darkgrid")
#titanic = sns.load_dataset("titanic")
total = float(len(df_titanic)) # one person per row 
ax = sns.countplot(x="Sex_New", data=df_titanic)
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x()+p.get_width()/2.,
            height + 3,
            '{:1.0f}'.format(height),
            #'{:1.2f}'.format(height/total),
            ha="center") 
plt.show()<jupyter_output><empty_output><jupyter_text>We can see in graph that there are 4 classes but ideally there should be only 2 clases.It seems like there is typo error. "fem" should be "female" and "mal" should be "male"<jupyter_code>## Lets manually map each value to either "male" or "female"

df_titanic["Sex_New"] = df_titanic["Sex_New"].map({"fem":"female","mal":"male","male":"male","female":"female"})
sns.countplot(x = "Sex_New",data = df_titanic)
df_titanic['Sex_New'].value_counts()
sns.set(style="darkgrid")
#titanic = sns.load_dataset("titanic")
total = float(len(df_titanic)) # one person per row 
ax = sns.countplot(x="Sex_New", data=df_titanic)
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x()+p.get_width()/2.,
            height + 3,
            '{:1.0f}'.format(height),
            #'{:1.2f}'.format(height/total),
            ha="center") 
plt.show()
# https://xspdf.com/resolution/58311140.html<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13188735350708414, 'fraction_numerical': 0.042329893300682174, 'mean_word_length': 3.2106038291605303, 'pattern_counts': {'":': 4, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '455366', 'license': 'MIT', 'n_tokens_mistral': 2218, 'n_tokens_neox': 2062, 'n_words': 520, 'path': 'Data_Cleaning/Handling Structural Error using Python - Data Cleaning Tutorial 3.ipynb', 'repo_name': 'atulpatelDS/Youtube', 'size': '5717'}