<jupyter_start><jupyter_text>Clean your Data With NumPy and Pandas<jupyter_code>#Import Library
import numpy as np
import pandas as pd
# Read Data from your datasets folder
data = pd.read_csv('../datasets/clean-data/Book.csv')
# Create DataFrame from the CSV file
data.head() # To see First Five entery with column names<jupyter_output><empty_output><jupyter_text>Data exploration and Basic Hygiene<jupyter_code># Print Missing value
print(data.isnull().sum())
# Drop columns with any missing values 
c=data.dropna(axis='columns')
c.head()
# Missing value is not Found as it is removed
print(c.isnull().sum())
# Drop the rows where at least one element is missing.
data.dropna()<jupyter_output><empty_output><jupyter_text>Link For Data operation using pandas https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html  We can remove columns by passing columns names directly to columns parameters  data.drop(columns=drop_col, inplace=True) data.head()<jupyter_code># drop unnessary columns 
# inplace and axis make change to object
# axis =1 drop column wise axis =0 for row wise
drop_col = ['Edition Statement',
           'Corporate Author',
           'Corporate Contributors',
           'Former owner',
           'Engraver',
           'Contributors',
           'Issuance type',
           'Shelfmarks']

data.drop(drop_col, inplace = True, axis = 1)
data.head()
# Missing value is not Found as it is removed
print(data.isnull().sum())<jupyter_output>Identifier                 0
Place of Publication       0
Date of Publication      181
Publisher               4195
Title                      0
Author                  1778
Flickr URL                 0
dtype: int64<jupyter_text>Change Index of Dataframe<jupyter_code># Check unique values in  Columns
data['Identifier'].is_unique
data['Place of Publication'].is_unique
data['Date of Publication'].is_unique
data['Publisher'].is_unique
data['Title'].is_unique
data['Author'].is_unique
data['Flickr URL'].is_unique
# Replace the existing index with column name with unique value using set_index :
# To make change directly to object
data.set_index('Identifier', inplace=True)
data.head()<jupyter_output><empty_output><jupyter_text>Replace the existing index with column name with unique value using set_index : data = data.set_index('Identifier') data.head()<jupyter_code># Access each record with loc[]
data.loc[480]
data.iloc[480]
data.iloc[5]<jupyter_output><empty_output><jupyter_text>We can see from above with    loc[] => help in accessing data based on index       whereas    iloc[] => help in accessing data based on position    index [480] data is same as in position [5]     Clean Fields in the Data<jupyter_code># All data types are in object dtype
data.get_dtype_counts()
data.loc[1905:, 'Date of Publication'].head(10)<jupyter_output><empty_output><jupyter_text>As stated above some book have multiple date of publication. Book can have only one date of publication. We need to enforce a numeric value for date of publication to do calculations as:Remove square brackets, like present: 1879 [1878]Convert date ranges to their “start date”, wherever present: 1860-63; 1839, 38-54Remove the dates like: [1897?] replace them with NumPy’s NaN : Convert the string NaN to NumPy’s NaN value<jupyter_code>ex = data['Date of Publication'].str.extract(r'^(\d{4})', expand=False)
ex.head()
# Convert to numerical values
data['Date of Publication'] = pd.to_numeric(ex)
data['Date of Publication'].dtypes
data['Date of Publication'].isnull().sum() / len(data)<jupyter_output><empty_output><jupyter_text>Use str Methods with NumPy to clean Columns To clean Place of Publication we use str method with NumPy<jupyter_code>data['Place of Publication'].head()
data.loc[data['Place of Publication']== 'London']
data.loc[data['Place of Publication']== 'Oxford']<jupyter_output><empty_output><jupyter_text>We see some row with Place of Publication as 'London' and 'Oxford' have unnecessary information. Clean this column in one go use str.contains() to get a boolean mask.<jupyter_code>london = data['Place of Publication'].str.contains('London')
london.head()
oxford = data['Place of Publication'].str.contains('Oxford')
oxford.head()
# Combine all with NumPy
data['Place of Publication'] = np.where(london, 'London',
                                      np.where(oxford, 'Oxford',
                                               data['Place of Publication'].str.replace('-', ' ')))
data.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10457371225577265, 'fraction_numerical': 0.014209591474245116, 'mean_word_length': 4.363095238095238, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '632115', 'license': 'MIT', 'n_tokens_mistral': 1396, 'n_tokens_neox': 1295, 'n_words': 443, 'path': 'SciPy/Clean your Data With NumPy and Pandas_1.ipynb', 'repo_name': 'byionline/Machine-Learning', 'size': '4504'}