<jupyter_start><jupyter_text>Reading the CSV file At first Save country.csv in a directory and read it<jupyter_code>df = pd.read_csv("country.csv")
df.head()<jupyter_output><empty_output><jupyter_text>Read it again this time with right header and check the 5 rows of the data frame<jupyter_code>df = pd.read_csv("country.csv", skiprows = 2) #header=2
df.head()
df.tail()<jupyter_output><empty_output><jupyter_text>Drop unwanted columns Drop the columnws 1 and CountryCode<jupyter_code>df.drop(['1','1.1', 'CountryCode'], axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Change the columns' labels Change the columns' labels to Name, pop_growth, pop, Area<jupyter_code>df.columns = ['Name', 'pop_growth', 'pop', 'Area']
df.columns
df.head()<jupyter_output><empty_output><jupyter_text>Change the index Change the index to the name of country<jupyter_code>df.set_index('Name', inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Duplicates Show the duplicated rows by using "duplicated()""<jupyter_code>duplicated_rows=df.duplicated()
duplicates = df[duplicated_rows]
duplicates
df.drop_duplicates(inplace=True)
df<jupyter_output><empty_output><jupyter_text>Check for duplicates again<jupyter_code>df[df.duplicated()]<jupyter_output><empty_output><jupyter_text>Missing Values Check if there is any NaN value by using "info()" or "isnull().sum()"<jupyter_code>df.info()
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Change NaN values by 0 , the value before, the value after and the mean of the column but do not replace df by the new dataframe<jupyter_code>df.replace(np.NaN, 0).head(20)

df.info()
df.fillna(method='ffill').head(20)
df.fillna(method='bfill').head(20)
df.fillna({'pop':df['pop'].mean(), 'pop_growth':df['pop_growth'].mean(), 'Area':df['Area'].mean()}).head(20)
from sklearn.impute import SimpleImputer
imp=SimpleImputer(missing_values=np.nan, strategy='mean')
imp = imp.fit(df[['pop_growth', 'pop', 'Area']])
transformed_nan = imp.transform(df[['pop_growth', 'pop', 'Area']])
transformed_nan
new_dataset=pd.DataFrame(transformed_nan, columns=['pop_growth', 'pop', 'Area'])
new_dataset.head()
#new_dataset['Code'] = df['Code']
#new_dataset.head(20)
#?????????<jupyter_output><empty_output><jupyter_text>Drop the NaN values and replace the new data frame by that<jupyter_code>df.isnull().sum()
df.dropna(inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Use info( to see if there is any more NaN value)<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 12 entries, Brazil to World
Data columns (total 3 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   pop_growth  12 non-null     float64
 1   pop         12 non-null     float64
 2   Area        12 non-null     float64
dtypes: float64(3)
memory usage: 384.0+ bytes<jupyter_text>dtypes, describe() Check the type of columns<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Use describe to see some statistics of the data frame<jupyter_code>df.describe().T<jupyter_output><empty_output><jupyter_text>Outliers Use boxplot to see if there is any outlier<jupyter_code>df.boxplot()<jupyter_output><empty_output><jupyter_text>Replace the outliers with the NaN values and then drop them<jupyter_code>df.columns
for label in ['pop_growth', 'pop', 'Area']:
    q3, q1 =np.percentile(df[label],[75,25])
    iqr = q3 - q1
    lower = q1 - 1.5*iqr
    upper = q3 + 1.5*iqr
    if lower<0:
        lower = df[label].min()
    df[label] = df[label].mask((df[label]<lower) | (df[label]>upper), np.nan)
df.head(20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14886642993717564, 'fraction_numerical': 0.016935263589183285, 'mean_word_length': 6.368209255533199, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '363016', 'license': 'Unlicense', 'n_tokens_mistral': 1386, 'n_tokens_neox': 1316, 'n_words': 286, 'path': 'data-science/data_processing_assignment.ipynb', 'repo_name': 'kate-morovat/kate-morovat.github.io', 'size': '3661'}