<jupyter_start><jupyter_text>Dog Breeds - Data Science project  Step 2 :  Scrubbing DataIn this step, we will Cleaning, formatting and filtering the data.In addition dealing the missing data with API.<jupyter_code>import pandas as pd
import numpy as np
import requests
import json
import re
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>Open CSV file that created on part 1.<jupyter_code>df = pd.read_csv("dog_breeds_data.csv",index_col = 0)
df.head()
# Check if there are missing values in the DataFrame
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Function that takes only the numbers from a string, and then returns the average between the numbers. Example:input: 10 to 12 yearsoutput: 11<jupyter_code>def get_average_from_str(string):
    numbers = re.findall('[0-9]+', string)
    if len(numbers) > 1:
        return ((int(numbers[0]) + int(numbers[1])) / 2)
    elif len(numbers) == 1:
        return int(numbers[0])
    else:
        return np.nan
# Checking function
get_average_from_str('50 to 60 pounds')
# Adding avg. columns for weight, height  and life span columns with NaN values.
df.insert(4,'Weight Avg.',np.nan)
df.insert(3,'Height Avg.',np.nan)
df.insert(7,'Life Span Avg.',np.nan)
df<jupyter_output><empty_output><jupyter_text>API connection function, to get the missing values.<jupyter_code>## API to search missing values by breed name and data type
def get_data_from_API(dtype,breed_name):
    # API Request
    API_TOKEN = 'b<PASSWORD>'
    headers = { 'x-api-key': API_TOKEN }
    queryParams = { 'q': breed_name }

    response = requests.get('https://api.thedogapi.com/v1/breeds/search',params = queryParams)
    
    # return specific missing data
    if dtype == 'weight' or dtype == 'height':
        try:
            json_data = json.loads(response.text)[0]
        except:
            print('Not Found!')
        # not found

        try:
            return json_data[dtype]['imperial']
            # '22 - 40'
        except:
            return np.nan
        
    else:
        try:
            json_data = json.loads(response.text)[0]
            return json_data[dtype]
            # '10 - 12 years'
            
        except:
            return np.nan
# Fill arr of breed names
def fill_arr_namesbreed(dframe,array):
    for breed in dframe:
        array.append(breed)<jupyter_output><empty_output><jupyter_text>Fill in the missing values by using the get_data_from_API function.<jupyter_code># Call to API get data and update the NaN values in DataFrame 
def update_dataframe_values(breed_arr,dtype,dframe):
    for breed in breed_arr:
        
        try:
            if dtype == 'Weight':
                value = get_data_from_API('weight',breed)
                
            elif dtype == 'Height':
                value = get_data_from_API('height',breed)
                
            elif dtype == 'Life Span':
                value = get_data_from_API('life_span',breed)
                
        except:
            continue
            
        if value:
            dframe.loc[dframe['breed name'] == breed, dtype] = value
# fill null values with API

breed_weight_null = df[df['Weight'].isnull() == True]['breed name'];
breed_height_null = df[df['Height'].isnull() == True]['breed name'];
breed_Life_Span_null = df[df['Life Span'].isnull() == True]['breed name'];

# build arr of name breed with missing values
arr_weight = []
arr_height = []
arr_lifespan = []

# fill arr of name breed with missing values
fill_arr_namesbreed(breed_weight_null,arr_weight)
fill_arr_namesbreed(breed_height_null,arr_height)
fill_arr_namesbreed(breed_Life_Span_null,arr_lifespan)

# update dataframe with new data
update_dataframe_values(arr_weight,'Weight',df)
update_dataframe_values(arr_height,'Height',df)
update_dataframe_values(arr_lifespan,'Life Span',df)<jupyter_output>Not Found!
Not Found!
Not Found!<jupyter_text>check if there are still missing values on the DataFrame.<jupyter_code>df.isnull().sum()<jupyter_output><empty_output><jupyter_text>After we filled in all the missing data values, we fill in the Avg. columns we created above by get_average_from_str function.<jupyter_code>df['Weight Avg.'] = df['Weight'].apply(lambda row: get_average_from_str(str(row)))
df['Height Avg.'] = df['Height'].apply(lambda row: get_average_from_str(str(row)))
df['Life Span Avg.'] = df['Life Span'].apply(lambda row: get_average_from_str(str(row)))
df
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Handling the rest missing data, by fill in the average values.<jupyter_code># Handling missing data
df['Tendency To Bark Or Howl'].fillna(3,inplace = True)
df['Prey Drive'].fillna(3,inplace = True)
df['Drooling Potential'].fillna(3,inplace = True)
df['Weight Avg.'].fillna(df['Weight Avg.'].mean(),inplace = True)
df['Height Avg.'].fillna(df['Height Avg.'].mean(),inplace = True)

df
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Export data to CSV to use it in the next step.<jupyter_code>df.to_csv("dog_breed_data_handling_missing_values.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11572525093485535, 'fraction_numerical': 0.007675654398740406, 'mean_word_length': 3.826210826210826, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '651302', 'license': 'Unlicense', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1652, 'n_words': 413, 'path': 'Step 2 - Scrubbing Data.ipynb', 'repo_name': 'yargisp/Dog-Breeds-Data-Science-Project', 'size': '5106'}