<jupyter_start><jupyter_text>Practice with Jupyter NotebookI'm reworking assignment 2 to practice commiting to GitHub from a local file.<jupyter_code>import pandas as pd
import numpy as np
# # fetch data with bash command
# !curl https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data-original

# # save data file locally
# !wget https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data-original

# !curl worked on Windows, but !wget didn't. *reminder* ask someone if !wget will work on Windows or if there's a different command
# Load data
# This UCI Machine Learning dataset has 2 links for the same data, but both datasets have different issues
mpg_data_url_original = 'https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data-original'
mpg_data_url_revised = 'https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data'

df_original = pd.read_csv(mpg_data_url_original, header=None, sep='\s+', names=['mpg', 'cylinders', 'displacement', 'horsepower',
                                                             'weight', 'acceleration', 'model year', 'origin', 'car name'])
df_revised = pd.read_csv(mpg_data_url_revised, header=None, sep='\s+',
                        names=['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'model year',
                              'origin', 'car name'])
print('Original Dataset: \n', df_original.head())
print('\n\nRevised Dataset: \n', df_revised.head())
# Check for missing data in original dataset
print(df_original.isnull().sum())
print(df_original.dtypes)<jupyter_output>mpg             8
cylinders       0
displacement    0
horsepower      6
weight          0
acceleration    0
model year      0
origin          0
car name        0
dtype: int64
mpg             float64
cylinders       float64
displacement    float64
horsepower      float64
weight          float64
acceleration    float64
model year      float64
origin          float64
car name         object
dtype: object<jupyter_text>In the original dataset we have missing data in the mpg and horsepower column, but all of the data types are correct<jupyter_code># Check for missing data in revised dataset
print(df_revised.isnull().sum())
print(df_revised.dtypes)<jupyter_output>mpg             0
cylinders       0
displacement    0
horsepower      0
weight          0
acceleration    0
model year      0
origin          0
car name        0
dtype: int64
mpg             float64
cylinders         int64
displacement    float64
horsepower       object
weight          float64
acceleration    float64
model year        int64
origin            int64
car name         object
dtype: object<jupyter_text>In the revised dataset we don't show any missing data (but we know it's there), and the horsepower column shows type object instead of float<jupyter_code># Convert '?' values to NaN
df_original_nan = df_original.replace('?', np.NaN)
df_revised_nan = df_revised.replace('?', np.NaN)

print('Original Data: \n', df_original_nan.isnull().sum())
print('Revised Data: \n', df_revised_nan.isnull().sum())
# Fill forward to replace NaN values
df_original_no_nan = df_original_nan.fillna(method='ffill')
df_revised_no_nan = df_revised_nan.fillna(method='ffill')

print('Original Data: \n', df_original_no_nan.isnull().sum())
print('Revised Data: \n', df_revised_no_nan.isnull().sum())
# Convert horsepower data in revised dataset from object to float
df_revised_no_nan['horsepower'] = df_revised_no_nan['horsepower'].astype(float)
df_revised_no_nan.dtypes
# Use one hot encoding to convert categorical data to boolean
df_original_cleaned = pd.get_dummies(df_original_no_nan, columns=['car name'])
df_revised_cleaned = pd.get_dummies(df_revised_no_nan, columns=['car name'])
# Set display options to see all columns
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

# Inspect cleaned original dataset
df_original_cleaned.head()
# Inspect cleaned revised dataset
df_revised_cleaned.head()
df_revised_cleaned.shape
df_original_cleaned.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09795819154107924, 'fraction_numerical': 0.014827418570734078, 'mean_word_length': 3.7848837209302326, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '214027', 'license': 'MIT', 'n_tokens_mistral': 1352, 'n_tokens_neox': 1284, 'n_words': 359, 'path': 'module2-loadingdata/Loading_Data_Practice.ipynb', 'repo_name': 'JLDaniel77/DS-Unit-1-Sprint-1-Dealing-With-Data', 'size': '4114'}