<jupyter_start><jupyter_text>Data mungingData munging (sometimes referred to as data wrangling) is the process of transforming and mapping data from one "raw" data form into another format with the intent of making it more appropriate and valuable for a variety of downstream purposes such as analytics.<jupyter_code>%matplotlib inline 
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
from sklearn.datasets import load_boston

# Make plots larger
plt.rcParams['figure.figsize'] = (15, 9)
tips = sns.load_dataset("tips")
tips.head(n=7)
tips.describe()
tips['time'].value_counts()
sns.boxplot(x="time", y="total_bill", data=tips);
sns.boxplot(x=tips["total_bill"])
tips['tip'].hist(bins=33)
sns.distplot(tips['tip'])
dataset = load_boston()
boston =  pd.DataFrame(dataset.data, columns=dataset.feature_names)
boston.head()<jupyter_output><empty_output><jupyter_text>Check missing values in the dataset<jupyter_code>sum(boston['NOX'].isnull()) 
boston.apply(lambda x: sum(x.isnull()),axis=0)<jupyter_output><empty_output><jupyter_text>Check the range of values in the dataset<jupyter_code>boston.describe()<jupyter_output><empty_output><jupyter_text>Pima Indians Diabetes Dataset[Pima Indians Diabetes Dataset](https://archive.ics.uci.edu/ml/datasets/pima+indians+diabetes) is known to have missing values.Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage. ADAP is an adaptive learning routine that generates and executes digital analogs of perceptron-like devices. It is a unique algorithm; see the paper for details.Attribute Information:1. Number of times pregnant 2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test 3. Diastolic blood pressure (mm Hg) 4. Triceps skin fold thickness (mm) 5. 2-Hour serum insulin (mu U/ml) 6. Body mass index (weight in kg/(height in m)^2) 7. Diabetes pedigree function 8. Age (years) 9. Class variable (0 or 1) Relevant Papers:<NAME>., <NAME>., <NAME>., <NAME>., & <NAME>. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care} (pp. 261--265). IEEE Computer Society Press.<jupyter_code>pima = pd.read_csv("http://nikbearbrown.com/YouTube/MachineLearning/DATA/pima-indians-diabetes.csv", sep=',')
pima.head()
pima.describe()
pima.apply(lambda x: sum(x.isnull()),axis=0)
pima.isnull().sum()
pima.info()
pima.isnull().sum()
(pima['serum_insulin'] == 0).sum()
(pima['serum_insulin'] < 50).sum()
pima_bak=pima.copy()<jupyter_output><empty_output><jupyter_text>How to fill missing values?<jupyter_code>pima_bak.isnull().sum()
pima_bak.info()
# drop rows with missing values
pima_bak.dropna(inplace=True)
pima_bak.isnull().sum()
pima_bak.info()
(pima_bak['serum_insulin'] < 50).sum()
pima_bak.loc[pima_bak['serum_insulin'] < 50]
pima_bak['serum_insulin'].loc[pima_bak['serum_insulin'] < 50]
pima_bak['serum_insulin'].mean()
pima_bak.head()
pima_bak.loc[pima_bak['serum_insulin'] < 50, 'serum_insulin'] = pima_bak['serum_insulin'].mean()
(pima_bak['serum_insulin'] < 50).sum()
pima_bak.head()
pima_bak['serum_insulin'].fillna(pima_bak['serum_insulin'].mean(), inplace=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1276405831597739, 'fraction_numerical': 0.0139839333531687, 'mean_word_length': 6.782407407407407, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '256296', 'license': 'MIT', 'n_tokens_mistral': 1238, 'n_tokens_neox': 1135, 'n_words': 312, 'path': 'Week_1/NBB_Data_Munging_Wrangling.ipynb', 'repo_name': 'nikbearbrown/INFO_6210', 'size': '3391'}