<jupyter_start><jupyter_text>Diabetes Prediction on Electronic Medical Records (EMR) Data Data Preparation and Exploratory Data Analysis  Imports<jupyter_code>%matplotlib inline
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from matplotlib import cm
import seaborn as sns
import pandas as pd
import pylab as pl
import random<jupyter_output><empty_output><jupyter_text>**Note:** SageMaker SDK is not used here.  Load & Prep Data<jupyter_code>df = pd.read_csv('./DATA/emr.csv')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>sns.countplot(df['class'],label='Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Box Plot and HistogramsVisualize distribution of the Input Variables<jupyter_code>df.plot(kind='box', 
        subplots=True, 
        layout=(2,2), 
        sharex=False, 
        sharey=False, 
        figsize=(9,9), 
        title='Input Variables')
plt.show()
df.hist(bins=30, figsize=(9,9))
pl.suptitle("Input Variables")
plt.show()<jupyter_output><empty_output><jupyter_text>Descriptive Statistics<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Scale, Split & Create Train/Test Datasets<jupyter_code>X = df[['bmi', 'diastolic_bp_change', 'systolic_bp_change', 'respiratory_rate']]
y = df['class']<jupyter_output><empty_output><jupyter_text>Encode the classes into numerical values using Sklearn's LabelEncoder<jupyter_code>label_encoder = LabelEncoder()
label_encoder.fit(['Yes', 'No'])
y = label_encoder.transform(y)
y<jupyter_output><empty_output><jupyter_text>Split X, y into Train and Test Sets<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=123)
X_train.shape
X_test.shape<jupyter_output><empty_output><jupyter_text>Scale Feature Columns<jupyter_code>scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Combine Scaled X & y into Train and Test DataFrames<jupyter_code>X_train = pd.DataFrame(X_train, columns=['bmi', 'diastolic_bp_change', 'systolic_bp_change', 'respiratory_rate'])
y_train = pd.DataFrame(y_train, columns=['class'])
train_df = pd.concat([y_train, X_train], axis=1)
train_df.head()
X_test = pd.DataFrame(X_test, columns=['bmi', 'diastolic_bp_change', 'systolic_bp_change', 'respiratory_rate'])
y_test = pd.DataFrame(y_test, columns=['class'])
test_df = pd.concat([y_test, X_test], axis=1)
test_df.head()<jupyter_output><empty_output><jupyter_text>Create a DataFrame for Batch Inference without the Class column<jupyter_code>batch_test_df = test_df.drop(['class'], axis=1)
batch_test_df.head()<jupyter_output><empty_output><jupyter_text>Write Train & Test Sets to Local Directory<jupyter_code>train_df.to_csv('./DATA/train/train.csv', header=False, index=False)
test_df.to_csv('./DATA/test/test.csv', header=False, index=False)
batch_test_df.to_csv('./DATA/batch_test/batch_test.csv', header=False, index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12710219922380336, 'fraction_numerical': 0.004527813712807244, 'mean_word_length': 8.63551401869159, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '164391', 'license': 'Apache-2.0', 'n_tokens_mistral': 1148, 'n_tokens_neox': 1109, 'n_words': 191, 'path': '0. Data Prep.ipynb', 'repo_name': 'arunprsh/SageMaker-Workshop-NLP', 'size': '3092'}