<jupyter_start><jupyter_text>Data Preparation Workflow  Set Up the Jupyter Notebook for AnalysisNote: We have our new package called swat - SAS Scripting Wrapper for Analytics Transfer - available on GitHub via pip install<jupyter_code># Import necessary packages and modules
import swat
%matplotlib inline

# Set the connection by specifying the hostname, port, username, and password
conn = swat.CAS(hostname, port, username, password)

# Get the hmeq csv file from SAS support documentation and lift into memory
castbl = conn.read_csv('http://support.sas.com/documentation/onlinedoc/viya/exampledatasets/hmeq.csv', casout = 'hmeq')
castbl.replace = True<jupyter_output><empty_output><jupyter_text>Examine the first few rows<jupyter_code># Assign the variable name df to the new CASTable object
df = conn.CASTable('hmeq')

# Perform the head method to return the first 5 rows
df.head()<jupyter_output><empty_output><jupyter_text>Create new features<jupyter_code># How much of their mortgage have they paid off?
df['MORTPAID'] = df['VALUE'] - df['MORTDUE']
df.head()
# What percent of the time does this happen?
df.query('MORTPAID < 0')['MORTPAID'].count()/len(df)<jupyter_output><empty_output><jupyter_text>Examine numeric variable distribution<jupyter_code># Use the pandas/matplotlib method for plotting a histogram of all numeric variables
df.hist(figsize = (15, 10));<jupyter_output><empty_output><jupyter_text>Examine summary statistics<jupyter_code># Use the pandas describe method, then switch rows and columns
summary = df.describe(include = 'all').transpose()
summary<jupyter_output><empty_output><jupyter_text>Is there an issue with missing data?<jupyter_code># Create percent missing column for plotting
summary['pctmiss'] = (len(df) - summary['count'])/len(df)

# Make a bar graph using pandas/matplotlib functionality
summary.query('pctmiss > 0')['pctmiss'].plot('bar', title = 'Pct Missing Values', figsize = (10, 6), color = 'c');<jupyter_output><empty_output><jupyter_text>The variables need to be imputed for missing values<jupyter_code># This is using the CAS action impute - really nice method for imputing all variables at once
## Impute the median for numeric, most common for categorical
df.impute(
    methodContinuous = 'MEDIAN',
    methodNominal    = 'MODE',
    inputs           = list(summary.index[1:]), # exclude target column
    copyAllVars      = True,
    casOut           = castbl
)<jupyter_output><empty_output><jupyter_text>Create partition indicator so models won't overfit<jupyter_code># Load the sampling actionset
conn.loadactionset('sampling')

# Do a simple random sample with a 70/30 split
df.srs(samppct = 30, partind = True, seed = 1, output = dict(casout = castbl, copyvars = 'all'))

# What percentage is in each split?
castbl['_PartInd_'].groupby('_PartInd_').count()/len(castbl)<jupyter_output>NOTE: Added action set 'sampling'.
NOTE: Using SEED=1 for sampling.<jupyter_text>Make sure the dataset looks good<jupyter_code>castbl.head()<jupyter_output><empty_output><jupyter_text>Promote the table to public memory<jupyter_code># This allows me to share my data across sessions and with other colleagues
conn.promote(name = castbl, targetlib = 'public', target = 'hmeq_prepped')<jupyter_output><empty_output><jupyter_text>Ensure that everything worked as intended<jupyter_code># Verify that hmeq_prepped has made it into the public caslib
conn.tableinfo(caslib = 'public')['TableInfo'].query('Name == "HMEQ_PREPPED"')
# End the session
conn.endsession()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10359467874327767, 'fraction_numerical': 0.00537786583639966, 'mean_word_length': 6.271604938271605, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '55548', 'license': 'Apache-2.0', 'n_tokens_mistral': 1163, 'n_tokens_neox': 1115, 'n_words': 338, 'path': 'webinars/PythonDataPrepWorkflow.ipynb', 'repo_name': 'c964309085/sas-viya-programming', 'size': '3533'}