<jupyter_start><jupyter_text>Sampling code for the example TEDS-A data sets A component of a final project by <NAME> Instructor: <NAME> Date: 4/20/20  Purpose of this sampling codeSince TEDS-A data tends to be very large, this code exists to show how the example data files stored on the github repository were truncated, and also to provide the ability to test the main pipeline with your own cutstom truncated TEDS-A data. The original 2015, 2016, and 2017 TEDS-A datasets were acquired from https://www.datafiles.samhsa.gov/study-series/treatment-episode-data-set-admissions-teds-nid13518. They were then imported and a simple random sample of 5% of the observations was selected from each. These subsampled datasets were then exported to csv (and pickle) files for use in the main code of this project.<jupyter_code># Import necessary packages
import numpy as np
import pandas as pd
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
# Assign the file paths
TEDSA1 = "C:/Users/mbrei/Desktop/TEDSA_2015.csv"
TEDSA2 = "C:/Users/mbrei/Desktop/TEDSA_2016.csv"
TEDSA3 = "C:/Users/mbrei/Desktop/TEDSA_2017.csv"
# Load the data in chunks and append
df1 = pd.read_csv(TEDSA1, chunksize = 100)
chunk_list = [] 
for chunk in df1:  
    chunk_list.append(chunk)  
df1_concat = pd.concat(chunk_list)

df2 = pd.read_csv(TEDSA2, chunksize = 100)
chunk_list = [] 
for chunk in df2:  
    chunk_list.append(chunk)  
df2_concat = pd.concat(chunk_list)

df3 = pd.read_csv(TEDSA3, chunksize = 100)
chunk_list = [] 
for chunk in df3:  
    chunk_list.append(chunk)  
df3_concat = pd.concat(chunk_list)
# Select a simple random sample of the data
df1s = df1_concat.sample(frac=0.05, replace=False, random_state=5)
df2s = df2_concat.sample(frac=0.05, replace=False, random_state=5)
df3s = df3_concat.sample(frac=0.05, replace=False, random_state=5)
# Get the number of observations of the original data
len(df1_concat.index)
len(df2_concat.index)
len(df3_concat.index)

# Get the number of observations of the sampled data
len(df1s.index)
len(df2s.index)
len(df3s.index)
# Preview
df1s.head(5)
df2s.head(5)
df3s.head(5)
# Drop the 'unnamed' columns
df1s.drop('Unnamed: 0', axis=1, inplace=True)
df2s.drop('Unnamed: 0', axis=1, inplace=True)
df3s.drop('Unnamed: 0', axis=1, inplace=True)
# Export the files as csv for the main code
df1s.to_csv('‪TEDSA_2015s.csv', index=False)
df2s.to_csv('‪TEDSA_2016s.csv', index=False)
df3s.to_csv('‪TEDSA_2017s.csv', index=False)
# Export the files as pickle for other purposes
df1s.to_pickle('TEDSA_2015spk')
df2s.to_pickle('‪TEDSA_2016spk')
df3s.to_pickle('TEDSA_2017spk')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10231516056758776, 'fraction_numerical': 0.04779686333084392, 'mean_word_length': 5.424460431654676, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '611056', 'license': 'FTL', 'n_tokens_mistral': 1054, 'n_tokens_neox': 974, 'n_words': 277, 'path': 'notebooks/ADS-Final-Project-Breitzig-Sampling.ipynb', 'repo_name': 'mbreitzig/ADS_2020_Breitzig', 'size': '2696'}