<jupyter_start><jupyter_text>Exploring a StatsCan DatasetAuthor: <NAME>.Created on: October 22 2010 AbstractIn this notebook we will explore a Stats Can Dataset, perform basic data exploration and visualization. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1310011101tables  Importing the dataset from StatsCan We first begin by downloading the data and importing Python libraries. This particular dataset contains information on "Number and rates of new cases of primary cancer, by cancer type, age group and sex". Reference: Statistics Canada.  Table  13-10-0111-01   Number and rates of new cases of primary cancer, by cancer type, age group and sexDOI:   https://doi.org/10.25318/1310011101-eng<jupyter_code># Importing libraries and modules 
import pandas as pd
import zipfile
import matplotlib.pyplot as plt 
from matplotlib.pyplot import figure
# Downloading the data 
!wget https://bit.ly/2JbKmPs

# Define a function to extract the data
def read_data_compute_df(data_file):
    # Read zipped file
    zf = zipfile.ZipFile(data_file)
    # Parse csv content without unzipping
    df = pd.read_csv(zf.open(str(13100111)+'.csv'),low_memory=False)
    
    return df
# Manual download file_name = "./13100111-eng.zip" 
file_name = "2JbKmPs"
cancer_data = read_data_compute_df(file_name)<jupyter_output><empty_output><jupyter_text>We look at the first five entries of the StatsCan dataset. We notice lots of internal use type fields. We will clean them up.<jupyter_code># Take a look at the data 
cancer_data.head()
# Clean up dataset
clean_cancer_data = cancer_data.drop(columns = 
                                    ["DGUID", "UOM_ID","UOM", "SCALAR_ID","VECTOR","COORDINATE","STATUS",\
                                    "TERMINATED","DECIMALS","SYMBOL"])<jupyter_output><empty_output><jupyter_text>In our case, and after cleaning up the data, we have dates which range from 1992 to 2007, all provinces in Canada, multiple age groups, male, female and bith sexes, multiple primary types od cancer, as well as whether the case is new, cancer incidence, as well as low and high 95% confidence interval cancer incidence.<jupyter_code>clean_cancer_data.head()<jupyter_output><empty_output><jupyter_text>Now that we have cleaned our dataset, we want to get all unique values for each category.<jupyter_code># Get data subsets
dates = clean_cancer_data.iloc[:,0].unique()
geog = clean_cancer_data.iloc[:,1].unique()
age_group = clean_cancer_data.iloc[:,2].unique()
sex = clean_cancer_data.iloc[:,3].unique()
cancer_type = clean_cancer_data.iloc[:,4].unique()
characteristics = clean_cancer_data.iloc[:,5].unique()<jupyter_output><empty_output><jupyter_text>For this example, I am interested in the number of new cancer cases, for Lung and Bronchus cancer, for multiple age groups, both sexes, all of Canada. Notice that you can use the subsets of the data to explore multiple sub tables.<jupyter_code># Suppose I am interested in the following subset
year = dates[15]
sp_sex = sex[0]
sp_age = age_group[7]
geoLoc = geog[0]
cancer = 'Lung and bronchus [C34.0-C34.9]'
charac = 'Number of new cancer cases'
# Extract entries in our dataset that contain the information I want
# ( clean_cancer_data["Age group"] == sp_age) &
subset_1 = clean_cancer_data[ (clean_cancer_data["REF_DATE"] == year ) & \
                             (clean_cancer_data["Sex"] == sp_sex ) & \
                            (clean_cancer_data["GEO"]==geoLoc) & \
                             (clean_cancer_data["Characteristics"]==charac) &\
                  (clean_cancer_data["Primary types of cancer (ICD-O-3)"] == cancer)
                 ] 
#subset_1<jupyter_output><empty_output><jupyter_text>I now create a barplot for the subset of the data that I am interested in.<jupyter_code># Plot our table 
# Specifying the figure size
figure(figsize=(12,4))
# Bar chart of data subset
plt.bar(subset_1["Age group"] , subset_1["VALUE"], label=cancer)
# Add a title
plt.title(str(charac) + ", " + str(year) + ", " + str(geoLoc) + ", " + str(sp_sex)\
        +", " + str(cancer)  , fontsize=15 )
# Add x y labels
plt.xlabel("Number of New Cancer cases ", fontsize=10)
plt.ylabel("Age group", fontsize=10)
# Rotate x axis label
plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>RemarksWe see that there is an increase in number of new cancer cases, lung and bronchus cancer, as individuals age.  GeneralizingIn the next section we will define a function that takes as input subcategories of the data and returns a barplot.<jupyter_code>def bar_plot_subcategories(year, sp_sex, geoLoc, charac,cancer):
    subset_1 = clean_cancer_data[ (clean_cancer_data["REF_DATE"] == year ) & \
                             (clean_cancer_data["Sex"] == sp_sex ) & \
                            (clean_cancer_data["GEO"]==geoLoc) & \
                             (clean_cancer_data["Characteristics"]==charac) &\
                  (clean_cancer_data["Primary types of cancer (ICD-O-3)"] == cancer)
                 ] 
    # Plot our table 
    # Specifying the figure size
    figure(figsize=(12,4))
    # Bar chart of data subset
    plt.bar(subset_1["Age group"] , subset_1["VALUE"], label=cancer)
    # Add a title
    plt.title(str(charac) + ", " + str(year) + ", " + str(geoLoc) + ", " + str(sp_sex)\
            +", " + str(cancer)  , fontsize=15 )
    # Add x y labels
    plt.xlabel("Number of New Cancer cases ", fontsize=10)
    plt.ylabel("Age group", fontsize=10)
    # Rotate x axis label
    plt.xticks(rotation=90);
# Suppose I am interested in the following subset
year = dates[15]
sp_sex = sex[0]
sp_age = age_group[7]
geoLoc = geog[0]
cancer = 'Lung and bronchus [C34.0-C34.9]'
charac = 'Number of new cancer cases'<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10741643454038997, 'fraction_numerical': 0.023502785515320333, 'mean_word_length': 3.8236775818639797, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '618902', 'license': 'MIT', 'n_tokens_mistral': 1919, 'n_tokens_neox': 1776, 'n_words': 550, 'path': 'DataWithJupyter.ipynb', 'repo_name': 'lfunderburk/JupyterDataScience', 'size': '5747'}