<jupyter_start><jupyter_text>Data Boot-Camp Final Project   American Time Use Study (ATUS)   <NAME> (sb5933), <NAME> (sj2280), <NAME> (vkk242) | May 5th, 2017   IntroductionIn this project, our aim is to thoroughly analyse the American Time Use Study (ATUS) data which primarily measures the amount of time Americans spend doing various activities such as personal work, paid work, and other daily duties. Nationally represented estimates of the time spent are processed to draw effective insights which will assist in further comprehending product business cycle, marketing, and in developing successful targeting strategy.With the help of Python and its extensive libraries, we then visualized the processed data to grasp the trends more clearly and consequently obtain valuable insights.  Contents1. Background2. About the Data    + 2.1 | Data Sources    + 2.2 | Python Libraries    + 2.3 | Dataframes3. Data Fetching, Cleaning, and Processing    + 3.1 | Fetching and Slicing Data         + 3.1.1 | User-defined Functions 'Extract_Main' / 'Extract_Sub' for Gender level Activity data        + 3.1.2 | Leisure Activity Data        + 3.1.3 | Age Level Activity Data        + 3.1.4 | Geography Level Activity Data     + 3.2 | Cleaning and Organizing Data4. Visualizing the Data    + 4.1 | Visualizing the time spent on main activities at gender level    + 4.2 | Visualizing trends for main activities through years 2011 - 2015    + 4.3 | Which activity is the primary focus for each gender?    + 4.4 | Does age play any role in the time spent on activities?    + 4.5 | Deep-Diving and visualizing data at sub-activity level for the top 4 main activities    + 4.6 | Leisure Activity Breakdown    + 4.7 | Visualizing time spent on Sports & Leisure activities at a geographic level    + 4.8 | Ad-Hoc Analysis: Time spent on Organizational, Civic, and Religious activities activities at a geographic level5. Conclusion  1 | BackgroundThe American Time Use Study (ATUS) is sponsored by the Bureau of Labor Statistics (BLS) and is conducted by the United States Census Bureau. It's main goal is to record the amount of time spent by Americans on numeorus activities ranging from work to leisure to childcare and house-hold activities. This data is further classified based on gender, age group, employment status, marital status, and geographic location. Participants are usually households which have completed all eight months of the Current Population Survey (CPS). Amongst this pool of participants, households are further categorized based on numerous demographic characteristics and then they are finally selected to participate in the survey. Any particular individual above the age of 15 in the household is called up and questioned regarding their time use. This data is utilized for many purposes by organizations such as Bureau of Economic Analysis (BEA), Bureau of Transportation Statistics (BTS) and the Economic Research Service (ERS). It has also been employed to understand worker productivity, effective promotion and targeting strategies and to deep dive into work-life balance ideology.  2 | About the DataThe data has been obtained from the Bureau of Labor Statistics - ATUS website. As mentioned above, ATUS provides nationally representative estimates of the time spent by Americans. It is the only federal survey providing data on the full range of non-market activities, from childcare to volunteering.The data is usually collected from over 170,000 interviews and we have focused on data spanning from 2011 - 2015 to maintain relevance.Main focus activities are: + Personal Care Activities+ Eating and Drinking+ Household Activities+ Purchasing goods and services+ Caring for and helping household members+ Caring for and helping non-household members+ Working and Work related activities+ Educational Activities+ Organizational, civic, and religious activities+ Leisure and Sports+ Telephone calls, mail, and email       2.1 | Data Sources**Website**                  : Bureau of Labor Statistics**Historical Data Span**     : 2011 – 2015**Data Sources URLs :**+ Activity Level Data        : https://www.bls.gov/tus/a1_all_years.xlsx+ Leisure Activity Data      : https://www.bls.gov/tus/charts/chart9.txt+ Students Activity Data     : https://www.bls.gov/tus/charts/chart6.txt+ Elders Activity Data       : https://www.bls.gov/tus/charts/chart4.txt+ Geography Level Data       : Obtained from ATUS on request - This data has been uploaded in the Dropbox    + Sports Data            : https://www.dropbox.com/s/c5ahh0ffb7tc3yv/ATUS_Geography_Data_Sports.csv?dl=1    + Organizational, Civic, and Religious Data :        https://www.dropbox.com/s/36gnok6gtn3u5gw/ATUS_Geography_Data_Religion.csv?dl=1    **Data Dictionaries**        : https://www.bls.gov/tus/dictionaries.htm**User Guide**               : https://www.bls.gov/tus/atususersguide.pdf 2.2 | Python LibrariesWe employed os and requests libraries for importing the data. We then used the pandas library to manipulate and display selected data. Finally, we worked with matplotlib, plotly, and other graphic libraries for visualization.<jupyter_code># Importing all the required libraries

%matplotlib inline 

import sys
import pandas as pd                 # data manipulation package
import datetime as dt               # date tools, used to note current date  
import matplotlib.pyplot as plt     # graphics package
import matplotlib as mpl            # graphics package
import plotly as pl                 # graphics package
import urllib.request               # To import data from Dropbox

# New Libraries

import os                          # operating system tools (check files)
import requests, io                # internet and input tools  
import zipfile as zf               # zip file tools 
import shutil                      # file management tools 
import numpy as np                 # scientific computing

# Geographical Views (Plotly Authentication)

import plotly.plotly as py
import plotly.graph_objs as go
py.sign_in('Vinee03', '0hNA8NplYEePfVAdDtUa')

# System Details

print('Python version:', sys.version)
print('Pandas version: ', pd.__version__)
print('Today: ', dt.date.today())
# To maintain color uniformity throughout all of the visualizations

colors = {1: "royalblue",
          2: "bisque",
          3: "navy",
          4: "silver",
          5: "darkmagenta",
          6: "pink",
          7: "chocolate",
          8: "orangered",
          9: "lime",
          10:"orange",
          11:"darkorchid",
          12:"black",
          13:"yellow"}<jupyter_output><empty_output><jupyter_text>2.3 | Dataframes+ The above mentioned excel workbook comprises of multiple tabs with each tab having data for one particular year (spanning from 2003 to 2015)+ There are two types of Activities:     + Main Activity (For example: Leisure & Sports, Educational Activities)    + Sub-Activity (For example: Watching TV, Participating in Sports, Attending Class)    + The sub-activity hours add up to the main activity hours+ To check the trends at both main activity and sub-activity level, two dataframes have been created for the entire time span i.e. from 2011 to 2015    + **Main Activity Dataframe** : df_main_final    + **Sub-Activity Dataframe**  : df_sub_final    + **Leisure Activity Dataframe**  : df_leisure_all+ **Consolidated Age Group Activity Dataframe**: df_age_data     + Students Activity Dataframe : df_students    + Elders Activity Dataframe   : df_elderly    + **Geographical Distribution Dataframe**    + Sports Data Dataframe   : df_geo_sports    + Organizational, Civic, and Religious Data Dataframe : df_geo_religion  3 | Data Fetching, Cleaning, and Processing 3.1 | Fetching and Slicing Data  3.1.1 | User-defined Functions 'Extract_Main' / 'Extract_Sub' for Gender level activity data+ Dataframe creation process by employing user-defined functions:    + Fetched the Excel workbook from the ATUS Website    + Read individual tabs within the Excel workbook into separate dataframes for each year    + Filtered out the activities into Main and Sub-Activities    + Merged all of the dataframes for each year based on the primary index (Activity)    + Created two final dataframes for Main and Sub-Activities    + Note:     + 'extract_main' function dynamically creates 5 dataframes for Main activities for each year (2011 to 2015)    + Merge all dataframes into a single dataframe 'df_main_final'    + Similarly, 'extract_sub' function dynamically creates 5 dataframes for sub-activities for each year (2011 to 2015)    + Merge all dataframes into a single dataframe 'df_sub_final'<jupyter_code># Fetching data from ATUS Website 

dls = "https://www.bls.gov/tus/a1_all_years.xlsx"
resp = requests.get(dls)
with open('a1_all_years.xlsx', 'wb') as output:
    c = output.write(resp.content)
    
# Creating a dictionary of dataframes for each year of analysis

df_names = ['df_2011','df_2012','df_2013','df_2014','df_2015']

# Defining a function to extract main activity data for each year ranging from 2011 to 2015

def extract_main(no_of_years):
    i= 0
    year = 2011 
    year_max = year + no_of_years
    while year < year_max: 
        year = str(year)
        df = pd.read_excel(open('a1_all_years.xlsx','rb'), sheetname= year, header= None)
        df_extract = df.loc[[5,11,14,25,37,46,52,58,62,73,85,91],[0,8,9]]    
        df_names[i] = df_extract
        df_names[i].columns = ['Main_Activity', 'AvgHrsMen_'+ year, 'AvgHrsWomen_' + year]
        df_names[i] = df_names[i].set_index(['Main_Activity'])
        year = int(year)
        year=year+1
        i = i+1
                
extract_main(5)

# Merging the year-level dataframes to obtain a consolidated main activity dataframe

df_main_final = pd.concat([df_names[0],df_names[1],df_names[2],df_names[3],df_names[4]], axis = 1)
df_main_final
# Fetching data from ATUS Website

dls = "https://www.bls.gov/tus/a1_all_years.xlsx"
resp = requests.get(dls)
with open('a1_all_years.xlsx', 'wb') as output:
    c = output.write(resp.content)

# Creating a dictionary of dataframes for each year of analysis

df_names = ['df_2011','df_2012','df_2013','df_2014','df_2015']

# Defining a function to extract sub-activity data for each year ranging from 2011 to 2015

def extract_sub(no_of_years):
    i= 0
    year = 2011 
    year_max = year + no_of_years
    while year < year_max: 
        year = str(year)
        df = pd.read_excel(open('a1_all_years.xlsx','rb'), sheetname= year, header= None)
        df_extract = df.loc[[6,7,8,9,10,12,13,15,16,17,18,19,20,21,22,23,24,26,28,32,35,38,42,45,47,48,51,53,54,55,56,57,59,60,61,
                            63,64,72,74,81,84,86,87,90],[0,8,9]]    
        df_names[i] = df_extract
        df_names[i].columns = ['Sub_Activity', 'AvgHrsMen_'+ year, 'AvgHrsWomen_' + year]
        df_names[i] = df_names[i].set_index(['Sub_Activity'])
        year = int(year)
        year=year+1
        i = i+1
                
extract_sub(5)

# Merging the year-level dataframes to obtain a consolidated sub-activity dataframe

df_sub_final = pd.concat([df_names[0],df_names[1],df_names[2],df_names[3],df_names[4]], axis = 1)
df_sub_final<jupyter_output><empty_output><jupyter_text>3.1.2 | Leisure Activity Data+ Fetched data from BLS website to further understand what leisure activities are most popular among Americans+ Cleaned the text file in order to accommodate the missing and multiple delimiters+ Handled the gap with numeric data to ensure data consistency<jupyter_code># Fetching Data for Leisure activity (will be used further down in the analysis)

dls = "https://www.bls.gov/tus/charts/chart9.txt"
resp = requests.get(dls)
with open('chart9.txt', 'wb') as output:
    c = output.write(resp.content)
    
# Extracting the text file data

lines = open("chart9.txt").readlines()
open('newfile.txt', 'w').writelines(lines[2:-4])
f_leisure = open("newfile.txt","r")
data_leisure = f_leisure.read()

# Replace the target string to fix the delimiters

data_leisure = data_leisure.replace("\t", '|').replace("||||", '|').replace("|||", '|').replace("||", '|')

# Write the file out again

with open('newfile.txt', 'w') as file:
    file.write(data_leisure)

f_leisure = open("newfile.txt","r")
data_leisure = f_leisure.read()

# Extracting/Cleaning the data and renaming certain columns

df_leisure_all = pd.read_csv(open("newfile.txt","r"), delimiter="|")
df_leisure_all = df_leisure_all.rename(columns={'Unnamed: 0': 'Activity_Leisure_SubActivity'})
df_leisure_all = df_leisure_all.drop(df_leisure_all.index[[7]])
df_leisure_all.iloc[0, df_leisure_all.columns.get_loc('Minutes')] = 167
df_leisure_all
df_leisure_all = df_leisure_all.set_index(['Activity_Leisure_SubActivity'])

df_leisure_all= df_leisure_all.astype(float)
df_leisure_all<jupyter_output><empty_output><jupyter_text>3.1.3 | Age Level Activity Data+ Fetched data from BLS website to further understand the time indulged in activities based on Age+ Age Brackets considered are:    + Ages 15-49    + Ages 55-64    + Ages 65-74    + Ages 75 + + Cleaned the text file in order to accommodate the missing and multiple delimiters+ Handled the gap with numeric data to ensure data consistency<jupyter_code># Fetching Students Data (will be used further down in the analysis)

dls = "https://www.bls.gov/tus/charts/chart6.txt"
resp = requests.get(dls)
with open('chart6.txt', 'wb') as output:
    c = output.write(resp.content)
    
# Extracting the text file data

lines = open("chart6.txt").readlines()
open('newfile.txt', 'w').writelines(lines[1:-5])
f_s = open("newfile.txt","r")
data_s = f_s.read()

# Replace the target string to fix the delimiters

data_s = data_s.replace("\t", '|').replace("||||", '|').replace("|||", '|').replace("||", '|').replace("activities", 'Working').replace("Educational Working", 'Household/Educational Activities')

# Write the file out again

with open('newfile.txt', 'w') as file:
    file.write(data_s)

f_s = open("newfile.txt","r")
data_s = f_s.read()

# Extracting the data and renaming certain columns

df_students = pd.read_csv(open("newfile.txt","r"), delimiter="|")
df_students = df_students.drop(df_students.index[[3,9,5,6,7,8]])
df_students = df_students.rename(columns={'Unnamed: 0': 'Main_Activity', 'Hours': 'Ages 15-49'})
df_students['Main_Activity'] = df_students['Main_Activity'].str.strip()
df_students = df_students.set_index(['Main_Activity'])
df_students
# Fetching Elders Data (will be used further down in the analysis)

dls = "https://www.bls.gov/tus/charts/chart4.txt"
resp = requests.get(dls)
with open('chart4.txt', 'wb') as output:
    c = output.write(resp.content)
    
# Extracting the text file data

lines = open("chart4.txt").readlines()
open('newfile.txt', 'w').writelines(lines[4:-4])
f_e = open("newfile.txt","r")
data_e = f_e.read()

# Replace the target string to fix the delimiters

data_e = data_e.replace("\t", '|').replace("||||", '|').replace("|||", '|').replace("||", '|').replace("Household activities", 'Household/Educational Activities')

# Write the file out again

with open('newfile.txt', 'w') as file:
    file.write(data_e)

f_e = open("newfile.txt","r")
data_e = f_e.read()

# Extracting the data and renaming certain columns

df_elderly = pd.read_csv(open("newfile.txt","r"), delimiter="|")
df_elderly.drop('Unnamed: 4', axis = 1, inplace = True)
df_elderly = df_elderly.rename(columns={'Unnamed: 0': 'Main_Activity'})
df_elderly['Main_Activity'] = df_elderly['Main_Activity'].str.strip()
df_elderly = df_elderly.set_index(['Main_Activity'])
df_elderly<jupyter_output><empty_output><jupyter_text>3.1.4 | Geography Level Activity Data+ Obtained the Geography level data upon request from ATUS+ Geography data considered are:    + Sports data    + Religion data+ Included the state code in order to easily visualize the data<jupyter_code># Extracting Geographical Distribution of Sports Data from Dropbox

url = "https://www.dropbox.com/s/c5ahh0ffb7tc3yv/ATUS_Geography_Data_Sports.csv?dl=1"
u_s = urllib.request.urlopen(url)
data_s = u_s.read()
u_s.close()

with open("ATUS_Geography_Data_Sports.csv", "wb") as f :
    f.write(data_s)

df_geo_sports = pd.read_csv(open("ATUS_Geography_Data_Sports.csv","r"),delimiter=",")
df_geo_sports
# Extracting Geographical distribution of Organizational, Civic, and Religious Data from Dropbox

url = "https://www.dropbox.com/s/36gnok6gtn3u5gw/ATUS_Geography_Data_Religion.csv?dl=1"
u_r = urllib.request.urlopen(url)
data_r = u_r.read()
u_r.close()

with open("ATUS_Geography_Data_Religion.csv", "wb") as f :
    f.write(data_r)

df_geo_religion = pd.read_csv(open("ATUS_Geography_Data_Religion.csv","r"),delimiter=",")
df_geo_religion<jupyter_output><empty_output><jupyter_text>3.2 | Cleaning and Organizing Data+ Cleaned the Activity data to ensure no outliers are present i.e. negative values, consistent decimal places, substitution of missing values+ Feature engineered additional data fields to observe distinct trends+ Merged multiple datasets in order to obtain a well-rounded view<jupyter_code># Cleaning the data and computing average time spent on activities at gender level

# Main Activity

df_main_final= df_main_final.apply(pd.to_numeric, errors='ignore')
df_main_final["AvgHrsMenMain"]= df_main_final[['AvgHrsMen_2011','AvgHrsMen_2012','AvgHrsMen_2013','AvgHrsMen_2014','AvgHrsMen_2015']].mean(axis = 1)
df_main_final["AvgHrsWomenMain"]= df_main_final[['AvgHrsWomen_2011','AvgHrsWomen_2012','AvgHrsWomen_2013','AvgHrsWomen_2014','AvgHrsWomen_2015']].mean(axis =1)
df_main_final = df_main_final.round(2)
df_main_final
# Cleaning the data and computing average time spent on activities at gender level

# Sub-Activity

df_sub_final=df_sub_final.replace('\(','',regex=True).replace('\)','',regex=True)   
df_sub_final= df_sub_final.apply(pd.to_numeric, errors='ignore')
df_sub_final["AvgHrsMenSub"]=df_sub_final[['AvgHrsMen_2011','AvgHrsMen_2012','AvgHrsMen_2013','AvgHrsMen_2014','AvgHrsMen_2015']].mean(axis =1)
df_sub_final["AvgHrsWomenSub"]=df_sub_final[['AvgHrsWomen_2011','AvgHrsWomen_2012','AvgHrsWomen_2013','AvgHrsWomen_2014','AvgHrsWomen_2015']].mean(axis =1)
df_sub_final = df_sub_final.round(2)
df_sub_final
# Sorting the dataframes as per the hours spent

df_main_final = df_main_final.sort_values('AvgHrsMenMain', ascending=True)

# Re-setting the Index

df_main_final = df_main_final.reset_index()
df_sub_final = df_sub_final.reset_index()

# Trimming the data to format the dataframe

df_main_final['Main_Activity'] = df_main_final['Main_Activity'].str.strip()
df_sub_final['Sub_Activity'] = df_sub_final['Sub_Activity'].str.strip()
# Combining the Students and Elderly dataframes to obtain a consolidated dataframe

df_age_data = pd.concat([df_students,df_elderly], axis = 1)
df_age_data = df_age_data.reset_index()
df_age_data = df_age_data.rename(columns={'index': 'Main_Activity'})
df_age_data = df_age_data.set_index('Main_Activity')
df_age_data<jupyter_output><empty_output><jupyter_text>4 | Visualizing the DataAfter processing the data, we have visualized it at different levels such as gender, age group, main activity/sub-activity, and geographical distribution to obtain a holistic understanding from all perspectives in order to create an effective targeting strategy plan as per the product requirements.   4.1 | Visualizing the time spent on main activities at gender level<jupyter_code># Plotting the Main Activity data as per the Average time

colortemp = [colors[x] for x in list(range(2,4))]
df_main_final = df_main_final.set_index(['Main_Activity'])
ax = df_main_final[['AvgHrsMenMain','AvgHrsWomenMain']].plot(kind='barh', title ="AVERAGE TIME SPENT ON MAIN ACTIVITIES (2011-2015) AT GENDER LEVEL", figsize=(8,8), legend=True, fontsize=10, color = colortemp )
ax.set_xlabel("AVERAGE HOURS FROM 2011 TO 2015", fontsize=12)
ax.set_ylabel("MAIN ACTIVITIES", fontsize=12)
L=plt.legend(loc = 'lower right')
L.get_texts()[0].set_text('Men')
L.get_texts()[1].set_text('Women')
plt.show()<jupyter_output><empty_output><jupyter_text>**From the visualization above, we see that majority of the Americans (both genders) tend to spend most time on the following activities on an average:**+ Personal Care Activities+ Working and Work related Activities+ Educational Activities+ Leisure and SportsAs the above visualization provides an idea regarding the top activities, we wanted to see this at a yearly level to further highlight the significance of these activities. By understanding the trends over the years, we can forecast as to which activities will pose relevance in future as well.  4.2 | Visualizing trends for main activities through years 2011-2015We can see the fluctuations in average hours per activity through each year from 2011 to 2015 for men and women. It is important to note these changes in trends to channelize advertisements/promotions in the right category for maximized results in the future as well (aiming for forecasting predictions).<jupyter_code># Plotting the view for main activities at yearly level

df_men = df_main_final.filter(regex="Men+",axis=1)
df_women = df_main_final.filter(regex="Women+",axis=1)
fig, ax = plt.subplots(nrows=1,ncols=2,figsize=(8,8))

# Creating a loop so that yearly data can be mapped

i = 0
for Activity in list(df_men.index):
    ax[0].plot(df_men.loc[Activity].tolist()[:-1],"ko",markersize=5)
    ax[0].plot(df_men.loc[Activity].tolist()[:-1],linestyle='dashed',label=Activity,color=colors[i+1])
    ax[1].plot(df_women.loc[Activity].tolist()[:-1],"ko",markersize=5)
    ax[1].plot(df_women.loc[Activity].tolist()[:-1],linestyle='dashed',label=Activity,color=colors[i+1])
    if i>=9:
        i=0
    else:
        i+=1
ax[0].legend(loc="upper left", fontsize=10,framealpha=0.2,bbox_to_anchor=(2.5, 1))
fig.subplots_adjust(wspace=0.3, hspace=0)
ax[0].set_title("Year level Main Activity trends (Men)")
ax[1].set_title("Year level Main Activity trends (Women)")

ax[0].set_xticks(list(range(0,5)))
ax[0].set_xticklabels(list(range(2011,2016)))
ax[1].set_xticks(list(range(0,5)))
ax[1].set_xticklabels(list(range(2011,2016)))

xaxis1 = ax[0].get_xaxis()
xaxis2 = ax[1].get_xaxis()

for ticks1,ticks2 in zip(xaxis1.get_ticklabels(),xaxis2.get_ticklabels()):
    ticks1.set_rotation(45)
    ticks1.set_color('k')
    ticks2.set_rotation(45)
    ticks2.set_color('k')

plt.show()<jupyter_output><empty_output><jupyter_text>We observe from the yearly trends that - + Time spent by both the genders has been gradually increasing from 2011 to 2015. This illustrates that a large emphasis is placed by individuals on personal care.+ Trends corresponding to Educational activities show a dip in the year 2014 for both genders but it is seen that it is slowly picking up in the year 2015.+ Overall, non-top 4 activities are consistent among their yearly trends which basically inidicates that higher emphasis on the same may not be extremely beneficial.  4.3 | Which activity is the primary focus for each gender? ** From the below visualization we see the Gender breakdown for some of the key primary activities. **This further provides an idea about which activity is of higher focus for each gender so that gender-localized targeting strategy can be developed.+ We clearly notice that though relatively the amount of time spent is in almost the same bracket for most of the activities, women tend to invest more time in house-hold activities. + Men tend to spend more time indulging in leisure activities+ The derivations above correlates with the normal convention of gender norms as well. Though we are seeing both genders involved in work related activies too, this disparity of activities is still distinct.<jupyter_code># Creating the visualization to distinctly identify important activies for each gender

fig, ax = plt.subplots(nrows=1,ncols=2, figsize=(6,6))
locx = list(range(0,10))
barwidth = 0.35
for loc, Activity in zip(range(len(df_men)),list(df_men.index)):
    bar1, = ax[0].barh(loc,round(df_men.loc[Activity]["AvgHrsMenMain"],2),label=Activity,color = colors[loc+1])
    ax[0].text(bar1.get_width()+1.5,bar1.get_y()+barwidth/2,bar1.get_width(),ha='left', va='bottom')
    bar2, = ax[1].barh(loc,round(df_women.loc[Activity]["AvgHrsWomenMain"],2),label=Activity,color = colors[loc+1])
    ax[1].text(bar2.get_width()*1.05,bar2.get_y()+barwidth/2,bar2.get_width())
ax[0].set_xticks([])
ax[0].set_yticks([])
ax[1].set_xticks([])
ax[1].set_yticks([])
ax[0].spines['top'].set_visible(False)
ax[0].spines['left'].set_visible(False)
ax[0].spines['bottom'].set_visible(False)
ax[0].spines['right'].set_visible(False)
ax[1].spines['top'].set_visible(False)
ax[1].spines['right'].set_visible(False)
ax[1].spines['bottom'].set_visible(False)

ax[0].invert_xaxis()                        #this invert provides the opposite comparison graphs between genders
ax[0].set_title("Average Hours for Men")
ax[1].set_title("Average Hours for Women")
box1 = ax[0].get_position()
box2 = ax[1].get_position()
fig.subplots_adjust(wspace=0.01, hspace=0)
l = ax[1].legend(loc="lower right", fontsize=10,framealpha=0.6, markerscale=5,labelspacing=0.1,borderpad=0.1
                 ,bbox_to_anchor=(2.2, -.1))
plt.show()<jupyter_output><empty_output><jupyter_text>4.4 | Does age play any role in the time spent on activities?** Follwing the understanding from the gender level analysis, we wanted to add another layer of analysis by viewing this data based on the age group demographic.** From the below stacked visualization we see the age-group breakdown for some of the key primary activities. As expected, sleeping takes the cake followed by Leisure and Sports. This further ties to the on-going analysis where it is best to cater to all age groups via TV or Internet or Social Media. Knowing that this platform is highly effective, it is extremely sensible to invest thoroughly in this domain and effectively grow the sales.<jupyter_code># Plotting the stacked bar chart for each age bracket

df_age_data = df_age_data.sort_values('Ages 15-49', ascending=False)
colortemp = [colors[x] for x in list(range(2,6))]
ax = df_age_data[['Ages 15-49','Ages 55-64', 'Ages 65-74', 'Ages 75+']].plot(kind='bar', stacked = True, color = colortemp,title ="AVERAGE TIME SPENT IN ACTIVITIES BY AGE GROUP (2011-2015)", figsize=(8, 8), legend=True, fontsize=10,rot=30)
ax.set_xlabel("MAIN ACTIVITIES", fontsize=12)
ax.set_ylabel("AVERAGE HOURS FROM 2011 TO 2015", fontsize=12)
plt.show()<jupyter_output><empty_output><jupyter_text>4.5 | Deep-Diving and visualizing data at sub-activity level for the top 4 main activitiesFrom all of the above visualizations, we can concretely see that few of the primary activities really stand out. Knowing this, we wanted to further delve into the data to decipher which sub-activities (within the above mentioned top 4 main activities) do Americans perform most. This granular analysis of the data will provide a deeper insight to more locally target the relevant customer base.We created main activity level segments to organize so that we can correlate easily.<jupyter_code># Filtering out the sub-activities of the Top 4 main activities

top_list = ['Sleeping', 'Grooming', 'Health-related self care', 'Personal activities', 'Travel related to personal care', 'Working',
            'Work-related activities', 'Other income-generating activities', 'Job search and interviewing', 'Travel related to work',
            'Attending class', 'Homework and research', 'Travel related to education', 'Socializing, relaxing, and leisure', 
            'Sports, exercise, and recreation', 'Travel related to leisure and sports']

df_sub_final_u = df_sub_final[df_sub_final['Sub_Activity'].isin(top_list)]
df_sub_final_u = df_sub_final_u.reset_index()

# Merging the sub-activity data with main activity to create categories/segements

data = {'Main_Activity': ['Personal care activities', 'Personal care activities', 'Personal care activities','Personal care activities','Personal care activities', 'Working and work-related activities', 'Working and work-related activities','Working and work-related activities', 'Working and work-related activities', 'Working and work-related activities',
'Educational activities', 'Educational activities', 'Educational activities', 'Leisure and sports', 'Leisure and sports', 'Leisure and sports']}  
df_main_act = pd.DataFrame(data)

df_sub_final_q = pd.merge(df_sub_final_u, df_main_act, left_index=True, right_index=True)
df_sub_final_q.drop('index', axis=1, inplace=True)

# Cleaning and sorting the data 

df_sub_final_Personal_Care = df_sub_final_q.groupby(['Main_Activity']).get_group('Personal care activities')
df_sub_final_Personal_Care = df_sub_final_Personal_Care.set_index(['Sub_Activity'])
df_sub_final_Personal_Care = df_sub_final_Personal_Care.sort_values('AvgHrsMenSub', ascending=False)

df_sub_final_Educational = df_sub_final_q.groupby(['Main_Activity']).get_group('Educational activities')
df_sub_final_Educational = df_sub_final_Educational.set_index(['Sub_Activity'])
df_sub_final_Educational = df_sub_final_Educational.sort_values('AvgHrsMenSub', ascending=False)

df_sub_final_Leisure_Sports = df_sub_final_q.groupby(['Main_Activity']).get_group('Leisure and sports')
df_sub_final_Leisure_Sports = df_sub_final_Leisure_Sports.set_index(['Sub_Activity'])
df_sub_final_Leisure_Sports = df_sub_final_Leisure_Sports.sort_values('AvgHrsMenSub', ascending=False)

df_sub_final_Work = df_sub_final_q.groupby(['Main_Activity']).get_group('Working and work-related activities')
df_sub_final_Work = df_sub_final_Work.set_index(['Sub_Activity'])
df_sub_final_Work = df_sub_final_Work.sort_values('AvgHrsMenSub', ascending=False)

# Plotting the graph at sub-activity level

fig, ax = plt.subplots(nrows=2, ncols=2)
colortemp = [colors[x] for x in list(range(4,6))]
df_sub_final_Personal_Care[['AvgHrsMenSub','AvgHrsWomenSub']].plot(kind='bar', ax=ax[0,0], color = colortemp, width=0.5, title ="AVERAGE TIME SPENT ON PERSONAL CARE ACTIVITIES", figsize=(15, 10), legend=True,fontsize=10, rot = 30)
df_sub_final_Work[['AvgHrsMenSub','AvgHrsWomenSub']].plot(kind='bar', ax=ax[0,1], color = colortemp, width=0.5, title ="AVERAGE TIME SPENT ON WORK OR WORK RELATED ACTIVITIES", figsize=(15, 10), legend=True, fontsize=10,sharey=ax[0,0],rot = 30 )
df_sub_final_Educational[['AvgHrsMenSub','AvgHrsWomenSub']].plot(kind='bar', ax=ax[1,0],color = colortemp, width=0.5, title ="AVERAGE TIME SPENT ON EDUCATIONAL ACTIVITIES", figsize=(15, 10), legend=True, fontsize=10,rot = 30 )
df_sub_final_Leisure_Sports[['AvgHrsMenSub','AvgHrsWomenSub']].plot(kind='bar', ax=ax[1,1],color = colortemp, width=0.5, title ="AVERAGE TIME SPENT ON LEISURE & SPORTS ACTIVITIES", figsize=(15, 10), legend=True, fontsize=10,sharey=ax[1,0],rot = 30)

# Setting the Legend and Axis Labels

L1 = ax[0,0].legend(loc = 'upper right')
L1.get_texts()[0].set_text('Men')
L1.get_texts()[1].set_text('Women')

L2 = ax[0,1].legend(loc = 'upper right')
L2.get_texts()[0].set_text('Men')
L2.get_texts()[1].set_text('Women')

L3 = ax[1,0].legend(loc = 'upper right')
L3.get_texts()[0].set_text('Men')
L3.get_texts()[1].set_text('Women')

L4 = ax[1,1].legend(loc = 'upper right')
L4.get_texts()[0].set_text('Men')
L4.get_texts()[1].set_text('Women')

ax[0,0].set_ylabel('Average Hours (2011-2015)', fontsize=12)
ax[0,1].set_ylabel('Average Hours (2011-2015)', fontsize=12)
ax[1,0].set_ylabel('Average Hours (2011-2015)', fontsize=12)
ax[1,1].set_ylabel('Average Hours (2011-2015)', fontsize=12)
ax[0,0].set_xlabel('Sub-Activity', fontsize=12)
ax[0,1].set_xlabel('Sub-Activity', fontsize=12)
ax[1,0].set_xlabel('Sub-Activity', fontsize=12)
ax[1,1].set_xlabel('Sub-Activity', fontsize=12)
fig.tight_layout()                                          # aligns all 4 charts within subplots 
fig.subplots_adjust(wspace=0.05, hspace=0.7)<jupyter_output><empty_output><jupyter_text>** We see that majority of the time is spent on either sleeping or working followed by attending class (target student crowd) or socializing, relaxing, and leisure.** All of these activities provide a profitable platform to advertise for a certain segment of population and develop a sales strategy. + For instance, since Americans spend an ample amount of time sleeping/resting or working, this could be leveraged such that the potential customer base is rightfully tapped based on the product. + Likewise, products catering to students and teens can be branded and publicized in and around academic and college campuses to generate maximum revenue.As we see from the plot above, socializing, relaxing, and leisure is also a primary activity with potential for convenient targeting. We wanted to further look into this category to clearly pin-point the modes by which advertising would be most effective. 4.6 | Leisure Activity Breakdown**From the below pie chart, we see that most of the time is consumed in:**+ **Watching TV** + **Socializing and Communicating**+ **Playing Games; Using Computer for Leisure**+ **Sports, exercise, and recreation**With the above insight into the type of leisure activities, we see that TV and Internet could pose to be very strategic platforms for exploiting a potential audience.<jupyter_code># Creating the exploded pie chart

fig, ax = plt.subplots(figsize=(8, 8))
explode = [0,0.1,0.2,.4,0.3,0.5,0.6]                                 
color = [colors[x] for x in list(range(1,len(df_leisure_all["Minutes"])+1))]
p,t = ax.pie(list(df_leisure_all["Minutes"]), explode=explode,shadow=True, startangle=90, radius=1.3,colors=color)
labels = ['{0} - {1:1.2f}'.format(i,j) for i,j in zip(list(df_leisure_all.index), list(df_leisure_all["Minutes"]))]
box1 = ax.get_position()
ax.set_position([box1.x0, box1.y0, box1.width * 0.9, box1.height])
l = ax.legend(p,labels , loc="upper right", fontsize=12,framealpha=0.2,bbox_to_anchor=(1.8, 1))
l.get_title().set_position((30, 0))

# Equal aspect ratio ensures that pie is drawn as a circle

ax.axis('equal')                                                     
ax.set_title("AVERAGE TIME SPENT BY AMERICANS IN LEISURE ACTIVITY (IN MINUTES)",fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>4.7 | Visualizing time spent on Sports & Leisure activities at a geographic level** From the below geographic distribution, we see the regions within United States of America with maximum interest in sports related activities **+ Northern USA is more inclined towards sports activities with Alaska, Montana, and Wyoming leading the race+ Southern USA is not very active in this particular domain<jupyter_code># Geographic Distribution of Sports and Leisure Activities

for col in df_geo_sports.columns:
    df_geo_sports[col] = df_geo_sports[col].astype(str)
#shows color gradient as hours increases
scl = [[0.0, 'rgb(242,240,247)'],[0.2, 'rgb(218,218,235)'],[0.4, 'rgb(188,189,220)'],\
       [0.6, 'rgb(158,154,200)'],[0.8, 'rgb(117,107,177)'],[1.0, 'rgb(84,39,143)']]

df_geo_sports['text'] = df_geo_sports['State']
layout = dict(
        title = 'Average Time spent on Sports & Leisure (in hours) <br>(Hover for breakdown)',
        geo = dict(
            scope='usa',
            projection=dict( type='albers usa' ),
            showlakes = True,
            lakecolor = 'rgb(255, 255, 255)'),
             )
data = [ dict(
        type='choropleth',
        colorscale = scl,
        autocolorscale = False,
        locations = df_geo_sports['code'],                      # picks 2 digit state code from csv
        z = df_geo_sports['Hours'].astype(float),               # picks activity hrs as float
        locationmode = 'USA-states',
        text = df_geo_sports['text'],
        marker = dict(
            line = dict (
                color = 'rgb(255,255,255)',
                width = 2
            ) ),
        colorbar = dict(
            title = "Time in Hours")                         
        ) ]
fig = dict( data=data, layout=layout )
py.iplot( fig, filename='d3-cloropleth-map' )<jupyter_output><empty_output><jupyter_text>4.8 | Ad-Hoc Analysis: Time spent on Organizational, Civic, and Religious activities at a geographic level** We were curious to see if religion and the corresponding activities are very popular among Americans.**We notice that religion is more strictly adhered to in the south-eastern part of the country. According to this ad-hoc analysis, the type of mentality and mind-set can also be worked into developing lucrative business strategies.<jupyter_code># Geographic Distribution of Religious and Civic Organization Activities

for col in df_geo_religion.columns:
    df_geo_religion[col] = df_geo_religion[col].astype(str)
#shows color gradient as hours increases
scl = [[0.0, 'rgb(243,205,174)'],[0.3, 'rgb(237,184,140)'],[0.6, 'rgb(227,142,72)'],\
            [0.9, 'rgb(72,39,11)'],[1.0, 'rgb(38,20,6)']]

df_geo_religion['text'] = df_geo_religion['State']  
layout = dict(
        title = 'Average time spent on Organizational, Civic & Religious Activities (in hours) <br>(Hover for breakdown)',
        geo = dict(
            scope='usa',
            projection=dict( type='albers usa' ),
            showlakes = True,
            lakecolor = 'rgb(255, 255, 255)'),
             )
data = [ dict(
        type='choropleth',
        colorscale = scl,
        autocolorscale = False,
        locations = df_geo_religion['code'],               # picks 2digit state code from csv
        z = df_geo_religion['Hours'].astype(float),        # picks activity hrs as float
        locationmode = 'USA-states',
        text = df_geo_religion['text'],
        marker = dict(
            line = dict (
                color = 'rgb(255,255,255)',
                width = 2
            ) ),
        colorbar = dict(
            title = "Time in Hours")
        ) ]
fig = dict( data=data, layout=layout )
py.iplot( fig, filename='d3-cloropleth-map' )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11199873029309068, 'fraction_numerical': 0.030684583641942653, 'mean_word_length': 4.763835950602226, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 63, 'https://': 15, 'lorem ipsum': 0, 'www.': 15, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '549401', 'license': 'MIT', 'n_tokens_mistral': 12714, 'n_tokens_neox': 11741, 'n_words': 3624, 'path': 'MBA_S17/Boddu-Jadwani-Kutty-American_Time_Use_Study.ipynb', 'repo_name': 'NYUDataBootcamp/Projects', 'size': '37825'}