<jupyter_start><jupyter_text>Project 1: Covid-19 *TODO: table of contents*  Table of ContentsBackground Knowledge: Spread of Disease1.   The Data Science Life Cycle    a. Formulating a question or problem     b. Acquiring and cleaning data    c. Conducting exploratory data analysis    d. Using prediction and inference to draw conclusions  The Data Science Life Cycle  *TODO:* Update resources. Update formulating a question or problem  Formulating a question or problem It is important to ask questions that will be informative and that will avoid misleading results. There are many different questions we could ask about Covid-19, for example, many researchers use data to predict the outcomes based on intervention techniques such as social distancing. Question: Take some time to formulate questions you have about this pandemic and the data you would need to answer the questions. In addition, add the link of an article you found interesting with a description an why it interested you.       You can find [resources](https://docs.google.com/document/d/1yGSQkqlkroF6Efj3mHvP4sbQXyZM9ddO43YV1FQ75uQ/edit?usp=sharing) here to choose from. Your questions: *here*Data you would need: *here*Article: *link* *TODO:* Update data background.  Acquiring and cleaning data We'll be looking at the COVID-19 Data Repository from Johns Hopkins University. You can find the raw data [here](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series). We've cleaned up the datasets a bit, but we will be investigating the number of cases, new cases, deaths, and new deaths for counties in states accross the US from March 2020 - May 2021.The following table, `covid_statistics`, contains the several statistics collected at the start of each month for every county in the United States. Columns dropped: `UID`, `iso2`, `iso3`, `code3`, `FIPS`, `Country_Region`, `Lat`, `Long_'`, `Combined_Key`<jupyter_code>covid_statistics = Table().read_table("data/covid_timeseries.csv").drop(0, 1, 2, 3, 4, 7, 
                                                                        'Lat', 'Long_', 'Combined_Key')
covid_statistics
#Here, we are relabeling the columns to have more accurate names
covid_statistics = covid_statistics.relabel(make_array('Admin2', 'Province_State', 'month', 'cases', 
                                                      'cases_new', 'deaths', 'deaths_new'),
                                            make_array('County', 'State', 'Date', 'Cases', 
                                                      'New Cases', 'Deaths', 'New Deaths'))
covid_statistics<jupyter_output><empty_output><jupyter_text>Question: It's important to evalute our data source. What do you know about Johns Hopkins University? What motivations do they have for collecting this data? What data is missing?    *Insert answer* Question: We want to learn more about the dataset. First, how many total rows are in this table?<jupyter_code>total_rows = ...<jupyter_output><empty_output><jupyter_text>*Insert answer here* Question: What does each row represent?        *Insert answer here*  Conducting exploratory data analysis  Visualizations help us to understand what the dataset is telling us. Compare the county with the most confirmed cases on April 1st with the next 9 most confirmed cases in a bar chart.   Part 1 Question: First, sort the dataset to show the counties with the highest number of new cases for a given month.<jupyter_code>new_cases_sorted = covid_statistics.sort('...', descending=...)
new_cases_sorted
#KEY
new_cases_sorted = covid_statistics.sort('New Cases', descending=True)
new_cases_sorted<jupyter_output><empty_output><jupyter_text>Question: Now, cut down the table to only have the top twenty from sorted_cases above.<jupyter_code>top_twenty = new_cases_sorted...(np.arange(20))
top_twenty
#KEY
top_twenty = new_cases_sorted.take(np.arange(20))
top_twenty<jupyter_output><empty_output><jupyter_text>Question: Next, create a bar chart to visualize the comparison between the top_ten counties for the number of cases on April 1st.<jupyter_code>top_twenty...("...", "...")
top_twenty.barh("County", "New Cases")<jupyter_output><empty_output><jupyter_text>Question: Let's look at the counties in California. First, return a table that only has the California counties. Then, select the counties from the table you want to compare to each other.<jupyter_code>ca_cases = covid_statistics.where("...", are.equal_to("..."))
ca_cases
#KEY
ca_cases = covid_statistics.where("State", are.equal_to("California"))
ca_cases
select_counties = ["Los Angeles", "Alameda", "Orange", "San Bernandino", "Bakersfield"]

#This will take the counties you choose for the comparison.
my_counties = ca_cases.where("County", are.contained_in(select_counties))
my_counties<jupyter_output><empty_output><jupyter_text>Question: Now make another bar chart using your selected counties and the number of cases on May. First, filter out the data to contain information about May only.    **Hint:** Use the number of the month.<jupyter_code>#Filter table to contain only May data
may_my_counties = my_counties.where('...', are.containing('...'))
may_my_counties
#KEY
may_my_counties = my_counties.where('Date', are.containing("5"))
may_my_counties
# Use this cell to make a bar chart of new cases in May 
...
#KEY
may_my_counties.barh("County", "New Cases")<jupyter_output><empty_output><jupyter_text>Question: What are some possible reasons for the disparities in certain counties? Why do counties appear twice? Hint: Think about the size of the counties.    *Insert answer here.*  Part 2 A disease will spread more when there are more people in a population to spread to. Let's look at the population of the states to compare the percentages based on the number of people. Here is a table with the states and their populations.<jupyter_code>pop_by_state = Table().read_table("data/pop_by_state.csv")
pop_by_state<jupyter_output><empty_output><jupyter_text>Question: First, group the covid statistics to show the number of cases for each state and the sum of the cases.<jupyter_code>#We are grouping all the counties into their states and taking the sum of the cases using this code.
grouped_by_state = covid_statistics.group("State", sum)
grouped_by_state
#Now we will drop the County sum and Combined_Key sum because they 
#do not have numbers to add and we do not need the columns anymore.
grouped_by_state = grouped_by_state.drop(1, 2)
grouped_by_state<jupyter_output><empty_output><jupyter_text>Question: Now that we have it grouped by state, let's first look at the number of cases in June so we can compare it to the percentages we will look at later.<jupyter_code>#Run this cell to see the number of
grouped_by_state.sort("June sum", descending = True).take(np.arange(10)).barh("State", "June sum")<jupyter_output><empty_output><jupyter_text>Question: Now join this table with the pop_by_state table.<jupyter_code>#We are going to join the two tables by providing the column they share which is "State".
with_pop = grouped_by_state.join("State", pop_by_state)
with_pop<jupyter_output><empty_output><jupyter_text>Question: Add a column called "Percentage" that has the number of cases collected in June divided by the population.<jupyter_code>#First, we want to find the columns that would make up an array of the percentages.
june_cases = with_pop.column("6/1/2020 sum")
population = ...
percentage = (.../...)*100
percentage
#KEY
june_cases = with_pop.column("6/1/2020 sum")
population = with_pop.column("Population")
percentage = (june_cases/population)*100
percentage
with_pct = with_pop.with_column("...", ...)
with_pct
#KEY
with_pct = with_pop.with_column("Percentage", percentage)
with_pct<jupyter_output><empty_output><jupyter_text>Question: Like we did in the previous section, sort with_pct and include the top ten states with the most cases on June 1st. Then, create a bar chart to compare the states with the highest percentages of cases.<jupyter_code>top_ten_pct = ...
#KEY
top_ten_pct = with_pct.sort("Percentage", descending = True).take(np.arange(10))
top_ten_pct
#fill in the code to make the bar chart looking at the States and their Percentages.
...
#KEY
top_ten_pct.barh("State", "Percentage")<jupyter_output><empty_output><jupyter_text>Question: What differences do you see from the bar chart of the states when we just saw the number of cases? Give some possible reasons for the differences.    *Insert answer here.*  Using prediction and inference to draw conclusions  Now that we have some experience making these visualizations, let's go back to exponential growth. We know that, without intervention, a disease can behave like a rumor and spread at an alarming rate. From the previous section, we also know that we need to take into account the population of the region when looking at the number of cases. Now we will read in two tables: Covid by State and Population by state in order to look at the percentage of the cases. And the growth of the<jupyter_code>covid_by_state = Table().read_table("data/covid_by_state.csv")
covid_by_state.show(5)
#run this cell to get a line plot!
def plot_states(state_1, state_2):
    covid_by_state.select(0, state_1, state_2).plot(0)
interact(plot_states, 
         state_1=Dropdown(options=covid_by_state.labels[1:]), 
         state_2=Dropdown(options=covid_by_state.labels[1:]));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08260684669359983, 'fraction_numerical': 0.009355730384860728, 'mean_word_length': 4.857409713574097, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '448794', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2807, 'n_tokens_neox': 2673, 'n_words': 1076, 'path': 'Project_2/COVID/Project_1.ipynb', 'repo_name': 'ds-modules/BUDS-SU21-Dev', 'size': '9406'}