<jupyter_start><jupyter_text>`ES 157` Notebook 3: Modeling COVID cases This week we will use everything we've been learning in class about systems and the Fourier transform so far, in order to make a simplistic model for the daily COVID cases. As a reminder from the first notebook, this is the data that we will work with.![covid_mass](https://i.imgur.com/bI1umxq.png)At the end of this notebook you will1. have used `pandas` and `python` to clean up data,2. have used the Fourier transform to analyze data, and3. have created a simple model for the daily new cases.As we always, let us import some needed libraries.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import scipy.signal as signal
import pandas as pd<jupyter_output><empty_output><jupyter_text>Loading up COVID data 💉 Along with this week's notebook you should have downloaded some COVID data. In the introductory video, Demba mentioned the COVID tracker of the [John Hopkins University](https://coronavirus.jhu.edu/data/new-cases-50-states). That tracker, like most of the COVID dashboards, actually use data from [this](https://github.com/CSSEGISandData/COVID-19) GitHub repo. The data we provide were downloaded from the same repo a few weeks ago. Let's load the data from our favorite state.<jupyter_code># load the csv

# check the data<jupyter_output><empty_output><jupyter_text>We can see that the data have lots of columns that we probably don't need for our simple analysis. In fact, we see that the first `11` columns have extraneous information. Let's extract the data only for a specific state, and calculate the number of daily cases.We see that the data is actually split into the numbers reported by each county. When trying to find the total number of daily cases, we have to aggregate the reported numbers by all the counties.Another thing we need to consider is that each day actually reports the _total_ number of cases thus far. So, again, we have to acount for that in order to find the number of daily cases.<jupyter_code># extract the list of columns

# restrict to your state's data

# find the new daily cases<jupyter_output><empty_output><jupyter_text>Plotting the data and the FFT Now that we've cleaned up the data a little bit, let us try and plot them to make sure we get a sensible image.**Optional**: create a sensible time axis using `pandas`/`matplotlib` (or, you might want to look into `plotly`; it has a lot of ways to manage plots with dates.)<jupyter_code># plot the daily new cases<jupyter_output><empty_output><jupyter_text>Okay, this looks good! It seems to be aligned with the image from the John Hopkins tracker. Let's examine now the frequency content of the daily new cases by using the _Fourier transform_.<jupyter_code># plot the FFT of the daily new cases

# create frequency axis

# plot the fft<jupyter_output><empty_output><jupyter_text>Analyzing the Fourier transform We see a very strong DC component in the Fourier transform of the signal. As we've talked about before, this is expected and encodes the mean of the signal. As a first analysis step, let's remove it and see if the image becomes clearer.<jupyter_code># remove the dc component from the FFT

# plot the fft<jupyter_output><empty_output><jupyter_text>We see a little bit clearer two symmetric peaks. What do they correspond to? Remember, our frequency axis is measured in _days_; therefore this implies that if we have a frequency $x$, then this means that the event happens every $\frac{1}{x}$ days (the period is the inverse of the frequency). Do the peaks make sense?Now, let's try the following; reconstruct the signal using the inverse FFT, but zero out everything except the DC and the interval $[0.12, 0.17]$ (and its symmetric one.)<jupyter_code># find the indices of interest

# zero out everything except the parts of interest

# reconstruct the signal

# plot the reconstructed signal<jupyter_output><empty_output><jupyter_text>It turns out that our intuition was correct; the DC component and the frequency content we identified are able to give us a pretty good approximation of the original curve. Obviously, there's some information lost since we zero'ed out most of the Fourier transform; however, we get a fairly sensible image with only a few coefficients.This idea is the basis of _compression_; we identify the most important coefficients in a base and we keep _only_ those coefficients.  Trying to model COVID Let's try and make a simple model for the COVID data. We will base our model on two observations- first, we saw that the number of daily cases exhibit some sort of periodicity, and- second, we see a Gaussian-shaped component.The simplest periodic signal we know is a _sine_; therefore, we will try to model the daily new cases by using the product of a sine and a Gaussian distribution.First, let us create a sine wave of the appropriate frequency.<jupyter_code># create a time variable

# choose an appropriate frequency/period

# create the sine

# plot the sine<jupyter_output><empty_output><jupyter_text>Hmm, but our sine takes negative values. To get a clearer picture on our graph, we could make sure we only keep the parts that are above zero.<jupyter_code># keep only the positive part of the sine

# plot the sine<jupyter_output><empty_output><jupyter_text>We're slowly making progress. 🤓 Let's move on to the Gaussian. As a reminder, the equation of a Gaussian distribution is given by$f(x) = \frac{1}{\sigma \sqrt{2 \pi}} e^{-\frac{1}{2}\left(\frac{x - \mu}{\sigma}\right)^2}$.Around what day does the Gaussian seem to be centered? And what would you say is it's standard deviation?Also, note that we need to acount for the height of the distribution. Notice that when $x = \mu$, the Gaussian simplifies to$f(\mu) = \frac{1}{\sigma \sqrt{2 \pi}}$.So, we should look at the corresponding desired height at that peak, and add an adjustment term to our distribution.<jupyter_code># make a guess for the mean and the std

# find the adjustment term

# create the Gaussian

# plot the Gaussian<jupyter_output><empty_output><jupyter_text>Okay, so now we have our Gaussian and we have our sine wave. Let's multiply them and plot the product in the same plot as the COVID data and see how we did.<jupyter_code># multiply the signals

# plot the product signal

# plot the covid data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05925925925925926, 'fraction_numerical': 0.004412923561859732, 'mean_word_length': 4.892293407613742, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '663742', 'license': 'MIT', 'n_tokens_mistral': 1737, 'n_tokens_neox': 1656, 'n_words': 941, 'path': 'notebooks/es157_notebook3.ipynb', 'repo_name': 'manosth/es157-notebooks', 'size': '6345'}