<jupyter_start><jupyter_text>Contents1&nbsp;&nbsp;Setup1.1&nbsp;&nbsp;Retrieve latest data1.2&nbsp;&nbsp;Process and add up all provinces/states into one row per country1.3&nbsp;&nbsp;List of all affected countries2&nbsp;&nbsp;Current situation plots2.1&nbsp;&nbsp;Interactive plot2.2&nbsp;&nbsp;Stats over calendar date2.3&nbsp;&nbsp;Stats scaled to population2.4&nbsp;&nbsp;Stats over days since start of local outbreak2.5&nbsp;&nbsp;Case fatality rate per country2.6&nbsp;&nbsp;Stats by continent2.7&nbsp;&nbsp;Smoothed growth factor2.8&nbsp;&nbsp;New confirmed/Active case ratio2.9&nbsp;&nbsp;Stats for some other countries, using linear scale2.10&nbsp;&nbsp;Pie charts2.11&nbsp;&nbsp;Country highlight3&nbsp;&nbsp;Forecast plots3.1&nbsp;&nbsp;Fit a logistic curve and extrapolate number of future cases3.2&nbsp;&nbsp;Modelling fatality probability per day of being infected3.3&nbsp;&nbsp;Generate history of active cases; how many people have been sick for how long?3.4&nbsp;&nbsp;Simulate future development based on history and probability models3.5&nbsp;&nbsp;Mitigation effectiveness  COVID-19 data visualizationThis notebook generates a series of visualizations from the raw data tables, showing some aspects of the ongoing Covid-19 epidemic that I couldn't find in other reports, or in many cases, not for all countries I was interested in. Data from Johns Hopkins University.An earlier version of the plot of cases per country since the start of the local outbreak was shared on Reddit [here](https://www.reddit.com/r/dataisbeautiful/comments/ff9jn4/oc_number_of_cases_per_country_counting_from_the/).   Another plot, of the average growth factor per country, was posted [here](https://www.reddit.com/r/dataisbeautiful/comments/fliec2/oc_covid19_growth_factor_over_time_in_various/).Note that the links in the Table of Contents above don't work in the Github preview, but they do on [nbviewer](https://nbviewer.jupyter.org/github/JeroenKools/covid19/tree/master/).Want to see these graphs for some other countries, or modify the values for the simulations? The easiest and quickest way to run the notebook and make modifications yourself is to open this notebook in [Colab](https://colab.research.google.com/github/JeroenKools/covid19/blob/master/COVID-19.ipynb) or [Binder](https://hub-binder.mybinder.ovh/user/jeroenkools-covid19-wt257qfb/notebooks/COVID-19.ipynb).  Setup Import code and parameters from [`covid19_util.py`](covid19_util.py) and [`covid_processing.py`](covid19_processing.py).Do some additional setup in case the notebook is being run on Google Colab.<jupyter_code>%load_ext autoreload
%autoreload 2
import sys
is_colab = 'google.colab' in sys.modules
if is_colab:
    if not "covid19" in sys.path:
        try:
            !git clone https://github.com/JeroenKools/covid19.git
        except e: 
            pass
    try:
        import geonamescache
    except ModuleNotFoundError:
        !pip install geonamescache
        import geonamescache
    sys.path.append("covid19")
    %config InlineBackend.print_figure_kwargs = {'bbox_inches':None}
    matplotlib.rcParams['figure.dpi'] = 75
    
    
from covid19_util import *
from covid19_processing import *<jupyter_output><empty_output><jupyter_text>Retrieve latest data<jupyter_code>data = Covid19Processing()<jupyter_output><empty_output><jupyter_text>Process and add up all provinces/states into one row per country This also adds in a few data points for China from before the start of the Johns Hopkins data, from [Wikipedia](https://en.wikipedia.org/wiki/Timeline_of_the_2019%E2%80%9320_coronavirus_outbreak_in_December_2019_%E2%80%93_January_2020).Then, display summaries of the confirmed cases by country and by continent.<jupyter_code>data.process(rows=20, debug=False)<jupyter_output><empty_output><jupyter_text>List of all affected countries<jupyter_code>data.list_countries()<jupyter_output><empty_output><jupyter_text>Current situation plots  Interactive plotWhen running locally, or on a server like [Google Colab](https://colab.research.google.com/github/JeroenKools/covid19/blob/master/COVID-19.ipynb) or MyBinder, this interactive plot allows easy customization of the selected countries, metrics and more. Unfortunately this doesn't work in the Github notebook viewer.<jupyter_code>import warnings
import ipywidgets as widgets
warnings.filterwarnings('ignore') 

data.set_default_countries([
    "World", "Brazil", "California, US", "Japan", "Mongolia", "United States", 
    "India", "Italy", "Russia", "Netherlands"])
interact(data.plot_interactive, 
         x_metric=["calendar_date", "day_number"], 
         y_metric=["confirmed", "deaths", "active", 
                   "new confirmed", "new deaths", 
                   "confirmed/population", "active/population", "deaths/population", 
                   "new confirmed/population", "new deaths/population", 
                   "recent confirmed", "recent deaths", 
                   "recent confirmed/population", "recent deaths/population",
                   "deaths/confirmed", "recent deaths/recent confirmed"],
         smoothing_days=widgets.IntSlider(min=0, max=31, step=1, value=7), use_log_scale=True)
print()<jupyter_output><empty_output><jupyter_text>Stats over calendar date<jupyter_code>for y_metric in ["confirmed", "deaths", "active", "new confirmed", "new deaths"]:
    data.plot("calendar_date", y_metric)<jupyter_output><empty_output><jupyter_text>Stats scaled to population<jupyter_code>for y_metric in ["confirmed", "deaths", "active", "new confirmed"]:
    data.plot("calendar_date", y_metric+"/population", sigma=7)<jupyter_output><empty_output><jupyter_text>Stats over days since start of local outbreak<jupyter_code>data.plot("day_number", "confirmed", min_cases=200)
data.plot("day_number", "deaths", min_cases=50)
data.plot("day_number", "active", min_cases=200)<jupyter_output><empty_output><jupyter_text>Case fatality rate per country The first graph below measures the cumulative case fatality rate, that is, the percentage of all confirmed cases that died. The high initial numbers in countries like Italy and the Netherlands are strongly influenced by the inavailability of tests at the time.The second graph measures the fatality rate on a rolling 30-day window. Due to the delay between disease onset and potential death, this will show a peak around the end of a wave in a country, when new cases are going down but deaths are still high. The countries selected for this graph are the ones that have had specific new virus variants appear.<jupyter_code>data.plot("calendar_date", "deaths/confirmed", min_cases=150, use_log_scale=False)
data.plot("calendar_date", "recent deaths/recent confirmed", 
          countries_to_plot=["United States", "Brazil", "United Kingdom", "South Africa", "World"], 
          min_cases=150, use_log_scale=False)<jupyter_output><empty_output><jupyter_text>Stats by continent<jupyter_code>continents = ["Asia", "Europe", "Africa", "North America", "South America", "Oceania"]

for y_metric in ["confirmed", "deaths", "active", "new confirmed"]:
    data.plot("calendar_date", y_metric, continents, "tab20", fixed_country_colors=False)<jupyter_output><empty_output><jupyter_text>Smoothed growth factorThe growth factor is the multiplier for the number of **new** cases per day. For example, if there are 100 new cases on day *n* and 150 new cases on day *n+1*, the growth factor is 1.5.* A growth factor larger than 1 means the outbreak is accelerating; it is growing exponentially.* A growth factor of 1 means the outbreak is growing linearly.* A growth factor of less than 1 means the outbreak is still growing, but slowing down.* A growth factor of 0 means there are no new cases.In this plot, the growth factor is averaged over 50 days to smooth out some of the shorter-scale variations.<jupyter_code>data.plot("calendar_date", "growth_factor", min_cases=500, sigma=50, 
          use_log_scale=False)<jupyter_output><empty_output><jupyter_text>New confirmed/Active case ratio<jupyter_code>data.plot("calendar_date", "new confirmed/active", min_cases=100, sigma=50, use_log_scale=False)<jupyter_output><empty_output><jupyter_text>Stats for some other countries, using linear scaleThis shows how on a linear Y-axis, it becomes hard to examine the trend in all but the top few countries.<jupyter_code>data.set_default_countries(["Australia", "Belgium", "Finland", "Malaysia", "India", "Mexico",
                            "Israel", "Russia", "Singapore", "Spain", "Vietnam"])


for y_metric in ["confirmed", "deaths", "active"]:
    data.plot("calendar_date", y_metric, use_log_scale=False)<jupyter_output><empty_output><jupyter_text>Pie chartsEverybody loves pie charts.<jupyter_code>for mode in ["country", "continent"]:
    data.plot_pie(["confirmed cases", "deaths", "active cases"], mode)<jupyter_output><empty_output><jupyter_text>Country highlight<jupyter_code>data.country_highlight("United States")<jupyter_output><empty_output><jupyter_text>Forecast plotsFirst, model the probability distribution of dying after a given number of days.We can consider a case recovered if they survive for a long enough time.  Fit a logistic curve and extrapolate number of future casesThis tries to fit a [logistic](https://en.wikipedia.org/wiki/Logistic_function) or 'S-shape' curve to the data so far. While it is reasonable to expect that the total number of cases in any epidemic will eventually roughly have such a shape, the crucial parameters are difficult to predict. For example, changes in spread rate due to policy changes, or the possibility of a second wave of infections from another country are not captured at all.For some countries the results look plausible enough, while for some others it can give extremely unlikely results, like predicting a number of cases that's many times the country's population.**Note**: This model is ill-suited for a trend that changes from logistic growth to linear growth, or one that has multiple logistic stages. It is forced to level off to completely flat even when that doesn't really align with the data we have.<jupyter_code>data.curve_fit("United States", days=100)
data.curve_fit("Germany", days=100)
data.curve_fit("World", days=200)<jupyter_output><empty_output><jupyter_text>Modelling fatality probability per day of being infected[Source](https://www.mdpi.com/2077-0383/9/2/538). This models the *hospitalization to death* distribution, because confirmed cases in the JHU dataset are not backdated to the onset of symptoms.<jupyter_code>death_chance_per_day(cfr=0.04, s=1.75, mu=0.5, sigma=10, length=30, do_plot=True)<jupyter_output><empty_output><jupyter_text>Generate history of active cases; how many people have been sick for how long?Data on **how long** people have been sick is not available, so it is reconstructed using the observed distribution of deaths by case duration.<jupyter_code>data.simulate_country_history(country="United States", history_length=7, show_result=True);<jupyter_output><empty_output><jupyter_text>Simulate future development based on history and probability modelsThis shows 3 simulations with the only difference being the effectiveness of lockdowns and other mitigation efforts.1. Effective lockdown: Assumes normal conditions at the start, followed by a quick and very strict lockdown for as long as it takes. This is the best case, but in my opinion somewhat unlikely.2. Ineffective lockdown: A late and halfhearted attempt at a lockdown is done, but the virus keeps spreading at a half speed. This would be pretty bad; most of the population would be infected before a vaccine is likely to be available.3.  Prolonged battle: Alternating periods of strict lockdowns with more relaxed periods softens the blow to the economy, and stalls the number of infections until a vaccine can hopefully be developed, or until eventually, herd immunity kicks in.Please note that this is *very* speculative, and highly dependent on:* The value chosen for the mitigation factor. Together with R0, this represents the reduction in spread based on the effectives of quarantines, lockdowns and other countermeasures, and determines the effective reproductive rate Reff . For example, a mitigation factor of 0.7 means the disease spreads 70% as fast as it would without any countermeasures. If R0 is 2, the mitigation factor is 0.8, and 98% of the population has not yet been infected,  Reff = 2 \* 0.8 \* 0.98 = 1.568,  meaning that effectively, every 1000 infected people will spread the disease to an average of 1568 uninfected people. The input for mitigation factor is a list that gets interpolated to the same length as the number of days to simulate. For example, simulating 5 days with a mitigation trend of `[1.0, 0.8]` will use growth rates of `[1.0, 0.95, 0.90, 0.85, 0.80]`.* The value chosen for case fatality rate (cfr); evidence so far suggests a range from 0.01 to 0.06Other limitations of the simulation:* Doesn't consider incubation time * Doesn't factor in international travel and potential recontamination/reseeding for second and later waves.* Doesn't consider effect on death rate when the number of active cases overwhelms capacity (**TODO**)<jupyter_code>S = 0.20              # mitigation effectiveness under severe lockdown
M = 0.30              # mitigation effectiveness under moderate lockdown
W = 0.50              # mitigation effectiveness under weak lockdown
N = 1.00              # mitigation effectiveness under normal conditions

# Values inspired by the mitigation effectiveness analysis below
I = 0.31   # Italy, Iran
C = 0.22   # China
K = 0.175  # South Korea
US = 0.37  # US

cfr = 0.025 # https://arxiv.org/abs/2003.09320; adjusted for recently lower deaths
r0 = 3.1    # https://arxiv.org/abs/2003.09320

scenarios = {
    "More effective mitigation (Best case)": [US,S],
    "No mitigation (Worst case)":            [N,N],
    "Continue current mitigation ":          [US,US],
}

for scenario in scenarios:
    simulation = data.plot_simulation(country="United States", days=365, 
                                  mitigation_trend=scenarios[scenario], 
                                  cfr=cfr, r0=r0, scenario_name=scenario)<jupyter_output><empty_output><jupyter_text>Mitigation effectivenessSimulation of developments after the start of lockdowns and other countermeasures in some countries that were affected early.By comparing the actual course of events with those of simulations with different mitigation effectiveness, we can approximate how effective the real countermeasures were.This primarily highlights the importance of an early response, and shows how effectiveluy countries' early lockdowns were. It becoms less relevant as time goes on, and waves and lockdowns come and go.<jupyter_code>cfr = 0.04
r0 = 3.1

country_start_days = {
    "China": 9,
    "South Korea": 33,
    "Italy": 37,
    "Iran": 13,
    "Netherlands": 18,
    "United States": 60
}

for country in country_start_days:
    s = country_start_days[country]
    actual, _ = data.simulate_country(country, days=0, cfr=cfr, r0=r0, from_day=-1)
    for i, f in enumerate([1.0, 0.5, 0.34, 0.31, 0.22, 0.175]):
        r_eff = r0 * f        
        sim, today = data.simulate_country(country, days=100, cfr=cfr, r0=r0, from_day=s, 
                                           mitigation_start=1, mitigation_end=f)
        plt.plot(sim.confirmed, label=f"simulated with {1-f:.1%} mitigation, trending to "+
                 f"r_eff={r_eff:.2f}", linestyle="-", lw=2, alpha=0.7)
        
    plt.plot(actual.confirmed, ":", label=f"{country}, actual", c="k")
    plt.scatter(sim.index[s], sim.confirmed[s], c="k", s=30)
    plt.grid()
    plt.legend()
    set_y_axis_format(actual.max().max(), True)
    plt.title(f"{country} confirmed cases, countermeasures starting on {sim.index[s].date()}", fontsize=16)
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09112090680100755, 'fraction_numerical': 0.023173803526448364, 'mean_word_length': 4.890578635014837, 'pattern_counts': {'":': 10, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 12, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '643105', 'license': 'MIT', 'n_tokens_mistral': 4881, 'n_tokens_neox': 4453, 'n_words': 1648, 'path': 'COVID-19.ipynb', 'repo_name': 'JeroenKools/covid19', 'size': '15880'}