<jupyter_start><jupyter_text>Notebook IntentionsThe purpose of this notebook is to report on the concentration of expenditure among the population over time. This notebook closely follows the methodology described in [STATISTICAL BRIEF 533:Concentration of Healthcare Expenditures and Selected Characteristics of High Spenders, U.S. Civilian Noninstitutionalized Population, 2018](https://www.meps.ahrq.gov/data_files/publications/st533/stat533.shtml), yet expands the research to the past decade. StructureFor each year we identify the total spending thresholds that intertect the population at various percentiles. For this notebook we will be evaluating all percentiles divisible by 10 as well as the 75th, 95th and 99th percentile. Once these thresholds have been identified we can calculate the percentage of annual spending and total annual spending attributed to members of each percentile.Using these values we can generate a concentration curve. The cumulative percentage of the population is represented along the X-axis, and the total expenditures is represented on the Y-axis. At the point where the x-axis is equal to 0.7 and the y-axis is equal to 214 this indicates the the bottom 70% of the population has a total of 214 billion dollars in expenditures. The label at each point represents the percentage of the sub-population's expenditures out of the full population's expenditures. The label at the coordinates (0.7, 214) of 10.8% indicates that the bottom 70% of the population accounts for 10.8% of overall spending. Conversely the top 30% accounts for 89.2% of overall spending. Rather than generate a concentration curve for each year we only display the curve for 2018. To allow for comparisons across years we display the curves as a bar graph with each percentile over 0.5 given it own bar. This reduces noise while only removing ~4% of total expenditures. We generate a graph that demonstrates the percentage of annual spending and a graph that demonstrates total annual spending attributed to members of each percentile. FindingsExpenditure concentration is highly skewed to the top 20, 5 and 1st percentiles for all years, accounting for 80%, 50% and 20% of all expenditures respectively. These proportions have remained relatively stable from 2008-2018 with no years or percentiles being statistically differentiable than the others in their groups. While the proportions have remained stable the total expenditures has increased consistently over the decade. With percentiles 0.5 through 0.9 increasing enough to be statistically significant.<jupyter_code>import os
from os.path import expanduser

import sys
sys.path.append(os.path.join(expanduser("~"), "meps", "meps_dev"))

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "meps_db.settings")
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

import django
django.setup();

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

from scipy import stats
from textwrap import wrap

from meps_db.components.reference import DATA_FILES_YEARS
from meps_db.analysis.expenses import ExpensesAnalyzer
sns.set_context("paper", rc={"font.size":12,"axes.titlesize":12,"axes.labelsize":12, "axes.ticksize": 12}) 
# Initializer analyzer
expenses_analyzer = ExpensesAnalyzer(years = DATA_FILES_YEARS[:11])
# calculate quantiles for decade
quantile_expenses = expenses_analyzer.calculate_quantile_expenses()
quantile_expenses_df = pd.DataFrame(quantile_expenses)<jupyter_output><empty_output><jupyter_text>Concentration Curve of Healthcare Expenditures 2018in 2018, spending on healthcare accounted for 17.7 percent of the United States gross domestic product, yet the majority of this spending was concentrated in a small percentage of the population. MEPS data show that about 13 percent of the U.S. civilian noninstitutionalized population had no personal healthcare expenditures in 2018. On the other hand, only 5 percent of the population accounted for nearly half of healthcare spending. This spending includes all sources of payments for medical care, including private insurance payments, Medicare, Medicaid, out-of-pocket spending, and other sources.Here we describe the overall concentration of healthcare expenditures across the U.S. civilian noninstitutionalized population in 2018.<jupyter_code>quans_2018 = quantile_expenses_df.loc[(quantile_expenses_df["year"]==2018)]
ax = sns.lineplot(
    data=quans_2018, 
    x="quantile", 
    y="total_exp", 
    marker= "o", 
    color="b"
)
ax.set_title("Concentration Curve of Healthcare Expenditures, 2018")
ax.set(xticks=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1])
ax.set(xlabel="Cumulative Percentile of the Population")
ax.set(ylabel="Total Expenditures in Billions")

# need to hardcode due to exponential curve
for x, y, label in zip(
    quans_2018["quantile"], quans_2018["total_exp"], quans_2018["cum_pct_exp"]
):
    if x == 0.1:
        x_fudge, y_fudge = -0.04, 100
    elif x <= 0.7:
        x_fudge, y_fudge = -0.05, 100
    elif x == 0.7:
        x_fudge, y_fudge = -0.05, -100
    elif x == 0.75:
        x_fudge, y_fudge = 0.01, -120
    elif x == 0.8:
        x_fudge, y_fudge = 0.03, -50
    elif x <0.95:
        x_fudge, y_fudge = 0.02, 0
    else:
        x_fudge, y_fudge = -0.12, 0
    
    ax.text(x+x_fudge, y+y_fudge, s=f"{label :.1f}%", color="black")

plt.savefig(f"concentration_curves/2018_concentration.png", bbox_inches='tight')
plt.show()<jupyter_output><empty_output><jupyter_text>Concentration Over TimeHere we explore how concentration curves have changed over the decade<jupyter_code>outliers = pd.DataFrame()
for quantile in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.75, 0.8, 0.9, 0.95, 0.99, 1]:
    df_slice = quantile_expenses_df.loc[(quantile_expenses_df["quantile"] == quantile)].copy()
    df_slice["z_score"] = np.abs(stats.zscore(df_slice["cum_pct_exp"]))

    outliers = pd.concat([outliers, df_slice.loc[(df_slice["z_score"]>2.5)]])

display(outliers)
#plt.figure(figsize=(15,8))
ax = sns.barplot(
    x="year", 
    y="cum_pct_exp", 
    hue="quantile", 
    data=quantile_expenses_df.loc[(quantile_expenses_df["quantile"]>=.5)],
    palette="viridis"
)
plt.legend(title="Percentile", bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set_title("\n".join(wrap(f"Cumulative Percentage of Annual Expenditure Distributions 2008-2018", 60)))
ax.set(xlabel="Year")
ax.set(ylabel="\n".join(wrap("Cumulative Percentage of Annual Expenditures", 30))) 
plt.savefig(f"concentration_curves/decade_pct_concentration.png", bbox_inches='tight')
plt.show()

outliers = pd.DataFrame()
for quantile in [0.5, 0.6, 0.7, 0.75, 0.8, 0.9, 0.95, 0.99, 1]:
    df_slice = quantile_expenses_df.loc[(quantile_expenses_df["quantile"] == quantile)].copy()
    df_slice["z_score"] = np.abs(stats.zscore(df_slice["total_exp"]))

    outliers = pd.concat([outliers, df_slice.loc[(df_slice["z_score"]>2.5)]])

display(outliers)
#plt.figure(figsize=(15,8))
ax = sns.barplot(
    x="year", 
    y="total_exp", 
    hue="quantile", 
    data=quantile_expenses_df.loc[(quantile_expenses_df["quantile"]>=.5)],
    palette="viridis"
)
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

plt.legend(title="Percentile", bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set_title("\n".join(wrap(f"Annual Expenditure Distributions 2008-2018", 60)))
ax.set(xlabel="Year")
ax.set(ylabel="\n".join(wrap("Total Expenditures in Billions", 40))) 
plt.savefig(f"concentration_curves/decade_amt_concentration.png", bbox_inches='tight')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10354403596932028, 'fraction_numerical': 0.04165564665432425, 'mean_word_length': 5.194103194103194, 'pattern_counts': {'":': 4, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '171019', 'license': 'MIT', 'n_tokens_mistral': 2702, 'n_tokens_neox': 2338, 'n_words': 778, 'path': 'research/expenditure_distribution/expenditure_distribution_over_time.ipynb', 'repo_name': 'explore-meps/meps_analytics', 'size': '7562'}