<jupyter_start><jupyter_text>Indicator ValidationThis notebook is aimed at assisting developers with tracking large scale indicator changes beyond what can be picked up by the unit tests. While unit tests perform local sanity checks on the operations, the tests here will be more qualitative in nature, comparing the live version of an indicator with the propagating changes. UsageSince each indicator will have different points of interest, this notebook will only provide a framework to get started. The goal is to support the comparison of the dataframes resulting from the data cleaning and shaping that our indicator code provides.<jupyter_code># Set your base directory (assuming this notebook is run from "covidcast_indicators/testing_utils/")
os.chdir("../")

%run testing_utils/indicator_validation.py
# The above will likely cause an error due to missing dependencies, run this to fix
%%capture
!pip install pandas matplotlib joblib covidcast<jupyter_output><empty_output><jupyter_text>InstallationInstall the utilities and the indicator you plan to test.<jupyter_code>%%capture
os.chdir(join(ROOT_DIR, "_delphi_utils_python"))
!pip install -e .
os.chdir(join(ROOT_DIR, "jhu"))
!pip install -e .<jupyter_output><empty_output><jupyter_text>Run the indicatorIf you are planning on testing your local receiving directory, you will need to generate those files. You can do that by running the cell below.<jupyter_code>%%capture
os.chdir(join(ROOT_DIR, "jhu"))
!python -m delphi_jhu<jupyter_output><empty_output><jupyter_text>Qualitative Comparisons  Loading DataTo load a local indicator and a remote `covidcast` indicator as dataframes for comparison use the function `load_signal_data(local_signal_dir, remote_signal_name, signal_type, start_day, end_day, geo_type)`. Separate functions for loading just the local and remote data exist as well. **Note that the local and remote values are cached to disk** to speed up computation and reduce API calls. See function docstring for instructions on clearing the cache.<jupyter_code>local_signal_dir = "jhu"
remote_signal_name = "usa-facts"
signal_type = "confirmed_incidence_prop"
start_day = date(2020, 8, 1)
end_day = date.today()
geo_type = "state"

# load_local_signal_data.clear()
local_data, remote_data = load_signal_data(local_signal_dir, remote_signal_name, signal_type, start_day, end_day, geo_type)<jupyter_output><empty_output><jupyter_text>Comparing Geocode SignalsA simple plotting demo.<jupyter_code>geo_code = "ca"
rd = remote_data.loc[geo_code, :]["value"]
ld = local_data.loc[geo_code, :]["val"]
plt.figure(figsize=(16, 6))
plt.plot(rd, label="remote")
plt.plot(ld, label="local")
plt.title(geo_code)
plt.legend()
print("Total difference: ", rd.sub(ld).sum())<jupyter_output>Total difference:  -0.18981468089412878<jupyter_text>Automatically Detecting OutliersDefining your own comparison statistics, we can automatically plot outlier signals.<jupyter_code>def simple_outlier_test(ld, rd):
    """
    Returns True if the rd and ld time series are similar, False otherwise.
    """
    max_percent_diff = rd.sub(ld).abs().max() / rd.abs().max()
    sum_percent_diff = rd.sub(ld).abs().sum() / rd.abs().sum()
    return False if max_percent_diff > 1.1 or sum_percent_diff > 1.1 else True

def plot_outliers(ld, rd, outlier_check):
    """
    Provides time series plots of geocodes that are sufficiently different.
    """
    local_geo_codes = set(local_data.reset_index()["geo_id"].unique())
    remote_geo_codes = set(remote_data.reset_index()["geo_value"].unique())
    geo_codes = local_geo_codes.intersection(remote_geo_codes)
    for geo_code in geo_codes:
        rd = remote_data.loc[geo_code, :]["value"]
        ld = local_data.loc[geo_code, :]["val"]
        if outlier_check(ld, rd) == False:
            plt.figure(figsize=(16, 6))
            plt.plot(rd, label="remote")
            plt.plot(ld, label="local")
            plt.title(geo_code)
            plt.legend()

plot_outliers(ld, rd, simple_outlier_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0968222442899702, 'fraction_numerical': 0.008440913604766634, 'mean_word_length': 5.425837320574162, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '385685', 'license': 'MIT', 'n_tokens_mistral': 1291, 'n_tokens_neox': 1215, 'n_words': 384, 'path': 'testing_utils/indicator_validation.template.ipynb', 'repo_name': 'davidkretch/covidcast-indicators', 'size': '4028'}