<jupyter_start><jupyter_text>Example of GISAID data filtering  Borrows from some basic outbreak.info and covidcg.org filters- outbreak.info methods: https://outbreak.info/situation-reports/methods- covidcg.org methods: https://covidcg.org/?tab=methodology This example uses the following filters:- sequences at least 20000 in length- collection date must not be in the future and must be at least year/month, not year alone- excluding sequences from the Nextstrain exclude list We may also want to filter out any sequence that is:- Less than 29700 bases (vs our 20000 threshold right now)- greater than 5% ambiguous base calls (N)<jupyter_code># how many B.1.1.7 sequences are in the US? Outbreak.info says 54,109, we're close
# If we were to use this data, we'd filter it using
gisaid_df.loc[
    (gisaid_df['country'] == 'USA') &
    (gisaid_df['Pango lineage'] == 'B.1.1.7') &
    (gisaid_df['Sequence length'] > 20000) &
    (~gisaid_df['Collection date'].isin(['2020', '2021'])) &
    (gisaid_df['Collection date'] < '2021-04-26') &
    (~gisaid_df['Virus name'].isin(exclude_sequences))
].shape
# example of still messy location data - there are a few locations in US still not nicely formatted
# as a tree structure
[x for x in location_set if 'USA' in x and not x.startswith('North America / USA')]<jupyter_output><empty_output><jupyter_text>outbreak.info API playing around<jupyter_code># daily b.1.1.7 counts in US
url = 'https://api.outbreak.info/genomics/prevalence-by-location?location_id=USA&cumulative=false&pangolin_lineage=B.1.1.7'
r = requests.get(url)
daily_sum = sum({x['date']: x['lineage_count'] for x in r.json()['results']}.values())
daily_sum
# total b.1.1.7 count in US
url = 'https://api.outbreak.info/genomics/prevalence-by-location?location_id=USA&cumulative=true&pangolin_lineage=B.1.1.7'
r = requests.get(url)
r.json()['results']['lineage_count'] == daily_sum
# Compare GISAID and outbreak.info

vocs = ['B.1.1.7', 'B.1.429', 'B.1.427', 'P.1', 'B.1.351']
vois = ['B.1.526', 'B.1.526.1', 'B.1.526.2', 'P.2', 'B.1.617']
def get_gisaid_US_lineage_count(lineage):
    return gisaid_df.loc[
        (gisaid_df['country'] == 'USA') &
        (gisaid_df['Pango lineage'] == lineage) &
        (gisaid_df['Sequence length'] > 20000) &
        (~gisaid_df['Collection date'].isin(['2020', '2021'])) &
        (gisaid_df['Collection date'] < '2021-04-26') &
        (~gisaid_df['Virus name'].isin(exclude_sequences))
    ].shape[0]

def get_outbreak_info_US_lineage_count(lineage):
    url = 'https://api.outbreak.info/genomics/prevalence-by-location?location_id=USA&cumulative=true&pangolin_lineage=%s' % lineage
    r = requests.get(url)
    return r.json()['results']['lineage_count']

counts = []
for lineage in vocs + vois:
    gisaid_lineage_count = get_gisaid_US_lineage_count(lineage)
    outbreak_info_lineage_count = get_outbreak_info_US_lineage_count(lineage)
    pct_difference = (gisaid_lineage_count - outbreak_info_lineage_count) / gisaid_lineage_count
    counts.append({
        'lineage': lineage,
        'GISAID_US_count': gisaid_lineage_count,
        'outbreak_info_US_count': outbreak_info_lineage_count,
        'pct_difference': pct_difference * 100
    })<jupyter_output><empty_output><jupyter_text>Cumulative counts for lineages in US from GISAID and outbreak.info (pct_difference is relative to GISAID count)<jupyter_code>pd.DataFrame(counts)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1388482899736919, 'fraction_numerical': 0.03507746273019585, 'mean_word_length': 4.849572649572649, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '365571', 'license': 'Apache-2.0', 'n_tokens_mistral': 1319, 'n_tokens_neox': 1209, 'n_words': 277, 'path': 'Variants exploration.ipynb', 'repo_name': 'julia326/covid-variant-tracking', 'size': '3421'}