<jupyter_start><jupyter_text>News Events **Purpose:** Compare outlet leadership within multi-article news events.**Steps:**1. Load/filter dataframe of multi-article clusters2. Make dyads per cluster3. Filter dyads per cluster    - Export filtered version to pickle4. Shift from cluster-level to dyad-level5. Comparisons**Note:** Each outlet is associated one-to-one with a `doctype`. Each `doctype` is categorized under a `doctype_ideo` value, which is either "alternative-right" or "established-right".<jupyter_code>import os
import pickle
import itertools
from pprint import pprint
%matplotlib inline
# matplotlib is logged even though disable_existing_loggers=yes in logging_config.yaml
# https://stackoverflow.com/a/51529172/7016397
# workaround is to manually set the level before creating my logger
import logging
logging.getLogger('matplotlib').setLevel(logging.WARNING)

from usrightmedia.shared.loggers import get_logger
df_dir = os.path.join("..", "..", "data", "02-intermediate", "07-newsevents", "05-softcosine-clusters", "dataframes")
dyads_dir = os.path.join("..", "..", "data", "02-intermediate", "07-newsevents", "06-softcosine-dyads")
from usrightmedia.shared.clusters_utils import *
# for exporting df_urls
urls_dir = os.path.join("..", "..", "data", "03-analysis")
figs_dir = os.path.join("..", "..", "figures", "dyadic")

def save_image(obj, name, height=4.0, width=5.5, units="in", dpi=600):
    obj.save(filename = os.path.join(figs_dir, "png", f'{name}.png'), height=height, width=width, units=units, dpi=dpi)<jupyter_output><empty_output><jupyter_text>Full version  1. Load dataframe of multi-article clusters- Select the softcosine threshold whose clusters represent news events- Add columns for:    - `doctype`: corresponds with the outlet names    - `doctype_ideo`: "est" ("established right") or "alt" ("alternative right")    - `doc_topic`: the primary topic assigned to the document per topic modeling    - `doc_tw`: list of `tweets2_url` IDs (the doc's URL was tweeted or re-tweeted 1+ times by congressional Republicans who served during 2016 - 2020)- `rmv_docs_with_odd_publish_date_ind=True`: remove documents published at exactly 05:00 UTC (midnight EST)- Filter for selected year(s) and topic(s). If the corresponding value(s) is an empty list, no filtering is used and all documents are kept in the dataset.<jupyter_code>threshold = "softcos06"
# LOGGER = get_logger(filename = f'06-dyadic-comparison-{threshold}', logger_type='main')
# %%time
# df_clusters = load_multi_doc_clusters_only(df_dir, threshold)
# df_clusters["doctype"] = df_clusters["doc_id"].map(lambda d: extract_doctype(d))
# df_clusters["doctype_ideo"] = df_clusters["doctype"].map(lambda d: get_ideo_subcategory(d))
%%time
# more efficient to query all doc_ids together than to send an HTTP request to ES for each doc_id.
# store the retrieved data in a helper dataframe (df_topics, df_tw) which is then used to populate df_clusters.

# df_tw = get_field_value_for_ids(list(df_clusters["doc_id"]), "tweets2_url_ids")
# df_clusters["doc_tw"] = df_clusters["doc_id"].map(lambda d: df_tw.at[d, "tweets2_url_ids"])
# df_clusters.to_pickle(os.path.join(dyads_dir, f"{threshold}_df_clusters_tw.pkl"))
# more efficient to query all doc_ids together than to send an HTTP request to ES for each doc_id.
# store the retrieved data in a helper dataframe (df_topics, df_tw) which is then used to populate df_clusters.

# TODO: topic assignment - currently uses placeholder values
# df_topic = get_field_value_for_ids(list(df_clusters["doc_id"]), "topic")
# df_clusters["doc_topic"] = df_clusters["doc_id"].map(lambda d: df_topic.at[d, "topic"])
# df_clusters["doc_topic"] = df_clusters["doc_id"].map(lambda d: get_doc_topic(d)) # placeholder values
# df_clusters_filtered = filter_clusters(
#                             df_clusters,
#                             rmv_docs_with_odd_publish_date_ind=True,
#                             filters={"years": [],
#                                      "topics": []}
#                         )
# df_clusters_filtered.to_pickle(os.path.join(dyads_dir,  f"{threshold}_df_clusters_filtered.pkl"))<jupyter_output><empty_output><jupyter_text>2. Make dyads per cluster- A cluster can consist of 0 or more dyads.    - The dyad list is empty if only 1 document remained after the filters above.    - The dyad list contains 1+ dyads if there were at least 2+ documents remaining in the cluster.    - For each dyad:    - The first position is occupied by the doctype (i.e., outlet) which publishes the oldest article within the cluster.    - The second position is occupied by a doctype which publishes a subsequent article within the cluster.    - `dyads`: two-item tuple where each item is a dict. Each dict represents a document.- `dyads_cx`: dictionary where the key is the `cluster_id` and the value is the list of dyads associated with it.- `dyads_cid`: the dyads for one cluster<jupyter_code>%%time
# dyads_cx = get_clusters_as_dyads(df_clusters_filtered)
# with open(os.path.join(dyads_dir, f'{threshold}_dyads_cx.pkl'), 'wb') as file:
#       pickle.dump(dyads_cx, file)<jupyter_output><empty_output><jupyter_text>3. Filter dyads per cluster`keep_dyads_bn_diff_doctypes_only=True`: - If the second outlet is the same as the first outlet, then the dyad is excluded.- Example, 4-article news event:    - Breitbart publishes the 1st article (D1)    - Fox News publishes the 2nd article (D2)    - Breitbart publishes the 3rd outlet (D3)    - Daily Stormer publishes the 4th outlet (D4)- Result:    - D1-D2: included    - D1-D3: excluded (same outlet in the first and second positions of the dyad)    - D1-D4: included    `keep_first_doctype_pair_only=True`:- Only the first instance of a doctype pair is kept.- Example, 3-article news event:    - Newsmax publishes the 1st article (D1)    - One America News publishes the 2nd article (D2)    - One America News publishes the 3rd article (D3)- Result:    - D1-D2: included    - D1-D3: excluded (repeat of "newsmax-oneamericanews" pair) - Filter dyads for dyadic comparison: `keep_first_doctype_pair_only=True` so that at most one instance is counted between outlets per news event.<jupyter_code>%%time
# dyads_cx_filtered = filter_dyads(dyads_cx,
#                                  keep_dyads_bn_diff_doctypes_only=True,
#                                  keep_first_doctype_pair_only=True)

# with open(os.path.join(dyads_dir, f'{threshold}_dyads_cx_filtered.pkl'), 'wb') as file:
#       pickle.dump(dyads_cx_filtered, file)<jupyter_output>CPU times: user 2 µs, sys: 1e+03 ns, total: 3 µs
Wall time: 4.77 µs<jupyter_text>- Filter dyads for indirect political resonance (IPR): `keep_first_doctype_pair_only=False`    - not used further in this notebook<jupyter_code># dyads_cx_filtered_IPR = filter_dyads(dyads_cx,
#                                  keep_dyads_bn_diff_doctypes_only=True,
#                                  keep_first_doctype_pair_only=False)

# with open(os.path.join(dyads_dir, f'{threshold}_dyads_cx_filtered_IPR.pkl'), 'wb') as file:
#       pickle.dump(dyads_cx_filtered_IPR, file)<jupyter_output><empty_output><jupyter_text>Load from files: describe clusters and dyads - `df_clusters` with `tweets2_url_ids`<jupyter_code># load from file if already calculated
df_clusters = pickle.load(open(os.path.join(dyads_dir, f'{threshold}_df_clusters_tw.pkl'),'rb'))
df_clusters
# check that number of unique clusters in dataframe matches the chart from 03-softcosine-clusters.ipynb
df_clusters["cluster_id"].nunique()
print(f"{len(df_clusters)} documents which are members of multi-doc clusters.")<jupyter_output>208975 documents which are members of multi-doc clusters.<jupyter_text>- cluster filter applied, dyadic filter not applied<jupyter_code># load from file if already calculated
dyads_cx = pickle.load(open(os.path.join(dyads_dir, f'{threshold}_dyads_cx.pkl'),'rb'))
df_dyads_cx = dyads_cx_as_df(dyads_cx).sort_values(by=['n_dyads'], ascending=False)
df_dyads_cx['n_dyads'].describe()
# run this cell if the plots don't show inline
%matplotlib inline
df_dyads_cx['n_dyads'].plot(kind = 'box')
df_dyads_cx_gt0 = df_dyads_cx.loc[df_dyads_cx['n_dyads']>0]
df_dyads_cx_gt0['n_dyads'].describe()
df_dyads_cx_gt0['n_dyads'].plot(kind = 'box')
print(f"{len(df_dyads_cx_gt0)} clusters after applying cluster-level filters.")
print(f"{df_dyads_cx_gt0['n_dyads'].sum()} dyads after applying cluster-level filters.")<jupyter_output>61269 clusters after applying cluster-level filters.
142537 dyads after applying cluster-level filters.<jupyter_text>- cluster filter applied, dyadic filter applied<jupyter_code># load from file if already calculated
dyads_cx_filtered = pickle.load(open(os.path.join(dyads_dir, f'{threshold}_dyads_cx_filtered.pkl'),'rb'))
df_dyads_cx_filtered = dyads_cx_as_df(dyads_cx_filtered).sort_values(by=['n_dyads'], ascending=False)
df_dyads_cx_filtered['n_dyads'].describe()
df_dyads_cx_filtered_gt0 = df_dyads_cx_filtered.loc[df_dyads_cx_filtered['n_dyads']>0]
df_dyads_cx_filtered_gt0['n_dyads'].describe()
df_dyads_cx_filtered_gt0['n_dyads'].plot(kind = 'box')
# example of a cluster's dyads
df_dyads_cx_filtered_gt0\
    .loc[df_dyads_cx_filtered_gt0['n_dyads']==3]\
    .reset_index(drop=False)\
    .iloc[0]['dyads']
print(f"{len(df_dyads_cx_filtered_gt0)} clusters after applying cluster-level and dyadic-level filters.")
print(f"{df_dyads_cx_filtered_gt0['n_dyads'].sum()} dyads after applying cluster-level and dyadic-level filters.")<jupyter_output>45128 clusters after applying cluster-level and dyadic-level filters.
71220 dyads after applying cluster-level and dyadic-level filters.<jupyter_text>4. Shift from cluster-level to dyad-level<jupyter_code># %%time
dyads_filtered = extract_dyads(dyads_cx_filtered)
print(f"{len(dyads_filtered)} dyads.")<jupyter_output>71220 dyads.<jupyter_text>5. Comparisons- In the comparison tables, the doctype (i.e., outlet) in the row is the leader and the doctype in the column is the follower.  5.1 Dyadic comparisons ![dyadic comparison 1](https://raw.githubusercontent.com/wlmwng/us-right-media/develop/usrightmedia/figures/desc/dyadic_comparison_explained_1.png)![dyadic comparison 2](https://raw.githubusercontent.com/wlmwng/us-right-media/develop/usrightmedia/figures/desc/dyadic_comparison_explained_2.png)<jupyter_code># spot-check the cells in dyadic_count_doctype
matched = get_matched_dyads(dyads=dyads_filtered,
                      d0_key="doctype",
                      d1_key="doctype",
                      d0_values=["dailycaller"],
                      d1_values=["newsmax"])

len(matched)<jupyter_output><empty_output><jupyter_text>To calculate the total number of dyads that an outlet participates in: `dyadic_count_doctype.at[DOCTYPE, "All"] + dyadic_count_doctype.at["All", DOCTYPE]`<jupyter_code>dyadic_count_doctype = make_dyadic_comparison(dyads=dyads_filtered, key="doctype", version="count")
dyadic_count_doctype
dyadic_prop_doctype = make_dyadic_comparison(dyads=dyads_filtered, key="doctype", version="proportion")
# displays as percentages
dyadic_prop_doctype.round(decimals=3).select_dtypes(include=["number"])*100
dyadic_count_ideo = make_dyadic_comparison(dyads=dyads_filtered, key="doctype_ideo", version="count")
dyadic_count_ideo<jupyter_output><empty_output><jupyter_text>5.2 Dyadic comparisons over time<jupyter_code># run this cell if the plots don't show inline
%matplotlib inline
# Run this cell if the following RuntimeWarning shows up

# RuntimeWarning: More than 20 figures have been opened.
# Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained
# until explicitly closed and may consume too much memory.

# import matplotlib.pyplot as plt
# plt.close('all')<jupyter_output><empty_output><jupyter_text>- The figure aggregates the dyads between right-wing alternative and established media over 2016-2020. Each category-pair is directional. For example, `alt_est` represents a dyad where an alternative outlet was the first to cover a news event which was later adopted by an established outlet. The percentages sum to 100% across the category-pairs per year, and indicate alternative and established outlets' relative resonance within the right-wing media sphere.- When considering all dyads, alternative-right outlets lead (alt_alt + alt_est)% of the time and established-right outlets lead (est_alt + est_est) of the time.  5.2.1 All dyads<jupyter_code># effectively no filter is applied since all doctypes are included
# the function returns two plots (one for each direction)
# in this case, the two plots are identical so just show idx 0
c0 = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                       key0="doctype",
                                       key1="doctype",
                                       values0=DOCTYPES,
                                       values1=DOCTYPES,
                                       plot_type="count")[0]

save_image(c0, "cnt_d0_ideo_d1_ideo")

p0 = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                       key0="doctype",
                                       key1="doctype",
                                       values0=DOCTYPES,
                                       values1=DOCTYPES,
                                       plot_type="percent")[0]

save_image(p0, "pct_d0_ideo_d1_ideo")
# effectively no filter is applied since all doctypes are included
# the function returns two plots (one for each direction)
for doctype in DOCTYPES:
    c0, c1 = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                       key0="doctype",
                                       key1="doctype",
                                       values0=[doctype],
                                       values1=DOCTYPES,
                                       plot_type="count")
    
    save_image(c0, f"cnt_d0_outl_d1_ideo_{doctype}")
    save_image(c1, f"cnt_d0_ideo_d1_outl_{doctype}")
    
    p0, p1 = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                           key0="doctype",
                                           key1="doctype",
                                           values0=[doctype],
                                           values1=DOCTYPES,
                                           plot_type="percent")
    
    save_image(p0, f"pct_d0_outl_d1_ideo_{doctype}")
    save_image(p1, f"pct_d0_ideo_d1_outl_{doctype}")
doctype
c0
c1
p0
p1
DOCTYPE_COMBOS = list(itertools.product(DOCTYPES, DOCTYPES))
DOCTYPE_COMBOS = [dc for dc in DOCTYPE_COMBOS if dc[0]!=dc[1]] # remove self-self
print(len(DOCTYPE_COMBOS))
DOCTYPE_COMBOS[0:5]
# plot_filtered_yearly_dyadic_comparison():
# when values0 and values1 each have exactly 1 doctype value,
# the function returns a 1-item list (1 plot) with both directions (displays at "doctype_to_doctype" level)
for combo in DOCTYPE_COMBOS:
    doctype0 = combo[0]
    doctype1 = combo[1]
    c = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                       key0="doctype",
                                       key1="doctype",
                                       values0=[doctype0],
                                       values1=[doctype1],
                                       plot_type="count")[0]
    
    save_image(c, f"cnt_d0_outl_d1_outl_{doctype0}_{doctype1}")

    p = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                           key0="doctype",
                                           key1="doctype",
                                           values0=[doctype0],
                                           values1=[doctype1],
                                           plot_type="percent")[0]
    
    save_image(p, f"pct_d0_outl_d1_outl_{doctype0}_{doctype1}")<jupyter_output>/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:719: PlotnineWarning: Saving 5.5 x 4.0 in image.
/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:722: PlotnineWarning: Filename: ../../figures/dyadic/png/cnt_d0_outl_d1_outl_americanrenaissance_breitbart.png
/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:719: PlotnineWarning: Saving 5.5 x 4.0 in image.
/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:722: PlotnineWarning: Filename: ../../figures/dyadic/png/pct_d0_outl_d1_outl_americanrenaissance_breitbart.png
/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:719: PlotnineWarning: Saving 5.5 x 4.0 in image.
/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:722: PlotnineWarning: Filename: ../../figures/dyadic/png/cnt_d0_outl_d1_outl_americanrenaissance_dailycaller.png
/home/jovyan/.local/lib/python3.8/site-packages/plotnine/ggplot.py:719: PlotnineWarning: Saving 5.5 x 4.0 in image.
[...]<jupyter_text>- compare selected established-right outlets based on their patterns as followers<jupyter_code># spot-check the yearly dyadic comparison
matched = get_matched_dyads(dyads=dyads_filtered,
                      d0_key="doctype_ideo",
                      d1_key="doctype",
                      d0_values=["alt", "est"],
                      d1_values=["foxnews"])

print(f"matched: {len(matched)} dyads")
print(f"matched per year: {sorted(Counter([m[0]['doc_publish_date'].year for m in matched]).items())}")
for DOCTYPE in ["foxnews", "washingtonexaminer"]:
    KEY0 = "doctype"
    KEY1 = "doctype_ideo"
    VALUES0 = [DOCTYPE]
    DOCTYPE0_IDEO = IDEOS[list(df_ideo.loc[df_ideo["outlet_std"]==VALUES0[0]]["ideo_subcategory"])[0]]
    VALUES1 = ["alt", "est"]

    plots = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                                   key0=KEY0,
                                                   key1=KEY1,
                                                   values0=VALUES0,
                                                   values1=VALUES1,
                                                   plot_type="count")

    # print("-"*120)
    # print(f"When {VALUES0} is a leader, it leads alternative-right outlets ({DOCTYPE0_IDEO}_alt)% of the time and established-right outlets ({DOCTYPE0_IDEO}_est)% of the time.")
    # print("-"*120)
    # print(f"Leading {KEY0}(s): {VALUES0}")
    # print(f"Following {KEY1}(s): {VALUES1}")
    # display(plots[0])

    print("-"*120)
    print(f"When {VALUES0} is a follower, it follows alternative-right outlets (alt_{DOCTYPE0_IDEO})% of the time and established-right outlets (est_{DOCTYPE0_IDEO})% of the time.")
    print("-"*120)
    print(f"Leading {KEY1}(s): {VALUES1}")
    print(f"Following {KEY0}(s): {VALUES0}")
    display(plots[1])
for DOCTYPE in ["foxnews", "washingtonexaminer"]:
    KEY0 = "doctype"
    KEY1 = "doctype_ideo"
    VALUES0 = [DOCTYPE]
    DOCTYPE0_IDEO = IDEOS[list(df_ideo.loc[df_ideo["outlet_std"]==VALUES0[0]]["ideo_subcategory"])[0]]
    VALUES1 = ["alt", "est"]

    plots = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                                   key0=KEY0,
                                                   key1=KEY1,
                                                   values0=VALUES0,
                                                   values1=VALUES1,
                                                   plot_type="percent")

    # print("-"*120)
    # print(f"When {VALUES0} is a leader, it leads alternative-right outlets ({DOCTYPE0_IDEO}_alt)% of the time and established-right outlets ({DOCTYPE0_IDEO}_est)% of the time.")
    # print("-"*120)
    # print(f"Leading {KEY0}(s): {VALUES0}")
    # print(f"Following {KEY1}(s): {VALUES1}")
    # display(plots[0])

    print("-"*120)
    print(f"When {VALUES0} is a follower, it follows alternative-right outlets (alt_{DOCTYPE0_IDEO})% of the time and established-right outlets (est_{DOCTYPE0_IDEO})% of the time.")
    print("-"*120)
    print(f"Leading {KEY1}(s): {VALUES1}")
    print(f"Following {KEY0}(s): {VALUES0}")
    display(plots[1])<jupyter_output><empty_output><jupyter_text>- see `doctype_to_doctype` for Fox News<jupyter_code># plot_filtered_yearly_dyadic_comparison():
# when values0 and values1 each have exactly 1 doctype value,
# the function returns 1 plot with both directions (displays at "doctype_to_doctype" level)

KEY0 = "doctype"
KEY1 = "doctype"
VALUES0 = ["breitbart"] # ALT_RIGHT_DOCTYPES
VALUES1 = ["foxnews"]

for DOCTYPE in VALUES0:
    display(plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                                   key0=KEY0,
                                                   key1=KEY1,
                                                   values0=[DOCTYPE],
                                                   values1=VALUES1,
                                                   plot_type="count"))
# plot_filtered_yearly_dyadic_comparison():
# when values0 and values1 each have exactly 1 doctype value,
# the function returns 1 plot with both directions (displays at "doctype_to_doctype" level)

KEY0 = "doctype"
KEY1 = "doctype"
VALUES0 = ["breitbart"] # ALT_RIGHT_DOCTYPES
VALUES1 = ["foxnews"]

for DOCTYPE in VALUES0:
    display(plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                                   key0=KEY0,
                                                   key1=KEY1,
                                                   values0=[DOCTYPE],
                                                   values1=VALUES1,
                                                   plot_type="percent"))<jupyter_output><empty_output><jupyter_text>- compare selected alternative-right outlets based on their patterns as leaders<jupyter_code>for DOCTYPE in ["breitbart", "gatewaypundit", "newsmax", "oneamericanews"]:
    KEY0 = "doctype"
    KEY1 = "doctype_ideo"
    VALUES0 = [DOCTYPE]
    DOCTYPE0_IDEO = IDEOS[list(df_ideo.loc[df_ideo["outlet_std"]==VALUES0[0]]["ideo_subcategory"])[0]]
    VALUES1 = ["alt", "est"]

    plots = plot_filtered_yearly_dyadic_comparison(dyads_filtered,
                                                   key0=KEY0,
                                                   key1=KEY1,
                                                   values0=VALUES0,
                                                   values1=VALUES1,
                                                   plot_type="percent")

    print("-"*120)
    print(f"When {VALUES0} is a leader, it leads alternative-right outlets ({DOCTYPE0_IDEO}_alt)% of the time and established-right outlets ({DOCTYPE0_IDEO}_est)% of the time.")
    print("-"*120)
    print(f"Leading {KEY0}(s): {VALUES0}")
    print(f"Following {KEY1}(s): {VALUES1}")
    display(plots[0])

    # print("-"*120)
    # print(f"When {VALUES0} is a follower, it follows alternative-right outlets (alt_{DOCTYPE0_IDEO})% of the time and established-right outlets (est_{DOCTYPE0_IDEO})% of the time.")
    # print("-"*120)
    # print(f"Leading {KEY1}(s): {VALUES1}")
    # print(f"Following {KEY0}(s): {VALUES0}")
    # display(plots[1])<jupyter_output><empty_output><jupyter_text>Dev version  1. Load/filter dataframe of multi-article clusters<jupyter_code>df_clusters_dev = load_multi_doc_clusters_only(df_dir, "softcos09_dev")
df_clusters_dev = df_clusters_dev.loc[df_clusters_dev["cluster_id"].isin(
    ["softcos09_0",
     "softcos09_1",
     "softcos09_1018",
     "softcos09_8",
     "softcos09_1019"])].reset_index(drop=True)

df_clusters_dev["doctype"] = df_clusters_dev["doc_id"].map(lambda d: extract_doctype(d))
df_clusters_dev["doctype_ideo"] = df_clusters_dev["doctype"].map(lambda d: get_ideo_subcategory(d))
df_clusters_dev["doc_topic"] = df_clusters_dev["doc_id"].map(lambda d: get_doc_topic(d))

# more efficient to query all doc_ids together than to send an HTTP request to ES for each doc_id.
# store the retrieved data in a helper dataframe (df_topics, df_tw) which is then used to populate df_clusters.

# TODO
# df_topic_dev = get_field_value_for_ids(list(df_clusters_dev["doc_id"]), "topic")
# df_clusters_dev["doc_topic"] = df_clusters_dev["doc_id"].map(lambda d: df_topic_dev.at[d, "topic"])

df_tw_dev = get_field_value_for_ids(list(df_clusters_dev["doc_id"]), "tweets2_url_ids")
df_clusters_dev["doc_tw"] = df_clusters_dev["doc_id"].map(lambda d: df_tw_dev.at[d, "tweets2_url_ids"])

df_clusters_dev_filtered = filter_clusters(
                            df_clusters_dev,
                            rmv_docs_with_odd_publish_date_ind=True,
                            filters={"years": [],
                                     "topics": []}
                        )<jupyter_output><empty_output><jupyter_text>`rmv_docs_with_odd_publish_date_ind=True`: only keep a document if it is not published at exactly 05:00 UTC (midnight EST)- `softcos09_0`: first 2 docs are odd -> first 2 docs are removed- `softcos09_1`: no docs are odd -> no docs removed- `softcos09_1018`: first doc is odd -> first doc is removed- `softcos09_8`: all docs are odd -> all docs removed- `softcos09_1019`: non-first doc is odd -> no docs removed`filters, years`: keep the cluster if its first document was published in one of the specified years`filters, topics`: keep the cluster if its first document was assigned one of the specified topics<jupyter_code>df_clusters_dev
df_clusters_dev_filtered<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10636367222945516, 'fraction_numerical': 0.021619915571862547, 'mean_word_length': 3.1404769683110096, 'pattern_counts': {'":': 4, '<': 67, '<?xml version=': 0, '>': 74, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 96, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '249941', 'license': 'MIT', 'n_tokens_mistral': 8544, 'n_tokens_neox': 7870, 'n_words': 1852, 'path': 'usrightmedia/code/09-analysis/01-newsevents-dyadic-comparison.ipynb', 'repo_name': 'wlmwng/us-right-media', 'size': '25347'}