<jupyter_start><jupyter_text>Analyzing hyperpartisan documents**<NAME>**The purpose of this assignment is to analyze the usage of words in documents which are considered to be *hyperpartisan*, or not. Hyperpartisan arguments are those that "exhibit blind, prejudiced, or unreasoning allegiance to one party, faction, cause, or person".You will be given two main documents:- [articles-validation-bypublisher-20181122.xml.zip](https://zenodo.org/record/5776081/files/articles-validation-bypublisher-20181122.zip?download=1) (332 MB): an XML document 150k News articles.- [ground-truth-validation-bypublisher-20181122.xml.zip](https://zenodo.org/record/5776081/files/ground-truth-validation-bypublisher-20181122.zip?download=1) (5.2MB): a document that states whether the News articles contain hyperartisan ar- guments or not.The dataset were used in the [semeval-2019 task 4](https://pan.webis.de/semeval19/semeval19-web/) on hyperpartisan News detection. Whereas the task on semeval was to design a system to auto- matically detect hyperpartisan news, in this exercise we are going to exploit both corpora (hyperpartisan and not hyperpartisan news), and analyze which terms are the most relevant in each of the sets. For this, analysis we will use the so-called log odd ratio.I recommend you to follow a two-step approach to obtain the log-odd ratios of the words. In the first step, you should generate text files from the XML documents. Then, in a separated step, you should create a le with the words along with its **log-odd ratio**.<jupyter_code>%cd /content/drive/MyDrive/LAP/Subjects/CL/assignments/hyperpartisan
import nltk
nltk.download('punkt')
nltk.download('stopwords')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>1. Generate text filesThe first step is to generate two text files for hyperpartisan and non-hyperpartisan news articles, respectively. You have to divide the News articles contained in [articles-validation-bypublisher-20181122.xml.zip](https://zenodo.org/record/1489920/files/articles-validation-bypublisher-20181122.zip?download=1) into two text les (`hyperpartisan.txt` and `non-hyperpartisan.txt`), according to their ground truth value in [ground-truth-validation-bypublisher-20181122.xml.zip](https://zenodo.org/record/1489920/files/ground-truth-validation-bypublisher-20181122.zip?download=1). You will need the `lxml` library in python to analyze the XML documents and extract the necessary information.When dividing the articles, it is highly recommended that you tokenize the text using a proper tokenizer. There are several tokenizers available for English in Python, chose the one that fits you best.<jupyter_code>"""Generate two text files for hyperpartisan and non-hyperpartisan
news articles."""
import string
from lxml import etree
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from tqdm import tqdm


PATH = "../../data/hyperpartisan/"
PUNCTUATION = string.punctuation + "‘’“”«»–…•"
STOPWORDS = set(stopwords.words('english'))
ENTITIES = set(["lt", "amp", "gt", "quot", "apos"])
IMAGE = set(["img", "alt", "src", "width", "height", "https", "datarecalcdims", "aligncenter"])


def clean_text(text):
    """Tokenize text, convert words to lowercase, remove puntuation,
    numbers, stop words, xml entities and image tags.

    Args:
        text (str): text to be cleaned

    Returns:
        str: cleaned text
    """
    # split into tokens
    words = word_tokenize(text)
    # convert to lower case and remove punctuation
    table = str.maketrans('', '', PUNCTUATION)
    words = [str.lower(word.translate(table)) for word in words]
    # remove numbers, stop words, xml entities and image tags
    words = [word for word in words if word.isalpha() and word not in STOPWORDS | ENTITIES | IMAGE]
    # join words to string
    text = " ".join(words)
    # remove extra whitespace
    text = " ".join(text.split())
    # strip whitespace at start and end
    text = text.strip()
    return text


def save_text(article_tree, class_dict, hyp_file, non_file):
    """Save text of hyperpartisan and non-hyperpartisan articles.

    Args:
        article_tree (xml.etree.ElementTree): articles xml document tree
        class_dict (dict): dictionary of article ids and their hyperpartisan class
        hyp_file (TextIOWrapper): hyperarpartisan output file
        non_file (TextIOWrapper): non-hyperpartisan output file
    """
    for article in tqdm(article_tree.xpath("/articles/article")):
        ident = article.get("id")
        title = article.get("title")
        title = clean_text(title)
        text = article.xpath("string(.)")
        text = clean_text(text)
        line = f"{title} {text}".strip()
        if class_dict[ident] == "true":
            hyp_file.write(f"{line}\n")
        else:
            non_file.write(f"{line}\n")


def get_class_dict(class_tree):
    """Get dictionary of article ids and their hyperpartisan class.

    Args:
        class_tree (xml.etree.ElementTree): class xml document tree

    Returns:
        dict: dictionary of article ids and their hyperpartisan class
    """
    class_dict = {}
    for article in class_tree.xpath("/articles/article"):
        class_dict[article.get("id")] = article.get("hyperpartisan")
    return class_dict


def generate_text_files():
    """Main function that generates hyperpartisan and non-hyperpartisan text files."""
    # discard whitespace nodes
    parser = etree.XMLParser(remove_blank_text=True)

    # parse article document
    article_tree = etree.parse(
        f"{PATH}articles-validation-bypublisher-20181122.xml", parser)

    # parse class document
    class_tree = etree.parse(
        f"{PATH}ground-truth-validation-bypublisher-20181122.xml", parser)

    class_dict = get_class_dict(class_tree)

    with open(f"{PATH}hyperpartisan.txt", 'w', encoding="utf-8") as hyp_file, \
            open(f"{PATH}non-hyperpartisan.txt", 'w', encoding="utf-8") as non_file:
        save_text(article_tree, class_dict, hyp_file, non_file)
generate_text_files()<jupyter_output>100%|██████████| 150000/150000 [26:15<00:00, 95.20it/s]<jupyter_text>2. Extract log-odd ratiosHaving the two text files, write a script that extracts the log-odd ratios of each word, by applying the equations. Because the log-odd ratio is sensitive to infrequent words, discard words that appear less than 20 times in the corpus. As a bonus, you can also extract the log-odd ratios of the bigrams in the corpus.The log odd ratio is a measure of words compared on two sets of documents ($i$ and $j$), which in our case corresponds to hyperpartisan and non-hyperpartisan documents, respectively. Each word then can be associated with its log-odd ratio $r_w$, which is a number that can be positive or negative: positive numbers are associated with set $i$, and negative numbers with set $j$.The log-odd ratio $r_w$ is defined as:$$p_w^{(i)} = \frac{f_w^{(i)}}{N^{(i)}}; p_w^{(j)} = \frac{f_w^{(j)}}{N^{(j)}}$$$$o_w^{(i)} = \frac{p_w^{(i)}}{1-p_w^{(i)}}; o_w^{(j)} = \frac{p_w^{(j)}}{1-p_w^{(j)}}$$$$r_w = \log{o_w^{(i)}} - \log{o_w^{(j)}}$$where $f_w^{(i)}$ is the frequency of word $w$ in group $i$ (hyperpartisan or non-hyperpartisan), and $N^{(i)}$ is the number of words in group $i$.For example, suppose that the word $gold$ appears $2,500$ times on hyperpartisan documents ($f_{gold}^{(i)} = 2500$), and $760$ times on non-hyperpartisan documents ($f_{gold}^{(j)} = 760$). Furthermore, suppose that there are $25,000$ words in hyperpartisan documents ($N^{i} = 25000$), and $17,500$ on non-hyperpartisan documents ($N^{j} = 17000$). Then:$$p_{gold}^{(i)} = \frac{2500}{25000} = 0.1; p_{gold}^{(j)} = \frac{760}{17500} = 0.045$$$$o_{gold}^{(i)} = \frac{0.1}{1-0.1} = 0.11; o_{gold}^{(j)} = \frac{0.045}{1-0.045} = 0.047$$$$r_{gold} = \log{0.11} - \log{0.047} = 0.369$$and therefore the log odd ratio of $gold$ is $0.369$.<jupyter_code>"""Extracts log odds ratios for words and bigrams."""
from collections import defaultdict
from math import log
from tqdm import tqdm

PATH = "../../data/hyperpartisan/"


def word_bigram_freq(filename):
    """Return two dictionaries with the word and bigram frequencies in a file.

    Args:
        filename (str): path to file

    Returns:
        defaultdict[str, int]: dictionary with word frequencies
        defaultdict[str, int]: dictionary with bigram frequencies
    """
    word_freq = defaultdict(int)
    bigram_freq = defaultdict(int)
    with open(filename, encoding='utf-8') as file:
        for line in tqdm(file.readlines()):
            words = line.split()
            for i in range(len(words) - 1):
                word_freq[words[i]] += 1
                bigram_freq[f"{words[i]}\t{words[i+1]}"] += 1
            word_freq[words[len(words) - 1]] += 1
    return word_freq, bigram_freq


def log_odd(freq, total):
    """Return the log odds for a word or bigram.

    Args:
        freq (int): word frequency
        total (int): total number of words or bigrams

    Returns:
        float: log odd for the word or bigram
    """
    prob = freq / total
    odd = prob / (1 - prob)
    return log(odd)


def log_odd_ratio(string_freq_hyp, string_freq_non, min_freq=20):
    """Return the log odd ratios for words or bigrams.

    Args:
        string_freq_hyp (defaultdict[str, int]): dictionary with word or bigram frequencies
            in hyperpartisan file
        string_freq_non (defaultdict[str, int]): dictionary with word or bigram frequencies
            in non-hyperpartisan file
        min_freq (int): minimum frequency for a word or bigram to be included

    Returns:
        defaultdict[str, float]: log odds ratios for each word or bigram
    """
    n_hyp = sum(string_freq_hyp.values())
    n_non = sum(string_freq_non.values())
    log_odds = defaultdict(float)
    strings = set(string_freq_hyp.keys())
    strings.update(string_freq_non.keys())
    for string in tqdm(strings):
        freq_hyp = string_freq_hyp[string]
        freq_non = string_freq_non[string]
        if freq_hyp < min_freq or freq_non < min_freq:
            continue
        log_odds[string] = log_odd(freq_hyp, n_hyp) - log_odd(freq_non, n_non)
    return log_odds


def write_results(log_odds, hyp_out, non_out, count=50):
    """Write the results to hyperpartisan and non-hyperpartisan files.

    Args:
        log_odds (defaultdict[str, float]): dictionary with log odds ratios
        hyp_out (file): file to write hyperpartisan results to
        non_out (file): file to write non-hyperpartisan results to
        count (int): number of results to write to each file
    """
    sorted_log_odds = sorted(
        log_odds.items(), key=lambda x: x[1], reverse=True)

    with open(hyp_out, "w", encoding="utf-8") as file:
        for string, odd in sorted_log_odds[:count]:
            file.write(f"{string}\t{odd}\n")

    with open(non_out, "w", encoding="utf-8") as file:
        for string, odd in sorted_log_odds[::-1][:count]:
            file.write(f"{string}\t{odd}\n")


def extract_log_odd_ratios():
    """Main function that extracts log odds ratios for words and bigrams."""
    hyp_inp = f"{PATH}hyperpartisan.txt"
    non_inp = f"{PATH}non-hyperpartisan.txt"
    word_freq_hyp, bigram_freq_hyp = word_bigram_freq(hyp_inp)
    word_freq_non, bigram_freq_non = word_bigram_freq(non_inp)

    log_odd_words = log_odd_ratio(word_freq_hyp, word_freq_non)
    hyp_words = "hyperpartisan_words.tsv"
    non_words = "non-hyperpartisan_words.tsv"
    write_results(log_odd_words, hyp_words, non_words)

    log_odd_bigrams = log_odd_ratio(bigram_freq_hyp, bigram_freq_non)
    hyp_bigrams = "hyperpartisan_bigrams.tsv"
    non_bigrams = "non-hyperpartisan_bigrams.tsv"
    write_results(log_odd_bigrams, hyp_bigrams, non_bigrams)
extract_log_odd_ratios()<jupyter_output>100%|██████████| 75000/75000 [01:03<00:00, 1190.04it/s]
100%|██████████| 75000/75000 [01:20<00:00, 932.29it/s]
100%|██████████| 1086179/1086179 [00:01<00:00, 578325.06it/s]
100%|██████████| 20925996/20925996 [00:42<00:00, 495757.36it/s]<jupyter_text>3. Analyze the resultsHaving the log-odd ratios, extract the most relevant 50 words in hyperpartisan and non-hyperpartisan documents. Analyze these words, and write a small document with your findings. Is there any interesting word on some set? Can you draw some conclusions regarding hyperpartisan text with respect to non-hyperpartisan ones? If you also computed the log-odd ratios for bigrams, repeat the analysis using bigrams.  WordsThere are many differences betweeen the 50 most relevant words of hyperpartisan and non-hyperpartisan news. Here are the main findings of each class.Words of hyperpartisan articles:- Hyperpartisan articles contain words ending in -ist/-ism/-ity(anarchist, anarchism, globalist, globalists, individualist, anarchists, zionists, vulgarity, profanity). These do not appear in non-hyperpartisan words.- Other hyperpartisan words that describe people (slager, teabagger, shep, lgbtq, courteous). Similar terms do not appear in non-hyperpartisan words.- Bad words in hyperpartisan articles (fucking, trolling, fuck, fck). There are no bad words in non-hyperpartisan.- News sites or webs in hyperpartisan articles (wonkette, realclearpolitics, newsbusters, vox, gofundme, newsmax, foxnewscom). This suggest that these news sites are commonly associated with hyperpartisan news. Most correspond to news agencies in the US. No news agencies appear in non-hyperpartisan words.- Other organizations in hyperpartisan news (usmc (United States Marine Corps), splc (Southern Poverty Law Center), emmys). They are US organizations.- People in hyperpartisan articles (oreilly, kilmeade, chomsky, cavuto, grahamcassidy, omalley, madsen, beyoncé, willard, odonell, kliff, machado, watters, susteren). They correspond to politicians, journalists and famous people.Words of non-hyperpartisan articles:- Demonyms in non-hyperpartisan articles (subsaharan, nigerians, thai, israelpalestinian, nigerian). They correspond to people from other countries. No demonyms appear in hyperpartisan words.- Places in non-hyperpartisan articles (bangkok, myanmar, rakhine, nigeria, thailand, tribune, kyoto, lima). Many places appear in non-hyperpartsan words, none in hyperpartisan words. They correspond to other countries and cities.- People in non-hyperpartisan articles (straus, zuma, newsom, hun, schwarzenegger, hu (Hu Jintao)). They correspond to politicians and famous people.- Organizations in non-hyperpartisan articles (treasuries, boko, haram, tic, utaustin (The University of Texas at Austin), anc (African National Congress, pri (Partido Revolucionario Institucional), nld (National League for Democracy)). Unlike hyperpartisan organizations, many organizations are from other countries different from the US.- There are many economics terms in non-hyperpartisan articles (renminbi, yen, rebalance, cfr, depreciation, exporters, aggregator, outflow). Some correspond to currencies and other to actions or people.<jupyter_code>import pandas as pd

hyp_words = "hyperpartisan_words.tsv"
non_words = "non-hyperpartisan_words.tsv"
df_hyp_words = pd.read_csv(hyp_words, delimiter="\t", header=None, names=["word_hyp", "ratio_hyp"])
df_non_words = pd.read_csv(non_words, delimiter="\t", header=None, names=["word_non", "ratio_non"])

df_words = pd.concat([df_hyp_words, df_non_words], axis=1)
df_words.to_csv("words.tsv", sep="\t", index=None)
display(df_words)
df_words.plot()<jupyter_output><empty_output><jupyter_text>BigramsThere are many differences betweeen the 50 most relevant bigrams of hyperpartisan and non-hyperpartisan news. Here are the main findings of each class.Bigrams of hyperpartisan articles:- More negative words than on non-hyperpartisan news (threats violence, hate group, divestment sanctions, overdose deaths, illegal alien)- People (obama, trump, bill oreilly, romney, <NAME>, mr comey, van susteren). They correspond to politicians or famous people- Media related terms (media research, independent journalism, media matters, corporate media, associate editor)- Politics related terms (trump obama, legislature, obamacare, america health, basic income, ruling class) Bigrams of non-hyperpartisan articles:- Demonyms in non-hyperpartisan articles (southeast asian, african).- Places in non-hyperpartisan news (texas, us, china, southeast asia, travis county, austin). Some places are repeated a lot in different bigrams: china, us and texas are the most repeated ones.- Many economics related bigrams also appear a lot (emerging economies, exchange rate, emerging markets, direct investment, private investors...).- Organizations (boko haram, international institutions, china gorvernment...)- People are also mentioned (<NAME>, <NAME>, <NAME>, president jacob, <NAME>)<jupyter_code>hyp_bigrams = "hyperpartisan_bigrams.tsv"
non_bigrams = "non-hyperpartisan_bigrams.tsv"
df_hyp_bigrams = pd.read_csv(hyp_bigrams, delimiter="\t", header=None, names=["word1_hyp", "word2_hyp", "ratio_hyp"])
df_non_bigrams = pd.read_csv(non_bigrams, delimiter="\t", header=None, names=["word1_non", "word2_non", "ratio_non"])

df_bigrams = pd.concat([df_hyp_bigrams, df_non_bigrams], axis=1)
df_bigrams.to_csv("bigrams.tsv", sep="\t", index=None)
display(df_bigrams)
df_bigrams.plot()
df_words_bigrams = pd.concat([df_words, df_bigrams], axis=1)
df_words_bigrams.to_csv("words_bigrams.tsv", sep="\t", index=None)
display(df_words_bigrams)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11094095627020589, 'fraction_numerical': 0.02314105836311043, 'mean_word_length': 4.4928348909657325, 'pattern_counts': {'":': 1, '<': 31, '<?xml version=': 0, '>': 24, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 14}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '264717', 'license': 'MIT', 'n_tokens_mistral': 5850, 'n_tokens_neox': 5360, 'n_words': 1778, 'path': 'code/hyperpartisan_words_bigrams.ipynb', 'repo_name': 'juletx/hyperpartisan-news-detection', 'size': '17663'}