<jupyter_start><jupyter_text>Finding (Problem Statement) Signal in Sentence PhrasingHere we look at intersecting (low signal) and exclusive (only in one class aka high signal) [n-grams](https://en.wikipedia.org/wiki/N-gram) of positive and negative labeled sentences.If there are n-grams that *almost* exclusively appear in one class (0 or 1) then they could make great matching phrases either directly for classifying a sentence without using a machine learning model or just to do last-mile quality assurance and flag suspicious model decisions<jupyter_code># default_exp core
import pandas as pd
df = pd.read_csv("downloads/40k_balanced_pm_acl.csv")

#export

def take_while(fn, coll):
    """Yield values from coll until fn is False"""
    for e in coll:
        if fn(e):
            yield e
        else:
            return

def partition(n, coll, step=None):
    return take_while(lambda e: len(e) == n,
        (coll[i:i+n] for i in range(0, len(coll), step or n)))

def partition_all(n, coll, step=None):
    return (coll[i:i+n] for i in range(0, len(coll), step or n))

def n_grams(texts, n_gram=2): return [" ".join(n) for t in texts for n in partition(n_gram, t.split(" "), 1)]
#export

import spacy
nlp = spacy.load("en_core_web_md")

def lemmatize(text, nlp=nlp):
    return " ".join([tok.lemma_ for tok in nlp(text)])
df["text_orig"] = df["text"]
df["text"] = df["text"].map(lemmatize)
df["labels"] = df["labels"].astype(str)
positives = df[df["labels"] == "1"]
negatives = df[df["labels"] == "0"]
positives[:3]
bi_grams_pos = n_grams(positives["text"], 2)
tri_grams_pos = n_grams(positives["text"], 3)
bi_grams_neg = n_grams(negatives["text"], 2)
tri_grams_neg = n_grams(negatives["text"], 3)
bi_grams_pos[:3]
from collections import Counter

d1 = dict(Counter(bi_grams_pos))
d2 = dict(Counter(bi_grams_neg))

d3 = dict(Counter(tri_grams_pos))
d4 = dict(Counter(tri_grams_neg))

bi_grams_both = {x:(d1[x], d2[x]) for x in d1 if x in d2}

tri_grams_both = {x:(d3[x], d4[x]) for x in d3 if d4.get(x)}
bi_grams_pos_only = {x:d1[x] for x in d1 if not d2.get(x)} # and d2.get(x) < 5
tri_grams_pos_only = {x:d3[x] for x in d3 if not d4.get(x)}
[(k,v) for k,v in Counter(tri_grams_both).items() if (v[0] + v[1]) > 20 ]
#n-gram: (pos, neg)
[(k,v) for k,v in Counter(bi_grams_both).items() if (v[0] + v[1]) > 100]<jupyter_output><empty_output><jupyter_text>Let's look at patterns that one class shows very rarely relative to the other **TODO: only show bigrams that weren't also in the SPIKE SEARCH PATTERN**<jupyter_code>factor = 15
#show n-grams that appear at least 20x more often in positive samples
[(k,v) for k,v in Counter(bi_grams_both).items() if (v[0]/v[1]) > factor], [(k,v) for k,v in Counter(tri_grams_both).items() if (v[0]/v[1]) > factor/2]<jupyter_output><empty_output><jupyter_text>Apparently "problem of" appears 54 in "problem statements" and only once in "non-problem" statement. Let's look at that one sentence!<jupyter_code>negatives[negatives["text"].str.contains('limitation of')].reset_index()["text"][4]
bgs = [(k,v) for k,v in Counter(bi_grams_pos_only).items() if v > 50]
tgs = [(k,v) for k,v in Counter(tri_grams_pos_only).items() if v > 30]
bgs, tgs
with open("ngrams_pos_only_lemmatized.txt", "w") as f:
    f.write("\n".join([t[0] for t in bgs+tgs]))
#trigrams that are not supersets of bigrams
#[bigram for bigram in bi_grams_pos_only.keys() if not any(bigram in tg for tg in tri_grams_pos_only.keys())]

texts = df[df["text"].str.contains("drawback be that")].reset_index()["text"]
texts
#!find downloads -maxdepth 1 -type f -exec du -h {} + | sort --human-numeric-sort --reverse<jupyter_output>128M	downloads/data.json
108M	downloads/sample_4k.json
28M	downloads/sample_1k.json
132K	downloads/oct7_test300.csv
{'fraction_non_alphanumeric': 0.14769394828045854, 'fraction_numerical': 0.021327645961077047, 'mean_word_length': 5.171052631578948, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '57595', 'license': 'Apache-2.0', 'n_tokens_mistral': 1489, 'n_tokens_neox': 1386, 'n_words': 413, 'path': '02_text_patterns.ipynb', 'repo_name': 'ScienceOctopus/octopus-core', 'size': '3751'}