<jupyter_start><jupyter_text>Read Data The [Toxic Commments Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) was a competition to classify comments into various categories of toxicity.In this tutorial, we will see how CMF can extract relevant topics to classification using the toxicity labels as supervision, and how that cannot be accomplished using other topic modeling techniques.Before running this code, you'll need to download the data from [here](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data) and put it in a directory of your choice (make sure the below DATA variable points to the directory where the data resides).<jupyter_code>DATA = Path("data")
train = pd.read_csv(DATA / "train.csv")<jupyter_output><empty_output><jupyter_text>Let's take a look at the data<jupyter_code>train.head(10)
train.describe()<jupyter_output><empty_output><jupyter_text>Preprocess Data We will now construct the text data to do topic modeling on<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
vectorizer = CountVectorizer(min_df=3, max_df=0.9, stop_words="english", binary=True)
%%time
X_train = vectorizer.fit_transform(train.head(10000).comment_text)
X_train.shape<jupyter_output><empty_output><jupyter_text>The label data to decompose<jupyter_code>from scipy.sparse import csc_matrix, csr_matrix
label_columns = list(train.columns[2:]); label_columns
Y_train = train.head(10000)[label_columns].values
Y_train.shape<jupyter_output><empty_output><jupyter_text>Attempt Decomposition<jupyter_code>N_COMPONENTS = 20<jupyter_output><empty_output><jupyter_text>Analysis functions we'll use later<jupyter_code>idx_to_word = np.array(vectorizer.get_feature_names())
def print_topics(H, topn=10):
    for i, topic in enumerate(H): 
        print("Topic {}: {}".format(i + 1, ",".join([str(x) for x in idx_to_word[topic.argsort()[-topn:]]])))<jupyter_output><empty_output><jupyter_text>NMF Let's see what topics we find using NMF<jupyter_code>from sklearn.decomposition import NMF
nmf = NMF(N_COMPONENTS)
%%time
W = nmf.fit_transform(X_train); H = nmf.components_
print_topics(H)<jupyter_output>Topic 1: used,right,case,said,point,make,fact,way,say,does
Topic 2: tildes,hello,questions,date,welcome,sign,using,editing,pages,help
Topic 3: placed,speedily,guidelines,tag,add,remove,speedy,note,deleted,deletion
Topic 4: message,added,comments,user,wp,comment,discussion,redirect,contribs,talk
Topic 5: pov,history,subject,discussion,good,main,created,needs,section,article
Topic 6: user,said,want,doesn,say,care,understand,need,really,don
Topic 7: considered,editors,users,http,stop,policy,en,wiki,org,wikipedia
Topic 8: bit,new,sounds,hey,feel,thing,really,things,looks,like
Topic 9: let,editor,right,mean,ll,really,going,wanted,say,just
Topic 10: say,wiki,dont,got,thing,didn,want,did,let,know
Topic 11: vandalize,make,stop,pages,continue,edits,editing,blocked,did,edit
Topic 12: post,read,message,delete,links,new,link,discussion,user,page
Topic 13: life,way,really,wiki,person,going,group,world,want,people
Topic 14: way,added,new,lot,ll,work,help,good,hi,thanks
Topic 15: right,way,section,su[...]<jupyter_text>None of these topics seems related to toxicity... The hyperparameter alpha determines the trade-off between how much to value the decomposition of one matrix over the other when computing the loss. It only affects the results when using the newton solver.We'll use a very heuristic method of computing alpha here.<jupyter_code>xnorm = np.sqrt(X_train.multiply(X_train).sum())
ynorm = np.sqrt((Y_train * Y_train).sum())
alpha = (ynorm / (xnorm + ynorm)) ** 0.75; alpha<jupyter_output><empty_output><jupyter_text>CMF with non-negative constraint on all matrices<jupyter_code>cmf_nn = pycmf.CMF(N_COMPONENTS,
                   U_non_negative=True, V_non_negative=True, Z_non_negative=True,
                   alpha=alpha, solver="mu", verbose=True)
%%time
U, V, Z = cmf_nn.fit_transform(csr_matrix(X_train.T), Y_train)
cmf_nn.print_topic_terms(vectorizer)<jupyter_output>Topic 1 [0.000,0.000,0.000,0.000,0.000,0.000]: right,said,case,used,point,fact,way,make,say,does
Topic 2 [0.000,0.000,0.000,0.002,0.000,0.000]: date,hello,ask,welcome,sign,questions,using,editing,pages,help
Topic 3 [0.000,0.000,0.000,0.000,0.000,0.000]: information,free,articles,tag,add,criteria,note,speedy,deletion,deleted
Topic 4 [0.000,0.000,0.000,0.000,0.000,0.000]: remove,message,comments,comment,wp,user,redirect,discussion,contribs,talk
Topic 5 [0.000,0.000,0.000,0.000,0.000,0.000]: version,history,wp,created,pov,main,needs,discussion,section,article
Topic 6 [6.421,0.970,4.473,0.233,4.192,0.721]: dick,life,suck,bitch,stop,ass,stupid,shit,fucking,fuck
Topic 7 [0.090,0.000,0.000,0.000,0.000,0.000]: vandalism,users,editors,stop,wiki,en,policy,org,articles,wikipedia
Topic 8 [0.017,0.000,0.000,0.000,0.000,0.000]: hey,bit,really,thing,new,sounds,look,things,looks,like
Topic 9 [0.004,0.000,0.000,0.000,0.000,0.000]: hi,got,didn,mean,ll,going,let,say,wanted,just
Topic 10 [0.001,0.000,0.00[...]<jupyter_text>The results seem better(?). Now we see some topics that are actually related to toxicity. Let's see what happens when we allow negativity.  CMF with non-negative constraint on only the topic matrices<jupyter_code>cmf = pycmf.CMF(N_COMPONENTS,
               U_non_negative=True, V_non_negative=True, Z_non_negative=False,
               x_link="linear", y_link="logit", alpha=alpha, l1_reg=2., l2_reg=5., max_iter=10,
               solver="newton", verbose=True)
%%time
U, V, Z = cmf.fit_transform(csr_matrix(X_train.T), Y_train)
cmf.print_topic_terms(vectorizer)<jupyter_output>Topic 1 [-0.047,-0.323,-0.057,-0.469,-0.015,-0.257]: hate,little,hell,life,hey,know,shit,ass,fucking,fuck
Topic 2 [0.139,-3.046,-2.502,-3.067,-2.442,-3.153]: say,wikipedia,like,articles,know,just,time,think,people,don
Topic 3 [-1.022,-0.368,0.075,-0.461,-0.314,-0.382]: non,ask,material,read,reason,article,page,want,wikipedia,deleted
Topic 4 [-5.596,-5.963,-4.806,-6.224,-5.267,-6.076]: don,confirm,accepted,companies,indicate,year,bands,company,tagging,club
Topic 5 [-6.810,-4.408,-6.029,-3.981,-5.526,-4.095]: way,page,wikipedia,right,talk,user,use,section,know,article
Topic 6 [-0.698,-0.271,-0.237,-0.501,-0.341,-0.307]: editors,way,talk,does,article,make,wikipedia,page,questions,like
Topic 7 [-4.465,-3.540,-4.501,-3.125,-4.039,-3.068]: edit,really,page,article,just,think,like,people,know,don
Topic 8 [0.516,0.147,0.988,-0.568,0.920,0.194]: gets,avoid,stupid,right,man,suck,nazi,bitch,faggot,fuck
Topic 9 [2.863,-1.721,1.025,-1.698,0.904,-1.538]: contributing,create,takes,learn,tests,policy,[...]
{'fraction_non_alphanumeric': 0.14913329348475793, 'fraction_numerical': 0.07680812910938434, 'mean_word_length': 7.515267175572519, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '648660', 'license': 'MIT', 'n_tokens_mistral': 2810, 'n_tokens_neox': 2447, 'n_words': 422, 'path': 'samples/toxic_comments.ipynb', 'repo_name': 'smn-ailab/PyCMF', 'size': '6692'}