<jupyter_start><jupyter_text>Raw text classification with BOW<jupyter_code># Load for Jupyter Notebook
import sys
sys.path.append('/home/elenaruiz/Documents/TFG/FNC')
import pandas as pd
import numpy as np
import gensim
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
from gensim import corpora, models

from sklearn.model_selection import train_test_split

from src.utils import io
from src.fake_news_detector.core.nlp import clean_text as ct<jupyter_output><empty_output><jupyter_text>1. Load dataset (tmp.json)The file `tmp.json` contains all raw data that has to be tokenize<jupyter_code>articles = io.read_json_file('/home/elenaruiz/Documents/TFG/FNC/src/data/tmp.json')
df = pd.DataFrame(data=articles['articles']) # Put in pandas dataframe
df.columns<jupyter_output><empty_output><jupyter_text>2. Join and clean raw textTokenize in words. For each article store list of Tokens<jupyter_code>corpus = []
i = 0
for _, row in df.iterrows():
    x = ct.clean_text_by_word(row['title'], True)
    y = ct.clean_text_by_word(row['subtitle'], True)
    z = []
    for sent in row['text']:
        z += ct.clean_text_by_word(sent, True)
    corpus.append(x + y + z)
    i = i + 1
len(corpus)
#Store in a dataframe
documents = pd.DataFrame(data={'corpus': corpus, 'label': df['fake']*1 })
documents.head()<jupyter_output><empty_output><jupyter_text>3. Bag of WordsCreate dictionary<jupyter_code>dictionary = gensim.corpora.Dictionary(documents['corpus'])
count = 0
for k, v in dictionary.iteritems():
    print(k, v)
    count += 1
    if count > 10:
        break
# Se quitan los menos usados
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)<jupyter_output><empty_output><jupyter_text>4. Vectorizar el contenidoRealizar proceso de doc2bow:<jupyter_code>bow_corpus = [dictionary.doc2bow(doc) for doc in documents['corpus']]

# EXAMPLE
bow_corpus[30]
bow_doc_30 = bow_corpus[30]

for i in range(len(bow_doc_30)):
    print("Word {} (\"{}\") appears {} time.".format(bow_doc_30[i][0], 
                                                     dictionary[bow_doc_30[i][0]], 
                                                     bow_doc_30[i][1]))<jupyter_output>Word 2 ("find") appears 1 time.
Word 5 ("he") appears 1 time.
Word 7 ("like") appears 1 time.
Word 8 ("make") appears 2 time.
Word 13 ("say") appears 4 time.
Word 16 ("time") appears 2 time.
Word 19 ("accord") appears 2 time.
Word 23 ("case") appears 3 time.
Word 32 ("seem") appears 1 time.
Word 46 ("it") appears 4 time.
Word 50 ("explain") appears 1 time.
Word 51 ("many") appears 1 time.
Word 52 ("people") appears 1 time.
Word 54 ("this") appears 1 time.
Word 55 ("years") appears 2 time.
Word 60 ("could") appears 1 time.
Word 62 ("point") appears 1 time.
Word 63 ("that") appears 1 time.
Word 64 ("another") appears 2 time.
Word 68 ("even") appears 1 time.
Word 69 ("take") appears 3 time.
Word 75 ("European") appears 2 time.
Word 81 ("we") appears 2 time.
Word 91 ("go") appears 1 time.
Word 95 ("still") appears 1 time.
Word 98 ("become") appears 1 time.
Word 100 ("next") appears 1 time.<jupyter_text>TF-IDFRealizar proceso de ordenar por frecuencias<jupyter_code>tfidf = models.TfidfModel(bow_corpus)
corpus_tfidf = tfidf[bow_corpus]

from pprint import pprint

for doc in corpus_tfidf:
    pprint(doc)
    break
### 5. LDA con BOW
lda_model = gensim.models.LdaMulticore(bow_corpus, num_topics=10, id2word=dictionary, passes=2, workers=2)
for idx, topic in lda_model.print_topics(-1):
    print('Topic: {} \nWords: {}'.format(idx, topic))<jupyter_output>Topic: 0 
Words: 0.053*"say" + 0.040*"I" + 0.037*"police" + 0.035*"right" + 0.028*"people" + 0.024*"know" + 0.024*"another" + 0.022*"country" + 0.021*"call" + 0.021*"case"
Topic: 1 
Words: 0.044*"use" + 0.037*"already" + 0.036*"point" + 0.033*"public" + 0.033*"they" + 0.029*"come" + 0.027*"consider" + 0.025*"way" + 0.025*"many" + 0.024*"in"
Topic: 2 
Words: 0.045*"say" + 0.041*"police" + 0.036*"euros" + 0.033*"would" + 0.028*"first" + 0.027*"in" + 0.024*"use" + 0.022*"new" + 0.021*"find" + 0.021*"two"
Topic: 3 
Words: 0.049*"in" + 0.045*"use" + 0.041*"make" + 0.039*"first" + 0.034*"open" + 0.033*"euros" + 0.031*"go" + 0.029*"since" + 0.028*"new" + 0.026*"it"
Topic: 4 
Words: 0.048*"would" + 0.045*"time" + 0.040*"could" + 0.035*"end" + 0.033*"say" + 0.029*"years" + 0.026*"company" + 0.025*"case" + 0.023*"European" + 0.023*"want"
Topic: 5 
Words: 0.053*"it" + 0.030*"time" + 0.028*"take" + 0.026*"they" + 0.025*"years" + 0.024*"a" + 0.024*"go" + 0.023*"people" + 0.022*"say" + 0.021*"he"
To[...]
{'fraction_non_alphanumeric': 0.16739894551845344, 'fraction_numerical': 0.0819420035149385, 'mean_word_length': 4.2213302752293576, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '423850', 'license': 'Apache-2.0', 'n_tokens_mistral': 2022, 'n_tokens_neox': 1691, 'n_words': 447, 'path': 'src/fake_news_detector/notebooks/raw_text_bow.ipynb', 'repo_name': 'elena20ruiz/FNC', 'size': '4552'}