<jupyter_start><jupyter_text>Tokenizing Sentences<jupyter_code>allsentences = ['<NAME>. — For a brief moment, former Vice President <NAME> placed himself in the crowded field of Democratic presidential candidates during a speech on Saturday night. Biden seemingly described himself as one of the people “running” while noting he gets criticized by some progressives.',

'I have the most progressive record of anybody running for the United—” Biden said, then quickly corrected himself to say, “of anybody who would run. I didn’t mean it! Of anybody who would run”',

'The former vice president’s speech was the keynote of the First State Democratic Dinner at the Dover Downs Hotel and Casino. Biden’s appearance in his home state came amid mounting rumors and speculation he will enter the 2020 presidential race.',

'Asked about the “running” remark, Biden spokesperson <NAME> pointed towards the “second half of the sentence” in which Biden corrected himself. During his career, Biden has earned a reputation for verbal misfires. A volunteer for the former vice president who was in the ballroom for the speech described the comment as a mistake.',

'He did not announce. That was the slip of the tongue,” said the volunteer, who declined to be identified by name.',

'As he has inched closer toward entering the presidential race, Biden has faced criticism from progressives who believe he is too moderate. Critics have pointed to his tough stance on crime when he served in the U.S. Senate, for instance. Biden has also faced backlash for a speech he gave last month in which he called the current vice president, <NAME>, a “decent guy.',

'Biden’s speech on Saturday was in many ways unapologetically moderate. He noted his home state has become known for “the Delaware way” where political rivals don’t “demonize” or “belittle” each other. Biden also said Delaware Democrats “might even say a nice word about a Republican when they do something good.” Biden suggested America should have more of this type of “consensus” on the national stage.',

'We need a little more of the Delaware way,” Biden said, adding that our current politics is “so mean,” “petty,” and “vicious.”',

'Despite his largely conciliatory tone, Biden offered an aggressive critique of President Trump and said “our Democracy is under threat” by the current administration. Specifically, Biden blasted Trump’s 2017 response to the Neo Nazi rallies in Charlottesville, Va., as well as the president’s attacks on the “free press,” Congress, and the “independent judiciary.',

'The president is systematically tearing down the guardrails of our democracy,” Biden said.',

'Biden also criticized Trump for endangering what he described as “the bargain that built the middle class” with his tax policies and cuts to social programs.',

'They’ve made a basic value judgement they’d rather reward the super wealthy and hope something trickles down than help anyone else,” Biden said of Republicans.',

'As he leveled these criticisms, Biden once again seemed to hint at a coming campaign.',

'You’re going to hear a hell of a lot more about it from me,” said Biden.',

'Even as he declared there is a raging “battle for the soul of America,” Biden struck a hopeful tone and repeatedly said this country has a higher “capacity” for greatness than any other.',

'As tough as things are right now, I’m optimistic, he said. It’s not about me and whether I run or not — and I mean that sincerely — it’s about the capacity to restore America and once again become that shining city on the hill.',

'While Biden stopped just short of launching a presidential bid, his fellow Delaware Democrats all seemed primed for him to enter the race. As he took the stage, Biden was greeted with chants of “Run Joe! Run!” Speaking to reporters before the speech, Delaware Sen. <NAME> indicated that, whether the night included an announcement, he expects Biden will throw his hat into the ring.'

'I am very optimistic that <NAME> will announce soon. I don’t know about tonight … I would be surprised, but it’s possible that the energy and passion of the room will lead him to do something bold and unexpected here,” Coons said, adding, “I do think he is all but certain to run. … He is 95 percent there.']
def word_extraction(sentence):
    p = set(stopwords.words('english'))
    ignore = list(p)
    words = re.sub("[^\w]", " ",  sentence).split()
    cleaned_text = [w.lower() for w in words if w not in ignore]
    return cleaned_text
def tokenize(sentences):
    words = []
    for sentence in sentences:
        w = word_extraction(sentence)
        words.extend(w)
        
    words = sorted(list(set(words)))
    return words
p= tokenize(allsentences)
print("Word List for Document \n{0} \n".format(p))
for sentence in allsentences:
    words = word_extraction(sentence)
    p= tokenize(allsentences)
    bag_vector = np.zeros(len(p))
    for w in words:
        for i,word in enumerate(p):
            if word == w: 
                bag_vector[i] += 1
    print("{0}\n{1}\n".format(sentence,np.array(bag_vector)))<jupyter_output>DOVER, Del. — For a brief moment, former Vice President <NAME> placed himself in the crowded field of Democratic presidential candidates during a speech on Saturday night. Biden seemingly described himself as one of the people “running” while noting he gets criticized by some progressives.
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 2. 0. 0. 0. 1. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. 0. 0. 1. 0. 0. 1. 0.
 0. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 1. 0. 0. 1.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 0. 0. 1. 0. 0. 0. 1. 0. 0. 1. 0.
 0. 0. 0. 0. 1. 1. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. [...]<jupyter_text>Build vocabulary and generate vectors<jupyter_code>def generate_bow(allsentences):    
    for sentence in allsentences:
        words = word_extraction(sentence)
        p= tokenize(allsentences)
        bag_vector = np.zeros(len(p))
        for w in words:
            for i,word in enumerate(p):
                if word == w: 
                    bag_vector[i] += 1
        print("{0}\n{1}\n".format(sentence,np.array(bag_vector)))
generate_bow(allsentences)
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(allsentences)
print(X.toarray())
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
import numpy as np
import pandas as pd
import re

cvec = CountVectorizer(stop_words='english', min_df=3, max_df=0.5, ngram_range=(1,2))
sf = cvec.fit_transform(allsentences)
transformer = TfidfTransformer()
transformed_weights = transformer.fit_transform(sf)
weights = np.asarray(transformed_weights.mean(axis=0)).ravel().tolist()
weights_df = pd.DataFrame({'term': cvec.get_feature_names(), 'weight': weights})

weights_df.sort_values(by='weight', ascending=False).head(50)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09879253567508232, 'fraction_numerical': 0.03526344676180022, 'mean_word_length': 3.3002949852507375, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '565237', 'license': 'MIT', 'n_tokens_mistral': 2469, 'n_tokens_neox': 2156, 'n_words': 1112, 'path': 'Uncovering Sentiments using EDGAR Datasets/Bag of Words Experiment byron.ipynb', 'repo_name': 'RajeshreeKale/PROJECTS', 'size': '7314'}