<jupyter_start><jupyter_text>Neural Networks with Sentdex - Processing your own data: Create Sentiment Feature Sets---We are going to use 'our own data set' which SentDex has store at https://pythonprogramming.net/static/downloads/machine-learning-data/pos.txtandhttps://pythonprogramming.net/static/downloads/machine-learning-data/neg.txtEach file is a list of 5000+ strings with different string lengths- We need to convert to numerical values to be processes like a dictionary format- DNNs (at the level) need to all be the same size    - there are DNNs that can process data of variable length and size    - such as sequence-to-sequence stuck -- don't have to be the same length    - and/or use bucketing and suchWe are going to use these libraries to generate a Lexicon -- or set of words that 'you' know- We are going to use the 'bag of words' model, where each word has an ID (here index)- Then we will incorporate this "not necessarilly `one hot`; but you can call it `hot` array or something" Simple Example of a Lexicon---Imagine that we only have 4 words in our lexicon -- [chair, table, spoon, television]- then we have a new sentence "I pulled the chair up to the table"- that is now going to have to be converted by the index that each element shows up in the lexicon:start with: value = [0 0 0 0]loop over the sentence and ask 1. "Does `I` exist in the table?" - No2. "Does `pulled` exist in the table?" - No3. "Does `the` exist in the table?" - No4. "Does `Chair` exist in the table?" - Yes  --> value = [1 0 0 0]5. "Does `up` exist in the table?" - No6. "Does `to` exist in the table?" - No7. "Does `the` exist in the table?" - No8. "Does `table` exist in the table?" - Yes --> value = [1 1 0 0]Convert this to a vector in the bag of words model, then `value = [1 1 0 0]` will become the vector for this sentence above.<jupyter_code>import nltk
nltk.download()
import nltk
from nltk.tokenize import word_tokenize<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08626850433894845, 'fraction_numerical': 0.014803471158754467, 'mean_word_length': 3.8514851485148514, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '359334', 'license': 'MIT', 'n_tokens_mistral': 584, 'n_tokens_neox': 552, 'n_words': 305, 'path': 'NeuralNetworks/Neural Networks with Sentdex - Processing your own data - Practical Machine Learning Part 48.ipynb', 'repo_name': 'exowanderer/PythonProgramming_PracticalMachineLearning', 'size': '1959'}