<jupyter_start><jupyter_text>Divide up my cognitive science and education books into subtopics based on entire document texts- Dataset: C:/Users/Arati/Documents/personal docs/cognitive science and educationall txt files (34 total)- Features: Bag of words- Method: LDA sklearn, then repeat with gensim 1. Load dataset into list of strings for each document<jupyter_code>import os
path = 'C:/Users/Arati/Documents/personal docs/cognitive science and education/'
fnames = []
rawtxt = {}
for r, d, f in os.walk(path):
    for file in f:
        if '.txt' in file:
            fnames.append(os.path.join(r, file))
            with open(os.path.join(r,file), "r",encoding = "utf-8") as myfile:
                rawtxt[file] = myfile.read()
            print('added ',file, ' to corpus.')
print('Total ',len(fnames), 'files added to corpus')<jupyter_output>added  52 Small Changes for the Mind.txt  to corpus.
added  A Mind For Numbers_ How to Excel at Math and Science (Even If You Flunked Algebra).txt  to corpus.
added  Affective Neuroscience_ The Foundations of Human and Animal Emotions (Series in Affective Science).txt  to corpus.
added  Building a Better Teacher_ How Teaching Works (and How to Teach It to Everyone).txt  to corpus.
added  Classroom Assessment Techniques_ A Handbook for College Teachers.txt  to corpus.
added  Cleverlands.txt  to corpus.
added  Essential Questions_ Opening Doors to Student Understanding.txt  to corpus.
added  Flow_ The Psychology of Optimal Experience (Harper Perennial Modern Classics).txt  to corpus.
added  Hare Brain, Tortoise Mind.txt  to corpus.
added  How the Mind Works.txt  to corpus.
added  How to Create a Mind_ The Secret of Human Thought Revealed.txt  to corpus.
added  How We Learn_ The Surprising Truth About When, Where, and Why It Happens.txt  to corpus.
added  Learn Like Einstein_ Mem<NAME>[...]<jupyter_text>Making a bag of words (frequency distribution) for each document.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
no_features= 1500
tf_vectorizer = CountVectorizer(max_df=0.99, min_df=1, max_features=no_features, stop_words='english')
tf = tf_vectorizer.fit_transform(rawtxt)
tf_feature_names = tf_vectorizer.get_feature_names()
print(tf_feature_names)<jupyter_output>['2nd', '52', 'abilities', 'accelerated', 'acquisition_', 'advanced', 'affective', 'algebra', 'animal', 'apply', 'art', 'assessment', 'better', 'blank', 'book', 'brain', 'brain_', 'break', 'building', 'changes', 'chef_', 'classics', 'classroom', 'cleverlands', 'college', 'comprehension', 'concentration', 'cooking', 'create', 'creative', 'design', 'diagrams_', 'discover', 'don_t', 'doors', 'ease', 'edition', 'einstein_', 'emotions', 'engagement', 'enhance', 'epistemology', 'essential', 'excel', 'excellence', 'expand', 'expanded', 'experience', 'expert', 'expertise', 'faster', 'flow_', 'flunked', 'focus', 'foundations', 'free', 'fulfillment', 'good', 'greater', 'guide', 'handbook', 'happens', 'hare', 'harper', 'hidden', 'hour', 'human', 'independence', 'information', 'jargon', 'journey', 'keys', 'knowledge_', 'language', 'lawrence', 'learn', 'learn_', 'learners', 'learning', 'learning_', 'life', 'like', 'living', 'logic', 'long', 'magic', 'make', 'makes', 'making', 'master', 'mastery_', [...]<jupyter_text>LDA for cognitive science and education books<jupyter_code>no_topics = 3
from sklearn.decomposition import LatentDirichletAllocation
lda = LatentDirichletAllocation(n_components=no_topics, max_iter=5, learning_method='online', learning_offset=50.,random_state=0).fit(tf)
lda
def display_topics(model, feature_names, no_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic",topic_idx)
        print(" ".join([feature_names[i] for i in topic.argsort()[:-no_top_words - 1:-1]]))

no_top_words = 10

display_topics(lda, tf_feature_names, no_top_words)
Y = lda.fit_transform(tf)
for i in range(len(fnames)):
    print('Book: ',fnames[i])
    print('Topic distribution: ',Y[i,:])<jupyter_output>Book:  C:/Users/Arati/Documents/personal docs/cognitive science and education/52 Small Changes for the Mind.txt
Topic distribution:  [0.07275846 0.0769751  0.85026644]
Book:  C:/Users/Arati/Documents/personal docs/cognitive science and education/A Mind For Numbers_ How to Excel at Math and Science (Even If You Flunked Algebra).txt
Topic distribution:  [0.0462383  0.90680906 0.04695264]
Book:  C:/Users/Arati/Documents/personal docs/cognitive science and education/Affective Neuroscience_ The Foundations of Human and Animal Emotions (Series in Affective Science).txt
Topic distribution:  [0.03689184 0.92558741 0.03752075]
Book:  C:/Users/Arati/Documents/personal docs/cognitive science and education/Building a Better Teacher_ How Teaching Works (and How to Teach It to Everyone).txt
Topic distribution:  [0.05297766 0.89369916 0.05332318]
Book:  C:/Users/Arati/Documents/personal docs/cognitive science and education/Classroom Assessment Techniques_ A Handbook for College Teachers.txt
Topic dis[...]
{'fraction_non_alphanumeric': 0.12554715479506565, 'fraction_numerical': 0.026661360923199363, 'mean_word_length': 5.511658031088083, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '251690', 'license': 'MIT', 'n_tokens_mistral': 1743, 'n_tokens_neox': 1563, 'n_words': 505, 'path': 'LDA/LDA_cognitive_science_and_education_books.ipynb', 'repo_name': 'rts1988/IntelligentTutoringSystem_Experiments', 'size': '5029'}