<jupyter_start><jupyter_text>In this notebook I will explore some of the results obtained by running the `run_lda.py` file!There are 3 main things that I do:- I take the names of those twitter handles which LDA couldnt not find a substantive 'topic(s)' for. That is to say, these twitter hadnles have been inactive for a while, have tweets that are extrmemely scattered or just simply do not have enough tweets for any analysis to be carried out. I place the names of these twitter users in a pickled list called `LDA_identified_bad_handles.pkl`.- I take the names of these bag handles and remove them from the list containing all the handles of the 2nd degree connnections we started with. I do this as I will need a 'pure' list of handles when running our tf-idf analysis. I will only be running tf-idf analysis on those twitter users that have valid LDA results. I place this list of valid hadnles in a pickle file called `verified_handles_lda.pkl`- I delete the the 'bad handle' dictionaries from our list of dictionaries. I pickle the resulting list into a file called `final_database_lda_verified.pkl`<jupyter_code>%run helper_functions.py
lst = unpickle_object("2nd_degree_connections_LDA_complete.pkl")
lst[8] #an example of a bad handle dictionary
handle_names = []
for dictionary in lst:
    name = list(dictionary.keys())
    handle_names.append(name)
handle_names = sum(handle_names, [])
#an example of me finding which user's in my LDA results tweet about "machine" --> alluding to "machine learning"
cnt = -1

for handle in handle_names:
    cnt +=1
    try:
        topics = lst[cnt][handle]['LDA']
        
        if "machine" in topics:
            print(handle)
    except:
        pass
# handles to be removed as they do not have valid LDA analysis
handle_to_remove = []
cnt = -1

for handle in handle_names:
    cnt += 1
    sub_dict = lst[cnt][handle]
    
    if "LDA" not in sub_dict:
        handle_to_remove.append(handle)

indicies = []

for handle in handle_to_remove:
    index = handle_names.index(handle)
    indicies.append(index)
#extracting the valid LDA handle
verified_handles_lda = [v for i,v in enumerate(handle_names) if i not in frozenset(indicies)]
handle_to_remove[:5] #a peek at the 'bad handles'
pickle_object(verified_handles_lda, "verified_handles_lda")
pickle_object(handle_to_remove, "LDA_identified_bad_handles")
#extracting the appropriate dictionaries to be used in TF-IDF analysis
final_database_lda_verified = [v for i,v in enumerate(lst) if i not in frozenset(indicies)]
pickle_object(final_database_lda_verified, "final_database_lda_verified")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06735159817351598, 'fraction_numerical': 0.003424657534246575, 'mean_word_length': 4.523109243697479, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '177462', 'license': 'MIT', 'n_tokens_mistral': 813, 'n_tokens_neox': 795, 'n_words': 326, 'path': '04-Project-Fletcher/Phases/Phase_3/LDA_result_exploration.ipynb', 'repo_name': 'igabr/Metis_Projects_Chicago_2017', 'size': '2628'}