<jupyter_start><jupyter_text>EDA-01 Questions:1. What rules did you put in place to refine/clean the data?2. What number of duplicates occurred?3. What amount of data was nonsensical?4. What number of unique posts/posters occurred?5. Is there surprising missingness or sparsity (e.g. by time, date, location, etc.)  Field	Description1. **tweet_id**: Unique identifier of the tweet2. **created_at**:	Time stamp of when the tweet was sent. 3. **full_text**:	The text of the tweet 4. **geo**	Geo tag of the tweet. Will only be included if user has location services enabled. 5. **coordinates** 	Latitude and Longitude of the tweet. Will only be included if the user has location services enabled. 6. **place**	Twitter identified place. Will only be included if the user has location services enabled. 7. **retweet_count**	The number of retweets the tweet received 8. **favorite_count**	The number of favorites the tweet received 9. **possible_sensitive**	Twitter flag if the tweet contains sensitive material. This is done by Twitter. If the tweet did not have this tagged by Twitter, we imputed NOT FOUND10. **lang**	The language of the tweet. We filtered towards English 'en' 11. **user_id**	The unique identiifer of the person or entity sending the tweet Data Collection Methodology The twitter public API allows for a query based keyword search to return 100 random samples of tweets containing keywords from the previous 7 days.Data was scraped from February 23 - March 3 searching for either a combination of a Covid-19 related keyword and a vaccine related key word, or just containing a hashtag keyword.The queries were randomly generated with the above logic in order to obtain unique tweets per API request.<jupyter_code>import os 
# set your cwd
my_cwd = "/home/mriveralanas/projects/Mar21-vaccine-uptake"
os.chdir(my_cwd)
# import data libraries
import pandas as pd
pd.options.mode.chained_assignment = None  
import datetime
from collections import Counter

# nlp
import re
import nltk
import numpy as np
from sklearn.cluster import AffinityPropagation
import distance
import unicodedata as ud
from langdetect import detect 

import time

# https://stackoverflow.com/questions/3094498/how-can-i-check-if-a-python-unicode-string-contains-non-western-letters

latin_letters= {}

def is_latin(uchr):
    try: 
        return latin_letters[uchr]
    except KeyError:
         return latin_letters.setdefault(uchr, 'LATIN' in ud.name(uchr))

def only_roman_chars(unistr):
    return all(is_latin(uchr) for uchr in unistr if uchr.isalpha())

def test_hashtag(unistr):    
    unistr_strip = list(filter(lambda x: only_roman_chars(x), unistr))
    return len(unistr_strip)>.5*len(unistr)

def clean_hashtag(unistr):    
    unistr = list(map(lambda x: x if only_roman_chars(x) else "", unistr))
    return unistr

#? ----- Affinity Propagation grouping
def hashtag_group(grouping, hashtag):
    hashtag_lower = hashtag.lower()
    hashtag_clean = re.sub(r'[^a-zA-Z0-9]', '', hashtag_lower)
    for key, value in grouping.items():
        if hashtag_clean in value:
            return key
        elif hashtag_clean == key:
            return key
    # error
    print(hashtag_clean)
    raise ValueError
    

import matplotlib.pyplot as plt

large = "22"
med = "18"
small = "16"
params = {"axes.titlesize": large, 
        "legend.fontsize":large,
        "figure.figsize":(20,8), 
        "axes.labelsize":med,
        "axes.titlesize":med,
        "xtick.labelsize":med,
        "ytick.labelsize":med,
        "figure.titlesize":large
        }
plt.rcParams.update(params)
plt.style.use('seaborn-darkgrid')

def plot_count_by(data, metric, groupby):

    data_groupby = data[[groupby,metric]].groupby([groupby])[metric].count().reset_index()

    x = data_groupby[groupby]
    y = data_groupby[metric]

    x_pos = [i for i, _ in enumerate(x)]
    plt.bar(x_pos, y, color='green')
    plt.xlabel("{} y {}".format(metric, groupby))
    plt.ylabel(metric)
    plt.title(groupby)

    plt.xticks(x_pos, x)

    plt.show()
    return None


# pull local twitter_text
raw_text = pd.read_csv("data/twitter_gdrive/twitter_text.csv")
raw_text.head()
# clean text data

def convert_date(d):
    if pd.isnull(d):
        return "NOT FOUND"
    else:
        new_d = datetime.datetime.strptime(d, "%a %b %d %H:%M:%S +0000 %Y").strftime("%Y-%m-%d")
        return new_d

raw_text['date'] = list(map(lambda d: convert_date(d), raw_text['created_at']))

# pull local twitter_user
raw_user = pd.read_csv("data/twitter_gdrive/twitter_user.csv")
raw_user.head()
# pull local twitter_hashtag
raw_hashtag = pd.read_csv("data/twitter_gdrive/twitter_hashtag.csv")

# convert tweet id to string
raw_hashtag['tweet_id'] = list(map(lambda x: str(x), raw_hashtag['tweet_id']))
raw_hashtag.head()
# # non english hashtags:
# {"中共病毒":"CCP Virus", # on twitter seems to linked to anti-CCP sentiment
#  "上本町わたなべクリニック":"",<jupyter_output><empty_output><jupyter_text>What is the Tweet location distribution over the 7 day window?<jupyter_code>tweet_time = raw_text[['tweet_id', 'date']]
plot_count_by(tweet_time, metric='tweet_id', groupby='date')<jupyter_output><empty_output><jupyter_text>Tweet ID by lang<jupyter_code>raw_text_lang = raw_text[['tweet_id', 'lang']]
raw_text_eng = raw_text_lang[list(map(lambda x: x == "en", raw_text_lang['lang']))]
raw_text_otherlang = raw_text_lang[list(map(lambda x: x != "en", raw_text_lang['lang']))]
# non english tweets
plot_count_by(raw_text_otherlang, metric='tweet_id', groupby='lang')<jupyter_output><empty_output><jupyter_text>Analysis of Hashtags- [Difficulty with time dependent, not purely lexical relationship among hashtags](https://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00277)    - Unlike other tagging systems (e.g., Flickr1), hashtags in Twitter are used more as a symbol of community membership than for organizing a message content, as remarked in Huang, Thornton, and Efthimiadis (2010).    - Together, these problems reduce the effectiveness of Twitter hashtags as a means both for tracing users’ interests (owning to obscurity and sense shifts), and for capturing the worldwide impact of trending topics (because of synonymy and multilinguality).    - Despite this, however, better methods for analyzing the semantics of hashtags are most definitely needed, since hashtags are readily available, whereas textual analysis techniques, when applied on large and lengthy micro-blog streams, are limited both by complexity constraints and by the very reduced dimension of micro-blog texts (140 characters). What is more, real-time detection of sense-related hashtags could be used to improve the task of hashtag recommendation, thereby facilitating the monitoring of online discussions    - The solution proposed in this article is to use an algorithm for hashtag sense clustering based on temporal co-occurrence and similarity of the related time series, named SAX*    - Ferragina, Piccinno, and Santoro (2015).    - **The underlying idea of SAX* is that hashtags with similar temporal behavior are semantically related**            - [Clustering using Edit distance, lexical similarity](https://stats.stackexchange.com/questions/123060/clustering-a-long-list-of-strings-words-into-similarity-groups)- [Affinity Propagation](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AffinityPropagation.htmlsklearn.cluster.AffinityPropagation)<jupyter_code># --------- cleaning

# limit to english lang tweets
hashtag_eng = raw_hashtag[list(map(lambda x: test_hashtag(x), raw_hashtag['hashtags']))]
hashtag_noneng = raw_hashtag[list(map(lambda x: not test_hashtag(x), raw_hashtag['hashtags']))]

# lower-case
hashtag_eng_list = list(map(lambda x: x.lower(), hashtag_eng['hashtags']))

#? strip non alpha numeric
hashtag_eng_list = list(map(lambda x: re.sub(r'[^a-zA-Z0-9]', '', x), hashtag_eng_list))

sorted(list(set(hashtag_eng_list)))
len(hashtag_eng_list)<jupyter_output><empty_output><jupyter_text>Run Affinity Clustering on Unique Cleaned Hashtags<jupyter_code>start_time = time.time()

words = list(set(hashtag_eng_list))
words = np.asarray(words)
lev_similarity = -1*np.array([[distance.levenshtein(w1,w2) for w1 in words] for w2 in words])

affprop = AffinityPropagation(affinity="precomputed", 
                              damping=0.7)
affprop.fit(lev_similarity)

print("--- %s seconds ---" % (time.time() - start_time))

affprop_dict = {}
for cluster_id in np.unique(affprop.labels_):
    exemplar = words[affprop.cluster_centers_indices_[cluster_id]]
    cluster = np.unique(words[np.nonzero(affprop.labels_==cluster_id)])
    affprop_dict[exemplar] = list(cluster)<jupyter_output><empty_output><jupyter_text>Prune Clusters:Remove members from cluster (basically create a new cluster) for members with average Lev distance less than the average of all members average pairwise lev distance to other cluster members**consider an alternative distance measure to lev distance**<jupyter_code>c = 0
for key, value in affprop_dict.items():
    if len(value) <2:
        continue
    print(key)
    print(value)
    print("==========================")
    c +=1
    
print(c)
affprop_dict_scrub = {}
t = np.array(list(map(lambda v: ((v - v.min()) / (v.max() - v.min())).mean(), lev_similarity)))
t > t.mean()<jupyter_output><empty_output><jupyter_text>Apply to main data<jupyter_code>hashtag_eng['hashtag_group'] = list(map(lambda x: hashtag_group(grouping=affprop_dict, hashtag=x), hashtag_eng['hashtags']))
hashtag_eng<jupyter_output><empty_output><jupyter_text>Check that Tweets are in english<jupyter_code>raw_text_en  = raw_text[list(map(lambda x: detect(x),raw_text['full_text']))]

#? add hashtag group
raw_text_en = raw_text_eng.merge(hashtag_eng, on = "tweet_id", how = "inner")
raw_text_en<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11086134668437723, 'fraction_numerical': 0.009808930213548585, 'mean_word_length': 4.9034981905910735, 'pattern_counts': {'":': 10, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '456512', 'license': 'MIT', 'n_tokens_mistral': 3299, 'n_tokens_neox': 3134, 'n_words': 922, 'path': 'notebooks/Twitter_EDA-01-HashtagCluster.ipynb', 'repo_name': 'datakind/Mar21-vaccine-uptake', 'size': '9787'}