<jupyter_start><jupyter_text>First look at entries  What are those?<jupyter_code>def list_entries():
    d = col_entries.distinct("type")
    return [x for x in d]<jupyter_output><empty_output><jupyter_text>What do we have the most?<jupyter_code>entries = Out[4]
def count_entry():
    entry_count = []
    for entry in entries:
        c = col_entries.find({"type":entry}).count()
        print("{}|{}".format(c,entry))
        entry_count.append((c,entry))
    return entry_count
entry_count = Out[13]
sorted(entry_count, key= lambda x: x[0])[::-1]<jupyter_output><empty_output><jupyter_text>In these sources of data, usernames seem interesting.  Usernames First, we want to look through what are the usernames we have in the database. To find all usernames, we can do:<jupyter_code>def find_all_usernames():
    return [x for x in col_entries.find({"type":"Username"})]<jupyter_output><empty_output><jupyter_text>Since there are about 50k usernames, let's just look at a random one:<jupyter_code>find_all_usernames()[30]<jupyter_output><empty_output><jupyter_text>Note the 'id' field. Since this correspond to the author field in the instance, we can use this to count the contributions of the username. Therefore, for each username, we can count or list all of the instance that they are associated with.<jupyter_code>def find_inst_by_username(userid):
    return [x for x in col_inst.find({"attributes.authors": userid})]
find_inst_by_username("LTRvjC")<jupyter_output><empty_output><jupyter_text>We could also go through the data and see which username "have the most to say".<jupyter_code>def username_ranking():
    usernames_ranks = []
    usernames = find_all_usernames()[:100]
    for u in tqdm(usernames):
        uid = u['id']
        name = u['name']
        count = col_inst.find({"attributes.authors":uid}).count()
        usernames_ranks.append((uid, name, count))
    return sorted(usernames_ranks, key=lambda x: x[2])<jupyter_output><empty_output><jupyter_text>On my computer, this process would take around **5 hours** to go through the complete dataset and count up all the usernames, we got about 3 username per seconds. So I only did about the first 100 usernames<jupyter_code>u_ranks = username_ranking()<jupyter_output>100%|██████████| 100/100 [00:33<00:00,  3.03it/s]<jupyter_text>For example, in this small subset, the user with the most instance connected to, is<jupyter_code>u_ranks[-1]
def get_inst_for_user(uid):
    return [x for x in col_inst.find({"attributes.authors":uid})]<jupyter_output><empty_output><jupyter_text>What does this user talks about?<jupyter_code>[x['attributes']['indicator'] for x in get_inst_for_user('KKYGPH')]
from nltk.tag import pos_tag
def find_nouns(sentence):
    words = sentence.split()
    tagged = pos_tag(words)
    return [w for w, t in tagged if t == 'NNP']<jupyter_output><empty_output><jupyter_text>Here, we will look at everything the user has said, and count all the proper nouns.<jupyter_code>def words_from_user(uid):
    insts = get_inst_for_user('KKYGPH')
    sentences = [x['fragment'] for x in insts]
    words = {}
    for s in sentences:
        ns = find_nouns(s)
        for n in ns:
            try:
                words[n] += 1
            except KeyError:
                words[n] = 1
    return words
sorted(((a,b) for a,b in words_from_user('KKYGPH').items()), key=lambda x: x[1])[::-1]<jupyter_output><empty_output><jupyter_text>Uhmm, apparently, this user talks about Dridex more than other. This still needs a lot of improvement. One big improvement this approach can use is to categorize the noun. This will give us some context to what the user is talking about.  Furthermore, we can compare the user and group them by what they are talking about. This come back to our instance database. For example, let's look at an our Dridex malware<jupyter_code>def people_and_instance(indicator):
    c = col_inst.find({"attributes.indicator": indicator})
    c = [x for x in c]
    authors = []
    authors_info = {}
    for entry in c:
        try:
            author = entry['attributes']['authors']
        except KeyError:
            author = None
        authors.append(author)
    for a in tqdm(authors):
        if a:
            en = col_entries.find_one({"id": a[0]})
            authors_info[en['id']] = {"name": en['name']}
    return authors_info
talks_of_dridex = Out[29]
len(Out[29]) #There are 468 entries about Dridex, sweet !!
people_and_instance('Dridex')<jupyter_output>100%|██████████| 468/468 [00:33<00:00, 19.11it/s]
{'fraction_non_alphanumeric': 0.11626878868258178, 'fraction_numerical': 0.015915119363395226, 'mean_word_length': 4.34869976359338, 'pattern_counts': {'":': 8, '<': 45, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '492165', 'license': 'MIT', 'n_tokens_mistral': 1555, 'n_tokens_neox': 1492, 'n_words': 447, 'path': 'rf/RF-Entries.ipynb', 'repo_name': 'zuik/stuff', 'size': '4524'}