<jupyter_start><jupyter_text>Demo of Sample text<jupyter_code>from __future__ import print_function, division, unicode_literals
#import example_helper
import json
import csv
import argparse

import numpy as np
import emoji

from torchmoji.sentence_tokenizer import SentenceTokenizer
from torchmoji.model_def import torchmoji_emojis
from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH

# Emoji map in emoji_overview.png
EMOJIS = ":joy: :unamused: :weary: :sob: :heart_eyes: \
:pensive: :ok_hand: :blush: :heart: :smirk: \
:grin: :notes: :flushed: :100: :sleeping: \
:relieved: :relaxed: :raised_hands: :two_hearts: :expressionless: \
:sweat_smile: :pray: :confused: :kissing_heart: :heartbeat: \
:neutral_face: :information_desk_person: :disappointed: :see_no_evil: :tired_face: \
:v: :sunglasses: :rage: :thumbsup: :cry: \
:sleepy: :yum: :triumph: :hand: :mask: \
:clap: :eyes: :gun: :persevere: :smiling_imp: \
:sweat: :broken_heart: :yellow_heart: :musical_note: :speak_no_evil: \
:wink: :skull: :confounded: :smile: :stuck_out_tongue_winking_eye: \
:angry: :no_good: :muscle: :facepunch: :purple_heart: \
:sparkling_heart: :blue_heart: :grimacing: :sparkles:".split(' ')

def top_elements(array, k):
    ind = np.argpartition(array, -k)[-k:]
    return ind[np.argsort(array[ind])][::-1]
maxlen = 30

text = ['I love mom''s cooking','I love how you never reply back..','I love cruising with my homies','I love messing with yo mind!!','I love you and now you''re just gone..','This is shit','This is the shit!']


# Tokenizing using dictionary
with open(VOCAB_PATH, 'r') as f:
        vocabulary = json.load(f)
        st = SentenceTokenizer(vocabulary, maxlen)
    # Loading model
        model = torchmoji_emojis(PRETRAINED_PATH)
    # Running predictions
        for i in text:
            tokenized, _, _ = st.tokenize_sentences([i])
            prob = model(tokenized)[0]
            emoji_ids = top_elements(prob, 5)
            #print(emoji_ids)
            emojis = map(lambda x: EMOJIS[x], emoji_ids)
            print(emoji.emojize("{} {}".format(i,' '.join(emojis)), use_aliases=True))
            print('\n')<jupyter_output>C:\Users\AjitC\Desktop\Application of Data Science\Project Proposal\torchMoji\torchmoji\model_def.py:159: UserWarning: nn.init.uniform is now deprecated in favor of nn.init.uniform_.
  nn.init.uniform(self.embed.weight.data, a=-0.5, b=0.5)
C:\Users\AjitC\Desktop\Application of Data Science\Project Proposal\torchMoji\torchmoji\model_def.py:161: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  nn.init.xavier_uniform(t)
C:\Users\AjitC\Desktop\Application of Data Science\Project Proposal\torchMoji\torchmoji\model_def.py:163: UserWarning: nn.init.orthogonal is now deprecated in favor of nn.init.orthogonal_.
  nn.init.orthogonal(t)
C:\Users\AjitC\Desktop\Application of Data Science\Project Proposal\torchMoji\torchmoji\model_def.py:165: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
  nn.init.constant(t, 0)
C:\Users\AjitC\Desktop\Application of Data Science\Project Proposal\torchMoji\torchmoji\model_def.py:167: UserWa[...]<jupyter_text>Demo of tweets<jupyter_code>from __future__ import print_function, division, unicode_literals
#import example_helper
import json
import csv
import argparse

import numpy as np
import emoji

from torchmoji.sentence_tokenizer import SentenceTokenizer
from torchmoji.model_def import torchmoji_emojis
from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH

# Emoji map in emoji_overview.png
EMOJIS = ":joy: :unamused: :weary: :sob: :heart_eyes: \
:pensive: :ok_hand: :blush: :heart: :smirk: \
:grin: :notes: :flushed: :100: :sleeping: \
:relieved: :relaxed: :raised_hands: :two_hearts: :expressionless: \
:sweat_smile: :pray: :confused: :kissing_heart: :heartbeat: \
:neutral_face: :information_desk_person: :disappointed: :see_no_evil: :tired_face: \
:v: :sunglasses: :rage: :thumbsup: :cry: \
:sleepy: :yum: :triumph: :hand: :mask: \
:clap: :eyes: :gun: :persevere: :smiling_imp: \
:sweat: :broken_heart: :yellow_heart: :musical_note: :speak_no_evil: \
:wink: :skull: :confounded: :smile: :stuck_out_tongue_winking_eye: \
:angry: :no_good: :muscle: :facepunch: :purple_heart: \
:sparkling_heart: :blue_heart: :grimacing: :sparkles:".split(' ')
maxlen = 30
def top_elements(array, k):
    ind = np.argpartition(array, -k)[-k:]
    return ind[np.argsort(array[ind])][::-1]

# Tokenizing using dictionary
with open(VOCAB_PATH, 'r') as f:
        vocabulary = json.load(f)

        st = SentenceTokenizer(vocabulary, maxlen)

    # Loading model
        model = torchmoji_emojis(PRETRAINED_PATH)
    # Running predictions
i =1       
for json_tweet in tweets:
        tokenized, _, _ = st.tokenize_sentences([json_tweet.full_text])
        prob = model(tokenized)[0]
        emoji_ids = top_elements(prob, 5)
        emojis = map(lambda x: EMOJIS[x], emoji_ids)
        print(emoji.emojize("{} {}".format(str(i)+') '+json_tweet.full_text,' '.join(emojis))+ '\n', use_aliases=True))
        i = i+1<jupyter_output>1) Here in Washington, we always vote by mail. It's a safe and secure way to shape the future of our country. But regardless of whether you vote by mail or in person this year, I hope you plan to make your voice heard. Visit https://t.co/TmsHAC9MGv for more information. 👍 😉 😊 😄 💙

2) @MohamedBinZayed The UAE is an essential partner in stopping polio in Pakistan. Thank you @MohamedBinZayed for your continued commitment to polio eradication. 👍 😉 👏 😊 💛

3) For decades, @Rotary has worked tirelessly to #EndPolio. Today, the health infrastructure they’ve built around the world is supporting the COVID-19 response. #WorldPolioDay https://t.co/67liH7akNH 👍 💪 👏 👊 🙏

4) RT @StephenCurry30: Even with his busy schedule, Dr. Fauci took the time to sit down with me (AGAIN) and talk about what we’ve gotten right… 👀 🙌 👏 👍 😂

5) I’ve been working for some time on a book about what we need to do over the next decade to avoid a climate disaster. I’m excited that it will be published on February 16. https[...]<jupyter_text>Splitting paragraphs into sentences to get precise emojies<jupyter_code>from __future__ import print_function, division, unicode_literals
#import example_helper
import json
import csv
import argparse

import numpy as np
import emoji

from torchmoji.sentence_tokenizer import SentenceTokenizer
from torchmoji.model_def import torchmoji_emojis
from torchmoji.global_variables import PRETRAINED_PATH, VOCAB_PATH

# Emoji map in emoji_overview.png
EMOJIS = ":joy: :unamused: :weary: :sob: :heart_eyes: \
:pensive: :ok_hand: :blush: :heart: :smirk: \
:grin: :notes: :flushed: :100: :sleeping: \
:relieved: :relaxed: :raised_hands: :two_hearts: :expressionless: \
:sweat_smile: :pray: :confused: :kissing_heart: :heartbeat: \
:neutral_face: :information_desk_person: :disappointed: :see_no_evil: :tired_face: \
:v: :sunglasses: :rage: :thumbsup: :cry: \
:sleepy: :yum: :triumph: :hand: :mask: \
:clap: :eyes: :gun: :persevere: :smiling_imp: \
:sweat: :broken_heart: :yellow_heart: :musical_note: :speak_no_evil: \
:wink: :skull: :confounded: :smile: :stuck_out_tongue_winking_eye: \
:angry: :no_good: :muscle: :facepunch: :purple_heart: \
:sparkling_heart: :blue_heart: :grimacing: :sparkles:".split(' ')

def top_elements(array, k):
    ind = np.argpartition(array, -k)[-k:]
    return ind[np.argsort(array[ind])][::-1]

# Tokenizing using dictionary
with open(VOCAB_PATH, 'r') as f:
        vocabulary = json.load(f)

        st = SentenceTokenizer(vocabulary, maxlen)

    # Loading model
        model = torchmoji_emojis(PRETRAINED_PATH)
    # Running predictions
i =1       
for k in sentence[0:20]:
    if len(k)>3:
        tokenized, _, _ = st.tokenize_sentences(sentence)
        prob = model(tokenized)[0]
        emoji_ids = top_elements(prob, 5)
        emojis = map(lambda x: EMOJIS[x], emoji_ids)
        print(emoji.emojize("{} {}".format(k,' '.join(emojis))+ '\n', use_aliases=True))
        i = i+1<jupyter_output>1)Dr. 😷 💓 🙌 👏 😎

2) <NAME>’s work with  was essential in stopping wild polio in Africa and will help create a polio-free world for all. 😷 💓 🙌 👏 😎

3) A well-deserved recognition. 😷 💓 🙌 👏 😎

4)This honor is well deserved. 😷 💓 🙌 👏 😎

5) COVID-19 is not just a health crisis—it is also now a food crisis, and  is there to respond. 😷 💓 🙌 👏 😎

6) I’m inspired by the thousands of people in the organization who have committed their lives to fighting hunger. 😷 💓 🙌 👏 😎

7)For the last 25 years, Dr. 😷 💓 🙌 👏 😎

8) <NAME>, an immunologist and infectious disease researcher in Bangladesh, has been working to protect entire communities from cholera epidemics. 😷 💓 🙌 👏 😎

9)I’m excited to see this kind of innovation. 😷 💓 🙌 👏 😎

10) Detecting and predicting outbreaks earlier is critical to preventing epidemics. 😷 💓 🙌 👏 😎

11)The only way to eliminate the threat of COVID-19 somewhere is to eliminate it everywhere. 😷 💓 🙌 👏 😎

12) By making sure poor countries are equipped to stop this disease,[...]
{'fraction_non_alphanumeric': 0.12520775623268698, 'fraction_numerical': 0.009529085872576178, 'mean_word_length': 4.561306223043746, 'pattern_counts': {'":': 3, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '69572', 'license': 'MIT', 'n_tokens_mistral': 3488, 'n_tokens_neox': 3327, 'n_words': 950, 'path': 'Source Code replication.ipynb', 'repo_name': 'avinashpandey1/COMP8240-Group-D', 'size': '9039'}