<jupyter_start><jupyter_text><jupyter_code>!pip install microprediction<jupyter_output><empty_output><jupyter_text>Let's grab a list of emoji streams from www.microprediction.com . Specifically, we will take the z1-streams. These are normalized relative to expectations, where the expectations are generated by a bunch of algorithms fighting to make the best distributional predictions.<jupyter_code>from microprediction import MicroReader
import pandas as pd
import time
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Hmmm... when will the debate start?(I'm writing this at 9:17 am EST on the morning of the debate)<jupyter_code>time.time()+60*60*11+60*43
LIVE = False
official_debate_start_time = 1601427660
debate_start_time = official_debate_start_time if time.time()-official_debate_start_time>30*60 else time.time()-30*60 
print(int((time.time()-debate_start_time))/60)
print(official_debate_start_time)
print(time.time())
mr = MicroReader()
streams = mr.get_streams_by_sponsor()
emoji_streams = [ name for name in streams.keys() if 'emojitracker-twitter' in name and 'z1~' in name and '3555' in name]
print(emoji_streams[:5])<jupyter_output>['z1~emojitracker-twitter-flushed_face~3555.json', 'z1~emojitracker-twitter-winking_face~3555.json', 'z1~emojitracker-twitter-weary_face~3555.json', 'z1~emojitracker-twitter-heavy_black_heart~3555.json', 'z1~emojitracker-twitter-person_with_folded_hands~3555.json']<jupyter_text>We want the z1-streams. And we want to use 1 hr ahead predictions to standardize. (See https://www.microprediction.org/stream_dashboard.html?stream=z1~emojitracker-twitter-winking_face~3555 for an example)<jupyter_code>plt.rcParams["figure.figsize"] = (15,10)
sorted_z1 = sorted( [ float(mr.get_current_value(name=name)) for name in emoji_streams ], reverse=True )
print(sorted_z1[:5])
TOP_FIVE_THRESHOLD = sorted_z1[5]+1e-6
df_top = None
for name in emoji_streams:
    cv = float(mr.get_current_value(name))
    if cv>TOP_FIVE_THRESHOLD:
      lagged_values = mr.get_lagged_values(name=name)
      lagged_times = mr.get_lagged_times(name=name)
      recent =[ (v,round(t,-1)) for v,t in zip(lagged_values,lagged_times) if t>debate_start_time ]
      recent_values = list(reversed([v for v,t in recent])) # Chronological ordering
      recent_times = list(reversed([t for v,t in recent]))
      df = pd.DataFrame(index=recent_times, data=recent_values,columns=[name])
      if df_top is None:
        df_top = df
      else:
        df_top = df_top.merge(df,left_index=True,right_index=True)
df_top.plot()<jupyter_output>[2.2865479513109825, 1.8339146358159142, 1.725460899561607, 1.6341744669080696, 1.3902781030993898]
{'fraction_non_alphanumeric': 0.1102803738317757, 'fraction_numerical': 0.06280373831775701, 'mean_word_length': 6.495798319327731, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '41056', 'license': 'MIT', 'n_tokens_mistral': 1087, 'n_tokens_neox': 983, 'n_words': 194, 'path': 'Presidential_Debate_Emotions.ipynb', 'repo_name': 'microprediction/microblog', 'size': '2675'}