<jupyter_start><jupyter_text>Installed Python Libraries now import them<jupyter_code>import pymongo
from pymongo import MongoClient
import json
import tweepy
import twitter
from pprint import pprint
import configparser
import pandas as pd<jupyter_output><empty_output><jupyter_text>Load Authorization Information<jupyter_code>config = configparser.ConfigParser()
config.read('config.ini')

CONSUMER_KEY      = config['mytwitter']['api_key']
CONSUMER_SECRET   = config['mytwitter']['api_secrete']
OAUTH_TOKEN       = config['mytwitter']['access_token']
OATH_TOKEN_SECRET = config['mytwitter']['access_secrete']

mongod_connect = config['mymongo']['connection']<jupyter_output><empty_output><jupyter_text>Connect to MongoDB Cluster<jupyter_code>client = MongoClient(mongod_connect)
db = client.lab9 # use or create a database named demo, we changed name from demo to lab 9
tweet_collection = db.tweet_collection #use or create a collection named tweet_collection
tweet_collection.create_index([("id", pymongo.ASCENDING)],unique = True) # make sure the collected tweets are unique<jupyter_output><empty_output><jupyter_text>Use the Rest API to Collect Tweets Authorization<jupyter_code>rest_auth = twitter.oauth.OAuth(OAUTH_TOKEN,OATH_TOKEN_SECRET,CONSUMER_KEY,CONSUMER_SECRET)
rest_api = twitter.Twitter(auth=rest_auth)<jupyter_output><empty_output><jupyter_text>Define query for rest API<jupyter_code>count = 100 #number of returned tweets, default and max is 100
#geocode = "38.4392897,-78.9412224,50mi"  # comment out location
q = "covid19"                               #define the keywords, tweets contain covid19<jupyter_output><empty_output><jupyter_text>The retained tweets will contain election and be located in harrisonburg<jupyter_code>search_results = rest_api.search.tweets( count=count,q=q) #you can use both q and geocode BUT you don't have to
statuses = search_results["statuses"]
since_id_new = statuses[-1]['id']
for statuse in statuses:
    try:
        tweet_collection.insert_one(statuse)
        pprint(statuse['created_at'])# print the date of the collected tweets
    except:
        pass<jupyter_output>'Thu Nov 04 19:01:38 +0000 2021'
'Thu Nov 04 19:01:37 +0000 2021'
'Thu Nov 04 19:01:37 +0000 2021'
'Thu Nov 04 19:01:37 +0000 2021'
'Thu Nov 04 19:01:37 +0000 2021'
'Thu Nov 04 19:01:37 +0000 2021'
'Thu Nov 04 19:01:37 +0000 2021'
'Thu Nov 04 19:01:36 +0000 2021'
'Thu Nov 04 19:01:36 +0000 2021'
'Thu Nov 04 19:01:34 +0000 2021'
'Thu Nov 04 19:01:34 +0000 2021'
'Thu Nov 04 19:01:34 +0000 2021'
'Thu Nov 04 19:01:34 +0000 2021'
'Thu Nov 04 19:01:33 +0000 2021'
'Thu Nov 04 19:01:33 +0000 2021'
'Thu Nov 04 19:01:32 +0000 2021'
'Thu Nov 04 19:01:32 +0000 2021'
'Thu Nov 04 19:01:31 +0000 2021'
'Thu Nov 04 19:01:31 +0000 2021'
'Thu Nov 04 19:01:31 +0000 2021'
'Thu Nov 04 19:01:30 +0000 2021'
'Thu Nov 04 19:01:28 +0000 2021'
'Thu Nov 04 19:01:28 +0000 2021'
'Thu Nov 04 19:01:28 +0000 2021'
'Thu Nov 04 19:01:27 +0000 2021'
'Thu Nov 04 19:01:27 +0000 2021'
'Thu Nov 04 19:01:27 +0000 2021'
'Thu Nov 04 19:01:26 +0000 2021'
'Thu Nov 04 19:01:26 +0000 2021'
'Thu Nov 04 19:01:25 +0000 2021'
'Thu Nov 0[...]<jupyter_text>Introduce the concept of time!<jupyter_code>since_id_old = 0
while(since_id_new != since_id_old):
    since_id_old = since_id_new
    search_results = rest_api.search.tweets( count=count,q=q,
                        max_id= since_id_new)
    statuses = search_results["statuses"]
    since_id_new = statuses[-1]['id']
    for statuse in statuses:
        try:
            tweet_collection.insert_one(statuse)
            pprint(statuse['created_at']) # print the date of the collected tweets
        except:
            pass<jupyter_output>'Thu Nov 04 19:00:57 +0000 2021'
'Thu Nov 04 19:00:57 +0000 2021'
'Thu Nov 04 19:00:57 +0000 2021'
'Thu Nov 04 19:00:57 +0000 2021'
'Thu Nov 04 19:00:56 +0000 2021'
'Thu Nov 04 19:00:56 +0000 2021'
'Thu Nov 04 19:00:56 +0000 2021'
'Thu Nov 04 19:00:56 +0000 2021'
'Thu Nov 04 19:00:55 +0000 2021'
'Thu Nov 04 19:00:55 +0000 2021'
'Thu Nov 04 19:00:54 +0000 2021'
'Thu Nov 04 19:00:53 +0000 2021'
'Thu Nov 04 19:00:53 +0000 2021'
'Thu Nov 04 19:00:53 +0000 2021'
'Thu Nov 04 19:00:53 +0000 2021'
'Thu Nov 04 19:00:52 +0000 2021'
'Thu Nov 04 19:00:52 +0000 2021'
'Thu Nov 04 19:00:52 +0000 2021'
'Thu Nov 04 19:00:51 +0000 2021'
'Thu Nov 04 19:00:50 +0000 2021'
'Thu Nov 04 19:00:49 +0000 2021'
'Thu Nov 04 19:00:49 +0000 2021'
'Thu Nov 04 19:00:49 +0000 2021'
'Thu Nov 04 19:00:48 +0000 2021'
'Thu Nov 04 19:00:47 +0000 2021'
'Thu Nov 04 19:00:47 +0000 2021'
'Thu Nov 04 19:00:47 +0000 2021'
'Thu Nov 04 19:00:47 +0000 2021'
'Thu Nov 04 19:00:46 +0000 2021'
'Thu Nov 04 19:00:46 +0000 2021'
'Thu Nov 0[...]<jupyter_text>View Collected Tweets Print number of tweets and unique twitter users<jupyter_code>print(tweet_collection.estimated_document_count())# number of tweets collected

user_cursor = tweet_collection.distinct("user.id")
print (len(user_cursor)) # number of unique Twitter users<jupyter_output>2773
2402<jupyter_text>Create a text index and print the Tweets containing specific keywords<jupyter_code>tweet_collection.create_index([("text", pymongo.TEXT)], name='text_index', default_language='english') # create a text index<jupyter_output><empty_output><jupyter_text>Create a cursor to query tweets with the created index<jupyter_code>tweet_cursor = tweet_collection.find({"$text": {"$search": "covid"}}) # return tweets contain covid<jupyter_output><empty_output><jupyter_text>Use pprint to display tweets<jupyter_code>for document in tweet_cursor[0:10]: # display the first 10 tweets from the query
    try:
        print ('----')
#         pprint (document) # use pprint to print the entire tweet document
   
        print ('name:', document["user"]["name"]) # user name
        print ('text:', document["text"])         # tweets
    except:
        print ("***error in encoding")
        pass
tweet_cursor = tweet_collection.find({"$text": {"$search": "vaccine"}}) # return tweets contain vote<jupyter_output><empty_output><jupyter_text>Use pandas to display the tweets<jupyter_code>tweet_df = pd.DataFrame(list(tweet_cursor ))
tweet_df[:10] #display the first 10 tweets
tweet_df["favorite_count"].hist() # create a histogram show the favorite count<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11777813146586026, 'fraction_numerical': 0.16138787203565175, 'mean_word_length': 5.19723865877712, 'pattern_counts': {'":': 4, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '570945', 'license': 'MIT', 'n_tokens_mistral': 3115, 'n_tokens_neox': 2210, 'n_words': 687, 'path': 'Lab9.ipynb', 'repo_name': 'DyMarti/ia340', 'size': '6283'}