<jupyter_start><jupyter_text>__Module 5/6 & Assignment 4__1. [Import](Import)1. [Module 5 walkthrough](Module-5-walkthrough)1. [Module 6 walkthrough](Module-6-walkthrough)1. [Assignment 4](Assignment-4) Explore1. Try set up a streaming for twitter using the geo-locations as the query parameter.2. Try the same with the search API as well.    Now that we have all this JSON data from twitter stored in a file, let's take a quick peak at what is JSON and how we can parse this data using python.3. Work on the twitter data that you have extracted :Try load the json from file into memoryCheck out the various keys for any one json object, there are some intresting data fields that can be used for different projects.Try extract the tweets alone from this json dataset.Depending on the questions,file uploads or screenshots are necessary to show your work.  Import<jupyter_code>import os
import sys
import twitter

modulePath = os.path.abspath(os.path.join("../../.."))
if modulePath not in sys.path:
    sys.path.append(modulePath)
import config  # stores the API and access keys for twitter. only on my machine so this will throw an error<jupyter_output><empty_output><jupyter_text>Module 5 walkthrough<jupyter_code># setup twitter API connection
api = twitter.Api(
    consumer_key=config.apiKey,
    consumer_secret=config.apiSec,
    access_token_key=config.accessToken,
    access_token_secret=config.accessSec,
)
# review credentials
print(api.VerifyCredentials())
# print sample tweet
for tweet in api.GetStreamFilter(track="life"):
    print(tweet)
    break
# capture small sample of tweets by keyword
import json

f = open("./streamingData.json", "w")
counter = 0
for tweet in api.GetStreamFilter(track="life"):
    counter += 1
    if counter <= 100:
        f.write(json.dumps(tweet))
        f.write("\n")
    else:
        break
# execute tweet search using keyword
results = api.GetSearch(raw_query="q=life%20&result_type=recent&count=100")
print(len(results))
print(results[0])<jupyter_output><empty_output><jupyter_text>Module 6 walkthrough<jupyter_code># load dummy json data
json_string = '{"name" : "brian", "age" : 24, "gender" : "male", "skills" : ["programming","musician"]}'
json_obj = json.loads(json_string)
# review dummy data
print(json_obj["name"])
print(json_obj["age"])
print(json_obj["gender"])
print(json_obj["skills"])
# load tweet sample
data = []
with open("./streamingData.json", "r") as jsonFile:
    for line in jsonFile:
        data.append(json.loads(line))
print("Total number of tweets loaded: {0}".format(len(data)))
# review keys in json and print sample tweet
print(data[0].keys())
print(data[0]["text"])
# count tweets
tweets = []
for item in data:
    if "text" in item.keys():
        tweet = item["text"]
        tweets.append(tweet)
print("Total number of tweets extracted from json: {0}".format(len(tweets)))<jupyter_output><empty_output><jupyter_text>Assignment 4<jupyter_code># tweets eminating around downtown Chicago, Illinois, United States. Write to file.
import json

f = open("./streamingData.json", "w")
for tweet in api.GetStreamFilter(
    locations=["-87.672529, 41.848849, -87.586613, 41.913966"]
):
    f.write(json.dumps(tweet))
    f.write("\n")
# search Elmhurst, Illinois, United States, area for tweets, print # of results, show first result
results = api.GetSearch(geocode="41.900456,-87.960835,10mi")
print(len(results))
print(results[0])
# load downtown Chicago, Illinois, United States data into memory
data = []
with open("./streamingData.json", "r") as jsonFile:
    for line in jsonFile:
        data.append(json.loads(line))
print("Total number of tweets loaded: {0}".format(len(data)))
# review keys available in a single tweet.
for key in data[0].keys():
    print(key)<jupyter_output><empty_output><jupyter_text>> Remarks - Of the keys shown here, I think the particularly interesting fields are reply_count, retweet_count, and favorite_count. These values speak to how popular a given tweet is, as well as how popular a given poster is.<jupyter_code># unpack all tweets in data
tweets = []
for item in data:
    if "text" in item.keys():
        tweet = item["text"]
        tweets.append(tweet)
print("Total number of tweets extracted from json: {0}".format(len(tweets)))
# print all tweets
for t in tweets:
    print(t)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1172127365020766, 'fraction_numerical': 0.02076603599446239, 'mean_word_length': 4.810991957104558, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '316444', 'license': 'MIT', 'n_tokens_mistral': 1421, 'n_tokens_neox': 1347, 'n_words': 450, 'path': 'academics/NLP/Assignment_04_NLP.ipynb', 'repo_name': 'sudhu26/data-science-portfolio', 'size': '4334'}