<jupyter_start><jupyter_text>Mining Twitter Data Using Python / Tweepy  Main script that parses the stream of tweets based on a filter word<jupyter_code>#Import the necessary methods from tweepy library
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream

#Variables that contains the user credentials to access Twitter API 
access_token = "your_access_token"
access_token_secret = "your_access_secret_token"
consumer_key = "your_consumer_key"
consumer_secret = "your_consumer_secret"


#This is a basic listener that just prints received tweets to stdout.
class StdOutListener(StreamListener):

    def on_data(self, data):
        print(data)
        return True

    def on_error(self, status):
        print(status)


if __name__ == '__main__':

    #This handles Twitter authetification and the connection to Twitter Streaming API
    l = StdOutListener()
    auth = OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    stream = Stream(auth, l)

    #This line filter Twitter Streams to capture data by the keywords: 'honda'
    stream.filter(track=['honda'])<jupyter_output><empty_output><jupyter_text>Then from your terminal, execute this script with output piped to a text file: your_script.py > tweets_data.txt  Then run this script below to create a Python dataframe of the tweets data text file<jupyter_code>%matplotlib inline
import json
import pandas as pd
import matplotlib.pyplot as plt
from os import path
pd.set_option("display.max_rows",1000)
pd.set_option("display.max_columns",20)
pd.set_option("display.max_colwidth",150)

d = path.dirname('/home/pybokeh/temp/')
#text = open(path.join(d, 'twitter_data.txt')).read()

tweets_data = []
tweets_file = open(path.join(d, 'nsx_tweets.txt'),'r')
for line in tweets_file:
    try:
        tweet = json.loads(line)
        if len(tweet) > 10:            # Ensure json data is complete
            tweets_data.append(tweet)
    except:
        continue

print(len(tweets_data))<jupyter_output>3487<jupyter_text>Create pandas dataframe<jupyter_code>tweets = pd.DataFrame()
tweets['text']      = [tweet['text'].lower() for tweet in tweets_data]
tweets['lang']      = [tweet['lang'] for tweet in tweets_data]
tweets['sample']      = [tweet['text'][:20] for tweet in tweets_data]
tweets['retweeted'] = [tweet['retweeted'] for tweet in tweets_data]<jupyter_output><empty_output><jupyter_text>You may be wondering why did I create a "sample" column?  The reason being, it will be used to determine if a tweet is a duplicate to another tweet.  Basically, my logic is, if the first 20 characters are the same between tweets, then I am assuming they are the exact same.  Whether or not this is a good logic to use, I dont' know, but seems to be the easiest way I can think of.  Let's remove redundant tweets, retweets, duplicates, and limit to English tweets<jupyter_code>english_tweets = tweets[(tweets['lang']=='en') & (tweets['retweeted']==False) 
                        & ~(tweets['text'].str.contains('vmware')) 
                        & ~(tweets['text'].str.contains('vm ware'))
                        & ~(tweets['text'].str.contains('virtualization'))
                       ]
english_tweets = english_tweets.drop_duplicates(subset='sample');
english_tweets.head(50)
text = ''
for line in english_tweets['text']:
    text = text + ' ' + line.replace("'s", " ")  # Replace 's with a space since I don't care about the possessive form
                                                 # of the noun<jupyter_output><empty_output><jupyter_text>Now create the word cloud<jupyter_code>%matplotlib inline
from os import path
from scipy.misc import imread
import matplotlib.pyplot as plt
from wordcloud import WordCloud, STOPWORDS
d = path.dirname('/home/pybokeh/Downloads/')

additional_words = [
    'rt',
    'ebay',
    'http',
    'https',
]

for word in additional_words:
    STOPWORDS.add(word)

wc = WordCloud(width=800, height=600, max_words=100).generate(text)

# store to file
wc.to_file(path.join(d, "nsx.png"))

# show
plt.imshow(wc)
plt.axis("off")
plt.show()
wc.layout_
wc.words_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1037284894837476, 'fraction_numerical': 0.007170172084130019, 'mean_word_length': 4.116136919315403, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '405558', 'license': 'MIT', 'n_tokens_mistral': 1393, 'n_tokens_neox': 1324, 'n_words': 388, 'path': 'word_cloud/Mining_Twitter_Example-Copy1.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '4184'}