<jupyter_start><jupyter_text>Twitter API Access  Here You will have to first go to this website and get started with the first app --> https://developer.twitter.com/en/apps Before going to the link just login with your twitter account and then click on the link. Then after clicking on the link click on Create a Developer Account. Go ahead with the options. Set Account name. Select Your country. Write 300 words description for using the API access.  Then read and accept the terms and conditions And Submit Application. Then go to Keys and Tokens. there you see Consumer API keys and Access Token details. For Access Tokens generation you have to click on generate the Access Token. Then write note it down and set at Consumet Key, Consumer Secret, Access Token, Access Token Secret accordingly. Do not share it (Access Keys and Consumer Keys) with anybody.  Import the Pickle and Os libraries of the Python<jupyter_code>import pickle
import os
if not os.path.exists('secret_twitter_credentials.pk1'):
    Twitter = {}
    Twitter['Consumer Key'] = ''
    Twitter['Consumer Secret'] = ''
    Twitter['Access Token'] = ''
    Twitter['Access Token Secret'] = ''
    with open('secret_twitter_credentials.pkl','wb') as f:
        pickle.dump(Twitter,f)
else:
    Twitter = pickle.load(open('secret_twitter_credentials.pk1','rb'))<jupyter_output><empty_output><jupyter_text>Install the Twitter package for interface with the Twitter API using this command if you don't have one !pip install twitter  Authorizing the Application to Access the Twitter Account Data<jupyter_code>import twitter
auth = twitter.oauth.OAuth(Twitter['Access Token'],
                          Twitter['Access Token Secret'],
                          Twitter['Consumer Key'],
                          Twitter['Consumer Secret'])
twitter_api = twitter.Twitter(auth=auth)
print(twitter_api)<jupyter_output><twitter.api.Twitter object at 0x00000166FFD55F60><jupyter_text>Collecting the Search Results<jupyter_code>#topic is the keyword for which we are looking for and count or number is the number of tweets we need
#statuses are in the JSON format
topic = '#MerryChristmas'
number = 100
search_results = twitter_api.search.tweets(q=topic,count=number)
statuses = search_results['statuses']
len(statuses)
#print(statuses)<jupyter_output><empty_output><jupyter_text>To filter out duplicate results, we filter it out from the output<jupyter_code>all_text =[]
filtered_statuses=[]
for s in statuses:
    if not s['text'] in all_text:
        filtered_statuses.append(s)
        all_text.append(s["text"])
statuses= filtered_statuses
# six duplicates are removed and 94 are remaining one
len(statuses)
import json
print(json.dumps(statuses[0], indent=1))
t = statuses[0]
print(t['retweet_count'])
print(t['retweeted'])<jupyter_output>12
False<jupyter_text>Extracting the text, screen names, and hashtags from the tweets<jupyter_code>status_texts = [status['text']
               for status in statuses]
screen_names = [user_mention['screen_name']
               for status in statuses
                   for user_mention in status['entities']['user_mentions']]
hashtags = [hashtag['text']
               for status in statuses
                   for hashtag in status['entities']['hashtags']]

#compute the collection of the words from all tweets
words = [w
        for t in status_texts

         for w in t.split()]
#We print the first 10 items for each
print(json.dumps(status_texts[:5],indent=1))
print(json.dumps(screen_names[:5],indent=1))
print(json.dumps(hashtags[:5],indent=1))
print(json.dumps(words[:5],indent=1))<jupyter_output>[
 "RT @BTS_army_Fin: Kyll\u00e4 on kivat joululahjat n\u00e4\u00e4 \nNaver x Dispatch @BTS_twt -joulukuvat \u2763\ufe0f\n#Hyv\u00e4\u00e4Joulua vaan ARMY! #MerryChristmas \n\n#Video\u2026",
 "Absolutely loved #clickandcollect fairplay, Andy made it up to Dev in the end and who doesn't like a happy ending #MerryChristmas",
 "\ud83d\ude0d\ud83d\ude0d\ud83d\ude0d #lakuno\u0107 #MerryChristmas https://t.co/TuZJ59ZmPM",
 "He Said, She Said: Beermas Day 11: Maybe read this one @tavour @lostabbey #craftbeer #blog #bloggers #beeroclock\u2026 https://t.co/vyhaazPXEY",
 "RT @THEANTHEMGIRL: Thank you to all who have served and are serving!  #MerryChristmas\u2060 \u2060 https://t.co/2kEGFQqUpB"
]
[
 "BTS_army_Fin",
 "BTS_twt",
 "tavour",
 "lostabbey",
 "THEANTHEMGIRL"
]
[
 "Hyv\u00e4\u00e4Joulua",
 "MerryChristmas",
 "clickandcollect",
 "MerryChristmas",
 "lakuno\u0107"
]
[
 "RT",
 "@BTS_army_Fin:",
 "Kyll\u00e4",
 "on",
 "kivat"
]<jupyter_text>Create the basic frequency distribution from the words in tweets<jupyter_code>from collections import Counter
for item in [words,screen_names,hashtags]:
    c=Counter(item)
    print(c.most_common()[:20]) #top 20
    print()<jupyter_output>[('RT', 72), ('#MerryChristmas', 44), ('a', 27), ('and', 24), ('you', 20), ('to', 17), ('the', 16), ('in', 11), ('all', 11), ('#merrychristmas', 11), ('Christmas', 10), ('😂', 10), ('Merry', 9), ('de', 8), ('be', 8), ('que', 7), ('of', 7), ('are', 6), ('we', 6), ('your', 6)]

[('BTS_twt', 2), ('NissyStaff', 2), ('MyMouseComic', 2), ('BTS_army_Fin', 1), ('tavour', 1), ('lostabbey', 1), ('THEANTHEMGIRL', 1), ('MarineLorphelin', 1), ('GWMFilm', 1), ('satomi_nissy_', 1), ('ProjetoFFans', 1), ('ShawnMendes', 1), ('artofmmignola', 1), ('TheRock', 1), ('masaki_1133', 1), ('iam_keiji', 1), ('SHOKICHI_927', 1), ('PayneFullHuman', 1), ('Zoysbe', 1), ('ELFSuJuMexico', 1)]

[('MerryChristmas', 55), ('merrychristmas', 12), ('Nissy', 3), ('BTS', 3), ('クリスマスイブ', 2), ('西島隆弘', 2), ('FelizNavidad', 2), ('feliznavidad', 2), ('JIMIN', 2), ('Christmas', 2), ('HappyNewYear', 2), ('HyvääJoulua', 1), ('clickandcollect', 1), ('lakunoć', 1), ('craftbeer', 1), ('blog', 1), ('bloggers', 1), ('beeroclock', 1), ('fi[...]<jupyter_text>Display the Previous output in Nice Tabular Format using Prettyprint function<jupyter_code>def prettyprint_counts(label,list_of_tuples):
    print("\n{:^20} | {:^6}".format(label,"Count"))
    print("*"*40)
    for k,v in list_of_tuples:
        print("{:20} | {:>6}".format(k,v))
        
for label, data in (('word',words),
                   ('Screen Name',screen_names),
                   ('Hashtag',hashtags)):
    c = Counter(data)
    prettyprint_counts(label,c.most_common()[:10])<jupyter_output>word         | Count 
****************************************
RT                   |     72
#MerryChristmas      |     44
a                    |     27
and                  |     24
you                  |     20
to                   |     17
the                  |     16
in                   |     11
all                  |     11
#merrychristmas      |     11

    Screen Name      | Count 
****************************************
BTS_twt              |      2
NissyStaff           |      2
MyMouseComic         |      2
BTS_army_Fin         |      1
tavour               |      1
lostabbey            |      1
THEANTHEMGIRL        |      1
MarineLorphelin      |      1
GWMFilm              |      1
satomi_nissy_        |      1

      Hashtag        | Count 
****************************************
MerryChristmas       |     55
merrychristmas       |     12
Nissy                |      3
BTS                  |      3
クリスマスイブ              |      2
西島隆弘                 |      2
Feli[...]<jupyter_text>Print Entire Tweets with their Count of Retweets<jupyter_code>retweets = [
            (status['retweet_count'],
             status['retweeted_status']['user']['screen_name'],
             status['text'].replace("\n","\\"))
    
             for status in statuses
                if 'retweeted_status' in status
          ]
row_template = "{:^7} | {:^15} | {:50}"
def prettyprint_tweets(list_of_tuples):
    print()
    print(row_template.format("count","Screen Name","Text"))
    print("*"*30)
    for count,screen_names,text in list_of_tuples:
        print(row_template.format(count,screen_names,text[:50]))
        if len(text)>50:
            print(row_template.format("","",text[50:100]))
            if len(text)>100:
                print(row_template.format("","",text[100:]))
prettyprint_tweets(sorted(retweets, reverse=True)[:10])<jupyter_output>count  |   Screen Name   | Text                                              
******************************
 53583  |     BTS_twt     | RT @BTS_twt: 행복한 크리스마스 보내고 있어요?\오늘하루 즐겁게 보내요\#JIMI
        |                 | N\#MerryChristmas https://t.co/cqTeerTTDw         
 10126  |   NissyStaff    | RT @NissyStaff: 🎄12/24本日発売🎄\【Nissy Entertainment 2
        |                 | ndLIVE -FINAL- inTOKYODOME】\✨Nissy史上初の一般店舗でも販売決定✨\
        |                 | https://t.co/gcGvKtOe1y\\#Nissy\#西島隆弘\#…          
 5684   |   NissyStaff    | RT @NissyStaff: 🎄12/24発売🎄\【Nissy Entertainment 2nd
        |                 | LIVE -FINAL- inTOKYODOME】\✨Nissy史上初の一般店舗でも販売決定✨\ht
        |                 | tps://t.co/gcGvKtOe1y\\#Nissy\#西島隆弘\#街や…          
 2624   | hf_dreamcatcher | RT @hf_dreamcatcher: 인썸니아에게 드리는 크리스마스 선물🎅🏻🤶🏻🎁\\드림캐
        |                 | 쳐 공식 인스타그램에서 확인해보세요💕\\여러분~❄\Merry Christmas🎄\\▶ ht
        |                 | tps://t.co/tcyOnzP8Bm\\#드림캐쳐 #Dreamcatc…          
 2328   |     DBak69[...]
{'fraction_non_alphanumeric': 0.1512778684067428, 'fraction_numerical': 0.03230016313213703, 'mean_word_length': 3.0, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 32, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '115681', 'license': 'Unlicense', 'n_tokens_mistral': 3470, 'n_tokens_neox': 3256, 'n_words': 780, 'path': 'Using Twitter API for Searching the data.ipynb', 'repo_name': 'jayeshmanani/Use-of-Twitter-API-for-Searching-the-keyword-', 'size': '9195'}