<jupyter_start><jupyter_text>Data decisionIn this notebook, I´m going to investigate both data set alternatives and make a decision on which data set to use.<jupyter_code># import needed libraries
import json
import pandas as pd<jupyter_output><empty_output><jupyter_text>Dataset<jupyter_code>#read in two data set

######## Trump Twitter Archive
# load data
with open('tt.json') as x:
    Twitter_archive = json.load(x)

######## Github - Twitter scraped data
# load the tweets
Twitter_scraped = pd.read_csv("trumptweets.csv", sep=';')<jupyter_output><empty_output><jupyter_text>Metrics to decide List of measurements:- Amount of tweets- data type- Frist and Last elementAlso- Text - The text needs to have the full length- Date- Retweets- Favorits - Retweets or his own  Twitter Archive<jupyter_code># amount of tweets
# length
length_Twitter_archive = len(Twitter_archive)
print("Length of data:", length_Twitter_archive)
# data type
print("Data type;", type(Twitter_archive))<jupyter_output>Length of data: 15161
Data type; <class 'list'><jupyter_text>Twitter scraped data<jupyter_code># amount of tweets
# length
length_Twitter_scraped = Twitter_scraped.shape[0]
print("Length of data:", Twitter_scraped.shape[0])
print("Data type;", type(Twitter_scraped))<jupyter_output>Length of data: 10250
Data type; <class 'pandas.core.frame.DataFrame'><jupyter_text>Amount of tweets - Check the amount of tweets without revtweets! Trumps twitter archive is much longer than the scraped data.<jupyter_code>length_Twitter_archive - length_Twitter_scraped
# check amount of retweets
counter = 0
for i in range(12500): # if I increase this amount (eg. length_Twitter_archive, it doesn´t run anymore)
    if(Twitter_archive[i]["is_retweet"] == True):
        counter = counter + 1
print("The amount of rewteets in Twitter Archive is: ", counter)<jupyter_output>The amount of rewteets in Twitter Archive is:  2094<jupyter_text>Even if I woud be able to check the whole amount of retweets, I looks like there are around **2500 more tweets in the *Twitter Archive*** data set.  Metrics  Trump Twitter Archive<jupyter_code>######## Trump Twitter Archive
# check if the sentence is complet
for i in range(50,90):
    print("----")
    print(Twitter_archive[i]["text"])
    print(Twitter_archive[i]["is_retweet"])<jupyter_output>----
RT @CarmineSabia: It is hilarious to see Speaker Pelosi so #Classless and #Broken. They know 2020 is gone. Who is their hero <NAME>…
True
----
RT @kurt_mcgert: Way to not only represent yourself and your party. She is an absolute disgrace and she is everything that is wrong with ou…
True
----
RT @DrewCHudson: Nancy Pelosi ripping up the State of the Union speech...I assume she planned to throw the trash on the sidewalk, given the…
True
----
RT @AuroraLightzz: This is pure gold, because she’s the one parading how immature and childish the president is but then look how she acted…
True
----
RT @HiredGun37: Today - The Senate gets to pull a #PelosiTantrum and do to the Impeachment Articles what she did to the SOTU. Karma is a bi…
True
----
RT @Real_Theology: This is what building bridges to the American people looks like.. The #PelosiTantrum is a tragic image of how divided ou…
True
----
RT @JonMart93888215: Where those the Articles of Impeachment Nancy was ripping up #Pelosi[...]<jupyter_text>- Text - Yes- Date - Yes- Retweets - Yes- Favorits - Yes- Retweets or his own - Yes  Twitter scraped data<jupyter_code>Twitter_scraped.head()
for i in range(50,90):
    print("---")
    print(Twitter_scraped.text[i])<jupyter_output>---
My Approval Rating in the Republican Party = 95%, a record! Big Iowa win. Approval Rating overall = 53%, a new high. With our great Economy and other major successes, would be 20 points higher without the ph<NAME> and Hoaxes???
---
Market up big today on very good economic news. JOBS, JOBS, JOBS!
---
The Democrat Party in Iowa really messed up, but the Republican Party did not. I had the largest re-election vote in the history of that great state, by far, beating President Obama’s previous record by a lot. Also, 97% Plus of the vote! Thank you Iowa!
---
When will the Democrats start blaming RUSSIA, RUSSIA, RUSSIA, instead of their own incompetence for the voting disaster that just happened in the Great State of Iowa?
---
It is not the fault of Iowa, it is the Do Nothing Democrats fault. As long as I am President, Iowa will stay where it is. Important tradition! https:// twitter.com/SteveKornacki/ status/1224682732208508928 …
---
The Democrat Caucus is an unmitigated disast[...]<jupyter_text>- Text - Yes- Date - Yes- Retweets - Yes- Favorits - Yes- Retweets or his own - Yes  Trump Twitter Archive In this part of the notebook I´m going to create data of *Twitter Archive*, without retweets!<jupyter_code>length_Twitter_archive
count1 = 0
count2 = 0
for i in range(length_Twitter_archive):
    count1 = count1 + 1
    if(Twitter_archive[i]["is_retweet"] == True):
        count2 = count2 + 1
print(count1)
print(count2)
Twitter_archive[12551]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09173766594016247, 'fraction_numerical': 0.01842678819100456, 'mean_word_length': 4.697516930022573, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '365856', 'license': 'Apache-2.0', 'n_tokens_mistral': 1655, 'n_tokens_neox': 1519, 'n_words': 641, 'path': 'Data/Decide on  Dataset/.ipynb_checkpoints/Twitter Data Decision-checkpoint.ipynb', 'repo_name': 'Philhoels/Emotions_and_StockMarkets', 'size': '5064'}