<jupyter_start><jupyter_text>DSTEP20 // Assignment 2assigned : **Jan 15, 2020**DUE : **Jan 22, 2020 11:59pm** Are NYC congresspeople tweeting about New Yorker's concerns?---  OVERVIEWSocial media use by congress is [booming](https://fas.org/sgp/crs/misc/R45337.pdf), with members using the platform to directly engage with their constiuents, take part in the national conversation around timely issues, and fundraise for (re)election.  Participation in social media discourse is quickly becoming [mandatory](https://fcw.com/articles/2018/08/08/social-media-congress.aspx) for public servants looking to both compete in the messaging game and solicit feedback from those who are affected by their policies.At the same time, from the social science perspective, social media is frought with participation bias of all sorts and so it is not clear that the conversation on social media platforms tracks with the lived experience of the electorate.  In this assignment, you will be determining whether members of Congress representing New York City are using social media to highlight the common issues that New Yorkers are facing as measured by 311 service requests.**Instructions for tasks that will be graded are in bold below.**  PART 1 - BackgroundProvide a brief (no more than 500 words) descriptive overview of 1. the use of social media by public servants and congress in particular, and2. the rise of the 311 service in New York City including the reasons for its implementation, operational methods, and characteristics of the datasets available via NYC's Open Data Portal,that can serve as the background for the results that you derive below.  As you lay out this Background for this assignment, an important characteristic to consider is bias (in a statistical as opposed to cultural sense).  For example:- Who is using social media platforms?- Who is calling into 311 in NYC?- Is the scope of 311 a reasonable tracer of the issues facing citizens?- What is divide between social media personality of a congressperson and their actual legislative agenda?Please include *references* within the description via weblinks (like the "booming" link in the Overview in the cell above). **1.	the use of social media by public servants and congress in particular, and**In the 21 century, the use of Twitter, Facebook and other social medias have been expanded by public servants and congress. The previous research found that Members of Congress such as Senators and almost all Representatives are using social medias, mainly using Twitter, adopted it as tools to complement their office communication strategies and disseminate information by automatically distribute vetted content ([Mergel, 2012](https://www.nomos-elibrary.de/10.5771/1865-4789-2012-3-108/connecting-to-congress-the-use-of-twitter-by-members-of-congress-jahrgang-5-2012-heft-); [Straus & Glassman, 2016](https://pdfs.semanticscholar.org/b20b/8ce45c502baf430c01c6681901a01dd38e0b.pdf)). The use of social media was also expanded by the governments’ aims for urban service provision and citizen-government interactions ([Minkoff, 2016](https://journals.sagepub.com/doi/full/10.1177/1078087415577796?casa_token=<KEY>)).  **2.	the rise of the 311 service in New York City including the reasons for its implementation, operational methods, and characteristics of the datasets available via NYC's Open Data Portal,**The mission of the 311 service in New York City (NYC) is to provide the public with quick, easy access to all New York City government services and information while offering the best customer service. According to [Minkoff(2016)](https://journals.sagepub.com/doi/full/10.1177/1078087415577796?casa_token=<KEY>), the city has implemented and expanded the 311 Program for citizens to contact government and report problems with infrastructure and city services based on urban service provision and citizen-government interactions. The contacts such as phone call and website are used for operational methods. The program includes the NYC’s Open Data Portal which covers datasets of Business, City Government, Education, Environment, and Health. •	**Who is using social media platforms?**The social media is known to be used by one-in-three people in the world, more than two-thirds of all internet users, and almost all young people in rich countries  [(Ortiz-Ispina, 2019)](https://ourworldindata.org/rise-of-social-media). Therefore, it is plausible to regard the social media platform as an important tools to distribute information and make networking effects. **•	Who is calling into 311 in NYC?**The 311 in New York City is used by local residents in the city. According to the previous study, it is known that New York City 'residents' report problems by phone call or online submission [(Minkoff, 2016)](https://journals.sagepub.com/doi/full/10.1177/1078087415577796?casa_token=<KEY>). In 2018, 20,9 million customer called to 311, 20,4 million visit online,and about 2,5 million people used mobile app, text, chat, and twitter, which was reported to be 44 million customer used 311 program [(NYC, 2019)](https://www1.nyc.gov/311/311-sets-new-record-in-2018.page).**•	Is the scope of 311 a reasonable tracer of the issues facing citizens?**Yes, the scope of 311 is the reasonable tracer of the issues facing citizens. According to the [311 program website](https://portal.311.nyc.gov/), it covers almost all economic, social, and environmental fields: Benefits & Support, Businesses & Consumers, Courts & Law, Culture & Recreation, Education, Employment, Environment, Garbage & Recycling, Government & Elections, Health, Housing & Buildings, Noise, Pets, Pests & Wildlife, Public Safety, Records, Sidewalks, Streets & Highways, Taxes, and Transportation.**•	What is divide between social media personality of a congressperson and their actual legislative agenda?**It is known that the information about legislative can easily flow from members to constituents, citizens will be capable of drawing policy judgements regarding congressional actions. [Feezell (2018)](https://www.researchgate.net/publication/322081120_Agenda_Setting_through_Social_Media_The_Importance_of_Incidental_News_Exposure_and_Social_Filtering_in_the_Digital_Era) analyzed that exposed to political information through social media such as Facebook yields an agenda setting effect by raising participants’ perceived importance of certain policy issues. Therefore, considering the social media effect, it is important to analyze whether the social media is measured by service requests.  **References:**<NAME> (2019, September 18) The rise of social media. Our World in Data. Retrieved from https://ourworldindata.org/rise-of-social-mediaFeezell, <NAME>. (2018). Agenda setting through social media: The importance of incidental news exposure and social filtering in the digital era. Political Research Quarterly, 71(2), 482-494.Merge<NAME>. (2012). “Connecting to Congress”: The use of Twitter by Members of Congress. ZPB Zeitschrift für Politikberatung, 5(3), 108-114.<NAME>. (2016). NYC 311: A tract-level analysis of citizen–government contacting in New York City. Urban Affairs Review, 52(2), 211-246.The Official Website of the City of New York (2019, Feburary 19) 311 Sets New Record with 44 Million Customer Interactions in 2018. Retrieved from: https://www1.nyc.gov/311/311-sets-new-record-in-2018.pageStraus, J. R., & <NAME>. (2016). Social media in congress: The impact of electronic media on member communications. Congressional Research Service.  PART 2 - Accessing and Analyzing Congressional TweetsYou will be working with two datasets for this assignment.  The first is all of the tweets from every member of congress from 2019.  This file can be found in the folder dstep20/data/congress_tweets/congress_tweets_2019.tsv.  The second is the 311 service requests from NYC that we accessed in class and which can be found [here](https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9). **Load the congressional tweets data from Google Drive into a DataFrame called tweets.$^{\dagger}$** $^{\dagger}$ Hint, the data is a tab separated file and the tab character can be represented by the string "\t".  Also, be sure to specify the lineterminator to be the newline character "\n".<jupyter_code>import pandas as pd
import numpy as np

# link google drive
from google.colab import drive
drive.mount("/content/drive")
fname = "drive/My Drive/dstep20/data/congress_tweets/congress_tweets_2019.tsv"

twt = pd.read_csv(fname, sep='\t', lineterminator="\n")
pd.set_option('display.max_columns', 100)
print(twt.head())
print("")
print(twt.dtypes)
print(twt["tweet"].iloc[0])
print(type(twt["tweet"].iloc[0]))<jupyter_output>RepLipinski:Happy New Year! Wishing everyone a healthy and prosperous 2019. I look forward to tackling new issues that arise in #IL03 &amp; having a productive 116th Congress in the Democratic majority. Here’s to success in the year ahead. http//pbs.twimg.com/media/Dv2VjrFWsAIsrG_.jpg
<class 'str'><jupyter_text>**Split the tweet column into a screen name and the tweet text, and add those back to the oringinal tweets DataFrame with column names "screen_name" and "text".**<jupyter_code>twt["tweet"] = twt["tweet"].str.split(":")
print(twt)
print(twt["tweet"].iloc[0])
ll_list = twt["tweet"].to_list()
print(ll_list)
column_names = ["screen_name", "text"]

df = pd.DataFrame(twt["tweet"].to_list(), columns = column_names)
print(df)
twt["screen_name"] = df["screen_name"]
twt["text"] = df["text"]
print(twt)<jupyter_output>date                                              tweet  \
0       2019-01-01  RepLipinski:Happy New Year! Wishing everyone a...   
1       2019-01-01  RosLehtinen:Hurrah @NikkiHaley! I fought every...   
2       2019-01-01  RepRubenGallego:I’ll be on @MSNBC shortly to d...   
3       2019-01-01  RepJohnYarmuth:Wishing you all a very happy he...   
4       2019-01-01  MarkWarner:Wishing everyone around the Commonw...   
...            ...                                                ...   
841598  2019-12-31  MichaelBennet:At the turn of this decade we ha...   
841599  2019-12-31  AOC:RT @corbintrent I know we can get #SouthDa...   
841600  2019-12-31  RepFredUpton:Happy New Year! Here’s wishing yo...   
841601  2019-12-31  MichaelBennet:At the turn of the last decade w...   
841602  2019-12-31  KenCalvert:Wishing you and your family a very ...   

            screen_name                                               text  
0           RepLipinski  Happy New Year! Wishi[...]<jupyter_text>The screen names for only congresspeople representing the citizens of NYC can be found in the file dstep20/data/congress_tweets/nyc_congress_screen_names.csv.**Sub-select only those tweets coming from congresspeople representing NYC using the list of screen names from the congress_screen_names.csv file and the .isin method of the tweets DataFrame.  Call this new DataFrame (consisting only of tweets for NYC) nyc_tw.**<jupyter_code>drive.mount("/content/drive")
fname2 = "drive/My Drive/dstep20/data/congress_tweets/nyc_congress_screen_names.csv"

cgr = pd.read_csv(fname2, sep=",")
print(cgr)
ind = twt["screen_name"].isin(cgr["screen_name"])
nyc_tw = twt[ind]
print(nyc_tw)<jupyter_output>date                                              tweet  \
35      2019-01-01  Grace4NY:My kids wrote a note to all the new #...   
51      2019-01-01  RepTomSuozzi:The Northport VA should be a mode...   
66      2019-01-01  RepEspaillat:You are cordially invited to join...   
129     2019-01-01  RepGregoryMeeks:Wishing everyone a happy new y...   
168     2019-01-01  RepYvetteClarke:The beginning of a new year sy...   
...            ...                                                ...   
841558  2019-12-31  CarolynBMaloney:We're so close to meeting our ...   
841560  2019-12-31  CarolynBMaloney:And here's 6-10 (...and someth...   
841562  2019-12-31  CarolynBMaloney:A lot happened in 2019 - here ...   
841592  2019-12-31  Grace4NY:@kulkarni_nina @TheDemocrats @TomPere...   
841599  2019-12-31  AOC:RT @corbintrent I know we can get #SouthDa...   

            screen_name                                               text  
35             Grace4NY  My kids wrote a note [...]<jupyter_text>**What is the total fraction of all congresspeople's tweets that are sent by congresspeople representing NYC?**<jupyter_code>All = len(twt["screen_name"])
print(All)
NYC = len(nyc_tw["screen_name"])
print(NYC)
print("The total fraction of all congresspeople's tweets that are sent by congresspeople representing NYC: {0}". format(NYC/All))<jupyter_output>The total fraction of all congresspeople's tweets that are sent by congresspeople representing NYC: 0.037703050012892064<jupyter_text>**How does that fraction compare to the fraction of the US population that are New Yorkers?** I regarded the New Yorkers as residents in City of New York, not State of New York, in this problem set. Estimate Population (2017)City of New York: 8,622,698State of New York: 19,849,399U.S. : 325,919,160Source:[Demographics of New York City](https://en.wikipedia.org/wiki/Demographics_of_New_York_Citycite_note-14).  [US Census](https://www.census.gov/popclock/).<jupyter_code>print("The fraction of the US population that are New Yorkers: {0}". format(8622698/325919160))
print("The fraction compare to the fraction above: {0}". format(0.037703050012892064/0.026456554441291514))<jupyter_output>The fraction compare to the fraction above: 1.425092980136817<jupyter_text>**Answer:** The total fraction of all congrespeople's tweets that are sent by congresspeople reprensenting NYC (0.038) is 1.4 times greater than the fraction of the US population that are New Yorkers (0.026). **Make a bar plot (including a descriptive caption) of the total number of tweets for each screen name in nyc_tw in descending order.**<jupyter_code>twt2 = nyc_tw.groupby("screen_name").count()
print(twt2)
ax = twt2.sort_values('tweet', ascending=False).reset_index().plot.bar("screen_name", "tweet")
xlab = ax.set_xlabel("Screen Name")
ylab = ax.set_ylabel("Total Number of Tweets")
ax.set_title("The total number of tweets for each screen name in NYC", loc="center")<jupyter_output><empty_output><jupyter_text>Before we compare the content of the tweets with 311 Service Requests, let's practice by searching the NYC tweets for certain topical words/phrases that are current.**What is the fraction of NYC tweets that contain the word "gun" or "guns"?**<jupyter_code>print(nyc_tw["text"].str.contains("gun", "guns"))
ngun = nyc_tw["text"].str.contains("gun", "guns").sum()
nall = len(nyc_tw["text"])
print("The fraction of NYC tweets that contain the word 'gun' or 'guns': {0}".format(ngun/nall))<jupyter_output>The fraction of NYC tweets that contain the word 'gun' or 'guns': 0.030159780656140684<jupyter_text>**What fraction of the NYC tweets contain the phrase "medicare for all"?**<jupyter_code>print(nyc_tw["text"].str.contains("medicare for all"))
nmed = nyc_tw["text"].str.contains("medicare for all").sum()
nall = len(nyc_tw["text"])
print("The fraction of the NYC tweets contain the phrase 'medicare care for all': {0}".format(nmed/nall))<jupyter_output>The fraction of the NYC tweets contain the phrase 'medicare care for all': 0.0<jupyter_text>**What fraction of the NYC tweets tag the President of the US's official twitter account?**<jupyter_code>print(nyc_tw["text"].str.contains("@POTUS", "@realDonaldTrump"))
ntrump = nyc_tw["text"].str.contains("@POTUS", "@realDonaldTrump").sum()
nall = len(nyc_tw["text"])
print("The fraction of the NYC tweets tag the President of the US's official twitter account: {0}".format(ntrump/nall))<jupyter_output>The fraction of the NYC tweets tag the President of the US's official twitter account: 0.003025432542308783<jupyter_text>PART 3 - Accessing 311 Service Request Data Use the sodapy module to load the 311 service requests for NYC from Jan 1st, 2019 to Jan 1st 2020 into a DataFrame.  The solution should include: - use pip to install sodapy- set the query return limit to 10000000 (to make sure you get them all)- set the query to return only the unique_key, created_date, and complaint_type columns- set an appropriate connection timeout- name the DataFrame nyc_311<jupyter_code># use pip to install sodapy
!pip install sodapy

import numpy as np
import pandas as pd
import sodapy
dom = "data.cityofnewyork.us"
dsid = "erm2-nwe9"
# set the query return to 10000000 
lim = 10000000 
client = sodapy.Socrata(dom, None, timeout=120)
result = client.get(dsid, limit=lim, select="unique_key,created_date,complaint_type") 
data = pd.DataFrame.from_records(result)
print(data)
print("")
print(data.columns)
print("")
print(data.isna().sum(axis=0))
# set the query to return only the unique_key, created_date, and complaint_type columns
# set an appropriate connection timeout
result_sel = client.get(dsid, limit=lim, select="unique_key, created_date, complaint_type", where="agency='NYPD'and created_date between '2019-01-01T00:00:00' and '2020-01-01T00:00:00.000'")
# name the DataFrame nyc_311
nyc_311 = pd.DataFrame.from_records(result_sel)
print(nyc_311)<jupyter_output>unique_key             created_date               complaint_type
0        44164461  2019-10-28T18:33:32.000                  Panhandling
1        44164518  2019-10-28T11:54:05.000             Blocked Driveway
2        44164520  2019-10-28T23:25:16.000             Blocked Driveway
3        44164521  2019-10-28T18:47:35.000             Blocked Driveway
4        44164522  2019-10-28T17:03:50.000             Blocked Driveway
...           ...                      ...                          ...
841087   44172066  2019-10-28T08:00:30.000  Non-Emergency Police Matter
841088   44173151  2019-10-28T15:23:23.000          Noise - Residential
841089   44172994  2019-10-28T14:25:22.000              Illegal Parking
841090   44172735  2019-10-28T13:58:39.000            Abandoned Vehicle
841091   44172089  2019-10-28T21:16:11.000          Noise - Residential

[841092 rows x 3 columns]<jupyter_text>**How many NaN's are there in each column?**<jupyter_code>nNaN1 = nyc_311["unique_key"].str.contains("NaN").sum()
nNaN2 = nyc_311["created_date"].str.contains("NaN").sum()
nNaN3 = nyc_311["complaint_type"].str.contains("NaN").sum()

print("The total number of NaN in each column: {0}".format(nNaN1+nNaN2+nNaN3))<jupyter_output>The total number of NaN in each column: 0<jupyter_text>**What are the unique values of complaint type?**<jupyter_code>cmplnt = np.unique(nyc_311["complaint_type"])
print(cmplnt)<jupyter_output>['Abandoned Vehicle' 'Animal Abuse' 'Animal-Abuse'
 'Bike/Roller/Skate Chronic' 'Blocked Driveway' 'Derelict Vehicle'
 'Disorderly Youth' 'Drinking' 'Drug Activity' 'Graffiti'
 'Homeless Encampment' 'Illegal Fireworks' 'Illegal Parking'
 'Noise - Commercial' 'Noise - House of Worship' 'Noise - Park'
 'Noise - Residential' 'Noise - Street/Sidewalk' 'Noise - Vehicle'
 'Non-Emergency Police Matter' 'Panhandling' 'Posting Advertisement'
 'Squeegee' 'Traffic' 'Urinating in Public' 'Vending']<jupyter_text>**Make a bar plot (including a descriptive caption) of the top 10 most common complaint types in the 2019 NYC 311 data in descending order left to right.**<jupyter_code>nyc_311_2 = nyc_311.groupby("complaint_type").count()
print(nyc_311_2)
ax = nyc_311["complaint_type"].value_counts().iloc[0:10].plot(kind = 'bar')
xlab = ax.set_xlabel("Complaint Types")
ylab = ax.set_ylabel("Total Number")
ax.set_title("The top 10 most common complaint types in the 2019 NYC 311 data", loc="center")<jupyter_output><empty_output><jupyter_text>PART 4 - Combining Congressional Tweets and 311 Service Request Analyses Combine the methodology for searching NYC congressional tweets that you developed in Part 2 with the 311 Service Request data that you have explored in Part 3 to address the question of whether congresspeople representing NYC are using social media as a platform to highlight these issues.This part of the assignment is purposefully open-ended as there are numerous ways to address the question. For whatever evidence that you provide, describe the assumptions and approximations as well as potential biases or missing perspectives that might impact your conclusions.<jupyter_code># Question is whether congresspeople representing NYC are using social media as a platform to highlight these issues

# Bring the NYC congresspeople data
drive.mount("/content/drive")
fname2 = "drive/My Drive/dstep20/data/congress_tweets/nyc_congress_screen_names.csv"
cgr = pd.read_csv(fname2, sep=",")
print(cgr)
# Compare Graphs

ax1 = twt2.sort_values('tweet', ascending=False).reset_index().plot.bar("screen_name", "tweet")
xlab = ax1.set_xlabel("Screen Name")
ylab = ax1.set_ylabel("Total Number of Tweets")
ax1.set_title("The total number of tweets for each screen name in NYC", loc="center")

ax2 = nyc_311["complaint_type"].value_counts().iloc[0:10].plot(kind = 'bar')
xlab = ax2.set_xlabel("Complaint Types")
ylab = ax2.set_ylabel("Total Number")
ax2.set_title("The top 10 most common complaint types in the 2019 NYC 311 data", loc="center")
# According to the previous background research, 
# Feezell (2018) analyzed that exposed to political information through social media yields an agenda setting effect by raising participants’ perceived importance of certain policy issues. 
# Therefore, considering the social media effect, it is important to analyze whether the social media is measured by service requests

# Assumption: Assume that the congresspeople are using social media by using some 'key words' in the 311 service requrests to raise residents' percieved importance of certain policy issues.

nall = len(nyc_tw["text"])
noise = nyc_tw["text"].str.contains("noise").sum()
resident = nyc_tw["text"].str.contains("resident").sum()
parking = nyc_tw["text"].str.contains("parking").sum()
drive = nyc_tw["text"].str.contains("drive").sum()
street = nyc_tw["text"].str.contains("street").sum()
vehicle = nyc_tw["text"].str.contains("vehicle").sum()
abandoned = nyc_tw["text"].str.contains("abandoned").sum()
derelict = nyc_tw["text"].str.contains("derelict").sum()
emergency = nyc_tw["text"].str.contains("emergency").sum()
police = nyc_tw["text"].str.contains("police").sum()
homeless = nyc_tw["text"].str.contains("homeless").sum()
camp = nyc_tw["text"].str.contains("camp").sum()


print("The fraction of the NYC tweets contain the phrase 'noise': {0}".format(noise/nall))
print("The fraction of the NYC tweets contain the phrase 'resident': {0}".format(noise/nall))
print("The fraction of the NYC tweets contain the phrase 'parking': {0}".format(parking/nall))
print("The fraction of the NYC tweets contain the phrase 'drive': {0}".format(drive/nall))
print("The fraction of the NYC tweets contain the phrase 'street': {0}".format(street/nall))
print("The fraction of the NYC tweets contain the phrase 'vehicle': {0}".format(vehicle/nall))
print("The fraction of the NYC tweets contain the phrase 'abandoned': {0}".format(abandoned/nall))
print("The fraction of the NYC tweets contain the phrase 'derelict': {0}".format(derelict/nall))
print("The fraction of the NYC tweets contain the phrase 'emergency': {0}".format(emergency/nall))
print("The fraction of the NYC tweets contain the phrase 'police': {0}".format(police/nall))
print("The fraction of the NYC tweets contain the phrase 'homeless': {0}".format(homeless/nall))
print("The fraction of the NYC tweets contain the phrase 'camp': {0}".format(camp/nall))<jupyter_output>The fraction of the NYC tweets contain the phrase 'noise': 0.001544231193470108
The fraction of the NYC tweets contain the phrase 'resident': 0.001544231193470108
The fraction of the NYC tweets contain the phrase 'parking': 0.0005672686016828969
The fraction of the NYC tweets contain the phrase 'drive': 0.006082380006933283
The fraction of the NYC tweets contain the phrase 'street': 0.005861775550723268
The fraction of the NYC tweets contain the phrase 'vehicle': 0.0010084775141029278
The fraction of the NYC tweets contain the phrase 'abandoned': 0.0007248432132614793
The fraction of the NYC tweets contain the phrase 'derelict': 0.00018908953389429895
The fraction of the NYC tweets contain the phrase 'emergency': 0.009139327471557783
The fraction of the NYC tweets contain the phrase 'police': 0.004853298036620339
The fraction of the NYC tweets contain the phrase 'homeless': 0.0035611862216759636
The fraction of the NYC tweets contain the phrase 'camp': 0.03510762345970817
{'fraction_non_alphanumeric': 0.09509227502858077, 'fraction_numerical': 0.06353094888126735, 'mean_word_length': 4.681512410113663, 'pattern_counts': {'":': 1, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 16, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '403107', 'license': 'MIT', 'n_tokens_mistral': 8342, 'n_tokens_neox': 7022, 'n_words': 2709, 'path': "Are NYC congresspeople tweeting about New Yorker's concerns?.ipynb", 'repo_name': 'KLeeDE/Data-Science', 'size': '24793'}