<jupyter_start><jupyter_text>DIC LAB 1 Problem 2 : Study response to an event Define the limit of numbers to be searched<jupyter_code>LIMIT = 20000<jupyter_output><empty_output><jupyter_text>Declare all the search, strings which will be searched in a vector, following are the hastags for SuperBowl<jupyter_code>superBowlStrings = c("#superbowl","#Patriots","#Falcons","#SB51")<jupyter_output><empty_output><jupyter_text>Declare your custom search strings, which will be searched, in a vector. Leave it empty vector if no custom string. Eg : Empty Vector will look like customUserSearchStrings = c("") **NOTE : Make sure you empty leave customUserSearchStrings vector empty if you want to see the results from superBowlStrings**<jupyter_code>customUserSearchStrings = c("") #c("#NASA")<jupyter_output><empty_output><jupyter_text>Create a boolean variable to check if the custom search string is present or not. **Custom search will be given preference over SuperBowl search.** Number of search limit will change too i.e. **LIMIT = 1000**<jupyter_code>isCustomSearch = FALSE
if(customUserSearchStrings[1]!=""){
    isCustomSearch = TRUE
    LIMIT = 1000
    userSearchStrings = customUserSearchStrings
}else{
    isCustomSearch = FALSE
    LIMIT = 20000
    userSearchStrings = superBowlStrings
}<jupyter_output><empty_output><jupyter_text>Define all the libraries which needs to be set up for operations below<jupyter_code>library("twitteR")
library("DBI")
library("RSQLite")
library("gtools")
library("bitops")
library("ggplot2")
library("RCurl")
library("RJSONIO")
library("ggmap")
library("sp")
library("mapdata")
library("maptools")
library("scales")
library("maps")
Sys.setlocale(category = "LC_ALL", locale = "C")<jupyter_output><empty_output><jupyter_text>Setup the twitter app key for authentication<jupyter_code>setup_twitter_oauth('YOUR KEY')<jupyter_output>[1] "Using direct authentication"<jupyter_text>Following function fetches the state and country from the address_components field in json response .Following is the response "address_components" : [ { "long_name" : "Buffalo", "short_name" : "Buffalo", "types" : [ "locality", "political" ] }, { "long_name" : "Erie County", "short_name" : "Erie County", "types" : [ "administrative_area_level_2", "political" ] }, { "long_name" : "New York", "short_name" : "NY", "types" : [ "administrative_area_level_1", "political" ] }, { "long_name" : "United States", "short_name" : "US", "types" : [ "country", "political" ] } ], It will generate the state as New York and country as United States.<jupyter_code>parseAddressComponents <- function(addr_comp){
    result = tryCatch(
        {
            addrSize = length(addr_comp)
            stateName = ""
            countryName = ""
            for(i in 1:addrSize){
                if(addr_comp[[i]]$types[[1]] == "administrative_area_level_1"){
                    stateName = addr_comp[[i]]$long_name
                }
                if(addr_comp[[i]]$types[[1]] == "country"){
                    countryName = addr_comp[[i]]$long_name
                }
            }
            if(stateName == ""){
                stateName = NA
            }
            if(countryName == ""){
                countryName = NA
            }
            return (c(stateName,countryName))
        },
        error=function(cond) {
            return (c(NA,NA))
        },
        warning=function(cond) {
        },
        finally={
        }
    )
    return (result);
}<jupyter_output><empty_output><jupyter_text>This function generates the url which will be used to query the Google server. You have to pass the address for this function Eg : https://maps.google.com/maps/api/geocode/json?address=Buffalo&key=YOUR_KEY_GOOGLE&sensor=false<jupyter_code>getGeoCodeUrl <- function(address, return.call = "json", sensor = "false") {
    root <- "https://maps.google.com/maps/api/geocode/"
    url <- paste(root, return.call, "?address=", address,"&key=YOUR_KEY_GOOGLE","&sensor=", sensor, sep = "")
    return(URLencode(url))
}<jupyter_output><empty_output><jupyter_text>This function generates the url which will be used to query the Google server. You have to pass the latitude and longitude for this function Eg. https://maps.google.com/maps/api/geocode/json?address=42.88644679999999,-78.8783689&key=AIzaSyCskJdqaLuNJBJ58IHDqfsjJAm9jkZbcKM&sensor=false<jupyter_code>getLatLngURL <- function(lat, lng, return.call = "json", sensor = "false") {
    root <- "https://maps.google.com/maps/api/geocode/"
    latlng = paste(lat, lng, sep = ",")
    url <- paste(root, return.call, "?address=", latlng,"&key=<KEY>","&sensor=", sensor, sep = "")
    return(URLencode(url))
}<jupyter_output><empty_output><jupyter_text>The below code decomposes the json response from the Google Map API code The output response is as follows for location Buffalo { "results" : [ { "address_components" : [ { "long_name" : "Buffalo", "short_name" : "Buffalo", "types" : [ "locality", "political" ] }, { "long_name" : "Erie County", "short_name" : "Erie County", "types" : [ "administrative_area_level_2", "political" ] }, { "long_name" : "New York", "short_name" : "NY", "types" : [ "administrative_area_level_1", "political" ] }, { "long_name" : "United States", "short_name" : "US", "types" : [ "country", "political" ] } ], "formatted_address" : "Buffalo, NY, USA", "geometry" : { "bounds" : { "northeast" : { "lat" : 42.9664549, "lng" : -78.795157 }, "southwest" : { "lat" : 42.826023, "lng" : -78.9337276 } }, "location" : { "lat" : 42.88644679999999, "lng" : -78.8783689 }, "location_type" : "APPROXIMATE", "viewport" : { "northeast" : { "lat" : 42.9664549, "lng" : -78.795157 }, "southwest" : { "lat" : 42.826023, "lng" : -78.9142665 } } }, "place_id" : "ChIJoeXfUmES04kRcYEfGKUEI5g", "types" : [ "locality", "political" ] } ], "status" : "OK" } From the above json repsonse the data is extracted in a data frame shown in following codes We remove all those locations which are not in United State or which are in United State but no information about the state.<jupyter_code>getGeoCode <- function(address,userScreenName,verbose=FALSE) {
    if(verbose) cat(address,"\n")
    url <- getGeoCodeUrl(address)
    doc <- getURL(url)
    docData <- fromJSON(doc,simplify = FALSE)
    
    if(docData$status=="OK") {
        
        lat <- docData$results[[1]]$geometry$location$lat
        lng <- docData$results[[1]]$geometry$location$lng        
        
        address_component <- parseAddressComponents(docData$results[[1]]$address_components)
        state = address_component[1]
        country = address_component[2]
        
        if(!invalid(country) & country == "United States"){
            
            if(invalid(state)){
                latlongUrl = getLatLngURL(lat,lng)
                latlngDoc <- getURL(latlongUrl)
                latlngDocData <- fromJSON(latlngDoc,simplify = FALSE)
    
                if(latlngDocData$status=="OK") {
                    address_component_latlng <- parseAddressComponents(latlngDocData$results[[1]]$address_components)
                    latlngState = address_component_latlng[1]
                    latlngCountry = address_component_latlng[2]
                    return(c(userScreenName, lat, lng, latlngState, latlngCountry))
                }
            }else{
                return(c(userScreenName, lat, lng, state, country))
            }
        } 
        Sys.sleep(0.5)
    }
}<jupyter_output><empty_output><jupyter_text>1st line creates a sqlite db file 2nd line store and load tweets database which is backend registered 3rd line store the tweets in a table named "tweets", which is automatically provided by twitterR<jupyter_code>if(!isCustomSearch){
    mydbName = toString("DIClab1.db")
}else{
    mydbName = toString("custom.db")
}
conn = dbConnect(SQLite(), dbname = mydbName)
register_sqlite_backend(mydbName)<jupyter_output><empty_output><jupyter_text>Searches the tweets from the user input search string vector and equally divides them in equal chunks per input string in vector. So that the total sum of all the tweets is LIMIT, i.e 20000 here.<jupyter_code>size = length(userSearchStrings)
for(searchStr in userSearchStrings){
    tweets <- searchTwitter(searchStr,LIMIT/size)
    store_tweets_db(tweets)
}<jupyter_output><empty_output><jupyter_text>Makes a connection with the database<jupyter_code>con = dbConnect(SQLite(), dbname = mydbName)<jupyter_output><empty_output><jupyter_text>Query the database for all the screen names of the users.<jupyter_code>tweetsUserNames = dbGetQuery(con,"SELECT screenName from tweets")<jupyter_output><empty_output><jupyter_text>Generate a vector by taking transpose of the above result.<jupyter_code>userNamesVec = c(t(tweetsUserNames))<jupyter_output><empty_output><jupyter_text>Prints a few user's screen names<jupyter_code>head(userNamesVec)<jupyter_output><empty_output><jupyter_text>Queries the twitter with all the screen names to get the user information<jupyter_code>lookupUserNames <- lookupUsers(userNamesVec)<jupyter_output><empty_output><jupyter_text>Converts the data collected in the previous step to a data frame<jupyter_code>userNamesDf = twListToDF(lookupUserNames)<jupyter_output><empty_output><jupyter_text>Save this data frame so that the previous heavy work needs not to be repeated everytime<jupyter_code>if(!isCustomSearch){
    saveRDS(userNamesDf,file = "userNamesDf.Rda")
}else{
    saveRDS(userNamesDf,file = "customUserNamesDF.Rda")
}<jupyter_output><empty_output><jupyter_text>Load the userNameDf from the saved file<jupyter_code>if(!isCustomSearch){
    userNamesDf = readRDS(file = "userNamesDf.Rda")
}else{
    userNamesDf = readRDS(file = "customUserNamesDF.Rda")
}<jupyter_output><empty_output><jupyter_text>Print a few items of the data frame<jupyter_code>head(userNamesDf)<jupyter_output><empty_output><jupyter_text>This is the heart of the whole code. It generates a data frame which will contain the screen name, latitude, longitude, state and country of the all the users. We keep storing the data frames as they arrive so that in case of any problem we can resume from the failure point. NOTE: Here in this experiment we generated some 5000+ locations.<jupyter_code>userLocationDataDf = data.frame(screenName = as.character(),Lat = double(),Long = double(),State = as.character(), Country = as.character())

sizeOfDf = length(userNamesDf$screenName)

for(i in 1:sizeOfDf){
    tempUserLocation = userNamesDf$location[i]
    tempUserScreenName = userNamesDf$screenName[i]
    if(tempUserLocation != ""){
        tempVec = getGeoCode(tempUserLocation,tempUserScreenName)
        if(!invalid(tempVec)){
            tempDf = data.frame(screenName = as.character(tempVec[1]),Lat = as.numeric(tempVec[2]),Long = as.numeric(tempVec[3]),State = as.character(tempVec[4]),Country = as.character(tempVec[5]))
            userLocationDataDf = rbind(userLocationDataDf,tempDf)
            if(!isCustomSearch){
                saveRDS(userLocationDataDf,file = "userLocationDataDf.Rda")
            }else{
                saveRDS(userLocationDataDf,file = "customUserLocationDataDf.Rda")
            }
        }
    }
}<jupyter_output><empty_output><jupyter_text>Save the above generated data frame since the above process is time consuming.<jupyter_code>if(!isCustomSearch){
    saveRDS(userLocationDataDf,file = "userLocationDataDf.Rda")
}else{
    saveRDS(userLocationDataDf,file = "customUserLocationDataDf.Rda")
}<jupyter_output><empty_output><jupyter_text>Load the data frame for further queries<jupyter_code>if(!isCustomSearch){
    userLocationDataDf = readRDS(file = "userLocationDataDf.Rda")
}else{
    userLocationDataDf = readRDS(file = "customUserLocationDataDf.Rda")
}<jupyter_output><empty_output><jupyter_text>Print and check if the data frame generated is in right format or not<jupyter_code>head(userLocationDataDf)<jupyter_output><empty_output><jupyter_text>Prints the number of items in the data frame<jupyter_code>print(length(userLocationDataDf$screenName))<jupyter_output>[1] 216<jupyter_text>This code basically aggregates the data as per the different state and club them together. This code also uses mean function to generate the latitude and logitude of the state the mean value of them will still be in the state.<jupyter_code>aggDataDf = aggregate(x = list(Latitude = userLocationDataDf$Lat, Longitude = userLocationDataDf$Long), by = list(State=userLocationDataDf$State), FUN = mean)<jupyter_output><empty_output><jupyter_text>This part of code generates the frequency table from the userLocationDataDf data frame per state.<jupyter_code>tweetLocationFreqDf = aggregate(x = list(Tweet_Count = userLocationDataDf$State), list(userLocationDataDf$State), length)<jupyter_output><empty_output><jupyter_text>Merges the two data frames generated above<jupyter_code>aggUserLocationDataDf = data.frame(aggDataDf,tweetLocationFreqDf)<jupyter_output><empty_output><jupyter_text>Save this data frame before moving further<jupyter_code>if(!isCustomSearch){
    saveRDS(aggUserLocationDataDf,file = "aggUserLocationDataDf.Rda")
}else{
    saveRDS(aggUserLocationDataDf,file = "aggCustomUserLocationDataDf.Rda")
}<jupyter_output><empty_output><jupyter_text>Reload the data in a data frame again for futher processing<jupyter_code>if(!isCustomSearch){
    aggUserLocationDataDf = readRDS(file = "aggUserLocationDataDf.Rda")
}else{
    aggUserLocationDataDf = readRDS(file = "aggCustomUserLocationDataDf.Rda")
}<jupyter_output><empty_output><jupyter_text>Print and check if the data frame is in right format or not.<jupyter_code>head(aggUserLocationDataDf)<jupyter_output><empty_output><jupyter_text>get_map give the map of the specified location.<jupyter_code>USAMap = get_map(location = 'United States', zoom = 4)<jupyter_output>Map from URL : http://maps.googleapis.com/maps/api/staticmap?center=United+States&zoom=4&size=640x640&scale=2&maptype=terrain&language=en-EN&sensor=false
Information from URL : http://maps.googleapis.com/maps/api/geocode/json?address=United%20States&sensor=false<jupyter_text>This code generates the points as per the data provided. We need to pass the map, generated in the previous step and all the points which needs to be plotted<jupyter_code>mapPoints <- ggmap(USAMap) +
    geom_point(aes(x = aggUserLocationDataDf$Longitude, y = aggUserLocationDataDf$Latitude, size = Tweet_Count), data = aggUserLocationDataDf, color = "blue")<jupyter_output><empty_output><jupyter_text>Draws the map with the marked points.<jupyter_code>mapPoints<jupyter_output>Warning message:
"Removed 2 rows containing missing values (geom_point)."
{'fraction_non_alphanumeric': 0.11615331964407939, 'fraction_numerical': 0.013826146475017112, 'mean_word_length': 4.927383367139959, 'pattern_counts': {'":': 0, '<': 158, '<?xml version=': 0, '>': 138, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '401895', 'license': 'Apache-2.0', 'n_tokens_mistral': 4899, 'n_tokens_neox': 4652, 'n_words': 1076, 'path': 'Lab1/DIC_LAB_1_QNS_2.ipynb', 'repo_name': 'vk3105/Data-Intensive-Programming-CSE-587', 'size': '14644'}