<jupyter_start><jupyter_text>Data Crawling Sadly, there are times wherein data will not be handed on a silver platter there are times wherein you have to gather data yourselves. Gaining exposure through various data collection methods will be a vital skill to have in order to thrive as a budding data scientist. In this notebook, we will be gathering data from through an Application Programming Interface(API). An API is an interface that allows programs to communicate with each other. A wide use of APIs can be found on online banking transactions. Usually, banks would expose APIs that allow developers cerain access to the bank's system (ex. money transfer, balance checking). In turn, developers will use these APIs to build the functionalities of their applications. In our case we use APIs to extract data to do our analysis and build our models.  Sourced from  <NAME><jupyter_code>import pandas as pd 
import datetime
import csv
import os 
import requests 
import datetime as dt
import time<jupyter_output><empty_output><jupyter_text>To familiarize ourselves with the use APIs we will be extracting data from reddit. Reddit is a media aggregation website that contains multiple communities. You guys can imagine it as a "forum-esque" website where users can discuss about anything under the sun. The tool that we will be using to get reddit data is  pushift. Pushshift, is an open source big data storage platform that copies submissions on reddit. The developer of this project has exposed an API which allows users to search for comments, aggregate data, query data for a specific range and many more. One limitation of the tool is that it saves the metadata of the submissions the time it was posted on reddit therefore scores, upvotes and downvotes are not updated.  Let's Start! To access pushshift we will be using the `requests` library of Python. This library allows us to do HTTP requests on the API. Basically, this library allows us to communicate with the API. There are 3 main entry to points for the API these are: `/reddit/comment/search` `/reddit/submission/search` `/reddit/subreddit/search` This means you can search based on texts coming from comments, submissions or subreddit. To filter your query the API offers a `param ` attribute. There are various parameters which you can find here. For this tutorial we will only be using 7. `after` - filter results to those made after a given date `before`- filter results to those made before a given date `sort_type` - sort by a specific attribute default value is "created_utc" other options are  ("score", "num_comments", "created_utc") `sort` - sort results in specific order ("asc" or "desc") `subreddit` - name of subreddit `size` - number of results to return `fields` - specifcy which fields to return   Sample API Query The query below gets submissions from DLSU's reddit community from April 1, 2020 to April 30, 2020 sorted in descending order based on score with only 20 results. We also indicate what fields the API should return. If you notice there is something weird about the date we placed in the `after` and `before` parameters. The API uses an epoch timestamp. This is the timestamp used by Unix. For us to interface with the API properly we can use this time converter. Just type in the date that you want to be converted to Unix. You may notice that after calling the requests library we use the `.json` method. This converts the results of the request into a JSON object. JSON is a widely used file format usually used to communicate between applications. In our case Reddit API and this notebook.<jupyter_code>URL = "https://api.pushshift.io/reddit/submission/search/"  #query submissions
PARAMS = {
    'after': 1483228800, #get dates after March 31, 2020
    'before': 1588291200, #get dates before May 1, 2020
    'sort_type': 'score', # sort by score
    'sort': 'desc', # sort in descending order
    'subreddit': 'dlsu', # do a search on DLSU subreddit
    'size': 20, # give only 20 search results
#     'fields': ["id","title","selftext","score","num_comments","created_utc"] #return only the following fields
}

#use the requests library to query pushshift api
r = requests.get(url = URL, params=PARAMS)
#parse returned data to a json object
r.json()<jupyter_output><empty_output><jupyter_text>Scaling the API Query  What if we want to scale the API and be able to get all reddit posts from April 1, 2020 to April 30, 2020? One way to do this would be to call the API for each day from Apirl 1 to April 30. For each day we set the size of the results to 500. The reason we are limiting it to 500 is the API only accepts values <= 500 for the size parameter. Note: We can see a limitation in this approach, what if a reddit page gets more than 500 posts a day. This a reality that you may encounter when dealing with APIs there will be limitations in how much you can query or extract. In the case of reddit, pushshift has provided a data dump through this link.<jupyter_code>def to_utc(date):
    #This function converts an object to UTC. This is to automate the conversion 
    #of dates instead of going to https://www.unixtimeconverter.io/ 
    return int(date.replace(tzinfo=dt.timezone.utc).timestamp())
    
def to_readable_date(timestamp):
    #This function converts the UTC format to a Year-Month-Day format 
    return dt.datetime.fromtimestamp(timestamp).strftime("%Y-%m-%d")

#Declare start and end of reddit posts to extract 
start_date = dt.datetime.strptime("2020-04-01", "%Y-%m-%d")
end_date = dt.datetime.strptime("2020-04-05", "%Y-%m-%d")

#Create a range of dates to iterate 
#Note: Periods here represents the number of days it will create from the start date 
#We also do a +2 since it will only generate up to April 29. We inlcude May 1 
#since we want to get data from the last day which is April 30 to May 1 
date_range = (pd.date_range(
                start_date, 
                periods=(end_date - start_date).days + 2)
              .tolist())

#prepare the parameters needed to call the API
sort_type="score"
sort="desc"
fields=["id","title","selftext","score","num_comments","created_utc"]
subreddit = 'dlsu'
url = "https://api.pushshift.io/reddit/submission/search/"
results = []
#loop through the dates 
for i, s_date in enumerate(date_range):
    #prevents us from getting an index out of range error
    if i != len(date_range)-1:
        #declare end date 
        e_date = date_range[i+1]
        #call the API
        r = requests.get(url = url, params={
            'after': to_utc(s_date),
            'before': to_utc(e_date),
            'sort_type': sort_type,
            'sort': sort,
            'subreddit': subreddit,
            'fields': fields,
            "size": 500
        })

        #add logs 
        print(f"Doing {s_date.strftime('%Y-%m-%d')} to {e_date.strftime('%Y-%m-%d')}")
        if r.status_code == 200:
            results.append(r.json()['data'])
            print("=====Done")
        else:
            print("=====Skipped")
        #so that we dont get blocked from abusing the API we call it after pausing for 1 second
        time.sleep(1)<jupyter_output>Doing 2020-04-01 to 2020-04-02
=====Done
Doing 2020-04-02 to 2020-04-03
=====Done
Doing 2020-04-03 to 2020-04-04
=====Done
Doing 2020-04-04 to 2020-04-05
=====Done
Doing 2020-04-05 to 2020-04-06
=====Done<jupyter_text>The results variable will be a list of lists and within a list is a dictionary object. Now for us to properly save this to a dataframe we would need to flatten the list and create a list of dictionary objects instead.<jupyter_code>results<jupyter_output><empty_output><jupyter_text>Converting data into a tableGiven that the JSON representation is a bit complex, additional processing would need to be done to be able to reogranize or restructure the data into a "flat" list of dictionary objects.Once we have the flat list, we can easily pass this to `pandas` which will make our data easier to analyze.<jupyter_code>flat_list = []
#loop through the reddit results
for sublist in results:
    #check if sublist is not empty. The reason we have empty lists is because there are days wherein there are no submissions
    if sublist is not None:
        #for each dictionary in the sublist add it to the flat list 
        for item in sublist:
            flat_list.append(item)

#pandas has a useful function called from_dict which will convert a list of dictionary objects into a dataframe
df = pd.DataFrame.from_dict(flat_list)
display(df.head())
df.to_csv("reddit_dlsu.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07301401869158879, 'fraction_numerical': 0.02219626168224299, 'mean_word_length': 3.8449349179400114, 'pattern_counts': {'":': 1, '<': 22, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '205694', 'license': 'MIT', 'n_tokens_mistral': 2473, 'n_tokens_neox': 2281, 'n_words': 1195, 'path': 'Module_1_Data_Collection/01_Reddit_Crawler.ipynb', 'repo_name': 'chechsap/foundations-of-data-science', 'size': '8568'}