<jupyter_start><jupyter_text>Data Mining IThis  notebook is intended to perform the following processes:    1.1 Read-in news articles from newsAPI for a given date range, and up to five queries (passed as a list).    1.2 Extract features native to the articles (e.g. url).    1.3 Perform data cleanup and preprocessing.    1.4 Split dataset into n-csv-files for distrubuted computation or batching.  __Documentation__newsAPI (https://newsapi.org/) has limited documentation as to how their backend works. However, when developing this notebook the following website was frequently referenced -- as their documentation appears to closely match the behavior of newsAPI. https://docs.aylien.com/newsapi/getting-startedMoreover, here is the user agreement:https://newsapi.org/terms ___ **Begin Data Mining I:** Read-in NewsAPI feed for a given date range<jupyter_code>### NEWSAPI RELATED ###
# keys: 
#rhkey = '<KEY>'
#r1key  = '<KEY>'
#r2key = '40d53e49ee3543a3b162e6a453e2e373'
m1key = '211fc2107848473e99c1f235b400a07f'
# m2key = '<KEY>'
# m3key = '658cd65a714349fdbb7e8dd6ce59e9c4'
#m4key  = '8ba091b7a47b4c9a9162a83ca72eb1ca'
#e1key  = '2bc85776a0c14af6b9937366ad683e2f'
#e2key = '22e5c3a8f0ee4fa59aaf384ba9395a86'
#e3key = 'c554f8fb27ca4be1862192b44ee4425d'


# Install API 
# !pip install newsapi-python

# Import Client
from newsapi import NewsApiClient

# Initialize Client (create object)
news_api = NewsApiClient(api_key = m1key)
print(type(news_api))<jupyter_output><class 'newsapi.newsapi_client.NewsApiClient'><jupyter_text>__1.1 Read-in news articles from newsAPI for a given date range__  Function: **get_news**Function establishes values to be used for control of loop then calls functions used to extract news article data<jupyter_code>def get_news(query, start, stop, sort, lang, article_count, page_count, init):
    """
    control function for iterating over 100 pages of newsAPI's content 
    function then controls subordinate fuctions that extract ~100 articles per each page
    """
    
    import math
    import time  
    
    # extract information about response file to ensure proper loop control
    params = get_params(query, start, stop, sort, lang, article_count, page_count)

    # variable referencing
    status = params['status']
    results = params['totalResults']

    # Confirmation of data extraction
    print("\nVerify Read-in Process:", status)
    print("Number of Articles Correctly Read: ", results)
    print(type(params), params.keys())
           
    # per page article extraction stop variable -- if number of articles is greater than number articles per page
    loops = math.ceil(results/article_count)
    
    # batching control
    begin = 0 + init
    terminate = article_count + init
    print("Total number of iterations (pages):",loops)
    
    # check to ensure loop does not extract over 100pages*100endpoints=10000 articles
    if loops < terminate:
        terminate = loops
        
    print("Page range being extracted", begin, terminate)
    
    if page_count == 'all' or article_count <  results:
        print("\n\nExtracting News Data...\n")
        full_df = pd.DataFrame()
    
        # function is called withinin while, is subject to number of pages available as a function of total no. articles
        while begin < terminate:
            begin += 1
            page = begin  # for referencing clarity
            
            ### newsAPI has MAX HIT LIMIT of 60 per minute ### 
            time.sleep(1)   # delay of 1 second
            print("Extracting Page", page)

            # call sequencial pages of articles and appends to df
            df = news_data(query, start, stop, sort, lang, article_count, page)
            full_df = full_df.append(df, ignore_index = True)
            
        print('Batch extraction completed:',begin,'of',terminate)
        return(full_df)            
    else:
        # extracts articles assuming articles >= pages
        print("Possible Invalid Parameters: Check values")
        brief_df = news_api.get_everything(q = query,
                                          from_parameter= start,
                                          to= stop,
                                          sort_by= sort,
                                          language= lang,
                                          page_size= int(article_count)
                                         )
        return(brief_df)<jupyter_output><empty_output><jupyter_text>Function: **get_params**Function runs an initial newsAPI call, used to store values for controlling loops<jupyter_code>def get_params(query, start, stop, sort, lang, article_count, page_count):
    """
    function accepts similar parameters to master function get_news.
    get_params is used to extract parameters to be used in controlling other functions 
    """
    
    print("\nExtracting Parameters for newsAPI...\n")
    params = news_api.get_everything(q = query,
                                     from_param= start,
                                     to= stop,
                                     sort_by= sort,
                                     language= lang,
                                     page_size= int(article_count)
                                    )
    
    # Confirmation of data extraction
    print("Read-in Status of Given Date Range:", params['status'])
    print("Number of Articles in Given Date Range: ", params['totalResults'])
    
    return(params)<jupyter_output><empty_output><jupyter_text>Function: **news_data**Function handles cases, and extracts values within 'articles'. Returns dataframe of contents: *Index(['author', 'description', 'publishedAt', 'source', 'title', 'url','urlToImage'],dtype='object')*<jupyter_code>def news_data(query, start, stop, sort, lang, article_count, page):
    """
    Principal data extraction function - can handle various relationships between no.pages and no.articles 
    """
    
    if isinstance(page, int):
        params = news_api.get_everything(q = query,
                                         from_param= start,
                                         to= stop,
                                         sort_by= sort,
                                         language= lang,
                                         page_size= int(article_count),
                                         page = int(page)
                                        )
    ########### if params['articles'] throws error ########### 
    # either endpoint limit was met (10000)                  #
    # too many endpoint requests in a month (1000 per month) #
    # change to new api key.                                 #
    ##########################################################
    return(pd.DataFrame(params['articles']))<jupyter_output><empty_output><jupyter_text>User provided parameters and function call.<jupyter_code>#01/26/18 to 03/26/18
query = 'Bitcoin'         # can handle a list of up to five search topics
start = '2018-10-09'      # yyyy-mm-dd
stop  = '2018-10-14'
sort  = 'publishedAt'
lang  = 'en'
article_count = 100       # default is 20
page_count = 'all'        # enter 1, 2, ... Notes: 'all' iterates over all articLes<jupyter_output><empty_output><jupyter_text>___--- __NOTE__Since newsAPI has a daily limit of endpoint requests (1000 per day), batching needs to be implemented.  The following cells controls the initialization for subsequent cells -- to ensure appropropriate, and sequential article extraction.Also, webAPI has a limit of 10,000 articles per key -- unless you want to pay for the monthly access.___---<jupyter_code>## ---------------------------------------------------------------------------- ##
## KEY: Initialize accroding to batch number (i.e. n*100, where n is the batch) ##
## USE: n > 0 ---- only once we can extract above 10000 articles
n = 0
init = n*100
## ---------------------------------------------------------------------------- ##
# object is the result of the following functions: 'get_params', 'get_news', and 'get_data'
news = get_news(query, start, stop, sort, lang, article_count, page_count, init)<jupyter_output>Extracting Parameters for newsAPI...

Read-in Status of Given Date Range: ok
Number of Articles in Given Date Range:  858

Verify Read-in Process: ok
Number of Articles Correctly Read:  858
<class 'dict'> dict_keys(['status', 'totalResults', 'articles'])
Total number of iterations (pages): 9
Page range being extracted 0 9


Extracting News Data...

Extracting Page 1
Extracting Page 2
Extracting Page 3
Extracting Page 4
Extracting Page 5
Extracting Page 6
Extracting Page 7
Extracting Page 8
Extracting Page 9
Batch extraction completed: 9 of 9<jupyter_text>Explore nested key/value pairs from newsAPI data<jupyter_code>## ENSURE API WAS ACCESSED ##
# if news.keys includes 'author', 'description', etc., success!

news.keys()
print(len(news))
print(news.keys())
news.head(5)
len(news['url'].unique())<jupyter_output>858
Index(['author', 'content', 'description', 'publishedAt', 'source', 'title',
       'url', 'urlToImage'],
      dtype='object')<jupyter_text>__1.2 Extract features native to the articles__  Function: **get_info**Function extracts variables from dataframe and stores each as a list, returning all of them as a single dataframe.__Note:__ *urlToImage* is not included in this process, as we are uncertain as to the value of the feature<jupyter_code>def get_info(df):
    """
    Accepts a dataframe of newsAPI articles, and controls all subbordinate functions that preprocess data
    """
    
    import copy
    
    author = []
    title = []
    publisher = []
    publish_url = []
    timeStamp = []
    content = []
    
    
    # loop appends rows to respective lists 
    for col_name in df:
        for index in df[col_name]:
            if col_name == 'author':
                author.append(index)
            elif col_name == 'title':
                title.append(index)
            elif col_name == 'source':
                name = index['name']
                publisher.append(name)
            elif col_name == 'url':
                publish_url.append(index)
            elif col_name == 'publishedAt':
                timeStamp.append(index)
            elif col_name == 'content':
                content.append(index)
            else:
                continue
    
    # merge lists and return them as dataframe.
    df = pd.DataFrame({'author' : author,
                       'title' : title,
                       'publisher' : publisher,
                       'source_url' : publish_url,
                       'timeStamp' : timeStamp,
                       'content' : content})
    
    return(df)<jupyter_output><empty_output><jupyter_text>Completed newsAPI Read-in Process:  newsDF contains features extracted from raw newsAPI feed, for a given data range, and query.<jupyter_code># Object creation
newsDF = get_info(news)
# Verifying correct data extraction
print("\nDataFrame Dimensions:", newsDF.shape, "\n")
newsDF.head(3)<jupyter_output>DataFrame Dimensions: (858, 6)<jupyter_text>__1.3 Perform data cleanup and preprocessing.__ The following functions perform basic clean up on a dataframe. The purpose is to prepare the file to write-out (csv).<jupyter_code># Replace 'None' values
def findNone(df):
    """
     Receives pandas datraframe, and removes null entries from author feature
    """
    print("Removed 'None' values in author feature...")
    author = df['author']
    publisher = df['publisher']
    
    for i in range(len(df)):
        if pd.isnull(author.loc[i]):
            author.loc[i] = publisher.loc[i]
    return(df)
# Remove gaps 
def gapStrip(df):
    """
    Receives pandas dataframe and leading and traling empty space`
    """
    df.columns = map(str.strip, df.columns) 
    print("Removed leading and trailing spaces and tabs...")
    # element-wise operation
    f = lambda x: x.strip() if (isinstance(x,str)) else x
    df = df.applymap(f)
    return(df)
# Standardize time stamps
def std_timeStamp(df):
    """
    Receives pandas dataframe and standardizes time stamps 
    """
    import datetime
    # Check to see time stamps are in zero timezones
    print("Converted Time Stamps to Desired Standard Formating...")
    for time in df['timeStamp']:
        if time.endswith('Z'):
            df['timeStamp'] = pd.to_datetime(df['timeStamp'],
                                             infer_datetime_format = True,
                                             utc = True)                       # returns a type '.Timestamp'
            return(df)
        else:
            print("Revisit appropriate variable or function to deal with time zones that are not zero")
def feature_clean(df):
    """
    Performs Generic Cleanup and Preprocessing on a given dataframe sourced from newsAPI
    """
    
    temp = findNone(df)           # removes missing values from author column
    temp2 = gapStrip(temp)        # remove leading and trailing white space
    temp3 = std_timeStamp(temp2)  # convert time stamps to 'utc' standard
    return(temp3)
riskEx_df = feature_clean(newsDF)
# Ensure data was preprocessed
riskEx_df.head(5), riskEx_df.tail(5)
## Check file size
riskEx_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 858 entries, 0 to 857
Data columns (total 6 columns):
author        858 non-null object
title         858 non-null object
publisher     858 non-null object
source_url    858 non-null object
timeStamp     858 non-null datetime64[ns, UTC]
content       820 non-null object
dtypes: datetime64[ns, UTC](1), object(5)
memory usage: 40.3+ KB<jupyter_text>__1.4 Write out to csv.__<jupyter_code># write out n-csv-files each with 100 rows. Process is done to reduce computational load
riskEx_df.to_csv('rawData_test1006.csv', index_label = False)
df = pd.read_csv('rawData_test1006.csv')
#print(df.info())
print(len(df))
len(df['content'].unique())

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
type(df['content'])
df['content'].dropna(axis = 0,inplace = True)
# df[df['description'].isnull()]
# df[df['description'].isnull()]
df.iloc[241,:]
bag_of_words = vectorizer.fit(df['content'].tolist())
bag_of_words = vectorizer.transform(df['content'].tolist())
words_cor = pd.DataFrame(bag_of_words.toarray(), columns=vectorizer.get_feature_names())
words_cor.head()
words_cor.sum().sort_values(ascending = False)
asc = words_cor.sum().sort_values(ascending = False).index
asc
from nltk.corpus import stopwords
stop_words = set(stopwords.words("english"))
type(words_cor.sum())
dtt = words_cor.sum().to_frame("freq")
filtered = [w for w in asc if not w in stop_words]
freqs = words_cor[filtered].sum().to_frame("freq")
freqs = freqs[:100]
freqs.to_csv('corporaZ*W.csv',index_label = False,index=False)
freqs<jupyter_output><empty_output><jupyter_text>__Note:__ if wanting to create batches of raw data files containing n-articles, use the following<jupyter_code>#def df_to_csvs(df):
#    articlesPage = int(100)
#    totalArticles = len(df)
#    batchSize=round(totalArticles/articlesPage)          # number of rows in single output file
        
#    for id, df_i in  enumerate(np.array_split(df, batchSize)):
#        df_i.to_csv('rawData_{id}.csv'.format(id=id), index_label = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1091324200913242, 'fraction_numerical': 0.022635355512067842, 'mean_word_length': 2.6853365384615384, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 52, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 37, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '374664', 'license': 'MIT', 'n_tokens_mistral': 4559, 'n_tokens_neox': 4244, 'n_words': 1362, 'path': 'past-team-code/Fall2018Team1/Exploration/Bag_of_words(Zy&Wb).ipynb', 'repo_name': 'shun-lin/project-paradigm-chatbot', 'size': '15411'}