<jupyter_start><jupyter_text>NY Time API Scrape Tools in this notebook are for the use of scraping news articles using the NY Times API. The following code will make requests to the API, scrape 1,000 of the most news recent articles, and parse the results for each article headline, snippet, and url into a DataFrame.   Please visit https://developers.nytimes.com/ before using this tool to review the NY Times API terms of service, obtain your personal NY Times developer API key (free), and research any additional information relating to the use of the API.  Imports<jupyter_code>import requests
import pandas as pd
import time<jupyter_output><empty_output><jupyter_text>NY Times API Scrape Enter your API key and the topic you which to search for, as a string in the cell below<jupyter_code>api_key = ''
topic = 'fire'<jupyter_output><empty_output><jupyter_text>This function makes multiple requests to the API, pulls 1,000 of the most recent articles (10 per request) for the topic designated above. This is the maximum amount of request the API allows for, adjust the range in the function to collect a lesser amount. Output is a list of dictionaries with all article specific information contained in the json.<jupyter_code>def nytimes_api_scrape(topic, api_key):
    article_list = []
    # adjust range to scrape less then 1,000
    for i in range(100):
        
        # gives an update every 100 articles
        if i % 10 == 0:
            print('{} articles gathered so far'.format(i*10))
        url = 'https://api.nytimes.com/svc/search/v2/articlesearch.json?q='+ topic + '&'+ str(i) +'&api-key='+ api_key
        res = requests.get(url)
        
        # checks to see if request was a success and adds to list
        if res.status_code == 200:
            the_json = res.json()
            article_list.extend(the_json['response']['docs'])
        
        else:
            print('Bad request status {}'.format(response.status_code))
            break
        
        # intentionally delay requests to the server
        time.sleep(5)
    
    print('You gathered {} articles about {}'.format(len(article_list), topic))    
    return article_list<jupyter_output><empty_output><jupyter_text>**Important Note:** total run time of the function is about 8.5 minutes<jupyter_code>fire_articles = nytimes_api_scrape(topic, api_key)
fire_articles[0].keys()<jupyter_output><empty_output><jupyter_text>This function takes the headline, snippet, and url for each article and returns a DataFrame for output<jupyter_code>def to_df(article_list):
    key_list = []
    for i in range(len(fire_articles)):
        key_list.append({
            'headline': article_list[i]['headline']['main'],
            'snippet': article_list[i]['snippet'],
            'web_url': article_list[i]['web_url']
        })
    df = pd.DataFrame(key_list)
    return df
df = to_df(fire_articles)<jupyter_output><empty_output><jupyter_text>Output<jupyter_code># save to csv to use for use in other notebook
df.to_csv('./datasets/{}'.format(topic))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.091324200913242, 'fraction_numerical': 0.010763209393346379, 'mean_word_length': 3.7403400309119013, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '148093', 'license': 'MIT', 'n_tokens_mistral': 924, 'n_tokens_neox': 881, 'n_words': 316, 'path': 'code and data/01-scraping-code.ipynb', 'repo_name': 'KevinJpotter/FEMA-News-Classifier', 'size': '3066'}