<jupyter_start><jupyter_text>AboutNotebook to ping Heathrow Arrival and Departure API endpoints InputsRequired Inputs:* `url` to ping* `headings` url request headings Outputs* Failure:    * Bugs up e.g. 404 Error if requested date is not recent enough.* Success    * Saves response soup as JSON        * Can be saved as Json        * List of dictionaries        * Each dictionary represents a flight  Setup<jupyter_code>import datetime
import json

from bs4 import BeautifulSoup
import requests<jupyter_output><empty_output><jupyter_text>Settings  Inputs & Parameters<jupyter_code># Date
days_before_today = 1
date_today = (datetime.date.today()-datetime.timedelta(days = days_before_today))
iso_date_str = date_today.isoformat()

# Flight Direction
direction = "departures"<jupyter_output><empty_output><jupyter_text>Derived & Constant Settings<jupyter_code># String Mapping
## Used to get the required url string
direction_order_by = {'arrivals':'localArrivalTime', 'departures': 'localDepartureTime'}

# Flight Data Request pattern
url = f'https://api-dp-prod.dp.heathrow.com/pihub/flights/{direction}?date={iso_date_str}&orderBy={direction_order_by[direction]}'
# Headers copied from Chrome DevTools XHR request
## Optional Cookie is removed
headers = {
"accept": "application/json, text/plain, */*",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7,ga;q=0.6",
"origin": "https://www.heathrow.com",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-site",
"user-agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36",
}<jupyter_output><empty_output><jupyter_text>Output Settings<jupyter_code>output_file_name = f'heathrow_{direction}_{iso_date_str}_test.json'<jupyter_output><empty_output><jupyter_text>Functions<jupyter_code>def saveJson(soup, fileName):
    """Save soup as json"""
    json_data = json.loads(str(soup))
    with open(fileName, 'w') as f:
        json.dump(json_data, f)<jupyter_output><empty_output><jupyter_text>Main Part Ping Endpoint<jupyter_code>url
r = requests.get(url, headers = headers)
soup = BeautifulSoup(r.text, 'html.parser')<jupyter_output><empty_output><jupyter_text>Checks<jupyter_code># Manually check response code
r
# Manually check Soup looks ok
soup # 04th<jupyter_output><empty_output><jupyter_text>Save<jupyter_code>print(output_file_name)
saveJson(soup, output_file_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12921348314606743, 'fraction_numerical': 0.01725521669341894, 'mean_word_length': 6.815047021943574, 'pattern_counts': {'":': 8, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '177114', 'license': 'MIT', 'n_tokens_mistral': 922, 'n_tokens_neox': 880, 'n_words': 180, 'path': 'notebooks/NB02 - Ping Endpoints - Heathrow.ipynb', 'repo_name': 'FergM/flight_tables', 'size': '2492'}