<jupyter_start><jupyter_text>Features APIThe Features API provides features for ML Models across all types of demand causal factors, including attended events and non-attended events.It allows you to go straight to feature-importance testing and improving your models rather than having to worry about first building a data lake and then aggregating the data. Features API is new and features will be expanded and added to over time. The endpoint currently supports:* PHQ Attendance features (for attendance based events)* PHQ Rank features (for scheduled, non-attendance based events)* PHQ Viewership features (for viewership based events)This notebook will guide you through how to use the Features API.- [Setup](setup)- [Getting Started](getting-started)- [PHQ Attendance](phq-attendance)- [PHQ Rank](phq-rank)- [Academic Events](academic-events)- [Wide Data ranges](wide-data-ranges)- [Exploring the data](exploring-the-data)- [Concert trends for San Francisco vs Oakland](concert)- [Exploring TV Viewership with the Feature API](tv-viewership)- [Retrieving Multiple Locations](multiple-locations)- [List of lat/lon](lat-lon)- [Multiple Categories](multiple-categories)  Setup If using Google Colab uncomment the following code block.<jupyter_code># %%capture
# !git clone https://github.com/predicthq/phq-data-science-docs.git
# %cd phq-data-science-docs/features-api
# !pip install aiohttp==3.7.4.post0 asyncio==3.4.3 backoff==1.10.0 bleach==3.3.0 calmap==0.0.9 iso8601==0.1.14 matplotlib==3.4.2 numpy==1.20.3 pandas==1.2.4 seaborn==0.11.1 uvloop==0.15.2<jupyter_output><empty_output><jupyter_text>RequirementsIf running locally, configure the required dependencies in your Python environment by using the [requirements.txt](https://github.com/predicthq/phq-data-science-docs/blob/master/features-api/requirements.txt) file which is shared alongside the notebook.These requirements can be installed by running the command `pip install -r requirements.txt`<jupyter_code>import requests
import json<jupyter_output><empty_output><jupyter_text>Getting Started Access TokenYou will need an access token that has the `events` scope. The following link will guide you through setting up an access token: [https://docs.predicthq.com/guides/quickstart/](https://docs.predicthq.com/guides/quickstart/) Basic ConceptsEvery request to the API must specify which features are needed. Throughout the notebook you will become familiar with the naming convention for features, and we will talk about features by name.Each request must specify a date range and location (that applies to all features in that request).Certain groups of features support additional filtering/parameters.Results are at the daily level.Each request can currently fetch up to 90 days worth - for longer date ranges, multiple requests must be made and we have some examples of how to do that in this notebook. There is no pagination in this API.<jupyter_code># Paste your access token here
ACCESS_TOKEN = '<TOKEN>'

headers = {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'Authorization': f'Bearer {ACCESS_TOKEN}'
}

url = 'https://api.predicthq.com/v1/features'<jupyter_output><empty_output><jupyter_text>PHQ AttendanceThis group of features is based on PHQ Attendance. The following features are supported:- `phq_attendance_sports`- `phq_attendance_conferences`- `phq_attendance_expos`- `phq_attendance_concerts`- `phq_attendance_festivals`- `phq_attendance_performing_arts`- `phq_attendance_community`- `phq_attendance_academic_graduation`- `phq_attendance_academic_social`Each of these features includes stats. You define which stats you need (or don't define any and receive the default set of stats). Supported stats are:- `sum` (included in default set of stats)- `count` (included in default set of stats)- `min`- `max`- `avg`- `median`- `std_dev`These features also support filtering by PHQ Rank as you'll see in the example below.<jupyter_code>payload = {
    'active': { 
        'gte': '2019-11-28',
        'lte': '2019-11-30'
    },
    'location': {
        'place_id': [5224323, 5811704, 4887398]
    },
    'phq_attendance_concerts': True,
    'phq_attendance_sports': {
        'stats': ['count', 'std_dev', 'median'],
        'phq_rank': { 
            'gt': 50
        }    
    }
}

response = requests.request('POST', url, headers=headers, json=payload)
print(json.dumps(response.json(), indent=4))<jupyter_output>{
    "results": [
        {
            "date": "2019-11-28",
            "phq_attendance_concerts": {
                "stats": {
                    "count": 4,
                    "sum": 1580
                }
            },
            "phq_attendance_sports": {
                "stats": {
                    "count": 0,
                    "median": 0.0,
                    "std_dev": null
                }
            }
        },
        {
            "date": "2019-11-29",
            "phq_attendance_concerts": {
                "stats": {
                    "count": 53,
                    "sum": 46089
                }
            },
            "phq_attendance_sports": {
                "stats": {
                    "count": 3,
                    "median": 18654.0,
                    "std_dev": 6343.987407161383
                }
            }
        },
        {
            "date": "2019-11-30",
            "phq_attendance_concerts": {
                "stats": {
        [...]<jupyter_text>Example with a lat/lonThe location filter supports Place IDs and lat/lon with a radius.<jupyter_code>payload = {
    'active': { 
        'gte': '2019-11-28', 
        'lte': '2019-11-30' 
    },
    'location': {
        'geo': {
            'lat': 47.62064,
            'lon': -117.40401,
            'radius': '50km'
        }
    },
    'phq_attendance_concerts': True,
    'phq_attendance_sports': {
        'stats': ['count', 'std_dev', 'median'],
        'phq_rank': { 
            'gt': 50
        }    
    }
}

response = requests.request('POST', url, headers=headers, json=payload)
print(json.dumps(response.json(), indent=4))<jupyter_output>{
    "results": [
        {
            "date": "2019-11-28",
            "phq_attendance_concerts": {
                "stats": {
                    "count": 0,
                    "sum": 0
                }
            },
            "phq_attendance_sports": {
                "stats": {
                    "count": 0,
                    "median": 0.0,
                    "std_dev": null
                }
            }
        },
        {
            "date": "2019-11-29",
            "phq_attendance_concerts": {
                "stats": {
                    "count": 0,
                    "sum": 0
                }
            },
            "phq_attendance_sports": {
                "stats": {
                    "count": 1,
                    "median": 7063.0,
                    "std_dev": 0.0
                }
            }
        },
        {
            "date": "2019-11-30",
            "phq_attendance_concerts": {
                "stats": {
                    "count": 1,[...]<jupyter_text>PHQ RankThis group of features is based on PHQ Rank for non-attendance based events (mostly scheduled non-attendance based). The following features are supported:- `phq_rank_public_holidays`- `phq_rank_school_holidays`- `phq_rank_observances`- `phq_rank_politics`- `phq_rank_daylight_savings`- `phq_rank_health_warnings`- `phq_rank_academic_session`- `phq_rank_academic_exam`- `phq_rank_academic_holiday`Results are broken down by PHQ Rank Level (1 to 5). Rank Levels are groupings of Rank and are grouped as follows:- 1 = between 0 and 20- 2 = between 21 and 40- 3 = between 41 and 60- 4 = between 61 and 80- 5 = between 81 and 100Additional filtering for PHQ Rank features is not currently supported.<jupyter_code>payload = {
    'active': { 
        'gte': '2019-11-28', 
        'lte': '2019-11-30' 
    },
    'location': {
        'place_id': [5224323, 5811704, 4887398]
    },
    'phq_rank_school_holidays': True,
    'phq_rank_public_holidays': True,
    'phq_rank_health_warnings': True
}

response = requests.request('POST', url, headers=headers, json=payload)
print(json.dumps(response.json(), indent=4))<jupyter_output>{
    "results": [
        {
            "date": "2019-11-28",
            "phq_rank_health_warnings": {
                "rank_levels": {
                    "1": 0,
                    "2": 0,
                    "3": 0,
                    "4": 0,
                    "5": 0
                }
            },
            "phq_rank_public_holidays": {
                "rank_levels": {
                    "1": 0,
                    "2": 0,
                    "3": 0,
                    "4": 0,
                    "5": 1
                }
            },
            "phq_rank_school_holidays": {
                "rank_levels": {
                    "1": 0,
                    "2": 7,
                    "3": 123,
                    "4": 58,
                    "5": 3
                }
            }
        },
        {
            "date": "2019-11-29",
            "phq_rank_health_warnings": {
                "rank_levels": {
                    "1": 0,
                    "2": 0,
        [...]<jupyter_text>Academic EventsAcademic events are slightly different in that they contain both attended and non-attended events. You may have noted above that there are features in PHQ Attendance and PHQ Rank for academic events.There are 5 different academic types - namely:- `graduation`- `social`- `academic-session`- `exam`- `holiday`The types `graduation` and `social` are attendance based events which (as noted earlier) means we have the following PHQ Attendance features:- `phq_attendance_academic_graduation`- `phq_attendance_academic_social`The types `academic-session`, `exam` and `holiday` are non-attendace based events which means we have the following PHQ Rank features:- `phq_rank_academic_session`- `phq_rank_academic_exam`- `phq_rank_academic_holiday`  Wide Date RangesAs mentioned earlier, the API currently supports a date range of up to 90 days. In order to fetch data across a wider range, multiple requests must be made. Here is an example using asyncio that fetches a few years worth of data in parallel.There are a few functions we'll define first.<jupyter_code>import asyncio
import aiohttp
import uvloop
import iso8601  
import backoff
from datetime import timedelta
asyncio.set_event_loop(uvloop.new_event_loop())
query = {
    'location': {
        'place_id': [5391959]  # 5391959 = San Francisco
    },
    'phq_attendance_concerts': {
        'stats': ['count', 'std_dev', 'median', 'avg'],
        'phq_rank': {
            'gt': 60
        }
    },
    'phq_attendance_sports': {
        'stats': ['count', 'std_dev', 'median'],
        'phq_rank': {
            'gt': 50
        }
    },
    'phq_attendance_community': {
        'stats': ['count', 'std_dev', 'median'],
        'phq_rank': {
            'gt': 50
        }
    },
    'phq_attendance_conferences': {
        'stats': ['count', 'std_dev', 'median'],
        'phq_rank': {
            'gt': 50
        }
    },
    'phq_rank_public_holidays': True,
    'phq_attendance_academic_graduation': True
}
format_date = lambda x: x.strftime('%Y-%m-%d')
parse_date = lambda x: iso8601.parse_date(x)

# The API has rate limits so failed requests should be retried automatically
@backoff.on_exception(backoff.expo, (aiohttp.ClientError), max_time=60)
async def get(
    session: aiohttp.ClientSession,
    query: dict,
    start: str,
    end: str,
    **kwargs ) -> dict:
    
    payload = {
        'active': {
            'gte': start,
            'lte': end
        },
        **query
    }

    resp = await session.request('POST', url=url, headers=headers, raise_for_status=True, json=payload, **kwargs)
    data = await resp.json()
    return data
async def gather_with_concurrency(n, *tasks):
    semaphore = asyncio.Semaphore(n)

    async def sem_task(task):
        async with semaphore:
            return await task
    
    return await asyncio.gather(*(sem_task(task) for task in tasks))


async def gather_stats(query: dict, start_date: str, end_date: str, **kwargs):
    date_ranges = []
    start_date = parse_date(start_date)
    end_date = parse_date(end_date)
    start_ref = start_date
    
    while start_ref + timedelta(days=90) < end_date:
        date_ranges.append({'start': format_date(start_ref), 
                            'end': format_date(start_ref + timedelta(days=90))})
        start_ref = start_ref + timedelta(days=91)
    
    date_ranges.append({'start': format_date(start_ref), 
                        'end': format_date(end_date)})
    
    async with aiohttp.ClientSession() as session:
        tasks = []
        for date_range in date_ranges:
            tasks.append(
                get(
                    session=session,
                    query=query,
                    start=date_range['start'],
                    end=date_range['end'], **kwargs))
        
        responses = await gather_with_concurrency(5, *tasks)
        results = []
        
        for response in responses:
            results.extend(response['results'])
        
        return results<jupyter_output><empty_output><jupyter_text>If using Google Colab uncomment the following code block to fix broken async functionality.This is as a result of how Google implemented notebooks and are running their async event loop.Google Colab's Tornado 5.0 (Currently running 5.1.x) update bricked asyncio functionalities after the addition of its own asyncio event loop.Thus, for any asyncio functionality to run on Google Colab Notebook you cannot invoke a `run_until_complete()` or `await`, since the loop you will receive from `asyncio.get_event_loop()` will be active and throw the exception seen.JupterHub/JupyterLab had the same issue but have since fixed this, which is why you can easily just `await` the resolution, processing and response of an async call with the await keyword.So we need to `"monkey patch"` the running event loop on Google Colab to hijack the event loop to allow us to issue tasks to be scheduled!<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)

# responses = awaitx(gather_stats(
#     query=query,
#     start_date='2016-01-01',
#     end_date='2021-12-31'))

# len(responses)<jupyter_output><empty_output><jupyter_text>Use the following code block only if running locally in JupyterHub or JupyterLab<jupyter_code>responses = await gather_stats(
    query=query,
    start_date='2016-01-01',
    end_date='2021-12-31')

len(responses)
print(json.dumps(responses[0], indent=4))<jupyter_output>{
    "date": "2016-01-01",
    "phq_attendance_academic_graduation": {
        "stats": {
            "count": 0,
            "sum": 0
        }
    },
    "phq_attendance_community": {
        "stats": {
            "count": 0,
            "median": 0.0,
            "std_dev": null
        }
    },
    "phq_attendance_concerts": {
        "stats": {
            "count": 2,
            "avg": 5000.0,
            "median": 5000.0,
            "std_dev": 0.0
        }
    },
    "phq_attendance_conferences": {
        "stats": {
            "count": 0,
            "median": 0.0,
            "std_dev": null
        }
    },
    "phq_attendance_sports": {
        "stats": {
            "count": 0,
            "median": 0.0,
            "std_dev": null
        }
    },
    "phq_rank_public_holidays": {
        "rank_levels": {
            "1": 0,
            "2": 0,
            "3": 0,
            "4": 0,
            "5": 1
        }
    }
}<jupyter_text>Exploring the Data<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
import calmap
import warnings

# To display more columns and with a larger width in the DataFrame
pd.set_option('display.max_columns', None)
pd.options.display.max_colwidth = 100<jupyter_output><empty_output><jupyter_text>Concert Trends for San Francisco vs Oakland<jupyter_code># IDs are from https://www.geonames.org/
sf_id = 5391959
oak_id = 5378538

# San Francisco
query = {
    'location': {
        'place_id': [sf_id]
    },
    'phq_attendance_concerts': {
        'stats': ['count', 'sum', 'median']
    },
    'phq_attendance_festivals': {
        'stats': ['count', 'sum', 'median']
    }
}<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# sf_responses = awaitx(gather_stats(
#     query=query,
#     start_date='2016-01-01',
#     end_date='2021-04-01'))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code>sf_responses = await gather_stats(
    query=query,
    start_date='2016-01-01',
    end_date='2021-04-01')
# Oakland
query = {
    'location': {
        'place_id': [oak_id]
    },
    'phq_attendance_concerts': {'stats': ['count', 'sum', 'median'], },
    'phq_attendance_festivals': {'stats': ['count', 'sum', 'median'], },
}<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# oak_responses = awaitx(gather_stats(
#     query=query,
#     start_date='2016-01-01',
#     end_date='2021-04-01'))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code>oak_responses = await gather_stats(
    query=query,
    start_date='2016-01-01',
    end_date='2021-04-01')
sf_df = pd.DataFrame.from_dict(sf_responses)
oak_df = pd.DataFrame.from_dict(oak_responses)

sf_df.head()<jupyter_output><empty_output><jupyter_text>Reshaping the responses into tabular form<jupyter_code>FIELDS = ['sum', 'count', 'median']

def parse_element(response: dict, fields: list) -> np.array:
    """
    Extracts the feature fields from a dictionary into an array
    """
    blank_response = np.zeros(shape=(len(fields)))
    if isinstance(response, dict):
        if not 'stats' in response: return blank_response
        stats = response['stats']
        if stats['count'] == 0: return blank_response
        return np.array([stats[field] for field in fields])
    
def add_flat_features(df: pd.DataFrame, column: str, 
                      feature_name: str, fields: list) -> pd.DataFrame:
    """
    Adds features into dataframe
    """
    list_feature = df[column].apply(parse_element, fields=fields)
    fts = pd.DataFrame(list_feature.to_list(), columns=fields)
    fts.columns = [f'{feature_name}_{col}' for col in fts.columns]
    return df.join(fts)
sf_flat = add_flat_features(sf_df, 'phq_attendance_concerts', 'concerts', fields=FIELDS)
sf_flat = add_flat_features(sf_flat, 'phq_attendance_festivals', 'festivals', fields=FIELDS)

oak_flat = add_flat_features(oak_df, 'phq_attendance_concerts', 'concerts', fields=FIELDS)
oak_flat = add_flat_features(oak_flat, 'phq_attendance_festivals', 'festivals', fields=FIELDS)

sf_flat['date'] = pd.to_datetime(sf_flat['date'])
oak_flat['date'] = pd.to_datetime(oak_flat['date'])
sf_flat.head(2)
oak_flat.head(2)
def truncate_to_month(date_col):
    return date_col - pd.Timedelta('1 day') * (date_col.dt.day - 1)

sf_flat['month'] = truncate_to_month(sf_flat['date'])
oak_flat['month'] = truncate_to_month(oak_flat['date'])
agg_dict = {'concerts_sum': 'sum',
            'concerts_count': 'sum',
            'concerts_median': 'median',
            'festivals_sum': 'sum',
            'festivals_count': 'sum',
            'festivals_median': 'median',}

sf_monthly = (sf_flat
 .groupby('month')
 .agg(agg_dict)
)

oak_monthly = (oak_flat
 .groupby('month')
 .agg(agg_dict)
)
sf_monthly.columns = [f'{col}_sf' for col in sf_monthly.columns]
oak_monthly.columns = [f'{col}_oak' for col in oak_monthly.columns]

both_monthly = sf_monthly.join(oak_monthly).reset_index()
both_monthly.head()
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'concerts_count_sf', 
                                    'concerts_count_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()

    sns.lineplot(data=both_monthly[['month', 'concerts_sum_sf', 
                                    'concerts_sum_oak']], ax=ax)
    # plt.xticks(both_monthly['month'])
    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()

    sns.lineplot(data=both_monthly[['month', 'concerts_median_sf', 
                                    'concerts_median_oak']], ax=ax)
    # plt.xticks(both_monthly['month'])
    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()

    sns.lineplot(data=both_monthly[['month', 'festivals_count_sf', 
                                    'festivals_count_oak']], ax=ax)
    # plt.xticks(both_monthly['month'])
    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>Exploring TV Viewership with the Features APIThis group of features is based on PHQ Viewership. The following features are some examples that are supported:- `phq_viewership_sports_american_football` (viewership across all supported American Football leagues)- `phq_viewership_sports_american_football_nfl` (viewership for NFL)- `phq_viewership_sports_baseball` (viewership across all supported Baseball leagues)- `phq_viewership_sports_soccer_mls` (viewership for MLS)- `phq_viewership_sports_basketball` (viewership across all supported Basketball leagues)- `phq_viewership_sports_basketball_nba` (viewership for NBA)- `phq_viewership_sports_ice_hockey_nhl` (viewership for NHL)The full list of available viewership feature fields are in our documentation [here](https://docs.predicthq.com/resources/features/viewership-based-feature-fields).For this section we will demonstrate how to apply the Features API to explore the viewership for two different sports. For one example we will be comparing viewership for all supported American Football leagues in San Francisco against viewership for all supported American Football leagues in Oakland. The second example we will be comparing viewership for all supported Basketball leagues in San Francisco against viewership for all supported Basketball leagues in Oakland.<jupyter_code># San Francisco
query = {
    'location': {
        'place_id': [sf_id]
    },
    'phq_viewership_sports_american_football': {
        'stats': ['count', 'sum', 'median'],
    },
    'phq_viewership_sports_basketball': {
        'stats': ['count', 'sum', 'median'], 
    },
}<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment the following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# sf_responses = awaitx(gather_stats(
#     query=query,
#     start_date='2020-01-01',
#     end_date='2020-12-31'))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code>sf_responses = await gather_stats(
    query=query,
    start_date='2020-01-01',
    end_date='2020-12-31')
sf_df = pd.DataFrame.from_dict(sf_responses)
sf_df.head()
# Oakland
query = {
    'location': {
        'place_id': [oak_id]
    },
    'phq_viewership_sports_american_football': {
        'stats': ['count', 'sum', 'median'], 
    },
    'phq_viewership_sports_basketball': {
        'stats': ['count', 'sum', 'median'], 
    },
}<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment the following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# oak_responses = awaitx(gather_stats(
#     query=query,
#     start_date='2020-01-01',
#     end_date='2020-12-31'))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code>oak_responses = await gather_stats(
    query=query,
    start_date='2020-01-01',
    end_date='2020-12-31')
oak_df = pd.DataFrame.from_dict(oak_responses)
oak_df.head()
sf_view_flat = add_flat_features(sf_df, 'phq_viewership_sports_american_football', 'american_football', fields=FIELDS)
sf_view_flat = add_flat_features(sf_view_flat, 'phq_viewership_sports_basketball', 'basketball', fields=FIELDS)

oak_view_flat = add_flat_features(oak_df, 'phq_viewership_sports_american_football', 'american_football', fields=FIELDS)
oak_view_flat = add_flat_features(oak_view_flat, 'phq_viewership_sports_basketball', 'basketball', fields=FIELDS)

sf_view_flat['date'] = pd.to_datetime(sf_view_flat['date'])
oak_view_flat['date'] = pd.to_datetime(oak_view_flat['date'])
sf_view_flat['month'] = truncate_to_month(sf_view_flat['date'])
oak_view_flat['month'] = truncate_to_month(oak_view_flat['date'])
sf_view_flat.head()
oak_view_flat.head()<jupyter_output><empty_output><jupyter_text>Reshaping the responses into tabular form<jupyter_code>agg_dict = {'american_football_sum': 'sum',
            'american_football_count': 'sum',
            'american_football_median': 'median',
            'basketball_sum': 'sum',
            'basketball_count': 'sum',
            'basketball_median': 'median',}

sf_monthly = (sf_view_flat
 .groupby('month')
 .agg(agg_dict)
)

oak_monthly = (oak_view_flat
 .groupby('month')
 .agg(agg_dict)
)
sf_monthly.columns = [f'{col}_sf' for col in sf_monthly.columns]
oak_monthly.columns = [f'{col}_oak' for col in oak_monthly.columns]

both_monthly = sf_monthly.join(oak_monthly).reset_index()
both_monthly.head()<jupyter_output><empty_output><jupyter_text>The results of viewership between San Francisco and Oakland for All American Football are graphed below.<jupyter_code>with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'american_football_count_sf', 
                                    'american_football_count_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'american_football_sum_sf', 
                                    'american_football_sum_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'american_football_median_sf', 
                                    'american_football_median_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>The results for viewership between San Francisco and Oakland for All Basketball are graphed below.<jupyter_code>with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'basketball_count_sf', 
                                    'basketball_count_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'basketball_sum_sf', 
                                    'basketball_sum_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);
with warnings.catch_warnings():
    
    warnings.simplefilter('ignore')
    fig, ax = plt.subplots()
    both_monthly['month'] = pd.to_datetime(both_monthly['month'])
    sns.lineplot(data=both_monthly[['month', 'basketball_median_sf', 
                                    'basketball_median_oak']], ax=ax)

    ax.set_xticklabels(both_monthly['month'].dt.strftime('%Y-%m'))
    ticks = [item.get_text() for item in ax.get_xticklabels()]

    plt.xticks(rotation=90);<jupyter_output><empty_output><jupyter_text>Retrieving Multiple LocationsYou can specify multiple locations in a single request and the results for all specified locations will be aggregated together as in the example below.<jupyter_code>query = {
    'location': {
        'place_id': [sf_id, oak_id]
    },
    'phq_attendance_concerts': {
        'stats': ['count', 'sum', 'median']
    },
    'phq_attendance_festivals': {
        'stats': ['count', 'sum', 'median']
    }
}<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# both_responses = awaitx(gather_stats(
#     query=query,
#     start_date='2016-01-01',
#     end_date='2021-04-01'))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code>both_responses = await gather_stats(
    query=query,
    start_date='2016-01-01',
    end_date='2021-04-01')
pd.DataFrame.from_dict(both_responses).head(5)<jupyter_output><empty_output><jupyter_text>List of lat/lonYou might need to fetch data for a wide date range as well as a large number of different locations. This example loads a list of lat/lon from a CSV file and fetches data for each of them.<jupyter_code>examples = pd.read_csv('./data/lat_lon_examples.csv').sample(50)

examples.head(5)
# Check number of locations
len(examples)
query_f = lambda lat, long: {
    'location': {
        'geo': {
            'lat': float(lat),
            'lon': float(long),
            'radius': '10km'
        }
    },
    'phq_attendance_sports': {
        'stats': ['count', 'avg'],
        'phq_rank': { 
            'gt': 50
        }    
    },
    "phq_attendance_concerts": {
        'stats': ['count', 'avg'],
        'phq_rank': { 
            'gt': 50
        }    
    },
}

async def pull_one_lat_long(city, lat, long):
    response = await gather_stats(
        query=query_f(lat, long),
        start_date='2016-01-01',
        end_date='2021-04-01')
    return {city: response}
import time

start_time = time.time()

all_locations = []

for ix , row in examples.iterrows():
    all_locations += [pull_one_lat_long(row['City'], 
                                        row['Latitude'], 
                                        row['Longitude'])]<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# # This might take a few seconds
# all_results = awaitx(gather_with_concurrency(5, *all_locations))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code># This might take a few seconds
all_results = await gather_with_concurrency(5, *all_locations)
end_time = time.time()
taken = (time.time() - start_time)
print(f"{taken}s processing time")
features_list = [
    "phq_attendance_sports",
    "phq_attendance_concerts"
]

all_counts = [val[key]['stats']['count'] for place_data in all_results
                for place_name, data in place_data.items() for val in data for key in features_list]
print(
    f"Features analysed for {sum(all_counts)} events across 21 quarters or ~1920 days for 2 different categories ")

FIELDS = ['count', 'avg']
all_locs = []
for city in all_results:
    one_loc = pd.DataFrame(list(city.values())[0])
    one_loc = add_flat_features(one_loc, 'phq_attendance_sports', 
                                 'sports', FIELDS)
    one_loc.drop(['phq_attendance_sports'],
                 axis=1, inplace=True)
    one_loc['City'] = list(city.keys())[0]
    all_locs.append(one_loc)
cities = pd.concat(all_locs)
cities.pivot_table(index='date', columns='City', values='sports_avg')<jupyter_output><empty_output><jupyter_text>Multiple Categories For San Francisco<jupyter_code>FIELDS = ['sum', 'count']
query = {
    'location': {
        'place_id': [sf_id]
    },
    'phq_attendance_concerts': {
        'stats': FIELDS
    },
    'phq_attendance_community': {
        'stats': FIELDS
    },
    'phq_attendance_conferences': {
        'stats': FIELDS
    },
    'phq_attendance_expos': {
        'stats': FIELDS
    },
    'phq_attendance_performing_arts': {
        'stats': FIELDS
    },
    'phq_attendance_sports': {
        'stats': FIELDS
    },
    'phq_attendance_festivals': {
        'stats': FIELDS
    },
    'phq_rank_public_holidays': True, 
    'phq_rank_school_holidays': True, 
    'phq_rank_observances': True
}<jupyter_output><empty_output><jupyter_text>If running on Google Colab uncomment following block to obtain results.<jupyter_code># import asyncio
# import nest_asyncio
# nest_asyncio.apply()


# def awaitx(x): return asyncio.get_event_loop().run_until_complete(x)
# sf_mlt_responses = awaitx(gather_stats(
#     query=query,
#     start_date='2016-01-01',
#     end_date='2021-04-01'))<jupyter_output><empty_output><jupyter_text>If running locally or within JupyterHub/JupyterLab use this block instead to obtain results.<jupyter_code>sf_mlt_responses = await gather_stats(
    query=query,
    start_date='2016-01-01',
    end_date='2021-04-01')<jupyter_output><empty_output><jupyter_text>Parse Rank Columns<jupyter_code>def parse_rank_column(col):
    rank_levels = col.apply(lambda x: x['rank_levels'])
    max_values = []
    for row in rank_levels:
        not_null = {k: v for k, v in row.items() if v>0}
        if not_null:
            max_values.append(int(max(not_null)))
        else:
            # all values were zero
            max_values.append(0)            
    return max_values
holidays_att = pd.DataFrame.from_dict(sf_mlt_responses)
holidays_att['public-holiday'] = \
     parse_rank_column(holidays_att['phq_rank_public_holidays'])
holidays_att['public-holiday'] = \
     parse_rank_column(holidays_att['phq_rank_public_holidays'])
holidats_pa = add_flat_features(holidays_att, 'phq_attendance_performing_arts', 
                  'performing_arts', FIELDS)
holidats_pa = add_flat_features(holidats_pa, 'phq_attendance_concerts', 
                  'concerts', FIELDS)
holidats_pa = add_flat_features(holidats_pa, 'phq_attendance_community', 
                  'community', FIELDS)
holidats_pa = add_flat_features(holidats_pa, 'phq_attendance_conferences', 
                  'conferences', FIELDS)
holidats_pa = add_flat_features(holidats_pa, 'phq_attendance_expos', 
                  'expos', FIELDS)
holidats_pa = add_flat_features(holidats_pa, 'phq_attendance_festivals', 
                  'festivals', FIELDS)
holidats_pa = add_flat_features(holidats_pa, 'phq_attendance_sports', 
                  'sports', FIELDS)
holidats_pa.set_index('date', inplace=True)
holidats_pa.index = pd.to_datetime(holidats_pa.index)
(holidats_pa
    .groupby('public-holiday')
    .agg({'conferences_sum': 'median',
          'community_sum': 'median',
          'conferences_count': 'median',
          'community_count': 'median',
         }
        )
)<jupyter_output><empty_output><jupyter_text>Visualize in a Calendar<jupyter_code>for category in ['performing-arts', 'conferences']:
    feature = category.replace('-','_')+'_count'
    fig, ax = calmap.calendarplot(holidats_pa[feature], cmap="YlGn")
    fig.set_size_inches(18, 15)
    fig.colorbar(ax[0].get_children()[1], ax=ax.ravel().tolist())
    _ = fig.suptitle(f"Number of {category} per day in calmap")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1222925868766963, 'fraction_numerical': 0.019796711191527858, 'mean_word_length': 3.1772813159942204, 'pattern_counts': {'":': 111, '<': 139, '<?xml version=': 0, '>': 142, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '29992', 'license': 'MIT', 'n_tokens_mistral': 12976, 'n_tokens_neox': 12314, 'n_words': 2659, 'path': 'features-api/features-api.ipynb', 'repo_name': 'CharlottePHQ/phq-data-science-docs', 'size': '37582'}