<jupyter_start><jupyter_text>Examples require an initialized GsSession and relevant entitlements. External clients need to substitute thier own client id and client secret below. Please refer to [Authentication](https://developer.gs.com/p/docs/institutional/platform/authentication/) for details.<jupyter_code>from gs_quant.session import GsSession
GsSession.use(client_id=None, client_secret=None, scopes=('read_product_data',))<jupyter_output><empty_output><jupyter_text>How to query data The Data APIs support many ways to query datasets to intuitively fetch only the data users need.More details on [Querying Data](https://developer.gs.com/p/docs/services/data/data-access/query-data/) can be found in the documentation<jupyter_code>from datetime import date, timedelta, datetime
from gs_quant.data import Dataset
import pydash<jupyter_output><empty_output><jupyter_text>Data in Marquee is available in the form of Datasets (collections of homogenous data). Each Dataset has a set of entitlements, a fixed schema, and assets in coverage.<jupyter_code>dataset_id = 'FXIVOL_STANDARD' # https://marquee.gs.com/s/developer/datasets/FXIVOL_STANDARD
ds = Dataset(dataset_id)<jupyter_output><empty_output><jupyter_text>Data for limited number of assets or spanning a small time frame can be queried in one go by specifying the assets to query and date/time range.<jupyter_code>start_date = date(2019, 1, 15)
end_date = date(2019, 1, 18)

data = ds.get_data(start_date, end_date, bbid=['EURCAD'])
data.head()<jupyter_output><empty_output><jupyter_text>Instead of a range, one can also specify a set of date/times to get data for just those specific date/times<jupyter_code>data = ds.get_data(dates=[date(2019, 1, 15), date(2019, 1, 18)],
                   bbid=['EURCAD'])
data.head()<jupyter_output><empty_output><jupyter_text>For larger number of assets or for longer time ranges, we recommend iterating over assets and time to avoid hitting API query limits.<jupyter_code># loop over assets
def iterate_over_assets(dataset, coverage, start, end, batch_size=5, query_dimension='assetId', delta=timedelta(days=6)):
    for ids in pydash.chunk(coverage[query_dimension].tolist(), size=batch_size):
        print('iterate over assets', ids)
        iterate_over_time(start, end, ids, dataset, delta=delta, query_dimension=query_dimension)

# loop over time
def iterate_over_time(start, end, ids, dataset, delta=timedelta(days=6), query_dimension='assetId'):
    iter_start = start
    while iter_start < end:
        iter_end = min(iter_start + delta, end)
        print('time iteration since', iter_start, 'until', iter_end)
        data = dataset.get_data(iter_start, iter_end, **{query_dimension: ids})
        # Add your code here to make use of fetched data
        
        iter_start = iter_end
dataset_id = 'EDRVOL_PERCENT_STANDARD'  # https://marquee.gs.com/s/developer/datasets/EDRVOL_PERCENT_STANDARD 
ds = Dataset(dataset_id)

coverage = ds.get_coverage()

iterate_over_assets(ds, coverage, date(2021, 5, 1), date(2021, 5, 31), batch_size=5)<jupyter_output>iterate over assets ['MA4B66MW5E27U8NN8LS', 'MA4B66MW5E27U8NN8N6', 'MA4B66MW5E27U8NN8SP', 'MA4B66MW5E27U8NN922', 'MA4B66MW5E27U8NN95P']
time iteration since 2021-05-01 until 2021-05-07
time iteration since 2021-05-07 until 2021-05-13
time iteration since 2021-05-13 until 2021-05-19
time iteration since 2021-05-19 until 2021-05-25
time iteration since 2021-05-25 until 2021-05-31
iterate over assets ['MA4B66MW5E27U8NPZQY', 'MA4B66MW5E27U8P3294', 'MA4B66MW5E27U8P3295', 'MA4B66MW5E27U8P32DM', 'MA4B66MW5E27U8P32GB']
time iteration since 2021-05-01 until 2021-05-07
time iteration since 2021-05-07 until 2021-05-13
time iteration since 2021-05-13 until 2021-05-19
time iteration since 2021-05-19 until 2021-05-25
time iteration since 2021-05-25 until 2021-05-31
iterate over assets ['MA4B66MW5E27U8P32KK', 'MA4B66MW5E27U8P32LE', 'MA4B66MW5E27U8P32LH', 'MA4B66MW5E27U8P32LQ', 'MA4B66MW5E27U8P32PC']
time iteration since 2021-05-01 until 2021-05-07
time iteration since 2021-05-07 until 2021-05-13
time[...]<jupyter_text>Similar approach can be used to download all data of a dataset<jupyter_code>coverage = ds.get_coverage(include_history=True)
coverage = coverage.sort_values(by='historyStartDate', axis=0)
start_date = datetime.strptime(coverage['historyStartDate'].values[0], '%Y-%m-%d').date()

# warning: long running operation
iterate_over_assets(ds, coverage, start_date, date.today())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10231832367365136, 'fraction_numerical': 0.0835934016941596, 'mean_word_length': 5.967391304347826, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 28, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '40469', 'license': 'Apache-2.0', 'n_tokens_mistral': 1800, 'n_tokens_neox': 1515, 'n_words': 398, 'path': 'gs_quant/documentation/00_data/00_datasets/examples/0000_query_dataset.ipynb', 'repo_name': 'dannyb2018/gs-quant', 'size': '4486'}