<jupyter_start><jupyter_text>Slow Start: A step by step breakdown of the Quickstart example The quickstart example demonstrated the fastest way to download and start working with data using Quest:<jupyter_code>import quest
data = quest.api.get_data(
    collection_name='quick-start',
    service_uri='svc://usgs-nwis:iv',
    search_filters={'bbox': [-91, 32.25, -90.8, 32.4]},
    download_options={'parameter': 'streamflow'},
)[0]
data.head()<jupyter_output><empty_output><jupyter_text>There is a lot going on in this seemingly simple example, so we're going to break it down and explain every step.The first thing to note is that the function ``quest.api.get_data``, is a workflow function, or in other words afunction that calls several other functions in succession. This provides a convenient way to get your data in one stepwhen you already know all of inputs you need. You can also use Quest to do the same workflow in a more interactive way.The ``quest.api.get_data`` call performs the following steps behind the scenes:  1. [Create a Collection](Create-a-Collection)  2. [Select a Data Service](Select-a-Data-Service)  3. [Search for Datasets](Search-for-Datasets)  4. [Add Datasets to Collection](Add-Datasets-to-Collection)  5. [Download Datasets](Download-Datasets)  6. [Open Datasets](Open-Datasets)The following sections will explain each of these steps in detail.  Create a Collection When Quest downloads data it needs to know where to put them. To keep data organized Quest provides a local organizationhierarchy to manage data (see :ref:`core-concepts-data-organization`). At the top of the hierarchy is a :term:`project`, and all Quest callswill always apply to whatever project is active. For more details about managing projects see (:ref:`examples-manage-projects`).Within :term:`projects` are :term:`collections` . All data that are downloaded by Quest are put in a :term:`collection` . In the:func:`quest.api.get_data` example above the ``collection_name`` argument specifies which :term:`collection` to put the datain. If there isn't already a :term:`collection` with the name specified by the ``collection_name`` argument then ``get_data``function will create it.This process can also be done manually. Using the Quest API we can get a list of the collections with the:func:`quest.api.get_collection` function:<jupyter_code>collections = quest.api.get_collections()
collections<jupyter_output><empty_output><jupyter_text>The collection called "quick-start" that was created as a result of the ``get_data``call made previously.  To create a new collection we manually we can use the :func:`quest.api.new_collection` function:<jupyter_code>new_collection = quest.api.new_collection('slow-start', exists_ok=True)
new_collection<jupyter_output><empty_output><jupyter_text>This function returns the metadata that is associated with this newly created collection. For more details about workingwith collection see :ref:`examples-manage-collections`.  Select a Data Service Once we have a place to store data locally we need to decide what data we want to download. Quest provides the abilityto search for data among many different data sources, or :term:`providers`. Each :term:`provider` will offer one or more data:term:`services` (see :ref:`core-concepts-data-repositories`). We can list the available :term:`services` by calling:func:`quest.api.get_services`:<jupyter_code>quest.api.get_services()<jupyter_output><empty_output><jupyter_text>Each :term:`service` is represented by a service URI. In our quickstart example we used the penultimate service URI listedhere: 'svc://usgs-nwis:iv'. This service URI is needed to tell Quest where to search for data.  Search for Datasets Each :term:`service` has a :term:`catalog` or listing of the data it provides. To search for data we need to tell Quest whichservice's or services' catalog to search. To limit our search we can pass in a dictionary of key-value pairs that specifyfilter criteria to filter the catalog entries by. In the quickstart example we filtered the catalog using a bounding box.```python...:     search_filters={'bbox': [-91, 32.25, -90.8, 32.4]},```To manually search the catalog we can call the Quest API function :func:`quest.api.search_catalog` and pass it theservice URI and the filters dictionary:<jupyter_code>quest.api.search_catalog(uris='svc://usgs-nwis:iv', filters={'bbox': [-91, 32.3, -90.8, 32.34]})<jupyter_output><empty_output><jupyter_text>The return value from :func:`quest.api.search_catalog` is a list of :term:`catalog entry` URIs. The :term:`catalog entry` URIlooks just like the :term:`service` URI that it came from with an appended catalog ID number. This :term:`catalog entry` URI isused to download the data associated to that :term:`catalog entry`.  Add Datasets to Collection Before we can download the data associated with a :term:`catalog entry` we need to create a :term:`dataset` derived from that:term:`catalog entry`. A Quest :term:`dataset` represents a piece of data and stores all of the metadata associated with thosedata. Every Quest :term:`dataset` has an associated :term:`catalog entry` that links it back to the :term:`service` where the datacame from, and an associated :term:`collection` that acts as a container for the data. We can create new :term:`datasets` bycalling :func:`quest.api.add_datasets` and passing it both the `collection` and the :term:`catalog entry` or entries fromwhich to create the :term:`datasets`.<jupyter_code>dataset_id = quest.api.add_datasets('slow-start', 'svc://usgs-nwis:iv/07289000')[0]
dataset_id<jupyter_output><empty_output><jupyter_text>The return value is a list of dataset IDs from the newly created datasets (in this case it's just a list of one ID. Wecan now use this dataset ID to download the data associated with it.  Download Datasets To download data using Quest we use the :func:`quest.api.download_datasets` function. We need to pass it the :term:`dataset`IDs for the data that we want to download. We also need to pass it a dictionary of download options. Each service specifiesit's own set of download options. To figure out what the download options are for a particular dataset we can eitherrefer to the documentation for that dataset's service or we can call :func:`quest.api.get_download_options` and pass itcan pass in either the :term:`service` URI the :term:`catalog entry` URI, or the :term:`dataset` ID.<jupyter_code>quest.api.get_download_options(dataset_id)<jupyter_output><empty_output><jupyter_text>This returns a dictionary keyed by the URIs that were passed to the fucntion. For each URI key the value is a dictionaryspecifying the download options or `properties` for that URI. In this case the download options we can specify are:  * `parameter`: one of 'gage_height', 'streamflow', or 'water_temperature'  * `start`: the start date for the period of data want  * `end`: the end date for the period of data you want  * `period`: a string representing a period of data you wantHere either the start and end date can be specified or a period string can be specified. If neither are specified thenthe default period 'P365D' (meaning a period of 365 days ending with today) will be used by default. In the quickstartexample we specified that we were interested in 'streamflow' data and we didn't specify a period so by default we gotthe past year of data. We can do the same here by calling :func:`quest.api.download_datasets`:<jupyter_code>quest.api.download_datasets(dataset_id, options={'parameter': 'streamflow'})<jupyter_output><empty_output><jupyter_text>The return value is a dictionary keyed by the dataset IDs that were passed in where the value is the status. In this case'downloaded' means that the data associated with the dataset were successfully downloaded.  Open Datasets When the data associated with a :term:`dataset` are downloaded they are by default stored on disk. Quest can be used totransform, visualize, or publish the data and will only require the :term:`dataset` ID as an argument. If you'd like to useother Python tools to work with your data you can use Quest to open your data and read it into a Python data structure.The data that we downloaded are a timeseries of streamflow values. The default data structure that Quest uses for thistype of data is a :obj:`pandas.DataFrame`. Therefore, when we call :func:`quest.api.open_dataset` we will getback ourdata in a DataFrame.<jupyter_code>data = quest.api.open_dataset(dataset_id)
data.head()<jupyter_output><empty_output><jupyter_text>Interactive Workflow with PyViz With a few additional tools from the PyViz tool suite Quest can be used to create an interactive workflow to select, download, and visualize data.<jupyter_code>import panel as pn
import geoviews as gv
import holoviews as hv
import hvplot.pandas
parameter_selector = quest.util.ParameterSelector(default='streamflow')
pn.Row(parameter_selector)
service_selector = quest.util.ServiceSelector(parameter=parameter_selector.value)
pn.Row(service_selector)
download_options = quest.api.get_download_options(service_selector.value, fmt='param')[service_selector.value]
download_options.parameter = parameter_selector.value
pn.Row(download_options)
collection_selector = quest.util.CollectionSelector()
pn.Row(collection_selector)
tiles = gv.tile_sources.StamenTerrain().options(width=950, height=600)
box_poly = gv.Polygons(
    hv.Bounds((-91, 32.25, -90.8, 32.4))
).options(fill_alpha=.2)
box_stream = hv.streams.BoxEdit(source=box_poly, num_objects=1)
tiles * box_poly
xs, ys = box_stream.element.array().T
bbox = [xs[0], ys[1], xs[2], ys[0]]
data = quest.api.get_data(
    collection_name=collection_selector.value,
    service_uri=service_selector.value,
    search_filters={'bbox': bbox},
    download_options=download_options,
)[0]

data.head()
data[parameter_selector.value].hvplot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0852595296025953, 'fraction_numerical': 0.008414436334144364, 'mean_word_length': 5.733788395904437, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '382473', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2791, 'n_tokens_neox': 2703, 'n_words': 1176, 'path': 'examples/notebooks/Slow_Start.ipynb', 'repo_name': 'sdc50/quest', 'size': '9864'}