<jupyter_start><jupyter_text>Basic Usage: datastore application----------------------------------The datastore is a tool for using the eemeter which automatesand helps to scale some of the most frequent tasks accomplished by theeemeter. These tasks include data loading and storage, meterrunning, result storage and warehousing. It puts a REST APIin front of the eemeter and uses a postgres backend.This tutorial is also available as a jupyter :download:`notebook `.<jupyter_code><div class="alert alert-info">

**Note:**

This tutorial assumes you have a working datastore instance.
If you do not, please follow the datastore development setup instructions
or contact Open EE to setting up a dedicated production deployment.

</div>
<div class="alert alert-info">

**Note:**

For small and large datasets, the ETL toolkit exists to ease and
speed up the process of loading your data.

This tutorial does not cover ETL toolkit usage.
For more information on the ETL toolkit, see its API documentation.

</div>
Setup
-----<jupyter_output><empty_output><jupyter_text>library importsimport pandas as pdimport requestsimport pytz<jupyter_code>If you followed the datastore development setup instructions, you will
already have run the command to create a superuser and access credentials.

    python manage.py dev_seed
    
If you haven't already done so, do so now. The `dev_seed` command
creates a demo admin user and a sample project.

- username: `demo`,
- password: `<PASSWORD>`,
- API access token: `tokstr`.

Ensure that your development server is running locally on port 8000 before continuing.

    python manage.py runserver
    
Each request will include an Authorization header

    Authorization: Bearer tokstr<jupyter_output><empty_output><jupyter_text>base_url = "http://0.0.0.0:8000"token = "tokstr"headers = {"Authorization": "Bearer {}".format(token)}<jupyter_code>Using the API to get loaded data
--------------------------------
We can use the API to inspect the data that is loaded into the datastore. (The API can also be used for loading data, but that is not covered here. See the ETL tutorial for more information on loading data.)
<div class="alert alert-info">

**Note:**

We will use the `requests` python package for making requests, but
you could just as easily use a tool like cURL or Postman.

If you have the `eemeter` package installed, you will also have the
`requests` package installed, but if not, you can install it with:

<div>
</div>

    $ pip install requests


A request using the requests library looks like this:

    import requests
    url = "https://example.com"
    data = {
        "first_name": "John",
        "last_name": "Doe"
    }
    requests.post(url + "/api/users/", json=data)

which is equivalent to:

    POST /api/users/ HTTP/1.1
    Host: example.com
    {
        "first_name": "John",
        "last_name": "Doe"
    }

</div>
Since the dev_seed command creates a sample project, this will return
a response showing that project. Projects all have a unique "project_id", 
which can be set to whatever is most appropriate (note: it is not used 
as primary key; that's the 'id' field).<jupyter_output><empty_output><jupyter_text>url = base_url + "/api/v1/projects/"projects = requests.get(url, headers=headers).json() projects<jupyter_code>Energy trace data will be associated with this project by foreign key
through a many-to-many table. This means that projects can have 0 to n
associated traces, and that traces can have 0 to n associated projects. 

Like projects and the project_id field, traces are identified by a unique
'trace_id' field, which can also be set to whatever is most appropriate.

There are API endpoints used to fetch trace data:

1. `/api/v1/traces/`: This stores trace ids, unit, and interpretation.
2. `/api/v1/trace_records/`: This stores time-series records associated with each trace.


These records are stored by record start timestamp, with the implicit
assumption that the `start` timestamp of the next temporal record is 
the end of the current record. The value of the last record is ignored, 
and serves as the final end timestamp (and is usually set to null).<jupyter_output><empty_output><jupyter_text>url = base_url + "/api/v1/traces/?projects={}".format(projects[0]['id'])traces = requests.get(url, headers=headers).json() traces<jupyter_code>We can also query for trace records by trace primary key.<jupyter_output><empty_output><jupyter_text>url = base_url + "/api/v1/trace_records/?trace={}".format(traces[0]['id'])trace_records = requests.get(url, headers=headers).json() trace_records[:3]   first 3 records<jupyter_code>Running meters
--------------

Running a meter means pulling trace data, matching it with
relevant project data, and evaluating its energy effiency performance. This is
the central task performed by the datastore, so if the specifics are
unfamiliar, there is a bit more background information worthy of review in the
Methods Overview section of the guides.

To run a meter, make a request to create a "meter run". This request
will start a job that runs a meter and saves its results. The result
of a meter run is called a "meter result".<jupyter_output><empty_output><jupyter_text>from collections import OrderedDictimport json<jupyter_code>Scheduling a single meter run
-----------------------------

The primary component of this request is a trace primary key.

The project data associated with the trace will be automatically pulled
in to be associated with the trace.<jupyter_output><empty_output><jupyter_text>created_meter_run = requests.post(    base_url + "/api/v1/meter_runs/",    json={        "trace": traces[0]['id']   single trace primary key    },    headers=headers).json(object_pairs_hook=OrderedDict)   retains order of keys print(json.dumps(created_meter_run, indent=2))<jupyter_code>This is a summary of the task to run the meter on the indicated project.

The response shows us the complete specification of the meter run behavior,
which is as follows:

1. `project`: the project primary key (determined implicitly from the trace).
2. `trace`: the trace primary key (given in API request).
3. `status`: the task status code (in this case `"PENDING"`), other options are:
   - `"PENDING"`: which means the tasks is scheduled but not yet running or completed.
   - `"RUNNING"`: task is currently running.
   - `"SUCCESS"`: successful completion.
   - `"FAILED"`: failed due to some sort of error.
4. `meter_result`: the primary key of the meter result.
5. `meter_input`: has not yet been created (this is the
   complete serialized input to the meter, as required by the eemeter.)
6. `model_class` and `model_kwargs`: The model class and arguments used in meter fitting.
   - If these are left blank, default values will be used.
7. `formatter_class` and `formatter_kwargs`: The formatter class and arguments used in meter fitting.
   - If these are left blank, default values will be used.

If you wish, you can also specify many of these properties explicitly
and we will do so in a following section.

Let's make another call to inspect the state of this meter run<jupyter_output><empty_output><jupyter_text>meter_run = requests.get(    base_url + "/api/v1/meter_runs/{}/".format(created_meter_run['id']),    headers=headers).json(object_pairs_hook=OrderedDict) print(json.dumps(meter_run, indent=2))<jupyter_code>The associated meter result is also available now and carries
a set of outputs that include the meter run value and additionally:

1. `meter_output`: serialized output of the meter run.
2. `eemeter_version` and `datastore_version`: software version
   of eemeter library and datastore application<jupyter_output><empty_output><jupyter_text>meter_result = requests.get(    base_url + "/api/v1/meter_results/{}/".format(created_meter_run['meter_result']),    headers=headers).json(object_pairs_hook=OrderedDict) print(json.dumps(meter_result, indent=2))<jupyter_code>Customizing meter runs
----------------------

Meter runs can also be customized by specifying various attributes
explicitly, such as custom arguments for the model class.<jupyter_output><empty_output><jupyter_text>custom_meter_run = requests.post(    base_url + "/api/v1/meter_runs/",    json={        "trace": 2,        "project": 1,        "model_kwargs": {            "heating_base_temp": 64,   different temperature            "cooling_base_temp": 64,        },    },    headers=headers).json(object_pairs_hook=OrderedDict) print(json.dumps(custom_meter_run, indent=2))<jupyter_code>Or, if you leave out the project and trace attributes, you can specify the
exact serialized input. This means that if serialized meter inputs are available,
you need not explicitly load traces and projects through ETL.<jupyter_output><empty_output><jupyter_text>.. If you are running tutorial as a jupyter notebook, the download is   available where you downloaded this notebook on eemeter.readthedocs.ioPlease download a preformatted input :download:`file `for this step.<jupyter_code>with open('meter_input_example.json', 'r') as f:
    meter_input = f.read()  # loaded as a serialized string
    meter_input_meter_run = requests.post(
        base_url + "/api/v1/meter_runs/",
        json={
            "meter_input": meter_input,
        },
        headers=headers
    ).json(object_pairs_hook=OrderedDict)
print(json.dumps(meter_input_meter_run, indent=2))
meter_run = requests.get(
    base_url + "/api/v1/meter_runs/{}/".format(meter_input_meter_run['id']),
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(meter_run, indent=2))
meter_result = requests.get(
    base_url + "/api/v1/meter_results/{}/".format(meter_input_meter_run['meter_result']),
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(meter_result, indent=2))<jupyter_output>{
  "id": 3,
  "trace": null,
  "project": null,
  "meter_run": 3,
  "meter_output": "https://storage.googleapis.com/my-storage-bucket/datastore/meter_run_outputs/7ab2bd31-a723-4c75-afa6-424d560ab284.json",
  "status": "SUCCESS",
  "eemeter_version": "0.4.12",
  "datastore_version": "0.2.3",
  "model_class": "SeasonalElasticNetCVModel",
  "model_kwargs": {
    "heating_base_temp": 65,
    "cooling_base_temp": 65
  },
  "formatter_class": "ModelDataFormatter",
  "formatter_kwargs": {
    "freq_str": "D"
  },
  "added": "2016-11-18T02:22:13.566866Z",
  "updated": "2016-11-18T02:22:13.583138Z"
}<jupyter_text>Meters can also be triggered in bulk; the next section covers this. Bulk-triggering meter runs--------------------------Often it is more convenient to trigger many meter runs at once than to do it trace-by-trace. This can be done either through the API or through a datastore management command. Through the APIThe following sends a list of "targets" to the datastore for triggering. Here, we're triggering a set of meter runs for one project, which will trigger meter runs for all associated traces. **Warning:**    The following may take a few minutes to complete.If you have enabled celery workers, it will execute more quicklyand computation will continue in the background. If this is thecase for you, you should wait until that computation has completedbefore continuing.For more information on background worker setup, see datastore setup instructions. To follow progress, watch the datastore logs or use the `meter_progress` command.In a development environment, these are printed in the `python manage.py runserver` output.<jupyter_code>bulk_created_meter_runs = requests.post(
    base_url + "/api/v1/meter_runs/bulk/",  # note: different url!
    json={
        "targets": [  # a list of targets can be provided
            {
                "project": projects[0]['id']
            },
        ]
    },
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(bulk_created_meter_runs, indent=2))<jupyter_output>[
  [
    {
      "id": 4,
      "trace": 3,
      "project": 1,
      "meter_result": 4,
      "meter_input": null,
      "status": "PENDING",
      "failure_message": null,
      "traceback": null,
      "model_class": null,
      "model_kwargs": null,
      "formatter_class": null,
      "formatter_kwargs": null,
      "added": "2016-11-18T02:22:14.088620Z",
      "updated": "2016-11-18T02:22:14.088658Z"
    },
    {
      "id": 5,
      "trace": 4,
      "project": 1,
      "meter_result": 5,
      "meter_input": null,
      "status": "PENDING",
      "failure_message": null,
      "traceback": null,
      "model_class": null,
      "model_kwargs": null,
      "formatter_class": null,
      "formatter_kwargs": null,
      "added": "2016-11-18T02:26:15.683349Z",
      "updated": "2016-11-18T02:26:15.683442Z"
    },
    {
      "id": 6,
      "trace": 5,
      "project": 1,
      "meter_result": 6,
      "meter_input": null,
      "status": "PENDING",
      "failure_message": null,
 [...]<jupyter_text>Note that results are returned grouped by target (as a list).If model or formatter class or kwarg arguments are supplied, they will beapplied to all meter_runs.  Through a management commandThe other way to bulk-trigger meter runs is through a management command.    python manage.py run_meters --all-traces    You can monitor the progress of these commands with:    python manage.py meter_progress --all-meters --poll-until-complete Meter result warehouse tables-----------------------------For easy access to summarized meter result data, it may be helpful touse the meter result "mart", which is part of the data warehousethat can be created in the postgres database.Data warehouse tables make it easier to query into results bysummarizing the most relevant information.To create warehouse tables, use the following management command:    $ python manage.py meterresultmart recreate    This is equivalent to running    $ python manage.py meterresultmart destroy    $ python manage.py meterresultmart create    Running the `create` command without first destroying will give duplicate rows. Using the `warehouse_meterresultmart` tableThe easiest way to access the results of the warehouse is to connect ananalytics service which can read from the database directly.If that is not available to you, you can also query directly with postgres. Assuming you have a database set up called "datastore" (yours may be named differently, depending on how you set it up), you can connect as follows:    $ psql datastore    psql (9.4.1)    Type "help" for help.    datastore= SELECT      trace_id      , differential_lower_bound as savings_lower_bound      , differential_value as savings      , differential_upper_bound as savings_upper_bound     FROM      warehouse_meterresultmart    WHERE      project_id='DEV_SEED_PROJECT'    AND      derivative_interpretation='gross_predicted'    ORDER BY      project_id      , trace_id      , derivative_interpretation;                        trace_id              | savings_lower_bound |      savings      | savings_upper_bound     ------------------------------------+---------------------+-------------------+---------------------     DEV_SEED_TRACE_ELECTRICITY_15MIN   |    2.21781163934072 |  4.93938974304001 |     7.6609678467393     DEV_SEED_TRACE_ELECTRICITY_HOURLY  |      10.47300113325 |  12.9734969290002 |    15.4739927247505     DEV_SEED_TRACE_NATURAL_GAS_DAILY   |   -12.4594891213768 | -6.03261538803008 |   0.394258345316612     DEV_SEED_TRACE_NATURAL_GAS_MONTHLY |   -774.348987437802 | -580.576019960851 |     -386.8030524839     DEV_SEED_TRACE_SOLAR_30MIN         |   0.848394785466816 |  3.81981394853938 |    6.79123311161194     DEV_SEED_TRACE_SOLAR_HOURLY        |                     |                   |                        (6 rows)    datastore=  Aggregations and groups-----------------------Traces can be aggregated by putting them into groups andtriggering aggregation runs.Groups must be named, and are defined by combinations offilters over project_id, trace_id, or arbitary project metadata.Filters are created with the following attributes as eithera "filter" or a "filter_boolean", which is combination of twofilters.Filter types:`"filter"`:    - `"target"`, can be:    - "project_id"    - "trace_id"    - "project_metadata|NAME_OF_ATTRIBUTE"  - `"comparison"`, can be:    - ">", ">=", "<", "<=", "==", "!="    - "in", "not in"  - `"value"`, can be:    - int, float, str (for comparisons ">", ">=", "<", "<=", "==", "!=")    - list of values (for comparisons "in", "not in")`"filter_boolean"`:  - `"boolean"`, can be:    - "and", "or"  - `"filter_a"`, can be:    - filter, filter_boolean  - `"filter_b"`, can be:    - filter, filter_booleanExample filter specification creation:<jupyter_code>filter_specification = {
    "filter": {
        "target": "project_id",
        "comparison": "==",
        "value": projects[0]["project_id"],
    }
}
trace_group = requests.post(
    base_url + "/api/v1/trace_groups/",  # note: different url!
    json={
        "name": "project_group",
        "filter_specification": filter_specification,
    },
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(trace_group, indent=2))
aggregation_run = requests.post(
    base_url + "/api/v1/aggregation_runs/",
    json={                                                                                                                 
        "group": trace_group['id'],                                                                                                                                              
        "trace_interpretation": "ELECTRICITY_CONSUMPTION_SUPPLIED",                                                                                                     
        "derivative_interpretation": "annualized_weather_normal",                                                                                                       
    },
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(aggregation_run, indent=2))
aggregation_run = requests.get(
    base_url + "/api/v1/aggregation_runs/{}/".format(aggregation_run["id"]),
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(aggregation_run, indent=2))
aggregation_result = requests.get(
    base_url + "/api/v1/aggregation_results/{}/".format(aggregation_run["aggregation_result"]),
    headers=headers
).json(object_pairs_hook=OrderedDict)
print(json.dumps(aggregation_result, indent=2))<jupyter_output>{
  "id": 1,
  "aggregation_run": 7,
  "trace_interpretation": "ELECTRICITY_CONSUMPTION_SUPPLIED",
  "derivative_interpretation": "annualized_weather_normal",
  "aggregation_interpretation": "SUM",
  "aggregation_output": "https://storage.googleapis.com/my-storage-bucket/datastore/aggregation_outputs/4295fd00-4ecd-494e-9eeb-884ef620ce14.json",
  "derivatives": [
    7,
    9
  ],
  "unit": "KWH",
  "baseline_value": 4863.15574521486,
  "baseline_lower": 1.5110421742195,
  "baseline_upper": 1.5110421742195,
  "baseline_n": 730.0,
  "reporting_value": 4860.88336194519,
  "reporting_lower": 0.545198012058528,
  "reporting_upper": 0.545198012058528,
  "reporting_n": 730.0,
  "differential_direction": "BASELINE_MINUS_REPORTING",
  "differential_value": 2.27238326967017,
  "differential_lower": 1.60639015330105,
  "differential_upper": 1.60639015330105,
  "differential_n": 1460.0,
  "eemeter_version": "0.4.12",
  "datastore_version": "0.2.3",
  "added": "2016-11-18T03:04:03.701863Z",
  "upda[...]
{'fraction_non_alphanumeric': 0.11187873788409981, 'fraction_numerical': 0.03413074860796041, 'mean_word_length': 3.4366422689844467, 'pattern_counts': {'":': 99, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '21069', 'license': 'Apache-2.0', 'n_tokens_mistral': 6182, 'n_tokens_neox': 5649, 'n_words': 1855, 'path': 'docs/datastore_basic_usage.ipynb', 'repo_name': 'tsennott/eemeter', 'size': '19399'}