<commit_before>NILM METADATA
=============

NILM Metadata (where `NILM' stands for `non-instrusive load
monitoring') is a metadata framework for describing appliances, meters,
measurements, buildings and datasets.

The aim is that NILM Metadata can be used as a stand-alone project to
specify the metadata for any NILM dataset; and that metadata can then
be used with the open-source energy disaggregation and analytics
framework [NILMTK](http://nilmtk.github.io/).

Please jump in and add to or modify the schema and documentation!

There are two sides to NILM Metadata:

### 1) A schema for  metadata that is included with disaggregated energy datasets

The schema covers:

* electricity meters (whole-home and individual appliance meters)
  * wiring hierarchy of meters
  * a controlled vocabulary for measurement names
* domestic appliances
  * a controlled vocabulary for appliance names
  * each appliance can contain any number of components (e.g. a
    light fitting can contain multiple lamps and a dimmer)
  * a list of time periods when each appliance was active
* a mapping of which appliances are connected to which meters
* buildings
* datasets

The metadata itself can be either
[YAML](http://en.wikipedia.org/wiki/YAML) or JSON.

### 2) Common information about appliances

Common info about appliances is stored in NILM Metadata.  This includes:

* Categories for each appliance type
* prior knowledge about the distribution of variables such as:
  * on power
  * on duration
  * usage in terms of hour per day
  * appliance correlations (e.g. that the TV is usually on if the
    games console is on)
* valid additional properties for each appliance

The common info about appliances uses a simple but powerful
inheritance mechanism to allow appliances to inherit from a other
appliances.  For example, `laptop computer` is a specialisation of
`computer` and the two share several properties (e.g. both are in the
`ICT` category).  So `laptop computer` inherits from `computer` and
modifies and adds any properties it needs.  In this way, we can
embrace the
["don't repeat yourself (DRY)"](http://en.wikipedia.org/wiki/Don%27t_repeat_yourself)
principal by exploiting the relationship between appliances.

## Research paper describing NILM metadata

The following paper describes NILM metadata in detail:

* <NAME> and <NAME> (2014). **Metadata for Energy
  Disaggregation**. In The 2nd IEEE International Workshop on Consumer
  Devices and Systems (CDS 2014) in Västerås, Sweden.
  [arXiv:1403.5946](http://arxiv.org/abs/1403.5946)

Please cite this paper if you use NILM metadata.

## JSON Schema has been depreciated

In order to automate the validation of metadata instances, we did
write a very comprehensive (and complex) schema using JSON Schema.
JSON Schema is a lovely language and can capture everything we need
but, because our metadata is quite comprehensive, we found that using
JSON Schema was a significant time drain and made it hard to move
quickly and add new ideas to the metadata.  As such, when we moved
from v0.1 to v0.2, the JSON Schema has been depreciated.  Please use
the human-readable documentation instead.  If there is a real desire
for automated validation then we could resurrect the JSON Schema, but
it is a fair amount of work to maintain, and is a little fragile.

## Examples

### Example of dataset metadata

dataset.yaml
```
name: UK-DALE
long_name: UK Domestic Appliance-Level Electricity
mains_voltage:
  nominal: 230
  upper_limit: 253
  lower_limit: 215
```

meter_devices.yaml
```
- model: EnviR
  manufacturer: Current Cost
  measurements:
  - physical_quantity: power
    ac_type: apparent
    lower_limit: 0
    upper_limit: 30000
```

building1.yaml
```
instance: 1
rooms:
- {name: kitchen, instance: 1}
- {name: lounge, instance: 1}
geo_location:
  locality: London
  country: GB
  latitude: 51.464462
  longitude: -0.076544
timezone: Europe/London
timeframe:
  start: 2012-11-09
  end: 2014-03-11
elec_meters: # Metadata about appliances

# Meter that measures whole-house mains:
- instance: 1
  device_model: EnviR
  site_meter: true
  sensors:
  - data_location: house1/channel_1.dat

# Meter that measures lighting circuit:
- instance: 2<commit_msg>more README formatting tweaks<commit_after>NILM METADATA
=============

NILM Metadata (where `NILM' stands for `non-instrusive load
monitoring') is a metadata framework for describing appliances, meters,
measurements, buildings and datasets.

The aim is that NILM Metadata can be used as a stand-alone project to
specify the metadata for any NILM dataset; and that metadata can then
be used with the open-source energy disaggregation and analytics
framework [NILMTK](http://nilmtk.github.io/).

Please jump in and add to or modify the schema and documentation!

There are two sides to NILM Metadata:

### 1) A schema for  metadata that is included with disaggregated energy datasets

The schema covers:

* electricity meters (whole-home and individual appliance meters)
  * wiring hierarchy of meters
  * a controlled vocabulary for measurement names
* domestic appliances
  * a controlled vocabulary for appliance names
  * each appliance can contain any number of components (e.g. a
    light fitting can contain multiple lamps and a dimmer)
  * a list of time periods when each appliance was active
* a mapping of which appliances are connected to which meters
* buildings
* datasets

The metadata itself can be either
[YAML](http://en.wikipedia.org/wiki/YAML) or JSON.

### 2) Common information about appliances

Common info about appliances is stored in NILM Metadata.  This includes:

* Categories for each appliance type
* prior knowledge about the distribution of variables such as:
  * on power
  * on duration
  * usage in terms of hour per day
  * appliance correlations (e.g. that the TV is usually on if the
    games console is on)
* valid additional properties for each appliance

The common info about appliances uses a simple but powerful
inheritance mechanism to allow appliances to inherit from a other
appliances.  For example, `laptop computer` is a specialisation of
`computer` and the two share several properties (e.g. both are in the
`ICT` category).  So `laptop computer` inherits from `computer` and
modifies and adds any properties it needs.  In this way, we can
embrace the
["don't repeat yourself (DRY)"](http://en.wikipedia.org/wiki/Don%27t_repeat_yourself)
principal by exploiting the relationship between appliances.

## Research paper describing NILM metadata

The following paper describes NILM metadata in detail:

* <NAME> and <NAME> (2014). **Metadata for Energy
  Disaggregation**. In The 2nd IEEE International Workshop on Consumer
  Devices and Systems (CDS 2014) in Västerås, Sweden.
  [arXiv:1403.5946](http://arxiv.org/abs/1403.5946)

Please cite this paper if you use NILM metadata.

## JSON Schema has been depreciated

In order to automate the validation of metadata instances, we did
write a very comprehensive (and complex) schema using JSON Schema.
JSON Schema is a lovely language and can capture everything we need
but, because our metadata is quite comprehensive, we found that using
JSON Schema was a significant time drain and made it hard to move
quickly and add new ideas to the metadata.  As such, when we moved
from v0.1 to v0.2, the JSON Schema has been depreciated.  Please use
the human-readable documentation instead.  If there is a real desire
for automated validation then we could resurrect the JSON Schema, but
it is a fair amount of work to maintain, and is a little fragile.

## Examples

### Example of dataset metadata

These files would all be in a `metadata` directory:

#### `dataset.yaml`
```
name: UK-DALE
long_name: UK Domestic Appliance-Level Electricity
mains_voltage:
  nominal: 230
  upper_limit: 253
  lower_limit: 215
```

#### `meter_devices.yaml`
```
- model: EnviR
  manufacturer: Current Cost
  measurements:
  - physical_quantity: power
    ac_type: apparent
    lower_limit: 0
    upper_limit: 30000
```

#### `building1.yaml`
```
instance: 1
rooms:
- {name: kitchen, instance: 1}
- {name: lounge, instance: 1}
geo_location:
  locality: London
  country: GB
  latitude: 51.464462
  longitude: -0.076544
timezone: Europe/London
timeframe:
  start: 2012-11-09
  end: 2014-03-11
elec_meters: # Metadata about appliances

# Meter that measures whole-house mains:
- instance: 1
  device_model: EnviR
  site_meter: true
  sensors:
  - data_location: house1/channel_1.dat

# Meter that measures lighting circuit:
- instance: 2
{'fraction_non_alphanumeric': 0.06672138836772983, 'fraction_numerical': 0.020403377110694183, 'mean_word_length': 4.354048964218456, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4055245', 'n_tokens_mistral': 2562, 'n_tokens_neox': 2326, 'n_words': 1137}