<commit_before>===============================================================================
The Public Utility Data Liberation Project (PUDL)
===============================================================================

.. readme-intro

.. image:: https://www.repostatus.org/badges/latest/wip.svg
   :alt: Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.
   :target: https://www.repostatus.org/#wip

.. image:: https://travis-ci.org/catalyst-cooperative/pudl.svg?branch=master
   :target: https://travis-ci.org/catalyst-cooperative/pudl
   :alt: Build Status

.. image:: https://readthedocs.org/projects/catalystcoop-pudl/badge/?version=latest
   :target: https://catalystcoop-pudl.readthedocs.io/en/latest/
   :alt: Documentation Status

.. image:: https://codecov.io/gh/catalyst-cooperative/pudl/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/catalyst-cooperative/pudl
   :alt: codecov

.. image:: https://api.codacy.com/project/badge/Grade/2fead07adef249c08288d0bafae7cbb5
   :target: https://app.codacy.com/app/zaneselvans/pudl
   :alt: Codacy Badge

`PUDL <https://catalyst.coop/pudl/>`__ makes US energy data easier to access
and work with. Hundreds of gigabytes of supposedly public information published
by government agencies, but in a bunch of different formats that can be hard to
work with and combine. PUDL takes these spreadsheets, CSV files, and databases
and turns them into easy to parse, well-documented
`tabular data packages <https://https://frictionlessdata.io/docs/tabular-data-package/>`__
that can be used to create a database, used directly with Python, R, Microsoft
Access, and lots of other tools.

The project currently contains data from:

* `EIA Form 860 <https://www.eia.gov/electricity/data/eia860/>`__
* `EIA Form 923 <https://www.eia.gov/electricity/data/eia923/>`__
* `The EPA Continuous Emissions Monitoring System (CEMS) <https://ampd.epa.gov/ampd/>`__
* `The EPA Integrated Planning Model (IPM) <https://www.epa.gov/airmarkets/national-electric-energy-data-system-needs-v6>`__
* `FERC Form 1 <https://www.ferc.gov/docs-filing/forms/form-1/data.asp>`__

We are especially interested in serving researchers, activists, journalists,
and policy makers that might not otherwise be able to afford access to this
data from commercial data providers.

Getting Started TEMPORARILY OUT OF DATE
---------------------------------------

Just want to play with some example data? Install
`Anaconda <https://www.anaconda.com/distribution/>`__
(or `miniconda <https://docs.conda.io/en/latest/miniconda.html>`__
if you like the command line) with at least Python 3.7. Then run the following
commands in your terminal:

**NOTE: (2019-09-02)** this next code block won't work unless you have the old
PostgreSQL PUDL database set up. We are in the process of deprecating that
database, and using tabular datapackages that feed into SQLite instead.
However, the code is temporarily out of sync with the docs. The
``getting_started`` docs from

`this commit <https://github.com/catalyst-cooperative/pudl/tree/14d6fb2a1e0cb4d114434e9277a83f480861f8e8/docs>`__
can help you get that set up in the interim if need be.

.. code-block:: console

    $ git clone https://github.com/catalyst-cooperative/pudl.git
    $ conda env create --name pudl --file pudl/environment.yml
    $ conda activate pudl
    $ pip install -e pudl
    $ mkdir pudl-work
    $ pudl_setup --pudl_in=pudl-work --pudl_out=pudl-work
    $ pudl_data --sources eia923 eia860 ferc1 epacems epaipm --years 2017 --states id
    $ pudl_etl pudl-work/settings/pudl_etl_example.yml
    $ jupyter-lab --notebook-dir=pudl_workspace/notebooks

This will install the PUDL Python package, create some local directories
inside a directory called ``pudl-work``, download the most recent year of
data from the public agencies, load it into a local PostgreSQL database,
and open up a folder with some example `Jupyter noteboooks <https://jupyter.org>`__
in your web browser.

We are transitioning to generating CSV/JSON based
`tabular data packages <https://frictionlessdata.io/docs/tabular-data-package/>`__,
which are then loaded into a local SQLite database to make setting up PUDL<commit_msg>Fix formatting error.<commit_after>===============================================================================
The Public Utility Data Liberation Project (PUDL)
===============================================================================

.. readme-intro

.. image:: https://www.repostatus.org/badges/latest/wip.svg
   :alt: Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.
   :target: https://www.repostatus.org/#wip

.. image:: https://travis-ci.org/catalyst-cooperative/pudl.svg?branch=master
   :target: https://travis-ci.org/catalyst-cooperative/pudl
   :alt: Build Status

.. image:: https://readthedocs.org/projects/catalystcoop-pudl/badge/?version=latest
   :target: https://catalystcoop-pudl.readthedocs.io/en/latest/
   :alt: Documentation Status

.. image:: https://codecov.io/gh/catalyst-cooperative/pudl/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/catalyst-cooperative/pudl
   :alt: codecov

.. image:: https://api.codacy.com/project/badge/Grade/2fead07adef249c08288d0bafae7cbb5
   :target: https://app.codacy.com/app/zaneselvans/pudl
   :alt: Codacy Badge

`PUDL <https://catalyst.coop/pudl/>`__ makes US energy data easier to access
and work with. Hundreds of gigabytes of supposedly public information published
by government agencies, but in a bunch of different formats that can be hard to
work with and combine. PUDL takes these spreadsheets, CSV files, and databases
and turns them into easy to parse, well-documented
`tabular data packages <https://https://frictionlessdata.io/docs/tabular-data-package/>`__
that can be used to create a database, used directly with Python, R, Microsoft
Access, and lots of other tools.

The project currently contains data from:

* `EIA Form 860 <https://www.eia.gov/electricity/data/eia860/>`__
* `EIA Form 923 <https://www.eia.gov/electricity/data/eia923/>`__
* `The EPA Continuous Emissions Monitoring System (CEMS) <https://ampd.epa.gov/ampd/>`__
* `The EPA Integrated Planning Model (IPM) <https://www.epa.gov/airmarkets/national-electric-energy-data-system-needs-v6>`__
* `FERC Form 1 <https://www.ferc.gov/docs-filing/forms/form-1/data.asp>`__

We are especially interested in serving researchers, activists, journalists,
and policy makers that might not otherwise be able to afford access to this
data from commercial data providers.

Getting Started TEMPORARILY OUT OF DATE
---------------------------------------

Just want to play with some example data? Install
`Anaconda <https://www.anaconda.com/distribution/>`__
(or `miniconda <https://docs.conda.io/en/latest/miniconda.html>`__
if you like the command line) with at least Python 3.7. Then run the following
commands in your terminal:

**NOTE: (2019-09-02)** this next code block won't work unless you have the old
PostgreSQL PUDL database set up. We are in the process of deprecating that
database, and using tabular datapackages that feed into SQLite instead.
However, the code is temporarily out of sync with the docs. The
``getting_started`` docs from
`this commit <https://github.com/catalyst-cooperative/pudl/tree/14d6fb2a1e0cb4d114434e9277a83f480861f8e8/docs>`__
can help you get that set up in the interim if need be.

.. code-block:: console

    $ git clone https://github.com/catalyst-cooperative/pudl.git
    $ conda env create --name pudl --file pudl/environment.yml
    $ conda activate pudl
    $ pip install -e pudl
    $ mkdir pudl-work
    $ pudl_setup --pudl_in=pudl-work --pudl_out=pudl-work
    $ pudl_data --sources eia923 eia860 ferc1 epacems epaipm --years 2017 --states id
    $ pudl_etl pudl-work/settings/pudl_etl_example.yml
    $ jupyter-lab --notebook-dir=pudl_workspace/notebooks

This will install the PUDL Python package, create some local directories
inside a directory called ``pudl-work``, download the most recent year of
data from the public agencies, load it into a local PostgreSQL database,
and open up a folder with some example `Jupyter noteboooks <https://jupyter.org>`__
in your web browser.

We are transitioning to generating CSV/JSON based
`tabular data packages <https://frictionlessdata.io/docs/tabular-data-package/>`__,
which are then loaded into a local SQLite database to make setting up PUDL
{'fraction_non_alphanumeric': 0.14415994387934059, 'fraction_numerical': 0.01800537822986087, 'mean_word_length': 4.276989512646514, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 48, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2077492', 'n_tokens_mistral': 2865, 'n_tokens_neox': 2625, 'n_words': 847}