<commit_before>The primary dataset is the information extracted for United States from **DOE Global Energy Storage Database**.
The data set roughly contains about **1629 rows and 106 columns**.
The **dimensionality**(usually, the number of columns for a given data set) is **106**.
The columns of primary importance in the data set are listed below:


* Project name
* Technology
* Technology type
* Rated Power in kW
* Duration
* Status
* Longitude and Longitude
* City and State
* Street Address of the Project
* Zip and Mail Code

The complete data set can be obtained from DOE Global Energy Storage Database at `http://www.energystorageexchange.org`.
It is possible to create subsets for the dataset for unit tests using integer or label based indexing in a python dataframe.

The other dataset is of the **United States EPA** (Environmental Protection Agency).
The data set roughly contains about **8505 rows and 120 columns**
The **dimensionality**(usually, the number of columns for a given data set) is **120**.
The columns of primary importance in the data set are listed below:

* Plant name
* Plant Latitude and Longitude
* Plant capacity factor
* Plant annual net generation for oil,coal,gas,nuclear power,hydropower,biomass,wind,solar,geothermal
  and other fossils
* Plant annual total non-renewable net generation
* Plant annual total renewable net generation
* Plant generation % for oil,coal,gas,nuclear power,hydropower,biomass,wind,solar,geothermal
  and other fossils
* Plant total non-renewables generation %
* Plant totals renewable generation %

The link for the above dataset is `https://www.epa.gov/energy/emissions-generation-resource-integrated-database-egrid`
It is possible to generate subsets for the dataset for unit tests by using integer or label based indexing in python dataframe
<commit_msg>Clarifications regarding data<commit_after>The primary dataset is the information extracted for United States from **DOE Global Energy Storage Database**.
The data set contains **1629 rows** (representing 1629 U.S. energy storage facilities) and **106 columns**.
The columns of primary importance in the data set are listed below:


* Project name
* Technology
* Technology type
* Rated Power in kW
* Duration
* Longitude and Longitude

The complete data set can be obtained from DOE Global Energy Storage Database at `http://www.energystorageexchange.org`.
It is possible to create subsets for the dataset for unit tests using integer or label based indexing in a python dataframe.

The other dataset is of the **United States EPA** (Environmental Protection Agency).
The complete dataset contains information at the level of power generators, power plants, states, grid regions, and more. Our project will focus on the power plant data and potentially also incorporate the state-level data.  
The power plant dataset contains **8505 rows** (representing 8505 U.S. power plants) and **120 columns**
The columns of primary importance in the data set are listed below:

* Plant name
* Plant Latitude and Longitude
* Plant capacity factor
* Plant annual net generation for oil,coal,gas,nuclear power,hydropower,biomass,wind,solar,geothermal and other fossil fuels

From the plant annual net generation columns, we will extract the primary energy type for each plant to be used in our classification model.  
The link for the above dataset is `https://www.epa.gov/energy/emissions-generation-resource-integrated-database-egrid`
It is possible to generate subsets for the dataset for unit tests by using integer or label based indexing in python dataframe

{'fraction_non_alphanumeric': 0.05727856943280246, 'fraction_numerical': 0.01173512154233026, 'mean_word_length': 5.098807495741056, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5291887', 'n_tokens_mistral': 926, 'n_tokens_neox': 839, 'n_words': 482}