<commit_before>wellbook
========
The wellbook concept is about a single view of an oil well and its history- something akin to a "Facebook Wall" for oil wells.

This repo is built from data collected and made available by the [North Dakota Industrial Commission](https://www.dmr.nd.gov/oilgas).

I used the wellindex.csv file to obtain a list of well file numbers (file_no), scraped their respective Production, Injection, Scout Ticket web pages, any available LAS format well logfiles, and loaded them into HDFS (/user/dev/wellbook/) for analysis.

To avoid the HDFS small files problem I used the Apache Mahout seqdirectory tool for combining my textfiles into SequenceFiles: the keys are the filenames and the values are the contents of each textfile.

Then I used a combination of Hive queries and the pyquery Python library for parsing relevant fields out of the raw HTML pages.

Tables:  
wellbook.wells -- [well metadata](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) including geolocation and owner  
wellbook.well_surveys -- [borehole curve](https://www.dmr.nd.gov/oilgas/feeservices/getsurveydata.asp?ID=8895135722786)  
wellbook.production -- how much [oil, gas, and water](https://www.dmr.nd.gov/oilgas/feeservices/getwellprod.asp?filenumber=22786) was produced for each well on a monthly basis  
wellbook.auctions -- how much was paid for each parcel of land at [auction](http://www.land.nd.gov/minerals/mineralapps/auctions/auctionhistorysale.aspx)  
wellbook.injections -- how much [fluid and gas](https://www.dmr.nd.gov/oilgas/feeservices/getwellinj.asp?filenumber=5600) was injected into each well (for enhanced oil recovery and disposal purposes) 
wellbook.log_metadata -- metadata for each [LAS well log file](http://pubs.usgs.gov/of/2007/1142/)  
wellbook.log_readings -- sensor readings for each depth step in all [LAS well log files](http://pubs.usgs.gov/of/2007/1142/)  
wellbook.log_key -- [map](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) of log mnemonics to their descriptions
wellbook.formations -- manually annotated [map](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) of well depths to rock formations
wellbook.formations_key -- [Descriptions](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) of rock formations

wellbook.water_sites -- [metadata](http://waterservices.usgs.gov/nwis/site/?stateCd=nd) for water quality monitoring stations in North Dakota  

Setup:
```<commit_msg>edited README formatting<commit_after>wellbook
========
The wellbook concept is about a single view of an oil well and its history- something akin to a "Facebook Wall" for oil wells.

This repo is built from data collected and made available by the [North Dakota Industrial Commission](https://www.dmr.nd.gov/oilgas).

I used the wellindex.csv file to obtain a list of well file numbers (file_no), scraped their respective Production, Injection, Scout Ticket web pages, any available LAS format well logfiles, and loaded them into HDFS (/user/dev/wellbook/) for analysis.

To avoid the HDFS small files problem I used the Apache Mahout seqdirectory tool for combining my textfiles into SequenceFiles: the keys are the filenames and the values are the contents of each textfile.

Then I used a combination of Hive queries and the pyquery Python library for parsing relevant fields out of the raw HTML pages.

Tables:  
wellbook.wells -- [well metadata](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) including geolocation and owner  
wellbook.well_surveys -- [borehole curve](https://www.dmr.nd.gov/oilgas/feeservices/getsurveydata.asp?ID=8895135722786)  
wellbook.production -- how much [oil, gas, and water](https://www.dmr.nd.gov/oilgas/feeservices/getwellprod.asp?filenumber=22786) was produced for each well on a monthly basis  
wellbook.auctions -- how much was paid for each parcel of land at [auction](http://www.land.nd.gov/minerals/mineralapps/auctions/auctionhistorysale.aspx)  
wellbook.injections -- how much [fluid and gas](https://www.dmr.nd.gov/oilgas/feeservices/getwellinj.asp?filenumber=5600) was injected into each well (for enhanced oil recovery and disposal purposes)  
wellbook.log_metadata -- metadata for each [LAS well log file](http://pubs.usgs.gov/of/2007/1142/)  
wellbook.log_readings -- sensor readings for each depth step in all [LAS well log files](http://pubs.usgs.gov/of/2007/1142/)  
wellbook.log_key -- [map](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) of log mnemonics to their descriptions
wellbook.formations -- manually annotated [map](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) of well depths to rock formations
wellbook.formations_key -- [Descriptions](https://www.dmr.nd.gov/oilgas/feeservices/flatfiles/flatfiles.asp) of rock formations
wellbook.water_sites -- [metadata](http://waterservices.usgs.gov/nwis/site/?stateCd=nd) for water quality monitoring stations in North Dakota  

Setup:
```
{'fraction_non_alphanumeric': 0.1001001001001001, 'fraction_numerical': 0.015215215215215214, 'mean_word_length': 4.749136939010357, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 16, 'lorem ipsum': 0, 'www.': 18, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7260668', 'n_tokens_mistral': 1601, 'n_tokens_neox': 1484, 'n_words': 496}