<jupyter_start><jupyter_text>Convert a MOS file into CSVs This notebook walks through opening a MOS file and breaking it down into a list of stations. It also has the begginings of parsing the MOS file into a spreadsheet. The end goal is to fully convert the station such that the dataframe matches the output of https://mesonet.agron.iastate.edu/mos/fe.phtml  CloneHere we are downloading the folder in which we will work thoughout the summer.1. open a terminal in the jupyterhub/jetstream 2.  Clone the mosparse repository into your jupyterhub instance```bashhttps://github.com/ccny-glasslab/mosparse.git```  InstallWe need to install external libraries and the library we will be modifying, which is called `mosparse`  Install external library unlzwWe need to use a special library called [unlzw](https://github.com/umeat/unlzw) to unzip our `unlzw` files. Let's install it here:<jupyter_code>%pip install unlzw<jupyter_output><empty_output><jupyter_text>Install mosparse as editable libraryFirst let's install the library that we're going to be editing. To do this1. open a terminal2. cd into mosparse    ```bash    cd mosparse    ```3. now type ls to list the contents of the folder:```bashjovyan@jupyter-:~/mosparse$ ls```The following should show on screen:```bashAUTHORS.rst  CONTRIBUTING.rst  docs  LICENSE  MANIFEST.in  mosparse  notes  README.rst  setup.cfg  setup.py  versioneer.py```4. If you see a setup.py file, type the following to install an editable version of the source code:```bashpython -mpip install -ve . ```5. restart the kernel:(FYI: You will have to take these last two step whenever you close and reopen your program)![imm.png](attachment:imm.png)Alternatively:<jupyter_code>%pip install -ve ../mosparse/<jupyter_output><empty_output><jupyter_text>The following two lines of code are so that the notebook reloads the libraries every time you make changes.  Importing LibrariesWe need to load the libraries we will be using. We use the `%` [ipython magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) to automatically reimport our libraries after we save changes to the source code.<jupyter_code>%load_ext autoreload
%autoreload 2
%reload_ext autoreload
%conda install pandas<jupyter_output>Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\story\Anaconda3\envs\mpp

  added / updated specs:
    - pandas


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mkl-service-2.3.0          |   py38h2bbff1b_0          53 KB
    pandas-1.1.3               |   py38ha925a31_0         9.7 MB
    pytz-2020.4                |     pyhd3eb1b0_0         227 KB
    ------------------------------------------------------------
                                           Total:        10.0 MB

The following NEW packages will be INSTALLED:

  blas               pkgs/main/win-64::blas-1.0-mkl
  intel-openmp       pkgs/main/win-64::intel-openmp-2020.2-254
  mkl                pkgs/main/win-64::mkl-2020.2-256
  mkl-service        pkgs/main/win-64::mkl-service-2.3.0-py38h2bbff1b_0
  mkl_fft         [...]<jupyter_text>The libraries that will be) used for this example. [Pathlib](https://docs.python.org/3/library/pathlib.html)<jupyter_code>from pathlib import Path

import mosparse as mp
import mosparse.mavreader as mpr
import mosparse.mavparse as mpp<jupyter_output><empty_output><jupyter_text>Download DataLets download one `.Z` file and one `.gz` file from https://sats.nws.noaa.gov/~mos/archives/mav.htmlOpen a terminal and following commands to download files:```bashwget https://sats.nws.noaa.gov/~mos/archives/avnmav/mav202006.t18z.gz```make sure to press enter before entering the next command```bashwget https://sats.nws.noaa.gov/~mos/archives/avnmav/mav201302.t00z.Z```Find the folder that the files are downloaded in by typing the `pwd` command. `pwd` is print working directory.```bashjovyan@jupyter-$ pwd```  Open FilesMy data is in `/home/jovyan/data`. Here I'm going to assign each location of each file to a variable. I'm using the [Pathlib](https://docs.python.org/3/library/pathlib.html) interface because it simplifies file manipulation. We can also tab complete our file path.<jupyter_code>zpath = Path('/home/jovyan/data/mav201302.t00z.Z')
gzpath = Path('/home/jovyan/data/mav202006.t18z.gz')<jupyter_output><empty_output><jupyter_text>Let's print out the contents of the`zpath` & `gzpath` variables:<jupyter_code>zpath
gzpath<jupyter_output><empty_output><jupyter_text>Let's check that the file exists, meaning that the computer can find it<jupyter_code>print(zpath.exists(), gzpath.exists())<jupyter_output><empty_output><jupyter_text>We're going to try to open our `.gz` file to test that our `MavReader` function works:<jupyter_code># see if the a *.gz file opens properly
mpr.MavReader(gzpath)<jupyter_output><empty_output><jupyter_text>Let's open the file and call it `f` and then print the first 10 lines:<jupyter_code>with mpr.MavReader(gzpath) as f:
    print(list(f)[:10])<jupyter_output><empty_output><jupyter_text>Raw TextNow we're going to use our function as a context manager and checking if it reads the content properly. The function should return a text [stream](https://docs.python.org/3/library/io.html) object.  Here we're going to only print the first 10 lines in the file. We're going to use [enumerate](https://docs.python.org/3/library/functions.htmlenumerate) and [break](https://docs.python.org/3/tutorial/controlflow.html?highlight=breakbreak-and-continue-statements-and-else-clauses-on-loops) to keep track of the  of lines and stop at 25.<jupyter_code># look at the contents of the file 
with mpr.MavReader(gzpath) as f:
    for linenumber, line in enumerate(f):
        #print(linenumber, line)
        pass
    print(linenumber)<jupyter_output><empty_output><jupyter_text>Now we're going to try this with a `.Z` file<jupyter_code># look at the contents of the file 
with mpr.MavReader(zpath) as f:
    for linenumber, line in enumerate(f):
        print(line)
        if linenumber>25:
            break<jupyter_output><empty_output><jupyter_text>Stations In the text above, all the stations are in one giant block. Since the [GFS MOS](https://www.weather.gov/mdl/mos_getbull) text product is interpreted on a station by station basis, it'd be easier for us to work with if we could break the text up into a list of stations. We do so using the `stations` keyword argument to MavReader. What it does is instead of return a text like above, instead the MavReader converts the text to a generator (which is like a list where the elements only exist on demand) where we can request the station one by one. We use a generator so that we only load the stations as needed. Here we print out 1 station to examine it:<jupyter_code>with mpr.MavReader(gzpath, stations=True) as station_generator:
    for i, station in enumerate(station_generator):
        print(f"i={i}, station={station}")
        if i > 1:
            break<jupyter_output><empty_output><jupyter_text>Parse station into spreadsheet (Pandas DataFrame)We need to convert each station in our list (like the one above) into a spreadsheet. We use the functions in `mav_parse.py` to do our conversion. We will walk through the steps here, but all these steps are combined in the `parse_station`<jupyter_code>station<jupyter_output><empty_output><jupyter_text>Get header informationIn the header we store the information common to all variables in the model. The first row in the station list is the header information as described in `mav_parse.get_header`<jupyter_code>station[0]
header = mpp.get_header(station[0])
header
header['station']<jupyter_output><empty_output><jupyter_text>Get the finish timeThis is the time at which the prediction is supposed to occur. Each mav text product predicts various weather variables at each time and stores predictions for multiple times. Each fntime is a row in our spreadsheet. We update the header dictionary. This information is stored in the 2nd and 3rd rows of the station list/text.<jupyter_code>station[1:3]
header['ftime'] = mpp.get_fntime(station[1], station[2], header)
header<jupyter_output><empty_output><jupyter_text>Get the measurementsNext we need to pull out the measurements and place them in the right row and column.<jupyter_code>df = mpp.get_rows(header, station)
df<jupyter_output><empty_output><jupyter_text>Combine all the stepsWe packaged all the steps into parse station to make the code most reusable<jupyter_code>df = mpp.parse_station(station)
df<jupyter_output><empty_output><jupyter_text>write the fileSometimes it's going to be more convieniant to write the file out into small .csvs that we'll later read into a database or something. This is especially useful in case code breaks.Here we have the write_station function return the path to the file it wrote:<jupyter_code>mpp.write_station(station)<jupyter_output><empty_output><jupyter_text>This generates a .csv file in the mos/modelrun folder. You can open and view this file by clicking on it.  log errorsif for whatever reason our parsing doesn't work, we can write the error in a log file.<jupyter_code>station_broken = station.copy()
#lets break the middle row
station_broken[10] = 'a b c d'
station_broken
mpp.write_station(station_broken)<jupyter_output><empty_output><jupyter_text>Next Steps: Try parsing a whole file into a set of spreadsheets1. Compose the functions described above to generate a set of spreadsheets that corresponds to each station record in the file.2. Monitor the logs to see what is broken as that will have to be fixed.<jupyter_code>## Hint 1: Looping over the stations
The following block of code loops over every station in /home/jovyan/data/mav201302.t00z.Z
```python
zpath = Path('/home/jovyan/data/mav201302.t00z.Z')
with mpr.MavReader(gzpath, stations=True) as station_generator:
    for station in station_generator:
```

## Hint 2: Write out a station
```python
mpp.write_station(station)
```
The `write_station` function takes in a station argument and either
* writes out a csv if code can parse the station correctly
* writes out a log of how the code fails at parsing the station<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09295120061967467, 'fraction_numerical': 0.015491866769945779, 'mean_word_length': 4.267210606833248, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 82, 'https://': 13, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '346638', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3290, 'n_tokens_neox': 3088, 'n_words': 1146, 'path': 'docs/mavreader_usage.ipynb', 'repo_name': 'valgur/mosparse', 'size': '10328'}