<jupyter_start><jupyter_text>Using the `python-vegindex` package command line toolsThe `python-vegindex` package can be used to extract greeness-index values from PhenoCam Network images.  In this notebook we'll download some images and a region-of-interest (ROI) mask list and extract the greeness-index timeseries. Input DataTo generate the standard PhenoCam timeseries you'll need images from one of the sites and you'll also need to specify the ROI for those images. Downloading image dataOn the [PhenoCam Network website](https://phenocam.sr.unh.edu/webcam/) you'll find a link to the "Data" page.  The page contains a form for downloading image data.  To access this page you must first register with the site using a valid e-mail.  After registration you can log in to the site and access the [Data Page](https://phenocam.sr.unh.edu/webcam/network/download/).  Fill out the form:![PhenoCam Network Download Form](./phenocam_download_form.png) If the data request was successfully processed, a `.zip` file download will start. Extract the images from the zipfileinto a directory of your choice.  For this notebook I've just placed the data in `./phenocamdata/`:    phenocamdata/bartlettir/    ├── 2016    │   ├── 01    │   │   ├── bartlettir_2016_01_01_113137.jpg    │   │   ├── bartlettir_2016_01_01_114107.jpg    │   │   ├── bartlettir_2016_01_01_115106.jpg    │   │   ├── bartlettir_2016_01_01_120107.jpg    │   │   ├── bartlettir_2016_01_01_121110.jpg    │   │   ├── bartlettir_2016_01_01_122107.jpg    │   │   ├── bartlettir_2016_01_01_123108.jpg     .     .     .    │       ├── bartlettir_2016_12_19_120107.jpg    │       ├── bartlettir_2016_12_19_121107.jpg    │       ├── bartlettir_2016_12_19_122107.jpg    │       ├── bartlettir_2016_12_19_123109.jpg    │       ├── bartlettir_2016_12_19_124108.jpg    │       └── bartlettir_2016_12_19_125106.jpg    ├── bartlettir_meta.json    └── bartlettir_meta.txtThe `python-vegindex` package will expect the data to be in this format.   Downloading the ROI dataThe other input data we need to use the package is an `ROI List` file and the associated `ROI masks`.  The list just specifies a date range and mask for the time period covered by the images.  If the field-of-view (FOV) of the camerais stable the list will only need a single entry which can cover the entire time range of the images.Each ROI page on the PhenoCam Network website (e.g. [bartlettir DB_0001](https://phenocam.sr.unh.edu/data/archive/bartlettir/ROI/bartlettir_DB_0001.html) has links for the `ROI List` and `ROI masks` in the list.  Use those links to download the list and masks.  These need to be placed in a sub-directory(a.k.a. folder) named `ROI` under the site directory:    phenocamdata/bartlettir/ROI/    ├── bartlettir_DB_0001_01.tif    ├── bartlettir_DB_0001_02.tif    ├── bartlettir_DB_0001_03.tif    ├── bartlettir_DB_0001_04.tif    └── bartlettir_DB_0001_roi.csvIn this example we only have data for a single year (2016) but the downloaded ROI list and ROI masks cover data for several years.  As long as the ROI list covers the time period for the image data we can generate the time series data.  Image data and ROI data for multiple sites can all be placed in the same `phenocamdata` directory.  Setting up a virtual environmentVirtual environments attempt to deal with the problem of creating an isolated and reproduceable environment in which to run a software package.  There are several ways to deal with virtual environments in python. I'll go over the basics here but a full discussion is beyond the scope of this document. `virtualenv` and `virtualenvwrapper`For `python2` these tools are widely used and you should be able to find installation instructions for your operating system.  Once these tools are installed you can do the following:    mkdir test_vegindex    cd test_vegindex/    mkvirtualenv test_vegindex    pip install -U pip     pip install vegindex    You should now have the `python-vegindex` package installed.  Because the `vegindex` package depends on several other packages you may encounter errors duing installation involving missing dependencies.  If this is case you can try installing the `Pillow` package before installing vegindex:    pip install Pillow    pip install vegindex  `anaconda` and `miniconda`Anconda python distributions   Command Line ToolsIf all goes well you should have two new commands available, `generate_roi_timeseries` and `generate_summary_timeseries`.  Both commands depend on a environment variable to specify the path to the input data.Depending on your operating system one of the following commands should work:    export PHENOCAM_ARCHIVE_DIR=./phenocamdataor    set PHENOCAM_ARCHIVE_DIR=./phenocamdata  `generate_roi_timeseries` The first command will generate summary statistics for the pixels within the ROI mask region for each image in the site directory.  The only filter applied to the image is on the total brightness (R + G + B).  Images which are too dark or too light will not have statistics calculated.<jupyter_code>%%bash
generate_roi_timeseries --help<jupyter_output>usage: generate_roi_timeseries [-h] [-v] [-n] site roiname

positional arguments:
  site           PhenoCam site name
  roiname        ROI name, e.g. DB_0001

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose  increase output verbosity
  -n, --dry-run  Process data but don't save results<jupyter_text>To actually run the tool you would do something like:    export PHENOCAM_ARCHIVE_DIR='./phenocamdata'    generate_roi_timeseries bartlettir DB_0001    Using the `-v` option will send the resulting output to the screen as well as to the output file.  The output file will be generated in the `ROI` directory and will have the name (in this case) of `bartlettir_DB_0001_timeseries.csv`.<jupyter_code>%%bash
head -22 ./phenocamdata/bartlettir/ROI/bartlettir_DB_0001_timeseries.csv<jupyter_output>#
# All-image time series for bartlettir
#
# Site: bartlettir
# Veg Type: DB
# ROI ID Number: 0001
# Lat: 44.0646
# Lon: -71.2881
# Elev: 268
# UTC Offset: -5
# Resize Flag: False
# Version: 1
# Creation Date: 2017-06-23
# Creation Time: 11:04:49
# Update Date: 2017-06-23
# Update Time: 11:08:57
#
date,local_std_time,doy,filename,solar_elev,exposure,mask_index,gcc,rcc,r_mean,r_std,r_5_qtl,r_10_qtl,r_25_qtl,r_50_qtl,r_75_qtl,r_90_qtl,r_95_qtl,g_mean,g_std,g_5_qtl,g_10_qtl,g_25_qtl,g_50_qtl,g_75_qtl,g_90_qtl,g_95_qtl,b_mean,b_std,b_5_qtl,b_10_qtl,b_25_qtl,b_50_qtl,b_75_qtl,b_90_qtl,b_95_qtl,r_g_correl,g_b_correl,b_r_correl
2016-01-01,11:31:37,1,bartlettir_2016_01_01_113137.jpg,22.85398,NA,4,0.35161,0.32339,88.88929,34.32057,52,58,66,78,102,140,161,96.64661,39.46101,56,61,70,83,113,157,182,89.32929,40.57407,48,53,62,75,106,151,177,0.97558,0.98391,0.96634
2016-01-01,11:41:07,1,bartlettir_2016_01_01_114107.jpg,22.94494,NA,4,0.35386,0.32823,95.51360,34.76272,58,63,73,85,109,147,169,102.97396[...]<jupyter_text>`generate_summary_timeseries`The second command reads in the output from `generate_roi_timeseries` tool and produces 1-day or 3-day summaries of the images taken during this period.  These summaries are used to reduce the noise in the `greeness` signal due to variations in illumination that occur duing the day.  These variations are a result of varying sun angle, atmospheric conditions, exposure, etc.<jupyter_code>%%bash
generate_summary_timeseries --help<jupyter_output>usage: generate_summary_timeseries [-h] [-v] [-n] [-p [{1,3}]] site roiname

positional arguments:
  site                  PhenoCam site name
  roiname               ROI name, e.g. canopy_0001

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         increase output verbosity
  -n, --dry-run         Process data but don't save results
  -p [{1,3}], --aggregation-period [{1,3}]
                        Number of Days to Aggregate (default=1)<jupyter_text>To actually run the tool you would do something like:    export PHENOCAM_ARCHIVE_DIR='./phenocamdata'    generate_summary_timeseries -p 3 bartlettir DB_0001    where we've specified a 3-day time period for aggregation.<jupyter_code>%%bash 
export PHENOCAM_ARCHIVE_DIR='./phenocamdata'
generate_summary_timeseries -p 3 bartlettir DB_0001;
%%bash
head -25 phenocamdata/bartlettir/ROI/bartlettir_DB_0001_3day.csv<jupyter_output>#
# {0}-day summary product timeseries for bartlettir
#
# Site: bartlettir
# Veg Type: DB
# ROI ID Number: 0001
# Lat: 44.0646
# Lon: -71.2881
# Elev: 268
# UTC Offset: -5
# Image Count Threshold: 1
# Aggregation Period: 3
# Solar Elevation Min: 10.0
# Time of Day Min: 00:00:00
# Time of Day Max: 23:59:59
# ROI Brightness Min: 100
# ROI Brightness Max: 665
# Creation Date: 2017-06-23
# Creation Time: 11:37:46
# Update Date: 2017-06-23
# Update Time: 11:37:47
#
date,year,doy,image_count,midday_filename,midday_r,midday_g,midday_b,midday_gcc,midday_rcc,r_mean,r_std,g_mean,g_std,b_mean,b_std,gcc_mean,gcc_std,gcc_50,gcc_75,gcc_90,rcc_mean,rcc_std,rcc_50,rcc_75,rcc_90,max_solar_elev,snow_flag,outlierflag_gcc_mean,outlierflag_gcc_50,outlierflag_gcc_75,outlierflag_gcc_90
2016-01-02,2016,2,21,bartlettir_2016_01_02_120108.jpg,90.50640,96.07558,76.63615,0.36500,0.34385,90.24518,6.16712,99.71275,6.28255,86.02489,6.31702,0.36120,0.00550,0.36157,0.36552,0.36805,0.32706,0.01626,0.32700,0.33595,0.3450[...]<jupyter_text>The resulting CSV files are in a format which can be easily plotted or used for further analysis.  Of course the data generated here is also available for download from the PhenoCam Network website.  So the real utility of the command-line tools is for generating your own ROI list and masks.I've shown how to use Pandas to plot the data in a separate notebook.<jupyter_code>%run Plotting_Output.ipynb<jupyter_output>./phenocamdata/bartlettir/ROI/bartlettir_DB_0001_timeseries.csv
{'fraction_non_alphanumeric': 0.08293525764813497, 'fraction_numerical': 0.08822034759630043, 'mean_word_length': 4.626072041166381, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '486921', 'license': 'CC0-1.0', 'n_tokens_mistral': 3967, 'n_tokens_neox': 3316, 'n_words': 1076, 'path': 'python-vegindex_CLI/Using_the_python-vegindex_package.ipynb', 'repo_name': 'tmilliman/phenocam_notebooks', 'size': '9839'}