<jupyter_start><jupyter_text>1. System Profile 2. CCBayou Data  Problem 3: Treaty Terms for the Colorado River BackgroundThe Colorado river originates in the Rocky mountains and has contributing tributariesand/or flows through the states of Wyoming, Colorado, Utah, Nevada, Arizona,and California. Prior to dams the river used to actually run to the ocean and at onetime in history actually had commerce by river boats. In the late 1800’s and early 1900’s the river and its tributaries were dammed and diverted. Around the 1920’sa legal document called the Colorado River Compact was developed. A Compact is similar to a treaty, except the parties to the treaty are states (essentially the individual states have entered into the treaty with the United States and Mexico). The Colorado River Compact divides the Colorado River into Upper and Lower Basins with the division being at Lee Ferry on the Colorado River one mile below the Paria River in Arizona.The Lower Basin states are Arizona, California, and Nevada, with small portions of New Mexico and Utah that are tributary to the Colorado River below Lee Ferry. The Upper Basin states are Colorado, New Mexico, Utah, and Wyoming, with a small portion of Arizona tributary to the Colorado River above Lee Ferry.Article III of the Compact apportions the waters of the Colorado River to the Upperand Lower Basins as follows:The Compact apportions the right to exclusive beneficial consumptive use of 7.5million acre-feet of water per year from the ”Colorado River System” in perpetuityto the Upper Basin and the Lower Basin.The Compact allows an additional 1.0 million acre-feet per year of increased beneficialconsumptive use to the Lower Basin.It provides water for Mexico pursuant to treaty. Water must first come from anysurplus over the waters allocated to the states in Article III(a) and (b). If thatsurplus is insufficient, then the burden of that deficiency shall be shared equally bythe Upper and Lower Basins.The Compact provides that the Upper Basin states will not cause the flow of theriver at Lee Ferry, Arizona to be depleted below an aggregate of 75 million acrefeetfor any period of ten consecutive years beginning with the ratification of theCompact.It provides that the Upper Basin states will not withhold water and the states of theLower Basin shall not require delivery of water which cannot reasonably be appliedto domestic and agricultural uses.  Problem StatementThe file named [colorado.txt](http://54.243.252.9/engr-1330-webroot/4-Databases/colorado.txt) contains annual natural flows volumes in the upperColorado river basin from 1906 to present. These flows are historical reconstructionat approximately the same location as the current USGS gage 09380000. Thelocation is approximately above (upstream of) Lee Ferry, hence the data representthe flows out of the upper basin.The data are in two columns, the first is the water year, the second column iscumulative discharge for the year in units of acre-feet.    YEAR ANNUAL.VOLUME    1906 18550021    1907 21201694    1908 12218817    1909 22356301    1910 14650616    ...Prepare an analysis of the data to support or refute the provisions of the Compact.You can assume that the data from 1906 to 1927 (the original Compact was ratifiedin 1922, many additional Compacts have been signed since that time, all trying to reapportionwater in the river basin witout violating the original agreement) were available to theauthors of the Colorado River Compact, analyze this period of data and decide if theallocations to the lower basin are realistic (8.5 million acre-feet-per year) assumingthe upper basin allocation is removed. Consumptive use means that the water willnot return to the basin. The data are ’natural’ flows and the upper basin allocationsare not removed in the data.   **Step 1:** Read the "colorado.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe.<jupyter_code># download the data file
# read into a dataframe
# plot
# describe the dataframe<jupyter_output><empty_output><jupyter_text>**Step 2:** Evaluate if the mean value in the pre-compact period is substantially different from the post-compact period.<jupyter_code># split the data into pre-compact and post-compact
# describe the two subsets
# box-plot both
# test for normal
# hypothesis test for difference in means
# conclusion(s)<jupyter_output><empty_output><jupyter_text>**Step 3:** Estimate the probability based on the the 20-year period that the lower basin allocation will not be met if upper basin allocation is satisfied. Estimate the probability that the allocation will not be met using the entire data set if the upper basin allocation is satisfied.<jupyter_code># empirical probability approach on the 20 years pre-compact; full data<jupyter_output><empty_output><jupyter_text>**Step 4:** Next assume that lower basin is satisfied first (this is the historical result of theoriginal and subsequent treaties). Estimate the probability that the upper basinallocation will not be met. Determine an upper basin allocation that will be met95% of the time. Again use the 20 year initial record, then the entire record.<jupyter_code># empirical probability approach on the 20 years pre-compact; full data<jupyter_output><empty_output><jupyter_text>**Step 5:** Was the initial 20 year period sufficiently different from the later data that theCompact authors made decisions based on unusually high or low flows?<jupyter_code># make a decision and report
import pandas as pd
import numpy as np
colorado_df = pd.read_table("colorado.txt") # Read the file as a table assign to a dataframe 
colorado_df.plot.line(x='YEAR', y='ANNUAL.VOLUME')
colorado_df.describe()<jupyter_output><empty_output><jupyter_text>We can kind of use the description to answer our questions.  Estimate the probability based on the the 20-year period that the lower basin allocation will not be met if upper basin allocation is satisfiedFirst filter the data<jupyter_code>junkdf = colorado_df[colorado_df['YEAR'] <= 1927] # screen by year
junkdf.plot.line(x='YEAR', y='ANNUAL.VOLUME') # plot the first 22 years
junkdf.describe()<jupyter_output><empty_output><jupyter_text>Then observe that the minimum is 12 million, so the upper basin's 7.5 million is satisfied all 22 years (yay for the upstreamers!)If the downstreamers get their allocation the total in the river needs to be 15 million, so the question is what fraction of observations is less than 15 million, that fraction is an estimate of the probability downstreamers won't get their water.<jupyter_code>result1 = junkdf.count()
result2 = junkdf[junkdf['ANNUAL.VOLUME'] < 15.0E6].count()
print ('Probability Downstreamers Wont Get their Agua =%.3f' % (result2[0]/result1[0]))<jupyter_output>Probability Downstreamers Wont Get their Agua =0.318<jupyter_text>The fraction is about 1/3; so 33% of the time, downstreamers won't get their water! Estimate the probability that the allocation will not be met using the entire data set if the upper basin allocation is satisfied.Here we use the same logic and scripts but dont need the date filter<jupyter_code>result1 = colorado_df.count()
result2 = colorado_df[colorado_df['ANNUAL.VOLUME'] < 15.0E6].count()
print ("Probability Downstreamers Wont Get their Agua =%.3f " % (result2[0]/result1[0]))<jupyter_output>Probability Downstreamers Wont Get their Agua =0.500<jupyter_text>So the downstreamers wont get water 1/2 the time.Next assume that lower basin is satisfied first (this is the historical result of the original and subsequent treaties). Determine an upper basin allocation that will be met 95% of the time. Again use the 22 year initial record, then the entire record.<jupyter_code>result1 = junkdf.count()
result2 = junkdf[junkdf['ANNUAL.VOLUME'] > 12.5e6].count()
print ('Probability Uptreamers Will Get their Agua based on 22 years =%.3f' % (result2[0]/result1[0]))
print ('Allocation Upstream =%.3f' % (12.5e6 - 7.5e6))
result1 = colorado_df.count()
result2 = colorado_df[colorado_df['ANNUAL.VOLUME'] > 8.5e6].count()
print ('Probability Uptreamers Will Get their Agua based on 98 years =%.3f' % (result2[0]/result1[0]))
print ('Allocation Upstream =%.3f' % (8.5e6 - 7.5e6))<jupyter_output>Probability Uptreamers Will Get their Agua based on 22 years =0.955
Allocation Upstream =5000000.000
Probability Uptreamers Will Get their Agua based on 98 years =0.949
Allocation Upstream =1000000.000<jupyter_text>So based on 22 years of record, 5 million acre feet allocation would be satisfied 95% of the time.Using the entire record, only 1 million acre feet was available for the upstream - interestingly that same extra 1 million already available to the downstreamers.Not a favorable contract for Upstreamers and Mexico, the big error was the choice of absolute volumes; flow fractions would have made more sense and still kept lawyers busy.  One observation in https://en.wikipedia.org/wiki/Cadillac_Desert notes that the base years were unusually wet - we can assess that claim using our hypothesis testing tools.  First we will check for normality<jupyter_code>newjunkdf = colorado_df[colorado_df['YEAR'] > 1927] # screen by year
stat, p = stats.normaltest(junkdf['ANNUAL.VOLUME'])
print('statistic=%.3f, p-value at rejection =%.3f' % (stat, p))
if p > 0.05:
    print('Probably normal distributed')
else:
    print('Probably Not-normal distributed')
    stat, p = stats.normaltest(newjunkdf['ANNUAL.VOLUME'])
print('statistic=%.3f, p-value at rejection =%.3f' % (stat, p))
if p > 0.05:
    print('Probably normal distributed')
else:
    print('Probably Not-normal distributed')<jupyter_output>statistic=7.070, p-value at rejection =0.029
Probably Not-normal distributed
statistic=2.071, p-value at rejection =0.355
Probably normal distributed<jupyter_text>So the first portion of the data seem non-normal, the later quite normal, likely a consequence of moderate sample size (80-ish) and a bounded physical process (infinite discharge makes no sense, less then zero also undefined).  Lets ask Shirely and Laverne what they think about variance.<jupyter_code>stat, p = stats.levene(junkdf['ANNUAL.VOLUME'],newjunkdf['ANNUAL.VOLUME'])
print('statistic=%.3f, p-value at rejection =%.3f' % (stat, p))
if p > 0.05:
    print('Probably homoscedastic')
else:
    print('Probably heteroscedastic')<jupyter_output>statistic=0.907, p-value at rejection =0.343
Probably homoscedastic<jupyter_text>So we have non-static variance, from the plots it looks to be increasing with respect to time, a quick check<jupyter_code>oldvar = (junkdf['ANNUAL.VOLUME'].std())**2
newvar = (newjunkdf['ANNUAL.VOLUME'].std())**2
print('Variance first 22 years =%.3e' % oldvar)
print('Variance last 86 years =%.3e' % newvar)<jupyter_output>Variance first 22 years =1.255e+13
Variance last 86 years =1.944e+13<jupyter_text>So the variance increased in the more reçent years, lets check means<jupyter_code>oldmean = (junkdf['ANNUAL.VOLUME'].mean())
newmean = (newjunkdf['ANNUAL.VOLUME'].mean())
print('Mean first 22 years =%.3e' % oldmean)
print('Mean last 86 years =%.3e' % newmean)<jupyter_output>Mean first 22 years =1.771e+07
Mean last 86 years =1.432e+07<jupyter_text>So there is evidence that the early years were wetter, by about 3 million acre feet.  Lets examine if the difference is significant using our hypothesis testing tools.<jupyter_code>stat, p = mannwhitneyu(junkdf['ANNUAL.VOLUME'],newjunkdf['ANNUAL.VOLUME'])
print('statistic=%.3f, p-value at rejection =%.3f' % (stat, p))
if p > 0.05:
    print('Probably the same distribution')
else:
    print('Probably different distributions')<jupyter_output>statistic=453.000, p-value at rejection =0.001
Probably different distributions<jupyter_text>Based on the Mann-Whitney test the statement in Cadillac Desert is defendible, the early years are indeed different (wetter) with a small p-value, so its  likely the difference is not random. ExerciseIf you were to rewrite the Compact provisions today, assuming that the originalintent was to divide the water between upper and lower basin in a 46% 54% split,what annual allocations would you recomend as ’firm’ water at 90% reliability?  Problem 5: Watershed Response Metrics (20 points) Background Rainfall-Runoff response prediction is a vital step in engineering design for mitigating flood-induced infrastructure failure.   One easy to measure characteristic of a watershed is its drainage area.  Harder to quantify are its characteristic response time, and its conversion (of precipitation into runoff) factor. Study DatabaseThe [watersheds.csv](http://54.243.252.9/engr-1330-webroot/4-Databases/watersheds.csv) dataset contains (measured)drainage area for 92 study watersheds in Texas (cite source), and the accompanying data:|Columns|Info.||:---|:---||STATION_ID |USGS HUC-8 Station ID code||TDA |Total drainage area (sq. miles) ||RCOEF|Runoff Ratio (Runoff Depth/Precipitation Depth)||TPEAK|Characteristic Time (minutes)||FPEAK|Peaking factor (same as NRCS factor)||QP_OBS|Observed peak discharge (measured)||QP_MOD|Modeled peak discharge (modeled)|  Assess the data:Using the following steps, assess the utility of the response metric.  **Step 1:** Read the "watersheds.csv" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe.<jupyter_code># import packages
# read data file
# summarize contents + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 2:** Using your laboratory scripts as a guide, assess the utility of **TDA** as a predictor of **TPEAK** (e.g. perform a regression and produce plots, with **TDA** on the x-axis)<jupyter_code># extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 3:**Using your laboratory scripts as a guide, assess the utility of **log(TDA)** as a predictor of **TPEAK** (e.g. perform a regression and produce plots, with **log(TDA)** on the x-axis)Which mapping of **TDA**  (arithmetic or logarithmic) produces a more useful graph?<jupyter_code># extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 4:**Using your laboratory scripts as a guide, assess the performance of the underlying (not disclosed) model, using **QP_MOD** as a predictor of **QP_OBS** (e.g. perform a regression and produce plots, with **QP_MOD** on the x-axis).Include an equal value line (x=y) on the plot and make an assertion of model bias (does the model overpredict or underpredict the observations?)<jupyter_code># extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# + markdown cell as needed<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07307974152288321, 'fraction_numerical': 0.027846246086203452, 'mean_word_length': 5.448453608247423, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 72, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '227336', 'license': 'CC0-1.0', 'n_tokens_mistral': 4565, 'n_tokens_neox': 4092, 'n_words': 1928, 'path': '5-ExamProblems/Exam3/fall2020/.ipynb_checkpoints/Exam3-2020-3-Draft-Copy1-checkpoint.ipynb', 'repo_name': 'dustykat/engr-1330-psuedo-course', 'size': '15011'}