<jupyter_start><jupyter_text>1. System Profile --- Problem 2: Water Quality Comparison (20 points) BackgroundCountry Club Bayou, formerly <NAME>, is located in southeast Houston.The bayou drains from east to west connecting to Brays Bayou. The upper portion of thebayou is conveyed in a concrete channel that was initially placed in the early 1900’s. Thelower portion of the bayou from the Hughes Street railroad bridge to the confluence withBraes Bayou is open, unlined channel.Pollution of the bayou has been problematic for at least a dozen years. Currentlysuspected high nutrient loading somewhere in the covered portion of the bayoucontributes to observed low dissolved oxygen values, a septic `odor`, and septic (black)color. The out-fall from the covered portion of the bayou to the open portion is justupstream of the Hughes Street Bridge. Samples collected at the bridge by the City ofHouston Health Department confirm these historical observations.At times the water at the out-fall just upstream of the Hughes Street Bridge has not meetstate water quality standards for unclassified waters. Unclassified waters are waterswhich are not specifically listed in Appendices A or D of §307.10 of Title:30, Part 1,Chapter 307 of the Texas Administrative Code. The table lists some of the relevant standards. Parameter Value Remarks    Dissolved Oxygen 2.0 mg/L - 24 hr. average                               1.5 mg/L - absolute minimum                             3.0 mg/L - proposed (circa 1998)                Sulfate  65 mg/L - proposed (circa 1998)                     pH  6.5-9.0 - proposed (circa 1998)         Fecal Coliform                    200 cfu/100mL  Contact recreation                                        2000 cfu/100mL  Non-contact recreation          Temperature 4oF above ambient Fall, Winter, Spring                     1.5oF above ambient Summer    These values are proposed for Segment 1014 (Buffalo Bayou above tidal) for contactrecreation and limited aquatic life use. See: [http://www.tnrcc.state.tx.us/water/quality/standards/revisions.html](http://www.tnrcc.state.tx.us/water/quality/standards/revisions.html)The figure below is a map of the area with storm and sanitary sewer systems drawn onto the map (the study pre-dates affordable GIS)![figure 1](CCBayoyMap.png)The current land-use in the area ranges includes residential, light-industrial, and severalmanufacturing facilities. The covered portion of the storm sewer system is ownedby the City of Houston, while the open portion appears to be privately owned except forthe portion through Wortham Park (COH). There are no known discharge permits issuedby any authority for discharge into Country Club Bayou.The figure below shows sample locations were water quality samples were collected![figure 1](SampleLocations.png)For this example, the data from the Hughes Street location are examined.  The data are stored in the file named HUGHES_STREET.txt and look like:    MM	DD	YY	pH	T_degC	DO_ppm	NH3_ppm	Odor    11	1	1999	6.63	22.5	4.54	0.63	0    5	28	1998	7.3	23.2	1.7	0.54	0    6	3	1998	7.2	24.1	1.2	0.31	0    6	8	1998	7.1	24.8	1	3.96	0    6	15	1998	7.2	25	2.9	0.57	0    6	26	1998	7.3	27.2	3.1	1.29	0    6	29	1998	7.7	26.2	7.9	0.28	0    7	8	1998	6.5	26.2	0.28	0.21	0    7	20	1998	6.5	28	8.1	0.32	0    ...    7	15	1999	7.08	27.5	3.19	0.52	0    5	12	1998	7	21.6	1.41	0.66	1    5	21	1998	7.2	23.9	1.2	1.23	1    5	27	1998	6.9	23.6	1.1	0.36	1    6	11	1998	7	24.9	1.7	2.5	1    ... Problem StatementDetermine if the mean value of **pH**,**T_degC**,**DO_ppm**, and **NH3_ppm** are significantly different when the data are separated using the Odor/No Odor code. The data are located at [http://54.243.252.9/engr-1330-webroot/4-Databases/HUGHES_STREET.txt](http://54.243.252.9/engr-1330-webroot/4-Databases/HUGHES_STREET.txt)  **Step 1:** Read the "HUGHES_STREET.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe.<jupyter_code># downloadload datafile
# read dataframe and describe contents<jupyter_output><empty_output><jupyter_text>**Step 2:** Split into two databases, No Odor and Odor<jupyter_code>#<jupyter_output><empty_output><jupyter_text>**Step 3:** Describe the resulting data series<jupyter_code># No Odor
# DO, Temp, NH3, pH - mean, variance<jupyter_output><empty_output><jupyter_text>**Step 4:** Perform appropriate hypothesis tests on the data to determine if the water quality data are different in Odor/No Odor cases.<jupyter_code>#<jupyter_output><empty_output><jupyter_text>References<NAME>., <NAME>., <NAME>., <NAME>., <NAME>. 2000. ``Investigation and Demonstration of Intervention Strategies to Improve Water Quality on Country Club Bayou.'' Final Report to Houston Wastewater Program, Texas Natural Resources Conservation Commission, City of Houston, and the Environmental Institute of Houston [https://172.16.58.3/documents/about-me/MyWebPapers/project_reports/ccbayou_report/](https://172.16.58.3/documents/about-me/MyWebPapers/project_reports/ccbayou_report/) --- Problem 3: Treaty Terms for the Colorado River (20 points) BackgroundThe Colorado river originates in the Rocky mountains and has contributing tributariesand/or flows through the states of Wyoming, Colorado, Utah, Nevada, Arizona,and California. Prior to dams the river used to actually run to the ocean and at onetime in history actually had commerce by river boats. In the late 1800’s and early 1900’s the river and its tributaries were dammed and diverted. Around the 1920’sa legal document called the Colorado River Compact was developed. A Compact is similar to a treaty, except the parties to the treaty are states (essentially the individual states have entered into the treaty with the United States and Mexico). The Colorado River Compact divides the Colorado River into Upper and Lower Basins with the division being at <NAME> on the Colorado River one mile below the Paria River in Arizona.The Lower Basin states are Arizona, California, and Nevada, with small portions of New Mexico and Utah that are tributary to the Colorado River below Lee Ferry. The Upper Basin states are Colorado, New Mexico, Utah, and Wyoming, with a small portion of Arizona tributary to the Colorado River above Lee Ferry.Article III of the Compact apportions the waters of the Colorado River to the Upperand Lower Basins as follows:The Compact apportions the right to exclusive beneficial consumptive use of 7.5million acre-feet of water per year from the ”Colorado River System” in perpetuityto the Upper Basin and the Lower Basin.The Compact allows an additional 1.0 million acre-feet per year of increased beneficialconsumptive use to the Lower Basin.It provides water for Mexico pursuant to treaty. Water must first come from anysurplus over the waters allocated to the states in Article III(a) and (b). If thatsurplus is insufficient, then the burden of that deficiency shall be shared equally bythe Upper and Lower Basins.The Compact provides that the Upper Basin states will not cause the flow of theriver at <NAME>, Arizona to be depleted below an aggregate of 75 million acrefeetfor any period of ten consecutive years beginning with the ratification of theCompact.It provides that the Upper Basin states will not withhold water and the states of theLower Basin shall not require delivery of water which cannot reasonably be appliedto domestic and agricultural uses.  Problem StatementThe file named [colorado.txt](http://54.243.252.9/engr-1330-webroot/4-Databases/colorado.txt) contains annual natural flows volumes in the upperColorado river basin from 1906 to present. These flows are historical reconstructionat approximately the same location as the current USGS gage 09380000. Thelocation is approximately above (upstream of) Lee Ferry, hence the data representthe flows out of the upper basin.The data are in two columns, the first is the water year, the second column iscumulative discharge for the year in units of acre-feet.    YEAR ANNUAL.VOLUME    1906 18550021    1907 21201694    1908 12218817    1909 22356301    1910 14650616    ...Prepare an analysis of the data to support or refute the provisions of the Compact.You can assume that the data from 1906 to 1927 (the original Compact was ratifiedin 1922, many additional Compacts have been signed since that time, all trying to reapportionwater in the river basin witout violating the original agreement) were available to theauthors of the Colorado River Compact, analyze this period of data and decide if theallocations to the lower basin are realistic (8.5 million acre-feet-per year) assumingthe upper basin allocation is removed. Consumptive use means that the water willnot return to the basin. The data are ’natural’ flows and the upper basin allocationsare not removed in the data.   **Step 1:** Read the "colorado.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe.<jupyter_code># download the data file
# read into a dataframe
# plot
# describe the dataframe<jupyter_output><empty_output><jupyter_text>**Step 2:** Evaluate if the mean value in the pre-compact period is substantially different from the post-compact period.<jupyter_code># split the data into pre-compact and post-compact
# describe the two subsets
# box-plot both
# test for normal
# hypothesis test for difference in means
# conclusion(s)<jupyter_output><empty_output><jupyter_text>**Step 3:** Estimate the probability based on the the 20-year period that the lower basin allocation will not be met if upper basin allocation is satisfied. Estimate the probability that the allocation will not be met using the entire data set if the upper basin allocation is satisfied.<jupyter_code># empirical probability approach on the 20 years pre-compact; full data<jupyter_output><empty_output><jupyter_text>**Step 4:** Next assume that lower basin is satisfied first (this is the historical result of theoriginal and subsequent treaties). Estimate the probability that the upper basinallocation will not be met. Determine an upper basin allocation that will be met95% of the time. Again use the 20 year initial record, then the entire record.<jupyter_code># empirical probability approach on the 20 years pre-compact; full data<jupyter_output><empty_output><jupyter_text>**Step 5:** Was the initial 20 year period sufficiently different from the later data that theCompact authors made decisions based on unusually high or low flows?<jupyter_code># make a decision and report<jupyter_output><empty_output><jupyter_text>Problem 4: Solids in Rivers  Problem 5: Watershed Response Metrics (20 points) Background Rainfall-Runoff response prediction is a vital step in engineering design for mitigating flood-induced infrastructure failure.   One easy to measure characteristic of a watershed is its drainage area.  Harder to quantify are its characteristic response time, and its conversion (of precipitation into runoff) factor. Study DatabaseThe [watersheds.csv](http://5172.16.17.32/engr-1330-webroot/4-Databases/watersheds.csv) dataset contains (measured)drainage area for 92 study watersheds in Texas (cite source), and the accompanying data:|Columns|Info.||:---|:---||STATION_ID |USGS HUC-8 Station ID code||TDA |Total drainage area (sq. miles) ||RCOEF|Runoff Ratio (Runoff Depth/Precipitation Depth)||TPEAK|Characteristic Time (minutes)||FPEAK|Peaking factor (same as NRCS factor)||QP_OBS|Observed peak discharge (measured)||QP_MOD|Modeled peak discharge (modeled)|  Assess the data:Using the following steps, assess the utility of the response metric.  **Step 1:** Read the "watersheds.csv" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe.<jupyter_code># import packages
# read data file
# summarize contents + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 2:** Using your laboratory scripts as a guide, assess the utility of **TDA** as a predictor of **TPEAK** (e.g. perform a regression and produce plots, with **TDA** on the x-axis)<jupyter_code># extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 3:**Using your laboratory scripts as a guide, assess the utility of **log(TDA)** as a predictor of **TPEAK** (e.g. perform a regression and produce plots, with **log(TDA)** on the x-axis)Which mapping of **TDA**  (arithmetic or logarithmic) produces a more useful graph?<jupyter_code># extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 4:**Using your laboratory scripts as a guide, assess the performance of the underlying (not disclosed) model, using **QP_MOD** as a predictor of **QP_OBS** (e.g. perform a regression and produce plots, with **QP_MOD** on the x-axis).Include an equal value line (x=y) on the plot and make an assertion of model bias (does the model overpredict or underpredict the observations?)<jupyter_code># extract lists to use Lab21 example scripts
# log transform as needed
# regression fits, include plots of data regression line, uncertainty lines

# + markdown cell as needed<jupyter_output><empty_output><jupyter_text>Problem 6: Currency Authentication (20 points)   Background The U.S. Secret Santas examined 1372 images of banknotes, and classified them as genuine or forgery. Data were extracted from images using an industrial camera for print inspection; The final images have 400x 400 pixels. Due to the object lens and distance to the investigated object gray-scale pictures with a resolution of about 660 dpi were gained. A Wavelet Transform model was used to extract features from images.  The resulting data model has 4 predictor variables (variance of image, skewness, kurtosis, entropy). The classification variable is encoded as 0 (authentic) or 1 (forgery).  Study DatabaseThe [data_banknote_authentication.txt](http://54.243.252.9/engr-1330-webroot/4-Databases/data_banknote_authentication.txt) dataset contains the data as 5 columns. Problem StatementBuild a classification engine (logistic regression) to classify a banknote image into authentic or forgery using a 70/30 training/testing split.Three banknotes are presented for authentication.  Determine which banknote images with with the following parameters are authentic or counterfeitBanknote 1:|Quality|Value||---|---||Variance|2.25040||Skewness|3.57570||Kurtosis|0.35273||Entropy|0.28360|Banknote 2: |Quality|Value||---|---||Variance|-1.3971||Skewness| 3.3191||Kurtosis|-1.3927||Entropy|-1.9948|Banknote 3: |Quality|Value||---|---||Variance|0.93574||Skewness|8.8865||Kurtosis|-1.68301||Entropy|-1.65999|  **Step 1:** Read the "data_banknote_authentication.txt" file as a dataframe. Explore the dataframe and in a markdown cell breifly describe the summarize the dataframe.<jupyter_code># import packages
# read data file
# summarize contents + markdown cell as needed<jupyter_output><empty_output><jupyter_text>**Step 2:** make a pairplot to visually examine if classification is feasible and identify outliers<jupyter_code># sns.pairplot ...
# comments<jupyter_output><empty_output><jupyter_text>**Step 3:**Build training and testing sets (use sklearn as guidance)<jupyter_code># inp_df = ...
# out_df = ...
# scaler = ...
# inp_df = ...
# X_train, X_test, y_train, y_test = ...<jupyter_output><empty_output><jupyter_text>**Step 4:**Fit the training set; examine accuracy from testing set<jupyter_code># from sklearn.linear_model import LogisticRegression
# clf = ...
# clf.fit(...)
# ...<jupyter_output><empty_output><jupyter_text>**Step 5:**Build the classification engine, determine classification of value(s) in problem statement<jupyter_code># 
# banknote 1 fake or real?
# banknote 2 fake or real?
# banknote 3 fake or real?
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
df = pd.read_csv('data_banknote_authentication.txt')
df.describe()
#df.columns =['Variance', 'Skewness', 'Kurtosis', 'Entropy','Class']
df.head()
sns.pairplot(df, hue='Class', height=2.5)
inp_df = df.drop(df.columns[[4]], axis=1)
out_df = df.drop(df.columns[[0,1,2,3]], axis=1)
#
scaler = StandardScaler()
inp_df = scaler.fit_transform(inp_df)
#
X_train, X_test, y_train, y_test = train_test_split(inp_df, out_df, test_size=0.2, random_state=42)
X_tr_arr = X_train
X_ts_arr = X_test
y_tr_arr = y_train.to_numpy()  
y_ts_arr = y_test.to_numpy()
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X_tr_arr, y_tr_arr)
print (clf.intercept_, clf.coef_)
pred = clf.predict(X_ts_arr)
print ('Accuracy from sk-learn: {0}'.format(clf.score(X_ts_arr, y_ts_arr)))
def mymodel(x1,x2,x3,x4,w1,w2,w3,w4,b):
    z = w1*x1 + w2*x2 +w3*x3 +w4*x4 +b #linear model to produce estimator to send to logistic fn
    yhat = sigmoid_activation(z)
    return(yhat)

df_fake = df[df["Class"]==1]
df_fake.head(30)
df_real = df[df["Class"]==0]
df_real.head(30)
sepal_l = 0.93584
sepal_w =  8.8855
petal_l = -1.68310
petal_w = -1.65990
myguess = mymodel(sepal_l,sepal_w,petal_l,petal_w,clf.coef_[0,0],clf.coef_[0,1],clf.coef_[0,2],clf.coef_[0,3],clf.intercept_[0])
print(myguess)
0.93584 	8.8855 	-1.68310 	-1.65990<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07591994103305551, 'fraction_numerical': 0.04241084084594886, 'mean_word_length': 4.819201583635763, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 7, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '249665', 'license': 'CC0-1.0', 'n_tokens_mistral': 5676, 'n_tokens_neox': 5047, 'n_words': 2192, 'path': '5-ExamProblems/Exam3/fall2020/.ipynb_checkpoints/Exam3-2020-3-Draft-Copy2-checkpoint.ipynb', 'repo_name': 'dustykat/engr-1330-psuedo-course', 'size': '17675'}