<jupyter_start><jupyter_text>Autograding Lessons LearnedSo I was able to implement some of what we leaned this semester in a notebook with tests that autogrades and outputs to a csv. I didn’t give them the test code.    Here is how it worked.  Students got these instructions:  *******************************   Final ExamThe goal of this is to submit the assignment both on the LMS and through github.  Both are secure means in case there are any issues.  For Github Accept Assignment[https://classroom.github.com/a/OlYXZLTw](https://classroom.github.com/a/OlYXZLTw)The link above will create a repository. 1. Go to [Collaboratory](https://colab.research.google.com) and click on git. 2. Enter your repository (mine is of the form techfundamentals-fall2018/final-starter-2019-jkuruzovich and click `Include private repos`.3. It will show the notebook `final_2019_student.ipynb`. 4. Use File->Save a Copy on Github5. When finished, download the .ipynb and upload it to the LMS under final part2.  Launch Notebook (Backup use only)If you cannot get it working with github, here is a backup.     *******************************  For all but 2 of the students, I was able to get them to clone repository and save to github.  2 students had repo clones that hung.    Travis-CI jobs were automatically scheduled after commits. I hadn't fully figured out grading at that time though.  Also, in retrospect, many (about half) of students submitted notebooks that had an error and thus wouldn't have been converted.    For most students I was able to get them to save to github.   I was able to download all notebooks from LMS, paste in my test code, and then run it.  My grading code (same as above) outputs results.  I was able to used the failed tests details to understand common patterns that students had (they failed to create dummies successfully)  To Do /Issues- TRAVIS-CI Grading requires conversion to .py. How to force students to commit an error free .py file.  Many entries would have never gotten to gradeing. - *I believe the nbconvert tool run by Travis-CI executes the ipython notebook in the native .ipynb format and converts the executed result to a markdown file -- Stephan*- TRAVIS-CI Should run tests on https://github.com/googlecolab/backend-container. That would take care of dependency issues between colab and Travis.  Though any local installs that the students do would not be installed in colab. Maybe standardize on a requirements.txt file in the tests folder?  - *I think it would be a smart idea to include a requirements.txt file in the repo which is run as part of the Travis-CI build.  Students could add package references to this for any 3rd party packages they introduce.  Additionally, instructions could be provided for how to install a library from within a notebook itself (see [this blog](https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/) for examples) -- Stephan*- Give them tests or not? How to grade B if it depends on A and they miss A. - Could commit grade.csv back to github. However, this could easily be altered by student. https://gist.github.com/willprice/e07efd73fb7f13f917ea - *I really like the idea of submitting the grades to some centralized service, but creating and running a custom service would be a very heavyweight solution and require the deployment of an authT/Z solution to ensure grade reliability.  Perhaps a better solution would be to continue to pursue integration with the university LMS?  If Travis-CI could update blackboard with a per-student assignment score then I think we would have the best overall solution -- Stephan*- When someone saves to github there is a likelyhood the file name will change.  Colab creates a "Copy of"  How to handle?  - *I think the simplest solution is to include explicit instructions on the expected notebook filename and how to rename the file generated by Colab in github.  Alternatively, the script that executes the notebook for grading could use wildcards -- Stephan*- In an exam/hm situation, students should be instructed to save the intermediate file in github. This creates a "copy_of" extension.-  Overall, it was painful to work through all this as an instructor. I think there could be something to said for automatically generating tests from a solution notebook.Full notebook with testing is below.  Tech Fundamentals Final enter your name here"I pledge under the RPI honor code that I have completed this work on my own." At any time a monitor may ask you to scroll up to the top of this document to view this.<jupyter_code>name='enter your name'
rcsid='rcsid'
#This just gets the data and preps the environment. 
data="https://github.com/jkuruzovich/final-starter-2019/raw/master/data.zip"
!wget $data && unzip -o data.zip<jupyter_output>--2019-06-25 15:54:41--  https://github.com/jkuruzovich/final-starter-2019/raw/master/data.zip
Resolving github.com (github.com)... 192.168.127.12
Connecting to github.com (github.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/jkuruzovich/final-starter-2019/master/data.zip [following]
--2019-06-25 15:54:41--  https://raw.githubusercontent.com/jkuruzovich/final-starter-2019/master/data.zip
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.3.11
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33651029 (32M) [application/zip]
Saving to: ‘data.zip’

data.zip            100%[===================>]  32.09M  11.7MB/s    in 2.7s    

2019-06-25 15:54:44 (11.7 MB/s) - ‘data.zip’ saved [33651029/33651029]

Archive:  data.zip
   creating: data/
   creating: data/ma/
  inflating:[...]<jupyter_text>This is healthcare data generated from a syntetic data generator.  We want to see how good it is.<jupyter_code>#Let's list the files
!ls ./data/ma

#Let's look at one of the dataframes
import numpy as np
import pandas as pd
careplans_ma= pd.read_csv('./data/ma/careplans.csv')
careplans_ny= pd.read_csv('./data/ny/careplans.csv')
careplans_ma<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis The data directory includes a wide variety of data related to health care.      (1. 5 points) Find the mean of the `COST` varaiable in the `encounters.csv` data for both NY and MA, assigning the values to `cost_ny` and `cost_ma`.<jupyter_code># Answer 1
encounters_ma= pd.read_csv('./data/ma/encounters.csv')
encounters_ny = pd.read_csv('./data/ny/encounters.csv')
cost_ma=0 # Fix this to calculate the mean of the cost. (5 points)
cost_ny=0 #Fix this to calculate the mean of the cost. (5 points)
cost_ma=encounters_ma.COST.mean()
cost_ny=encounters_ny.COST.mean()
print( cost_ma, cost_ny)
print ("___", "    If you are seeing a face, probably aren't there yet. ")<jupyter_output>17.906257207081175 15.188246329742485
___     If you are seeing a face, probably aren't there yet.<jupyter_text>Cost Regression Analysis Dummy Creation and Splint to Train and Test for NY ONLY The goal of the code below is to predict the `COST` from `CODE` and `ENCOUNTERCLASS`.  (2. 5 points) First create dummies from `CODE` and `ENCOUNTERCLASS`.  Don't use any other variables.  split your data into an 70% train and 30% validation using a `random_state` of 111. For validation, set: ```splittest1 = X_train.iloc[5,2]splittest2 = X_test.iloc[7,4]```<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

test_size=0.3
cols=['CODE','ENCOUNTERCLASS']
X = pd.get_dummies(encounters_ny.loc[:,cols], columns=cols, dummy_na = True )
y = encounters_ny['COST']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=111)
#verify split
splittest1 = X_train.shape
splittest2 = X_test.iloc[7,4]
print(splittest1, splittest2)<jupyter_output>(72930, 40) 0<jupyter_text>Predict Cost via Linear Regression (3. 10 points) Using linear regression analysis, use the `CODE` and `ENCOUNTERCLASS` as independent variables to predict `COST`. This is your naive model. Report R-Squared for both training `r2_train_cost` and validation `r2_test_cost`.<jupyter_code>#3 Answer
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit( X_train, y_train )
r2_train_cost=0 # Make equal to the R2. 
r2_train_cost = lm.score( X_train, y_train )
r2_test_cost = lm.score(X_test, y_test)
print('R2 for Train:', r2_train_cost)
print('R2 for Test (cross validation)：', r2_test_cost )<jupyter_output>R2 for Train: 0.6964659105321847
R2 for Test (cross validation)： 0.6984160299386837<jupyter_text>Creat Graph of Encounter class vs Cost(4. 10 points) Create a graph of encounter class vs cost. It should look like the graph below (created via seaborn).<jupyter_code>%matplotlib inline
#4 Answer
import seaborn as sns
dev = sns.barplot(y="ENCOUNTERCLASS", x="COST", data=encounters_ny)
pic =dev.get_figure()<jupyter_output>/anaconda3/lib/python3.7/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>Create this graph. ENCOUNTERCLASS is on they axis and  and COST on X axis.![Imgur](https://i.imgur.com/asCDZSM.png)  High Cost Patients Classification A second Challenge is determinine the most expensive individuals.  This is set as all those groups that are more than 1 standard deviation above the mean, indicated as 'HIGHCOST' in the dataset.  (5. 5 points) Overall, count the total number of individuals who are high cost in NY (i.e., where encounters_ny['HIGHCOST'] is equal to 1.) Assign to the value `total_high_cost_ny`. Overall, count the total number of individuals who are high cost in MA (i.e., where encounters_ny['HIGHCOST'] is equal to 1.) Assign to the value `total_high_cost_ma`.<jupyter_code>#5 Answer 
total_high_cost_ny=encounters_ny['HIGHCOST'].sum()
total_high_cost_ma=encounters_ma['HIGHCOST'].sum()

print(total_high_cost_ny, total_high_cost_ma)<jupyter_output>21558 22505<jupyter_text>Split the Train and test set for Classification with you DV set to HIGHCOST for NY Only. (6. 5 points) First split your data into an 70% train and 30% validation. Make sure that the results are *stratified* (equal classes in train and test) with `random_state = 111`.  Also set the following:```splittest3 = y_train.iloc[5]splittest4 = y_test.iloc[3]```<jupyter_code>#6 Answer 
#Set y equal to train encounters_ny['HIGHCOST'] 
y = encounters_ny['HIGHCOST']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=111,stratify=y)

#This will be used for testing. 
splittest3 = y_train.iloc[3]
splittest4 = y_test.iloc[17]
print (splittest3, splittest4)<jupyter_output>1 1<jupyter_text>Classification with Random Forrest(8. 5 points) Use a RandomForestClassifier to predict the 'HIGH_COST' with use the `CODE` and `ENCOUNTERCLASS` as independent variables.  Also set the following:```train_accuracytest_accuracy```<jupyter_code>from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier()
#This fits the model object to the data.
classifier.fit(X_train,  y_train)
train_accuracy = classifier.score(X_train, y_train)
test_accuracy = classifier.score( X_test,  y_test)
#This creates the prediction. 
print("Classifier Accuracy for Train: ", train_accuracy )
print("Classifier Accuracy for Test: ", test_accuracy )<jupyter_output>/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
  "10 in version 0.20 to 100 in 0.22.", FutureWarning)<jupyter_text>SubmissionFor the final submission, please submit a link to this notebook.<jupyter_code>#This just gets the data and preps the environment. 
tests="https://github.com/jkuruzovich/final-starter-2019/raw/master/tests.zip"
ok="https://github.com/jkuruzovich/final-starter-2019/blob/master/final.ok"
!wget $tests && unzip -o tests.zip && wget $ok
!pip uninstall okgrade -y && pip install git+https://github.com/data-8/Gofer-Grader  

from client.api.notebook import Notebook
import grade
import importlib
importlib.reload(grade)
ok = Notebook('final.ok')
_ = ok.auth(inline=True)

name="final"
points_per_test=2.5
comments=""

grade.grade(name, points_per_test, comments, ok)
%%capture  
#This capture command supresses output. 

#***********
#Add the manually graded ones
q4=5
comments=""
points_per_test=2.5

#***********


from client.api.notebook import Notebook
ok = Notebook('final.ok')
_ = ok.auth(inline=False)
from bs4 import BeautifulSoup
import pandas as pd
import os

#Grade Results
results= {q[:-3]:ok.grade(q[:-3]) for q in os.listdir("tests") if q.startswith('q')}

#If running locally with lots of notebooks load the grades. 
df = pd.DataFrame()
row=df.shape[0]
df.loc[row,'student']=name  #This is set in the last.
df.loc[row,'rcsid']=rcsid   #This is set in the last. 
total_grade=0
#This loops through the results
for key, val in results.items(): 
    df.loc[row,key]=val.grade
    results_key=str(key)+"-failed"
    df.loc[row,key]=val.grade*points_per_test
    #We use beautiful soup to parse the tests. 
    soup = BeautifulSoup(str(val.failed_tests), "lxml")
    #There are multiple components, but the expected data seems most valuable. 
    got = soup.get_text().split('\\n')[16:20]
    df.loc[row,results_key]=str(got)
    total_grade+=df.loc[row,key]  #total grade
df.loc[row, 'q4']=q4
total_grade+=q4
df.loc[row,'total_grade']=total_grade
df.loc[row,'comments']=comments

if not os.path.isfile('grades.csv'):
   df.to_csv('grades.csv', index=False)
else: # else it exists so append without writing the header
   df.to_csv('grades.csv', mode='a', header=False,index=False)


!cat grades
cols=df.columns
row=df.shape[0]-1
for x in range(df.shape[1]):
  print(cols[x]," = ",  df.iloc[row,x])
  
#See All Records
df= pd.read_csv('grades.csv')
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10055434706336397, 'fraction_numerical': 0.03038383271349379, 'mean_word_length': 4.925987525987526, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '580284', 'license': 'MIT', 'n_tokens_mistral': 4768, 'n_tokens_neox': 4424, 'n_words': 1658, 'path': 'Autograding_example.ipynb', 'repo_name': 'RPI-DATA/summer_2019', 'size': '14257'}