<jupyter_start><jupyter_text>Based on similar work with Twin Cities Pioneer Press [Schools that Work](http://www.twincities.com/2010/07/10/lessons-from-a-school-that-works/)<jupyter_code>import pandas as pd
import numpy as np 
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Setting things up Let's load the data and give it a quick look.<jupyter_code>df = pd.read_csv('data/apib12tx.csv')
df.describe()<jupyter_output>/home/ec2-user/anaconda3/lib/python3.5/site-packages/numpy/lib/function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile
  RuntimeWarning)<jupyter_text>Checking out correlations Let's start looking at how variables in our dataset relate to each other so we know what to expect when we start modeling.<jupyter_code>df.corr()<jupyter_output><empty_output><jupyter_text>The percentage of students enrolled in free/reduced-price lunch programs is often used as a proxy for poverty.<jupyter_code>df.plot(kind="scatter", x="MEALS", y="API12B")<jupyter_output><empty_output><jupyter_text>Conversely, the education level of a student's parents is often a good predictor of how well a student will do in school.<jupyter_code>df.plot(kind="scatter", x="AVG_ED", y="API12B")<jupyter_output><empty_output><jupyter_text>Running the regression Like we did last week, we'll use scikit-learn to run basic single-variable regressions. Let's start by looking at California's Academic Performance index as it relates to the percentage of students, per school, enrolled in free/reduced-price lunch programs.<jupyter_code>data = np.asarray(df[['API12B','MEALS']])
x, y = data[:, 1:], data[:, 0]
lr = LinearRegression() 
lr.fit(x, y)
# plot the linear regression line on the scatter plot
lr.coef_
lr.score(x, y)
plt.scatter(x, y, color='blue')
plt.plot(x, lr.predict(x), color='red', linewidth=1)<jupyter_output><empty_output><jupyter_text>In our naive universe where we're only paying attention to two variables -- academic performance and free/reduced lunch -- we can clearly see that some percentage of schools is overperforming the performance that would be expected of them, taking poverty out of the equation.A handful, in particular, seem to be dramatically overperforming. Let's look at them:<jupyter_code>df[(df['MEALS'] >= 80) & (df['API12B'] >= 900)]<jupyter_output><empty_output><jupyter_text>Let's look specifically at Solano Avenue Elementary, which has an API of 922 and 80 percent of students being in the free/reduced lunch program. If you were to use the above regression to predict how well Solano would do, it would look like this:<jupyter_code>lr.predict(80)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09805361733382299, 'fraction_numerical': 0.015056922511935366, 'mean_word_length': 6.5041322314049586, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '126802', 'license': 'MIT', 'n_tokens_mistral': 897, 'n_tokens_neox': 831, 'n_words': 284, 'path': '07 Teaching Machines/regression_review.ipynb', 'repo_name': 'barjacks/algorithms_mine', 'size': '2723'}