<jupyter_start><jupyter_text>Homework 6  Problem 0It is highly recommended that you work with your group to fully complete the recent Discussion assignments related to the Palmer Penguins data set, as these will directly help with your project.   Problem 1Gapminder is a foundation, based on Sweden, that aims to enhance basic awareness of basic facts about the socioeconomic global world. As part of their efforts, they collect detailed statistics on life expectancy, population, and GDP, sometimes going back over many years. Here, we'll work with an excerpt of the Gapminder data. This excerpt has been packaged up and made available via <NAME>'s [`gapminder` repo](https://github.com/jennybc/gapminder) on Github. Run the code below to retrieve the data and take a look. As usual, you can also directly download the data by pasting the url into your browswer, saving the file, and reading it in locally via `pandas.read_csv`.<jupyter_code>import pandas as pd
        
url = "https://philchodrow.github.io/PIC16A/datasets/gapminder.csv"
gapminder = pd.read_csv(url)
gapminder<jupyter_output><empty_output><jupyter_text>Use the `gapminder` data to create the folowing visualization. Here, each trendline corresponds to a distinct country, which in turn lies on the stated continent.     You should achieve this result **without for-loops**, and also without manually creating the plot on each axis. You may find it useful to define additional data structures, such as dictionaries, that assign colors or axis indices to each continent. Feel free to modify aesthetic details of the plots, such as the colors. **Hint**: Try `df.groupby(["country"]).apply(f)`, where `f` is some very smart function that you have figured out.<jupyter_code># your solution here<jupyter_output><empty_output><jupyter_text>Problem 2In this problem, you will create several attractive visualizations of the `gapminder` data set using the [Seaborn package](https://seaborn.pydata.org/). Seaborn is a high-level interface to Matplotlib specially designed for working with Pandas data frames. Seaborn makes it relatively easy to create complex, multifaceted data graphics. The drawback is that it can sometimes be more difficult to exercise control over figure details, such as axis labels or figure size.Part of the learning objective for this problem is for you to **practice learning from documentation and examples**. I have given you some pointers in each problem, but it will be up to you to fill in the blanks. Import the `seaborn` package with the line `import seaborn as sns`. Then, run the code block below to create a scatterplot of `gdpPercap` (gross domestic product per capita) against `lifeExp` (life expectancy), with different colors for each continent.<jupyter_code>from matplotlib import pyplot as plt
import seaborn as sns

fgrid = sns.relplot(x = "gdpPercap", 
                    y = "lifeExp", 
                    hue = "continent", 
                    data = gapminder)<jupyter_output><empty_output><jupyter_text>The result of the above code is that `fgrid` is now a `FacetGrid` object (we'll see why it's called that in a minute). However, under the hood, we are still dealing with Matplotlib. To retrieve the individual plotting axis, we can access it from the `fgrid.axes` attribute, which is a 2d Numpy array containing the axes. For example, to make the horizontal axis logarithmic and labels, we can do this:<jupyter_code>fgrid = sns.relplot(x = "gdpPercap", 
                    y = "lifeExp", 
                    hue = "continent", 
                    data = gapminder)

fgrid.axes[0][0].semilogx()

labels = fgrid.axes[0][0].set(xlabel = "GDP per capita", 
                              ylabel = "Life Expectancy",
                              title  = "Impact of Economic Output on Life Expectancy")<jupyter_output><empty_output><jupyter_text>Not bad! Note that seaborn has effectively looped over the different continents, without us needing to use `for`-loops or `apply`.   Part ACreate the following plot using Seaborn:   <img src="https://philchodrow.github.io/PIC16A/homework/seaborn-example-1.png" alt="" width="800px">In this figure, each point represents a country, and the size of the point corresponds to the population. In this part and the next, it is not necessary that your plot *precisely* match mine -- feel free to modify the sizes of labels, change the color scheme, etc.  Hints- You should use [`sns.relplot`](https://seaborn.pydata.org/generated/seaborn.relplot.html) to create the figure. - Look in the [Seaborn Gallery](https://seaborn.pydata.org/examples/index.html) for examples illustrating how to create a multi-panel figure without using `plt.subplots()`. - The size of the figure is controlled by the `height` and `aspect` ratio arguments to `relplot`. I used `height = 2` and `aspect = 1.3`, but other choices are also fine. - When using column wrapping to create the `3 x 4` axis array, the resulting `fgrid.axes` array is no longer 2-dimensional, but 1-dimensional.   Part BCreate the following plot using Seaborn:  <img src="https://philchodrow.github.io/PIC16A/homework/seaborn-example-2.png" alt="" width="500px"> Hints- The easiest way is to again use `relplot`. This function defaults to producing scatterplots, but if you check the [gallery](https://seaborn.pydata.org/examples/index.html) you'll find examples of how to specify keywords that create line charts instead.   Part CCreate the following (very long) plot using Seaborn:  <img src="https://philchodrow.github.io/PIC16A/homework/seaborn-example-3.png" alt="" width="300px"> Hints- Use `sns.barplot`. - Pick a single year of the `gapminder` data frame to show. In the example, I showed data from 2002 only. - To sort the bars in ascending order by continent, you should sort the data frame that you pass to `sns.barplot`. - The argument `dodge = False` is a helpful keyword that will keep your bars centered. - I used a different color palette just to shake things up a little.   Part DCreate the one last plot using Seaborn:  <img src="https://philchodrow.github.io/PIC16A/homework/seaborn-example-4.png" alt="" width="500px"> Hints- This is a boxplot. - You should again pick a single year of the `gapminder` data frame to show. In the example, I show data from 2002 only.   Problem 3In our first lecture on machine learning, we did linear regression "by hand." In this problem, we will similarly perform logistic regression "by hand." This homework problem is closely parallel to the lecture, and so you might want to have the [notes](https://nbviewer.jupyter.org/github/PhilChodrow/PIC16A/blob/master/content/ML/ML_2.ipynb) handy.  Although logistic regression is a relatively simple model, it is a foundation for many modern deep neural nets. Additioanlly, logistic regression itself is still often used in applied contexts even now. &quot;When we raise money it’s AI, when we hire it&39;s machine learning, and when we do the work it&39;s logistic regression.&quot;(I&39;m not sure who came up with this but it&39;s a gem 💎)&mdash; Dr. <NAME> (@daniela_witten) September 26, 2019 Whereas linear regression is often used to measure arbitrary quantities like GDP or happiness scores, logistic regression is primarily used to estimate *probabilities*. For example, we might use logistic regression to estimate the probability of a passenger surviving the Titanic crash, a borrower defaulting on a loan, or an email being spam.For concreteness, let's say that we are considering the latter case. Suppose that we wish to model the probability that an email is spam as a function of the proportion of flag words (like "investment", "capital", "bank", "account", etc.) in the email's body text. Call this proportion $x$. $x$ is then a variable between $0$ and $1$. In logistic regression, we suppose that the probability $p$ that an email is spam has the form $$p = \frac{1}{1+e^{-ax - b}}\;,$$where $a$ and $b$ are again parameters. Let's see how this looks.<jupyter_code># run this block

import numpy as np
from matplotlib import pyplot as plt

n_points = 1000

a = 10
b = -5

x = np.sort(np.random.rand(n_points))
p = 1/(1+np.exp(-a*x - b))

fig, ax = plt.subplots(1)
ax.plot(x, p, color = "black")<jupyter_output><empty_output><jupyter_text>As usual, in practice we don't have access to the true function telling us the probability that an email is spam. Instead, we have access to data telling us whether or not the email really IS spam. We can model this situation by flipping a biased coin for each email, with the probability of heads determined by the modeled probability.<jupyter_code># run this block
y = 1.0*(np.random.rand(n_points) < p)<jupyter_output><empty_output><jupyter_text>A value of 1 indicates that the email is indeed spam, while a value of 0 indicates that the email is not spam.<jupyter_code># run this block
ax.scatter(x, y,  alpha = 0.5)
fig<jupyter_output><empty_output><jupyter_text>Notice that there are more spam emails where the model gives a high probability, and fewer where the model gives a lower probability. However, there may be some non-spam emails with even high probability -- sometimes we get legitimate emails about bank accounts, investments, etc.  Of course, we don't have access to the true model, so our practical situation looks more like this:<jupyter_code># run this block
fig, ax = plt.subplots(1)
ax.scatter(x, y, alpha = 0.5)<jupyter_output><empty_output><jupyter_text>We would like to use logistic regression to try to recover something close to the true model.   Part AWrite the model function `f`. The arguments of `f` should be the predictor variables `x` and the parameters `a` and `b`. The output of `f` should be the spam probabilities under the logistic model (see equation above) for these data. Use `numpy` tools, without `for`-loops. If you scan the above code carefully, you'll observe that most of this code is already written for you. This is a simple function, but **please add a short docstring indicating** what kinds of input it accepts and how to interpret the output. Comments are necessary only if your function body exceeds one line.<jupyter_code># your solution<jupyter_output><empty_output><jupyter_text>Part BPlot 10 candidate models against the data, using randomly chosen values of `a` between 5 and 15 and randomly chosen values of `b` between -2.5 and -7.5. Your plot should resemble in certain respects the third plot in [these lecture notes](https://nbviewer.jupyter.org/github/PhilChodrow/PIC16A/blob/master/content/ML/ML_2.ipynb). Comments are not necessary in this part.<jupyter_code># your solution here<jupyter_output><empty_output><jupyter_text>Part CThe *loss function* most commonly used in logistic regression is the *negative cross-entropy*. The negative cross-entropy of the `i`th observation is $$-\left[y_i \log \hat{p}_i + (1-y_i)\log(1-\hat{p}_i)\right]$$where $y_i \in \{0,1\}$ is the `i`th entry of the target data and $\hat{p}_i$ is the model's estimated probability that $y_i = 1$. The negative cross-entropy of the entire data set is the sum of the negative cross-entropies for each individual observation. Write a function that computes the negative cross entropy as a function of `x`, `y`, `a`, and `b`. This can be done in no more than two lines using `numpy`, without `for`-loops. Don't forget which logarithm is \BestLogarithm.  As in Part B, please write a short docstring describing what your function does and what inputs it accepts. Comments are necessary only if your function body exceeds two lines.<jupyter_code># your solution here<jupyter_output><empty_output><jupyter_text>Part DOn a single axis, plot 100 distinct models (using the code that you wrote in) in Part B. Highlight the one with the lowest negative cross entropy in a different color -- say, red. Compare the best values of `a` and `b` that you found to the true values, which were `a = 10` and `b = -5`. Are you close? The plot you produce should resemble, in some respects, the fifth plot in [these lecture notes](https://nbviewer.jupyter.org/github/PhilChodrow/PIC16A/blob/master/content/ML/ML_2.ipynb). It is not necessary to write comments in this part.<jupyter_code># your solution here<jupyter_output><empty_output><jupyter_text>It is not required to use `scipy.optimize` to more accurately estimate `a` and `b` for this homework assignment, but you are  free to do so if you wish. You may then use the optimal estimates in the following part.   Part EIn classification tasks, we evaluate not just the standard loss function, but also the *accuracy* -- how often does the model correctly classify the data? Let's say that the model classifies an email as spam according to the following rule: 1. If $\hat{p}_i$ (the model probability plotted above) is larger than $c$, classify the email as spam. 2. If $\hat{p}_i$ is less than or equal to $c$, classify the email as not-spam. Write a function called `positive_rates` which accepts the following arguments: 1. The data, `x` and `y`. 2. The best parameters `best_a` and `best_b`. 3. A threshold `c` between 0 and 1. This function should output two numbers. The first of these is *false positive rate*: the proportion of non-spam emails that the model incorrectly labels as spam. The second is the *true positive rate*: the proportion of spam emails that the model correctly labels as spam. For example: ```python positive_rates(x, y, best_a, best_b, c = 0.5)``````(0.05058365758754864, 0.7469135802469136)```**Note**: due to randomization, your numerical output may be  different. Please write a descriptive docstring for your function. Comments are necessary only if your function body exceeds five lines.<jupyter_code># your solution here

# demonstrate your function here<jupyter_output><empty_output><jupyter_text>Part FPlot the *receiver operating characteristic* (ROC) curve for the logistic model with parameters `best_a` and `best_b`. The ROC curve is the plot of the `false_positive` rate (on the horizontal axis) against the `true_positive` rate (on the vertical axis) as the threshold `c` is allowed to vary. Additionally, plot a diagonal line ("the line of equality") between the points (0,0) and (1,1). Your ROC curve should lie noticeably above the line of equality. Plot your curves in different colors and add a legend to help your reader understand the plot. It is ok to use `for`-loops and list comprehensions in this part. Your result should look something like this, although it won't be exactly the same due to randomness.<jupyter_code># plot code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07737768004398021, 'fraction_numerical': 0.011338647608576141, 'mean_word_length': 4.51877133105802, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 63, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '386398', 'license': 'MIT', 'n_tokens_mistral': 4199, 'n_tokens_neox': 3937, 'n_words': 2013, 'path': 'homework/HW6.ipynb', 'repo_name': 'mm5110/PIC16A', 'size': '14565'}