<jupyter_start><jupyter_text>Homework 2**For exercises in the week 20-25.11.19****Points: 6 + 1b**Please solve the problems at home and bring to class a [declaration form](http://ii.uni.wroc.pl/~jmi/Dydaktyka/misc/kupony-klasyczne.pdf) to indicate which problems you are willing to present on the blackboard.  Declartation| Exercise || 1 | 2 | 3 | 4 |  5  | 6 | 7 | 8 ||----------||---|---|---|---| --- |---|---|---|| Points   || 0 | 1 | 0 | 1 | 0.5 | 0 | 1 | 0 |  Problem 1 [1p]Let $(x^{(i)},y^{(i)})$ be a data sample with $x^{(i)}\in\mathbb{R}^D$, $y^{(i)}\in\mathbb{R}$. Let $\Theta \in\mathbb{R}^D$ a parameter vector.Find the closed form solution $\Theta^*$ to $$\min_\Theta \left(\frac{1}{2}\sum_i (\Theta^Tx^{(i)} - y^{(i)})^2 + \frac{\lambda}{2}\sum_{d=1}^D \Theta_d^2\right).$$ Ans:$\Theta = (X^T X+\lambda I)^{-1}X^T Y$  Problem 2 [1p]Let $v\in\mathbb{R}^D$ be a vector. Define the gradient of $f(v)\in\mathbb{R}$ with respect to $v$ to be $\frac{\partial f}{\partial v} = \left[\frac{\partial f(v)}{\partial v_1}, \frac{\partial f(v)}{\partial v_2}, ..., \frac{\partial f(v)}{\partial v_D}\right]$Find the following functions' gradients with respect to vector $[x, y, z]^T$:1. $f_1([x, y, z]^T) = x + y$2. $f_2([x, y, z]^T) = xy$3. $f_3([x, y, z]^T) = x^2y^2$4. $f_4([x, y, z]^T) = (x + y)^2$5. $f_5([x, y, z]^T) = x^4 + x^2 y z + x y^2 z + z^4$6. $f_6([x, y, z]^T) = e^{x + 2y}$7. $f_7([x, y, z]^T) = \frac{1}{x y^2}$8. $f_8([x, y, z]^T) = ax + by + c$9. $f_9([x, y, z]^T) = \tanh(ax + by + c)$  Ans:1. [1, 1, 0]2. [y, x, 0]3. [$2y^2x$, $2x^2y$, 0]4. [$2x + 2y$, $2x + 2y$, 0] 5. [$4x^3 +2yzx +y^2z$, $2xzy + zx^2$, $3x^4 + x^2y + xy^2$]6. [$e^{x + 2y}$, $2e^{x + 2y}$, 0]7. [$-\frac{1}{y^2x^2}$, $-\frac{2}{xy^3}$, 0]8. [$a$, $b$, 0]9. [$a*sech^2(ax + by + c)$, $b*sech^2(ax + by + c)$, 0]  Problem 3 [0.5p]Find the following functions' gradients or Jacobians with respect to vector $\mathbf{x}$, where $\mathbf{x}, \mathbf{b} \in \mathbb{R}^{n}$, $\mathbf{W} \in \mathbb{R}^{n \times n}$:1. $\mathbf{W} \mathbf{x} + \mathbf{b}$2. $\mathbf{x}^T \mathbf{W} \mathbf{x}$, Ans:1. $\mathbf{W}$2. $x^T(W^T + W)$  Problem 4 [1p]Find the derivative of$-\log(S(\mathbf{x})_j)$, where $S$ is the    softmax function    (https://en.wikipedia.org/wiki/Softmax_function) and we are    interested in the derivative over the $j$-th output of the    Softmax.  Ans:$$(-log(S(x_j)))' = - \frac{1}{S(x_j)} * S'(x_j) = - \frac{1}{S(x_j)} * (\frac{e^{x_j}}{\sum_i e^{x_i}})' = $$$$= - \frac{1}{S(x_j)} * \frac{(e^{x_j})' * (\sum_i e^{x_i}) - e^{x_j} * (\sum_i e^{x_i})'}{(\sum_i e^{x_i})^2} =$$$$= - \frac{\sum_i e^{x_i}}{e^{x_j}} * \frac{e^{x_j} * \sum_i e^{x_i} - e^{x_j} * e^{x_j}}{(\sum_i e^{x_i})^2} =$$$$= \frac{\sum_i e^{x_i} - e^{x_j}}{\sum_i e^{x_i}} = S(x_j) - 1$$  Problem 5 [0.5p]Consider a dataset with 400 examples of class C1 and 400 of class C2. Let tree A have 2 leaves with class distributions:| Tree A   | C1    | C2  ||----------|-------|-----|| Leaf 1   | 100   | 300 || Leaf 2   | 300   | 100 |and let tree B have 2 leaves with class distribution:| Tree B   | C1    | C2  ||----------|-------|-----|| Leaf 1   | 200   | 400 || Leaf 2   | 200   |   0 |What is the misclassification rate for both trees? Which tree is more pure according to Gini or Infogain?  Ans:<jupyter_code># misclassification rate: 
1/2*1/4 + 1/2 * 1/4, 1/2 * 1/2 + 1/2 * 0 
import pandas as pd
import numpy as np
def entropy(counts):
    _sum = np.sum(counts.values, axis=0)
    return -np.sum(counts/_sum * np.log2(counts/_sum), axis=0)

def gini(counts):
#     print(type(counts))
#     print(counts)
    _sum = np.sum(counts.values, axis=0)
    return np.sum(counts/_sum * (1 - counts/_sum), axis=0)

def mean_err_rate(counts):
    _sum = np.sum(counts.values, axis=0)
    return np.min(counts/_sum, axis=0)


data1 = pd.DataFrame.from_dict({"c1": [100,300], "c2": [300,100]}).transpose()
data2 = pd.DataFrame.from_dict({"c1": [200,200], "c2": [400,0]}).transpose()

np.average(gini(data1)), np.average(gini(data2), weights=[2/3, 1/3])
np.average(entropy(data1)), np.average(entropy(data2), weights=[2/3, 1/3])<jupyter_output>/Users/michal/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in log2
  """<jupyter_text>Problem 6 [1p]Consider regresion problem, with $M$ predictors $h_m(x)$ trained to aproximate a target $y$. Define the error to be $\epsilon_m(x) = h_m(x) - y$.Suppose you train $M$ independent classifiers with average least squares error$$E_{AV} = \frac{1}{M}\sum_{m=1}^M \mathbb{E}_{x}[\epsilon_m(x)^2].$$Further assume that the errors have zero mean and are uncorrelated:$$\mathbb{E}_{x}[\epsilon_m(x)] = 0\qquad\text{ and }\qquad\mathbb{E}_{x}[\epsilon_m(x)\epsilon_l(x)] = 0\text{ for } m \neq l$$Let the mean predictor be$$h_M(x) = \frac{1}{M}h_m(x).$$What is the average error of $h_M(x)$?  Problem 7 [1p]Suppose you work on a binary classification problem and train 3 weak classifiers. You combine their prediction by voting. Can the training error rate of the voting ensemble smaller that the error rate of the individual weak predictors? Can it be larger? Show an example or prove infeasibility.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
A = np.arange(1,7)
B = np.arange(5,11)
C = np.hstack([np.arange(1,5),[7,8]])
Res = np.arange(1, 11)
ResC = ['g'] * 8  + ['r'] * 2 
plt.scatter(A, A*0+1)
plt.scatter(B, B*0+1.5)
plt.scatter(C, C*0+2, c='purple')
plt.scatter(Res, Res*0, c=ResC )<jupyter_output><empty_output><jupyter_text>3*60% -> 80%<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
A = np.arange(1,7)
B = np.hstack([np.arange(1,5),[9,10]])
C = np.hstack([np.arange(1,5),[7,8]])
Res = np.arange(1, 11)
ResC = ['g'] * 4  + ['r'] * 6
plt.scatter(A, A*0+1)
plt.scatter(B, B*0+1.5)
plt.scatter(C, C*0+2, c='purple')
plt.scatter(Res, Res*0, c=ResC )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.23887189942235815, 'fraction_numerical': 0.05164797825348284, 'mean_word_length': 3.9263598326359834, 'pattern_counts': {'":': 4, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '136946', 'license': 'MIT', 'n_tokens_mistral': 2685, 'n_tokens_neox': 2492, 'n_words': 637, 'path': 'homework2/Homework2.ipynb', 'repo_name': 'iCarrrot/ML', 'size': '5886'}