<jupyter_start><jupyter_text>Podržano učenje s Scikit Learn bibliotekom.  Biblioteke- [scikit-learn](http://scikit-learn.org/stable/)- pandas- matplotlibCilj današnje vježbe je naučiti koristiti osnovne funkcionalnosti scikit-learn biblioteke za treniranje modela podržanog strojnog učenja. Modeli koje ćemo obraditi su Linearna Regresija, $k$ najbližih susjeda, logistička regresija i Slučajne šume.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import mean_squared_error, auc, roc_curve, classification_report

# import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Linearna regresija - Predviđanje prodaje iz cijene oglašavanja  Podatci<jupyter_code>data = pd.read_csv('data/Advertising.csv', index_col=0)
data<jupyter_output><empty_output><jupyter_text>Koje su **značajke**?- TV: dolari potrošeni na TV oglašavanje za jedan proizvod na određenom tržištu (u tisućama dolara)- Radio: dolari potrošeni na oglašavanje na radiju- Novine: dolari potrošeni na oglašavanje u novinamaŠto je **izlaz**?- Prodaja: prodaja jednog proizvoda na određenom tržištu (u tisućama jedinica)  Vizualizacija odnosa između značajki i izlaza<jupyter_code>fig, axs = plt.subplots(1, 3, sharey=True)
data.plot(kind='scatter', x='TV', y='sales',
          ax=axs[0], figsize=(16, 5), grid=True)
data.plot(kind='scatter', x='radio', y='sales', ax=axs[1], grid=True)
data.plot(kind='scatter', x='newspaper', y='sales', ax=axs[2], grid=True)<jupyter_output><empty_output><jupyter_text>Procjena ("učenje") koeficijenata modelaOpćenito govoreći, koeficijenti se procjenjuju korištenjem **metodom najmanjih kvadrata**, što znači da pronalazimo liniju (matematički) koja minimizira **zbroj kvadrata reziduala** (ili "zbroj grešaka na kvadrat"): Pitanja:- S koliko parametara je opisana ovakva funkcija?- Što bi svaki od tih parametara pokazivao na ovoj slici?  Učenje modelaPodijelimo prvo podatke na matricu značajki (X) i vektor razreda (y)<jupyter_code>feature_cols = ['TV', 'radio', 'newspaper']
X = data[feature_cols]
y = data.sales

X.describe()<jupyter_output><empty_output><jupyter_text>Dobro je provjeriti nedostaju li podatci u skupu.<jupyter_code>X.isna().sum(axis=0)<jupyter_output><empty_output><jupyter_text>**Scikit-learn** pruža vrlo jednostavan način učenja modela<jupyter_code>model = LinearRegression()  # create the model
model.fit(X, y)  # train it<jupyter_output><empty_output><jupyter_text>Da vidimo kako izgleda formula modela<jupyter_code>print(f"{model.intercept_:.3}", end="")
for coef, feature in zip(model.coef_, feature_cols):
    print(f" + {coef:.3} * {feature}", end="")<jupyter_output>2.94 + 0.0458 * TV + 0.189 * radio + -0.00104 * newspaper<jupyter_text>$$y = \beta_0 + \beta_1  \times TV + \beta_2  \times radio + \beta_3  \times newspaper$$$$y = 2.94 + 0.0458 \times TV + 0.189  \times radio + -0.00104  \times newspaper$$<jupyter_code>lr = LinearRegression()

# cross_val_predict returns an array of the same size as `y` where each entry
# is a prediction obtained by cross validation:
y_pred = cross_val_predict(lr, X, y, cv=5)


# Plot the results
fig, ax = plt.subplots(figsize=(12, 8))
ax.scatter(y, y_pred, edgecolors=(0, 0, 0))
ax.plot([min(y), max(y)], [min(y), max(y)], 'r--', lw=4)
ax.set_xlabel('Original')
ax.set_ylabel('Predicted')
plt.show()

print(f"error = {mean_squared_error(y, y_pred)}")<jupyter_output><empty_output><jupyter_text>Dodatne informacije o unakrsnoj validaciji - https://en.wikipedia.org/wiki/Cross-validation_(statistics)  Regularizacija U našem skupu podataka imamo samo 3 značajke, ali 200 zapisa, što je dovoljno za učenje dobrog linearnog modela. No što kad bi imali puno manje zapisa, recimo, 5?<jupyter_code>N = 5
X_s = X[:N]
y_s = y[:N]

lr = LinearRegression()

# cross_val_predict returns an array of the same size as `y` where each entry
# is a prediction obtained by cross validation:
y_pred = cross_val_predict(lr, X_s, y_s, cv=5)

# Plot the results
fig, ax = plt.subplots(figsize=(12, 8))
ax.scatter(y_s, y_pred, edgecolors=(0, 0, 0))
ax.plot([min(y_s), max(y_s)], [min(y_s), max(y_s)], 'r--', lw=4)
ax.set_xlabel('Original')
ax.set_ylabel('Predicted')
plt.show()

print(f"error = {mean_squared_error(y_s, y_pred)}")<jupyter_output><empty_output><jupyter_text>**Problem**: Model pamti zapise iz skupa za treniranje (prenaučenost).**Rješenje**: RegularizacijaRegularizacija se odnosi na metode koje pomažu u smanjenju prenaučenosti. Pokušajmo koristiti Ridge Regresiju, koja kažnjava velike težine $\beta_i$ i prisiljava ih da budu manjih magnituda. Time se smanjuje složenost modela.<jupyter_code>ridge = Ridge(alpha=6)

# cross_val_predict returns an array of the same size as `y` where each entry
# is a prediction obtained by cross validation:
y_pred = cross_val_predict(ridge, X_s, y_s, cv=5)

# Plot the results
fig, ax = plt.subplots(figsize=(12, 8))
ax.scatter(y_s, y_pred, edgecolors=(0, 0, 0))
ax.plot([min(y_s), max(y_s)], [min(y_s), max(y_s)], 'r--', lw=4)
ax.set_xlabel('Original')
ax.set_ylabel('Predicted')
plt.show()

print(f"error = {mean_squared_error(y_s, y_pred)}")<jupyter_output><empty_output><jupyter_text>Logistička regresija - Predviđanje vjerojatnosti preživljavanja titanikaZa to ćemo koristiti podatke dostupne na _Encyclopedia Titanica_.   Podatci<jupyter_code>data = pd.read_excel('data/titanic.xls')
data<jupyter_output><empty_output><jupyter_text>Koje su **značajke**?- name: <NAME>- sex: muški ili ženski- age: Starost u godinama- sibsp: broj braće i sestara / supružnika na Titanicu- parch: broj roditelja/djece na Titanicu- ticket: <NAME>- fare: cijena karte- cabin: broj kabine- embarked: Luka ukrcajaKakav je **odgovor**?- preživio: je li putnik preživio katastrofu ili ne<jupyter_code>dead = data[data['survived'] == 0]
survived = data[data['survived'] == 1]

print("Survived {0}, Dead {1}".format(len(survived), len(dead)))
print("Survived {:.2%}".format(len(survived)/len(data)))<jupyter_output>Survived 500, Dead 809
Survived 38.20%<jupyter_text>Za potrebe ove vježbe uzeti ćemo samo neke značajke i pretpostaviti da ostale nisu prediktivne.<jupyter_code>feature_cols = ['sex', 'age', 'sibsp', 'parch', 'fare']
X = pd.get_dummies(data[feature_cols])
y = data.survived

X.isna().sum(axis=0)<jupyter_output><empty_output><jupyter_text>Pokušajmo popraviti podatke osnovnom metodom imputacije: zamjenjujući nedostajuće vrijednosti srednjom vrijednosti.Više informacija: https://en.wikipedia.org/wiki/Imputation_(statistics)<jupyter_code>X = X.fillna(X.mean())
X.isna().sum(axis=0)<jupyter_output><empty_output><jupyter_text>Treniranje modela i ispis izvještaja<jupyter_code>logistic = LogisticRegression()
y_pred = cross_val_predict(logistic, X,  y, cv=10)

print(classification_report(y, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.81      0.84      0.82       809
           1       0.72      0.68      0.70       500

    accuracy                           0.77      1309
   macro avg       0.76      0.76      0.76      1309
weighted avg       0.77      0.77      0.77      1309<jupyter_text>Ako logistička regresija klasifikator, zašto se onda zove regresija? Logistička regresija koristi prag vjerojatnosti za odlučivanje kojoj klasi dodijeliti predviđanje. U nekim slučajevima nas zanima kako se model ponaša na različitim razinama ovog praga.Pogledajmo ROC krivulju!https://en.wikipedia.org/wiki/Receiver_operating_characteristic<jupyter_code># Predict the probabilities with a cross validationn
y_pred_proba = cross_val_predict(logistic, X, y, cv=10, method="predict_proba")
# Compute the False Positive Rate and True Positive Rate
fpr, tpr, _ = roc_curve(y, y_pred_proba[:, 1])
# Compute the area under the fpt-tpf curve
auc_score = auc(fpr, tpr)

plt.plot(fpr, tpr)
plt.plot([0, 1], [0, 1], 'r--')
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve - Area = {:.5f}".format(auc_score))
logistic.fit(X, y)
test = pd.DataFrame([
    [25, 0, 0, 100, 0, 1], # 25 year old man
    [25, 0, 0, 100, 1, 0], # 25 year old woman
], columns=X.columns)

logistic.predict(test)
logistic.predict_proba(test)<jupyter_output><empty_output><jupyter_text>Slučajna šuma - podatci s Titanika<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score

number_trees = [n for n in range(1, 21)]

accuracy_scores = []
f1_scores = []


for nt in number_trees:
    clf = RandomForestClassifier(max_depth=5, random_state=0, n_estimators=nt)
    clf.fit(X, y)
    y_pred = cross_val_predict(clf, X, y, cv=5)

    accuracy_scores.append(accuracy_score(y, y_pred))
    f1_scores.append(f1_score(y, y_pred))
fig, ax = plt.subplots(1, figsize=(6, 4))

ax.plot(number_trees, accuracy_scores, label="Accuracy")
ax.plot(number_trees, f1_scores, label="F1 scores")

ax.set_ylabel("Score value")
ax.set_xlabel("Number of trees")
ax.legend()<jupyter_output><empty_output><jupyter_text>--------- Kratki pregled rada K-NN modelaPrvo je potrebno generirati kopleksne oblike kako bi pokazali rad k-NN modela.<jupyter_code>from sklearn.datasets import make_moons, make_gaussian_quantiles
from sklearn.model_selection import train_test_split

X_moons, y_moons = make_moons(500, noise=0.2, random_state=0)
X_circles, y_circles = make_gaussian_quantiles(n_samples=100, random_state=0)


fig, axs = plt.subplots(1, 2, figsize=(9,4))

axs[0].scatter(X_moons[:,0], X_moons[:,1], c=y_moons)
axs[0].set_title('Moon Data')

axs[1].scatter(X_circles[:,0], X_circles[:,1], c=y_circles)
axs[1].set_title('Circles')
# Plotting helper 
# Source: https://scikit-learn.org/stable/auto_examples/neighbors/plot_classification.html
from matplotlib.colors import ListedColormap

def plot_prediction(model, X, y, ax, K):
    # step size in the mesh
    h = .02
    # Create color maps
    cmap_light = ListedColormap(['orange', 'cyan', 'cornflowerblue'])
    cmap_bold = ListedColormap(['darkorange', 'c', 'darkblue'])

    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    ax.pcolormesh(xx, yy, Z, cmap=cmap_light, shading='auto')

    # Plot also the training points
    ax.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold,
                edgecolor='k', s=20)
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_title("K = {}".format(K))<jupyter_output><empty_output><jupyter_text>Stvaranje modela sa k=1 i k=15<jupyter_code># K = 1
clf_moons_1 = KNeighborsClassifier(1)
clf_moons_1.fit(X_moons, y_moons)
clf_circles_1 = KNeighborsClassifier(1)
clf_circles_1.fit(X_circles, y_circles)

# K = 15
clf_moons_15 = KNeighborsClassifier(15)
clf_moons_15.fit(X_moons, y_moons)
clf_circles_15 = KNeighborsClassifier(15)
clf_circles_15.fit(X_circles, y_circles)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12817777777777778, 'fraction_numerical': 0.023733333333333332, 'mean_word_length': 5.499711149624495, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '440194', 'license': 'Apache-2.0', 'n_tokens_mistral': 4764, 'n_tokens_neox': 4547, 'n_words': 1011, 'path': 'labosi/lab-07/2021-22/by_sheriffHorsey/lab7.ipynb', 'repo_name': 'studosi-fer/UUZOP', 'size': '11262'}