<jupyter_start><jupyter_text>Toy Regression      [Run in Google Colab](https://colab.research.google.com/drive/1adhqoV6b0uEavLDmMfkiwtRjam0DrXux?usp=sharing)     [View on GitHub](https://github.com/adapt-python/notebooks/blob/d0364973c642ea4880756cef4e9f2ee8bb5e8495/Regression.ipynb) You will find here the application of DA methods from the ADAPT package on a simple one dimensional DA regression problem.First we import packages needed in the following. We will use ``matplotlib Animation`` tools in order toget a visual understanding of the selected methods:<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from matplotlib import rc
rc('animation', html='jshtml')<jupyter_output><empty_output><jupyter_text>Experimental Setup We now set the synthetic regression DA problem using the [make_regression_da](https://adapt-python.github.io/adapt/generated/adapt.utils.make_regression_da.html)function from ``adapt.utils``.<jupyter_code>from adapt.utils import make_regression_da

Xs, ys, Xt, yt = make_regression_da()
tgt_index_lab_ = np.random.choice(100,3)
Xt_lab = Xt[tgt_index_lab_]; yt_lab = yt[tgt_index_lab_]<jupyter_output><empty_output><jupyter_text>We define here a ``show`` function which we will use in the following to visualize the algorithms performanceson the toy problem.<jupyter_code>def show(ax, y_pred=None, X_src=Xs, weights_src=50, weights_tgt=100):
    ax.scatter(X_src, ys, s=weights_src, label="source", edgecolor="black")
    ax.scatter(Xt, yt, s=50, alpha=0.5, label="target", edgecolor="black")
    ax.scatter(Xt_lab, yt_lab, s=weights_tgt, 
                c="black", marker="s", alpha=0.7, label="target labeled")
    if y_pred is not None:
        ax.plot(np.linspace(-0.7, 0.6, 100), y_pred, c="red", lw=3, label="predictions")
        index_ = np.abs(Xt - np.linspace(-0.7, 0.6, 100)).argmin(1)
        score = np.mean(np.abs(yt - y_pred[index_]))
        score = " -- Tgt MAE = %.2f"%score
    else:
        score = ""
    ax.set_xlim((-0.7,0.6))
    ax.set_ylim((-1.3, 2.2))
    ax.legend(fontsize=16)
    ax.set_xlabel("X", fontsize=16)
    ax.set_ylabel("y = f(X)", fontsize=16)
    ax.set_title("Toy regression DA issue"+score, fontsize=18)
    return ax
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
show(ax=ax)
plt.show()<jupyter_output><empty_output><jupyter_text>As we can see in the figure above (plotting the output data ``y`` with respect to the inputs ``X``),source and target data define two distinct domains. We have modeled here a classical supervised DA issue where the goal is to build a good model on orange data knowing only the labels (``y``) of the blueand black points.We now define the base model used to learn the task. We use here a neural network with two hidden layer.We also define a ``SavePrediction`` callback in order to save the prediction of the neural network ateach epoch.<jupyter_code>import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Input, Dense, Reshape
from tensorflow.keras.optimizers import Adam

def get_model():
    model = Sequential()
    model.add(Dense(100, activation='elu', input_shape=(1,)))
    model.add(Dense(100, activation='relu'))
    model.add(Dense(1))
    model.compile(optimizer=Adam(0.01), loss='mean_squared_error')
    return model
from tensorflow.keras.callbacks import Callback

class SavePrediction(Callback):  
    """
    Callbacks which stores predicted
    labels in history at each epoch.
    """
    def __init__(self):
        self.X = np.linspace(-0.7, 0.6, 100).reshape(-1, 1)
        self.custom_history_ = []
        super().__init__()
            
    def on_epoch_end(self, batch, logs={}):
        """Applied at the end of each epoch"""
        predictions = self.model.predict_on_batch(self.X).ravel()
        self.custom_history_.append(predictions)<jupyter_output><empty_output><jupyter_text>TGT Only First, let's fit a network only on the three labeled target data. As we could have guessed,this is not sufficient to build an efficient model on the whole target domain.<jupyter_code>np.random.seed(0)
tf.random.set_seed(0)

model = get_model()
save_preds = SavePrediction()
model.fit(Xt_lab, yt_lab, callbacks=[save_preds], epochs=100, batch_size=64, verbose=0);
def animate(i, *fargs):
    ax.clear()
    y_pred = save_preds.custom_history_[i].ravel()
    if len(fargs)<1:
        show(ax, y_pred)
    else:
        show(ax, y_pred, **fargs[0])
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ani = animation.FuncAnimation(fig, animate, frames=100, blit=False, repeat=True)
ani<jupyter_output><empty_output><jupyter_text>![tgtOnly](../images/tgtOnly.gif)  Src Only We would like to use the large amount of labeled source data to improvethe training of the neural network on the target domain. However,as we can see on the figure below, using only the sourcedataset fails to provide an efficient model.<jupyter_code>np.random.seed(0)
tf.random.set_seed(0)

model = get_model()
save_preds = SavePrediction()
model.fit(Xs, ys, callbacks=[save_preds], epochs=100, batch_size=100, verbose=0);
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ani = animation.FuncAnimation(fig, animate, frames=100, blit=False, repeat=True)
ani<jupyter_output><empty_output><jupyter_text>![srcOnly](../images/srcOnlyReg.gif)  All Same thing happen when using both source and target labeled data. As the source sample ovewhelms the target one,the model is not fitted enough on the target domain.<jupyter_code>np.random.seed(0)
tf.random.set_seed(0)

model = get_model()
save_preds = SavePrediction()
model.fit(np.concatenate((Xs, Xt_lab)),
          np.concatenate((ys, yt_lab)),
          callbacks=[save_preds],
          epochs=100, batch_size=110, verbose=0);
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ani = animation.FuncAnimation(fig, animate, frames=100, blit=False, repeat=True)
ani<jupyter_output><empty_output><jupyter_text>![all](../images/all.gif)  CORAL Let's now consider the domain adaptation method [CORAL](https://adapt-python.github.io/adapt/generated/adapt.feature_based.CORAL.html) This "two-stage" method first perfroms a feature alignment on source data and then fitan estimator on the new feature space.<jupyter_code>from adapt.feature_based import CORAL

save_preds = SavePrediction()
model = CORAL(get_model(), lambda_=1000., random_state=0)
model.fit(Xs.reshape(-1, 1), ys, Xt,
          callbacks=[save_preds], epochs=100, batch_size=110, verbose=0);
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
X_transformed = model.predict_features(Xs.reshape(-1, 1), domain="src").ravel()
ani = animation.FuncAnimation(fig, animate, frames=100, blit=False, repeat=True,
                              fargs=(dict(X_src=X_transformed),))
ani<jupyter_output><empty_output><jupyter_text>![coral](../images/coral.gif) As we can see. when using CORAL method, source input data are translated closer totarget data. However, for this example, this is not enough to obtain a good modelon the target domain.  TrAdaBoostR2 We now consider an instance-based method: [TrAdaBoostR2](https://adapt-python.github.io/adapt/generated/adapt.instance_based.TrAdaBoostR2.html).This method consists in a reverse boosting algorithm decreasing the weights of source data poorly predictedat each boosting iteraton.<jupyter_code>from adapt.instance_based import TrAdaBoostR2

model = TrAdaBoostR2(get_model(), n_estimators=30, random_state=0)

save_preds = SavePrediction()
model.fit(Xs.reshape(-1, 1), ys, Xt_lab, yt_lab,
          callbacks=[save_preds], epochs=100, batch_size=110, verbose=0);
def animate_tradaboost(i):
    ax.clear()
    i *= 10
    j = int(i / 100)
    y_pred = save_preds.custom_history_[i].ravel()
    weights_src = 10000 * model.sample_weights_src_[j]
    weights_tgt = 10000 * model.sample_weights_tgt_[j]
    show(ax, y_pred, weights_src=weights_src, weights_tgt=weights_tgt)
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ani = animation.FuncAnimation(fig, animate_tradaboost, frames=299, blit=False, repeat=True)
ani<jupyter_output><empty_output><jupyter_text>![trada](../images/tradaboost.gif) As we can see on the figure above, [TrAdaBoostR2](https://adapt-python.github.io/adapt/generated/adapt.instance_based.TrAdaBoostR2.html) perfroms very wellon this toy DA issue! The importance weights are described by the size of data points.We observe that the weights of source instances close to 0 are decreased as the weights of target instances increase.This source instances indeed misleaded the fitting of the network on the target domain. Decreasing their weights helpsthen a lot to obtain a good target model.  RegularTransferNN Finally, we consider here the paremeter-based method [RegularTransferNN](https://adapt-python.github.io/adapt/generated/adapt.parameter_based.RegularTransferNN.html).This method fits the target labeled data with a regularized loss. During training, the mean squared error on target data isregularized with the euclidean distance between the target model parameters and the ones of a pre-trained source model.<jupyter_code>from adapt.parameter_based import RegularTransferNN

np.random.seed(0)
tf.random.set_seed(0)

save_preds = SavePrediction()
model_0 = get_model()
model_0.fit(Xs.reshape(-1, 1), ys, callbacks=[save_preds], epochs=100, batch_size=110, verbose=0);
model = RegularTransferNN(model_0, lambdas=1.0, random_state=0)
model.fit(Xt_lab, yt_lab, callbacks=[save_preds], epochs=100, batch_size=110, verbose=0);
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ani = animation.FuncAnimation(fig, animate, frames=200, blit=False, repeat=True)
ani<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11553246753246753, 'fraction_numerical': 0.02566233766233766, 'mean_word_length': 5.111746031746032, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 41, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '90404', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 3390, 'n_tokens_neox': 3176, 'n_words': 875, 'path': 'src_docs/examples/Regression.ipynb', 'repo_name': 'antoinedemathelin/adapt', 'size': '9625'}