<jupyter_start><jupyter_text>In this problem setting we are interested in the case given two data sets, a labelled $\mathcal{D}_{Train} = \{(\mathbf{x}_i,y_i)\}_{i=1}^n$ and an unlabelled  $\mathcal{D}_{Target} = \{\mathbf{x}_j\}_{j=n+1}^{n+m}$, with $\mathbf{x} \in \mathbb{R}^d$, a $d$-dimensional feature vector,  and $y \in \mathbb{R}$, a real-valued output, assuming an underlying generative process: $$(X_i,Y_i) \in \mathcal{D}_{Train} \overset{i.i.d.}{\sim} P = P_X \times P_{Y|X}$$$$X_j \in \mathcal{D}_{Target} \overset{i.i.d.}{\sim} Q_X$$We first set up our synthetic example data, where both $P_X$ and $Q_X$ will take the form of Normal distributions, and $P_{Y|X} = \mathcal{N}(f(\mathbf{x}),\sigma^2)$, i.e. $Y$ is some deterministic function of $X$ corrupted by some Gaussian noise.<jupyter_code>class feature_dist():
    '''
    Just a little class to wrap the behaviour of the distrbutions over
    features. Defines a Gaussian distribution, allows for pdf evaluation 
    and sampling.
    '''
    def __init__(self,mean=0,sd=1):
        self.mean = mean 
        self.sd = sd
    
    def pdf(self,x):
        return(ss.norm.pdf(x,loc=self.mean,scale=self.sd))
    
    def sample(self,num):
        return npr.normal(loc=self.mean,scale=self.sd,size = (num,1))
P_x = feature_dist(7,2)
Q_x = feature_dist(11,2)

x = np.linspace(0, 18,1000)

pdf_p = P_x.pdf(x)
pdf_q = Q_x.pdf(x)

plt.plot(x,pdf_p, label = 'P_x')
plt.plot(x,pdf_q, label = 'Q_x')
plt.legend()
plt.show()
num_samples = 50

samples_p = P_x.sample(num_samples)
samples_q = Q_x.sample(num_samples)

plt.hist(samples_p, alpha = 0.5, label = 'Samples from P_x')
plt.hist(samples_q, alpha = 0.5, label = 'Samples from Q_x')
plt.legend()
plt.show()
def y_target(x):
    return np.sin(x)/2 + x/4 - x**2/100

def y_noise(n,sd):
    return npr.normal(loc=0,scale=sd,size=(n,1))


y_samples_p = y_target(samples_p) + y_noise(num_samples, 0.1)
y_samples_q = y_target(samples_q) + y_noise(num_samples, 0.1)
true_func = y_target(x)

plt.scatter(samples_p,y_samples_p, label = 'P_x')
plt.scatter(samples_q,y_samples_q, label = 'Q_x')

plt.plot(x, true_func, c = 'r', label = 'True function')

plt.legend()
plt.show()
    
from models import transductive, mc_dropout_concrete
from utils import nice_plot, nice_plot_conc
model_concrete = mc_dropout_concrete([1,32,64,1],d_rate = [0.5,0.5,0.5])
model_concrete.train(samples_p, y_samples_p, 10000)
nice_plot_conc(model_concrete,samples_p,y_samples_p,samples_q,y_samples_q,true_func)
model = transductive([1,32,64,1],d_units = 8)
model.train(samples_p, y_samples_p, samples_q, 1000)
nice_plot(model,samples_p,y_samples_p,samples_q,y_samples_q,true_func)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15986646884272998, 'fraction_numerical': 0.02373887240356083, 'mean_word_length': 5.006681514476615, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '602834', 'license': 'MIT', 'n_tokens_mistral': 1140, 'n_tokens_neox': 1073, 'n_words': 229, 'path': 'TD/tutorial.ipynb', 'repo_name': 'XanderJC/transductive_dropout', 'size': '2696'}