<jupyter_start><jupyter_text>Picking the gamma (length scale) value for Regression with RBF kernelHere we use the quantile heuristics same as [SVM_rbf_gamma notebook](https://github.com/chengsoonong/didbits/blob/master/Estimation/SVM_rbf_gamma.ipynb).<jupyter_code># direct to proper path
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path)
import numpy as np

import pandas
import matplotlib.pyplot as plt
import itertools
from collections import defaultdict

from codes.embedding import Embedding
from codes.environment import Rewards_env
from codes.ucb import GPUCB
from codes.evaluations import evaluate, plot_eva
from codes.regression import Regression
from codes.kernels_pairwise import spectrum_kernel_pw, mixed_spectrum_kernel_pw, WD_kernel_pw, WD_shift_kernel_pw

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import PairwiseKernel, DotProduct, RBF
from sklearn.kernel_ridge import KernelRidge

from ipywidgets import IntProgress
from IPython.display import display
import warnings
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Reading the Dataset<jupyter_code># Data downloaded from https://github.com/synbiochem/opt-mva
# Paper https://pubs.acs.org/doi/abs/10.1021/acssynbio.8b00398

Path = '../data/trainset.rbs1.v2.csv'

df = pandas.read_csv(Path, delimiter=',')
df.head()
Log_flag = False # indicates whether take log label
Norm_method = 'minmax' # indicates how to normalize label (one of 'mean', 'minmax', None)
data = np.asarray(df.iloc[:, 2:26])
data.shape<jupyter_output><empty_output><jupyter_text>Picking a good gamma valueThe ``gamma`` parameter appears as follows.Using a rbf kernel, we measure similarity between datapoints by $k(x, y) = \exp(-\gamma||x-y||^2)$.If $\gamma||x-y||^2$ is very small -- i.e., if $\frac{1}{\gamma}$ is much larger than $||x-y||^2$ -- $k(x, y)$ will be close to 1.So a small $\gamma$ value, relative to the distances between datapoints, means the influence of a single datapoint reaches over most of the dataset.If $\gamma||x-y||^2$ is very large, then $k(x, y)$ will be close to 0.That is, for a large $\gamma$ value, a given datapoint has influence only on datapoints very close to it.To pick a good ``gamma`` value, we would use cross validation. To do cross validation, we need to know what range sensible ``gamma`` values lie in.To find this range, we use a heuristic.We want the influence of each datapoint to extend over some but not all of the dataset -- that is, we want ``gamma`` to be some value such that $\gamma||x-y||^2$ is neither very large nor very small.We thus want $\frac{1}{\gamma}$ to be of similar magnitude to "typical" values of $||x-y||^2$.<jupyter_code>pairs = itertools.combinations(data, r = 2)
distances = [np.linalg.norm(a-b)**2 for a,b in pairs]
distances.sort()
gamma_values = [distances[int(len(distances)*frac)] for frac in [0.1, 0.3, 0.5, 0.7, 0.9]]
print(gamma_values)
# PS. in sklearn gpr.kernel RBF using length scales instead

len_scale = np.sqrt(0.5 * np.asarray(gamma_values))
len_scale = list(set(len_scale))
len_scale<jupyter_output><empty_output><jupyter_text>We also compare with length scale of 0.1, 1, 10 and 100, to show values outside this range.<jupyter_code>len_scale.extend([0.1, 1, 10, 100])
len_scale<jupyter_output><empty_output><jupyter_text>Regression<jupyter_code>df['RBS'] = df['rbs1'] + df['rbs2']

# take log FC -- possiblely provide Gaussain distribution?
if Log_flag:
    df['FC'] = np.log(df['FC'])

if Norm_method == 'mean':
    # mean normalization
    df['FC'] = (df['FC'] - df['FC'].mean())/df['FC'].std()
elif Norm_method == 'minmax':
    # min-max normalization 
    df['FC'] = (df['FC'] - df['FC'].min())/(df['FC'].max() - df['FC'].min())
else:
    assert Norm_method == None

data = np.asarray(df[['RBS', 'FC']])
data.shape
# indicates whether cross validation (KFOLD)
cross_val_flag = False 

# indicates whether plot predict_label vs. true label
plot_flag = True 

# name dictionaries
regression_name = {KernelRidge: 'KR',
                  GaussianProcessRegressor: 'GPR'}


train_scores = {}
test_scores = {}
def run_regression(model, kernel, embedding, l):
    reg = Regression(model(kernel = kernel), data, embedding)
    reg.train()
    train_score, test_score = reg.evaluate(cross_val_flag = cross_val_flag, plot_flag = plot_flag)
    key = regression_name[model] + '_RBF(' +  str(round(l,2)) + ')'
    train_scores[key] = train_score
    test_scores[key] = test_score
    print()

for model in [KernelRidge, GaussianProcessRegressor]:
    for l in len_scale:
        run_regression(model, RBF(l), 'onehot', l)    
sorted_test_scores = sorted(test_scores.items(), key=lambda kv: kv[1])

import collections

sorted_test_scores = collections.OrderedDict(sorted_test_scores)

plt.bar(range(len(sorted_test_scores)), list(sorted_test_scores.values()), align='center')
plt.xticks(range(len(sorted_test_scores)), list(sorted_test_scores.keys()), rotation = 70)
plt.ylabel('RMSE')
plt.title('Test Scores')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1270772238514174, 'fraction_numerical': 0.012316715542521995, 'mean_word_length': 5.30826140567201, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '434832', 'license': 'Apache-2.0', 'n_tokens_mistral': 1788, 'n_tokens_neox': 1706, 'n_words': 494, 'path': 'notebook/others/regression/Regression_RBF_gamma.ipynb', 'repo_name': 'mholowko/Solaris', 'size': '5115'}