<jupyter_start><jupyter_text>Active Learning via Membership Query Synthesis for Protein Yield Prediction > (work in progress) Membership Query Synthesis via state of the art generative models for active learning.- toc: true - badges: true- comments: true- categories: [jupyter]  Sketch: - Idea 1 is to use a fitted "oracle" as a proof of concept to label generative instances (data leakage prone). May be worth looking at if the learner is far more onerous to train than the generator for some reason, or as a proof of concept.- Idea 2 is to use a generative model to query, naïvely, a human oracle (```input(...)```) with a single instance (use case here would be lots of unlabeled data for the generator to learn from, but with very few and very expensive _labeled_ instances).- Idea 3 is to do something like the paper. We generate a ton of unlabeled instances per active learning iteration, then via uncertainty sampling choose a few to query either to a human/oracle, or do a nearest neighbors search for real labeled instances (for testing purposes).- Idea 4 is to do what the paper did but implement it in modAL, and also implement a regression version (could be written up too).<jupyter_code>!pip install -q sdv
!pip uninstall -y numpy
!pip install numpy
!pip install -q ModAL
import pandas as pd

data = pd.read_csv('./DataPool.csv').dropna()
data
import torch; torch.cuda.is_available()<jupyter_output><empty_output><jupyter_text>I will evaluate each model on these data to decide what to use for production.<jupyter_code>import warnings
warnings.filterwarnings('ignore')

from tqdm.notebook import tqdm
from sdv.tabular import GaussianCopula, CopulaGAN, CTGAN, TVAE

models = [GaussianCopula(), CopulaGAN(), CTGAN(), TVAE()]
for model in tqdm(models):
  model.fit(data)

synthetic_data = [model.sample() for model in models]
synthetic_data[0].head()
from sdv.evaluation import evaluate
evaluations=[evaluate(s_data, data, aggregate=False) for s_data in synthetic_data]
evaluations[0]
evaluations[1]
evaluations[2]
evaluations[3]
for i, generated_data in enumerate(synthetic_data):
  print(f'The aggregated evaluation result for {models[i].__class__.__name__} is {evaluate(generated_data, data, aggregate=True)}. \n')
aggregated_evaluations = [evaluate(sd, data, aggregate=True) for sd in synthetic_data]
aggregated_evaluations<jupyter_output><empty_output><jupyter_text>We can now see how these generative models behave in a supervised setting. This fits a Machine Learning model on the synthetic data and then evaluates it making predictions on the real data.<jupyter_code>from sdv.metrics.tabular import LinearRegression, MLPRegressor #only two are built in...

for i, generated_data in enumerate(synthetic_data):
  for regressor in (LinearRegression, MLPRegressor):
    print(f'''
        The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model {models[i].__class__.__name__} and regression model {regressor.__name__}
        is {regressor.compute(data,generated_data,target='Yield')}.''')<jupyter_output>The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model GaussianCopula and regression model LinearRegression
        is -13.266503943446004.

        The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model GaussianCopula and regression model MLPRegressor
        is -6980.369229257133.

        The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model CopulaGAN and regression model LinearRegression
        is -0.750010067105459.

        The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model CopulaGAN and regression model MLPRegressor
        is -3.0565872071910247.

        The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model CTGAN and regression model LinearRegression
        is [...]<jupyter_text>Given the active learning setting, we also could see how fitting to the real data and then fitting afterward to the fake data could improve performance.<jupyter_code>#baseline

for regressor in (LinearRegression, MLPRegressor):
  print(f'''
      The evaluation R^2 result for real training data and real inference/scoring data with regression model {regressor.__name__}
      is {regressor.compute(data,data,target='Yield')}.''')
combined_data=[pd.concat([sd,data]) for sd in synthetic_data]

for i, generated_data in enumerate(combined_data):
  for regressor in (LinearRegression, MLPRegressor):
    print(f'''
        The evaluation R^2 result for synthetic training data and real inference/scoring data with
        generative model {models[i].__class__.__name__} and regression model {regressor.__name__}
        is {regressor.compute(data,generated_data,target='Yield')}.''')
data<jupyter_output><empty_output><jupyter_text>We need an Oracle. That is, we want some way to accurately label the generative instances. We can try to train a regressor on all available data and say this is a "ground-truth" approximation.<jupyter_code>from sklearn.ensemble import RandomForestRegressor 

regr=RandomForestRegressor()
regr.fit(X=data.drop('Yield',1).values,y=data['Yield'])
regr.score(X=data.drop('Yield',1).values,y=data['Yield'])
synthetic_data[0] #GaussianCopula
s_data=combined_data[0] #example GaussianCopula
s_targets=s_data['Yield']
s_targets_oracle=regr.predict(s_data.drop('Yield',1))
from sklearn.metrics import r2_score
r2_score(s_targets, s_targets_oracle) #we compare generative labels to random-forest predicted labels, such that the random forest was trained on REAL data

#not a terrible result. This indicates that the random forest is not great at prediction, or the genarative model is not great at label approximation, or both.

#in any event, let a fitted regressor stand-in for an "oracle" for our membership query synthesis!<jupyter_output><empty_output><jupyter_text>MQS steps: we have a budget of 510 additional instances to be labeled via MQS by an oracle. The active learning setup is as follows:- 1: train an "oracle" on ALL data, even test data.- 2: Train a query synthesizer to generate instances based on ALL (unlabeled) data available.- 3: train a freshly instantiated learner on some all available complete feature/label instances (510+507 - test amount)- 4: generate 510 membership queries and let the "Oracle" label them.- 5: compare to random sampling.<jupyter_code>#from the top:
from sklearn.ensemble import RandomForestRegressor
import pandas as pd

data = pd.read_csv('./DataPool.csv').dropna()
oracle=RandomForestRegressor()
oracle.fit(X=data.drop('Yield',1).values,y=data['Yield'])
oracle.score(X=data.drop('Yield',1).values,y=data['Yield'])<jupyter_output><empty_output><jupyter_text>We now must generate a held-put test set for evaluation that the synthesizer cannot see.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    data.drop('Yield',1), data['Yield'], test_size=0.15, random_state=42)
from sdv.tabular import GaussianCopula

synthesizer = GaussianCopula()
synthesizer.fit(X_train)

synthetic_data=synthesizer.sample() #assess quality of synthesizer

eval = evaluate(synthetic_data, X_train, aggregate=False)
eval<jupyter_output><empty_output><jupyter_text>We now instantiate a blind learner and start "fresh" with the dataset, such that we have a total of 510 labeling experiments. We can first try with 100 initial instances and 410 membership synthesis queries.<jupyter_code>import numpy as np
from tqdm.notebook import tqdm,trange
n_call_to_oracle=410
history=[]
curr_training_pool=X_train.sample(100).values #random

for _ in trange(n_call_to_oracle):
  synthetic_datapoint = synthesizer.sample(1)
  curr_training_pool=np.vstack((curr_training_pool,synthetic_datapoint))
  learner = RandomForestRegressor()
  learner.fit(curr_training_pool,oracle.predict(curr_training_pool))
  history.append(learner.score(X_test,y_test))
from matplotlib import pyplot as plt
pd.DataFrame(history).plot(colormap='viridis')
plt.legend('')
plt.title('Pseudo-Oracle Membership Query Synthesis')
import copy
from modAL.models import ActiveLearner

def random_sampling(classifier, X_pool):
    n_samples = len(X_pool)
    query_idx = np.random.choice(range(n_samples))
    return query_idx, X_pool[query_idx]

X_train,y_train=X_train.values,y_train.values

n_initial = 100
initial_idx = np.random.choice(range(len(X_train)), size=n_initial, replace=False)
X_initial, y_initial = X_train[initial_idx], y_train[initial_idx]

X_pool=copy.deepcopy(X_train)
y_pool=copy.deepcopy(y_train)

X_pool=np.delete(X_pool,initial_idx,axis=0)
y_pool=np.delete(y_pool,initial_idx,axis=0)

learner = ActiveLearner(
    estimator=RandomForestRegressor(),
    query_strategy=random_sampling,
    X_training=X_initial, y_training=y_initial
)

n_call_to_oracle=410

history = []

for _ in trange(n_call_to_oracle):
    query_idx, _ = learner.query(X_pool)
    learner.teach(
        X_pool[query_idx].reshape(1,-1),
        y_pool[query_idx].reshape(1,))
    X_pool = np.delete(X_pool, query_idx, axis=0)
    y_pool = np.delete(y_pool, query_idx, axis=0)

    r2 = learner.score(X_test,y_test)
    history.append(r2)

import matplotlib.pyplot as plt

pd.DataFrame(history).plot(colormap='Spectral')
plt.legend('')
plt.title('Random Sampling')
import copy
from modAL.models import ActiveLearner

def GP_regression_std(regressor, X):
    _, std = regressor.predict(X, return_std=True)
    query_idx = np.argmax(std)
    return query_idx, X[query_idx]




from sklearn.model_selection import train_test_split
data = pd.read_csv('./DataPool.csv').dropna()
X_train, X_test, y_train, y_test = train_test_split(
    data.drop('Yield',1), data['Yield'], test_size=0.15, random_state=42)

X_train,y_train=X_train.values,y_train.values



from modAL.models import ActiveLearner
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import WhiteKernel, RBF



n_initial = 100
initial_idx = np.random.choice(range(len(X_train)), size=n_initial, replace=False)
X_initial, y_initial = X_train[initial_idx], y_train[initial_idx]

X_pool=copy.deepcopy(X_train)
y_pool=copy.deepcopy(y_train)

X_pool=np.delete(X_pool,initial_idx,axis=0)
y_pool=np.delete(y_pool,initial_idx,axis=0)


kernel = RBF(length_scale=1.0, length_scale_bounds=(1e-2, 1e3)) \
         + WhiteKernel(noise_level=1, noise_level_bounds=(1e-10, 1e+1))

learner = ActiveLearner(
    estimator=GaussianProcessRegressor(kernel=kernel),
    query_strategy=GP_regression_std,
    X_training=X_initial, y_training=y_initial
)



n_call_to_oracle=410

history = []

for _ in trange(n_call_to_oracle):
    query_idx, _ = learner.query(X_pool)
    learner.teach(
        X_pool[query_idx].reshape(1,-1),
        y_pool[query_idx].reshape(1,))
    X_pool = np.delete(X_pool, query_idx, axis=0)
    y_pool = np.delete(y_pool, query_idx, axis=0)

    r2 = learner.score(X_test,y_test)
    history.append(r2)
import matplotlib.pyplot as plt

pd.DataFrame(history).plot(colormap='cividis')
plt.legend('')
plt.title('GPR Std. Sampling')<jupyter_output><empty_output><jupyter_text>Finished initial MQS implementation. We have a pseudo "oracle" composed of a random forest regressor with access to ALL data, a Gaussian Cupola generator fitted to all unlabeled instances (minus a held-out test set), and a separate random forest learner. I've compared it to random sampling. Note the danger of overfitting is strong here since we have an identical "oracle" to a learner. However, considering the blinded implementation, and if we consider the "oracle" to purely be a high-fidelity labeler, we can see convergence improves in both time and fidelity compared to random sampling.As such, MQS is compelling when we may learn a good generative model based on ALL unlabeled data, and then spend the experimental budget purely on labeling.One other side to this: we may generate additional combinations of the design matrix (it was given as a discrete combinatorial problem).The real weakness here is the overlap in the oracle and serious data leakage. Another way I was thinking of getting around this is to use "pool"-based sampling in modAL, but really alter it significantly such that a generative model is used to either generate naïve instances to be labeled by a human oracle (implemented below), or do something like https://doi.org/10.1016/j.neucom.2014.06.042.My thought with this was to generate many many instances at each query iteration based on the underlying distribution of the unlabeled dataset, and then use something like uncertainty sampling to choose generated instance to either label by a human, or test out how doing a nearest neighbors search to an existing labeled pool could perform.  Implementing a ModAL MQS learner.- without modifying the ActiveLearner class, we can do MQS as follows: consider the active learning "pool" to be a "pool" of unlabeled instances by which we train a generative model. It is up to the user whether this is inclusive or exlcusive of instances seen by the model or not.- by default, we will use SDV and choose the<jupyter_code>from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
import copy


data = pd.read_csv('./DataPool.csv').dropna()
X_train, X_test, y_train, y_test = train_test_split(
    data.drop('Yield',1), data['Yield'], test_size=0.15, random_state=42)
data
n_initial = 100
initial_idx = np.random.choice(range(len(X_train)), size=n_initial, replace=False)

X_training, y_training = X_train.iloc[initial_idx], y_train.iloc[initial_idx]

# X_pool=copy.deepcopy(X_train.values)
# y_pool=copy.deepcopy(y_train.values)

# X_pool=np.delete(X_pool,initial_idx,axis=0)
# y_pool=np.delete(y_pool,initial_idx,axis=0)
# defining the custom query strategy, which uses the linear combination of
# classifier uncertainty and classifier margin

def fit_tabular_generator(X:pd.DataFrame, generator = None, return_fitted=True):
    '''
    Train a generator on some initial pool of [tabular] data.

    Keyword arguments:
      X -- the [tabular] feature pool to train a generator on, an np.ndarray or pd.DataFrame
      generator -- an existing SDV.tabular instance, or something with a .sample() method. Make a new one if none provided
      return_fitted -- return an instance of a fitted generator if true, else a newly instantiated model.
    '''
    if generator:
      return generator.fit(X)
    else:

      print('Creating and fitting GaussianCopula, CopulaGAN, CTGAN, and TVAE...')

      from sdv.tabular import GaussianCopula, CopulaGAN, CTGAN, TVAE
      from sdv.evaluation import evaluate

      generators = [GaussianCopula(), CopulaGAN(), CTGAN(), TVAE()]

      for model in generators:
        model.fit(X)
      
      print('Sampling data...')

      synthetic_data = [model.sample() for model in generators]

      print('Evaluating data...')

      aggregated_evaluations = [evaluate(X_synthetic, X, aggregate=True) for X_synthetic in synthetic_data]
      best_idx=np.argmax(aggregated_evaluations)#choose best performing

      print(f'Choosing best generator... \n done.\n Chose {generators[best_idx].__class__.__name__}')

      return generators[best_idx] if return_fitted else [GaussianCopula(), CopulaGAN(), CTGAN(), TVAE()][best_idx]

import warnings
warnings.filterwarnings('ignore')

# generator = fit_tabular_generator(X_training)
from sdv.tabular import GaussianCopula
generator=GaussianCopula()
generator.fit(X_training)
def ask_oracle_for_label(instance,labels,regr_or_clf='regression'):
  '''
  Query oracle for a label. Careful as this does not handle bad inputs and typing!
  '''
  if labels.dtype=='float64':
    possible_labels = labels.describe()
  else:
    possible_labels = np.unique(labels)

  oracle_label = None
  label_type=np.array(possible_labels).dtype

  # Start a loop that will run until the user enters a valid discrete label or a float.
  while not oracle_label:
      # Ask the user for a label.
      oracle_label = input(f"The possible labels for this instance should be from \n {possible_labels}.\n\n \
                        Please provide a suitable dtype {label_type} label for {instance}: ")
      
  return oracle_label
def membership_query_synthesis(generator, human_annotator = True, labels = None,regr_or_clf='regression'): ##, regr_or_clf:str='classification'):
  
    """Generate a synthetic query based on the underlying distribution of some
       inputted "pool" of tabular data, X.

       Note that this could be easily extended to other data modalities,
       such as images, by specifying your own 'generator' object

       Given the synthetic data, we either ask the user for a label
       on this instance and feed it to the learner.

       ***[WIP]***
        Another option is to
        find the nearest real instance
        per https://doi.org/10.1016/j.neucom.2014.06.042
        (only works for classification), and then query an oracle or simply
        pull from the existing labeled pool for testing purposes.


      Keyword arguments:

      X -- the pool that the generator and nearest-neighbors algorithm will draw from.

      generator -- the synthetic [tabular] data generator, based on SDV library.

      human_annotator -- boolean denoting presence of a human annotator.
                If True, asks for a label to the synthetic instance from the human/oracle.

      possible_labels -- optional list of possible labels.
                Recommend to default to unique values or range of values
                based on provided y data in learning loop.

      regr_or_clf -- string specifying whether 'regression' or 'classification' problem.
    """

    X_synthetic = generator.sample(1)
    y_oracle    = ask_oracle_for_label(instance=X_synthetic,labels=labels,regr_or_clf='regression')

    return X_synthetic,y_oracle<jupyter_output><empty_output><jupyter_text>We now can query a user for labels to a single generative instance. This is somewhat naïve. a better approach may be to generate a ton of X/y pairs, then use uncertainty sampling, etc, on those instances, then ask for true labels from the oracle for just those features.To start, here is the naïve method.<jupyter_code>from modAL import ActiveLearner
from sklearn.ensemble import RandomForestRegressor

learner = ActiveLearner( #should change, we don't actually sample.
    estimator=RandomForestRegressor(),
    query_strategy=membership_query_synthesis,
    X_training=X_training.values, y_training=y_training.values
)

# pool-based sampling --> MQS
n_queries = 5

for idx in range(n_queries):
    X_synthetic, y_oracle = membership_query_synthesis(generator,labels=y_training, regr_or_clf='regression')
    # typically, we should do learner.query(generator) here.
    # instead, we hope the generative model was useful
    learner.teach(
        X=X_synthetic.values,
        y=np.array([y_oracle]).reshape(-1,)
    )<jupyter_output>The possible labels for this instance should be from 
 count    100.00000
mean      14.29433
std        8.20773
min        0.22200
25%        8.13525
50%       13.64450
75%       21.49800
max       32.98300
Name: Yield, dtype: float64.

                         Please provide a suitable dtype float64 label for    Mg-glutamate (mM)  K-glutamate (mM)  ...  3-PGA (mM)  NTP (mM)
0           2.548255         72.746426  ...   16.425252  1.451154

[1 rows x 11 columns]: 4
The possible labels for this instance should be from 
 count    100.00000
mean      14.29433
std        8.20773
min        0.22200
25%        8.13525
50%       13.64450
75%       21.49800
max       32.98300
Name: Yield, dtype: float64.

                         Please provide a suitable dtype float64 label for    Mg-glutamate (mM)  K-glutamate (mM)  ...  3-PGA (mM)  NTP (mM)
0           3.682443         55.098293  ...    5.140474  1.017392

[1 rows x 11 columns]: 4
The possible labels for this instance should be from 
 count[...]
{'fraction_non_alphanumeric': 0.08274630419684492, 'fraction_numerical': 0.02103383272150015, 'mean_word_length': 4.469072164948454, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '547650', 'license': 'Apache-2.0', 'n_tokens_mistral': 6505, 'n_tokens_neox': 5964, 'n_words': 2063, 'path': '_notebooks/WIP/2021-04-10-mqs_regression.ipynb', 'repo_name': 'simonlevine/simonsays', 'size': '20158'}