<jupyter_start><jupyter_text>Transfer LearningThere are various methods for transfer learning such as **fine tuning** and **frozen feature extraction**. In this tutorial, we will demonstrate how to perform a **frozen feature extraction** type of transfer learning in XenonPy.This tutorial will use **Refractive Index** data, which are collected from [Polymer Genome](https://www.polymergenome.org). We do not provide these data directly in this tutorial. If you want to rerun this notebook locally, you must collect these data yourself.  frozen feature extractionA frozen feature extraction type of transfer learing can be splitted into 2 steps:1. you need to have pre-trained model(s) as source model(s). This can be done by accessing **XenonPy.MDL**.2. you need a feature extractor to generate "neural descriptors" from the source model(s).Here, we would like to introduce you to our feature extractor, ``xenonpy.descriptor.FrozenFeaturizer``.The following codes show a case study of transfer learning between **Refractive Index** of inorganic and organic materials. In this example, the source models will be trained on inorganic compounds and the target will be polymers.Let's do this transfer learning step-by-setp.  useful functionsRunning the following cell will load some commonly used packages, such as [NumPy](https://numpy.org/), [pandas](https://pandas.pydata.org/), and so on. It will also import some in-house functions used in this tutorial. See *'tools.ipynb'* file to check what will be imported.<jupyter_code>%run tools.ipynb<jupyter_output><empty_output><jupyter_text>access pre-trained models with MDL classWe prepared a wide range of APIs to let you query and download our models.These APIs can be accessed via any HTTP requests.For convenience, we implemented some of the most popular APIs and wrapped them into XenonPy.All these functions can be accessed using `xenonpy.mdl.MDL`.<jupyter_code>from xenonpy.mdl import MDL

mdl = MDL()
mdl

mdl.version<jupyter_output><empty_output><jupyter_text>1. query **Refractive Index** models<jupyter_code>query = mdl(modelset_has="Stable", property_has="refractive")
query
summary = query('id', 'modelset', 'meanAbsError', 'pearsonCorr').sort_values('meanAbsError')
summary.head(3)<jupyter_output><empty_output><jupyter_text>2. download the best performing model based on MAE<jupyter_code>results = mdl.pull(summary.id[0].item())
results<jupyter_output>100%|██████████| 1/1 [00:00<00:00,  1.26it/s]<jupyter_text>3. load **Refractive Index** data from Polymer Genome and calculate the ``Composition`` descriptors<jupyter_code>pg = <load your polymer genome data>
pg.head(3)
from xenonpy.descriptor import Compositions

pg_desc = Compositions().transform(pg['composition'])
pg_desc.head(3)<jupyter_output><empty_output><jupyter_text>4. predict Polymer Genomer **Refractive Index** directly from a model trained on inorganic compounds<jupyter_code>from xenonpy.model.training import Checker

checker = Checker(results.model[0])
checker.checkpoints
# --- pre-trained model for prediction
from xenonpy.model.training import Trainer

trainer = Trainer.load(from_=checker)
trainer
trainer.reset(to='mae_1')

y_pred = trainer.predict(x_in=torch.tensor(pg_desc.values, dtype=torch.float)).detach().numpy().flatten()
y_true = pg['Refractive Index'].values

draw(y_true, y_pred, prop_name='Refractive Index')<jupyter_output>Missing directory and/or file name information!<jupyter_text>5. frozen feature extraction``FrozenFeaturizer`` accepts a [Pytorch](https://pytorch.org) model as its input.<jupyter_code>from xenonpy.descriptor import FrozenFeaturizer

# --- init FrozenFeaturizer with NN model
ff = FrozenFeaturizer(model=trainer.model)
ff<jupyter_output><empty_output><jupyter_text>The following code will generate new "neural descriptors" from the corresponding neural network model.<jupyter_code>neural_descriptors = ff.transform(pg_desc, depth=2 ,return_type='df')<jupyter_output><empty_output><jupyter_text>Here, ``depth=x`` means that the last x hidden layer from the output neuron(s) will be concatenated and used as the neural descriptor.<jupyter_code>neural_descriptors.head(3)<jupyter_output><empty_output><jupyter_text>As an example, **-1** in the column names denotes the last layer.<jupyter_code>from xenonpy.visualization import DescriptorHeatmap

sorted_prop = pg['Refractive Index'].sort_values()
sorted_desc = neural_descriptors.loc[sorted_prop.index]

heatmap = DescriptorHeatmap( 
        bc=True,  # use box-cox transform 
#         save=dict(fname='heatmap_density', dpi=150, bbox_inches='tight'),  # save fingure to file
        figsize=(70, 10))
heatmap.fit(sorted_desc)
heatmap.draw(sorted_prop)<jupyter_output><empty_output><jupyter_text>6. use neural descriptors to train new models.In this case, **Random Forest** model and **Bayesian Ridge Linear** model will be trained.<jupyter_code># split data
from xenonpy.datatools import Splitter

y = pg['Refractive Index']
splitter = Splitter(len(y), test_size=0.2)

X_train, X_test, y_train, y_test = splitter.split(neural_descriptors, y.values.reshape(-1, 1))
# random forest
from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(n_estimators=100)
rf.fit(X_train, y_train.ravel())
y_pred = rf.predict(X_test)
y_fit_pred = rf.predict(X_train)
draw(y_test.ravel(), y_pred, y_train.ravel(), y_fit_pred, prop_name='refractive index')
# bayesian linear
from sklearn.linear_model import BayesianRidge

br = BayesianRidge()
br.fit(X_train, y_train.ravel())
y_pred = br.predict(X_test)
y_fit_pred = br.predict(X_train)
draw(y_test.ravel(), y_pred, y_train.ravel(), y_fit_pred, prop_name='refractive index')<jupyter_output>Missing directory and/or file name information!
{'fraction_non_alphanumeric': 0.10290514525726287, 'fraction_numerical': 0.00840042002100105, 'mean_word_length': 6.660857908847185, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 43, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '127160', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1889, 'n_tokens_neox': 1785, 'n_words': 536, 'path': 'samples/transfer_learning.ipynb', 'repo_name': 'chouki-zhang/Digit-chemical-lab', 'size': '5714'}