<jupyter_start><jupyter_text>USPTO data Exploration> Looking into the USPTOyield data<jupyter_code># hide 
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline 
# hide
default_plot_kwargs = {
    'color': '#541FA3',
}
title_fontsize = 18
label_fontsize = 16<jupyter_output><empty_output><jupyter_text>Getting the data filesThe USPTO yields data files are too large to be uploaded to this GitHub repo and have to be downloaded from this [Box folder](https://ibm.ent.box.com/v/uspto-yields-data) and placed into `../data/uspto`. The data was extracted from the data set published by Lowe [Chemical reactions from US patents (1976-Sep2016)](https://figshare.com/articles/Chemical_reactions_from_US_patents_1976-Sep2016_/5104873), which has catalized research on ML for chemical reactions in recent years.  USPTO data - mass scalesWe observed that `milligram scale` and `gram scale` reactions had different yield distributions. Gram scale reaction are often optimized, whereas milligram scale are not optimized and yields might be often lower. In academia predicting the non-optimized yield is more relevant. Therefore, we splitted the USPTO reactions, which contained yield and product mass information into two categories (gram, milligram) at a threshold of 1 gram.<jupyter_code># data
gram_df = pd.read_csv('../data/uspto/yields_above_200622.tsv', sep='\t', index_col=0)
gram_df['calc_yield'] = gram_df.calc_yield_above

milligram_df = pd.read_csv('../data/uspto/yields_below_200622.tsv', sep='\t', index_col=0)
milligram_df['calc_yield'] = milligram_df.calc_yield_below
# data
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))
fig.suptitle('Yield distributions - USPTO', fontsize=title_fontsize)
ax1.set_ylabel('Count', fontsize=label_fontsize)
ax1.set_xlabel('Yields [%] - gram scale', fontsize=label_fontsize)
gram_df.calc_yield.hist(bins=20, ax=ax1, **default_plot_kwargs)
ax2.set_xlabel('Yields [%] - milligram scale', fontsize=label_fontsize)
milligram_df.calc_yield.hist(bins=20, ax=ax2, **default_plot_kwargs)
# hide
# data
# fig.tight_layout()
fig.savefig('images/uspto_yield_distributions.pdf')
fig.savefig('images/uspto_yield_distributions.png')
# data
pd.concat([gram_df['calc_yield'].describe(),milligram_df['calc_yield'].describe()], axis=1)<jupyter_output><empty_output><jupyter_text>Prepare the data<jupyter_code># data
milligram_df['rxn'] = milligram_df.tokenized_reactants_inputs.str.replace(' ', '') + '>>' + milligram_df.tokenized_products_inputs.str.replace(' ', '')
milligram_df['scaled_yield'] = milligram_df.calc_yield / 100
milligram_train_random_split_df = milligram_df[milligram_df.random_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
milligram_test_random_split_df = milligram_df[milligram_df.random_split=='test'][['rxn', 'scaled_yield']]
milligram_train_time_split_df = milligram_df[milligram_df.time_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
milligram_test_time_split_df = milligram_df[milligram_df.time_split=='test'][['rxn', 'scaled_yield']]
#hide
# data
milligram_train_random_split_df.to_csv('../data/uspto/milligram_train_random_split.tsv', sep='\t')
milligram_test_random_split_df.to_csv('../data/uspto/milligram_test_random_split.tsv', sep='\t')
milligram_train_time_split_df.to_csv('../data/uspto/milligram_train_time_split.tsv', sep='\t')
milligram_test_time_split_df.to_csv('../data/uspto/milligram_test_time_split.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>Gram scale<jupyter_code># data
gram_df['rxn'] = gram_df.tokenized_reactants_inputs.str.replace(' ', '') + '>>' + gram_df.tokenized_products_inputs.str.replace(' ', '')
gram_df['scaled_yield'] = gram_df.calc_yield / 100
gram_train_random_split_df = gram_df[gram_df.random_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
gram_test_random_split_df = gram_df[gram_df.random_split=='test'][['rxn', 'scaled_yield']]
gram_train_time_split_df = gram_df[gram_df.time_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
gram_test_time_split_df = gram_df[gram_df.time_split=='test'][['rxn', 'scaled_yield']]
#hide
# data
gram_train_random_split_df.to_csv('../data/uspto/gram_train_random_split.tsv', sep='\t')
gram_test_random_split_df.to_csv('../data/uspto/gram_test_random_split.tsv', sep='\t')
gram_train_time_split_df.to_csv('../data/uspto/gram_train_time_split.tsv', sep='\t')
gram_test_time_split_df.to_csv('../data/uspto/gram_test_time_split.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>USPTO - smoothed<jupyter_code># data
gram_df = pd.read_csv('../data/uspto/yields_above_200622.tsv', sep='\t', index_col=0)
gram_smoothed_df = pd.read_csv('../data/uspto/yields_above_smooth_200622.tsv', sep='\t', index_col=0) 
gram_df['calc_yield'] = gram_smoothed_df['4NN-2'].values
milligram_df = pd.read_csv('../data/uspto/yields_below_200622.tsv', sep='\t', index_col=0)
milligram_smoothed_df = pd.read_csv('../data/uspto/yields_below_smooth_200622.tsv', sep='\t', index_col=0)

milligram_df['calc_yield'] = milligram_smoothed_df['4NN-2'].values
# data
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))
fig.suptitle('Yield distributions - USPTO smoothed 4NN-2', fontsize=title_fontsize)
ax1.set_ylabel('Count', fontsize=label_fontsize)
ax1.set_xlabel('Yields [%] - gram scale', fontsize=label_fontsize)
gram_df.calc_yield.hist(bins=20, ax=ax1, **default_plot_kwargs)
ax2.set_xlabel('Yields [%] - milligram scale', fontsize=label_fontsize)
milligram_df.calc_yield.hist(bins=20, ax=ax2, **default_plot_kwargs)
# hide
# data
# fig.tight_layout()
fig.savefig('images/uspto_yield_distributions_4nn-2.pdf')
fig.savefig('images/uspto_yield_distributions_4nn-2.png')
#hide
# data

milligram_df['rxn'] = milligram_df.tokenized_reactants_inputs.str.replace(' ', '') + '>>' + milligram_df.tokenized_products_inputs.str.replace(' ', '')
milligram_df['scaled_yield'] = milligram_df.calc_yield / 100
milligram_train_random_split_df = milligram_df[milligram_df.random_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
milligram_test_random_split_df = milligram_df[milligram_df.random_split=='test'][['rxn', 'scaled_yield']]
milligram_train_time_split_df = milligram_df[milligram_df.time_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
milligram_test_time_split_df = milligram_df[milligram_df.time_split=='test'][['rxn', 'scaled_yield']]


milligram_train_random_split_df.to_csv('../data/uspto/milligram_smooth_train_random_split.tsv', sep='\t')
milligram_test_random_split_df.to_csv('../data/uspto/milligram_smooth_test_random_split.tsv', sep='\t')
milligram_train_time_split_df.to_csv('../data/uspto/milligram_smooth_train_time_split.tsv', sep='\t')
milligram_test_time_split_df.to_csv('../data/uspto/milligram_smooth_test_time_split.tsv', sep='\t')
#hide
# data
gram_df['rxn'] = gram_df.tokenized_reactants_inputs.str.replace(' ', '') + '>>' + gram_df.tokenized_products_inputs.str.replace(' ', '')
gram_df['scaled_yield'] = gram_df.calc_yield / 100
gram_train_random_split_df = gram_df[gram_df.random_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
gram_test_random_split_df = gram_df[gram_df.random_split=='test'][['rxn', 'scaled_yield']]
gram_train_time_split_df = gram_df[gram_df.time_split=='train'][['rxn', 'scaled_yield']].sample(frac=1., random_state=42)
gram_test_time_split_df = gram_df[gram_df.time_split=='test'][['rxn', 'scaled_yield']]

gram_train_random_split_df.to_csv('../data/uspto/gram_smooth_train_random_split.tsv', sep='\t')
gram_test_random_split_df.to_csv('../data/uspto/gram_smooth_test_random_split.tsv', sep='\t')
gram_train_time_split_df.to_csv('../data/uspto/gram_smooth_train_time_split.tsv', sep='\t')
gram_test_time_split_df.to_csv('../data/uspto/gram_smooth_test_time_split.tsv', sep='\t')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1548223350253807, 'fraction_numerical': 0.019416243654822335, 'mean_word_length': 8.610975609756098, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '77434', 'license': 'MIT', 'n_tokens_mistral': 3214, 'n_tokens_neox': 3025, 'n_words': 386, 'path': 'nbs/02_uspto_data_exploration.ipynb', 'repo_name': 'himanshubhansali/rxn_yields', 'size': '7880'}