<jupyter_start><jupyter_text>Smiles Data To Memory Efficient Tfrecord (For GCS Bucket)* Setup the following parameters:    * **DATA_SMI='/path/to/some/dataset.smi'**    * **TFRECORD_PATH='/folder/path/to/save/tfrecords/'**    * **EVAL_SPLIT=fraction of data for validation**<jupyter_code>DATA_SMI     = '/home/ansary/RESEARCH/nCov/pySMD/data/sample_dataset_c_34_128.smi' # @param
TFRECORD_PATH= '/home/ansary/RESEARCH/nCov/pySMD/data/tfrecSMDsample/'  # @param
EVAL_SPLIT   = 0.2<jupyter_output><empty_output><jupyter_text>Imports and Fixed Parameters<jupyter_code>import os
import numpy as np
from rdkit import Chem
from tqdm.notebook import tqdm
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline
import cv2
from coreLib.utils import SmilesTokenizer,create_dir,create_records,data_input_fn
# tokenizer
tokenizer=SmilesTokenizer()
# train and eval dir
TRAIN_DIR= os.path.join(TFRECORD_PATH,'Train')
EVAL_DIR = os.path.join(TFRECORD_PATH,'Eval')

create_dir(TFRECORD_PATH)
create_dir(TRAIN_DIR)
create_dir(EVAL_DIR)<jupyter_output><empty_output><jupyter_text>Sample Data Visualization<jupyter_code>with open(DATA_SMI) as f:
    smiles = [s.rstrip() for s in f]
print('Total Smile Data:',len(smiles))    
print('Smile Example Data:',smiles[0])   
molecule=Chem.MolFromSmiles(smiles[0])
molecule<jupyter_output>Total Smile Data: 438552
Smile Example Data: O=C1Nc2cc(NC(=O)c3c[nH]cc(-c4ccc(C(F)(F)F)cc4)c3=O)ccc2C1=Cc1ccc[nH]1<jupyter_text>Saving the data* Tokenizing the smiles * Padding the tokenized smiles as per [LSTM_Chem:data_loader](https://github.com/topazape/LSTM_Chem/blob/b469e6aff2df28de99b9c4541a1a76d4d4fd77b0/lstm_chem/data_loader.py)* Saving to tfrecords<jupyter_code># tokenizing the smiles
print('Tokenizing the smiles')
tokenized_smiles = [tokenizer.tokenize(smi) for smi in tqdm(smiles)]
# Get MaxLength of smiles 
print('Getting MaxLength of smiles')
SMI_MAX_LEN=0
for smi in tqdm(tokenized_smiles):
    _smi_len=len(smi)
    if _smi_len > SMI_MAX_LEN:
        SMI_MAX_LEN=_smi_len
# padding the smiles with max length 
print('Padding the smiles with max length')
padded_smiles = [['G'] + tokenized_smi + ['E']+
                 ['A' for _ in range(SMI_MAX_LEN -len(tokenized_smi))]
                 for tokenized_smi in tqdm(tokenized_smiles)]

# data Split
eval_data_len=int(len(padded_smiles)*EVAL_SPLIT)
train_smiles= padded_smiles[eval_data_len:]
eval_smiles = padded_smiles[:eval_data_len]

# saving records
create_records(train_smiles,TRAIN_DIR,'Train',tokenizer)
create_records(eval_smiles,EVAL_DIR,'Eval',tokenizer)<jupyter_output>Tokenizing the smiles<jupyter_text>But why use ".png" bytes instead of "float32" one hot encoded data?* Simple Answer: **DataSize** Data Inspection<jupyter_code>import sys

example_smile=smiles[0]
print('Example Smile:',example_smile)
print()
tokenized=tokenizer.tokenize(example_smile)
print('Tokenized Smile:',tokenized)
print()
padded=['G'] + tokenized + ['E']+ ['A' for _ in range(SMI_MAX_LEN -len(tokenized))]
print('Padded Smile:',padded)
X = np.array([tokenizer.one_hot_dict[symbol] for symbol in padded[:-1]])
y = np.array([tokenizer.one_hot_dict[symbol] for symbol in padded[1:]])
print('One hot Encoded Input Data:',X)
print()
print('Input Shape:',X.shape)
print()
print('One hot Encoded Target Data:',y)
print()
print('Target Shape:',y.shape)
print()
print('Input Size:',sys.getsizeof(X),' bytes')
print('Target Size:',sys.getsizeof(y),' bytes')
print('Approximate Total Space needed to store the data as it is:',
      round((2*len(padded_smiles)*sys.getsizeof(X))/10**9,2),'Giga bytes')


# Data as unit8
print('float32 Data visualization')
print('Input')
plt.imshow(X)
plt.show()
print('Data Sign:',np.unique(X))
print('Target')
plt.imshow(y)
plt.show()
print('Data Sign:',np.unique(y))

X=X.astype(np.uint8)
y=y.astype(np.uint8)

print('uint8 Data visualization')
print('Input')
plt.imshow(X)
plt.show()
print('Data Sign:',np.unique(X))
print('Target')
plt.imshow(y)
plt.show()
print('Data Sign:',np.unique(y))
# Png encoded data
_,X_coded = cv2.imencode('.png',X)
_,y_coded = cv2.imencode('.png',y)
# Byte conversion
x_bytes = X_coded.tobytes()
y_bytes = y_coded.tobytes()
print('Encoded bytes datasize:Input:',sys.getsizeof(x_bytes),' bytes')
print('Encoded bytes datasize:Target:',sys.getsizeof(y_bytes),' bytes')
print('Approximate Total Space needed to store the data in .png raw bytes:',
      round((2*len(padded_smiles)*sys.getsizeof(x_bytes))/10**9,2),'Giga bytes')<jupyter_output>Encoded bytes datasize:Input: 363  bytes
Encoded bytes datasize:Target: 363  bytes
Approximate Total Space needed to store the data in .png raw bytes: 0.32 Giga bytes<jupyter_text>Data Pipeline Check<jupyter_code>BATCH_SIZE=1024
BUFFER_SIZE=2048
DATA_DIM=(SMI_MAX_LEN+1,len(tokenizer.table))


example_ds = data_input_fn(TFRECORD_PATH,'Train',DATA_DIM,BATCH_SIZE,BUFFER_SIZE)
for x,y in example_ds.take(1):
  print('Batched Input Data Shape:',x.shape)
  print('Batched Target Data Shape:',y.shape)
  print('Sample Input Data:',x[0])
  print('Sample Target Data:',y[0])

print()
print()
print()

print('The DATA_DIM variable for training:',DATA_DIM)<jupyter_output>Batched Input Data Shape: (1024, 78, 47)
Batched Target Data Shape: (1024, 78, 47)
Sample Input Data: tf.Tensor(
[[0. 0. 0. ... 1. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 1. 0.]
 [0. 0. 0. ... 0. 1. 0.]
 [0. 0. 0. ... 0. 1. 0.]], shape=(78, 47), dtype=float32)
Sample Target Data: tf.Tensor(
[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 1. 0.]
 [0. 0. 0. ... 0. 1. 0.]
 [0. 0. 0. ... 0. 1. 0.]], shape=(78, 47), dtype=float32)



The DATA_DIM variable for training: (78, 47)
{'fraction_non_alphanumeric': 0.15667074663402691, 'fraction_numerical': 0.03444658157020458, 'mean_word_length': 5.470588235294118, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '168281', 'license': 'MIT', 'n_tokens_mistral': 2396, 'n_tokens_neox': 2229, 'n_words': 493, 'path': 'data_lstm_chem.ipynb', 'repo_name': 'mnansary/pySMD', 'size': '5719'}