<jupyter_start><jupyter_text>Using Deepchem DatasetsIn this tutorial we will have a look at various deepchem `dataset` methods present in `deepchem.datasets`.<jupyter_code>import deepchem as dc
import numpy as np
import random<jupyter_output>/Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
RDKit WARNING: [23:36:48] Enabling RDKit 2019.09.3 jupyter extensions
/Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Pas[...]<jupyter_text>Using NumpyDatasets The `dc.data.NumpyDatasets` class is used when you have your data in numpy arrays. It provides a simple wrapper around a collection of Numpy datasets.<jupyter_code># data is your dataset in numpy array of size : 20x20.
data = np.random.random((4, 4))
labels = np.random.random((4,)) # labels of size 20x1
from deepchem.data.datasets import NumpyDataset # import NumpyDataset
dataset = NumpyDataset(data, labels) # creates numpy dataset object
dataset<jupyter_output><empty_output><jupyter_text>Extracting X, y from NumpyDataset ObjectExtracting the data and labels from the NumpyDataset is very easy.<jupyter_code>dataset.X # Extracts the data (X) from the NumpyDataset Object

dataset.y # Extracts the labels (y) from the NumpyDataset Object<jupyter_output><empty_output><jupyter_text>Weights of a dataset - wSo apart from `X` and `y` which are the data and the labels, you can also assign weights `w` to each data instance. The dimension of `w` is same as that of `y`(which is `Nx1` where `N` is the number of data instances).**NOTE:** By default `w` is a vector initialized with equal weights (all being 1).<jupyter_code>dataset.w # printing the weights that are assigned by default. Notice that they are a vector of 1's
w = np.random.random((4,)) # initializing weights with random vector of size 20x1
dataset_with_weights = NumpyDataset(data, labels, w) # creates numpy dataset object
dataset_with_weights.w<jupyter_output><empty_output><jupyter_text>Iterating over NumpyDatasetIn order to iterate over NumpyDataset, we use `itersamples` method. We iterate over 4 quantities, namely `X`, `y`, `w` and `ids`. The first three quantities are the same as discussed above and `ids` is the id of the data instance. By default the id is given in order starting from `1`<jupyter_code>for x, y, w, id in dataset.itersamples():
    print(x, y, w, id)<jupyter_output>[0.01109257 0.62277556 0.02058281 0.83395641] 0.9069618791084421 1.0 0
[0.29591717 0.69673525 0.67001604 0.68693823] 0.45977403789888005 1.0 1
[0.84156563 0.02039639 0.83506678 0.4422977 ] 0.9692269600648693 1.0 2
[0.39966698 0.57210768 0.4434791  0.16909073] 0.24167063686752832 1.0 3<jupyter_text>You can also extract the ids by `dataset.ids`. This would return a numpy array consisting of the ids of the data instances.<jupyter_code>dataset.ids<jupyter_output><empty_output><jupyter_text>MNIST ExampleJust to get a better understanding, lets take read MNIST data and use `NumpyDataset` to store the data.<jupyter_code>from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# Load the numpy data of MNIST into NumpyDataset
train = NumpyDataset(mnist.train.images, mnist.train.labels)
valid = NumpyDataset(mnist.validation.images, mnist.validation.labels)

import matplotlib.pyplot as plt
# Visualize one sample 
sample = np.reshape(train.X[5], (28, 28))
plt.imshow(sample)
plt.show()<jupyter_output><empty_output><jupyter_text>Converting a Numpy Array to tf.data.dataset()Let's say you want to use the `tf.data` module instead of DeepChem's data handling library. Doing this is straightforward and is quite similar to getting a `NumpyDataset` object from numpy arrays.<jupyter_code>import tensorflow as tf
data_small = np.random.random((4,5))
label_small = np.random.random((4,))
dataset = tf.data.Dataset.from_tensor_slices((data_small, label_small))
print ("Data\n")
print (data_small)
print ("\n Labels")
print (label_small)<jupyter_output>Data

[[0.78113293 0.01674453 0.48489516 0.69356293 0.91605677]
 [0.70025413 0.66522493 0.03279785 0.1810656  0.34951665]
 [0.8357952  0.68600992 0.19022591 0.6087858  0.61117143]
 [0.02318132 0.85849407 0.31825101 0.83070808 0.13985736]]

 Labels
[0.41737946 0.83331863 0.89246031 0.21424502]<jupyter_text>Extracting the numpy dataset from tf.dataIn order to extract the numpy array from the `tf.data`, you first need to define an `iterator` to iterate over the `tf.data.Dataset` object and then in the tensorflow session, run over the iterator to get the data instances. Let's have a look at how it's done.<jupyter_code>iterator = dataset.make_one_shot_iterator() # iterator
next_element = iterator.get_next()
numpy_data = np.zeros((4, 5))
numpy_label = np.zeros((4,))
sess = tf.Session() # tensorflow session 
for i in range(4):
    data_, label_ = sess.run(next_element) # data_ contains the data and label_ contains the labels that we fed in the previous step
    numpy_data[i, :] = data_
    numpy_label[i] = label_
    
print ("Numpy Data")
print(numpy_data)
print ("\n Numpy Label")
print(numpy_label)<jupyter_output>WARNING:tensorflow:From <ipython-input-18-f67e6d094179>:1: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
Numpy Data
[[0.78113293 0.01674453 0.48489516 0.69356293 0.91605677]
 [0.70025413 0.66522493 0.03279785 0.1810656  0.34951665]
 [0.8357952  0.68600992 0.19022591 0.6087858  0.61117143]
 [0.02318132 0.85849407 0.31825101 0.83070808 0.13985736]]

 Numpy Label
[0.41737946 0.83331863 0.89246031 0.21424502]<jupyter_text>Now that you have the numpy arrays of `data` and `labels`, you can convert it to `NumpyDataset`.<jupyter_code>dataset_ = NumpyDataset(numpy_data, numpy_label) # convert to NumpyDataset
dataset_.X  # printing just to check if the data is same!!<jupyter_output><empty_output><jupyter_text>Converting NumpyDataset to `tf.data`This can be easily done by the `make_iterator()` method of `NumpyDataset`. This converts the `NumpyDataset` to `tf.data`. Let's look how it's done!<jupyter_code>iterator_ = dataset_.make_iterator() # Using make_iterator for converting NumpyDataset to tf.data
next_element_ = iterator_.get_next()

sess = tf.Session() # tensorflow session 
data_and_labels = sess.run(next_element_) # data_ contains the data and label_ contains the labels that we fed in the previous step


print ("Numpy Data")
print(data_and_labels[0])  # Data in the first index 
print ("\n Numpy Label")
print(data_and_labels[1])  # Labels in the second index<jupyter_output>WARNING:tensorflow:From /Users/bharath/opt/anaconda3/envs/deepchem/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
Numpy Data
[[0.02318132 0.85849407 0.31825101 0.83070808 0.13985736]
 [0.78113293 [...]
{'fraction_non_alphanumeric': 0.09370645610775662, 'fraction_numerical': 0.09359033906177427, 'mean_word_length': 5.227765726681128, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '200904', 'license': 'MIT', 'n_tokens_mistral': 3354, 'n_tokens_neox': 2868, 'n_words': 936, 'path': 'examples/notebooks/Using_DeepChem_Datasets.ipynb', 'repo_name': 'test-user-18/deepchem', 'size': '8612'}