<jupyter_start><jupyter_text>Before we train a model, we need to prepare the dataset.First we download nearly 40MB examples of neutral and smiling faces.<jupyter_code>!curl -L -O https://github.com/hromi/SMILEsmileD/archive/master.zip
!unzip -q master.zip
!rm master.zip<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122    0   122    0     0     31      0 --:--:--  0:00:03 --:--:--    31
100 18.4M    0 18.4M    0     0   160k      0 --:--:--  0:01:57 --:--:--     0<jupyter_text>Then we list all the image files, and create a big list of examples as `(path, class)` tuples.<jupyter_code>from utils import list_all_files
negative_paths = list(list_all_files('SMILEsmileD-master/SMILEs/negatives/negatives7/', ['.jpg']))
print 'loaded', len(negative_paths), 'negative examples'
positive_paths = list(list_all_files('SMILEsmileD-master/SMILEs/positives/positives7/', ['.jpg']))
print 'loaded', len(positive_paths), 'positive examples'
examples = [(path, 0) for path in negative_paths] + [(path, 1) for path in positive_paths]<jupyter_output>loaded 9475 negative examples
loaded 3690 positive examples<jupyter_text>For each path, we will load the image with `imread` and resize from 64x64 to 32x32 with `block_reduce`. This can take up to **10 seconds**.<jupyter_code>import numpy as np
from skimage.measure import block_reduce
from skimage.io import imread

def examples_to_dataset(examples, block_size=2):
    X = []
    y = []
    for path, label in examples:
        img = imread(path, as_grey=True)
        img = block_reduce(img, block_size=(block_size, block_size), func=np.mean)
        X.append(img)
        y.append(label)
    return np.asarray(X), np.asarray(y)

%time X, y = examples_to_dataset(examples)<jupyter_output>/opt/anaconda3/envs/py2version/lib/python2.7/site-packages/skimage/io/_io.py:49: UserWarning: `as_grey` has been deprecated in favor of `as_gray`
  warn('`as_grey` has been deprecated in favor of `as_gray`')<jupyter_text>Then we turn `X` and `y` into numpy arrays and coerce them into the right shape and range.<jupyter_code>X = X.astype(np.float32) / 255.
y = y.astype(np.int32)
print X.dtype, X.min(), X.max(), X.shape
print y.dtype, y.min(), y.max(), y.shape<jupyter_output>float32 0.0 1.0 (13165, 32, 32)
int32 0 1 (13165,)<jupyter_text>Let's take a look at a few of the images.<jupyter_code>from utils import make_mosaic, show_array
show_array(255 * make_mosaic(X[:len(negative_paths)], 8), fmt='jpeg') # negative at the beginning
show_array(255 * make_mosaic(X[-len(positive_paths):], 8), fmt='jpeg') # positive at the end<jupyter_output><empty_output><jupyter_text>Now we save the data to disk for so we can load it quickly when working with it later. Keras likes to use data in the (rows, cols, channels) format so we make a small change to the images before saving them.<jupyter_code>X = np.expand_dims(X, axis=-1)
np.save('X.npy', X)
np.save('y.npy', y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1220949263502455, 'fraction_numerical': 0.03764320785597381, 'mean_word_length': 4.093333333333334, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '614234', 'license': 'MIT', 'n_tokens_mistral': 1156, 'n_tokens_neox': 1058, 'n_words': 315, 'path': '1 Dataset Preparation.ipynb', 'repo_name': 'yulin6666/SmileCNN', 'size': '3055'}