<jupyter_start><jupyter_text>Deep500 Recipe TutorialA recipe is composed of: fixed components (that cannot change if benchmarked), mutable components (that can be tuned), and acceptable metrics.First thing's first, let's import Deep500:<jupyter_code>import deep500 as d5<jupyter_output>WARNING: mpi4py not available, distributed optimization disabled<jupyter_text>Now we will set up the fixed components of the recipe. In this example, we will use a LeNet CNN on the MNIST dataset, both of which are available in `deep500.networks` and `deep500.datasets`. We will also set an epoch budget of 10. More information may be found by browsing the available [datasets](https://github.com/deep500/deep500/tree/master/deep500/datasets) and [networks](https://github.com/deep500/deep500/tree/master/deep500/networks).<jupyter_code>fixed_components = {
    'dataset': 'mnist',
    'train_sampler': d5.ShuffleSampler,
    'model': 'simple_cnn',
    'epochs': 10
}<jupyter_output><empty_output><jupyter_text>Our benchmark may want to allow different executors (i.e., deep learning frameworks such as TensorFlow), batch sizes and optimizers. We will define them as mutable components. Before we do that, we must import an executor, for instance PyTorch:<jupyter_code>from deep500.frameworks import pytorch as d5fw
mutable_components = {
    'executor': d5fw.from_model,
    'executor_kwargs': dict(device=d5.GPUDevice()),  # Let the executor know that the default device is GPU
    'batch_size': 64,
    'optimizer': d5fw.MomentumOptimizer,  # Use a framework built-in optimizer for performance
    'optimizer_args': (0.1, 0.9) # Set learning rate to 0.1, momentum to 0.9
}<jupyter_output><empty_output><jupyter_text>Now, all that's left is to define what are our success criteria. Since MNIST is relatively easy to train, we will set 90% validation accuracy as our target:<jupyter_code>acceptable_metrics = [
    (d5.TestAccuracy(), 90)
]<jupyter_output><empty_output><jupyter_text>Let's run the training process:<jupyter_code>d5.run_recipe(fixed_components, mutable_components, acceptable_metrics)<jupyter_output>C:\Users\XL\miniconda3\envs\denv\lib\site-packages\deep500\networks\pytorch_mnist.py:11: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
  init.xavier_uniform(m.weight, gain=np.sqrt(2))
C:\Users\XL\miniconda3\envs\denv\lib\site-packages\deep500\networks\pytorch_mnist.py:12: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
  init.constant(m.bias, 0)<jupyter_text>If the dataset is not there, Deep500 will automatically download it for you (or provide instructions to set it up). As we can see, the test accuracy goes up from 5.02% before training to 92.2% on the first epoch, maxing out at around 98%. We can improve this by changing the neural network, or by adding regularization:<jupyter_code># Add "1e-4 * w" as a term to the gradient
def weight_decay(name: str, param, grad):
    grad += 1e-4 * param
    return grad

# Set the optimizer to modify the gradients to include this
mutable_components['optimizer_kwargs'] = dict(gradient_modifier=weight_decay)
d5.run_recipe(fixed_components, mutable_components, acceptable_metrics)<jupyter_output>graph(%input.1 : Float(64, 1, 28, 28),
      %conv1.weight : Float(10, 1, 5, 5),
      %conv1.bias : Float(10),
      %conv2.weight : Float(20, 10, 5, 5),
      %conv2.bias : Float(20),
      %fc1.weight : Float(50, 320),
      %fc1.bias : Float(50),
      %fc2.weight : Float(10, 50),
      %fc2.bias : Float(10)):
  %9 : Float(64, 10, 24, 24) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[5, 5], pads=[0, 0, 0, 0], strides=[1, 1]](%input.1, %conv1.weight, %conv1.bias), scope: Net/Conv2d[conv1] # C:\Users\XL\miniconda3\envs\denv\lib\site-packages\torch\nn\modules\conv.py:340:0
  %10 : Float(64, 10, 12, 12) = onnx::MaxPool[kernel_shape=[2, 2], pads=[0, 0, 0, 0], strides=[2, 2]](%9), scope: Net # C:\Users\XL\miniconda3\envs\denv\lib\site-packages\torch\nn\functional.py:487:0
  %11 : Float(64, 10, 12, 12) = onnx::Relu(%10), scope: Net # C:\Users\XL\miniconda3\envs\denv\lib\site-packages\torch\nn\functional.py:913:0
  %12 : Float(64, 20, 8, 8) = onnx::Conv[dilations=[1, 1], group=1, k[...]
{'fraction_non_alphanumeric': 0.1260345235280208, 'fraction_numerical': 0.05178529203121305, 'mean_word_length': 5.068866571018652, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '311699', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1599, 'n_tokens_neox': 1448, 'n_words': 412, 'path': '1_mnist.ipynb', 'repo_name': 'deep500/eurompi19', 'size': '4229'}