<jupyter_start><jupyter_text>Please run the following block to ensure TVM is setup for *this notebook*.  Each notebook may have its own runtime.<jupyter_code>try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False

if IN_COLAB:
    ! gsutil cp "gs://tvm-fcrc-binariesd5fce43e-8373-11e9-bfb6-0242ac1c0002/tvm.tar.gz" /tmp/tvm.tar.gz
    ! mkdir -p /tvm
    ! tar -xf /tmp/tvm.tar.gz --strip-components=4 --directory /tvm
    ! ls -la /tvm
    ! bash /tvm/package.sh
    # Add TVM to the Python path.
    import sys
    sys.path.append('/tvm/python')
    sys.path.append('/tvm/topi/python')
    ! pip install mxnet
else:
    print('Notebook executing locally, skipping Colab setup ...')<jupyter_output>Copying gs://tvm-fcrc-binariesd5fce43e-8373-11e9-bfb6-0242ac1c0002/tvm.tar.gz...
\ [1 files][119.5 MiB/119.5 MiB]                                                
Operation completed over 1 objects/119.5 MiB.                                    
total 164
drwxr-xr-x 21 root root  4096 Jun 22 17:54 .
drwxr-xr-x  1 root root  4096 Jun 22 17:54 ..
drwx------  8 root root  4096 May 31 08:14 3rdparty
drwx------ 12 root root  4096 May 31 08:14 apps
drwx------  3 root root  4096 Jun 19 07:58 build
drwx------  4 root root  4096 May 31 08:14 cmake
-rw-------  1 root root 11053 Jun 19 04:54 CMakeLists.txt
drwx------  6 root root  4096 May 31 08:14 conda
-rw-------  1 root root  5736 Jun 19 04:54 CONTRIBUTORS.md
drwx------  3 root root  4096 May 31 08:14 docker
drwx------ 11 root root  4096 May 31 08:14 docs
drwx------  4 root root  4096 May 31 08:14 golang
drwx------  3 root root  4096 May 31 08:14 include
-rw-------  1 root root 10607 Jun 19 04:54 Jenkinsfile
drwx------  6 root root  4096 May 31 [...]<jupyter_text>μTVM: Deep Learning on Bare-Metal Devices**Authors:** [<NAME>](https://github.com/weberlo) (<EMAIL>) and [<NAME>](https://homes.cs.washington.edu/~patelp1/) (<EMAIL>)<jupyter_code>import os
import pprint

import numpy as np
import tvm
from tvm.contrib import graph_runtime, util
from tvm import relay
import tvm.micro as micro<jupyter_output><empty_output><jupyter_text>The proliferation of low-cost, AI-powered IoT devices has led to widespread interest in running ML on "bare-metal" (lacking an operating system) devices among ML researchers and practitioners. On the other hand, computer architects building custom ML hardware need a way to rapidly experiment with and optimize an increasing variety of ML models on their hardware. μTVM brings rapid deployment and optimization of deep learning code generated by TVM to such bare-metal devices.This tutorial aims to provide a broad understanding of how μTVM works, and primarily focuses on the steps you need to follow to support μTVM on your own bare-metal device.In this notebook, we're going to cover:*   the C code generation backend*   the cross-compiler interface*   the low-level device interface*   the graph runtime*   a demo of ResNeT-18 using the graph runtime **Note**: We have only tested μTVM using a Linux host machine.  It may or may not work on Mac.**Disclaimer**: μTVM is still experimental and its API may change in the future.  C CodegenIf we're going to work with bare-metal devices, we need some type of code generation backend that targets them.  At first glance, LLVM seems like a great option, but unfortunately, we can't use it because bare-metal devices often lack LLVM support.However, most bare-metal devices ***do*** support C.  We have taken advantage of this fact by developing a C code generation backend for TVM.  With this backend, any programs expressible in TVM's low-level IR can be compiled into C.Note that although we made this choice because most devices can run C code, it is still possible to use LLVM or any other code generation backend supported by TVM if your target bare-metal device supports it.<jupyter_code># Builds a C source module from a Relay function.
def build_c_module(func):
  """Builds a C source module by compiling `func` with Relay."""
  # Note: μTVM currently does not support vectorized instructions.
  with tvm.build_config(disable_vectorize=True):
    # We set `target` to "c", and the build is now routed to the C backend.
    _, src_mod, _ = relay.build(func, target='c', params={})
  return src_mod<jupyter_output><empty_output><jupyter_text>Now, let's write a simple Relay program to add two vectors and examine the generated C code.<jupyter_code># First, we use Relay to construct our function which computes `x + y`.
ty = relay.TensorType(shape=(1024,), dtype='float32')
x = relay.var('x', ty)
y = relay.var('y', ty)
func = relay.Function([x, y], relay.add(x, y))
print(func)
# Then build it into a C source module.
src_mod = build_c_module(func)
# Now, we can see the source that was generated.
print(src_mod.get_source())<jupyter_output>#include "tvm/runtime/c_runtime_api.h"
#include "tvm/runtime/c_backend_api.h"
#include "tvm/runtime/micro/utvm_device_lib.h"
extern void* __tvm_module_ctx = NULL;
#ifdef __cplusplus
extern "C"
#endif
TVM_DLL int32_t fused_add( void* args,  void* arg_type_ids, int32_t num_args) {
  if (!((num_args == 3))) {
    TVMAPISetLastError("fused_add: num_args should be 3");
    return -1;
  }
  void* arg0 = (((TVMValue*)args)[0].v_handle);
  int32_t arg0_code = (( int32_t*)arg_type_ids)[0];
  void* arg1 = (((TVMValue*)args)[1].v_handle);
  int32_t arg1_code = (( int32_t*)arg_type_ids)[1];
  void* arg2 = (((TVMValue*)args)[2].v_handle);
  int32_t arg2_code = (( int32_t*)arg_type_ids)[2];
  float* placeholder = (float*)(((TVMArray*)arg0)[0].data);
  int64_t* arg0_shape = (int64_t*)(((TVMArray*)arg0)[0].shape);
  int64_t* arg0_strides = (int64_t*)(((TVMArray*)arg0)[0].strides);
  if (!(arg0_strides == NULL)) {
    if (!((1 == ((int32_t)arg0_strides[0])))) {
      TVMAPISetLastError("arg0.strides: e[...]<jupyter_text>The Cross-Compiler InterfaceWith the ability to generate C, we need access to a cross-compiler toolchain for our device, in order to create binaries for it.How you obtain the toolchain will vary, depending on the device you're targetting.  Once you have a toolchain, you need to make sure it's accessible from a terminal by adding it to your system's PATH variable.All of the binaries in the toolchain should have a common prefix.  For example, if you're targetting RISC-V, you might want "gcc" to be named "riscv-gcc", "ld" to be named "riscv-ld", etc., and in this case, the prefix is "riscv-".  Having a common prefix is important, because we use this prefix to tell μTVM which binaries to use for binary compilation/manipulation.In the example below, we create a session using the host emulated device type, which creates a region of memory on the host machine to emulate device memory.  Because this device type is running on the host machine, we don't need a cross-compiler, and in fact, we can use plain old "gcc".  In this case, there is no prefix on the toolchain, so we use the empty string.<jupyter_code>device_type = 'host'
toolchain_prefix = ''
with micro.Session(device_type, toolchain_prefix) as sess:
  # Do μTVM things...
  pass<jupyter_output><empty_output><jupyter_text>The Low-Level Device InterfaceNow we have the ability to generate binaries for our device, but we don't yet have a method of loading and running these binaries onto the device.  And even if we could run it, we couldn't read the outputs.  To satisfy these requirements, we require an implementation of a low-level device interface for every new device in μTVM.  The low-level device interface exposes three important operations to interact with the bare-metal device: - Reading from device memory- Writing to device memory- Starting function executionThis interface appears in C++ asIn the above interface, `DevBaseOffset` is a wrapper type for a `uintptr_t` that represents an offset from the base address of the device.  The full definition of the inteface can be found [here](https://github.com/uwsampl/tvm/blob/tutorial/src/runtime/micro/low_level_device.h).To add support for your own MicroDevice in μTVM, you simply need to implement the low-level device interface for your MicroDevice.  Doing so will require some form of communication with the device.  For instance, communication may occur over [JTAG](https://en.wikipedia.org/wiki/JTAG) or over a network connection. Host Low-Level DeviceTo give an example of an implementation of this interface, we will walk through the already-existing `HostLowLevelDevice` (found [here](https://github.com/uwsampl/tvm/blob/tutorial/src/runtime/micro/host_low_level_device.cc)), because it's implementation is relatively simple.This device uses an allocated region of memory on the host machine to simulate a bare-metal device. It provides μTVM users with an easy setup to experiment with, without having to communicate with and debug external MicroDevices. ConstructorTo create a host device, we call `mmap` to allocate a memory region that is at least as large as the requested device size, and we assign read/write/execute permissions to that region.  The base address of the device is just the start address of the region. DestructorTo destroy a host device, we simply unmap the `mmap`'d region. ReadTo read from the host device, we convert the device offset into a host virtual address, then use `memcpy` to transfer from the mapped memory into the output buffer `buf`. WriteWriting to the host device is similar to reading, except now `buf` serves as an input buffer. ExecuteFor function execution, we are given an offset to the target function, but also a breakpoint indicating the memory offset at which code execution should stop.We don't need to use `breakpoint` here, because with the host device, control flow never leaves the host machine, so we can simply call the function as we would any other function.For real devices, we may need to use `breakpoint` to stop execution, because otherwise, the device may continue executing indefinitely.  Graph Runtime ExecutionThe primary execution strategy for running models with μTVM is TVM's graph runtime.When we use the graph runtime, the host drives the control flow of the model.  For each node in the graph, the host calls out to the device to execute the corresponding operator for that node. Such a split execution framework is useful because it allows you to optimize kernels for your target bare-metal device easily using AutoTVM.Note that this means models do not run entirely on the device (i.e., they aren't self-hosted).  Fully self-hosted models are not currently supported, as this would require using the Relay ahead-of-time compiler, which is still experimental.<jupyter_code>shape = (1024,)
dtype = 'float32'
    
# Construct Relay program.
x = relay.var('x', relay.TensorType(shape=shape, dtype=dtype))
xx = relay.multiply(x, x)
z = relay.add(xx, relay.const(1.0))
func = relay.Function([x], z)
print(func)
with micro.Session('host', '') as sess:
  mod, params = sess.build(func)
  mod.set_input(**params)

  x_in = np.random.uniform(size=shape[0]).astype(dtype)
  print(f'input:\t\t\t{x_in}')
  # Run with `x_in` as the input. 
  mod.run(x=x_in)
  result = mod.get_output(0).asnumpy()

  expected = (x_in * x_in) + 1.0
  print()
  print(f'expected result:\t{expected}')
  print(f'μTVM result:\t\t{result}')
  print()
  if np.array_equal(expected, result):
    print('Looks good!')
  else:
    print('Something\'s wrong!')<jupyter_output>input:			[0.56248146 0.27734974 0.0507655  ... 0.6762357  0.9093801  0.2733077 ]

expected result:	[1.3163854 1.0769229 1.0025772 ... 1.4572947 1.8269721 1.0746971]
μTVM result:		[1.3163854 1.0769229 1.0025772 ... 1.4572947 1.8269721 1.0746971]

Looks good!<jupyter_text>ResNet-18 Demo<jupyter_code>import io
import mxnet as mx
from mxnet.gluon.model_zoo.vision import get_model
from mxnet.gluon.utils import download
from PIL import Image
import IPython.display

RESNET_INPUT_IMG_SHAPE = (1, 3, 224, 224)

def get_image():
  img_name = 'cat.png'
  download('https://github.com/dmlc/mxnet.js/blob/master/data/cat.png?raw=true',
           img_name)
  image = Image.open(img_name)
  return image


def show_image(image):
  img_byte_arr = io.BytesIO()
  image.save(img_byte_arr, format='png')
  img_byte_arr = img_byte_arr.getvalue()
  IPython.display.display(IPython.display.Image(img_byte_arr))


def convert_image(image):
  dtype = 'float32'
  image = image.resize(RESNET_INPUT_IMG_SHAPE[2:])
  image = np.array(image) - np.array([123., 117., 104.])
  image /= np.array([58.395, 57.12, 57.375])
  image = image.transpose((2, 0, 1))
  image = image[np.newaxis, :]
  image = tvm.nd.array(image.astype(dtype))
  return image


def get_resnet():    
  block = get_model('resnet18_v1', pretrained=True)
  module, params = relay.frontend.from_mxnet(
    block, shape={'data': RESNET_INPUT_IMG_SHAPE})
  func = module['main']
  return func, params
  
  
# Fetch a mapping from class IDs to human-readable labels.
synset_url = ''.join(['https://gist.githubusercontent.com/zhreshold/',
                      '4d0b62f3d01426887599d4f7ede23ee5/raw/',
                      '596b27d23537e5a1b5751d2b0481ef172f58b539/',
                      'imagenet1000_clsid_to_human.txt'])
synset_name = 'synset.txt'
download(synset_url, synset_name)
with open(synset_name) as f:
  synset = eval(f.read())

    
def get_predictions(mod):
  output = mod.get_output(0).asnumpy()[0]
  top_k = output.argsort()[-3:][::-1]
  predictions = list(map(lambda x: synset[x], top_k))
  return predictions<jupyter_output>Downloading synset.txt from https://gist.githubusercontent.com/zhreshold/4d0b62f3d01426887599d4f7ede23ee5/raw/596b27d23537e5a1b5751d2b0481ef172f58b539/imagenet1000_clsid_to_human.txt...<jupyter_text>It's great that we can prototype models and immediately run them on a device, but the function we just ran was pretty simple.  Let's try a more interesting example.In this section, we will run ResNet-18 on μTVM to perform image recognition.  And our example input is...<jupyter_code>image = get_image()
show_image(image)
# Convert image to a format the model can understand.
image = convert_image(image)<jupyter_output>Downloading cat.png from https://github.com/dmlc/mxnet.js/blob/master/data/cat.png?raw=true...<jupyter_text>...a cat!Luckily, MxNet is one of the many frameworks that TVM can import models from, so we can use a pretrained ResNet-18 model from the Gluon Model Zoo.Below, we grab a relay function that expresses our model, along with its set of pretrained parameters.<jupyter_code>resnet, params = get_resnet()
print(resnet)
print()
pprint.pprint(list(params.keys()))<jupyter_output>Downloading /root/.mxnet/models/resnet18_v1-a0666292.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/resnet18_v1-a0666292.zip...
v0.0.1
fn (%data: Tensor[(1, 3, 224, 224), float32], %resnetv10_conv0_weight: Tensor[(64, 3, 7, 7), float32], %resnetv10_batchnorm0_gamma: Tensor[(64,), float32], %resnetv10_batchnorm0_beta: Tensor[(64,), float32], %resnetv10_batchnorm0_running_mean: Tensor[(64,), float32], %resnetv10_batchnorm0_running_var: Tensor[(64,), float32], %resnetv10_stage1_conv0_weight: Tensor[(64, 64, 3, 3), float32], %resnetv10_stage1_batchnorm0_gamma: Tensor[(64,), float32], %resnetv10_stage1_batchnorm0_beta: Tensor[(64,), float32], %resnetv10_stage1_batchnorm0_running_mean: Tensor[(64,), float32], %resnetv10_stage1_batchnorm0_running_var: Tensor[(64,), float32], %resnetv10_stage1_conv1_weight: Tensor[(64, 64, 3, 3), float32], %resnetv10_stage1_batchnorm1_gamma: Tensor[(64,), float32], %resnetv10_stage1_batchnorm1_beta: Tensor[(64,), float32], %re[...]<jupyter_text>With that, we can create a μTVM session, build a graph runtime module, then run the model.<jupyter_code>with micro.Session('host', '') as sess:
  print('Building...')
  module, params = sess.build(resnet, params=params)
  module.set_input(**params)
  print()
  print('Running...', end='')
  # Execute with `image` as the input.
  module.run(data=image)
  print()
  print('Predictions')
  for i, prediction in enumerate(get_predictions(module)):
    print(f'  {i + 1}: {prediction}')<jupyter_output>Building...
{'fraction_non_alphanumeric': 0.10405523075009329, 'fraction_numerical': 0.046585396193556415, 'mean_word_length': 4.402889784946237, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '462404', 'license': 'Apache-2.0', 'n_tokens_mistral': 5643, 'n_tokens_neox': 5128, 'n_words': 1837, 'path': 'notebook/06_TVM_Tutorial_MicroTVM.ipynb', 'repo_name': 'deepakbabel23/tutorial', 'size': '16127'}