The Deep Learning Compiler: A Comprehensive Survey <s> Deep Learning Frameworks <s> Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models. The framework is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures. Caffe fits industry and internet-scale media needs by CUDA GPU computation, processing over 40 million images a day on a single K40 or Titan GPU ($\approx$ 2.5 ms per image). By separating model representation from actual implementation, Caffe allows experimentation and seamless switching among platforms for ease of development and deployment from prototyping machines to cloud environments. Caffe is maintained and developed by the Berkeley Vision and Learning Center (BVLC) with the help of an active community of contributors on GitHub. It powers ongoing research projects, large-scale industrial applications, and startup prototypes in vision, speech, and multimedia. <s> BIB001 </s> The Deep Learning Compiler: A Comprehensive Survey <s> Deep Learning Frameworks <s> Deep learning models with convolutional and recurrent networks are now ubiquitous and analyze massive amounts of audio, image, video, text and graph data, with applications in automatic translation, speech-to-text, scene understanding, ranking user preferences, ad placement, etc. Competing frameworks for building these networks such as TensorFlow, Chainer, CNTK, Torch/PyTorch, Caffe1/2, MXNet and Theano, explore different tradeoffs between usability and expressiveness, research or production orientation and supported hardware. They operate on a DAG of computational operators, wrapping high-performance libraries such as CUDNN (for NVIDIA GPUs) or NNPACK (for various CPUs), and automate memory allocation, synchronization, distribution. Custom operators are needed where the computation does not fit existing high-performance library calls, usually at a high engineering cost. This is frequently required when new operators are invented by researchers: such operators suffer a severe performance penalty, which limits the pace of innovation. Furthermore, even if there is an existing runtime call these frameworks can use, it often doesn't offer optimal performance for a user's particular network architecture and dataset, missing optimizations between operators as well as optimizations that can be done knowing the size and shape of data. Our contributions include (1) a language close to the mathematics of deep learning called Tensor Comprehensions offering both imperative and declarative styles, (2) a polyhedral Just-In-Time compiler to convert a mathematical description of a deep learning DAG into a CUDA kernel with delegated memory management and synchronization, also providing optimizations such as operator fusion and specialization for specific sizes, (3) a compilation cache populated by an autotuner. [Abstract cutoff] <s> BIB002 </s> The Deep Learning Compiler: A Comprehensive Survey <s> Deep Learning Frameworks <s> Frameworks for writing, compiling, and optimizing deep learning (DL) models have recently enabled progress in areas like computer vision and natural language processing. Extending these frameworks to accommodate the rapidly diversifying landscape of DL models and hardware platforms presents challenging tradeoffs between expressivity, composability, and portability. We present Relay, a new compiler framework for DL. Relay's functional, statically typed intermediate representation (IR) unifies and generalizes existing DL IRs to express state-of-the-art models. The introduction of Relay's expressive IR requires careful design of domain-specific optimizations, addressed via Relay's extension mechanisms. Using these extension mechanisms, Relay supports a unified compiler that can target a variety of hardware platforms. Our evaluation demonstrates Relay's competitive performance for a broad class of models and devices (CPUs, GPUs, and emerging accelerators). Relay's design demonstrates how a unified IR can provide expressivity, composability, and portability without compromising performance. <s> BIB003
In this sub-section, we provide an overview of popular DL frameworks. The discussion might not be exhaustive but is meant to provide a guideline fo DL practitioners. Figure 1 , presents the landscape of DL frameworks including currently popular frameworks, historical frameworks and ONNX supported frameworks. TensorFlow -TensorFlow was originally released by Google in late 2015. Among all the DL frameworks, TensorFlow has the most comprehensive support for language interfaces, including C ++, Python, Java, Go, R, and Haskell. TensorFlow employs a dataflow graph of primitive operators extended with restricted control edges to represent differentiable programs BIB003 . TensorFlow Lite is designed for mobile and embedded deep learning and provides an Android neural network API. To reduce the complexity of using TensorFlow, Google added Keras as a frontend to the TensorFlow core. Furthermore, TensorFlow eager-mode applies an approach similar to PyTorch to support dynamic computation graphs better. Keras -Keras [32] was first released in March 2015 with Google Support. Keras is a high-level neural network library for quickly building DL models, written in pure Python. Though not a DL framework on its own, Keras provides a high-level API that integrates with TensorFlow, MXNet, Theano, and CNTK. With Keras, DL developers can build a neural network with just a few lines of code. Besides, Keras can integrate other common DL packages, such as scikit-learn for Python. However, Keras is not flexible enough due to over-encapsulation, which makes it too difficult to add operations or obtain low-level data information. PyTorch -PyTorch was launched by Facebook in early 2017. Facebook rewrote the Lua-based DL framework Torch in Python and refactored all modules on Tensor level. Besides, As the most popular dynamic framework, PyTorch embeds primitives for constructing dynamic data flow graphs in Python, where the control flow is executed in the Python interpreter. PyTorch 1.0 integrated the codebases of PyTorch 0.4 and Caffe2 to create a unified framework. This allows PyTorch to absorb the benefits of Caffe2 to support efficient graph execution and mobile deployment. FastAI [50] is an advanced API layer based on PyTorch's upper-layer encapsulation. It fully borrows Keras to improve PyTorch's ease of use. Caffe/Caffe2 -Caffe BIB001 was designed for deep learning and image classification at UC Berkeley in 2014. Caffe supports the command line, Python, and MATLAB APIs. An important feature of Caffe is the ability to train and deploy models without writing code. Caffe's simple framework makes its code easy to extend, suitable for developers to analyze in-depth. Therefore, Caffe is mainly positioned in research, which has made it popular from the beginning to the present. Caffe2 is built upon the original Caffe project, supporting both mobile (e.g., iOS and Android) and server (e.g., Linux, Windows, and Mac) build platforms. Caffe2 is similar in structure to TensorFlow, albeit with a lighter API and making access to the intermediate results in the computation graph much easier BIB002 .