There is a perceived trade-off between machine learning code that is easy to write, and machine learning code that is scalable or fast to execute. In machine learning, imperative style libraries like Autograd and PyTorch are easy to write, but suffer from high interpretive overhead and are not easily deployable in production or mobile settings. Graph-based libraries like TensorFlow and Theano benefit from whole-program optimization and can be deployed broadly, but make expressing complex models more cumbersome. We describe how the use of staged programming in Python, via source code transformation, offers a midpoint between these two library design patterns, capturing the benefits of both. A key insight is to delay all type-dependent decisions until runtime, via dynamic dispatch. We instantiate these principles in AutoGraph, a software system that improves the programming experience of the TensorFlow library, and demonstrate usability improvements with no loss in performance compared to native TensorFlow graphs. We also show that our system is backend agnostic, and demonstrate targeting an alternate IR with characteristics not found in TensorFlow graphs.
Providing easier deferred execution using staged programming or multiple dispatch has a long history. Notable examples include Lightweight Modular Staging's type-based deferred execution model @cite_8 , the paired use of Lua and Terra to stage high-performance numerical code @cite_22 , and Julia's multiple dispatch system @cite_19 . Libraries implementing or using code rewriting in Python have been in limited use, including the privacy- and confidentiality-aware Jeeves system @cite_7 , which relies on MacroPy @cite_17 , as well as the Hy system, a Lisp dialect (which includes the macro facilities of Lisp) embedded in Python @cite_6 . However, each of these approaches alone, without substantial modification, are inappropriate for the Python language.
@cite_22 @cite_7 @cite_8 @cite_6 @cite_19 @cite_17
2154697693 2132598718 2626217303 2890397703
Software engineering demands generality and abstraction, performance demands specialization and concretization. Generative programming can provide both, but the effort required to develop high-quality program generators likely offsets their benefits, even if a multi-stage programming language is used. We present lightweight modular staging, a library-based multi-stage programming approach that breaks with the tradition of syntactic quasi-quotation and instead uses only types to distinguish between binding times. Through extensive use of component technology, lightweight modular staging makes an optimizing compiler framework available at the library level, allowing programmers to tightly integrate domain-specific abstractions and optimizations into the generation process. We argue that lightweight modular staging enables a form of language virtualization, i.e. allows to go from a pure-library embedded language to one that is practically equivalent to a stand-alone implementation with only modest effort. High-performance computing applications, such as auto-tuners and domain-specific languages, rely on generative programming techniques to achieve high performance and portability. However, these systems are often implemented in multiple disparate languages and perform code generation in a separate process from program execution, making certain optimizations difficult to engineer. We leverage a popular scripting language, Lua, to stage the execution of a novel low-level language, Terra. Users can implement optimizations in the high-level language, and use built-in constructs to generate and execute high-performance Terra code. To simplify meta-programming, Lua and Terra share the same lexical environment, but, to ensure performance, Terra code can execute independently of Lua's runtime. We evaluate our design by reimplementing existing multi-language systems entirely in Terra. Our Terra-based auto-tuner for BLAS routines performs within 20 of ATLAS, and our DSL for stencil computations runs 2.3x faster than hand-written C. We present a novel approach to proving the absence of timing channels. The idea is to partition the programâ s execution traces in such a way that each partition component is checked for timing attack resilience by a time complexity analysis and that per-component resilience implies the resilience of the whole program. We construct a partition by splitting the program traces at secret-independent branches. This ensures that any pair of traces with the same public input has a component containing both traces. Crucially, the per-component checks can be normal safety properties expressed in terms of a single execution. Our approach is thus in contrast to prior approaches, such as self-composition, that aim to reason about multiple (kâ � 2) executions at once. We formalize the above as an approach called quotient partitioning, generalized to any k-safety property, and prove it to be sound. A key feature of our approach is a demand-driven partitioning strategy that uses a regex-like notion called trails to identify sets of execution traces, particularly those influenced by tainted (or secret) data. We have applied our technique in a prototype implementation tool called Blazer, based on WALA, PPL, and the brics automaton library. We have proved timing-channel freedom of (or synthesized an attack specification for) 24 programs written in Java bytecode, including 6 classic examples from the literature and 6 examples extracted from the DARPA STAC challenge problems. Generic sequence-to-sequence models have trouble generating outputs with highly-structured dependencies such as source code. Motivated by the observation that editing is easier than writing from scratch, we propose a general retrieve-and-edit paradigm that can leverage any base sequence-to-sequence model: given a test input, we first retrieve a training example and then edit the retrieved output into the final predicted output using the base model. The key challenge is to efficiently learn a retriever that is sensitive to the prediction task. We propose first learning a joint variational autoencoder over input-output pairs and then regressing a conditional retriever on the joint embeddings. On the Hearthstone cards benchmark, we show that applying the retrieve-and-edit paradigm to a vanilla sequence-to-sequence model results in BLEU scores approaching those of specialized AST-based code generation models. Additionally, we introduce a new autocomplete task on Python code from GitHub, on which we demonstrate the benefits of retrieve-and-edit.