From Principles to Practice with Class in the First Year

Abstract: We propose a bridge between functional and object-oriented programming in the first-year curriculum. Traditionally, curricula that begin with functional programming transition to a professional, usually object-oriented, language in the second course. This transition poses obstacles for students, and often results in confusing the details of development environments, syntax, and libraries with the fundamentals of OO programming that the course should focus on. Instead, we propose to begin the second course with a sequence of custom teaching languages which minimize the transition from the first course, and allow students to focus on core ideas. After working through the sequence of pedagogical languages, we then transition to Java, at which point students have a strong command of the basic principles. We have 3 years of experience with this course, with notable success.

\section{Introduction}
\label{sec:intro}

Many universities and colleges aim to teach their students proficiency
in an industrial object-oriented programming language by the end of
the students' first year.  The most common approach to achieve this
aim is to teach an industrial language in the first course, often
Java, starting on the first day.  Other curricula take a more indirect
route by teaching functional programming in the first semester,
followed by a second semester in Java.  The latter approach is an
improvement over the first, as pointed out by numerous
observers~\cite{dvanhorn:Felleisen2004Structure, local:spolsky,
  dvanhorn:Chakravarty2004Risks, dvanhorn:Ragde2008Chilling}, but
both suffer serious flaws.

As an example, Northeastern University teaches introductory programming
in the first semester using \emph{How to Design
  Programs}~\cite{dvanhorn:Felleisen2001How}, followed by
object-oriented programming using \emph{How to Design
  Classes}~\cite{local:htdc} in the second semester.  This sequence
was designed to provide a smooth path for incoming students with
 a competence in high-school level algebra to reach
proficiency in Java by the end of their first year
\cite{dvanhorn:Felleisen2004Structure}.  It was a major improvement
over the previous Java-first curriculum in terms of student success,
attrition, and preparation for subsequent
courses~\cite{dvanhorn:Proulx2006Design}.  However, significant
problems remain; in particular, the second semester course violates
the designers' own principles (as recalled
in~\cite{dvanhorn:Bloch2000Scheme}):
\begin{enumerate}
\item \emph{introduce only those language constructs that are necessary to
  teach programming principles}, and
\item \emph{choose a language with as
  few language constructs as possible, and one in which they can be
  introduced one at a time}.
\end{enumerate}

The problem is that the first semester ends with an advanced
pedagogical functional language and the second semester starts with
Java, although it focuses on a side-effect free subset.  Despite this focused
subset, this transition is too abrupt to meaningfully bridge the gap
between functional and object-oriented programming, because
several other significant transitions happen in concert:
\begin{itemize}
\item from a highly regular and minimal syntax to a complicated
  irregular syntax,

\item from an untyped language to a typed language,

\item from a pedagogical programming environment (DrRacket) to a professional
programming environment (Eclipse),

\item from a language with numeric values corresponding to
  mathematical objects to a language with numeric values corresponding
  to common machine representations,\footnote{While this may seem like
    a minor point, details of numeric representation can crop up
    quickly in a Java-based course---for example, $1/3$ cannot be
    represented by any Java numeric types.}

\item from a language with image literals and graphical libraries to
  one in which graphical programming is tedious,

\item from an interaction-oriented language and tool suite to a
  compiled, batch-oriented language.
\end{itemize}

This abrupt transition has several negative consequences which we have
experienced first-hand: the principles of object-oriented programming
are obscured and de-emphasized, struggling with the programming
environment is frustrating and can cause potentially good students to
leave the program, it favors students with prior exposure to the
particular tools (a set that is demographically skewed), it inhibits
students from experimenting by allowing them to rely upon past skills,
and it creates the false impression that courses are discrete units of
instruction that can be discarded after successful completion rather
than being part of a continuous and cumulative educational experience.

We contribute an alternative approach to the second semester that
overcomes these problems and provides a gradual introduction
to object-oriented programming.  Our approach
starts the second semester by introducing \emph{only} the
concept of programming with objects, while all other aspects of the course
remain where they were left off in the previous semester.  This
 allows other concepts to be introduced at the point at
which they are relevant and motivated.  Despite this more gradual
approach, the course accomplishes the goal of reaching industrial
language competence by the end of the semester, covering a super-set
of the concepts and topics covered in the \emph{How to Design Classes}-based
course.

\paragraph{Outline} The remainder of this paper is organized as follows:
 section~\ref{sec:background} provides background on \emph{How to
  Design Programs} and the context and constraints involved in the
first year at Northeastern.  Section~\ref{sec:shift} describes our
approach to the second semester, which starts with a small shift in
perspective to bridge the gap between functional programming and
object oriented programming.  Section~\ref{sec:industrial} describes
the path from our pedagogical languages to an industrial object-oriented
programming language.  Section~\ref{sec:related-work} discusses the
relation to existing work and section~\ref{sec:conclusion} concludes.

\section{Background: the context at Northeastern}
\label{sec:background}

At Northeastern, the College of Computer \& Information Science (CCIS)
requires a four course introductory sequence for Computer Science majors in the first year.  The
first semester features both a course on discrete mathematics and
 an introduction to programming following the \emph{How to Design
  Programs} curriculum.  The second semester follows with a course on
object-oriented programming and one featuring formal reasoning about
programs, both on paper and with the ACL2 theorem
prover~\cite{dvanhorn:Kaufmann2000ComputerAided}.

After the first year, students take a wide variety of
follow-up courses, ranging from a required course in ``Object-oriented
design'' to architecture, operating systems, robotics, and programming
languages.  No standard language is used in these courses.

More significantly, Northeastern distinctively emphasizes experiential
education, with almost all Computer Science majors participating in a
6 month ``co-op'' internship after their third semester.  These co-ops
take place at a wide variety of companies, and while most students do
some software development, there is no single platform or
set of tools that commands majority use.
In particular, there is wide variation in the languages students use
while on co-op. This combination sets the constraints under which we
designed our approach.

\subsection{A first course on \emph{How to Design Programs}}

In the first semester, students are introduced to the ``design
recipe'', a step-by-step process for going from English descriptions
of problems to working programs.  The design recipe involves six steps:
\begin{enumerate}
\item Analyze the information involved in the problem and express
  how to represent it as data.
\item Write down a function signature, a summary of the purpose
  of the function, and a function stub.
\item Illustrate the signature and the purpose statement with some
  functional examples.
\item Take an inventory of the input data that can be used to compute
  an answer.
\item Write the code for the function.
\item Verify the behavior of the program against the functional
  examples given earlier.
\end{enumerate}

Students explore program design using this process in the context of a
series of successively richer pedagogical programming language
levels~\cite{dvanhorn:Felleisen2004Structure,
  dvanhorn:Felleisen2001How} that are included in the DrRacket
(formerly DrScheme) programming
environment~\cite{dvanhorn:Findler2002DrScheme}.
These languages are called the beginning student language (BSL), the
intermediate student language (ISL), and the advance student language
(ASL).
The language and environment include several tools in support of the
design recipe.  For example, functional examples can be written as
executable tests by writing \texttt{check-expect}
expression~\cite{local:check-expect}; an algebraic stepper and REPL
are provided to interact with programs at each step of the design
process.

Finally, the first semester course makes extensive use of a library
for developing interactive animations and games using functional
programming and functional
graphics~\cite{dvanhorn:Felleisen2009Functional, local:barland-sfp10}.

\subsection{The goal of the second course}

After the second course, students should both (a) be prepared for
subsequent courses in the curriculum, which expect familiarity with
Java and standard Java libraries, (b) be prepared for co-ops in which
they will use professional-grade languages and tools which will almost
certainly be object-oriented.  More significantly, we aim to teach the
key insights behind the object-oriented approach to program design.

These constraints, while in detail specific to Northeastern and the
CCIS curriculum, are broadly similar to the requirements for the first
year at many universities.  Our course also attends to smaller and
more idiosyncratic elements of our curriculum, ranging from formal
reasoning to algorithmic analysis, as described in the following
sections.

\section{A small shift of focus}
\label{sec:shift}

On the first day of the second semester, we introduce a single
linguistic concept to an otherwise unchanged context of the previous
semester: the idea of an object.
An object is a new kind of value that can, as a first cut, be
understood as a pairing together of two familiar concepts: data and
functionality.

\begin{itemize}
\item An object is like a structure in that it has a fixed number of
fields, thus an object (again, like a structure) can represent
compound data. But unlike a structure, an object contains not just
data, but functionality too;

\item An object is like a (set of) function(s) in that it has behavior---it computes; it is
not just inert data.
\end{itemize}

This suggests that objects are a natural fit for well-designed programs
since good programs are organized around data definitions and
functions that operate over such data. An object, in essence, packages
these two things together into a single programming apparatus. This
has two important consequences:

\begin{enumerate}

\item Students already know how to design programs oriented around objects.

Since objects are just the combination of two familiar concepts that
students already use to design programs, they already know how to
design programs around objects, even if they have never heard the term
``object'' before.

\item Objects enable new kinds of abstraction and composition.

Although the combination of data and functionality may seem simple,
objects enable new forms of abstraction and composition. That is,
objects open up new approaches to the construction of computations. By
studying these new approaches, we can distill new design
principles. Because we understand objects are just the combination of
data and functionality, we can understand how all of these principles
apply in the familiar context of programming with functions.
\end{enumerate}

\subsection{The basics of objects}

To begin with, we introduce the notion of a \emph{class definition},
which can be thought of at first as a structure definition in that
it defines a new class of compound data.  A class is defined using the
{\tt define-class} form:
\begin{verbatim}
    (define-class posn (fields x y))
\end{verbatim}
This is similar to the {\tt define-struct} form of the first semester,
used as follows:
\begin{verbatim}
    (define-struct posn (x y))
\end{verbatim}

An \emph{object} is a value that is a member of this class of data,
which can be constructed with the {\tt new} keyword, a class name, and
the appropriate number of arguments for the fields of the object:
\begin{verbatim}
    (new posn 3 4)
\end{verbatim}
An object understands some set of \emph{messages}.  Simple
structure-like objects understand messages for accessing their fields
and messages are sent by using the {\tt send} keyword, followed by an object,
a message name, and some number of arguments:
\begin{verbatim}
    (send (new posn 3 4) x) ;=> 3
    (send (new posn 3 4) y) ;=> 4
\end{verbatim}
The {\tt send} notation is simple, but syntactically heavy.  Once
students are comfortable with {\tt send}, we introduce
shorthand to make it more convenient, writing {\tt (x . m)} for {\tt
  (send x m)}.  The \emph{dot notation} can be nested, so {\tt (x . m
  . n)} is shorthand for {\tt (send (send x m) n)}.  (The approach of
introducing a simple, uniform syntax and later introducing a
convenient shorthand that would have been confusing to start with
follows the approach of first introducing {\tt cons} and then later {\tt list}
and {\tt quote} in the first semester.)

It is possible to endow objects with functionality by defining \emph{methods},
which extend the set of messages an object understands.  A method definition
follows the same syntax as a function definition, but is located inside of a
class definition.  Here is a more complete development of the {\tt posn} class
with two methods:
\begin{verbatim}
    ;; A Posn is a (new posn Number Number),
    ;; which represents a point on the Cartesian plane
    (define-class posn (fields x y)

      ;; dist : Posn -> Number
      ;; Distance between this posn and that posn
      (check-expect ((new posn 0 0) . dist (new posn 3 4)) 5)
      (define (dist that)
        (sqrt (+ (sqr (- (this . x) (that . x)))
                 (sqr (- (this . y) (that . y))))))

      ;; dist-origin : -> Number
      ;; Distance of this posn from the origin
      (check-expect ((new posn 0 0) . dist-origin) 0)
      (check-expect ((new posn 3 4) . dist-origin) 5)
      (define (dist-origin)
        (this . dist (new posn 0 0))))
\end{verbatim}

This class definition defines a new class of values which are {\tt
  posn} objects.  Such objects are comprised of two numeric values and
understand the messages {\tt x}, {\tt y}, {\tt dist}, and {\tt
  dist-origin}.  Unit tests have been included with each method
definition, following the principles of the design recipe studied in
the first semester.  Although {\tt check-expect} forms can appear
within class definitions, they are lifted to the top-level when a
program is run.

Methods can be defined to consume any number of arguments, but they
are implicitly parameterized over {\tt this}, the object that received
the message.

\subsection{Where did the {\tt cond} go?}

Unions, and recursive unions in particular, are a fundamental kind of
data definition that students are well-versed in from the previous
semester.  A fundamental early lesson is how to represent (recursive)
unions using classes and how to write recursive methods.  As an
example, figure~\ref{fig:tree} defines binary trees of numbers
(an archetypal recursive union data definition)
using the Beginning Student language (BSL) as used at the start of the
first semester, and also using the Class language of our course.

\begin{figure}[h!]
\begin{minipage}[t]{3.5in}
\begin{verbatim}
#lang bsl
;; A Tree is one of:
;; - (make-leaf Number)
;; - (make-node Tree Number Tree)
(define-struct leaf (v))
(define-struct node (left v right))

;; sum : Tree -> Number
;; sums the elements of the given tree
(define (sum a-tree)
  (cond [(leaf? a-tree) (leaf-v a-tree)]
        [else
         (+ (sum (node-left  a-tree))
            (node-v a-tree)
            (sum (node-right a-tree)))]))

(check-expect (sum (make-leaf 7)) 7)
(check-expect
  (sum (make-node
         (make-leaf 1)
         5
         (make-node (make-leaf 0)
                    10
                    (make-leaf 0))))
  16)
\end{verbatim}
\end{minipage}
\begin{minipage}[t]{3in}
\begin{verbatim}
#lang class/1
;; A Tree is one of:
;; - (new leaf Number)
;; - (new node Tree Number Tree)
;; and implements
;; sum : -> Number
;; sums the elements of this tree

(define-class leaf
  (fields v)
  (define (sum) (this . v)))

(define-class node
  (fields left v right)
  (define (sum)
    (+ (this . left . sum)
       (this . v)
       (this . right .sum))))

(check-expect ((new leaf 7) . sum) 7)
(check-expect
  ((new node
        (new leaf 1)
        5
        (new node (new leaf 0)
                  10
                  (new leaf 0))))
    . sum)
  16)
\end{verbatim}
\end{minipage}
\caption{Binary tree sum in Beginning Student and in the Class language}
\label{fig:tree}
\end{figure}

The structure of this program is analogous to the approach of
the previous semester but this example brings to light an important
difference with the functional approach.  The method for computing the
sum of a leaf is defined in the {\tt leaf} class, while the
method for computing the sum of a node is in the {\tt node} class.
When a tree object is sent the {\tt sum} method, there is no function
with a conditional to determine whether the object is a
leaf---instead, the object itself takes care of computing the sum
based on its own {\tt sum} method.  This shift in perspective
is at the core of object-orientation: objects contain their own
behavior and the case analysis previously done in functions is
eliminated.

\subsection{Worlds and animations}

At Northeastern, Programming in the first semester is often oriented
around interactive event-driven video games.  The basic design of a
video game involves defining a data representation for states of the
game and functions for transitioning between states based on events
such as clock ticks, keyboard input, or mouse events.  The design of a
game thus involves the design of data and operations on that data; in
other words, the game involves the design of objects.  We therefore
continue in the second semester with the use of programming video
games but supplement the course with a library for doing so in an
object-oriented style.  Figure~\ref{fig:world} gives an example
written in both the functional style and object-oriented style.

The key difference between these two programs is that the functional
program uses the {\tt 2htdp/universe} library, which provides a {\tt
  big-bang} form that consumes the initial state of the world and has
a declarative form of associating event-handler functions, while the
object-oriented program uses an alternative library developed for our
class: {\tt class/universe}.  It also provides a {\tt big-bang} form
but it consumes a single argument, the initial state of the world
represented as an object.  Event handlers are simply methods of this
object; for example, clock ticks trigger the {\tt on-tick} method.

The program on the left is the first program of the first semester,
while the one on the right is the first program of the second
semester.  Our approach is able to make the conceptual connection
between functional and object-oriented programming quite clear while
appealing to the familiar event-driven interactive programs developed
throughout the year.

The move to object-oriented style immediately and naturally leads to
designs that are enabled by organizing programs around objects.  For
example, the state pattern~\cite{samth:GOF} becomes useful almost
immediately.  The programs in figure~\ref{fig:world} animate a rocket
(rendered as a circle in this example) taking off.  An illustrative
follow-up exercise is to animate a rocket that \emph{lands}.  The
natural design is to have two variants for states of the rocket: one
for descending rockets and one for landed rockets (an example is given
in appendix~\ref{sec:appendix}).  While in the functional approach it
is easy to use the state-pattern for the \emph{data} representing a
rocket, it is more difficult to have states of \emph{behavior}.  The
typical solution adds conditionals to all of the event handlers.  In
the object-oriented approach, states of behavior are just as natural
as data.  It is therefore straightforward to design programs with easy
to observe invariants such as ``a landed rocket never changes
position.''  In the functional approach, even such simple properties
are more involved to establish, because all event handlers must be
inspected.

This approach also leads naturally to discussion of inheritance.
Often programs with multiple states wish to share the implementation
of some methods.  We first show that this can be accomplished at the
cost of minor boilerplate with delegation, and then show how
inheritance allows the programmer to avoid duplication and boilerplate
entirely.  Once inheritance is able to group identical methods,
overriding is a natural next step when some but not all of the
implementations are identical across the variants.

\begin{figure}
\begin{minipage}[t]{3.4in}
\begin{verbatim}
#lang bsl
(require 2htdp/image 2htdp/universe)

;; A World is a Number

;; on-tick : World -> World
(define (tick w)
  (add1 w))

;; to-draw : World -> Image
(define (draw w)
  (place-image
    (circle 10 "solid" "red")
    w 200 (empty-scene 400 400)))

;; on-key : KeyEvent World -> World
(define (on-key k w) 10)

(big-bang 10
          [to-draw draw]
          [on-tick tick])
\end{verbatim}
\end{minipage}
\begin{minipage}[t]{3in}
\begin{verbatim}
#lang class/1
(require 2htdp/image class/universe)

;; A World is a (new world Number)
(define-class world
  (fields n)

  ;; on-tick : -> World
  (define (on-tick)
    (new world (add1 (this . n))))

  ;; to-draw : -> Image
  (define (to-draw)
    (place-image
     (circle 10 "solid" "red")
     (this . w) 200 (empty-scene 400 400)))

  ;; on-key : KeyEvent -> World
  (define (on-key k) (new world 10)))

(big-bang (new world 10))
\end{verbatim}
\end{minipage}
\caption{World programs}
\label{fig:world}
\end{figure}

\subsection{Language levels}

Our introduction to object-oriented programming is built on a series
of ``language levels'', each of which introduces additional features,
adding complexity to the programming model and expressiveness to the
programs.  Each language is  \texttt{class/}\textit{N}
for some \textit{N}, with features appearing in the following order.
\begin{enumerate}
  \setcounter{enumi}{-1}
\item Classes and Objects
\item Abbreviated notation for method calls
\item Super classes
\item Overriding
\item Constructors
\end{enumerate}

Several commonalities run through all of these languages.  First, they
are all purely functional; we do not introduce imperative I/O or
side-effects until after transitioning to Java in the second half of
the course.  Second, they all are a super set of the
\emph{Intermediate Student} language from \emph{How to Design Programs}\xspace, meaning that they
support higher-order functional programming and lists.

One key principle that we adhere to in the design of the language
levels is that no features of the language are added purely to support
``software engineering'' concerns such as specification mechanisms.
Not only does that language not support declaring types or contracts,
but interfaces are described purely in comments.

This is not to say that interfaces and contracts are optional; in
fact, they are mandatory.  But the focus of the first part of the
course is on the fundamentals of object-orientation.  Teaching the use
of software engineering tools such as type systems, while vital, is a
topic which we defer to the second half of the course when we
transition to Java.

We made this decision after experience in which students were confused
about the relationship between explicit interface specifications, type
systems, and the informal data definitions and contracts which
students are required to write for all methods.  After removing
interfaces from the language and making them purely a specification
construct, this confusion disappeared.

\section{From principles to industrial languages}
\label{sec:industrial}

The transition from custom teaching languages to a professional
language takes place about half-way through the course.  At this
point, students already have experience with many of the essential
concepts of object-oriented programming. In particular: objects,
classes, fields and methods, dynamic dispatch, inheritance, and
overriding.

From this point, almost any language that students might encounter in
future co-op positions, summer internships, or upper-level courses
would be an appropriate follow-up.  Our course transitions to Java,
but C\#, Python, Ruby, Eiffel, or JavaScript would all work naturally.
The key lesson of the transition is that the fundamental principles
underlying object-oriented programming remain the same between
languages, and that learning a new language is primarily a matter of
mapping these concepts to specific constructs in the new language.  Of
course, particular languages also use unique specific mechanisms which
need to be taught to use the language effectively, but these are
rarely as vital as the cross-language principles.

We chose the half-way point as the time for transition based on
experience with earlier versions of this course. In particular, we
found that a later transition, while allowing us to present additional
concepts in a controlled environment, did not give students sufficient
time and experience with Java.  Subsequent classes found that students
were strong on fundamentals but weak on Java practice.  The other
alternative, transitioning earlier, would not provide sufficient time
to cover the fundamental topics before the transition.

\subsection{Functional Java}

The transition begins with replicating the object-oriented style of
our teaching languages in Java.  In particular, we do not introduce
mutation, for loops, or mutable data structures such as arrays or
\texttt{ArrayList}s until later in the semester.  Instead, students
design data representations using classes, with \texttt{interface}s
representing unions of data.  Additionally, we avoid mention of the
distinction between primitive and other values in Java, which is made
easier by not using standard libraries early.  An example of this
style of programming is presented in figure~\ref{fig:java}, repeating
the binary tree sum from the previous section.

\begin{figure}
\begin{verbatim}
import tester.*;

interface Tree {
    // sums the elements of this tree
    Integer sum();
}

class Leaf implements Tree {
    Integer v;
    Leaf(Integer v) { this.v = v; }
    public Integer sum() { return this.v; }
}

class Node implements Tree {
    Tree left; Integer v; Tree right;
    Node(Tree l, Integer v, Tree r) {
        this.left = l;
        this.v = v;
        this.right = r;
    }

    public Integer sum() {
        return this.left.sum() + this.v + this.right.sum();
    }
}

class Examples {
    void test_tree(Tester t) {
        t.checkExpect(new Leaf(7).sum(), 7);
        t.checkExpect(new Node(new Leaf(1),
                               5,
                               new Node(new Leaf(0), 10, new Leaf(0))).sum(),
                      16);
    }
}
\end{verbatim}
\caption{Binary tree sum in the style of \emph{How to Design Classes}}
\label{fig:java}
\end{figure}

Comparing this figure to the previous example illustrates a number of the
differences that students are exposed to upon transition to Java.

\begin{enumerate}
\item Explicit representation of unions and interfaces in the
  language.  Previously, interfaces were simply described in stylized
  comments, following the \emph{How to Design Programs}\xspace approach.
\item Types are now specified as part of the program and are
  (statically) enforced.  Data definitions and interfaces can be
  transformed from the stylized comments into {\tt interface}
  definitions and method signatures annotated with types.  Students
  are taught the benefits of type systems, which impose syntactic
  restrictions sufficient to prove objects meet (the structural
  aspects of) their interface definitions.  Students also encounter
  the downside of types when they discover the type system cannot
  always follow valid reasoning about program invariants and may reject
  perfectly good programs.
\item Java syntax is substantially different and more verbose. For
  example, constructors must be defined explicitly.
\item The testing environment is somewhat different, and requires
  additional boilerplate, although we are
  able to use the JavaLib framework~\cite{local:java-world} to support testing with
  structural equality.
\end{enumerate}

\noindent
There are other differences which cannot be seen from a code snippet.

\begin{enumerate}
  \setcounter{enumi}{4}
\item Students must use a new development environment and compiler.
  In class, we primarily develop in a text editor and run the Java
  compiler at the command line.  In labs and on homeworks, students
  typically use the Eclipse IDE.
\item Installing and configuring libraries is now required.  Because
  we use a custom library for testing, students must cope with library
  installation and class paths on the first day.
\end{enumerate}

All but the first two of these changes are unrelated to the
fundamental lessons we hope to teach---the rest merely present
additional hurdles for students.  However, at this point in the
semester, the students are far better equipped to meet these
challenges.  They are already familiar with objects, classes, and the
other concepts we have covered.  They are also fully engaged in the
class, instead of making the transition in the midst of the transition
between semesters.  Finally, they have now been programming for 50\%
longer than they had at the start of the semester.

\subsection{Traditional Java}

Thanks to the preparation in the first half of the course, we can
cover OO programming in a functional subset of Java in a just a few
lectures.  We then increase the subset of the language we use to
encompass mutation, loops, and mutable data structures and introduce
the underlying design principles. We present \texttt{ArrayList}s,
followed briefly by arrays. Students use, and then implement, hash
tables as well as other mutable and immutable data structures.
Conventional input and output are treated only very briefly, as we
focus instead on both fundamentals and exercises making use of real
APIs such as hashing functions or Twitter posting.  Finally,
\texttt{while}, \texttt{for}, and for-each loops are presented,
following the methodology of \emph{How to Design Classes}\xspace which connects loops to stylized
use of recursive functions with accumulators, a technique the students
now have used for two semesters.

\subsection{Beyond Traditional Java}

Finally, at the end of the course, we are able to build on the two
major segments to examine less-well-explored topics in object-oriented
programming.  Typically, we cover the basics of representing objects
in a functional language, advanced OO techniques such as mixins and
prototypes, and a new OO language such as Ruby or JavaScript.
Additionally, we emphasize the ability to embed functional programming
in an OO context, using techniques such as the command pattern and the
visitor patterns.  Again, the key message is the transferability of
concepts across languages.

\section{Related work}
\label{sec:related-work}

Teaching programming principles in a functional style has a long
history, with Abelson and Sussman's \emph{Structure and Interpretation
  of Computer Programs}~\cite{dvanhorn:sicp} being a prominent
example.  Our work follows in the tradition of the \emph{Program by
  Design} (PbD)
project\footnote{\url{http://www.programbydesign.org/}} (previously
known as the \emph{TeachScheme!} project), which emphasizes a
systematic approach to program construction.

Since the introduction of functional-first curricula, and more
specifically in the Program by Design framework, numerous courses have
tackled the problem of transition.  Typically they, as we, transition
to Java in the second course.  We discuss first the approach developed
by some of the principal creators of PbD, and then other approaches.

\subsection{Program by Design and ProfessorJ}

The Program by Design project initially focused only on the first course,
with the second course typically taught in Java in
institution-specific ways. Subsequently, the pedagogical approach was
extended to Java, but without the tool support and textbook of the
first course.  An example of this approach is described by
\citet{dvanhorn:Bloch2000Scheme}, who presents the experience
integrating these courses at Adelphi.  He reports that ``many of
Java's concepts could be introduced more easily in a second course
than a first.''

With these lessons in mind, the PbD project set out to apply
the lessons of teaching languages and IDE support to Java, as well as
to present the approach to object-oriented programming in textbook
form. ProfessorJ~\cite{dvanhorn:Gray2003ProfessorJ} is the resulting
system, accompanying the draft textbook \emph{How to Design
  Classes}~\cite{local:htdc}.  In parallel to our course, Northeastern
teaches the remainder of its computer science majors following this
approach.

ProfessorJ and \emph{How to Design Classes} maintain many of the ideas
of the first course.  In particular, ProfessorJ brings language levels
to Java, in an attempt to smooth the transition for students from the
first course and provide more helpful feedback.  ProfessorJ is also
embedded in the DrRacket IDE, increasing familiarity for the students
and supporting tools such as an interactive read-eval-print loop.

However, the ``day 1'' transition from the student languages used with
\emph{How to Design Programs}\xspace to {ProfessorJ}\xspace is too abrupt and too large.
Most significantly, changing languages from the first semester
immediately rather than simply adding a new concept confuses too many
issues for students.  On the first day of a \emph{How to Design Classes}\xspace-based course,
students see object-orientation, a new programming paradigm; Java, a
new language with new syntax, and a static type system, a crucial but
orthogonal concept.
In contrast, our course presents just one of these concepts on the
first day, but  covers all of them by the end of the semester.

{ProfessorJ}\xspace also takes on the dual challenges of implementing Java as
well as subsetting it.  This ultimately resulted in both a limited
Java environment as well as the eventual abandoning of the tool since
it was too difficult to maintain, let alone keep up with advances in
Java.

Committing to Java on the first day, regardless of the environment
provided to students, has significant limitations.  First, the
syntactic and semantic heaviness of Java is a burden for beginning
students, and discourages interactive and experimental programming.
The very first chapter of \emph{How to Design Classes}\xspace discusses the fixed size of Java
integers, a topic avoided entirely in the first course.  Second, by
committing to a particular industrial-strength language, it closes off
possibilities in the curriculum.  Third, it commits entirely to the
new paradigm, making it more difficult for students to compare the
approaches.

Since {ProfessorJ}\xspace is no longer available, students are faced with an even
starker change on the first day.  Even with a student-oriented
environment such as DrJava or BlueJ~\cite{dvanhorn:Allen2002DrJava,
  dvanhorn:Hsia2005Taming}, students must learn an entirely new tool,
along with new libraries.
If the course uses a typical professional development environment such
as Eclipse, students must also contend with compilation, loss of
interactivity, and subtle issues such as classpaths, none of which are
fundamental to the concepts that the course focuses on.

\subsection{Other transitions}

Not every curriculum that begins with \emph{How to Design Programs}\xspace transitions to Java after
the first course.  \citet{dvanhorn:Ragde2008Chilling} describes a
second course that includes both more advanced work in Scheme beyond
teaching-oriented languages as well as low-level programming in C,
taught to computer science majors at University of Waterloo.  Radge's
course intentionally does not use student-oriented languages, although
the recently-developed C0 language~\cite{local:c0} could provide such
a language. Other discussions of functional programming in the first
year~\cite{dvanhorn:Chakravarty2004Risks} do not discuss the problems
of transition.

\subsection{Other approaches to Java}

The problems of teaching Java in introductory courses have been
well-explored; we mention only a few related directions here.
DrJava~\cite{dvanhorn:Allen2002DrJava} and
BlueJ~\cite{dvanhorn:Hsia2005Taming,dvanhorn:Kolling2003} are
introductory environments for Java, which alleviate some but not all
of the drawbacks we have outlined.
For example, both of these systems immediately present students with
(1) type systems and (2) Java syntax, and (3) do not support the image
values and exact numeric values that we rely on in our course.

Several teaching-oriented graphics libraries for Java have been
proposed~\cite{dvanhorn:Bruce2001Library,dvanhorn:Alphonce2003Using},
but these are significantly more complex than the graphics and
interaction libraries we are able to use in the introductory language
we present.

\section{Experience and outlook}
\label{sec:conclusion}

We have now completed the third iteration of this course, teaching
approximately 35 students each time.  Our experience has been
uniformly positive, and the students have gone on to significant
success in the subsequent courses, despite the curriculum differing
from what the bulk of Northeastern University computer science majors
take.  Anecdotally, the class has also had notable success in the
recruitment and retention of female students, as compared to the other
versions of the second-semester course. However, the classes are
sufficiently different as to make a precise comparison impossible.

The course has provided a vantage point to introduce topics that will
be taken up later in the curriculum.  We present types, contracts,
invariants, and properties of functions, all of which tie into both
the concurrent course on logic and computation, as well as later
classes on formal methods.  The emphasis on
representation-independence and interfaces both tie into later classes
on software engineering, as well as preparing students for algorithms
and data structures courses.  Finally, the use of interactive and
distributed systems connects to later courses on operating systems and
networks.

Despite our success, much remains to be done.  Type systems are a
fundamental concept, but their introduction accompanies the rest of
Java.  Developing a typed version of our introductory languages would
allow a smoother introduction of this idea.

Our class's use of Eclipse could also be improved by first
transitioning to a pedagogically-oriented Java environment, but we
have not evaluated the specific options.  Alternatively, introducing
Java-like syntax for the teaching languages we have developed would
help tease apart the difficult transitions still present in the course.

Finally, the Java portion of the class does not continue the use of
``World''-style interactive graphical programming, although a version
of the ``World'' library has been developed for Java
\cite{local:java-world}.  Instead, our course focuses on coverage of
standard Java libraries, as well as introductory algorithmic and data
structure topics.  Continuing to use World-style programming in
motivating examples might be valuable for continuity between the two
halves of the course.

\subsection*{Acknowledgments}

Matthias Felleisen's approach to pedagogy and passion for
undergraduate teaching has inspired this work from the beginning.
CCIS Dean Larry Finkelstein entrusted two postdocs with the redesign
of a key undergraduate course, which made this experiment possible.
Our teaching assistants, Dan Brown, Asumu Takikawa, and Nicholas
Labich, as well as the tutors and graders, contributed enormously to
the success of our courses. Finally, and most importantly, our
students at Northeastern for the last three years have put up with a
curriculum in progress, and the opportunity to teach them has been
truly rewarding.

\bibliographystyle{eptcs}


Task-Aware Neural Architecture Search

Abstract: The design of handcrafted neural networks requires a lot of time and resources. Recent techniques in Neural Architecture Search (NAS) have proven to be competitive or better than traditional handcrafted design, although they require domain knowledge and have generally used limited search spaces. In this paper, we propose a novel framework for neural architecture search, utilizing a dictionary of models of base tasks and the similarity between the target task and the atoms of the dictionary; hence, generating an adaptive search space based on the base models of the dictionary. By introducing a gradient-based search algorithm, we can evaluate and discover the best architecture in the search space without fully training the networks. The experimental results show the efficacy of our proposed task-aware approach.

\section{Introduction}
\label{sec:intro}

These guidelines include complete descriptions of the fonts, spacing, and
related information for producing your proceedings manuscripts. Please follow
them and if you have any questions, direct them to Conference Management
Services, Inc.: Phone +1-979-846-6800 or email
to \\\texttt{papers@2021.ieeeicassp.org}.

\section{Formatting your paper}
\label{sec:format}

All printed material, including text, illustrations, and charts, must be kept
within a print area of 7 inches (178 mm) wide by 9 inches (229 mm) high. Do
not write or print anything outside the print area. The top margin must be 1
inch (25 mm), except for the title page, and the left margin must be 0.75 inch
(19 mm).  All {\it text} must be in a two-column format. Columns are to be 3.39
inches (86 mm) wide, with a 0.24 inch (6 mm) space between them. Text must be
fully justified.

\section{PAGE TITLE SECTION}
\label{sec:pagestyle}

The paper title (on the first page) should begin 1.38 inches (35 mm) from the
top edge of the page, centered, completely capitalized, and in Times 14-point,
boldface type.  The authors' name(s) and affiliation(s) appear below the title
in capital and lower case letters.  Papers with multiple authors and
affiliations may require two or more lines for this information. Please note
that papers should not be submitted blind; include the authors' names on the
PDF.

\section{TYPE-STYLE AND FONTS}
\label{sec:typestyle}

To achieve the best rendering both in printed proceedings and electronic proceedings, we
strongly encourage you to use Times-Roman font.  In addition, this will give
the proceedings a more uniform look.  Use a font that is no smaller than nine
point type throughout the paper, including figure captions.

In nine point type font, capital letters are 2 mm high.  {\bf If you use the
smallest point size, there should be no more than 3.2 lines/cm (8 lines/inch)
vertically.}  This is a minimum spacing; 2.75 lines/cm (7 lines/inch) will make
the paper much more readable.  Larger type sizes require correspondingly larger
vertical spacing.  Please do not double-space your paper.  TrueType or
Postscript Type 1 fonts are preferred.

The first paragraph in each section should not be indented, but all the
following paragraphs within the section should be indented as these paragraphs
demonstrate.

\section{MAJOR HEADINGS}
\label{sec:majhead}

Major headings, for example, "1. Introduction", should appear in all capital
letters, bold face if possible, centered in the column, with one blank line
before, and one blank line after. Use a period (".") after the heading number,
not a colon.

\subsection{Subheadings}
\label{ssec:subhead}

Subheadings should appear in lower case (initial word capitalized) in
boldface.  They should start at the left margin on a separate line.

\subsubsection{Sub-subheadings}
\label{sssec:subsubhead}

Sub-subheadings, as in this paragraph, are discouraged. However, if you
must use them, they should appear in lower case (initial word
capitalized) and start at the left margin on a separate line, with paragraph
text beginning on the following line.  They should be in italics.

\section{PRINTING YOUR PAPER}
\label{sec:print}

Print your properly formatted text on high-quality, 8.5 x 11-inch white printer
paper. A4 paper is also acceptable, but please leave the extra 0.5 inch (12 mm)
empty at the BOTTOM of the page and follow the top and left margins as
specified.  If the last page of your paper is only partially filled, arrange
the columns so that they are evenly balanced if possible, rather than having
one long column.

In LaTeX, to start a new column (but not a new page) and help balance the
last-page column lengths, you can use the command ``$\backslash$pagebreak'' as
demonstrated on this page (see the LaTeX source below).

\section{PAGE NUMBERING}
\label{sec:page}

Please do {\bf not} paginate your paper.  Page numbers, session numbers, and
conference identification will be inserted when the paper is included in the
proceedings.

\section{ILLUSTRATIONS, GRAPHS, AND PHOTOGRAPHS}
\label{sec:illust}

Illustrations must appear within the designated margins.  They may span the two
columns.  If possible, position illustrations at the top of columns, rather
than in the middle or at the bottom.  Caption and number every illustration.
All halftone illustrations must be clear black and white prints.  Colors may be
used, but they should be selected so as to be readable when printed on a
black-only printer.

Since there are many ways, often incompatible, of including images (e.g., with
experimental results) in a LaTeX document, below is an example of how to do
this \cite{Lamp86}.

\section{FOOTNOTES}
\label{sec:foot}

Use footnotes sparingly (or not at all!) and place them at the bottom of the
column on the page on which they are referenced. Use Times 9-point type,
single-spaced. To help your readers, avoid using footnotes altogether and
include necessary peripheral observations in the text (within parentheses, if
you prefer, as in this sentence).

\begin{figure}[htb]

\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{image1}}
  \centerline{(a) Result 1}\medskip
\end{minipage}
\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image3}}
  \centerline{(b) Results 3}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image4}}
  \centerline{(c) Result 4}\medskip
\end{minipage}
\caption{Example of placing a figure with experimental results.}
\label{fig:res}
\end{figure}

\section{COPYRIGHT FORMS}
\label{sec:copyright}

You must submit your fully completed, signed IEEE electronic copyright release
form when you submit your paper. We {\bf must} have this form before your paper
can be published in the proceedings.

\section{RELATION TO PRIOR WORK}
\label{sec:prior}

The text of the paper should contain discussions on how the paper's
contributions are related to prior work in the field. It is important
to put new work in  context, to give credit to foundational work, and
to provide details associated with the previous work that have appeared
in the literature. This discussion may be a separate, numbered section
or it may appear elsewhere in the body of the manuscript, but it must
be present.

You should differentiate what is new and how your work expands on
or takes a different path from the prior studies. An example might
read something to the effect: "The work presented here has focused
on the formulation of the ABC algorithm, which takes advantage of
non-uniform time-frequency domain analysis of data. The work by
Smith and Cohen \cite{Lamp86} considers only fixed time-domain analysis and
the work by Jones et al \cite{C2} takes a different approach based on
fixed frequency partitioning. While the present study is related
to recent approaches in time-frequency analysis [3-5], it capitalizes
on a new feature space, which was not considered in these earlier
studies."

\vfill\pagebreak

\section{REFERENCES}
\label{sec:refs}

List and number all bibliographical references at the end of the
paper. The references can be numbered in alphabetic order or in
order of appearance in the document. When referring to them in
the text, type the corresponding reference number in square
brackets as shown at the end of this sentence \cite{C2}. An
additional final page (the fifth page, in most cases) is
allowed, but must contain only references to the prior
literature.

\bibliographystyle{IEEEbib}

\section{Introduction}
\label{sec:intro}
Neural Architecture Search (NAS) has been a major focal point for work on automated machine learning (AutoML). Initially studied through the lens of reinforcement learning~\cite{zoph2016neural}, a modern development of NAS algorithms largely focuses on minimizing both search time and prior knowledge. Though NAS techniques have greatly improved, many recently proposed methods require significant prior knowledge, e.g. the explicit architecture search domain, or the specific task at hand, as input. This requirement restricts their ability to adapt to situations in which future tasks are potentially unknown.

In this work, we propose a novel, flexible NAS framework, which we call Task-Aware Neural Architecture Search (TA-NAS). The ultimate goal of TA-NAS is to develop an algorithm that dynamically learns an appropriate architecture for a given task at hand, making decisions based on prior history and any information input by the user. Our pipeline is composed of three key components. First, we start with a dictionary of base tasks, the atoms of which consist of architectures that accurately perform said tasks. The dictionary serves as a base on which we dynamically build architectures for new tasks not in the dictionary. Based on the idea that similar tasks should require similar architectures, an often-used assumption in both transfer and lifelong learning, we propose a novel similarity measure for tasks to find the closest base tasks to the new task. Then, we construct a dynamic search space, based on the combined knowledge from the related tasks, without the need for prior domain knowledge. Finally, we present a gradient-based search algorithm, called \emph{Fusion Search} (FUSE). The FUSE algorithm is designed to evaluate the performance of network candidates without fully train any of them. Our experimental evaluation will show the efficacy of our proposed approach.

\section{Related Work}
\label{related}
Many recently proposed NAS techniques have resulted in architectures with performance comparable to those of hand-tuned architectures. The techniques themselves are based on a wide range of techniques, including evolutionary algorithms~\cite{real2018regularized}, reinforcement learning (RL)~\cite{zoph2018learning}, and sequential model-based optimization (SMBO)~\cite{liu2018progressive}. All of these approaches, however, are very time-consuming and need require computational resources, e.g. potentially thousands of GPU-days. To alleviate these issues, differentiable search~\cite{cai2018proxylessnas,liu2018darts,noy2019asap,luo2018neural,xie2018snas} and random search together with sampling sub-networks from a one-shot super-network~\cite{bender2018understanding, li2019random,cho2019one} have been introduced in the literature. For instance, DARTS~\cite{liu2018darts} smooths the architecture search space using a softmax operation. It then solves a bilevel optimization problem which can accelerate the discovery of the final architecture by orders of magnitude~\cite{zoph2016neural, zoph2018learning, real2018regularized, liu2018progressive}. Other recent methods include random search~\cite{li2018massively,li2019random,sciuto2019evaluating,xie2019exploring}, RL based approaches via weight-sharing ~\cite{pham2018efficient}, and network transformations~\cite{cai2018efficient,elsken2018efficient,jaderberg2017population}.

Besides, \cite{bender2018understanding} has thoroughly analyzed the one-shot architecture search using weight-sharing and correlation between the super-graph and sub-networks. None of the above techniques have yet explored the role of the closeness of tasks in the search neural architecture space. Consequently, the search space used by these techniques is often biased and based on the domain knowledge from the well-performed handcrafted neural network architectures. Here, we propose an approach to encode the similarities between tasks for a more efficient search strategy.

\section{Proposed Approach}
\label{sec:NAS}
The pseudocode of TA-NAS is given in Algorithm~\ref{alg1}. At time $t$, we assume that we have access to a dictionary consisting of both previous pairs $(T_{1},X_{1}),...(T_{t-1},X_{t-1}),$ of tasks $T_{k}$ and data sets $X_{k}$ is a given data set, as well as a collection of such pairs that were available upon initialization. Each pair is represented in our dictionary by a trained network. Given the target pair of $(T_{t},X_{t}),$ our goal is to find an architecture for achieving high performance on the target task. In summary, TA-NAS works as follows:
\begin{enumerate}
    \item \textbf{Task Similarity.} Given a new task-data set pair, TA-NAS finds the most related task-data set pairs in the dictionary.
    \item \textbf{Search Space.} TA-NAS defines a suitable search space for the incoming (target) task-data set pair, based on the related pairs.
    \item \textbf{Search Algorithm.} TA-NAS searches to discover an optimal architecture in term of performance for the target task-data set pair on the search space.
\end{enumerate}

\begin{algorithm}[t]
\label{alg1}
\SetAlgoLined
 \textbf{Initialization}: A set of baseline task-data set pairs $B$\;
 \textbf{Input}: Task-data set pairs $(T_{1},X_{1}),...,(T_{K},X_{K})$, Threshold $\tau$, $\epsilon$\;
 \textbf{Output}: Best architecture for the incoming tasks t\;

 \For{$t = (K+1), (K+2),...$}{
    \For{$b \in B$}{
        Calculate distance $d_{b,t}$ to find the related tasks\;
        }

    Define search space by combining operations, cells, skeleton from related tasks\;

    \While{criteria not met}{
        Sample C candidates from search space\;
        Evaluate these candidates using FUSE\;
    }

    Add the task t and its architecture to $B.$
 }

\caption{Task-Aware NAS}
\end{algorithm}

\subsection{Task Similarity}
\label{ssec:task-taxonomy}
The TA-NAS pipeline heavily depends on the notion of similarity between task-data set pairs. We define the similarity between task-data set pairs in terms of a model-transformation complexity, $N_t$. In particular, we first construct a dictionary with the atoms given by the by trained architectures performing well in each base task-data set pairs. More precisely, let $\ell_{(T,X)}(N)$ be a function that measures the performance of a given architecture $N$ on task $T$ with input data $X$. Here, $\ell_{(T,X)}(N)=1$ denotes the highest performance, while $\ell_{(T,X)}(N)=0$ denotes a complete failure of architecture $N$ on task $T$. For a fixed $0 < \epsilon < 1$, we say that an architecture $N$ is $\epsilon$-representative for $(T,X)$ if $\ell_{T,X}(N) \geq 1 - \epsilon.$ In other words, an architecture $N$ is $\epsilon$-representative if it performs sufficiently well on the given task-data set pair. We thus restrict our attention to $\epsilon$-representative networks.

We now define our measure of dissimilarity between two task-data set pairs using the notion of $\epsilon$-representative architecture. Let $A =(T_A, X_A)$ and $B=(T_B, X_B)$ be two task-data set pairs, where $N_{A}$ and $N_{B}$ are two trained architectures that are $\epsilon$-representative for $A$ and $B$, respectively. We can define a dissimilarity measure between $A$ and $B$ as follows (hence, the similarity between $A$ and $B$ is given by $1- d^{\epsilon}_{A,B}$):

\begin{figure}[t]
	\centering
	\centerline{\includegraphics[width=8.5cm]{task-taxonomy.eps}}
	\caption{Illustration of the procedure to compute the distance from task A to task B.}\label{fig: 1}
\end{figure}

\begin{equation}
    d^{\epsilon}_{A,B} = \min_{\{N_{t} \in S_{t} : \ell_{B}(N_t \circ N_A) \geq (1 - \epsilon)\}} O(N_t),
\end{equation}
where $S_{t}$ is a given transform network search space, and $O$ is a general measure of complexity (e.g., the number of parameters in a network). The symbol $\circ$ denotes function (network) composition. In this setting, we assume that the first layer of $N_t$ is the same as the {\emph{penultimate}} layer of $N_A$. In practice, the role of $N_t$ is to transform the features of the penultimate layer of $N_A$ generated by elements of $X_B$ and transform them into elements of $N_B(X_B)$. In other words, the model transformation network $N_t$ should be such that it can transfer the $N_A(X_B)$ into $N_B(X_B)$ with minimum complexity. For instance, if $N_t$ is the identity matrix, it means that $T_A$ and $T_B$ are exactly the same as $N_B(X_B) = N_A(X_B)$. In general, $d^{\epsilon}_{A,B}$ is asymmetric measure, i.e., $d^{\epsilon}_{A,B}\neq d^{\epsilon}_{B,A}$. Ideally, $O$ should be zero for two identical tasks, implying that the architecture for $A$ is also suitable for $B$. In practice, finding the least-complex transform network can be achieved by iterative pruning some super-network for which $N_{t}$ is a sub-graph with the performance as good as the super-network. The dissimilarity measure is the percentage of the non-zero parameters in the pruned $N_t$, and it ranges between 0 and 1. This is illustrated in Figure~\ref{fig: 1}.

\subsection{Search Space}
\label{ssec:search-space}
Defining a meaningful search space is the key to efficiently finding the best architecture for a specific task. In the NAS literature, the search space is typically defined by stacking a structure called \emph{cell}, as illustrated in Figure~\ref{fig: 1b}. A cell is a densely connected directed-acyclic graph (DAG) of nodes, where all nodes are connected by operations. Other NAS techniques such as one-shot approaches (e.g., DARTS~\cite{liu2018darts}, NAS-Bench201~\cite{dong2020bench}) have also introduced another structure in the search space referred to as \emph{skeleton}. A skeleton is a combination of cells with other operations, forming the complete network architecture. A skeleton is normally predefined, and the goal of NAS algorithms is to find the optimal cells. In this paper, we similarly define the search space in terms of skeletons and cells. Specifically, we focus our search on cells and their operations. As mentioned, cells consist of nodes and operations. Each node has $2$ inputs and $1$ output. The operations (e.g., identity, zero, convolution, pooling) are set so that the dimension of the output is the same as that of the input. If $\mathbf{n}$ is the number of nodes in a cell and $\mathbf{m}$ denotes the number of operations, the total number of possible cells is given by: $\mathbf{m} \times \exp \big({\frac{\mathbf{n}!}{2(\mathbf{n}-2)!}}\big)$.

\begin{figure}[t]
	\centering
	\centerline{\includegraphics[width=6.5cm]{cell-skeleton.eps}}
	\caption{Illustration of the cell and the skeleton.}\label{fig: 1b}
\end{figure}

Our use of a dissimilarity measure gives us knowledge about how related two tasks are. Build upon this knowledge, we can define the search space of the target task-data set pair by combining the skeletons, cells, and operations from only the most similar pairs in the dictionary. Since the search space is restricted to only related tasks, the architecture search algorithm can perform efficiently and requires few GPU hours to find the best candidate network. We have illustrated this in the experimental section.

\subsection{Search Algorithm}
\label{ssec:IDARTS}
The Fusion Search (FUSE) is a novel search algorithm that considers the network candidates as a whole and performs the optimization using gradient descent. Let $C$ be the set of candidate networks on which we define the search space. Given $c \in C$ and training data $X$, denote by $c(X)$ the output of the network candidate $c$. The FUSE algorithm, as illustrated in Algorithm 2, is based on the continuous relaxation of the network outputs. It is capable of searching through all networks in the relaxation space without fully training them. We use as our relaxed space $C$ the set of all convex combinations of candidate networks, which each weight in the combination given by exponential weights:
\begin{equation}
    \Bar{c}(X) = \sum_{c\in C}\frac{\exp{(\alpha_c)}}{\sum_{c'\in C}\exp{(\alpha_{c'})}} c(X),
\end{equation}
where $\Bar{c}$ is the weighted output of network candidate $c$, and $\alpha_c$ is a continuous variable that assigned to candidate $c$'s output. We then conduct our search by jointly training the network candidates and optimizing their $\alpha$ coefficients. Let $X_{train}$, $X_{val}$ be the training and validation data set. The training procedure is based on alternative minimization and can be divided into: (i) freeze $\alpha$ coefficients, jointly train network candidates, (ii) freeze network candidates, update $\alpha$ coefficients. Initially, $\alpha$ coefficients are set to $1/|C|$. While freezing $\alpha$, we update the weights in network candidates by jointly train the relaxed output $\Bar{c}$ with cross-validation loss on training data:
\begin{equation}{\label{eq3}}
    \min_w \mathcal{L}_{train}(w; \alpha, \Bar{c}, X_{train}),
\end{equation}

where $w$ are weights of network candidates in $C$. Next, the weights in those candidates are fixed while we update the $\alpha$ coefficients on validation data:
\begin{equation}{\label{eq4}}
    \min_\alpha \mathcal{L}_{val}(\alpha; w, \Bar{c}, X_{val}).
\end{equation}
These steps are repeated until $\alpha$ converges. The most promising candidate will be selected by: $c^* = \arg\max_{c \in C} \alpha_c$. This training procedure will deliver the best candidate among candidates in $C$ without fully training all of them. In order to go through the entire search space, this process is repeated until certain criteria, such as the number of iterations, the performance of the current most promising candidate, is met.

\begin{algorithm}[t]
\SetAlgoLined
 \textbf{Input}: search space S, $X_{train}$, $X_{val}$, I\;
 \textbf{Initilization}: $c^*$, $\alpha$\;
 \textbf{Output}: Best architecture\;

 \For{$i = 1,...,I$}{
    C = [$c^*$, and candidates sampled from S]\;
    \While{$\alpha$ not converge}{
        Update $C$ by descending $\nabla_{w} \mathcal{L}_{train}(w; \alpha, \Bar{c})$\;
        Update $\alpha$ by descending $\nabla_{\alpha} \mathcal{L}_{val}(\alpha; w, \Bar{c})$\;
    }
    $c^* = \arg\max_{c \in C}\alpha_c$\;
 }

 \caption{FUSE Algorithm}
\end{algorithm}

\section{Experimental Study}
\label{sec:experiment}
We evaluate the TA-NAS algorithm on image data sets and classification tasks. For our experiment, we initialize the TA-NAS with a set of base binary classification tasks consisting of finding specific digits in  MNIST~\cite{lecun2010mnist} and specific objects in Fashion-MNIST~\cite{xiao2017fashion}. We find $\epsilon$-representatives for each task by pre-training networks on the same architecture ($\text{conv}(32\times5\times5) \rightarrow \text{dense}(1024) \rightarrow \text{dense}(2)$). Here, we pick representative architectures that achieve at least $96\%$ accuracy on their tasks.

In order to compute the dissimilarity between architectures, we consider for $A$ and $B$ (two task-data set pairs) the first two layers of their trained $\epsilon$-representative networks, which we denote by $N_{A}$ and $N_{B}$, respectively. We then wish to find the least complex architecture that maps hidden features from one task to the other. We thus consider a transform network $N_{t}$ with a  $\text{dense}(2048) \rightarrow \text{dense}(512) \rightarrow \text{dense}(1024)$ architecture. We train $N_{t}$  with mean-square error (MSE) loss on a data set consisting of $N_{A}(X_{B})$ and those of $N_{B}(X_{B})$; here, the goal is to transform $N_{A}(X_{B})$ into $N_{B}(X_{B})$. We then iteratively prune the trained $N_{t}$ as much as possible while maintaining similar performance to $N_{t}$. We take our dissimilarity measure to be the percentage of the remaining non-zero parameters in $N_{t}$ after pruning. We show our results in Figure 2. Our results suggest that two tasks from the same data set (e.g., MNIST or Fashion-MNIST) are often more similar than tasks involving different data sets. It is perhaps interesting to note that the similarity from MNIST tasks to Fashion-MNIST tasks is greater than the similarity from Fashion-MNIST tasks to those in MNIST. Consequently, we can often use Fashion-MNIST knowledge on MNIST, but not vice-versa.

The task on which we perform NAS is binary classification on Quick, Draw!~\cite{jongejan2016quickdraw} dataset. The Quick, Draw! is a doodle drawing dataset of 345 categories. In this experiment, we select a subset of the Quick, Draw! with a similar format to MNIST and fashion-MNIST, by choosing only 10 categories (e.g., apple, baseball bat, bear, envelope, guitar, lollipop, moon, mouse, mushroom, rabbit) with each has 60,000 data points. Our task of interest is the moon indicator from this subset of Quick, Draw! data set. We pre-train this task and compute dissimilarities to other tasks in the same manner as above. To conduct our search, we choose the top three most similar tasks to the baselines discussed above: (i) digit 0, (ii) trouser, (iii) digit 3 indicators. Due to the similarity in the shape of digit 0 and the moon, the base task of digit 0 indicator is the most related task to the moon indicator.

\begin{figure}[t]
	\centering
	\centerline{\includegraphics[width=6.4cm]{distance.eps}}
	\caption{The distance matrix of baseline tasks.}\label{fig: 2}
\end{figure}

After obtaining the related tasks, we combine the operations and cell structures to generate a suitable search space for the target task. The cell consists of 4 nodes, with 6 edges of operations. The list of operations includes identity, zero, dil-conv3x3, sep-conv3x3, maxpool2x2. Next, the search algorithm is used to find the best architecture in this search space. Initially, three network architectures are randomly generated from the search space. At each iteration, FUSE quickly evaluates these candidates and only saves the best architecture for the next iteration. The search stops only when all criteria (e.g., a prespecified maximum number of iteration or best architecture converges) are met. The results in Table 1 give the best test error of the optimal architecture found by TA-NAS  after 20 trials, in comparison with a random search algorithm and other state-of-art handcrafted networks (e.g., DenseNet~\cite{huang2017densely}, ResNet~\cite{he2016deep}). The architecture produced by TA-NAS is competitive with manual-designed networks while having a smaller number of parameters. When comparing with the random search method using our pre-defined search space, TA-NAS approach achieves a higher accuracy model with less search time in terms of GPU days (i.e., the number of days for a single GPU to perform the task). Hence, this framework can utilize the knowledge of related tasks to find the efficient network architecture for the target task.

\begin{table}
\begin{tabular}{ p{4cm} | p{0.9cm} | p{1.1cm} | p{0.8cm} }
 \hline
 \multirow{2}{3cm}{\textbf{Architecture}} & \textbf{Error} & \textbf{Param}  & \textbf{GPU}\\
 & \textbf{(\%)} & \textbf{(M)} & \textbf{days}\\
 \hline
 ResNet-18~\cite{he2016deep}                & 1.42 & 11.44 & -\\
 ResNet-34~\cite{he2016deep}                & 1.2 & 21.54 & -\\
 DenseNet-161~\cite{huang2017densely}       & 1.17 & 27.6 & -\\
 \hline
 Random Search                              & 1.33 & 2.55 & 4\\
 FUSE w. standard space                     & 1.21 & 2.89 & 2\\
 FUSE w. task-aware space                   & 1.18 & 2.72 & 2\\
 \hline
\end{tabular}
\caption{\label{tab:result}Comparison with state-of-art image classifiers on Quick, Draw! dataset.}
\end{table}

\section{Conclusion}
\label{sec:conclusion}
We proposed TA-NAS, a novel task-aware framework to address the Neural Architecture Search problem. By introducing a similarity measure for given pairs of tasks and data sets, we can define a restricted, dynamic architecture search space for a new task-data set pair based on similar previously observed pairs. Additionally, we proposed the gradient-based search algorithm, FUSE, to quickly evaluate the performance of network candidates in the search space. This search algorithm can be applied to find the best way to grow or to compress the current network.

\vfill\pagebreak
\balance
\bibliographystyle{IEEEbib}

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Task-Aware Neural Architecture Search