Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Three-dimensional modeling has long been regarded as an ideal application for virtual reality (VR), but current VR-based 3D modeling tools suffer from two problems that limit creativity and applicability: (1) the lack of control for freehand modeling, and (2) the difficulty of starting from scratch. To address these challenges, we present Lift-Off, an immersive 3D interface for creating complex models with a controlled, handcrafted style. Artists start outside of VR with 2D sketches, which are then imported and positioned in VR. Then, using a VR interface built on top of image processing algorithms, 2D curves within the sketches are selected interactively and “lifted” into space to create a 3D scaffolding for the model. Finally, artists sweep surfaces along these curves to create 3D models. Evaluations are presented for both long-term users and for novices who each created a 3D sailboat model from the same starting sketch. Qualitative results are positive, with the visual style of the resulting models of animals and other organic subjects as well as architectural models matching what is possible with traditional fine art media. In addition, quantitative data from logging features built into the software are used to characterize typical tool use and suggest areas for further refinement of the interface. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Sketching designs has been shown to be a useful way of planning and considering alternative solutions. The use of lo-fidelity prototyping, especially paper-based sketching, can save time, money and converge to better solutions more quickly. However, this design process is often viewed to be too informal. Consequently users do not know how to manage their thoughts and ideas (to first think divergently, to then finally converge on a suitable solution). We present the Five Design Sheet (FdS) methodology. The methodology enables users to create information visualization interfaces through lo-fidelity methods. Users sketch and plan their ideas, helping them express different possibilities, think through these ideas to consider their potential effectiveness as solutions to the task (sheet 1); they create three principle designs (sheets 2,3 and 4); before converging on a final realization design that can then be implemented (sheet 5). In this article, we present (i) a review of the use of sketching as a planning method for visualization and the benefits of sketching, (ii) a detailed description of the Five Design Sheet (FdS) methodology, and (iii) an evaluation of the FdS using the System Usability Scale, along with a case-study of its use in industry and experience of its use in teaching. <s> BIB002 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Hand-Drawn sketches are natural means by which abstract descriptions of environments can be provided. They represent weak prior information about the scene, thereby enabling a robot to perform autonomous navigation and exploration when a full metrical description of the environment is not available beforehand. In this paper, we present an extensive evaluation of our navigation system that uses a sketch interface to allow the operator of a robot to draw a rough map of an indoor environment as well as a desired trajectory for the robot to follow. We employ a theoretical framework for sketch interpretation, in which associations between the sketch and the real world are modeled as local deformations of a suitable metric manifold. We investigate the effectiveness of our system and present empirical results from a set of experiments in real-world scenarios, focusing both on the navigation capabilities and the usability of the interface. <s> BIB003 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Mid-air 3D sketching has been mainly explored in Virtual Reality (VR) and typically requires special hardware for motion capture and immersive, stereoscopic displays. The recently developed motion tracking algorithms allow real-time tracking of mobile devices, and have enabled a few mobile applications for 3D sketching in Augmented Reality (AR). However, they are more suitable for making simple drawings only, since they do not consider special challenges with mobile AR 3D sketching, including the lack of stereo display, narrow field of view, and the coupling of 2D input, 3D input and display. To address these issues, we present Mobi3DSketch, which integrates multiple sources of inputs with tools, mainly different versions of 3D snapping and planar/curves surface proxies. Our multimodal interface supports both absolute and relative drawing, allowing easy creation of 3D concept designs in situ. The effectiveness and expressiveness of Mobi3DSketch are demonstrated via a pilot study. <s> BIB004 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Augmented Reality(AR) tools are currently primarily targeted at programmers, making designing for AR challenging and time-consuming. We developed an interactive prototype, PintAR, that enables the authoring and rapid-prototyping of situated experiences by allowing designers to bring their ideas to life using a digital pen for sketching and a head-mounted display for visualizing and interacting with virtual content. In this paper, we explore the versatility such a tool could provide through case studies of a researcher, an artist, a ballerina, and a clinician. <s> BIB005 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Users within a Virtual Environment often need support designing the environment around them with the need to find relevant content while remaining immersed. We focus on the familiar sketch-based interaction to support the process of content placing and specifically investigate how interactions from a tablet or desktop translate into the virtual environment. To understand sketching interaction within a virtual environment, we compare different methods of sketch interaction, i.e., 3D mid-air sketching, 2D sketching on a virtual tablet, 2D sketching on a fixed virtual whiteboard, and 2D sketching on a real tablet. The user remains immersed within the environment and queries a database containing detailed 3D models and replace them into the virtual environment. Our results show that 3D mid-air sketching is considered to be a more intuitive method to search a collection of models; while the addition of physical devices creates confusion due to the complications of their inclusion within a virtual environment. While we pose our work as a retrieval problem for 3D models of chairs, our results are extendable to other sketching tasks for virtual environments. <s> BIB006 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> User Interface (UI) prototyping is a necessary step in the early stages of application development. Transforming sketches of a Graphical User Interface (UI) into a coded UI application is an uninspired but time-consuming task performed by a UI designer. An automated system that can replace human efforts for straightforward implementation of UI designs will greatly speed up this procedure. The works that propose such a system primarily focus on using UI wireframes as input rather than hand-drawn sketches. In this paper, we put forward a novel approach wherein we employ a Deep Neural Network that is trained on our custom database of such sketches to detect UI elements in the input sketch. Detection of objects in sketches is a peculiar visual recognition task that requires a specific solution that our deep neural network model attempts to provide. The output from the network is a platform-independent UI representation object. The UI representation object is a dictionary of key-value pairs to represent the UI elements recognized along with their properties. This is further consumed by our UI parser which creates code for different platforms. The intrinsic platform-independence allows the model to create a UI prototype for multiple platforms with single training. This two-step approach without the need for two trained models improves over other methods giving time-efficient results (average time: 129 ms) with good accuracy. <s> BIB007 </s> Deep Learning for Free-Hand Sketch: A Survey <s> POTENTIAL RESEARCH DIRECTIONS <s> Despite their renowned predictive power on i.i.d. data, convolutional neural networks are known to rely more on high-frequency patterns that humans deem superficial than on low-frequency patterns that agree better with intuitions about what constitutes category membership. This paper proposes a method for training robust convolutional networks by penalizing the predictive power of the local representations learned by earlier layers. Intuitively, our networks are forced to discard predictive signals such as color and texture that can be gleaned from local receptive fields and to rely instead on the global structures of the image. Across a battery of synthetic and benchmark domain adaptation tasks, our method confers improved generalization out of the domain. Also, to evaluate cross-domain transfer, we introduce ImageNet-Sketch, a new dataset consisting of sketch-like images, that matches the ImageNet classification validation set in categories and scale. <s> BIB008
Nowadays, not only deep learning is developing rapidly, but also various data collection equipment is constantly updated. This brings many new opportunities and challenges to free-hand sketch community. There are some potential research directions in future: (i) With the developments of data collection equipment, collecting 3D sketches  is easier than previous. There are many interesting 3D sketch related research topics, e.g., combining virtual reality (VR) BIB001 and augmented reality (AR) BIB004 , BIB005 , BIB006 . 3D sketch studies will help to bring the sketch-based human-computer interaction from the 2D planes of the touch-screens to the 3D spaces, enabling users have immersive experience. (ii) Existing sketch works focus on the sketches that depict objects and scenes. In the practical application scenarios, users want to machines understand the input sketching contents as many as possible. Thus, beyond object and scene, other practical sketch contents also need to be studied, e.g., sheets, curves, histograms BIB002 , maps BIB003 , user interface (UI) prototype drawings BIB007 . Sketch also can be studied with hand-written character together. (iii) All the existing free-hand sketches are collected by common touch-screen devices, e.g., mobile phone, ipad. Hence, the position information of strokes is the main feature in previous works. Currently, professional hand-drawn collection equipment is able to sense the pressure along strokes, thus more features for sketch stroke analysis can be collected. This is also worth studying in future. (iv) Sketch is particularly diverse, abstract, and noisy, so that the sketch-oriented deep models need more robustness. However, the deep model robustness is really under-studied in sketch community to date. In future, sketch can be studied from the perspective of model robustness. Furthermore, adversarial attack learning also can be involved to sketch study. (v) Sketch is a special data domain. Beyond humancomputer interaction related applications, sketch has some other potential uses. Recently, sketch-like images (e.g., ImageNet-Sketch BIB008 ) are used to evaluate the generalization ability of deep models. Thus, for both uni-modal and cross-modal cases, free-hand sketch can be used as a more difficult/challenging test-bed to evaluate the domain adaptation/transferability/robustness of the deep models. For instance, essentially speaking, both of fine-grained sketchbase image retrieval (FG-SBIR) and person re-identification (Re-ID) are instance-level matching problems. Therefore, researchers can evaluate the generalization ability of person re-identification models on FG-SBIR tasks.