Any variable parameter (signal) that can be controlled by human (the user) and recorded in a digital form can be used to encode input information affecting the computer functionality. However, the computer functionality to interact with the user is limited to a passive presentation of information in different types of messages delivered through signals of different modalities: auditory, visual and tactile. Both the size and power consumption of wearable devices restrict the output capabilities, the computer is able to perform to support interaction techniques. Touch screens, in particular, are becoming increasingly popular because of their high sensitivity to detect a location and pressure of an input device (a stylus tip) or a finger tip and advantages of the direct manipulation interface. In this context, manipulation still means the user input that has an effect on the program execution.
However, finger-based interaction has a limited duration of the contact between fingertip and touchscreen (direct input) to avoid visual occlusion of imaging (in a case of the use a visual output). The short contact constraints the time during that it would be possible to apply and perceive mechanical energy variations in a kind of tactile signals (forces, vibrations and/or movements) in fingers to present complex information through tactile channel. On the other hand, tactile information presented during finger movements (gesture) cannot be referred to a specific point of contact at a specific moment of time. Though, instead of the direct feedback the user is able to feel different sensations by other hand continuously holding the portable electronic device, an interpretation of tactile information delivered from the back of the device and presented to the contralateral hand can cause a spatial discordance of the depth perception and localization when tactile signals have to be integrated with the visual information in a specific manner (e.g., in a case of ambiguous images, depth cues, etc).
A stylus is a universal mediator of the user input that can be used with various devices in a manner as a regular pen or pencil, e.g. with Personal Digital Assistants (PDA), personal computers, mobile phones, smart watches (US-2014/0035884-A) and any other portable electronic device to input information in a textual, graphical and pictorial form (US-2014/0078109-A). It also improves the precision of the touch input rather than using a finger, allowing use of handwriting and micro-movements.
Nowadays, advanced styluses have extended input-output capabilities for input spell check and notification (US-2014/0253469-A, US-2005/0125228-A), by interacting with a smart phone as a Bluetooth headset (US-2012/0139879-A) and for input as a wireless 3D manipulandum (US-2014/0191967-A, US-2002/0084114-A, US-2013/0002614-A, WO-2013-003128-A). A stylus input can be used for simulating different physical qualities (stiffness, compliance, elasticity, rigidity, inertia, friction, impedance etc.) and associated perceptual qualities of materials, textures and other sensory experience. Through the stylus mediating human-computer interaction, the user is able to perceive different local properties of the virtual objects and materials such as hard, soft, sticky, and other region/shape descriptors and features as a texture gradient (slippery, silky, velvety, bumpy, smooth), concavity and convexity, edges and so on (US-2009/0079703-A, US-2014/0043242-A, U.S. Pat. No. 8,681,130-B, U.S. Pat. No. 8,749,533-B, U.S. Pat. No. 8,773,403-B, US-2012/0293463-A, US-2012/0127088-A, Wintergerst G. et al., “Reflective Haptics: Enhancing Stylus based interactions on touch screen”, EuroHaptics Conference Proceedings, 2010, Part I, LNCS 6191, 360-366).
Nowadays, in advanced multimodal interfaces besides spatial audio, enhancement of graphic cues is often based on the use of a haptic sense (Evreinova T. V et al., “Virtual Sectioning and Haptic Exploration of Volumetric Shapes in the Absence of Visual Feedback”, Advances in Human-Computer Interaction, 2013, Article ID: 740324). Many stationary (desktop) and mobile (linkage-free) input devices have been developed to augment visual interaction with three-dimensional objects through complementary haptic sense (U.S. Pat. No. 5,642,469-A, Yang “Design and Control of an integrated haptic interface for touch screen applications”, Ph.D. thesis in Lille1 University, France, 2013, Evreinova T. V. et al., “From Kinesthetic Sense to New Interaction Concepts: Feasibility and Constraints”, International Journal of Advanced Computer Technology, 2014, 3, 4, 1-33). Various solutions for different kinds of devices in three-dimensional pointing mostly rely on enhanced visual feedback even when the user can apply different pressure on a stylus tip to change the cursor location along the direction of the normal force applied (e.g., US-2009/0079703-A, WO-2011-061395-A, US-2008/0225007-A, US-2012/0206419-A, Withana A. et al., “ImpAct: Immersive haptic stylus to enable direct touch and manipulation for surface computing”, ACM Computers in Entertainment, 2010, 8, 2, Article 9, Lee J. et al., “Beyond-Collapsible Tools and Gestures for Computational Design”, CHI2010, 2010, 3931-3936, Nagasaka S. “Haptistylus: Stylus for Unified Manipulations”, 2015, available at: http://oshiro.bpe.es.osaka-u.ac.jp/research/cgvr.html).
However, to our knowledge and understanding the solutions mentioned above are able only to simulate the feelings that the user can sense by moving a stylus/pen or a paintbrush across a piece of paper or a canvas, while these solutions cannot actively move the stylus/pen or/and the user's hand with respect to the stylus or with respect to the surface of interaction (Moscatelli A. et al., “A change in the fingertip contact area induces an illusory displacement of the finger”, EuroHaptics Conference Proceedings, 2014, Part II, LNCS 8619, 72-79). For example, US-2012/0127088-A discloses that “In some embodiments, the haptic actuator may further generate haptic feedback that can be felt by the nerves of a user's fingers without physically moving the body of the haptic input device.” (paragraph [0042]). US-2014/0043242-A discloses a method for guiding a stylus on a surface of a touchscreen by moving the stylus across the surface while varying the friction based on a location of the stylus. The friction is modulated such that a region of the surface has a higher friction than areas immediately surrounding the region so as to bias the stylus towards the region, by guiding the user towards appropriate strokes (claim 20, par [0061], [0062]).
Nevertheless, the guidance that contains ambiguity and is not able to present an exact way to solve the task can fail in the absence of visual feedback and prior knowledge (preexisting attitudes, experiences, and mental templates). By other words, when the user is not able to exactly track/cross the areas having a low coefficient of friction in a specific direction, an exploration of areas surrounding the stylus tip region disintegrates kinesthetic information or complicates filtering and integration of kinesthetic information by hindering an appearance of the holistic mental representation of the path (appropriate strokes) needed to follow to complete a task. To optimize learning for the specific handwriting skills, the guidance has to facilitate filtering and integration of the kinesthetic information by applying the tangential vector of force moving the stylus along the needed pathways, while avoiding any exploratory extra movements. Still besides the known solutions for desktop devices (U.S. Pat. No. 5,642,469-A, U.S. Pat. No. 8,432,361-B, U.S. Pat. No. 8,725,292-B, US-2005/0065649-A, US-2010/0042258-A, WO-2004-095170-A, Evreinova T. V et al., “From Kinesthetic Sense to New Interaction Concepts: Feasibility and Constraints”, International Journal of Advanced Computer Technology, 2014, 3, 4, 1-33), there have not been published any attempts of implementing autonomously movable (self-propelling) stylus, pen or stick for a mobile interaction (Nagasaka S. “Haptistylus: Stylus for Unified Manipulations”, 2015, available at: http://oshiro.bpe.es.osaka-u.ac.jp/research/cgvr.html).
For example, by producing push and pull forces in synchronization with altering static and kinetic friction forces, it is possible to create a displacement vector to an object in a given direction with respect to a supporting surface (U.S. Pat. No. 3,957,162-A, U.S. Pat. No. 8,230,990-B, U.S. Pat. No. 6,841,899-B, Reznik D. S., “The Universal Planar Manipulator”, Ph. D. thesis in University of California at Berkeley, 2000, Darby A. P. et al., “Modeling and Control of a Flexible Structure Incorporating Inertial Slip-Stick Actuators”, Journal of Guidance, Control, And Dynamics, 1999, 22, 1, 36-42, Awrejcewicz J. et al., “Occurrence of Stick-Slip Phenomenon”, Journal of Theoretical and Applied Mechanics, 2007, 45, 1, 33-40). Herewith, there is a strong interest in a haptic society to apply the controllable friction, tangential force and displacement in the absence of any mechanical linkage to the user's fingerpad when s/he interacts via touchscreen with a portable electronic device (U.S. Pat. No. 8,525,778-B, Wiertlewski M. et al., “A High-Fidelity Surface-Haptic Device for Texture Rendering on Bare Finger”, EuroHaptics Conference Proceedings, 2014, Part II, LNCS 8619, 241-248, Dai X. et al., “LateralPaD: A Surface-Haptic Device That Produces Lateral Forces on A Bare Finger”, IEEE Haptics Symposium, 2012, 7-14, Chubb E. C. et al., “ShiverPaD: A Glass Haptic Surface That Produces Shear Force on a Bare Finger”, IEEE Transactions on Haptics, 2010, 3, 3, 189-198, Giraud F. et al., “Design of a transparent tactile stimulator”, Haptics Symposium, 2012, 485-489, Gleeson B. T. et al., “Perception of Direction for Applied Tangential Skin Displacement: Effects of Speed, Displacement, and Repetition”, IEEE Transactions on Haptics, 2010, 3, 3, 177-188, Winfield L. et al., “T-PaD: Tactile Pattern Display through Variable Friction Reduction”, Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2007, 421-426, Kaye J. J., “Sawtooth planar waves for haptic feedback”, Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology, 2010, 5-6, Roudaut A. et al., “Gesture Output: Eyes-Free Output Using a Force Feedback Touch Surface”, CHI2013, 2013, 2547-2556, Saga S. et al., “Simultaneous geometry and texture display based on lateral force for touchscreen”, IEEE World Haptics Conference, 2013, 437-442). For example, Derler S. et al., “Stick-slip phenomena in the friction of human skin”, Wear, 2013, 301, 324-329 mentions the stick-slip behavior of the index fingerpad sliding on wet, smooth glass as a function of normal force and sliding velocity in friction measurements using a tri-axial force plate.
In general, the friction coefficients during the stick-slip phase of sliding were 30% lower than those in a stationary phase of sliding. During a stick-slip phase, the amplitude of the friction coefficient also varied more than twice greater than during a stationary phase of sliding. As soon as new materials and actuators are available, the stick-slip phenomenon can be realized on a portable electronic device and in designing the stylus as a mediator of interaction with portable electronic devices. This can significantly extend functionalities of the stylus-based interaction that has been realized in the present invention.