Because of the parallels found with human language production and acquisition, birdsong is an ideal animal model to study general mechanisms underlying complex, learned motor behavior. The rich and diverse vocalizations of songbirds emerge as a result of the interaction between a pattern generator in the brain and a highly nontrivial nonlinear periphery. Much of the complexity of this vocal behavior has been understood by studying the physics of the avian vocal organ, particularly the syrinx. A mathematical model describing the complex periphery as a nonlinear dynamical system leads to the conclusion that nontrivial behavior emerges even when the organ is commanded by simple motor instructions: smooth paths in a low dimensional parameter space. An analysis of the model provides insight into which parameters are responsible for generating a rich variety of diverse vocalizations, and what the physiological meaning of these parameters is. By recording the physiological motor instructions elicited by a spontaneously singing muted bird and computing the model on a Digital Signal Processor in real-time, we produce realistic synthetic vocalizations that replace the bird' s own auditory feedback. In this way, we build a bio-prosthetic avian vocal organ driven by a freely behaving bird via its physiologically coded motor commands. Since it is based on a low-dimensional nonlinear mathematical model of the peripheral effector, the emulation of the motor behavior requires light computation, in such a way that our bio-prosthetic device can be implemented on a portable platform.
The complex motor behavior originating the rich vocalizations of adult oscine birds results from the interaction between a central pattern generator (the brain) and a nonlinear biomechanical periphery (the bird' s vocal organ) [1], [2]. The fact that this complex behavior is learned, together with the parallels between the physical mechanisms of birdsong and human speech production, make birdsong an ideal model to study how a complex motor behavior is acquired, produced and maintained [3]. In an effort to understand what gives rise to complexity in this behavior, a part of the birdsong community has set focus on the capabilities of the periphery to produce vocalizations owning a diverse set of nontrivial acoustic features [2]. The avian vocal organ, comprised mainly by the respiratory system, the syrinx and the vocal tract, is a highly nonlinear biomechanical device. The complexity of its dynamics leaves traces in the sounds that can be produced in it. In this way, several acoustic features found in vocalizations can be related to nonlinear phenomena occurring in the syrinx [4], [5] or introduced by acoustic interactions between the syrinx and the tract [6]–[9]. In all these cases, the complexity of the behavior does not require a complex motor pattern to drive the vocal organ, but rather simple, smooth gestures. Through a combination of experimental observations and theoretical analysis, low-dimensional mathematical models have been proposed that account for the physical mechanisms of sound production in the avian vocal organ [10], [11]. In particular, a model based on Titze' s proposed flapping mechanism for oscillations in human vocal folds [12] was recently used to synthesize the song of the Zebra finch (Taeniopygia guttata) [13], [14]. This model captures the nonlinear dynamics of the folds oscillating to produce sound, in a way that a variety of complex vocalizations are generated by the tuning of parameters related to physiologically observable motor gestures elicited by the bird. Part of the appeal of counting with this model is the prospect of applying it to the construction of a bio-prosthetic device. In this scenario computation is relatively inexpensive because of the low dimension of the mathematical model. In addition, the physical description of the peripheral effectors led to the identification of a set of smoothly varying parameters that determine the behavior [13]. By recording the physiological activity related to the parameters and feeding it to a device that solves the equations of the model in real-time, vocal behavior can be emulated by a prosthesis controlled by a subject via its motor instructions. The usual strategy of BCIs and BMIs (Brain Computer Interfaces and Brain Machine Interfaces) is to decode motor commands from recordings of physiological activity in the brain and use this activity to control bio-mimetic devices [15]–[17]. In [17], for instance, multi-electrode recordings of tens to hundreds of neurons in different cortical areas of primates are used to drive a robotic arm. In recent work, Cichocki et. al. discuss the perspectives of using electroencephalographic (EEG) recordings to generate noninvasive BCI solutions [16]. In these examples (as well as in many other BCI implementations), the crucial problem is the classification of the features of the large data set which correspond to a determined set of motor tasks. The feature extraction is performed by different techniques which include linear decomposition in a diversity of vector spaces and machine learning algorithms [15], [16], [18]. In this way, accurate control of bio-mimetic effectors is achieved for a finite number of specific tasks, such as grasping or cursor moving. Our current understanding of the biophysics of the avian vocal organ, particularly our capacity to identify the dynamical mechanisms by which complex behavior occurs when the peripheral systems are driven by low dimensional, smooth instructions, allows us to propose an example of a different kind of bio-prosthetic solution. The model predicts a diversity of qualitatively different solutions to the system for continuous paths in a parameter space. Not only is this parameter space suggested by the model, but it is also physiologically pertinent. We present a device that is driven by a freely behaving Zebra finch to produce realistic, synthetic vocalizations in real-time. The device is based on the real time integration of the mathematical model of the vocal organ on a Digital Signal Processor (DSP). It is controlled by the bird' s subsyringeal air sac pressure gesture, which is transduced, digitized and fed to the DSP to provide the model with the appropriate path in parameter space. The work is organized as follows. In the Methods section we describe the Zebra finch vocal organ and the mathematical model that accounts for its dynamics. We also discuss its applicability to the construction of a model-based bio-prosthetic device, and introduce the physiological motor gestures that relate to parameters of the model. We present the steps leading to the real-time implementation of the model on a device controlled by a spontaneously singing bird. In the Results section we show the example of a successful case. Finally, we summarize the results and discuss the impact of this device as an example of a kind of bio-prosthetic device enabled by the low-dimensional dynamical model of the peripheral effector.
All experiments were conducted in accordance with the Institutional Animal Care and Use Committee of the University of Utah. One of the most studied species of songbirds is the Zebra finch. Its song presents a set of diverse acoustic features, which can be accounted for by the dynamics displayed by the mathematical model of its vocal organ [5], [11]. This low-dimensional mathematical model, when driven by the appropriate gesture in parameter space, is capable of producing realistic, synthetic birdsong. By implementing this model on a Digital Signal Processor, we are able to construct a bio-prosthetic vocal organ. The mathematical model for the vocal organ, the reduction of the system ruling the dynamics of the sound source, and the identification of the pertinent parameters accounting for its motor control, they all make way for the construction of a bio-prosthetic device. The parameters determining acoustic properties of vocalizations in the normal form (5) are physiologically meaningful and the set of differential equations is easy enough to compute in a portable platform such as a Digital Signal Processor. By fitting the parameters and integrating the system in real-time, synthetic song can be produced in a device controlled by the motor instructions elicited by a freely behaving bird. In many bio-prosthetic solutions, the physiological motor gestures used to drive the device are degraded respect to those recorded in the intact subjects [15]. One application of this device is the performance of altered auditory feedback experiments [32]–[34]. In experiments enabled by this device, the bird' s own auditory feedback can be replaced by synthetic birdsong computed in real-time. Since the synthetic feedback is produced by the integration of the model when fed with actual physiological motor gestures elicited by a freely behaving bird, alterations of the feedback are possible that are consistent with alterations in the motor gestures intended to produce them. Here, the prosthetic vocal organ is driven by a bird that is muted via the insertion of a cannula through its inter-clavicular air sac. Phonation is prevented as the airflow is bypassed away from the syrinx. As a side effect, the pressure pattern registered on the muted bird differs from that recorded in the intact animal. The device reconstructs the intended motor gesture from this degraded pressure gesture to trigger integration of the model. When the pressure pattern corresponding to the syllables comprising a motif are identified, the mathematical model for the vocal organ is computed with the appropriate paths in parameter space, to produce the corresponding synthetic output.
The device succeeds in synthesizing song online when driven by the pressure gesture of a muted bird. From the altered motor gesture, the algorithm infers the segment of a motif intended by the bird and computes the model to produce the vocalizations. An example is illustrated in Fig. 5. The upper panels of the figure display the recorded subsyringeal pressure and sonogram of a segment of a bout with its preceding introductory call. A song bout of this bird (B06) is composed of a number of introductory notes (O) and the repetition of a simple motif containing two syllables (A and B), indicated by different colors and opacity of shading in Fig. 5. An initial segment of syllable A (marked with clearer shading in the figure) is used to detect the intention to elicit a motif in the pressure gesture of the muted bird. The part that is produced upon triggering of the synthesis appears in a darker shade of the same color. These recordings were used to fit the parameters of the model to produce synthetic vocalizations showing a match in fundamental frequency and spectral content. The bird is then muted and placed in the setup to drive the electronic vocal organ with its pressure gesture. In the lower panel of Fig. 5 we show the pressure gesture of the muted bird and the sonogram of the sound recorded by a microphone placed close (about) to the bird and the speaker. It can be noted in the sonogram that no sound is produced during the bird' s attempts to phonate an introductory note. When the pressure motor gesture corresponds to the first syllable in the bout (syllable A), the instruction is recognized and the corresponding song is synthesized and played through the speaker. In the first bout, the bird only elicits the gesture corresponding to the first syllable, and then stops. In this case the algorithm detects the interruption and turns off the integration. In the second bout, the bird continues with the second syllable and drives the electronic syrinx to the end of the motif. This example illustrates how this device works, and shows that it is successful in synthesizing the song motif as the bird drives it. We evaluate its success by counting the times the motif was properly detected and synthesized, and how many times a false trigger occurred. During a session of hours (the second day after the muting took place), a muted bird elicited about calls, out of which less than generated false triggering of synthetic song. In most cases the false trigger event was recognized by the algorithm and silenced after less than. The rate of success in detecting the beginning of a bout was of about in attempted bouts elicited by the bird. Despite the variability of the altered pressure gesture in the subsequent days (days after the muting), a brief calibration before each daily session allowed the rates of success and false triggers to be maintained. To do this calibration, the pressure gesture was recorded during and these data were used to re-set the cross-correlation thresholds, while keeping the test segment of the intact pressure gesture previously selected. We show here that realistic vocal behavior is synthesized in real-time by our device, as it is controlled by the spontaneous behavior of a muted bird, a physiological signal (its air sac pressure) that is degraded in respect to the one recorded in the intact bird. The computing platform is a low cost, portable processor, and the initial rate of success is high. This is an encouraging example of the plausibility of a kind of interface between the central motor pattern generator and the synthetic, bio-mimetic behavior. DSP technology is being implemented in a variety of biologically inspired problems, and together with Field Programmable Gate Array technology (FPGA) is likely to become a standard solution for a variety of bio-mimetic applications [15], [35]. Brain computer and brain machine interfaces (BCI and BMI) typically read physiological data and attempt to decode motor instructions that drive peripheral devices in order to produce synthetic behavior [15], [17], [35], [36]. Because we have a physical model of the peripheral effector, the origins of the complexity of the behavior were linked to smooth paths in a low dimensional parameter space. Following the identification of the pertinent parameters and their physiological link to the pattern generator (activity of the ventral syringeal muscle and sub-syringeal air sac pressure), a further simplification of the system was carried out, on dynamical grounds, by eliminating irrelevant nonlinear terms (performing a reduction of the model to its normal form). This led to the possibility of implementing our bio-prosthetic device on a programmable electronic platform. Since the computing capabilities of the platform are greatly enhanced by the low costs of our implementation, technological advances in this front will have great impact on the complexity of the peripheral biomechanical system that can be emulated.
We have built a device that emulates complex motor behavior when driven by a subject by its actual (yet degraded) physiological motor gestures. It successfully reproduces the result of the stereotyped motor gesture that leads to the behavior, i. e. , the diverse and complex set of sounds comprising the bird' s song bout. The realistic synthetic vocalizations are produced in real-time, by computing a mathematical model of the vocal organ on a portable Digital Signal Processor. The relative computational and technological simplicity of the device relies on the current level of understanding of the peripheral biomechanical effector [1], [2], [12], [24]. We have been able to construct a physical model of the syrinx that presents the adequate level of description and converges to a low dimensional (as low as two dimensions) dynamical system. The deterministic model of the vocal organ on which our device is based is not a statistical attempt to capture causal relationships between motor commands and behavior; instead, it is a hierarchization of the interactions within the biomechanical periphery and with the pattern generator. It aims to identify the dynamical mechanisms by which the behavior is produced. Furthermore, exploration of the model leads to the finding that much of the diversity and complexity of the behavior can be explained in terms of the dynamical features of this nonlinear system [5], [28], requiring only simple instructions of the nervous system to produce a rich variety of vocalizations. Just as it is identified that a low dimensional system reproduces the main features of the complexity of the vocal organ, it can also be concluded that the control parameters are few and their behavior is simple (i. e. , the physiological motor gestures linked to the paths in space are smooth). In this way, the parameter space of the model not only suggests the pertinent physiological instructions determining the main properties of the output but also how they are expected to behave. Paths in parameter space reconstructed in order for the model to produce vocalizations matching experimental recordings are indeed effective in predicting the physiological motor gestures [14]. In addition, knowledge of nonlinear dynamics allows us to find the simplest system with equivalent oscillatory behavior. The reduction of the low dimensional mathematical model for the syrinx to its normal form reduces the computational requirements and makes way for the implementation on a real-time computing solution, such as a DSP. Realistic vocal behavior is synthesized online, controlled by the motor gesture of a freely behaving muted bird, which is a physiological signal that is degraded respect to the one recorded in the intact bird. This was achieved by computing in real time a mathematical model describing the mechanisms of sound production in the interface between the motor pattern generator and the behavior, the highly nonlinear vocal organ. The computing platform is a low cost, portable processor. This successful avian vocal prosthesis is an encouraging example of the plausibility of a kind of interface between the central motor pattern generator and the synthetic, bio-mimetic behavior. An advance towards models in which certain complex features of the motor behavior are understood in terms of the underlying nonlinear mechanisms of the peripheral effectors has the potential to enhance solutions of brain-bio-mimetic effector interfaces in many ways.
Brain Machine Interfaces (BMIs) decode motor instructions from neuro-physiological recordings and feed them to bio-mimetic effectors. Many applications achieve high accuracy on a limited number of tasks by applying statistical methods to these data to extract features corresponding to certain motor instructions. We built a bio-prosthetic avian vocal organ. The device is based on a low-dimensional mathematical model that accounts for the dynamics of the bird' s vocal organ and robustly relates smooth paths in a physiologically meaningful parameter space to complex sequences of vocalizations. The two physiological motor gestures (sub-syringeal pressure and ventral syringeal muscular activity), are reconstructed from the bird' s song, and the model is implemented on a portable Digital Signal Processor to produce synthetic birdsong when driven by a freely behaving bird via the sub-syringeal pressure gesture. This exemplifies the plausibility of a type of synthetic interfacing between the brain and a complex behavior. In this type of devices, the understanding of the bio-mechanics of the periphery is key to identifying a low dimensional physiological signal coding the motor instructions, therefore enabling real-time implementation at a low computational cost.
biotechnology
signal processing
biomimetics
biomechanics
biological systems engineering
mathematics
computational neuroscience
coding mechanisms
bioengineering
biology
nonlinear dynamics
biophysics
speech signal processing
biophysic al simulations
computational biology
engineering
Prosthetic Avian Vocal Organ Controlled by a Freely Behaving Bird Based on a Low Dimensional Model of the Biomechanical Periphery