Trial-to-trial variability in the execution of movements and motor skills is ubiquitous and widely considered to be the unwanted consequence of a noisy nervous system. However, recent studies have suggested that motor variability may also be a feature of how sensorimotor systems operate and learn. This view, rooted in reinforcement learning theory, equates motor variability with purposeful exploration of motor space that, when coupled with reinforcement, can drive motor learning. Here we review studies that explore the relationship between motor variability and motor learning in both humans and animal models. We discuss neural circuit mechanisms that underlie the generation and regulation of motor variability and consider the implications that this work has for our understanding of motor learning. Expected final online publication date for the Annual Review of Neuroscience Volume 40 is July 8, 2017. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Trial-and-error learning requires evaluating variable actions and reinforcing successful variants. In songbirds, vocal exploration is induced by LMAN, the output of a basal ganglia-related circuit that also contributes a corrective bias to the vocal output. This bias is gradually consolidated in RA, a motor cortex analogue downstream of LMAN. We develop a new model of such two-stage learning. Using stochastic gradient descent, we derive how the activity in 'tutor' circuits (e.g., LMAN) should match plasticity mechanisms in 'student' circuits (e.g., RA) to achieve efficient learning. We further describe a reinforcement learning framework through which the tutor can build its teaching signal. We show that mismatches between the tutor signal and the plasticity mechanism can impair learning. Applied to birdsong, our results predict the temporal structure of the corrective bias from LMAN given a plasticity rule in RA. Our framework can be applied predictively to other paired brain areas showing two-stage learning.
Rapid and reversible manipulations of neural activity in behaving animals are transforming our understanding of brain function. An important assumption underlying much of this work is that evoked behavioural changes reflect the function of the manipulated circuits. We show that this assumption is problematic because it disregards indirect effects on the independent functions of downstream circuits. Transient inactivations of motor cortex in rats and nucleus interface (Nif) in songbirds severely degraded task-specific movement patterns and courtship songs, respectively, which are learned skills that recover spontaneously after permanent lesions of the same areas. We resolve this discrepancy in songbirds, showing that Nif silencing acutely affects the function of HVC, a downstream song control nucleus. Paralleling song recovery, the off-target effects resolved within days of Nif lesions, a recovery consistent with homeostatic regulation of neural activity in HVC. These results have implications for interpreting transient circuit manipulations and for understanding recovery after brain lesions.
Addressing how neural circuits underlie behavior is routinely done by measuring electrical activity from single neurons during experimental sessions. While such recordings yield snapshots of neural dynamics during specified tasks, they are ill-suited for tracking single-unit activity over longer timescales relevant for most developmental and learning processes, or for capturing neural dynamics outside of task context. Here we describe an automated platform for continuous long-term recordings of neural activity and behavior in freely moving animals. An unsupervised algorithm identifies and tracks the activity of single units over weeks of recording, dramatically simplifying the analysis of large datasets. Months-long recordings from motor cortex and striatum made and analyzed with our system revealed remarkable stability in basic neuronal properties, such as firing rates and inter-spike interval distributions. Interneuronal correlations and the representation of different movements and behaviors were similarly stable. This establishes the feasibility of high-throughput long-term extracellular recordings in behaving animals.
Temporally precise motor sequences underlie many motor skills and innate actions, yet the flexibility with which the timing of such stereotyped behaviors can be modified is not known. To probe this, we induced adaptive changes to the temporal structure of birdsong. We find that the duration of specific song segments can be modified without affecting the timing in other parts of the song. We derive formal prescriptions for how neural networks can implement such flexible motor timing. We find that randomly connected recurrent networks, a common approximation for how neocortex is wired, do not generally conform to these, though certain implementations can approximate them. We show that feedforward networks, by virtue of their one-to-one mapping between network activity and time, are better suited. Our study provides general prescriptions for how pattern generator networks can implement flexible motor timing, an important aspect of many motor skills, including birdsong and human speech.
Motor cortex is widely believed to underlie the acquisition and execution of motor skills, but its contributions to these processes are not fully understood. One reason is that studies on motor skills often conflate motor cortex's established role in dexterous control with roles in learning and producing task-specific motor sequences. To dissociate these aspects, we developed a motor task for rats that trains spatiotemporally precise movement patterns without requirements for dexterity. Remarkably, motor cortex lesions had no discernible effect on the acquired skills, which were expressed in their distinct pre-lesion forms on the very first day of post-lesion training. Motor cortex lesions prior to training, however, rendered rats unable to acquire the stereotyped motor sequences required for the task. These results suggest a remarkable capacity of subcortical motor circuits to execute learned skills and a previously unappreciated role for motor cortex in "tutoring" these circuits during learning.
Motor skill learning is characterized by improved performance and reduced motor variability. The neural mechanisms that couple skill level and variability, however, are not known. The zebra finch, a songbird, presents a unique opportunity to address this question because production of learned song and induction of vocal variability are instantiated in distinct circuits that converge on a motor cortex analogue controlling vocal output. To probe the interplay between learning and variability, we made intracellular recordings from neurons in this area, characterizing how their inputs from the functionally distinct pathways change throughout song development. We found that inputs that drive stereotyped song-patterns are strengthened and pruned, while inputs that induce variability remain unchanged. A simple network model showed that strengthening and pruning of action-specific connections reduces the sensitivity of motor control circuits to variable input and neural 'noise'. This identifies a simple and general mechanism for learning-related regulation of motor variability.
To signal the onset of salient sensory features or execute well-timed motor sequences, neuronal circuits must transform streams of incoming spike trains into precisely timed firing. To address the efficiency and fidelity with which neurons can perform such computations, we developed a theory to characterize the capacity of feedforward networks to generate desired spike sequences. We find the maximum number of desired output spikes a neuron can implement to be 0.1-0.3 per synapse. We further present a biologically plausible learning rule that allows feedforward and recurrent networks to learn multiple mappings between inputs and desired spike sequences. We apply this framework to reconstruct synaptic weights from spiking activity and study the precision with which the temporal structure of ongoing behavior can be inferred from the spiking of premotor neurons. This work provides a powerful approach for characterizing the computational and learning capacities of single neurons and neuronal circuits.
Individual differences in motor learning ability are widely acknowledged, yet little is known about the factors that underlie them. Here we explore whether movement-to-movement variability in motor output, a ubiquitous if often unwanted characteristic of motor performance, predicts motor learning ability. Surprisingly, we found that higher levels of task-relevant motor variability predicted faster learning both across individuals and across tasks in two different paradigms, one relying on reward-based learning to shape specific arm movement trajectories and the other relying on error-based learning to adapt movements in novel physical environments. We proceeded to show that training can reshape the temporal structure of motor variability, aligning it with the trained task to improve learning. These results provide experimental support for the importance of action exploration, a key idea from reinforcement learning theory, showing that motor variability facilitates motor learning in humans and that our nervous systems actively regulate it to improve learning.
Nature Neuroscience. January 2014. (pdf). Write up in the Harvard Gazette here. News and Views from Nature Neuroscience (pdf).*Co-senior authors.
Executing a motor skill requires the brain to control which muscles to activate at what times. How these aspects of control-motor implementation and timing-are acquired, and whether the learning processes underlying them differ, is not well understood. To address this, we used a reinforcement learning paradigm to independently manipulate both spectral and temporal features of birdsong, a complex learned motor sequence, while recording and perturbing activity in underlying circuits. Our results uncovered a striking dissociation in how neural circuits underlie learning in the two domains. The basal ganglia was required for modifying spectral, but not temporal, structure. This functional dissociation extended to the descending motor pathway, where recordings from a premotor cortex analog nucleus reflected changes to temporal, but not spectral, structure. Our results reveal a strategy in which the nervous system employs different and largely independent circuits to learn distinct aspects of a motor skill.
Neuron. 80(2):494-506. September 2013 (pdf). Write up in the Harvard Gazette here. Software package and users manual for implementing the CAF experiments described in the paper can be downloaded here.
Addressing the neural mechanisms underlying complex learned behaviors requires training animals in well-controlled tasks, an often time-consuming and labor-intensive process that can severely limit the feasibility of such studies. To overcome this constraint, we developed a fully computer-controlled general purpose system for high-throughput training of rodents. By standardizing and automating the implementation of predefined training protocols within the animal's home-cage our system dramatically reduces the efforts involved in animal training while also removing human errors and biases from the process. We deployed this system to train rats in a variety of sensorimotor tasks, achieving learning rates comparable to existing, but more laborious, methods. By incrementally and systematically increasing the difficulty of the task over weeks of training, rats were able to master motor tasks that, in complexity and structure, resemble ones used in primate studies of motor sequence learning. By enabling fully automated training of rodents in a home-cage setting this low-cost and modular system increases the utility of rodents for studying the neural underpinnings of a variety of complex behaviors.
The ability to chronically record from populations of neurons in freely behaving animals has proven an invaluable tool for dissecting the function of neural circuits underlying a variety of natural behaviors, including navigation(1) , decision making (2,3), and the generation of complex motor sequences(4,5,6). Advances in precision machining has allowed for the fabrication of light-weight devices suitable for chronic recordings in small animals, such as mice and songbirds. The ability to adjust the electrode position with small remotely controlled motors has further increased the recording yield in various behavioral contexts by reducing animal handling.(6,7) Here we describe a protocol to build an ultra-light motorized microdrive for long-term chronic recordings in small animals. Our design evolved from an earlier published version(7), and has been adapted for ease-of use and cost-effectiveness to be more practical and accessible to a wide array of researchers. This proven design (8,9,10,11) allows for fine, remote positioning of electrodes over a range of ~ 5 mm and weighs less than 750 mg when fully assembled. We present the complete protocol for how to build and assemble these drives, including 3D CAD drawings for all custom microdrive components.
Premotor circuits help generate imitative behaviors and can be activated during observation of another animal's behavior, leading to speculation that these circuits participate in sensory learning that is important to imitation. Here we tested this idea by focally manipulating the brain activity of juvenile zebra finches, which learn to sing by memorizing and vocally copying the song of an adult tutor. Tutor song-contingent optogenetic or electrical disruption of neural activity in the pupil's song premotor nucleus HVC prevented song copying, indicating that a premotor structure important to the temporal control of birdsong also helps encode the tutor song. In vivo multiphoton imaging and neural manipulations delineated a pathway and a candidate synaptic mechanism through which tutor song information is encoded by premotor circuits. These findings provide evidence that premotor circuits help encode sensory information about the behavioral model before shaping and executing imitative behaviors.
Nature Neuroscience. 15(10):1454-9. October 2012 (PDF). *Co-senior authors.
Neural circuits underlying complex learned behaviors, such as speech in humans, develop under genetic constraints and in response to environmental influences. Little is known about the rules and mechanisms through which such circuits form. We argue that songbirds, with their discrete and well studied neural pathways underlying a complex and naturally learned behavior, provide a powerful model for addressing these questions. We briefly review current knowledge of how the song circuit develops during learning and discuss new possibilities for advancing the field given recent technological advances.
The acquisition of complex motor sequences often proceeds through trial-and-error learning, requiring the deliberate exploration of motor actions and the concomitant evaluation of the resulting performance. Songbirds learn their song in this manner, producing highly variable vocalizations as juveniles. As the song improves, vocal variability is gradually reduced until it is all but eliminated in adult birds. In the present study we examine how the motor program underlying such a complex motor behavior evolves during learning by recording from the robust nucleus of the arcopallium (RA), a motor cortex analog brain region. In young birds, neurons in RA exhibited highly variable firing patterns that throughout development became more precise, sparse, and bursty. We further explored how the developing motor program in RA is shaped by its two main inputs: LMAN, the output nucleus of a basal ganglia-forebrain circuit, and HVC, a premotor nucleus. Pharmacological inactivation of LMAN during singing made the song-aligned firing patterns of RA neurons adultlike in their stereotypy without dramatically affecting the spike statistics or the overall firing patterns. Removing the input from HVC, on the other hand, resulted in a complete loss of stereotypy of both the song and the underlying motor program. Thus our results show that a basal ganglia-forebrain circuit drives motor exploration required for trial-and-error learning by adding variability to the developing motor program. As learning proceeds and the motor circuits mature, the relative contribution of LMAN is reduced, allowing the premotor input from HVC to drive an increasingly stereotyped song.
How neural circuits underlie the acquisition and control of learned motor behaviors has traditionally been explored in monkeys and, more recently, songbirds. The development of genetic tools for functional circuit analysis in rodents, the availability of transgenic animals with well characterized phenotypes, and the relative ease with which rats and mice can be trained to perform various motor tasks, make rodents attractive models for exploring the neural circuit mechanisms underlying the acquisition and production of learned motor skills. Here we discuss the advantages and drawbacks of this approach, review recent trends and results, and outline possible strategies for wider adoption of rodents as a model system for complex motor learning.
Certain ganglion cells in the retina respond sensitively to differential motion between the receptive field center and surround, as produced by an object moving over the background, but are strongly suppressed by global image motion, as produced by the observer's head or eye movements. We investigated the circuit basis for this object motion sensitive (OMS) response by recording intracellularly from all classes of retinal interneurons while simultaneously recording the spiking output of many ganglion cells. Fast, transient bipolar cells respond linearly to motion in the receptive field center. The synaptic output from their terminals is rectified and then pooled by the OMS ganglion cell. A type of polyaxonal amacrine cell is driven by motion in the surround, again via pooling of rectified inputs, but from a different set of bipolar cell terminals. By direct intracellular current injection, we found that these polyaxonal amacrine cells selectively suppress the synaptic input of OMS ganglion cells. A quantitative model of these circuit elements and their interactions explains how an important visual computation is accomplished by retinal neurons and synapses.
Due to fixational eye movements, the image on the retina is always in motion, even when one views a stationary scene. When an object moves within the scene, the corresponding patch of retina experiences a different motion trajectory than the surrounding region. Certain retinal ganglion cells respond selectively to this condition, when the motion in the cell's receptive field center is different from that in the surround. Here we show that this response is strongest at the very onset of differential motion, followed by gradual adaptation with a time course of several seconds. Different subregions of a ganglion cell's receptive field can adapt independently. The circuitry responsible for differential motion adaptation lies in the inner retina. Several candidate mechanisms were tested, and the adaptation most likely results from synaptic depression at the synapse from bipolar to ganglion cell. Similar circuit mechanisms may act more generally to emphasize novel features of a visual stimulus.
Songbirds learn their songs by trial-and-error experimentation, producing highly variable vocal output as juveniles. By comparing their own sounds to the song of a tutor, young songbirds gradually converge to a stable song that can be a remarkably good copy of the tutor song. Here we show that vocal variability in the learning songbird is induced by a basal-ganglia-related circuit, the output of which projects to the motor pathway via the lateral magnocellular nucleus of the nidopallium (LMAN). We found that pharmacological inactivation of LMAN dramatically reduced acoustic and sequence variability in the songs of juvenile zebra finches, doing so in a rapid and reversible manner. In addition, recordings from LMAN neurons projecting to the motor pathway revealed highly variable spiking activity across song renditions, showing that LMAN may act as a source of variability. Lastly, pharmacological blockade of synaptic inputs from LMAN to its target premotor area also reduced song variability. Our results establish that, in the juvenile songbird, the exploratory motor behavior required to learn a complex motor sequence is dependent on a dedicated neural circuit homologous to cortico-basal ganglia circuits in mammals.