Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and cognitive systems. However, this interaction between fields is less developed in the study of motor control. In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artificial models of embodied control. We then use this platform to study motor activity across contexts by training a model to solve four complex tasks. Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent's behavior and goals. We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and task-invariant behavioral kinematics. These representations are reflected in the sequential activity and population dynamics of neural subpopulations. Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.
How the basal ganglia contribute to the execution of learned motor skills has been thoroughly investigated. The two dominant models that have emerged posit roles for the basal ganglia in action selection and in the modulation of movement vigor. Here we test these models in rats trained to execute highly stereotyped and idiosyncratic task-specific motor sequences. Recordings and manipulations of neural activity in the striatum were not well explained by either model, and suggested that the basal ganglia, in particular its sensorimotor arm, are crucial for controlling the detailed kinematic structure of the learned behaviors. Importantly, the neural representations in the striatum, and the control functions they subserve, did not depend on the motor cortex. Taken together, these results extend our understanding of basal ganglia function, by suggesting that they can control and modulate lower-level subcortical motor circuits on a moment-by-moment basis to generate stereotyped learned motor sequences.
The acquisition and execution of learned motor sequences are mediated by a distributed motor network, spanning cortical and subcortical brain areas. The sensorimotor striatum is an important cog in this network, yet how its two main inputs, from motor cortex and thalamus respectively, contribute to its role in motor learning and execution remains largely unknown. To address this, we trained rats in a task that produces highly stereotyped and idiosyncratic motor sequences. We found that motor cortical input to the sensorimotor striatum is critical for the learning process, but after the behaviors were consolidated, this corticostriatal pathway became dispensable. Functional silencing of striatal-projecting thalamic neurons, however, disrupted the execution of the learned motor sequences, causing rats to revert to behaviors produced early in learning and preventing them from re-learning the task. These results show that the sensorimotor striatum is a conduit through which motor cortical inputs can drive experience-dependent changes in subcortical motor circuits, likely at thalamostriatal synapses.
Trial-to-trial movement variability can both drive motor learning and interfere with expert performance, suggesting benefits of regulating it in context-specific ways. Here we address whether and how the brain regulates motor variability as a function of performance by training rats to execute ballistic forelimb movements for reward. Behavioral datasets comprising millions of trials revealed that motor variability is regulated by two distinct processes. A fast process modulates variability as a function of recent trial outcomes, increasing it when performance is poor and vice versa. A slower process tunes the gain of the fast process based on the uncertainty in the task's reward landscape. Simulations demonstrated that this regulation strategy optimizes reward accumulation over a wide range of time horizons, while also promoting learning. Our results uncover a sophisticated algorithm implemented by the brain to adaptively regulate motor variability to improve task performance. VIDEO ABSTRACT.
Addressing how neural circuits underlie behavior is routinely done by measuring electrical activity from single neurons in experimental sessions. While such recordings yield snapshots of neural dynamics during specified tasks, they are ill-suited for tracking single-unit activity over longer timescales relevant for most developmental and learning processes, or for capturing neural dynamics across different behavioral states. Here we describe an automated platform for continuous long-term recordings of neural activity and behavior in freely moving rodents. An unsupervised algorithm identifies and tracks the activity of single units over weeks of recording, dramatically simplifying the analysis of large datasets. Months-long recordings from motor cortex and striatum made and analyzed with our system revealed remarkable stability in basic neuronal properties, such as firing rates and inter-spike interval distributions. Interneuronal correlations and the representation of different movements and behaviors were similarly stable. This establishes the feasibility of high-throughput long-term extracellular recordings in behaving animals.
Trial-to-trial variability in the execution of movements and motor skills is ubiquitous and widely considered to be the unwanted consequence of a noisy nervous system. However, recent studies have suggested that motor variability may also be a feature of how sensorimotor systems operate and learn. This view, rooted in reinforcement learning theory, equates motor variability with purposeful exploration of motor space that, when coupled with reinforcement, can drive motor learning. Here we review studies that explore the relationship between motor variability and motor learning in both humans and animal models. We discuss neural circuit mechanisms that underlie the generation and regulation of motor variability and consider the implications that this work has for our understanding of motor learning. Expected final online publication date for the Annual Review of Neuroscience Volume 40 is July 8, 2017. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Trial-and-error learning requires evaluating variable actions and reinforcing successful variants. In songbirds, vocal exploration is induced by LMAN, the output of a basal ganglia-related circuit that also contributes a corrective bias to the vocal output. This bias is gradually consolidated in RA, a motor cortex analogue downstream of LMAN. We develop a new model of such two-stage learning. Using stochastic gradient descent, we derive how the activity in 'tutor' circuits (e.g., LMAN) should match plasticity mechanisms in 'student' circuits (e.g., RA) to achieve efficient learning. We further describe a reinforcement learning framework through which the tutor can build its teaching signal. We show that mismatches between the tutor signal and the plasticity mechanism can impair learning. Applied to birdsong, our results predict the temporal structure of the corrective bias from LMAN given a plasticity rule in RA. Our framework can be applied predictively to other paired brain areas showing two-stage learning.
Rapid and reversible manipulations of neural activity in behaving animals are transforming our understanding of brain function. An important assumption underlying much of this work is that evoked behavioural changes reflect the function of the manipulated circuits. We show that this assumption is problematic because it disregards indirect effects on the independent functions of downstream circuits. Transient inactivations of motor cortex in rats and nucleus interface (Nif) in songbirds severely degraded task-specific movement patterns and courtship songs, respectively, which are learned skills that recover spontaneously after permanent lesions of the same areas. We resolve this discrepancy in songbirds, showing that Nif silencing acutely affects the function of HVC, a downstream song control nucleus. Paralleling song recovery, the off-target effects resolved within days of Nif lesions, a recovery consistent with homeostatic regulation of neural activity in HVC. These results have implications for interpreting transient circuit manipulations and for understanding recovery after brain lesions.
Addressing how neural circuits underlie behavior is routinely done by measuring electrical activity from single neurons during experimental sessions. While such recordings yield snapshots of neural dynamics during specified tasks, they are ill-suited for tracking single-unit activity over longer timescales relevant for most developmental and learning processes, or for capturing neural dynamics outside of task context. Here we describe an automated platform for continuous long-term recordings of neural activity and behavior in freely moving animals. An unsupervised algorithm identifies and tracks the activity of single units over weeks of recording, dramatically simplifying the analysis of large datasets. Months-long recordings from motor cortex and striatum made and analyzed with our system revealed remarkable stability in basic neuronal properties, such as firing rates and inter-spike interval distributions. Interneuronal correlations and the representation of different movements and behaviors were similarly stable. This establishes the feasibility of high-throughput long-term extracellular recordings in behaving animals.
Motor cortex is widely believed to underlie the acquisition and execution of motor skills, but its contributions to these processes are not fully understood. One reason is that studies on motor skills often conflate motor cortex's established role in dexterous control with roles in learning and producing task-specific motor sequences. To dissociate these aspects, we developed a motor task for rats that trains spatiotemporally precise movement patterns without requirements for dexterity. Remarkably, motor cortex lesions had no discernible effect on the acquired skills, which were expressed in their distinct pre-lesion forms on the very first day of post-lesion training. Motor cortex lesions prior to training, however, rendered rats unable to acquire the stereotyped motor sequences required for the task. These results suggest a remarkable capacity of subcortical motor circuits to execute learned skills and a previously unappreciated role for motor cortex in "tutoring" these circuits during learning.
Motor skill learning is characterized by improved performance and reduced motor variability. The neural mechanisms that couple skill level and variability, however, are not known. The zebra finch, a songbird, presents a unique opportunity to address this question because production of learned song and induction of vocal variability are instantiated in distinct circuits that converge on a motor cortex analogue controlling vocal output. To probe the interplay between learning and variability, we made intracellular recordings from neurons in this area, characterizing how their inputs from the functionally distinct pathways change throughout song development. We found that inputs that drive stereotyped song-patterns are strengthened and pruned, while inputs that induce variability remain unchanged. A simple network model showed that strengthening and pruning of action-specific connections reduces the sensitivity of motor control circuits to variable input and neural 'noise'. This identifies a simple and general mechanism for learning-related regulation of motor variability.
To signal the onset of salient sensory features or execute well-timed motor sequences, neuronal circuits must transform streams of incoming spike trains into precisely timed firing. To address the efficiency and fidelity with which neurons can perform such computations, we developed a theory to characterize the capacity of feedforward networks to generate desired spike sequences. We find the maximum number of desired output spikes a neuron can implement to be 0.1-0.3 per synapse. We further present a biologically plausible learning rule that allows feedforward and recurrent networks to learn multiple mappings between inputs and desired spike sequences. We apply this framework to reconstruct synaptic weights from spiking activity and study the precision with which the temporal structure of ongoing behavior can be inferred from the spiking of premotor neurons. This work provides a powerful approach for characterizing the computational and learning capacities of single neurons and neuronal circuits.
Individual differences in motor learning ability are widely acknowledged, yet little is known about the factors that underlie them. Here we explore whether movement-to-movement variability in motor output, a ubiquitous if often unwanted characteristic of motor performance, predicts motor learning ability. Surprisingly, we found that higher levels of task-relevant motor variability predicted faster learning both across individuals and across tasks in two different paradigms, one relying on reward-based learning to shape specific arm movement trajectories and the other relying on error-based learning to adapt movements in novel physical environments. We proceeded to show that training can reshape the temporal structure of motor variability, aligning it with the trained task to improve learning. These results provide experimental support for the importance of action exploration, a key idea from reinforcement learning theory, showing that motor variability facilitates motor learning in humans and that our nervous systems actively regulate it to improve learning.
Nature Neuroscience. January 2014. (pdf). Write up in the Harvard Gazette here. News and Views from Nature Neuroscience (pdf).*Co-senior authors.
Executing a motor skill requires the brain to control which muscles to activate at what times. How these aspects of control-motor implementation and timing-are acquired, and whether the learning processes underlying them differ, is not well understood. To address this, we used a reinforcement learning paradigm to independently manipulate both spectral and temporal features of birdsong, a complex learned motor sequence, while recording and perturbing activity in underlying circuits. Our results uncovered a striking dissociation in how neural circuits underlie learning in the two domains. The basal ganglia was required for modifying spectral, but not temporal, structure. This functional dissociation extended to the descending motor pathway, where recordings from a premotor cortex analog nucleus reflected changes to temporal, but not spectral, structure. Our results reveal a strategy in which the nervous system employs different and largely independent circuits to learn distinct aspects of a motor skill.
Neuron. 80(2):494-506. September 2013 (pdf). Write up in the Harvard Gazette here. Software package and users manual for implementing the CAF experiments described in the paper can be downloaded here.
Addressing the neural mechanisms underlying complex learned behaviors requires training animals in well-controlled tasks, an often time-consuming and labor-intensive process that can severely limit the feasibility of such studies. To overcome this constraint, we developed a fully computer-controlled general purpose system for high-throughput training of rodents. By standardizing and automating the implementation of predefined training protocols within the animal's home-cage our system dramatically reduces the efforts involved in animal training while also removing human errors and biases from the process. We deployed this system to train rats in a variety of sensorimotor tasks, achieving learning rates comparable to existing, but more laborious, methods. By incrementally and systematically increasing the difficulty of the task over weeks of training, rats were able to master motor tasks that, in complexity and structure, resemble ones used in primate studies of motor sequence learning. By enabling fully automated training of rodents in a home-cage setting this low-cost and modular system increases the utility of rodents for studying the neural underpinnings of a variety of complex behaviors.
The ability to chronically record from populations of neurons in freely behaving animals has proven an invaluable tool for dissecting the function of neural circuits underlying a variety of natural behaviors, including navigation(1) , decision making (2,3), and the generation of complex motor sequences(4,5,6). Advances in precision machining has allowed for the fabrication of light-weight devices suitable for chronic recordings in small animals, such as mice and songbirds. The ability to adjust the electrode position with small remotely controlled motors has further increased the recording yield in various behavioral contexts by reducing animal handling.(6,7) Here we describe a protocol to build an ultra-light motorized microdrive for long-term chronic recordings in small animals. Our design evolved from an earlier published version(7), and has been adapted for ease-of use and cost-effectiveness to be more practical and accessible to a wide array of researchers. This proven design (8,9,10,11) allows for fine, remote positioning of electrodes over a range of ~ 5 mm and weighs less than 750 mg when fully assembled. We present the complete protocol for how to build and assemble these drives, including 3D CAD drawings for all custom microdrive components.
Premotor circuits help generate imitative behaviors and can be activated during observation of another animal's behavior, leading to speculation that these circuits participate in sensory learning that is important to imitation. Here we tested this idea by focally manipulating the brain activity of juvenile zebra finches, which learn to sing by memorizing and vocally copying the song of an adult tutor. Tutor song-contingent optogenetic or electrical disruption of neural activity in the pupil's song premotor nucleus HVC prevented song copying, indicating that a premotor structure important to the temporal control of birdsong also helps encode the tutor song. In vivo multiphoton imaging and neural manipulations delineated a pathway and a candidate synaptic mechanism through which tutor song information is encoded by premotor circuits. These findings provide evidence that premotor circuits help encode sensory information about the behavioral model before shaping and executing imitative behaviors.
Nature Neuroscience. 15(10):1454-9. October 2012 (PDF). *Co-senior authors.
Neural circuits underlying complex learned behaviors, such as speech in humans, develop under genetic constraints and in response to environmental influences. Little is known about the rules and mechanisms through which such circuits form. We argue that songbirds, with their discrete and well studied neural pathways underlying a complex and naturally learned behavior, provide a powerful model for addressing these questions. We briefly review current knowledge of how the song circuit develops during learning and discuss new possibilities for advancing the field given recent technological advances.