In mammalian animal models, high-resolution kinematic tracking is restricted to brief sessions in constrained environments, limiting our ability to probe naturalistic behaviors and their neural underpinnings. To address this, we developed CAPTURE (Continuous Appendicular and Postural Tracking Using Retroreflector Embedding), a behavioral monitoring system that combines motion capture and deep learning to continuously track the 3D kinematics of a rat’s head, trunk, and limbs for week-long timescales in freely behaving animals. CAPTURE realizes 10- to 100-fold gains in precision and robustness compared with existing convolutional network approaches to behavioral tracking. We demonstrate CAPTURE’s ability to comprehensively profile the kinematics and sequential organization of natural rodent behavior, its variation across individuals, and its perturbation by drugs and disease, including identifying perseverative grooming states in a rat model of fragile X syndrome. CAPTURE significantly expands the range of behaviors and contexts that can be quantitatively investigated, opening the door to a new understanding of natural behavior and its neural basis.
To generate adaptive behaviors, animals must learn from their interactions with the environment. Describing the algorithms that govern this learning process and how they are implemented in the brain is a major goal of neuroscience. Careful and controlled observations of animal learning by Thorndike, Pavlov and others, now more than a century ago, identified intuitive rules by which animals (including humans) can learn from their experiences by associating sensory stimuli and motor actions with rewards. But going from explaining learning in simple paradigms to deciphering how complex problems are solved in rich and dynamic environments has proven difficult (Figure 1). Recently, this effort has received help from computer scientists and engineers hoping to emulate intelligent adaptive behaviors in machines. Inspired by the animal behavior literature, pioneers in artificial intelligence developed a rigorous and mathematically principled framework within which reward-based learning can be formalized and studied. Not only has the field of reinforcement learning become a boon to machine learning and artificial intelligence, it has also provided a theoretical foundation for biologists interested in deciphering how the brain implements reinforcement learning algorithms. The ability of reinforcement learning agents to solve complex, high-dimensional learning problems has been dramatically enhanced by using deep neural networks (deep reinforcement learning, Figure 1). Indeed, aided by ever-increasing computational resources, deep reinforcement learning algorithms can now outperform human experts on a host of well-defined complex tasks …
Though the temporal precision of neural computation has been studied intensively, a data-driven determination of this precision remains a fundamental challenge. Reproducible spike patterns may be obscured on single trials by uncontrolled temporal variability in behavior and cognition and may not be time locked to measurable signatures in behavior or local field potentials (LFP). To overcome these challenges, we describe a general-purpose time warping framework that reveals precise spike-time patterns in an unsupervised manner, even when these patterns are decoupled from behavior or are temporally stretched across single trials. We demonstrate this method across diverse systems: cued reaching in nonhuman primates, motor sequence production in rats, and olfaction in mice. This approach flexibly uncovers diverse dynamical firing patterns, including pulsatile responses to behavioral events, LFP-aligned oscillatory spiking, and even unanticipated patterns, such as 7 Hz oscillations in rat motor cortex that are not time locked to measured behaviors or LFP.
Parallel developments in neuroscience and deep learning have led to mutually productive exchanges, pushing our understanding of real and artificial neural networks in sensory and cognitive systems. However, this interaction between fields is less developed in the study of motor control. In this work, we develop a virtual rodent as a platform for the grounded study of motor activity in artificial models of embodied control. We then use this platform to study motor activity across contexts by training a model to solve four complex tasks. Using methods familiar to neuroscientists, we describe the behavioral representations and algorithms employed by different layers of the network using a neuroethological approach to characterize motor activity relative to the rodent's behavior and goals. We find that the model uses two classes of representations which respectively encode the task-specific behavioral strategies and task-invariant behavioral kinematics. These representations are reflected in the sequential activity and population dynamics of neural subpopulations. Overall, the virtual rodent facilitates grounded collaborations between deep reinforcement learning and motor neuroscience.
How the basal ganglia contribute to the execution of learned motor skills has been thoroughly investigated. The two dominant models that have emerged posit roles for the basal ganglia in action selection and in the modulation of movement vigor. Here we test these models in rats trained to execute highly stereotyped and idiosyncratic task-specific motor sequences. Recordings and manipulations of neural activity in the striatum were not well explained by either model, and suggested that the basal ganglia, in particular its sensorimotor arm, are crucial for controlling the detailed kinematic structure of the learned behaviors. Importantly, the neural representations in the striatum, and the control functions they subserve, did not depend on the motor cortex. Taken together, these results extend our understanding of basal ganglia function, by suggesting that they can control and modulate lower-level subcortical motor circuits on a moment-by-moment basis to generate stereotyped learned motor sequences.
The acquisition and execution of learned motor sequences are mediated by a distributed motor network, spanning cortical and subcortical brain areas. The sensorimotor striatum is an important cog in this network, yet how its two main inputs, from motor cortex and thalamus respectively, contribute to its role in motor learning and execution remains largely unknown. To address this, we trained rats in a task that produces highly stereotyped and idiosyncratic motor sequences. We found that motor cortical input to the sensorimotor striatum is critical for the learning process, but after the behaviors were consolidated, this corticostriatal pathway became dispensable. Functional silencing of striatal-projecting thalamic neurons, however, disrupted the execution of the learned motor sequences, causing rats to revert to behaviors produced early in learning and preventing them from re-learning the task. These results show that the sensorimotor striatum is a conduit through which motor cortical inputs can drive experience-dependent changes in subcortical motor circuits, likely at thalamostriatal synapses.
Trial-to-trial movement variability can both drive motor learning and interfere with expert performance, suggesting benefits of regulating it in context-specific ways. Here we address whether and how the brain regulates motor variability as a function of performance by training rats to execute ballistic forelimb movements for reward. Behavioral datasets comprising millions of trials revealed that motor variability is regulated by two distinct processes. A fast process modulates variability as a function of recent trial outcomes, increasing it when performance is poor and vice versa. A slower process tunes the gain of the fast process based on the uncertainty in the task's reward landscape. Simulations demonstrated that this regulation strategy optimizes reward accumulation over a wide range of time horizons, while also promoting learning. Our results uncover a sophisticated algorithm implemented by the brain to adaptively regulate motor variability to improve task performance. VIDEO ABSTRACT.
The development of increasingly sophisticated methods for recording and manipulating neural activity is revolutionizing neuroscience. By probing how activity patterns in different types of neurons and circuits contribute to behavior, these tools can help inform mechanistic models of brain function and explain the roles of distinct circuit elements. However, in systems where functions are distributed over large networks, interpreting causality experiments can be challenging. Here we review common assumptions underlying circuit manipulations in behaving animals and discuss the strengths and limitations of different approaches.
Addressing how neural circuits underlie behavior is routinely done by measuring electrical activity from single neurons in experimental sessions. While such recordings yield snapshots of neural dynamics during specified tasks, they are ill-suited for tracking single-unit activity over longer timescales relevant for most developmental and learning processes, or for capturing neural dynamics across different behavioral states. Here we describe an automated platform for continuous long-term recordings of neural activity and behavior in freely moving rodents. An unsupervised algorithm identifies and tracks the activity of single units over weeks of recording, dramatically simplifying the analysis of large datasets. Months-long recordings from motor cortex and striatum made and analyzed with our system revealed remarkable stability in basic neuronal properties, such as firing rates and inter-spike interval distributions. Interneuronal correlations and the representation of different movements and behaviors were similarly stable. This establishes the feasibility of high-throughput long-term extracellular recordings in behaving animals.
Trial-to-trial variability in the execution of movements and motor skills is ubiquitous and widely considered to be the unwanted consequence of a noisy nervous system. However, recent studies have suggested that motor variability may also be a feature of how sensorimotor systems operate and learn. This view, rooted in reinforcement learning theory, equates motor variability with purposeful exploration of motor space that, when coupled with reinforcement, can drive motor learning. Here we review studies that explore the relationship between motor variability and motor learning in both humans and animal models. We discuss neural circuit mechanisms that underlie the generation and regulation of motor variability and consider the implications that this work has for our understanding of motor learning. Expected final online publication date for the Annual Review of Neuroscience Volume 40 is July 8, 2017. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Trial-and-error learning requires evaluating variable actions and reinforcing successful variants. In songbirds, vocal exploration is induced by LMAN, the output of a basal ganglia-related circuit that also contributes a corrective bias to the vocal output. This bias is gradually consolidated in RA, a motor cortex analogue downstream of LMAN. We develop a new model of such two-stage learning. Using stochastic gradient descent, we derive how the activity in 'tutor' circuits (e.g., LMAN) should match plasticity mechanisms in 'student' circuits (e.g., RA) to achieve efficient learning. We further describe a reinforcement learning framework through which the tutor can build its teaching signal. We show that mismatches between the tutor signal and the plasticity mechanism can impair learning. Applied to birdsong, our results predict the temporal structure of the corrective bias from LMAN given a plasticity rule in RA. Our framework can be applied predictively to other paired brain areas showing two-stage learning.
Rapid and reversible manipulations of neural activity in behaving animals are transforming our understanding of brain function. An important assumption underlying much of this work is that evoked behavioural changes reflect the function of the manipulated circuits. We show that this assumption is problematic because it disregards indirect effects on the independent functions of downstream circuits. Transient inactivations of motor cortex in rats and nucleus interface (Nif) in songbirds severely degraded task-specific movement patterns and courtship songs, respectively, which are learned skills that recover spontaneously after permanent lesions of the same areas. We resolve this discrepancy in songbirds, showing that Nif silencing acutely affects the function of HVC, a downstream song control nucleus. Paralleling song recovery, the off-target effects resolved within days of Nif lesions, a recovery consistent with homeostatic regulation of neural activity in HVC. These results have implications for interpreting transient circuit manipulations and for understanding recovery after brain lesions.
Addressing how neural circuits underlie behavior is routinely done by measuring electrical activity from single neurons during experimental sessions. While such recordings yield snapshots of neural dynamics during specified tasks, they are ill-suited for tracking single-unit activity over longer timescales relevant for most developmental and learning processes, or for capturing neural dynamics outside of task context. Here we describe an automated platform for continuous long-term recordings of neural activity and behavior in freely moving animals. An unsupervised algorithm identifies and tracks the activity of single units over weeks of recording, dramatically simplifying the analysis of large datasets. Months-long recordings from motor cortex and striatum made and analyzed with our system revealed remarkable stability in basic neuronal properties, such as firing rates and inter-spike interval distributions. Interneuronal correlations and the representation of different movements and behaviors were similarly stable. This establishes the feasibility of high-throughput long-term extracellular recordings in behaving animals.
Motor cortex is widely believed to underlie the acquisition and execution of motor skills, but its contributions to these processes are not fully understood. One reason is that studies on motor skills often conflate motor cortex's established role in dexterous control with roles in learning and producing task-specific motor sequences. To dissociate these aspects, we developed a motor task for rats that trains spatiotemporally precise movement patterns without requirements for dexterity. Remarkably, motor cortex lesions had no discernible effect on the acquired skills, which were expressed in their distinct pre-lesion forms on the very first day of post-lesion training. Motor cortex lesions prior to training, however, rendered rats unable to acquire the stereotyped motor sequences required for the task. These results suggest a remarkable capacity of subcortical motor circuits to execute learned skills and a previously unappreciated role for motor cortex in "tutoring" these circuits during learning.
Motor skill learning is characterized by improved performance and reduced motor variability. The neural mechanisms that couple skill level and variability, however, are not known. The zebra finch, a songbird, presents a unique opportunity to address this question because production of learned song and induction of vocal variability are instantiated in distinct circuits that converge on a motor cortex analogue controlling vocal output. To probe the interplay between learning and variability, we made intracellular recordings from neurons in this area, characterizing how their inputs from the functionally distinct pathways change throughout song development. We found that inputs that drive stereotyped song-patterns are strengthened and pruned, while inputs that induce variability remain unchanged. A simple network model showed that strengthening and pruning of action-specific connections reduces the sensitivity of motor control circuits to variable input and neural 'noise'. This identifies a simple and general mechanism for learning-related regulation of motor variability.
To signal the onset of salient sensory features or execute well-timed motor sequences, neuronal circuits must transform streams of incoming spike trains into precisely timed firing. To address the efficiency and fidelity with which neurons can perform such computations, we developed a theory to characterize the capacity of feedforward networks to generate desired spike sequences. We find the maximum number of desired output spikes a neuron can implement to be 0.1-0.3 per synapse. We further present a biologically plausible learning rule that allows feedforward and recurrent networks to learn multiple mappings between inputs and desired spike sequences. We apply this framework to reconstruct synaptic weights from spiking activity and study the precision with which the temporal structure of ongoing behavior can be inferred from the spiking of premotor neurons. This work provides a powerful approach for characterizing the computational and learning capacities of single neurons and neuronal circuits.
Individual differences in motor learning ability are widely acknowledged, yet little is known about the factors that underlie them. Here we explore whether movement-to-movement variability in motor output, a ubiquitous if often unwanted characteristic of motor performance, predicts motor learning ability. Surprisingly, we found that higher levels of task-relevant motor variability predicted faster learning both across individuals and across tasks in two different paradigms, one relying on reward-based learning to shape specific arm movement trajectories and the other relying on error-based learning to adapt movements in novel physical environments. We proceeded to show that training can reshape the temporal structure of motor variability, aligning it with the trained task to improve learning. These results provide experimental support for the importance of action exploration, a key idea from reinforcement learning theory, showing that motor variability facilitates motor learning in humans and that our nervous systems actively regulate it to improve learning.
Nature Neuroscience. January 2014. (pdf). Write up in the Harvard Gazette here. News and Views from Nature Neuroscience (pdf).*Co-senior authors.