Modeling the Impact of Internal State on Sensory Processing: An Introduction

Current wisdom says that–with the exception of those that go on to great scientific fame–a PhD student’s thesis will be read by at most the 5 or so professors on their dissertation committee. Because most of the content in a thesis is already or soon-to-be published as separate papers this is not much of a loss. However, the introduction to a thesis is something that is usually written special-purpose for that document and rarely has another outlet for publication. These introductions, however, offer a space for young researchers to catalogue their journey through the scientific literature and share some accumulated wisdom from years of learning to do science. So to get a little more mileage out of my own thesis introduction and encourage others to do the same, I’ve published it here, along with links to where elements of each chapter of the thesis work are available online.

Thesis Abstract: Perception is the result of more than just the unbiased processing of sensory stimuli. At each moment in time, sensory inputs enter a circuit already impacted by signals of arousal, attention, and memory. This thesis aims to understand the impact of such internal states on the processing of sensory stimuli. To do so, computational models meant to replicate known biological circuitry and activity were built and analyzed. Part one aims to replicate the neural activity changes observed in auditory cortex when an animal is passively versus actively listening. In part two, the impact of selective visual attention on performance is probed in two models: a large-scale abstract model of the visual system and a smaller, more biologically-realistic one. Finally in part three, a simplified model of Hebbian learning is used to explore how task context comes to impact prefrontal cortical activity. While the models used in this thesis range in scale and represent diverse brain areas, they are all designed to capture the physical processes by which internal brain states come to impact sensory processing.

Thesis Introduction

This thesis takes a computational approach to the task of understanding how internal brain states and representations of sensory inputs combine. In particular, mathematical models of neural circuits are built to replicate and elucidate the physical processes by which internal state modulates sensory processing. Ultimately, this work should contribute to an understanding of how complex behavior arises from neural circuits.

Cognition is the result of internal state and external influences

While the neural mechanisms that underly it remain an open question to this day, the observation that internal mental state causes changes in sensory perception dates back millennia [Hatfield, 1998]. One of the earliest such observations comes from Aristotle in  350 B.C.E. in his treatise On Sense and the Sensible, wherein he remarks that “…persons do not perceive what is brought before their eyes, if they are at the time deep in thought, or in a fright, or listening to some loud noise.” The notion that internal state can be purposefully controlled in order to enhance processing was also noted by Lucretius in the first century B.C.E: “Even in things that are plainly visible, you can note that if you do not direct the mind, the things are, so to speak, far removed and remote for the whole time.” Philosophers continued to make these observations for centuries, with Descartes, for example, writing in 1649 that, “The soul can prevent itself from hearing a slight noise or feeling a slight pain by attending very closely to some other thing…” While these early documentations provide evidence that this phenomenon is universal and perceptually relevant, a more direct link to the field of experimental psychology comes in the early 18th century through the work of Gottfried Leibniz. While Leibniz’s work posits many elements considered outside the realm of today’s science, he does provide insights on the role of memory (plausibly a form of internal state) on sensory processing: “It is what we see in an animal that has a perception of something striking of which it has previously had a similar perception; the representations in its memory lead it to expect this time the same thing that happened on the previous occasion, and to have the same feelings now as it had then” [Leibniz, 2004]. He also, through the notion of “apperception,” expounded on the ways in which motivation and will influence perception. But the particular significance of Leibniz’s work for modern psychology comes through his influence on Wilhelm Wundt. Wundt, who founded what is considered the first experimental psychology lab in 1879, is explicit about the role of Leibniz’s work in his own thought. With a particular focus on the notion of “apperception,” Wundt took up the task of scientifically measuring and studying central mental control processes [Rieber and Salzinger, 2013]. Modern studies of internal state and sensory processing are direct descendants of his initial work on developing the field of experimental psychology.

Through centuries of experimental research, a myriad of ways in which internal state can impact processing have been documented. Arousal levels, for example, have been shown to impact perceptual thresholds and reaction times in an inverted-U manner [Tomporowski and Ellis, 1986]; that is, beneficial effects on perception come from moderate levels of arousal, while too low or too high arousal can impair performance. When awakened from sleep (and presumably in a state of low arousal), people are slower to respond to auditory stimuli [Wilkinson and Stretton, 1971]. Under conditions of sleep deprivation, responses to visual stimuli are slower and misses are more common [Belenky et al., 2003]. In the study of human psychology, mood and emotional state have also been related to changes in sensory processing. For example, patients with major depressive disorder showed higher thresholds for odor detection than healthy controls, but this difference went away after successful treatment [Pause et al., 2001].

Selective attention differs from arousal and mood in that it is controllable and directed to a subset of the perceptual experience. When participants expect a stimulus in a given sensory modality (e.g. a visual input), they are slower to respond to a relevant stimulus in a different modality (e.g. a tactile one) [Spence et al., 2001]. When cued to attend to a subset of the input within a sensory modality, similar benefits and costs are found for the attended and unattended stimuli, respectively [Carrasco, 2011]. In a particularly well-known example of “inattentional blindness,” subjects asked to count the number of basketball passes in a video did not report awareness of a person in a gorilla suit walking across the frame [Simons and Chabris, 1999].

Interestingly, the internal state generated by a stimulus in one sensory modality may also alter the perception from another. For example, hearing animal noises prior to image presentation increases detection of animal images and lowers reaction time [Schneider et al., 2008]. In addition, certain forms of memory and stimulus history within a modality can impact sensory processing. For example, trial history has complex effects on future behavior that are at least in part due to changes in stimulus expectation as well as low-level sensory facilitation [Cho et al., 2002].

While this ability of internal state to alter perception and decision-making seems perhaps a hallmark of mammalian, or even primate, neurophysiology, it has been observed across the evolutionary tree [Lovett-Barron et al., 2017]. For example, being in a food-deprived state alters the response of C. elegans to chemical gradients [Ghosh et al., 2016].

Different types of internal state modulation are believed to have different neural underpinnings. Overall arousal levels, for example, have broad impacts on various sensory and cognitive functions. A likely candidate for such modulation is thus the brainstem, as it contains nuclei that send diffuse connections across the brain [Sara and Bouret, 2012]. The axons from these areas release a cocktail of neuromodulators that can have diverse impacts. For example, noradrenaline released from the locus coeruleus is believed to play a role in neural synchronization [Sara and Bouret, 2012]. Switching between tasks that have different goals or require information from different sensory modalities, however, requires more targeted manipulations that can impact different brains areas separably. In a study that monitored fMRI activity during switches between auditory and visual attention, activity in frontal and parietal cortices were correlated with the switch [Shomstein and Yantis, 2004]. Further along this spectrum, selective attention within a sensory modality implies a targeting of individual cell populations that represent the attended stimulus. Such fine-grained modulation by attention has been observed [Martinez-Trujillo and Treue, 2004], and is assumed to be controlled by top-down connections originating in the frontal cortex [Bichot et al., 2015].

The alteration of sensory processing by internal state is an important component of the cognitive processes that lead to adaptive behavior. Allowing perception to be influenced by context, goals, and history creates a more flexible mapping between sensory input and behavioral output. This can be viewed as a useful integration of many different information sources for the purposes of decision-making. The importance of this is made clear by the cases in which it goes wrong. An underlying cognitive deficit in schizophrenia, for example, is the inability to incorporate context into perceptual processing [Bazin et al., 2000].

Circuit modeling as an approach for connecting structure and function

The notion that structure begets function in the brain has appeared throughout the history of neuroscience. Even without any significant evidence, phrenologists proposed different anatomical foci for different cognitive functions [Parssinen, 1974], so natural is the structure-function relationship. Some of the earliest examples of observed structure being related to function came in the late 19th century from Santiago Ramón y Cajal. Through careful anatomical investigation of a variety of neural circuits, Cajal came to hypothesize—correctly—that a “nervous current” travels from the dendritic tree through the soma and out through the axon [Llinás, 2003]. This relationship between structure and function extends beyond individual neuron morphology to the structure of entire circuits. The presence of a repeating laminar circuit motif—with, for example, inputs from lower areas targeting cells in layer 4, which project to cells in layers 2/3, which send outputs to layer 5—is frequently cited as evidence that such structure is a functional unit of the brain [Douglas and Martin, 2004]. A more direct investigation of how the structure of neural connections leads to functionally-interpretable activity came from Hubel and Weisel. Particularly, they documented two different types of neurons in primary visual cortex—simple cells (which respond to on- and off-patterns of light with spatial specificity) and complex cells (which have more spatial invariance in their responses to light patterns)—and came to the conclusion that the responses of the complex cells could be understood if it is assumed that they receive input from multiple simple cells, each representing a slightly different spatial location [Hubel and Wiesel, 1962]. Thus, the connections of the neural circuit were mapped to functional properties of the neural responses. This level of understanding should ultimately be possible for all neural responses, insofar as all are the result of the neuron’s place in a circuit.

While experimental results have facilitated an understanding of the importance of structure in neural circuits, such descriptive approaches have limitations. To truly understand a neural circuit, as Richard Feynman would say, we must be able to build it. Mathematical models allow for the precise formulation and testing of a hypothesis. In neural circuit modeling, neurons are represented by an equation or set of equations that represent how the neuron’s inputs are combined and transformed into an output measure, such as firing rate. A weight matrix dictates the impact any given neuron’s activity has on other neurons in the network. When designed to incorporate facts about the connectivity and neural response properties of a particular brain area, circuit models can serve as powerful mechanistic explanation of neural activity. As such, they can be used to test and generate hypotheses about the relationship between structure and activity. In neuroscience, where tools for observation and manipulation are limited and/or expensive, being apply to perform experiments in silico can be of immense value. Furthermore, mathematical analysis and simulation is of particular use when working with large and complex systems, which can display counterintuitive and difficult-to-predict behavior.

Circuit modeling exists within a larger set of quantitative approaches that comprise computational/theoretical neuroscience. Other approaches in this category focus on devising advanced tools for data analysis tailored to the problems of neuroscience. Another subset of methods involves more abstract mathematical analysis for the purpose of deriving statements about qualities such as optimality, stability, or memory capacity. While these other quantitative approaches have much to offer the field, circuit modeling is particularly well-suited for incorporating and explaining data. Theoretical constructs are only useful insofar as they can be related to biologically observable values, and circuit models are built to be directly comparable to existing biological structures. Therefore, predications from a circuit model are straightforward to interpret, and lead to predictions for the data. Practically, certain predictions from circuit models may be difficult to explore experimentally due to technical limitations. However, this creates a role for theory in driving the development of tools, as circuit models make clear which components of the biology are most worth measuring.

To encourage an integration of experimental and computational work, it is important for there to be a common language and set of ideas. Practically speaking, this can be achieved by building models that have explicit one-to-one correspondence with biological entities. It is also helpful to design models in a way that allows for the same set of analyses to be performed on the data as well as the model. In this case then, even if a one-to-one correspondence isn’t possible, derivative measurements can still be compared directly. This thesis contains examples from along this spectrum. A tension that always comes with model building, however, is the desire to make a model that is both detailed and accurate while also conceptually useful. Highly detailed, complex models may be good at capturing the data but can be unwieldy and do not open themselves up for easy mathematical, or even informal conceptual, inspection. While simpler models can be worked with and interrogated more easily, the rich dynamics of the brain is unlikely to be captured by a simple model. Again, this thesis includes models from across this spectrum.

Thesis overview

The parts of this thesis are arranged according to the brain area studied as well as the type of internal state being explored.

In the first, the impact of task engagement on responses in mouse auditory cortex is explored. The modeling approach used in this chapter allows for a direct comparison of the firing rates of different neuronal subtypes in the model with those found experimentally under two circumstances: during an active tone discrimination task and during passive exposure to tones. The aim of this model is to understand the physical structure of the circuit and the input signals that allow for different neural responses to the same tones under different conditions. To read more about this work, see the following paper: Parallel processing by cortical inhibition enables context-dependent behavior

In the second part, selective visual attention is the focus. In particular, the mechanisms that allow for certain visual features to be enhanced across the visual field are recapitulated in a large scale model of the visual system. While modeling of this type doesn’t allow for a direct comparison to data on the neural level, it has the benefit of providing a behavioral output. Thus, this model is used to understand how voluntary shifts in selective visual attention lead to changes in performance on complex visual tasks. To learn more about this work, see this video: Understanding Biological Visual Attention Using Convolutional Neural Networks – CCN 2017 or the associated manuscript on bioRxiv (now published in eLife). The second chapter in this part includes an extension to these models that is meant to make them more biologically-realistic and thus more comparable to data.

Finally, the third part more directly addresses the ability of context to alter the mapping from sensory inputs to behavior. Here, again, task types are changed in blocks. These different task conditions alter the way in which visual stimuli are encoded in prefrontal cortical neurons, which then allows for a more flexible mapping to behavioral outputs. To understand how these encoding changes come to be, a simplified model that includes Hebbian learning is introduced and analyzed in comparison to analysis of the data. To learn more about this work, see the following paper: Hebbian Learning in a Random Network Captures Selectivity Properties of the Prefrontal Cortex

References

Gary Hatfield. Attention in early scientific psychology. Visual attention, 1:3–25, 1998.
Gottfried Wilhelm Leibniz. The principles of philosophy known as monadology. Jonathan
Bennett Edition, 2004.
Robert W Rieber and Kurt Salzinger. Psychology: Theoretical–historical perspectives.
Academic Press, 2013.
Phillip D Tomporowski and Norman R Ellis. Effects of exercise on cognitive processes: A
review. Psychological bulletin, 99(3):338, 1986.
Robert T Wilkinson and M Stretton. Performance after awakening at different times of
night. Psychonomic Science, 23(4):283–285, 1971.
Gregory Belenky, Nancy J Wesensten, David R Thorne, Maria L Thomas, Helen C Sing,
Daniel P Redmond, Michael B Russo, and Thomas J Balkin. Patterns of performance
degradation and restoration during sleep restriction and subsequent recovery: A sleep
dose-response study. Journal of sleep research, 12(1):1–12, 2003.
Bettina M Pause, Alejandra Miranda, Robert Göder, Josef B Aldenhoff, and Roman Ferstl.
Reduced olfactory performance in patients with major depression. Journal of psychiatric
research, 35(5):271–277, 2001.
Charles Spence, Michael ER Nicholls, and Jon Driver. The cost of expecting events in the
wrong sensory modality. Perception & Psychophysics, 63(2):330–336, 2001.
Marisa Carrasco. Visual attention: The past 25 years. Vision research, 51(13):1484–1525,
2011.                                                                                                                                                    Daniel J Simons and Christopher F Chabris. Gorillas in our midst: Sustained inattentional
blindness for dynamic events. Perception, 28(9):1059–1074, 1999.
Till R Schneider, Andreas K Engel, and Stefan Debener. Multisensory identification of
natural objects in a two-way crossmodal priming paradigm. Experimental psychology,
55(2):121–132, 2008.
Raymond Y Cho, Leigh E Nystrom, Eric T Brown, Andrew D Jones, Todd S Braver, Philip J
Holmes, and Jonathan D Cohen. Mechanisms underlying dependencies of performance on stimulus history in a two-alternative forced-choice task. Cognitive, Affective, & Behavioral Neuroscience, 2(4):283–299, 2002.
Matthew Lovett-Barron, Aaron S Andalman, William E Allen, Sam Vesuna, Isaac Kauvar,
Vanessa M Burns, and Karl Deisseroth. Ancestral circuits for the coordinated modulation
of brain state. Cell, 2017.
D Dipon Ghosh, Tom Sanders, Soonwook Hong, Li Yan McCurdy, Daniel L Chase, Netta
Cohen, Michael R Koelle, and Michael N Nitabach. Neural architecture of hunger-
dependent multisensory decision making in c. elegans. Neuron, 92(5):1049–1062, 2016.
Susan J Sara and Sebastien Bouret. Orienting and reorienting: the locus coeruleus mediates cognition through arousal. Neuron, 76(1):130–141, 2012.
Sarah Shomstein and Steven Yantis. Control of attention shifts between vision and audition in human cortex. Journal of neuroscience, 24(47):10702–10706, 2004.
Julio C Martinez-Trujillo and Stefan Treue. Feature-based attention increases the selectivity of population responses in primate visual cortex. Current Biology, 14(9):744–751, 2004.
Ying Zhang, Ethan M Meyers, Narcisse P Bichot, Thomas Serre, Tomaso A Poggio, and
Robert Desimone. Object decoding with attention in inferior temporal cortex. Proceedings of the National Academy of Sciences, 108(21):8850–8855, 2011.
Nadine Bazin, Pierre Perruchet, Marie Christine Hardy-Bayle, and André Feline. Context-
dependent information processing in patients with schizophrenia. Schizophrenia research, 45(1):93–101, 2000.
Terry M Parssinen. Popular science and society: the phrenology movement in early victorian britain. Journal of Social History, 8(1):1–20, 1974.
Rodolfo R Llinás. The contribution of santiago ramon y cajal to functional neuroscience.
Nature Reviews Neuroscience, 4(1):77–80, 2003.
Rodney J Douglas and Kevan AC Martin. Neuronal circuits of the neocortex. Annu. Rev.
Neurosci., 27:419–451, 2004.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of physiology, 160(1):106–154, 1962.