Intracranial Pressure Monitor

Dartmouth College, Hanover, Nh, Usa

Note: This chapter is forthcoming in Intracranial EEG for Cognitive Neuroscience

Draft Current As Of February 7, 2022

Abstract. Identifying stimulus-driven neural activity patterns is critical for studying the neural basis of cognition. This can be particularly chal- lenging in intracranial datasets, where electrode locations typically vary across patients. This chapter first presents an overview of the major chal- lenges to identifying stimulus-driven neural activity patterns in the gen- eral case. Next, we will review several modality-specific considerations and approaches, along with a discussion of several issues that are partic- ular to intracranial recordings. Against this backdrop, we will consider a variety of within-subject and across-subject approaches to identifying and modeling stimulus-driven neural activity patterns in multi-patient intracranial recordings. These approaches include generalized linear mod- els, multivariate pattern analysis, representational similarity analysis, joint stimulus-activity models, hierarchical matrix factorization models, Gaus- sian process models, geometric alignment models, inter-subject correla- tions, and inter-subject functional correlations. Examples from the recent literature serve to illustrate the major concepts and provide the conceptual intuitions for each approach.

Keywords: stimulus-driven · multi-subject · signal processing · computa-

Overview

Studying brain function often requires identifying brain responses to a given stimulus or set of stimuli. For some stimuli, and for some systems, this identifica- tion problem is relatively straightforward. For example, when a photopigment in a retinal photoreceptor absorbs light, this triggers a cascade of responses that is ultimately sent from the retina to other brain areas via the optic nerve .

In the general sense, however (i.e., for arbitrarily complex stimuli and arbi- trary brain areas), the problem of identifying neural responses to known (or unknown) stimuli can be incredibly challenging .

41.1.1

Why is it challenging to identify stimulus-driven brain activity? To illustrate the enormity of the challenge of identifying stimulus-driven brain responses in the general sense, it can be useful to start by considering what form a complete solution might take. First, we need some means of defining (and measuring) what brain activity is. For example, should we concern ourselves with measuring membrane potentials or firing rates of individual neurons? Or population activity in a given brain structure or network? And is it more appropriate to analyze or interpret activity patterns in the time domain (e.g., firing rate or voltage as a function of time), or in the frequency domain (e.g., characterizing the signal through the relative contributions of its constituent sinusoidal components at different frequencies)? Should we consider neurons and/or brain structures in isolation, or should we instead interpret each “unit” of activity within the context of the network(s) it participates in or contributes to? We discuss several different approaches to these questions (and their relative trade-offs) in Section 41.1.2.

Second, we need some means of characterizing (and ideally, quantifying) the stimulus itself. For a simple stimulus, such as a single photon of light, emitted from a known location in an otherwise completely dark room, constructing a sufficiently comprehensive model of the stimulus might be straightforward– and perhaps even trivial. For other stimuli, such as real-world experiences, constructing a comprehensive model of “what is happening” can be highly complex (at best). Essentially, building a stimulus model entails quantifying how different features, or properties, change over time. In our single-photon example, we might represent the stimulus as a timeseries of zeros (no photon present) and ones (photon present). For more complex stimuli, however, it may not even be clear what the features are. We discuss considerations and approaches to building explicit stimulus models in Section 41.1.3.

Characterizing brain activity and building stimulus models are each complex challenges in their own right. Linking the two provides its own set of additional challenges. We discuss these issues in Section 41.2.

41.1.2

How can we measure neural “activity” in the human brain? The brain is a complex organ comprising myriad cell types that interconnect to form a vast network. When neuroscientists use the term brain activity, this can refer to a variety of possible physical and physiological phenomena. To contextualize what brain activity means, it can be helpful to first consider the brain’s structure and function.

Neurons are the dominant cell type in the brain; the adult human brain contains roughly 100 billion neurons. The cerebral cortex, commonly associated with high-level brain function, comprises roughly 80% of the adult human brain’s mass, but only roughly 20% of its neurons . In general, brain activity refers to changes in cellular processes that neurons undergo. These processes can take many forms, including : rapid changes in membrane voltage that result in neurotransmitter release, called action potentials ; sub-threshold

Mri (Structural)

Fig. 1. Spatial versus temporal resolution. Each colored region represents the temporal (x-axis) and spatial (y-axis) limits of a recording method or neuroimaging modality [fig- ure inspired by 32, 136]. Green shading denotes in vitro methods. Blue shading denotes invasive in vivo methods. Purple, red, and orange denote non-invasive in vivo methods.

The gray shading denotes suggested ranges of maximum structural and functional rel- evance to experimental-scale behaviors. The shaded region outlined in yellow denotes overlap between the horizontal and vertical gray regions. Note: axes are not drawn to scale.

changes in the membrane voltages of individual neurons or populations of neurons; neurotransmitter release; changes in the composition or distribution of cell surface proteins (including ion channels); metabolic changes (such as increased or decreased blood flow); changes in ion concentrations; structural or anatomical changes, and so on. Because the term brain activity often refers to the activities of neurons specifically, the term neural activity can be a more precise way of referring to these phenomena. However, it is worth noting that neurons are not the only cell type in the brain. For example, the brain also contains roughly 100 billion glial cells [5, 103] , which also play an important role in supporting the neuronal function, synapse formation, and brain metabolism .

After considering the various forms that brain (neural) activity can take, it can also be useful to define a relevant spatiotemporal scale (Fig. 1). For example, a biologist concerned with the structure and function of individual ion channels embedded in the neuronal cell membrane may be most interested in processes that happen over a span of picoseconds or nanoseconds (e.g., the amount of time it takes for an ion to pass through an ion channel, or the amount of time it takes for an ion channel to change its conformation). They may also be most interested in spatial resolutions on the orders of angstroms (e.g., the approximate scale of an individual ion channel). At another extreme, a neuroanatomist studying the comparative anatomy of different species of primates might be most interested in timescales on the order of decades (e.g., an entire lifetime) and spatial scales on the order of decimeters (e.g., the size of an adult brain). As summarized in Figure 1, different neuroimaging approaches are each associated with a range of temporal and spatial scales that they are best suited to measure.

J. R. Manning

If we are specifically interested in stimulus-driven neural activity, this im- plies focusing in on a limited range of spatiotemporal resolutions (gray shading in Fig. 1). Neuroimaging approaches that enable insights at those resolutions may provide particularly useful measures of stimulus-driven neural activity.

Non-invasive approaches. Neuroimaging approaches that rely on measure- ments taken using sensors that are placed without requiring surgery are referred to as non-invasive. In general, non-invasive neuroimaging entails placing one or more sensors on or near the subject’s head. Examples include scalp electroen- cephalography (EEG; i.e., recording voltages from small electrodes placed on the scalp); magnetoencephalography (MEG; i.e., measuring tiny changes in the magnetic field outside of the head caused by neuronal firing); and functional magnetic resonance imaging (fMRI; i.e., inferring changes in blood flow asso- ciated with neural activity using a powerful magnet placed around the head).

A related approach, magnetic resonance imaging (MRI) uses strong magnetic fields, magnetic field gradients, and radio waves to produce a static anatomical image of the brain. Each of these approaches is widely used by neuroscientists interested in studying the neural basis of cognition and behavior. A benefit of relying on non-invasive neuroimaging is that these approaches are low-risk and may be safely used on healthy (non-patient) participants, and without the supervision of a physician. The main drawback of non-invasive neuroimaging is that, because these approaches all rely on sensors placed outside of the head, any relevant activity that is filtered out by the skull, or that is too weak to be mea- sured from distant sensors, cannot be captured. This means that non-invasive neuroimaging approaches tend to have lower spatiotemporal resolution than invasive approaches.

Invasive approaches. Neuroimaging approaches that require surgery are re- ferred to as invasive. Invasive in vivo techniques entail placing sensors directly on the surface of and/or in direct contact with deep structure inside of a living person’s brain. Examples include intracranial electroencephalography (iEEG; i.e., recording voltages from tiny wires implanted in the brain) and electrocor- ticography (ECoG; i.e., recording voltages from small electrodes lying directly onthe brain’scortical surface).BothiEEG andECoGare similarto (non-invasive) EEG, in that all three approaches entail recording aggregate voltages from pop- ulations of many neurons. The key differences between these approaches are the locations and sizes of the electrodes. When sensors are larger and are placed far from the signal sources (i.e., neurons), as in EEG, the sensors pick up on relatively large populations of neurons that are spread over a large portion of the brain. When sensors are smaller and placed in direct contact with signal sources, as in ECoG and iEEG, the sensors pick up on smaller populations of neurons that are closer to the recording surface of the electrodes. When tiny microwires are used to generate iEEG recordings, it is even possible to record action potentials from individual neurons.

5

Invasive in vitro neuroimaging approaches entails taking measurements from brain slices or other structures that have been excised from an intact brain and placed in an isolated environment such as a petri dish. For example, patch clamp recordings use tiny glass micropipettes placed directly on the cell membrane to capture changes in membrane potential associated with the open- ing and closing of individual ion channels. In vitro approaches are not generally used to study stimulus-driven neural activity in humans, since excising the to-be-recorded tissues typically isolates the corresponding neurons from their sensory inputs. In vitro approaches also require destroying the to-be-recorded tissues, which presents ethical and safety concerns.

Because invasive neuroimaging approaches require physically cutting into the brain, they are not appropriate for use in non-patient populations. Rather, invasive recordings in humans are typically taken from neurosurgical patients who have their electrodes implanted as part of a treatment protocol. For exam- ple, people suffering from drug-resistant epilepsy may elect to undergo invasive monitoring (from implanted electrodes) in order to help neurologists localize the most likely source(s) of their seizures. During an extended hospital stay, the patients may elect to participate in research studies that are not directly related to their treatment, in the interest of advancing scientific knowledge by providing access to high quality recordings from their brain. Stimulus-driven responses in individual neurons or small circuits that unfold over sub-millisecond timescales can only be measured using invasive approaches like iEEG and ECoG. By pro- viding measurements at both high spatial resolution and high temporal resolu- tion, intracranial recordings can be ideally suited to studying stimulus-driven neural activity patterns (Fig. 1).

Single-channel neural signals. A single ECoG or iEEG electrode implanted in a patient’s brain measures changes in membrane potential (voltage) in individ- ual neurons, other cells and signal sources, and populations of cells. Because neurons can most effectively transmit signals to other cells via action potentials, the timings of individual action potentials from a given neuron, or the firing rate of a neuron, can provide putative insights into that cell’s function (Fig. 2A).

For example, if a neuron changes its firing rate when the patient is exposed to a particular stimulus or experience, this could suggest that the neuron plays some role in processing information pertaining to that stimulus or experience.

Local field potentials (LFPs) reflect the aggregate neuronal firing and sub- threshold changes in membrane potential across thousands of neurons near the recording surface of an electrode (Fig. 2B). When LFPs change during expo- sure to a stimulus or experience, this can suggest that the underlying population of neurons plays some role in processing that stimulus or experience. These changes may be aperiodic, as in Figure 2B, or sinusoidal, as in Figure 2C. Rhyth- mic (sinusoidal or periodic) changes in the LFP tend to reflect coordinated firing patterns across neurons in the population, whereas uncoordinated changes are reflected as changes in the volatility of the LFP [22, 36, 46, 94].

D

Fig. 2. Measuring and processing single-channel neural signal. All of the examples shown in this figure are constructed using simulated data. A–D. Neural signals. The illustrated examples show voltage (y-axis, arbitrary units) as a function of time (x-axis, arbitrary units). A. Neuronal firing. The vertical lines illustrate the times at which an ar- tificial neuron fired action potentials (spikes). The smooth curve shows the timeseries of firing rates, computed by convolving the spike timeseries with a Gaussian kernel. B. Lo- cal field potentials (LFPs). LFPs reflect the aggregate neuronal firing and sub-threshold changes in membrane potential across thousands of neurons near the recording surface of an electrode. C. Oscillations. When the local field potential exhibits sinusoidal fluctu- ations, this can reflect coordinated changes in membrane potential across a population of neurons. D. Phase coding. The timing of an individual neuron’s action potentials with respect to oscillations in the local field potential can code information via the phase (angle) relative to a sine wave at the oscillation’s frequency. E. Spike-triggered average.

Phase coding may be identified by sampling the LFP before and after each spike and then averaging across all spikes. The spike-triggered average in this panel is computed using the artificial LFP displayed in Panel D. F. Power spectrum and broadband power.

Oscillatory contributions to the local field potential may be summarized as a power spec- trum that shows the extent to which oscillations at each frequency contribute to the LFP. The power spectrum in this panel is computed using the artificial LFP displayed in Panel C. The underlying height of the power spectrum, called broadband power may be estimated by using robust regression to fit a line to the power spectrum in log-log space.

The area under the robust fit line may be used to estimate the firing rates of neurons in the underlying population .

7

When both spike timing information and LFP recordings are available, it is possible to examine whether a given neuron’s spikes are modulated according to the activity of the surrounding population. For example, phase-locked neu- rons tend to fire action potentials during a particular phase (angle) of an oscillation that appears in an LFP (Fig. 2D). Other neurons exhibit phase cod- ing by changing their preferred phase according to properties of the stimulus, ongoing experience, or behavior. For these neurons, the phases (of LFP oscilla- tions) at which spikes occur can carry additional information beyond firing rate alone [70, 76, 91, 142]. One way of characterizing phase-depending firing is to compute a spike-triggered average of the LFP preceding and proceeding each spike (Fig. 2E). When a neuron exhibits a phase preference, its spike-triggered average will look like an oscillation centered at the neuron’s preferred phase.

Although oscillations can sometimes be detected visually by examining a raw LFP recording, a more rigorous approach is to use signal processing meth- ods to quantify the presence of oscillatory components of the LFP. A power spectrum (Fig. 2F) plots the power at each frequency– i.e., the extent to which oscillations at each frequency contribute to the LFP. When the LFP exhibits an oscillation, this appears as a peak (centered on the oscillation’s frequency) in the LFP’s power spectrum. An LFP may also exhibit multiple oscillations, which appear as multiple peaks in the power spectrum.

In addition to true (sinusoidal) oscillations in the LFP, the volatility of the LFP can also change its power spectrum. For example, an increase in the standard deviation of the LFP’s changes in voltages across successive timepoints will result in an increase in power at all frequencies. So-called broadband shifts in power can occur when the neurons in the underlying population change their firing rates .

Given the many ways to measure and characterize neural responses, which approach is best? The answer depends in part on what we hope to learn. For example, if we are interested in processes that we expect to depend on very precise timing and relatively simple neural computations, then neuron-centric signals like spike timing and firing rate may be especially promising. If we are instead interested in processes that we expect to depend on large-scale com- putations carried out by populations of thousands of neurons, then we may instead benefit from focusing on periodic and aperiodic features of local field potentials recorded from relatively large electrodes. In general, lower-level pro- cesses (e.g., signal transduction) tend to rely on smaller numbers of neurons and occur over shorter timescales. Approaches that operate over few neurons and that support high temporal resolution are often best-suited to studying these low-level processes. By contrast, high-level processes (e.g., scene understand- ing, complex planning, emotional processing) tend to rely on large populations of neurons and occur over relatively long timescales . Approaches that record from larger populations of neurons and that measure processes or changes that unfold over longer timescales are often best-suited to studying these high-level processes. While these general principles have tended to hold across many stud- ies and recording modalities, it is worth noting some exceptions. For example,

Intersubject Functional Connectivity

Fig. 3. Univariate and multivariate patterns. Individual electrodes (channels) or features (e.g., firing rate, phase, power at a given frequency, etc.) may be considered in isolation (univariate patterns) or in combination with other channels or features (multivariate patterns). Each feature or pattern may be considered as an “independent” functional unit, or within the context of its broader network. Finally, analyses may be carried out on data from a single patient’s brain (within-brain) or in aggregate across multiple individuals (across-brain). Note: portions of this figure are adapted from .

single-neuron responses in humans and non-human primates can sometimes exhibit selectivity for high-level stimuli and semantic concepts. Units and patterns versus networks. After identifying a set of signals that we think will be appropriate for studying our phenomenon or cognitive process of interest, the next key decisions regard whether we should treat those signals in isolation, or as part of a broader network. Essentially this comes down to a deci- sion about how to combine signals and features within and across participants (Fig. 3).

Early single-neuron recordings (in cats and non-human primates) played a central role in Hubel and Wiesel’s Nobel Prize-winning work on mapping out receptive fields of visual cortical neurons [66, 67]. They mapped out the recep- tive fields of neurons in the primary visual cortex by measuring their firing rates as a function of the visual stimulus shown on the retina. In general, a neu- ron’s receptive field describes the stimulus to which it is maximally responsive.

Hubel and Wiesel’s work showed that the primary visual cortex is organized into orientation columns of neurons whose receptive fields are tilted dark or light bars at a particular orientation relative to horizontal. Several decades later, researchers used high-field fMRI to show analogous orientation columns in human primary visual cortex . The receptive fields of individual neurons can be enormously complex. In contrast to the simple stimuli preferred by pri- mary sensory neurons, neurons in other brain regions can have receptive fields that correspond to high-level concepts. For example, hippocampal place cells fire preferentially when an animal travels to a particular location in an envi- ronment [40, 82, 150]. Other work has shown that some medial temporal lobe neurons appear to increase their firing in response to photographs of specific faces, animals, objects, or scenes .

9

When recordings from several neurons are available, the set of firing rates across the population can provide additional information beyond that contained in the firing rates of individual neurons [4, 121]. For example, if a single place cell responds to one area of an environment, a population of many place cells that each respond to a different location in the environment can provide a rich cognitive map.

Although they do not provide information about spike timing, macro-scale LFP recordings also reflect population-level neuronal activity. Multi-electrode LFP recordings from one or more brain areas can be especially informative. For example, multi-channel LFPs may be used to decode visual stimuli [12, 13], auditory stimuli [13, 143], speech production [52, 112, 122], acute pain onset and intensity , and even semantic representations [39, 97].

Intracranial recordings (of individual neurons, populations of neurons, and LFPs) may also be considered within the context of the larger brain networks to which they belong or contribute . One set of approaches to character- izing brain networks is informed by graph theory, a branch of mathematics concerned with characterizing network architectures, influence, and member- ship [10, 15, 21, 64, 110, 129, 141, 145, 146]. For example, a timeseries of recorded responses from multiple channels may be used to infer functional or causal interactions between the associated neural populations. After mapping out a network of pairwise connections between the responses, graph theoretic mea- sures may be applied to estimate or compare the influence of a given channel or set of channels. Considering interactions can provide information beyond the responses of individual channels or patterns. For example, patterns of in- teractions between neurons or populations can show selective modulation in response to stimuli or features, even when the underlying individual neurons or populations do not appear responsive to the stimulus when considered in isolation .

Static versus dynamic measures of brain activity. When we attempt to discover the neural patterns associated with a particular stimulus or representation, we need to consider two fundamental questions about how the relevant patterns might change over time. The first question is whether brain representations are fundamentally stable. For example, each time you think of a concept, like the meaning of the word “automobile” do the brain areas relevant to representing that concept display the same basic activity patterns? Or do the neural rep- resentations of concepts change in meaningful ways over time, such that the representation of a concept looks fundamentally different each time we mea- sure it? The second question is about whether representations themselves are static or dynamic. For example, when you think of the concept “cat,” does the entire representation essentially become activated as a single unit? Or do dif- ferent components of the representation (e.g., “fur,” “mammal,” “whiskers,” “tail,” etc.) come online in sequence, perhaps in a stereotyped way that adds

J. R. Manning

Some of our conceptual knowledge, and presumably the underlying neural representations of that knowledge, is acquired over timescales on the order of several years. For example, as they develop, children acquire new representa- tions of concepts and how they are related or organized [16, 85]. Changes in neural representations that occur over the course of years are unlikely to be captured by intracranial recordings, which are typically made over timescales on the order of days or weeks.

Another process that leads to changes in the neural representations is pat- tern separation. Pattern separation refers to the phenomenon of differentiating the neural representations of two or more related stimuli or concepts. For ex- ample, pattern separation can occur when we learn to identify and focus in on subtle differences between stimuli or concepts that initially seemed (nearly) identical. Pattern separation can occur over relatively short timescales, and can be identified using intracranial recordings [7, 90, 128, 160].

Although research on the development of conceptual knowledge and pat- tern separation shows that neural representations can change over time, there is also substantial evidence that neural representations are at least somewhat stable over timescales of hours to days, and even across different individu- als [57, 101, 138]. For the most part, this body of work treats the neural repre- sentations of concepts and stimulus responses as essentially static. By assuming that the neural patterns evoked by a fixed stimulus are stable (within and across presentations), one can use a machine learning approach called pattern classi- fication to learn mappings between neural patterns and stimulus labels .

Once these mappings are learned, they may be applied to new neural patterns to estimate the stimulus or “thoughts” associated with those neural patterns. This allows researchers to estimate the cognitive dynamics that occur during neuroimaging [31, 51, 93, 118, 119].

Some stimuli, such as words, images, pure tones, etc., are perceptually static. Most or all of the information in the stimulus is “made available” to our sensory systems at the same time. Certainly it may take some time for higher-order information to unfold in our minds, for example when we are presented with a complex or thought-provoking image. However, those dynamics are driven by internal processes rather than (directly) by the stimulus itself. Other stimuli, such as movies, motion sequences, and dynamic sounds like speech or mu- sic, are fundamentally dynamic. For example, if we were presented with only the average (across time) visual or auditory information in a popular feature- length film, we would be missing nearly all of the structure that made that film engaging or interesting. Dynamic stimuli, particularly naturalistic stim- uli with rich spatiotemporal structure reminiscent of real-world experiences, can evoke dynamic neural responses that are often highly reliable across re- peated presentations to a single participant participant, as well as across par- ticipants [26, 28, 43, 44, 53, 68, 69, 79, 80, 98, 105, 110]. While it is possible to temporally average across the timepoints of responses to some classes of dynamic stimuli while still achieving high reliability , this is not univer- sally true. For example, in the domains of speech comprehension and speech

11

production, temporal information is a primary indicator of meaning. Studying neural responses to speech therefore requires considering how neural correlates of speech unfold over time [52, 122].

Independent of whether a given stimulus is fundamentally static or dy- namic, neural responses can also change according to which other stimuli were experienced nearby in time. For example, interpreting the neural response to ‘B’ in the sequence A B C D B might entail accounting for whether the given instance of ‘B’ is the one that follows ‘A’ or ‘D’. Randomizing stimulus order and averaging over repeated trials effectively removes this sort of contextual infor- mation. However, in some cases, the context in which a stimulus occurs—i.e., the set of other stimuli and thoughts that were experienced nearby in time—can play a critical role in how we process, interpret, and remember incoming infor- mation . For example, priming participants using different cues can reliably bias them to interpet an ambiguous narrative in a particular way . Account- ing for these sorts of contextual effects often entails factoring in stimulus order or content to the corresponding analyses and models.

Building Explicit Stimulus Models

Identifying neural responses to a stimulus requires formalizing what the stimu- lus in question is and when the participant is exposed to it. Broadly, this entails building explicit or implicit models that describe the composition and/or dy- namics of the stimulus. The features, outputs, or predictions of these models may then be related to neural patterns.

What is a stimulus “model”? From an analytic perspective, describing a stimu- lus typically entails characterizing how different aspects of the stimulus change over time. If we are solely interested in the presence or absence of a stimulus, or the timings of a discrete sequence of trials (Fig. 4A), then the “stimulus” might be describable as a simple binary sequence (Fig. 4B). One could then examine or compare neural activity patterns recorded during the “on” timepoints versus “off” timepoints, or estimate how neural responses change during the transi- tions between those binary states. If stimuli are drawn from a well-defined set of categories, then category information may be conveyed using one binary sequence per category (Fig. 4C).

In other instances, we may be interested in understanding how neural re- sponses relate to specific stimulus values, or how those values change over time. For example, we might describe the brightness or salience of a visual stimulus (or the loudness of an auditory stimulus, etc.) as a sequence of real-valued num- bers (Fig. 5A). This could enable us to understand graded neural responses to the stimulus, such as how neural responses change as a function of the stimulus values. Event-triggered averages (analogous to spike-triggered averages such as Fig. 2E) can also provide insights into how the stimulus tended to change during a time window centered on a particular neural event (e.g., an action potential or the appearance of a specific activity pattern).

Presentation Time

Fig. 4. Discrete stimulus timeseries. A. Example stimulus sequence. A succession of images are presented to the participant on a computer screen, interspersed by intervals of blank screen. Images are drawn from three categories: faces, outdoor scenes, and concrete objects. B. Onset and offset timing. Stimulus timing, but not stimulus category, may be conveyed using a single binary timeseries. C. Stimulus identity. Stimulus identity (e.g., category) may be represented using a single binary timeseries for each stimulus category or feature.

Some stimuli are best described by multivariate real-valued feature vectors whose elements (i.e., “features”) describe the absense, presence, or values of specific stimulus properties (Fig. 5B). Describing a stimulus as a timeseries of multivariate feature vectors can facilitate more nuanced mappings between those stimulus properties and different aspects of neural activity. For example, the firing rates of different neurons, or the patterns of power spectra across the electrodes in a given region of interest, might display different sensitivity to different stimulus features.

A fourth (general) way of describing how a stimulus changes over time is to use a multivariate timeseries with explicit event-level or trial-level boundaries (Fig. 5C). For example, in a movie, the scene cuts could constitute event bound- aries– i.e., moments of transition where the stimulus features exhibit rapid “jumps” that are substantially larger than usual between-timepoint changes.

Event boundaries can delineate changes in the focus of an ongoing conversa- tion, scenes in a story or movie, environmental changes, or other transitions in the low-level or high-level content of the stimulus. Experimental trials can also be considered as a sort of event boundary. For example, a multivariate timeseries like that in Figure 5C could also be used to describe the content of a sequence of short video clips presented in succession. Within each clip, the features might change comparatively less than across clips.

As described next, there are many ways to define what the stimulus features are. This requires making assumptions about which aspects of the stimulus “matter” (e.g., in terms of evoking neural responses, predicting behaviors, etc.), and about how the moments of the stimulus timecourse should be matched up with moments of a neural recording. While these assumptions can have a large influence on the outcome of an analysis, there is unfortunately no universal way of describing or modeling stimuli (or of relating stimulus dynamics to neural responses). Rather, one must make informed decisions about how to procede

C

Fig. 5. Continuous stimulus timeseries. A. Univariate real-valued stimulus. The stim- ulus takes on any value at any timepoint. In this example the stimulus values are au- tocorrelated. B. Multivariate real-valued stimulus. The stimulus comprises multiple features, each of which can take on any value at any moment. When projected into 3D, the stimulus traces out a trajectory describing how the values of its features change over time. C. Multivariate real-valued stimulus with event-level or trial-level dynamics.

This stimulus is similar to the one displayed in Panel B, but here the stimulus values exhibit occasional event boundaries. based on the sorts of insights one hopes to gain from an analysis, or based on what approaches have performed well in prior related work.

Manual approaches. In trial-based experiments where stimuli (e.g., words, sounds, images, etc.) have well-defined onset and offset times, the stimulus onset and offset timeseries (e.g., Fig. 4B) can serve as a simple stimulus “model.” A binary sequence that solely describes stimulus onset and offset times ignores stimulus identity (e.g., the category or label) and stimulus features (e.g., the values of the corresponding feature vectors). In this way, modeling the stimulus using a binary sequence makes the implicit assumptions that (a) stimulus timing is the main factor of interest with respect to the associated neural responses and (b) stimulus identity and stimulus features may be safely ignored.

To model stimulus identity and timing (when both identity and timing in- formation are well-defined, e.g. as in trial-based experiments), the stimulus may also be modeled using a set of binary timeseries (Fig. 4C). For example, one might define a separate binary timeseries describing the onsets and offsets of only one stimulus category or trial type. This approach makes the implicit assumption that the specific identities of different exemplars (e.g., within a stim- ulus category– such as face images of different people’s faces) may be safely ig- nored in favor of prioritizing coarser-scale information such as broad stimulus category or trial type labels.

When parameterized stimuli are constructed to vary along one or more ex- plicit stimulus dimensions (e.g., visual stimuli that vary in brightness, contrast, spatial frequency, etc.), the values along each dimension and/or the parame- ters themselves can serve as a representation of the stimulus (e.g., Fig. 5). Each stimulus dimension (or parameter) may be represented by its own real-valued

J. R. Manning

timeseries. This approach makes the implicit assumption that the only relevant sources of variation in the stimulus are those characterized by the specified stimulus dimensions or parameters. All other stimulus features or properties are effectively ignored.

Some stimuli cannot be adequately characterized or described using their associated parameters and/or features that can be directly mapped onto specific physical or perceptual properties. For example, such stimuli might be best described using high-level perceptual or conceptual properties of the stimulus such as the presence of specific objects or high-level content, emotional tone, etc. When these properties may be readily judged or rated by human observers, normed ratings or judgements (typically collected by an independent set of participants) may be used as another means of quantifying stimulus features.

These judgements may take the form of integer-valued or real-valued responses (e.g., rating how “happy” an image or sound is, on a particular scale) or binary “yes/no” judgements (e.g., indicating whether or not a tree is present in an image).

Automated approaches. Many types of stimuli, including natural images, text, complex sounds (e.g., speech, music, recordings of natural environments, etc.), movies, and others, cannot always be easily categorized or manually labeled or rated. In other cases, even if manually labeling stimuli might be possible in principle, in practice it may be too expensive in money or time to generate manual labels. Automated approaches to building stimulus models can scale to millions of stimuli and thousands of stimulus features.

One class of automated approaches to generating stimulus feature models entails applying probabilistic models and deep neural networks such as con- volutional neural networks [84, 88], text embedding models [14, 17, 19, 24, 34, 37, 81, 87, 99, 100, 102, 107, 115, 116, 147, 156], transformers [20, 38, 125], and others to the stimuli. The activations of the hidden or output layers of these networks may be used as feature vectors for the corresponding stimuli.

Another class of automated approaches includes visual entity tagging , image-to-text models [2, 6, 148], speech-to-text models [47, 74, 157, 166], and other algorithms for generating text data from non-text stimuli like still im- ages, video, and sound. After training these models on large corpora of labeled examples, novel stimuli may be automatically tagged with text annotations. In turn, these annotations may be passed onto text embedding models to construct feature vectors for each stimulus. Alternatively, the annotations may be treated directly as stimulus features, for example by treating each unique keyword as a binary feature that is either present or absent in each stimulus.

Human-in-the-loop techniques provide a balance between purely manual and purely automated methods. These techniques entail combining human feedback with classic machine learning approaches. For example, mul- tidimensional scaling [23, 41, 151] may be applied to pairwise similarity judge- ments from human participants to derive n-dimensional feature vectors whose

B

Fig. 6. Within-patient versus across-patient electrode coverage. Each dot denotes the location of the recording surface of one neurosurgically implanted electrode. A. Example patient. The locations of the 169 electrodes implanted in one patient’s brain are displayed.

B. Across-patient electrode locations. The locations of n = 5023 electrodes are displayed. Colors denote different patients (electrodes from m = 53 patients are displayed). Both panels: implantation locations are taken from and filtered using a thresholding procedure to remove noisy signals reported by .

pairwise correlations or (inverse) distances are consistent with those judge- ments. To facilitate comparisons or other analyses, it can be useful to partition con- tinuous stimuli into discrete “states” or “events”. For example, hidden Markov models may be applied to a multidimensional timeseries of observations (e.g., feature vectors) to estimate the moments of transition that are interspersed be- tween periods of relative stability [8, 61, 124]. Neural responses during different events, or at transition points, may then be examined further.

41.1.4

What are some modality-specific challenges to identifying stimulus-driven brain activity from intracranial recordings? Whileintracranialrecordingsprovidehighspatiotemporalresolutiondataabout neural activity (Fig. 1), the coverage afforded by intracranial recordings—i.e., the proportion of the brain volume captured across all electrodes—is relatively poor compared with popular non-invasive approaches like fMRI, MEG, and scalp EEG. The locations of electrodes implanted in a representative patient’s brain are shown in Figure 6A. Across the 169 electrodes, coverage is limited primarily to the left frontal and temporal lobes. The recordings from this ex- ample patient cannot provide direct information about activity outside of these regions.

The precise electrode numbers and implantation locations are determined by teams of clinicians whose primary goal is (typically) to locate the seizure focus for that patient. For some patients, the clinical team may have a relatively good sense of where the patient’s seizures likely originate. These patients may

J. R. Manning

be implanted with relatively few electrodes, spread over a relatively small area of the brain. For other patients where the seizure focus is less clear, they may be implanted with a larger number of electrodes spread throughout the brain.

Taken together, these factors mean that each patient is implanted with a different number of electrodes, in different locations, according to a unique clinical plan. Whereasnon-invasive recordingscan oftenbe easilyaligned acrosspeople(since the sensor locations are typically held relatively constant across people), the alignment problem is highly non-trivial for intracranial recordings.

One benefit to having varied electrode locations across patients is that, for a sufficiently large number of patients, it becomes possible to obtain data from most of the brain (Fig. 6B). Full-brain maps and analyses may be obtained from intracranial recordings by constructing maps that are stitched together across patients. These maps obtained from intracranial recordings may then be compared to analogous maps obtained using other neuroimaging approaches to provide clues about the reliability of the across-patient findings. However, because full-brain maps derived from intracranial recordings typically require combining data across patients, it can be difficult to identify reliable within- patient effects, or to compare responses across patients.

Identifying Stimulus-Driven Neural Activity

Thus far, we have surveyed a variety of approaches for measuring or charac- terizing neural activity patterns and stimuli. When applied in conjunction, the result of these approaches is a set of two timeseries: one describing the patient’s neural responses and the other describing the stimulus the patient experienced as they exhibited those neural responses. The final step is to combine these characterizations in order to relate changes in neural activity to changes in the stimulus. Broadly, this combination step may be carried out within participant or across participant.

Within-Participant Approaches

Within-participant analyses are carried out on data from a single person. Iden- tifying stimulus-driven neural activity using within-participant analyses typ- ically entails combining data over time (e.g., across runs, conditions, trials, etc.). The objective is to estimate maps, patterns, or response profiles that are unique to each individual. These within-participant estimates may then be com- bined across participants to examine general tendencies in the population and/or individual-specific markers.

Generalized linear models and multivariate pattern analysis. Given a neural recordings (Sec. 41.1.2) and a stimulus model (Sec. 41.1.3), the two most widely used approaches to identifying stimulus-driven neural activity are generalized linear models and multivariate pattern analysis. Broadly, generalized linear

17

models (GLMs) are an approach to predicting a set of labels, Y, typically repre- sented by one or more feature vectors, from a set of an equal number of inputed observations, also represented by feature vectors, X . Formally, we say that

Yt = F Xt, Β
,

where yt is the N-dimensional vector of output labels for observation t, xt is a the M-dimensional tth observation, and β is an M×N matrix of weights. The link function, f(·) takes as inputs a set of observations and weights and produces as output a transformed version of the N-dimensional vector β⊤xt. For example, if the elements β⊤xt are Real-valued and lie within the interval (−∞, +∞), then a sigmoidal link function (e.g., the logistic or hyperbolic tangent functions) would transform the elements to lie within the interval (−1, +1). The “power” of the generalized linear model framework comes from the flexibility in how the link function may be defined. By choosing an appropriate link function, it is possible to take arbitrary Real-valued inputs and transform them into outputs that match a wide variety of useful formats– e.g., (unbounded) Real-valued outputs; probability-like values bounded between 0 and 1; indicator vectors (i.e., vectors where all values are 0s except for one element whose value is 1); binary-like values (i.e., where extreme values are “pulled” towards one of two boundaries, as in sigmoid functions); and many more. When X reflects neural activity and Y reflects the stimulus features during the corresponding moments, the fitted GLM weights (i.e., β) describe how different aspects of neural activity relate to different stimulus features. These fitted weights may also be used to “decode” stimulus features from new, previously unobserved, neural data.

Multivariatepatternanalysis(MVPA)describesasecondclassofapproaches for connecting stimulus features and neural activity [56, 109]. Like GLMs, the goal of MVPA is to predict a set of labels from a set of observations. Typi- cally the “labels” comprise stimulus features and the “observations” comprise neural responses. GLMs are a special case of MVPA for which the output fea- tures reflect a (potentially transformed) linear combination of the input features.

However, MVPA also includes a variety of other approaches for which the re- lations between input and output features are non-linear, and potentially even non-monotonic. The umbrella term for such algorithms is pattern classifiers, and includes GLMs, support vector machines , boosting , naive Bayes classifiers [35, 89, 108], nearest neighbor-based classifiers , and (deep) neural network-based classifiers [153, 164], among others.

Representational similarity analysis. Comparing the neural temporal correla- tion matrix (i.e., the correlations between the neural patterns recorded at every pair of timepoints) to the stimulus temporal correlation matrix (i.e., the cor- relations between the stimulus feature vectors at every pair of timepoints) can reveal similarities and differences between how neural and stimulus feature change over time. Representational similarity analysis (RSA) entails comput- ing the element-wise correlation between the upper triangles of the neural and stimulus correlation matrices . Following the logic of , to the extent that

J. R. Manning

neural patterns show a similar temporal correlation structure to stimulus fea- tures, we can interpret this as evidence that the neural and stimulus features are related.

RSA may also be carried out across a series of searchlights. Similar to how a searchlight illuminates a well-defined area of darkness, a searchlight analysis provides insights into the functionality or responses profiles of a focused region of interest. For each of a series of spherical volumes tiled throughout the brain, RSA may be performed for each volume by limiting the neural features under consideration to only those captured by electrodes within that sphere’s radius.

This yields, for each sphere (i.e., searchlight) a single correlation coefficient between that sphere’s neural temporal correlation matrix and the stimulus tem- poral correlation matrix. Examining which searchlights displayed high versus low correlations can highlight which brain areas might represent the stimulus in a way consistent with a given stimulus model (i.e., the model used to construct or estimate the stimulus features).

A convenient property of RSA is that, unlike approaches like GLMs or MVPA, RSA does not require learning an explicit mapping between neural features and stimulus features. This is because RSA is driven solely by pat- tern similarities across timepoints, rather than by the specific properties of the patterns themselves. In this way, RSA can sometimes be a more sensitive way of identifying stimulus-driven neural patterns (e.g., compared with GLMs and MVPA). For example, high levels of noise during many of the exposures to a particular stimulus category will mean that neural decoding approaches will likely fail to effectively learn mappings between the neural and stimulus fea- tures for that category. However, RSA analyses effectively “average” over all timepoints, thereby highlighting aspects of neural activity with a similar tem- poral correlation structure to any stimulus features (even if the associations with a subset of the stimulus features are noisy).

Joint stimulus-activity models. Thus far, we have reviewed two approaches to identifying stimulus-driven neural activity. MVPA attempts to learn mappings between stimulus features and neural features, and RSA attempts to identify stimulus features and neural features that exhibit similar temporal correlation patterns. A third (related) approach entails building models that jointly con- sider the timecourses of neural and stimulus features. Whereas MVPA and RSA implicitly treat stimulus and neural features as a “ground truth,” joint stimulus- activity models allow the stimulus features and neural features to mutually inform each other. These models assume that, while the mappings between stim- ulus and neural features may be relatively stable, there may be some times when the stimulus features provide a more reliable signal and other times when neural features provide a more reliable signal .

Figure 7 provides some geometric intuitions for the idea of joint stimulus- activity models. First, consider how we might go about estimating the reliability of some neural responses we measure as people experience a given stimulus, such as watching a movie. One approach might be to expose a single partic-

Brain

Fig. 7. Joint geometric models of stimulus and neural features. A. Neural features from different individuals. Group-averaged trajectory of fMRI activity from ventral visual cortex split into two randomly-selected groups of subjects (group 1: n = 6, group 2: n = 5) watching the same movie . B. A common geometric space for stimulus and neural features. Group-averaged trajectory of fMRI activity from ventral visual cortex and tra- jectory of movie (pixel intensities over time) hyperaligned to a common space. In Panels A and B, the high-dimensional stimulus and neural trajectories have been pro- jected onto three dimensions to facilitate visualization . C. Interpreting coordinates in the common feature space. Stimuli (e.g., movie frames) may be reconstructed from ventral visual brain activity by mapping the coordinates of neural features in the common space onto coordinates in the stimulus space. Note: this figure is adapted from .

ipant to the same stimulus multiple times. Or alternatively, we might expose several different people to the same stimulus. We could then examine the sim- ilarities and differences between the neural responses recorded across trials or individuals.

Geometrically, the neural features recorded from one individual, during one moment, can be conceptualized as a single point in a high dimensional feature space (whose dimensions each correspond to a single neural feature). Over time, as the individual’s neural activity changes, the successive activity patterns trace out a trajectory through this neural activity space. When the timecourses of neural activity patterns are similar across trials or individuals, this results in similar neural trajectory shapes (Fig. 7A).

The stimulus features from a single moment, as well as the timecourse de- scribing how stimulus features change over time, may also be conceptualized as a trajectory through a (different) high dimensional feature space, whose di- mensions each correspond to a single stimulus feature. It could be interesting to ask whether the neural and stimulus trajectories are similarly shaped, or whether there are particular moments or circumstances under which the shapes converge or diverge. However, because the dimensions of neural trajectories and stimulus trajectories do not (typically) match, an additional step is needed before such comparisons may be made. The procrustean transformation is a geometric transformation for bringing two sets of coordinates into an optimal point-by-point alignment. This entails computing the affine transformations (i.e., rotations, reflections, and scalings) that, when applied to coordinates in one set, minimize the average Euclidean distance between the corresponding coordinates in the second set. The resulting aligned coordinates may then be

J. R. Manning

directly compared, since the transformations map the coordinates in the first set into the same coordinate system (with the same dimensions) as the second set . Mapping a neural trajectory into the stimulus feature space (Fig. 7B, C) provides a common coordinate system for describing both stimulus features and neural features.

We can use this geometric framework to conceptualize what it means to jointly model the stimulus and neural responses. Consider, for example, the stimulus and neural trajectories displayed in Figure 7B. While both trajecto- ries look similar in some respects (e.g., they have roughly similar coarse-scale shapes), they also differ in potentially important ways (e.g., the stimulus trajec- tory is “spikier” than the neural trajectory). Which trajectory is “correct”? On one hand, the stimulus trajectory provides a relatively clean characterization of the stimulus that exactly reflects specific measurable aspects of what partici- pants were exposed to. In this sense, the stimulus trajectory is not “corrupted” by measurement noise, inattention, or other factors unrelated to the stimulus itself. On the other hand, the stimulus trajectory is (by definition) a reflection only of the specific stimulus features that we, as the experimentalists, decided were likely to be important. Those features might be at best different and at worst unrelated to the stimulus properties that participants actually care about or respond to. In this sense, one could argue that neural responses reflect the most direct representation of aspects of the stimulus the brain is responding to, since those neural responses are uncorrupted by the experimentalists’ as- sumptions about which stimulus features are important. Taken together, it is clear that neither the stimulus trajectory nor the neural response trajectory, in isolation, provide a complete reflection of an individual’s internal mental rep- resentations of the stimulus. Instead, it might be most accurate to incorporate aspects of both the stimulus and neural trajectories. This joint stimulus-activity modeling approach acknowledges that the true representation(s) to which an individual’s brain is responding may lie somewhere between the stimulus and neural trajectories.

Across-Participant Approaches

Across-participant analyses are carried out on data from multiple individuals. Identifying stimulus-driven neural activity using across-participant analyses typically entails building an across-participant model or developing analyses that characterize similarities or differences in responses across participants. The objective is to estimate a single map, pattern, or response profile that is common across individuals. Some approaches also attempt to estimate individual differ- ences that characterize how each individual’s responses differ from the group’s (aggregated) responses.

Across-participant models. Building across-participant models for identifying stimulus-driven neural activity requires defining a common representation for describing neural activity (and, potentially, linking neural features with stim- ulus features). Each participant’s data must first be mapped into the common

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.