Below please find information in regards to Psychology (Behavioral and Cognitive Neuroscience) PhD student lvan Iotzov’s second exam (defense of research proposal), which is open to all and will take place on Tuesday, May 25th at 11:00am via Zoom (email iiotzov@gradcenter.cuny.edu for Zoom ID). Ivan's abstract is also below.

Neural Speech Tracking: Mechanisms and Practical Applications

Speech signals have a strong and consistent effect on brain activity. Many previous studies have demonstrated the ability to find correlations between the amplitude envelope of ongoing speech and evoked responses measured through EEG or MEG. This correlation appears to be modulated by attention, as well as other high-level factors. It is of particular interest because of the possible practical applications of speech tracking in the steering of hearing aid devices and other assistive hearing devices. These devices are typically difficult to tune and the ability to use an objective neural signal as the basis for their tuning would be a great advancement in comfort and efficacy for their users. In this proposal, investigate the correlation between speech intelligibility and the neural tracking of a speech segment. We show a link between the neural tracking of speech and performance on a behavioral word-recognition task. We also develop a novel behavioral paradigm for the investigation of these effects and show preliminary data demonstrating the validity of this new paradigm. Additionally, we propose further experiments to illuminate the mechanisms behind this speech tracking phenomenon using novel manipulations of speech stimuli. Together, these aims and methods provide a basis for the use of speech tracking as an objective neural measure of intelligibility of speech and look to shed light on the oscillatory mechanisms that create the speech tracking phenomenon.

Maximilian Nentwich, a PhD student in the lab of Dr. Lucas Parra will present his defense of her research proposal on Monday, May 2, 2021 at 1pm. A copy of his abstract is below. If you would like to attend, please contact Maximilian at mnentwi000@citymail.cuny.edu for the Zoom meeting ID.

NEURAL RESPONSES TO NATURALISTIC STIMULI

Maximilian Nentwich

Department of Biomedical Engineering

Mentor: Lucas C. Parra

Abstract

While experiments in neuroscience traditionally focus on well-defined static stimuli, the

results of these studies often fail to explain neural processes in naturalistic settings. Naturalistic

stimuli, like movies, allow for free eye movements and contain various complex visual, semantic

and narrative features. However, defining these features requires subjective and labor-intensive

manual annotations. Alternatively, the reliability of neural signals between brain areas or

subjects can be analyzed. This has led to the identification of brain areas that are correlated

between subjects, dependent on the narrative content of movies and the subjects attention.

Similarly, patterns of correlations between brain areas, termed ‘functional connectivity’ (FC), are

reliably activated during resting state and movie tasks.

FC has been studied extensively with fMRI, is reliable across methods and laboratories

and related to various psychiatric and demographic phenotypes. However, FC has not been

studied well in EEG. Therefore, aim 1 is to compare FC between fMRI and EEG. We

hypothesize that patterns of FC measured by fMRI and EEG are similar, and that FC in

both modalities is related to phenotypic variables. To test this, we analyzed a database of

EEG and fMRI recorded from over 1600 children and adolescents during resting state and

movie tasks. We computed FC matrices in fMRI with Pearson’s correlation, and in EEG with the

imaginary part of coherence (iCOH), a measure of phase-coupling. We then compared the

spatial patterns of FC by correlating connectivity matrices of EEG and fMRI. FC matrices of both

measures were related to phenotypes by multivariate distance matrix regression (MDMR), which

determines if differences of connectivity matrices correspond to differences in phenotypic

measures. Contrary to our hypothesis, we found that the spatial patterns of FC in EEG and fMRI

are distinct. However, FC in both modalities is related to phenotypes. We conclude that EEG

and fMRI FC reflect different neural processes.

To investigate which features of movies drive neural responses most reliably we

analyzed an additional dataset of intracranial EEG using the same movies as the FC dataset.

Movies contain several visual features, particularly temporal contrast, scene cuts, and elicit

saccades. Previous studies on fMRI and EEG suggest that motion, particularly of socially

relevant stimuli, elicits the strongest neural responses in movies. In aim 2 we test whether

motion in movies leads to neural responses in iEEG. We hypothesise that motion elicits

stronger responses than other visual features. Further, we predict that motion of

semantic objects, elicits stronger responses than other motion. We analysed the data with

a linear systems identification approach to identify the neural responses to stimuli extracted

from the movies. We could not find any significant response to optical flow, a measure of

motion. However, we found strong responses after scene cuts and saccades. Additionally,

scene cuts identified as semantic lead to different responses than scene cuts without semantic

content. We conclude that opposed to visual motion, salient novelty events drive neural

responses to movies. We further propose a recognition memory task to test whether semantic

scene cuts are better encoded in memory.

Below please find information in regards to BME PhD student Jason Ki’s dissertation defense, which is open to all and will take place on Tuesday, March 30th at 3:00pm via Zoom. Please email Jason at jki00@citymail.cuny.edu for the Zoom link. Jason’s abstract is also below.

When the Brain Plays a Game:

Neural responses to visual dynamics during naturalistic visual tasks.

Department of Biomedical Engineering

Jason Ki

Mentor: Lucas C Parra

ABSTRACT

Many day-to-day tasks involve processing of complex visual information in a continuous stream. While much of our knowledge on visual processing has been established from reductionist approaches in lab-controlled settings, very little is known about the processing of complex dynamic stimuli experienced in everyday scenarios. Traditional investigations employ event-related paradigms that involve presentation of simple stimuli at select locations in visual space or discrete moments in time. In contrast, visual stimuli in real-life are highly dynamic, spatially-heterogeneous, and semantically rich. Moreover, traditional experiments impose unnatural task constraints (e.g. inhibited saccades), thus, it is unclear whether theories developed under the reductionist approach apply in naturalistic settings. Given these limitations, alternative experimental paradigms and analysis methods are necessary. Here, we introduce a new approach for investigating visual processing, applying the system identification (SI) framework. We investigate the modulation of stimulus-evoked responses during a naturalistic task (i.e. kart race game) using non-invasive scalp recordings.

In recent years, multivariate modeling approaches have become increasingly popular for assessing neural response to naturalistic stimulus. Encoding models use stimulus patterns to predict brain responses and decoding models use patterns of brain responses to predict stimulus that drove these responses. In this dissertation, we employ a hybrid method that “encodes” the stimulus to predict “decoded” brain responses. With this approach, we measure the stimulus-response correlation (SRC, i.e. temporal correlation of neural response and dynamic stimulus) to assess the strength of stimulus-evoked activity to uniquely experienced naturalistic stimulus. To demonstrate this, we measured the SRC during a kart race videogame. We find that SRC increased with active play of the game, suggesting that stimulus-evoked activity is modulated by the visual task demands. Furthermore, we analyzed the selectivity of neural response across the visual space. While it is well-established that neural response is spatially selective to discrete stimulus, it is unclear whether this is true during naturalistic stimulus presentation. To assess this, we measured the correlation of neural response with optical flow magnitude at individual locations on the screen during the videogame. We find that the SRC is greater for locations in space that are task-relevant, enhancing during active play. Moreover, the spatial selectivity differs across scalp locations, which suggest that individual brain regions are spatially selective to different visual dynamics.

Overall, we leveraged the SI framework to investigate visual processing during a naturalistic stimulus presentation, extending visual research to ecologically valid paradigms. Our findings shed new insights about the stimulus-evoked neural response to visual dynamics during a uniquely experienced naturalistic visual task. We show that selectivity of neural response can be spatially-resolved at pixel-level from a low-SNR EEG. In the future, by further probing other spatial and temporal dimensions of the stimuli (beyond optical flow), we may gain new insights into how neural signals convey visual processing during dynamic natural visual experiences.