publications

2024

1. Vision and language representations in multimodal AI models and human social brain regions during natural movie viewing

   Hannah Small, H. Lee Masson, S.H. Mostofsky, and L. Isik

   In Proceedings of UniReps: the Second Workshop on Unifying Representations in Neural Models, 2024

   Abs PDF

   Recent work in NeuroAI suggests that representations in modern AI vision and language models are highly aligned with each other and human visual cortex. In addition, training AI vision models on language-aligned tasks (e.g., CLIP-style models) improves their match to visual cortex, particularly in regions involved in social perception, suggesting these brain regions may be similarly "language aligned". This prior work has primarily investigated only static stimuli without language, but in our daily lives, we experience the dynamic visual world and communicate about it using language simultaneously. To understand the processing of vision and language during natural viewing, we fit an encoding model to predict voxel-wise responses to an audiovisual movie using visual representations from both purely visual and language-aligned vision transformer models and paired language transformers. We first find that in naturalistic settings, there is remarkably low correlation between representations in vision and language models and both predict social perceptual and language regions well. Next, we find that language alignment does not improve a vision model embedding’s match to neural responses in social perceptual regions, despite these regions being well predicted by both vision and language embeddings. Preliminary analyses, however, suggest that vision-alignment does improve a language model’s ability to match neural responses in language regions during audiovisual processing. Our work demonstrates the importance of testing multimodal AI models in naturalistic settings and reveals differences between language alignment in modern AI models and the human brain.

2. Precision fMRI reveals that the language network exhibits adult-like left-hemispheric lateralization by 4 years of age

   O. Ozernov-Palchik, A.M. O’Brien, E. Lee, H. Richardson, and 8 more authors

   2024

   Preprint, under review

   Abs DOI

   Left hemisphere damage in adulthood often leads to linguistic deficits, but many cases of early damage leave linguistic processing preserved, and a functional language system can develop in the right hemisphere. To explain this early apparent equipotentiality of the two hemispheres for language, some have proposed that the language system is bilateral during early development and only becomes left-lateralized with age. We examined language lateralization using functional magnetic resonance imaging with two large pediatric cohorts (total n=273 children ages 4-16; n=107 adults). Strong, adult-level left-hemispheric lateralization (in activation volume and response magnitude) was evident by age 4. Thus, although the right hemisphere can take over language function in some cases of early brain damage, and although some features of the language system do show protracted development (magnitude of language response and strength of inter-regional correlations in the language network), the left-hemisphere bias for language is robustly present by 4 years of age. These results call for alternative accounts of early equipotentiality of the two hemispheres for language.

2022

1. Probabilistic atlas for the language network based on precision fMRI data from >800 individuals

   B. Lipkin, G. Tuckute, J. Affourtit, Hannah Small, and 16 more authors

   Scientific Data, 2022

   Abs DOI PDF

   Two analytic traditions characterize fMRI language research. One relies on averaging activations across individuals. This approach has limitations: because of inter-individual variability in the locations of language areas, any given voxel/vertex in a common brain space is part of the language network in some individuals but in others, may belong to a distinct network. An alternative approach relies on identifying language areas in each individual using a functional ‘localizer’. Because of its greater sensitivity, functional resolution, and interpretability, functional localization is gaining popularity, but it is not always feasible, and cannot be applied retroactively to past studies. To bridge these disjoint approaches, we created a probabilistic functional atlas using fMRI data for an extensively validated language localizer in 806 individuals. This atlas enables estimating the probability that any given location in a common space belongs to the language network, and thus can help interpret group-level activation peaks and lesion locations, or select voxels/electrodes for analysis. More meaningful comparisons of findings across studies should increase robustness and replicability in language research.

2. Precision fMRI reveals that the language-selective network supports both phrase-structure building and lexical access during language production

   J. Hu^*, Hannah Small^*, H. Kean, A. Takahashi, and 6 more authors

   Cerebral Cortex, 2022

   Abs DOI PDF

   A fronto-temporal brain network has long been implicated in language comprehension. However, this network’s role in language production remains debated. In particular, it remains unclear whether all or only some language regions contribute to production, and which aspects of production these regions support. Across 3 functional magnetic resonance imaging experiments that rely on robust individual-subject analyses, we characterize the language network’s response to high-level production demands. We report 3 novel results. First, sentence production, spoken or typed, elicits a strong response throughout the language network. Second, the language network responds to both phrase-structure building and lexical access demands, although the response to phrase-structure building is stronger and more spatially extensive, present in every language region. Finally, contra some proposals, we find no evidence of brain regions—within or outside the language network—that selectively support phrase-structure building in production relative to comprehension. Instead, all language regions respond more strongly during production than comprehension, suggesting that production incurs a greater cost for the language network. Together, these results align with the idea that language comprehension and production draw on the same knowledge representations, which are stored in a distributed manner within the language-selective network and are used to both interpret and generate linguistic utterances.

3. Frontal language areas do not emerge in the absence of temporal language areas: A case study of an individual born without a left temporal lobe

   G. Tuckute, A. Paunov, H. Kean, Hannah Small, and 3 more authors

   Neuropsychologia, 2022

   Abs DOI PDF

   Language processing relies on a left-lateralized fronto-temporal brain network. How this network emerges ontogenetically remains debated. We asked whether frontal language areas emerge in the absence of temporal language areas through a ’deep-data’ investigation of an individual (EG) born without her left temporal lobe. Using fMRI methods that have been validated to elicit reliable individual-level responses, we find that-as expected for early left-hemisphere damage-EG has a fully functional language network in her right hemisphere (comparable to the LH network in n = 145 controls) and intact linguistic abilities. However, we detect no response to language in EG’s left frontal lobe (replicated across two sessions, 3 years apart). Another network-the multiple demand network-is robustly present in frontal lobes bilaterally, suggesting that EG’s left frontal cortex can support non-linguistic cognition. The existence of temporal language areas therefore appears to be a prerequisite for the emergence of the frontal language areas.

2018

1. A Xenopus oocyte model system to study action potentials

   A. Corbin, Hannah Small, L.M. Boland, and C. Villalba-Galea

   Journal of General Physiology, 2018

   Abs DOI PDF

   Action potentials (APs) are the functional units of fast electrical signaling in excitable cells. The upstroke and downstroke of an AP is generated by the competing and asynchronous action of Na+- and K+-selective voltage-gated conductances. Although a mixture of voltage-gated channels has been long recognized to contribute to the generation and temporal characteristics of the AP, understanding how each of these proteins function and are regulated during electrical signaling remains the subject of intense research. AP properties vary among different cellular types because of the expression diversity, subcellular location, and modulation of ion channels. These complexities, in addition to the functional coupling of these proteins by membrane potential, make it challenging to understand the roles of different channels in initiating and “temporally shaping” the AP. Here, to address this problem, we focus our efforts on finding conditions that allow reliable AP recordings from Xenopus laevis oocytes coexpressing Na+ and K+ channels. As a proof of principle, we show how the expression of a variety of K+ channel subtypes can modulate excitability in this minimal model system. This approach raises the prospect of studies on the modulation of APs by pharmacological or biological means with a controlled background of Na+ and K+ channel expression.