Embodied Language Conflict

Last night, I read a cool paper by Bergen and colleagues on the role of embodiment in understanding language. The idea is that portions of the brain that are used for perception and motor activity also play a role in understanding language via a process referred to as “simulation”.

Variations of the Perky effect can be used to study language understanding. For example, if a person is simulating while understanding language, it may be harder for him to use that same part of the brain in a visual or motor task. This is exactly what Bergen et al. found:

In Experiment 1, participants viewed sentences whose verbs literally denoted up or down, such as “The cork rocketed,” an “UP” sentence. At the same time, they had to characterize pictures of objects that were either located at the top or bottom of a screen. When the objects were located at the top, they were slower to do so, demonstrating an interference effect that may have occurred because they were simulating an “UP” sentence. This effect was also observed for “DOWN” sentences and objects located at the bottom of the screen.

When reading that "the cork rocketed," you probably simulated something in the upward direction, like this. Image: www.thewinectr.com
When reading that “the cork rocketed,” you probably simulated something in the upward direction, like this.
Image: http://www.thewinectr.com

Experiment 2 was the same, except up/down nouns were used instead of verbs. The experimenters again found an interference effect in the same direction. This suggests that the specific lexical entry isn’t what causes the simulation, but instead understanding the sentence as a whole may.

In Experiment 3, sentences containing verbs that expressed metaphorical motion were used (for example, “The prices climbed.”). There was no interference effect, nor was there an effect in Experiment 3, in which abstract, non-metaphorical verbs (such as “the percentage decreased”) were used. Together, these add support to the idea that the meaning of a sentence as a whole triggers simulation, rather than individual words.

Then this morning, I read a post about a paper that counters Bergen et al.’s findings. In the fMRI study reported, participants were shown nouns, verbs, noun-like nonwords, and verb-like nonwords (their endings were what signaled whether they were noun- or verb-like). The authors found that when viewing verbs and verb-like nonwords, participants’ premotor cortices were activated more than when viewing nouns and noun-like nonwords. They took this as an indication that the observed cortical responses to action words result from ortho-phonological probabilistic cues to grammar class, as opposed to embodied motor representations.

But, what about context? We rarely come in contact with words in isolation, but instead with words embedded in the context of a sentence, and sentences in their contexts too. Since the methods in the anti-embodied language study aren’t reflective of the real-life situations in which we encounter language, are they meaningful? How can we reconcile a these two studies?

Imagining an apple vs. a banana

I’ve written previously about my skepticism regarding the use of fMRI to localize a function in the brain. Multivariate pattern analysis (MVPA) is commonly used to detect subtle differences in patterns of brain activity in fMRI data.

This post articulated the misleading potential of MVPA really clearly.

In the new paper, Todd et al make a very simple point: all MVPA really shows is that there are places where, in most people’s brain, activity differs when they’re doing one thing as opposed to another. But there infinite reasons why that might be the case, many of them rather trivial.

The authors give an example of two similar tasks, imagining apples and imagining bananas. Although standard fMRI analysis might find no significant differences in activation, an MVPA might reveal one area in which the different tasks produced differing patterns of activity (differences in blood flow). At first, it might seem that the MPVA revealed the area of the brain responsible for encoding specific fruits, but this conclusion overlooks context. Every person has had different experiences with apples and bananas. Maybe the participant likes one and not the other. Maybe he ate a banana right before participating in the study and therefore has bananas on his mind. The potential contextual differences in experiences that any individual has had with the two fruits is endless, and would lead to an apparent difference in individual activation patterns for different tasks.

Apples and bananas are linked to different experiences and thoughts for everyone.  Image: http://www.ireallylikefood.com/708661250/do-you-like-to-eat-eat-eat-apples-or-bananas/
Apples and bananas are linked to different experiences and thoughts for everyone.
Image: http://www.ireallylikefood.com/708661250/do-you-like-to-eat-eat-eat-apples-or-bananas/

Here’s another way of looking at the hypothetical task (image below). For Participant A (the dark line), task A (imagining the apple) requires more concentration than task B (imagining the banana) does. For Participant B, the difficulties are reversed. Normally, these individual differences would cancel each other, demonstrating no significant difference between the brain activity when people imagine apples and the activity when they imagine bananas, but MVPA is designed not to allow individual differences to cancel each other out.

MVPA confound
Image from Todd et al.: http://www.sciencedirect.com/science/article/pii/S1053811913002887

The authors of the original paper suggest that MVPA confounds can be controlled by classical Generalized Linear Model (GLM) analysis or by post hoc linear regressions, so  I don’t take this as a sign that MVPA or fMRI data aren’t meaningful, but more of a warning to be very cautious when interpreting them. Further, I take it as a reminder of the importance of context- we’re all such different creatures and our vastly different experiences inevitably color our cognition and behavior.