The Eyes Have It
Pioneered by Rochester professor Michael Tanenhaus, the eyetracker is proving to be an exceptionally insightful way to explore language. By Scott Hauser. Photography by Elizabeth Torgerson-Lamark.
Michael Tanenhaus can remember thinking, “This isn’t working,” as he sat at a table in the virtual reality lab in the Computer Studies Building in 1993.
The professor of brain and cognitive sciences was wearing a helmet-like headpiece outfitted with a camera that was supposed to pinpoint where he was looking as he listened to commands to organize assorted objects on the table. Sitting at a nearby video monitor, Tanenhaus’s then graduate student, Michael Spivey ’96 (PhD), could see what his advisor couldn’t—a pair of crosshairs flitting across the screen as Tanenhaus turned his attention from cup to napkin to plate.
The two were part of a four-person team testing a new idea: whether technology designed to track eye movements could shed light on how humans comprehend language in “real-life” situations. Would the crosshairs on the monitor move in sync with the mention of each object’s name?
Ever the teacher, Tanenhaus’s next thought was, “How am I going to explain in a good, advisorly way that sometimes experiments just don’t work the way you would hope?”
But as he took off the headpiece and turned to Spivey, it was evident that no explanation was necessary.
“Spivey had this huge grin on his face,” Tanenhaus says.
On the monitor, each move of Tanenhaus’s eyes had been recorded down to the millisecond, an unexpectedly rich potential source of data for studying how humans process language. When he later showed such videotapes at professional conferences, there were audible gasps in the audience.
Says Tanenhaus: “It was clear that this was really promising.”
Eye-poppingly clear. Out of that initial trial has grown an entire line of research that has helped psycholinguists, psychologists, cognitive scientists, linguists, vision researchers, and others better understand how humans comprehend spoken language in everyday, natural settings.
First published in Science in 1995, the work and its influence is the subject of three separate, edited volumes (one co-edited by Tanenhaus) scheduled to be published this year.
In their forthcoming book The Interface of Language, Vision, and Action: Eye Movements and the Visual World, Michigan State University researchers John Henderson and Ferranda Ferreira credit Tanenhaus and his lab with ushering in a new era of exploring how humans understand language:
“It is an understatement to say that this technique has caught on,” they write. “In the last five years, psycholinguists have used eye movements to study how speech and language are understood, to learn about the processes involved in language production, and even to shed light on how conversations are managed. [The] second era of research combining psycholinguistics and the study of eye movements promises to be at least as productive as the first.”
Mary Hayhoe, professor of brain and cognitive sciences and herself an expert on how human vision and movement interact, says Tanenhaus’s work is especially insightful because the approach allows researchers to study language in a natural context. Tanenhaus, who now has a multiroom eye-tracking lab in Meliora Hall, conducted his original trials in a lab that Hayhoe shares with Dana Ballard, professor of computer science. “His work was quite revolutionary,” Hayhoe says. “It really started a whole new field.”
Spivey, who now is an associate professor of psychology at Cornell University, says the excitement was palpable.
“We really did feel as if we were coming up with a new way of measuring things in psycholinguistics,” he says. “They were heady times. I haven’t had lab meetings like that in a long time.”
Language—a highly structured system of communication, yet filled with ambiguity, dependent on context, and learnable by infants—has intrigued scientists and poets alike for millennia. But like vision, language is difficult to separate out as an object of study because the tools used to study it—that is, words, meanings, sounds—make up the very stuff of language itself.
By the middle of the 20th century, psycholinguists, those who study the connection between language and the brain, had divided into two broad camps: those who adhere to the “language as product” tradition, which argues that the best way to understand language is to understand the constituent parts that make it up; and the “language as action” tradition, which argues that language cannot be fully understood without considering the context in which it is used.
But both camps agreed that getting a picture of language in its full richness was limited by the artificial settings of most laboratories and experimental environments. They had been looking for a methodology that could capture some of the subtleties in the way people process language.
Intrigued by the challenge, Tanenhaus and his group of doctoral students began experimenting with the eyetracker, a technology first developed for use in psychology (one of its earliest applications was by advertising consultants who were curious about where eyeballs were pointing). The system uses miniature video cameras and computer software to follow the pupils of the eyes as they move with each change of focus. That information is fed to a computer, where crosshairs and a timecode are synchronized to match the movements.
“At the time, it was remarkable what we were seeing,” Tanenhaus says. “You feel a little as if you are inside someone’s head, seeing what they’re seeing. Of course, nothing is a direct window into someone else’s head, but when I first saw the videotapes, it was clear that this was going to be really exciting.”
While Tanenhaus was not the first to notice the connection between eye movements and attention, he and his team were the first to systematically record how the technology could be used to analyze language comprehension.
As people move their attention to a particular object—whether in conversation or not—their eyes naturally shift to bring that object into focus on a speck of nerves called the fovea in the sensitive macular area of the retina. The shift happens with blinding speed —it takes about 40 milliseconds—and occurs unconsciously many billions of times a day.
Tanenhaus, a specialist in the study of how humans deal with ambiguity in comprehending words and sentences, says that the eye needs another 150 to 200 milliseconds to program itself to move in response to a change in attention. For example, when people are presented with a picture of a beagle and a picture of a beaker, within a quarter second—about 250 milliseconds—of hearing the syllable “bea,” and the start of the next syllable, their eyes have figured out where to look and have begun moving in the right direction. If the second sound is “gle,” they look at the dog, and if that sound is “ker,” they look at the glassware.
For a researcher interested in how conversation works, knowing exactly when one speaker recognizes what the other is talking about provides an unimaginable perspective on language processing. The eyetracker, it turns out, is the perfect tool for “seeing” such previously invisible details.
“No research method solves all the problems, but for a whole range of topics, this has proved to be a major breakthrough,” Tanenhaus says.
Tanenhaus credits his interest in language, in large part, to his academic-minded family. Growing up in New York City and Iowa City, where his father was a political scientist and his mother surrounded the family with books and literature, he eventually enrolled at the University of Iowa to study speech pathology and audiology. On the verge of graduation, one of his mentors suggested that he continue his studies at Columbia University.
Originally planning a career in medicine, Tanenhaus thought he would give research a try and ended up getting a Ph.D.
“Psycholinguistics was a young, exciting field at the time,” he says. “It just got more and more interesting.”