Robert Jacobs

Leveraging his expertise in cognitive science and computer science, Robert Jacobs is using experimental and computational methods to study human perception, learning, memory, and decision making.


Bridging Cognition and Computation

Robert Jacobs

Robert Jacobs has always been fascinated by human behavior and artificial intelligence. “From very early on, I’ve had this idea that if we could understand the brain better, we could build smarter computers and, ideally, make the world a better place.”

He is helping to realize this notion through his research. As a professor of both cognitive and computer sciences and a member of the University’s Center for Computation and the Brain, Jacobs bridges these worlds. He uses experimental and computational methods to study human perception, learning, memory, and decision making.

A large part of what Jacobs does involves studying how people understand concepts. He then builds computational models to represent that knowledge to help us understand the way the brain works.

One big question he explores is how we understand concepts through our senses. For example, we can learn about the shape of an object visually; we can also learn about it tactilely, just by holding it in our hands. It’s the same object shape, yet we can learn about it through visual or tactile modalities.

Here’s another example. A person can examine one object and put it into category A and then look at another object and place it into category B. If that person is blindfolded and handed an object that he’s never touched before, he will know into which category to place it. We learn concepts through a variety of modalities. Jacobs is fascinated by the mind’s ability to grasp information from sensory data.

Jacobs has also begun a new project with Brad Mahon, assistant professor of brain and cognitive sciences and of neurosurgery, to learn how our brains process information. They are using functional magnetic resonance imaging (fMRI) to study the brains of subjects. “While in the scanner, we might have subjects look at one object, and then we might have them close their eyes and touch another object,” says Jacobs. He and Mahon look at what regions of the brain are active when they use their vision, what areas are active when they use touch, and what regions show the same activity irrespective of vision or touch.

“There’s a lot of information in fMRI data, and there’s a lot of noise in it,” he adds. “You have to use data science tools to make sense of it.”

Our brains are robust and flexible. Part of what makes them so is the ability to understand hierarchal structures—from “gist” or summary information, to intermediary levels of understanding, to the fine details of a situation.

“Our brains just know what to do,” Jacobs says. “For instance, a student just walked into this office for the first time. She’s never been here before but recognized this as an office scene. She also knew the sorts of objects she would find here. She didn’t know the specifics, but she could assess the situation. This is our brain taking in high-level gist information.”

Short-term visual memory is also reflective of gist information. “If we leave a room in which there was a group of people, we will likely be able to recall about how many people were there, and we’ll remember general details,” he adds. “We are encoding information at multiple levels of abstraction—our brain is the most advanced computing system out there.” 

Basic research like this is driving discoveries about human cognition. One clinical application that is not too far off involves “sensory substitution.” Imagine there’s a blind person. Obviously, he can’t “see” an object, but what if he could understand object shapes because they have been recoded from a visual image into an auditory signal? Jacobs is developing new machine learning algorithms for learning and representing the kind of knowledge that will impact people’s lives.

Data science is facilitating the application of much more advanced machine learning and statistical tools than have been ever been available. Having faster computers that can run effectively in parallel along with algorithmically better software tools is advancing research and discovery.

For Jacobs and others at the helm of cognitive and computer science, data science is opening the doors to a golden age of discovery. The implications are vast, from understanding decision making and action taking to addressing and treating diseases and medical conditions.