Scientists Use Brain Scans and AI to ‘Decode’ Thoughts

AFP, Sunday 7 May 2023

Scientists said they have found a way to use brain scans and artificial intelligence modeling to transcribe “the gist” of what people are thinking, in what was described as a step toward mind reading.

While the main goal of the language decoder is to help people who have lost the ability to communicate, the U.S. scientists acknowledged that the technology raised questions about “mental privacy.”

Aiming to assuage such fears, they ran tests showing that their decoder could not be used on anyone who had not allowed it to be trained on their brain activity over long hours inside a functional magnetic resonance imaging (fMRI) scanner.

Previous research has shown that a brain implant can enable people who can no longer speak or type to spell out words or even sentences.

These “brain-computer interfaces” focus on the part of the brain that controls the mouth when it tries to form words.

Alexander Huth, a neuroscientist at the University of Texas at Austin and co-author of a new study, said his team’s language decoder “works at a very different level.”

“Our system really works at the level of ideas, of semantics, of meaning,” Huth told an online press conference.

It is the first system to be able to reconstruct continuous language without an invasive brain implant, according to the study in the journal Nature Neuroscience.

‘Deeper than language’

For the study, three people spent a total of 16 hours inside an fMRI machine listening to spoken narrative stories, mostly podcasts such as The New York Times’ “Modern Love.”

This allowed the researchers to map out how words, phrases and meanings prompted responses in the regions of the brain known to process language.

They fed this data into a neural network language model that uses GPT-1, the predecessor of the AI technology later deployed in the hugely popular ChatGPT.

The model was trained to predict how each person’s brain would respond to perceived speech, then narrow down the options until it found the closest response.

To test the model’s accuracy, each participant then listened to a new story in the fMRI machine.

The study’s first author, Jerry Tang, said the decoder could “recover the gist of what the user was hearing.”

For example, when the participant heard the phrase, “I don’t have my driver’s license yet,” the model came back with, “She has not even started to learn to drive yet.”

The decoder struggled with personal pronouns such as “I” or “she,” the researchers admitted.

But even when the participants thought up their own stories — or viewed silent movies — the decoder was still able to grasp the “gist,” they said.

This showed that “we are decoding something that is deeper than language, then converting it into language,” Huth said.

Because fMRI scanning is too slow to capture individual words, it collects a “mishmash, an agglomeration of information over a few seconds,” Huth said. “So, we can see how the idea evolves, even though the exact words get lost.”

Ethical warning

David Rodriguez-Arias Vailhen, a bioethics professor at Spain’s Granada University, which is not involved in the research, said it went beyond what had been achieved by previous brain-computer interfaces.

This brings us closer to a future in which machines are “able to read minds and transcribe thought,” he said, warning this could possibly take place against people’s will, such as when they are sleeping.

The researchers anticipated such concerns.

They ran tests showing that the decoder did not work on a person if it had not already been trained on their own particular brain activity.

The three participants were also able to easily foil the decoder.

While listening to one of the podcasts, the users were told to count by sevens, name and imagine animals or tell a different story in their mind. All these tactics “sabotaged” the decoder, the researchers said.

Next, the team hopes to speed up the process so they can decode the brain scans in real time.

They also called for regulations to protect mental privacy.

“Our mind has so far been the guardian of our privacy,” Rodriguez-Arias said. “This discovery could be a first step towards compromising that freedom in the future.”

This article was originally published by Agence France-Presse.


When a bird brain tops Harvard students on a test

Experiment tests human vs. parrot memory in a complex shell game

What happens when an African grey parrot goes head-to-head with 21 Harvard students in a test measuring a type of visual memory? Put simply: The parrot moves to the head of the class.

Harvard researchers compared how 21 human adults and 21 6- to 8-year-old children stacked up against an African grey parrot named Griffin in a complex version of the classic shell game.

It worked like this: Tiny colored pom-poms were covered with cups and then shuffled, so participants had to track which object was under which cup. The experimenter then showed them a pom-pom that matched one of the same color hidden under one of the cups and asked them to point at the cup. (Griffin, of course, used his beak to point.) The participants were tested on tracking two, three, and four different-colored pom-poms. The position of the cups were swapped zero to four times for each of those combinations. Griffin and the students did 120 trials; the children did 36.

The game tests the brain’s ability to retain memory of items that are no longer in view, and then updating when faced with new information, like a change in location. This cognitive system is known as visual working memory and is the one of the foundations for intelligent behavior.

So how did the parrot fare? Griffin outperformed the 6- to 8-year-olds across all levels on average, and he performed either as well as or slightly better than the 21 Harvard undergraduates on 12 of the 14 of trial types.

That’s not bad at all for a so-called bird brain.

Three graphs indicting the parrot’s success against the human participants

“Think about it: Grey parrot outperforms Harvard undergrads. That’s pretty freaking awesome,” said Hrag Pailian, the postdoctoral fellow at the Graduate School of Arts and Sciences who led the experiment. “We had students concentrating in engineering, pre-meds, this, that, seniors, and he just kicked their butts.”

Full disclosure: Griffin has been the star of past cognitive studies, like showing he’s smarter than the typical 4-year-old and as intelligent as a 6- to 8-year-old child. But making Harvard students do a double take on their own intelligence is quite the step up.

To be fair, the Harvard students did manage to keep (some) of their Crimson pride intact. On the final two tests, which involved the most items and the most movement, the adults had the clear edge. Griffin’s average dipped toward the children’s performance — though never below it. The researchers were unable to determine the precise reason for this drop, but they believe it has something to do with the way human intelligence works (arguably making the Harvard students’ victory a matter of performance enhancement of the genetic variety).

The experiment was part of a study published in Scientific Reports in May. Pailian was the lead author and he collaborated with comparative psychologist Irene Pepperberg, Henry A. Morss Jr. and Elisabeth W. Morss Professor of Psychology Susan Carey, and Justin Halberda at John Hopkins University.

The researchers were investigating the limits of the brain’s ability to process and update mental representations. In other words, they were looking at the “working” portion of the visual working memory system. The ability is referred to as manipulation. And ultimately, they were hoping to gain insights into the development and origin of the visual working memory system and the nature of human intelligence.

“Any operation that you perform in your mind, it takes place in visual working memory,” Pailian said. “You store information from the outside world; you play around with it; and then you shuttle it up for higher cognition. It helps fuel STEM aptitude, mental wellness, and all these different types of important cognitive attributes …. We think that one of the main components of human intelligence, the key characteristic is that we’re able to think about all these things in our minds and do these mental manipulations, but if we find that other animals, other species can perform those manipulation operations [and also see how ancient this ability is], maybe that can help us inform what delineates human intelligence from other animal intelligence, as well.”

At a broad level, the paper’s findings hint at the possible evolutionary origins of the ability to manipulate visual memory. Griffin’s success suggests it is not limited to humans and might be shared across species derived from a common ancestor. In this case, the ancestor would be the dinosaurs, since humans and parrots are separated by more than 300 million years of evolution.

“We’re suggesting that it’s possible — we can’t prove this — that dinosaurs, our common ancestor, may have had some basic capacity,” said Pepperberg, a research associate in Harvard’s Psychology Department. “Then this [advanced manipulation] ability could have evolved in parallel [in primates and birds]. The other possibility is that our common ancestor lacked this ability, and it somehow arose independently in these two lines. But we’re arguing that because manipulation is built on storage capacity, and so many different species have similar storage capacities, that some simple form of manipulation likely existed in a common ancestor.”

In the paper, the researchers note that future work is needed to confirm manipulation ability across a wider variety of species and identify its origins.

“It’s not that we proved everything provable,” Pepperberg added. “It’s that we’ve demonstrated a behavior that leads to a lot of different questions.”

Griffin was a prime candidate for this experiment because the researchers needed an animal whose brain was functionally similar to humans but evolutionarily distant for comparison. It was also likely that parrots possessed the manipulation ability because of environmental pressures in the wild, like tracking their hungry fledglings or threats like predators. Plus, Griffin is always ready to show off his brain power and earn a few cashews as a reward.

“He’s the kind of student who asks you, ‘What do I have to do to get the A?’” and then goes and does it, Pepperberg said.


When a bird brain tops Harvard students on a test.

 Experiment tests human vs. parrot memory in a complex shell game.


Copyright (c) 2020 by All rights reserved.

Know Your Brain


The brain is the most complex part of the human body. This three-pound organ is the seat of intelligence, interpreter of the senses, initiator of body movement, and controller of behavior. Lying in its bony shell and washed by protective fluid, the brain is the source of all the qualities that define our humanity. The brain is the crown jewel of the human body.

For centuries, scientists and philosophers have been fascinated by the brain, but until recently they viewed the brain as nearly incomprehensible. Now, however, the brain is beginning to relinquish its secrets. Scientists have learned more about the brain in the last 10 years than in all previous centuries because of the accelerating pace of research in neurological and behavioral science and the development of new research techniques. As a result, Congress named the 1990s the Decade of the Brain. At the forefront of research on the brain and other elements of the nervous system is the National Institute of Neurological Disorders and Stroke (NINDS), which conducts and supports scientific studies in the United States and around the world.

This fact sheet is a basic introduction to the human brain. It may help you understand how the healthy brain works, how to keep it healthy, and what happens when the brain is diseased or dysfunctional.

Image 1

The Architecture of the Brain

The brain is like a committee of experts. All the parts of the brain work together, but each part has its own special properties. The brain can be divided into three basic units: the forebrain Image 2  , the midbrain Image3, and the hindbrain   Image4 .

The hindbrain includes the upper part of the spinal cord, the brain stem, and a wrinkled ball of tissue called the cerebellum (Image 1). The hindbrain controls the body’s vital functions such as respiration and heart rate. The cerebellum coordinates movement and is involved in learned rote movements. When you play the piano or hit a tennis ball you are activating the cerebellum. The uppermost part of the brainstem is the midbrain, which controls some reflex actions and is part of the circuit involved in the control of eye movements and other voluntary movements. The forebrain is the largest and most highly developed part of the human brain: it consists primarily of the cerebrum (Image 2) and the structures hidden beneath it (see “The Inner Brain Image 5).

When people see pictures of the brain it is usually the cerebrum that they notice. The cerebrum sits at the topmost part of the brain and is the source of intellectual activities. It holds your memories, allows you to plan, enables you to imagine and think. It allows you to recognize friends, read books, and play games.

The cerebrum is split into two halves (hemispheres) by a deep fissure. Despite the split, the two cerebral hemispheres communicate with each other through a thick tract of nerve fibers that lies at the base of this fissure. Although the two hemispheres seem to be mirror images of each other, they are different. For instance, the ability to form words seems to lie primarily in the left hemisphere, while the right hemisphere seems to control many abstract reasoning skills.

For some as-yet-unknown reason, nearly all of the signals from the brain to the body and vice-versa cross over on their way to and from the brain. This means that the right cerebral hemisphere primarily controls the left side of the body and the left hemisphere primarily controls the right side. When one side of the brain is damaged, the opposite side of the body is affected. For example, a stroke in the right hemisphere of the brain can leave the left arm and leg paralyzed.

The Forebrain
The Midbrain
The Hindbrain

Each cerebral hemisphere can be divided into sections, or lobes, each of which specializes in different functions. To understand each lobe and its specialty we will take a tour of the cerebral hemispheres, starting with the two frontal lobes (Image1), which lie directly behind the forehead. When you plan a schedule, imagine the future, or use reasoned arguments, these two lobes do much of the work. One of the ways the frontal lobes seem to do these things is by acting as short-term storage sites, allowing one idea to be kept in mind while other ideas are considered. In the rearmost portion of each frontal lobe is a motor area (Image1), which helps control voluntary movement. A nearby place on the left frontal lobe called Broca’s area (Image 1) allows thoughts to be transformed into words.

When you enjoy a good meal—the taste, aroma, and texture of the food—two sections behind the frontal lobes called the parietal lobes (Image 1) are at work. The forward parts of these lobes, just behind the motor areas, are the primary sensory areas (Image 1). These areas receive information about temperature, taste, touch, and movement from the rest of the body. Reading and arithmetic are also functions in the repertoire of each parietal lobe.

As you look at the words and pictures on this page, two areas at the back of the brain are at work. These lobes, called the occipital lobes (Image 1), process images from the eyes and link that information with images stored in memory. Damage to the occipital lobes can cause blindness.

The last lobes on our tour of the cerebral hemispheres are the temporal lobes (Image 1), which lie in front of the visual areas and nest under the parietal and frontal lobes. Whether you appreciate symphonies or rock music, your brain responds through the activity of these lobes. At the top of each temporal lobe is an area responsible for receiving information from the ears. The underside of each temporal lobe plays a crucial role in forming and retrieving memories, including those associated with music. Other parts of this lobe seem to integrate memories and sensations of taste, sound, sight, and touch.

Continue reading

Scientists find ally in fight against brain tumors: Ebola

Here glioblastoma cells from a human brain are growing. Addition of the Ebola-VSV oncolytic virus results in tumor infection and cell death, seen here as black cells. Over time the infection spreads to other glioblastoma cells.

Glioblastomas are relentless, hard-to-treat, and often lethal brain tumors. Yale scientists have enlisted a most unlikely ally in efforts to treat this form of cancer — elements of the Ebola virus.

“The irony is that one of the world’s deadliest viruses may be useful in treating one of the deadliest of brain cancers,” said Yale’s Anthony van den Pol, professor of neurosurgery, who describes the Yale efforts Feb. 12 in the Journal of Virology.

The approach takes advantage of a weakness in most cancer tumors and also of an Ebola defense against the immune system response to pathogens.

Unlike normal cells, a large percentage of cancer cells lack the ability to generate an innate immune response against invaders such as viruses. This has led cancer researchers to explore the use of viruses to combat a variety of cancers.

Continue reading

How the future of brain wave research may be etched in silicon

In the 1950s, engineers faced a challenge. The parts they were using to wire computers – namely transistors – were too bulky for their plans to build more powerful machines. In response, they did something remarkable: they showed that it was possible to greatly shrink a computer’s main circuitry by etching, or chemically burning, the transistors onto tiny chips of silicon. Since then manufacturers have used the same basic process to cram many more circuits onto tinier chips that, ultimately, have powered today’s smartphones, PCs, and the internet.

In a recent article published in Science Translational Medicine, a team of NIH BRAIN Initiative®-funded researchers showed how this chip manufacturing process may also help neuroscientists overcome similar challenges they face today in recording brain wave activity.

Led by Jonathan Viventi, Ph.D., an assistant professor at Duke University, John A. Rogers, S.M., Ph.D., director of the Center on Bio-Integrated Electronics at Northwestern University, and Bijan Pesaran, Ph.D., professor of Neural Science at New York University, the team described how they made the Neural Matrix, a thinner-than-hair, flexible electrocorticography device that has the potential to record brain activity with higher fidelity and for longer periods  than existing devices.

Continue reading