Brain Computer Interface transforms mental handwriting into on-screen text-ScienceDaily

Scientists are exploring several ways for people with disabilities to communicate with their thoughts. The latest and fastest way is to go back to handwriting, a vintage way to express yourself.

For the first time, researchers have deciphered brain activity associated with trying to write letters by hand. The team worked with a paralyzed participant who had a sensor embedded in his brain to identify the character using an algorithm when he tried to write it. The system then displayed the text on the screen in real time.

Research co-author Krishnachenoy, a researcher at the Howard Hughes Medical Institute at Stanford University and co-author of the work with Stanford neurosurgeon Jaimie Henderson, has further evolved with this innovation. It states that it will allow people with paralysis to type quickly without using their hands.

By attempting handwriting, study participants entered 90 characters per minute. This is more than double the previous recording of inputs using such a “brain computer interface”. Nature May 12, 2021.

Jose Carmena, a neural engineer at the University of California, Berkeley, who was not involved in the study, said the technique and other techniques could help people with all kinds of disabilities. The findings are preliminary, but “a big step forward in this area,” he says.

Brain computer interfaces transform thoughts into actions, says Carmena. “This paper is a perfect example. The interface deciphers the idea of ​​writing and generates actions.”

Thinking communication

When a person becomes immobile due to injury or illness, the neural activity of the brain to walk, drink coffee, and speak remains. Researchers can use this activity to help people with paralysis or amputation regain their lost abilities.

The need depends on the nature of the disorder. Some people who are out of hand can use a computer with voice recognition and other software. For those who have difficulty speaking, scientists have developed other ways to help people communicate.

In recent years, Shenoy’s team has deciphered speech-related neural activity in the hope of reproducing speech. We also devised a way for participants with embedded sensors to move the cursor on the screen, using ideas related to arm movement attempts. When you point to and click on a character this way, people can enter about 40 characters per minute. This is a previous speed record of the input using the Brain Computer Interface (BCI).

But no one was looking at the handwriting. Frank Willett, a neuroscientist in Shenoy’s group, wondered if it was possible to take advantage of the brain signals triggered by placing a pen on paper. “We want to find new ways to help people communicate faster,” he says. He was also motivated by the opportunity to try something different.

The team collaborated with participants enrolled in a clinical trial called BrainGate 2, which tests the safety of BCI, which relays information directly from the participants’ brains to the computer. (The director of the study is Leigh Hochberg, a neurologist and neuroscientist at Massachusetts General Hospital, Brown University, and Providence VA Medical Center.) Henderson says two small sensors on the part of the brain that control the hands and arms. Was embedded. For example, a person who moves a robot’s arm or cursor on the screen in an attempt to move his paralyzed arm.

Participants, 65 years old at the time of the study, suffered a spinal cord injury and remained paralyzed below the neck. Machine learning algorithms used signals picked up by sensors from individual neurons when imagining human writing to recognize patterns generated by the brain at each letter. With this system, men can copy sentences and answer questions as fast as people of the same age type on their smartphones.

This so-called “brain to text” BCI is very fast. This is because each character elicits a very distinctive activity pattern, making it relatively easy for algorithms to distinguish from each other, Willett says.

New system

Shenoy’s team envisions using handwritten trials for text input as part of a more comprehensive system that includes point-and-click navigation as used on today’s smartphones and voice decoding trials. doing. “It’s what we do naturally to have these two or three modes and switch between them,” he says.

Next, Shenoy says the team intends to work with participants who can’t speak, such as amyotrophic lateral sclerosis and people with degenerative neuropathy that results in loss of movement and speech.

Henderson adds that the new system could potentially help people suffering from paralysis caused by many conditions. They include the brainstem attacks that plagued Jean-Dominique Bauby, the author of the book The Diving Bell and the Butterfly Bell. “He was able to write this inspiring and beautiful book by carefully selecting characters one by one using eye movements,” says Henderson. “Imagine what he could do with Frank’s handwriting interface!”

Video: https: // v = pcApwQxbagg

Note: Clinical trial device. It is restricted to clinical trial use by federal law.

Back to top button