Two University of Michigan labs are using artificial intelligence and, in one case, brain implants to help people with mobility issues, including paralysis and amputations, lead fuller lives by controlling devices with only their thoughts. Paralyzed people with injuries or neurological problems that prevent them from moving their arms and legs don't have many therapies to help restore lost motor function, said Matthew Willsey, an assistant professor of neurosurgery and biomedical engineering at the University of Michigan Medical School. "Most of the therapies are focused around assistive devices to help them use the function that they have left to accomplish a variety of tasks, like activities of daily living," Willsey said. One approach to restore lost motor function is to use brain-computer interfaces or devices that translate brain activity into commands that control an external device. Willsey works on clinical trials that test the safety and feasibility of these devices. One trial conducted in 2023, when Willsey was at Stanford University, allowed a man with tetraplegia, or paralysis in all four limbs, to control a virtual drone, or quadcopter, with only his thoughts. "This participant in particular had an extremely strong passion for flight," Willsey said. "One of the reasons why he joined the study is the thought that, one day, he could use a brain-computer interface to fly or control a drone." The research is technically innovative and could open new doors for paralyzed people, potentially helping those with limited mobility operate robotic arms, UM researchers said. It could also expand recreational opportunities for activities such as video games, making them more accessible. For Rocky Stoutenburgh, 37, of Southgate, video games are far more than just a hobby. Stoutenburgh suffered a spinal cord injury when he was 19 years old that left him paralyzed and without the use of his hands. "It’s amazing to see this kind of research happening right here in Michigan," Stoutenburgh, who has a large following on social media who follow his gaming, said in an email to The Detroit News that was written by his manager. "Technology is evolving so fast, and anything that helps more people with disabilities engage with gaming and virtual activities is a huge win." Mind-controlled drone One of Willsey's interests is enabling people with paralysis to play video games. The ability to control multiple fingers on both hands simultaneously makes this significantly easier, he said. Using individual fingers is essential for many aspects of everyday life, from typing to playing instruments and video games, Willsey said. While brain-computer interfaces have long been used to allow research participants to control two-dimensional cursors on a computer screen, using them to mimic finger movement is more complicated. "What is making it novel, from a technical standpoint, is our ability to take a neural network, like a sophisticated algorithm that interprets brain activity in someone who has paralysis, and ... train it in a way that they're able to attempt movement and control multiple individuated fingers at once and control them continuously," Willsey said. The device used in Willsey's virtual drone experiment was used in a clinical trial under an investigational device exemption approved by the Food and Drug Administration. It was implanted in the study participant in 2016, Willsey said. He had paralysis of the arms and legs, and in these research studies, participants are typically enrolled after they plateau in their rehabilitation efforts to regain movement, the UM researcher added. The study was part of the BrainGate2 clinical trial. It was designed to collect information about the safety and feasibility of people with tetraplegia using the BrainGate2 brain-computer interface system to control assistive devices with their thoughts. "For us, it was like the testing of this kind of technical capability, but for him, it was the realization of kind of a dream that he had had that he couldn't achieve because of his injury," said Willsey, who completed his neurosurgery residency at UM in 2022 and returned after a one year post-graduate appointment as a clinical instructor at Stanford University. The researchers targeted the part of the brain believed to control hand movement and surgically implanted electrodes — small devices that read electrical signals used by the brain to process information — near those areas in the man's brain. The study patient, who remained anonymous, knew what it was like to move his fingers before his injury and would attempt to do so while hooked up to the brain-computer interface, Willsey said. While he couldn't actually move his fingers, video from the study showed he could move a virtual hand avatar on a screen in front of him. Moving specific virtual fingers allowed the study participant to move the virtual drone forward, backward, left, right, up and down, Willsey said. Eventually, the participant maneuvered the drone in a game through virtual obstacle courses, he added. "He could see the fingers move as he would attempt to move them, and then we would map the position of that finger to one of the control axes for the drone," Willsey said. A wire that goes under the scalp connects the electrodes in the brain to a pedestal that comes through the scalp and connects to a computer, Willsey said. Using the implanted electrodes, researchers record electrical activity in that part of the brain and use algorithms to interpret what the person is trying to do, he said. "We mainly rely on artificial intelligence to build algorithms that train on lots of data, and they see lots of patterns of electrical activity when a participant moves a certain finger, and then it can recognize these patterns when the participant tries to do it on future or subsequent attempts," Willsey said. While controlling two-dimensional cursors is useful for computers, it is not the most functional movement, said Cynthia Chestek, a biomedical engineering and robotics professor at the University of Michigan who has collaborated with Willsey in the past. Individual finger control is much more complex than moving a computer cursor in two dimensions, Chestek said. Willsey's experiment decoded the activity of individual fingers in three dimensions and real time, she explained. "Those fingers could stop at any point, start at any point — it was a true, continuous control of finger movement, and you can imagine that someday being applied to stimulate a paralyzed hand," Chestek said. Other applications Chestek researches motor-neuroprosthetics, or using signals from both the brain and nerves to control prosthetic hand movements. "It sounds like science fiction, and it's very similar to what you're seeing in Matt Willsey's paper. The difference is just the neural control source," Chestek said. "In practice, this is done in a lab in very controlled circumstances. ... A person will come into the lab and there will be, right now, a physical connector that goes to wires that pass through the skin. So, I do this for nerves, and Professor Willsey does this for (the) brain." Chestek said she uses techniques developed for brain-computer interfaces on individual nerves to control prosthetic hands. Patients with electrodes implanted in a muscle, which is wrapped around an amputated nerve, are hooked up to computers running very fast artificial intelligence programs. The programs try to interpret nerve activity fast enough to control a prosthetic hand, similar to how Willsey uses them to control the virtual quadcopter, the UM robotics professor said. "In both cases, you're using real-time AI," Chestek said. "It's not that we necessarily understand these signals. It's that we learn from the person trying to do the task." With a prosthetic hand, patients in Chestek's studies can make four different grasping patterns, ranging from a fist to a pinch, point and active open position. In her lab, research study participants have used prosthetic hands to try to complete practical tasks such as making coffee. But since that work is focused on peripheral nerves, it wouldn't be helpful for people with paralysis. "There's a lot of times where there's a healthy brain, it's just that the signal can't get where it's going," Chestek said. A therapy called functional electrical stimulation uses electrical impulses to stimulate muscles in people with neurological disorders and spinal cord injuries. Chestek said she would love to work with Willsey using these methods in the future, recording activity in the brain and using it to stimulate the arms of research participants with spinal cord injuries or suffering from stroke-related damage. "The good thing is that the neurons below the site of injury remain healthy, and so if there's still neurons in the spinal cord, then we can still stimulate the nerve and have it actually work," Chestek said. "As long as the spinal cord is alive, you can stimulate a paralyzed limb." Making video games accessible Willsey was an electrical engineer before he became a physician and has always been interested in using brain-computer interfaces to help people who otherwise have no other recourse for regaining mobility. The virtual drone research could allow patients with paralysis or limited mobility to play more complicated games and interact with those communities. "Now we're looking at ... enabling technology where they're not limited to playing simpler games, but they can actually interact with the general community and participate in these massive multiplayer games for the social benefits that they provide, kind of the sporting and competitive activities that they provide and just overall recreation," Willsey said. Stoutenburgh, the Southgate gamer, learned how to play video games using something called a Quadstick, a controller for quadriplegics, that is operated entirely by the mouth. He said figuring out how to play games using the device was "life-changing" and brought back a sense of control and independence he thought he'd lost after the injury. "Gaming has been more than just a hobby; it’s been a way to connect with people, build friendships, and feel part of a community again," Stoutenburgh said in an email written by his manager. "After my injury, I wasn’t sure how I’d fit into the world, but gaming gave me that outlet." Over the last several years, Stoutenburgh has amassed more than 1 million followers on TikTok, going by the moniker "RockyNoHands." He is also on Instagram, YouTube and Twitch. "Over the years, I’ve been introduced to other adaptive technologies, like the Augmental, which shows how far things have come in terms of accessibility," Stoutenburgh said. "It’s incredible to think that new technologies might open up those same doors for others who feel limited by physical barriers. Anything that helps more people experience that kind of connection and freedom is something I’m fully behind." What's next Willsey remains focused on improving the devices to be worth the risk of undergoing brain surgery to have an implant put in place. "We did three finger groups where the thumb is moved in two dimensions (using the person's mind), but I would like to see all 10 fingers moving in a very naturalistic way with wrist and arm movements, so that the ultimate goal is kind of full body restoration," Willsey said. "It can be through controlling an exoskeleton, maybe controlling a robotic arm that's attached to some kind of proxy. Maybe it is controlling a computer in a way through virtual reality or video games or remote work." He said he would also like to see the devices become more practical and potentially wireless, rather than using a pedestal to connect to a computer. From strokes to visual impairments, many medical conditions could potentially be improved by interacting with the brain, said Chestek, the robotics professor. "We finally have the ability to interact with the brain at the level of neurons," Chestek said, "and we think that's going to apply to a lot of things we haven't even thought about yet." hmackay@detroitnews.com This article originally appeared on The Detroit News: A paralyzed man flew a drone with his mind. How UM researchers are opening virtual doors