Non-hand-worn, load-free VR hand rehabilitation system assisted by deep learning based on ionic hydrogel
Graphical Abstract
Abstract
Many individuals suffer from stroke, osteoarthritis, or accidental hand injuries, making hand rehabilitation greatly significant. The current hand rehabilitation therapy requires repetitive task-oriented hand exercises, relying on exoskeleton mechanical gloves integrated with different sensors and actuators. However, these conventional mechanical gloves require wearing heavy mechanical components that need weight-bearing and increase hand burden. Additionally, these devices are usually structurally complex, complicated to operate, and require specialized medical institutions. Here, a Virtual Reality (VR) hand rehabilitation system is developed by integrating deep-learning-assisted electromyography (EMG) recognition and VR human-machine interfaces (HMIs). By applying a wet-adhesive, self-healable, and conductive ionic hydrogel electrode array assisted by deep learning, the system can realize 14 Jebsen hand rehabilitation gestures recognition with an accuracy of 97.9%. The recognized gestures further communicate with the VR platform for real-time interaction in a virtual scenario to accomplish VR hand rehabilitation. Compared with present hand rehabilitation devices, the proposed system enables patients to perform immersive hand exercises in real-life scenarios without the need for hand-worn weights, and offers rehabilitation training without time and location limitations. This system could bring great breakthroughs for the development of a load-free hand rehabilitation system available in home-based therapy.
Keywords
Electronic Supplementary Material
References
Powell, M. P.; Verma, N.; Sorensen, E.; Carranza, E.; Boos, A.; Fields, D. P.; Roy, S.; Ensel, S.; Barra, B.; Balzer, J. et al. Epidural stimulation of the cervical spinal cord for post-stroke upper-limb paresis. Nat. Med. 2023, 29, 689–699.
Hunter, D. J.; Schofield, D.; Callander, E. The individual and socioeconomic impact of osteoarthritis. Nat. Rev. Rheumatol. 2014, 10, 437–441.
Liu, Y.; Xie, W. C.; Tang, Z. B.; Tan, Z. F.; He, Y. Z.; Luo, J.; Wang, X. L. A reconfigurable integrated smart device for real-time monitoring and synergistic treatment of rheumatoid arthritis. Sci. Adv. 2024, 10, eadj0604.
Li, T. F.; Zhao, H.; Li, H. F.; Zhang, Z. B.; Han, F.; Fang, P.; Li, G. L.; Xie, R. J.; Li, Q. S.; Gao, X. H. et al. Dual-modal stretchable sensing patches for in situ monitoring of electromyography and acoustic myography. Adv. Funct. Mater. 2024, 34, 2409780.
Wang, H.; Ding, Q. L.; Luo, Y. B.; Wu, Z. X.; Yu, J. H.; Chen, H. Z.; Zhou, Y. B.; Zhang, H.; Tao, K.; Chen, X. L. et al. High-performance hydrogel sensors enabled multimodal and accurate human-machine interaction system for active rehabilitation. Adv. Mater. 2024, 36, 2309868.
Luo, J. Z.; Li, Y. S.; He, M.; Wang, Z. M.; Li, C. Y.; Liu, D.; An, J.; Xie, W. Q.; He, Y. Q.; Xiao, W. F. et al. Rehabilitation of total knee arthroplasty by integrating conjoint isometric myodynamia and real-time rotation sensing system. Adv. Sci. 2022, 9, 2105219.
Li, S. Y.; Liu, A. D.; Qiu, W.; Wang, Y. M.; Liu, G. Q.; Liu, J. R.; Shi, Y. T.; Li, Y. X.; Li, J. N.; Cai, W. J. et al. An all-protein multisensory highly bionic skin. ACS Nano 2024, 18, 4579–4589.
Siviy, C.; Baker, L. M.; Quinlivan, B. T.; Porciuncula, F.; Swaminathan, K.; Awad, L. N.; Walsh, C. J. Opportunities and challenges in the development of exoskeletons for locomotor assistance. Nat. Biomed. Eng. 2023, 7, 456–472.
Sui, M. L.; Ouyang, Y. M.; Jin, H.; Chai, Z. Y.; Wei, C. Y.; Li, J. Y.; Xu, M.; Li, W. H.; Wang, L.; Zhang, S. W. A soft-packaged and portable rehabilitation glove capable of closed-loop fine motor skills. Nat. Mach. Intell. 2023, 5, 1149–1160.
Zhang, X. B.; Mo, X. Y.; Li, C. B.; Li, F. L.; Jin, J.; Xie, P.; Yao, G.; Lin, Y.; Yao, D. Z.; Xu, P. A wearable master-slave rehabilitation robot based on an epidermal array electrode sleeve and multichannel electromyography network. Adv. Intell. Syst. 2023, 5, 2200313.
Ben-Tzvi, P.; Ma, Z. Sensing and force-feedback exoskeleton (SAFE) robotic glove. IEEE Trans. Neural Syst. Rehabil. Eng. 2015, 23, 992–1002.
Han, Y. L.; Xu, Q.; Wu, F. Design of wearable hand rehabilitation glove with bionic fiber-reinforced actuator. IEEE J. Transl. Eng. Health Med. 2022, 10, 2100610.
Tang, Z. C.; Wang, H.; Cui, Z. X.; Jin, X. N.; Zhang, L. K.; Peng, Y. X.; Xing, B. X. An upper-limb rehabilitation exoskeleton system controlled by MI recognition model with deep emphasized informative features in a VR scene. IEEE Trans. Neural Syst. Rehabil. Eng. 2023, 31, 4390–4401.
Dragusanu, M.; Iqbal, M. Z.; Baldi, T. L.; Prattichizzo, D.; Malvezzi, M. Design, development, and control of a hand/wrist exoskeleton for rehabilitation and training. IEEE Trans. Robot. 2022, 38, 1472–1488.
Verdel, D.; Bruneau, O.; Sahm, G.; Vignais, N.; Berret, B. The value of time in the invigoration of human movements when interacting with a robotic exoskeleton. Sci. Adv. 2023, 9, eadh9533.
Tran, P.; Jeong, S.; Herrin, K. R.; Desai, J. P. Review: Hand exoskeleton systems, clinical rehabilitation practices, and future prospects. IEEE Trans. Med. Robot. Bion. 2021, 3, 606–622.
Zhu, M. L.; Sun, Z. D.; Zhang, Z. X.; Shi, Q. F.; He, T. Y. Y.; Liu, H. C.; Chen, T.; Lee, C. Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications. Sci. Adv. 2020, 6, eaaz8693.
Molinaro, D. D.; Kang, I.; Young, A. J. Estimating human joint moments unifies exoskeleton control, reducing user effort. Sci. Robot. 2024, 9, eadi8852.
Chen, X. S.; Gong, L.; Wei, L.; Yeh, S. C.; Xu, L. D.; Zheng, L. R.; Zou, Z. A wearable hand rehabilitation system with soft gloves. IEEE Trans. Ind. Informat. 2021, 17, 943–952.
Gao, S.; He, T. Y. Y.; Zhang, Z. X.; Ao, H. R.; Jiang, H. Y.; Lee, C. A motion capturing and energy harvesting hybridized lower-limb system for rehabilitation and sports applications. Adv. Sci. 2021, 8, 2101834.
Kim, J. J.; Wang, Y.; Wang, H. Y.; Lee, S.; Yokota, T.; Someya, T. Skin electronics: Next-generation device platform for virtual and augmented reality. Adv. Funct. Mater. 2021, 31, 2009602.
Amin, F.; Waris, A.; Syed, S.; Amjad, I.; Umar, M.; Iqbal, J.; Omer Gilani, S. Effectiveness of immersive virtual reality-based hand rehabilitation games for improving hand motor functions in subacute stroke patients. IEEE Trans. Neural Syst. Rehabil. Eng. 2024, 32, 2060–2069.
Huang, Y.; Zhou, J. K.; Ke, P. C.; Guo, X.; Yiu, C. K.; Yao, K. M.; Cai, S. Y.; Li, D. F.; Zhou, Y.; Li, J. et al. A skin-integrated multimodal haptic interface for immersive tactile feedback. Nat. Electron. 2023, 6, 1020–1031.
Xiao, X.; Fang, Y. S.; Xiao, X.; Xu, J.; Chen, J. Machine-learning-aided self-powered assistive physical therapy devices. ACS Nano 2021, 15, 18633–18646.
Jha, C. K.; Shukla, Y.; Mukherjee, R.; Rathva, P.; Joshi, M.; Jain, D. A glove-based virtual hand rehabilitation system for patients with post-traumatic hand injuries. IEEE Trans. Biomed. Eng. 2024, 71, 2033–2041.
Oh, J.; Kim, S.; Lee, S.; Jeong, S.; Ko, S. H.; Bae, J. A liquid metal based multimodal sensor and haptic feedback device for thermal and tactile sensation generation in virtual reality. Adv. Funct. Mater. 2021, 31, 2007772.
Zhu, M. L.; Sun, Z. D.; Lee, C. Soft modular glove with multimodal sensing and augmented haptic feedback enabled by materials’ multifunctionalities. ACS Nano 2022, 16, 14097–14110.
Yang, S. J.; Cheng, J. H.; Shang, J.; Hang, C.; Qi, J.; Zhong, L. N.; Rao, Q. Y.; He, L.; Liu, C. Q.; Ding, L. et al. Stretchable surface electromyography electrode array patch for tendon location and muscle injury prevention. Nat. Commun. 2023, 14, 6494.
Kumar, D. K.; Pah, N. D.; Bradley, A. Wavelet analysis of surface electromyography. IEEE Trans. Neural Syst. Rehabil. Eng. 2003, 11, 400–406.
Lin, Y.; Li, Q. S.; Ding, C.; Wang, J. Y.; Yuan, W.; Liu, Z. Y.; Su, W. M.; Cui, Z. High-resolution and large-size stretchable electrodes based on patterned silver nanowires composites. Nano Res. 2022, 15, 4590–4598.
Zhao, Q. N.; Gribkova, E.; Shen, Y. Y.; Cui, J. L.; Naughton, N.; Liu, L. S.; Seo, J.; Tong, B. X.; Gazzola, M.; Gillette, R. et al. Highly stretchable and customizable microneedle electrode arrays for intramuscular electromyography. Sci. Adv. 2024, 10, eadn7202.
Jeon, J.; Park, J. W. Stretchable electrodes for interconnects in soft electronics. Nano Lett. 2024, 24, 9553–9560.
Farina, D.; Jiang, N.; Rehbaum, H.; Holobar, A.; Graimann, B.; Dietl, H.; Aszmann, O. C. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: Emerging avenues and challenges. IEEE Trans. Neural Syst. Rehabil. Eng. 2014, 22, 797–809.
Zhang, Q.; Lambeth, K.; Sun, Z. Y.; Dodson, A.; Bao, X. F.; Sharma, N. Evaluation of a fused sonomyography and electromyography-based control on a cable-driven ankle exoskeleton. IEEE Trans. Robot. 2023, 39, 2183–2202.
Wang, S. Y.; Wang, X.; Zhang, W. F.; Shi, X. H.; Song, D. K.; Zhang, Y.; Zhao, Y.; Zhao, Z. H.; Liu, N. Electronic tattoos based on large-area Mo2C grown by chemical vapor deposition for electrophysiology. Nano Res. 2023, 16, 4100–4106.
LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444.
Wainberg, M.; Merico, D.; Delong, A.; Frey, B. J. Deep learning in biomedicine. Nat. Biotechnol. 2018, 36, 829–838.
Ran, X.; Luo, F. Y.; Lin, Z. M.; Zhu, Z. Y.; Liu, C. J.; Chen, B. Blood pressure monitoring via double sandwich-structured triboelectric sensors and deep learning models. Nano Res. 2022, 15, 5500–5509.
Li, N.; Wang, Z. S.; Yang, X. R.; Zhang, Z. Y.; Zhang, W. D.; Sang, S. B.; Zhang, H. L. Deep-learning-assisted thermogalvanic hydrogel e-skin for self-powered signature recognition and biometric authentication. Adv. Funct. Mater. 2024, 34, 2314419.
Fang, Y. S.; Xu, J.; Xiao, X.; Zou, Y. J.; Zhao, X.; Zhou, Y. H.; Chen, J. A deep-learning-assisted on-mask sensor network for adaptive respiratory monitoring. Adv. Mater. 2022, 34, 2200252.
Liu, Y.; Li, C.; Yu, L. X.; Wu, Z. W.; Fan, S. C.; Lv, R. T. Ultra-high sensitivity fiber optic microphone with corrugated graphene-oxide diaphragm for voice recognition. Nano Res. 2024, 17, 7593–7602.
Zhang, P. P.; Guo, W. B.; Guo, Z. H.; Ma, Y.; Gao, L.; Cong, Z. F.; Zhao, X. J.; Qiao, L. J.; Pu, X.; Wang, Z. L. Dynamically crosslinked dry ion-conducting elastomers for soft iontronics. Adv. Mater. 2021, 33, 2101396.
Ding, H. Y.; Liang, X. X.; Wang, Q.; Wang, M. M.; Li, Z. J.; Sun, G. X. A semi-interpenetrating network ionic composite hydrogel with low modulus, fast self-recoverability and high conductivity as flexible sensor. Carbohydr. Polym. 2020, 248, 116797.
Kong, B.; Liu, R.; Hu, X. J.; Li, M. Y.; Zhou, X. T.; Zhao, Y. J.; Kong, T. T. Cornea-inspired ultrasound-responsive adhesive hydrogel patches for keratitis treatment. Adv. Funct. Mater. 2024, 34, 2310544.
Zhang, J.; Chen, X. M.; Lin, J. S.; Zhang, P.; Lei, I. M.; Tao, Y.; Zhang, J. J.; Luo, T.; Liu, J. Hydrogel bioadhesives harnessing nanoscale phase separation for Achilles tendon repairing. Nano Res. 2024, 17, 778–787.
Yang, C. C.; Ji, C. C.; Guo, F. J.; Mi, H. Y.; Wang, Y. W.; Qiu, J. S. Wireless sensor system based on organohydrogel ionic skin for physiological activity monitoring. ACS Appl. Mater. Interfaces 2024, 16, 25181–25193.
Wang, S. H.; Wang, Z. M.; Zhang, L.; Song, Z. Q.; Liu, H.; Xu, X. Sweat-adaptive adhesive hydrogel electronics enabled by dynamic hydrogen bond networks. Chem. Eng. J. 2024, 492, 152290.
Li, X.; Zhu, P. C.; Zhang, S. C.; Wang, X. C.; Luo, X. P.; Leng, Z. W.; Zhou, H.; Pan, Z. F.; Mao, Y. C. A self-supporting, conductor-exposing, stretchable, ultrathin, and recyclable kirigami-structured liquid metal paper for multifunctional e-skin. ACS Nano 2022, 16, 5909–5919.
Leng, Z. W.; Zhu, P. C.; Wang, X. C.; Wang, Y. F.; Li, P. S.; Huang, W.; Li, B. C.; Jin, R.; Han, N. N.; Wu, J. et al. Sebum-membrane-inspired protein-based bioprotonic hydrogel for artificial skin and human-machine merging interface. Adv. Funct. Mater. 2023, 33, 2211056.
Li, Y.; Gu, Y. Z.; Qian, S.; Zheng, S. W.; Pang, Y. C.; Wang, L. L.; Liu, B. G.; Liu, S. J.; Zhao, Q. An injectable, self-healable, and reusable PEDOT:PSS/PVA hydrogel patch electrode for epidermal electronics. Nano Res. 2024, 17, 5479–5490.
Xu, Y. D.; Zhao, G. G.; Zhu, L.; Fei, Q. H.; Zhang, Z.; Chen, Z. Y.; An, F. F.; Chen, Y. Y.; Ling, Y.; Guo, P. J. et al. Pencil-paper on-skin electronics. Proc. Natl. Acad. Sci. USA 2020, 117, 18292–18301.
京公网安备11010802044758号