direkt zum Inhalt springen

direkt zum Hauptnavigationsmenü

Logo der TU Berlin

Inhalt des Dokuments

Hand Gesture Tracking and Control

From Fachgebiet Regelungssysteme TU Berlin

Jump to: navigation, search


Abstract

Research at our group aims at developing a feedback-controlled hand neuroprosthesis that consists of an inertial hand sensor system (HSS), an embedded computer, and two surface electrode arrays on the forearm. The current project aims at the development of algorithms that estimate hand gestures and motions in real time based on the inertial measurements of the forearm sensor, the base unit and the sensor strips. We propose a method that uses dual quaternions and a simplified biomechanical hand model to estimate the fingertip positions. The accuracy of these positions is evaluated with respect to an optical motion capture system. Results show that the HSS is capable of tracking the tip of thumb and the other four fingers. The proposed method yields accurate estimates even if spatially unrestricted hand motions are considered and magnetometer readings are disregarded.

People involved

Description

© Michael Gromotka

Fig. 1: Modular sensor strips of the Hand Sensor System (HanSS). The black inertial measurement chips are connected by a ribbon-shaped flexible circuit board that can be linked to the HanSS base unit.

Functional electrical stimulation (FES) has proven to be a promising technology to assist patients with a stroke or spinal cord injury related motor impairment in their rehabilitation process. Various neuroprostheses to support the wrist and hand movement have been proposed. Most of them are open-loop approaches. As such, these methods do not tap the full potential of the FES for the complex and highly nonlinear system of the hand. In contrast, research at our group aims at developing a feedback-controlled hand neuroprosthesis that adjusts the stimulation based on information about its effect on the motion of the hand. The core elements of that neuroprosthesis are an inertial hand sensor system (HSS), an embedded computer, and two surface electrode arrays on the forearm.

The HSS itself consists of a mandatory base unit that is placed on the back of the hand, an inertial sensor for the forearm and up to five optional sensor strips that can be fixed to the dorsal side of the fingers. The current project aims at the development of algorithms that estimate hand gestures and motions in real time based on the inertial measurements of the forearm sensor, the base unit, and the sensor strips.

We propose a method that uses dual quaternions and a simplified biomechanical hand model to estimate the fingertip positions. In the model, each finger is represented as a kinematic chain of rigid segments that are connected by joints. These joints are constrained to one degree of freedom for the DIP and PIP joint, i.e. allowing only flexion/extension, and two degrees of freedom for the MCP and wrist joints, i.e. allowing flexion/extension and abduction/adduction. Refer to the figure on the below for further details.

The hand posture and motion estimation is based on a simplified biomechanical model of the hand with 1-dimensional PIP and DIP joints and 2-dimensional MCP joints.


The motion estimation algorithms are briefly described as follows. The accelerations and angular rates measured by each sensor unit are combined using a recently developed inertial sensor fusion algorithm. This yields sensor orientations from which we calculate joint angles by means of algorithms that exploit kinematic constraints of the joints. We use literature results on biometrical ratios between finger segment lengths to estimate the functional joint distances of each finger from easily determined hand dimensions. Finally, joint angles and segment lengths are combined to calculate the fingertip positions in a forearm-based coordinate system.

The accuracy of these positions is evaluated with respect to an optical motion capture system. We analyze in detail how various modifications of the biomechanical hand model influence the quality of the estimated fingertip positions. A second aspect of particular interest is to minimize the deteriorative influence of non-homogeneous magnetic fields, which are common in indoor environments.

Fig. 3: Video of a moving hand wearing the Hand Sensor System (HanSS) and a realtime 3D visualization based on the proposed estimation method. Click on figure to view video!

Our experimental results show that the HSS is capable of tracking the tip of thumb, index and middle finger. Small errors in the range of 1cm are found to result from simplifications in the biomechanical model. The predicted positions of the ring and little finger are slightly less accurate. We further find that the proposed method yields accurate estimates even if spatially unrestricted hand motions are considered and magnetometer readings are disregarded. The figure above shows a real-time 3D visualization of the estimated hand pose and motion. Click on the figure to view the video!

Related Publications

M. Valtin, C. Salchow, T. Seel, D. Laidig, T. Schauer. Modular finger and hand motion capturing system based on inertial and magnetic sensors. Current Directions in Biomedical Engineering, 3 (1):19–23, 2017.
C. Salchow, M. Valtin, T. Seel, T. Schauer. Development of a Feedback-Controlled Hand Neuroprosthesis: FES Supported Mirror Training. In Workshop AUTOMED, Wismar, Germany, 2016.
T. Seel, S. Ruppin. Eliminating the Effect of Magnetic Disturbances on the Inclination Estimates of Inertial Sensors. IFAC-PapersOnLine, 50 (1):8798–8803, 2017.
C. Salchow, M. Valtin, T. Seel, T. Schauer. A new semi-automatic approach to find suitable virtual electrodes in arrays using an interpolation strategy. European Journal of Translational Myology, 26 (2):715–718, 2016.
D. Laidig, T. Schauer, T. Seel. Exploiting Kinematic Constraints to Compensate Magnetic Disturbances when Calculating Joint Angles of Approximate Hinge Joints from Orientation Estimates of Inertial Sensors. In Proc. of 15th IEEE Conference on Rehabilitation Robotics (ICORR), page 971–976, London, UK, 2017.
M. Ruppel, T. Seel, S. Dogramadzi. Development of a novel functional electrical stimulation system for hand rehabilitation using feedback control. In Proc. of the 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), page 1135–1139, Singapore, Singapore, 2016.

Recommend this page