Predicting Actions from Finger and Hand Forces

Problem Description and Data Collection

Using a force glove during execution of 5 different actions on 2 objects, we studied whether it is possible to distinguish the actions just from the force measurements. Data was collected by instructing a subject to execute the same action with its left and right hand, where one of them was wearing the force glove, and the other one was a bare hand, filmed by a Kinect sensor. For this experiment we used only the time series of 6 forces (for the 5 fingers and the palm) from the glove. 2 subjects performed the 10 actions (5 per object), and repeated each 5 times. This results in 50 time series of 6 forces. The data was annotated, such that they would start at the onset of the movement, but the different executions have different durations.

The Method

Looking at the different raw force measurements, one can recognize already differences in the temporal patterns of forces between different actions. However, there are also strong differences in execution speed, which make it impossible to use simple measures like correlation to quantify similarity of the time series. Furthermore, the magnitude of forces at the 5 fingers and palm are quite different. However, during a single execution of most actions, forces of all fingers were strongly correlated, except for actions like flip (of a sponge), where not all fingers are involved.

Raw force measurements  for the sponge object, 2 subjects and 5 actions Finger force correlations

We used the Dynamic Time Warping (DTW) method to align force sequences. This method creates a non-linear warping of the time axis through dynamic programming, such that the six trajectories can be aligned as closely as possible. We used a maximum warping window of 20 time steps to avoid degenerate solutions. DTW returns the optimal warping distance, and we also reconstructed the optimal warping path such that curves can be plotted aligned.


After alignment, both the magnitude-normalized and unnormalized trajectories showed clear patterns for individual actions and subjects. The DTW distance matrix also exhibited clear block-diagonal structure, which means that actions of the same type are more similar to each other than to trajectories of other actions, even after trying to align them.

Normalized and DTW-aligned force trajectories

As a result, we could use a simple nearest neighbour classifier using DTW distance to classify actions from force trajectories. This gave us excellent results for the sponge object, where only one trial was misclassified (2% error). On the cup object, the results were still very good, resulting in a classification error of 18%.

DTW distance matrix for the sponge object