Emg Controlled Wheelchair

Multimodal Surface EMG Hand Gesture Recognition Using

Query-Based Transformers For Prosthetic Control

Federico Del Pupa,b,∗, Elisa Tentorib,c and Manfredo Atzorib,d,e

A B S T R A C T

Hand gesture recognition via surface electromyography (sEMG) is fundamental to human-machine interaction and prosthetic control. In this field, deep learning approaches have become the gold standard. However, current architectures struggle to scale; model performance typically decreases as the number of hand movements increases. Performance degradation is tied to the increased statistical complexity of decoding expanded gesture sets and compounded by the limitations of state- of-the-art approaches, which primarily rely on low-latency unimodal convolutional architectures.

Convolutions operate locally, limiting model’s ability to capture long-range sequential patterns. Unimodal setups cannot leverage complementary information from coordinated signals characterizing movement execution, such as inertial and eye-tracking data. These limitations motivate architectures that integrate local and global features across multimodal physiological sequences. To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end hybrid convolutional-transformer for seamless multimodal integration. EMG-CrossFormer combines representations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers, and decodes the fused representations using learnable gesture queries. EMG-CrossFormer was evaluated on four NinaPro datasets (DB2, DB3, DB7, and DB10) and benchmarked against six state-of-the-art models using increasing number of modalities. Using only sEMG signals, EMG-CrossFormer achieved mean accuracies of 72.33%, 52.48%, 79.16%, and 73.49% on DB2, DB3, DB7, and DB10, respectively, consistently achieving the highest performance. Incorporating inertial signals improved performance to 90.66%, 80.40%, 92.79%, and 92.06%. These results show that joint local-global feature modeling improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition in prosthetics.

Surface Electromyography (Semg)-Controlled Upper-

limb prostheses have been used clinically since the 1960s , and decades of subsequent research have progressively refined their control algorithms and improved their reliabil- ity. These myoelectric devices have been shown to enhance the quality of life for trans-radial amputees by partially restoring limb motor functions [2, 3, 4].

To date, commercial myoelectric solutions integrate pattern-recognition algorithms for automatic sEMG gesture recognition, typically supporting a limited set of functional grasps with high accuracy and minimal latency . Concur- rently, advances in the computational capabilities of embed- ded systems have driven a shift in the research community from classical machine learning approaches to deep learning paradigms . However, this transition has not yet been ∗Corresponding author.

This document is the results of the research project by the European Union’s Horizon Europe research and innovation programme under Grant agreement no 101137074 - HEREDITARY.

Orcid(S): 0009-0004-0698-962X (F. Del Pup);

0000-0002-4755-1577 (E. Tentori); 0000-0001-5397-2063

(M. Atzori)

reflected in commercial pattern-recognition systems, which still predominantly rely on classical algorithms. In research settings, several studies have reported out- standing decoding performance, frequently exceeding 90% accuracy on commonly used gesture sets comprising up to 17 hand movements representative of activities of daily living [7, 8, 9, 10]. However, current literature in this domain often suffers from a lack of reproducibility, complicating the direct benchmarking of different architectures under identi- cal experimental settings. Furthermore, common evaluation protocols frequently omit various movements identified as highly similar by quantitative taxonomies , simplifying the classification task, potentially inflating reported perfor- mance, and decreasing the usefulness of benchmark datasets.

These evaluation gaps make it difficult to critically assess the true benefits of deep learning frameworks over classical machine learning models, such as Random Forests (RF) or Support Vector Machines (SVM), which continue to offer highly competitive performance .

These limitations of current deep learning approaches become particularly evident when models are scaled to larger sets of hand gesture. Under these demanding scenar- ios, the performance of deep learning models drops signif- icantly, often falling below that of machine learning classi- fiers. For example, in , the authors evaluated a shallow

Arxiv:2607.22779V1 [Cs.Lg] 24 Jul 2026

Multimodal Hand Gesture Recognition Using Quey-Based Transformers Convolutional Neural Network (CNN) against a set of 51 distinct hand movements from the Non Invasive Adaptive Prosthetics (NinaPro) dataset , revealing a performance degradation of more than 10% compared to a RF across both intact and amputee cohorts.

The reported reduction in performance highlights some limitations of the existing deep learning frameworks. State- of-the-art networks rely predominantly on unimodal con- volutional architectures optimized for low latency. While conventional convolutions enable highly parallelized and efficient computations that keep system latency below the recommended real-time threshold of 125 ms , they are inherently limited to extracting local features. This local receptive field limits the model’s ability to distinguish com- plex neural activation patterns from noise, thereby reducing decoding accuracy and preventing the model from fully exploiting the performance gains offered by deep neural networks. In contrast, attention layers offer a complementary solution that allows sEMG deep learning models to cap- ture longer temporal dynamics . However, their integra- tion into low-latency, resource-constrained models remains scarcely investigated.

Similarly, unimodal (sEMG-only) approaches prevent these architectures from effectively leveraging complemen- tary information from other signal modalities, such as ac- celerometer data, which can help characterize both motor intent and the physical execution of movement. Previous work has demonstrated that combining sEMG signals with accelerometer data can improve classification accuracy in amputee subjects to levels comparable to intact individuals [17, 18]. Consistent with these findings, the authors in achieved comparable results with a novel multimodal convo- lutional model that integrates sEMG and accelerometer data within a compact, end-to-end framework.

Despite the benefits highlighted, multimodal solutions integrating further modalities (e.g., eye tracking, scene videos) remain less investigated. These limitations motivate the development of novel architectures capable of simulta- neously extracting and combining local and global features across heterogeneous multimodal physiological sequences.

Contributions: To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end, hybrid convolutional- transformer framework designed for seamless multimodal signal integration. EMG-CrossFormer combines represen- tations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers. Then, it decodes the fused representations via cross-attention with learnable gesture queries, inspired by consolidated query- based decoding approaches applied in other domains (e.g.,

Detr ). Emg-Crossformer Was Evaluated On Four

NinaPro databases (DB2, DB3, DB7, and DB10) and bench- marked against six state-of-the-art models using an incre- mental number of input modalities. While this study focuses on the integration of sEMG with accelerometer and eye- tracking data, EMG-CrossFormer has been designed to be easily extended to additional data modalities, facilitating the development of multimodal architectures incorporating, for instance, scene camera or EEG data.

Paper structure: the outline of this paper is as follows. Section 2 describes in detail the model architecture and the experimental setting. Section 3 presents the comparative analysis between EMG-CrossFormer and the other selected models using an incremental number of data modalities. Sec- tion 4 critically discusses the results, highlighting potential limitations and future directions. Finally, a conclusion is drawn in section 5.

2. Methods

This section outlines the methodological design of the study, providing thorough details on the procedure and the architecture, and fostering the reproducibility of the results.

Specifically, it covers dataset selection, data preprocessing, model architecture, training hyperparameters, performance evaluation, and statistical analysis.

The Proposed Emg-Crossformer Model Was Evaluated

using four distinct datasets from the public NinaPro repos- itory1 : datasets 2, 3, 7, and 10, referred to as DB2, DB3, DB7, and DB10, respectively. Table 1 summarizes their main acquisition configurations.

The datasets were selected to ensure a comprehensive evaluation of decoding performance across heterogeneous cohorts of both intact and trans-radial amputee subjects, while also investigating increasing levels of sensor multi- modality. DB2, DB3, and DB7 provide synchronized sur- face electromyography (sEMG) and tri-axial accelerometer recordings acquired with a 12-channel Delsys Trigno™ system (Delsys, Natick, MA, USA). They include a broad set of hand-motion and grasping movements collected from both intact subjects (DB2, DB7) and trans-radial amputee subjects (DB3, DB7). Movements were organized into three sessions, referred to as Exercises B, C, and D, comprising 17, 23, and 9 movements, respectively. Each gesture was repeated six times; active trials lasted 5 seconds and were followed by 3 seconds of rest. DB7 only includes Exercises B and C. Therefore, Exercise D was excluded from DB2 and DB3 to harmonize the label space across the three datasets, yielding 40 discrete gestures plus the resting state. sEMG signals were acquired at a sampling rate of 2 kHz, whereas accelerometer signals were sampled at 148 Hz and upsam- pled by the original authors to match the sEMG sampling rate. All subjects were included in the analysis except for two DB3 subjects, for whom the number of electrodes was reduced due to insufficient residual limb space, according to the NinaPro authors’ usage notes .

DB10 includes 10 grasping movements and the resting state, selected from exercises B and C based on activities of daily living. Data were collected from a cohort of 30 intact subjects and 15 trans-radial amputee subjects . Each movement repetition lasted between 5 and 6 seconds and was

Page 2 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers

Table 1

Main acquisition configurations of the NinaPro datasets used in this work.

10

∗A stands for Amputees; ∗∗Subjects 6 and 7 were discarded according to usage notes provided in ∗∗∗Six subjects and unreliable repetitions were discarded according to usage notes provided in followed by 4 seconds of rest. DB10 was acquired with three recording modalities at the same time: sEMG, accelerometer and gaze signals. sEMG signals were recorded using a 12- channel Delsys Trigno™Wireless system at a sampling frequency of 1926 Hz. Upper-limb kinematics were captured via tri-axial accelerometers at a native sampling rate of 148 Hz, and eye-gaze dynamics were recorded using Tobii Pro Glasses 2 at 100 Hz. Both accelerometer and gaze signals were upsampled by the original authors to match the sEMG sampling rate and ensure temporal alignment across modalities . Following the authors’ usage notes, six subjects were discarded due to poor signal quality, and repetitions marked as unreliable by the original authors were excluded .

2.2. Data Preprocessing

Raw sEMG signals were preprocessed using the follow-

Ing Pipeline:

∙scale conversion: sEMG signals were converted to milli- volts to improve numerical stability and avoid representa- tion issues during mixed-precision (FP16) training.

∙DC component removal: the channel mean was subtracted from each sEMG channel, which is equivalent to removing the DC component.

∙Filtering: sEMG signals were filtered using a second-order (12 dB/oct) forward-backward Butterworth band-pass fil- ter with cutoff frequencies of 20 Hz and 500 Hz. The filter order and cutoff frequencies were selected according to guidelines for biomechanical and clinical applications . Power-line noise and its harmonics had already been removed by the original authors using a Hampel filter .

∙Resampling: sEMG signals were resampled to 1 kHz to reduce memory footprint and the number of floating- point operations (FLOPs) required by the deep learning architectures.

∙Window extraction: sEMG data were partitioned into win- dows of 100, 150, or 200 ms with a 10% shift (correspond- ing to 90% overlap) to increase the number of samples.

∙Rest class balancing: windows belonging to the resting class were undersampled to match the gesture class ratios. Accelerometer data underwent only resampling and win- dow extraction to preserve temporal consistency. Gaze sig- nals, represented as (𝑥, 𝑦) coordinates in image space, were processed in the same way after short intervals of missing values were imputed by linear interpolation, following the guidelines provided in .

2.3. Emg-Crossformer Architecture

EMG-CrossFormer is a hybrid convolutional-transformer model designed to jointly process and fuse multiple input modalities (signals in this study) for hand-movement de- coding. As illustrated in Figure 1, the architecture consists of four main modules: modality-specific backbones, which map each input modality to a compact sequence of feature tokens; a fusion module that combines the token sequences through a cascade of cross-attention fusion blocks; a trans- former decoder with learnable gesture queries; and a two- layer feed-forward network (FFN) for hand-gesture recogni- tion.

The model is highly flexible and was designed to be easily adaptable to different experimental setups, including different types and numbers of input modalities. Unimodal backbones may differ across modalities, provided that they output representations in the form of a sequence of tokens.

Furthermore, the number of modalities can also vary by adding or removing fusion blocks in the cascade. A PyTorch implementation of the model is provided in the openly available source code2. Initialization hyperpa- rameters are also reported in the Supplementary Materials.

2.3.1. Unimodal Backbone

Let 𝑚denote the input modality, 𝐶𝑚the number of chan- nels and 𝑊𝑚the number of samples within the input window. Given an input multi-channel signal 𝐱sig𝑚∈ℝ𝐶𝑚×𝑊𝑚, the 𝑚-th unimodal backbone generates a compact representation 𝐳enc𝑚∈ℝ𝐿𝑚×𝑑(Figure 1), where 𝐿𝑚is the sequence length (number of output tokens) and 𝑑is the token embedding di- mension. 𝐿𝑚may vary across modalities, particularly when the corresponding signals have different sampling rates. By contrast, 𝑑is kept fixed across modalities to enable the cross- attention in the fusion and decoding modules.

To assess the architecture’s adaptability to different encoder designs, two unimodal backbone implementations were evaluated: a 1D depthwise convolutional encoder and a 2D multi-scale convolutional encoder.

Page 3 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers

Pe : Positional Encoding

Figure 1: Schematic representation of EMG-CrossFormer. The model combines representations (𝐳enc𝑚) from unimodal backbones through cascaded fusion layers (transformer decoder layers). The transformer decoder block decodes the fused representation (𝐳fuse) using learnable hand movement queries. A final shared FFN outputs the hand movement predictions from the decoded representation (𝐳dec). EMG-CrossFormer can integrate an arbitrary number of modalities. To illustrate this extensibility, the third modality branch and its corresponding fusion layer are shown as optional (dashed box).

The 1D Depthwise Implementation, Based On , Ex-

tracts channel-specific temporal features through stacked depthwise convolutions and average pooling, with low pa- rameter cost. Each block doubles the feature dimension while halving the temporal resolution. Because convolutions are depthwise, this implementation maintains a low parame- ter count and restricts feature extraction to temporal patterns, leaving the modeling of inter-channel patterns to subsequent stages of the architecture.

The 2D multi-scale implementation, based on , treats the multi-channel signals as a single-channel pseudo-image and applies parallel convolutional branches with different kernel sizes, to capture patterns at multiple frequency scales.

The extracted features are subsequently combined through separable convolutions to generate a compact representation while reducing the parameter count. Compared to the 1D depthwise backbone, the 2D multi-scale architecture extracts spatio-temporal features at the cost of increased parameters and computational load.

To integrate these backbones into the EMG-CrossFormer architecture while preserving the architectural designs pro- posed by the original authors, a learnable linear projection layer is appended to each encoder to map its output into a shared 𝑑-dimensional embedding space. The resulting EMG-CrossFormer variants are denoted as EMG-CF1D and EMG-CF2D, respectively.

The Fusion Stage Combines A Variable Number 𝑀of

input modalities through a sequential cascade of non-causal transformer decoder layers. Let 𝐳enc𝑚∈ℝ𝐿𝑚×𝑑represent the compact tokenized sequence generated by the 𝑚-th unimodal backbone, with 𝑚∈{1, … , 𝑀}. Here, 𝑚= 1 denotes the primary sEMG modality, while 𝑚> 1 denotes subsequent auxiliary signals.

Before fusion, a sinusoidal positional encoding 𝐸𝑚

Pos ∈

ℝ𝐿𝑚×𝑑is injected into each unimodal feature set. For aux- iliary modalities, a learnable modality-specific bias, defined

Mode ∈ℝ1×𝑑, Is Also Added Directly To The

sequence (see "Mode token" in Figure 1). This token acts as an indicator to help fusion blocks contextualize non-sEMG signal representations. For each modality, the processed

(1)

where 𝟏𝐿𝑚= (1, … , 1)𝑇∈ℝ𝐿𝑚×1. The mixing of representations across modalities is per- formed via a directional cascade, as schematized in Figure 1.

For notational reasons, the representations of the first pri- mary modality are considered the initial state of the fusion

Fuse =̄ 𝐳enc1. For Each Subsequent Auxiliary

modality 𝑚= 2, … , 𝑀, the (𝑚−1)-th fusion block applies the set of operations of a non-causal transformer decoder layer.

More formally, the output of the 𝑚-th block in the cascade is

)) (3)

Here, LN denotes the layer normalization, FFN a point- wise feed-forward network, and MHA the multi-head atten- tion mechanism. For an in-depth description of the multi- head attention mechanism, the reader is referred to the original implementation of the transformer model .

Page 4 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers Following the final cascade layer, the module outputs a highly descriptive, multimodal representation of the input

ℝ𝐿1×𝑑. This Representation

fully compresses the dynamics of all available modalities while preserving the native token sequence length 𝐿1 of the underlying sEMG driver.

In this study, transformer decoder layers were initialized with an embedding dimension 𝑑= 128, 8 heads, SwiGLU activation , and no dropout.

The Fused Multimodal Representation 𝐳fuse ∈ℝ𝐿1×𝑑

is fed into a query-based transformer decoder. Following the formulation originally introduced for object detection tasks in computer vision , this module implements the standard transformer decoder operations described in Equa- tion 2, using a set of learnable query embeddings as de- coder inputs. Analogously, EMG-CrossFormer uses learn- able hand gesture queries, each encoding a candidate hand movement. Through self-attention among queries and cross- attention over 𝐳fuse, the model jointly reasons over all ges- tures, capturing pairwise relationships while using the mul- timodal fused representation as contextual information. In this study, the number of layers in the query-based decoder was set to 4, with each layer initialized using the same hyperparameters adopted for the fusion stage layers.

2.3.4. Feed-Forward Network

Final predictions are produced by a shared two-layer Feed-Forward Network (FFN) applied independently to each decoded hand gesture query. The hidden layer has dimen- sionality 𝑑= 128 and is followed by a ReLU activation.

The FFN outputs class logits for each query, which are subsequently normalized with a softmax function to obtain class probabilities.

2.3.5. Model Configuration

In the unimodal setting, only sEMG data are used. To apply the fusion architecture within this setting, the sEMG signals are divided into two model input modalities: forearm channels, corresponding to the first eight channels, and upper-arm channels, corresponding to the last four channels.

The forearm channels are provided as the primary input modality, while the upper-arm channels are provided as an auxiliary input modality, potentially allowing the model to learn muscle synergies [29, 30]. In multimodal settings, the model receives sEMG together with accelerometer signals and, when available, gaze information. In this case, sEMG is kept as a single primary input modality, rather than being split into forearm and upper-arm channels. Accelerometer signals define the second input modality, while gaze infor- mation is used as the third input modality when present. Ad- ditional details regarding model implementation, auxiliary training outputs, and hyperparameter selection are provided in the Supplementary Materials.

Deep Learning Models Were Implemented And Trained

using PyTorch , while conventional machine learning models were implemented with Scikit-learn . Statistical analyses were performed using SciPy and statsmodels

. Figures Were Generated With Seaborn . Experi-

computing cluster equipped with four NVIDIA A30 GPUs. Further implementation details are available in the open- source codebase.

2.4.1. Data Partition

Results were obtained using an intra-subject evaluation protocol, consistent with the domain [19, 10], where pros- thetic devices are required to work robustly on a specific subject. In this setting, models are trained and evaluated on different repetitions of hand movements from the same subject. Specifically, repetitions 1, 3, 4, and 6 were assigned to the training set, while repetitions 2 and 5 were assigned to the test set. For DB10, additional repetitions (7 and 8) were included in the training set.

2.4.2. Model Comparison

To provide a fair benchmark, four representative deep learning models were selected for comparison: Shallow

Ral Network (Mkcnn) , And Narrow Kernel Dual-View

Feature Fusion Convolutional Neural Network (NKDFF) . These models were selected because they achieve com- petitive performance on sEMG-based hand gesture recog- nition while representing different architectural paradigms, ranging from lightweight convolutional networks to residual and multi-scale architectures. Moreover, their implementa- tion details are sufficiently documented to enable faithful reproduction. NKDFF also provides a multimodal variant that incorporates accelerometer data. Although additional recent models were considered, they were not included because insufficient implementation details, the absence of open-source code, or substantial differences in the original experimental protocols prevented a reliable reproduction and fair comparison.

The open-source codebase was designed to be extensi- ble, enabling the straightforward integration of additional models and future benchmark extensions. This design choice is intended to encourage community contributions to the source code, support the expansion of the benchmark anal- ysis, and promote fair and reproducible evaluation of novel approaches.

Traditional machine learning models were also included in the evaluation, as they remain competitive for sEMG- based hand gesture recognition , as discussed in Sec- tion 1 and further confirmed in Section 3. Specifically, a RF classifier and an SVM were evaluated. These models were trained using the same set of handcrafted features listed in . Additional details on the features and the hyper- parameter search grids are provided in the Supplementary Materials.

Page 5 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers

2.4.3. Data Augmentation

Data augmentation was incorporated during training to mitigate overfitting. Specifically, a signal warping strategy was implemented to randomly stretch or compress portions of the input signal along the temporal axis, similarly to

[26, 36]. The Procedure Is Defined As Follows:

1. The input window is divided into multiple segments. 2. Up to half of these segments are randomly stretched, while the remaining segments are squeezed.

3. A non-uniform temporal grid is constructed according to the selected stretch and compression operations.

The Grid Values Are Determined By The Stretch And

compression strength hyperparameters. 4. The input window is interpolated onto the non-uniform grid using the Piecewise Cubic Hermite Interpolating Polynomial (PCHIP) .

5. The resulting signal is treated as uniformly sampled

Pchip.

Mini-batches were augmented with a probability of 70% dur- ing training. When signal warping was applied, the number of segments (4, 6, or 8), compression strength (1.25, 1.5, or 2.0), and stretch strength (1.0, 1.5, or 2.0) were randomly sampled from a predefined hyperparameter grid. To improve computational efficiency, the same augmentation parameters were applied to all samples within a mini-batch through broadcasting along the batch dimension as well as to all input signals to preserve temporal consistency across modalities.

2.4.4. Training Loss

EMG-CrossFormer was trained using a composite loss function designed to provide deep supervision across the encoding, fusion, and decoding stages. The loss jointly optimizes classification performance and embedding-space structure. More formally, let 𝑀denote the number of input

∙𝐬be The Final Ffn Output Logits;

∙𝐬enc𝑚and 𝐬fuse be auxiliary logits obtained from a linear projection of the global average pooling output of 𝐳enc𝑚

∙𝐳dec Be The Query-Based Decoder Output;

∙𝐡be the ground-truth handcrafted sEMG features used by machine learning models (see Supplementary Materials); ∙̂ 𝐡be the handcrafted sEMG feature estimates, obtained by applying a linear predictor of the global average pooling output of 𝐳dec.

(4)

Here, 𝐶𝐸denotes the Cross-Entropy with label smooth- ing (𝛼smooth = 0.1), 𝐿1 denotes the Mean Absolute Error (L1) loss, and 𝑆𝑢𝑝𝐶𝑜𝑛denotes the supervised contrastive loss formalized in . Based on empirical tuning, the weighting hyperparameters were set to 𝜆𝑐𝑒= 3.0, 𝜆𝑠𝑢𝑝𝑐𝑜𝑛= 1.5, 𝜆𝑎𝑢𝑥=1.0, and 𝜆ℎ𝑎𝑛𝑑=2.0.

Emg-Crossformer Was Trained Using The Lamb Opti-

mizer with default parameters (𝛽1 = 0.9, 𝛽2 = 0.999, no weight decay) . LAMB was preferred over the stan- dard ADAM optimizer , as it provided greater training stability across all investigated models. Gradient clipping with maximum norm 0.1 was applied to further stabilize training. A standard maximum norm value of 1.0 was also tested but yielded worse results. The batch size was set to 128. The initial learning rate was set to 5.0 ⋅10−4 for the unimodal backbones and 5.0⋅10−5 for the fusion and decoder modules. An exponential scheduler with 𝛾= 0.99 was used to decrease the learning rate after each epoch. All models were trained using mixed precision to accelerate training, reduce memory usage, and better reflect potential real-world deployment on embedded devices.

The Number Of Epochs Was Set To 200 For Semg-Only

training and to 100 for multimodal approaches, as multi- modal training showed faster convergence. Early stopping was not adopted, since creating a separate validation set from the training gestures excessively reduced the number of available training samples. The custom training loss previ- ously described was used to provide deep supervision during training.

All other selected deep learning models (Shallow CNN,

Resnet-1D, Mkcnn, And Nkdff) Were Trained Using

the same set of training hyperparameters. However, unlike EMG-CrossFormer, the learning rate was set to 5.0 ⋅10−4 for the entire network, and categorical cross-entropy was used as the training loss. For the machine learning models, hyperparameter tuning was performed using 4-fold cross- validation on the training repetitions, followed by refitting on the entire training set using the optimal hyperparameters.

Handcrafted features were standardized using a Stan- dardScaler fitted on the training data and incorporated into the hyperparameter search pipeline.

2.4.6. Performance Evaluation and Statistical Analysis Model performance was evaluated using balanced accu- racy, to account for class imbalance. Additional evaluation metrics, including the F1-score and Cohen’s kappa, are reported in the Supplementary Materials. Pairwise model comparisons were performed using the Wilcoxon signed- rank test applied to subject-level performance estimates.

This non-parametric paired test matches the intra-subject design: model comparisons are computed within subjects, while the resulting subject-level differences are treated as independent observations across subjects. 𝑝-values were

Page 6 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers corrected for multiple comparisons using the Benjamini- Hochberg method . In addition to statistical significance, the mean paired improvement and its uncertainty, estimated by non-parametric bootstrap resampling, are reported to quantify the magnitude and practical relevance of perfor- mance differences.

3. Results

This section summarizes the results of nearly 3,000 train- ing runs across different models, datasets, window lengths,

And Number Of Input Modalities. In Particular:

∙Subsection 3.1 presents the comparison of models in the

Semg-Only Setting, Showing That Emg-Cf2D Performs

better than the other considered models across DB2, DB3 and DB7 for all window lengths. ∙Subsection 3.2 presents the performance gain obtained by adding accelerometer signals as a second input modality.

∙Subsection 3.3 evaluates EMG-CrossFormer performance on DB10, where gaze data are incorporated as a third input modality.

∙Subsection 3.4 compares the computational cost, mem- ory footprint, and inference latency of different EMG- CrossFormer variants.

For each configuration (dataset, window length, and num- ber of input modalities), the description of the results fo- cuses on the mean paired balanced accuracy difference be- tween the best-performing EMG-CrossFormer variant and the strongest competing model listed in subsubsection 2.4.2.

Complete model-level statistical analyses for unimodal and multimodal settings, together with additional performance metrics, are provided in the Supplementary Materials.

3.1. Single Modality: Semg-Only Input

Table 2 summarizes the results of the selected models in the unimodal-input setting (sEMG-only). EMG-CF2D achieves the highest mean balanced accuracy for every dataset and window length, with consistent but modest im- provements over the strongest competing model. The closest competitors are RF and MKCNN, the latter providing the 2D multi-scale backbone used in EMG-CF2D. By contrast, EMG-CF1D performs below the 2D variant, suggesting that explicit spatio-temporal feature extraction is beneficial when only sEMG signals are available.

On Db2, Emg-Cf2D Surpasses Rf By 0.79 Percentage

points (pp) at 200 ms (CI = [0.23, 1.31], 𝑝FDR = 0.003), and MKCNN by 1.44 pp at 150 ms (CI = [0.97, 1.85], 𝑝FDR < 0.001) and 1.50 pp at 100 ms (CI = [−0.16, 2.42], n.s.). On DB3, EMG-CF2D surpasses RF by 1.46 pp at 200 ms (CI = [−0.20, 3.11], n.s.) and 1.44 pp at 150 ms (CI = [−0.67, 3.35], n.s.), and MKCNN by 1.28 pp at 100 ms (CI = [−0.16, 2.42], n.s.). On DB7, EMG-CF2D surpasses RF by 1.04 pp at 200 ms (CI = [0.36, 1.69],

0.006), And Mkcnn By 1.81 Pp At 150 Ms

(CI = [1.42, 2.20], 𝑝FDR < 0.001) and by 2.09 pp at 100 ms (CI = [1.75, 2.45], 𝑝FDR < 0.001).

Table 3 Summarizes The Results Obtained When Semg

and accelerometer signals are jointly used as input. In this multimodal setting, the two EMG-CrossFormer variants achieve the highest mean balanced accuracy across all datasets and window lengths. EMG-CF2D ranks first on

Db2 And Db3, While Emg-Cf1D Achieves The Best Db7

performance at 150 ms and 200 ms. In addition, EMG- CrossFormer exhibits one of the largest improvements from the unimodal to the multimodal setting, surpassed only by NKDFF; however, NKDFF always achieves a lower absolute balanced accuracy.

On Db2, Emg-Cf2D Surpasses Nkdff By 1.77 Pp At

200 ms (CI = [1.31, 2.29], 𝑝FDR < 0.001), 2.37 pp at 150 ms (CI = [1.94, 2.79], 𝑝FDR < 0.001), and 2.88 pp at 100 ms (CI = [2.40, 3.36], 𝑝FDR <0.001). On DB3, EMG-CF2D out- performs NKDFF by 3.02 pp at 200 ms (CI = [0.56, 5.64], n.s.), 4.37 pp at 150 ms (CI = [2.31, 6.59], 𝑝FDR = 0.004), and 6.39 pp at 100 ms (CI = [4.43, 8.30], 𝑝FDR = 0.002).

On DB7, EMG-CF1D surpasses RF by 1.05 pp at 200 ms (CI = [0.28, 1.88], n.s.) and 1.61 pp at 150 ms (CI

=

[0.92, 2.38], 𝑝FDR < 0.001), whereas EMG-CF2D surpasses RF by 2.16 pp at 100 ms (CI = [1.40, 2.99] 𝑝FDR < 0.001).

Emg-Crossformer Variants On Db10 As The Number Of

input modalities increases. The transition from sEMG-only to multimodal decoding led to mean paired accuracy gains of up to +20.65 pp, with the best accuracy obtained by EMG-CF2D using sEMG plus accelerometers and a 100 ms window ((92.06 ± 3.91)%). However, adding gaze does not always further improve decoding accuracy over sEMG plus accelerometers, which supports prior evidence that gaze is more informative for predicting imminent movements than for decoding executed ones. This limited gain may also be related to the specific experimental setting adopted in the original DB10 study.

With A Window Length Of 200 Ms, Both Emg-Cf1D And

EMG-CF2D achieved their highest mean paired accuracy in the sEMG plus accelerometers setting, with (85.00 ± 7.07)% (+17.57 pp over sEMG-only) and (91.69 ± 4.38)% (+18.20 pp over sEMG-only), respectively. With a window length of 150 ms, EMG-CF1D achieved its highest mean paired accuracy in the three-modality setting, with (86.40±6.38)% (+18.54 pp over sEMG-only), whereas EMG-CF2D achieved its top performance in the sEMG plus accelerometers setting, with (91.80 ± 4.30)% (+18.85 pp over sEMG-only). With a window length of 100 ms, EMG-CF1D achieved its highest mean paired accuracy in the three-modality setting, with (86.19 ± 5.19)% (+20.43 pp over sEMG-only), whereas EMG-CF2D achieved its top performance in the sEMG plus accelerometers setting, with (92.06±3.91)% (+20.65 pp over sEMG-only).

Page 7 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers

Table 2

Balanced Accuracy (%) across models, datasets, and window lengths using only sEMG signals

Table 3

Balance Accuracy (%) across models, datasets, and window lengths, using sEMG and accelerometer signals

(+12.96)

Parenthesized values indicate the mean balanced accuracy paired difference (pp) relative to the corresponding sEMG-only setting. Table 4 further stratifies EMG-CF accuracies by healthy and amputee subject groups. Multimodal integration nar- rowed the performance gap between the two groups, most clearly for EMG-CF2D: the healthy–amputee difference de- creased from approximately 15 pp with sEMG alone to 3.8–4.8 pp with sEMG plus accelerometers across window lengths, although mean accuracy remained lower in am- putees.

Table 5 Summarizes The Computational Cost, Memory

footprint, and latency of both EMG-CF1D and EMG-CF2D under unimodal and multimodal configurations. Perfor-

Mance Metrics Were Benchmarked On A Single Nvidia

A30 GPU using CUDA 12.2 and PyTorch 2.12.0+cu126. Reported values correspond to the average profiling statistics collected over 1000 independent inference runs with a batch size of one. Results are intended as reference benchmarks, as embedded processors used in prosthetic systems generally provide significantly lower computational capabilities than the evaluation hardware.

The results demonstrate that EMG-CF1D achieves the best trade-off between predictive performance and inference latency, particularly in the multimodal setting. First, the multimodal EMG-CF1D model requires 5.8× fewer FLOPs than its 2D counterpart. This reduction is associated to the lightweight and efficient design of the 1D depthwise convolutional backbone, which scales more effectively with the number of input channels. Second, the model’s compiled version (through TorchInductor) yields an 8.98× latency reduction for the 1D variant compared to the compiled 2D- backbone configuration under identical multimodal input.

EMG-CF1D is the only model achieving sub-millisecond inference latency (0.77 ± 0.10 ms) on the tested hardware. Although EMG-CF2D achieves higher classification ac- curacy across different datasets and window lengths, this performance gap becomes smaller in the multimodal setting.

This similarity in multimodal decoding accuracy, coupled with the lower computational overhead, supports the se- lection of the EMG-CF1D variant for real-time, resource- constrained edge applications such as myoelectric prosthetic control systems.

Decoding Complex Hand Movements From Semg Signals

remains a challenging task. To achieve clinical and practical viability, deep learning architectures must be designed to handle the unique characteristics of this modality. In partic- ular, models must extract informative representations from short temporal windows to maintain low latency. These fea- tures must effectively capture spatio-temporal relationships that characterize distinct motor patterns, overcoming the low signal-to-noise ratio intrinsic to sEMG data.

Emg-Crossformer Was Developed To Address Several

limitations of conventional unimodal convolutional archi- tectures and to facilitate multimodal data integration in sEMG architectures. By leveraging a 2D multi-scale back- bone capable of modeling localized spatio-temporal dynam- ics during feature extraction, the proposed model achieved superior performance across multiple databases and tempo- ral window configurations within an sEMG-only baseline

Page 8 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers

Db10

Balanced Accuracy Across Window Lengths, Models and Modalities

Semg + Acc + Gaze

Figure 2: Hand-movement decoding performance of EMG-CrossFormer across datasets, window lengths, model variants, and input modalities. Balanced accuracy is shown for EMG-CF1D, based on a 1D depthwise convolutional backbone, and EMG-CF2D, based on a 2D multi-scale convolutional backbone. Results are stratified by window length (100 ms, 150 ms, and 200 ms) and input configuration: sEMG-only decoding is shown in blue, sEMG + ACC decoding is shown in orange, sEMG + ACC + Gaze in green. Each point represents one subject, and gray lines connect paired unimodal and multimodal results from the same subject.

Subplots correspond to NinaPro DB2, DB3, DB7, and DB10.

Table 4

Balance Accuracy (%) of EMG-CrossFormer variants on DB10, across window lengths and input modalities.

87.38±3.64

∗S stands for sEMG, A stands for Accelerometer, G stands for Gaze setup. Although performance gains are statistically signifi- cant according to FDR-corrected signed-rank tests, the mean paired accuracy gap between EMG-CrossFormer and other competitive approaches, such as RFs or MKCNN, remains modest when considering the observed variability of results.

Furthermore, sEMG-only decoding accuracy for trans-radial amputees (DB3) degrades by 20 pp or more compared to intact individuals (DB2 and DB7). These findings suggest that sEMG signals alone may be insufficient to reliably discriminate among a large number of partially correlated hand movements. Consequently, integrating additional in- put modalities may help distinguish between similar move- ments by combining complementary information related to both motor intent and the physical execution of movement.

However, the effectiveness of multimodal systems strongly depends on the design of the fusion strategy, which must enable the model to fully exploit the information contained in each modality.

Most existing multimodal deep learning approaches, such as NKDFF, combine modality-specific representations through feature concatenation before the final classification stage. While straightforward, this strategy limits the model’s ability to learn complex interactions between modalities.

In practice, concatenation may encourage the network to rely predominantly on the most informative modality while underutilizing the complementary information provided by the others. EMG-CrossFormer addresses this limitation through a dedicated fusion module based on cross-attention mechanisms. By integrating multimodal interactions before

Page 9 Of 21

Multimodal Hand Gesture Recognition Using Quey-Based Transformers

Table 5

Computational analysis of EMG-CrossFormer Variants

19.57

the transformer decoder stage, the model learns representa- tions in which each modality can condition its features on information extracted from the others. This design promotes the learning of richer cross-modal relationships that cannot be captured through simple feature aggregation. Further- more, the supervised contrastive component of the loss function described in section 2 encourages modality-specific representations to be projected into a shared embedding space while preserving class separability, facilitating more effective multimodal integration.

Thanks to this design, EMG-CrossFormer achieves one of the highest performance improvements when switching from unimodal to a multimodal setup, even if its unimodal performance in subsection 3.1 is already superior to other models. In particular, mean paired accuracy on amputees (DB3) improves by 31.43 pp when using the shortest win- dow length (100 ms). This improvements reduce the ob- served differences in decoding accuracy between intact sub- jects and trans-radial amputees. Furthermore, the accuracy gap between EMG-CrossFormer and the closest-performing model on the same dataset and window length increases from +1.27 pp in the unimodal setting (against MKCNN) to +6.37 pp (against NKDFF). Such an increase confirms the improved ability of the model to combine information from different signals, even for amputees, who are the targets for real-world prosthetic applications.

Beyond the overall benefit of multimodal fusion, the choice of the additional modality also represents an impor- tant practical factor. Among the investigated configurations, the inclusion of eye-tracking information leads to improve- ments in decoding performance for EMG-CF1D, suggesting that gaze can provide discriminative cues related to motor intention. However, this improvement comes at the cost of increased prosthetic-system complexity, since eye tracking requires dedicated hardware, calibration procedures, and in- tegration within a wearable or clinically usable setup. A sim- ilar trade-off has been reported for video-based modalities, whose integration can improve decoding accuracy but gen- erally increases computational cost due to image acquisition and processing requirements . Conversely, accelerom- eter signals provide complementary information related to the physical execution of movement with a comparatively simpler sensing configuration. Therefore, the selection of auxiliary modalities should consider not only decoding accu- racy, but also hardware complexity, computational cost, us- ability, and translational feasibility for real-world prosthetic applications.

Despite these promising results, several limitations should be acknowledged. Although EMG-CrossFormer achieves superior performance while maintaining reduced FLOPs, memory footprint, and inference latency (see Table 5), it re- mains a black box model. In rehabilitation and other biomed- ical applications, interpretability is essential for improving model reliability, understanding failure scenarios, and in- creasing robustness to out-of-distribution samples. The 1D depthwise convolutional encoder adopted in EMG-CF1D extracts channel-specific features characterizing local por- tions of the input signal. Still, the physiological meaning of these learned representations, as well as the temporal patterns that activate them, remains unclear. Future work should investigate architectures and analysis methods that improve the interpretability of unimodal representations.

Interpretable modality-specific features would also facil- itate the analysis of multimodal interactions by enabling inspection of cross-attention matrices and quantification of attention flow through transformer layers using explainable artificial intelligence (XAI) techniques such as attention rollout .

A Second Limitation Concerns Benchmarking And Re-

producibility. Although the results presented in section 3 demonstrate that EMG-CrossFormer consistently ranks first among the top-performing models across all investigated datasets and configurations, comparisons between studies remain challenging. As anticipated in section 1, differences in preprocessing pipelines, train-test splitting strategies, hy- perparameter optimization procedures, and evaluation proto- cols can substantially influence reported performance. Fur- thermore, the lack of implementation details and open repos- itories identified in many studies prevents the inclusion of other models in the presented benchmarking. As the number of deep learning studies for hand gesture decoding contin- ues to grow, there is an increasing need for standardized benchmarking frameworks that facilitate fair comparisons and fully reproducible evaluations. The source code made openly available in this study enables an easy integration of different models in the same experimental setting described in section 2. The community is therefore encouraged to add more models and support the design of an open benchmark- ing library.

Finally, the experiments were conducted primarily on the NinaPro database, which represents one of the largest and most widely adopted resource for hand movement decoding.

Nevertheless, both the number of subjects and the diversity of modalities remain limited for large multimodal deep learning applications. The acquisition and public release of larger multimodal datasets could accelerate progress in

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.