Motor Imagery Bci Classification

Michael Tangermann1,2[0000−0001−6729−0290]

1 Donders Institute for Brain, Cognition and Behaviour, Abstract. We present a simple deep learning-based framework com- monly used in computer vision and demonstrate its effectiveness for cross-dataset transfer learning in mental imagery decoding tasks that are common in the field of Brain-Computer Interfaces (BCI). We inves- tigate, on a large selection of 12 motor-imagery datasets, which ones are well suited for transfer, both as donors and as receivers.

Challenges. Deep learning models typically require long training times and are data-hungry, which impedes their use for BCI systems that have to minimize the recording time for (training) examples and are subject to constraints induced by experiments involving human subjects. A solution to both issues is transfer learning, but it comes with its own challenge, i.e., substantial data distribution shifts between datasets, subjects and even between subsequent sessions of the same subject.

Approach. For every pair of pre-training (donor) and test (receiver) dataset, we first train a model on the donor before training merely an additional new linear classification layer based on a few receiver trials. Performance of this transfer approach is then tested on other trials of the receiver dataset.

Significance. First, we lower the threshold to use transfer learning be- tween motor imagery datasets: the overall framework is extremely simple and nevertheless obtains decent classification scores. Second, we demon- strate that deep learning models are a good option for motor imagery cross-dataset transfer both for the reasons outlined in the first point and because the framework presented is viable in online scenarios. Finally, analysing which datasets are best suited for transfer learning can be used as a reference for future researchers to determine which to use for pre-training or benchmarking.

Keywords: EEG · BCI · Motor Imagery · Deep Learning · Transfer

Learning · Cross-Dataset

⋆Supported by the Donders Center for Cognition (DCC).

Introduction

Electroencephalography (EEG) offers a non-invasive modality for capturing the electrical activity of the brain. Utilizing machine learning techniques, specific brain activities can be decoded in single trial. For instance, event-related poten- tials (ERPs) signify the brain’s response to the occurrence of an awaited and attended event . Likewise, the sensorimotor cortices, being relatively large areas of the cortical surface, allow for the decoding of motor execution and mo- tor imagery activities. The decoded brain activities can be mapped to computer commands. In so-called Brain-Computer Interfaces (BCIs) , the commands are used to drive applications that are directly controlled by the brain.

In a typical setting, each BCI session starts with an initial phase where the user is prompted to produce predefined outputs. This is typically called the calibration phase. The acquired data enables the supervised training of a decoding model. Only afterwards, the user can interact freely with the BCI application. This productive phase is typically called the online mode. However, calibration can be a tedious and uninteresting process for a user, and may require up to an hour of attention-demanding activities, even though the interesting phase we aim to start with as quickly as possible is the online phase.

Specifically for BCIs that make use of motor imagery tasks, the tedious cal- ibration process prompts the consideration of Transfer Learning (TL) methods. TL leverages information from various sources, such as earlier sessions of the same user, of other users, or datasets obtained under slightly different exper- imental conditions, to reduce the need for calibration data samples collected during the ongoing session. In the most rudimentary approach to TL, a model is trained on a dataset we will call donor, and tested on another one we will call receiver. The necessity for TL becomes even more pronounced in the context of deep learning, given its inherent requirements for large training datasets and a substantial training time which would further delay the online phase . How- ever, TL presents its own challenges in the form of distributional shifts across different datasets, subjects, or even sessions. They can be attributed to factors like variations in experimental protocol, EEG recording systems, brain morphol- ogy, level of fatigue, medication intake, and more . While TL has been widely studied in BCI, the focus has been mostly on cross-session transfer , cross- subject transfer , or both . However, cross-dataset transfer learning has been a relatively underexplored domain. The recent BEETL competition has begun to fill this void by drawing attention to the challenges and oppor- tunities cross-dataset transfer represents for BCI . Notably, the top three winning methods in this competition were based on deep learning, thus indicat- ing promising potential for such models. Furthermore, it was recently suggested to use cross-dataset transfer to evaluate the structure of cognitive tasks .

In this study, we extend a TL method already standard in the field of com- puter vision, the ImageNet linear evaluation protocol , to BCIs. Following our previous work which showcased its efficacy for cross-subject transfer , here we focus on its applicability for cross-dataset transfer. We experiment with one- Transfer Learning between Motor Imagery datasets using Deep Learning

3

to-one transfer across twelve motor imagery datasets, leveraging the MOABB library for both, dataset acquisition and normalized evaluation. In this article, we will examine several pertinent questions: How challeng- ing is actually TL between motor imagery datasets? Is deep learning genuinely a good option for TL in BCI? If so, what constitutes the minimal configura- tion necessary? Which datasets serve as the most effective donors? And which datasets are the best receivers? Our contributions can be summarized as follows: 1. We present a TL framework that can deal with a minimum of data from the receiver dataset and still reach good performance; 2. We lower the threshold to use TL between motor imagery datasets consid-

Ering The Extreme Simplicity Of This Framework;

3. We provide evidence that deep learning models and our TL framework are apt choices for cross-dataset transfer, further substantiated by the viability

Of Our Framework For Online Scenarios;

4. We characterize the motor imagery datasets currently available in MOABB; 5. We present an analytical investigation into the suitability of these datasets for TL, serving as a valuable resource for future research in pre-training or benchmarking models.

Supplementary materials, including source code, pre-trained models, and comprehensive results, are available online34.

Neural Network Architecture: Eegnet

We used EEGNet as neural network architecture to conduct our experiments. It consists of an initial temporal convolutional layer that captures temporal cor- relations in EEG signals. This is followed by a depthwise spatial convolution layer that accounts for spatial correlations across electrodes. The network con- cludes with a separable convolution for dimensionality reduction before reaching the linear classification layer.

Our choice of EEGNet as the neural network architecture is motivated by three primary factors. First, its versatility is demonstrated through proficient performance across multiple BCI paradigms . Second, its simplicity and com- putational efficiency make it particularly lightweight. Third, EEGNet has earned widespread community acceptance for EEG decoding, as evidenced by several studies .

Data: Motor Imagery

We conducted our experiments on EEG-based motor imagery (MI) datasets. They are constituted of EEG recordings capturing subjects as they either imag- ine or execute specific motor tasks. These motor tasks can range from imagined 3 Source code: https://gitlab.com/PierreGtch/motor embedding benchmark 4 Pre-trained models, and results: https://huggingface.co/PierreGtch/EEGNetv4

P. Guetschel, M. Tangermann

actions such as squeezing a fist to wiggling toes. In each dataset, multiple mo- tor imagery tasks are represented and the classification problem is to determine which task was imagined/executed in each example, which typically consists of a few seconds of EEG recordings called a trial.

Table 1. Summary of Datasets Evaluated in the Study.

Alexmi

f, r, rh.

Bnci2014001

f, lh, rh, t.

Bnci2014004

lh, rh.

Bnci2015001

f, rh.

Bnci2015004

f, n, rh, s, wa.

Cho2017

lh, rh.

Lee2019 Mi

lh, rh.

R, Ree, Ref, Rhc,

rho, rp, rs.

Physionetmi

bh, f, lh, r, rh.

23

Schirrmeister2017 f, lh, r, rh.

Bh, F, Lh, Lhrf,

r, rh, rhlf.

Zhou2016

f, lh, rh.

50

Description of Datasets In this study, we included all the Motor Imagery datasets available with the MOABB library . However, three datasets were excluded due to technical limitations such as missing electrode names and impos- sibility of downloading the data. The complete list of datasets that were included along with their respective descriptions can be found in Table 1. The column tained for each combination of subject, session and class. The imagined tasks which form the class label information in each dataset are described using the

– Wa: Word Association

Transfer Learning between Motor Imagery datasets using Deep Learning

5

It is noteworthy that the datasets included in this study employ diverse imagination strategies. For example, the Lee2019 dataset required subjects to imagine grasping with the appropriate hand. In contrast, the Cho2017 dataset instructed subjects to imagine moving their fingers from the index to the lit- tle finger. The Schirrmeister2017 dataset contains executed instead of imagined movements, where subjects engaged in sequential finger-tapping. Of particular note is the Ofner2017 dataset, which includes single instances of both executed and imagined movements, diverging from other datasets that sustain or repeat the imagination tasks over several seconds. Some authors did not report the ex- plicit instructions they gave on the imagination strategies and might have left it to the discretion of the subjects.

Pre-processing The pre-processing protocol we used for all the datasets in- volved setting the trial windows to 0 to 3 seconds post-cue. We strongly limited the EEG channels used to channels C3, C4, and Cz, which are placed over the sensorimotor cortices. The data was resampled at a frequency of 128Hz, aligning with EEGNet’s default setting. A bandpass filter with the range 0.5 to 40 Hz was applied to the data.

Method

In the current study, we employ the transfer learning framework previously pro- posed in our work . However, a salient distinction is that, unlike the prior study which focused solely on cross-subject transfer, the present study extends this to cross-dataset transfer. A schematic representation of this methodology is encapsulated in Figure 1. This transfer framework has two phases: a pre-training phase using the donor dataset and a fine-tuning phase using the receiver dataset.

Pre-training Phase During the pre-training phase, a neural network architec- ture is trained from scratch on a set of source datasets, or donors. The aim is to categorize the motor task being imagined in each example/trial. To maintain a simple framework, the hyperparameters chosen are consistent with those recom- mended by the EEGNet authors for cross-subject training . Upon completion of this phase, the network’s classification head is discarded, and the rest of the architecture is frozen, i.e., the remaining parameters will stay unchanged in the following phase.

Fine-tuning Phase In the subsequent fine-tuning phase, the pre-trained net- work is utilized as a feature extractor, wherein its parameters are frozen. A following linear classification layer is then trained from scratch to categorize the embeddings produced by the feature extractor. This classification layer is trained and tested using examples from the same session, i.e. the receiver dataset, adopt- ing a within-session cross-validation (CV) approach through multiple random

Transfer

Fig. 1. Schematic representation of the transfer learning framework we introduced in and have now extended to cross-dataset transfer. The upper block describes the pre-training phase. The lower block describes the fine-tuning phase.

shuffles of the examples. Importantly, during the fine-tuning phase, only the lin- ear classification layer is subject to training, while the rest of the architecture remains static. The examples used to train the linear classification layer will be called calibration examples. The dataset employed for this fine-tuning is inter- changeably termed as the receiver dataset, test dataset or calibration dataset, given that both fine-tuning and testing are executed within the same session of a single dataset, even though using separate examples.

Experimental Settings

Pre-training Phase In this phase, a distinct EEGNet architecture was trained for each dataset, utilizing the entire dataset and all available classes therein. Con- sequently, a total of 12 pre-trained neural network architectures were generated.

Fine-tuning Phase During fine-tuning, we systematically examined varying numbers of calibration examples—specifically 1, 2, 4, 8, 16, 32 (where applica- ble), and 64 (where applicable) per class—across each session for every subject in the test datasets. Three distinct analyses were conducted: left hand vs. right hand (lh-rh), right hand vs. feet (rh-f ), and an all-classes classification. The Transfer Learning between Motor Imagery datasets using Deep Learning

7

classes specified for each analysis, are the only ones from the receiver used dur- ing the fine-tuning phase. The ROC-AUC (area under the receiver operating characteristic curve) metric was adopted for lh-rh and rh-f classification scores, while accuracy metrics were used for the all-classes scenario. The first two binary classification analyses allow for a fair comparison between datasets. Whereas the classification with all classes is a fallback case in which the chance level varies with the number of classes. We specifically focussed our analysis on the lh-rh and rh-f class pairs, as they are the most ubiquitously represented across the various datasets. The number of folds for within-session CV is set to 16. While this may appear excessive, it is crucial to mitigate the significant impact that the selection of just one or two calibration examples can have on the classification performance metrics.

Interpretation Of Results

Data Volume and Analysis The three analyses culminated in an extensive set of 871,488 scores, derived from the different CV folds, number of calibration examples, sessions, subjects, test datasets, and pre-trained models.

Tabulated Scores For ease of interpretation, scores were averaged across CV folds, sessions, and subjects. This resulted in three summary tables correspond- ing to each of the three analytical scenarios. Each table contains an average score for every unique combination of pre-training dataset, test dataset, and number of calibration examples. Due to editorial limitations, the main text includes only black-and-white tables restricted to the case with 16 calibration examples per class. 16 calibration examples per class represents the maximum number that allows for the inclusion of all test datasets in the analysis. However, the complete color-coded tables are available in the supplementary materials, the location of which was indicated in the introductory section 1. Results of the rh-f analysis with 16 calibration examples per class are in Table 2(a), those related to the lh-rh one are in Table 2(b), and those covering the all-classes one can be located in Table 2(c).

Graphical Representations Line plots showing test scores with respect to the number of calibration example per class were generated from tabulated re- sults, averaged either across test datasets in Figure 2 or pre-training datasets in Figure 3. Notably, in the averaged line plots, the datasets, even with disparate subject counts, hold equal weight.

Specific Observations and Cautions The crenellated blue line on the right- hand y-axis of Figure 2, denotes the number of test datasets included in each average. Its fluctuation is attributed to the variable number of examples across test datasets which does not allow to include all the datasets when using 32 or 64 calibration examples. Consequently, the average curves’ progressions should be

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.