Respiratory Rate Sensor System

Graphical Abstract

A Skin-Tone-Aware Dual-Representation Remote Photoplethysmography Framework for Contact-

Less Respiratory Rate Estimation

Trishna Saikia, Anup Kumar Gupta, Puneet Gupta, Pasi Liljeberg

Highlights

A Skin-Tone-Aware Dual-Representation Remote Photoplethysmography Framework for Contact-

Less Respiratory Rate Estimation

Trishna Saikia, Anup Kumar Gupta, Puneet Gupta, Pasi Liljeberg • Dual-representation rPPG for contactless respiratory rate estimation. • Skin-tone-aware rPPG extraction improves respiratory signal quality.

• RR-rPPG: An Indian demographic dataset for respiratory rate research.

A Skin-Tone-Aware Dual-Representation Remote

Photoplethysmography Framework for Contactless Respiratory Rate

Estimation⋆

Trishna Saikiaa,∗, Anup Kumar Guptaa, Puneet Guptaa and Pasi Liljebergb aIndian Institute of Technology (IIT) Indore, 452020, India

A B S T R A C T

Background and Objective: Respiratory rate is a vital indicator of pulmonary and cardiovascular health, yet conventional methods for estimating respiratory rate are often intrusive and unsuitable due to their contact-based nature. To this end, remote photoplethysmography offers a promising non-contact alternative and has been widely used for heart rate estimation; however, its potential for respiratory rate estimation remains underexplored. Existing methods typically adapt conventional re- mote photoplethysmography techniques, such as green and chrominance-based projections, originally designed for heart rate estimation, which only partially capture respiratory dynamics. Most prior work focuses on the Eulerian representation with fixed or empirically selected RGB projections, lacking adaptive mechanisms to extract maximal physiological information.

Methods: To address these gaps, we propose a skin-tone-aware dynamic RGB signal projection that optimally captures respiratory information. To mitigate the sensitivity of the Lagrangian representa- tion to non-respiratory motion, we introduce a dedicated denoising network for motion-based remote photoplethysmography signals. Furthermore, while some methods have initiated exploration of dual- representation fusion, they lack effective training strategies to transfer complementary information across representations. We address this limitation by designing a phase-independent contrastive loss that enables both representations to collaboratively learn relevant respiratory rate information.

We also introduce RR-rPPG, a new respiratory-rate facial video dataset with Indian demographic representation, addressing the limited diversity of currently available resources. Results: We evaluate the proposed method on both the RR-rPPG dataset and the publicly available COHFACE dataset, where it consistently outperforms the comparison methods and achieves up to a 42.1% reduction in mean absolute error across the evaluated settings.

Conclusion: The proposed framework demonstrates the effectiveness of jointly leveraging skin-tone- aware Eulerian and denoised Lagrangian representations for contactless respiratory rate estimation from facial videos. In addition, the RR-rPPG dataset contributes a diverse benchmark resource for future research in remote respiratory monitoring. The code and dataset will be made publicly available upon paper acceptance.

1. Introduction

Respiratory rate (RR) is a vital physiological parameter that provides early insight into a person’s pulmonary and cardiovascular health. Abnormal RR patterns are often early indicators of critical conditions such as respiratory distress, cardiac arrest, and metabolic imbalances . Conventional RR estimation methods, such as chest belts, spirometry, and capnography, require physical contact with the body, making them less suitable for continuous, unobtrusive, or remote health monitoring . These constraints have driven signif- icant interest in video-based physiological signal analysis methods, such as remote photoplethysmography (rPPG), which leverage regular RGB cameras to extract physiolog- ical signals [3, 4].

⋆This work of Trishna Saikia is supported by the Prime Minister’s Research Fellowship (PMRF), the Ministry of Education, Government of India (2102743).

Orcid(S):

The rPPG method captures subtle skin color variations caused by blood volume changes in superficial facial ves- sels [5, 6]. These variations appear as temporal changes in red, green, and blue (RGB) pixel intensities across video frames . By processing these RGB signals, a composite physiological signal, referred to as the rPPG signal, can be derived . While rPPG is primarily used for cardiac signal extraction, it can also reveal respiratory patterns due to respi- ratory sinus arrhythmia (RSA), a physiological phenomenon where heart rate (HR) varies in sync with the breathing cycle . However, extracting clean respiratory signals from rPPG in real-world scenarios remains challenging due to factors such as skin tone diversity and other appearance-related variations.

In the context of rPPG-based RR estimation, two com- plementary visual modalities are employed: Eulerian and Lagrangian, which correspond to color-based and motion- based methods, respectively . Eulerian-based methods extract rPPG signals by analyzing subtle temporal variations in skin color intensity, typically in exposed facial regions . In contrast, Lagrangian-based methods estimate RR by tracking periodic respiratory-induced displacements in

Elite-Rr

facial regions, such as those on the nostrils or cheeks . Notably, Lagrangian-based methods do not require direct visibility of skin tissue and can remain effective even when portions of the face are occluded . However, they are more susceptible to confounding artifacts from facial expres- sions, speech, or head movements.

While Eulerian-based methods are comparatively robust to such motion artifacts, their performance is heavily influ- enced by skin tone diversity. This highlights the need for a dynamic, skin-aware projection algorithm that can adap- tively extract high-fidelity rPPG signals from RGB signals based on skin characteristics. Additionally, the incorpora- tion of a dedicated denoising network can further refine Lagrangian signals by suppressing non-respiratory noise. Fi- nally, integrating both Eulerian and Lagrangian representa- tions through a unified dual-representation framework holds the potential to improve the robustness of RR estimation across varied subjects.

Despite the complementary nature of Eulerian and La- grangian methods, most rPPG-based RR estimation methods are predominantly built upon Eulerian methods such as methods in [8, 9, 12]. To this end, many of these methods utilize fixed empirical projection algorithms like CHROM to extract respiration-induced rPPG signals from the RGB channels. However, such static methods overlook in- dividual variations in skin tone, thereby limiting their gener- alizability and physiological fidelity. Moreover, the majority of existing methods treat Eulerian and Lagrangian signals independently [8, 14], failing to leverage their complemen- tary strengths. Although some recent efforts have explored dual-representation integration for robust RR estimation , they generally lack an adaptive fusion mechanism that dynamically adjusts based on skin characteristics and signal quality. Furthermore, none of these methods explicitly facil- itates cross-modal knowledge sharing to refine one represen- tation using insights from the other. In response, we propose a unified, skin-aware method that not only combines both representations but also employs a novel phase-independent contrastive learning strategy. This enables alignment in the frequency domain across representation, despite temporal mismatches, thereby supporting joint learning and robust RR estimation.

Another limiting factor in advancing rPPG-based RR es- timation is the scarcity of diverse, publicly available datasets specifically curated for this task. Existing datasets are either limited in demographic diversity or not specifically curated for RR estimation task. Notably, there is an absence of datasets representing Indian skin tones and facial features, which restricts the applicability and generalizability of cur- rent methods across different population groups.

To address these limitations, we introduce RR-rPPG, a novel dataset comprising facial videos of subjects from diverse Indian demographics, accompanied by synchronized respiration and pulse signals. In addition to this, we propose ELITE-RR (Eulerian and Lagrangian Integration with skin- Tone awarEness for RR estimation), a dual-representation rPPG-based RR estimation method that adaptively combines Eulerian and Lagrangian methods through a quality-guided skin-aware fusion strategy. Our method is designed to en- hance robustness across varying skin tones and improve physiological signal fidelity. Its key contributions are as

Follows:

1. We propose a novel, data-driven method that adap- tively transforms RGB pixel intensities into a physi- ological rPPG signal using skin tone-aware features.

Unlike conventional methods that apply static or em- pirically derived projections, our method learns dy-

Shallow Neural Network Conditioned On Handcrafted

skin-tone-aware features. This enables personalized and robust signal extraction across varying skin tones. 2. To harness the complementary information of Eu-

Lerian And Lagrangian Representations, We Propose

a contrastive loss that aligns signals based on their frequency content rather than phase. This design ac- counts for phase misalignment across representations and allows our networks to learn robust, modality- agnostic representations focused on respiratory fre- quency content.

3. We present the first rPPG dataset designed for RR estimation that includes Indian demographic diversity. This dataset will enable the development and bench- marking of more inclusive and generalizable methods in the field.

2.1. Remote Photoplethysmography (Rppg)

The rPPG is a non-contact method that estimates phys- iological parameters like HR and RR by analyzing subtle changes in skin color captured through RGB cameras .

These color variations arise from blood volume fluctuations beneath the skin, which affect the amount of reflected light and appear as temporal changes in the RGB channels of facial videos . In the rPPG domain, two complementary representations are commonly used: the Eulerian represen- tation, which analyzes pixel intensity changes at fixed loca- tions to detect color-based signals , and the Lagrangian representation, which tracks the motion of discriminative fa- cial landmarks over time to capture movement-based signals . Respiration affects both representations: it influences HR through RSA, which alters the color-based signals, and it causes subtle head and facial movements that are captured in motion-based signals .

Representation:

To extract respiratory signals using the Eulerian repre- sentation, earlier methods have employed signal decompo- sition techniques such as Independent Component Analysis (ICA) and Principal Component Analysis (PCA), or signal amplification approaches like Eulerian Video Magnification (EVM) . In recent literature, methods such as and primarily adopt the CHROM method to obtain the Eulerian

Elite-Rr

rPPG signal. These methods typically rely on predefined rules or empirically derived projections to combine RGB channels. While such strategies perform well in controlled environments, they often fail to generalize across individuals with diverse skin tones. This makes the Eulerian modality particularly sensitive to appearance-related variations.

Representation:

Existing methods employing the Lagrangian represen- tation estimate RR by extracting rPPG signals through the tracking of subtle facial motions, particularly around regions like the nostrils or cheeks . This modality has proven effective, especially in scenarios where skin visibility is limited, and has been shown to outperform Eulerian-based methods under such conditions [11, 14]. However, these motion signals are highly vulnerable to contamination from non-respiratory movements like head shifts or facial expres- sions. Despite this challenge, prior works have not explicitly addressed the denoising of such signals. To overcome these limitations, we design a dedicated denoising network aimed at refining the Lagrangian rPPG signals for robust RR esti- mation.

2.4. Dual-Representation Rr Estimation:

Recent research has focused on dual-representation methods that fuse both Eulerian and Lagrangian represen- tations to overcome the limitations of using either represen- tation alone. This integration leverages both of their com- plementary strengths. A few existing methods estimate RR independently from each representation and then rely on a machine learning model to select the more reliable one .

However, such methods do not enhance the performance of the complementary representation. They treat the two signal representations independently, lacking a mechanism for mutual refinement or knowledge transfer. To address these limitations, we design a unified dual-representation method that performs three coordinated tasks: (1) a projec- tion network that dynamically fuses RGB signals based on skin tone variations, (2) a denoising network that enhances the reliability of Lagrangian signals, and (3) a novel training strategy using a phase-independent contrastive loss that en- ables the model to align spectral characteristics of signals across representations, despite possible phase differences, thereby facilitating effective knowledge transfer and joint learning for robust RR estimation. Moreover, the absence of demographically diverse datasets, particularly those rep- resenting Indian skin tones, restricts the generalizability of existing methods. In response, we introduce a new dataset specifically curated for this purpose.

3. Proposed Method

We propose a dual-representation method, ELITE-RR, for RR estimation from facial videos by integrating both Eulerian and Lagrangian representations (refer to Figure 1).

The input video is divided into fixed-length non-overlapping clips, and for each clip, facial landmarks are detected to extract two regions of interest (ROIs): the forehead and cheeks (including nostrils). In the Eulerian branch, each ROI is divided into equal-sized patches. For every patch, we extract temporal RGB signals and compute a feature vector that encodes skin tone characteristics. A shallow neural network then predicts a set of dynamic projection weights to combine RGB channels into a single rPPG signal. The patch with the highest signal-to-noise ratio (SNR) is selected, and RR is estimated via spectral analysis. In the Lagrangian branch, vertical motions within the ROIs are tracked to cap- ture respiration-induced movements. The trajectory with the best SNR is further denoised using a CNN-based network, and RR is computed from its frequency spectrum. Finally, RR estimates from both representations are fused using an SNR-based quality-weighted strategy, yielding a robust and enhanced RR estimation.

3.1. Roi Extraction

We extract facial ROIs relevant for rPPG signal estima- tion from each video clip. At first, we utilize the MediaPipe Framework to detect the face and its corresponding landmarks in each frame. These landmarks enable precise localization of facial sub-regions that are physiologically informative and less susceptible to expression-induced ar- tifacts. We intentionally exclude areas around the eyes and mouth to avoid the influence of facial expressions, which can introduce noise into the rPPG signal. Instead, we focus on two primary regions that have been shown to contain strong respiratory-related signals: the forehead and cheeks (including the nostrils). The forehead ROI is defined by applying a minimum enclosing rectangle around the de- tected forehead landmarks as suggested in . Similarly, the cheek and nostril regions are extracted based on their respective landmarks. To further enhance signal quality and account for inconsistencies in rPPG information across dif- ferent facial regions, each ROI is subdivided into smaller, non-overlapping square patches, as described in . This division not only facilitates localized rPPG signal extraction but also increases the robustness of signal processing by allowing the selection of high-quality patches during infer- ence.

3.2. Rr Estimation From Eulerian Representation

After extracting the facial ROIs and dividing them into smaller patches, we compute temporal signals of the red, green, and blue channels, denoted as 𝐬𝑟(𝑡), 𝐬𝑔(𝑡), and 𝐬𝑏(𝑡), respectively. Each of these signals is obtained by averaging the corresponding color channel pixel values within the patch across video frames. To suppress noise and retain only respiration-relevant components, each signal is passed through a 4th-order Butterworth band-pass filter with a cutoff frequency of [0.08, 0.5] Hz, a range corresponding to normal respiratory frequencies . Finally, to project these filtered signals into a single physiological rPPG signal that maximally captures respiratory activity, we employ a shal- low neural network, 𝑝that generates a dynamic projection vector 𝐩𝑗= [𝛼, 𝛽, 𝛾] based on the patch-specific skin tone feature vector, 𝐟𝑠. Details of the 𝑝architecture are provided

Elite-Rr

Figure 1: Overview of the proposed method, ELITE-RR. It combines two complementary representations: Eulerian and Lagrangian. In the Eulerian branch (top), the facial ROI is divided into patches. Each patch’s skin-tone-aware features guide a projection network (𝑝) to transform RGB signals into reliable physiological signals. In the Lagrangian branch (bottom), vertical displacements of facial landmarks are tracked to extract rPPG signals, which are refined using a denoising network (𝑑). A phase-independent contrastive loss (𝑐) facilitates knowledge transfer between representations. Finally, RR is estimated through a quality-weighted fusion of both representations.

in Section 3.7, while the construction of the skin-tone feature vector 𝐟𝑠is described in the next section.

3.3. Skin-Tone-Aware Feature Vector

To adaptively extract high-quality rPPG signals from diverse facial regions, it is essential to account for variations in skin tone. For this purpose, we construct a 12-dimensional feature vector 𝐟𝑠∈ℝ12 for each patch, summarizing its color characteristics and temporal stability. This vector is used as input to a shallow neural network 𝑝(detailed in Section 3.7) to derive an optimal projection for rPPG signal extraction. The components of 𝐟𝑠are organized into four groups, each capturing a distinct visual property:

3.3.1. Mean Channel Intensities

These values reflect the average color composition of the patch and provide a baseline estimate of the subject’s skin tone. For that, we calculate the mean RGB pixel intensities from the first frame of the video clip within the selected

𝜇𝐵= Mean(𝐵1)

Here, 𝑅1, 𝐺1, and 𝐵1 denote the red, green, and blue pixel values of the selected patch in the first frame, respectively.

3.3.2. Temporal Standard Deviations

These features measure the temporal fluctuations in av- erage channel intensities, capturing how stable the patch’s color is over time. Thereby reflecting appearance variability

)

Here, 𝑠𝑡𝑑(⋅) refers to the standard deviation operation.

3.3.3. Chromatic Ratios

Further, we compute inter-channel mean intensity ratios to capture the color balance of the patch as follows:

𝜇𝐵+ 𝜀

These ratios capture color dominance, aiding in distinguish- ing skin tones variations.

3.3.4. Normalized Rgb Vector

To isolate pure color composition from brightness, we normalize the mean RGB values of a patch to form a unit

]

This normalization reduces the influence of overall intensity changes while preserving the relative color composition of the patch.

Final Feature Vector: All computed features are concate- nated to form the final skin-tone-aware feature vector 𝐟𝑠,

Ratio𝑅𝐺, Ratio𝑅𝐵, Ratio𝐺𝐵,̂ 𝜇𝑅,̂ 𝜇𝐺,̂ 𝜇𝐵] ∈ℝ12

The resulting feature vector provides a comprehensive sum- mary of the patch’s skin tone properties. It is then fed into the projection network 𝑝, which outputs a dynamic projection vector 𝐩𝑗= [𝛼, 𝛽, 𝛾] used to compute the patch-level rPPG signal, 𝐬𝑒for the Eulerian modality: 𝐬𝑒= 𝛼⋅𝐬𝑟(𝑡)+𝛽⋅𝐬𝑔(𝑡)+ 𝛾⋅𝐬𝑏(𝑡). This process is repeated for all patches. In training, each patch is treated as a separate sample. At inference, we compute SNR for each patch’s signal (following ) and select the one with highest SNR. Final RR is estimated using method in Section 3.5.

Representation

After extracting the facial ROIs, we apply the Lucas- Kanade tracker to identify and track a set of discrim- inatory feature points across frames. For each point, we obtain a temporal trajectory that records its motion in both horizontal and vertical directions. Since vertical motion has been shown to be more sensitive to respiration-induced variations , we retain only the 𝑦-axis component of each trajectory. Let 𝑛denote the total number of frames and 𝑝the number of tracked points. The Lagrangian rPPG

𝑛] Denotes The Vertical Motion Trajectory

of the 𝑖th point across 𝑛frames. Although these extracted signals reflect periodic respiratory-induced movements, they may still be contaminated by noise due to non-respiratory facial motion or tracking inaccuracies. To suppress irrelevant frequency components and retain only respiration-related variations, each signal is first passed through a band-pass filter with a frequency range of [0.08, 0.5] Hz, consistent with the typical respiratory frequency range . Following filtering, we evaluate the signal quality using the SNR met- ric and select the best-quality signal. This selected signal is then refined using a dedicated denoising network, 𝑁𝑑, which mitigates residual artifacts and enhances respiratory features. Finally, the denoised signal is subjected to spectral analysis to estimate the RR, following the procedure outlined in Section 3.5.

3.5. Representation-Wise Rr Estimation

Once the rPPG signals are obtained from each represen- tation, RR is estimated by analyzing their frequency spectra using the Fast Fourier Transform (FFT), as outlined in .

This is based on the observation that respiratory-induced oscillations in rPPG signals typically occur within a specific frequency range, while other components are attributed to unrelated physiological signals. The frequency correspond- ing to the highest spectral peak is assumed to represent the respiratory rate. Let 𝐬𝐟denote the FFT spectrum of the rPPG signal. The respiratory rate 𝑟𝑟(in breathing rate per minute, BrPM) is calculated as: 𝑟𝑟= 𝑚(𝐬𝐟) × 60. Here, 𝑚(𝐬𝐟) is the peak frequency (Hz); multiplying by 60 converts it to BrPM.

3.6. Dual-Representation Rr Estimation

While both Eulerian and Lagrangian representations in- dependently provide valuable insights into respiratory activ- ity, each has its own limitations. The Eulerian representation is sensitive to skin color variations, whereas the Lagrangian representation is prone to motion artifacts and tracking noise. To mitigate these limitations and improve robustness, we performed a fusion of the RR estimates from both representations. Specifically, we adopt a quality-weighted fusion strategy based on the SNR of the respective rPPG signals. Let 𝑟𝑟𝑒and 𝑟𝑟𝑙denote the respiratory rate estimates from the Eulerian and Lagrangian representation, and let 𝑞𝑒 and 𝑞𝑙represent their corresponding signal qualities. The final fused RR estimate, 𝑟𝑟𝑓, is computed as a weighted

. This Formulation Ensures That

the representation providing a cleaner signal (higher SNR) contributes more significantly to the final estimation.

3.7. Network Architecture Details

This section describes the architectural and training details of two neural networks used in ELITE-RR: the skin-aware projection network (𝑝) and the denoising net- work (𝑑). To enable effective learning across representa- tion despite temporal misalignment, we introduce a phase- independent contrastive spectral loss that emphasizes fre- quency consistency over phase coherence, facilitating cross- modal training.

𝐩: The 𝑝takes as input a 12-dimensional skin tone- aware feature vector 𝐟𝑠(as described in Section 3.3), and produces a projection vector 𝐩𝑗= [𝛼, 𝛽, 𝛾] that linearly combines the RGB temporal signals into a single physio- logical rPPG signal, 𝐬𝑒. The architecture of 𝑝is a shallow

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.