Biofeedback Device Design

Clara Moge, Katherine Wang, and Youngjun Cho. 2022. Shared User Interfaces of Physiological Data: Systematic Review of Social Biofeedback Systems and Contexts in HCI. In CHI Conference on Human Factors in Computing Systems (CHI ’22), ACM,

Https://Doi.Org/10.1145/3491102.3517495

Shared User Interfaces of Physiological Data: Systematic Review of Social

Biofeedback Systems And Contexts In Hci

Clara Moge1, Katherine Wang1, and Youngjun Cho1,2*

Abstract

As an emerging interaction paradigm, physiological computing is increasingly being used to both measure and feed back information about our internal psychophysiological states. While most applications of physiological computing are designed for individual use, recent research has explored how biofeedback can be socially shared between multiple users to augment human-human communication. Reflecting on the empirical progress in this area of study, this paper presents a systematic review of 64 studies to characterize the interaction contexts and effects of social biofeedback systems. Our findings highlight the importance of physio-temporal and social contextual factors surrounding physiological data sharing as well as how it can promote social-emotional competences on three different levels: intrapersonal, interpersonal, and task-focused. We also present the Social Biofeedback Interactions framework to articulate the current physiological-social interaction space. We use this to frame our discussion of the implications and ethical considerations for future research and design of social biofeedback interfaces.

CCS CONCEPTS • Human-centered computing → Human computer interaction (HCI): Interaction techniques; Interactive systems and tools; Interaction paradigms; Interaction devices; HCI theory, concepts and models. Additional Keywords and Phrases: Physiological computing; social biofeedback; physiological data sharing; computer-mediated communication; emotion communication; systematic review.

1 Introduction

Designing computing systems able to sense, analyze, and respond to emotional states is essential in bringing human-computer interaction closer to human-human communication . These physiological and affective computing technologies (also called emotional artificial intelligence) rely upon input data from .,physiological signals such as blood volume pulse, heart rate variability, temperature and respiration [18, 21, 22, 122, 139], as well as behavioral cues (e.g. facial expression, gesture, and tone [103, 134]). Through the process of psychophysiological inference (i.e., the mapping of phenomenologically distinct psychological states to patterns of physiological activity ), these inputs are analyzed to interpret cognitive, emotional and motivational states.

This allows the computing agent to modulate its response as a function of user state to drive task efficiency and improve user experiences . On the other hand, instead of modulating its response, the computer can also return real-time physiological information back to users in a process known as biofeedback . This involves externalizing physiological state in an accessible form for users to monitor and subsequently learn to control their own physiological activity [39, 139]. For this reason, biofeedback-based physiological systems have been particularly beneficial in therapeutic contexts, including in anxiety disorder management , substance abuse

2

treatment and general stress management , and have also gained popularity in health and fitness self-tracking . Recently, physiological, affective, and social computing research agendas have converged to consider the question: if biofeedback can be used on an intrapersonal level to help people understand their psychophysiology, can it be used interpersonally to promote an understanding of others? This question is fueled by studies showing physiological signals are inherently emotional and social . For example, physiological interaction between pairs is correlated with attachment, engagement, and team performance (see for a review), and individual physiology reflects changes in social emotions like embarrassment , guilt and empathy as well as attention in various interpersonal contexts. While physiological activity reflects our reactions to social interactions and can inform us about the emotional states of others , physiological signals do not constitute an explicit channel of information that humans use to interpret others’ states as most signals are subtle and displayed internally . However, some research has argued that externalizing such data through biofeedback in social contexts can help individuals recognize their own and others’ emotions [16, 76], and in turn augment social communication . This shift from using physiological input to promote human-computer and intrapersonal interactions to now enhancing human-human interaction has significant implications in fulfilling people’s need to belong . For the remainder of this paper, we refer to technologies enabling the sharing of physiological signals as social biofeedback systems.

While it is promising, there are limitations to our understanding of social biofeedback, its effects and the environmental factors surrounding its use. A reconceptualization of physiological signals as a communication medium has been proposed ; however, there is no systematic understanding of the contexts of social biofeedback systems. This is important given the significant moderating influence that context can have on the quality of communication and thus social outcomes. To address this, we systematically review the existing literature on social biofeedback systems to understand how these technologies can enrich social interaction.

We synthesize the physical, temporal, and social contexts in which such systems are embedded and articulate the current physiological-social space through a novel framework, the Social Biofeedback Interactions framework. We also provide a qualitative analysis of the effects of social biofeedback on socio-emotional skills, and a meta-analysis of its effects on positive emotions.

2 Literature Review

2.1 Situating physiological computing in a social context Biofeedback-based computing systems are increasingly being embedded into everyday ambulatory contexts which naturally involve the presence of other people. However, of specific interest to this review is how biofeedback can be intentionally designed to promote outcomes of social interaction. This idea is highlighted in Chanel and Mühl ’s seminal paper , which outlines two principal directions for physiological computing in social interactions: (1) using physiology as a social cue and (2) using physiology as a measurement of social interactions. As physiological data carries socially relevant information, allowing others to perceive it can help people make mental inferences, regardless of whether psychological synchrony is involved . The asymmetry involved in user-observer interactions has also been useful in assistive contexts, specifically to tailor communication to support educational (e.g., ) and work-related outcomes (e.g., ). Additionally, since

3

physiological activity fluctuates in response to social factors, biofeedback systems can also be used to assess group-level social processes like collaboration. This can be done where the computing module of the system assesses the interaction (e.g., synchrony) between physiological inputs from multiple users .

To the best of our knowledge, only one systematic review has since explored the landscape of social biofeedback technologies . This review conceptualized physiological data sharing as a novel communication medium from a media psychology and communication perspective. By assessing the communication characteristics of physiological data sharing, they found that while most current social biofeedback systems enable communication in real-time, they do not afford much autonomy nor opportunities to revise input before sharing it with others. Potentially reflecting trends toward interactional (as opposed to informational) displays of biofeedback in HCI , researchers also found many systems afforded reciprocity (bi- or multi-directional data sharing). This is interesting because the support of multi-user input shows a new physiologically-mediated mode of communication .

It was also highlighted that sensemaking with displays of physiological data is highly context-dependent, with notable contextual influences being the way data is visualized and the relationship between interactional parties . This is consistent with Bradley and Dunlop’s model of contextual information in computer science and HCI, which considers both the computing and social environments. The types of relationships between users, which fall under the social context category, have not yet been studied alongside the physical and temporal dimensions of context. Characterizing social biofeedback systems based on these moderating factors surrounding their contexts of cases is important to understand the mechanisms through which these systems lead to beneficial effects.

2.2 Towards an understanding of transferable benefits of social biofeedback In their systematic review, Feijt et al. found that social biofeedback systems can benefit interpersonal relationships through increasing feelings of intimacy, connectedness and shared experiences. This is consistent with other research showing that social connectedness can be enhanced by adding modality-specific features designed to express emotions like facial expressions, gestures and emoticons (e.g., Hug Over a Distance , the Sensing Beds , and the Cube and the Picture Frame ).

While results from shed light on the benefits of social biofeedback from a user experience (UX) standpoint, what remains unclear is whether these effects are transitory. We turn to the development of social- emotional skills as a potentially enduring and transferrable benefit of social biofeedback as they enable people to express, regulate and understand their affective, cognitive and behavioral states in everyday life and social interactions . Interestingly, studies have focused on exploring biofeedback systems to specifically support processes of individual emotional expression, regulation and understanding . However, there have not been considerations of shared user interface perspectives, particularly amidst concerns that excessive screen-time and Internet use may hinder social and emotional skill development .

2.3 Problem Statement And Contributions

While the communication characteristics of biofeedback have been explored , no studies have systematically identified the contexts of social biofeedback systems. It is also unclear whether such systems can help users practice socio-emotional skills, beyond providing transitory experiences. Addressing these points, the contributions of this paper are: (1) a systematic understanding of the physio-temporal and social

4

characteristics of the interaction contexts of social biofeedback systems, (2) the presentation of a novel framework, the Social Biofeedback Interactions framework, articulating the current physiological-social interaction space, (3) a qualitative analysis of socio-emotional competences involved in interacting through shared interfaces of physiological data, and (4) a meta-analytic review of the effectiveness of social biofeedback interactions on positive emotions.

3 Method

Our review is guided by two research questions: what types of interactions occur in physiological-social space, and with whom, and what kind of socio-emotional competencies can be practiced and/or developed by sharing displays of physiological data? Our method for identifying and screening research papers follows guidelines by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; see our detailed flowchart in Appendix A). We also conduct a thematic analysis and meta-analysis on the included studies.

3.1 Search Strategy

Xplore, Scopus and SpringerLink. Google Scholar was also used as an additional resource to ensure all relevant papers were included and locate others potentially missed by the database search. References of select papers [16, 58, 119] which were key to the research questions were also checked for relevant papers.

The Boolean search string was iteratively developed across two concept categories, physiological computing and social interaction. The final search string used across full texts: (“physiological sens*” OR “physiological signal*” OR “physiological data” OR biosens* OR biosignal* OR biodata OR biofeedback) AND (social OR interpersonal). To narrow search results from the broader Scopus database, the following search filters were applied: journal and conference proceedings, written in English, within Psychology, Engineering or Computer Science categories, and with suggested keywords “human”, “physiology” or “biofeedback”. Similarly, the filters applied for the SpringerLink database were: articles in “Computer Science and User Interfaces” and “HCI” categories.

3.2 Screening Process

The initial search returned a total of 3,951 research articles which imported into Zotero reference management software . 1,233 duplicates were automatically deleted by the software and five additional duplicates were then deleted manually. The remaining articles (n= 2,713) were then screened for relevance which was assessed based on the inclusion of keywords present in the search string and a focus on computer-mediated human- human interaction. For instance, papers explicitly describing physiological signals only in monitoring systems and for evaluation of clinical/rehabilitative interventions were excluded at this stage.

3.3 Selection Process And Eligibility Criteria

The screening resulted in 180 full-text research articles to assess for eligibility. For journal articles, conference proceedings and dissertations to be eligible for review, studies were required to (1) be published in English, (2) be peer-reviewed, (3) include at least one defined, indirect physiological measure, (4) involve human participants and (5) describe an experiment, prototype or system involving a minimum of two interactional partners. Studies investigating interpretations of manipulated physiological data (e.g., , ) and from

5

imagined interactional partners (e.g., , ) were also included, as we were interested in the perception of physiological data in social contexts, even if hypothetical. Where authors published several papers on a given system or prototype, papers were included if considered as making a novel, directly relevant contribution.

Otherwise, only the most comprehensive paper was included. To further narrow the scope of the review, studies were excluded if they (1) described only physiological measurement by a third-party, such as using physiology for patient monitoring purposes or (2) used only external measures (e.g., facial expression, gesture), endocrine or behavioral measures of physiology. We exclude external measures as they represent affect in the physical rather than physiological domain .

Papers capturing a combination of direct and indirect measures (e.g., , ) were included. Those describing (3) a prototype or design concept without evaluation or (4) physiological computing applications only in assistive contexts like therapy (e.g., ), education (e.g., ) or occupation (e.g., ) were excluded.

Papers were also excluded if they (5) described only individual uses of biofeedback contexts. We also included papers in which individual biofeedback was designed to be visible to others, such as in a public space (e.g., , ) or multiuser environment (e.g., ), as we considered such contexts to be social. Finally, papers were excluded if they (6) adopted a purely theoretical perspective, or (7) constituted research proposals, system demonstrations or doctoral consortiums.

The three authors developed the inclusion and exclusion criteria which the first author initially applied to the papers. To ensure inter-rater reliability, the second author verified screening decisions for a randomly selected subset of 20 papers, and the third author verified the complete sets of included and excluded papers. All three authors met regularly to define eligibility criteria, and the list of included papers was reviewed and assessed continually.

Leading causes of exclusion were the use of physiology for measurement and monitoring purposes (n= 23) and a lack of reporting of prototype evaluations (n= 21). The full text of 1 paper was not available on any online database, despite the authors’ best efforts to locate it, and was therefore excluded from the corpus. In addition, two papers ( and ) were excluded on the basis that more comprehensive papers describing identical prototypes were already included for review ( and , respectively). Finally, 72 papers were included.

3.4 Quality Assessment

To assess risks of bias, we conducted a quality assessment using the Critical Appraisal Skills Programme (CASP) qualitative research checklist , the most recommended tool for qualitative studies on individual experiences and in social contexts . The checklist was adapted for studies describing a quantitative or mixed-methods approach. Based on this assessment, papers were rated as having “good” (n= 46), “fair” (n= 12) or “poor” (n= 8) methodological quality. The initial assessment was done by the first author and verified by the remaining authors where the quality of a paper was deemed poor, predominantly due to a lack of rigorous data analysis. All papers receiving a “poor” rating were excluded, resulting in the final inclusion of 64 studies.

3.5 Data Extraction

Key bibliometrics from each paper (e.g., title, author(s), publication year) were exported from Zotero into an Excel spreadsheet. We then manually extracted relevant data from each study: study participants, materials and prototypes, design and methodology, key outcomes of biofeedback sharing, and social effects reported.

6

For papers wherein multiple studies were reported, data was extracted for each relevant study reporting user feedback on a prototype or outcomes from an intervention with an implemented prototype. For example, data was not extracted for user studies conducted for preliminary user research (e.g., in ), pilot tests for general usability (e.g., in ) or specific prototype features (e.g., in ), and for physiological measurement validation or calibration before intervention (e.g., in ).

3.6 Thematic Analysis

A thematic analysis was conducted to examine the effects of human-human interaction mediated by social biofeedback on socio-emotional skills. The aim was to qualitatively describe these effects across affective, cognitive, and behavioral levels. Following the process for deductive thematic analysis outlined by Braun and Clark , we started by familiarizing ourselves with the data and taking preliminary notes. The data consisted of the authors’ interpretations of their primary data. Initial codes were then generated and iteratively sorted into higher-level themes, reviewed for validity, and further refined. In total, the thematic analysis was performed on 52 studies, after excluding those which reported only quantitative results (n=12).

3.7 Meta-Analysis

A meta-analysis was conducted to estimate effect sizes for the effectiveness of social biofeedback interactions on positive affect. We included all studies reporting a quantitative measure of positive affect captured after social biofeedback-mediated interaction (n=6). Mean scores and standard deviations (SDs) of affective questionnaires measuring positive affect were extracted. Cohen’s d was used to estimate the standardized mean difference (SMD) with 95% confidence intervals (CI). Inferential statistical test values (e.g., t-test values, F-values) were used to calculate effect size when means and standard deviations were not reported.

4.1 General Study Characteristics

Studies on social physiological data sharing included in this work were published between 2007-2021 (see Appendix B for details). The number of studies on the topic is gradually increasing, particularly since 2013, despite a potentially lower emergence of studies from 2020 due to feasibility issues during the COVID-19 pandemic. General methodology and design characteristics of each paper are detailed in Appendix C. Further details of physiological sensors, biofeedback modalities and forms explored in the studies can be found in Appendix D.

The most common application contexts for social biofeedback are social play and tele-social communication (both n=14, 22% each). Social play use cases range from multiplayer virtual reality (VR) gameplay (n=5, [29– 31, 45, 63]) to video gaming (n=3, [32, 95, 107, 142]), tabletop gameplay (n=3, [2, 32, 40]), mobile (n=2, [71, 73]) and outdoor gameplay (n=1, ). Use cases for tele-social communication are primarily in remote non- verbal communication (n=9, [3, 41, 58, 65, 77, 91, 125, 133, 135]), followed by text-messaging (n=4, [52, 67, 80, 81]) and video chatting (n=1, ). Other prominent application contexts for social biofeedback include public interactive displays of emotion (n=11, 17%) in the form of installations (n=6), performances (n=3) and wearables (n=3), as well as mediated social interaction (n=7), social meditation and relaxation (n=6), social

7

exertion (n=5), online content sharing (n=4) and face-to-face communication (n=2). Overall, the oldest applications of social biofeedback are in social entertainment, play and exertion, with studies emerging since 2007. Mostly from 2015 onwards, an increasing number of studies have focused on tele-social and augmented emotion communication, with exploration through artistic means in the last few years.

4.2 Characteristics of physio-temporal and social contexts

4.2.1 Physical-Temporal Context

To understand the physical and temporal characteristics of physiological data sharing in social contexts, we use the Time-Space taxonomy of groupware proposed by Ellis et al. , which distinguishes co-located vs. distributed physical locations and synchronous (real-time) vs. asynchronous temporal dimensions of interaction.

Our analysis is shown in Figure 1.

Temporal Context

Figure 1: 2x2 matrix showing studies categorized according to the Time-Space Taxonomy of groupware . Distributed systems in general (n=39, 61%) were more commonly described than co-located ones (n=25 while synchronous systems (n=51, 80%) were more common than asynchronous ones (n=23). Specifically, distributed synchronous social biofeedback systems constitute the biggest category (24 studies, 38%), followed by co-located synchronous (n=17, 27%), distributed asynchronous (n=15, 23%) and co-located asynchronous systems (n=8, 13%).

4.2.2 Articulating the physiological-social interaction space Given the range of social contexts and relationships between interactional partners described in the literature, we considered how to represent the social interaction space for current biofeedback systems. First, driven by the concept of using one individual’s biofeedback as a social cue for third-party observers , we formalize the notion of asymmetrical interaction in physiological-social space. While the participant-observer model of biofeedback has previously been described , we considered how to build on this to reflect differences in biofeedback access, extending the idea that in media space, making information (e.g., physiological state) available is independent of necessarily obtaining it . We refer to access as opposed to visibility to encompass multi-modal forms of biofeedback. Hence, within asymmetrical interactions, we distinguish between systems that afford biofeedback access to both primary and secondary users (observers; Figure 2a) and those which only display biofeedback to secondary users (Figure 2b). As we intentionally excluded studies describing

8

researchers as recipients of users’ physiological data, we emphasize here that secondary user(s) refers to observers without an experimental agenda. Figure 2. Social biofeedback data flows in (a, b) asymmetrical and (c - e) symmetrical systems. The heart icon represents any general indirect physiological activity. Solid black arrows show physiological input, curved grey arrows show biofeedback of self-data, and dotted arrows show social biofeedback.

In addition, we extend the notion of symmetry in media space to describe a theoretically unexplored class of symmetrical interactions in physiological-social space where physiological input is obtained from more than one user. Unrelated to time, symmetry distinguishes itself from synchrony (e.g., text messaging can be both symmetrical and asynchronous). In practice, sharing physiology in multi-user environments goes beyond individual displays to what we refer to as ‘collective’ representations of physiological data. Within collective displays, we also consider variations in biofeedback content, as opposed to access, since both primary users are given access in symmetrical communication systems. Biofeedback content can include data from the other user only (Figure 2c), or data from both users either in raw (Figure 2d) or aggregated forms (Figure 2e). The characteristics of symmetrical and asymmetrical systems are summarized in Table 1.

Table 1: Summary of discriminating characteristics of symmetrical and asymmetrical social biofeedback systems We articulate the full scope of the current physiological-social space in Figure 3: the Social Biofeedback Interactions Framework. The framework shows the basic physiological data flows and multi-user interactions around individual and collective physiological data displays. It also considers behavioral forms of feedback between users.

9

Figure 3. The Social Biofeedback Interactions Framework Overall, 37 studies (58%) designed symmetrical social biofeedback systems, 23 studies (36%) designed asymmetrical ones and four studies engineered opportunities for both. In symmetrical systems, it was most common for users to receive biofeedback about their own physiological state together with that of other users’ (n=14; [2, 52, 54, 67, 70, 72, 73, 75, 77, 85, 95, 116, 119, 121]). In 12 studies, users only received feedback about states of co-users and not their own [3, 29, 30, 45, 55, 65, 87, 91, 94, 110, 125, 133], while in four studies only aggregated measures of physiology were shown to both players [83, 100, 107, 142]. Six studies combined the sharing physiology of all users with additional aggregated measures like average metrics and synchrony level between users [46, 60, 61, 82, 113]. One study manipulated both receiving biofeedback from self and others, instead of only others’ .

In asymmetrical systems, it was most common for the primary user not to be in the loop, with only secondary users having access to the individual biofeedback (n=13, 54% of asymmetrical systems, [26, 27, 31, 41, 58, 78, 81, 88–90, 130, 135, 136]). Moreover, biofeedback access was granted to both primary and secondary users in 11 studies [37, 63, 64, 80, 93, 98, 109, 112, 118, 120]. Finally, the four studies designed for both symmetrical and asymmetrical interaction styles involved two or more symmetrically-engaged users and bystanders who could observe the interaction [32, 33, 122] and provided behavioral feedback .

4.2.3 Social Communication Context

Previous research has shown that the effects of physiological data sharing depend on the relationship between interacting individuals [119, 122]. To understand the roles of social biofeedback systems as a function of their symmetry affordances, we categorize prototypes with respect to four types of social communication contexts: interpersonal, group, public and mass (see Figure 4). Within each, we identify different types of social relationships.

10

Figure 4: Social communication contexts for (left) symmetrical and (right) asymmetrical social biofeedback systems Most symmetrical social biofeedback systems were geared towards connecting dyads (n=28) with strong attachments, such as close relations [3, 52, 54, 65, 67, 70, 75, 91, 116, 125] and romantic partners [65, 77, 110, 119, 133]. Such systems were also used to connect people with similar hobbies, like video gaming [45, 95, 107] and running . Bidirectional sharing of physiological data in groups (n=8) was used mostly as a mechanic in social gameplay with friends [2, 32, 40, 87] and during shared physical exertion with fellow athletes [85, 121]. Symmetrical systems deployed in public contexts (n=5) focused on connecting members of the public by socially engaging audience members [100, 123] and designing opportunities for transformative interpersonal experiences with strangers [33, 55, 122].

Asymmetrical sharing in dyadic interaction was mainly designed for experimental purposes to study mechanisms of social perception [27, 31, 58, 78, 81, 89, 90, 135, 136], with one study using it to introduce strategic interdependence in two-player VR games . In group contexts, two studies shared physiological metrics of leaders during group exercise [88, 93]. In public, such systems (n=10) involved externalizing emotional state as a form of self-expression with loved ones , friends and colleagues , but also in general day-to-day social interactions with strangers [37, 98]. Asymmetrical biofeedback was also used to celebrate physical effort among athletes [64, 130]. Lastly, mass communication contexts involved one user sharing their physiological activity on social networking platforms, including live broadcasts during specific events [26, 80] as well as during video streaming and sharing .

4.3 The Effects Of Social Biofeedback Systems

Figure 5 shows the six themes synthesized from 61 codes through thematic analysis: (1) mindful self- awareness, (2) self-reflection and regulation of affective states, (3) empathy, (4) compassion and caregiving, (5) relationship skills for authentic connectedness, and (6) motivation, performance, and coordinative effort.

These themes are also sorted according to the key conceptual domains of socio-emotional competencies outlined in the domains and manifestations of social-emotional competences (DOMASEC) model . The framework delineates how individuals perceive themselves (self-orientation) from how they interact with people around them (others-orientation) and how they engage with tasks in the environment they are in (task- orientation). We use this multi-disciplinary framework as it allows us to clearly distinguish between affective, cognitive, and behavioral manifestations of the effects of social biofeedback systems.

11

Figure 5: Overview of major themes and corresponding DOMASEC domains identified through thematic analysis.

4.3.1 Theme 1: Mindful Self-Awareness

As an intrapersonal skill, self-awareness describes the ability to recognize our emotions. In defining this theme, we associate this with mindfulness to describe non-reflective and non-judgmental awareness of our own affective states . In this work, mindful self-awareness was found in meditative states of relaxation, noticing new internal and external phenomena, and attitudes of openness and acceptance.

Several studies found that sharing physiological data in immersive environments was a relaxing and calming experience [33, 41, 46, 52, 55, 120, 122, 125, 133]. While generally being positive in valence, these experiences were not associated with positive high arousal emotions (e.g., joy) but were instead associated with low-arousal states (e.g., pleasant, nice, and peaceful) [31, 41, 125]. Physiologically, peaceful states were found in trends towards decreasing arousal of users and a natural synchronization of physiological activity [41, 122].

Some studies also found that users made fewer efforts to communicate with co-users in meditative states, instead becoming more quiet [31, 55, 120, 125]. In others, users aware of their own mindful self-awareness expressed wanting to help co-users enter the same state [33, 46]. The emergence of contemplative meditation also helped to elicit new perspectives on the self through feelings of individual vulnerability and uniqueness . Finally, where social biofeedback was explicitly used for meditative relaxation, studies found an increased sense of focus and non-mindful hyper-awareness of one’s weaknesses relative to other users . The cognitive mechanisms behind this line of heightened awareness are described in Theme 2.

Some studies found forms of detached awareness allowing users to perceive internal bodily sensations better, noticing new relationships between their physical bodies and the environment [3, 71, 94]. The meta- awareness of new relationships was an interesting finding in a few studies, with links observed between body and mind , as well as between users and wider lifeforms such as nature [55, 122]. Some studies also found that users noticed relationships between emotions and between users over time . Studies also found that users were curious about learning more about themselves, consciously experimenting with their bodies [52, 64, 116, 119, 120].

Our analysis also revealed important attitudinal shifts towards openness and acceptance, both key to mindfulness practice . Several studies found that the mere act of sharing physiological data was perceived as a gesture of openness [33, 71, 77, 80, 119]. Openness was also found in that some users made conscious efforts to observe both the objective display of their physiology and their subjective states, inviting the possibility of differences between the two [27, 54]. In terms of fostering acceptance, some studies found users embracing a loss of agency despite privacy concerns [64, 88, 116, 130], as well as accepting negative emotions as part of the process of physiological data sharing .

12

4.3.2 Theme 2: Self-reflection and regulation of affective states On a cognitive level, self-reflection involves the active examination of one’s beliefs, thoughts, and knowledge. Unlike mindful self-awareness, it describes the elaborative processing of experience .

Studies found that engaging with social biofeedback systems encouraged individual users to reflect on the specific causes of their emotions [37, 52, 54, 83, 98, 120], judge others’ emotions [89, 90, 110] and identify triggers for negative states in particular [52, 112]. Shared physiological data displays also stimulated comparative judgments and reflected on one’s own states relative to another user’s [27, 120, 121]. After making judgements about their own affective states, biofeedback displays were used to validate these judgments as a form of confirmation [27, 54, 98, 100, 116]. Other studies found this was also the case when judgements were made about the states of others [75, 110].

These evaluations of personal emotional states also led to behavioral efforts to self-regulate. For instance, some studies found users censored or tailored their behavior to avoid potential tension with other users [37, 110, 112]. Behavioral self-regulation was not only found for those whose physiological data was displayed but also for secondary users . Other studies found self-regulation occurring for impression management, where users consciously manipulated their self-presentation to deceive, for example, in games [32, 40, 46] and sometimes during communication with a partner [67, 80].

We found that reflection during interaction with shared displays of physiological data can also elicit worries about self-revelation. For example, studies found that users were concerned about potential discrepancies between their own appraisal of their emotional state and the objective display [37, 98, 112, 119], as well as between their own appraisal and that of other users [32, 33, 54, 75, 78, 80, 112, 116]. This was related to maladaptive cognitions such as inflated self-consciousness [93, 120] and preoccupations and fears of embarrassment [32, 33, 85].

4.3.3 Theme 3: Empathy

We found that both affective and cognitive forms of empathy were relevant competences used during human- human interaction in physiological-social space. Affective empathy describes the ability to share another person’s emotional state, while cognitive empathy is the ability to accurately recognize the state of mind of another .

Several studies found affective empathy in the form of shared positive emotions like happiness, vitality and enthusiasm [3, 55, 64, 118, 123]. Positive emotions were also shared when users went through hardship together, for instance, in social physical exertion [94, 121, 130] or during games [63, 95]. Some studies found direct effects of one user’s emotional state on another’s, suggesting emotional convergence [63, 122, 125].

Others found evidence of emotional convergence where users tried to mimic a pattern of physiological response [41, 65, 67] or where physiological activity naturally became synchronized between interactional partners [52, 122, 123, 125]. Haptic ‘feeling’ was also found to directly increase feelings of empathy between users [72, 118].

Some studies also found evidence of empathic concern, where users became worried when another user’s physiology was too high or low [45, 75, 130]. However, several studies also found that having access to another person’s physiological data had the opposite effect of connecting people, such as alienation when physiology was too different [3, 93, 121].

We also found that cognitive empathy was fostered by having access to the physiological activity of another. Studies found that when presented with the biofeedback of another person, users were able to accurately detect

13

others’ internal states, a construct of empathy called mind perception [27, 37, 52, 65, 116, 125, 130]. Beyond perception, other studies found biofeedback of co-user was helpful to understand the emotional state of that person [31, 75, 78, 89, 90, 118]. In some cases, users went further by actually asking probing questions about the biofeedback to fully understand the context of their co-user [52, 75, 110, 120]. Finally, many studies also found users made guesses and explicit inferences about others’ states, another sub-construct of empathy called mentalizing [40, 52, 54, 65, 75, 83, 89, 90, 110, 116, 119, 120].

Some mixed-methods studies also measured empathy or its sub-constructs using quantitative measures. These studies found positive main effects of social biofeedback on empathy , self-reported affect [31, 63], and usage of affect-related terminology [67, 110]. While excluded from our formal qualitative analysis, it must be noted that many quantitative studies in the review measured empathy or its sub-constructs. These found that social biofeedback leads to increased emotional perspective-taking [81, 136] and self-reported empathy [60, 61, 82, 95]. However, both positive and null results have been reported for emotional convergence.

These findings were not formally meta-analyzed due to the diverse range of empathy sub-constructs and questionnaires (including custom ones) employed.

4.3.4 Theme 4: Compassion And Caregiving

Compassion is the feeling that arises when another person is suffering, which motivates helping behavior . It differs from empathy in that it involves active support and not necessarily feeling the same emotion . The most explicit examples of compassion were found in behavioral manifestations of helping behavior after seeing the physiological response of another user. Several studies found that users actively tried to soothe or calm down others [52, 65, 83, 110, 130] and sometimes censored their conversations in accordance with another person’s physiological display [37, 110, 112]. In some studies, users reported consistently checking up on another’s physiological display [52, 75, 109] and directly helping their co-user improve their performance on a task [46, 65, 130]. Altruistic behaviors were even noted in competitive settings [45, 73, 94], as well as expressions of sympathy and concern in adversarial contexts [89, 90].

Studies also found that users made conscious efforts to show thoughtfulness and respond to the emotions of others to make them feel validated [67, 77, 98, 110]. Specifically, in studies describing asymmetrical social biofeedback systems, compassion was found in that secondary users were eager to show sympathetic support for primary users going through hardship such as physical exertion [26, 64, 88, 130] and effortful gameplay . Finally, an intriguing way of showing compassion was through humor; indeed, some studies found that users initiated humor to dissipate feelings of social embarrassment when one person felt overly-exposed [32, 54, 64]. Moreover, in one quantitative study of prosocial behavior (charitable donating), no effect of social biofeedback was found .

4.3.5 Theme 5: Relationship skills for authentic social connection Relationship skills describe the actions taken to establish and maintain positive, healthy, and rewarding relationships , ultimately fostering emotional experiences of belonging. These are separate from empathetic responses because they do not necessarily entail feeling or fully understanding another person’s emotions and are also separate from exercising compassion in that they do not require an individual to be suffering. Our analysis showed that social biofeedback systems fostered relationship skills in two ways: by encouraging users to initiate new connections and maintain positive intimacy in existing relationships.

14

Shared displays of physiological data were used to naturally spark new conversations between people [32, 52, 64, 72, 75, 80, 98, 130] and support novel and spontaneous interactions with strangers [37, 64, 72, 88, 109]. Some studies reported particularly meaningful and spiritually rewarding interactions with strangers or acquaintances through shared displays [33, 55, 93, 122]. With regards to existing relationships, simply having access to the physiological activity of another person without explicit affiliative action seemed to enhance feelings of social presence [3, 27, 55, 65, 119, 125]. However, users went further to maintain connections by initiating spontaneous play, such as playfully bothering others [32, 40, 46, 64, 65, 95, 120] and naturally collaborating without it being required in a task [2, 31, 72, 73, 130]. Studies also found that opportunities for unfiltered emotion sharing were welcomed to practice reciprocal trust and honesty, leading to enhanced emotional closeness [54, 77, 116, 133].

Interestingly, some studies also found a heightened sense of responsibility for others [63, 88, 120]. However, some studies found that social biofeedback may instead promote impersonal connections, by reducing necessary effort to understand others and distracting from authentic interactions [77, 78]. Likewise, intimate awareness and connectedness were also reported when viewing social biofeedback without explicit effort or action made by either senders or receivers [91, 133].

4.3.6 Theme 6: Motivation, performance, and coordinative effort A recurrent finding in the reviewed studies was that interacting with social biofeedback systems motivated task- based persistence. In some studies, feeling observed by others motivated more effort to perform [26, 88, 123] and created opportunities for shared goal setting [93, 130]. Social biofeedback also naturally created a sense of healthy competition between athletes [94, 121] and even created feelings of competition when there was no task at all. In those studies, the controlling of physiological responses itself became a competitive endeavor [3, 64]. In competitive contexts, social biofeedback was used to strategically enhance chances of better performance [32, 40]. However, being intimately aware of others’ physical capabilities also created inopportune pressure to perform, leading to giving up [85, 93, 121] and risks of over-exertion due to harmful competition [85, 130]. Despite this, high levels of engagement and fun were reported across many studies where users were engaged in tasks [31, 32, 40, 46, 64, 72, 87, 88, 95, 107, 130].

Coordinative effort was also an essential aspect in social biofeedback systems embedded in task-specific contexts. Studies found that users took the initiative to find creative ways to interact with a co-user to achieve a common goal, including developing new norms, rules and vocabularies being socially constructed around shared displays [2, 65, 95, 107, 130]. Teamwork also became prominent in asymmetrical systems where users depended on each other for feedback about their physiology [63, 95, 130]. Other studies found that spontaneous leadership initiatives emerged to maximize the efficiency of coordinated efforts, although this led to both positive and negative outcomes. The presence of social biofeedback also encouraged users to put more effort into communicating with other users [31, 46, 63, 72, 73, 88, 98, 130].

4.4 Meta-Analysis

The meta-analysis for synthesizing the effectiveness of social biofeedback interactions on positive emotions included six papers involving 204 participants (30.5 years ± 6.42) (see Appendix E for study characteristics). Outcomes of assessments measuring positive affect were synthesized (e.g., Positive and Negative Affect Schedule (PANAS) , Networked Minds Social Presence (NMSP) , Self-Assessment Mannikin (SAM)

15

). The main results of the meta-analysis demonstrate effect sizes of biofeedback from a partner on positive affect measurements that vary from small (d=0 and d=0.146 for audio-haptic biofeedback on SAM and d=0.01, d=0,02 and d=0.19 for audio-haptic biofeedback on PANAS ) to medium (d=0.62 for audio, visual, and audio-visual biofeedback on SAM ; 0.62 for visual biofeedback on PANAS ) to large (d=1.34, d=1.38, d=1.46 for visual biofeedback on the NMSP measurement ) (see Appendix F for details). The most effective study, which measured positive affect using the NMSP measurement , was the only study to report a significant effect of feedback from three conditions: EEG (d=1.46; 95% CI=[0.78, 2.11]), respiration (d=1.38; 95% CI=[0.71, 2.04]), and EEG and respiration combined (d=1.34; 95% CI=[0.68, 2.00]). The other five studies reported effects of social HR biofeedback on self-reports of positive affect; however, none demonstrated significant effects.

5 Discussion

Despite methodological advances in psychophysiological data analysis and biosensing technology, our understanding of physiological computing as an interaction paradigm remains fragmented. Existing research focusing on social uses of biofeedback has suggested that physiological data sharing constitutes a communication medium often associated with positive user experiences. However, it is unclear what communication contexts are associated with prosocial outcomes, as well as whether social biofeedback can promote more lasting interpersonal effects. In this paper, we systematically reviewed empirical progress in social applications of biofeedback over the last two decades. We identified physio-temporal and social contextual characteristics surrounding biofeedback-mediated communication, and highlighted socio-emotional competences associated with physiological data sharing. We also developed the Social Biofeedback Interactions framework to articulate the current physiological-social space based on the literature we reviewed.

In the following section, we discuss the implications of our synthesis and propose opportunities for research and design with social biofeedback. 5.1 What types of interactions occur in physiological-social space, and with whom? 5.1.1 Physio-temporal context of social biofeedback As new communication technologies expand humans’ reach in time and space , we suggest this is no different for social biofeedback systems. Using the Time-Space taxonomy of groupware , we found that current social biofeedback systems are designed for use more in synchronous time than asynchronous ways.

The idea of synchronicity is generally important in computer-mediated communication because it provides an understanding of user context . We found synchronicity was important for overcoming physical distance while performing social activities involving both competition and collaboration, including games like Space Connection and FitBirds and activities from meditation to distributed running. In these situations, biofeedback cues were used to inform co-users of each other’s real-time states and as a mechanic to drive the social task itself. Interestingly, we also found this was the case for synchronous and co-located social biofeedback systems, e.g., [2, 32, 46, 54, 64, 87] demonstrating the informational role of social biofeedback in synchronous interactions.

However, synchronicity is not always practical over distance and may create unwanted pressure to constantly share biofeedback and undue responsibility for others’ wellbeing when used for continuous, direct

16

communication. Asynchronicity bypasses these drawbacks with two major types of interactions: awareness systems and instant messaging. First, following Weiser’s notion of calm computing, we found awareness systems were all conceived to minimize attentional requirements (e.g., BioCrystal , ExternalEyes , Wigglears , Open Heart Helmet and MoodLight ). With these systems becoming increasingly ambulatory, asynchronous co-located systems encouraged interactions that were not necessarily always focused on the display itself. For instant messaging, users also felt an increased connection to each other after using social biofeedback-enhanced systems (e.g., Significant Otter , HeartChat and Animo ). We suggest that asynchronous systems create subliminal nudges like reminding others to be mindful of each other’s emotional states. This supports the ideas of ‘ambient co-presence’ in polymedia , and that asynchrony in communication is more useful to stay in touch than to relate specific information about a person’s wellbeing .

5.1.2 The Social Biofeedback Interactions framework To articulate the current physiological-social interaction space, we formulated the Social Biofeedback Interactions framework based on ’s directions for social biofeedback and concurrent ideas of symmetry in media space . This framework allows us to make informed observations about the different implications of social biofeedback system designs, as shown in Table 2.

Table 2: Table showing biofeedback parameters and implications for both symmetrical and asymmetrical social biofeedback

Secondary Users

The main purpose of using this type of asymmetrical social biofeedback set-up is for celebrating self-expression of emotional state while raising awareness for bystanders in the surrounding environment. As emotional self-expression through biofeedback can be daunting, granting access to the primary user can alleviate self- presentation concerns as well as promote introspection.

Secondary User

Intentionally designing the primary user out of the social biofeedback loop necessitates the acceptance of a loss of agency associated with self-disclosure, and trust. For those reasons, the most common application of this type of system to date is in the laboratory to conduct experiments of social perception of physiological signals. However, this type of asymmetricity has been useful in sporting events, where physiology reflects exertion effort as opposed to more intimate emotions.

From Both Users

Seeing another user’s biofeedback as well as one’s own has mostly been implemented in games and sport to engender competition and drive individual performance to rival that of a co-user’s. This system design is also useful for mobile messaging applications, in a similar way to text-based texting where both users can see their own and their interlocuter’s responses. Data from both users can also be presented in aggregated form, which can dispel concerns about social image, create opportunities for social play and for collaborative efforts.

Other User

This type of design enables users to receive physiological information of one another while not being explicitly aware of what physiological data they themselves are transmitting. The most common application for this was to increase intimate connectedness between two users in a way resembling natural communication.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.