Abstract
Quorum systems are a key abstraction in distributed fault-tolerant computing for capturing trust as- sumptions. They can be found at the core of many algorithms for implementing reliable broadcasts, systems that model subjective trust. Every process is free to choose which combinations of other pro- cesses it trusts and which ones it considers faulty. Asymmetric quorum systems strictly generalize standard Byzantine quorum systems, which have only one global trust assumption for all processes.
This work also presents protocols that implement abstractions of shared memory, broadcast primi- tives, and a consensus protocol among processes prone to Byzantine faults and asymmetric trust. The model and protocols pave the way for realizing more elaborate algorithms with asymmetric trust.
Ntroduction
Byzantine quorum systems are a fundamental primitive for building resilient distributed systems from untrusted components. Given a set of nodes, a quorum system captures a trust assumption on the nodes in terms of potentially malicious protocol participants and colluding groups of nodes. Based on quorum systems, many well-known algorithms for reliable broadcast, shared memory, consensus and more have been implemented; these are the main abstractions to synchronize the correct nodes with each other and to achieve consistency despite the actions of the faulty, so-called Byzantine nodes.
Traditionally, trust in a Byzantine quorum system for a set of processes P has been symmetric. In other words, a global assumption specifies which processes may fail, such as the simple and prominent threshold quorum assumption, in which any subset of P of a given maximum size may collude and act against the protocol. The most basic threshold Byzantine quorum system, for example, allows all subsets of up to f < n/3 processes to fail. Some classic works also model arbitrary, non-threshold symmetric quorum systems [35, 27], but it is unknown if these have been used in practice.
However, trust is inherently subjective. De gustibus non est disputandum – There is no disputing about taste. Estimating which processes will function correctly and which ones will misbehave may depend on personal taste. A myriad of local choices influences one process’ trust in others, especially because there are so many forms of “malicious” behavior. Some processes might not even be aware of all others, yet a process should not depend on unknown third parties in a distributed collaboration. How can one model asymmetric trust in distributed protocols? Can traditional Byzantine quorum systems be extended to subjective failure assumptions? How do the standard protocols generalize to this model? 1This work combines multiple preliminary publications on asymmetric trust and protocols with asymmetric trust, which appeared at OPODIS 2019 , DISC 2021 , and ESORICS-CBT 2021 3DFINITY, CH-8000 Z¨urich, Switzerland. Work done at IBM Research – Zurich.
Arxiv:1906.09314V2 [Cs.Dc] 2 May 2024
Asymmetric trust. In this paper, we answer these questions and introduce models and protocols for asymmetric distributed trust. We formalize asymmetric (Byzantine) quorum systems for asynchronous protocols, in which every process can make its own assumptions about Byzantine faults of others. We introduce several protocols with asymmetric trust that strictly generalize the existing algorithms, which require common trust.
Our formalization takes up earlier work by Damg˚ard et al. and starts out with the notion of a fail-prone system that forms the basis of a symmetric Byzantine quorum system. A global fail-prone system for a process set P contains all maximal subsets of P that might jointly fail during an execution.
In an asymmetric quorum system, every process specifies its own fail-prone system and a corresponding all processes and a local availability condition, and generalize symmetric Byzantine quorum system according to Malkhi and Reiter .
Quorum systems are used within various fault-tolerant dis- tributed protocols, here specifically within protocols for systems subject to Byzantine faults. An impor- tant aspect of our notion concerns its relation to existing protocols: it should be easy to generalize the known protocols to the asymmetric model, ideally simply by replacing the symmetric quorums with their asymmetric counterparts. Indeed this is the case for many, but not for all protocols described here. A different, generalized analysis is necessary in any case.
We show first that two existing protocols for emulating a shared regular register also work in the asymmetric model. Second, we introduce asymmetric Byzantine consistent and reliable broadcast prim- Third, we address consensus, one of the most important primitives in distributed computing, and extend a randomized binary consensus protocol for asynchronous networks to work with asymmetric trust. The protocol relies on a common coin abstraction, for which a different implementation is needed.
Our randomized consensus takes up the award-winning, randomized, and signature-free implemen- tation of consensus by Most´efaoui et al. . In its 2014 version, however, this protocol suffered from a liveness issue, which was corrected subsequently , although the fix added considerable complex- ity. The corrected algorithm offers the same asymptotic complexity in message and time as the original algorithm, but it requires more communication steps.
Through our randomized asymmetric consensus, we also introduce a novel way of fixing the problem in the original protocol. It retains the latter protocol’s simplicity, which is an appealing property. Ob- viously, our asymmetric consensus protocol can also be instantiated with symmetric threshold quorums to work in the same model as the protocol of Most´efaoui et al. . In order to clearly demonstrate the liveness issue and to show how our approach avoids it, we also include in this work a discussion of this randomized consensus algorithm in the symmetric-trust model.
In the traditional models for quorum-based systems, all correct processes uniformly benefit from the guarantees of a protocol as long as the initial assumption expressed by the fail-prone system holds. With subjective trust, this symmetry no longer exists. Some of the correct processes may have made assumptions that proved appropriate in an execution with actually faulty processes F ⊂P; we call these processes wise. Other correct processes, however, may have assumed that only a proper subset of F actually fails; these processes are na¨ıve and they do not enjoy the same guarantees as the wise ones, even though they are correct. In particular, our protocols typically ensure safety only for wise processes and liveness depends on the existence of a sufficiently large group of wise processes.
Motivation. Interest in consensus protocols based on Byzantine quorum systems has surged recently because of their application to permissioned blockchain networks [14, 6]. Typically run by a consor- tium, such distributed ledgers often use Byzantine-fault tolerant (BFT) protocols like PBFT , Ten- Bitcoin blockchain and many other cryptocurrencies, which triggered this development, started from dif- ferent assumptions and use so-called permissionless protocols, in which everyone may participate. Those algorithms capture the relative influence of the participants on consensus decisions by an external factor, such as invested “work” or “stake” in the system.
A middle ground between permissionless blockchains and BFT-based ones has been introduced by the blockchain networks of Ripple (https://ripple.com) and Stellar (https://stellar. org). Their stated model for achieving network-level consensus uses subjective trust in the sense that each process declares a local list of processes that it “trusts” in the protocol.
Consensus in the Ripple blockchain (and for the XRP cryptocurrency on the XRP Ledger) is executed by its validator nodes. Each node declares a Unique Node List (UNL), which are validators that this node trusts, in the sense that “the given participant believes [they] will not conspire to defraud [the node].” At least up to around 2020, however, nodes have not really been free in their trust choice since “Ripple provides a default and recommended list which we [Ripple] expand based on watching the history of validators operated by Ripple and third parties” . As of 2023, the XRP ledger documentation states that “currently the XRP Ledger Foundation and Ripple are known to publish recommended default lists of high quality validators . . ” . It is clear that two nodes that transact via the XRP ledger need to have some validators that they trust in common. But many questions are left open about the kind of decentralization offered by the Ripple protocol.
Stellar was created as an evolution of Ripple that shares much of the same design philosophy. The Stellar consensus protocol powers the Stellar Lumen (XLM) cryptocurrency and introduces federated Byzantine quorum systems (FBQS); they also capture subjective trust assumptions, but differ technically of a quorum that can convince one particular node of agreement.” In an FBQS, “each node chooses its own quorum slices” and “the system-wide quorums result from these decisions by individual nodes” .
Contribution. The main motivation for this work is to understand how existing ideas of subjective formalization of asymmetric quorums provides a sound foundation for protocols with asymmetric trust.
The protocols described here generalize well-known, classic algorithms in the literature and therefore look similar. This should be seen as a feature, actually, because simplicity and modularity are important guiding principles in science.
Our Contributions Are As Follows:
• We introduce asymmetric Byzantine quorum systems formally in Section 4 as an extension of standard Byzantine quorum systems and discuss some of their properties. • In Section 5, we show two implementations of a shared register, with single-writer, multi-reader • We examine broadcast primitives in the Byzantine model with asymmetric trust in Section 6. In particular, we define and implement Byzantine consistent and reliable broadcast protocols.
• In Section 7, we present the first asynchronous Byzantine consensus protocol with asymmetric trust. It uses randomization, provided by an asymmetric common coin protocol, to circumvent the impossibility of asynchronous consensus.
Before presenting the technical contributions, we discuss related work in Section 2 and state our system model in Section 3. A detailed discussion of the liveness issue in the existing signature-free Byzantine consensus protocol and of our approach to fixing it appears in Appendix A.
Related Work
Practical systems: Ripple and Stellar. The Ripple consensus protocol is run by an open set of val- idator nodes. The protocol uses votes, similar to standard consensus protocols, whereby each validator only communicates with the validators in its UNL. Each validator chooses its own UNL, which makes it possible for anyone to participate, in principle, similar to proof-of-work blockchains. Early investiga- tions suggested that the intersection of the UNLs of every two validators should be at least 20% of each list , assuming that also less than one fifth of the validators in the UNL of every node might be faulty.
An independent analysis by Armknecht et al. later argued that this bound must be more than 40%. A technical report of Chase and MacBrough [20, Thm. 8] concludes, under the same assumption of f < n/5 faulty nodes in every UNL of size n, that the UNL overlap should actually be at least 90%.
However, the same paper also derives a counterexample to the liveness of the Ripple consensus protocol [20, Sec. 4.2] as soon as two validators don’t have “99% UNL overlap.” By generalizing the example, this essentially means that the protocol can get stuck unless all nodes have the same UNL.
According to the standards of the field of distributed systems, though, a protocol needs to satisfy safety and liveness because achieving only one of these properties is trivial. Amores-Sesar et al. confirm the prior analysis and exhibit a wider set of examples how safety and liveness may be violated in executions of the Ripple consensus protocol. They first show that the network may fork, even under the standard condition stated by Ripple on the overlap of UNLs, and then that the consensus protocol may lose liveness in the presence of only one Byzantine process, even if all the processes have the same UNL. These works, however, exploit arbitrary message delays, i.e., a period of asynchronous network behavior, which is not assumed by Ripple and arguably also unlikely to occur in practice.
The Stellar consensus protocol (SCP) also features open membership and lets every node express its own set of trusted nodes [36, 31]. Generalizing from Ripple’s flat lists of unique nodes, every node declares a collection of trusted sets called quorum slices, whereby a slice is “the subset of a quorum convincing one particular node of agreement.” A quorum in Stellar is a set of nodes “sufficient to reach agreement,” defined as a set of nodes that contains one slice for each member node. The quorum choices of all nodes together yield a federated Byzantine quorum systems (FBQS). The literature on Stellar gives properties for FBQS and contains protocols that build on them, which have been implemented in the Stellar blockchain . However, standard Byzantine quorum systems and FBQS are not comparable because (1) an FBQS when instantiated with the same trust assumption for all processes does not reduce to a symmetric quorum system and (2) existing protocols do not directly generalize to FBQS.
Models of asymmetric trust. Starting from Stellar’s notions, Garc´ıa-P´erez and Gotsman build a link from FBQS to existing quorum-system concepts by investigating a Byzantine reliable broadcast abstraction in an FBQS. They show that the federated voting protocol of Stellar is similar to Bracha’s reliable broadcast and that it implements a variation of Byzantine reliable broadcast on an FBQS for executions that contain, additionally, a set of so-called intact nodes. Losa et al. have later formulated an abstraction of the consensus mechanism in the Stellar network by introducing Personal Byzantine quorum systems (PBQS). In contrast to the other notions of “quorums”, their definition does not require each one satisfies agreement and liveness on its own.
The FBQS and PBQS concepts, however, differ from the notion of a Byzantine quorum system in the literature. In particular, the characterization of their properties seems to take into account knowledge of which nodes are Byzantine, and their effects are therefore analyzed in the context of particular executions.
Existing notions of symmetric quorum systems in the literature [35, 27] start from an a-priori assumption about all potentially faulty sets of nodes, through a fail-prone system . This permits to study protocol- Another approach for designing Byzantine fault-tolerant (BFT) consensus protocols has been intro- duced by Malkhi et al. , namely Flexible BFT. This notion guarantees higher resilience by intro- ducing a new alive-but-corrupt fault type, which denotes processes that attack safety but not liveness.
Malkhi et al. also define flexible Byzantine quorums that allow processes in the system to have different faults models. Our work, in contrast, goes back to the model of Damg˚ard et al. . It already contains the basic formulation of asymmetric trust and expresses it in the context of synchronous protocols for secure property, but omits liveness. Damg˚ard et al. also state a characterization of when an asymmetric Byzantine quorum system exists (with the so-called B3), but give no proof. Their work has remained without impact until research on cryptocurrencies has revived interest in heterogeneous and subjective trust models.
Signature-free randomized consensus. Most´efaoui et al. present a randomized, signature-free, and round-based asynchronous consensus algorithm for binary values. It achieves optimal resilience and takes O(n2) constant-sized messages. Randomization is achieved through a common coin as defined by Rabin . Their binary consensus algorithm has been taken up for constructing the “Honey Badger BFT” protocol by Miller et al. , for instance. One important contribution of Most´efaoui et al.
is a new binary validated broadcast primitive with a non-deterministic termination property; it has also found applications in other protocols . Tholoniat and Gramoli observe a liveness issue in the protocol by Most´efaoui et al. in which an adversary is able to prevent progress among the correct processes by controlling messages between them and by sending them values in a specific order.
In a later work, Most´efaoui et al. present a different version of their randomized consensus algorithm that does not suffer from the liveness problem anymore. The resulting algorithm offers the same asymptotic complexity in message and time as their previous algorithm , but requires more communication steps.
System Model
Processes. We consider a system of n processes P = {p1, . . , pn} that communicate with each other. The processes interact asynchronously with each other through exchanging messages. The system itself is asynchronous, i.e., the delivery of messages among processes may be delayed arbitrarily and the processes have no synchronized clocks. Every process is identified by a name, but such identifiers are not made explicit. A protocol for P consists of a collection of programs with instructions for all processes.
Protocols are presented in a modular way using the event-based notation of Cachin et al. . Executions and faults. An execution starts with all processes in a special initial state; subsequently the processes repeatedly trigger events, react to events, and change their state through computation steps.
Every execution is fair in the sense that, informally, processes do not halt prematurely when there are still steps to be taken or events to be delivered (we refer to the standard literature for a formal definition ). A process that follows its protocol during an execution is called correct. On the other hand, a faulty process may crash or even deviate arbitrarily from its specification, e.g., when corrupted by an adver- sary; such processes are also called Byzantine. We consider only Byzantine faults here and assume for simplicity that the faulty processes fail right at the start of an execution.
Functionalities. A functionality is an abstraction of a distributed computation, either a primitive that may be used by the processes or a service that they will provide. Every functionality in the system is specified through its interface, containing the events that it exposes to protocol implementations that may call it, and its properties, which define its behavior. A process may react to a received event by changing their state and triggering further events.
There are two kinds of events in an interface: input events that the functionality receives from other abstractions, typically to invoke its services, and output events, through which the functionality delivers information or signals a condition to a process. The behavior of a functionality is usually stated through a number of properties or through a sequential implementation.
Multiple functionalities may be composed together modularly. In a modular protocol implementa- tion, in particular, every process executes the program instructions of the protocol implementations for all functionalities in which it participates.
Links. We assume there is a low-level functionality for sending messages over point-to-point links be- tween each pair of processes. In a protocol, this functionality is accessed through the events of “sending a message” and “receiving a message.” Point-to-point messages are authenticated and delivered reliably among correct processes.
Moreover, we assume FIFO ordering on the reliable point-to-point links for every pair of correct processes. This means that if a correct process has “sent” a message m1 and subsequently “sent” a message m2, then every correct process does not “receive” m2 unless it has earlier also “received” m1.
FIFO-ordered links are actually a very common assumption. Protocols that guarantee FIFO order on top of (unordered) reliable point-to-point links are well-known and simple to implement [26, 11]. We remark that there is only one FIFO-ordered reliable point-to-point link functionality in the model; hence, FIFO order holds among the messages exchanged by the implementations for all functionalities used by a protocol.
Idealized digital signatures. A digital signature scheme provides two operations, signi and verifyi. The invocation of signi specifies a process pi and takes a bit string m ∈{0, 1}∗as input and returns a signature σ ∈{0, 1}∗with the response. Only pi may invoke signi. The operation verifyi takes a putative signature σ and a bit string m as parameters and returns a Boolean value with the response. Its implementation satisfies that verifyi(σ, m) returns TRUE for any i ∈[1, n] and m ∈{0, 1}∗if and only if pi has executed signi(m) and obtained σ before; otherwise, verifyi(σ, m) returns FALSE. Every process may invoke verify.
Asymmetric Byzantine Quorum Systems
This section defines asymmetric Byzantine quorum systems and the notions of a guild and a tolerated system, which are used in protocols later. To set the stage, symmetric Byzantine quorum systems are reviewed first.
Review Of Symmetric Trust
Quorum systems are well-known in settings with symmetric trust. As demonstrated by many applications to distributed systems, ordinary quorum systems and Byzantine quorum systems play a crucial role in formulating resilient protocols that tolerate faults through replication . A quorum system typically ensures a consistency property among the processes in an execution, despite the presence of some faulty processes.
For the model with Byzantine faults, Byzantine quorum systems have been introduced by Malkhi and Reiter . This notion is defined with respect to a fail-prone system F ⊆2P, a collection of subsets of P, none of which is contained in another, such that some F ∈F with F ⊆P is called a fail-prone set and contains all processes that may at most fail together in some execution . A fail-prone system is the same as the basis of an adversary structure, which was introduced independently by Hirt and Maurer .
A fail-prone system captures an assumption on the possible failure patterns that may occur. It speci- fies all maximal sets of faulty processes that a protocol should tolerate in an execution; this means that a protocol designed for F achieves its properties as long as the set F of actually faulty processes satisfies F ∈F∗. Here and from now on, the notation A∗for a system A ⊆2P, denotes the collection of all subsets of the sets in A, that is, A∗= {A′|A′ ⊆A, A ∈A}.
Definition 1 (Byzantine quorum system ). A Byzantine quorum system for F is a collection of sets of processes Q ⊆2P where no set is contained in another and each Q ∈Q is called a quorum, such the
Following Properties Hold:
Consistency: The intersection of any two quorums contains at least one process that is not faulty, i.e.,
∀Q1, Q2 ∈Q, ∀F ∈F : Q1 ∩Q2̸ ⊆F.
Availability: For any set of processes that may fail together, there exists a disjoint quorum in Q, i.e.,
∀F ∈F : ∃Q ∈Q : F ∩Q = ∅.
The above notion is also known as a Byzantine dissemination quorum system and allows a pro- tocol to be designed despite arbitrary behavior of the potentially faulty processes. The notion generalizes the usual threshold failure assumption for Byzantine faults , which considers that any set of f pro- cesses may fail.
We say that a set system T dominates another set system S if for each S ∈S there is some T ∈T such that S ⊆T . In this sense, a quorum system for F is minimal whenever it does not dominate any other quorum system for F. A maximal set system is defined analogously.
Similarly to the threshold case, where n > 3f processes are needed to tolerate f faulty ones in many Byzantine protocols, Byzantine quorum systems can only exist if not “too many” processes fail. Definition 2 (Q3-condition [35, 27]). A fail-prone system F satisfies the Q3-condition, abbreviated as
∀F1, F2, F3 ∈F : P̸ ⊆F1 ∪F2 ∪F3.
In other words, Q3(F) means that no three fail-prone sets together cover the whole system of pro- cesses. A Qk-condition can be defined like this for any k ≥2 . The following result of Malkhi and Reiter [35, Theorem 5.4] considers the bijective complement of a process set S ⊆2P, which is defined as S = {P \ S|S ∈S}, and turns F into a Byzantine quorum system. A related theorem was formulated also by Hirt and Maurer .
Lemma 1. Given a fail-prone system F, a Byzantine quorum system for F exists if and only if Q3(F). In particular, if Q3(F) holds, then F, the bijective complement of F, is a Byzantine quorum system. The quorum system Q = F is called the canonical quorum system of F. According to the duality between Q and F, properties of F are sometimes ascribed to Q as well. However, note that the canonical quorum system is not always minimal. For instance, if F consists of all sets of f ≪n/3 processes, then each quorum in the canonical quorum system has n −f members, but also the family of all subsets of P
With ⌈N+F+1
⌉< n −f processes forms a quorum system. Core sets. A core set C for F is a minimal set of processes that contains at least one correct process in every execution. More precisely, C ⊆P is a core set whenever (1) for all F ∈F, it holds P \F ∩C̸ = ∅ (and, equivalently, C̸ ⊆F) and (2) for all C′ ⊊C, there exists F ∈F such that P \ F ∩C′ = ∅(and, equivalently, C′ ⊆F). With the threshold failure assumption, every set of f + 1 processes is a core set.
A core-set system C is the minimal collection of all core sets, in the sense that no set in C is contained in another. Core sets can be complemented by survivor sets, as shown by Junqueira et al. . This yields a dual Kernels.
Given a symmetric Byzantine quorum system Q, we define a kernel K as a minimal set of processes that overlaps with every quorum. A kernel generalizes the notion of a core set . Definition 3 (Kernel system). A set K ⊆P is a kernel of a quorum system Q if an only if
∀K′ ⊊K : ∃Q ∈Q : Q ∩K′ = ∅.
We also define the kernel system K of Q to be the set of all kernels of Q. For example, under a threshold failure assumption where any f processes may fail, every set of
processes is a kernel. In particular, n = 3f + 1 if and only if every kernel has f + 1 processes. The definition of a kernel is related to that of a core set in the following sense. Lemma 2. Let F be a fail-prone system and Q = F be the canonical quorum system of F. Then the kernel system of Q is the same as the core-set system for F.
Proof. Consider a kernel system K of a Byzantine quorum system Q. By definition, the following two properties hold with respect to every kernel K ∈K: (i) For every quorum Q in Q, the intersection with the kernel K is non-empty, i.e., K ∩Q̸ = ∅.
(ii) For any proper subset K′ of K, there exists a quorum Q in Q such that K′ does not intersect with Q, i.e., Q ∩K′ = ∅. Given the canonical quorum system Q derived from the fail-prone system F, by definition of canonical quorum system of F we have that for every Q in Q, there exists a unique fail-prone set F in F such that Q is precisely the complement of F within P, that is, Q = P \ F. Consequently, the concepts of a kernel and a core set are equivalent in this context, as a core set is defined with respect to sets of the form P \ F.
Lemma 3. Let F, Q, and K be a fail-prone system, a Byzantine quorum system for F, and the kernel system of Q, respectively. Then, for every quorum Q ∈Q, there exists a kernel K ∈K such that K ⊆Q. Proof. Consider the quorum system Q for F. Let F be any such fail-prone set in F. For a given quorum Q ∈Q, define the set K = Q \ F. By definition, K is a subset of Q, i.e., K ⊆Q. The consistency property of the Byzantine quorum system now implies that any two quorums Q, Q′ ∈Q have an intersection Q ∩Q′ that is not fully contained within F. Therefore, K intersects with Q′ since (Q\F)∩Q′ = K∩Q′ is not empty. This property holds for every Q′ ∈Q and confirms that K intersects with every quorum in Q. As such, K satisfies the first property of a kernel of Q.
For the second property, minimality, let us consider such a K. To construct a kernel contained in Q, we progressively remove elements from K, ensuring that the resultant subset retains the property of without losing the intersection property. The minimality of K∗is guaranteed by the contradiction that arises from the assumption that a proper subset of K∗could intersect with all quorums, as this would violate the termination of our removal process. Therefore, K∗is a kernel by definition since it is the minimal intersecting set with every quorum in Q, and it is contained within the original quorum Q from which we subtracted F. This shows that K∗is a kernel of Q.
Asymmetric Trust
In our model with asymmetric trust, every process is free to make its own trust assumption and to express this with a fail-prone system. Hence, an asymmetric fail-prone system F = [F1, . . , Fn] consists of an array of fail-prone systems, where Fi denotes the trust assumption of pi. One often assumes pi̸ ∈Fi for practical reasons, but this is not necessary. This notion has earlier been formalized by Damg˚ard et al.
Definition 4 (Asymmetric Byzantine quorum system). An asymmetric Byzantine quorum system for F is an array of collections of sets Q = [Q1, . . , Qn], where Qi ⊆2P for i ∈[1, n]. The set Qi ⊆2P is called the quorum system of pi and any set Qi ∈Qi is called a quorum (set) for pi. It satisfies: Consistency: The intersection of two quorums for any two processes contains at least one process for which either process assumes that it is not faulty, i.e., ∀i, j ∈[1, n], ∀Qi ∈Qi, ∀Qj ∈Qj, ∀Fij ∈Fi∗∩Fj∗: Qi ∩Qj̸ ⊆Fij.
Availability: For any process pi and any set of processes that may fail together according to pi, there
Exists A Disjoint Quorum For Pi In Qi, I.E.,
∀i ∈[1, n], ∀Fi ∈Fi : ∃Qi ∈Qi : Fi ∩Qi = ∅. Recall that the consistency condition for a (symmetric) Byzantine quorum system requires that at least one process in the intersection of every two quorums is correct. In the asymmetric case, quorums are subjective and defined according to the quorum system for each process. The asymmetric consistency property states that in the intersection of every two subjective quorums of two processes there exists at least one process that is correct according to one of the two processes. On the other hand, the availability condition in the above definition is a direct extension of the symmetric case, since it considers the quorum system of each process separately. We remark that availability suffices for implementing some protocols but a stronger assumption (i.e., the existence of a guild, introduced below) is needed for others.
The existence of asymmetric quorum systems can be characterized with a property that generalizes the Q3-condition for the underlying asymmetric fail-prone systems as follows. Definition 5 (B3-condition). An asymmetric fail-prone system F satisfies the B3-condition, abbreviated
As B3(F), Whenever It Holds That
∀i, j ∈[1, n], ∀Fi ∈Fi, ∀Fj ∈Fj, ∀Fij ∈Fi∗∩Fj∗: P̸ ⊆Fi ∪Fj ∪Fij The following result is the generalization of Lemma 1 for asymmetric quorum systems; it was stated by Damg˚ard et al. without proof.
Theorem 4. An asymmetric fail-prone system F satisfies B3(F) if and only if there exists an asymmetric quorum system for F. Proof. Suppose that B3(F). We let Q = [Q1, . . , Qn], where Qi = Fi is the canonical quorum system of Fi, and show that Q is an asymmetric quorum system. Indeed, let Qi ∈Qi, Qj ∈Qj, and Fij ∈ Fi∗∩Fj∗for any i and j. Then Fi = P \Qi ∈Fi and Fj = P \Qj ∈Fj by construction, and therefore, Fi ∪Fj ∪Fij̸ = P holds according to B3(F). This means there is some pk ∈P \ (Fi ∪Fj ∪Fij).
Because pk̸ ∈Fi, it holds pk ∈Qi and analogously pk ∈Qj. This implies in turn that pk ∈Qi ∩Qj but pk /∈Fij and proves the consistency condition. The availability property holds by construction of the To show the reverse direction, let Q be a candidate asymmetric Byzantine quorum system for F that satisfies availability and assume towards a contradiction that B3(F) does not hold. We show that consistency cannot be fulfilled for Q. By our assumption there are sets Fi, Fj, Fij in F such that Fi∪Fj ∪ Fij = P, which means also that P \ (Fi ∪Fj) ⊆Fij. The availability condition for Q then implies that there are sets Qi ∈Qi and Qj ∈Qj with Fi ∩Qi = ∅and Fj ∩Qj = ∅. Now for every pk ∈Qi ∩Qj it holds that pk /∈Fi ∪Fj by availability and therefore pk ∈P \ (Fi ∪Fj). Taken together this means that Qi ∩Qj ⊆P \ (Fi ∪Fj) ⊆Fij. Hence, Q does not satisfy the consistency condition and the statement follows.
Asymmetric core sets and kernels. Let F = [F1, . . , Fn] be an asymmetric fail-prone system. An asymmetric core-set system C is an array of collections of sets [C1, . , Cn] such that each Ci is a core set system for the fail-prone system Fi. We call a set Ci ∈Ci a core set for pi.
Given an asymmetric quorum system Q for F, an asymmetric kernel system for Q is defined analo- gously as the array K = [K1, . . , Kn] that consists of the kernel systems for all processes in P. A set Ki ∈Ki is called a kernel for pi. This means that every kernel for pi has a non-empty intersection with every quorum of pi.
Na¨ıve and wise processes. Recall that the guarantees of quorum-based protocols apply to correct processes only, but not to faulty ones. The faults or corruptions occurring in a protocol execution with an underlying quorum system induce a set F of actually faulty processes. However, no process knows F and this information is only available to an observer outside the system. With a traditional quorum system Q designed for a fail-prone set F, the guarantees of a protocol usually hold as long as F ∈F∗, and if F is not contained in F∗, no useful properties can be derived for any process.
With asymmetric quorums, we further distinguish between two kinds of correct processes, depending on whether they considered F in their trust assumption or not. Given a protocol execution, the processes
Are Therefore Partitioned Into Three Types:
Faulty: A process pi ∈F is faulty. Na¨ıve: A correct process pi for which F̸ ∈Fi∗is called na¨ıve. Wise: A correct process pi for which F ∈Fi∗is called wise.
The na¨ıve processes are new for the asymmetric case, as all correct processes are wise under a symmetric trust assumption. Protocols for asymmetric quorums cannot guarantee the same properties for na¨ıve processes as for wise ones, since the na¨ıve processes may have the “wrong friends.” In one formalization of the Stellar protocol, correct nodes that find themselves in a similar situation have been called “befouled” .
Example 1. We define an example of asymmetric fail-prone system FA on P = {p1, p2, p3, p4, p5}. The
Notation Θn
k(S) for a set S with n elements denotes the “threshold” combination operator and enumerates all subsets of S of cardinality k. W.l.o.g. every process trusts itself. The diagram below shows fail-prone
Sets As Shaded Areas And The Notation N
k in front of a fail-prone set stands for k out of the n processes in the set.
P1
The operator ∗for two sets satisfies A ∗B = {A ∪B|A ∈A, B ∈B}. As one can verify in a straightforward way, B3(FA) holds. Let QA be the canonical asymmetric quo- rum system for FA. Note that since FA contains the fail-prone systems of p3 and p5 that permit two faulty processes each, this fail-prone system cannot be obtained as a special case of Θ5 1({p1, p2, p3, p4, p5}).
When F = {p2, p4}, for example, then processes p3 and p5 are wise and p1 is na¨ıve. Guilds. If too many processes are na¨ıve or even fail during a protocol run with asymmetric quorums, then protocol properties cannot be ensured. A guild is a set of wise processes that contains at least one quorum for each member; by definition this quorum consists only of wise processes. A guild ensures liveness and consistency for typical protocols. This generalizes from protocols with symmetric trust, where the correct processes in every execution form a quorum by definition. A guild represents a group of influential and well-connected wise processes, like in the real world.
Definition 6 (Guild). Given a fail-prone system F, an asymmetric quorum system Q for F, and a protocol execution with faulty processes F, a guild G for F and Q satisfies two properties:
Wisdom: G Is A Set Of Wise Processes:
∀pi ∈G : F ∈Fi∗. Closure: G contains a quorum for each of its members: ∀pi ∈G : ∃Qi ∈Qi : Qi ⊆G.
A guild is related to an “intact set” in the Stellar consensus protocol [36, 31], but the two notions differ in how they are defined. Observe that the union of two guilds is again a guild, since the union consists only of wise processes and contains again a quorum for each member. All guilds overlap, as the next result shows.
Lemma 5. In any execution with a guild G, every two guilds intersect. Proof. Let P be a set of processes, G be a guild, and F be the set of actually faulty processes. Further- more, suppose that there is another guild G′. Let pi ∈G and pj ∈G′ be two processes and consider a quorum Qi ⊆G for pi and a quorum Qj ⊆G′ for pj. From the definition of an asymmetric quorum system it must hold Qi ∩Qj ⊈F, with Qi ∩Qj̸ = ∅and F ∈Fi∗∩Fj∗. It follows that there exists a wise process pk ∈Qi ∩Qj with pk ∈G and pk ∈G′. Notice also that G and G′ both contain a quorum for pk.
It follows that every execution with a guild contains a unique maximal guild Gmax. The next lemma shows that if a guild exists, no quorum for any process contains only faulty processes. Lemma 6. Let Gmax be the maximal guild for a given execution and let Q be the canonical asymmetric quorum system. Then, there cannot be a quorum Qj ∈Qj for any process pj consisting only of faulty processes.
Proof. Given an execution with F as set of faulty processes, suppose there is a guild Gmax. This means that for every process pi ∈Gmax, a quorum Qi ⊆Gmax exists such that Qi ∩F = ∅. It follows that for every pi ∈Gmax, there is a set Fi ∈Fi such that F ⊆Fi. Recall that since Q is a quorum system, B3(F) holds. From Definition 5, we have that for all i, j ∈[1, n], all Fi ∈Fi, ∀Fj ∈Fj, and all Fij ∈Fi∗∩Fj∗, it holds P̸ ⊆Fi ∪Fj ∪Fij.
Towards a contradiction, assume that there is a process pj such that there exists a quorum Qj ∈Qj for pj with Qj = F. This implies that there exists Fj ∈Fj such that Fj = P \ F. Let Fi be the fail-prone system of pi ∈Gmax such that F ⊆Fi and let Fj = P \ F as just defined.
Then, Fi ∪Fj ∪Fij = P. This follows from the fact that Fi contains F and that Fj = P \ F. This contradicts the B3-condition for F. Lemma 7. Let Gmax be the maximal guild for a given execution and let pi be any correct process. Then, every quorum for pi contains at least one process in Gmax.
Proof. The claim naturally derives from the consistency property of an asymmetric quorum system. Consider any correct process pi and one of its quorums, Qi ∈Qi. For any process pj ∈Gmax, let Qj be a quorum of pj such that Qj ⊆Gmax, which exists because Gmax is a guild. Then, the quorum consistency property implies that Qi ∩Qj̸ = ∅. Thus, Qi contains a process in the maximal guild.
Finally, we show with an example that it is possible for a wise process to be outside the maximal guild. Example 2. Let us consider a seven-process asymmetric quorum system QB, defined through its fail- prone system FB.
F7
One can verify that B3(FB) holds; hence, let QB be the canonical quorum system of FB.
=
{{p1, p2, p3, p4}, {p1, p2, p4, p5}, {p1, p3, p4, p5}, {p2, p3, p4, p5}}
=
{{p1, p2, p3, p5}, {p1, p2, p4, p5}, {p1, p3, p4, p5}, {p2, p3, p4, p5}}
{{P1, P2, P6, P7}}
With F = {p4, p5}, for instance, processes p1, p2, p3 and p7 are wise, p6 is na¨ıve, and Gmax = {p1, p2, p3}. It follows that process p7 is wise but outside the guild Gmax, because the unique maximal quorum in Q7 contains the na¨ıve process p6.
Lemma 7 reveals the interesting result that for an execution with a guild, each quorum of every correct process pi contains at least one process that is also in the maximal guild Gmax. Since a kernel for pi is a process set that has some member in common with every quorum of pi, this implies that Gmax contains a kernel for pi.
Corollary 8. In every execution with a guild, the maximal guild Gmax contains a kernel for every correct process. It follows that whenever all processes in the maximal guild send some particular message, then every correct process will eventually receive this message from all processes in one of its kernels. This is exploited by protocols that use kernels, such as Algorithm 4 (in Section 6).
A guild can also be seen as a set of sufficiently many wise processes that allow a protocol to make progress, in the following sense. Lemma 9. Consider an execution, in which the processes in F are faulty and let Gmax be the maximal guild for F. Let A be a superset of F that is disjoint from Gmax, i.e., F ⊆A ⊆P \ Gmax.
Then, in any execution where the processes in A fail, Gmax is also the maximal guild for A. Proof. Let Gmax be the maximal guild in an execution with set of faulty processes F ⊆P \ Gmax. By definition of a guild, Gmax contains a quorum for each of its members. This means that there exists a quorum Qi for every pi ∈Gmax such that Qi ∩F = ∅. This also implies that for every set A ⊇F, with A ⊆P \ Gmax, we have that Qi ∩A = ∅, and the lemma follows.
Given the importance of a guild for an asymmetric Byzantine quorum system, we introduce the following notion. Definition 7 (Tolerated system). Given an asymmetric Byzantine quorum system Q and an execution with faulty processes F, a set of processes T is called tolerated (by Q) if a non-empty guild G for F and Q exists such that T = P \ G.
The tolerated system T of an asymmetric Byzantine quorum system Q is the maximal collection of tolerated sets, where F ranges over all possible executions. Intuitively, the tolerated system of an asymmetric Byzantine quorum system reflects its resilience: even when all processes in a tolerated set fail, there still exists a non-empty guild. Therefore, the tolerated system characterizes the executions in which some processes will be able to operate correctly and make progress (where progress is defined by the protocol they are running). In that sense, the tolerated system of an asymmetric Byzantine quorum system can be seen as a counterpart of the fail-prone system in the symmetric model.
Notice that the tolerated system is a global notion emerging from the subjective trust choices of the participating processes; any process that knows the fail-prone and quorum systems of all processes can calculate it. We remark that the tolerated system is a central concept for composing asymmetric Byzantine quorum system, as shown by Alpos et al. .
The following lemma shows that the tolerated system T of a canonical asymmetric Byzantine quorum system is itself a symmetric fail-prone system. In particular, τ builds a connection to symmetric quorum- based protocols. This property will be used in Section 7 to construct an asymmetric common coin protocol.
Lemma 10. Let Q be an asymmetric Byzantine quorum system among processes P with asymmetric fail- prone system F = Q, i.e., such that Q is a canonical asymmetric Byzantine quorum system, and let T be the tolerated system of Q. If B3(F), then Q3(T ).
Proof. Towards a contradiction, let us assume that T does not satisfy the Q3-condition. This means that there exist T1, T2, T3 ∈T such that T1 ∪T2 ∪T3 = P. Also, let G1, G2, G3 be the corresponding guilds, i.e., G1 = P \T1, G2 = P \T2 and G3 = P \T3. By assumption, every guild contains at least one process and at least one quorum for this process is fully contained in the guild. By the consistency property of an asymmetric Byzantine quorum system, these quorums must intersect pairwise, hence the guilds also intersect pairwise. This means that there exist processes pi ∈G1 ∩G2 and pj ∈G2 ∩G3. Now, because pi is a member of G1, we can make the following reasoning: pi has a quorum Qi ∈Qi such that Qi ⊆G1, the quorum system is canonical, so pi has a fail-prone set Fi = P \ Qi ∈Fi, thus we get T1 ⊆Fi, i.e., T1 ∈Fi. With similar reasoning, we get T2 ∈Fi (because pi ∈G2), T2 ∈Fj (because pj ∈G2), and T3 ∈Fj (because pj ∈G3). But this is a contradiction because pi and pj with fail-prone sets T1, T2, and T3 violate the B3-condition in Q.
Shared Memory
This section illustrates a first application of asymmetric quorum systems: how to emulate shared memory, represented by a register. Maintaining a shared register reliably in a distributed system subject to faults is perhaps the most fundamental task for which ordinary, symmetric quorum systems have been introduced, in the models with crashes and with Byzantine faults .
Efinitions
Operations and precedence. For the particular shared-object functionalities considered here, the pro- cesses interact with an object Λ through operations provided by Λ. Operations on objects take time and are represented by two events occurring at a process, an invocation and a response. The history of an execution h consists of the sequence of invocations and responses of Λ occurring in h. An operation is complete in a history if it has a matching response.
An operation o precedes another operation o′ in a sequence of events h, denoted o Semantics. A register with domain X provides two operations: write(x), which is parameterized by a value x ∈X and outputs a token ACK when it completes; and read, which takes no parameter for invocation but outputs a value x ∈X upon completion. We consider a single-writer (or SW) register, where only a designated process pw may invoke write, and permit multiple readers (or MR), that is, every process may execute a read operation. The register is initialized with a special value x0, which is written by an imaginary write operation that occurs be- fore any process invokes operations. We consider regular semantics under concurrent access ; the extension to other forms of concurrent memory, including an atomic register, proceeds analogously. It is customary in the literature to assume pw writes every value in X at most once. Furthermore, the writer and the reader are correct; with asymmetric quorums we assume explicitly that readers and writers are wise. We illustrate below why one cannot extend the guarantees of the register to na¨ıve processes. Definition 8 (Asymmetric Byzantine SWMR regular register). A protocol emulating an asymmetric Liveness: If a wise process p invokes an operation on the register, p eventually completes the operation. Safety: Every read operation of a wise process that is not concurrent with a write returns the value written by the most recent, preceding write of a wise process; furthermore, a read operation of a wise process concurrent with a write of a wise process may also return the value that is written concurrently. In Algorithm 1, we describe a protocol for emulating a regular SWMR register with an asymmetric Byzantine quorum system, for a designated writer pw and a reader pr ∈P. The protocol uses data authentication implemented with digital signatures. This protocol is the same as the classic one of Malkhi and Reiter that uses a Byzantine dissemination quorum system and where processes send messages to each other over point-to-point links. The difference lies in the individual choices of quorums by the processes and that it ensures safety and liveness for wise processes. In more detail, every process stores a triple (ts, v, σ), which consists of a timestamp ts, a value v, and a signature σ. The idea is that the writer maintains a timestamp that increases with every write operation. The writer pw signs the timestamp/value pair and sends it in a message together with the signature to the processes, who will store the data if the timestamp within the received message is higher than the timestamp ts stored locally. A process then responds to pw with an ACK message. The change from the classic protocol is the writer pw obtains ACK messages from all processes in a quorum Qw ∈Qw for itself. The reader pr sends a READ message to all processes. It then waits to receive responses, which carry a triple of value, timestamp, and signature such that the signature is valid, from processes in a quorum Qr for pr. The returned value is the one from the triple with the highest timestamp. The function highestval(S) takes a set of timestamp/value pairs S as input and outputs the value in the pair with the largest timestamp, i.e., v such that (ts, v) ∈S and ∀(ts′, v′) ∈S : ts′ < ts∨(ts′, v′) = (ts, v). Note that this v is unique in Algorithm 1 because pw is correct. The protocol uses digital signatures, modeled by operations signi and verifyi, as introduced earlier. Theorem 11. Algorithm 1 emulates an asymmetric Byzantine SWMR regular register. Proof. First we show liveness for wise writer pw and reader pr, respectively. Since pw is wise by as- sumption, F ∈Fw∗, and by the availability condition of the quorum system there is Qw ∈Qw with F ∩Qw = ∅. Therefore, the writer will receive sufficiently many [ACK] messages and the write will Algorithm 1 Emulation of an asymmetric SWMR regular register (process pi). wts: sequence number of write operations, stored only by writer pw rid: identifier of read operations, used only by reader ts, v, σ: current state stored by pi: timestamp, value, signature wait for receiving a message [ACK] from all processes in some quorum Qw ∈Qw wait for receiving messages [VALUE, rj, tsj, vj, σj] from all processes in some Qr ∈Qr such that 16: upon receiving a message [WRITE, ts′, v′, σ′] from pw do return. As pr is wise, F ∈Fr∗, and by the analogous condition, there is Qr ∈Qr with F ∩Qr = ∅. Be- cause pw is correct and by the properties of the signature scheme, all responses from processes pj ∈Qr satisfy the checks and read returns. Regarding safety, it is easy to observe that any value output by read has been written in some preced- ing or concurrent write operation, and this even holds for na¨ıve readers and writers. This follows from the properties of the signature scheme; read verifies the signature and outputs only values with a valid signature produced by pw. We now argue that when both the writer and the reader are wise, then read outputs a value of either the last preceding write or a concurrent write and the protocol satisfies safety for a regular register. On a high level, note that F ∈Fw∗∩Fr∗since both are wise. So if pw writes to a quorum Qw ∈Qw and pr reads from a quorum Qr ∈Qr, then by consistency of the quorum system Qw ∩Qr̸ ⊆F because pw from pw and returns it to pr. Example 3. We show why the guarantees of this protocol with asymmetric quorums hold only for wise readers and writers. Consider QA from the last section and an execution in which p2 and p4 are faulty, and therefore p1 is na¨ıve and p3 and p5 are wise. A quorum for p1 consists of p1 and three processes in {p2, . . , p5}; moreover, every process set that contains p3, one of {p1, p2} and one of {p4, p5} is a quorum for p3. We illustrate that if na¨ıve p1 writes, then a wise reader p3 may violate safety. Suppose that all correct processes, especially p3, store timestamp/value/signature triples from an operation that has terminated and that wrote x. When p1 invokes write(u), it obtains [ACK] messages from all processes except p3. This is a quorum for p1. Then p3 runs a read operation and receives the outdated values representing x from itself (p3 is correct but has not been involved in writing u) and also from the faulty p2 and p4. Hence, p3 outputs x instead of u. Analogously, with the same setup of every process initially storing a representation of x but with wise p3 as writer, suppose p3 executes write(u). It obtains [ACK] messages from p2, p3, and p4 and terminates. When p1 subsequently invokes read and receives values representing x, from correct p1 and p5 and from faulty p2 and p4, then p1 outputs x instead of y and violates safety as a na¨ıve reader. Since the sample operations are not concurrent, the implication actually holds also for registers with only safe semantics. This section describes a second protocol emulating an asymmetric Byzantine SWMR regular register. In contrast to the previous protocol, it does not use digital signatures for authenticating the data to the reader. Our algorithm generalizes the construction of Abraham et al. and also assumes that only a finite number of write operations occur (FW-termination). Furthermore, this algorithm illustrates the use of asymmetric core-set systems in the context of an asymmetric-trust protocol. This protocol extends Algorithm 1 and every process stores the most recently written timestamp- value pair (ts, v). Every write operation performs two rounds instead of one, a pre-write round and a write round. In addition to the previous protocol, every process stores the most recently pre-written timestamp-value pair (pts, pv). From the perspective of the writer pw, each round proceeds like the single round in Algorithm 1, except that pw does not produce a digital signature. In particular, pw waits in each round for responses that form a quorum Qw ∈Qw for itself. The reader pr exchanges one round of messages with the processes and waits for responses that form a quorum Qr ∈Qr for pr. Every response contains the pre-written and the written timestamp- value pairs from the sending process. The reader collects these in an array readlist until the following condition is satisfied. A pair (ts∗, v∗), a core set Cr for pr of entries in readlist, and a quorum Qr for pr of entries in readlist exist such that (1) the pair (ts∗, v∗) is either the pre-written or the written pair in all entries of readlist in Cr; and (2) (ts∗, v∗) is the pair with the highest timestamp among the entries in Qr. Intuitively, the initial pre-write round and the core set Cr that reports this value to pr replace the step of authenticating the value through a digital signature. This respects safety because Cr, for a wise pr, contains at least one correct process that has not altered the value. The full protocol appears in Algorithm 2. Theorem 12. Algorithm 2 emulates an asymmetric Byzantine SWMR regular register, provided there are only finitely many write operations. Proof. We first establish safety when the writer pw and the reader pr are wise. In that case, F ∈Fw∗∩ Fr∗. During in a write operation, pw has received PREACK and ACK messages from Qw ∈Qi and w ∈Qi, respectively, and for all Qr ∈Qr it holds that Qw ∩Qr̸ ⊆F and Q′ w ∩Qr̸ ⊆F. We now argue that any pair (ts∗, v∗) returned by pr was written by pw either in a preceding or a concurrent write. From the properties of the core set Cr, because pr is wise, and together with the condition that (ts∗, v∗) satisfies, it follows that at least one correct process exists in Cr that stores (ts∗, v∗) as a pre-written or as a written value. Thus, the pair was written by pw before. Next we argue that for every completed write(v∗) operation, in which pw has sent [WRITE, wts, v∗], and for any subsequent read operation that selects (ts∗, v∗) and returns v∗, it must hold wts ≤ts∗. Namely, the condition on Qr implies that ts∗≥tsk for all pk ∈Qr. By the consistency of the quorum w ∩Qr that has sent tsℓto pr. Then ts∗≥tsℓ≥wts follows because the timestamp variable of pℓonly increases. The combination of the above two paragraphs implies that for read operations that are not concurrent with any write, the pair (ts∗, v∗) chosen by read was actually written in the immediately preceding write. If the read operation occurs concurrently with a write, then the pair (ts∗, v∗) chosen by read may also originate from the concurrent write. This establishes the safety property of the SWMR regular register. We now show liveness. First, if pw is wise, then there exists a quorum Qw ∈Qw such that Qw ∩ F = ∅. Second, any correct process will eventually receive all [PREWRITE, wts, v] and [WRITE, wts, v] messages sent by pw and process them in the correct order by the assumption of FIFO links. This means that pw will receive [PREACK] and [ACK] messages, respectively, from all processes in one of its quorums, since at least the processes in Qw will eventually send those. Algorithm 2 Double-write emulation of an asymmetric SWMR regular register (process pi). pts, pv, ts, v: current state stored by pi: pre-written timestamp and value, written timestamp and value wait for receiving a message [PREACK] from all processes in some quorum Qw ∈Qw 14: upon receiving a message [VALUE, rj, ptsj, pvj, tsj, vj] from pj such that if there exist ts∗, v∗, a core set Cr ∈Cr for pr, and a quorum Qr ∈Qr for pr such thatSwmr Regular Register Satisfies:
Protocol With Authenticated Data
:
:
:
:
:
Return Highestval({(Tsj, Vj)|J ∈Qr})
Send Message [Value, R, Ts, V, Σ] To Pr
Ouble-Write Protocol Without Data Authentication
Q′
W ∩Qr̸ ⊆F, So There Is A Correct Process Pℓ∈Q′
:
:
Send Message [Read, Rid] To All Pj ∈P
:
∧