Cross Silo Federated Learning | AI Final Year Project Report

Complete Technical Paper Content — Cross Silo Federated Learning

1. Executive Abstract, Problem Formulation & Background

Interoperability is a characteristic of a product or system to work with other products or systems. While the term was initially defined for information technology or systems engineering services to allow for information exchange, a broader definition takes into account social, political, and organizational factors that impact system-to-system performance. Types of interoperability include syntactic interoperability, where two systems can communicate with each other, and cross-domain interoperability, where multiple organizations work together and exchange information.

If two or more systems use common data formats and communication protocols, then they are capable of communicating with each other, and they exhibit syntactic interoperability. XML and SQL are examples of common data formats and protocols. Low-level data formats also contribute to syntactic interoperability, ensuring that alphabetical characters are stored in the same ASCII or a Unicode format in all the communicating systems. Beyond the ability of two or more computer systems to exchange information, semantic interoperability is the ability to automatically interpret the information exchanged meaningfully and accurately in order to produce useful results as defined by the end users of both systems. To achieve semantic interoperability, both sides must refer to a common information exchange reference model. The content of the information exchange requests are unambiguously defined: what is sent is the same as what is understood. Cross-domain interoperability involves multiple social, organizational, political, legal entities working together for a common interest or information exchange.

Closed feature set (no extensions or subsetting): The specification must be normative and bounded. The standard must state that a compliant implementation supports the entirety of the base standard and nothing else. The standard must not allow vendor-specific extensions. The standard may define allowable subsets, but must exclude vendor-specific or vendor-defined subsetting. Later versions of a standard need not be 100% backward compatible, but deviations are precisely specified. Deterministic Output: The standard must define explicit and invariant requirements around input-to-output mappings. The standard must permit no “implementation defined” behaviors. The standard may permit implementations to set limits on maximum size, capacity, etc., but must require uniform, conforming behaviors within those limits. The standard document uses the word "shall" essentially exclusively, and "may" or "should" in only rare cases (for example, error handling that must be delegated to an external system). For standards governing real-time processes, Exhaustive Error Handling: Rather than leaving errors to "implementation-defined" logic, the standard must mandate exact error codes, state transitions, and fallback behaviors to maintain system stability. Platform & Vendor Neutrality: The definitions must remain divorced from specific internal system architectures, programming languages, or operating environments. Reference implementation: while it is not essential to specify a reference implementation, that is common. The standard may provide for resolution of ambiguities or omissions to ensure uniform implementation. The standard may include a test suite for conformance testing.

2. Related Work, Literature Survey & Algorithmic Foundations

Open standards rely on a broadly consultative and inclusive group, including representatives from vendors, academics and others holding a stake in the development, that discusses and debates the technical and economic merits, demerits and feasibility of a proposed common protocol. After the doubts and reservations of all members are addressed, the resulting common document is endorsed as a common standard. This document may be subsequently released to the public, and henceforth becomes an open standard. It is usually published and is available freely or at a nominal cost to any and all comers, with no further encumbrances. Various vendors and individuals (even those who were not part of the original group) can use the standards document to make products that implement the common protocol defined in the standard and are thus interoperable by design, with no specific liability or advantage for customers for choosing one product over another on the basis of standardized features. The vendors' products compete on the quality of their implementation, user interface, ease of use, performance, price, and a host of other factors, while keeping the customer's data intact and transferable even if they choose to switch to another competing product for business reasons. However, while "open" is a necessary condition for an interoperabilty standard, it is not a sufficient condition. For example, the standard-setting processes for the Fortran, C, and C++ languages are open, with a broadly-inclusive consultative group. But the resulting standards are not interoperability standards.

Post facto interoperability may be the result of the absolute market dominance of a particular product in contravention of any applicable standards, or if any effective standards were not present at the time of that product's introduction. The vendor behind that product can then choose to ignore any forthcoming standards and not cooperate in any standardization process at all, using its near-monopoly to insist that its product sets the de facto standard by its very market dominance. This is not a problem if the product's implementation is open and minimally encumbered, but it may well be both closed and heavily encumbered (e.g., by patent claims). Because of the network effect, achieving interoperability with such a product is both critical for any other vendor if it wishes to remain relevant in the market, and difficult to accomplish because of lack of cooperation on equal terms with the original vendor, who may well see the new vendor as a potential competitor and threat. The newer implementations often rely on clean-room reverse engineering in the absence of technical data to achieve interoperability. The original vendors may provide such technical data to others, often in the name of encouraging competition, but such data is invariably encumbered and may be of limited use. Availability of such data is not equivalent to an open standard, because:

The data is provided by the original vendor on a discretionary basis, and the vendor has every interest in blocking the effective implementation of competing solutions, and may subtly alter or change its product, often in newer revisions, so that competitors' implementations are almost, but not quite completely interoperable, leading customers to consider them unreliable or of lower quality. These changes may not be passed on to other vendors at all, or passed on after a strategic delay, maintaining the market dominance of the original vendor. The data itself may be encumbered, e.g., by patents or pricing, leading to a dependence of all competing solutions on the original vendor, and possibly leading to a revenue stream from the competitors' customers back to the original vendor. This revenue stream is the result of the original product's market dominance and not a result of any innate superiority. Even when the original vendor is genuinely interested in promoting healthy competition (so that he may also benefit from the resulting innovative market), post-facto interoperability may often be undesirable as many defects or quirks can be directly traced back to the original implementation's technical limitations. Although in an open process, anyone may identify and correct such limitations, and the resulting cleaner specification may be used by all vendors, this is more difficult post-facto, as customers already have valuable information and processes encoded in the faulty but dominant product, and other vendors are forced to replicate those faults and quirks for the sake of preserving interoperability even if they could design better solutions. Alternatively, it can be argued that even open processes are subject to the weight of past implementations and imperfect past designs and that the power of the dominant vendor to unilaterally correct or improve the system and impose the changes to all users facilitates innovation. Lack of an open standard can also become problematic for the customers, as in the case of the original vendor's inability to fix a certain problem that is an artifact of technical limitations in the original product. The customer wants that fault fixed, but the vendor has to maintain that faulty state, even across newer revisions of the same product, because that behavior is a de facto standard and many more customers would have to pay the price of any interoperability issues caused by fixing the original problem and introducing new behavior.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

Force interoperability is defined in NATO as the ability of the forces of two or more nations to train, exercise and operate effectively together in the execution of assigned missions and tasks. Additionally, NATO defines interoperability more generally as the ability to act together coherently, effectively and efficiently to achieve Allied tactical, operational and strategic objectives. At the strategic level, interoperability is an enabler for coalition building. It facilitates meaningful contributions by coalition partners. At this level, interoperability issues center on harmonizing world views, strategies, doctrines, and force structures. Interoperability is an element of coalition willingness to work together over the long term to achieve and maintain shared interests against common threats. Interoperability at the operational and tactical levels is where strategic interoperability and technological interoperability come together to help allies shape the environment, manage crises, and win wars. The benefits of interoperability at the operational and tactical levels generally derive from the interchangeability of force elements and units. Technological interoperability reflects the interfaces between organizations and systems. It focuses on communications and computers but also involves the technical capabilities of systems and the resulting mission compatibility between the systems and data of coalition partners. At the technological level, the benefits of interoperability come primarily from their impacts at the operational and tactical levels in terms of enhancing flexibility.

Desktop interoperability is a subset of software interoperability. In the early days, the focus of interoperability was to integrate web applications with other web applications. Over time, open-system containers were developed to create a virtual desktop environment in which these applications could be registered and then communicate with each other using simple publish–subscribe patterns. Rudimentary UI capabilities were also supported, allowing windows to be grouped with other windows. Today, desktop interoperability has evolved into full-service platforms that include container support, basic exchange between web and web, but also native support for other application types and advanced window management. The very latest interop platforms also include application services such as universal search, notifications, user permissions and preferences, 3rd party application connectors and language adapters for in-house applications.

With respect to software, the term interoperability is used to describe the capability of different programs to exchange data via a common set of exchange formats, to read and write the same file formats, and to use the same communication protocols. The lack of interoperability can be a consequence of a lack of attention to standardization during the design of a program. Indeed, interoperability is not taken for granted in the non-standards-based portion of the computing world. According to ISO/IEC 2382-01, Information Technology Vocabulary, Fundamental Terms, interoperability is defined as follows: "The capability to communicate, execute programs, or transfer data among various functional units in a manner that requires the user to have little or no knowledge of the unique characteristics of those units". Standards-developing organizations provide open public software specifications to facilitate interoperability; examples include the Oasis-Open organization and buildingSMART (formerly the International Alliance for Interoperability). Another example of a neutral party is the RFC documents from the Internet Engineering Task Force (IETF). The Open Service for Lifecycle Collaboration community is working on finding a common standard in order that software tools can share and exchange data e.g., bugs, tasks, requirements etc. The final goal is to agree on an open standard for interoperability of open source application lifecycle management tools. Java is an example of an interoperable programming language that allows for programs to be written once and run anywhere with a Java virtual machine. A program in Java, so long as it does not use system-specific functionality, will maintain interoperability with all systems that have a Java virtual machine available. Applications will maintain compatibility because, while the implementation is different, the underlying language interfaces are the same.

4. Neural Network Backbone, Layer Specifications & Loss Formulations

Product testing Products produced to a common standard, or to a sub-profile thereof, depend on the clarity of the standards, but there may be discrepancies in their implementations that system or unit testing may not uncover. This requires that systems formally be tested in a production scenario – as they will be finally implemented – to ensure they actually will intercommunicate as advertised, i.e., they are interoperable. Interoperable product testing is different from conformance-based product testing as conformance to a standard does not necessarily engender interoperability with another product that is also tested for conformance. Product engineering Implements the common standard, or a sub-profile thereof, as defined by the industry and community partnerships with the specific intention of achieving interoperability with other software implementations, also following the same standard or sub-profile thereof. Industry and community partnership Industry and community partnerships, either domestic or international, sponsor standard workgroups with the purpose of defining a common standard that may be used to allow software systems to intercommunicate for a defined purpose. At times, an industry or community will sub-profile an existing standard produced by another organization to reduce options and thus make interoperability more achievable for implementations. Common technology and intellectual property The use of a common technology or intellectual property may speed up and reduce the complexity of interoperability by reducing variability between components from different sets of separately developed software products and thus allowing them to intercommunicate more readily. This technique has some of the same technical results as using a common vendor product to produce interoperability. The common technology can come through third-party libraries or open-source developments. Standard implementation Software interoperability requires a common agreement that is normally arrived at via an industrial, national or international standard. Each of these has an important role in reducing variability in intercommunication software and enhancing a common understanding of the end goal to be achieved.

Interoperability tends to be regarded as an issue for experts and its implications for daily living are sometimes underrated. The European Union Microsoft competition case shows how interoperability concerns important questions of power relationships. In 2004, the European Commission found that Microsoft had abused its market power by deliberately restricting interoperability between Windows Workgroup servers and non-Microsoft work group servers. By doing so, Microsoft was able to protect its dominant market position for work group server operating systems, the heart of corporate IT networks. Microsoft was ordered to disclose complete and accurate interface documentation, which could enable rival vendors to compete on an equal footing (the interoperability remedy). Interoperability has also surfaced in the software patent debate in the European Parliament (June–July 2005). Critics claim that because patents on techniques required for interoperability are kept under RAND (reasonable and non-discriminatory licensing) conditions, customers will have to pay license fees twice: once for the product and, in the appropriate case, once for the patent-protected program the product uses.

Railways have greater or lesser interoperability depending on conforming to standards of gauge, couplings, brakes, signalling, loading gauge, and structure gauge to mention a few parameters. For passenger rail service, different railway platform height and width clearance standards may also affect interoperability. North American freight and intercity passenger railroads are highly interoperable, but systems in Europe, Asia, Africa, Central and South America, and Australia are much less so. The parameter most difficult to overcome (at reasonable cost) is incompatibility of gauge, though variable gauge axle systems can be used on rolling stock.

5. Training Protocol, Convergence Behavior & Experimental Evaluation

The ability to provide services to and accept services from other systems, and to use the services exchanged to enable them to operate effectively together. ITU-T provides standards for international telecommunications. The condition achieved among communications-electronics systems or items of communications-electronics equipment when information or services can be exchanged directly and satisfactorily between them or their users. The degree of interoperability should be defined when referring to specific cases. In two-way radio, interoperability is composed of three dimensions:

compatible communications paths (compatible frequencies, equipment and signaling), radio system coverage or adequate signal strength, and; scalable capacity.

The Soviet Union, officially the Union of Soviet Socialist Republics (USSR), was a transcontinental country that spanned much of Eurasia from its formation in 1922 until its dissolution in 1991. It was the world's third-most populous country, the largest by area, and bordered twelve other countries. A diverse multinational state, it was organized as a federal union of national republics, with the largest and most populous being the Russian Soviet Federative Socialist Republic (SFSR). The world's flagship communist state, it was governed by the Communist Party under Marxism–Leninism, through soviet councils. The Soviet economy was also centrally planned. Its capital and largest city was Moscow. The Soviet Union's roots lay in the October Revolution of 1917. The new government, led by Vladimir Lenin, established the Russian SFSR, the world's first constitutionally communist state. Following the Bolshevik victory in the Russian Civil War, the Russian SFSR and its subordinate republics were merged into the Soviet Union in 1922. Following Lenin's death in 1924, a six-year power struggle saw Joseph Stalin emerge as the country's undisputed personalist ruler. He initiated rapid industrialization and forced collectivization that led to significant growth, but contributed to a devastating famine in the 1930s that killed millions. The Soviet forced labour camp system, known as the Gulag, was expanded. During the late 1930s, Stalin's government conducted the Great Purge to eliminate opposition, resulting in deportations, executions, and the Moscow trials. Failing to build an anti-Nazi coalition in Europe, the Soviet Union signed a non-aggression pact with Nazi Germany in 1939. However, in 1941, Germany invaded the Soviet Union in the largest land invasion in history, opening the Eastern Front of World War II. The Soviet Union played a decisive role in defeating the Axis powers as part of the Allies, while extending its sphere of influence to Central and Eastern Europe. With around 27 million casualties, it suffered the most deaths of any country in World War II. In the war's aftermath, the Soviet Union consolidated the territories occupied by the Red Army into satellite states and undertook rapid economic development, cementing its status as a superpower. Geopolitical tensions with the United States led to the Cold War. The US-led Western Bloc coalesced into the NATO military alliance in 1949, prompting the Eastern Bloc to form the Warsaw Pact in 1955. With little direct combat, the blocs engaged in ideological and proxy wars. In 1953, following Stalin's death, Nikita Khrushchev initiated a campaign of de-Stalinization, which led to ideological tensions with communist China, under Mao Zedong, and culminated in an acrimonious split. The Soviet military suppressed uprisings in East Germany, Hungary and Czechoslovakia, while the resolution of the Cuban Missile Crisis narrowly averted a global conflict. Under Leonid Brezhnev, prosperity shifted toward stagnation, although relations with the US eased. In 1985, Mikhail Gorbachev sought reform through his policies of glasnost and perestroika. In 1989, most Warsaw Pact countries overthrew their Soviet-backed regimes, effectively ending the Eastern Bloc. Nationalist movements across the Soviet republics declared sovereignty, and in 1991, a successful referendum to establish a renewed federation was followed by a failed coup by hardliners. This prompted Russia, Ukraine, and Belarus to secede. On 26 December, Gorbachev officially recognized the dissolution of the Soviet Union. Boris Yeltsin, president of the Russian SFSR, oversaw its reconstitution into the Russian Federation, the Soviet Union's successor state. All fifteen republics became independent states; all except the Baltic states joined the Commonwealth of Independent States. The post-Soviet states experienced a humanitarian disaster and continuing wars and insurgencies. The Soviet Union was one of the world's two superpowers, with hegemony in Eastern Europe, global diplomacy, ideological influence (particularly in the Global South), military might, economic strength, and scientific and technological accomplishments. Its space program made significant achievements in the Space Race. It had the world's second-largest economy and the largest standing military. As a nuclear state, it wielded the largest arsenal of nuclear weapons in the world. As an Allied nation, it was a founding member of the United Nations and one of five permanent members of its Security Council.