Complete Technical Paper Content — Horizontal Federated Learning
1. Executive Abstract, Problem Formulation & Background
Federated learning (also known as collaborative learning) is a machine learning technique in a setting where multiple entities (often called clients) collaboratively train a model while keeping their data decentralized, rather than centrally stored. A defining characteristic of federated learning is data heterogeneity. Because client data is decentralized, data samples held by each client may not be independently and identically distributed. Federated learning is generally concerned with and motivated by issues such as data privacy, data minimization, and data access rights. Its applications involve a variety of research areas including defence, telecommunications, the Internet of things, and pharmaceuticals.
f ( x 1 , … , x K ) = 1 K ∑ i = 1 K f i ( x i ) {\displaystyle f(\mathbf {x} _{1},\dots ,\mathbf {x} _{K})={\dfrac {1}{K}}\sum _{i=1}^{K}f_{i}(\mathbf {x} _{i})}
where K {\displaystyle K} is the number of nodes, x i {\displaystyle \mathbf {x} _{i}} are the weights of model as viewed by node i {\displaystyle i} , and f i {\displaystyle f_{i}} is node i {\displaystyle i} 's local objective function, which describes how model weights x i {\displaystyle \mathbf {x} _{i}} conforms to node i {\displaystyle i} 's local dataset. The goal of federated learning is to train a common model on all of the nodes' local datasets, in other words:
Optimizing the objective function f ( x 1 , … , x K ) {\displaystyle f(\mathbf {x} _{1},\dots ,\mathbf {x} _{K})} . Achieving consensus on x i {\displaystyle \mathbf {x} _{i}} . In other words, x 1 , … , x K {\displaystyle \mathbf {x} _{1},\dots ,\mathbf {x} _{K}} converge to some common x {\displaystyle \mathbf {x} } at the end of the training process.
Initialization: according to the server inputs, a machine learning model (e.g., linear regression, neural network, boosting) is chosen to be trained on local nodes and initialized. Then, nodes are activated and wait for the central server to give the calculation tasks. Client selection: a fraction of local nodes are selected to start training on local data. The selected nodes acquire the current statistical model while the others wait for the next federated round. Configuration: the central server orders selected nodes to undergo training of the model on their local data in a pre-specified fashion (e.g., for some mini-batch updates of gradient descent). Reporting: each selected node sends its local model to the server for aggregation. The central server aggregates the received models and sends back the model updates to the nodes. It also handles failures for disconnected nodes or lost model updates. The next federated round is started returning to the client selection phase. Termination: once a pre-defined termination criterion is met (e.g., a maximum number of iterations is reached or the model accuracy is greater than a threshold) the central server aggregates the updates and finalizes the global model. The procedure considered before assumes synchronized model updates. Recent federated learning developments introduced novel techniques to tackle asynchronicity during the training process, or training with dynamically varying models. Compared to synchronous approaches where local models are exchanged once the computations have been performed for all layers of the neural network, asynchronous ones leverage the properties of neural networks to exchange model updates as soon as the computations of a certain layer are available. These techniques are also commonly referred to as split learning and they can be applied both at training and inference time regardless of centralized or decentralized federated learning settings.
Covariate shift: local nodes may store examples that have different statistical distributions compared to other nodes. An example occurs in natural language processing datasets where people typically write the same digits/letters with different stroke widths or slants. Prior probability shift: local nodes may store labels that have different statistical distributions compared to other nodes. This can happen if datasets are regional and/or demographically partitioned. For example, datasets containing images of animals vary significantly from country to country. Concept drift (same label, different features): local nodes may share the same labels but some of them correspond to different features at different local nodes. For example, images that depict a particular object can vary according to the weather condition in which they were captured. Concept shift (same features, different labels): local nodes may share the same features but some of them correspond to different labels at different local nodes. For example, in natural language processing, the sentiment analysis may yield different sentiments even if the same text is observed. Unbalanced: the amount of data available at the local nodes may vary significantly in size. The loss in accuracy due to non-iid data can be bounded through using more sophisticated means of doing data normalization, rather than batch normalization.
Number of federated learning rounds: T {\displaystyle T}
Total number of nodes used in the process: K {\displaystyle K}
Fraction of nodes used at each iteration for each node: C {\displaystyle C}
Local batch size used at each learning iteration: B {\displaystyle B}
2. Related Work, Literature Survey & Algorithmic Foundations
Number of iterations for local training before pooling: N {\displaystyle N}
Those parameters have to be optimized depending on the constraints of the machine learning application (e.g., available computing power, available memory, bandwidth). For instance, stochastically choosing a limited fraction C {\displaystyle C} of nodes for each iteration diminishes computing cost and may prevent overfitting, in the same way that stochastic gradient descent can reduce overfitting.
Heterogeneity between the different local datasets: each node may have some bias with respect to the general population, and the size of the datasets may vary significantly; Temporal heterogeneity: each local dataset's distribution may vary with time; Interoperability of each node's dataset is a prerequisite; Each node's dataset may require regular curations; Hiding training data might allow attackers to inject backdoors into the global model; Lack of access to global training data makes it harder to identify unwanted biases entering the training e.g. age, gender, sexual orientation; Partial or total loss of model updates due to node failures affecting the global model; Lack of annotations or labels on the client side. Heterogeneity between processing platforms
A further governance-related issue concerns the distribution of value among participants as multi-party federated learning systems scale. As more clients contribute to training, model accuracy may approach a performance threshold, after which additional contributions yield diminishing marginal utility. This dynamic can create tensions within collaborative arrangements, as early participants may perceive later contributions as offering reduced value while still benefiting from the shared model. Such asymmetries may affect incentives to join or remain in a consortium and complicate the design of fair participation, ownership, or reward structures. Establishing transparent rules for contribution assessment and benefit allocation is therefore an important organisational challenge and represents a direction for future research.
HyFDCA is a provably convergent primal-dual algorithm for hybrid FL in at least the following settings. Hybrid Federated Setting with Complete Client Participation Horizontal Federated Setting with Random Subsets of Available Clients The authors show HyFDCA enjoys a convergence rate of O(1⁄t) which matches the convergence rate of FedAvg (see below). Vertical Federated Setting with Incomplete Client Participation The authors show HyFDCA enjoys a convergence rate of O(log(t)⁄t) whereas FedBCD exhibits a slower O(1⁄sqrt(t)) convergence rate and requires full client participation. HyFDCA provides the privacy steps that ensure privacy of client data in the primal-dual setting. These principles apply to future efforts in developing primal-dual algorithms for FL. HyFDCA empirically outperforms HyFEM and FedAvg in loss function value and validation accuracy across a multitude of problem settings and datasets (see below for more details). The authors also introduce a hyperparameter selection framework for FL with competing metrics using ideas from multiobjective optimization. There is only one other algorithm that focuses on hybrid FL, HyFEM proposed by Zhang et al. (2020). This algorithm uses a feature matching formulation that balances clients building accurate local models and the server learning an accurate global model. This requires a matching regularizer constant that must be tuned based on user goals and results in disparate local and global models. Furthermore, the convergence results provided for HyFEM only prove convergence of the matching formulation not of the original global problem. This work is substantially different from HyFDCA's approach which uses data on local clients to build a global model that converges to the same solution as if the model was trained centrally. Furthermore, the local and global models are synchronized and do not require the adjustment of a matching parameter between local and global models. However, HyFEM is suitable for a vast array of architectures including deep learning architectures, whereas HyFDCA is designed for convex problems like logistic regression and support vector machines. HyFDCA is empirically benchmarked against the aforementioned HyFEM as well as the popular FedAvg in solving convex problem (specifically classification problems) for several popular datasets (MNIST, Covtype, and News20). The authors found HyFDCA converges to a lower loss value and higher validation accuracy in less overall time in 33 of 36 comparisons examined and 36 of 36 comparisons examined with respect to the number of outer iterations. Lastly, HyFDCA only requires tuning of one hyperparameter, the number of inner iterations, as opposed to FedAvg (which requires tuning three) or HyFEM (which requires tuning four). In addition to FedAvg and HyFEM being quite difficult to optimize hyperparameters in turn greatly affecting convergence, HyFDCA's single hyperparameter allows for simpler practical implementations and hyperparameter selection methodologies.
Google Cloud is a suite of cloud computing services offered by Google that provides a series of modular cloud services including computing, data storage, data analytics, and machine learning, alongside a set of management tools. It runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, and Google Docs, according to Verma et al. Registration requires a credit card or bank account details. Google Cloud provides infrastructure as a service, platform as a service, and serverless computing environments. In April 2008, Google announced App Engine, a platform for developing and hosting web applications in Google-managed data centers, which was the first cloud computing service from the company. The service became generally available in November 2011. Since the announcement of App Engine, Google added multiple cloud services to the platform. Google Cloud includes Google Cloud public cloud infrastructure, as well as Google Workspace (G Suite), enterprise versions of Android and ChromeOS, and application programming interfaces (APIs) for machine learning and enterprise mapping services. Since at least 2022, Google's official materials have stated that "Google Cloud" is the new name for "Google Cloud Platform," which may cause naming confusion.
For those familiar with other notable cloud service providers, a comparison of similar services may be helpful in understanding Google Cloud Platform's offerings.
April 2008 – Google App Engine announced in preview May 2010 – Google Cloud Storage launched May 2010 – Google BigQuery and Prediction API announced in preview October 2011 – Google Cloud SQL is announced in preview June 2012 – Google Compute Engine is launched in preview May 2013 – Google Compute Engine is released to GA August 2013 – Cloud Storage begins automatically encrypting each Storage object's data and metadata under the 128-bit Advanced Encryption Standard (AES-128), and each encryption key is itself encrypted with a regularly rotated set of master keys February 2014 – Google Cloud SQL becomes GA May 2014 – Stackdriver is acquired by Google June 2014 – Kubernetes is announced as an open source container manager June 2014 – Cloud Dataflow is announced in preview October 2014 – Google acquires Firebase November 2014 – Alpha release Google Kubernetes Engine (formerly Container Engine) is announced January 2015 – Google Cloud Monitoring based on Stackdriver goes into Beta March 2015 – Google Cloud Pub/Sub becomes available in Beta April 2015 – Google Cloud DNS becomes generally available April 2015 – Google Dataflow launched in beta July 2015 – Google releases v1 of Kubernetes; Hands it over to The Cloud Native Computing Foundation August 2015 – Google Cloud Dataflow, Google Cloud Pub/Sub, Google Kubernetes Engine, and Deployment Manager graduate to GA November 2015 – Bebop is acquired, and Diane Greene joins Google February 2016 – Google Cloud Functions becomes available in Alpha September 2016 – Apigee, a provider of application programming interface (API) management company, is acquired by Google September 2016 – Stackdriver becomes generally available November 2016 – Qwiklabs, an EdTech company is acquired by Google February 2017 – Cloud Spanner, highly available, globally-distributed database is released into Beta March 2017 – Google acquires Kaggle, world's largest community of data scientists and machine learning enthusiasts April 2017 – MIT professor Andrew Sutherland breaks the record for the largest ever Compute Engine cluster with 220,000 cores on Preemptible VMs. May 2017 – Google Cloud IoT Core is launched in Beta November 2017 – Google Kubernetes Engine gets certified by the CNCF February 2018 – Google Cloud IoT Core becomes generally available February 2018 – Google announces its intent to acquire Xively February 2018 – Cloud TPUs, ML accelerators for Tensorflow, become available in Beta May 2018 – Google Cloud Memorystore becomes available in Beta April 2019 – Google Cloud Run (fully managed) Beta release April 2019 – Google Anthos announced November 2019 – Google Cloud Run (fully managed) General availability release March 2020 – Due to the COVID-19 pandemic, Google Cloud postponed the online streaming version of its Google Cloud Next mega-conference, two weeks after it canceled the in-person version. October 2020 – Google Cloud announced that it will become a block producer candidate for the EOS network and EOS.IO protocol. Currently the top block producers are cryptocurrency exchanges like OKEx and Binance. February 2021 – Google Kubernetes Engine Autopilot is introduced. May 2021 – Vertex AI announced at Google.io June 2021 – In 2021, Apple was Google Cloud's biggest customer. April 2022 – MobiledgeX acquired and joins Google Cloud. March 2023 – Google brings generative AI capabilities to Google Cloud. May 2024 Google Cloud partnered with Airtel. April 2025 – At Google Cloud Next 2025 in Las Vegas, Google announced the seventh-generation Tensor Processing Unit, named Ironwood, claiming 3,600 times better performance than its first publicly available TPU. Google also announced Cloud Wide Area Network (Cloud WAN), making its private global network available to enterprise customers, and introduced Gemini 2.5 and Gemini 2.5 Flash reasoning models. April 2026 – At Google Cloud Next '26, Sundar Pichai announced expanded AI infrastructure investments, including the introduction of Trillium TPUs (8th generation) and the Gemini Enterprise Agent Platform, with Google reporting that 75% of new internal code was AI-generated.
Federated_Learning_Horizontal An Implementation of the Federated Averaging Algorithm as described in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data by H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas Description This repository contains code to implement the Federated Averaging Algorithm as described in in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data. The code consists of: FederatedAveraging_Android : An Android Application that 1. Uses the TensorFlow for Java API to perform On-Device Training 2. Sends weights to the Server as well as receives model from the server FederatedAveragingServer : A Web Application that: 1. Receives model weights from various devices 2. Uses the FederatedAveraging algorithm to aggregate the weights and update the global model. 3. Push the weights to the client devices
An Implementation of the Federated Averaging Algorithm as described in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data by H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
This repository contains code to implement the Federated Averaging Algorithm as described in in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data. The code consists of:
An Implementation of the Federated Averaging Algorithm as described in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data by H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas
Federated_Learning_Horizontal An Implementation of the Federated Averaging Algorithm as described in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data by H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas Description This repository contains code to implement the Federated Averaging Algorithm as described in in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data. The code consists of: FederatedAveraging_Android : An Android Application that 1. Uses the TensorFlow for Java API to perform On-Device Training 2. Sends weights to the Server as well as receives model from the server FederatedAveragingServer : A Web Application that: 1. Receives model weights from various devices 2. Uses the FederatedAveraging algorithm to aggregate the weights and update the global model. 3. Push the weights to the client devices
An Implementation of the Federated Averaging Algorithm as described in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data by H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas
This repository contains code to implement the Federated Averaging Algorithm as described in in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data. The code consists of:
An Implementation of the Federated Averaging Algorithm as described in the Paper - Communication-Efficient Learning of Deep Networks from Decentralized Data by H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas
Horizontal Federated Learning In this project, an Erlang Communication Layer and a Java Web Server were realized for the support of Horizontal Federated Learning Techniques, with a particular focus on KMeans as the target Machine Learning Algorithm. Distributed Software Architecture The software architecture of the Erlang nodes is layered as described in the following diagram. The supervisor process and the server process will be hosted in the same node, while the client processes can be hosted in the same or different nodes. One pyrlang process will be available in each of the client nodes. Pyrlang is a Python library that implements the Erlang distribution protocol and creates an Erlang-compatible node in your Erlang cluster. The supervisor is in charge of supervision & control for a designated experiment. The server is in charge of monitoring the execution of rounds. The client process will be in charge of executing the algorithm rounds. The project architecture considering both the Java Web Server and the distributed Erlang nodes is described in the following diagram. The web application uses the three-tier architecture: Presentation Layer. This layer interacts with the client and it’s built using JSPs, HTML, CSS, and Javascript. Logic Layer. This layer contains the logic of the application. One part of this layer is written in Java and uses a Tomcat Server, while the other is written using Erlang. Data Access Layer. This layer consists of a Key/Value Repository and the corresponding DBMS. Project Structure The project is organized as follows: ErlangCommunicationLayer/ contains the source code of the Erlang nodes (supervisor, server, client) JavaWebServer/ contains the source code of the Java Web Server More information can be found in the documentation file attached. Execution Proof We've proven that our algorithm is able to train in a distributed and privacy-oriented fashion the K-Means algorithm, reaching the same result that it could reach if it was centralized.
In this project, an Erlang Communication Layer and a Java Web Server were realized for the support of Horizontal Federated Learning Techniques, with a particular focus on KMeans as the target Machine Learning Algorithm.
The software architecture of the Erlang nodes is layered as described in the following diagram. The supervisor process and the server process will be hosted in the same node, while the client processes can be hosted in the same or different nodes. One pyrlang process will be available in each of the client nodes. Pyrlang is a Python library that implements the Erlang distribution protocol and creates an Erlang-compatible node in your Erlang cluster.
We've proven that our algorithm is able to train in a distributed and privacy-oriented fashion the K-Means algorithm, reaching the same result that it could reach if it was centralized.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
An Erlang communication layer and a Java web server as support to horizontal federated learning, targeting KMeans as Machine Learning algorithm.
A reference implementation of Horizontal Federated Learning (HFL) using only Snowflake native features. Model parameters are exchanged across 3 Snowflake accounts via Private Listings to collaboratively train a binary classification model for diabetes prediction on CDC BRFSS 2024 data.
Horizontal Federated Learning (HFL) enables multiple organizations to collaboratively train machine learning models without sharing raw data with each other. This implementation focuses on the healthcare domain. Patient health and lifestyle data is extremely valuable for improving early disease detection and prediction accuracy, but direct data sharing between healthcare institutions and organizations faces significant regulatory and ethical barriers due to patient privacy concerns. With federated learning, each institution retains its data locally and exchanges only model parameters (weights), enabling collaborative training. This achieves generalization performance unattainable from any single institution's data alone, without compromising privacy.
In this implementation, we use CDC BRFSS 2024 (Behavioral Risk Factor Surveillance System) data, geographically splitting 53 U.S. states across 10 clients to train a diabetes prediction model. Each client holds data from only a limited set of states, but through federated learning, a global model reflecting patterns across all states is constructed, outperforming every single client's local model. Even with differential privacy (DP) noise applied, performance remains above all local baselines.
This approach is not limited to healthcare. Applicable use cases include: fraud detection across financial institutions (banks collaboratively learn fraud patterns without sharing transaction data), network anomaly detection among telecommunications carriers, cross-supply-chain quality prediction in manufacturing, and consumer behavior analysis across retail companies. Any domain where data sensitivity is high yet integrating data from multiple organizations creates significant value is a candidate for federated learning.
┌─────────────────────────────────────────────────┐ │ ORCHESTRATOR_ACCOUNT (Orchestrator) │ │ │ │ HFL_APP_DB.HFL_META │ │ - HFL_ROUNDS │ │ - HFL_GLOBAL_WEIGHTS │ │ - HFL_EVAL_RESULTS │ │ - EVAL_DATA_SAMPLE (10K) │ │ - Model Registry │ │ │ │ ┌──────────────────┐ ┌─────────────────┐ │ │ │AWS2_HFL_CLIENT_DB│ │AZ1_HFL_CLIENT_DB│ │ │ │ (Shared DB) │ │ (Shared DB) │ │ │ └────────┬─────────┘ └────────┬────────┘ │ └───────────┼─────────────────────┼───────────────┘ │ Private Listing │ Private Listing │ (param receive) │ (param receive) ▼ ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ CLIENT_ACCOUNT_AWS │ │ CLIENT_ACCOUNT_AZURE │ │ (Clients C1–C5) │ │ (Clients C6–C10) │ │ │ │ │ │ HFL_APP_DB.HFL_LOCAL │ │ HFL_APP_DB.HFL_LOCAL │ │ TRAIN_DATA_SAMPLE_C1│ │ TRAIN_DATA_SAMPLE_C6│ │ TRAIN_DATA_SAMPLE_C2│ │ TRAIN_DATA_SAMPLE_C7│ │ TRAIN_DATA_SAMPLE_C3│ │ TRAIN_DATA_SAMPLE_C8│ │ TRAIN_DATA_SAMPLE_C4│ │ TRAIN_DATA_SAMPLE_C9│ │ TRAIN_DATA_SAMPLE_C5│ │ TRAIN_DATA_SAMPLE_C10│ │ HFL_CLIENT_UPDATES │ │ HFL_CLIENT_UPDATES │ │ │ │ │ │ GLOBAL_HFL_DB │ │ GLOBAL_HFL_DB │ │ (Shared from Orch.) │ │ (Shared from Orch.) │ └──────────────────────┘ └──────────────────────┘
All 53 U.S. states/territories are distributed across 10 clients using greedy bin packing, balancing each client's training pool to ~28K–30K rows. Each client subsamples 1,000 rows from its pool.
Round 0 (Initialization) Orchestrator: Insert R00_INIT (zero vector) → distribute to clients via Private Listing Round N (N = 1, 2, 3) 1. CLIENT_AWS/CLIENT_AZURE: Fetch previous round's global weights from GLOBAL_HFL_DB 2. 10 clients: Train with warm_start on local data (1K each) → Store NONE + DP (noise_scale=0.05) updates in HFL_CLIENT_UPDATES 3. Orchestrator: Poll client weights via Private Listing shared DBs 4. Orchestrator: Aggregate via FedAvg → store in HFL_GLOBAL_WEIGHTS 5. Orchestrator: Register in Model Registry Final Orchestrator: SP_EVAL_MODELS evaluates all 12 models via ROC-AUC - FEDERATED_NONE, FEDERATED_DP - LOCAL_ONLY_{AWS2_C1..C5, AZ1_C6..C10}
Each client's trained model and the federated global models are evaluated on client-local eval data whose geographic distribution matches the client's assigned states (IMONTH > '09'). This measures how well models generalize to each client's own population, rather than a global average.
horizontal-federated-learning/ ├── README.md ├── LICENSE ├── run_multi_account_hfl.sh # Full HFL workflow orchestration script ├── populate_state_split.py # Client data population (state-based 1K subsample) ├── populate_client_eval.py # Client-local eval table population ├── data/ │ ├── brfss_2024_pipeline.sh # CDC → Parquet → Snowflake pipeline │ └── brfss_2024_work/ │ └── convert_brfss_xpt_to_parquet.py # XPT → Parquet conversion ├── sql/ │ ├── 00_prepare_data.sql # TRAIN/EVAL data preparation (all accounts) │ ├── 01_common_ddl.sql # DB, schemas, warehouse │ ├── 02_orchestrator_meta.sql # Meta tables + eval data (orchestrator) │ ├── 03_client_local_tables.sql # Client local tables │ ├── 04_aws2_views.sql # AWS client training data │ ├── 05_azure1_views.sql # Azure client training data │ ├── 06_aws1_views.sql # Orchestrator training data │ ├── 07_alter_add_model_type.sql # Add MODEL_TYPE column (XGBoost support) │ ├── 10_orchestrator_sp_aggregate_global_model.sql # FedAvg / Federated Forest SP │ ├── 11_orchestrator_sp_eval_models.sql # Evaluation SP │ ├── 12_orchestrator_sp_register_global_model.sql # Global model registration SP │ ├── 13_orchestrator_sp_register_baseline_model.sql # Baseline registration SP │ ├── 20_client_sp_train_local_baseline.sql # Baseline training SP │ ├── 21_client_sp_train_local_federated.sql # Federated training SP │ └── 22_client_sp_eval_client_models.sql # Client-local evaluation SP
5. Training Protocol, Convergence Behavior & Experimental Evaluation
Note : The connection names ORCHESTRATOR , CLIENT_AWS , and CLIENT_AZURE are placeholders. Replace them with the actual connection names in your ~/.snowflake/connections.toml .
for CONN in ORCHESTRATOR CLIENT_AWS CLIENT_AZURE ; do snow sql -c $CONN -f sql/00_prepare_data.sql done
# Step 1: Common DDL (all 3 accounts) for CONN in ORCHESTRATOR CLIENT_AWS CLIENT_AZURE ; do snow sql -c $CONN -f sql/01_common_ddl.sql done # Step 2: Orchestrator meta tables + eval data snow sql -c ORCHESTRATOR -f sql/02_orchestrator_meta.sql # Step 3: Client local tables (all 3 accounts) for CONN in ORCHESTRATOR CLIENT_AWS CLIENT_AZURE ; do snow sql -c $CONN -f sql/03_client_local_tables.sql done # Step 4: Training data views snow sql -c ORCHESTRATOR -f sql/06_aws1_views.sql snow sql -c CLIENT_AWS -f sql/04_aws2_views.sql snow sql -c CLIENT_AZURE -f sql/05_azure1_views.sql
Important : CREATE OR REPLACE TABLE breaks Private Listing share references. Use TRUNCATE TABLE instead (safe).
# Orchestrator SPs snow sql -c ORCHESTRATOR -f sql/10_orchestrator_sp_aggregate_global_model.sql snow sql -c ORCHESTRATOR -f sql/11_orchestrator_sp_eval_models.sql snow sql -c ORCHESTRATOR -f sql/12_orchestrator_sp_register_global_model.sql snow sql -c ORCHESTRATOR -f sql/13_orchestrator_sp_register_baseline_model.sql # Client SPs (AWS and Azure) for CONN in CLIENT_AWS CLIENT_AZURE ; do snow sql -c $CONN -f sql/20_client_sp_train_local_baseline.sql snow sql -c $CONN -f sql/21_client_sp_train_local_federated.sql done
populate_state_split.py reads BRFSS_2024_TRAIN from the orchestrator account, then subsamples 1,000 rows per client based on state assignments:
This writes 1,000 rows to each TRAIN_DATA_SAMPLE_C1 – C5 on CLIENT_AWS and TRAIN_DATA_SAMPLE_C6 – C10 on CLIENT_AZURE.
bash run_multi_account_hfl.sh # LogReg (default) bash run_multi_account_hfl.sh XGBOOST # XGBoost Federated Forest
WH= " USE WAREHOUSE WH_HFL_XS; " # Baseline training (10 clients) for i in 1 2 3 4 5 ; do snow sql -c CLIENT_AWS -q " $WH CALL HFL_APP_DB.HFL_LOCAL.SP_TRAIN_LOCAL_BASELINE('AWS2_C ${i} ', 'LOGREG') " done for i in 6 7 8 9 10 ; do snow sql -c CLIENT_AZURE -q " $WH CALL HFL_APP_DB.HFL_LOCAL.SP_TRAIN_LOCAL_BASELINE('AZ1_C ${i} ', 'LOGREG') " done # Federated training rounds (N = 1, 2, 3) for N in 1 2 3 ; do for i in 1 2 3 4 5 ; do snow sql -c CLIENT_AWS -q " $WH CALL HFL_APP_DB.HFL_LOCAL.SP_TRAIN_LOCAL_FEDERATED( $N , 'AWS2_C ${i} ', 0.05, 'LOGREG') " done for i in 6 7 8 9 10 ; do snow sql -c CLIENT_AZURE -q " $WH CALL HFL_APP_DB.HFL_LOCAL.SP_TRAIN_LOCAL_FEDERATED( $N , 'AZ1_C ${i} ', 0.05, 'LOGREG') " done snow sql -c ORCHESTRATOR -q " $WH CALL HFL_APP_DB.HFL_META.SP_AGGREGATE_GLOBAL_MODEL( $N , 'NONE', 600, 15, 'LOGREG') " snow sql -c ORCHESTRATOR -q " $WH CALL HFL_APP_DB.HFL_META.SP_AGGREGATE_GLOBAL_MODEL( $N , 'DP', 600, 15, 'LOGREG') " snow sql -c ORCHESTRATOR -q " $WH CALL HFL_APP_DB.HFL_META.SP_REGISTER_GLOBAL_MODEL( $N , 'NONE', 'LOGREG') " snow sql -c ORCHESTRATOR -q " $WH CALL HFL_APP_DB.HFL_META.SP_REGISTER_GLOBAL_MODEL( $N , 'DP', 'LOGREG') " done # Final evaluation snow sql -c ORCHESTRATOR -q " $WH CALL HFL_APP_DB.HFL_META.SP_EVAL_MODELS(3, 'LOGREG') " # Check results snow sql -c ORCHESTRATOR -q " SELECT MODEL_TYPE, METRIC_VALUE FROM HFL_APP_DB.HFL_META.HFL_EVAL_RESULTS ORDER BY METRIC_VALUE DESC; "
BRFSS_2024_TRAIN from orchestrator account (278K rows, 53 states) ↓ pandas Load into DataFrame ↓ _STATE filter + df.sample(n=1000) Subsample 1,000 rows per client ↓ TRUNCATE + write_pandas TRAIN_DATA_SAMPLE_C{1-5} on CLIENT_AWS TRAIN_DATA_SAMPLE_C{6-10} on CLIENT_AZURE
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)