From Simulation to Cross-Silo Cloud Deployment

After the local simulation works, teams can begin preparing for a cross-silo cloud deployment. At this stage, the objective is not yet to operate a production FL service. The objective is to confirm that the reference architecture can run across controlled cloud environments, that each participant can operate its client node, and that the required data, security, and governance owners are in place.

4.1 Implementation path

A common failure mode is trying to solve every production concern before proving the FL loop. This guide recommends a staged path: first prove the local simulation, then deploy a cloud pilot with synthetic data, then add privacy, security, governance, and operational controls as the use case matures.

Stage	Objective	Typical evidence
Local simulation	Confirm that the server, clients, task configuration, and aggregation loop work.	Completed synthetic smoke task, local logs, basic metrics.
Cloud pilot with synthetic data	Confirm that networking, certificates, deployment scripts, and cloud services work without real data.	Successful training run across cloud client nodes, audit logs, basic monitoring.
Representative-data test	Assess data schema, baseline model performance, heterogeneity, privacy controls, and failure modes.	Local and federated baselines, privacy-test results, participant-level metrics.
Governed pilot	Run with approved data under documented governance controls.	DPIA or equivalent review, federation agreement, signed residual-risk decision.
Production deployment	Operate as a supported service with monitoring, support, incident response, and model lifecycle controls.	Runbooks, VAPT or security testing, monitoring dashboards, release approvals.

4.2 Minimum readiness checklist

Readiness area	Minimum expectation before a cloud pilot
Use case	A specific ML task, target users, expected benefit, and success metric are defined.
Participants	At least two participants are identified, with named technical and governance contacts.
Data	Each participant knows what local data is available, its schema, quality, volume, sensitivity, and constraints.
Baseline ML	A local or centralised baseline has been attempted so the FL result can be compared meaningfully.
Infrastructure	Each participant can run a client node with the required compute, storage, network, and access controls.
Security	Participant authentication, network paths, container supply chain, logging, and secrets management have owners.
Governance	The team has a path for data protection review, model governance, participant agreement, and operational support.

4.3 Roles and responsibilities

Role	Primary responsibility
Sponsor	Confirms that the use case is worth piloting, funds the work, and accepts residual risks with the appropriate governance leads.
Model owner	Defines the ML task, baseline, metrics, data requirements, training configuration, and model release criteria.
Participant data owner	Approves local data use, confirms permitted processing, and ensures local data remains governed.
Platform operator	Deploys and runs the coordinator and client nodes, manages configuration, monitoring, certificates, and runbooks.
Security architect	Reviews trust boundaries, access control, network design, supply-chain controls, and incident-response requirements.
DPO or compliance lead	Reviews data protection risks, DPIA or equivalent assessment, participant agreements, and residual-risk documentation.
On-call owner	Responds to production alarms, coordinates incident triage, and escalates to model, security, or governance owners where needed.

For production use, each operational runbook should have an owner, backup owner, escalation route, and review cadence. The sponsor should confirm that the production support model is funded and realistic before production data is used.

4.4 Deployment pattern

The reference cloud deployment uses one coordinator endpoint and one client node per participant. The client node initiates outbound communication to the coordinator. The coordinator should not require inbound access into the participant environment. Identity should rely on certificates and application-level controls, not on network location alone.

Design choice	Recommended starter approach
Coordinator topology	Centralised coordinator for the pilot. This is simpler to operate and reason about than a decentralised topology.
Client topology	One client node per participating organisation or controlled environment.
Network connectivity	Client-initiated outbound connection to the coordinator over a restricted endpoint.
Participant authentication	mTLS with per-client certificates issued through the project PKI or approved enterprise PKI.
Data location	Raw data remains inside the participant environment. Only model updates leave the client.
First cloud run	Synthetic data only. Representative or production data should follow after approval.

4.5 Participant onboarding

Each participant should be onboarded deliberately. A participant is not just a network endpoint; it is a data controller or data holder with its own local data, infrastructure, governance obligations, and operational responsibilities.

Onboarding item	What to confirm
Participant identity	Legal entity or organisational unit, technical owner, data owner, and escalation contact.
Client certificate	Certificate common name or equivalent identifier, expiry, revocation process, and owner.
Local data	Dataset name, schema, sensitivity, consent or permitted-use basis, quality, and expected row count.
Client infrastructure	Compute type, storage, local access controls, logging, and maintenance owner.
Training participation	Expected availability, time windows, dropout tolerance, and contact during failed rounds.
Governance commitments	Participant agreement, model access terms, incident response, and exit procedure.

4.6 Infrastructure prerequisites

The exact implementation will depend on the deploying organisation’s cloud standards. The following prerequisites should be treated as the minimum architecture checklist for a cross-silo FL pilot on cloud. Detailed environment configuration, pre-flight validation, and deployment reference material is provided in Appendix C.

Area	Minimum expectation
Cloud account or tenancy	A controlled environment with the organisation’s required baseline security controls.
Compute	Coordinator instance or container service; one client instance or container service per participant. GPU is optional depending on model type.
Storage	Encrypted storage for model checkpoints, configuration, logs, and run artefacts. Raw data remains local to clients.
Networking	Private or restricted network paths; no broad inbound access to client nodes; egress limited to required endpoints.
Identity and access	Named roles for deployer, operator, auditor, data owner, and security reviewer. Avoid standing administrator access.
Certificates	Server certificate and per-client certificates for mTLS, with a revocation process.
Secrets and keys	Secrets managed through approved secrets management tooling. Keys should be separated by purpose where practical.
Logging	Coordinator and client events should be logged to an auditable location with appropriate retention.

4.7 Deployment sequence

The first cloud deployment should be as small as possible while still exercising the cross-silo architecture. A typical sequence is:

1. Provision the VPC, private subnets, endpoints, DNS, and security groups.

2. Create purpose-separated keys for deployment, PKI audit, logs, and storage.

3. Build, sign, scan, and push the FL container image.

4. Provision the subordinate CA or approved PKI path for server and client certificates.

5. Provision the coordinator service and TLS termination or mTLS-verifying proxy.

6. Provision two client environments using synthetic data only.

7. Issue the coordinator and client certificates.

8. Deploy the coordinator service and confirm that it accepts only trusted clients.

9. Deploy each client service and confirm outbound connectivity to the coordinator.

10. Wire observability, dashboards, alarms, and audit-log shipping.

11. Run the smoke task in cloud mode using synthetic data.

12. Confirm that logs, metrics, checkpoints, and run records are produced.

13. Introduce secure aggregation and test client dropout behaviour.

14. Introduce DP-SGD and confirm that the DP budget is computed and logged.

15. Complete handover, hardening evidence, and governance sign-offs before representative or production data is used.

Figure 2. Example CDK or IaC stack dependency order. The exact stack names should be updated to match the final repository.

4.8 Validation gates

Teams should treat the transition from local simulation to cloud pilot, and from cloud pilot to representative-data testing, as formal validation gates. Detailed pre-flight and post-deployment checks should be maintained in the repository or operator runbooks. The main guide records the expected evidence for each gate.

Gate	Minimum evidence
Before cloud provisioning	Landing zone confirmed, required roles available, network assumptions reviewed, repository and tooling available.
Before first cloud smoke task	Coordinator and two clients deployed with synthetic data only; mTLS working; logs and metrics enabled.
Before representative data	Data manifests validated, local baselines completed, secure aggregation and DP settings tested where applicable.
Before governed pilot	Data protection, security, and participant agreement reviews completed; residual risks and support model documented.
Before production release	Monitoring, rollback, incident response, backup/DR, model release criteria, and sponsor sign-off completed.

4.9 Suggested pilot plan

Week	Focus	Outputs
1	Use-case and participant alignment	Confirmed use case, participants, data owners, model owner, and success metrics.
2	Local simulation	Synthetic smoke task completed, baseline task configuration drafted.
3	Cloud pilot setup	Coordinator and two client nodes deployed with synthetic data, mTLS working.
4	Federated training pilot	Cloud FL smoke task completed, logs and metrics reviewed.
5	Data readiness and baselines	Schema validation, local baselines, data-profile comparison, evaluation plan.
6	Privacy and security controls	Secure aggregation, DP budget projection, privacy-test plan, security review findings.
7	Governance review	DPIA or equivalent review, participant agreement draft, model governance checklist.
8	Pilot decision	Go/no-go decision for representative-data pilot or production hardening.

Experienced operators who are already familiar with the concepts in this guide may prefer to work from the condensed Appendix I operator checklist.