Appendices

Appendix A: Framework Alternatives

Framework	Consider when	Notes
Flower	General cross-silo FL, tutorial path, PyTorch/TensorFlow/JAX flexibility.	Recommended reference framework for this guide.
NVIDIA FLARE	Healthcare imaging, GPU-oriented workflows, or organisations already using NVIDIA/MONAI tooling.	Strong ecosystem for clinical and imaging use cases.
OpenFL	Regulated consortia and deployments that require hardware-assisted trust patterns.	Often discussed in healthcare and consortium settings.
FATE	Vertical FL in finance or use cases needing mature HE-based VFL workflows.	Heavier infrastructure footprint, but strong VFL support.
TensorFlow Federated	Research, simulation, or teams already committed to TensorFlow research workflows.	More simulation-oriented for many teams.
Substra	Regulated consortia where provenance and traceability are central requirements.	Useful where auditability is a core deliverable.

---

Appendix B: Repository Documentation Checklist

The reference repository should contain enough material for a technical team to run the quickstart, deploy the reference architecture, configure privacy and security controls, validate data and model readiness, and operate the system safely.

Repository area	Suggested contents
README.md	Overview of the reference implementation, scope, supported FL setting, prerequisites, and link back to this guide.
docs/quickstart.md	Local simulation steps, dependency setup, expected outputs, common quickstart issues, and reset instructions.
docs/configuration.md	Environment variables, configuration files, parameter descriptions, and example values.
docs/deployment.md	Cloud deployment sequence, coordinator setup, client setup, deployment commands, and validation checks.
infra/	CDK or Terraform modules, infrastructure README, stack dependencies, IAM assumptions, networking assumptions, and deployment outputs.
configs/	Example task configuration, privacy configuration, secure aggregation configuration, and environment-specific templates.
scripts/	Preflight checks, deployment scripts, validation scripts, diagnostics, cost reporting, and log collection.
templates/	Data manifest, training manifest, model card, dataset summary, federation agreement checklist, and decision log templates.
privacy/	DP budget tooling, privacy accountant usage, privacy-test scripts, and example privacy reports.
secagg/	Secure aggregation configuration, quorum settings, dropout handling notes, and test cases.
psi/	Optional secure entity alignment module, test data, configuration examples, and governance notes.
runbooks/	Certificate rotation, CA rollover, failed training rounds, rollback, disaster recovery, incident response, and participant offboarding.
docs/troubleshooting.md	Detailed troubleshooting decision tree and known failure modes.
docs/cost-reporting.md	How to estimate per-run cost, use cost-reporting scripts, and update indicative cost assumptions.

Where a feature is described in the guide but not yet implemented in the repository, the repository should mark it clearly as planned, experimental, or not currently supported. This is especially important for optional or advanced features such as secure entity alignment, secure inference, enclave-based components, and production-hardening scripts.

---

Appendix C: Environment Configuration and Deployment Reference

This appendix preserves detailed deployment reference material that may change as the repository, infrastructure modules, and organisation-specific deployment standards evolve. Command-level procedures should be maintained in the repository.

Pre-flight validation

Before provisioning production-like infrastructure, the operator should run a pre-flight validation. The goal is to catch missing access, invalid configuration, or unacceptable network posture before a partial deployment is created.

Pre-flight area	Checks
Landing zone	Correct account or tenancy, target region, baseline controls, approved network ranges, tagging standards.
Operator workstation	Approved CLI tooling, SSO or STS access, no long-lived local credentials, repository cloned, dependency versions checked.
Network	Private subnets, required VPC endpoints, security groups, DNS resolution, no unrestricted inbound paths.
Identity and access	Named deployer/operator/auditor roles, permission boundaries, just-in-time access where available.
Compute and storage	Baseline instance or container capacity, encrypted block/object storage, backup targets.
PKI and certificates	Root/subordinate CA plan, server name, client identities, revocation and renewal approach.

Operational Guidance: Pre-flight command pattern

The repository should provide a script such as scripts/preflight.sh that checks the landing zone, validates required IAM permissions, confirms required endpoints and DNS, checks local tooling versions, and exits non-zero on missing prerequisites. The output should be stored as deployment evidence.

Environment configuration

A single set of environment-specific configuration values should be the source of truth for each deployment. These values should not be checked into source control. The repository should instead provide a commit-safe template carrying placeholders, which the operator populates through the approved configuration process.

Configuration area	Values to define
Region and account	Region, account or tenancy identifier, organisation name, environment name.
DNS	Private hosted-zone name, coordinator DNS name, certificate subject alternative names.
Network	VPC, subnet IDs, endpoint IDs, security groups, allowed egress destinations.
PKI	Root CA reference, subordinate CA reference, certificate validity, CRL publication path.
Keys	Purpose-separated KMS or equivalent keys for deployment, PKI audit, logs, storage, and backups.
Storage	Model checkpoint bucket, audit-log bucket, run-record location, retention rules.
Participants	Client identifiers, certificate common names, contact owners, expected availability windows.
Training defaults	Default task, aggregation strategy, round count, local epochs, DP preset, secure aggregation setting.

Operational Guidance: Configuration placeholders

• Use $PLACEHOLDER values in documentation and command examples, never unnamed manual steps.

• Keep env.example.yaml in the repository and render deployment-specific values into a local ignored file.

• Separate configuration by environment, for example dev, pilot, representative-data test, and production.

• Record the configuration hash or version in each training run record so results are reproducible.

Post-deployment validation and handover

Validation item	Expected evidence
Coordinator health	Health check succeeds; service version and configuration hash logged.
mTLS enforcement	Known client succeeds; unknown or revoked client fails; result logged as evidence.
Synthetic cloud smoke task	Completed run across two client nodes; metrics and run record produced.
Secure aggregation behaviour	Round succeeds when quorum is met and fails safely when quorum is not met.
DP budget logging	Configured DP parameters and projected budget are recorded.
Audit-log shipping	Coordinator and client logs arrive in the auditable location within expected delay.
Operator handover	Runbooks, dashboards, escalation contacts, and rollback procedures reviewed.

Rolling upgrades

Production deployments should support controlled upgrades of the coordinator, clients, task definitions, and container images. An upgrade should not change model behaviour or privacy controls without explicit model-owner and governance review.

Upgrade step	Minimum control
Prepare release	Record code version, image digest, task changes, migration notes, and rollback target.
Validate in non-production	Run local simulation and cloud smoke task using synthetic data.
Stage clients	Upgrade a small client cohort first where operationally possible.
Monitor	Compare health, dropout, metrics, and privacy-budget logs before and after upgrade.
Rollback	Restore previous image and configuration if health, security, or model criteria fail.

---

Appendix D: PKI and Certificate Operations

This appendix preserves the certificate and PKI operations that are important for production use, but too detailed for the main guide. The repository should maintain the command-level procedures and environment-specific values.

PKI hierarchy and certificate lifecycle

Production deployments should define the certificate hierarchy and lifecycle before onboarding participants. This includes certificate issuance, renewal, revocation, CRL or equivalent revocation publication, emergency replacement, and CA rollover.

Figure 4. Example two-tier PKI hierarchy with an offline root CA and a subordinate CA for server and client certificates.

Operational Guidance: PKI lifecycle controls

• Keep the root CA isolated and offline where that is the selected trust model; routine operations should use a subordinate CA or approved enterprise PKI.

• Issue separate certificates for coordinator and each participant client. Do not reuse client certificates across participants or environments.

• Track certificate owner, common name or identifier, expiry, issuance path, revocation status, and renewal date.

• Automate certificate-expiry monitoring and trigger alerts before certificates are within the renewal window.

• Define a revocation process for departed participants, compromised clients, or emergency containment.

• Test CRL or revocation-list refresh and alert if the revocation source becomes stale.

• Plan CA rollover and server cutover in advance, including dual-trust windows and rollback criteria.

Rollback and CA rollover

Operation	When used	Minimum steps
Model rollback	New model is corrupted, degraded, unapproved, or unsafe.	Restore previous approved model, block export of failed version, log decision and incident record.
Application rollback	Deployment causes service, security, or training failures.	Redeploy previous signed image and configuration, validate smoke task, monitor alarms.
Certificate rollback or replacement	Certificate error, expiry, or suspected compromise.	Issue replacement, revoke old certificate, refresh revocation lists, validate mTLS.
CA rollover	Subordinate CA expiry, compromise, or planned rotation.	Introduce new CA, support dual-trust window where needed, rotate certificates, revoke old CA path, validate all clients.

---

Appendix E: Secure Inference with FHE and Nitro Enclave (AWS)

This is an advanced optional extension. It should remain clearly separated from the main cross-silo HFL training path unless the reference repository implements and supports it as part of the published deployment.

Some deployments may also need to protect inference queries, not only training data. This is a separate design from training-time secure aggregation. Secure aggregation protects training updates from the coordinator; FHE-based secure inference protects inference inputs by allowing the server to compute over encrypted queries for supported model types.

When this applies

Use secure inference when	Do not use it when
Inference queries are sensitive and should not be visible to the model-serving environment.	Plain inference is acceptable under the trust model and approved governance controls.
The model type is supported by the FHE library and the performance trade-off is acceptable.	The model is too complex for practical FHE inference or latency requirements cannot be met.
A separate key-handling and attestation design is approved.	The team has not resourced enclave operations, attestation monitoring, and key rotation.

Supported modes

Mode	Description	Operator implications
Plain inference	The model receives plaintext input under normal application controls.	Simpler and faster; requires standard access control and logging.
FHE secure inference	The client encrypts the query; the server computes over ciphertext for supported models.	Higher compute cost, model-type constraints, key management, and specialised monitoring.
Enclave-assisted key handling	A Nitro Enclave or equivalent protects sensitive keys used in the secure-inference flow.	Requires image reproducibility, attestation checks, sealed storage, and failure runbooks.

Key handling and attestation

Figure 7. Example FHE secure-inference key boundary with an enclave protecting sensitive key material.

Operational Guidance: Enclave controls

• Define the enclave image, expected PCR or equivalent attestation values, and approval process for image changes.

• Do not release sensitive key material unless attestation succeeds.

• Monitor attestation failures as high-severity security events.

• Define ciphertext lifetime and avoid retaining ciphertexts across key rotations unless explicitly approved.

• Document side-channel mitigations, rate limits, and error-handling behaviour.

Inference request flow

1. Client prepares and encrypts the query using the approved scheme.

2. Server receives ciphertext and routes it to the supported inference path.

3. Where used, enclave attestation is checked before sensitive key use.

4. Model computation runs over the permitted encrypted representation.

5. Result is returned to the authorised caller and the request is logged without exposing sensitive input.

Rate limiting and side-channel mitigations

Risk	Control
Query flooding	Apply per-client rate limits, quotas, and anomaly detection.
Timing side channels	Monitor latency patterns and avoid exposing unnecessary diagnostic detail to callers.
Error-message leakage	Return generic errors externally; preserve detailed errors only in protected logs.
Model extraction	Control API access, throttle repeated probing, and monitor abnormal query patterns.

---

Appendix F: Indicative Cost Estimate

This appendix provides indicative cost estimates as planning references. Actual costs will vary depending on the deployment environment, infrastructure architecture, cloud or hosting provider, procurement assumptions, exchange rates, and assurance requirements. Cost estimates should be reviewed and updated during solution design and before sponsor approval.

The table below is intended as a cost-estimation template. Amounts should not be treated as fixed commitments and should be validated against the target deployment environment, prevailing pricing, procurement arrangements, and vendor quotations.

Cost item	Driver	How to estimate
Coordinator compute	Instance or container service, availability requirements, runtime.	Estimate by instance/container type, hours per month, and expected training windows.
Client compute	Number of participants, CPU/GPU type, local runtime.	Estimate per participant and multiply by expected active clients.
Key management	Number of purpose-separated keys and API calls.	Include keys for deployment, storage, logs, and PKI audit where used.
Logging and monitoring	Metric volume, log volume, retention, alarms, dashboards.	Estimate by retention period and expected run frequency.
Object and block storage	Model checkpoints, run records, audit evidence, backups.	Estimate by artefact size, retention, and backup copies.
PKI and certificate operations	CA, certificate issuance, revocation, CRL distribution, monitoring.	Estimate platform costs plus operator time.
Security assurance	VAPT, architecture review, code review, container scanning.	Use procurement quotes or internal chargeback estimates.
AI governance and model review	AI Verify or equivalent, model risk review, documentation effort.	Estimate per major model version or release.
Operational support	Runbook maintenance, on-call, incident response, retraining.	Estimate by support model and expected frequency of runs.

Operational Guidance: Indicative budget discipline

The engineer draft contained indicative steady-state monthly and one-time assurance cost ranges. These figures should be treated as planning references only. Before publication or sponsor approval, validate cost estimates against the target deployment environment, current pricing, procurement quotations, and the organisation’s charging model. Sponsors should approve both the expected steady-state operating costs and one-time assurance costs in writing.

---

Appendix G: Incident Response and Operator Runbooks

This appendix preserves operational runbook material that should also be maintained in the repository for production operators.

Dead-man heartbeats

A dead-man heartbeat detects silent probe failure. Each probe emits a heartbeat only when the probe itself succeeds. If the metric disappears, the operator treats the probe as unhealthy rather than assuming the control is working.

Probe function	Schedule	Primary metric	Dead-man alarm
mTLS enforcement probe	Every 5 minutes	mTLS enforcement indicator	mTLS probe dead.
Certificate-expiry probe	Daily	Days until certificate expiry	PKI probe dead.
CRL refresh probe	Every 4 hours	CRL age in hours	CRL probe dead.
Enclave-attestation probe, if used	Every 15 minutes	Attestation indicator	Attestation probe dead.

DR drill

Drill step	Evidence
Declare scenario	Scenario, scope, participants, and expected recovery target.
Restore configuration	Restoration logs and configuration version.
Restore model artefacts	Model registry restored and rollback target verified.
Validate certificates	mTLS success and revoked-client failure verified.
Run synthetic smoke task	Successful end-to-end cloud run after recovery.
Review	Lessons learned, gaps, and remediation owners.

Incident severity ladder

Severity	Examples	Response
P0	Confirmed data exposure, unauthorised participant accepted, malicious model release, attestation failure affecting protected keys.	Immediate containment, page on-call and security lead, suspend affected run, preserve evidence, start notification assessment.
P1	DP budget exhausted, CRL stale, mTLS probe dead, validation failure on approved data, production training unavailable.	Pause affected workflow, page on-call, involve model/security/governance owner as relevant.
P2	SecAgg abort rate elevated, audit-log lag, repeated client dropouts, degraded performance.	Investigate during support hours or as agreed by production SLA.
P3	Documentation issue, non-critical dashboard bug, low-priority cost variance.	Track in backlog and resolve before next release or drill.

P0/P1 runbook

1. Stop or pause the affected training or inference workflow where safe to do so.

2. Preserve logs, run records, model artefacts, manifests, and deployment evidence.

3. Identify affected participants, model versions, tasks, and time windows.

4. Contain access by revoking certificates, disabling clients, rolling back images, or blocking model export where needed.

5. Notify the sponsor, security lead, DPO/compliance lead, and participant contacts according to the escalation matrix.

6. Assess whether any statutory, contractual, or sector-specific notification is triggered.

7. Recover using approved rollback or restoration procedures.

8. Document root cause, corrective actions, and approval to resume.

Decision tree

Figure 8. Example troubleshooting decision tree for production operators.

Adding a new client site

Step	Action
Confirm participation basis	Update federation agreement and governance records.
Validate local readiness	Check local data, schema, infrastructure, owner contacts, and support model.
Issue identity	Create participant record and certificate; record expiry and revocation owner.
Deploy client node	Deploy approved image and configuration in the participant environment.
Run synthetic validation	Confirm mTLS, logs, smoke task, secure aggregation behaviour, and run records.
Run representative-data readiness checks	Validate manifest, baselines, heterogeneity profile, and model impact before using production data.

---

Appendix H: Glossary

Term	Meaning
FL coordinator	The central service that orchestrates rounds and aggregates model updates.
FL client node	Participant-side service that trains locally and sends model updates.
HFL	Horizontal federated learning: similar feature spaces, different records.
VFL	Vertical federated learning: different feature spaces, overlapping entities.
FTL	Federated transfer learning: limited overlap in both records and features.
mTLS	Mutual TLS, where both client and server authenticate cryptographically.
PKI	Public Key Infrastructure used to issue and manage certificates.
SecAgg	Secure aggregation, which prevents the coordinator from seeing individual participant updates.
DP-SGD	Differentially private stochastic gradient descent.
PSI	Private Set Intersection, used for privacy-preserving entity alignment.
FHE	Fully homomorphic encryption, used in this guide only for optional secure inference.
SBOM	Software bill of materials, listing software components and dependencies.

---

Appendix I: Experienced Operator Checklist

Phase	Experienced operator checks
Before provisioning	Use case approved; participants identified; repository cloned; pre-flight script passes; configuration placeholders rendered.
Infrastructure	Network, keys, PKI, coordinator, clients, storage, logs, and dashboards deployed through version-controlled IaC.
Validation	mTLS probe, revoked-client test, cloud smoke task, secure aggregation dropout test, DP budget logging.
Data readiness	Manifests approved, schema validation passes, baselines recorded, heterogeneity profile reviewed.
Security readiness	Image signed and scanned; SBOM generated; hardening checklist evidence captured; VAPT or equivalent review scheduled/completed.
Governance readiness	DPIA or equivalent review, participant agreement, residual-risk decision, model release criteria.
Production operations	On-call matrix, alarms, runbooks, rollback, backup/restore, CA rollover, DR drill.
Release	Model card, dataset summary, privacy tests, per-participant metrics, cost report, sponsor sign-off.

---

Closing Note

A good FL deployment is not simply a model-training loop distributed across several environments. It is a collaboration system: participants must agree on the task, trust boundaries, controls, evaluation, model use, and operational responsibilities. This guide provides a starter path and a production operator reference for that journey. Teams should begin small, prove the loop with synthetic data, and add governance and production hardening in proportion to the sensitivity and operational importance of the use case.