A production FL service should have an observability stack covering service health, training progress, security controls, privacy-budget state, audit-log delivery, model performance, and cost. The exact tooling may vary, but the operator should have dashboards and alarms before production data is used.
| Signal type | Examples |
|---|
| Service health | Coordinator health, client heartbeats, endpoint latency, CPU/memory/storage. |
| Training progress | Round start/end, selected clients, completed clients, dropout rate, metric trend. |
| Security controls | mTLS enforcement, certificate expiry, CRL age, failed authentication, image version. |
| Privacy controls | Secure aggregation quorum, DP budget, privacy-test status. |
| Audit delivery | Log shipping latency, missing logs, integrity check status. |
| Cost and energy | Runtime, compute type, storage generated, per-run cost estimate. |
Operational monitoring should cover both system health and federation-specific behaviour. Where an enterprise SIEM is used, security-relevant coordinator, client, certificate, and audit-log events should be forwarded according to the organisation’s logging standards. Detailed log-forwarding and heartbeat procedures should be maintained in repository runbooks.
| Dashboard | Minimum panels |
|---|
| Operations dashboard | Service health, client availability, active run, last successful run, failed runs. |
| Security dashboard | mTLS failures, certificate expiry, revocation status, image digest, failed access attempts. |
| Training dashboard | Round metrics, convergence curve, dropout rate, per-participant metrics. |
| Privacy dashboard | DP budget spent, secure aggregation status, privacy-test status. |
| Cost dashboard | Per-run compute, storage, network transfer, monitoring volume, cumulative monthly estimate. |
| Alarm | Trigger | Action |
|---|
| Coordinator unavailable | Health check fails or no heartbeat. | P1 or P0 depending on production impact; page on-call. |
| mTLS enforcement regression | Unknown or invalid client unexpectedly accepted, or probe fails. | Treat as suspected security regression until verified manually. |
| Certificate near expiry | Days until expiry below threshold, such as 30 days. | Trigger rotation runbook. |
| CRL or revocation stale | Revocation source age above threshold or refresh probe missing. | Page on-call; fix distribution. |
| SecAgg abort rate | Abort rate above defined threshold over monitoring window. | Investigate client health, quorum, timeouts, and network stability. |
| DP budget exhausted | Epsilon spent at or above configured budget. | Halt or pause training and escalate to model owner and DPO. |
| Validation failed | Client reports schema or manifest validation failure. | Block run and resolve data integrity issue. |
| Audit-log delivery lag | Cross-account or central log delivery exceeds threshold. | Verify audit pipeline and retention. |
Detailed incident response procedures, severity ladder, dead-man heartbeats, and DR drill guidance are provided in Appendix G.
After release, the model should be monitored for performance degradation, data drift, and operational incidents. Retraining should be scheduled or event-driven, and retraining requires the same evidence discipline as initial training.
| Trigger | Response |
|---|
| Scheduled retraining window | Run approved retraining workflow and compare to prior model. |
| Performance degradation | Investigate per-participant metrics, data drift, and recent schema changes. |
| Data drift | Review feature distributions, label distribution, and missingness. |
| Participant change | Re-run onboarding, update certificates, update federation agreement where needed. |
| New model version | Complete model card, privacy tests, security review, and release gate. |
| Backed-up item | Notes |
|---|
| Infrastructure configuration | IaC state, approved configuration templates, deployment parameters. |
| Certificates and PKI evidence | Certificate inventory, issuance records, revocation records, CA rollover plans. |
| Model checkpoints | Approved model versions and rollback targets. |
| Run records and manifests | Training manifests, data-manifest hashes, participant lists, privacy settings. |
| Audit logs | Operational logs, security events, control validation evidence. |
| Runbooks | Current operational procedures and escalation contacts. |
The default production assumption should be Singapore-resident recovery unless the deploying organisation has an approved data-residency waiver or equivalent governance decision. Disaster recovery plans should define recovery time objective, recovery point objective, backup frequency, restoration steps, and drill cadence.
| DR decision | Minimum expectation |
|---|
| Recovery environment | Approved environment in the permitted residency boundary. |
| Backups | Encrypted, access-controlled, restorable backups for configuration, model artefacts, and evidence. |
| Restoration procedure | Documented and tested steps for restoring coordinator, client configuration, certificates, and model registry. |
| Participant coordination | Contacts and procedures for client reconnection after recovery. |
| DR drill | Periodic drill with evidence and lessons learned. |