A working FL deployment does not guarantee that the resulting model will be useful. Before using real data, participants should confirm that their datasets, task definitions, validation approach, and model assumptions are sufficiently aligned for federated training.
| Area | What to confirm |
|---|
| Task definition | Prediction target, intended users, intended use, and out-of-scope uses are clearly defined. |
| Feature set | Required input features are available at each participant, or differences are explicitly documented. |
| Label definition | Labels are defined consistently across participants, including how outcomes are measured and over what time window. |
| Population scope | Inclusion and exclusion criteria are consistent enough for joint training. |
| Time period | Data covers comparable time windows, or temporal differences are documented. |
| Data volume | Each participant has enough usable records for local training, validation, or evaluation. |
| Missingness | Missing values are understood and handled consistently. |
| Data sensitivity | Data classification, permitted use, and any restrictions on local processing are confirmed. |
| Local capability | Each participant can run the required preprocessing, training, and evaluation steps locally. |
Before a participant joins a representative-data test, it should prepare a data manifest. The manifest should describe the dataset used for training and evaluation without exposing raw records to the coordinator.
| Manifest item | Purpose |
|---|
| Dataset identifier and version | Supports reproducibility and auditability. |
| Participant identifier | Links the dataset to the participating organisation or client node. |
| Task name | Links the dataset to the intended FL task. |
| Schema version | Confirms which agreed schema the dataset follows. |
| Feature list and data types | Confirms that required inputs are present and correctly typed. |
| Label definition | Records how the target label is defined and generated. |
| Row counts | Shows whether each participant has sufficient records. |
| Missingness summary | Highlights fields with high or uneven missingness. |
| Time window | Records the period covered by the dataset. |
| Data split | Records how training, validation, and test sets are separated locally. |
| Data sensitivity | Records the relevant data classification or handling requirements. |
| Preprocessing steps | Documents local transformations applied before training. |
| Manifest signature or approval | Provides evidence that the participant has reviewed and approved the dataset description. |
The schema version, validation status, manifest approval, and any local data-quality exceptions should be recorded before a participant joins a representative-data test. Detailed validation scripts and schema templates should be maintained in the repository.
Teams should establish baselines before starting federated training. This helps separate model-quality issues from FL-specific issues such as client drift, aggregation behaviour, network failures, or privacy-control effects.
| Baseline | Purpose |
|---|
| Local baseline | Each participant trains and evaluates the model locally on its own data. This shows whether the task is learnable at each site. |
| Simple pooled or proxy baseline, where permitted | A centralised or synthetic/proxy baseline helps estimate the upper bound of model performance. This should only be used where lawful and approved. |
| Federated baseline without advanced controls, using synthetic or approved test data | Confirms whether the basic FL workflow can train and aggregate successfully. |
| Federated baseline with planned controls | Shows the effect of secure aggregation, DP-SGD, participant dropout, and other planned safeguards. |
| Per-participant evaluation | Shows whether the global model performs consistently across sites, rather than only reporting aggregate performance. |
Data heterogeneity is expected in cross-silo FL. Participants often serve different populations, use different source systems, apply different coding standards, or collect data over different periods. These differences can affect convergence, model performance, fairness across participants, and the usefulness of the final global model.
| Heterogeneity issue | Practical check | Possible response |
|---|
| Quantity skew | Compare record counts across participants. | Use participant weighting carefully, monitor whether large participants dominate, and report per-participant metrics. |
| Label skew | Compare label distribution or outcome prevalence across participants. | Use stratified evaluation, adjust sampling where appropriate, or consider aggregation methods that better handle non-IID data. |
| Feature distribution skew | Compare summary statistics for key features across participants. | Standardise preprocessing, review feature definitions, or remove unstable features. |
| Missingness differences | Compare missingness rates by feature and participant. | Agree imputation rules, remove unreliable features, or document site-specific limitations. |
| Temporal skew | Compare data collection periods and recency. | Align time windows where possible, or evaluate whether older data degrades performance. |
| Population differences | Compare permitted high-level population summaries. | Use per-participant and subgroup evaluation where permitted. |
| Missing classes | Identify participants with no examples of important classes. | Use local evaluation carefully, consider data augmentation, participant grouping, or revised task scope. |
| System or coding differences | Check whether source systems, coding standards, or measurement units differ. | Harmonise definitions, apply local mappings, and document transformations in the manifest. |
| Area | What to define |
|---|
| Model architecture | Model type, parameter count, dependencies, and expected compute requirements. |
| Task configuration | Task name, features, label, loss function, metrics, and evaluation method. |
| Training configuration | Number of rounds, local epochs, batch size, learning rate, random seed, and stopping criteria. |
| Aggregation strategy | Aggregation method, participant weighting approach, and dropout behaviour. |
| Privacy configuration | Whether DP-SGD is used, privacy budget, clipping settings, and expected utility impact. |
| Evaluation configuration | Global metrics, per-participant metrics, subgroup metrics where permitted, and release thresholds. |
| Failure criteria | Conditions under which training should stop, be re-run, or be escalated for review. |
| Evidence | Why it matters |
|---|
| Signed or approved data manifests | Shows which datasets, schema versions, and preprocessing steps were used. |
| Training manifest | Records the participating clients, task configuration, model version, aggregation strategy, seed, and privacy settings. |
| Baseline results | Shows how local, centralised/proxy, and federated performance compare. |
| Per-participant metrics | Identifies whether the global model performs unevenly across sites. |
| Heterogeneity profile | Documents material differences in data volume, labels, feature distributions, missingness, time periods, and population coverage. |
| Privacy-test results | Records the results of membership-inference, gradient-leakage, or other relevant privacy tests where used. |
| Fairness or subgroup metrics, where permitted | Supports review of uneven model performance across relevant groups. |
| Model card | Documents intended use, out-of-scope use, model details, training process, evaluation results, limitations, and approval status. |
| Dataset card or dataset summary | Documents data source, coverage, known biases, sensitivity, retention, and permitted use. |
| Decision log | Records whether the model should proceed, be revised, or stop. |