5. Privacy-preserving Data Synthesis

While synthetic data attempts to break direct one-to-one mappings between real and synthetic records, its “synthetic” label alone does not guarantee privacy. Like any AI/ML model, a generator—also called a synthesizer—learns patterns from real data, and under certain conditions, those patterns can leak sensitive or identifiable information.

What is a privacy leak? When synthetic data contains information that could identify or reveal sensitive details about real people from the real dataset. For e.g., if a synthetic patient record is nearly identical to a real patient’s record, it could expose that person’s medical information.

A context-driven privacy evaluation is therefore essential before sharing or using synthetic data in sensitive scenarios. Privacy risks should be assessed using complementary methods, each providing a different lens on potential leakage.

Two-Pronged Privacy Risk Evaluation

As no single metric can capture every vector of privacy leakage, our two-pronged methodology includes two type of evaluation metrics:

1. Empirical Privacy Risk Assessment (Smoke Testing): Computationally lightweight proxy metrics that detect memorization. These serve as a fast sanity check and first line of defense.
2. Attack-Based Evaluation (Adversarial Testing): Rigorous modeling of specific privacy attacks (singling out, linkability, inference, membership inference) under a defined threat model. This provides a realistic measure of an attacker’s potential success.

Before any evaluation begins, strictly partition your data:

• D_train — the real data ingested by the generator for training.
• D_control — a holdout set representing the general population. The generator must not see this data.
• D_synth — the generated synthetic dataset.

The control partition is essential because it provides the counterfactual baseline necessary to measure excess risk. It answers a critical question: is an attacker exploiting leaked information specific to the individuals in the training data, or simply leveraging the broader, population-level trends the model was supposed to learn? By comparing evaluation metrics on D_train against D_control, we isolate genuine privacy leakage from benign pattern learning.

The following subsections sections walk through each type of evaluation metrics.

1. Empirical Privacy Risk Assessment: Smoke Testing

Empirical checks are a quick way to flag potential privacy leaks—situations where synthetic data reveals information about real individuals from the real dataset. These tests look for signs that the synthesizer memorized specific real records instead of learning general patterns.

Common “smoke tests” include:

• Distance-based measures (similarity/proximity checks): Compare synthetic records directly against the real dataset to find close matches. If many synthetic records closely resemble real ones, this suggests the generator copied rather than learned patterns.

• Duplicate detection: Check if any synthetic records are exact copies or near-duplicates of real records. Exact matches are clear privacy violations.

• Outlier detection: Identify unusual synthetic records (rare combinations of attributes) and check if they match real outliers. Copying rare cases is particularly risky for privacy.

Figure 1: Example of distance-based privacy evaluation using DCR (distance to closest record) measurements to evaluate the overall distance between real and synthetic data. For every point of synthetic data (blue), the closest point of real data (black) is identified. The DCR (red line) is the distance to that real data point. In this example, there are three columns of data (X, Y and Z); the same DCR metric can be applied to any number of columns.

Image reference:

SDV DCRBaselineProtection

Known Limitations of DCR-Based Privacy Evaluation

DCR-based metrics are computationally efficient screening tools that provide a useful first line of defence against memorization. However, they should not be relied upon in isolation. Recent research has shown that synthetic datasets deemed “truly anonymous” or “private” by DCR’s pass/fail statistical tests can still be highly vulnerable to membership inference attacks (MIAs) and reconstruction attacks that recover 78%–100% of training outliers. Furthermore, the continuous DCR score shows no meaningful correlation with actual vulnerability to MIAs, indicating that passing a DCR threshold alone does not guarantee privacy protection.

Two-Pronged Approach: Treat DCR as an early screening step within the empirical smoke-testing stage, but supplement it with attack-based evaluation methods (e.g., membership inference, attribute inference, reconstruction attacks) and, where feasible, formal differential privacy guarantees to achieve a rigorous privacy assessment.

Sources:

• The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against ‘Truly Anonymous’ Synthetic Datasets

• The DCR Delusion: Measuring the Privacy Risk of Synthetic Data

2. Attack-Based Evaluation: Adversarial Testing

Attack-based evaluations assess privacy risks by testing how vulnerable or safe synthetic data is to privacy attacks. Before interpreting attack results, consider your threat model: what are the attacker’s goals, background knowledge, and available resources?

Scope and Future Development

This guide scopes release-only and black-box settings—attackers can access the synthetic dataset and possibly query a synthesizer API but cannot inspect internals (weights, gradients, code). Future revisions will add white-box scenarios, where internals are visible, enabling gradient-based membership tests and model-inversion/reconstruction with internal signals. We also note orthogonal attack surfaces—data poisoning during training and query/prompt injection at inference—which depend on an attacker’s ability to modify or influence the system rather than merely observe it, and can arise under either black- or white-box access.

Common “adversarial tests” include:

• Membership inference: Determine whether a target individual’s record was used in training by probing model outputs or analyzing the synthetic release.¹

• Attribute inference (sensitive-attribute disclosure): Infer a hidden/sensitive attribute about a target using patterns learned by the synthesizer or models trained on the synthetic data.

• Record reconstruction: Recover an approximate training-set record (e.g., a near-duplicate) from model behavior or the synthetic dataset.

Figure 2: Privacy attack goals against synthetic data arranged along an information-gain spectrum: membership inference, attribute inference, and record reconstruction.

Attack-based testing² provides useful risk signals because it works under realistic attacker assumptions, but it must be framed within a defined threat model to avoid over- or underestimating risk.

Evaluating Worst-Case Privacy Risks

When simulating privacy attacks, the choice of target records matters. Research demonstrates that launching attacks against randomly selected records can mask true worst-case risks. Averaging attack success rates across a dataset hides highly vulnerable outliers, whereas rational attackers naturally focus on the easiest, highest-confidence targets.

To accurately measure worst-case privacy risk, evaluations should target the most vulnerable records rather than relying on a random sample. Research shows that the “mean distance to a record’s closest neighbors” is a reliable measure of vulnerability, outperforming traditional ad-hoc methods (such as random sampling, rare attribute value and lowest log-likelihood outlier records). Consequently, attack simulations should focus on the records with the highest mean distance to their closest neighbors.

Source:

• Achilles’ Heels: Vulnerable Record Identification in Synthetic Data Publishing

Getting started with privacy risk evaluation for tabular synthetic data

These tools operationalize the two-pronged evaluation approach described above. Each targets different privacy vectors:

• Anonymeter — Simulates singling out, linkability, and attribute inference attacks.

• DOMIAS — A density-based membership inference attack that detects local overfitting in generative models. It estimates the synthetic data density around a target point and compares it to the true data distribution (approximated by D_control). Unnaturally high local synthetic density signals memorization rather than smooth distribution learning.

• SDMetrics — Part of the SDV ecosystem. Offers empirical and attack-based privacy metrics.

• SynthCity — Offers empirical and attack-based privacy metrics.

Interpreting Privacy Test Results

Privacy test results require careful contextual interpretation. The following guidelines apply across all tools and evaluation approaches:

• Consider your specific situation: Test results depend on who might attack your data and how they would use it. For e.g., if you’re sharing data publicly, privacy attacks are more concerning than if you’re only using synthetic data internally within your organization.

• High scores don’t always mean high risk: A tool might report that an attack “succeeded,” but this could require the attacker to have unrealistic access to information or computational resources. Consider whether the attack scenario actually applies to your real-world situation.

• Dataset specifics: Results vary significantly based on data size, dimensionality, and data quality. For e.g., if your real dataset is small or has many unique records, privacy tests will often show high vulnerability regardless of how good your synthetic data generator is. This reflects the inherent challenge of protecting privacy in small, diverse datasets.

• Baseline comparisons: Always compare against appropriate baselines (e.g., attacks on real data) to interpret results meaningfully. If attacks succeed equally well on real and synthetic data, the issue may be dataset characteristics rather than privacy leakage.

Differential Privacy: Formal Guarantees

Differentially private

synthesizer training provides a mathematical guarantee that the trained model parameters, and any subsequent model outputs, are relatively unaffected by the addition, removal or change of any single user’s training examples. Unlike empirical or attack-based testing, DP’s protection does not rely on assumptions about attacker behavior.

Figure 3: Synthetic data generation without and with differential privacy (DP). (a) Non-DP: a model learns patterns from real data to sample synthetic records—often high utility but no formal privacy guarantee and potential inference/memorization risks. (b) DP: training uses clipping and calibrated noise (tuned to (ε,δ)), yielding a model and synthetic outputs with a formal individual-level DP guarantee, typically with some utility trade-off.

Image reference:

On Renyi Differential Privacy in Statistics-Based Synthetic Data Generation

The guarantee works by adding carefully calibrated randomness (noise) to the synthesizer’s training process measured by privacy parameters ε (epsilon) and δ (delta). Most practical deployments use (ε,δ)-DP where δ accounts for the small probability of privacy failure. Smaller ε values mean stronger privacy but typically lower utility. Intuitively, DP ensures that each person’s data has almost no impact on the final synthetic dataset, making it highly uncertain whether their information was included at all.

When a synthesizer protects privacy using DP while generating data, that process is differentially private synthetic data generation (DP-SDG).

Getting started with DP-SDG for tabular data

Packages that offer DP-SDG models include

OpenDP’s SmartNoise ,

Gretel Synthetics,

and

SynthCity .

For robust DP auditing, DP-Auditorium, developed by a team at Google, provides comprehensive tests to verify that DP mechanisms correctly protect privacy by analyzing output changes under data perturbations and detecting implementation bugs resulting in privacy violations.

An Illustrative Example of Using DP for LLM Fine-tuning For Text Generation

Figure 4: Researcher from Microsoft fine-tuned an LLM with DP on private data corpus. The model can be used to generate synthetic examples that resemble the private corpus.

Image reference:

Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe

An Example of Using Differentially Private Gaussian Copula for Tabular Generation

While various models can incorporate DP for synthetic data generation, the Differentially-Private Gaussian Copula serves as a practical example for tabular data. It introduces quantifiable mathematical bounds on privacy loss, providing rigorous, tunable safeguards against individual re-identification.

Copula-based models are computationally efficient and mathematically interpretable alternatives to resource-intensive methods like deep neural networks. They reduce training time and hardware requirements while mitigating the overfitting and mode-collapse risks common in neural networks. By utilizing Sklar’s Theorem, this approach fundamentally decouples the distribution of individual variables from the underlying dependency structure that binds them together.

How Gaussian Copula Works

Gaussian Copula offers a fast, easily interpretable way to replicate the global statistical properties of a dataset by capturing dataset-wide rank correlations through a single global matrix. Commonly implemented in libraries like the Synthetic Data Vault (SDV), the generation relies on a two-step framework:

1. Marginal Modeling: The algorithm models the marginal distributions of each column independently, transforming the data into a uniform space, and subsequently into a standard normal space.
2. Global Covariance: It computes a single, global covariance matrix in this latent space, effectively capturing the pairwise rank correlations of the source dataset.

To generate new data, the model samples from a multivariate normal distribution mapped to this matrix and applies the corresponding inverse transformation to each column to recover the mapped values in their original marginal distributions.

Applying Differential Privacy

During the training phase of a Differentially Private Gaussian Copula, formal privacy guarantees are achieved by injecting calibrated statistical noise into the dataset’s foundational statistics. This noise is injected using either the Laplace mechanism to achieve pure ε-Differential Privacy, or the Gaussian mechanism to yield approximate (ε,δ)-Differential Privacy. Specifically, the algorithm computes one-way marginal distributions and two-way attribute conjunctions, adding this calibrated noise to the query results rather than the raw data itself. These noisy measurements are then used to estimate the latent global covariance matrix. However, because independent noise injection often distorts the matrix and breaks its mathematical validity, an essential post-processing step projects it to its nearest positive semi-definite equivalent. Sampling from this structurally valid, privatized copula guarantees quantifiable privacy protection. Yet, like all DP mechanisms, it inherently introduces a privacy-utility trade-off, as both the injected noise and the subsequent matrix projection may degrade the statistical fidelity of the generated data compared to a standard non-private model.

Notable Real-World Deployments of DP Synthetic Data

1. Israel’s Ministry of Health (2014 Live Births Dataset): In February 2024, Israel’s Ministry of Health released a DP synthetic dataset of its National Registry of singleton live births from 2014. The dataset, designed in collaboration with researchers and stakeholders, is protected by DP with an ε = 9.98, balancing usability with privacy safeguards. Read more in Real Deployments chapter.

2. Microsoft & International Organization for Migration (IOM): In late 2022, the IOM and Microsoft Research partnered to release the Global Victim–Perpetrator DP synthetic dataset, a privacy-preserving dataset capturing cases of human trafficking. It reflects statistical properties of the real sensitive records without exposing identities.

3. UNHCR Registration Data: UNHCR applied DP synthetic data to registration datasets of displaced individuals, enabling secure sharing of sensitive refugee data for humanitarian program planning. The organization found this methodology particularly effective for full enumeration datasets and developed a practical guide for implementation.

4. Google’s On-Device Safety Classifier: In 2024, Google published an approach using DP-SDG to train on-device safety classifiers for language models. By using DP, they ensure that the synthetic data matches sensitive real data without risking user privacy.

More on Risk Reduction in Practice: Federated Learning and Privacy Filters

Synthetic data works best as part of a layered privacy strategy. Rather than relying on generation alone, combine multiple techniques to reduce single points of failure. Some approaches include:

Federated Learning (FL) keeps raw data at its source: training runs locally, and only model parameter updates (i.e., updates to the synthesizer) are shared centrally (combined at a central server), reducing exposure risk. Multiple organizations can collaboratively train a synthesizer without directly sharing their datasets.

Privacy Filters reduce risk before or after synthesis. Before SDG, by suppressing rare values (removing uncommon entries), dropping PIIs, generalizing identifiers (making specific details less precise), or removing outliers that could create unique signatures in the training data. After SDG, by filtering synthetic records that too closely match real ones, removing unrealistic combinations (unrealistic data patterns), or applying additional anonymization techniques (e.g., k-anonymity grouping, noise addition, or geographic rounding).

Refer to our detailed case study on Google’s Gboard deployment for an example of how synthetic data, FL, DP and privacy filters can be combined to train on-device typing correction models while preserving user privacy.

Privacy isn’t a binary yes-or-no question—it’s about managing risk based on your specific context and threat model. Start with simple smoke tests to catch obvious issues, use attack-based evaluations for realistic threat assessment, and consider DP-SDG when you need formal guarantees with strong privacy protection.

---

The next chapter shifts focus to exploring practical synthesis considerations by taking a systematic, pipeline-driven approach that helps you generate useful synthetic data for your specific applications.

Footnotes

1. Membership inference is trickier for data such as images, videos and audio. For e.g., even if a synthetic image is far away (in terms of some distance metric) from a real image, it could be perceptually similar, leaking the sensitive identity information. So “perceptual similarity” evaluation is critical for such data. ↩

2. In a recent empirical work, the authors systematically performed various attacks on synthetic data with differential privacy guarantees and demonstrated that it can still leak sensitive information. However, we argue that gaps in how differential privacy was applied may have contributed to these failures and then there is lack of details on the implementation. ↩