Skip to content
GovTech GTO Practices Portal

3. Tools, Performance & Deployment

Beyond the concepts above, we carried out practical work in early 2023 to help practitioners apply differential privacy: a comparison of open-source tools, an empirical benchmark of their accuracy and scalability, and a guide to deploying them at scale on the cloud. The summaries below describe what each covers and the durable lessons from them. The full detail remains available in the original articles and code repository.

Comparing Differential Privacy Tools

Section titled “Comparing Differential Privacy Tools”

We evaluated four open-source tools that bring differential privacy to non-experts: OpenDP (Harvard), Tumult Analytics (originally Tumult Labs, now part of the OpenDP project), PipelineDP (Google and OpenMined), and Diffprivlib (IBM). The comparison assesses each across four lenses that remain a useful way to evaluate any differential privacy tool:

  • Analytics: the statistics and query types supported, including aggregate statistics and GROUP BY queries.
  • Security: use of cryptographically secure randomness and protection against floating-point vulnerabilities.
  • Usability: scalability, accuracy adjustment, parameter search, and pre- and post-processing.
  • Differential privacy features: privacy-budget accounting, available mechanisms, privacy definitions, and composition.

It closes with guidance on matching a tool to your priorities, whether that is high security, feature breadth, or automated parameter search. The specific feature-by-feature comparison reflects early-2023 versions; read it as a snapshot.

Full detail: Part 2: Practitioners’ Guide to Accessing Emerging Differential Privacy Tools.

Benchmarking Tool Performance

Section titled “Benchmarking Tool Performance”

We benchmarked the same four tools on utility (accuracy) and scalability, using synthetic datasets that varied in scale, skew, and size, across the count, sum, mean, and variance queries and a range of ε values. Utility was measured with mean relative error and scalability with execution time, in both standalone and distributed (Spark) environments. The durable lessons are methodological:

  • Accuracy depends on more than ε. The scale, skew, and size of the data, and the bounded or unbounded neighbouring definition a tool uses to compute sensitivity, all affect the result.
  • Tool rankings shift with data characteristics, so there is no single best tool across all queries and settings.

The specific accuracy figures and head-to-head rankings reflect the tool versions tested in early 2023 and should not be cited as current.

Full detail and open-sourced experiments: Part 3: Evaluating Differential Privacy Tools’ Performance, and the benchmarking repository.

Deploying at Scale on the Cloud

Section titled “Deploying at Scale on the Cloud”

We documented how to run the two distributed-capable tools, Tumult Analytics and PipelineDP, at scale on Amazon Web Services, using AWS Glue (serverless) and Amazon EMR (managed clusters), with a worked example computing a differentially private mean over US Census data. The durable takeaway is the deployment pattern: keep raw data in your own storage, run the differential privacy computation on managed Spark, and treat the privacy budget as a governed resource.

The specific service versions, package pins, and console steps have since changed and should be taken from current AWS documentation and the code repository rather than reused as-is.

Full detail: Part 4: Getting Started with Scalable Differential Privacy Tools on the Cloud, and the deployment scripts.

Where to Find the Original Material

Section titled “Where to Find the Original Material”