Holdout-Based Empirical Assessment of Mixed-Type Synthetic Data

Platzer, Michael and Reutterer, Thomas (2021) Holdout-Based Empirical Assessment of Mixed-Type Synthetic Data. Frontiers in Big Data, 4 (79939). pp. 1-12. ISSN 2624-909X

Available under License Creative Commons: Attribution 4.0 International (CC BY 4.0).

Download (4MB) | Preview


AI-based data synthesis has seen rapid progress over the last several years and is increasingly recognized for its promise to enable privacy-respecting high-fidelity data sharing. This is reflected by the growing availability of both commercial and open-sourced software solutions for synthesizing private data. However, despite these recent advances, adequately evaluating the quality of generated synthetic datasets is still an open challenge. We aim to close this gap and introduce a novel holdout-based empirical assessment framework for quantifying the fidelity as well as the privacy risk of synthetic data solutions for mixed-type tabular data. Measuring fidelity is based on statistical distances of lower-dimensional marginal distributions, which provide a model-free and easy-to-communicate empirical metric for the representativeness of a synthetic dataset. Privacy risk is assessed by calculating the individual-level distances to closest record with respect to the training data. By showing that the synthetic samples are just as close to the training as to the holdout data, we yield strong evidence that the synthesizer indeed learned to generalize patterns and is independent of individual training records. We empirically demonstrate the presented framework for seven distinct synthetic data solutions across four mixed-type datasets and compare these then to traditional data perturbation techniques. Both a Python-based implementation of the proposed metrics and the demonstration study setup is made available open-source. The results highlight the need to systematically assess the fidelity just as well as the privacy of these emerging class of synthetic data generators.

Item Type: Article
Keywords: synthetic data, privacy, fidelity, structured data, anonymization, self-supervised learning, statistical disclosure control, mixed-type data
Divisions: Departments > Marketing > Service Marketing und Tourismus
Forschungsinstitute > Kryptoökonomie
Forschungsinstitute > Rechenintensive Methoden
Version of the Document: Published
Depositing User: Gertraud Novotny
Date Deposited: 02 Jul 2021 09:03
Last Modified: 02 Jul 2021 09:03
Related URLs:
FIDES Link: https://bach.wu.ac.at/d/research/results/100567/
URI: https://epub.wu.ac.at/id/eprint/8191


View Item View Item


Downloads per month over past year

View more statistics