Validation & Quality Report
Every generation run is followed by a comprehensive validation report that measures how faithfully the synthetic data reproduced the original's statistical structure. This is what lets you trust a synthetic dataset rather than take its fidelity on faith -- and what you cite when you report that your synthetic data preserved the source's properties.
The validation compares the synthetic data against the targets on three levels -- univariate (each variable's own distribution), bivariate (each pairwise relationship), and multivariate (the overall correlation structure) -- and combines them into a single overall quality score.
The three levels of validation
| Level | What it checks | Examples |
|---|---|---|
| Univariate | Each variable's marginal distribution matches the target | Mean, standard deviation, range bounds, success rate (binary), category frequencies (categorical) |
| Bivariate | Each pairwise relationship matches the target | Correlations, Cohen's d, Cramer's V, odds ratios, hazard ratios |
| Multivariate | The whole correlation structure matches | Correlation Matrix Distance (Frobenius norm) |
Each individual check produces a status of pass, warning, or fail, plus a numeric quality score.
How each metric is scored
The report uses three scoring methods, each suited to a different kind of metric. All three map a deviation between original and synthetic into a quality score in [0, 1] and an interpretation label (excellent / good / acceptable / poor / failed).
Standardised Mean Difference (SMD) -- for means and similar metrics
The SMD expresses the difference between original and synthetic in standard-error units:
SMD = (generated - original) / standard_error
Scored on the absolute SMD, following Cohen-style conventions:
| |SMD| | Quality score | Interpretation | |---|---|---| | < 0.1 | 1.0 | Excellent | | 0.1 -- 0.2 | 0.9 | Good | | 0.2 -- 0.5 | 0.7 | Acceptable | | 0.5 -- 1.0 | 0.4 | Poor | | >= 1.0 | Granular decay | Failed |
For catastrophic deviations (|SMD| >= 1.0), the score decays granularly rather than dropping to zero:
| |SMD| | Score range | |---|---|---| | 1.0 -- 2.0 | 0.20 to 0.10 (linear) | | 2.0 -- 5.0 | 0.10 to 0.05 (linear) | | 5.0 -- 10.0 | 0.05 to 0.02 | | > 10.0 | logarithmic decay toward 0.01 |
Absolute difference -- for normalised metrics
For metrics already on a bounded scale (correlations, Cramer's V), the report uses the absolute difference:
absDiff = |generated - original|
| absDiff | Quality score | Interpretation |
|---|---|---|
| < 0.05 | 1.0 | Excellent |
| 0.05 -- 0.10 | 0.9 | Good |
| 0.10 -- 0.30 | granular | Acceptable |
| 0.30 -- 0.50 | 0.30 to 0.15 | Poor |
| 0.50 -- 0.70 | 0.15 to 0.10 | Poor |
| > 0.70 | decays toward 0.05 | Failed |
Log-ratio difference -- for ratio metrics
For ratio measures (odds ratios, hazard ratios), the report compares values on the log scale:
logDiff = |ln(generated) - ln(original)|
| logDiff | Quality score | Interpretation |
|---|---|---|
| < 0.14 | 1.0 | Excellent |
| 0.14 -- 0.22 | 0.9 | Good |
| 0.22 -- 0.41 | 0.7 | Acceptable |
| 0.41 -- 0.69 | 0.4 | Poor |
| >= 0.69 | granular decay | Failed |
A log difference of 0.69 is approximately ln(2), meaning the synthetic ratio is about double or half the target -- the boundary at which the metric is judged to have failed.
Univariate checks
For each variable, the report verifies that the synthetic marginal matches the target:
| Variable type | Checks |
|---|---|
| Continuous / time-to-event | Mean (SMD), Standard Deviation, Range Bounds (whether synthetic values stay within expected min/max) |
| Binary | Success rate -- the proportion of 1s vs 0s, compared against original counts |
| Categorical | Category frequency -- each category's proportion compared against the original |
Bivariate checks
For each configured relationship, the report compares the synthetic effect size against the target:
| Relationship | Metric | Scoring method |
|---|---|---|
| Continuous x continuous | Correlation | Absolute difference |
| Continuous x binary/categorical | Cohen's d | SMD-style |
| Categorical x categorical | Cramer's V | Absolute difference |
| Binary x binary | Odds ratio | Log-ratio |
| Time-to-event x binary | Hazard ratio | Log-ratio |
A pairwise correlation check passes when the synthetic and original correlations differ by less than 0.10.
Multivariate check -- Correlation Matrix Distance
To assess the whole correlation structure at once, the report computes the Frobenius norm of the difference between the original and synthetic correlation matrices:
Matrix Distance = sqrt(sum_ij (R_original[i][j] - R_synthetic[i][j])^2)
This single number summarises how well the entire web of correlations was preserved. The check passes when the matrix distance is less than 0.10.
Status thresholds
Each check is assigned a status:
| Status | Meaning |
|---|---|
| Pass | The metric is reproduced within the tight tolerance (e.g. correlation difference < 0.10, matrix distance < 0.10) |
| Warning | The metric is reproduced within an acceptable but looser band |
| Fail | The metric deviates beyond the acceptable band |
The report tallies the number of passed, warning, and failed checks.
The overall quality score
The individual checks are combined into a single overall quality score in [0, 1], using a weighted average across the three levels:
overallQuality = 0.25 * (univariate quality)
+ 0.50 * (bivariate quality)
+ 0.25 * (multivariate quality)
The bivariate level carries the most weight (50%) -- a deliberate choice, because the relationships between variables are usually what matter most for downstream statistical inference. Getting each variable's marginal right (univariate, 25%) and the overall correlation web right (multivariate, 25%) round out the score.
Interpreting the overall score
| Overall quality | Reading |
|---|---|
| >= 0.70 | The synthetic data reproduced the targets well |
| < 0.70 | Some metrics failed; the report flags them and recommends reviewing data quality, configuration, or sample size |
When quality falls below 0.70, the report generates recommendations identifying which categories of metric failed (e.g. categorical frequencies, binary success rates, Cohen's d effects, correlations, Cramer's V, or odds ratios) so you know exactly what to investigate.
Visual validation
Alongside the numeric report, the module produces visual diagnostics:
| Visualisation | What it shows |
|---|---|
| Love plot | Standardised mean differences for every variable on a single plot -- the standard visual for confirming that synthetic and original marginals align (SMD near 0 indicates good balance) |
| Correlation matrices | Original and synthetic correlation matrices side by side, so structural differences are visible |
| SMD comparison | Standardised-mean-difference comparisons across variables |
The Love plot is the conventional way to present synthetic-vs-original balance in a publication -- every variable's SMD should sit close to zero if the synthetic data is faithful.
How validation connects to the optimisation loop
The validation report is not just an after-the-fact summary -- it is the objective function of the optimisation loop. Each iteration of generation produces a validation report, and its overall quality score drives the loop:
- If the score reaches the target: the loop stops (target_reached)
- If 3 iterations pass without reaching the target: the loop stops (max_iterations)
- If an iteration fails to improve on the previous score: the loop stops (no_improvement)
The report you see at the end reflects the best of up to three generation attempts, each scored by exactly the metrics documented above.