The Generation Engine (Statistical Methods)
This section documents how the synthetic data is actually generated -- the algorithms, formulas, and the order in which they run. It is a complete account of the engine's mathematics, suitable for citing in a methods section.
The engine's job is to produce a dataset that simultaneously satisfies three things: each variable's marginal distribution, the correlation structure among continuous variables, and the effect-size targets between variables. It does this through a multi-phase generation pipeline built on Cholesky decomposition, followed by an optimisation loop that refines the result.
The correlated-normal foundation
The core of the engine is the standard technique for generating correlated continuous data: draw independent standard-normal variables, then linearly combine them with the Cholesky factor of the desired correlation matrix.
Standard normal generation -- Box-Muller transform
Independent standard-normal values (mean 0, SD 1) are generated with the Box-Muller transform:
u1, u2 ~ Uniform(0, 1)
z = sqrt(-2 * ln(u1)) * cos(2 * pi * u2)
Each z is an independent draw from N(0, 1).
Cholesky decomposition
To impose a target correlation matrix R on a set of independent normals, the engine computes the Cholesky factor L -- the lower-triangular matrix such that L * L^T = R:
For each i, for each j <= i:
if i == j:
L[j][j] = sqrt(R[j][j] - sum(L[j][k]^2 for k < j))
else:
L[i][j] = (R[i][j] - sum(L[i][k] * L[j][k] for k < j)) / L[j][j]
If a diagonal value would be <= 1e-10 (the matrix is not positive definite), the decomposition returns null and the engine falls back to the SVD method.
Applying the Cholesky factor
Given k correlated variables, the engine generates a k x n matrix Z of independent standard normals, then forms the correlated matrix Y = L * Z:
For each observation:
Y[i] = sum(L[i][j] * Z[j] for j <= i)
The resulting Y rows are standard-normal variables (z-scores) with the target correlation structure. These z-scores are the raw material that subsequent phases transform into the final variables.
Robustness -- non-positive-definite matrices
Real correlation matrices -- especially when manually edited -- are not always positive definite. The engine handles this in two layers:
| Layer | Method |
|---|---|
| Nearest positive-definite correction | Before decomposition, if the matrix is not positive definite, it is replaced with the nearest positive-definite matrix (eigenvalue-based correction that adjusts negative eigenvalues to small positive values) |
| SVD-based fallback | If Cholesky still fails, the engine switches to SVD-based generation, which produces correlated normals without requiring positive definiteness |
This two-layer safety net means the engine produces a valid result even from imperfect or hand-specified correlation matrices.
The multi-phase generation pipeline
The engine generates variables in a deliberate phased order, so that higher-priority relationships are established first:
| Phase | What it does |
|---|---|
| Phase 0 | Generate the independent grouping variables (binary/categorical) that participate in Cohen's d relationships |
| Phase 1 | Generate all continuous and time-to-event variables as correlated z-scores via Cholesky |
| Phase 2 | Apply Cohen's d transformations -- reshape continuous z-scores so the mean difference between groups matches the target |
| Phase 3 | Transform remaining continuous variables (not involved in Cohen's d) to their target distribution via value = mean + std * z |
| Phase 4 | Process categorical-categorical relationships -- generate joint or conditional categorical variables |
| Phase 5 | Generate any remaining independent binary and categorical variables from their marginal distributions |
| Phase 6 | Adjust marginal proportions -- post-process binary and categorical variables so proportions exactly match targets |
Phase 2 -- Cohen's d transformation
For a continuous variable related to a binary group by a target Cohen's d, the engine assigns each observation a value drawn from its group's distribution:
For each observation i:
if group[i] == 1: value = mean1 + sd1 * z[i]
else: value = mean0 + sd0 * z[i]
Because the z-scores already carry the correlation structure from Phase 1, this transformation reproduces the target standardised mean difference while keeping correlations with other continuous variables intact.
Phase 4 -- Categorical relationships
Categorical-categorical associations are reproduced by sampling from the joint probability distribution:
- If neither variable exists yet: generate them jointly from the joint probability table
- If one already exists: generate the other conditionally, using conditional probabilities from the joint table
- The conditional case handles either direction by transposing the joint table
Phase 6 -- Exact marginal proportions
After generation, binary and categorical variables are post-processed so their marginal proportions match targets exactly:
- Binary variables are adjusted to the target proportion p
- Two-category categorical variables are treated as binary and adjusted the same way
- Multi-category variables are adjusted to their target category probabilities
This correction removes sampling error from the marginals that the random sampling in earlier phases introduces.
Distribution generation formulas
The four directly-implemented distribution generators:
| Distribution | Generation method |
|---|---|
| Normal | Box-Muller: value = mean + std * z, where z = sqrt(-2 ln u1) * cos(2 pi u2) |
| Uniform | value = min + (max - min) * u, where u ~ Uniform(0,1) |
| Binomial | Sum of n Bernoulli trials: count of draws where Uniform(0,1) < p |
| Categorical | Inverse-CDF sampling: draw u ~ Uniform(0,1), walk cumulative category probabilities, select the category whose band contains u |
In the multivariate pipeline, continuous and time-to-event variables are both generated through the correlated-normal path (Phase 1 then Phase 3). Binary variables are generated as exact Bernoulli draws, and categorical variables by inverse-CDF sampling.
Bounds enforcement
After a continuous value is generated, it is constrained by the bounds mode via the applyBounds step. If no min/max is defined, no bounds are applied. Otherwise:
Strict mode -- hard-clamped to the original observed range:
value = max(min, min(max, value))
Flexible mode (default) -- range widened by +/-15% before clamping:
range = max - min
flexibleMin = min - 0.15 * range
flexibleMax = max + 0.15 * range
value = max(flexibleMin, min(flexibleMax, value))
Non-negativity protection: if the original minimum is >= 0, the flexible lower bound is floored at 0:
if (min >= 0): flexibleMin = max(0, flexibleMin)
This prevents logically non-negative variables from ever taking negative values in the synthetic data.
The optimisation loop
After initial generation, the engine runs an optimisation loop that measures reproduction quality and refines the result:
| Property | Value |
|---|---|
| Maximum iterations | 3 |
| Per-iteration step | Generate data, validate against targets, compute overall quality score |
| Adjustment | If score is below target, correlations are nudged and data is regenerated |
| Convergence reasons | target_reached (score met), max_iterations (ran out), or no_improvement (iteration failed to improve) |
Each iteration produces a validation report and an overall score. The loop stops when the target score is reached, after 3 iterations, or when no improvement is detected. The best result is kept.
The full generation flow, end to end
A single generation run proceeds as follows:
- Assemble configuration -- variables, distributions, correlation matrix, effect sizes, sample size, bounds mode
- Multivariate generation -- run the 7-phase pipeline:
- Build correlation matrix, correct to positive-definite if needed
- Cholesky-decompose (or SVD fallback) and generate correlated z-scores
- Apply Cohen's d transformations, distribution transformations, categorical relationships, and marginal corrections
- Bounds enforcement -- constrain each continuous value per bounds mode
- Optimisation loop -- validate, score, and refine up to 3 times
- Format output -- assemble the final rows
- Validate -- produce the full validation report
If a variable is left ungenerated by the multivariate pipeline (an unusual edge case), the engine falls back to generating it from its marginal distribution directly, or through a dependent-variable routine that conditions on an already-generated variable.