NOVA

Banks in West Africa hold the data that could power local AI. They cannot share it. Privacy law, customer trust, and competitive advantage keep loan books and transaction histories on bank servers. For rural borrowers, the informal economy, and most of the interior, though, the data does not exist anywhere to begin with. I did not want to build another model that copies a dataset somebody already has. I wanted two things. Learn a real dataset when one exists. Generate realistic data from domain knowledge when none does. Privacy-safe either way, with the validation numbers shown rather than claimed.

• The CTGAN paper (Xu et al., 2019) is the basis for the model. Two ideas make tabular GANs work: mode-specific normalisation for multimodal continuous columns, and conditional training-by-sampling for imbalanced categoricals. Neither is obvious, and both are implemented from the paper rather than inherited from a library.
• The synthetic-data survey (Jordon et al., 2022) and the SDMetrics scoring conventions set the evaluation. They are why the first validation run was measuring the wrong thing. A KS p-value is meaningless at n=10,000. The field scores the KS statistic, the effect size, instead.
• WGAN-GP (Gulrajani et al., 2017) is the stabiliser. Plain GAN training on tabular data is unstable. The gradient penalty keeps the critic honest, and PacGAN packing is a cheap defence against mode collapse.
• TimeGAN (Yoon et al., 2019) covers sequential data. This project is tabular, so it was not used. It is the right next step for real fraud data.
• West African microfinance dynamics set the calibration. Loan sizes run 50 to 2,000 USD. APRs sit at 8 to 30 percent. Default rates at 20 to 30 percent. Group-lending lowers that. Rural borrowers skew agricultural, urban borrowers skew trading. None of this is in the architecture papers, and without it the ground-truth data would be a fantasy.

• No real financial data. Banks do not hand loan books to students, and the compliance burden makes it a non-starter even when someone is willing. Every number had to be either synthetic-by-design or sourced. Nothing invented and passed off as real.
• No generic training set either. German Credit and Kaggle loan dumps do not reflect West African dynamics: group lending, sector-specific default, the informal economy. A model trained on those learns the wrong structure, and I would not be able to defend it.
• No GPU. Everything trains on a CPU laptop, so the architecture and the training budget had to stay CPU-feasible. That ruled out the largest networks and forced real choices about width and epochs.
• The deployment target is a 512 MB machine. PyTorch plus a RandomForest validation pass is heavy. The backend had to fit a tight memory and image-size budget. That shaped the Docker build and how much I generate per request.

1. Decision 01

   Build CTGAN from scratch in PyTorch, not on top of SDV.

   SDV ships a perfectly good CTGAN. Using it would have been a black box. I needed to understand every piece: mode-specific normalisation, the conditional generator and its cross-entropy term, the gradient penalty, the training-by-sampling loop. So I implemented all of it from the paper, with no sdv or ctgan import. That is also what later made the criteria-engine mode possible, since the generator internals were mine to reuse.

2. Decision 02

   Build the ground truth as a structural-causal model, not independent draws.

   Sampling every column independently and forcing a correlation matrix on top produces incoherent data. Validating that a GAN preserves correlations becomes meaningless. So the relationships run through shared latent drivers: education and age load onto income, income onto loan size, risk drivers onto a calibrated default logit. Income really does drive loan size. The generator verifies all ten target correlations and seven integrity constraints on every run. Nested default flags, a borrower cannot default more loans than they took, valid ranges. It fails loudly if a change breaks them.

3. Decision 03

   Validate four ways, and pick the metric that tells the truth over the one that flatters.

   I scored four things rather than one, because they measure different properties: statistical similarity, correlation preservation, train-on-synthetic-test-on-real utility, and privacy. Shape, relationships, usefulness, safety. Twice I changed the metric itself rather than the result. Effect size instead of KS p-values. Distance-to-closest-record instead of a real-versus-synthetic detector. The convenient metric was lying both times.

4. Decision 04

   Make NOVA two modes, and refuse to brute-force the second one with seven more GANs.

   Copying a real dataset is one problem. Generating data for populations where none exists is a different problem, and it needs a rule engine, not a GAN. So I built a criteria engine: define columns, distributions, domain rules, and it generates from nothing. Later, the brief asked for seven separate CTGANs, one per financial domain. I did not train them. Roughly a day of CPU, seven model files, and a weaker create-from-nothing story. The seven domains became presets of the criteria engine instead. A user can define an eighth, or rural-Gambia student records, themselves.

5. Decision 05

   Ship a deployed web app on a real backend, not a notebook.

   A notebook serves an analyst. A running app serves whoever needs the output. The model is served by a FastAPI backend behind a Next.js studio. Upload a CSV and copy it, or pick a domain and create from rules. Getting it live meant solving the unglamorous parts: a CPU-only torch image, a checkpoint that only unpickles under the right library versions. Those decide whether a thing runs in production or just on my machine.

• My first validation run reported a 4 percent statistical pass rate. That looked like failure. It was the metric, not the model. At n=10,000 a KS p-value collapses to near zero for differences too small to matter. I switched to effect size, mean column-shape similarity, which is sample-size independent and the SDMetrics convention. Kept the p-values as context. Real fidelity: 0.94.
• The CTGAN over-produced defaulters. 41 percent against a real 25. Cause: log-frequency conditioning, which I had used to help rare categories during training. It taught the generator a more balanced default base rate than reality. I switched to true-frequency conditioning and added conditional generation so a user can dial the rate directly. The marginal snapped back to real. TSTR rose from about 0.82 to 0.92.
• Two columns had a problem. `interest_rate_apr` and `interest_rate_daily` were independent in the generator. They should not be. Daily is apr divided by 36,500, a deterministic identity the GAN had no reason to respect. I added a post-processing step that re-imposes known identities. Synthetic data is now internally consistent instead of subtly contradictory.
• My first privacy metric was a real-versus-synthetic detector. It scored about 0.99. I almost reported it as a privacy failure. It is not a privacy metric at all. It measures distinguishability, and a model that simply memorised the training data would be undetectable yet maximally unsafe. I switched to distance-to-closest-record: are synthetic rows abnormally close to real training rows? Ratio 1.10, with 1.1 percent near-duplicates. Kept the detector on as a fidelity diagnostic.
• First Docker build was heading for 2.5 GB. The default PyTorch wheel bundles CUDA I never use on a CPU box. Installing the CPU-only wheel cut the image to 367 MB. Then the container crashed on boot. The checkpoint had been pickled under NumPy 2.x and scikit-learn 1.7. Pinning the older versions the brief suggested raised 'No module named numpy._core'. Matching the library majors fixed it.

• I did not appreciate how much of this is domain knowledge rather than architecture. The CTGAN paper gives you the network. It does not tell you that microfinance default sits at 20 to 30 percent, that group lending lowers it, or that rural and urban borrowers borrow for different things. I had to learn the domain before the data meant anything.
• Conditional sampling was the lever for utility, and I missed it the first time. My train-on-synthetic score was 0.81 and I assumed the model was under-trained. It was not. I was generating wrong. Sampling the target freely instead of conditioning on the reference prevalence. The same checkpoint jumped to 0.92 once I used it properly.
• The four metrics are not redundant, and I only learned that by watching them disagree. A model can match every marginal and still break the correlations. It can be useful for prediction and still sit too close to a real row. Seeing which one disagreed is what stopped me from chasing a single score.
• I started out thinking synthetic data meant copying. Building the criteria engine changed that. The more useful thing is generating data that should exist but does not. Different problem, different tool. I would not have seen it if I had stopped at the GAN.

• A fifth measure the headline four leave out: a classifier tells real from synthetic 89% of the time against a 0.50 baseline. Reported as a diagnostic rather than a gate.
• A structural-causal ground-truth set: 10,000 rows by 29 columns of realistic West African microfinance data, with all ten correlations and seven integrity constraints verified on every generation
• A CTGAN implemented from scratch in PyTorch. Mode-specific normalisation, PacGAN critic, WGAN-GP, conditional training-by-sampling, early stopping. No SDV or ctgan import.
• All four validation metrics pass: statistical similarity 0.94, correlation L1 0.05, TSTR 0.92 (AUC ratio 0.94), and distance-to-closest-record privacy 1.10 with 1.1 percent near-duplicates
• A criteria engine that generates data from columns, distributions, and domain rules with no source dataset, behind a whitelist expression evaluator so user-supplied rules cannot inject code
• Seven financial-domain presets for create-mode: banking, payments and fraud, insurance, remittances, macro indicators, wealth, and corporate statements
• A FastAPI backend serving both copy and create endpoints, live on Fly.io. A Next.js studio with a Create/Copy toggle, live on Vercel. All open source.

The backend runs on Fly's no-card trial, so the machine sleeps after five minutes and cold-starts on the next request. Making it always-on is a card away. 512 MB is tight for the heavier copy-mode validation. DCR shows no memorisation empirically, but it is not a formal guarantee. I would add a differentially-private training option for a formal guarantee. Transaction data here is tabular. Real fraud is sequential, which is why I read TimeGAN. Extending both modes to time series is the next build. I would also push TSTR across all seven domains rather than mainly loans, and add a third mode that blends a handful of real rows with domain rules, for the user who has a little data and a lot of knowledge.