Evidence

What we have shown, and what is still to test.

A prototype of the engine reproduces published attendance figures in two school systems. Whether it predicts what a decision will do is tested at later gates, with serving school leaders and real cohorts.

School systems calibrated: 2
Published figures matched within 0.2 points: 5 of 5
Fresh synthetic schools tested in each system: 200
Processes the prototype covers so far: Attendance

Calibration

Fitted to published figures, then tested on fresh schools.

In each system, the engine was fitted on a pool of 80 synthetic schools. It then ran 200 fresh schools from new random seeds. The fit holds within 0.2 points of every published figure.

System A · secondary schools, 2024–25

Attendance, students from the most deprived fifth of areas

Published

84.2%

Engine

84.4%

Attendance, students from the least deprived fifth of areas

Published

92.4%

Engine

92.4%

Persistent absence, missing 10% or more of sessions

Published

37.4%

Engine

37.4%

System B · state-funded secondary schools, 2024–25

System B calibration: targets and engine results, state-funded secondary schools, 2024–25
Measure	Target	Basis of target	Engine, mean of 200 fresh schools
Attendance, all students	91.56%	Published	91.42%
Persistent absence, missing 10% or more of sessions	23.37%	Published	23.57%
Attendance, students eligible for free school meals	86.73%	Worked out from the all-schools ratio	86.52%
Attendance, students not eligible for free school meals	93.24%	Worked out from the all-schools ratio	93.12%

Two of System B's four targets are published. The other two split attendance by eligibility for free school meals. No secondary-only figure could be read, so both were worked out from the ministry's ratio for all schools, and the engine lands within 0.21 points of each.

Engine figures are the mean of 200 synthetic schools generated from new random seeds after fitting. Published figures for System A and System B: Scottish Government, School attendance, absence and exclusions statistics 2024–25; Department for Education, Pupil absence in schools in England 2024/25 (26 March 2026). The ratio comes from Pupil attendance in schools, end of the 2024/25 academic year (7 August 2025). The share eligible for free school meals, 25.85%, comes from Schools, pupils and their characteristics 2024/25.

Spread

A school system is more than its average.

A simulation can match a mean and still get the schools wrong. So the engine is also checked on how much its schools differ, and on two figures the fit never aimed at.

Persistent absence across secondary schools

System B, 2024–25

Synthetic schools, 10th to 90th percentile17.7% to 28.9%

System B's 152 local areas, 10th to 90th percentile19.5% to 27.2%

System A, 2024–25

300 synthetic schools, 10th to 90th percentile32% to 43%

System A's local areas, lowest to highest25.4% to 45.8%

Published figure for the system: 23.37% in System B, 37.4% in System A Persistent absence means missing 10% or more of sessions. System A's range runs from lowest to highest, a wider measure than a 10th to 90th percentile range.

Wider than area averages, as it should be

In System B, synthetic schools spread more widely than local area averages. Single schools should vary more than the areas they sit in.

A gap the fit did not aim at

In the engine, students eligible for free school meals are 2.73 times as likely as others to be persistently absent. For all schools, the Department for Education puts the figure at "over two and a half times".

The prototype

The properties the product depends on.

Replay

The same seed and pack give identical snapshots on every run.

Speed

A year of attendance for a 1,000-student school runs in about 0.02 s. The same year for 300 schools takes about 5 s.

Clean branching

Two branches share every random draw until a decision differs. Measured that way, the effect of supporting 40 students varied a quarter to a third as much as when each branch drew its own luck.

Scope

The prototype covers attendance only. The full engine adds the cast, the school leader's chair and the debrief, and each stage passes a gate before the next begins.

Validation

Nine checks, and every gate uses at least one.

Validation tests whether outcomes match published figures and whether agents act like the people they play. Every pass mark is a starting figure, to be revised after the first panel round. The results of each gate are published as a short note, so buyers and peers can check them.

The nine validation checks and the gate at which each one applies
Check	Question	How it works	Gate
Calibration	Do the engine's processes produce real distributions?	Each process is fitted to published figures for its system, checking spread and relationships as well as averages, such as the gap by deprivation. Any published pattern it misses is reported.	Gate 1
Noise floor	Do outcomes land where experienced school leaders expect?	Leaders predict what happens next in five scenarios. The engine passes when, in at least four of the five, its median falls inside their middle half and its spread is no wider. Leaders who helped tune a rule never score it.	Gate 2
Recognition panels	Do serving school leaders find the school recognisable and usable?	Leaders judge a blind mix of agent and human transcripts. The agents pass if leaders pick them out no more than 60% of the time, and every round reports how far the raters agree. A pass shows the school is recognisable and usable, and does not count towards validity.	Gate 2
Traceability	Can any outcome be explained?	Every outcome links to the rules, random draws and stored model replies that produced it.	Every gate
Retrodiction	Does an area started from its published figures follow its later results?	Synthetic schools built from a district's published figures replay an earlier year. The result is compared with what that district went on to publish, and no school is named or reproduced.	After Gate 3
Identical-policy test	Do two branches with the same policy stay identical?	Two branches run the same policy from one save point. Any difference in any outcome fails the run, because it means the shared random draws have fallen out of step. The test runs on every engine change.	Gate 1
Sensitivity analysis	How far do results rest on the engine's assumptions?	Each behavioural assumption is varied across its published range. Every policy-lab result reports that spread beside its paired intervals, which measure simulation noise only.	Gate 1
Identity audits	Are some groups portrayed worse than others?	Panels review personas and sampled narration group by group, students with additional support needs included. The persona tests repeat their probes for each group.	Gate 2
State checks	Does what agents say match what the engine holds?	Sampled lines are checked against the engine's state at that moment, such as a student's attendance or a budget line. Every mismatch is logged and fixed.	Gate 2

Where the methods come from

Calibration checks spread as well as averages, because synthetic survey samples have matched human means while varying far less (Bisbee et al., 2024).
The noise floor follows a study that scored agents against people's own consistency when they were retested two weeks later (Park et al., 2024).
Blind mixes follow the Social Science Turing Test, one of four criteria for algorithmic fidelity (Argyle et al., 2023).
Each process keeps its own random stream, because a single generator falls out of step once one branch draws a number the other does not (Klein et al., 2024).
Sensitivity analysis follows the call for models to report how far their results rest on their assumptions (Saltelli et al., 2020).
Identity audits answer evidence that language models flatten and misportray identity groups (Wang et al., 2025). State checks answer the 59.9% result for GPT-4 described below (Wang et al., 2024).

What the research says

The design rests on evidence from other fields.

Published research supports how Synthetic School is built more firmly than any claim it could make about leaders or students. No study yet links simulation-based leadership preparation to leaders' practice or students' outcomes, and most of the evidence comes from the United States.

Most new leaders start underprepared

Only 27% of new principals in New Zealand felt prepared or very prepared when they started (Education Review Office, 2023). In the UK, "most headteachers do not feel fully prepared when they first take on the role" (Greany et al., 2025).

Leaders' measured effects shrink under newer methods

In six US studies, a principal one standard deviation more effective added 0.13 SD a year in maths (Grissom et al., 2021). A 2024 re-estimate finds about 0.03 SD (Bartanen et al., 2024, read here through a reproduced abstract).

Simulation training works, with smaller gains against active teaching

Across 609 health-professions studies, effects of simulation against no intervention fell from 1.20 for knowledge to 0.50 for direct effects on patients (Cook et al., 2011). Against other instruction in emergency medicine, effects were 0.33 or less and not significant (Ilgen et al., 2013).

Debriefs and decisions count for more than polish

Debriefs improved performance by about 25% (d = 0.67) across 46 samples (Tannenbaum and Cerasoli, 2013). High- and low-fidelity simulators differed by 1% to 2% across 24 health studies (Norman et al., 2012).

Language-model agents reproduce averages better than individuals

Synthetic survey samples had means near human averages but far less spread (Bisbee et al., 2024). GPT-4 predicted the full next state in at most 59.9% of text-world transitions that changed it (Wang et al., 2024). That is the case for letting the engine decide.

Attendance support cuts a few days a year

Randomised parent mailings in the US cut absence by about 6% to 8% (Rogers and Feller, 2018; Robinson et al., 2018). In a Chicago trial, mentoring cut absences in grades 5 to 7 by 4.22 days, or 22.9% (Guryan et al., 2020, working paper). It had no detectable effect in grades 1 to 4.

Evidence for each design choice

Ratings are for a school-leadership setting, where most of the evidence is indirect.

Strength of evidence for each design choice
Design choice	Main support	Main challenge or gap	Strength
A trainee leads a whole synthetic school	Novice leaders struggle with the volume and unpredictability of problems (US, New Zealand)	No study links simulation to leaders' practice or students' outcomes	Weak and indirect
AI agents play the staff and families	Agents remove the cost of live actors (US)	Identity groups flattened; sycophancy; drift; no school-specific tests	Moderate for cost, weak for realism
The engine decides and agents narrate	GPT-4 got at most 59.9% of state-changing transitions right; model effect sizes run high	No head-to-head test	Strong argument, untested directly
Weekly save points with shared draws	Per-decision random streams cut runs needed by up to four orders of magnitude in one model	No study of branching replay for learning	Strong for the statistics, weak for learning
Debriefs	d of 0.67 to 0.79 in meta-analyses; coaching gave 1.44 SD in mixed reality (US)	Gains measured inside simulators; AI-led debriefs untested	Moderate, from adjacent fields
Decision demands over visual polish	High- and low-fidelity simulators differed by 1% to 2% (health)	No fidelity study for school leadership	Moderate, from health
Calibration to two systems' official statistics	Detailed official data; established validation standards	Over-parameterised models can fit any result; the systems differ by about ten points	Strong data, untested method
A policy lab across hundreds of schools	A Washington State model of teacher sorting	Prediction judged infeasible for most social phenomena	Moderate, for exploring what could happen
The five-point attendance setting	One non-randomised New York City estimate	About twice the best randomised result	Weak, at the top of the range
Scoring trainees, which Synthetic School does not do	Decision logs are more honest than self-ratings	Assessment-centre validity "very low"	Weak

Read: Does simulation training work for school leaders?

The full literature review has 172 references. Read the literature review

The pilot trial

The next test needs real cohorts.

The pilot compares Synthetic School with case-based learning, with the same coach-led debrief in both groups. Learning is judged on new scenarios that trainees have not played, scored blind by two raters.

A small first stage checks that the trial can run. A larger second stage follows only if the first meets its progression criteria.

How the pilot works

Sources cited on this page

Help run the next test.

The pilot trial needs leadership programmes and school groups to run its first cohorts. Pilots are free or at cost while Synthetic School is validated.

Request a pilot place