Summary
A two-stage randomised trial will test whether a term of Synthetic School improves trainee leaders' practice on scenarios they have not seen. The comparison group spends the same time on case-based learning with the same coach-led debrief, then gets Synthetic School afterwards.
Stage 1 is a feasibility trial with about 80 trainees in two or three leadership programmes over one term. It checks recruitment, retention, rater reliability and safety, and it estimates the spread of scores that Stage 2 needs.
Stage 2 is the efficacy trial, with about 150 trainees in four to six programmes across at least two school systems. It can detect an effect of 0.4 standard deviations, and an independent team runs the assessment and the analysis.
The primary outcome is performance on held-out scenarios, scored by two trained raters who don't know which group each trainee was in. Mentors or line managers rate practice in the job six months later.
The trial does not measure students' outcomes. School leaders' measured effects on test scores are small even in large administrative datasets (Bartanen et al., 2024), and no trial of this size could detect them. The results will be published whatever they show, against a pre-registered plan.
Questions
The primary question is whether a term of Synthetic School improves trainees' practice on scenarios they have not seen. The comparison is case-based learning given the same time and the same coach-led debrief.
Secondary questions:
Does any gain carry into the job, as rated by mentors or line managers six months later?
Does it transfer to a kind of scenario neither group practised?
Do trainees find the school recognisable and usable? This counts only as usability evidence.
How much do trainees use it, and which parts, such as the debrief and branch replays?
Does the effect differ by experience or by school system? This is exploratory.
What does it cost per trainee?
Stage 1 answers an earlier question: can the trial run as designed? Its tests are under Success criteria.
A later study, outside this design, will compare debriefs led by a coach with debriefs led by AI. No published study has made that comparison, as the literature review in Synthetic School white paper sets out.
Design
The trial randomises individual trainees within their programme, either to Synthetic School or to case-based learning. Both groups get the same time and the same coach-led debrief, and the comparison group gets Synthetic School after the six-month follow-up.
Trainees are randomised after the baseline assessment. The comparison group gets Synthetic School only after the six-month follow-up.
| Synthetic School | Case-based learning | |
|---|---|---|
| Weekly session | 60 minutes running the synthetic school for a week of school time, with some events arriving mid-week | 60 minutes working through a written case on the same leadership problem |
| Time | 90 minutes a week for 12 weeks | 90 minutes a week for 12 weeks |
| Debrief | 30 minutes with a trained coach, working from the decision log | 30 minutes with a trained coach, working from the trainee's written decisions |
| After the follow-up | No change | A term of Synthetic School |
The comparison is active because simulation beats no training by a wide margin and other teaching by much less (Cook et al., 2011; Ilgen et al., 2013). A comparison with nothing would overstate the effect. Matching the time, the problems and the debrief isolates what running the school adds.
The debrief is held constant because debriefs alone raise performance by about 25% on average (d = 0.67) against no debrief (Tannenbaum & Cerasoli, 2013). Five minutes of coaching between sessions produced large gains over self-reflection, 1.44 SD across five randomised mixed-reality trials with teacher candidates (Cohen et al., 2021).
An independent statistician allocates trainees 1:1 within each programme, in randomly varying blocks, after the baseline assessment. Programme staff, coaches and the developer never see the allocation sequence.
Stage 1 and Stage 2 follow the same design. Stage 2 starts only if Stage 1 meets its progression criteria, and Stage 1 data are not pooled into the Stage 2 analysis.
The comparison group waits until after the follow-up so that the six-month ratings still compare trainees with and without Synthetic School. During the wait they keep their full programme and the case-based sessions, so no one loses training they would otherwise have had.
Participants and setting
Participants are aspiring and newly appointed school leaders in formal leadership programmes, in at least two school systems. They are recruited through their programme, never through their employer.
| Stage 1 | Stage 2 | |
|---|---|---|
| Trainees | About 80 | About 150 |
| Programmes | 2 or 3 | 4 to 6 |
| School systems | The two systems whose packs are in build | Those two plus at least one more, such as the other system the engine is already calibrated to |
| Partners | Two or three programmes from pilot requests and the founder's network, including one public-system headship programme and one school group's leadership academy | Four to six programmes from the same pipeline, recruited after Stage 1 reports |
Who can take part:
Trainees on a programme for aspiring school leaders, or school leaders in their first two years with an induction programme.
People who can give 90 minutes a week for 12 weeks, plus the assessments.
Who can't:
Anyone who wrote, reviewed or tested Synthetic School's scenarios.
Anyone employed by the developer's employer, and any programme it runs, to keep the trial clear of a conflict of interest.
Programme leads invite every eligible trainee, who then reads an information sheet and gives consent to the evaluation team directly. Randomisation is stratified by programme and by role, aspiring or in post.
Leaders of international schools face their own pressures, with governance and board relations among the main challenges (Morrison, 2024). A programme for international schools therefore needs its own scenarios on boards and owners, reviewed by serving leaders of those schools.
Measures
The primary outcome is the mean rubric score across two held-out scenarios at post-test, scored by two trained raters who don't know each trainee's group. Both groups sit the same assessment, which takes place outside Synthetic School so that neither group is tested on its own practice format.
| Outcome | Measure | When | Scored by |
|---|---|---|---|
| Practice on new scenarios (primary) | Two held-out scenarios: an in-tray of documents to act on, and a 15-minute conversation with a trained actor playing a parent or member of staff | Baseline, on a parallel form, and post-test | Two raters, blind to group |
| Transfer to a new kind of problem | One scenario of a type neither group practised: a budget shortfall that forces a staffing decision | Post-test | Two raters, blind to group |
| Practice in the job | Structured ratings on the same rubric dimensions | Baseline, and six months after the post-test | Mentor or line manager, not told the group |
| Self-efficacy | The Principal Sense of Efficacy Scale (Tschannen-Moran & Gareis, 2004) Used under the author's published permission letter for research use, with the citation it asks for. | Baseline, post-test, and six months after it | Trainee |
| Usability and recognition | System Usability Scale and recognition ratings | Post-test, Synthetic School group | Trainees and coaches |
| Use | Sessions completed, debriefs attended, branch replays and time on task | Throughout | Usage logs |
| Cost | Coach, rater, actor and model costs per trainee | End of each stage | Evaluation team |
| Retention | Still in a school leader's or other senior role | 12 months after the post-test, trainees in post only | Programme records |
The rubric dimensions map onto each system's own leadership standard, named for the evaluation team in an annex. Each dimension is scored 1 to 4 against written anchors.
Self-reports are secondary outcomes only. Researchers in teacher education have called for moving "beyond reliance on teacher self-reports" (Anderson & Calandra, 2026). Weak designs and methods limit the evidence on leadership simulations (Hallinger et al., 2025), and trainees' self-ratings can run ahead of their assessed ability (Dotger, 2015).
The trial measures neither students' outcomes nor simulated test scores. Estimates of school leaders' effects on test scores fall to about 0.03 of a standard deviation in maths and 0.01 in reading in tighter designs (Bartanen et al., 2024). Leadership practice links more strongly to teacher and organisational outcomes (Liebowitz & Porter, 2019).
Scenario bank and scoring
Every assessment scenario comes from a held-out bank that no trainee meets in either group. Scenarios come in parallel forms, so baseline and post-test differ in content but match in difficulty.
How the bank is built:
Serving school leaders in each system write at least three parallel pairs of scenarios, plus the transfer scenario. They draw on the problems new school leaders report as hardest: the volume and unpredictability of problems, and operational work such as budgets, buildings, staffing and legal compliance (Spillane & Lee, 2013; ERO, 2023).
Each scenario passes the same child protection and cultural review as a pack before Stage 1.
Stage 1 tests every form for difficulty and timing. A pair whose forms differ in mean score by more than 0.2 standard deviations is rewritten before Stage 2.
Forms are counterbalanced: half of each group takes form A at baseline and form B at post-test, and the other half the reverse.
Actors work from standardised protocols with set triggers: the exact statements, questions and mannerisms each actor must present. Teacher education uses the same method for standardised parents and students (Dotger, 2015). Every conversation is recorded for scoring.
Raters are serving or recent school leaders from the same system. They train on anchor responses, then recalibrate halfway through scoring. Responses reach them anonymised and shuffled across groups and time points, so they can tell neither.
Two raters score every primary response, and a third settles any dimension where they differ by more than one point. The pre-specified reliability target is an intraclass correlation of at least 0.75 for the two-rater mean.
AI scoring plays no part in the primary outcome. It may be tested as an extra rater, checked against the human scores, in a later study.
Sample size
Stage 2 needs 63 trainees per group to detect an effect of 0.4 standard deviations, so it recruits 75 per group to allow for 15% loss. The calculation assumes a two-sided 5% test, 80% power and an analysis that adjusts for the baseline score.
| Effect to detect (SD) | Per group, adjusting for baseline (r = 0.6) | Per group, no adjustment |
|---|---|---|
| 0.3 | 112 | 175 |
| 0.4 | 63 | 99 |
| 0.5 | 41 | 63 |
The correlation of 0.6 between baseline and post-test scores is an assumption, and Stage 1 measures it. If the true value is lower, Stage 2 grows towards the right-hand column.
An effect of 0.4 sits below the 0.63 found for role-play simulations across higher education (Chernikova et al., 2020). That is deliberate, because effects against active teaching run smaller than effects against no training (Ilgen et al., 2013). Detecting 0.3 would need about 260 trainees, a size worth planning for if partners can supply them.
Coaches may add clustering, because each coach works with several trainees. If Stage 1 finds coach effects, Stage 2 grows by the design effect: 45% more trainees at an intraclass correlation of 0.05 with 10 trainees per coach.
Stage 1 recruits about 80 so that at least 70 complete the post-test, 35 per group. That is the minimum Teare and colleagues recommend for estimating a continuous outcome's standard deviation from a pilot (Teare et al., 2014). Stage 1 is not powered to detect an effect, and its effect estimate plays no part in the decision to proceed.
Procedure and timeline
Each stage takes about ten months from consent to the six-month follow-up, and a further six months for the retention check. Stage 1 starts once Synthetic School passes the roadmap gate that covers the cast and the debrief, in the first school term after that gate.
Weeks 1 and 2: consent, then the baseline assessment on form A or B, baseline mentor ratings and the self-efficacy scale.
Week 3: randomisation, once every baseline assessment is in.
Weeks 3 to 14: twelve weekly sessions in both groups, each giving 60 minutes to the school or the case, then 30 minutes to a coach-led debrief.
Weeks 15 and 16: post-test on the other form plus the transfer scenario. The Synthetic School group also completes the usability and recognition measures.
Six months after the post-test, about month 10: mentor or line-manager ratings and the self-efficacy scale. The data are then locked for analysis.
After that follow-up: the comparison group gets its term of Synthetic School.
Twelve months after the post-test: the retention check for trainees in post.
The decision to start Stage 2 comes after the Stage 1 post-test data are in. It rests on the feasibility criteria alone, so it doesn't wait for the six-month ratings.
Analysis
The primary analysis compares post-test scores between the groups, adjusting for baseline score, by intention to treat. Every randomised trainee is analysed in the group they were allocated to.
The plan and the analysis code are registered before randomisation, Stage 1 on the Open Science Framework and Stage 2 on ISRCTN. The analyst works with masked group labels until the registered analysis has run.
Model: a linear mixed model with group, baseline score, programme and role as fixed effects and coach as a random effect.
Effect size: Hedges' g with its 95% confidence interval, reported whatever its size.
Missing data: multiple imputation, assuming data are missing at random, with a sensitivity analysis that relaxes that assumption.
Use: a complier average causal effect for trainees who complete at least 10 of the 12 sessions, as a secondary estimate.
Secondary outcomes: the same model, labelled secondary, with no claim of effect unless the primary outcome supports one.
Moderators: experience, role and school system, as interaction terms, reported as exploratory.
Stage 1 reports against the CONSORT extension for pilot and feasibility trials (Eldridge et al., 2016), and Stage 2 against CONSORT. Anonymised data are shared where consent allows.
Implementation and fidelity
Logs, debrief recordings and interviews show whether each group received what the design promised. Without them, a null result couldn't be told apart from a trial that was never delivered.
| What | How | Who |
|---|---|---|
| Use of Synthetic School | Sessions, minutes, events met, decisions taken, branch replays | Usage logs |
| Attendance in the comparison group | Sessions and minutes | Coaches |
| Debrief quality | A 10% sample of debriefs, recorded with consent and rated against the debrief protocol | Evaluation team |
| Contamination | An exit question on whether trainees saw the other group's materials | Evaluation team |
| Experience of taking part | Interviews with 10 to 12 trainees and every coach in each stage | Evaluation team |
| Technical faults | Narration that contradicts the engine's state, persona problems, outages | Usage logs and coach reports |
The debrief protocol keeps reflection private and formative. Accountability to superiors suppresses the kind of thinking about what could have gone better that drives learning from incidents (Morris & Moore, 2000).
The same coaches debrief both groups, trained to one protocol. That spreads any coach's strengths evenly across the groups, and the fidelity ratings show whether a coach favours one format.
Ethics, consent and data
The trial runs only with approval from the evaluation partner's research ethics committee, written consent from every trainee, and an independent team holding the outcome data.
Formative only: no score from the trial reaches an employer, and none feeds any selection, appraisal or programme grade.
Withdrawal: trainees can leave at any time without giving a reason and without any effect on their programme. Their data can be withdrawn until the analysis is locked.
Data: every school, student and family in Synthetic School is synthetic, and no real student data enters the trial. Trial data are pseudonymised and stored in a UK or EU region.
Recordings: kept only until scoring and fidelity rating are finished, then deleted within 30 days of the final rating, and never later than 12 months after recording.
Play records: each trainee owns theirs and decides who sees it, as in the product.
Distress: child protection storylines carry a content note and can be skipped. Coaches follow a distress protocol, with each programme's wellbeing lead as the named contact in its trial agreement.
The developer has an interest in a good result, so the design keeps him away from the outcome data. The evaluation partner runs randomisation, assessment and analysis, and JS sees no unmasked outcome data before the registered analysis has run. The results are published whatever they show, with JS's interest declared.
Success criteria
Stage 2 goes ahead only if Stage 1 meets every green criterion, or meets amber ones with a fix agreed before Stage 2 recruits. Any red criterion stops the trial until the design is changed and tested again.
| Criterion | Green | Amber | Red |
|---|---|---|---|
| Trainees consented in the recruitment window | 80 or more | 60 to 79 | Under 60 |
| Trainees completing the post-test | 70 or more | 56 to 69 | Under 56 |
| Synthetic School group completing at least 10 of 12 sessions | 75% or more | 50% to 74% | Under 50% |
| Rater reliability, intraclass correlation for the two-rater mean | 0.75 or more | 0.60 to 0.74 | Under 0.60 |
| Difference in mean score between parallel forms | Under 0.10 SD | 0.10 to 0.20 SD | Over 0.20 SD |
| Comparison trainees who saw Synthetic School materials | Under 10% | 10% to 20% | Over 20% |
| Serious adverse events linked to scenario content | None | One, with a fix in place | More than one, or one without a fix |
Usability and recognition ratings inform design changes but don't decide progression, because they are usability evidence rather than evidence of effect.
In Stage 2, the trial claims an effect only if the 95% confidence interval for the primary outcome excludes zero. A null or negative result is published in full, and the debrief and scenarios are redesigned before any claim of effect is made.
Even a positive result shows better practice on assessed scenarios, plus whatever the six-month ratings add. It does not show effects on students, and the white paper should say so.
Risks to validity
The largest risk is a trial that measures what Synthetic School practises rather than what leaders do. The held-out scenarios, the neutral format and the transfer scenario exist to stop that.
| Risk | How the design handles it |
|---|---|
| Teaching to the test | Held-out scenarios in a format outside Synthetic School, parallel forms, and a transfer scenario of a new type |
| Rater bias | Raters blind to group and time point, double scoring, calibration halfway through |
| Novelty and attention | An active comparison with the same time, problems and debrief |
| Contamination within a programme | Separate sessions, no shared materials and an exit check. If Stage 1 shows contamination above the red line, Stage 2 randomises whole cohorts instead |
| Coaches favouring one format | The same coaches for both groups, one protocol, fidelity ratings |
| Unequal drop-out | Intention-to-treat analysis, multiple imputation and reasons recorded for every loss |
| The developer's interest | An independent evaluator, a registered plan and publication whatever the result |
| Inflated self-ratings | Self-report kept to secondary outcomes |
| Scenarios that don't fit a system | Written and reviewed by serving school leaders in each system |
| Too few trainees per system | No effect claims per system; system is an exploratory moderator only |
| Every decision self-paced at a save point | Some events arrive mid-week, with no pause |
Adapting a leadership simulation to a new system takes real work. A Chinese adaptation of one of Hallinger's simulations followed a staged research-and-development cycle, and its author reports different local constraints of policy and culture (Tang, 2015).
Aviation research flags a gap between self-paced events in a simulator and surprises in the real cockpit (Landman et al., 2022). The mid-week events answer that risk in the product. Delivering the transfer scenario as an unannounced interruption during the in-tray would test it.
Roles and budget
Raters and coaches are the largest costs. Raters need roughly 400 hours in Stage 1 and 750 in Stage 2, and coaches roughly 480 and 900. The hours below are estimates; the rates are for JS to set.
| Role | Who | Stage 1 | Stage 2 | Cost |
|---|---|---|---|---|
| Evaluation lead and analyst | Independent university team | To be quoted | To be quoted | To be quoted by the evaluation unit |
| Raters | Serving or recent school leaders | About 400 hours | About 750 hours | Fixed day rate, agreed per system |
| Coaches | Programme coaches, trained to one protocol | About 480 hours | About 900 hours | In kind from programmes |
| Actors | Trained actors for the conversations | About 70 hours | About 125 hours | The agency's published rates |
| Scenario authors | Serving school leaders in each system | To be agreed | To be agreed | Fixed day rate, agreed per system |
| Model costs | Synthetic School seats, including the comparison group's later term | 80 seats, about US$9,600 | 150 seats, about US$18,000 | At the US$120 seat cap, an upper bound |
| Product and support | JS | Not costed | Not costed |
How the hours were estimated:
Raters: two raters score every primary response and every transfer response, at an assumed 30 minutes a rating, to be timed during rater training. Stage 1 has 800 ratings and Stage 2 has 1,500.
Coaches: twelve 30-minute debriefs for every trainee in both groups.
Actors: one 15-minute conversation per trainee at baseline and post-test, plus 10 minutes' turnaround each.
Third-rater adjudication, rater training and coach training add to these figures, and Stage 1 measures how much.
References
Full details of every study behind these choices are in the literature review. On 18 October 2026 an independent check opened every source below and compared it with each sentence that cites it, and the text was corrected to match.
Anderson, E., & Calandra, B. (2026). Mixed-reality simulations in teacher education: A scoping review. Contemporary Issues in Technology and Teacher Education, 26(2). Link
Bartanen, B., Husain, A. N., & Liebowitz, D. D. (2024). Rethinking principal effects on student outcomes. Journal of Public Economics, 234, 105115. doi.org/10.1016/j.jpubeco.2024.105115
Chernikova, O., Heitzmann, N., Stadler, M., Holzberger, D., Seidel, T., & Fischer, F. (2020). Simulation-based learning in higher education: A meta-analysis. Review of Educational Research, 90(4), 499-541. doi.org/10.3102/0034654320933544
Cohen, J., Krishnamachari, A., & Wong, V. C. (2021). Experimental evidence on the robustness of coaching supports in teacher education (EdWorkingPaper 21-468). Annenberg Institute at Brown University. doi.org/10.26300/dgf9-ca95
Cook, D. A., Hatala, R., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J., & Hamstra, S. J. (2011). Technology-enhanced simulation for health professions education: A systematic review and meta-analysis. JAMA, 306(9), 978-988. doi.org/10.1001/jama.2011.1234
Dotger, B. H. (2015). Core pedagogy: Individual uncertainty, shared practice, formative ethos. Journal of Teacher Education, 66(3), 215-226. doi.org/10.1177/0022487115570093
Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., Lancaster, G. A., & the PAFS consensus group. (2016). CONSORT 2010 statement: Extension to randomised pilot and feasibility trials. BMJ, 355, i5239. doi.org/10.1136/bmj.i5239
Education Review Office, New Zealand. (2023). 'Everything was new': Preparing and supporting new principals [summary report]. Link
Hallinger, P., Banoğlu, K., & Gümüş, S. (2025). Scoping review of research on digital simulations and serious games in educational leadership and management. Leadership and Policy in Schools, 24(1), 178-199. doi.org/10.1080/15700763.2024.2447315
Ilgen, J. S., Sherbino, J., & Cook, D. A. (2013). Technology-enhanced simulation in emergency medicine: A systematic review and meta-analysis. Academic Emergency Medicine, 20(2), 117-127. doi.org/10.1111/acem.12076
Landman, A., Mol, D., van Emmerik, M. L., & Groen, E. L. (2022). Negative transfer of training: Simulator study into effects of going beyond alarms during stall recovery training (DOT/FAA/TC-22/10). doi.org/10.21949/1524474
Liebowitz, D. D., & Porter, L. (2019). The effect of principal behaviors on student, teacher, and school outcomes: A systematic review and meta-analysis of the empirical literature. Review of Educational Research, 89(5), 785-827. doi.org/10.3102/0034654319866133
Morris, M. W., & Moore, P. C. (2000). The lessons we (don't) learn: Counterfactual thinking and organizational accountability after a close call. Administrative Science Quarterly, 45(4), 737-765. doi:10.2307/2667018. Link
Morrison, A. R. (2024). Survival at the top: Issues and challenges affecting the tenure of international school heads [EdD thesis, OISE, University of Toronto]. Link
Spillane, J. P., & Lee, L. C. (2013, January draft). Novice school principals' sense of ultimate responsibility: Problems of practice in transitioning to the principal's office (IPR Working Paper WP-13-05). Northwestern University. Link
Tang, S. (2015). Learning to make change happen in Chinese schools: Adapting a problem-based computer simulation for training school leaders [EdD thesis, Hong Kong Institute of Education]. Link
Tannenbaum, S. I., & Cerasoli, C. P. (2013). Do team and individual debriefs enhance performance? A meta-analysis. Human Factors, 55(1), 231-245. doi.org/10.1177/0018720812448394
Teare, M. D., Dimairo, M., Shephard, N., Hayman, A., Whitehead, A., & Walters, S. J. (2014). Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: A simulation study. Trials, 15, 264. doi.org/10.1186/1745-6215-15-264
Tschannen-Moran, M., & Gareis, C. R. (2004). Principals' sense of efficacy: Assessing a promising construct. Journal of Educational Administration, 42(5), 573-585. doi.org/10.1108/09578230410554070