aes26 International Evaluation Conference: Full Schedule

Presenter and chair information page, resources for conference presenters, authors and session chairs.

arrow_back View All Dates

3:30pm ACST

Uncertain impacts, real decisions: measuring and communicating causal uncertainty in quantitative evaluations

Thursday September 17, 2026 3:30pm - 4:00pm ACST

Author: Dimitria Gavalyugova, NSW Department of Education
Evaluators face a core dilemma: decision-makers typically expect a definitive measure of program impact, yet the designs capable of delivering one are rarely feasible for large-scale government programs. Quasi-experimental methods and administrative data are often the only tools available to quantify long-term outcomes, but the assumptions needed to establish causal impact cannot always be met – or tested – with available data. This mismatch has driven evaluations into a sub-optimal equilibrium where uncertainty is understated, even though addressing it could lead to better-informed decisions. This presentation explores how to reconcile the need for reporting actionable findings with what the evidence can support.

The session draws on systematic analysis of the 22 combined outcome and economic evaluations published in the NSW Treasury library. By mapping the confidence of causal language attributing outcomes to programs across report sections, the analysis identifies a persistent "within-document gap." In many cases, findings are treated as causal in executive summaries and recommendations, even when underlying methodologies note that they should be interpreted with caution. Scoring each estimation strategy against a risk-of-bias framework reveals observable misalignment between evaluations’ methodological constraints and the confidence of causal claims.

When causality is uncertain, the program effect that gets monetised in economic evaluations contains the true impact plus some degree of bias. Existing guidance addresses uncertainty around monetary parameters and discount rates, but not around estimated program impacts. In the absence of such guidance, the evidence shows that potentially biased estimates routinely enter cost-benefit analyses as true effects. Using an anonymised real-world example, the presentation demonstrates how even small levels of bias can alter benefit-cost ratios and funding recommendations.

The session invites participants to collectively explore practical solutions across four areas: leveraging available methods and data to strengthen evidence, turning limitations into guidance for future evaluation design, communicating uncertainty effectively, and modelling uncertain impacts within economic analyses.

Chair

Megan Anderson

Associate Director, HealthConsult

Speakers

Dimitria Gavalyugova

Senior Research and Evaluation Officer, NSW Department of Education

Thursday September 17, 2026 3:30pm - 4:00pm ACST
Waterfront 1 Stokes Hill Rd, Darwin City NT, Australia

Traditional and new ways

Format Short paper
Audience Level Intermediate, Advanced
Industry Education and training, Health and disability, Economic development tourism and infrastructure, Community and social services, Public administration and safety
ID 323

aes26 International Evaluation Conference

3:30pm ACST

Megan Anderson

Dimitria Gavalyugova

Get help with the event