aes26 International Evaluation Conference

Author: Dorothea Huber
Lessons management in emergency management typically relies on mixed method approaches, with qualitative analysis carrying much of the analytical burden. Evaluators are increasingly expected to synthesise large volumes of unstructured material under tight timeframes, often in resource constrained public sector environments. Against this backdrop, artificial intelligence is alternately framed as a threat to professional judgement or as a solution to chronic capacity pressures. This paper argues that both framings are unhelpful.

Rather than replacing evaluative expertise, this presentation positions AI as a methodological assistant that can undertake defined, low risk tasks while leaving interpretation, sense making and ethical judgement firmly with the evaluator. Using real world examples drawn from emergency management lessons processes, the paper explores where AI has demonstrated practical value across the evaluation lifecycle. These include rapid document triage, support for qualitative coding within pre specified frameworks, identification of recurring themes and contradictions, synthesis of lessons learned, and surfacing gaps that may be missed under time pressure.

The paper also addresses common methodological and governance concerns, including transparency, bias, over reliance on fluent outputs, and the risk of mistaking confidence for insight. It outlines practical strategies for supervised AI use that protect rigour, credibility and accountability, particularly in settings where evaluative findings must withstand scrutiny and inform high stakes decisions.

Structured as a short paper, the presentation will focus on three key messages: where AI adds genuine value; where it should not be used; and how evaluators can establish clear boundaries for its application. Audience interaction will be built in through targeted questions and discussion, inviting participants to share their own experiences of using—or choosing not to use—AI in evaluation practice. The presentation reframes the central question from whether AI is the enemy, to how evaluators can use it well.