aes26 International Evaluation Conference

Authors: Ethel Karskens (Clear Horizon), Maree Dibella (Clear Horizon)
What participants will learn: This hands-on session teaches evaluators how to configure and use AI assistants for real evaluation tasks. Participants will leave with a working AI tool customised for their own work, whether that's coding interview transcripts, analysing open-ended survey responses, or synthesising progress reports.

Why this skill matters: Most evaluators have experimented with ChatGPT or similar tools, but few have moved beyond ad-hoc prompting to building reusable, reliable AI workflows. The gap between "asking ChatGPT a question" and "using AI as a systematic evaluation tool" is significant. This session bridges that gap by teaching the practical setup skills that turn general-purpose AI into specialized evaluation assistants that produce consistent, auditable outputs.

As AI becomes standard in evaluation practice, knowing how to configure these tools properly - with the right instructions, quality controls, and workflow integration - is becoming a core professional capability. This session responds directly to evaluators' need for practical AI implementation skills, not just conceptual understanding.

How we'll teach the skill:
Minutes 0-5: Quick orientation - what makes an AI "assistant" different from a chatbot
Minutes 5-15: Live demonstration - the facilitator builds an assistant for interview coding from scratch, narrating decisions
Minutes 15-40: Guided hands-on practice -participants configure their own assistant for a task they choose (interview coding, survey analysis, or report summarisation), using either sample data provided or their own files
Minutes 40-50: Group debrief -participants share one success and one challenge; facilitator troubleshoots common issues

How participants will engage: Participants will work on their own laptops throughout the session, following a structured build process with real-time facilitator support. They'll leave with a configured tool, a workflow template, and practical troubleshooting strategies they can apply immediately in their work.