Learner surveys: an introduction

What learner surveys work best for

Learner surveys, like all evaluation tools, are good at measuring some things and poor at measuring others. So, it’s important to start by clarifying what you want to measure. Your overarching key evaluation areas (KEAs) could be, for example: 

• Effectiveness: Are your courses delivering their intended outcomes?
• Relevance: Did the course match the learners’ needs and job context?
• Quality: Did the course design and delivery provide a good learning experience?
• Application: Are participants applying what they learned in the workplace?

Read more about key evaluation areas and explore a case study from the APS Academy outlining our current KEAs. It’s important to note that KEAs are not survey questions, even though we have framed them here as questions.

To decide whether a learner survey is the right tool for you, consider your KEAs alongside the characteristics of learner surveys listed below.

Consider your local context before getting started

There are many ways to design a learner survey. Throughout these learning resources, our goal is to help you develop a fit for purpose survey that suits your unique context. Here are some things to keep in mind:

1. Evaluation frameworks, plans, and program logic models. Does your agency have an evaluation strategy or framework in place? Is there an existing evaluation plan or program logic model for the program/project which you can (or must) use?
2. Evaluation approaches. Is there a particular evaluation approach you will follow (e.g. Kirkpatrick, Brinkerhoff, Thalheimer, Phillips, Capability, Opportunity and Motivation-Behaviour)? You don’t need to strictly follow one of these models, however they may provide you with a useful structure as you design and administer your survey. If your agency already evaluates other learning activities, consider using standardised survey questions so data can be compared.
3. Existing data collection tools and data sources. Can any of your questions be answered by existing tools (e.g. APS Employee Census, other internal surveys) or data sources (e.g. enrolment data, capability reviews)?
4. Agency-specific needs and priorities. What insights matter most in the context of your agency and key stakeholders? Are there any reporting requirements or performance metrics you need to consider?
5. Resources available. What time, resources, and budget do you have for this work? Does that align with what you have planned?
6. Local expertise. Can you seek advice or support from any internal evaluation or L&D colleagues?
7. Tools available. What software is available to you (e.g. Qualtrics, MS Forms, Power BI, LMS functionality)?
8. Final product. How easy will it be to visualise your survey data to tell a persuasive and accurate story?

Decide when to survey your learners

A learning survey can be administered at any time, from the moment a participant enrols in the course, to months or years after they complete it. So, when is the right time? That depends on what you need to know and what participants can tell you.

Before the training

A pre-program survey can collect data that captures a baseline for future comparison. Comparing a snapshot of ‘before’ and ‘after’ training can be valuable but be mindful that participants can only give you an opinion of their level of skill or knowledge. 

Before training, you might also want to collect participant information to help you tailor the learning experience for a particular cohort. For example, you might ask participants about their learning goals, prior experience or reason for enrolling.

During the training

A midpoint survey can be useful if you are delivering a program over several days, weeks, or months. For example, you might check for quality issues or affirm that the focus has been relevant for the participants thus far. If the facilitators can make meaningful positive changes for that cohort, it is worth asking questions during the training. Online polling tools, sticky notes, or facilitated reflections can also achieve this purpose.

Immediately after the training

Post course surveys collect feedback and information about a learner’s experience while it’s still fresh in their minds. By scheduling 5-10 minutes at the end of the training, you can maximise your response rate and scaffold the survey as an embedded part of the learning experience.

At this moment, learners can also tell you about their intentions to apply learning, the characteristics of the work environment they are going back to, and what they expect the impact of the training on their performance will be. But they can’t tell you about whether the learning has had a long-term impact on their performance.

Well after the training

Surveys administered a few weeks or months after the training can be useful for measuring if and how participants have applied learnings in their job. This is self-reported information rather than direct evidence of behaviour change but can still yield useful insights. Surveying a manager can provide another perspective on whether the performance of a course participant has improved and if not, why not?

Re-engaging with learners can also serve as a ‘nudge’, a behavioural economics term referring to a design choice that seeks to influence behaviour. This might simply be a prompt to recall and reflect on key learnings and their application. But it can also nudge learners towards behaviours associated with successful learning transfer, including sharing with peers or taking on a relevant work project soon after training.

There is no clear consensus on the best time to administer a follow-up learner survey. The sooner you ask participants, the less time they will have had to apply their learning. However, the longer you wait, the less likely participants will respond to your survey and be able to accurately recall the focus of the training. Somewhere between three weeks and three months after training is likely appropriate, depending on your context.