Reasons for Evaluation in Virtual Simulation

There are numerous reasons for evaluating virtual simulation. Some of these include: 

• Measuring achievement of specific learning objectives.
• Documenting what virtual simulation teaching and learning processes work and what needs to change.
• Taking stock of resources used to inform future resource allocation.
• Identifying best processes for integrating virtual simulation in the curriculum.
• Gathering quality evidence to share with other educators.

An important principle of evaluation is not to try to answer all questions in one evaluation. It is a good idea to clarify the scope of the evaluation in the initial planning stage.  A helpful way to get started is to develop a list of questions that the teaching team most wants answered, and which can be answered (Table 6.1).

Click here to download an accessible PDF version of Table 6.1.

Table 6.1: Key Questions for Planning the Evaluation

The more specific the evaluation question, the more likely the educator is to get a clear answer. Also, when designing the evaluation plan, it is important to ensure that the plan is feasible (Freeth et al., 2005). Educators should reflect on how much time and energy the evaluation will take, what skills will be needed and if there are sufficient resources to get the job done. Many evaluation methods do not require any specialized data collection or analysis skills; however, some do. For example, a basic knowledge of how to run and interpret statistical tests will be needed for measuring pre and post activity knowledge gains. Educators should think through their choice of evaluation method and ensure they have the necessary resources. Avoid being overly ambitious!

Expert’s Corner: Evaluating Learning

In response to the question, ‘What do we really want to know?’ educators decide they want to know if learners gained knowledge by playing the virtual simulation on neonatal care. They ask learners to complete a 10 item multiple choice quiz based on neonatal care virtual simulation learning outcomes, online, before playing the virtual simulation. They ask learners to complete the test again,  one week after playing the virtual simulation. A member of their team agrees to analyze the data using statistics.

There are many different methods that can be used to evaluate the virtual simulation experience: learner testing, focus groups, surveys, facilitator self-reflection, peer observation, and feedback. Just as specific learning outcomes drive an educator’s choice of virtual simulation, they also drive the evaluation strategy an educator will use (Figure. 6.1). For example, if an educator simply wants to know if learners gained knowledge, testing is an appropriate strategy. If, however, the educator wants an in-depth understanding of the impact of the simulation on clinical practice, discussion or focus group interviews would be the best approach.

Examples in Action: Focus Groups

Educators used a virtual simulation to help developmental service worker (DSW) students prepare for their first community visit. The wanted to evaluate the impact of the virtual simulation on learners’ knowledge and confidence levels. They conducted a focus group with learners and asked these questions:

1. Tell me what you were thinking about your role as a DSW as you worked through the simulation.
2. Did the simulation help to prepare you for your first visit in the community? Tell me why or why not.
3. What were some key learnings you had as a result of using the simulation? How will they help you in the field?
4. How would you rate your confidence level about making your first visit before you played the simulation? After? Can you tell me about that?
5. What can we do differently with this virtual simulation to further promote your learning?

Click here to download an accessible PDF version of Figure 6.1.

Figure 6.1: The Evaluation Cycle

Many evaluation strategies can be conducted though a learning management system or survey administration site thereby streamlining data collection and analysis. Sample outcomes and evaluation strategies are outlined in Table 6.2.

Click here to download an accessible PDF version of Table 6.2.

Table 6.2: Evaluating Learner Outcomes and Evaluation Strategies in Virtual Simulation

Another important point to consider is how the evaluation results will be used. There are two main types of evaluation that apply to virtual simulation: formative and summative.