Learning Barometer Constructs

The barometer consists of pre-simulation and post-simulation surveys, which are used to measure changes in student perceptions of learning outcomes. The barometer also includes problem-based questions designed to measure cognitive change for a number of commonly used simulations. This approach is encouraged and supported by other researchers (Cronan & Douglas, 2012; Foster, 2011; Hsu, 1989; Seethamraju, 2011). The barometer is built on the premise that the impact of simulations can be measured by monitoring different variables that before, during and after the simulation (see figure on next page).

Figure 1. Key constructs measured by the Simulation Learning Barometer

Pre-simulation

Antecedents to learning include attitudes about learning expectations, teamwork and enjoyment as well as self-efficacy (Ineson, Jung, Hains, & Kim, 2013). The barometer measures a number of these items using a set of self-reported rating scales developed from the literature, student focus groups and trial surveys.

The learning outcomes of simulations have been tested with cognitive measures, such as student grades, and with affective measures, such as student perceptions of learning and student satisfaction. The complexities of measuring learning outcomes have been acknowledged and attempts to measure the learning outcomes of simulations have resulted in mixed results (Anderson & Lawton, 2009; De Freitas & Jarvis, 2007; Keys & Wolfe, 1989). Gosen and Washbush (2004) reviewed and categorized the problems of simulation research into three areas: correlates of performance, validity of simulations, and the nature of the instruments used to measure the effectiveness of simulations. Initial research in the field assumed that simulation performance could be used as a proxy for learning but this has proved to be inaccurate (Batista & Cornachione, 2005). A meta-analysis of research on simulations found that of the 248 studies evaluated, only 32 could be included in the meta-analysis owing to methodological and reporting flaws (Vogel et al., 2006). The absence of a theoretical framework and lack of rigor in design have been identified as problems with previous efforts to measure the impact of simulations (Ruben, 1999; Wu et al., 2012). Finally, much of the research is conceptual rather than empirical (Feinstein & Parks, 2002), as is evident by the lack of empirical testing and measurement.

In evaluating simulations, very few studies have reported both subjective and objective measures (Anderson & Lawton, 2009; Cronan, Léger, Robert, Babin, & Charland, 2012). These are sometimes described as indirect or direct measures (Lo, 2010).

In the Simulation Learning Barometer objective measures will be captured through a problem-based scenario while behavioural learning will be measured through students’ performance in the simulation. The problem-based scenario provides a baseline for the post-simulation survey but it will be necessary to adapt the scenario for different simulators. Subjective measures will include self-reported expectations and perceptions of knowledge and skills acquired during the simulation (Batista & Cornachione, 2005). These skills are measured before the simulation by asking students what they expect to learn and this is then compared with perceptions of what they learned in the post-simulations survey.

Table 1. Pre-simulation survey items

During the Simulation

Although these items relate to inputs and processes during the simulation they are measured at the end of the simulation using the post-simulation survey. Chaparro-Peláez et al. (2013) found three factors that affect students’ perceived learning: satisfaction, time dedication, and collaborative learning. Online business simulations generally require students to work in teams to plan, coordinate and manage a virtual business. Students learn by developing knowledge and understanding from their experiences and interactions with others through a process of social constructivism (Boulos, Maramba, & Wheeler, 2006; Jonassen, Peck, & Wilson, 1999). Simulations provide fertile opportunities for constructivist learning because they provide multiple representations of reality, attempt to represent the natural complexity of the real world and attempt to replicate authentic tasks (Lainema & Makkonen, 2003). The learning barometer includes items designed to measure engagement and collaboration through teamwork.

The problem-based case study used in the pre-simulation survey is presented again on the post-simulation survey to measure whether students have developed a better understanding of key concepts/processes.

Post-simulation

The barometer draws on Bloom’s taxonomy to evaluate the learning outcomes of simulations. Hsu (1989) argues that the outcomes of simulations should be measured across all three of Bloom’s domains of cognitive, affective, and psychomotor learning. Previous simulation research found that learning exists when a personally responsible participant cognitively, affectively and behaviourally processes knowledge, skills and/or attitudes in a learning situation (Agnello, Pikas, Agnello, & Pikas, 2011). Cognitive learning can be described as developing an understanding of basic facts. Affective learning is where the simulation participants perceive that they learn, hold positive attitudes and satisfaction. Whilst behavioural learning might be described as simulation participants taking the facts and formulating correct decisions or actions (Agnello et al., 2011). Behavioural learning should demonstrate problem analysis and decision-making and the application of cross functional skills (Hermens & Clarke, 2009). The barometer requires students to reflect on what they have learned by responding to a series of scale items representing cognitive, affective and behavioural outcomes and skills at different levels of Bloom’s taxonomy.

Table 2. Post-simulation Survey items

Subjective measures include student’s perceptions of learning from the simulation, attitudes toward the simulation, collaboration and self-efficacy.

• Evaluation of the simulation includes perceived cognitive outcomes, which is an indirect measure of cognitive outcomes. Students perceived cognitive outcomes are their perceptions of learning which include a range of skills.  These include for example, development of skills in finance, marketing, and HR.  More advanced skills were also included in line with Blooms Taxonomy, such as problem solving and critical thinking.
• Students attitudes toward the simulation include affective attitudinal statements of enjoyment and satisfaction.  It also includes perceptions of the simulation assisting their future career prospects and communication skills.  Positive attitudes and satisfaction have been found to improve student learning.
• Students experience surrounding attitudes toward group work, perceptions of collaborative and social cognitive learning are included.
• Individual engagement is captured through motivation, level of self-directed learning, and self-efficacy.

As a major objective of the project surrounds pedagogy, the project also measures students attitudes regarding resources, learning activities and assessment tasks. Student performance and grades are an important part of the barometer but are not captured using the survey because educators would already have this information. Students are asked for their ID number so that survey responses can be matched to these grades.