Evaluation of integrated e-learning

* * * *
Theo Bastiaens, Jo Boon and Rob Martens (2004). Evaluating integrated e-learning. In Jochems, W., van Merriënboer, J., & Koper, R. (Eds.) (2004). Integrated E-Learning: implications for pedagogy, technology and organization. London, UK: RoutledgeFalmer.
Chapter 14 Evaluating integrated e-learning
Keywords: evaluation, Kirkpatrick VBe-L
In this chapter the authors explained the four-level model for evaluating of Kirkpatrick and an example of a VBe-L framework for the evaluation of integrated e-learning is presented in this chapter.
Quotes

• …hybrid learning environments which contain a combination of face-to-face and Web-based formats are often referred to as “blended learning” (Baestians, Martens and Stijnen, 2002). P187
• Evaluation is needed to improve the quality and effectiveness of the e-learning initiative and check the design assumptions. P187
• Evaluation of e-learning is a relatively new topic in the field, requiring an approach on different levels at the same time. There are many different stakeholders, and a range of problem areas extending from content to technological questions. Concerns about the quality of the content, the learning access level, technological features, costs and sustainability call for a broad evaluation design. P188
• The focus is on evaluating the effectiveness and efficiency of e-learning components as part of a lesson plan, curriculum or a course administered by an educational institute. P188
• Kirtpatrick’s (1998) four level evaluation framework has been popular mainly in the commercial training world for about 40 years… Kirkpatrick’s framework describes four levels of evaluation. Level 1, the reaction level is defined as what the participants think of the course or training: in a word, do they like the initiative? Level 2 is concerned with measuring the learning results, and raises the question of what participants learn. Level 3, behaviour, determines to what extent the participants change their behaviour on the job. What is the transfer of what was learned? Level 4, results, monitors the organizational improvement such as quality changes, effectiveness and costs. Did the initiative affect the organization? P188
• In short:

Level 1 (reaction): do learners like the initiative?
Level 2 (Learning) result measuring;
Level 3 (Performance)What is the transfer of what was learned?
Level 4 (Organization) Did the initiative affect the organization?

• The most difficult aspect of evaluation of e-learning is the question whether the media mix is successful. Do the different media support the learning process optimally? This brings us to the third level of the model, namely performance. At this level, the transfer of learning to actual performance, it is very difficult to measure the impact of learning. P196
• …level 4 of the framework considers the results of the evaluation on the organizational level, considering questions of costs, improvement of work or of interpersonal communication for the organization as a whole. P196
• In general, these topics can be categorized into technical, educational, institutional economic and implementation aspects, and can be evaluated at different levels. We recommend that Occam’s razor to be applied. It is not wise to perform a four-level evaluation for every e-learning event, but only for each new type of medium that is adopted, or for each new course type.

Further reading:

• Kirkpatrick, D L (!998) Evaluation Training Programs: The four levels, Berreth-Koehler, San Francisco, CA.
• Singh, H (2001) Online implementation of Kirkpatrick’s four levels of evaluation using web databases, in Web-Based Training, ed B H Khan, pp 523-36, Educational Technology Publications, Englewood Cliffs, NJ.
• Walker (1998) The value of building skills with online technology: online training costs and evaluation at the Texas natural resource conservation commission. In Distance Training: How innovative organizations are using technology to maximize learning and meet business objectives, ed D Schreiber and A Berge, pp 270-87 Jossey-Bass, San Francisco, CA.

Alberto Ramirez Martinell