Overview of ImREAL Evaluation activities and plans
This is an overview of all the evaluation activities that are currently in progress: Use Case A, Use Case B and a cost-benefit study outlined below.
Afterwards, you will find a detailed explanation of the three plannes evaluation phases.
Use Case A - Trial 1
Target Group: Trinity College Dublin Medical Students.
Sample size: Approximately 150.
Meta-cognitive scaffolding service for Use Case A: Evaluation of the effects of the meta-cognitive scaffolding service developed in WP5 on self-regulated learning, learning performance and experiences, and identification of aspects for further improvement of scaffolding. Students will learn to carry out interviews with patients using the ETU simulator as part of the curriculum of their medical degree – after the ‘learning’ period using the simulator the ‘trained’ students will be asked to complete a survey on their experiences. In the second half of the evaluation a subset of the students will be questioned after they have conducted ‘real world’ interviews (this will consist of a short questionnaire and ‘real’ world performance assessed by experts).
Use Case B - Trial 1
Target: Instructional Designers and Subject Matter Experts
Sample size: Minimum number 10-15
Inform and collect content for the simulation development: Investigation of the benefit brought by the (initial version of the) storyboarding environment to the process of developing contents for simulations, serious games, and e-learning modules. This evaluation should also highlight any aspects that should be further improved. Participants will work on a task (for instance, creating the ‘script’ for a job interview in a ‘multicultural’ setting) using the storyboarding environment. Afterwards, participants will be asked to describe their authoring experience with a questionnaire and an interview.
Cost-benefit Study
Target: ImREAL partners (especially SMEs), business organizations.
Subjective cost-benefit analysis of the ImREAL ‘services’: Initial investigation of the readiness of the ‘intelligent’ services for the market and their cost-effectiveness will be carried out. Costs and benefits for service/simulator development and integration will be determined and a market analysis for simulations will be undertaken.
Overview of ImREAL Evaluation phases
The following three phases of evaluation have been planned.
1) Baseline Evaluation
The initial phase of evaluation in ImREAL has been denoted baseline evaluation; it aimed at collecting benchmark data that serves to support the identification of requirements for the ImREAL intelligent services and therefore no ImREAL services had been required at this stage. This baseline also provides the basis for comparison with evaluation outcomes using the augmented simulations, which include ImREAL services.
2) Phase 1 Evaluation
The Phase 1 evaluation in ImREAL corresponds to the notion of formative evaluation and has been carried out using the first versions of the ImREAL services and demonstrators to gather information on the initial benefits and, in particular on how to further improve the ImREAL services – to inform work on the revised versions of ImREAL technologies and to increase benefit and efficient augmentation for simulations (i.e. for the simulation environment, in which context the services has been evaluated, as well as for different simulators). The Phase 1 evaluation outcomes have been compared to the results of the baseline evaluation and have provided evidence of the benefit of the augmentation through ImREAL services. The evaluation report has been used to feed back evaluation outcomes to the developers for the implementation of the second version of the ImREAL services.
3) Phase 2 Evaluation
The phase 2 evaluation of the ImREAL project is actually two-fold and covers two consecutive sub-phases. The first one will use the revised ImREAL demonstrator integrating the revised versions of the services and can be understood as a hybrid form of formative and summative evaluation in the final year of the project. The evaluation outcomes will be compared to the results of the baseline evaluation as well as of the first user trials, in order to give evidence of the general benefit of simulation augmentation through ImREAL intelligent services, as well as to determine improvements compared to the initial implementations. In addition, this evaluation will allow identifying final issues for fine-tuning the ImREAL services, which can be taken into account for the final implementations (see Figure below).
Iterative evaluation approach of ImREAL for the different evaluation stages.
If you are interested in participating in any of these evaluation activities, please contact us through our LinkedIn group: ImREAL.
News about these activities will be published both on the LinkedIn group and on Twitter: follow ImREAL_project!
