Volume 18-Issue2-2009-BALOG

Developing a Measurement Scale for the Evaluation of AR-Based Educational Systems

A. BALOG, C. PRIBEANU
National Institute for Research and Development in Informatics – ICI Bucharest
Bd. Mareşal Averescu Nr. 8-10, 011455 Bucureşti, Romania

Abstract: Educational systems based on the augmented reality (AR) technology are creating a new kind of user learning experience by bringing real life objects into a computer environment. The mix of the real and virtual world requires the design of new interaction techniques which have to be tested with users early in the development process. For these new e-learning systems to be effective traditional usability evaluation is not enough. The adoption of AR-based e-learning systems in schools also requires an investigation into the perceived usefulness and increase in students’ motivation. This paper presents a measurement model for the usability evaluation of AR-based e-learning systems that is targeting the educational and motivational values. The model was developed during a European research project and is inspired from the technology acceptance theories. The scale development was carried on in a methodological approach starting with the definition of a conceptual model from which an initial scale of 28 items was generated. The evaluation of the measurement model which is based on a confirmatory factor analysis resulted in a reliable scale with 19 items organized into five constructs.

Keywords: Technology acceptance models, AR, e-learning, usability evaluation, user experience.

Alexandru Balog received his PhD degree in Economic Informatics from the Academy of Economic Studies in Bucharest, Romania, in 1994. Since 1990 he has been a Senior Researcher in ICI Bucharest. His research interests include quality evaluation, software quality, e-services quality, quantitative methods, and information systems.

Costin Pribeanu received his PhD degree in Economic Informatics from the Academy of Economic Studies in Bucharest, Romania, in 1997. Since 1990 he has been a Senior Researcher in ICI Bucharest. Costin Pribeanu is also the Chair of the Romanian CHI group (RoCHI – SIGCHI Romania) since 2001. His research interests include usability evaluation, task analysis, and user interface design.

>Full text
CITE THIS PAPER AS:
Alexandru BALOG, Costin PRIBEANU, Developing a Measurement Scale for the Evaluation of AR-Based Educational Systems, Studies in Informatics and Control, ISSN 1220-1766, vol. 18 (2), pp. 137-148, 2009.

1. Introduction

The proliferation of AR (Augmented Reality) technologies is creating a strong opportunity for teachers to apply new teaching methods in schools. Desktop AR configurations are bringing real life objects into a computing environment thus making the e-learning process more natural and enjoyable for young students. Instead of interacting with representations of real objects and processes displayed onto a computer monitor, the user is manipulating a real object (e.g. a torso of the human body used in Biology lessons) which is observable on a see-through screen where computer generated images are superimposed. From a pedagogical point of view, AR systems have a great potential to support a learning-by-doing approach to education.

AR systems are expensive since a lot of research and design effort is needed to develop visualization and rendering software. On another hand, the mix of the real and the virtual requires appropriate interaction techniques, which have to be tested with users early in the development process in order to avoid usability problems. While the usability of interaction techniques is critical for a good user experience the successful adoption of AR technologies in school require to investigate also the educational and motivational values of a given application.

The main objective of the ARiSE (Augmented Reality for School Environments) project is to test the pedagogical effectiveness of introducing AR (Augmented Reality) in schools and creating remote collaboration between classes around AR display systems. The project has developed a new technology, the Augmented Reality Teaching Platform (ARTP), by adapting an existing augmented reality system for museums to the needs of students in primary and secondary classes.

The specific objectives of the project are: (1) To adapt the AR technology for the specific needs of schools; (2) To develop interaction scenarios to promote collaborative work between students; (3) To develop tools for easy use of the ARTP by teachers; (4) To demonstrate the pedagogical effectiveness of activities using the AR platform; (5) To build on a design framework able to support the usability of interactive systems.

To address the project’s objectives, we developed a usability questionnaire that goes beyond the traditional usability evaluation approaches, by targeting the educational and motivational value. The measurement instrument (scale) is based on a conceptual model inspired by the technology acceptance theories and consists of five constructs: ergonomics of the platform, perceived ease of using the application, perceived usefulness, perceived enjoyment and intention to use. By addressing issues like perceived enjoyment and perceived usefulness, the usability evaluation results could be better integrated with the pedagogical evaluation results.

The questionnaire was firstly administered during and after a summer school in 2007, with the aim of improving the usability of the implemented learning scenarios. This way, it served as an instrument for the user-centered formative evaluation of the ARTP. After the installation of the improved version of the software, it was administered again to 278 students. The results were used to evaluate the measurement model.

The methodological approach to the development and validation of the measurement scale consists in five main steps: (1) conceptualization of constructs and item generation, (2) pilot test and preliminary item analysis, (3) data analysis and processing, (4) preliminary scale evaluation, and (5) model testing and validation. In this paper we will present the first four steps undertaken to define a reliable and sensitive scale as a pre-requisite for validating the measurement model (cf. Gediga et al., 1999). The preliminary evaluation of the measurement scale was performed by carrying on a exploratory factor analysis. The final result is a scale with 19 items organized into 5 constructs.

7. Conclusion and Future Work

While the usability of a software system is a critical feature, usability evaluation per-se does not ensure a successful adoption of the underlying information technology. The evaluation of new systems should go beyond the traditional usability approach and investigate the user acceptance in order to understand the various factors that influence the intention to use.

In this paper we presented the development of a measurement scale that is intended to measure three core features of an AR-based teaching platform: usability, pedagogical and motivational value. Our evaluation strategy is based on the integration of formative and summative usability evaluation with the methods and procedures used in the technology acceptance theories.

The preliminary scale evaluation leaded to a measurement model of 19 items grouped into 5 constructs: ergonomics of the ARTP, perceived ease of use, perceived usefulness, perceived enjoyment and intention to use. As such, the model is able to answer the research questions of the ARiSE project and makes it easier to integrate the usability evaluation results with the pedagogical evaluation results obtained with specific qualitative methods.

Nevertheless, the measurement model is a prerequisite for the final model evaluation and validation. The next step is the evaluation of the measurement model and the validation of the structural model. This work will be carried on with a structural equations modeling approach. Then base on the causal relationships between constructs we will be able to statistically accept / reject the research hypotheses.

ACKNOWLEDGEMENT

This work was supported by the ARiSE research project funded by EU under FP6 027039.

[/one_half]

REFERENCES:

Churchill, G.A. (1979). A Paradigm for Developing better Measures of Marketing Constructs. Journal of Marketing Research, Vol. XVI, February 1979, pp. 64-73.
Costello, A.B., Osborne, J.W. (2005). Best Practices in Exploratory Factor Analysis, Practical Assessment, Research and Evaluation vol. 10, no. 7, pp. 1-9.
Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly vol.13, no.3, pp. 319-339.
Davis, F.D., Bagozzi, R.P., Warshaw, P.R. (1989). User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Management Science, Vol. 35, No. 8, pp. 982-1003.
Fabriger, L. R., MacCallum, R. C., Wegener, D. T., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, no. 4, pp. 272-299.
Field, A. (2005). Discovering Statistics Using SPSS. Second edition. SAGE Publications Ltd.
Gediga, G., Hamborg, K.-C. & Duntsch, I. (1999) The IsoMetrics Usability Inventory – An operationalisation of ISO 9241-10. Behavior and Information Technology, 18, pp. 151-164.
Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E., Tatham, R.L. (2006). Multivariate Data Analysis. 6^th ed., Prentice Hall, 2006.
Hornbaek, K. (2006) Current practice in measuring usability: Challenges to usability studies and research. Int. J. Human Computer Studies. 64 (2006).pp. 79-102.
ISO/DIS 9241-11:1994 Information Technology – Ergonomic requirements for office work with visual display terminal (VDTs) – Guidance on usability.
ISO 9126-1:2001 Software Engineering – Software product quality. Part 1: Quality Model.
Law, E., Hvannberg, E.T., Hassenzahl, M. (2006). User Experience – Towards a unified view. Procedings of UX Workshop NordiCHI 2006, pp. 1-3, ACM Press, Oslo.
Legris, P., Ingham, J., & Collerette, P. (2003). Why people use information technology? A critical review of technology acceptance model. Information & Management, vol. 40, pp. 191-204.
Scriven, M.: Evaluation thesaurus. 4th ed. Newbury Park, CA: Sage Publications (1991).
Silva, L. (2007). Post-positivist Review of Technology Acceptance Model. Journal of the Association for Information Systems, vol. 8, issue 4, pp. 255-266.
Theofanos, M. & Quesenbery, W. (2005). Towards the Design of Effective Formative Test Reports. Journal of Usability Studies, Issue 1, Vol. 1. pp. 27-45.
Tabachnick, B. G., and Fidell, L. S. (2007). Using Multivariate Statistics, 5th ed. Boston: Allyn and Bacon.
Venkatesh, V. (2000). Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model. Information Systems Research, Vol. 11, No. 4, pp. 342-365.
Venkatesh, V., Davis, F.D. (2000). A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management Science Vol. 46, No. 2, pp. 186-204.
Venkatesh, V., Morris, M., Davis, G., Davis, F. (2003). User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly Vol. 27, No. 3, pp. 425-478.
Worthington, R.L., Whittaker, T.A (2006). Scale Development Research. A Content Analysis and Recommendations for Best Practices. The Counseling Psychologist vol. 34, no. 6, pp. 806-838.