Monday , May 29 2023

Testing a Multidimensional and Hierarchical Quality
Assessment Model for Digital Libraries

Alexandru BALOG
I C I Bucharest
(National Institute for R & D in Informatics)

8-10 Averescu Blvd.
011455 Bucharest 1, Romania

Abstract: This paper proposes and tests a multidimensional and hierarchical model for quality evaluation of digital libraries (LibEval). A convenience sample of 252 undergraduate and graduate students was used. Data was collected in October 2010 by means of a paper-and-pencil questionnaire. Results from a confirmatory factor analysis utilizing structural equation modelling techniques confirm the existence of a second-order factor construct (Quality of Digital Library) and five first-order dimensions, namely: interface quality, system quality, information quality, service quality, and contextual factors. The results from this study show empirically that the five dimensions are distinct manifestations of the quality of digital libraries. The user perceives and evaluates the quality of a digital library globally and in each dimension. The LibEval instrument can be utilized to assess the quality of digital libraries from the general user’s perspective.

Keywords: digital library, quality model, multidimensional and hierarchical model, model testing, CFA, SEM.

>>Full text
Alexandru BALOG, Testing a Multidimensional and Hierarchical Quality Assessment Model for Digital Libraries, Studies in Informatics and Control, ISSN 1220-1766, vol. 20 (3), pp. 233-246, 2011.

1. Introduction

The term “digital library” (DL) covers different applications and has been used interchangeably for digitized collections, e-journal platforms, network databases, library websites. As Chowdhury [7] pointed out, “a modern digital library is a space – a centre of intellectual activities – with content, available in different forms and formats in a distributed network environment, as well as tools and facilities for user-centric access, use, interactions, collaborations and sharing”.

One major area of library and information science research is the evaluation of digital library. In line with the growing number of operational DLs, there is also growing interest among researchers to investigate the quality of DLs [1], [13], [22], [34], [42], [44]. In the context of Romania, no studies were found addressing this topic. The overall quality of DLs is insufficiently studied and reported [44], and DL quality and evaluation is a very underrepresented research area in the digital library literature [13]. While previous studies have provided useful and interesting information regarding the topic, this study proposes and tests a multidimensional and hierarchical model which can be used to assess the quality of digital libraries. The model is based on Zhang model [44] and DeLone & McLean IS success model [9].

This paper is organized as follows. In Section 2 we briefly review the related work and studies on DL quality. In Section 3 we present the proposed research model, which is followed by the description of the research methods used and statistical results in Section 4 and 5, respectively. A summary and conclusions are presented in Section 6.


  1. AGOSTI, M., N. FERRO, E. A. FOX, M. A. GONCALVES, Towards a Reference Quality Model for Digital Libraries, First Intl. Workshop on DLs Foundations, Vancouver, Canada, June 23, 2007.
  2. ANDERSON, J. C., D. W. GERBING, Structural Equation Modeling in Practice: A Review and Recommended Two-Step Approach. Psychological Bulletin, vol. 103(3), 1998, pp. 411-423.
  3. BALOG, AL., Dezvoltarea preliminară a modelului LibEval de evaluare a calităţii bibliotecilor digitale. Revista Romană de Informatică şi Automatică, vol. 21(1), 2011, pp. 11-26.
  4. BALOG, AL., Dezvoltarea şi testarea unui model ierarhic de evaluare a calităţii bibliotecilor on-line: un studiu exploratoriu. Revista Română de Interacţiune Om-Calculator nr. 4, 2011, pp. 1-22.
  5. BERTOT, J. C., J. T. SNEAD, P. T. JAEGER, C. R. MCCLURE, Functionality, Usability, and Accessibility. Iterative User-centered Evaluation Strategies for Digital Libraries. Performance Measurement and Metrics 7 (1), 2006, pp. 17-28.
  6. BYRNE, B., Structural Equation Modeling with AMOS. Basic Concepts, Applications, and Programming. Lawrence Erlbaum Association, Publishers, 2001.
  7. CHOWDHURY, G., From Digital Libraries to Digital Preservation Research: the Importance of Users and Context. J. of Documentation 66(2), 2010, pp. 207-223.
  8. COOK, C., F. HEATH, B. THOMPSON, Users’ Hierarchical Perspectives on Library Service Quality: A LibQUAL+ Study. College & Research Library, 2001, pp. 147-153.
  9. DELONE, W. H., E. R. MCLEAN, The DeLone and McLean Model of Information Systems Success: A Ten-year Update. Journal of MIS vol. 19(4), 2003, pp. 9-30.
  10. EDWARDS, J. R., Multidimensional Constructs in Organizational Behavior Research: An Integrative Analytical Framework. Organizational Research Methods vol. 4(2), 2001, pp. 144-192.
  11. FORNELL, C., D. F. LARCKER, Evaluating Structural Equations Models with Unobservable Variables and Measurement Error. Journal of Marketing Research vol. 18(1), 1981, pp. 39-50.
  12. FUHR, N., G. TSAKONAS ET AL., Evaluation of Digital Libraries. Intl. J. on Digital Libraries, vol. 8(1), 2007, pp. 21-38.
  13. GONÇALVES, M. A., B. L. MOREIRA, E. A. FOX, What is a Good Digital Library? A Quality Model for Digital Libraries. Inf. Proc. & Manag., 43, 2007, pp. 1416-1437.
  1. HAIR, J. F., W. C. BLACK, B. J. BABIN, R. E. ANDERSON, R. L. TATHAM, Multivariate Data Analysis. 6th ed., Prentice Hall, 2006.
  2. HERNON, P., P. CALVERT, E-service Quality in Libraries: Exploring its Features and Dimensions. Library & Info. Science Research. 27(3), 2005, pp.377-404.
  3. HU, L. T., P. M. BENTLER, Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives. Structural Equation Modeling vol. 6(1), 1999, pp. 1-55.
  4. JARVIS, C. B., S B. MACKENZIE, P. M. PODSAKOFF, A Critical Review of Construct Indicators and Measurement Model Misspecification in Marketing and Consumer Research. J. of Consumer Research vol. 30(2), pp. 199-218.
  5. JENG, J., Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency, Satisfaction, and Learnability. Libri, vol. 55(2-3), 2005, pp. 96-121.
  6. KNIGHT, S. A., J. M. BURN, Developing a Framework for Assessment Information Quality on the World Wide Web. Informing Science nr. 8, 2005, pp. 159-172.
  7. KOUFTEROS, X. A., Testing a Model of Pull Production: a Paradigm for Manufacturing Research using Structural Equation Modeling. Journal of Operations Management, vol. 17(4), 1999, pp. 467-488.
  8. KOUFTEROS, X. A., S. BABBAR, M. KAIGHOBADI, A Paradigm for Examining Second-order Factor Models Employing Structural Equation Modeling, Intl. J. of Prod. Ec. 120, 2, 2009, pp. 633-652.
  9. KYRILLIDOU, M., S. GIERSCH, Developing the DigiQUAL Protocol for Digital Library Evaluation, in: JCDL’05: Proc. the 5th ACM/IEEE-CS Joint Conf. on DLs, NY, ACM Press, 2005, pp. 172-173.
  10. LAW, K. S., C. S. WONG, W. H. MOBLEY, Toward a Taxonomy of Multidimensional Constructs, Academy of Management Review vol. 23(4), 1998, pp. 741-755.
  11. LEE, Y. W., D. M. STRONG, B. K. KAHN, R. Y. WANG, AIMQ: a Methodology for Information Quality Assessment. Information & Management 40, 2, 2002, pp. 133-146.
  12. MACKENZIE, S. B., P. M. PODSAKOFF, N. P. PODSAKOFF, Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques. MIS Quarterly, vol. 35(2), 2011, pp. 293-334.
  13. MARCHIONINI, G., Evaluating Digital Libraries: A Longitudinal & Multifaceted View. Library Trends, vol. 49(2), 2000, pp. 304-333.
  14. MARSH, H. W., D. HOCEVAR, Application of Confirmatory Factor Analysis to the Study of Self-concept: First- and Higher Order Factor Models and Their Invariance Across Groups. Psych. Bulletin, vol. 97(3), 1985, pp. 562-582.
  15. MASREK, M. N., A. JAMALUDIN, S. A. MUKHTAR, Evaluating Academic Library Portal Effectiveness. A Malaysian case study. Library Review 59, 3, 2010, pp. 198-212.
  16. NICHOLSON, S., A Conceptual Framework for the Holistic Measurement and Cumulative Evaluation of Library Services. Journal of Documentation vol. 60,(2), 2004, pp. 164-182.
  17. PETTER, S., W. DELONE, E. MCLEAN, Measuring Information Systems Success: Models, Dimensions, Measures, and Interrelationships, European J. of Info. Sys. vol. 17(3), 2008, pp. 236-263.
  18. PRIBEANU, C., A Usability Assistant for the Heuristic Evaluation of Interactive Systems, Studies in Informatics and Control vol. 18(4), 2009, pp. 355-362.
  19. RAMAYAH, T., Interface Characteristics, Perceived Ease of Use and Intention to Use an Online Library in Malaysia. Information Development, vol. 22(2), 2006, pp. 123-133.
  20. SARACEVIC, T., Digital Library Evaluation: Toward an Evolution of Concepts. Library Trends vol. 49(3), 2000, pp. 350-369.
  21. SARACEVIC, T., Evaluation of Digital Libraries: An Overview, in: Agosti, M., Fuhr, N. (eds.): Notes of the DELOS WP7 Workshop on the Evaluation of DLs. Padova, 2004.
  22. SEGARS, A. H., V. GROVER, Strategic Information Systems Planning Success: An Investigation of the Construct and Its Measurement, MIS Quarterly, vol. 22(2), 1998, pp. 139-163.
  23. SHEN, R., Applying the 5S Framework to Integrating Digital Libraries. PhD thesis, Virginia Tech CS Department, Virginia, 2006. theses/available/etd-04212006-135018/.
  24. TSAKONAS, G., C. PAPATHEODO ROU, Exploring Usefulness and Usability in the Evaluation of Open Access Digital Libraries, Information Processing and Management, vol. 44(3), 2008, pp. 1234-1250.
  25. TSAKONAS, G., C. PAPATHEODO ROU, Analysing and Evaluating Usefulness and Usability in Electronic Information Services. J. of Information Sci., vol. 32(5), 2006, pp. 400-419.
  26. WANG, R. Y., D. M. STRONG, Beyond Accuracy: What Data Quality Means to Data Consumers, Journal of MIS, vol. 12(4), 1996, pp. 5-34.
  27. XIE, H., Evaluation of Digital Libraries: Criteria and Problems from Users’ Perspectives. Library & Information Science Research, vol. 28(3), 2006, pp. 433-452.
  28. XIE, H., Users’ Evaluation of Digital Libraries (DLs): Their Uses, Their Criteria, and Their Assessment. Inf. Processing and Management. vol. 44(3), 2008, pp. 1346-1373.
  29. XIE, I., S. JOO, Evaluation Constructs and Criteria for Digital Libraries: a Document Analysis. TEFKO 2010, Nov. 7-8, 2010, Brunswick, NJ, USA.
  30. ZHANG, Y., Developing a Holistic Model for Digital Library Evaluation. Dissertation, May 2007. ttp://hdl.rutg [accessed 20 June 2010].
  31. ZHANG, Y., Developing a Holistic Model for Digital Library Evaluation. Journal of the American Society for Information Science and Technology vol. 61(1), 2010, pp. 88-110.