Loughborough University
Leicestershire, UK
LE11 3TU
+44 (0)1509 263171
Loughborough University

Loughborough University Institutional Repository

Please use this identifier to cite or link to this item: https://dspace.lboro.ac.uk/2134/20497

Title: Bridging the edtech evidence gap: A realist evaluation framework refined for complex technology initiatives
Authors: King, Melanie R.N.
Rothberg, Steve
Dawson, Ray
Batmaz, Firat
Issue Date: 2016
Publisher: © Emerald
Citation: KING, M. ...et al., 2016. Bridging the edtech evidence gap: A realist evaluation framework refined for complex technology initiatives. Journal of Systems and Information Technology, 18 (1), pp. 18-40.
Abstract: Purpose - There are five factors acting as a barrier to effective evaluation of educational technology (edtech) these being: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this investigation was to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives. Design/methodology/approach - An initial investigation, of traditional evaluative approaches used within the technology domain, was broadened to investigate evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps, to support mapping the technical complexity of initiatives. Findings - A refined illustrative example of a realist evaluation framework is presented including two novel architectural edtech domain reference models to support mapping. Practical implications - Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology ‘context’ then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity. Originality/value - This paper makes a novel contribution by arguing the imperative for a theory based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surface ‘what works, for whom, in which contexts and why’ benefiting educators, IT managers, funders, policy makers and future learners.
Description: This paper was accepted for publication in the journal Journal of Systems and Information Technology and the definitive published version is available at http://dx.doi.org/10.1108/JSIT-06-2015-0059
Version: Accepted for publication
DOI: 10.1108/JSIT-06-2015-0059
URI: https://dspace.lboro.ac.uk/2134/20497
Publisher Link: http://dx.doi.org/10.1108/JSIT-06-2015-0059
ISSN: 1328-7265
Appears in Collections:Published Articles (Computer Science)
Published Articles (Mechanical, Electrical and Manufacturing Engineering)

Files associated with this item:

File Description SizeFormat
KING_etal_full_formatted_18-11-15_repository.pdfAccepted version428.68 kBAdobe PDFView/Open


SFX Query

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.