Vol. 17, No. 3, September 2001

Special Issue: Assessing learning with ICT

Guest editorial: assessing learning with ICT

A.McDougall, The University of Melbourne

Email: a.mcdougall@edfac.unimelb.edu.au

This Special Issue of JCAL was prompted initially by a question sent to the Editor by a colleague who, ‘after listening to a lot of presentations where people simply describe their experiences and conclude that the computer somehow helped’, sought a good evaluation strategy to enable him to describe precisely how and why (if at all) students learned differently with the computer model he was using. Throughout the development of computer-assisted learning, extensive work has been undertaken on formative evaluation of software and of innovative projects, but no similar range of effective and reliable ways of assessing any real learning gains attributable to the use of Information and Communications Technologies (ICT) has so far been developed. It is crucial that relationships between ICT use and learning be investigated, given the major developments and growing investment in ICT resources for education.

This is not to say that no work at all has been done on this problem. Johnson et al. (1994) reported on the ImpacT study, which addressed the question of the contri-bution of IT to pupils’ learning. The study showed that in particular circumstances IT had a highly positive impact on children’s achievements; it also illustrated the complexities involved in a large-scale longitudinal study of this kind. Evaluation studies with similar aims have revealed problems such as a tendency to ask ‘technocentric’ questions (Papert, 1985), limitations of measuring techniques such as traditional forms of testing, the variety of evaluation strategies needed for different types of computer use and the necessary dependence of evaluations on context, setting, teaching approaches, etc.

Comparisons with non-ICT strategies are sought but controlled experimental approaches can prove problematic due to: interrelationships among key variables resulting in an inability to control only selected ones; ethical and political questions arising with non-ICT groups; attitudes to computer use in contexts where exams are still conducted using pen and paper; and inherent assumptions that curriculum will remain unchanged despite the introduction of ICT. Is it correct to assume that the tasks undertaken with innovative uses of ICT are the same as those done previously? If a task can be done without ICT by a control group, how much better does an experimental group need to perform to justify the use of expensive technologies?

Despite the difficulties, it is important, perhaps now more than ever, to address the question: Can learning be improved with use of ICT, and if so in what ways? And if success in answering that question is limited, there is a need to explain why.

This Special Issue of JCAL is a collection of papers that show what can be done and, as appropriate, to provide good arguments to illustrate what cannot. It was not intended to include formative or summative evaluation of software environments as such, neither was it to be limited to an examination of computer-based simulation and modelling activities — although the initial query that stimulated the development of the issue did concern modelling.

The issue is titled Assessing learning with ICT, with deliberate intent to exploit the ambiguity in that title. Depending on how these words are grouped, the issues are examined from two rather different, though related, viewpoints. Questions such as whether ICT has enabled improvements in learning environments and facilitated learning processes, and whether the large investment in learning technologies has been, or will be, worthwhile look at the matter from the viewpoint of assessing learning with ICT. However answers to such questions depend, at least in part, on the development of appropriate strategies and methods for assessing students’ learning in technology-rich environments — assessing learning with ICT.

The first two papers provide a valuable introduction to the issue, suggesting frameworks for thinking about the problem and drawing particular attention to the importance of context in the description and assessment of learning with ICT. Next a group of six papers describe empirical research studies that address the question of assessing learning with ICT in a range of learning contexts. The final three papers address issues in and strategies for assessing learning of students in environments where ICT is extensively and increasingly used.

Perspectives on the relationships between ICT and assessment, outlines roles for ICT in schools: a set of skills; a vehicle for teaching and learning of school subjects; an agent of change. The paper raises issues about teaching and testing of skills, and argues for assessment of ICT skills through context dependent use of technology, not simple testing of techniques. McFarlane outlines findings reported from attempts to investigate effects on learning resulting from use of ICT. She describes difficulties inherent in this endeavour, including the lack of theoretical models for the work and problems with interpreting findings when little evidence of context is provided. Drawing attention to the variety of functionality and application of ICT, she argues strongly against treating the term as an aggregation of all its forms, as a ‘homogeneous description of a set of possible interventions’. She asserts that a more developed discourse is required if the relationship between a form of ICT, a model of use, and any impact on the user is to be appropriately examined. The importance of context is further considered in the paper, Examining learning in relation to the contexts of use of ICT. Tolmie emphasises that the same technology or software may have unexpectedly diverse effects, according to the specific setting in which it is employed, and argues for a ‘context-sensitive approach to both the introduction and the evaluation of ICT in education’. He provides examples from primary, secondary and university levels of education to illustrate this, and to indicate the types of issues to be examined when considering the fit between technology, the context of its use and the effects on learning. He draws attention to the difficulty of using information from context-sensitive evaluations suggesting that management effort must be located more within the process of implementation than within design. This implies that evaluation findings should be planned for and disseminated to users, in contrast to the present focus on evaluation for developers.

The next two papers, the first in the group on assessing learning with ICT, present contrasting findings from two Australian studies. These papers clearly demonstrate the importance of describing the specific application of ICT being examined and the learning context, as argued by McFarlane and Tolmie.

The role of visually rich technology in facilitating children’s writing describes work with a case study group of five upper primary aged students, previously very low achievers in writing, with strongly preferred visual learning styles. The investigation showed that student use of the visual and animation features of the MicroWorlds multimedia programming environment, concurrent with engagement in narrative writing tasks, mediated a substantial improvement in the fluency and the quality of writing of these students. Vincent argues that the far greater complexities in structures, observed in the students’ written work in the multimedia development environment, suggested that those structures were inherent in the minds of the students and were triggered by the application of the software.

A contrasting result from a somewhat similar study is reported in The computer as a Trojan Horse. Boyle worked in a relatively similar school environment, with students only slightly older than those in Vincent’s study, but Boyle’s study concerned a class of boys considered low performers in formal school mathematics. He investigated the efficacy of the use of scaffolded multimedia software development as a vehicle for incidental learning of mathematical techniques. The researcher’s records showed that all but one of the students used a broad range of the essential mathematical concepts and skills and peer reviews of their software judged that all but one project adequately covered the mathematical material required and were worthy of use for teaching students in the following year. However regular school testing gave no evidence to suggest that the intervention had any effect on the performance of any of these students in normal school mathematics. Boyle uses Minsky’s idea of a devil demon frame to explain the apparently insurmountable gap between these students’ successful experience in the mathematical software design activity and their unsuccessful experience in formal school mathematics.

The next two papers describe studies involving comparisons between technology-rich teaching strategies and more traditional approaches. A problem-solving based computer-assisted tutorial for the earth sciences, by Chang compares the use of a problem-solving based computer-assisted tutorial system with a lecture-internet-discussion teaching approach, for Taiwanese high school students. The problem-solving approach was associated with some valuable learning activities; however, gains in student achievement were not found to be significant, and no significant changes were found in attitudes towards earth sciences. ICT in higher education: evaluating outcomes for health education, by Lockyer and colleagues, describes an examination of collaborative tutorial activities carried out in web-based and face-to-face environments at an Australian university. The effectiveness of each approach was measured in terms of observed learning outcomes, analysis of learner interactions and reports of learner perceptions of the experience. Many of the comparative findings did not provide significant differences; however, the study showed that web-based environments with embedded collaborative activities can effectively foster rich learning experiences and positive learning outcomes. The major disadvantage of the web-based approach, as perceived by the students, was the reduced level of contact with lecturers. While both Chang’s and this paper provide useful information about the technologically-rich environments they describe, the lack of significant results in both reports appears to substantiate the problematic nature of comparative experimental studies in this area.

The final two papers in this group, while still focusing on assessing learning with ICT provide a transition toward the next group, as these two are also concerned with strategies for assessing learning of the students in the technology-rich environments being investigated. Discovery simulations and the assessment of intuitive knowledge by Swaak and DeJong, examines relations between the features of discovery simulations, the learning processes elicited, the knowledge that results and the methods used to measure this knowledge. They conclude that assignments associated with the technology-based tasks contribute to the effectiveness of simulations, and that tests of intuitive knowledge appear to be able to measure the results of learning with discovery simulations. In ‘Bridging Tools’ for change: evaluating a collaborative learning network, Bonamy and colleagues discuss the development and evaluation of a collaborative learning environment in which higher education students in five European countries worked together on team-based academic activities. The students are supported by tutors from any of the participating institutions. Analysis of the logbooks and reflection reports of students using the environment showed learning mostly of ICT skills, followed by methodological skills and finally metacognitive skills. Unanticipated effects of implementing this form of collaborative distance learning in traditional institutions were noted, and the authors provide lists of enabling and disincentive factors from the experience.

The final three papers focus particularly on strategies for and approaches to the assessment of students’ learning in technology-rich situations. Collis and colleagues in Feedback for web-based assignments identify types of web-based tools that might be used for feedback. They illustrate these with examples of feedback strategies supported by a web-based course management system, discussing instructor support in this environment and suggesting that new forms of feedback supported by web-based environments can enable the provision of enriched learning for students. In Experiences of assessment: using phenomenography for evaluation, Jones reports on interviews with academics from five universities in the UK, all of whom had used computer conferencing systems on the web for delivery of courses, and with 10 students using computer conferencing for study and assessment in an Open University course. The academics raised issues of difficulty of designing online activities and courses, and drew attention to assessment criteria as a key feature of course design. The students revealed important variation in their interpretations of assessment guidelines; many of them were concerned, far more than the course designer had intended, with the collaborative aspects of the assessment activities, with commensurately less emphasis on the content of the course. Jones argues that a phenomenographic approach to such variation could have implications for the design of networked learning environments.

The necessary linking between new learning approaches associated with developments in ICT and methods for formal assessment of learning are the focus of What have examinations got to do with computers in education? Harding describes several projects exploring the impact of new technologies on learning and assessment, based at the Interactive Technologies in Assessment and Learning Unit in the University of Cambridge Local Examinations Syndicate.

Finally I want to thank all the contributors to this Special Issue for their submissions and for their co-operation in meeting deadlines. I have appreciated enormously the support, patience and wisdom of Bob Lewis throughout the development of the issue.

References

Johnson, D.C., Cox, M.J. & Watson, D.M. (1994) Evaluating the impact of IT on pupils’ achievements. Journal of Computer Assisted Learning, 10, 138–156.

Papert, S. (1985) Computer criticism versus technocentric thinking. Logo 85 Theoretical Papers. (pp. 53–67). Massachusetts Institute of Technology, Cambridge, MA.

Perspectives on the relationships between ICT and assessment

A.McFarlane, Graduate School of Education, University of Bristol

Email: a.e.mcfarlane@bristol.ac.uk

There are at least three perspectives on ICT in schools which influence policy and practice. This paper considers the relationship between each of these perspectives; as tool, learning support and revolutionary agent, and their relationships to assessment. It proposed that ultimately what determines the role ICT plays in school-based learning depends on the value attributed to learning outcomes. While national assessments continue to reward specified content knowledge above knowledge building abilities, the use of ICT will continue to cause tensions in the classroom.

Keywords: Assessment; IT-use; Knowledge building; Learning outcomes; Practice; Policy; School

Journal of Computer Assisted Learning (2001) 17, 3, 227-234 Accepted 14 May 2001

Examining learning in relation to the contexts of use of ICT

A.Tolmie, Department of Psychology, University of Strathclyde

Email: a.k.tolmie@strath.ac.uk

Although ICT resources are commonly expected to produce uniform benefits, they are necessarily employed within pre-existing contexts of educational and social activity, and the outcome in terms of both pattern of use and learning depends on how they fit in with these. As a result, the same technology or software may have unexpectedly diverse effects, according to specific setting. If the object is to exercise control over outcome, then the conditions of use need to be planned for within the design and implementation of the technology. In order to do this, it is crucial that research gathers data on how outcomes are affected by the interplay between technology and context. This raises questions about the methods that would be appropriate for the conduct and dissemination of such research. These points are discussed in relation to three studies, one each at primary, secondary and university levels of education.

Keywords: Conferencing; Context; Dissemination; Gender; Group; Illuminative evaluation; School; University

Journal of Computer Assisted Learning (2001) 17, 3, 235-241 Accepted 1 May 2001

The role of visually rich technology in facilitating children’s writing

J. Vincent, Department of Science and Mathematics Education, University of Melbourne

Email: jlv@alphalink.com.au

A significant sample of children in any class may be categorised as having a preferred visual learning style. The writing outcomes of a case study group of five upper primary children with a strongly preferred visual learning style were examined when the children were encouraged to write with the aid of a computer and a visually rich software medium. A constructionist classroom environment was established in which children were exhorted to dare to experiment, play and problem set, especially when using microworlds and laptop computers. The writing outcomes of this group grew in volume and linguistic complexity when the students were offered the opportunity to use the visual and animation features of the software at the same time as engaging in narrative writing tasks. These outcomes are situated in recent neuro-psychological theory of the role of the right brain hemisphere as a platform for language development and comprehension, and suggest the possible vital place of computers in the full realisation of their potential by children with a visually preferred learning style.

Keywords: Assessment; Constructivism; Interview; Learning styles; Observation; School; Visual preference; Writing

Journal of Computer Assisted Learning (2001) 17, 3, 242-251 Accepted 16 April 2001

The computer as a Trojan Horse

M. Boyle, Department of Science & Mathematics Education, The University of Melbourne

Email: wmboyle@unimelb.edu.au

This paper reports on a study of boys in an Australian secondary school who were under achieving in school mathematics. The subjects were asked to develop computing software to teach fellow students about aspects of Ancient Egyptian mathematics. The paper reports four example case studies of the experience of the subjects during the course of the intervention. The paper examines the use of computing software development as a vehicle for the incidental learning of mathematical technique and for changes in attitude towards mathematics. Four different mathematical activities are reported in the case studies with a screen-dump of the subjects work included as an illustrative example.

Keywords: Attitude; Computing; Incidental learning; Interview; Mathematics; Satisfaction; Secondary; Software development

Journal of Computer Assisted Learning (2001) 17, 3, 251-262 Accepted 18 May 2001

A problem-solving based computer-assisted tutorial for the earth sciences

C.Y. Chang, Department of Earth Sciences, National Taiwan Normal University

Email: changcy@cc.ntnu.edu.tw

This study investigated the effects of a Problem-Solving based Computer-Assisted Tutorial (PSCAT) on earth science achievement and attitudes toward earth science of tenth graders (16-year olds) in a senior high school in Taiwan. A total of 137 students who were enrolled in four earth science classes participated in this pre-test/post-test control-group experiment. The experimental groups received the PSCAT; whereas the comparison groups received a Lecture-Internet-Discussion (LID) teaching approach. A multivariate analysis of covariance on the post-test scores of the Earth Science Achievement Test and Attitudes Toward Earth Science Inventory, with students’ pre-test scores as the covariates, suggested that PSCAT produced (almost) significantly greater gains on students’ earth science achievement than did the LID approach and that no statistically significant increase or decrease in student attitudes toward earth science was found for either group.

Keywords: Achievement; Attitudes; Computer; Control group; Problem solving; Science; Secondary school

Journal of Computer Assisted Learning (2001) 17, 3, 263-274 Accepted: 7 January 2001

ICT in higher education: evaluating outcomes for health education

L. Lockyer, J. Patterson & B. Harper, University of Wollongong, Australia

Email: llockyer@uow.edu.au

This paper presents an investigation that examined and compared the effectiveness of collaborative tutorial activities carried out in both web-based and face-to-face learning environments within an undergraduate health education subject. Effectiveness of the different learning environments was measured in terms of observed learning outcomes, analysis of learner interactions and reported perceptions of the learners regarding their experience. Results demonstrated that web-based environments, with embedded collaborative activities, can effectively foster rich learning experiences that result in attaining positive learning outcomes.

Keywords: Collaboration; Health education; Interaction; Learning outcome; Tutorial; Undergraduate; World-wide web

Journal of Computer Assisted Learning (2001) 17, 3, 275-283 Accepted 25 May 2001

Discovery simulations and the assessment of intuitive knowledge

J. Swaak & T. de Jong, Telematica Instituut and University of Twente, Enschede

Email: swaak@telin.nl

The objective of the present work is to have a closer look at the relations between the features of discovery simulations, the learning processes elicited, the knowledge that results, and the methods used to measure this acquired knowledge. It is argued that discovery simulations are ‘rich’, have a relatively low transparency, and require active involvement of learners. Discovery simulations are suited to support data-driven, partly implicit learning. Discovery learning leads to intuitive knowledge. To complement this conceptual investigation, a series of five experimental studies is described. In all five studies, learners were pre-tested and post-tested with several knowledge measures. Central to the set of tests was one with the objective of measuring intuitive knowledge. One conclusion of these experimental studies is that assignments contribute most clearly to the instructional effectiveness of simulations. Another conclusion is that the intuitive knowledge tests seem able to measure the results of learning with discovery simulations.

Keywords: Assessment; Discovery learning; Explicit knowledge; Intuitive knowledge; Physics; Post-secondary; Pre-/post-test; Simulation

Journal of Computer Assisted Learning (2001) 17, 3, 284-294 Accepted 18 May 2001

‘Bridging Tools’ for change: evaluating a collaborative learning network

J. Bonamy, B. Charlier & M. Saunders

Université Lumière Lyon 2, Université de Namur & Lancaster University

Email: bonamy@gate.cnrs.fr

This paper outlines the evaluation approach associated with the Learn-Nett project under the SOCRATES initiative. It describes the project, locates the role of evaluation within it and finally conceptualises the role evaluation might play in a change process. The paper argues that evaluation in these circumstances can be essentially formative but also be prompted by a search for new knowledge in the area of changing practices in learning. It identifies ‘transition’ as anomic and the need to develop ‘provisional stability’ for change to occur effectively. In such periods of change, evaluation products can act as useful ‘bridging tools’ which can support the development of ‘provisional stability’.

Keywords: Change; Collaboration; Group; IT-use; Online; Professional; Teachers; Qualitative; University; Virtual campus

Journal of Computer Assisted Learning (2001) 17, 3, 295-305 Accepted 24 May 2001

Feedback for web-based assignments

B. Collis, W. De Boer & K. Slotman, Faculty of Educational Science and Technology, University of Twente,

Email: collis@edte.utwente.nl

In the Faculty of Educational Science and Technology at the University of Twente an educational concept based on the ideas of increased flexibility in learning options and the active student, has been in place since 1998. In this approach, many courses have moved from a model of lectures and a final examination or single large project, to a model of six or seven assignments or subparts of a larger overall project, each submitted via a web-based course-management system and each worth a certain portion of the overall course grade. Feedback has become a key aspect of teaching as well as assessment. This paper gives an overview of feedback as part of the assessment process in learning settings making use of web-based environments. This is followed by some examples of feedback strategies supported by the web-based system in use in the faculty and a description of a web-based decision-support tool, currently under development, for instructors relating to feedback.

Keywords: Action research; Assessment; Feedback; Mediated; Post-graduate; Student-centred; Web-based system

Journal of Computer Assisted Learning (2001) 17, 3, 306-313 Accepted 16 June 2001

Experiences of assessment: using phenomenography for evaluation

C. Jones & M. Asensio, Departments of Educational Research & Management Learning, Lancaster University

Email: c.r.jones@lancaster.ac.uk

The aim of this paper is to explore the use of assessment as a tool for structuring students’ experiences within a networked learning environment. It is suggested that this investigation may have a wider bearing on issues raised by the idea of aligning teaching approach with the students’ approach to learning. The methods used are broadly phenomenographic and the use of this approach is proposed for the evaluation of networked learning in higher education. The work is drawn from the initial phase of a two-year study being undertaken at Lancaster University. The choice of phenomenography as the preferred methodological approach is explained and how this is appropriate for evaluation. An emphasis is placed upon the evaluative aspects of phenomenography, its focus on varieties of experience and the relationship between approaches adopted to learning and the outcomes of learning. The example, drawn from the research, examines student approaches in relation to the declared intentions of the course designers.

Keywords: Collaboration; Distance; Evaluation; Illuminative; Interview; IT-use; Conferencing; Phenomenography; Qualitative; Undergraduate

Journal of Computer Assisted Learning (2001) 17, 3, 306-313 Accepted 24 May 2001

What have examinations got to do with computers in education?

R. Harding, University of Cambridge Local Examinations Syndicate

Email: r.d.harding@ucles-red.cam.ac.uk

Courseware, computer based learning materials, usually includes some self-assessment for formative assessment. However, insufficient attention has been paid to the link between formal assessment and use of courseware: teachers and learners focus on examination syllabuses. Examinations bodies can use only the simplest forms of ICT-based assessment (objective or multiple choice tests) until learners are familiar with newer styles. This problem can be addressed on several fronts: with new ICT styles of tests, by developing websites supporting learners and teachers, by conducting field trials using computers under examination conditions and by commissioning new types of examination questions built around online interactive simulations.

Keywords: Assessment; Change; English; Examination; Internet; Mathematics; Professional; Secondary; Vocational

Journal of Computer Assisted Learning (2001) 17, 3, 322-328 Accepted 10 May 2001