eLearning > Quality Regulation

China eLearning QA

1. Background
2. Purpose of Quality
3. Method of State assessment

4. Implications for the management of online institutes
5. Issues and drivers
6. Concluding comments

1.      Background

Education is an area of significant concern for Chinese government; higher education, in particular, has enjoyed considerable attention within recent years.  The nature of political concern with expansion in the university sector could be summarised in terms of the following three issues:

Elearning has been a significant strategic investment in a striving to achieve these aims.  The institutionalisation and growth of elearning in China has been documented under the eLearning > History section.   However, it has become clear from our own research that acceptance of this particular development strategy is not straightforward.  So, there is some public uneasiness about the value of qualifications attained by eLearning routes. 

However, concern extends beyond popular public opinion.  We have interviewed a sample of tutors from the 8 leading online schools that deliver programmes in English Language.  Their responses indicate a more professional level of unease. The majority believed that a campus-based course is always superior to an online course.  (Although the majority also believed that an online course is more demanding on students.)  Moreover, tutors were themselves uncertain regarding their own preparation to support learners in these novel forms of study.

In addition, we interviewed institutional managers from these eight schools. All but one had modelled their web-based courses on existing, conventional ones.  Yet two of these managers were not willing to answer questions about comparability of standards in the delivery and impact of these two modes of learning, and the others were clearly divided about the probable comparability of standards.  In everyday talk, we could say that there is here an issue of managing the “quality” of new educational practice.

At present, t he educatinal systems is seeking to address concerns over quality through closer monitoring and review of online distance education - as a way to build confidence in what is being offered

2.      Purpose of Quality

The goals of the online institutes are to serve the needs of the nationally-defined curriculum within a rapidly expanding public education system.  The anticipated knowledge and skills that a learner is expected to acquire through the study programme in any subject are laid down by the Ministry of Education.  These are specified across the various points of student progression and are laid down in great detail e.g. in the case of English language learning, it will be stipulated, for instance, how many words a learner can write in a minute.  The level of central planning across all sectors of education in China therefore provides public statements about what a learner might reasonably be expected to know at the end of his/her course – much in the same way, within Britain, the national curriculum provides such an account of learners in the primary and secondary levels of schooling, though not in the tertiary sector. 

Online education requires the effective deployment of staffing, resources, facilities, technology, assessment activities and learner support services to enable students to work (mostly independently) towards their desired educational aims.  The design and interplay of these aspects have cumulative impact upon the online enterprise and are reliant upon learner access to (networked) technology and for study to be accommodated within their daily activities. 

The extent to which the institute profiles and/or makes visible the various strands of its activities to learners may vary though learners will typically receive orientation or induction in the early stages of their online education as to what is available and how to make the most of the available resources.  Equally, the extent to which online institutes seek to promote their activities and/ or differentiate themselves by the services and reputation they offer to prospective learners is a moot point.  However, under the terms of the MoE licence what requirements are there upon institutes to reveal specifications for their operations in terms of the structures and mechanisms in place?

In other words, what are the higher orders of organisation and arrangement that have to be put in place?  It may be inferred that at a system-wide level, these strands are considered the most vital in formulating the groundwork for assessments of whether the online institutes have met their goals. 

So, how is the performance of the online institutes judged?  To what extent does an institute have to trap data in order to show good stewardship?  These are substantial questions in their own right, and the project work to date, will only offer mere glimpses into this world.  Two aspects are worth a preliminary address: how are the online institutes judged to succeed by the MoE (Methods of State assessment), and secondly, how deliberate are online institutes efforts in tracking and recovering traces of activity within their own system e.g. quality functions (Implications for the management of online institutes)?

3.      Method of State assessment
In 2001, the State introduced four key instruments to track and monitor the performance of online institutes, including their distributed learning centres:

i. Online institute self-assessment
As part of the reporting procedures, online institutes are required to prepare a self-report under the following headings:

Table1: Self-assessment checklist issued by MoE

This checklist is accompanied by an annual statistical return; both of which are submitted to the MoE.  (Deadline: typically December of each year).

ii. Online institute annual report
This report is prepared by the institute following the annual self-assessment exercise, and similarly submitted to MoE (Deadline: February of the following year).

Basic tables:

Each item unpacks into a series of regulatory requirements (apparently) mainly concerned with summative evaluation indicators.

iii. Selective examination / on-site visits
A range of visits to both headquarters and distributed learner centres may be undertaken (These usually occur in late Spring):

iv. MoE Annual assessment/ inspection
Yearly inspections are carried out by the Web-based Education Guiding Committee of Higher Education Schools under the auspices of MoE to review all the foregoing items.  Following approval, the MoE release the results of their findings in the late Spring of the new year.

Results are collated centrally and fall into one of two categories: Pass/ Fail.  Only those which pass may proceed in their operations.

Failure is marked in cases of serious violation of relevant administrative regulations; signs of serious problems in teaching; and/or failure to participate or meet deadlines within the annual review process.  According to the gravity of the breach, the MoE may resort to several punitive actions:

Summary
It is not clear how many institutes have been issued warnings or failed in the system given the national need to maintain confidence in the scheme.  It would seem that the overall number of online institutes has remained intact over the past couple of years, though some are now changing their orientation e.g. to focus on internal elearning provision within the campus rather than through distance education options.  However, in 2005, there are about 3000 learning centres in China though it seems c.200 failed their annual assessment (Liu, Y., 2005).  As part of this growing management and administrative requirements to oversee this process, a colleague mentioned that a Higher Education Evaluation Centre (which was part of MoE) might be established as a standalone unit – akin to some of the other international quality agencies - to take a lead on the quality review of pilot universities.  It is unclear whether this happened.  However, 2005 was a critical juncture for the online institutes as the MoE undertook a major evaluative review of the pilot universities online education initiative.  The outcomes of this examination were not visible to this research team during the project, but the results will likely influence the next phase of development for online education.

4.      Implications for the management of online institutes

Part of the audit trail might reasonably ask - what additional measures need to be put in place and/or managed within this process to provide capture of an institute’s internal activities so that it can be held to account?  Thus, what are the implications of this form of reporting upon the online institutes?  From our limited encounter with staff from various online institutes, concerns over quality are often expressed though it is difficult to probe what is meant by these issues.  One thing is certain, a considerable amount of time and effort is required from senior staff in order to compile accounts, coordinate reporting and prepare for site-visits etc.  Colleagues at one leading online institute estimated that the annual report paperwork alone required over half a month’s work from the senior staff team co-ordinating the various elements for the 2005 exercise. 

In their case, they report that the onus to develop traces for trapping and demonstrating the quality of learner interaction within their system originates with them rather than being driven by the central administration.  In their case, they have produced an internal quality handbook (through the auspices of an internal quality unit) which systematises all their activities with associated flowcharts and criteria, including detailed specifications of people’s roles, functions and tasks within the system, against which actual activities are measured.  In this way, the internal planning and decision-making sits comfortably with formal reporting needs of this particular activity.  Other institutes also seem to engage in similar practices. 

Thus, for individual specified activities associated with the institute, some form of audit trail exists.  Two key questions emerge – as with any quality system.  Do those items which are privileged within the audit trail provide reliable indicators of what is actually going on within the system?  And secondly, what inferences may be made about the ‘health’ of activities based upon those indicators.  For example, tutors at the remote study centres are subject to the rigours of the institute’s quality monitoring programme.  While, it is possible to specify the qualifications of the tutors and the nature of the desired engagement that they should have with learners e.g. contact time with learners, the pace and terms of student feedback etc. is this contingent on students need?  Are these interactions driven by learner demands or by the constraints of timetabling and process organisation?

In our follow-up interviews with the senior managers of eight online institutes, all reported that they used student feedback as a means of monitoring tutor performance (as well as other aspects).  However, it would require a further set of investigations to make visible what students feel about the student feedback process.  To what extent do individuals modify their responses in light of having to share them with others?

Further research is therefore required to ascertain what is actually being measured by these trails and the degree of correspondence they have to how students actually conduct, regulate and engage with the various learning opportunities open to them.  This represents a substantial endeavour and is probably best approached through building up detailed institute case studies of the interactions between programme staff, tutors, learners and those responsible for audit.

5.      Issues and drivers

The policy landscape for operating online institutes has evolved rapidly since 2001.  Major revisions to the regulatory framework are evident.  On the basis of the available data, we would like to highlight the following factors as potential influences on the trajectory of State-level quality control of online learning:

1. Enhanced proceduralisation of activities
The requirements upon online institutes to make explicit their activities have been increasingly systematised and specified into standardised forms of presentation and reporting.  This trend seems likely to continue. 

As part of the MoE’s strategy for improving the quality of online distance education, various Expert Panels and Strategic Groups were established to advise on developments e.g. the Chinese eLearning Technology Standardisation Committee (CELTSC).  Nine sub-projects were also set up to develop and trial new instruments for the purpose of capturing (and by implication advancing system-wide) good practice in quality assurance and to advise on possible new instruments and regimes for monitoring, assessing and validating activities. 

Recent ‘experimental’ work at Beijing Normal University has sought to devise weighted models for assessing the quality of various aspects of online education activity.  An example development project aimed to produce courseware quality certification standards.  Attention was paid to specifying primary (n=6), secondary (n=32) and tertiary (n=44) level indicators of institute activities, the latter of which comprised 17 mandatory objects, 23 optional objects and 4 extra objects.  The primary level addressed: teaching design, teaching content, technical content, information delivery, usability and resources.   Scores were then assigned to each object.  The intended aim of the project was to be “less concerned with the expertise and qualities of teaching professionals but the performance, behaviour and outcomes of the various participants in the web-based teaching process.” (Huang, 2003a).  Following a pilot study with 3 programme areas, revisions to the potential scoring system were suggested. 

Other state-level projects have sought to address the development of accreditation systems for online teaching (Huang, 2003b).  While the extent to which the findings of these development projects will be adopted and integrated into mainstream practices remains unclear, it indicates a strong predilection to ‘anchor’ points in the educational system that can be monitored and measured.

2. Regulatory orientation towards that which can readily be measured: infrastructure and management concerns
In the earliest documentation, online institutes were asked to make explicit items which relate easily to campus activity:  principal’s assurance; methods of student recruitment; methods of school examinations; teaching approach; student support and service; the standard of learning centres; graduation and degree; conditions of the pilot university and self-assessment (2001: a).  In so far as there is reference to specific designs of teaching and learning practices for “web-based” provision (under item 4), their characterisations and projected measures for assessment are those that overlap with traditional (campus) activities: e.g. the staging and invigilation of exams; the schedule and venue arrangement for the interactive teaching; the source of experienced teaching staff, documenting student attendance; the ratio of tutors to students; the standard and design of off-campus learning centres. 

Over time, the online institutes were increasingly asked to provide more information and monitoring of the remote learning centres, in particular, in recognition of some problems which had been identified with individual instances of poor operation.  Key problems such as the “ill adjustment to new conceptions of web-based delivery, unsound management systems, relatively backward school facilities, and a lack of quality teaching resources at some pilot schools” (2002: b) forced greater attention to be paid to the general infrastructure and environment. It is arguable that these represent elements which can be remedied given that they lie within the control of the online institutes.

3. Limited regulatory drivers or direction for specifically online learning quality issues
Although MoE documentation released in 2002 indicates more urgent concerns for web-specific issues in response to the problems outlined above, little prescription has been given regarding how institutes might ensure improved study practices and routines amongst students, other than outline provisions around providing ‘good’ resources and tutorial interaction.  This is broadly accepted as a limitation to the quality process:

“In dealing with this question, it has to be recognised that the quality of learning for online students is in many ways different from the notion of quality that applies in traditional learning.  How to define the quality of eLearning is, therefore, a challenge for eLearning institutions.  Equally difficult problems are involved in the practical application of quality assurance, especially at a time of rapidly expanding enrolments in eLearning,” (Ding et.al., 2005, p.71).

Once again, this highlights the need for further research into what might constitute effective models of online learning. 

4.   “Modern distance education”: Tensions between ‘elite education’ and ‘public education’
The online institutes were set up to be self-financing over time, particularly in those instances where private or venture capital was involved.  The market orientation results in a level of competitiveness among institutes for the recruitment and retention of students (item 2 of the MoE checklist).  However, it is generally acknowledged that the entry qualification of these students will not be on a par with campus based learners, and in addition that online learner typically face more constraints in developing their study routines compared to their campus counterparts.  Following some instances of very poor student performance, a number of the online institutes instituted their own entrance exams.  However, this leads onto a more fundamental issue of accreditation (item 8 on the MoE checklist).  What kind of degree are learners being issued upon successful completion of their online programme of study?  How does it equate to other forms of degree?  What is its standing vis-à-vis the campus based programmes?  What are the respective pass-rates and are the outputs consistent across different institutes? 

Currently online institutes may offer two types of degree: (i) adult distance education or (ii) on-campus, full-time degree of the parent campus institution (though special wording is inserted on the degree certificate).  The form of accreditation has to be specified in the official documentation and remains a source of tension between the various interests given the issue of validating the worth of the required accreditation.  State, university, online institute and learners (both online and campus) all have vested interests in how the degrees are perceived and accorded status. 

In general, it seems that online and campus programmes offered in the same subject are required to use different course textbooks even though the national curriculum will specify the final outcomes for the learners.  However, the current debate about the future of online learning in China circles about this issue.  Cross-validating the course objectives, assessment and accreditation against actual achievements of learners on both online and campus-based settings is still in its early days and therefore uncertainty remains over the value and respect with which the online degrees are held. (Previously identified under Background section).

6 Concluding comments

It is arguable that other forms of State-level checks and balances for monitoring and tracking educational activities with the institutes (and universities) are available.  For instance, some forms of national competency based examinations provide a common benchmark (if learners elect to sit the qualification) e.g. TEM in English language learning.  A national examination thus provides an indirect form of monitoring of institutional performance against the core goals for educational reform.  The results may help make visible those institutions which have failed to meet some required standard.

In addition, all publications, especially textbooks, have to receive MoE approval prior to publication.  Book outlines containing course content and aims have to be submitted jointly by the academic and publisher for approval prior to being issued with a publication number.  In this way, it may be possible to ensure alignment to a common goal given that the assessment and examination procedures seem to align closely to the content of course texts.  Although the exact nature of changes ongoing within Chinese higher education publication are still unclear, particularly given the rise of e-publications, it may in part help explain the critical role and continued vitality of the core text book within study programmes.  As noted by the project teams, each course (and course leader) will have its authorised text.

As a concluding note to this report, it should also be considered how innovation and improvement are captured within the quality assurance system, given that the notion of compliance or conformity traditionally sits at the heart of the quality agenda.  From our initial investigations, it seems that there is innovation ‘space’ afforded within the system which is continually negotiated within the existing system – the eChina-UK programme being an example of this.

References

Ding X., Gu X. and Zhu, Z. (2005), The Chinese Approach in McIntosh, C. and Varoglu, Z., Lifelong Learning and Distance Higher Education (Commonwealth of Learning/ UNESCO), pp. 63-78 Available at http://www.col.org/LLLinHigher/PSeries_LLLDHE_CH06.pdf  (Retrieved 20 May 2006)

Huang, R. (2003, a). Research and experiments on quality certification of online curricula. China Distance Education, 21. [Link]

Huang, R., & Cao, Y. (2003, b). On the teaching quality of current online education. China Distance Education, 11. [Link]

Liu Y., (2005) Quality assurance and monitor system of China inland modern distance higher education.  Conference on Quality Assurance for Online Education in China (Hong Kong, June 2005).

The QA framework for online institutes was adapted from the following policy documents:

MoE (2000) Document on developing network-education colleges of several higher education schools (July 2000)

MoE (2001, a) General Department, MoE, Notification of conducting annual report assessment to the School of Web-based Education at the universities piloting modern distance education, (Document Number 10 of Higher Education, 2001)
Annexes:
i. Suggestions for the implementation of the 2001 annual report assessment to the School of Web-based Education at the universities modern distance education – from the MoE
ii. The 2001 self-assessment checklist for the School of Web-based Education at the universities piloting modern distance education
iii. The 2001 annual statistics of the School of Web-based Education at the universities piloting modern distance education.

MoE (2001, b) Emergent notices of the Ministry of Education about strengthening admission managements in distant learning (Aug, 2001).

MoE (2002, a) Modern Distance Education Off-campus Learning Centre (site) Temporary Management Regulations (Jan 2002)
 
MoE (2002, b)  Suggestions for strengthening the management of the School of Networked Education in Higher Education Schools and Raising its teaching quality (Document No, 8 of Higher Education, July, 2002)

MoE (2002, c) Notice for the approval of 2002 autumn admission plans of network college in higher education (Aug, 2002)

MoE (2002, d) Notice from the Ministry of Education about strengthening managements on Higher Education accredited certificates

MoE (2002, e) Notices from the MoE regarding how modern distance education schools develop annual review reports and yearbook (2002)

MoE (2002, f) Opinions from the MoE on how to write an annual review report, (2002)

MoE (2003) Principals on establishment and management of off-campus learning centre in modern distance education (March 2003).