Statement by Lingnan University University Rankings 2007 by Education 18.com Mislead the Public (26 July 2007)
Lingnan University (“Lingnan”) expresses regret over the survey results on university rankings just released by Education 18.com. With serious doubt about the professionalism and accuracy of the poll, Lingnan considers the poll results can seriously mislead the public, including parents and students.
Lingnan thinks that the problem lies not with the rankings themselves but rather the way the survey was carried out, including the methods used, which did not follow statistical and scientific principles. The rankings were derived partly from the “opinion survey” conducted over telephone by the University of Hong Kong's Public Opinion Programme. Of concern to Lingnan are three areas:
1) The people polled were randomly selected members of the public, most of whom lacked an adequate understanding of the various universities. The questions in the poll were not designed professionally, eg. respondents were asked about institutions’ local and international reputation, facilities, staff qualifications and academic research performance etc. Even secondary school principals, not to mention members of the public, cannot give a score to each institution on the basis of such wide-ranging areas. Therefore, what was obtained eventually was just the general public's impression of various institutions, not their actual academic standing.
Yet the education web site determined its rankings by combining the opinion poll results with other incomplete assessment indicators. The result is a seriously misleading report causing the public to have a wrong understanding of the actual situation of the institutions.
2) Of all the important evaluation data (except library resources and electronic databases), such as research performance, research grants allocation, the education web site's survey only looked at the aggravate rather than per capita figures. In terms of research output, the web site only calculated the output reported by institutions, even including publications in non-academic journals. The 2006 Research Assessment Exercise conducted among the eight publicly-funded institutions by the University Grants Committee showed that Lingnan had outstanding achievement in the three academic disciplines of business, social sciences and humanities. Institutions’ research outcome was measured in a highly vigorous manner by an independent academic panel. But the education web site had failed to make reference to such internationally recognized assessment methods. Because of its flawed assessment methods, the web site's poll produced results different from that of the UGC exercise.
3) Also seriously misleading is the figure on each institution's “graduate performance”, determined on the basis of the opinion poll. The education web site did not mention which fields the more than 100 employers surveyed came from, the size of their companies, nor whether they had direct supervision over the graduates etc. Therefore, the survey lacked a scientific basis and its results did not tie in with that of employers’ surveys, such as those conducted by the Education and Manpower Bureau (now the Education Bureau) and the Hong Kong Economic Journal Monthly, in which Lingnan graduates were given high ratings. The interviewees of these two polls were all direct employers, and their evaluation of graduates was far more credible. Yet without adopting a representative surveying approach, the web site produced a biased survey that can seriously mislead the public.
Another source of bias comes from the fact that despite heavy weightings of 35 per cent and 20 per cent were given repectively to both rankings on admissions grades, there are fundamental discrepancies in the ways the web site calculated institutions’ scores. Even the years in which some of the data came from were different.
Lingnan expresses deep regret at the release of university rankings by Education 18.com and HKU's Public Opinion Programmes, year after year, with no attempt ever made to review and improve the polling exercise.