Pierre-Sofiane Kadri
Launched on 13 May, the site U-Multirank lets you compare 850 universities around the world. Supported financially and politically by the European Union, the tool offers a personalized search and various performance indicators. Complex and sometimes imprecise, the first version of this innovative project, imagined as the antithesis to classical academic rankings, is not entirely free of faults.
This article was originally published in French: U-Multirank : un projet ambitieux pour comparer les universités. It was translated to English by Abby Tabor.
Don’t expect a definitive list of the 100 best universities; the objective of U-Multirank is to shed light on the strengths and weaknesses of each institution. Since the project is aimed at students, it’s not about finding the ultimate course of study, or absolutely superior instruction, but rather the program best adapted to one needs and interests. The site is, above all, a tool. You can carry out a personalized comparison between two schools of the same type, in a particular field, or limit the search to a few criteria.
Like any other ranking system, the purpose of U-Multirank is to list and to score universities, with the help of indicators: for example, the number of patents or post-docs, the international mobility of students, or the number of internships carried out in the region. The multidimensional approach, on the other hand, is new, its indicators concerning not only research, but reflecting the quality of teaching and the local involvement of a university, too. In the end, five areas were defined to cover these indicators—teaching and learning, research, knowledge transfer, international orientation and regional engagement—with scores from A to E attributed to each criterion. The interest is, thus, less in the ranking of universities, as in their comparison.
Although the user is invited to play with the tool, since its launch, the designers of this ranking with several dimensions have warned against misuse of the site’s data and particularly rail against the idea of creating a global ranking table by adding up the different scores.
Those who make league tables with our data run the risk of making serious methodological mistakes & misinforming their audiences. (2of2)
— U-Multirank (@UMultirank) May 16, 2014
The consortium behind the project is led by Professor Frank Ziegele of the Center for Higher Education (CHE) in Gütersloh, Germany and Professor Frans van Vught of the Center for Higher Education Policy Studies (CHEPS) at the University of Twente in the Netherlands. The initial reason for U-Multirank, supported to the amount of 2 million euros by the European Commission: there is no theoretical argument that justifies a definitive, unidimensional ranking, like the Shanghai Ranking, for example, which, for that matter, regularly ranks European universities poorly.
Innovative, surprising, responding to numerous criticisms of previous rankings, U-Multirank is, however, still not perfect. While the design of the internet site is fluid and colorful, actually using it takes some getting used to. The user is quickly submerged in the quantity of numbers and in criteria that are not always easy to understand (despite the existence of a glossary). This is the price to pay for a transparent tool: the user is not led by the hand. The possibilities are exhilarating, but the complexity can be off-putting. In this respect, one is also sorry to see that no translations have been done: understanding English is imperative. All in all, these problems are surmountable, on the condition that an effort be made to train people on the use of the tool, which could be the role of universities.
We attempted to compare the University of Paris-Sud with equivalent establishments. A lot of data is missing, especially regarding the quality of instruction.
The reliability of data, however, poses a more serious problem: in several cases, the information collected by U-Multirank is erroneous. The Masters graduation rate caught our attention: 11.74% for Agro ParisTech, for example, or even 6.8% for the University of Technology of Belfort-Montbéliard. Contacted by phone, these two institutions refuted these abnormal rates (spare a thought for the Medical University of Gdansk, in Poland, and its 0.64% of graduates). Of the 850 universities, a little more than 500 responded to the online questionnaire in order to submit certain information directly. For the others, the data was collected from indirect sources, which probably explains these errors (we have contacted the U-Multirank team and await their response.)
Update (26/05/2014) : The incorrect data for masters graduation rates has disappeared from the U-Multirank pages for Agro ParisTech and UTBM. This happened after the site was down for a few hours earlier this afternoon. The Medical University of Gdansk can still boast about its incredibly low graduation rate
The development and the methodology of the U-Multirank project are detailed by Frank Ziegele and Frans van Vught in Multidimensional Ranking, published in 2012. Their approach falls within a desire for transparence, in response to a remark often made about the statistical approach: numbers give the illusion of objectivity, but the choice to measure one dimension over another is necessarily subjective. The authors, thus, propose a participative approach to the evaluation of universities, in which the logic lies in involving the principal beneficiaries of the tool—university students—in the choice of indicators and data collection. For this to work, the parties first need to cooperate ahead of the project, by completing forms and responding to surveys. Once the product has been delivered, their feedback must be used to refine the previous criteria.
The missing or erroneous data are, thus, a call for more collaboration on the part of universities, in order to avoid the statistical slips and to yield an efficient, refined, reliable tool. Therefore, it is not in the nature of this ranking to be final, but rather to be constantly perfected. The next version of U-Multirank is planned for March 2015. Let’s wager that this multidimensional ranking will have found its audience and, thus, been improved.
