This article first appeared in Forum, The Edge Malaysia Weekly on December 25, 2023 - January 7, 2024

On Dec 5, 2023, the Programme for International Student Assessment (PISA) scores of 77 countries were released worldwide. PISA is an international study started in 2000 and owned by the Organisation of Economic Co-operation and Development (OECD), a club of 38 high-income countries. Every three years, PISA/OECD randomly selects a group of 15-year-olds to take tests in three domains — mathematics, reading and science. In 2022, 7,069 Malaysian students from 199 Malaysian secondary schools took the PISA.

The results are sobering in three main ways. One, since 2008, Malaysia has been consistently below the OECD average in all three domains of maths, reading and science. Two, although the 2022 PISA scores declined on average versus 2018, Malaysia declined sharper in all three domains compared with most countries. And three, 15-year-old Malaysians in 2022 did worse than 15-year-old Malaysians in 2012 in all three domains, reversing the gains from 2012 to 2018. For comparison, five Asian cities/countries had higher scores in 2022 versus 2018 in at least one domain: Hong Kong, Japan, Singapore, Taiwan and South Korea.

This author is not an educationist, so cannot analyse the reasons, mitigating factors and solutions to Malaysia’s decline or the strengths and weaknesses of PISA’s methodology. However, there are many lessons from PISA that we can apply to health systems in general, including Malaysia’s.

The first lesson is that there should be a measurement framework for health system performance, no matter how imperfect such a measurement framework is. There are many examples of measurement frameworks, starting with the under-publicised Universal Health Coverage Service Coverage Index (UHC SCI) maintained by the World Health Organization (WHO).

The UHC SCI is a composite of 14 tracer indicators. In the SCI, Malaysia has a score of 76/100 against a global average score of 68, ranking the country at No 50 worldwide. For context, Canada has the highest score (89), the top 25 countries have scores 83, and Malaysia is the world’s 55th richest country measured by GDP/capita on a purchasing power parity (PPP) basis. This context is important, because one number or statistic alone is misleading if interpreted in a vacuum.

There are other measurement frameworks: The International Health Regulations core capacity score (measuring pandemic preparedness and maintained by the WHO); the Global Health Security Index (maintained at the Johns Hopkins University, which ironically ranked the US at No 1 in 2019 before the US had 1.4 million excess deaths during Covid in 2020 to 2023); and the Commonwealth Fund scorecards. And then there are the “commercial rankings” offered by various magazines or companies that bravely declare rankings like “Best Healthcare in the World 2023” without robust methodology. This author believes that these bold commercially motivated rankings should not be used to endorse any country’s health system at all.

A famous phrase about statistics is, “There are three types of untruths: lies, damned lies, and statistics”. In other words, any measurement framework, ranking or statistical model can be interpreted to tell almost any story. For example, Malaysia’s decline in PISA scores (2022 versus 2018) looks acceptable because of the overall average decline in PISA scores, but does not look acceptable because of our sharper decline compared with most countries and in the light of the improvements by some Asian neighbours.

To borrow another famous phrase on statistics, “All rankings are wrong, but some rankings are useful”. In other words, a standardised measurement framework will provide a country ranking, which is useful to mobilise political support for important health reforms or to identify which projects should “start, continue or stop” in different health subsectors. To do that, health leaders must avoid focusing on only one number, even if that number is a composite number of 14 tracer indicators.

Therefore, because all frameworks and rankings have nuances, they cannot be so quickly or superficially accepted as either good news or bad news. Rankings may hide weaknesses amid strengths (like the US No 1 pre-Covid ranking for health security being exposed during Covid), and hide strengths amid weaknesses. The nuances of frameworks and rankings are especially important in healthcare or education, which are complex systems that cannot be reduced into a single, neat score or ranking.

Scores and rankings frequently elicit emotional responses. A high ranking may elicit (self-)satisfaction, happiness and pride. A low ranking may elicit dissatisfaction, defensiveness manifesting as excuses or blaming others, or frustration/anger manifesting as criticism of the ranking organisation being biased or using the wrong methodology. Examples include the annual positive-or-negative responses to the global university rankings or commercially motivated healthcare system rankings. These emotional responses distract from the real work that must be done.

All rankings are wrong, but some rankings are useful. In other words, a calmer and more constructive response to rankings will help leaders, policymakers and decision-makers focus on the real work to identify parts of the system that are working well (“continue the plan”), parts that need major improvement (“replace the plan”) and parts that need minor improvement (“tweak the plan”). A single ranking can never fully represent a complex system like health or education, and do not deserve simplistic emotional responses.

Malaysia is the 55th richest country in the world, out of the 193 member states of the United Nations. That already puts us in the 28th percentile of countries, which means that Malaysia should benchmark our health system performance against the 10th percentile, for example, and not against the mean or median performance.

If given enough time, all large organisations have bureaucratic inertia, path dependencies and a regression to the mean. Those three terms are well studied in the fields of economics, public administration and management, and collectively mean that “organisations will find the path of least resistance”. In other words, it is easy for any country to say that they are “slightly above average”, and stay contented and complacent. Malaysia must resist that temptation and must benchmark against the best, not against the average.

Health is so complex that it cannot be reduced to the Ministry of Health (MoH) alone. For example, the Ministry of Higher Education (MoHE) trains doctors, Public Service Division policies help to retain doctors in the public service, and the Ministry of Human Resources determines the employment laws that protect employee health. In other words, healthcare rankings depend on all-of-government and all-of-society contributions.

Similarly, education attainment and rankings depend on non-Ministry of Education (MoE) contributions, like high parental involvement, anti-bullying efforts from the police, and 5G efforts from the Ministry of Digital to provide rural coverage for online learning.

All of the above require good political choices. Yes, more money to MoH and MoE will help, without increasing our national debt and with strong accountability for that additional spending. Yes, more doctors and more teachers will help, without compromising other nation-building professions like engineers, entrepreneurs and scientists. But good political choices are the most important success factor to increase health rankings.

Rankings are important to human society for several reasons. One, humans have a deep-seated psychological need to understand the world by using rankings, categories, sequences and taxonomies. Two, humans are competitive as a species, and rankings allow us to measure against each other. Three, rankings allow human societies to allocate resources, attention and efforts, in a continuous improvement effort.

Although all rankings are imperfect (and therefore “wrong”), some rankings are useful. Therefore, the public policy maker would be best served with a small measure of calmness, wisdom and judgement to make the best use of rankings.

Dr Khor Swee Kheng is CEO of Angsana Health. He specialises in health systems.

Save by subscribing to us for your print and/or digital copy.

P/S: The Edge is also available on Apple's App Store and Android's Google Play.