Ratings and rankings: Voodoo or Science?

Composite indicators aggregate a set of variables using weights which are
understood to reflect the variables’ importance in the index. In this paper we propose
to measure the importance of a given variable within existing composite indicators via
Karl Pearson’s ‘correlation ratio’; we call this measure ‘main effect’. Because socioeconomic
variables are heteroskedastic and correlated, (relative) nominal weights are
hardly ever found to match (relative) main effects; we propose to summarize their
discrepancy with a divergence measure. We further discuss to what extent the mapping
from nominal weights to main effects can be inverted. This analysis is applied to
five composite indicators, including the Human Development Index and two popular
league tables of university performance. It is found that in many cases the declared
importance of single indicators and their main effect are very different, and that the
data correlation structure often prevents developers from obtaining the stated importance,
even when modifying the nominal weights in the set of nonnegative numbers
with unit sum.