A person could be forgiven for being as confused as a cow at Nkandla when reviewing the MOST Awards results this year. How can a company that has achieved a higher average MOST Awards score be ranked lower than a company that has achieved a lower score?
One of the challenges that faced the MOST Awards methodology in recent years has been the “narrowing of the gap” between companies with respect to their criteria scores. This was especially pervasive amongst the top performers where evidently it was difficult for respondents to discern differences in performance. The result was that the variances in MOST Award scores between some of the top companies in certain categories in recent years have been very small.
The crux of the matter is that in research one most often draws a sample of a population in order to be able to draw inferences into the perceptions and/or behaviours of that population. The implication of sample-based research is that there is always the probability that the behaviour of the sample is not 100% consistent with the behaviour of the whole population. This ‘probability of variance’ between the behaviour of the sample and the behaviour of the population is commonly termed the ‘margin of error’.
In layman’s terms from a MOST Awards perspective, this means that there is a probability that the final score achieved for a company might be a little higher or a little lower than the score calculated from the sample collected (due to the margin of error). So, from a research perspective, the fact that the final MOST Awards score for a company is marginally higher than that for another company is not necessarily a conclusive result (because the scores for both companies may be little higher or a little lower, according to their respective margins of error).
In order to discern with confidence whether a result is conclusive, a statistical method called a ‘significance test’ is often conducted. In 2014, significance testing was conducted on all of the MOST Awards data. A significance test determines whether the variance between two results is statistically significant or insignificant. Simply, the test takes into account margins of error between MOST Awards scores to determine with a high level of confidence whether the difference between the scores is ‘valid’ or conversely, statistically insignificant.
In the latter instance, taking into account margins of error between sample-based research data, MOST Awards scores between companies that are statistically insignificant, are technically and statistically ‘the same’. In other words, they are ‘technically tied’.
Every year, Wag The Dog Publishers, publisher of the MOST Awards survey, hosts an informal focus group discussion with Industry stakeholders to review the MOST Awards methodology, the relevance of the performance criteria to be measured and to discuss any other relevant issues. In 2014, the challenge of diminishing variances between MOST Awards scores was discussed. It was agreed that a methodology should be sought to yield greater variance between the MOST Awards scores so that the rankings of companies could be discerned with absolute confidence.
To this end, the MOST Awards research partner, Freshly Ground Insights, consulted with and commissioned assistance from an independent third party analytics specialist, Consulta. A primary outcome of this consultation was that the scoring methodology for the MOST Awards was adjusted (where required) in 2014. However, it was anticipated that the possibility of achieving ‘technical ties’ between companies still existed.
In order to break the tie in such instances, MOST Awards voters were asked to nominate the company that they deemed to be the “overall best performer in the context of the MOST Awards” in each category that they scored. The number of “best-in-category” votes were counted for each company and in the event of a technical tie, used as the metric to break the tie. A company achieving a significantly higher number of “best in category” votes than another company would be positioned as the higher of the two in the ranking.
As a final measure to maintain the validity and integrity of the MOST Awards results, the survey method and data analytics were audited by the analytics department at the University of Pretoria, and given a clear rating.
A contributing factor to the success of the MOST Awards since its inception six years ago is surely due to its principles of a robust research method, a transparent approach towards improvement, and ongoing relevance.
IMAGE: Gordon Muller (red beret) interviews FGI’s Brad Aigner about MOST methodology
