Behind the numbers: revealing a better story, July 6th, Royal Statistical Society
David Walker, director of the RSS’s getstats statistical literacy campaign, opened the meeting by relating reliable reporting of numbers and statistics to the current wider debate about accuracy and veracity in journalism. He also commented that interrogating data can now make and lead the news in a way unlike previous ages.
Richard Knight and Richard Vadon, More or Less (BBC Radio 4)
(audio unfortunately cuts out between around 5:30 and 9:00)
First up, Richard Vadon, editor, and Richard Knight, producer, of Radio 4’s More or Less gave their three lessons for interrogating numbers, illustrated with stories they have covered on the programme. Their first lesson – asking “is it a big number?” – focused on the story that 584 women became pregnant while using the contraceptive implant Implanon. They put this in context by calculating the failure rate for the 1.4 million women using the implant, which in fact compares very favourably to other methods of contraception and to vasectomy. But the media made it into a scandal because it had seemingly big numbers and all the elements of a good story.
The second lesson was distinguishing correlation and causation. Here they looked at calls for a ban on alcohol advertising, and in particular a claim that young people’s knowledge of alcohol brands is linked with increased alcohol consumption. But in fact, couldn’t the causation be the other way round, with those who drink more being more likely to know about brands? They also handed out a hoax press release put out by mathematician Matt Parker showing a strong link between the number of mobile phone masts in an area and the number of babies born there.
Their third and final lesson was to go beyond the press release and be sceptical of claims, even those made by good causes. It’s not just politicians, but also charities that can be guilty of distorting numbers to fit the narrative they want to promote and just because it’s a good cause doesn’t mean journalists should put aside their analytical skills.
Nigel Hawkes, Straight Statistics
Nigel Hawkes runs the Straight Statistics campaign which attempts to shame people who misuse statistics, and said his main message would be that journalists need to be more careful about uncritically cutting and pasting data. Although the natural instinct of a journalist is to listen carefully and identify exaggeration, numbers tend to slip by unexamined.
Survey data is a common source of problems, through a failure to examine how the survey was carried out, who the respondents were, how they were selected and what the results actually mean. Nigel gave an example of a recent survey on favourite foods around the world. The problem was that the categories didn’t make sense: “Italian”, “pizza”, “pasta”, “lasagne” and “spaghetti” all appeared separately, and in Russia the 9th most popular choice was “delicious”.
Another example (also picked up by Ben Goldacre) was a press release by the Department of Communities and Local Government claiming that improvements in council procurement could save £450 per household. This was based on a tenuous extrapolation from three items of spending by three councils.
Nigel explained how he debunked a claim that knife injuries cost NHS Scotland £500m per year. He looked up publicly available statistics on hospital admissions for knife injuries, average length of stay and cost per night in hospital to calculate the inpatient cost as less than £1m. He also put the figures in context of an £11bn total budget for NHS Scotland and an academic study which put the cost of knife crime to the NHS in England (which has ten times the population) at £3m per year.
These examples make the point that by doing some digging you can often uncover a better story than by accepting what you are told, and that the internet provides the means. However, the Office of National Statistics website is difficult to use and you may be better off googling the term you are interested in with ‘ONS’ to get to the right page. Nigel also looked at the problems in making comparisons over time when the methodology for collecting statistics may have changed.
Finally he pointed out that without having an advanced knowledge of maths or statistics, by asking whether the numbers make sense, journalists can probe even complex medical studies. He looked at a study which claimed that consumption of sweets by children was associated with criminal violence later in life. Although the study followed 17,000 children, the conclusions were drawn from only 33 criminal convictions, and there are many possible confounding factors.
Rob Cook, Bazian
Rob Cook works for Bazian, a small company which analyses the evidence behind health stories for the NHS Choices website. They cover two stories a day, chosen by the NHS, and Rob explained their methodology and some of the things they look out for.
In particular, it’s important to ask where the story came from. Studies published in peer reviewed journals are more likely to be trustworthy than preliminary research presented at a conference. They also look for possible conflicts of interest. Rob showed the results of a study of 200 press releases that were covered by the media. 40% were based on evidence classified as ‘weak’, for example because of small sample sizes, and 12% were based on unpublished research. Two thirds of the press releases based on animal research said the results would translate to humans, but in fact two thirds of animal studies do not.
Most of the stories they look at are about the cause of diseases, new treatments or a new test. Rob briefly described how Bazian appraise the design and quality of the original stud, looking at a study showing that coffee cuts the risk of prostate cancer by 60%. Questions they look at include how participants were recruited, whether they are representative of the wider population, how they were allocated to exposure and control groups and whether they remain in the same groups over time (e.g. did their coffee consumption stay the same for 20 years?)
Finally he looked at the way results are reported, in particular whether relative risk or absolute risk is reported. In one study, it was reported that a drug for melanoma reduced the number of deaths after 6 months by 56%, but after 10 months only two people of the 700 in the study were still alive.
Patrick Casey, Full Fact
The final talk was from Patrick Casey, a writer for Full Fact, which campaigns for more accurate information to be used in politics and the media, mainly through fact checking and asking for mistakes to be corrected. He pointed out that you don’t need to be a statistical expert to find blatant inaccuracies – often you can just follow some very simple maxims. The first thing is to get to the original source and not be scared by figures.
Patrick gave an example of a story in the Daily Express headlined “House prices set to surge” based on a press release by the Royal Institute of Chartered Surveyors. But by going back to the press release, it was easy to see mistakes and misquotes in the story. Another example was the reporting of residents of a housing estate in Merthyr Tydfil having a life expectancy of 58, lower than Haiti and Iraq. It turned out the figure was actually for healthy life expectancy and Full Fact got the papers involved to print corrections.
He pointed out how false claims can be propagated between think tanks, charities, the media and politicians, with everyone assuming that someone else is checking them. In fact it’s not that hard to get to the original data – common sources are ONS bulletins, government publications (though the press offices don’t seem to like journalists talking to their statisticians), freedom of information requests, parliamentary answers and House of Commons library research. Getting hold of a statistician to discuss the data with can really help.
He concluded that just by going back to the original report you can often see mistakes, and begin to get used to where data comes from and its potential misuses. Don’t just assume someone else had done the checking because often they haven’t.
