One of the more tentative findings in the UK Cyber Security Breaches Survey 2017 was played down when the report was published this week.

This was the average cost of a cyber breach to large businesses, which is just over half the figure recorded in a comparable survey (by the same organisations) in late 2015-16.

The authors, Ipsos MORI and Portsmouth University, do not in fact mention the previous year’s figure, which was £36,500. Instead they just say that the absolute difference between the two figures was large but was found to be “not statistically significant at the 0.05 level, based on a t-test”.

This does not mean there was no significant difference, merely that none could be demonstrated. The total number of large firms in the sample that reported breaches was around 100 although only 50 reported any associated costs.

As the sample was small and the variance probably large, the t-test could well have found no significant difference even if one existed. Data breaches are precisely the kind of phenomenon that can give rise to extremely varied costs, from nothing at all in most cases, to millions (or in Yahoo’s case) hundreds of millions of dollars.

All of which means that even if researchers can obtain reliable estimates for the costs of cyber security failures they may yet struggle to show any relationship between those costs and security spending.

Doubt must surround the cost estimates in these surveys. Commenting on the 2017 survey this week one analyst pointed out that its estimate of the average cost of a cyber breach is a fraction of other figures doing the rounds.

In March a study from Opinium, commissioned by ISP Beaming, estimated that breaches have been costing an average of £10,000 per incident for all business sizes. Another study from the Ponemon Institute in 2016 put the consolidated cost of a ‘material’ breach where sensitive, protected or confidential data were lost or stolen at $4 million.

By contrast this latest survey, in which most of the incidents did not result in a material loss, arrives at an average of just £1,570 per breach (for all firms) rising to £19,600 for large firms. Respondents were invited to consider all costs including loss of revenue from downtime, lost assets, recovery and long term costs.

Another finding is that only 6% of respondents said they had anything in place to monitor the cost of cyber security incidents, prompting doubts as to how thoroughly the costs were estimated.

Also doubtful in the 2017 survey results are the responses to questions about how breaches were identified. Very few respondents reported that breaches were discovered by people outside the firm, and around 80% were said to have been identified within 24 hours.

Yet Oliver Pinson-Roxburgh a cyber security expert with Alert Logic told Retail Risk News in February that data breaches in the retail sector are taking an average of 207 days to detect, and detection is normally made by a bank. This can be because criminals do not take advantage of breaches immediately, he said, but also reflects a general lack of security awareness in most retail businesses.

The latter point at least is borne out by the survey. But since the survey method did not allow for anonymity, many of the other findings must be treated, as the authors suggest, with caution.

One thing is clear: the cost of data breaches is set to rise in 2018 when the EU General Data Protection regulation comes into force, at which point penalties could amount to four percent of annual turnover.