Picture credit: iStockPhoto
In April a trade publication reported that Ofcom planned to buy off-the-shelf handsets and use them to run a UK-wide drive and walk mobile data testing program. According to the reporter contacted by Ofcom, the proposed testing would examine “whether UK networks have achieved consistency” but would stop short of measuring “quality of service.”
Put simply, the planned tests would identify where in the UK you can get a signal from EE, Vodafone, O2 and 3 — but would reveal little about actual network performance. They would not tell you whether you can reliably load a web page, send an email or a tweet, or upload a photo to Instagram.
Would results from that kind of testing help consumers? Some argue that any testing done for consumers is beneficial, but I have reservations. Mobile network testing is difficult to do well, and the results are often hard to interpret. If testing methods or interpretation are flawed, the conclusions can be misleading.
How can testing itself go wrong? Using the Ofcom proposal as an example, imagine surveyors roaming areas, checking handsets for signal strength and noting available data networks such as 3G and 4G. Will every handset have identical hardware and software? Two devices of the same make and model can behave differently depending on firmware or operating system versions, distribution variants, or carrier-specific settings. How frequently will testers take measurements? What metrics will they record? How long will they remain in a location before deciding they understand the “consistency” there? Without rigorous, standardized methodology across all tests, the data collected can be anomalous or arbitrary.
Then there’s interpretation. Consider a recent report claiming to show mobile coverage on Britain’s roads and railways. The report came from a company that relies on crowd-sourced data: it aggregates spot measurements from the mobile devices of hundreds of thousands of users running its app, and uses that combined dataset to draw conclusions. The report stated how much time app users spent on 2G, on 3G/4G, or with no signal, and mainstream media picked up the headlines.
Those headlines — for example, that rail passengers “will have coverage” 72% of the time — were widely repeated without deeper scrutiny. No one asked whether a large share of the measurements came from idle devices, which would skew the results for several reasons, or whether the report included success rates for calls, drops, or data throughputs. In short, many readers did not realize that “coverage” does not equal the ability to complete basic tasks like making a call or sending a photo. Based on what I know about the methodology behind that dataset, and from my own experience traveling in the UK, connectivity on trains and roads is likely worse than the media headlines suggested.
Incorrectly collected or misinterpreted data harms consumers who need reliable information to push for network and service improvements — particularly when the published picture is rosier than reality.
By August, Ofcom published a report on voice calls in the UK drawing on a consumer poll, crowd-sourced data from a third party, and data supplied by operators. Like the earlier proposed mobile data testing, Ofcom’s voice report faces criticisms: it mixes methodologies, relies on potentially arbitrary crowd-sourced measurements taken outside a controlled environment, and includes operator-provided data that could present conflicts of interest. That mix makes the findings harder to evaluate and compare.
So what should Ofcom do to provide useful, trustworthy information for consumers? The answer is straightforward. Ofcom should commission regular, controlled mobile data and voice testing from an independent engineering firm with the capability to record and interpret layer-3 network signalling and other low-level network metrics. Such testing would capture the messages exchanged between the network and devices, offering reliable indicators of real-world performance rather than just presence of signal.
Ofcom should publish those findings and share them with the major operators. Operators are likely to welcome accurate, repeatable testing because transparent, reliable data benefits consumers and helps providers target and prioritize network improvements.
In April, Ofcom acknowledged the growing confusion caused by multiple, sometimes contradictory, network reports. Ofcom has the authority and the opportunity to reduce that confusion by commissioning rigorous, repeatable testing and making the results available. If it chooses to act, consumers and operators alike would gain a clearer, more useful picture of mobile network performance across the UK.