An algorithm dunnit: anatomy of Spark's mid-match surrender

By Pattrick Smellie

Sept. 23 (BusinessDesk) - It had all been going so well.

Friday night's Rugby World Cup opening ceremony and first match, Japan vs Russia, rolled down the international telecommunications cables to New Zealand so seamlessly for Spark Sport that freshly departed Spark chief executive Simon Moutter couldn't contain himself.

Tweeting at just after midnight, he had a crack at the media for being "clearly disappointed that Spark Sport delivered superbly tonight" and was still trying to "find a few customers who had issues."

At the coalface were some 200 staff on the Spark Sport help team - expanded from the usual dozen or so - fielding the predictable flood of calls from customers with last minute trouble setting up their livestreams at home.

A team of senior Spark executives, including CEO Jolie Hodson, had "eyes on glass" in a situation room, monitoring with technical teams a bewildering number of devices from high-end TVs to tablets and smartphones, taking the feed from Tokyo.

"We were rapt with how the platform performed" on Friday, says corporate affairs manager Andrew Pirie. Of course, the Spark team had "kick-arse" connectivity, unlike the many customers who were just discovering that fibre to their front porch won't make the new TV in the lounge play the rugger if the two are connected by a weak wi-fi signal.

"Hundreds of thousands of New Zealanders watch Netflix and Lightbox every night," and streaming TV is second nature to them, says Pirie."But there's hundreds of thousands who have never experienced Netflix and who need to get into the world of streaming to watch their beloved rugby."

Spark's willingness to cop flak for forcing irritated punters to buy a Spark Sport subscription and risk the kind of failure that occurred during the All Blacks vs Springboks match on Saturday evening boils down to this simple calculus: having all those newbies switched on to livestreaming will be good for business in the long run.

By 6pm Saturday, customers had logged 10,000 set-up queries just that day by phone, online chat and social media. Three hours before kick-off in Yokohama stadium, new subscriptions were still selling at a rate of 90 a minute. By mid-evening, the Spark executives were daring to hope that the first really big game of the tournament for Kiwi fans would go off without a hitch.

The Australia vs Fiji game had gone well, likewise Argentina vs France.

"There was pressure on the customer care channels, but from a technical point of view, we were prepared for that level of capacity. The audience was building nicely," says Pirie.

But Spark's kick-arse connection at HQ also meant Hodson, Pirie and the other senior executives in the room could see immediately when the picture began to play up, early in the first half of the All Blacks/Springboks clash.

"About 10 minutes in, there was a bit of a blip in the bit rates," he said. A few anxious moments later, the feed seemed to come right.

"Five minutes later, it happened again."

There was concern but again, things seemed to come right.

"Then it happened again," says Pirie, and everyone knew they had to take action. If they were seeing a few blips on their excellent connection, sure as eggs there would be customers seeing far worse effects.

"People were starting to huddle around the screens and we were getting onto our content partners asking what was happening."

Watching the game at home and with no apparent problems was Spark's general counsel, Melissa Anastasiou, who was surprised to get a call saying she needed to come into the office.

However, the call centres were getting hit with a spike in complaints, with some customers apoplectic to be missing parts of the game.

"The real concern we had was that the symptoms customers might have seen were very similar to the symptoms they will see with poor in-home connectivity," says Pirie.

Spark's contingency plan was to make judgements based on three key factors: numbers of customers affected; how bad it was; and context.

That last factor - context - was crucial. As far as Spark could tell, perhaps 10 percent of all viewers would be seeing a "degraded" livestream. While that wasn't great, the numbers seeing pixellation or the buffering 'wheel' would be far smaller again.

However, this was the All Blacks vs the Springboks in the first round of the Rugby World Cup. The context dictated that emergency measures were required.

"If this had happened in another game, would we have made the same decision? Depending on the game, we might not have," Pirie says.

By this time, word had reached the CEO at Spark Sport's American streaming partner, Akamai, and the Beehive was being kept in the loop. Yet there was still no technical issue with the livestream itself and plenty of capacity in the New Zealand broadband system.

However, with half time approaching, Pirie says it was a case of "getting the right people in the room and achieving a quick consensus" that there was no choice but to switch to live broadcast on TVNZ's Duke channel. At TVNZ, a stand-by team flicked the switch and, ironically, Duke was broadcasting the game before Spark had a chance to notify customers.

A flaw in contingency planning meant a banner on the Spark Sport livestream telling viewers they could switch to Duke was delayed. That was largely, it seems, because Spark had planned for its worst nightmare: a complete service failure with nothing on the screen at all, rather than the halfway house that befell it on Saturday.

While some customers ask why Spark didn't text all subscribers, the reality is that many would only have received the text long after the final whistle.

By Sunday morning, a post mortem had determined where the problem lay: it was in the way livestream had been allocated to various internet 'pathways' that brought the game to New Zealand.

While there was more than enough capacity across the whole system, an automated process caused bottlenecks by directing more of the rugby content into one of those pathways. The issue was exacerbated by heavy mid-Saturday evening demand from the rest of New Zealand who weren't watching the rugby, but were watching something on, say, Netflix.

"Algorithms did this," says Pirie, a little haplessly. Every other part of the process was working exactly as it should and that there was no technical reason to prevent everyone being able to watch whatever they liked, without interruption.

That bottleneck has now, Spark believes, been manually unpicked. It should not be possible for the exact same problem to happen again. However, no one can be sure until the same level of demand hits the New Zealand broadband network again.

To be on the safe side, all games through Sunday were broadcast on TVNZ Duke as well.

"We're bracing ourselves for tonight for going back to Spark Sport only," says Pirie, who knows some customers are still having trouble with their set-up at home and that any further failure will gain maximum publicity.

As to the refunds it's offering, Spark knows some people will claim their money back even if their service was OK. A "surprising" number bought a one-match pass for Saturday's game and they can claim a refund, no questions asked, if they say the quality was lacking and notify Spark before next Sunday night.

But, in the end, the company has no choice but to be seen to be doing the right thing, says Pirie.

"We accept that with the Rugby World Cup, you've got no choice, so we are giving people the option to opt-out if they want to."

© Scoop Media