Customers Shocked By Level 3's Internet Disruption

Level 3 Communications said it warned Cogent Communications before pulling the plug on parts of the Internet last Thursday. But not everyone received an alert.

Partners of the two Internet communication companies said they and their customers were taken completely by surprise when Level 3 cut off the free peering connection between it and Cogent, which severed access to tens of millions of IP addresses. Peering is a common practice that Internet backbones such as Level 3 and Cogent use to give customers the simplest access to all of the Internet.

Level 3 has temporarily restored the peering connection, though it remains in a heated spat with Cogent over shared bandwidth usage.

Tim Bradley, senior vice president of VoIP at New Global Telecom, a Level 3 master agent in Golden, Colo., said the sudden shutdown of major parts of the Internet caught him and his customers off-guard. "We were given no prior warning. We learned about it from a sudden burst of sporadic customer calls," he said.

The lack of advance notice about the shutdown from Level 3 didn’t please Bradley. Customers calling New Global Telecom about the service interruption expected the company to know what was going on, and customer service staff had to scramble to stay on top of things, he noted. "We very calmly and very coolly took care of everyone," he said.

Such an abrupt Internet shutdown isn’t very likely, so building in redundancy for such an event--be it adding a failover network via a second Internet provider loop or a firewall that redirects IP traffic--isn’t something that’s high on anyone’s checklist, according to Bradley. In fact, most of New Global's business customers just happened to have both a Cogent loop and a second Internet provider loop, "so the impact was minimized," he said.

Yet one of Cogent's largest customers wasn’t so lucky, said Quy "Q" Nguyen, founder and CEO of Allyance Communications Networks, Irvine, Calif.

The customer, an online gaming company that has needed to increase its bandwidth monthly, saw its Internet traffic drop at least 25 percent when Level 3 pulled the plug on Cogent, said Nguyen. And the gaming company had no failover in place. "They were just down," Nguyen said.

Another Cogent customer, which lost 50 percent of its Internet traffic when Level 3 cut off Cogent, was quickly convinced by Level 3 to leave Cogent and sign a one-year contract, said Nguyen. "Then, only minutes after they signed the contract, [Level 3] announced they would turn back on the peering connection. Now they're stuck under a one-year contract at $25 per [megabyte], when they were paying $10 per meg with Cogent. They told me as soon as the contract is up, they are leaving Level 3," he said.

Sureel Choksi, executive vice president at Broomfield, Colo.-based Level 3, said in a statement that there was nothing sudden about Level 3 cutting Cogent's peering connection. At least three times over the past three months, Level 3 warned Cogent that it would take such action if the two companies couldn’t negotiate "alternative commercial terms to allow the continued exchange of traffic," Choksi said. Choksi claimed that Cogent was running more traffic over Level 3’s network than vice versa, an imbalance that was costing Level 3 money in terms of maintaining the bandwidth eaten up by Cogent.

"It is important to keep in mind that traffic received by Level 3 in a peering relationship must be moved across Level 3's network at considerable expense," Choksi said in the statement. "Simply put, this means that, without paying, Cogent was using far more of Level 3's network, far more of the time, than the reverse. Following our review, we decided that it was unfair for us to be subsidizing Cogent's business."

Level 3 last Friday said it would again shut down the Cogent peer connection if the company fails to make certain changes by 6:00 a.m. EST on Nov. 9. Choksi’s statement didn’t specify the necessary changes, and a Level 3 spokeswoman declined to comment further.

Meanwhile, Cogent has begun offering Level 3 customers one year of free Internet access if they leave Level 3. The wording of the Cogent offer, delivered in a statement from Cogent CEO Dave Schaeffer, was less-than-gracious to Level 3. "Cogent feels that Internet users deserve better treatment while this situation is ironed out. That is why Cogent is appealing directly to Level 3 single-homed customers and offering them one year of free, dedicated Internet access to provide them with access to the full Internet, not just the Level 3 portion of it," Schaeffer said in the statement.

The increase in Cogent traffic that Level 3 complained about stems in part from Cogent’s acquisitions of other Internet backbones and their peering traffic, said Jeff Henriksen, director of marketing communications at Washington-based Cogent. He said Cogent thought Level 3’s warnings leading up to last week' outage were requests to update the peering contracts of networks Cogent had acquired.

"We thought it was more a maintenance issue, clearing up old contracts--house cleaning," Henriksen said. Cogent has 425 other peer networks, he said.

Tom Snyder, COO of Xantrion, an Oakland, Calif.-based consultant and MSP, said Level 3’s action caused as many as 45,000,000 IP addresses to become unreachable. Xantrion customers running SonicWall firewalls, however, were spared any long-term effects of the “de-peering” thanks to failover Internet features in those products, he added.

Xantrion was able to remotely reprogram the SonicWall firewalls in customer networks to take alternative paths through the Internet, quickly restoring full access, Snyder said. Like New Global’s Bradley, Snyder thinks awareness of such Internet failover techniques is limited.

"Companies servicing the small- to midsize-business market are, I think, just becoming aware of these new tools for failover and redundancy,” Snyder said. “I base this on the fact that most of our new customers are transfers from other service providers in our market, and none of them have had failover before we put it in. Also, none of the engineers we have hired have been aware of this technology. So awareness is one issue." The cost of Internet failover has dropped significantly in the last three years, he added.