The Network Backbone - Not Just an IP Highway

Posted: 03/1999

The Network Backbone - Not Just an IP Highway
By Charlotte Wolter

Internet protocol (IP) architecture in the backbones of large long distance carriers is a new phenomenon in telecom. As recently as two years ago, most network providers had no IP in their backbone, except as traffic carried along on another transport technology.

Today, big carriers are putting raw IP directly onto their optical networks as IP over synchronous optical network (SONET), IP over dense wave-division multiplexing (DWDM) or even packet over optical. IP has been tapped as the "converged" network technology of tomorrow: a single economical network architecture that will carry any kind of traffic and provide any level of quality of service (QoS) the traffic demands.

But is IP up to the task? Clearly, it has gained increasing favor for a number of applications. Internet service providers (ISPs) and the network providers who service them are using more and more pure IP backbone.

It also has spawned one of the more acrimonious debates among telecom technologists, as IP proponents have squared off against those backing another newer technology, asynchronous transfer mode (ATM), as the king of the backbone services.

In 1998, IP was in the media spotlight, but ATM led the way in new users. "The story that grabbed the headlines in the trade press was the ascendance of IP, and ATM is supposedly yesterday's news and a corpse by road," says John Ryan, chief analyst, Ryan Hankin Kent Inc. (RHK), San Francisco. "In fact it was a terrific year for ATM and will be again this year."

In the real world where network providers must serve customers and make money, both technologies are deployed--IP not as widely as ATM, but growing. And they have to share the spotlight with other legacy technologies, notably frame relay, that have provided very effective solutions to particular customer needs.

For the service providers, the real-world demands of customers have dictated that they come up with their own solutions to the question of which will be the converged network of tomorrow. For carriers, the overwhelming requirement has been that they must be able to carry a great variety of traffic, from time division multiplexing (TDM) voice to IP. No matter what their predominant backbone strategy, they have to make accommodation for this diversity in their payloads.

In the ATM-dominant networks, the strategy is to encapsulate everything in ATM packets and simply compensate for the increased overhead with more bandwidth. ATM has the ability to carry almost any traffic, but the packet structure adds about 10 percent in overhead.

"IP is a send-and-pray method of transmission. But IP will get there, and the reason is because AT&T and other carriers will begin to provide virtual private networks using IP. As we get more security and screening functions and can guarantee capacity in IP, we will see more and more customers use IP." --Dan Sheinbein, vice president, network architecture and development, AT&T Corp.

Another strategy has been to build several parallel networks. A wide range of operators has used this. IXC Communications Inc., Austin, Texas, added its new all-IP "hierarchical-meshed" network beside an existing SONET-based network. Other operators, such as AT&T Corp., have IP, ATM and frame relay running simultaneously.

QoS demands expected from customers also is a critical factor in choice of technology. Where there is expected to be a demand for mixtures of voice, video and data--as with Sprint Corp.--ATM is a technology of choice. As in networks designed to service ISPs, where data represents the dominant feature and voice is an add-on, IP is the dominant technology.

Private networks for intranets or virtual private networks (VPNs) are predominantly IP-based now be-cause they carry mostly data. However, as corporations gradually integrate voice with data in their enterprise networks, they also will be carrying voice over IP (VoIP).

This does not necessarily mean that these networks will migrate to ATM for QoS. Being private, they can tightly control use of the bandwidth to keep the quality up for VoIP. Therefore, it is considered likely that they will remain IP-based, even as voice becomes a larger part of their traffic mix.

One overall trend that will favor IP-based networks, experts say, is the burgeoning growth of data on all networks. By 2005, according to several estimates, data will outweigh voice by a factor of 30 to one on backbone networks, a powerful force that is expected to bring IP into predominance.

Sprint on Cruise Control with ATM

As network operators have made their choices, some interesting solutions to the question have emerged. Sprint has been one of the most vocal proponents of all-ATM backbones, and is well along in its deployment of services. "We have technologies that we are in the process of deploying--one in particular that runs voice over ATM that allows us to talk to the PSTN (public switched telephone network)," says Charles Fleckenstein, technology spokesperson. "IP at this point, instead of being a transport mechanism, is still seen as an application. We don't see IP on the backbone, but consumers would see IP in the home."

Sprint's arguments for ATM are flexibility and quality. "What we can do with ATM is, it can roll up IP, frame relay, voice and video, and run them all over a single ATM backbone," Fleckenstein says. "ATM is the only technology that gives the quality of service we are looking for."

"IP at this point, instead of being a transport mechanism, is still seen as an application. We don't see IP on the backbone, but consumers would see IP in the home." --Charles Fleckenstein, technology spokesman, Sprint Corp.

One of the efficiencies that Sprint hopes to introduce in its long-term vision of its nine-month-old Integrated On-Demand Network (ION), is an element of subscriber management and provisioning of their own services, even for consumers. Each subscriber premises, home or business, is equipped with a hub that talks to a service node on the edge of the network. Services are provisioned on the service node. A customer chooses a service, such as conference call, and the hub talks to the service node, which automatically dedicates bandwidth to the user. If it is a voice call with highest QoS requirements, the network will automatically allocate 64 kilobits per second (kbps); if it is asymmetrical digital subscriber line (ADSL), as much as 6 megabits per second (mbps) will be dedicated to the service. "With ION the idea was customers are never turned off, like electricity," Fleckenstein says.

Ryan says it is true that Sprint is making great efforts to use ATM "as a sort of convergence technology so they can carry voice and frame relay and ATM and X.25 in a single managed infrastructure. That is a lot of what they are trying to do with the ION network."

Beth Gage, a Tulsa, Okla.-based analyst with TeleChoice Inc., Owasso, Okla., says, "For IP-type services it would be an advantage in not losing overhead, not adding latency in processing times. You do add latency when translating IP to ATM and frame relay back to IP. If you are in a pure IP environment, then you don't lose anything to that translation."

However, at the company's own ISP, the Sprint Link Group, the backbone technology is packet over SONET. "So, if you look at their network, they have IP routers, and the output goes directly into a SONET terminal without passing through a Layer 2 box--ATM or frame relay," Ryan says.

"So what does that prove? What it proves is that at this particular juncture the attempt to create a single converged network is extremely premature. Instead, it makes senses to separate the needs of pure IP networks from other networks."

Qwest Rides IP

In apparent contrast to Sprint--but very much like the architecture of the Sprint ISP--is Qwest Communications International Inc., Denver, Colo., which is building a nationwide backbone of IP over SONET with OC-48 capacity. About half of the company's intended mileage of 18,500 miles is "lit," with the rest planned to be in operation by June 1999. A number of customers are connected already, including Netscape Communica-tions Corp., Mountain View, Calif., which has OC-12 capacity. Qwest recently announced it had reached 1.5 million miles of OC-3 (155mbps) capacity. The network also includes some capacity of OC-192 using IP over DWDM.

The company says the network has one-to-one redundancy, with 50-millisecond-to-200-millisecond restore times and extended switching down to the router, so customers are not aware of downtimes.

"Due to these capabilities, we are able to provide the same quality of service and uptime that customers are used to in the old telephony world," says Vab Goel, Qwest's director of IP network engineering, "but at a cheaper cost with more flexibility for leased types of capacity and phone."

The services include IP data, VoIP (launched late in 1998) multicast-enabled video for videoconferencing, and the network can carry frame relay and ATM for private-line services. "We have thrown a lot of capacity on the backbone. We did not build it just for web traffic," Goel says. "We are not offering it like an ISP. It is not a backbone; it is an infrastructure."

Goel says the trend in private networks is that companies are going to build them over public networks using VPN technology. "VPN is going to be the key feature set offered on our backbone, and there is a tremendous amount of demand in the marketplace for companies to move out of their legacy frame relay or ATM networks and get onto VPN so they can take advantage of the flexibility of the IP protocol," he explains.

Qwest has parallel nationwide frame relay and ATM networks for those customers still concerned with getting value out of their legacy equipment. Its parallel networks offer customers a migration path that will enable them to get from frame and ATM to IP.

Goel says QoS is not an issue on the Qwest network, which is built for 0 percent packet loss and real-time operation. The capacity in the backbone provides enough bandwidth to ensure QoS. Qwest also achieves QoS with one-to-one operation. "Other IP backbones do oversubscription of five-to-one or 10-to-one," Goel says. "We have the capacity to do one-to-one, so we don't drop packets and, in case of failure, we have SONET autoswitching. So we are able to provide real-time applications on the network."

Qwest, with its brand new, built-to-order IP network, is fairly unique in its ability to call up more bandwidth when it has to ensure QoS. "Not everyone is in Qwest's situation, so it's not an answer anyone can use," Gage says, "and networks tend to fill up pretty fast anyway."

Tacking on more DWDM channels is not necessarily the answer, she adds. "Even with new carriers, like Williams (Tulsa, Okla.) and Qwest, and technology like DWDM, [try to] balance that with the phenomenal growth of the Internet and the continuing evolution of data applications," she says.

With cable modem access and DSL just starting to hit the market, bandwidth usage will ramp up considerably over the next five years to 10 years. "What happens to all this space when we have 10 million users with 1mbps access to the Internet?" Gage says.

"Fundamentally it is an unresolved issue," Ryan says. "The terabit-router people are not the only ones who think overprovisioning the network with more capacity than needed makes sense. The unresolved part is that it's not clear whether, in fact, it is cheaper to build a massive network than a very good network, and no one knows the answer to that one."

AT&T Changes Lanes

While Sprint and Qwest espouse a particular network architecture, AT&T is carrying the torch for technology agnosticism. The company's architecture carries a mix of backbone technologies, each aimed at a different service category. The basic network is SONET or DWDM, with private-line, frame relay, ATM and IP transports all sharing that platform. Until recently, the whole network was SONET, but recent developments, including a deal with @Home Networks, Redwood City, Calif., for OC-48 DWDM channels, has put some IP directly on to DWDM.

The network has 35,000 miles of SONET in 57 rings across the United States. The choice of backbone technology comes down to customer needs rather than absolute considerations about QoS, says Dan Sheinbein, vice president, network architecture and development for AT&T. Putting IP directly onto DWDM means the network loses some of the protections of SONET. If there is a fiber cut, the network may not be able to achieve the quick 100-millisecond restore time of SONET. And, because an IP network is fully meshed, a cut may mean that 2.5 gigabits per second (gbps) of traffic suddenly gets dumped on other routes that may or may not have the capacity to accommodate them.

"Many ISPs are saying, 'I don't want you to go through the SONET layer and give 100-millisecond restoration. I will do it internally to my network. Just give me OC-48 straight into DWDM,'" Sheinbein says. "We at AT&T are agnostic. If the customer does want fast restoration, and most non-ISPs do want it, we will provide it."

In its own ISP, AT&T still is using the SONET layer, "but we will see how things go," he adds.

The actual traffic carried by the different transport methods in use in the AT&T network--ATM, frame relay, IP--is usually IP, Sheinbein says. But the network continues to use frame relay and ATM because of their ability to deliver guaranteed bandwidth. "You can't do that with IP today, and I emphasize today. IP is a send-and-pray method of transmission," Sheinbein says, "But IP will get there, and the reason is because AT&T and other carriers will begin to provide virtual private networks using IP. As we get more security and screening functions and can guarantee capacity in IP, we will see more and more customers use IP."

Sheinbein also believes frame relay and ATM will not disappear, because of large installed bases and the difficulties of migrating from one network technology to another. "So we are more likely to see interworking between frame and ATM networks, which we have begun, and between those networks and the IP network as we get more QoS [in IP]. As we move forward, we will see lots more IP on the backbone," he says.

For VoIP, AT&T provides its own consumer VoIP, called Connect and Save, that uses two-stage dialing and essentially is taking advantage of IP arbitrage opportunities. AT&T uses overengineering to ensure QoS on that service.

A newer play, but the one that is expected to grow exponentially, is VoIP over intranets. Here, some companies are using overengineering for QoS while others are doing the opposite, compressing the voice and putting it in gaps in the data transmissions. Today, this is used primarily for communications within a company, particularly between remote offices, with public branch exchanges (PBXs) directing some traffic onto IP and other traffic onto the switched network.

But, compared to data, IP still is rather thinly deployed in the voice realm, and one reason is that backbone providers are being fed so meagerly with IP traffic by incumbent local exchange carriers (ILECs) and competitive local exchange carriers (CLECs). That is not surprising, considering that VoIP to consumers is seen as a very long-term play in the incumbents' local loop. CLECs are expected to see more IP voice, particularly as corporate customers start to integrate voice and data networks, but that market has been slow to develop.

"Most CLEC applications that we have seen so far for IP have been rather limited," says Cody Bowman, AIN product marketing manager, Ericsson, Stockholm Sweden. "We talked to a number of customers over the past six months to a year about where they see IP unfolding in the local loop, but overall we see VoIP applications tied more to interexchange carrier (IXC) use."

IXC Builds New Road

IXC Communications has taken pure IP to a new network architecture that combines some features of SONET rings with IP's "meshed" architecture, one it calls "hierarchical meshed." Basically all routers in the network eventually are connected to a mesh as in a typical IP network, but they are organized into eight "regions" that act as sort of hubs or "superrouters." They also enable cross-country links of just two hops, from one regional hub to another, which is virtually unknown in other IP network topologies.

IXC Communications puts packet over SONET, with very-high-capacity DWDM-based links, now at OC-48 and able to scale to OC-192. IXC has revealed few details about pricing and specific services available on the network, leaving analysts unsure about its impact as a new approach to IP. "A lot of companies are talking about all the bandwidth they can deliver and high-speed backbones, some packet over SONET, packet over optics, ATM over SONET," says RHK analyst Joe Struber. "To me, the real issue is what are the services they can deliver and how are they priced? Maybe it's a great packet-over-optic network, but what if it costs twice as much as packet over ATM or I can get it only in three cities?"

The biggest problem for IP at this point is likely a QoS issue, which is far from any resolution. "If you get into multimedia traffic over IP or voice over IP added to data or if you want to have different quality of service for different applications or different customer classes, that is still where IP has some maturing to do," Gage says, "and new protocols, such as DiffServ (differentiated services) and MPLS (multiprotocol label switching), are being defined to address that."

A huge part of the problem, Ryan says, is that currently there is no way to control routers in IP networks so they either set up a virtual circuit or look farther ahead when choosing routes. If network operators could just tell a router which route to use, "the estimates that we have seen are that traffic engineering like that can save 25 percent to 30 percent of capital cost, not just bandwidth," Ryan says. The alternative way to solve the problem of a congested link would be to expand the router and buy more bandwidth, more OC-48s. And capacity on other routes in the network could continue to be underutilized, a stranded investment. "So the debate over how to build next-generation, high-capacity networks remains a very, very vigorous debate," Ryan says.

Carriers Drive Defensively

In sum, despite bold pronunciations by some of the newer network operators about the ascendancy of IP, none are running truly IP-only networks. And, even an ATM stalwart such as Sprint operates a pure IP-over-SONET architecture for its ISP.

"The answer is, because someone has to pay the rent," Ryan says. "Although for the services that are most mature, meaning technologically most stable, there is more money in them, they are not native IP services. They are voice and data, like frame relay, maybe IP inside the frame relay, but the company is signed up for frame relay service or signed up for ATM."

RHK estimates that IP revenues are well below 10 percent of telecom services revenues. "Here is where the big argument for creating a converged network becomes more difficult," Ryan says. Nevertheless, he says, reliable industry estimates are that IP traffic is rising each year by a factor of 10. If that estimate is true, within a few years voice will be a very small part of an IP-dominated network. That network may well be close to the 30-to-1 estimate of the ratio of IP data to voice traffic.

The fly in the ointment is that IP service revenues, however, are not also increasing by a factor of 10 or eight, but in fact just doubling. "So there are interesting strains on network architecture that don't, in the short term, point to happiness in converged networks," Ryan says.

Charlotte Wolter is infrastructure editor for PHONE+ magazine.