Case Studies> WAN Design

The Problem

A large electric utility had a large problem. It's WAN and voice network were out of control, and users were not happy.

In 1996, this company changed its operating structure, from three separate operating companies to a single company. In the years following, the company experienced phenomenal growth, adding offices and acquiring several smaller companies. Two data centers contained almost all application servers.

WAN planning was poor at best. Often point-to-point T1s would be purchased so that an Add/Drop CSU could be used to provide internal calling. Circuit selection was primarily determined by cost for that site only (the total cost was never considered). If a new site was built, the WAN may connect to another remote site, because it was thought to be less expensive than routing it to a data center. The circuit was generally provided by the LEC (Local Exchange Carrier), like Verizon. Not all sites were served by the same LEC. The IXEC (inter Exchange Carrier) was used as little as possible, only when it was necessary to cross a LATA line, because it was thought to be more expensive.

The WAN was a mess to administer. Some of the circuits were under contract, but most were not. Costs were impossible to monitor, because there were nearly one hundred circuits with prices changing nearly every month. There were no less than 12 different contracts, and no less than 30 separate bills. Some circuits would have multiple carriers. One circuit was carried in part by three different carriers. There were three different bills for this one circuit. Application support staff had problems too. They couldn't determine where to place file, print, and cache servers.

When a user would experience performance problems, it was impossible to determine where the root of the problem was. Was it at that site or at one of the seven sites the data "hopped" through? When a site was down multiple providers may need called, or it could be caused by equipment failure. Staff had to be mobilized regularly to root out problems. There were costs in time and productivity, as well as vehicles and test equipment.

The Solution

Innovative Thinking employees developed a plan to solve the problem. The goal was to rearrange the data network so that every site would have a Frame Relay connection, with a PVC to each of the two data centers. This would ensure that every site had a "direct" connection to both data centers, eliminating over seventy single points of failure, simplifying maintenance, and eliminating all "hops". This also expedites problem resolution, as latency would be uniform, and bandwidth utilization could be monitored closely. If a site didn't have enough bandwidth, or had too much bandwidth, the port or PVC could be changed for that site with no affect on other sites.

Several problems faced this resolution.

• The voice network was made up partially of point-to-point T1s and Add/Drop CSUs
• The cost was prohibitive
• The implementation would need to be closely coordinated and well managed

Innovative Thinking employees developed a Voice over IP solution to replace the TDM portion of the voice network. The switches and routers were Cisco, but the PBXs were primarily Nortel Networks. Cisco was clearly a better decision. Equipment was leased to distribute the cost over time. Some sites had circuits leased solely for voice. By converting these sites to VoIP, a significant savings was attainable. The Add/Drop CSUs were eliminated in favor of integrated CSU cards for use on the Cisco routers. This standardized the end equipment, and provided secure access to CSUs. Our first problem was resolved.

Next up was trying to make the design affordable. The company had a contract with an IXEC primarily for Long Distance, but did have rates for Frame Relay. After interviewing several other IXECS, the incumbent agreed to renegotiate their data AND voice rates. The end result was a slight increase in WAN costs, but a significant decrease in Long Distance. The end result was a slightly lower net bill. There were additional savings when the routers came off lease and were replaced with scaled down models. Maintenance was drastically reduced.

With the design phase problems out of the way, only the implementation problem remained. A schedule was developed and accepted. Innovative Thinking employees developed a "WAN cutover checklist". This document contained all relevant information including circuit IDs and contact phone numbers. This document was distributed to all necessary staff involved in the cutover. Weekly conference calls were scheduled with the carrier and company staff to review progress. A weekly email update was sent to all stakeholders, keeping them apprised of progress. The project was managed from beginning to end.

The Result

The end result? NOT ALL PROBLEMS DISAPPEARED. Some users still experienced slow performance from one particular application. Further analysis revealed flaws in the way the application was performing error checking and transmitting data. The problem was resolved by changing the application, not the network. Instead of throwing money away by increasing the available bandwidth, a small amount of money was spent purchase Citrix servers and licenses.

Today, performance is consistent across the network. Bandwidth utilization is closely monitored. There are two WAN bills. The VoIP solution has allowed a smooth transition to IP Telephony. The maintenance staff was reduced because of reorganization. Because of the change, fewer people are able to support the WAN, efficiently and economically. Even better, the next step, a transition to a MPLS based network is a relatively minor step - just a few logical changes, no truck rolls. All this, thanks to some Innovative Thinking!