Review
of the
ISUnet
Core Upgrade
Project
February 1998
Revision 2
Richard J. Scafuto
Network Systems Engineer
Kent Datacomm
Schaumburg, IL
Table of Contents
Introduction * Background * Current ISUnet Configuration * Future ISUnet Configuration Requirements * ISU’s Concept as Proposed for ISUnet * Technology Overview * Networking * Raw Speed vs. Throughput * Fast Ethernet * Gigabit Ethernet * FDDI * ATM * VLANs and Address Consolidation * QOS * RSVP * LANE * MPOA * NHRP * DHCP/DNS/DDNS * Multicast * Internet2 * MREN * Network Management * Concepts * Best of Breed vs. One Vendor * Redundancy vs. Hot Spares * Centralized vs. Distributed * Migrated Path * Routing at the Core vs. at the Edge * Security * Tactical vs. Strategic Decisions * Tactical * Strategic * Marketing Issues * Funding * Arguments * Fast Ethernet/Gigabit Ethernet Backbone * ATM Backbone * Conclusion * Appendix A * Consultant Qualifications * References *
On December 15, 1997 Scott Genung, Manager of Networking Systems, Office of Telecommunications, requested a meeting with Kent Datacomm to discuss the future ISUnet. At that and subsequent meetings, Scott went over the current architecture and the future requirements of the University.
*Statistics as reported by Scott Genung.
Future ISUnet Configuration Requirements
The future requirements of ISUnet are based on some core issues:
Educational
Students are being required, as part of their general education requirements, to use the network for research. They are being encouraged, if not mandated, to use web searches, newsgroups, list servers, email, etc., to do their research.
Experimental
Students are being encouraged to experiment with technology. Many lab areas have been, and are being built with this in mind.
Educational
Faculty must stay up to date on emerging technologies, if they are to be able to present information in an accurate and timely manner.
Use of on line student records for registration is also a major requirement.
Administrative
Administrative use of the network will focus on grant proposal writing and on line student records.
Each of these groups within ISUnet are permitted to operate their own systems. Operational requirements therefore dictate that there must be an "any to any environment" allowing as much bandwidth as required, at any time, to any group or user.
The University has produced a document titled "Vision 2000 Strategic Plan". Listed below are a few of the highlights of this plan, as quoted in the ISUnet Core Upgrade Project document:
In conjunction with the "Vision 2000 Strategic Plan" the University Networking Systems Group has identified a number of items that need to be addressed. These items, as stated by Scott Genung, are as follows:
ISU’s Concept as Proposed for ISUnet
In preparation for a "Future ISUnet", the Networking Systems Group at ISU has developed a new network design. This design is focused around a core distributed ATM backbone that is distributed to various buildings around the campus. There, "edge switches", are deployed that will extend the core ATM backbone to various strategic points on campus. These edge switches will also be the entry points into the ATM core for multiple building backbones. The edge switches will be responsible for all routing functions and Segmentation and Re-Assembly (SAR) requirements, for packet to cell conversion, when access into the ATM core is required. The building switches that feed into the edge switches can and will be, of various technologies, such as Ethernet and Token Ring. This design will allow the end nodes and their connectivity options, to be maintained, rather than a complete redesign from core to end node.
The design also takes into account that ISUnet is a production network and must be operational during the phased in cut over to the new design. This design is a phased approach and set to function in the beginning phases, side by side with the existing network. As the buildings are brought on line, live testing will be available to work out any issues that will arise, while maintaining the production system.
In the early 1980’s, systems were built on the centralized model. All devices were terminals connected to a central computer. During the middle to later 1980’s, file server system models were being phased in . In file server models the processing is done at the end node and files are saved to a centralized computer.
Today’s networks are client server based, which is a combination of the centralized model and the file server model. Some or all of the processing is done on the end node and some of the processing can be done on the server. Today, the computing system is no longer held within a metal box. The computing system is the network itself. Users can directly access information on servers, on one another’s desktop, print across the network or for that matter the world, and talk, using picture phones, over today’s networks. These networks have become mission critical systems with processing power, beyond our imagination. These computers can produce two to three hundred megabits per second of information through a single network interface card.
Networks are living networks, growing and adapting as the environment changes. As these systems evolve, users will create ways to use them that have not yet been considered. Trying to "see the future" in this business is an arduous task.
Many people look at the "horsepower under the hood" as the main issue in building a network. Just because ATM is stated to have 155 Mbps, or Ethernet at 100 Mbps, can it, in fact, produce that amount of data?
The same holds true for all networking technologies. Several factors come into play. For each technologies there are multiple layers of overhead at different levels. For ATM there is the SONET, SAR, and cell overhead. For Ethernet there is the CSMA/CD and preamble overhead. Usable throughput will always be less than what is stated.
There is also the issue of bottlenecks. Even if it were possible to push one gigabit through the network, where could it be pushed to? What computer or sub system could handle that much data? "The maximum amount of data that current-generation servers are expected to be able to transmit is around 250 Mbps to 350 Mbps"..
100BaseX is the IEEE 802.3u specification for the 100-Mbps Ethernet implementation over unshielded twisted-pair (UTP) and fiber. The Media Access Control (MAC) layer is compatible with the existing IEEE 802.3 MAC layer. It is nearly identical, other than some distance issues, to 10BaseX, except for its speed.
Fast Ethernet has been around since 1994 and is proven technology, providing a larger pipe for the aggregation of 10BaseX networks. Some vendors have implemented a proprietary technology that allows multiple 100BaseX ports, to be run in parallel, to their devices. This allows these ports to aggregate the data and provide gigabits of data over 100BaseX.
Gigabit Ethernet is not currently an IEEE standard, but still being studied. Standardization is expected in the first half on 1998 in the 802.3z subcommittee. The current standards efforts are based on a Fiber Channel physical layer and other high-speed networking components, so standardization should be easier than having to write a physical layer specification also.
To accommodate unshielded twisted-pair (UTP) cabling, a logical media independent interface will be specified between the MAC and PHY layers that will enable Gigabit Ethernet to operate over unshielded twisted-pair (UTP) cabling.
Today, several companies are shipping pre-standard products for customers who can not wait until the standard is in place or just simply wish to do testing or proof of concept studies.
"Twelve companies demonstrate that products can work together. The companies included in the testing were 3Com Corp., Acacia Networks, Inc., Alteon Networks Inc., Bay Networks Inc., Cabletron Systems Inc., Cisco Systems Inc., Extreme Networks Inc., Foundry Networks Inc., Hewlett-Packard Co., Prominet Corp. (which is being acquired by Lucent Technologies Inc.), Sun Microsystems Inc. and XLNT Designs Inc."
Fiber Distributed Data Interface (FDDI) is the ANSI X3T9.5 standard for the 100 Mbps implementation over fiber. It was originally based on the IEEE 802.5 Token Ring Standard.
FDDI uses a dual-ring architecture with traffic on each ring flowing in opposite directions (called counter-rotating). The dual-rings consist of a primary and a secondary ring. During normal operation, the primary ring is used for data transmission, and the secondary ring remains idle.
FDDI is frequently used as high-speed backbone technology because of its support for high bandwidth and greater distances than copper. FDDI’s appeal for the backbone was strong a few years ago. Before Fast Ethernet, it was the only viable way to aggregate the 10BaseX and the 16 Mbps Token Ring networks, and build server farms. Unfortunately FDDI did not have the price reductions that Ethernet products did. FDDI is still in use today at legacy networks, but due to its price, it is not a viable technology for the future.
Asynchronous Transfer Mode (ATM) is defined by a set of switch interface standards developed by the International Telegraph and Telephone Consultative Committee (CCITT), now called the ITU.
ATM is a switching technology based on a cell-relay protocol that uses a connection oriented, fixed-length, 53 byte cell structure. The fixed-length cell structure provides predictability in delay, so that ATM can support applications requiring continuous bit rate transmissions.
These characteristics allow ATM switches to be simple, and this simplicity allows them to be fast. ATM does not guarantee the correct delivery of cells, or even that they will be delivered at all. What it does guarantee, is that the cells that are delivered, will be delivered quickly, and in order. The ATM protocol is simple enough to allow cells to be switched in hardware which adds to its speed.
ATM links can operate at different physical layer speeds, for example, 25 Mbps, 100 Mbps, 155 Mbps, 622 Mbps, and beyond. It can consist of different media types such as, multimode fiber, single mode fiber, coax and shielded or unshielded twisted pair. These physical layers can be arbitrarily mixed in a single network.
Since this technology is connection oriented, each device coming on line will request a path through the network before it can transfer information. A circuit is set up with the switch for as long as needed and then torn down when not needed, much like how a home telephone works. During this set up, a request is made for a Quality of Service (QOS). The switches along the path can either guarantee or deny the level or service requested depending on parameters set by the administrator or by the level of congestion already in the network. The decision is then to transmit, or in the case of denial, to request a lesser amount or quality.
One of the major advantages of ATM is that the same protocol is used worldwide, in the Wide Area Network (WAN), and in the Local Area Network (LAN). Only the bandwidth pipe (physical transport layer) changes, therefore the difficult, costly and time consuming (latency) issues of traditional LANs are greatly reduced.
VLANs and Address Consolidation
In a traditional LAN, workstations search for each other through broadcast or multicast messages. Switches and other layer 2 type devices, will propagate these broadcasts messages allowing anyone within the area to be able to see these packets, process them, and possibly respond to them. The area within which broadcasts and multicasts are confined is called a broadcast domain. A router is usually used to prevent broadcasts and multicasts from traveling beyond the broadcast domain of the network. This means that everyone on a LAN (broadcast domain) must be located in the same area, behind the same router. Beyond the broadcast domain, the router must help you to get to your destination.
Virtual Local Area Networks (VLANs) allow a network manager to logically segment a LAN into different broadcast domains. Since this is a logical segmentation, and not a physical one, workstations do not have to be physically located together, as is the usual approach. Users on different floors of the same building, or even in different buildings, can now belong to the same LAN, or broadcast domain.
VLANs can offer a substantial performance increase if designed correctly. Network traffic consists of a high percentage of broadcast and multicast traffic. VLANs will confine these to the users who need to see them and reduce the need to send such traffic across the network.
A major amount of network costs are the result of adds, moves, and changes of users in the network. Every time a user is moved in a LAN, new station IP (Internet Protocol) addressing, and possibly reconfiguration of hubs and routers becomes necessary. Some of these tasks can be simplified with the use of VLAN's.
Periodically, sensitive data may be broadcast on a network. In such cases, placing only those users who can have access to that data on a VLAN can reduce the chances of an outsider gaining access to the data.
With VLANs, a consolidation of IP addresses can take place. Networks that were dispersed, yet all belonging to the same group, can now be placed in the same VLAN and share an IP subnet. Without this, it is often the case that many subnets may have only used a small amount of the possible 254 addresses available, on them.
With this savings, VLANs will add a larger layer of administrative complexity. It will also become necessary to manage virtual workgroups, rather than physical ones. The only way to do this, is by being very diligent at record keeping, and by employing excellent management tools specifically designed for VLANs.
Another issue with setting up virtual workgroups is the implementation of centralized server farms. Server farms are essentially collections of servers and major resources for operating a network, at a central location. The advantages here are numerous, since it is more efficient and cost-effective to provide security, uninterrupted power supply, backup, and a proper operating environment, in a single area. Server farms can cause problems when setting up virtual workgroups, if servers cannot be placed on more than one VLAN. In such a case, the servers would be placed on a single VLAN and all other VLANs wanting access would have to go through a router. This will reduce performance, and defeats the original concept. The way around this, is to pick systems which allow multiple VLANs.
Quality of Service (QOS) is an important concept in any network, and especially, ATM networks. It includes parameters like the amount of bandwidth required and the delay requirements of a connection. In ATM, such requirements are included in the signaling messages used to establish a connection. Although there is some basic "Type of Service" in TCP, it is rarely used. Current IP (IPv4) has no such inherent QOS concepts, and each packet is forwarded on a best effort basis by the routers.
"Unlike routers, ATM switches and technology have been designed from the start to be able to guarantee all service levels, even the most stringent (CBR), allowing very predictable performance. For example, the need to guarantee stringent performance is the reason that ATM cell payloads are only 48 bytes, in marked contrast to IP packets. ATM hardware technology is also inherently better for multicast, and for multicast quality of service, than is router technology".
A LAN based protocol, the Resource reSerVation Protocol (RSVP) is an IETF signaling protocol that can be used by applications to request special treatment for particular IP traffic streams. Priorities can be set and buffers reserved at all intermediate routing or switching points, as long as the routers and switches understand RSVP. Using a connection separate from the data connection, a receiving application sends an RSVP request message to the network. The message identifies the IP data connection involved (using the IP destination address, port number, and protocol) and requests a level of service. RSVP can handle both point-to-point and multicast addresses, distributing the RSVP requests to all concerned routers in the path.
Routers are not required to offer all RSVP levels of service, and they can unilaterally alter the service level provided at any time during the connection. The user can attempt to renegotiate or can abandon the connection. The user can also request an alteration in the quality of service at any time. Because of the stringent requirements for the Guaranteed Service Level, router overhead is large.
Another difficulty facing RSVP, is the request is made after the route is selected. The routing protocol, being completely unaware of the quality of service needs, could easily choose a path through the network, using routers that are unwilling or unable to grant RSVP requests. As all routers in a path must agree to an RSVP request, this can easily lead to rejection of a request, even though a path that could have supplied the needed quality of service, remains idle.
Another issue is scaling. Each router must maintain state information about all of the RSVP controlled data flows traversing it, and this can become overwhelming for those routers used by many different flows.
An Internet Draft that analyzes RSVP, states that "it will generally not be appropriate to deploy RSVP on high-bandwidth backbones at the present time.""
In LAN Emulation (LANE), an ATM network is configured to simulate an Ethernet or Token Ring LAN. FDDI LANE does not exist, therefore, in the case of FDDI, all FDDI traffic must be converted to Ethernet or Token Ring first. The resulting LAN is called an Emulated LAN (ELAN). LANE defines a service interface that looks to any protocol exactly the same as Ethernet or Token Ring. In this way the IP/IPX/etc. software that is running previously on Ethernet and Token Ring can be run on the ATM network without modification.
While LANE sounds like the perfect solution, it has its faults. Many of today’s networks use FDDI as a backbone system. All the traffic would have to be converted. This process would take up valuable time and negate any bandwidth realized.
All devices on the ATM, known as a LAN Emulation Client (LEC), must register with a LAN Emulation Server (LES): Each ELAN contains a LES, which acts as the coordinator. Each LEC will register with the LES its MAC address. Based on such information, the LES resolves MAC addresses into corresponding ATM addresses.
Each ELAN includes a Broadcast and Unknown Server (BUS) to emulate the broadcast capability of Ethernet and Token Ring, because ATM is a point to point connection oriented network. A LEC who broadcasts a packet sends it to the BUS, which forwards every ELAN member a copy. Before the direct data connection is built between two LECs, the data between them is forwarded through the BUS. Once a direct data connection between two LECs is established with a switched virtual circuit, then the LECs talk directly to each other.
The last piece, the LAN Emulation Configuration Server (LECS) stores the configuration information of each ELAN including the LECs, LESs and BUSs belonging to that ELAN.
LECs that belong to different ELANs and requiring to communicate with each other, must go through routers. The router port must belong to each of the ELANs.
If this sounds complex, it is. Before any data is actually passed, over 6 switched virtual circuits (SVCs) are brought on line. Management and troubleshooting of these circuits can be very challenging.
Described below, MPOA solves this problem by creating shortcuts between ELANs.
Multi Protocol Over ATM (MPOA), is a specification from the ATM Forum that is a combination of LANE and NHRP (Next Hop Resolution Protocol, discussed next). MPOA improves LANE by allowing inter-ELAN traffic to go through shortcut connections rather than routers. To build such a shortcut, NHRP is used to resolve destination IP address into ATM address. In this sense, MPOA is a combination of Layer 3 routing and Layer 2 bridging. MPOA based networks communicate with conventional routers via standard routing protocols, such as RIP (Routing Information Protocol) and OSPF (Open Shortest Path First). In essence, MPOA identifies data "flows" and maps them directly to ATM virtual channels.
A typical MPOA environment is an ATM network with ATM hosts and edge devices attached to it. An edge device can be a switch or a router which brings legacy LANs into ELANs. There are a number of ELANs on any given ATM network and the job of the MPOA protocol is to find the most efficient way for any two hosts to communicate with each other.
MPOA also supports the separation of route calculation and data forwarding. A traditional router has both functions. In an MPOA environment, if every edge device is a full-fledged router, it would be too costly. Also, the speed of today’s router would not be able to keep up with the flows.
The general concept involves splitting forwarding and routing functions, traditionally supported within conventional multi-protocol routers, between MPOA Clients (MPC) and MPOA Servers (MPS). Address management and topology discovery, for example, are performed by the MPOA Server, while traffic forwarding is provided by MPOA Clients via the ATM switch fabric. The MPS typically resides in an ATM switch-router or a stand-alone ATM attached route server, while MPCs reside in edge devices and ATM attached hosts. This provides a physical separation between the devices that calculate the inter-network route and those that forward the data. While traditional routers are limited by the speed of their back-planes, an MPOA based routing system uses products such as ATM switches, resulting in a multi-gigabit routing infrastructure.
ELANs and devices on different IP sub nets have to go through routers, which is not an optimal solution when both parties involved are attached to the same ATM network. Direct connection between them is desired. All we need is a mechanism for an end system to resolve the IP address of another end system in a Logical IP Subnets (LISs) into its corresponding ATM address. NHRP does this.
NHRP provides an extended address resolution protocol that permits Next Hop Clients (NHCs) to send queries between different Logical IP Subnets (LISs) sometimes referred to as Local Address Groups (LAGs). Queries are propagated using Next Hop Servers (NHSs) along paths discovered by standard routing protocols such as RIP and OSPF. This enables the establishment of ATM SVCs across subnet boundaries, allowing inter-subnet communications without using intermediate routers.
Each LIS contains at least one NHRP server and each end system is an NHRP client. When an end system needs to resolve an IP address, it sends a request to an NHRP server in charge of its LIS. The NHS keeps a table of IP address to ATM address pairs for all the hosts that belong to the LISs it is serving. If the IP address to be queried belongs to these LISs, NHS expects to find an entry that matches the IP address and replies with the corresponding ATM address. Otherwise, a negative reply will be returned. At this point the "router" does not need to be involved any further, and the data can flow directly between end nodes.
"With the ATM Forum’s MPOA standard, the latency associated with the multiple routing segmentation and re-assembly is reduced by distributing L3 (Layer 3) switching at the edge via NHRP. CTL3 (Cut-Through Layer 3 Routing) avoids setting up SVCs for short-lived transactions, providing scalability".
Constant increases in the number of computers, along with the trend towards remote and mobile computing, have made managing TCP/IP configuration information for each computer a nightmare for administrators. One tool that can help with IP address management is a DHCP server.
Dynamic Host Configuration Protocol (DHCP) is a network protocol that enables a DHCP server to automatically assign an IP address to an individual computer's TCP/IP stack. DHCP assigns a number based on a defined range of numbers set by the network administrator. Although this helps on one hand, with not having to manually configure each device, it further complicates the process. The DHCP data must be synchronized with the DNS. Although DHCP does not need DNS to function, many IP services require that there be a DNS entry.
Domain Naming System (DNS) is a program that converts domain names to their IP addresses. Without it, it would be necessary to remember numbers, for example, 138.87.7.99, instead of words, Joe_Computer, to get around the network. DNS is usually set to update all other DNS servers at specified intervals. Maintaining DNS information in text-based configuration files is time-consuming and error-prone. One small mistake or a duplicate IP address can bring the entire network down.
Dynamic Domain Naming System (DDNS) alleviates this problem by synchronizing the activities of DNS and DHCP servers, easing the administrative burden imposed by ever-changing network configurations.
Multicast is a method of providing information from one to many users in an efficient manner. Most applications, such as viewing a distance learning video file across the network, would require each end node be provided individual streams of the same video. Consider the bandwidth being wasted if 20 people in one area were to view the same video from another area. Multicast handles that by providing a standard way of bringing the information as close to the end user as possible before allowing each user to have their own packets.
As distance learning becomes more prevalent, ISUnet will be required to have a Multicast Scheme in effect, thereby reducing the overall bandwidth required within the core.
Internet2 (I2) is a collaborative effort by universities, the private sector and the federal government, to develop a next generation gigabit Internet technology and applications. Internet2 will renew the government, academic and industry partnership that built the first generation Internet.
"The university Internet2 effort and the federal NGI initiative are parallel programs with many common elements. The Internet2 program is already in partnership with the National Science Foundation's (NSF) meritorious High Performance Connections program, from which over 50 Internet2 institutions have received competitively awarded grants. In particular, Internet2 intends to assist in achieving the NGI goal of establishing a high-performance academic network operating at speeds 10-100 times greater than today. Additional cooperative relationships are being planned as part of NGI implementation. As Internet2 develops and NGI programs go forward, it will be important to ensure advanced network services are available on interoperable backbone networks that are competitively provided by many vendors".
The way this works is that there are Network Access Point (NAPs) located all over the U.S. All of these are considered GigaPOPs. The name for GigaPOPs comes from the giga in gigabits and POP as in Point of Presence. Ameritech, Chicago, is this area’s GigaPOP. The Ameritech NAP in Chicago is based on Cisco products, such as 7500 series routers and Lightstream ATM equipment. This NAP provides access to other universities and higher end research facilities in the area who are also connected to this GigaPOP. Further, it provides connectivity to the Metropolitan Research and Education Network (MREN, discussed later). This connectivity is provided through ATM based links. Today, this site operates with OC3c but will soon be updated to OC12c and is planned to go to gigabits in the future.
"The Metropolitan Research and Education Network (MREN) is a collaborative effort between Argonne National Laboratory (ANL), Fermi National Accelerator Laboratory (FNAL), the University of Chicago (UoC), the University of Illinois at Chicago (UIC), and Northwestern University (NWU). The purpose of the collaboration is to deploy a very high-speed network between the sites which will provide an infrastructure which will support high-end, collaborative research for scientists and educators. The network technology which was chosen to interconnect the sites is ATM-based. The topology is star shaped with high-speed network connections from the sites to an Ameritech owned and operated ATM switch. ANL and UIC currently utilize OC3 (155Mbps) links; FNAL and UoC use DS3 (45Mbps) links; and NWU is in the process of connecting at OC3. The network not only provides an infrastructure in support of leading edge research, but it also provides a test bed for investigating advanced networking concepts such as traffic shaping."
As has been discussed previously, all of this technology is complex. Add to that, the amount of users and the services that they require. This totals up to a very complex system that no one person can effectively manage. There are numerous systems available to watch over a network. A network, is only as good as its weakest link. A mis-configured workstation, an under trained console administrator or worse, a system that is not correctly configured, will only serve to add to the problems when they occur on the network.
"Graves said the database that generates the DNS files, which contain all the information necessary to locate Internet URLs, crashed and generated incomplete files for all the .com and .net domain names. Even though a monitoring system indicated that the files were bad, a network operator sent the data out to the primary root server anyway, he said".
For proper network management, two types of tools are required. One pro-active, such as Hewlett Packard OpenView, and one which is usually used as reactive, such as Network General Sniffer, a protocol analyzer.
Using Hewlett Packard OpenView, all networking products are monitored for potential faults before they become hard failures. Further, it will also provide trend analysis information.
A protocol analyzer is used when things go very wrong. This is the only product that can be used to help diagnose the actual packets that are on the wire and the protocol stacks themselves. As a reference, prices of a Network General Sniffer is $10,000-$20,000 for Ethernet and more than $30,000 for ATM.
Consider these two conditions to using a "Sniffer" type of product.
There are two competing philosophies in the networking world. One is to use the Best of Breed, the other, a One Vendor solution.
Some of the pros of Best of Breed philosophy:
Some of the pros of using a One Vendor philosophy:
There are two basic philosophies to redundancy. First, a network can be built where there is at least two of everything, and it is all running, on line, at all times. In the event of a major failure, no user will be affected.
The second, is to provide no redundancy at all, but to only provide spare parts that are available at a moment’s notice to repair a down system.
Each philosophy has its merits and its downfalls. To provide a completely redundant system, cost would be the only factor. Since it would be necessary to literally place at least two of everything in the network. This would be overkill for ISUnet. However, it must be remembered that ISUnet will be a complete system at some point, handing voice, video and data, so redundancy must be provided at some point.
Usually, full redundancy is provided in the core of the system. This is where completely identical systems are placed on line, so that if one system fails, the other system would continue, without user intervention.
The edge systems usually will have redundancy to the level of power receptacle, power supply units, extra ports and slots for on line hot standby. At this level, user intervention may be needed but only for very hard failures.
End user areas usually provide no redundancy, but adopt a swap out philosophy. The standard practice is that for every 10 units of each product, one should be purchased as a spare. Usually a vendor will provide spares at a substantially reduced rate. One of the most over looked items in the spare inventory is upgrading. Each time software or hardware revisions occur in the main system, those upgrades must also be applied to the spares. There are times that only certain revisions of a product will work together.
In a centralized system, all network resources are under the control of the network manager.
In a distributed system, the network manager provides the connectivity between systems and centralized services, such as email, internet connectivity, etc.
In a migrated path, bandwidth is slowly incremented in the areas of the network that require it. In order to implement a migrated path, it would require:
The migration plan addressed here is not to be confused with the planned equipment phase-in already built into the upgrade of the new ISUnet. An upgrade of this size would be conducted as a planned phase-in, regardless of the technology. The plan being discussed here promotes a gradual migration of technologies, providing bandwidth only when necessary, versus the planned phase-in of the equipment.
Routing at the Core vs. at the Edge
Routing at the core provides for an effective use of time and resources, both during the installation and troubleshooting thereafter. Whereas all packets that are routed through the network eventually end up in a centralized area, therefore, it is easier to do the initial install saving time and money on the moves, adds and changes to that routing system. There are however, some major problems with this environment.
If routers are placed in the core, current technology routers will not be able to handle the traffic loads already on the network. Moving to route servers may help, but it is possible that they will not be able to handle the anticipated traffic loads in the near future.
Scalability is an issue. While more nodes are being added to the network, the routers are basically the same out of date products, which will ultimately cause the network as a whole to slow down.
Security also plays a role in the central model, because all data traffic travels through one area. Even if redundant routers were to be provided the risk of failure will still exist at the core of the network.
Although security is beyond the scope of this report, it is being mentioned as an issue that requires serious study. The new ISUnet will provide ubiquitous connectivity to the whole of the University. With the network becoming a switched environment vs. the existing shared and switched environment, the amount of security has, by default, been raised. Users who once were able to snoop all information on their segment will now only see packets that are destined for them only, making it much harder to pick up passwords, log on IDs, etc. But, do not become complacent. As the future unfolds, the University’s intention is to place all of the voice, video and data over one common network. This network can become vulnerable to attacks by hackers.
On the note of physical security, diligence should be exercised, to insure all areas of the network are under lock and key. Simply turning off one switch can bring down an entire area or building. Consider the effect on the University, for example, if the computers and the phones went off line during registration.
Regarding the physical layout, care should be taken that there are multiple physical routes for the fiber plant. Often products are placed with redundancy in place, only to have a cut in the fiber cable bring down the network. It is recommended that the fiber plant be constructed as a ring with a matrix. This will provide a minimum of two complete paths in the event of a cable cut.
Tactical vs. Strategic Decisions
"…(T)he best kind of solution to the performance problem is not a single big jump in bandwidth, but a scalable performance upgrade that allows the network's capacity to gracefully keep pace with user demands. In the ideal, the network manager would have the ability to add bandwidth to the network when and where it is needed, with minimal disruption to existing applications and investments in desktop connectivity."
"…(A) user considering both Fast Ethernet and ATM for its highest-bandwidth applications could cut its network ‘cost of ownership’ in half by purchasing ATM today and eliminating the cost of building, maintaining and the scrapping a Fast Ethernet solution implemented only as a tactical solution."
"While many analysts have written off desktop ATM as dead on arrival, this product shows that the technology is on track and is, cost-wise, a viable alternative to much less powerful Ethernet and token-ring solutions. In fact, all at once, the major technical and market impediments to desktop ATM are being swept away, revealing what looks to be a technology sector finally ready to blossom." Here, Kevin Tolly anticipated an awakening of ATM technology to the desktop.
Of course, we can look back with 20/20 hindsight and see that this statement is not entirely correct. ATM proponents thought that ATM would be installed from end node to end node universally.
In 1997, we watched a strong desktop ATM player, Madge Networks, literally scrap it’s entire line up of ATM products. "Interestingly, the new Madge also almost ATM-free — a radical turnabout for a company that was a strong proponent of ATM — even desktop 25 Mbit/s ATM."
Just 11 months later Kevin Tolly changed his mind about ATM:
"Unfortunately, for the proponents of desktop ATM (25M bit/sec), technical merit alone was not enough to make the technology viable. And several recent events indicate desktop ATM's demise may be close at hand…The situation has become only worse. With Fast Ethernet network interface cards (NIC) now available for under $100 and 100M bit/sec switch port prices plunging, few would find even a $200 25M bit/sec ATM NIC to be a bargain – and ATM NIC prices remain twice that…the flickering flame that is Desktop 25 will soon be snuffed out for good."
From these quotes, it is clear that the marketing aspects of technologies have a lot to do with what comes to market and stays at market. The technical merits of a product or technology do not necessarily bode its success.
In business organizations, plans are laid for systems to be built over long periods, with various upgrades taking place. Using a product’s predicted useful life, depreciation is calculated. Eventually this product is considered completely depreciated and can be replaced any time thereafter. If a product reaches the end of its useful life before the end of the depreciation period, for possibly technical reasons, a write off is taken, and the product is replaced with the "latest and greatest".
At the University level, and other not for profit organizations, funding is provided through grants, donations and tax dollars. Rather than a steady cash flow each year being dedicated to network personnel and projects, funds arrive in lump sums. With this type of environment, it is important to look further into the future during implementation planning, because systems must be installed that can carry the University far into the future.
In this study, various aspects of technologies that are anticipated as the future of ISUnet unfolds have been discussed. This has turned into somewhat of a religious argument for some people. And so this study will provide a basic overview of the leading two technologies, ATM and Ethernet. No system can be based solely on one consideration, such as price, therefore all arguments must be considered. Each technology has it advantages and disadvantages, and each could be used effectively in the new ISUnet. Each technology also has the potential to provide for a network that will deliver the requirements set forth.
Fast Ethernet/Gigabit Ethernet Backbone
Each end user is, or will be, provided with a switched 10 Mbps link, except for legacy connections. The legacy connections can be shared or dedicated Ethernet or Token Ring. But for the argument here, the focus will be on switched Ethernet, since that is the stated direction of ISUnet. Each of these users ports will be aggregated to a switch. This switch, and others, will be aggregated to an edge device. The edge device will be responsible for routing and the segmentation and re-assembly (SAR) functions of getting a user onto the backbone. Knowing today’s machines and the applications the users are running, it is not likely that the user will be using their 10 Mbps connections at 100% capacity, 100% of the time. To provide the back end of the network with enough bandwidth to accomplish this would be overkill. A blocking architecture could be installed that would provide enough bandwidth for a reasonable amount of time.
Using Fast Ethernet as backbone technology alone would not provide enough bandwidth, so a higher level will be needed. Gigabit Ethernet would fit the bill, but it is not standardized yet . Standardization is anticipated for the first half of 1998, well within the time period of this project. Until it becomes standardized, the University could use a proprietary technology, offered by most of the top tier vendors, in which they aggregate multiple Fast Ethernet connections. Often this can provide in excess of 400 Mbps half duplex. Now assume that this is switch to switch, with full duplex mode, providing in excess of 800 Mbps. As Gigabit Ethernet becomes available in standard mode, the University can slowly replace these connections with it. A slower replacement allows the prices of all technologies to drop due to the incline of demand, and the passage of time.
Even today pre-standard Gigabit Ethernet is significantly less expensive than ATM. "When 3Com announced the switch in November, it also announced pricing for high speed modules. The ATM variety lists for $10,000 for two 622Mbps connections; the packet version costs $4,000 for two gigabit-Ethernet ports. Few networks can realize two-and-a-half times the benefit by going ATM."
Other technologies such as RSVP were made for the LAN, so why should the University try to map it into QOS over ATM? It should also be asked why use LANE, MPOA and SAR, if the University will only use ATM as a large pipe to carry LAN traffic across campus?
The reasons for ATM are clear. ATM is a worldwide standard. ATM is scalable from 25 Mbps to multi gigabits, without having to use proprietary products. The same ATM cells that run over a 155 Mbps (OC3c) pipe are the same that run over a 2.48 Gbps (OC48c) pipe, alleviating all of the protocol changes we see in LAN to WAN, such as Ethernet to T1. The advantages of ATM do not necessarily come at a low initial cost.
While ATM may be more expensive initially, it will be an investment that is more cost effective over the long term. ATM is generally recognized, in the networking community, as the basis of backbone networking for years to come, and the strategic decision. "While Ethernet switches may involve simple tactical purchase decisions made largely on the basis of raw performance and price per port, 10/ATM switching and ATM itself are obviously strategic investments in LAN infrastructure that must be decided based on tradeoffs among a much broader range of purchase criteria". "A strategy that does not provide for voice/data integration does not take full advantage of ATM’s potential for improved management and cost savings." Once the initial investment is made, ATM’s advantages can be realized. One of these advantages is Quality of Service (QOS).
QOS is built into ATM and has been built that way since its inception. Without QOS, what happens when ports, switches, router servers and backplanes become congested and overrun with traffic? Frame based products will simply start dropping packets arbitrarily, because it is not scalable. With ATM, the QOS can guarantee that the important information will get through. Could you imagine only hearing every other word a person says while you are speaking with them over the phone? The switches can use protocols to signal each other to have end nodes throttle back, or take different routes through the network.
This leads to the question of load balancing. ATM can provide a true load balancing environment. Load balancing is inherent in ATM-there is no extra effort involved in configuring the system to balance loads properly. As more services, such as voice and video are added there will be a need to maintain a quality of service that is acceptable to the users.
However, there is no way to provide voice, data, and video over frame based equipment with the quality that is needed. Frame based systems are not built to do this. You can force them to do it, but you would need to provide a pipe so large, that it could guarantee that it would never be over run with data. The slightest blockage will create an atmosphere of dropped packets.
For example, consider that a student may be watching a live broadcast of a class on campus. The network is frame based, so upon connecting to the conference, they use RSVP to "guarantee" bandwidth. They now have a path that will allow them the bandwidth. But, unfortunately the way RSVP works, the computer data may take a different route through the network, which does not have the guarantee of the bandwidth. As long as there is no congestion over the network, all is fine. If congestion were to occur, the student would miss parts of the class, and since it is live, there is no way to replay it. The switches that are carrying this traffic are usually small. Cell based switches, however have a larger capacity.
Cell based switches are usually built with backplanes in the 20 gigabit range, because it is expected that they will be pushing that kind of traffic around the network. Frame switches are usually in the 5 gigabit range. Why build a system that is obsolete in a short time? Considering obsolescence, ATM incorporates legacy systems into its system by using LANE and MPOA technology.
It is recognized that each user will not be installing ATM Network interface Cards (NICs) into their PCs, and most current applications are not aware of QOS. Therefore, in order to provide backward compatibility, LANE and MPOA will be used.
Finally, one of the major advantages of installing ATM has to do with the connection to Internet2. The Internet2 system is only operational on ATM. The resources provided by internet2, along with the MREN, UIC, etc. requires the use of ATM.
From a logical standpoint, regardless of what system is installed, whether it is ATM or Ethernet, each must be carefully watched, studied, analyzed and adjusted on a steady basis. But from a physical standpoint, once an ATM system goes in, other than the occasional upgrading of a few ports, it is essentially complete. If more bandwidth is needed, just add another OC12c or OC48c. A port can be easily brought on line and incorporated into the mix. This is basically a one install, done, scenario.
Kent Datacomm has studied the technologies, the marketing aspects, the various concepts, and have used our long standing knowledge of the environment to address the issues that face the University. Each of these issues have been taken into account to form a conclusion.
Seen in this study, there are many ways to evolve the new ISUnet. The requirements can be placed in front of 5 engineers and the University will be given 5 correct, but different, configurations. One of the easiest decisions to make would be to just recommend the cheapest or the easiest system to build. If we were to do that, we would not provide the University with the correct system for the long term.
The new ISUnet will be a living entity that requires constant attention. The University must provide dedicated resources to the study, testing, and implementation of these technologies. One of the issues discussed in this study is that the funding of this network may not be accomplished with a steady, regular flow of funds and with a planned phase in of new technologies. Rather, the funding situation impels the University complete the installation in one effort. This then, leads away from the Migrated Path Concept.
With the installation of some leading edge and some bleeding edge technologies it becomes much more compelling to implement a one vendor solution. Further, since the number of staff at the University is limited, calling on the vendor for technical support during the testing, installation and post sales time periods will be required. This approach also alleviates the finger pointing when something goes wrong. Therefore, the One Vendor Concept is a more acceptable solution, and the one that we would recommend.
With a one vendor solution, the University will be able to share products across campus. This will become evident in stocking spare parts, which can help with the overall cost of the system. Further, as the networks in each building become too full with either ports or bandwidth consumption, the University can easily change the connectivity hierarchy of a device. Devices that were originally installed as an "edge switch" can be easily converted to a "building switch". This is accomplished by adding a higher bandwidth uplink port. The reverse can also be true, as the situation may arise that a building becomes less dense, although this is unlikely.
In the original design of the new ISUnet, we see that the best of both worlds has been included in the design: redundancy and spares. In the core, we see a design which is truly redundant. This redundancy allows any one quadrant to go offline, not affecting the backbone. The University must insure that the redundant fiber takes different pathwaysand that the pathways look like a meshed ring. All too often redundant fiber is placed in the same sheath along the same path.
Each of the edge devices are semi-redundant, because they provide for power redundancy, but use the hot spare methodology to keep the system running. The plan for dual homing (connecting the edge device to two different core switches) should be implemented in the initial phases and not considered an upgrade.
Each building switch redundancy is provided for with power redundancy and hot spares. This is enough for most environments, such as dorms, but for areas other than dorms, it should be considered that a second fiber, not in the same path, be run to the edge devices.
The LANE, MPOA and DDNS servers should be redundant and the server farm switch should be dual homed.
Since ISUnet is already set as a distributed model, a network systems manager must plan for the unknown. The manager has no idea if, when, or how much bandwidth is needed. At any time, a group may bring up a system that is not placed in the most desirable area (pertaining to network flow), yet, they must be accommodated. If a system is put on line that requires all of the bandwidth or services available, the manager must provide the services required for that area, and all other areas that it will affect. In the case of ISUnet, bandwidth and quality of service must be available at a moments notice, from any area, to any area.
ISUnet will be providing "any to any" connectivity. The concept of a core router is, that all traffic, having to be routed, will come to the core and them be redistributed. In the ISUnet environment, that would be a waste of bandwidth. Considering "any to any" traffic, the University must allow the traffic flows and the horsepower of the router, to dictate the placement of these routers. The University should strive to deploy all services as close to the end user as possible, therefore, provide the fastest possible, and least congested path for their traffic. Further, most routers today can not keep up with the traffic flows of the core.
The University must be very careful in picking its technology. The situations where the best technology did not prevail has happened again and again. For example, many people would agree that IBM’s OS/2 was a better desktop environment than Windows 3.0, but Windows prevailed. This shows that marketing and third party support MUST be taken into account, regardless of a technology’s merits. This is another area where a one vendor solution can be used. The University should investigate writing into the contract that the vendor has major responsibility to guarantee that the leading/bleeding edge technologies will work or your money will be refunded.
In the ATM versus Ethernet argument, we can clearly see that to accommodate the Vision 2000 and other requirements, only ATM can meet those needs. If the University is to run these applications and add voice to the mix, vast gains will be realized with this technology.
The same hold true for Tactical versus Strategic, ATM is the definitive answer. If the University were to choose to go Tactical, it would not be able to provide all the services required, day one, as stated in the Vision 2000 Plan.
In conclusion, the Core Upgrade Project put together by the Networking Systems Group at ISU is a well thought out, well researched project with excellent viability. It is a very aggressive plan in implementing a state of the art system to meet the demands of running voice, video and data over it. Because it is an aggressive plan, it is important to remember that some of the technology being used in this project is still evolving. Therefore, with careful administration of the project, flexibility with minor design changes needed along the way, and careful testing at each stage, there is no doubt that this project, with a few changes along the way, can be implemented as designed.
Richard J. Scafuto is uniquely qualified to consult on this project for the following reasons: