Inter-Domain Policy Routing Working Group M. Lepp and M. Steenstrup Internet Draft BBN Systems and Technologies June 1992 Expires 15 December 1992 An Architecture for Inter-Domain Policy Routing Status of this Memo This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a ``working draft'' or ``work in progress''. Please check the 1id-abstracts.txt listing contained in the internet-drafts Shadow Directories on nic.ddn.mil, nnsc.nsf.net, nic.nordu.net, ftp.nisc.sri.com, or munnari.oz.au to learn the current status of any Internet Draft. This Internet Draft will be submitted to the RFC editor as an architecture specification. Distribution of this Internet Draft is unlimited. Please send comments to idpr-wg@bbn.com. Abstract We present an architecture for inter-domain policy routing (IDPR). The objective of IDPR is to construct and maintain routes, between source and destination administrative domains, that provide user traffic with the requested services within the constraints stipulated for the domains transited. The IDPR architecture is designed to accommodate an internetwork containing tens of thousands of administrative domains with heterogeneous service requirements and restrictions. Internet Draft IDPR Architecture June 1992 Contributors The following people have contributed to the IDPR architecture: Bob Braden, Lee Breslau, Ross Callon, Noel Chiappa, Dave Clark, Pat Clark, Deborah Estrin, Marianne Lepp, Mike Little, Martha Steenstrup, Zaw-Sing Su, Paul Tsuchiya, and Gene Tsudik. Yakov Rehkter supplied many useful comments on a previous draft of this document. 1 Internet Draft IDPR Architecture June 1992 Contents 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 The Internet Environment. . . . . . . . . . . . . . . . . . . . 2 2 Approaches to Policy Routing. . . . . . . . . . . . . . . . . . . . 3 2.1 Policy Route Generation. . . . . . . . . . . . . . . . . . . . .3 2.1.1 Distance Vector Approach. . . . . . . . . . . . . . . . . 3 2.1.2 Link State Approach. . . . . . . . . . . . . . . . . . . .5 2.2 Routing Information Distribution. . . . . . . . . . . . . . . . 5 2.2.1 Distance Vector Approach. . . . . . . . . . . . . . . . . 6 2.2.2 Link State Approach. . . . . . . . . . . . . . . . . . . .7 2.3 Message Forwarding along Policy Routes. . . . . . . . . . . . . 8 2.3.1 Hop-by-Hop Approach. . . . . . . . . . . . . . . . . . . .8 2.3.2 Source Specified Approach. . . . . . . . . . . . . . . . .9 3 The IDPR Architecture. . . . . . . . . . . . . . . . . . . . . . . 11 3.1 IDPR Functions. . . . . . . . . . . . . . . . . . . . . . . . .11 3.2 IDPR Entities. . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2.1 Path Agents. . . . . . . . . . . . . . . . . . . . . . . 13 3.2.2 IDPR Servers. . . . . . . . . . . . . . . . . . . . . . .14 3.2.3 Entity Identifiers. . . . . . . . . . . . . . . . . . . .16 3.3 Security and Reliability. . . . . . . . . . . . . . . . . . . .16 3.3.1 Retransmissions and Acknowledgements. . . . . . . . . . .17 3.3.2 Integrity Checks. . . . . . . . . . . . . . . . . . . . .17 3.3.3 Source Authentication. . . . . . . . . . . . . . . . . . 17 3.3.4 Timestamps. . . . . . . . . . . . . . . . . . . . . . . .18 i Internet Draft IDPR Architecture June 1992 3.4 An Example of IDPR Operation. . . . . . . . . . . . . . . . . .18 4 Accommodating a Large, Heterogeneous Internet. . . . . . . . . . . 22 4.1 Domain Level Routing. . . . . . . . . . . . . . . . . . . . . .22 4.2 Route Generation. . . . . . . . . . . . . . . . . . . . . . . .23 4.3 Super Domains. . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4 Domain Communities. . . . . . . . . . . . . . . . . . . . . . .26 4.5 Robustness in the Presence of Failures. . . . . . . . . . . . .26 4.5.1 Path Repair. . . . . . . . . . . . . . . . . . . . . . . 27 4.5.2 Partitions. . . . . . . . . . . . . . . . . . . . . . . .28 ii Internet Draft IDPR Architecture June 1992 1 Introduction As data communications technologies evolve and user populations grow, the demand for internetworking increases. Internetworks usually proliferate through interconnection of autonomous, heterogeneous networks administered by separate authorities. We use the term administrative domain (AD) to refer to any collection of contiguous networks, gateways, links, and hosts governed by a single administrative authority who selects the intra-domain routing procedures and addressing schemes, defines service requirements for locally-generated traffic, and specifies service restrictions for transit traffic. Interconnection of administrative domains can broaden the range of services available in an internetwork. Hence, traffic with special service requirements is more likely to receive the service requested. However, administrators of domains offering special transit services are more likely to establish stringent access restrictions, in order to maintain control over the use of their domains' resources. An internetwork composed of many domains with diverse service requirements and restrictions requires policy routing to transport traffic between source and destination. Policy routing constitutes route generation and message forwarding procedures for producing and using routes that simultaneously satisfy user service requirements and respect transit domain service restrictions. With policy routing, each domain administrator sets transit policies that dictate how and by whom the resources within its domain should be used. Transit policies are usually public, and they specify offered services comprising: Access restrictions: e.g., applied to traffic to or from certain domains or classes of users. Quality: e.g., delay, throughput, or error characteristics. Monetary cost: e.g., charge per byte, message, or unit time. Each domain administrator also sets source policies for traffic originating within its domain. Source policies are usually private, and they specify requested services comprising: Access restrictions: e.g., domains to favor or avoid in routes. Quality: e.g., acceptable delay, throughput, or reliability. 1 Internet Draft IDPR Architecture June 1992 Monetary cost: e.g., acceptable session cost. In this document, we describe an architecture for inter-domain policy routing (IDPR), and we provide a set of functions which can form the basis for a suite of IDPR protocols. 1.1 The Internet Environment The Internet currently comprises over 4000 operational networks and over 10,000 registered networks. In fact, for the last several years, the number of constituent networks has approximately doubled annually. Although we do not expect the Internet to sustain this growth rate, we must provide an architecture for IDPR that can accommodate the Internet five to ten years in the future. According to the functional requirements for inter-autonomous system (i.e., inter-domain) routing set forth in [6], the IDPR architecture and protocols must be able to handle O(100,000) networks distributed over O(10,000) domains. Internet connectivity has increased along with the number of component networks. In the early 1980s, the Internet was purely hierarchical, with the ARPANET as the single backbone. The current Internet possesses a semblance of a hierarchy in the collection of backbone, regional, metropolitan, and campus domains that compose it. However, technological, economical, and political incentives have prompted the introduction of inter-domain links outside of those in the strict hierarchy. Hence, the Internet has the properties of both hierarchical and mesh connectivity. We expect that the Internet will evolve in the following way. Over the next five years, the Internet will grow to contain O(10) backbone domains, most providing connectivity between many source and destination domains and offering a wide range of qualities of service, for a fee. Most domains will connect directly or indirectly to at least one Internet backbone domain, in order to communicate with other domains. In addition, some domains may install direct links to their most favored destinations. Domains at the lower levels of the hierarchy will provide some transit service, limited to traffic between selected sources and destinations. However, the majority of Internet domains will be stubs, that is, domains that do not provide any transit service for other domains. The bulk of Internet traffic will be generated by hosts in these stub domains, and thus, the applications running in these hosts will determine the traffic service requirements. We expect application diversity encompassing electronic mail, desktop videoconferencing, scientific visualization, and distributed simulation, to list a few. Many of these applications have strict requirements on 2 Internet Draft IDPR Architecture June 1992 delivery, delay, and throughput. Ensuring that Internet traffic traverses routes that provide the required services without violating domain usage restrictions will be the task of policy routing in the Internet in the next several years. Refer to [1]- [10] for more information on the role of policy routing in the Internet. 3 Internet Draft IDPR Architecture June 1992 2 Approaches to Policy Routing In this section, we provide an assessment of candidate approaches to policy routing, concentrating on the distance vector and link state alternatives for routing information distribution and route generation and on the hop-by-hop and source specified alternatives for data message forwarding. The IDPR architecture supports link state routing information distribution and route generation in conjunction with source specified message forwarding. We justify these choices for IDPR below. 2.1 Policy Route Generation We present policy route generation from the distance vector perspective and from the link state perspective. 2.1.1 Distance Vector Approach Distance vector route generation distributes the computation of a single route among multiple routing entities along the route. Hence, distance vector route generation is potentially susceptible to the problems of routing loop formation and slow adaptation to changes in an internetwork. However, there exist several techniques that can be applied during distance vector route generation to reduce the severity of, or even eliminate, these problems. For information on a loop-free, quickly adapting distance vector routing procedure, consult [13] and [14]. During policy route generation, each recipient of a distance vector message assesses the acceptability of the associated route and determines the set of neighboring domains to which the message should be propagated. In the context of policy routing, both of the following conditions are necessary for route acceptability: 1. The route is consistent with at least one transit policy for each domain, not including the current routing entity's domain, contained in the route. To enable each recipient of a distance vector message to verify consistency of the associated route with the transit policies of all constituent domains, each routing entity should include its domain's identity and transit policies in each acceptable distance vector message it propagates. 2. The route is consistent with at least one source policy of at least one domain in the Internet. To enable each recipient of a distance vector 4 Internet Draft IDPR Architecture June 1992 message to verify consistency of the associated route with the source policies of particular domains, each domain must provide other domains with access to its source policies. In addition, at least one of the following conditions is necessary for route acceptability: 1. The route is consistent with at least one of the transit policies of the current routing entity's domain. In this case, the routing entity accepts the distance vector message and then proceeds to compare the associated route with its other routes for the given destinations. If the routing entity decides that the new route is preferable, it updates the distance vector message with its domain's identity and transit policies and then propagates the message to the appropriate neighboring domains. We discuss distance vector message distribution in more detail in section ??. 2. The route is consistent with at least one of the source policies of the current routing entity's domain. In this case, the routing entity need not propagate the distance vector message but does retain the associated route for use by traffic, from local hosts, bound for the given destinations. The routing entity discards any distance vector message that does not meet these necessary conditions. With distance vector policy route generation, a routing entity may select and store multiple routes of different characteristics, such as qualities of service, to a single destination. A routing entity uses the quality of service information provided in the transit policies contained in accepted distance vector messages to discriminate between routes based on quality of service. Moreover, a routing entity may select routes that are specific to certain source domains, provided that the routing entity has access to the source policies of those domains. In the distance vector context, the flexibility of policy route generation afforded by accounting for other domains' transit and source policies in route selection has the following disadvantages: 1. Each recipient of a distance vector message must bear the cost of verifying the consistency of the associated route with the constituent domains' transit policies. 2. Source policies must be made public. Thus, a domain must divulge 5 Internet Draft IDPR Architecture June 1992 potentially private information. 3. Each recipient of a distance vector message must bear the potentially high costs of selecting routes for arbitrary source domains. In particular, a routing entity must store the source policies of other domains, account for these source policies during route selection, and maintain source-specific forwarding information. Moreover, there must be a mechanism for distributing source policy information among domains. Depending on the mechanism selected, distribution of source policies may add to the costs paid be each routing entity in supporting source-specific routing. We note, however, that failure to distribute source policies to all domains may have unfortunate consequences. In the worst case, a domain may not learn of any acceptable routes to a given destination, even though acceptable routes do exist. For example, suppose that AD V is connected to AD W and that AD W can reach AD Z through either AD X or AD Y. Suppose also that AD W, as a recipient of distance vector messages originating in AD Z, prefers the route through AD Y to the route through AD X. Furthermore, suppose that AD W has no knowledge of AD V's source policy precluding traffic from traversing AD Y. Hence, AD W distributes to AD V the distance vector message containing the route WYZ but not the distance vector message containing the route WXZ. AD V is thus left with no known route to AD Z, although a viable route traversing AD W and AD X does exist. 2.1.2 Link State Approach Link state route generation permits concentration of the computation of a single route within a single routing entity at the source of the route. In the policy routing context, entities within a domain generate link state messages containing information about the originating domain, including the set of transit policies that apply and the connectivity to adjacent domains, and they distribute these messages to neighboring domains. Each recipient of a link state message stores the routing information for anticipated policy route generation and also distributes it to neighboring domains. Based on the set of link state messages collected from other domains and on its domain's source and transit policies, a routing entity constructs and selects policy routes from its domain to other domains in the Internet. We have selected link state policy route generation for IDPR for the following reasons: 6 Internet Draft IDPR Architecture June 1992 1. Each domain has complete control over policy route generation from the perspective of itself as source. 2. The cost of computing a route is completely contained within the source domain. Hence, routing entities in other domains need not bear the cost of generating policy routes that their domain's local hosts may never use. 3. Source policies may be kept private and hence need not be distributed. Thus, there are no memory, computation, or link bandwidth costs incurred for distributing and storing source policies. 2.2 Routing Information Distribution A domain's routing information and the set of domains to which that routing information is distributed each influence the set of generable policy routes that include the given domain. In particular, a domain administrator may promote the generation of routes that obey its domain's transit policies by ensuring that its domain's routing information: 1. Includes resource access restrictions. 2. Is distributed only to those domains that are permitted to use the resources. Both of these mechanisms, distributing restrictions with and restricting distribution of a domain's routing information, can be applied in both the distance vector and link state contexts. 2.2.1 Distance Vector Approach A routing entity may distribute its domain's resource access restrictions by including the appropriate transit policy information in each distance vector it accepts and propagates. Also, the routing entity may restrict distribution of an accepted distance vector message by limiting the set of neighboring domains to which it propagates the message. In fact, restricting distribution of routing information is inherent in the distance vector approach, as a routing entity propagates only the preferred routes among all the distance vector messages that it accepts. Although restricting distribution of distance vector messages is easy, coordinating restricted distribution among domains may not be. Each domain may have a set of distribution restrictions that apply to all distance 7 Internet Draft IDPR Architecture June 1992 vector messages generated by that domain as well as sets of distribution restrictions that apply to distance vector messages generated by other domains. As a distance vector message propagates among domains, each routing entity should exercise the distribution restrictions associated with each domain constituting the route thus far constructed. In particular, a routing entity should send an accepted distance vector message to a given neighbor, only if distribution of that message to that neighbor is not precluded by any domain contained in the route. To enable a routing entity to exercise these distribution restrictions, each domain must permit other domains access to its routing information distribution restrictions. However, we expect that domains will prefer to keep distribution restictions, like source policies, private. There are at least two ways to make a domain's routing information distribution restrictions generally available to other domains: 1. Prior to propagation of an accepted distance vector message, a routing entity includes in the message its domain's distribution restrictions (all or only those to that apply to the given message). This method requires no additional protocol for disseminating the distribution restrictions, but it may significantly increase the size of and the processing required for each distance vector message. 2. Each domain independently disseminates its distribution restrictions to all other domains, so that each domain will be able to exercise all other domains' distribution restrictions. This method requires an additional protocol for disseminating the distribution restrictions, and it may require a significant amount of memory at each routing entity for storing all domains' distribution restrictions. We note that a domain administrator may describe the optimal distribution pattern of distance vector messages originating in its domain, as a directed graph rooted at its domain. Furthermore, if all domains in the directed graph honor the directionality and if the graph is also acyclic, no routing loops may form, because no two domains are able to exchange distance vector messages pertaining to the same destination. However, an acyclic graph also means that some domains may be unable to discover alternate paths when connectivity between adjacent domains fails, as we show below. We reconsider the example from section 2.2.1. Suppose that the distance vector distribution graph for AD Z is such that all distance vectors 8 Internet Draft IDPR Architecture June 1992 originating in AD Z flow toward AD V. In particular, distance vectors from AD Z enter AD W from AD X and AD Y and leave AD W for AD V. Now, suppose that the link between the AD Z and AD X breaks. AD X no longer has knowledge of any viable route to AD Z, although such a route exists through AD W. To ensure discovery of alternate routes during connectivity failures, the distance vector distribution graph for AD Z must contain bidirectional links between AD W and AD X and between AD W and AD Y. 2.2.2 Link State Approach With link state routing information distribution, all recipients of a domain's link state message gain knowledge of that domain's transit policies and hence service restrictions. For reasons of efficiency or privacy, a domain may also restrict the set of domains to which its link state messages should be distributed. Thus, a domain has complete control over distributing restrictions with and restricting distribution of its routing information. A domain's link state messages automatically travel to all other domains if no distribution restrictions are imposed. Moreover, to ensure that distribution restrictions, when imposed, are applied, the domain may use source specified forwarding of its link state messages, such that the messages are distributed and interpreted only by the destination domains for which they were intended. Thus, only those domains receive the given domain's link state messages and hence gain knowledge of that domain's service offerings. We have selected link state routing information distribution for IDPR for the following reasons: 1. A domain has complete control over the distribution of its own routing information. 2. Routing information distribution restrictions may be kept private and hence need not be distributed. Thus, there are no memory, computation, or link bandwidth costs incurred for distributing and storing distribution restrictions. 9 Internet Draft IDPR Architecture June 1992 2.3 Message Forwarding along Policy Routes To transport data messages along a selected policy route, a routing entity may use either hop-by-hop or source specified message forwarding. 2.3.1 Hop-by-Hop Approach With hop-by-hop message forwarding, each routing entity makes an independent forwarding decision based on a message's source, destination, and requested services and on information contained in the entity's forwarding information database. Hop-by-hop message forwarding follows a source-selected policy route only if all routing entities along the route have consistent routing information and make consistent use of this information when generating and selecting policy routes and when establishing forwarding information. In particular, all domains along the route must have consistent information about the source domain's source policies and consistent, but not necessarily complete, information about transit policies and domain adjacencies within the Internet. In general, this implies that each domain should have knowledge of all other domains' source policies, transit policies, and domain adjacencies. When hop-by-hop message forwarding is applied in the presence of inconsistent routing information, the actual route traversed by data messages not only may differ from the route selected by the source but also may contain loops. In the policy routing context, private source policies and restricted distribution of routing information are two potential causes of routing information inconsistencies among domains. Moreover, we expect routing information inconsistencies among domains in a large Internet, independent of whether the Internet supports policy routing, as some domains may not want or be able to store routing information from the entire Internet. A Clarification. In a previous draft, we presented the following example which results in persistent routing loops, when hop-by-hop message forwarding is used in conjunction with distance vector routing information distribution and route selection. Consider the sequence of events: 10 Internet Draft IDPR Architecture June 1992 1. AD X receives a distance vector message containing a route to AD Z, which does not include AD Y. AD X selects and distributes this route as its primary route to AD Z. 2. AD Y receives a distance vector message containing a route to AD Z, which does not include AD X. AD Y selects and distributes this route as its primary route to AD Z. 3. AD X eventually receives the distance vector message containing the route to AD Z, which includes AD Y but not AD X. AD X prefers this route over its previous route to AD Z and selects this new route as its primary route to AD Z. 4. AD Y eventually receives the distance vector message containing the route to AD Z, which includes AD X but not AD Y. AD Y prefers this route over its previous route to AD Z and selects this new route as its primary route to AD Z. Thus, AD X selects a route to AD Z that includes AD Y, and AD Y selects a route to AD Z that includes AD X. Suppose that all domains along the route selected by AD X, except for AD Y, make forwarding decisions consistent with AD X's route, and that all domains along the route selected by AD Y, except for AD X, make forwarding decisions consistent with AD Y's route. Neither AD X's selected route nor AD Y's selected route contains a loop. Nevertheless, data messages destined for AD Z and forwarded to either AD X or AD Y will continue to circulate between AD X and AD Y, until there is a route change. The reason is that AD X and AD Y have conflicting notions of the route to AD Z, with each domain existing as a hop on the other's route. We note that while BGP-3 [8] is susceptible to such routing loops, BGP-4 [9] is not. We thank Tony Li and Yakov Rekhter for their help in clarifying this difference between BGP-3 and BGP-4. 2.3.2 Source Specified Approach With source specified message forwarding, the source domain dictates the data message forwarding decisions to the routing entities in each intermediate domain, which then forward data messages according to the source specification. Thus, the source domain ensures that any data 11 Internet Draft IDPR Architecture June 1992 message originating within it follows its selected routes. For source specified message forwarding, each data message must carry either an entire source specified route or a path identifier. Including the complete route in each data message incurs a per message transmission and processing cost for transporting and interpreting the source route. Using path identifiers does not incur these costs. However, to use path identifiers, the source domain must initiate, prior to data message forwarding, a path setup procedure that forms an association between the path identifier and the next hop, in the routing entities in each domain along the path. Thus, path setup may impose an initial delay before data message forwarding can begin. We have selected source specified message forwarding for IDPR data messages for the following reasons: 1. Message forwarding respects the source policies of the source domain, regardless of whether intermediate domains along the route have knowledge of these source policies. 2. Message forwarding is loop-free, regardless of whether the all domains along the route maintain consistent routing information. Also, we have chosen path identifiers over complete routes, to affect source specified message forwarding, because of the reduced transmission and processing cost per data message. 12 Internet Draft IDPR Architecture June 1992 3 The IDPR Architecture We now present the architecture for IDPR, including a description of the IDPR functions, the entities that perform these functions, and the features of IDPR that aid in accommodating Internet growth. 3.1 IDPR Functions Inter-domain policy routing comprises the following functions: 1. Collecting and distributing routing information including domain transit policies and inter-domain connectivity. 2. Generating and selecting policy routes based on the routing information distributed and on the source policies configured or requested. 3. Setting up paths across the Internet using the policy routes generated. 4. Forwarding messages across and between domains along the established paths. 5. Maintaining databases of routing information, inter-domain policy routes, forwarding information, and configuration information. 3.2 IDPR Entities From the perspective of IDPR, the Internet comprises administrative domains connected by virtual gateways, which are in turn connected by intra-domain routes supporting the transit policies configured by the domain administrators. Each domain administrator defines the set of transit policies that apply across its domain and the virtual gateways between which each transit policy applies. Several different transit policies may be configured for the intra-domain routes connecting a pair of virtual gateways. Moreover, a transit policy between two virtual gateways may be directional. That is, the transit policy may apply to traffic flowing in one direction, between the virtual gateways, but not in the other direction. Virtual gateways (VGs) are the only connecting points recognized by IDPR between adjacent administrative domains. Each virtual gateway is 13 Internet Draft IDPR Architecture June 1992 actually a collection of directly-connected policy gateways (see below) in two adjacent domains, whose existence has been sanctioned by the administrators of both domains. Domain administrators may agree to establish more than one virtual gateway between their domains. For example, if two domains are to be connected at two geographically distant locations, the domain administrators may wish to preserve these connecting points as distinct at the inter-domain level, by establishing two distinct virtual gateways. Policy gateways (PGs) are the physical gateways within a virtual gateway. Each policy gateway forwards transit traffic according to the service restrictions stipulated by its domain's transit policies applicable to its virtual gateway. A single policy gateway may belong to multiple virtual gateways. Within a domain, two policy gateways are neighbors if they are in different virtual gateways. Within a virtual gateway, two policy gateways are peers if they are in the same domain and are adjacent if they are in different domains. Peer policy gateways must be able to communicate over intra-domain routes that support the transit policies that apply to their virtual gateways. Adjacent policy gateways are directly connected if they are the only Internet addressable entities attached to the connecting medium. Note that this definition implies that not only point-to-point links but also multiaccess networks may serve as direct connections between adjacent policy gateways. Combining multiple policy gateways into a single virtual gateway affords three advantages: 1. A reduction in the amount of IDPR routing information that must be distributed and maintained throughout the Internet. 2. An increase in the reliability of IDPR routes through redundancy of physical connections between domains. 3. An opportunity for load sharing of IDPR traffic among policy gateways. Several different entities are responsible for performing the IDPR functions: 1. Policy gateways collect and distribute routing information, participate in path setup, forward data messages along established paths, and maintain forwarding information databases. 14 Internet Draft IDPR Architecture June 1992 2. Path agents act on behalf of hosts to select policy routes, to set up and manage paths, and to maintain forwarding information databases. 3. Special-purpose servers maintain all other IDPR databases as follows: (a) Each route server is responsible for both its database of routing information, including domain connectivity and transit policy information, and its database of policy routes. Also, each route server generates policy routes on behalf of its domain, using entries from its routing information database and source policy information supplied through configuration or obtained directly from the path agents. (b) Each mapping server is responsible for its database of mappings that resolve Internet names and addresses to administrative domains. (c) Each configuration server is responsible for its database of configured information that applies to policy gateways, path agents, and route servers in the given administrative domain. The configuration information for a given domain includes source and transit policies and mappings between local IDPR entities and their Internet addresses. To maximize IDPR's manageability, one should embed all of IDPR's required functionality within the IDPR protocols and procedures. However, to minimize duplication of implementation effort, one should take advantage of required functionality already provided by mechanisms external to IDPR. Two such cases are the mapping server functionality and the configuration server functionality. The functions of the mapping server can be integrated into an existing name service such as DNS, and the functions of the configuration server can be integrated into the domain's existing network management system. Within the Internet, only policy gateways, path agents, and route servers must be able to generate, recognize, and process IDPR messages. The existence of IDPR is invisible to all other gateways and hosts. Mapping servers and configuration servers perform necessary but ancillary functions for IDPR, and they are not required to execute the IDPR protocols. 15 Internet Draft IDPR Architecture June 1992 3.2.1 Path Agents Any Internet host can reap the benefits of IDPR, as long as there exists a path agent configured to act on its behalf and a means by which the host's messages can reach that path agent. Path agents select and set up policy routes for hosts, accounting for service requirements. To obtain a host's service requirements, a path agent may either consult its IDPR source policy configuration information or extract service requirements directly from the host's data messages, provided such information is available in these data messages. Separating the path agent functions from the hosts means that host software need not be modified to support IDPR. Moreover, it means that a path agent can aggregate onto a single policy route traffic from several different hosts, as long as the source domains, destination domains, and service requirements are the same for all of these host traffic flows. Policy gateways are the natural choice for the entities that perform the path agent functions on behalf of hosts, as policy gateways are the only inter-domain connecting points recognized by IDPR. Each domain administrator determines the set of hosts that its domain's path agents will handle. We expect that a domain administrator will normally configure path agents in its domain to act on behalf of its domain's hosts only. However, a path agent can be configured to act on behalf of any Internet host. This flexibility permits one domain to act as an IDPR proxy for another domain. For example, a small stub domain may wish to have policy routing available to a few of its hosts but may not want to set up its domain to support all of the IDPR functionality. The administrator of the stub domain can negotiate the proxy function with the administrator of another domain, who agrees that its domain will provide policy routes on behalf of the stub domain's hosts. If the source domain supports IDPR and limits all domain egress points to policy gateways, then each message generated by a host in the source domain and destined for a host in another domain must pass through at least one policy gateway, and hence path agent, in the source domain. A host need not know how to reach any policy gateways in its domain; it need only know how to reach a gateway on its own local network. Gateways within the source domain direct inter-domain host traffic toward policy gateways, using default routes or routes derived from 16 Internet Draft IDPR Architecture June 1992 other inter-domain routing procedures. If the source domain does not support IDPR and requires an IDPR proxy domain to provide its hosts with policy routing, the administrator of the source domain must carefully choose the proxy domain. All intervening gateways between hosts in the source domain and path agents in the proxy domain forward traffic according to default routes or routes derived from other inter-domain routing procedures. In order for traffic from hosts in the source domain to reach the proxy domain with no special intervention, the proxy domain must lie on an existing non-IDPR inter-domain route from the source to the destination domain. Hence, to minimize the knowledge a domain administrator must have about inter-domain routes when selecting a proxy domain, we recommend that a domain administrator select its proxy domain from the set of adjacent domains. In either case, the first policy gateway to receive messages from an inter-domain traffic flow originating at the source domain acts as the path agent for the host generating that flow. 3.2.2 IDPR Servers IDPR servers are the entities that manage the IDPR databases and that respond to queries for information from policy gateways or other servers. Each IDPR server may be a dedicated device, physically separate from the policy gateway, or it may be part of the functionality of the policy gateway itself. Separating the server functions from the policy gateways reduces the processing and memory requirements for and increases the data traffic carrying capacity of the policy gateways. The following IDPR databases: routing information, route, mapping, and configuration, may be distributed hierarchically, with partial redundancy throughout the Internet. This arrangement implies a hierarchy of the associated servers, where a server's position in the hierarchy determines the extent of its database. At the bottom of the hierarchy are the local servers that maintain information pertinent to a single domain; at the top of the hierarchy are the global servers that maintain information pertinent to all domains in the Internet. There may be zero or more levels in between the local and global levels. 17 Internet Draft IDPR Architecture June 1992 Hierarchical database organization relieves most IDPR servers of the burden of maintaining information about large portions of the Internet, most of which their clients will never request. Distributed database organization, with redundancy, allows clients to spread queries among IDPR servers, thus reducing the load on any one server. Furthermore, failure to communicate with a given IDPR server does not mean the loss of the entire service, as a client may obtain the information from another server. We note that some IDPR databases, such as the mapping database, may grow so large that it is not feasible to store the entire database at any single server. IDPR routing information databases need not be completely consistent for proper policy route generation and use, because message forwarding along policy routes is completely specified by the source path agent. The absence of a requirement for consistency among IDPR routing information databases implies that there is no requirement for strict synchronization of these databases. Such synchronization is costly in terms of the message processing and transmission bandwidth required. Nevertheless, each IDPR route server should have a query/response mechanism for making its routing information database consistent with that of another route server, if necessary. A route server uses this mechanism to update its routing information database following detection of a gap or potential error in database contents, for example, when the route server returns to service after disconnection from the Internet. A route server in one domain wishing to communicate with a route server in another domain must establish a policy route to the other route server's domain. To generate and establish a policy route, the route server must have sufficient routing information and it must know the other route server's domain. As route servers usually intercommunicate in order to obtain routing information, one might assume an ensuing deadlock in which a route server requires routing and mapping information to establish a policy route but must establish a policy route in order to obtain that routing and mapping information. However, such a deadlock should seldom persist, if the following IDPR functionality is in place: 1. There should be a mechanism that allows a route server to gain access to the identities of the other route servers within its domain, during route server initialization. Using this information, the route server may query these other route servers 18 Internet Draft IDPR Architecture June 1992 in order to update its routing information database. 2. There should also be a mechanism that allows a route server to gain access to its domain's adjacencies, during route server initialization. Using this information, the route server may establish policy routes to the adjacent domains in order to query their route servers for routing information when none is available within its own domain. 3. Once operational, a route server should collect all the routing information to which it has access. A domain usually does not restrict distribution of its routing information but instead distributes its routing information to all other Internet domains. Hence, a route server in a given domain is likely to receive routing information from most Internet domains. 4. There should be a mechanism that allows an operational route server to obtain the identities of external route servers from which it can obtain routing information and of the domains containing these route servers. Furthermore, this mechanism should not require mapping server queries. Rather, each domain should distribute in its routing information messages the identities of all route servers, within its domain, that may be queried by clients outside of its domain. When a host in one domain wishes to communicate with a host in another domain, the path agent in the source domain must establish a policy route to a path agent in the destination domain. However, the source path agent must first query a mapping server, to determine the identity of the destination domain. The queried mapping server may in turn contact other mapping servers to obtain a reply. As with route server communication, one might assume an ensuing deadlock in which a mapping server requires routing and mapping information to establish a policy route but must establish a policy route in order to obtain that routing and mapping information. We have previously described how to minimize the potential for deadlock in obtaining routing information. To minimize the potential for deadlock in obtaining mapping information, there should be a mechanism that allows a mapping server to gain access to the identities of other mapping servers and the domains in which they reside, during mapping server initialization. Hence, no external mapping server queries are required to obtain this information. 19 Internet Draft IDPR Architecture June 1992 3.2.3 Entity Identifiers Each domain has a unique identifier within the Internet, specifically an ordinal number in the enumeration of Internet domains, determined by an Internet coordinator responsible for maintaining such information. Each virtual gateway has a unique local identifier within a domain, derived from the adjacent domain's identifier together with the virtual gateway's ordinal number within an enumeration of the virtual gateways connecting the two domains. The administrators of both domains mutually agree upon the enumeration of the virtual gateways within their shared set of virtual gateways; selecting a single virtual gateway enumeration that applies in both domains eliminates the need to maintain a mapping between separate virtual gateway ordinal numbers in each domain. Each policy gateway and route server has a unique local identifier within its domain, specifically an ordinal number in the domain administrator's enumeration of IDPR entities within its domain. This local identifier, when combined with the domain identifier, produces a unique identifier for the policy gateway or route server within the Internet. 3.3 Security and Reliability The correctness of control information, and in particular routing-related information, distributed throughout the Internet is a critical factor affecting the Internet's ability to transport data. As the number and heterogeneity of Internet domains increases, so too does the potential for both information corruption and denial of service attacks. Thus, we have imbued the IDPR architecture with a variety of mechanisms to: 1. Promote timely delivery of control information. 2. Minimize acceptance and distribution of corrupted control information. 3. Verify authenticity of a source of control information. 4. Reduce the chances for certain types of denial of service attacks. 20 Internet Draft IDPR Architecture June 1992 Consult [11] for a general security architecture for routing and [12] for a security architecture for inter-domain routing. 3.3.1 Retransmissions and Acknowledgements All IDPR entities must make an effort to accept and distribute only correct IDPR control messages. Each IDPR entity that transmits an IDPR control message expects an acknowledgement from the recipient and must retransmit the message up to a maximum number of times when an acknowledgement is not forthcoming. An IDPR entity that receives an IDPR control message must verify message content integrity and source authenticity before accepting, acknowledging, and possibly redistributing the message. 3.3.2 Integrity Checks Integrity checks on message contents promote the detection of corrupted information. Each IDPR entity that receives an IDPR control message must perform several integrity checks on the contents. Individual IDPR protocols may apply more stringent integrity checks than those listed below. The required checks include confirmation of: 1. Recognized message version. 2. Consistent message length. 3. Valid message checksum. Each IDPR entity may also apply these integrity checks to IDPR data messages. Although the IDPR architecture only requires data message integrity checks at the last IDPR entity on a path, it does not preclude intermediate policy gateways from performing these checks as well. 3.3.3 Source Authentication Authentication of a message's source promotes the detection of a rogue entity masquerading as another legitimate entity. Each IDPR entity that receives an IDPR control message must verify the authenticity of 21 Internet Draft IDPR Architecture June 1992 the message source. We recommend that the source of the message supply a digital signature for authentication by message recipients. The digital signature should cover the entire message contents, so that it can serve as the message checksum as well as the source authentication information. Each IDPR entity may also authenticate the source of IDPR data messages; however, the IDPR architecture does not require source authentication of data messages. Instead, we recommend that higher level (end-to-end) protocols, not IDPR, assume the responsibility for data message source authentication, because of the amount of computation involved in verifying a digital signature. 3.3.4 Timestamps Message timestamps promote the detection of out-of-date messages as well as message replays. Each IDPR entity that receives an IDPR control message must check that the message is timely. The IDPR control message must carry a timestamp supplied by the source, which serves to indicate the age of the message. IDPR entities use the absolute value of a timestamp to confirm that the message is current and use the relative difference between timestamps to determine which message contains the most recent information. Hence, all IDPR entities must possess internal clocks that are synchronized to some degree, in order for the absolute value of a message timestamp to be meaningful. The synchronization granularity required by the IDPR architecture is on the order of minutes and can be achieved manually. Any IDPR control message whose timestamp lies outside of the acceptable range may contain stale or corrupted information or may have been issued by a source whose internal clock has lost synchronization with the message recipient's internal clock. IDPR data messages also carry timestamps; however, the IDPR architecture does not require timestamp acceptability checks on IDPR data messages. Instead, we recommend that IDPR entities only check IDPR data message timestamps during problem diagnosis, for example, when checking for suspected message replays. 22 Internet Draft IDPR Architecture June 1992 3.4 An Example of IDPR Operation We illustrate how IDPR works by stepping through an example. In this example, we assume that all domains support IDPR and that all domain egress points are policy gateways. Suppose host HX in domain AD X wants to communicate with host HY in domain AD Y. HX need not know the identity of its own domain or of HY's domain in order to send messages to HY. Instead, HX simply forwards a message bound for HY to one of the gateways on its local network, according to its local forwarding information. If the recipient gateway is a policy gateway, the resident path agent determines how to forward the message outside of the domain. Otherwise, the recipient gateway forwards the message to another gateway in AD X, according to its local forwarding information. Eventually, the message will arrive at a policy gateway in AD X. The path agent resident in the recipient policy gateway uses the message header, including source and destination addresses and any requested service information (for example, type of service), in order to determine whether it is an intra-domain or inter-domain message, and if inter-domain, whether it requires an IDPR policy route. Specifically, the path agent attempts to locate a forwarding information database entry for the given traffic flow. The forwarding information database will already contain entries for all of the following: 1. All intra-domain traffic flows. Intra-domain forwarding information is integrated into the forwarding database as soon as it is received. 2. Inter-domain traffic flows that do not require IDPR policy routes. Non-IDPR inter-domain forwarding information is integrated into the forwarding database as soon as it is received. 3. IDPR inter-domain traffic flows for which a path has already been set up. IDPR forwarding information is integrated into the forwarding database only during path setup. The path agent uses the message header contents to guide the search for a forwarding information database entry for a traffic flow; we suggest a radix search to locate a database entry. When the search terminates, it either produces a forwarding information database entry or a 23 Internet Draft IDPR Architecture June 1992 directive to generate such an entry for an IDPR traffic flow. If the search terminates in an existing database entry, the path agent forwards the message according to that entry. Suppose that the search terminates indicating that the traffic flow between HX and HY requires an IDPR route and that no forwarding information database entry yet exists for this flow. In this case, the path agent first determines the source and destination domains associated with the message's source and destination addresses, before attempting to obtain a policy route. The path agent relies on the mapping servers to supply the domain information, but it caches all mapping server responses locally to limit the number of future queries. When attempting to resolve an address to a domain, the path agent always checks its local cache before contacting a mapping server. After obtaining the source and destination domain information, the path agent attempts to obtain a policy route to carry the traffic from HX to HY. The path agent relies on the route servers to supply policy routes, but it caches all route server responses locally to limit the number of future queries. When attempting to locate a suitable policy route, the path agent consults its local cache before contacting a route server. A policy route contained in the cache is suitable provided that its associated source domain is AD X, its associated destination domain is AD Y, and it satisfies the service requirements specified in the data message or through configuration. If no suitable cache entry exists, the path agent queries the route server, providing it with the source and destination domains together with the service requests carried in the data message or specified through configuration. Upon receiving a policy route query, a route server consults its route database. If it cannot locate a suitable route in its route database, the route server attempts to generate at least one route to domain AD Y, consistent with the requested services for HX. The response to a successful route query consists of a set of candidate routes, from which the path agent makes its selection. We expect that a path agent will normally choose a single route from a candidate set. Nevertheless, the IDPR architecture does not preclude a path agent from selecting multiple routes from the candidate set. A path agent may desire multiple routes to support features such as fault tolerance or load balancing; however, the IDPR architecture does not specify how the path agent should use multiple routes. In any case, a route server 24 Internet Draft IDPR Architecture June 1992 always returns a response to a path agent's query, even if it is not successful in locating a suitable policy route. If the policy route is a new route provided by the route server, there will be no existing path for the route and thus the path agent must set up such a path. However, if the policy route is an existing route extracted from the path agent's cache, there may well be an existing path for the route, set up to accommodate a different host traffic flow. The IDPR architecture permits multiple host traffic flows to use the same path, provided that all flows sharing the path travel between the same endpoint domains and have the same service requirements. Nevertheless, the IDPR architecture does not preclude a path agent from setting up distinct paths along the same policy route to preserve the distinction between host traffic flows. The path agent associates an identifier with the path, which will be included in each message that travels down the path and will be used by the policy gateways along the path in order to determine how to forward the message. If the path already exists, the path agent uses the preexisting identifier. However, for new paths, the path agent chooses a path identifier that is different from those of all other paths that it manages. The path agent also updates its forwarding information database to reference the path identifier and modifies its search procedure to yield the correct forwarding information database entry given the data message header. For new paths, the path agent initiates path setup, communicating the policy route, in terms of requested services, constituent domains, relevant transit policies, and the connecting virtual gateways, to policy gateways in intermediate domains. Using this information, an intermediate policy gateway determines whether to accept or refuse the path and to which policy gateway to forward the path setup information. The path setup procedure allows policy gateways to set up a path in both directions simultaneously. Each intermediate policy gateway, after path acceptance, updates its forwarding information database to include an entry that associates the path identifier with the appropriate previous and next hop policy gateways. Paths remain in place until they are torn down because of failure, expiration, or when resources are scarce, preemption in favor of other paths. When a policy gateway in AD Y accepts a path, it notifies the source path agent in AD X. We expect that the source path agent will normally wait until a path has been successfully established before 25 Internet Draft IDPR Architecture June 1992 using it to transport data traffic. However, the IDPR architecture does not preclude a path agent from forwarding data messages along a path prior to confirmation of successful path establishment. In this case, the source path agent transmits data messages along the path with full knowledge that the path may not yet have been successfully established at all intermediate policy gateways and thus that these data messages will be immediately discarded by any policy gateway not yet able to recognize the path identifier. We note that data communication between HX and HY may occur over two separate IDPR paths: one from AD X to AD Y and one from AD Y to AD X. The reasons are that within a domain, hosts know nothing about policy gateways nor IDPR paths, and policy gateways know nothing about other policy gateways' existing IDPR paths. Thus, in AD Y, the policy gateway that terminates the path from AD X may not be the same as the policy gateway that receives traffic from HY destined for HX. In this case, receipt of traffic from HY forces the second policy gateway to set up a new path from AD Y to AD X. 26 Internet Draft IDPR Architecture June 1992 4 Accommodating a Large, Heterogeneous Internet The IDPR architecture must be able to accommodate an Internet containing O(10,000) domains, supporting diverse source and transit policies. Thus, we have endowed the IDPR architecture with many features that allow it to function effectively in such an environment. 4.1 Domain Level Routing The IDPR architecture provides policy routing among administrative domains. In order to construct policy routes, route servers require routing information at the domain level only; no intra-domain details need be included in IDPR routing information. The size of the routing information database maintained by a route server depends not on the number of Internet gateways, networks, and links, but on how these gateways, networks, and links are grouped into domains and on what services they offer. Therefore, the number of entries in an IDPR routing information database depends on the number of domains and the number and size of the transit policies supported by these domains. Policy gateways distribute IDPR routing information only when detectable inter-domain changes occur and may also elect to distribute routing information periodically (for example, on the order of once per day) as a backup. We expect that a pair of policy gateways within a domain will normally be connected such that when the primary intra-domain route fails, the intra-domain routing procedure will be able to construct an alternate route. Thus, an intra-domain failure is unlikely to be visible at the inter-domain level and hence unlikely to force an inter-domain routing change. Therefore, we expect that policy gateways will not often generate and distribute IDPR routing information messages. IDPR entities rely on intra-domain routing procedures operating within domains to transport inter-domain messages across domains. Hence, IDPR messages must appear well-formed according to the intra-domain routing and addressing procedures in each domain traversed. Recall that source authentication information (refer to section 3.3.3 above) may cover the entire IDPR message. Thus, the IDPR portion of such a message cannot be modified at intermediate domains along the path without causing source authenticity checks to fail. Therefore, at domain boundaries, IDPR messages require encapsulation and decapsulation according to the routing procedures and addressing schemes operating with the given 27 Internet Draft IDPR Architecture June 1992 domain. Only policy gateways and route servers must be capable of handling IDPR-specific messages; other gateways and hosts simply treat the encapsulated IDPR messages like any other message. Thus, for the Internet to support IDPR, only a small proportion of Internet entities require special IDPR software. With domain level routes, many different traffic flows may use not only the same policy route but also the same path, as long as their source domains, destination domains, and service requirements are compatible. The size of the forwarding information database maintained by a policy gateway depends not on the number of Internet hosts but on how these hosts are grouped into domains, which hosts intercommunicate, and on how much distinction a source domain wishes to preserve among its traffic flows. Therefore, the number of entries in an IDPR forwarding information database depends on the number of domains and the number of source policies supported by those domains. Moreover, memory associated with failed, expired, or disused paths can be reclaimed for new paths, and thus forwarding information for many paths can be accommodated in a policy gateway's forwarding information database. 4.2 Route Generation Route generation is the most computationally complex part of IDPR, because of the number of domains and the number and heterogeneity of policies that it must accommodate. Route servers must generate policy routes that satisfy the requested services of the source domain and respect the offered services of the transit domains. We distinguish requested qualities of service and route generation with respect to them as follows: 1. Optimal requested services include minimum route delay, minimum route delay variation, minimum session monetary cost, and maximum available route bandwidth. In the worst case, the computational complexity of generating a route that is optimal with respect to a given requested service is O(N + L) for breadth-first (BF) search and O((N + L) log N) for Dijkstra's shortest path first (SPF) search, where N is the number of nodes and L is the number of links in the search graph. Multi-criteria optimization, for example finding a route with minimal delay variation and minimal session monetary cost, may be defined in several ways. One approach to multi-criteria optimization is to assign each link a 28 Internet Draft IDPR Architecture June 1992 single value equal to a weighted sum of the values of the individual offered qualities of service and generate a route that is optimal with respect to this new criterion. However, selecting the weights that yield the desired route generation behavior is itself an optimization procedure and hence not trivial. 2. Requested service limits include upper bounds on route delay, route delay variation, and session monetary cost and lower bounds on available route bandwidth. Generating a route that must satisfy more than one quality of service constraint, for example route delay of no more than X seconds and available route bandwidth of no less than Y bits per second, is an NP-complete problem. To contain the combinatorial explosion of processing and memory costs associated with route generation, we supply the following guidelines for generation of suitable policy routes: 1. Each route server should only generate policy routes from the perspective of its own domain as source; it need not generate policy routes for arbitrary source/destination domain pairs. Thus, we can distribute the computational burden over all route servers. 2. Route servers should precompute routes for which they anticipate requests and should generate routes on demand only in order to satisfy unanticipated route requests. Hence, a single route server can distribute its computational burden over time. 3. Route servers should cache the results of route generation, in order to minimize the computation associated with responding to future route requests. 4. To handle multi-criteria optimization in route selection, a route server should generate routes that are optimal with respect to the first specified optimal requested service listed in the source policy. The route server should resolve ties between otherwise equivalent routes by evaluating these routes according to the other optimal requested services, in the order in which they are specified. With respect to the route server's routing information database, the selected route is optimal according to the first optimal requested service but is not necessarily optimal according to any other optimal requested service. 5. To handle requested service limits, a route server should always 29 Internet Draft IDPR Architecture June 1992 select the first route generated that satisfies all of the requested service limits. 6. To handle a mixture of requested service limits and optimal requested services, a route server should generate routes that satisfy all of the requested service limits. The route server should resolve ties between otherwise equivalent routes by evaluating these routes as described in the multi-criteria optimization case. 7. All else being equal, a route server should always prefer minimum-hop routes, because they minimize the amount of network resources consumed by the routes. All domains need not execute the identical route generation procedure. Each domain administrator is free to specify the IDPR route generation procedure for route servers in its own domain, making the procedure as simple or as complex as desired. 4.3 Super Domains A super domain is itself an administrative domain, comprising a set of contiguous domains with similar transit policies and formed through consensus of the administrators of the consituent domains. Super domains provide a mechanism for reducing the amount of IDPR routing information distributed throughout the Internet. Given a set of n contiguous domains with consistent transit policies, the amount of routing information associated with the set is approximately n times smaller when the set is considered as a single super domain than when it is considered as n individual domains. When forming a super domain from constituent domains whose transit policies do not form a consistent set, one must determine which transit policies to distribute in the routing information for the super domain. The range of possibilities is bounded by the following two alternatives, each of which reduces the amount of routing information associated with the set of constituent domains: 1. The transit policies supported by the super domain are derived from the union of the access restrictions and the intersection of the qualities of service, over all constituent domains. In this case, the formation of the super domain reduces the number of services 30 Internet Draft IDPR Architecture June 1992 offered by the constituent domains, but guarantees that none of these domains' access restrictions are violated. 2. The transit policies supported by the super domain are derived from the intersection of the access restrictions and the union of the qualities of service. In this case, the formation of the super domain increases the number of services offered by the constituent domains, but requires these domains to relax their access restrictions. Thus, we recommend that domain administrators refrain from arbitrarily grouping domains into super domains, unless they fully understand the consequences. The existence of super domains imposes a hierarchy on domains within the Internet. For model consistency, we assume that there is a single super domain at the top of the hierarchy, which contains the set of all high-level domains. A domain's identity is defined relative to the domain hierarchy. Specifically, a domain's identity may be defined in terms of the domains containing it, the domains it contains, or both. For any domain AD X, the universe of distribution for its routing information usually extends only to those domains contained in AD X's immediate super domain and at the same level of the hierarchy as AD X. However, the IDPR architecture does not preclude AD X from distributing its routing information to domains at arbitrarily high levels in the hierarchy, as long as these domains share a common super domain with AD X. For example, the administrator of an individual domain within a super domain may wish to have one of its transit policies advertised outside of the immediate super domain, so that other domains can take advantage of a quality of service not offered by the super domain itself. In this case, the super domain and the consituent domain may distribute routing information at the same level in the domain hierarchy, even though one domain actually contains the other. We note that the existence of super domains may restrict the number of routes available to source domains with access restrictions. For example, suppose that a source domain AD X has source policies that preclude its traffic from traversing a domain AD Y and that AD Y is contained in a super domain AD Z. If domains within AD Z do not advertise routing information separately, then route servers within AD X do not have enough routing information to construct routes that 31 Internet Draft IDPR Architecture June 1992 traverse AD Z but that avoid AD Y. Hence, route servers in AD X must generate routes that avoid AD Z altogether. 4.4 Domain Communities A domain community is a group of domains to which a given domain distributes routing information, and hence domain communities may be used to limit routing information distribution. Domain communities not only reduce the costs associated with distributing and storing routing information but also allow concealment of routing information from domains outside of the community. Unlike a super domain, a domain community is not necessarily an administrative domain. However, formation of a domain community may or may not involve the consent of the administrators of the member domains, and the definition of the community may be implicit or explicit. Each domain administrator determines the extent of distribution of its domain's routing information and hence unilaterally defines a domain community. By default, this community encompasses all Internet domains. However, the domain administrator may restrict community membership by describing the community as a neighborhood (defined in terms of domain hops) or as a list of member domains. A group of domain administrators may mutually agree on distribution of their domains' routing information among their domains and hence multilaterally define a domain community. By default, this community encompasses all Internet domains. However, the domain administrators may restrict community membership by describing the community as a list of member domains. In fact, this domain community may serve as a multicast group for routing information distribution. 4.5 Robustness in the Presence of Failures The IDPR architecture possesses the following features that make it resistent to failures in the Internet: 1. Multiple connections between adjacent policy gateways in a virtual gateway and between peer and neighbor policy gateways across an administrative domain minimize the number of single component failures that are visible at the inter-domain level. 32 Internet Draft IDPR Architecture June 1992 2. Policy gateways distribute IDPR routing information immediately after detecting a connectivity failure at the inter-domain level, and route servers immediately incorporate this information into their routing information databases. This ensures that new policy routes will not include those domains involved in the connectivity failure. 3. The routing information database query/response mechanism ensures rapid updating of the routing information database for a previously failed route server following the route server's reconnection to the Internet. 4. To minimize user service disruption following a failure in the primary path, policy gateways attempt local path repair immediately after detecting a connectivity failure. Moreover, path agents may maintain standby alternate paths that can become the primary path if necessary. 5. Policy gateways within a domain continuously monitor domain connectivity and hence can detect and identify domain partitions. Moreover, IDPR can continue to operate properly in the presence of partitioned domains. 4.5.1 Path Repair Failure of one or more entities on a given policy route may render the route unusable. If the failure is within a domain, IDPR relies on the intra-domain routing procedure to find an alternate route across the domain, which leaves the path unaffected. If the failure is in a virtual gateway, policy gateways must bear the responsibility of repairing the path. Policy gateways nearest to the failure are the first to recognize its existence and hence can react most quickly to repair the path. Relinquishing control over path repair to policy gateways in other domains may be unacceptable to some domain administrators. The reason is that these policy gateways cannot guarantee construction of a path that satisfies the source policies of the source domain, as they have no knowledge of other domains' source policies. Nevertheless, limited local path repair is feasible, without distributing either source policy information throughout the Internet 33 Internet Draft IDPR Architecture June 1992 or detailed path information among policy gateways in a domain or in a virtual gateway. We say that a path is locally repairable if there exists an alternate route between two policy gateways on the path, separated by at most one policy gateway. This definition covers path repair in the presence of failed routes between consecutive policy gateways as well as failed policy gateways themselves. A policy gateway attempts local path repair, proceeding in the forward direction of the path, upon detecting that the next policy gateway on a path is no longer reachable. The policy gateway must retain enough of the original path setup information to repair the path locally. Using the path setup information, the policy gateway attempts to locate a route around the unreachable policy gateway. Specifically, the policy gateway attempts to establish contact with either: 1. A peer of the unreachable policy gateway. 2. A peer of itself, if the unreachable policy gateway is an adjacent policy gateway and if the given policy gateway no longer has direct connections to any adjacent policy gateways. The contacted policy gateway attempts to locate the next policy gateway following the unreachable policy gateway, on the original path. If successful, the contacted policy gateway informs the requesting policy gateway. In this case, the requesting, contacted, and next policy gateways update their forwarding information databases to conform to the new part of the path. If not successful, the contacted policy gateway initiates teardown of the original path; in this case, the source path agent must find a new route to the destination. 4.5.2 Partitions A domain partition exists whenever there are at least two entities within the domain that can no longer communicate over any intra-domain route. Domain partitions not only disrupt intra-domain communication but also may interfere with inter-domain communication, particularly when the partitioned domain is a transit domain. Therefore, we have designed the IDPR architecture to permit effective use of partitioned domains and hence maximize Internet connectivity in the presence of domain partitions. When a domain is partitioned, it becomes a set of multiple distinct 34 Internet Draft IDPR Architecture June 1992 components. A domain component is a subset of the domain's entities such that all entities within the subset are mutually reachable via intra-domain routes, but no entities in the complement of the subset are reachable via intra-domain routes from entities within the subset. Each domain component has a unique identifier, namely the identifier of the domain together with the ordinal number of the lowest-numbered operational policy gateway within the domain component. No negotiation among policy gateways is necessary to determine the domain component's lowest-numbered operational policy gateway. Instead, within each domain component, all policy gateway members discover mutual reachability through intra-domain reachability information. Therefore, all members have a consistent view of which is the lowest-numbered operational policy gateway in the component. IDPR entities can detect and compensate for all domain partitions that isolate at least two groups of policy gateways from each other. They cannot, however, detect any domain partition that isolates groups of hosts only. Note that a domain partition may segregate portions of a virtual gateway, such that peer policy gateways lie in separate domain components. Although itself partitioned, the virtual gateway does not assume any additional identities. However, from the perspective of the adjacent domain, the virtual gateway now connects to two separate domain components. Policy gateways use partition information to select routes across virtual gateways to the correct domain components. They also distribute partition information to route servers as part of the IDPR routing information. Thus, route servers know which domains are partitioned. However, route servers do not know which hosts reside in which components of a partitioned domain; tracking this information would require extensive computation and communication. Instead, when a route server discovers that the destination of a requested route is a partitioned domain, it attempts to generate a suitable policy route to each component of the destination domain. Generation of multiple routes, on detection of a partitioned destination domain, maximizes the chances of obtaining at least one policy route that can be used for communication between the source and destination hosts. 35 Internet Draft IDPR Architecture June 1992 References [1] Y. Rehkter. EGP and policy based routing in the new NSFNET backbone. IETF RFC 1092. February 1989. [2] D. Clark. Policy routing in internet protocols. IETF RFC 1102. May 1989. [3] H-W. Braun. Models of policy based routing. IETF RFC 1104. June 1989. [4] B. Leiner. Policy issues in interconnecting networks. IETF RFC 1124. September 1989. [5] D. Estrin. Requirements for policy based routing in the research internet. IETF RFC 1125. November 1989. [6] M. Little. Goals and functional requirements for inter-autonomous system routing. IETF RFC 1126. July 1989. [7] J. Honig, D. Katz, M. Mathis, Y. Rekhter, and J. Yu. Application of the border gateway protocol in the internet. IETF RFC 1164. June 1990. [8] K. Lougheed and Y. Rekhter. A border gateway protocol 3 (BGP-3). IETF RFC 1267. October 1991. [9] Y. Rekhter and T. Li, editors. A border gateway protocol 4 (BGP-4). IETF Internet Draft. April 1992. [10] ANSI. Intermediate system to intermediate system inter-domain routeing information exchange protocol. ANSI Document X3S3.3/90-132, June 1990. [11] R. Perlman. Network Layer Protocols with Byzantine Robustness. Ph.D. Thesis. Department of Electrical Engineering and Computer Science. MIT. August 1988. [12] D. Estrin and G. Tsudik. Secure Control of Transit Internetwork Traffic. TR-89-15. Computer Science Department. University of Southern California. To appear in Computer Networks and ISDN Systems, 1991. [13] J.J. Garcia-Luna-Aceves. A unified approach for loop-free routing using link states or distance vectors. ACM Computer Communication Review, Vol. 19, No. 4, SIGCOMM 1989, pp. 212-223. [14] W.T. Zaumen and J.J. Garcia-Luna-Aceves. Dynamics of distributed shortest-path routing algorithms. ACM Computer Communication Review, Vol. 21, No. 4, SIGCOMM 1991, pp. 31-42. 36