From owner-freebsd-net@FreeBSD.ORG Wed Jan 21 01:45:06 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F2B316A4CF for ; Wed, 21 Jan 2004 01:45:06 -0800 (PST) Received: from ns2.alphaque.com (ns2.alphaque.com [202.75.47.153]) by mx1.FreeBSD.org (Postfix) with SMTP id C3E8D43D53 for ; Wed, 21 Jan 2004 01:44:57 -0800 (PST) (envelope-from dinesh@alphaque.com) Received: (qmail 52960 invoked by uid 0); 21 Jan 2004 09:44:51 -0000 Received: from lucifer.net-gw.com (HELO prophet.alphaque.com) (202.75.47.153) by lucifer.net-gw.com with SMTP; 21 Jan 2004 09:44:51 -0000 Received: from localhost (localhost.alphaque.com [127.0.0.1]) by prophet.alphaque.com (8.12.10/8.12.9) with ESMTP id i0L9i4DQ000711 for ; Wed, 21 Jan 2004 17:44:04 +0800 (MYT) (envelope-from dinesh@alphaque.com) Date: Wed, 21 Jan 2004 17:44:04 +0800 (MYT) From: Dinesh Nair To: freebsd-net@freebsd.org In-Reply-To: <20040121023111.X312-100000@prophet.alphaque.com> Message-ID: <20040121170911.Y532-100000@prophet.alphaque.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: Two ISP lines X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jan 2004 09:45:06 -0000 On Wed, 21 Jan 2004, Dinesh Nair wrote: > will do. i'm on 4.9-STABLE. will try the following in this order: > > 1. the multipath patches > 2. security/pf port > 3. rewriting ng_ether to plonk in the new ip address directly into the > mbuf. along with ng_one2many, this should provide the same > functionality. i downloaded the multipath patches, and applied them to 4.9-STABLE. by itself, the kernel didn't compile, and broke in if_ether.c due to the added argument list into rtrequest(). this was easily remedied with an #ifdef, and after a new kernel and new userland route and netstat binaries followed by a reboot, i attempted to test out multipath routing over two interfaces, fxp0 and aue0 (see below). i added a default route using both gateways like so: route add default -gateway 10.1.1.1 -gateway 192.168.0.1 and the routing table (netstat -rn) was as follows: Destination Gateway Flags/ Refs/ Use Netif Expire metric left default UGSc 0 0 aue0 192.168.0.1 10 0 fxp0 *10.1.1.1 10 0 0 aue0 10.1/16 link#8 UC 1 0 aue0 10.1.1.1 00:00:1c:d5:03:63 UHLW 2 2 aue0 1115 127.0.0.1 127.0.0.1 UH 0 362 lo0 192.168.0 link#2 UC 1 0 fxp0 192.168.0.1 00:30:ab:10:6c:0d UHLW 1 0 fxp0 981 i then attempted a tcp connection to an external machine as well as snooping on both interfaces and came to this observation. while this multipath patch does round robin the packets (according to pathmetric) across the two gateways, it does not change the source ip address of the packet to correspond with the interface it went out on. as such, the round robin works this way. the first 10 packets are sent with a src ip of 192.168.0.5 (fxp0's address) out via fxp0. the next 10 are sent still with a src ip of 192.168.0.5 out via aue0. then the cycle repeats with 10 more being sent out via fxp0. this magic number 10 is equivalent to the -pathmetric argument given when creating the route. the default, obviously, is 10. but i digress. this obviously doesn't work for me, since higher upstream from aue0 (ip address 10.1.105.26), the src address of 192.168.0.5 is address translated to a public IP address. the router/natd which does that will obviously not have any address translation maps for 192.168.0.5, and thus silently drop the packet. packets going out fxp0 work the way they should. after all, why shouldn't they ? :) it would however work if both the interfaces were assigned a public ip address, though all this still would do is to round robin outgoing packets, but incoming packets will still come down the same interface (due to the src address not changing). i'm guessing the the picked source ip address depends on the current active gateway for that route, as i've seen the same happen in reverse with a source ip address of 10.1.105.26, the address on aue0. which leads me to deduce that the src address is picked on a per connection basis, depending on the current active gateway, and held for the duration of the connection. thus this does not ensure proper load balancing over both links, even if public ip addresses were used. this does not do what i want to do, i.e. multipath routing in a round robin fashion over x number of interfaces with src ip address changed to the address of the interface the packet goes out on. this would ensure that the reply packets from the destination are routed back in correctly. i'm probably going to experiment with creating a new netgraph node type to do this. looking thru /usr/src/sys/netgraph, and reading the ng_one2many(4) man page, i think it would be possible to base a new node type on one2many which takes input from one interface (hooked to the upper hook of the ether node) and writes to the lower nodes of the interfaces we want to round robin multipath over. in between reading from one and writing to the lower interfaces, i'd need to figure out how to do the following: for outgoing packets: 1. modify the passed in mbuf to change the src ip address of the packet from what it is to what is currently tagged on the chosen outgoing interface. 2. modify the ethernet dest address of the mbuf to point to the next hop router for the chosen outgoing interface for incoming packets: 3. modify the destination ip address on incoming packets to what is currently tagged on the interface hooked to the one hook. conceptually, you'd have just one default route pointing to one interface, and the netgraph nodes would take care of the round robin and packet delivery. i'm new to netgraph, though i think i understand how it works and have read the source in /usr/src/sys/netgraph. i'm absolutely new to mbuf manipulation however. any pointers in this endeavour will be much appreciated, as well as any gotchas i'd need to watch out for. also, would the method i'm proposing above be the way to do it ? Regards, /\_/\ "All dogs go to heaven." dinesh@alphaque.com (0 0) http://www.alphaque.com/ +==========================----oOO--(_)--OOo----==========================+ | for a in past present future; do | | for b in clients employers associates relatives neighbours pets; do | | echo "The opinions here in no way reflect the opinions of my $a $b." | | done; done | +=========================================================================+