From owner-freebsd-net@FreeBSD.ORG Tue Sep 14 07:05:24 2010 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B383106566B; Tue, 14 Sep 2010 07:05:24 +0000 (UTC) (envelope-from dave@seddon.ca) Received: from postfix.xen.seddon.ca (seddon.ca [203.209.212.18]) by mx1.freebsd.org (Postfix) with ESMTP id 98AA58FC08; Tue, 14 Sep 2010 07:05:22 +0000 (UTC) Received: from [127.0.0.1] (unknown [172.16.100.140]) by postfix.xen.seddon.ca (Postfix) with ESMTP id 17117C11D; Tue, 14 Sep 2010 06:32:11 +0000 (UTC) From: Dave Seddon To: Julian Elischer In-Reply-To: <4C8EC845.2060306@elischer.org> References: <1284107762.5923.306.camel@das8530.vic.bigpond.net.au> <532349FC-9269-4674-872F-FA84292E264C@mimectl> <1284130306.6282.6.camel@das8440.seddon.ca> <009101cb5308$514066d0$f3c13470$@com> <1284423495.5238.99.camel@das8530.vic.bigpond.net.au> <4C8EC845.2060306@elischer.org> Content-Type: text/plain; charset="UTF-8" Date: Tue, 14 Sep 2010 16:31:39 +1000 Message-ID: <1284445899.5238.155.camel@das8530.vic.bigpond.net.au> Mime-Version: 1.0 X-Mailer: Evolution 2.28.1 Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Robert Watson , Andrew Hannam , dave@seddon.ca Subject: Re: FreeBSD route tables limited 16? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dave@seddon.ca List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Sep 2010 07:05:24 -0000 Greetings, Thanks for the quick response. It sounds like dedicating some space for this in the mbuf would be the best way forward, but the question is how much. I'm worried that most freebsd users won't go for lots of route tables, which is why you went for 4 bits originally. Within the network service provider space there is frequently a requirement for lots of virtual-routing with MPLS. I imagine there are others in my situation, including vendors and people working on equipment like Cisco/Juniper/Lucatel. Regarding the size to dedicate, the best number might be 12 bits or 4096. This would allow a route table per VLAN on a 802.1q interface. (Actually I'm lying a little because the first and last vlan IDs aren't usable :) ). Perhaps a separate option for non-common users who want many route tables would be best. e.g. GIANT_ROUTETABLES=12 Seems like there would need to be changes in multiple places although perhaps this list isn't exhaustive. So far the files to edit are: /usr/src/sys/net/route.h /sys/sys/mbuf.h Regarding firewalls and these multiple route tables, have you considered having a separate firewall rule table per route table? I haven't looked at the vnet jails, yet. Will do. Thanks. Kind regards, Dave -----Original Message----- From: Julian Elischer To: dave@seddon.ca Cc: Andrew Hannam , FreeBSD Net , Robert Watson Subject: Re: FreeBSD route tables limited 16? Date: Mon, 13 Sep 2010 17:56:37 -0700 Mailer: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.9) Gecko/20100825 Thunderbird/3.1.3 On 9/13/10 5:18 PM, Dave Seddon wrote: > Greetings Julian, > > I've been wondering if it's possible to increase the number of FreeBSD > route tables to a larger number. It seems this is currently 4 bits, > however I was wondering about perhaps 16 bits? Yes the code is designed to handle many more and if you do create more then everything SHOULD handle it. The bottleneck is that we need to store an associated fib with each outgoing (or for that matter incoming) packet, bit we do not at this time want to dedicate a whole word in the mbuf to the task. My "hack" for 8.x (before it was done) was to hide the information in the flags word of the mbuf. I only took 4 bits to make sure I didn't trample on other people's use of bits there. The plan is/was to make a separate entry in the mbuf some time after 7.x branched (say, "now" for example :-) ) you could just steal more bits for now, but if you take 8 bits there will only be one spare. (see /sys/sys/mbuf.h) It may just be time to bite the bullet and steal the entry. Out of curiosity, why do you need > 16 fibs? have you considered using vnet jails a well? > > /* MRT compile-time constants */ > #ifdef _KERNEL > #ifndef ROUTETABLES > #define RT_NUMFIBS 1 > #define RT_MAXFIBS 1 > #else > /* while we use 4 bits in the mbuf flags, we are limited to 16 */ > #define RT_MAXFIBS 16 > #if ROUTETABLES> RT_MAXFIBS > #define RT_NUMFIBS RT_MAXFIBS > #error "ROUTETABLES defined too big" > #else > #if ROUTETABLES == 0 > #define RT_NUMFIBS 1 > #else > #define RT_NUMFIBS ROUTETABLES > #endif > #endif > #endif > #endif > > Really liked your announcement years ago: > http://lists.freebsd.org/pipermail/freebsd-arch/2007-December/007331.html > > Kind regards, > Dave Seddon > +61 447 SEDDON > dave@seddon.ca > > -----Original Message----- > From: Andrew Hannam > To: dave@seddon.ca > Subject: RE: FreeBSD route tables - limited to 16 :( > Date: Mon, 13 Sep 2010 15:55:47 +1000 > Mailer: Microsoft Office Outlook 12.0 > > I think the gentleman is confusing route-tables with routes. > 150K routes is easily possible but it is obvious there is currently only support for up to 16 route tables. > > I think that you are right and the number of bits will need to be updated. > > I don't know the answer to the 'route leaking' question and it has been a long time since I looked at this code. > > You really need to speaking the specialist responsible for the multiple route table code. This person should be clearly marked in the code headers. > > I'm guessing that no-one has thought about using it the way you are planning to use it. > > If I get some time I will have a look - but don't hold your breath. > > Regards, > Andrew. > > -----Original Message----- > From: Dave Seddon [mailto:dave@seddon.ca] > Sent: Saturday, 11 September 2010 12:52 AM > To: Aldous, Matthew D > Cc: dave@seddon.ca; Andrew Hannam; Truman Boyes > Subject: RE: FreeBSD route tables - limited to 16 :( > > Greetings, > > I'm guessing we need to adjust the number of bits defined for the route > table in the mbufs structure definition (where ever that is), then we > can update the route.h to match. > > I guess really we should make the mbufs codes _and_ route.h code pickup > the KERNCONF definition of the variable ROUTETABLES. > > Andrew - thoughts on this? > > I'm not sure if the firewall rules allow you to update the route table > variable in the mbuf, but if it doesn't we should allow this. This > would be equivelant to what they call 'route leaking' in MPLS speak, > when you can pop traffic from one VPN to another (very nasty, but > sometimes handy). yes ipfw does allow you to do this but it needs some more work.. It only really works as the naive user may expect on incoming packets. > > Regards, > Dave > > On Fri, 2010-09-10 at 19:05 +1000, Aldous, Matthew D wrote: >> ________________________________ >> From: Dave Seddon [dave@seddon.ca] >> Sent: Friday, 10 September 2010 6:36 PM >> To: Andrew Hannam >> Cc: dave@seddon.ca; Aldous, Matthew D; Truman Boyes >> Subject: FreeBSD route tables - limited to 16 :( >> >> I just tried compiling up FreeBSD 8.1 with 1024 route tables. It's >> throwing an error, which is tracked down to the >> vi /usr/src/sys/net/route.h (line 99ish). The limit is 16, because as >> the comments say this is 4 bits. Need to look into increasing this to >> say 16 bits :). Given each mbuf will have this, it could cause a >> significant increase in memory usage for a system with a large number of >> packets (although who cares, ram is cheap). >> >> >> /* MRT compile-time constants */ >> #ifdef _KERNEL >> #ifndef ROUTETABLES >> #define RT_NUMFIBS 1 >> #define RT_MAXFIBS 1 >> #else >> /* while we use 4 bits in the mbuf flags, we are limited to 16 */ >> #define RT_MAXFIBS 16 >> #if ROUTETABLES> RT_MAXFIBS >> #define RT_NUMFIBS RT_MAXFIBS >> #error "ROUTETABLES defined too big" >> #else >> #if ROUTETABLES == 0 >> #define RT_NUMFIBS 1 >> #else >> #define RT_NUMFIBS ROUTETABLES >> #endif >> #endif >> #endif >> #endif >> >> > > >