From owner-freebsd-net@FreeBSD.ORG Fri Sep 7 19:38:35 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 385CE16A418; Fri, 7 Sep 2007 19:38:35 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from noop.in-addr.com (unknown [IPv6:2001:5c0:8fff:fffe::214d]) by mx1.freebsd.org (Postfix) with ESMTP id DC8AD13C4A7; Fri, 7 Sep 2007 19:38:34 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from uucp by noop.in-addr.com with local-rmail (Exim 4.54 (FreeBSD)) id 1ITjew-000JRW-En; Fri, 07 Sep 2007 15:38:30 -0400 Received: from localhost.home.in-addr.com ([127.0.0.1]:62955 helo=rimmer.home.in-addr.com) by rimmer.home.in-addr.com with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1ITjel-0002Jm-Nx; Fri, 07 Sep 2007 20:38:19 +0100 Message-ID: <46E1A8AB.7070708@freebsd.org> Date: Fri, 07 Sep 2007 20:38:19 +0100 From: Gary Palmer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.6) Gecko/20070808 SeaMonkey/1.1.4 MIME-Version: 1.0 To: Kirc Gover References: <517436.12027.qm@web44802.mail.sp1.yahoo.com> In-Reply-To: <517436.12027.qm@web44802.mail.sp1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: OS choice for an edge router X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Sep 2007 19:38:35 -0000 Kirc Gover wrote: > Hi Gary, Thanks for your response. > > Yes, the bus architecture would be either PCI-X or PCI-Express. Host CP= U would be a high performance multi-core such as Xeon and NICs would be I= ntel. > > One of my concern is on the native forwarding capability of FreeBSD OS = and the execution of critical userland processes. I have experience befor= e that a FreeBSD box configured as router appears to slow down the userla= nd processes when the traffic load is high. I have verified this lately o= n 6.1, running on Athlon64 with 1G NIC cards with PF and ALTQ (queuing) e= nabled. I'm not so sure if this is caused by PF or ALTQ. Looking at the = processes using top, it could see that "swi(x) net" process is almost as = near 100% cpu utilization. At this state, the box can still forward traff= ic (not sure yet if the was a change in throughput) but I could notice th= e userland processes to be very slow, like invoking any command from the= shell (e.g. ls) will take so long to be executed and completed. Is this = a know limitation or bug? > =20 I'm probably not the best person to address that, but its my=20 understanding that interrupts will preempt other activity such as=20 userland processing. Since there is often a requirement to service=20 interrupts as a matter of priority (e.g. before the receive buffer in=20 the NIC overflows) I'm not sure theres a way around it. The performance = improvements in 7.x, especially relating to multi-core/CPU environments=20 might help. The complexity of the PF / ALTQ rules can also have an=20 impact, although I'm a little surprised that they counted towards=20 interrupt activity, which your message seems to imply. Gary P.S. Please don't top post. > Thanks a lot. > Kirc > > Gary Palmer wrote: On Fri, Sep 07, 2007 at 03:06:= 54AM +1000, Kirc Gover wrote: > =20 >> We are in the stage of planning and research for a commercial developm= ent of an edge router that will be based mostly on OpenSource software. I= would like to solicit for information and recommendation if FreeBSD is a= suitable OS. The router is expected to withstand forwarding of sustained= traffic from 10Mbps to 1Gbps and maybe more than that. Are there any kno= wn limitations of FreeBSD in terms of architecture and performance? Can I= just take out a FreeBSD as is and put it with the hardware without any s= pecific or major refinements in its code? I'm very much concerned with i= ts capability in forwarding heavy sustained traffic. Packet loss should b= e at minimum and critical userland processes should working normally eve= n under heavy load. Are there any known specific limitations of FreeBSD? = I have browsed through the archives and found a lot of hangups, deadlocks= and freeze issues. What is the usual or minimum hardware requirement? Is= soekris box enough, or dual core or ASIC >> based platforms? I'm aware that there are so many FreeBSD based route= rs and network based devices in the market. Is this a way to go over real= time and embedded OS such as VxWorks and others (mostly commercial) witho= ut putting the licensing cost in picture? I really appreciate any help, s= uggestions and recommendations. More power to FreeBSD! >> =20 > > Kirc, > > There are two factors to consider: > > - bus architecture (PCI, PCI-X, PCI-Express, etc) will dictate the > maximum throughput in bits/sec. Allow some overhead for bus arbitrat= ion > activities, and remember that the packet crosses the bus twice, once > on the inbound leg and once on the outbound leg. > > - Host CPU (and perhaps to a limited extent the interface cards used) > will dictate the packets per second (PPS) > > Most commercial routers run out of packets per second (in real-world > situations, not lab mockups) long before the theoretical maximum > throughput is achieved. Thinks like ICMP ping packets and TCP RST > packets are small (less than 100 bytes usually) but normally take as mu= ch > CPU to process & route as a 1500 byte (or larger) packet. > > The more you put in the processing path (e.g. packet filters, complex > routing tables) the more you reduce the PPS. > > Gary > > > =20 > --------------------------------- > Sick of deleting your inbox? Yahoo!7 Mail has free unlimited storage. G= et it now. > =20