From owner-freebsd-net@FreeBSD.ORG Fri Feb 10 21:39:45 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1FCDE16A420 for ; Fri, 10 Feb 2006 21:39:45 +0000 (GMT) (envelope-from _pppp@mail.ru) Received: from f16.mail.ru (f16.mail.ru [194.67.57.46]) by mx1.FreeBSD.org (Postfix) with ESMTP id AC11D43D46 for ; Fri, 10 Feb 2006 21:39:44 +0000 (GMT) (envelope-from _pppp@mail.ru) Received: from mail by f16.mail.ru with local id 1F7fzS-000GUW-00; Sat, 11 Feb 2006 00:39:42 +0300 Received: from [85.140.69.145] by koi.mail.ru with HTTP; Sat, 11 Feb 2006 00:39:42 +0300 From: dima <_pppp@mail.ru> To: Marcos Bedinelli Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [85.140.69.145] Date: Sat, 11 Feb 2006 00:39:42 +0300 In-Reply-To: <711b7ec873f31bc5be50ce477313fac3@madhaus.cns.utoronto.ca> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-net@freebsd.org Subject: Re: Network performance in a dual CPU system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2006 21:39:45 -0000 > Hello all, > > thanks for the replies. Most of you have suggested that I turn on > polling and give it a try. The machine is in production, hence I need > to schedule downtime for that. > > The system is mainly being used as a dedicated router. It runs OSPF, > BGP and IPFW (around 150 rules). OSPF and BGP are managed by Quagga. > The box has 2 gigabit interfaces that handle on average 200Mbp/s - 50K > packets/s (inbound and outbound combined), each one of them. The second CPU wouldn't help you for sure. There's only one [swi1: net] kernel thread which deals with all the kernel traffic. The option of per-CPU [swi: net] threads was discussed on freebsd-arch@ several months ago, but it wouldn't be implemented soon. So, the only hardware option is installing the fastest CPU possible. There are several software (FreeBSD specific) options though: 1. You should surely try polling(4). 50kpps mean 50000 interrupts and the same amount of context switches, which are quite expensive. 2. FastForwarding. It's the most suitable for you. As I know, Quagga inserts its dynamic routes to the system routing table. And FastForwarding is aware of routing table and firewall rules. And the most exciting: you can switch it on/off without reboot: # sysctl net.inet.ip.fastforwarding=1 The only limitation is it applies to IPv4 unicast traffic only. There's no documentation on this feature as i know (am I wrong, or should I report this as a documentation bug?) but you can look at the comments in the beginning of /sys/netinet/ip_fastfwd.c The authors reported up to 1Mpps (see page 10 at http://people.freebsd.org/~andre/FreeBSD-5.3-Networking.pdf) > > > Some of you have asked for the following information: > > > - As I indicated before, polling is currently disabled. > > > - Hyperthreading (HTT) is disabled. > > > mull [~]$vmstat -i > interrupt total rate > irq1: atkbd0 3466 0 > irq6: fdc0 10 0 > irq13: npx0 1 0 > irq14: ata0 47 0 > irq21: fxp1 20462527 8 > irq28: bge0 3511765157 1444 > irq29: bge1 3633124373 1494 > irq30: aac0 1842472 0 > cpu0: timer 566751007 233 > Total 7733949060 3181 > > > mull [~]$netstat -m > 644/646/1290 mbufs in use (current/cache/total) > 643/407/1050/17088 mbuf clusters in use (current/cache/total/max) > 0/5/4528 sfbufs in use (current/peak/max) > 1447K/975K/2422K bytes allocated to network (current/cache/total) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > > > Thank you, > > -- > Marcos