Date: Sat, 11 Feb 2006 17:48:04 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: dima <_pppp@mail.ru> Cc: Marcos Bedinelli <bedinelli@madhaus.cns.utoronto.ca>, freebsd-net@freebsd.org Subject: Re: Network performance in a dual CPU system Message-ID: <20060211174648.X90460@fledge.watson.org> In-Reply-To: <E1F7fzS-000GUW-00._pppp-mail-ru@f16.mail.ru> References: <E1F7fzS-000GUW-00._pppp-mail-ru@f16.mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 11 Feb 2006, dima wrote: >> The system is mainly being used as a dedicated router. It runs OSPF, BGP >> and IPFW (around 150 rules). OSPF and BGP are managed by Quagga. The box >> has 2 gigabit interfaces that handle on average 200Mbp/s - 50K packets/s >> (inbound and outbound combined), each one of them. > > The second CPU wouldn't help you for sure. There's only one [swi1: net] kernel thread which deals with all the kernel traffic. The option of per-CPU [swi: net] threads was discussed on freebsd-arch@ several months ago, but it wouldn't be implemented soon. So, the only hardware option is installing the fastest CPU possible. > There are several software (FreeBSD specific) options though: If you set net.isr.direct=1, the netisr workload is moved from the netisr thread to the thread performing the dispatch -- typically, the ithread. If you have multiple interfaces and they are assigned different ithreads, then the work can occur in parallel. However, there are some other properties of this setting that are important, so it affects different workloads in different ways. Robert N M Watson > 1. You should surely try polling(4). 50kpps mean 50000 interrupts and the same amount of context switches, which are quite expensive. > 2. FastForwarding. It's the most suitable for you. As I know, Quagga inserts its dynamic routes to the system routing table. And FastForwarding is aware of routing table and firewall rules. And the most exciting: you can switch it on/off without reboot: > # sysctl net.inet.ip.fastforwarding=1 > The only limitation is it applies to IPv4 unicast traffic only. There's no documentation on this feature as i know (am I wrong, or should I report this as a documentation bug?) but you can look at the comments in the beginning of /sys/netinet/ip_fastfwd.c > The authors reported up to 1Mpps (see page 10 at http://people.freebsd.org/~andre/FreeBSD-5.3-Networking.pdf) > >> >> >> Some of you have asked for the following information: >> >> >> - As I indicated before, polling is currently disabled. >> >> >> - Hyperthreading (HTT) is disabled. >> >> >> mull [~]$vmstat -i >> interrupt total rate >> irq1: atkbd0 3466 0 >> irq6: fdc0 10 0 >> irq13: npx0 1 0 >> irq14: ata0 47 0 >> irq21: fxp1 20462527 8 >> irq28: bge0 3511765157 1444 >> irq29: bge1 3633124373 1494 >> irq30: aac0 1842472 0 >> cpu0: timer 566751007 233 >> Total 7733949060 3181 >> >> >> mull [~]$netstat -m >> 644/646/1290 mbufs in use (current/cache/total) >> 643/407/1050/17088 mbuf clusters in use (current/cache/total/max) >> 0/5/4528 sfbufs in use (current/peak/max) >> 1447K/975K/2422K bytes allocated to network (current/cache/total) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> 0 calls to protocol drain routines >> >> >> >> Thank you, >> >> -- >> Marcos > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060211174648.X90460>