From owner-freebsd-net@FreeBSD.ORG Wed Jan 14 13:25:30 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3E91F16A4CE; Wed, 14 Jan 2004 13:25:30 -0800 (PST) Received: from sizone.org (mortar.sizone.org [65.126.154.242]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E8BE43D66; Wed, 14 Jan 2004 13:25:28 -0800 (PST) (envelope-from dgilbert@daveg.ca) Received: by sizone.org (Postfix, from userid 66) id DB665307D5; Wed, 14 Jan 2004 15:56:43 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id B74911D20A1; Wed, 14 Jan 2004 15:56:39 -0500 (EST) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16389.44295.593077.330791@canoe.dclg.ca> Date: Wed, 14 Jan 2004 15:56:39 -0500 To: Adrian Penisoara In-Reply-To: References: X-Mailer: VM 7.17 under 21.4 (patch 14) "Reasonable Discussion" XEmacs Lucid cc: freebsd-isp@freebsd.org cc: freebsd-net@freebsd.org Subject: Handling 100.000 packets/sec or more X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 21:25:30 -0000 >>>>> "Adrian" == Adrian Penisoara writes: Adrian> Hi, At one site that I administer we have a gateway server Adrian> which services a large SOHO LAN (more than 300 stations) and Adrian> I'm facing a serious issue: very often we see strong spoofed Adrian> floods (variable source IP and port, variable destination IP, Adrian> destination port 80) which can go as far as 100 000 Adrian> packets/sec! Adrian> Of course, the server (FreeBSD 5.2-REL, PIII 733Mhz, 256Mb Adrian> RAM, 3COM 3C905B-TX aka xl0 with checksum offloading support) Adrian> has a hard time swallowing this kind of traffic. The main Adrian> issue are the IRQ interrupts: over 15000 interrupts/sec which Adrian> consume more than 90% of the CPU time. We got ingress Adrian> filtering so the packets go no further than the firewall Adrian> (which, BTW, is not the issue, even disabling it it's the same Adrian> problem). The system is still responsive but the load average Adrian> goes as high as 10 and the interface is losing packets (input Adrian> errors) which dramatically affects legitimate traffic, besides Adrian> mbuf(9) starvation. We are taking down the culprit clients, Adrian> but this takes time and we need the other clients not to be Adrian> affected by it. Adrian> What can I do to make the system better handle this kind of Adrian> traffic ? Could device polling(8) or just increasing the Adrian> kernel frequency clock to 1000Hz or more improve the situation Adrian> ? What kind of network cards could face a lot better this Adrian> burden ? Are there any other solutions ? Adrian> On a side note: what would be a adequate formula to Adrian> calculate the NMBCLUSTERS and MBUFS we should set on this Adrian> server (via boot-time kern.ipc.nmbclusters and Adrian> kern.ipc.nmbufs) ? In our experience, switch to fxp ethernet cards, test several motherboards and enable polling. fxp and em cards appear to have the best performance ... outrunning other cards by a fair margin. Different motherboards have several orders of magnitude different performance with the same processor. Polling (as others have mentioned) roughly doubles the throughput of a server and eliminates live lock. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can only be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================