From owner-freebsd-net@FreeBSD.ORG Tue Feb 14 07:46:10 2006 Return-Path: X-Original-To: freebsd-net@FreeBSD.org Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F031716A420 for ; Tue, 14 Feb 2006 07:46:10 +0000 (GMT) (envelope-from glebius@FreeBSD.org) Received: from cell.sick.ru (cell.sick.ru [217.72.144.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2697843D45 for ; Tue, 14 Feb 2006 07:46:09 +0000 (GMT) (envelope-from glebius@FreeBSD.org) Received: from cell.sick.ru (glebius@localhost [127.0.0.1]) by cell.sick.ru (8.13.3/8.13.3) with ESMTP id k1E7k7Ov066135 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 14 Feb 2006 10:46:07 +0300 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.sick.ru (8.13.3/8.13.1/Submit) id k1E7k6D0066134; Tue, 14 Feb 2006 10:46:06 +0300 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.sick.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Tue, 14 Feb 2006 10:46:06 +0300 From: Gleb Smirnoff To: Marcos Bedinelli Message-ID: <20060214074606.GH86448@FreeBSD.org> Mail-Followup-To: Gleb Smirnoff , Marcos Bedinelli , freebsd-net@freebsd.org References: <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <7bb8f24157080b6aaacb897a99259df9@madhaus.cns.utoronto.ca> User-Agent: Mutt/1.5.6i Cc: freebsd-net@FreeBSD.org Subject: Re: Network performance in a dual CPU system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2006 07:46:11 -0000 On Fri, Feb 10, 2006 at 08:46:00AM -0500, Marcos Bedinelli wrote: M> We have a 2.4GHz Intel Xeon machine running FreeBSD 6.0-RELEASE-p2. Due M> to heavy network traffic, CPU utilization on that machine is 100%: M> M> === M> M> mull [~]$top -S M> last pid: 94989; load averages: 3.69, 4.02, 4.36 up M> 25+07:21:34 14:51:43 M> 105 processes: 2 running, 46 sleeping, 57 waiting M> CPU states: 0.0% user, 0.0% nice, 0.3% system, 99.4% interrupt, M> 0.3% idle M> Mem: 20M Active, 153M Inact, 84M Wired, 4K Cache, 60M Buf, 237M Free M> Swap: 999M Total, 999M Free M> M> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND M> 60 root 1 -44 -163 0K 8K WAIT 355.6H 72.17% swi1: M> net M> 39 root 1 -68 -187 0K 8K WAIT 52.3H 5.22% irq28: M> bge0 M> 40 root 1 -68 -187 0K 8K WAIT 28.3H 2.25% irq29: M> bge1 M> 11 root 1 171 52 0K 8K RUN 166.6H 0.00% idle M> 63 root 1 -16 0 0K 8K - 121:55 0.00% yarrow M> 61 root 1 -32 -151 0K 8K WAIT 46:21 0.00% swi4: M> clock sio M> [...] You didn't described what the box is actually. According to top(1) output, looks like your system is routing between bge0 and bge1. If this is true, then you should try: sysctl net.inet.ip.fastforwarding=1 This will split the work into the ithreads and also this will use a faster code path. The performance will improve, and obtaining a second CPU will give you a benefit, since you will have two heavy loaded threads. You should also try upgrade to 6.1, since bge(4) driver has undergone some improvements. If my guess was incorrect, and you aren't routing traffic, then you should follow Robert's advice and try: sysctl net.isr.direct=1 And see whether load is splitted betweeen processes. If it is, then purchase second CPU. M> Does anyone know whether a dual CPU system can help us improve the M> situation? I was wondering if the software interrupt threads would be M> divided between the two processors. -- Totus tuus, Glebius. GLEBIUS-RIPN GLEB-RIPE