From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 14:13:26 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7BA2F484 for ; Fri, 25 Apr 2014 14:13:26 +0000 (UTC) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) by mx1.freebsd.org (Postfix) with ESMTP id 0D6F71917 for ; Fri, 25 Apr 2014 14:13:25 +0000 (UTC) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id E4F8C20110A; Fri, 25 Apr 2014 16:13:23 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id C4434405889; Fri, 25 Apr 2014 16:13:23 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id EBFAA406AF1; Fri, 25 Apr 2014 16:13:22 +0200 (CEST) Received: from cascade.aei.uni-hannover.de ([10.117.15.111]) by intranet.aei.uni-hannover.de (Lotus Domino Release 8.5.3FP6) with ESMTP id 2014042516131167-88690 ; Fri, 25 Apr 2014 16:13:11 +0200 Date: Fri, 25 Apr 2014 16:13:12 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: "Steven Hartland" Subject: Re: NFS over LAGG / lacp poor performance Message-Id: <20140425161312.3ad95e3a.gerrit.kuehn@aei.mpg.de> In-Reply-To: <4E7A280CACC84910A3DE068085976E30@multiplay.co.uk> References: <535A1354.2040309@wp.pl> <20140425113711.e7c7d1c2.gerrit.kuehn@aei.mpg.de> <535A482E.1030106@wp.pl> <20140425140123.a76c18f9.gerrit.kuehn@aei.mpg.de> <535A5268.100@wp.pl> <8247FE6336414E1F97ADA561D0680097@multiplay.co.uk> <535A5DD9.9060206@wp.pl> <535A69CE.9010800@wp.pl> <4E7A280CACC84910A3DE068085976E30@multiplay.co.uk> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.1.3 (GTK+ 2.24.19; amd64-portbld-freebsd8.2) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 8.5.3FP6|November 21, 2013) at 04/25/2014 16:13:11, Serialize by Router on intranet/aei-hannover(Release 8.5.3FP6|November 21, 2013) at 04/25/2014 16:13:21, Serialize complete at 04/25/2014 16:13:21 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.4.25.140019 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_2000_2999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __C230066_P5 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_PATH 0, __URI_NO_WWW 0, __URI_NS ' Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 14:13:26 -0000 On Fri, 25 Apr 2014 15:02:15 +0100 "Steven Hartland" wrote about Re: NFS over LAGG / lacp poor performance: SH> We find that large numbers of queues causes high interrupt issues Like the thing I am seeing with igb1 on my system? --- root@storage:/root # vmstat -i interrupt total rate irq1: atkbd0 2743 0 irq18: ehci0 uhci5 4445560 2 irq21: uhci1 29 0 cpu0:timer 355724275 227 irq256: igb0:que 0 99437514 63 irq257: igb0:que 1 61534816 39 irq258: igb0:que 2 101725601 65 irq259: igb0:que 3 100864440 64 irq260: igb0:link 2 0 irq261: igb1:que 0 1689527 1 irq262: igb1:que 1 2357590958 1510 irq263: igb1:que 2 1584474 1 irq264: igb1:que 3 1923144 1 irq265: igb1:link 2 0 irq266: mps0 332232450 212 irq267: mps1 194207894 124 irq268: mps2 176700834 113 irq269: ahci0 59175548 37 cpu1:timer 419838321 268 cpu3:timer 329696415 211 cpu2:timer 328219053 210 Total 4926593600 3156 --- irq262 sticks out like a sore thumb... SH> however at a guess you did this to enable the machine to boot with SH> all nics due to lack of auto mbuf tuning in 9.x. SH> I'd go with ~2 queues per nic. I was wondering what to try next for my system: either manually set the queues back to 2 or 1 per NIC, or try upgrading to either 9.2 or 10 as it looked like there have been improvements in the igb driver. Do you have any recommendations on that? cu Gerrit