From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 13:27:58 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D508227 for ; Fri, 25 Apr 2014 13:27:58 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id F23B0138D for ; Fri, 25 Apr 2014 13:27:57 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 382C620E7088B; Fri, 25 Apr 2014 13:27:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.3 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,HELO_NO_DOMAIN,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 31C0D20E70885; Fri, 25 Apr 2014 13:27:52 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Marek Salwerowicz" , =?iso-8859-1?Q?Gerrit_K=FChn?= References: <535A1354.2040309@wp.pl> <20140425113711.e7c7d1c2.gerrit.kuehn@aei.mpg.de> <535A482E.1030106@wp.pl> <20140425140123.a76c18f9.gerrit.kuehn@aei.mpg.de> <535A5268.100@wp.pl> <8247FE6336414E1F97ADA561D0680097@multiplay.co.uk> <535A5DD9.9060206@wp.pl> Subject: Re: NFS over LAGG / lacp poor performance Date: Fri, 25 Apr 2014 14:27:56 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 13:27:58 -0000 ----- Original Message ----- From: "Marek Salwerowicz" To: "Steven Hartland" ; "Gerrit Kühn" Cc: Sent: Friday, April 25, 2014 2:06 PM Subject: Re: NFS over LAGG / lacp poor performance >W dniu 2014-04-25 14:55, Steven Hartland pisze: >> ----- Original Message ----- From: "Marek Salwerowicz" >> >> >>> W dniu 2014-04-25 14:01, Gerrit Kühn pisze: >>>> Thanks for your input. As far as I understood so far, there should >>>> be one >>>> igb queue created per cpu core in the system by default (and this is >>>> what >>>> I see on my system). But my irq rate looks quite high to me (and it is >>>> only on one of these queues). >>> >>> >>> My CPU has 8 cores: >>> >>> http://ark.intel.com/products/75267/Intel-Xeon-Processor-E5-2640-v2-20M-Cache-2_00-GHz >>> >>> >>> So why do I have only 1 queue ? >> >> What does "sysctl hw.igb.num_queues" report? > > storage1% sysctl hw.igb.num_queues > hw.igb.num_queues: 1 >> >> num_queues does default to 1 for Legacy or MSI so you might be hitting >> that. >> >> Do you see "Using MSIX interrupts with" in your dmesg? > storage% dmesg | grep MSIX > igb0: Using MSIX interrupts with 2 vectors > igb1: Using MSIX interrupts with 2 vectors > igb2: Using MSIX interrupts with 2 vectors > igb3: Using MSIX interrupts with 2 vectors > igb0: Using MSIX interrupts with 2 vectors > igb1: Using MSIX interrupts with 2 vectors > igb2: Using MSIX interrupts with 2 vectors > igb3: Using MSIX interrupts with 2 vectors In that case I believe you've hard coded the number of queues, check /boot/loader.conf for references to this. Regards Steve