From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 13:51:06 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 23DC2A8A for ; Fri, 25 Apr 2014 13:51:06 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id DA8401602 for ; Fri, 25 Apr 2014 13:51:05 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqYEAMphWlODaFve/2dsb2JhbABZg1VXgmW6AoZnUYEldIIlAQEBAwEBAQEgKyALBQcPDgMEAQEBAgINGQIpAQkeCAYIAgUEARwEiAwDCQgNpzCcTiGGWBeBKYxOEAIBBhUBMwcGgmmBSgSWK4QSkSSDTSExfEE X-IronPort-AV: E=Sophos;i="4.97,927,1389762000"; d="scan'208";a="117571142" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 25 Apr 2014 09:51:04 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 2C9D7B3EA2; Fri, 25 Apr 2014 09:51:04 -0400 (EDT) Date: Fri, 25 Apr 2014 09:51:04 -0400 (EDT) From: Rick Macklem To: Steven Hartland Message-ID: <1778189736.824550.1398433864172.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: NFS over LAGG / lacp poor performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-net@freebsd.org, Marek Salwerowicz , Gerrit =?utf-8?B?S8O8aG4=?= X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 13:51:06 -0000 Steven Hartland wrote: >=20 > ----- Original Message ----- > From: "Marek Salwerowicz" > To: "Steven Hartland" ; "Gerrit K=C3=BChn" > > Cc: > Sent: Friday, April 25, 2014 2:06 PM > Subject: Re: NFS over LAGG / lacp poor performance >=20 >=20 > >W dniu 2014-04-25 14:55, Steven Hartland pisze: > >> ----- Original Message ----- From: "Marek Salwerowicz" > >> > >> > >> > >>> W dniu 2014-04-25 14:01, Gerrit K=C3=BChn pisze: > >>>> Thanks for your input. As far as I understood so far, there > >>>> should > >>>> be one > >>>> igb queue created per cpu core in the system by default (and > >>>> this is > >>>> what > >>>> I see on my system). But my irq rate looks quite high to me (and > >>>> it is > >>>> only on one of these queues). > >>> > >>> > >>> My CPU has 8 cores: > >>> > >>> http://ark.intel.com/products/75267/Intel-Xeon-Processor-E5-2640-v2-2= 0M-Cache-2_00-GHz > >>> > >>> > >>> So why do I have only 1 queue ? > >> > >> What does "sysctl hw.igb.num_queues" report? > > > > storage1% sysctl hw.igb.num_queues > > hw.igb.num_queues: 1 > >> > >> num_queues does default to 1 for Legacy or MSI so you might be > >> hitting > >> that. > >> > >> Do you see "Using MSIX interrupts with" in your dmesg? > > storage% dmesg | grep MSIX > > igb0: Using MSIX interrupts with 2 vectors > > igb1: Using MSIX interrupts with 2 vectors > > igb2: Using MSIX interrupts with 2 vectors > > igb3: Using MSIX interrupts with 2 vectors > > igb0: Using MSIX interrupts with 2 vectors > > igb1: Using MSIX interrupts with 2 vectors > > igb2: Using MSIX interrupts with 2 vectors > > igb3: Using MSIX interrupts with 2 vectors >=20 > In that case I believe you've hard coded the number of queues, check > /boot/loader.conf > for references to this. >=20 Not really replying to Steve's email, but... NFS uses a single TCP connection for a mount. I still know nothing about lagg, but if lagg/lacp requires multiple TCP connections to spread the load..I'd just switch to using something like ftp, given you are only moving a few large files. If you must use NFS, then to get multiple TCP connections, you'll need to do multiple mounts and then do the file transfers concurrently over the different mounts. rick > Regards > Steve >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >=20