From owner-freebsd-net@FreeBSD.ORG  Thu Jul  3 21:26:39 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 691C77CE
 for <freebsd-net@freebsd.org>; Thu,  3 Jul 2014 21:26:39 +0000 (UTC)
Received: from quine.pinyon.org (quine.pinyon.org [65.101.5.249])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 3E92C2604
 for <freebsd-net@freebsd.org>; Thu,  3 Jul 2014 21:26:38 +0000 (UTC)
Received: by quine.pinyon.org (Postfix, from userid 122)
 id 816C41602CD; Thu,  3 Jul 2014 14:26:37 -0700 (MST)
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on quine.pinyon.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00
 autolearn=ham autolearn_force=no version=3.4.0
Received: from feyerabend.n1.pinyon.org (feyerabend.n1.pinyon.org [10.0.10.6])
 (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits))
 (No client certificate requested)
 by quine.pinyon.org (Postfix) with ESMTPSA id 59DFF1600DF
 for <freebsd-net@freebsd.org>; Thu,  3 Jul 2014 14:26:35 -0700 (MST)
Message-ID: <53B5CA8B.1050707@pinyon.org>
Date: Thu, 03 Jul 2014 14:26:35 -0700
From: "Russell L. Carter" <rcarter@pinyon.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: freebsd-net@freebsd.org
Subject: Re: NFS client READ performance on -current
References: <53B49AE0.4030902@pinyon.org>
 <371130768.6608219.1404345846086.JavaMail.root@uoguelph.ca>
 <CAHu1Y738u1OGG6MC29kT4860pwCUSvDobRnNrhcdKCTubJN2SQ@mail.gmail.com>
 <53B4A605.8000604@pinyon.org> <20140703090733.5a648056@tmu.ulm.sysgo.com>
In-Reply-To: <20140703090733.5a648056@tmu.ulm.sysgo.com>
X-Enigmail-Version: 1.6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 03 Jul 2014 21:26:39 -0000



On 07/03/14 00:07, Thomas Mueller wrote:
> On Wed, 02 Jul 2014 17:38:29 -0700, Russell L. Carter wrote:
>> On the second suggestion:
>>
>> root@feyerabend> sysctl -a | grep dev.em | grep enable_
>> root@feyerabend>
> 
>  tmu:~$ sysctl -a |  grep msix
>  hw.em.enable_msix: 1
>  hw.igb.enable_msix: 1
>  hw.ix.enable_msix: 1
>  [...]
> 
> 

I set hw.em.enable_msix=0 on first the server and then the server +
client and NFSv4.1 read throughput is unchanged.  For a 5GB transfer
it settles down to a range of 58-68MB/s eventually.  I then set it
back to enabled for both server and client and saw the same rates.

Russell