From owner-freebsd-stable@FreeBSD.ORG Fri Sep 26 18:30:32 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7818106564A; Fri, 26 Sep 2008 18:30:32 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id A95168FC13; Fri, 26 Sep 2008 18:30:32 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m8QII6UT004334; Fri, 26 Sep 2008 11:18:06 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m8QII6jF004333; Fri, 26 Sep 2008 11:18:06 -0700 (PDT) Date: Fri, 26 Sep 2008 11:18:06 -0700 (PDT) From: Matthew Dillon Message-Id: <200809261818.m8QII6jF004333@apollo.backplane.com> To: Jeremy Chadwick References: <20080926081806.GA19055@icarus.home.lan> <20080926095230.GA20789@icarus.home.lan> <20080926143153.GA26978@icarus.home.lan> Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: bad NFS/UDP performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2008 18:30:32 -0000 :> -vfs.nfs.realign_test: 22141777 :> +vfs.nfs.realign_test: 498351 :> :> -vfs.nfsrv.realign_test: 5005908 :> +vfs.nfsrv.realign_test: 0 :> :> +vfs.nfsrv.commit_miss: 0 :> +vfs.nfsrv.commit_blks: 0 :> :> changing them did nothing - or at least with respect to nfs throughput :-) : :I'm not sure what any of these do, as NFS is a bit out of my league. ::-) I'll be following this thread though! : :-- :| Jeremy Chadwick jdc at parodius.com | A non-zero nfs_realign_count is bad, it means NFS had to copy the mbuf chain to fix the alignment. nfs_realign_test is just the number of times it checked. So nfs_realign_test is irrelevant. it's nfs_realign_count that matters. Several things can cause NFS payloads to be improperly aligned. Anything from older network drivers which can't start DMA on a 2-byte boundary, resulting in the 14-byte encapsulation header causing improper alignment of the IP header & payload, to rpc embedded in NFS TCP streams winding up being misaligned. Modern network hardware either support 2-byte-aligned DMA, allowing the encapsulation to be 2-byte aligned so the payload winds up being 4-byte aligned, or support DMA chaining allowing the payload to be placed in its own mbuf, or pad, etc. -- One thing I would check is to be sure a couple of nfsiod's are running on the client when doing your tests. If none are running the RPCs wind up being more synchronous and less pipelined. Another thing I would check is IP fragment reassembly statistics (for UDP) - there should be none for TCP connections no matter what the NFS I/O size selected. (It does seem more likely to be scheduler-related, though). -Matt