From owner-freebsd-stable@FreeBSD.ORG Fri Mar 9 15:00:26 2007 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4678516A402 for ; Fri, 9 Mar 2007 15:00:26 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.190]) by mx1.freebsd.org (Postfix) with ESMTP id 976E613C48D for ; Fri, 9 Mar 2007 15:00:25 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by nf-out-0910.google.com with SMTP id k27so1010454nfc for ; Fri, 09 Mar 2007 07:00:24 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=qMyQNPq2jFgP9oDUqTeJRDSHFBIvzP37kRle2LHOy4nnvQ6HRN/VygLJ3CjzpFK2x5aJiVVctjfQ4vt79vyY1E24K/omceOriVcylq8EgNBpAv7l7NHopyjtnsKR3BzrFIq5xdkzYwaA0qLujGg69WKz65w9WFKo6k/inALyHFY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=CwZ/uBkOWSHVSsu24uTXgtk+qTZTv6ezGEYiY7U5tnGhSd1rs5JXJCnJ9saAaY2jp8wvjDhOMGnp2NEmxP/zvOEuUKQFJmmdpVLDSDM9LMjII01fUnVb+QzadV6IUCA7fzUvF2lAL/VsESm8FcqTcqMc0WEleaf+z7LccO5BZOQ= Received: by 10.82.175.2 with SMTP id x2mr2905483bue.1173452423203; Fri, 09 Mar 2007 07:00:23 -0800 (PST) Received: by 10.82.148.3 with HTTP; Fri, 9 Mar 2007 07:00:23 -0800 (PST) Message-ID: <7ad7ddd90703090700q708a82ddr84a35b32da7b04df@mail.gmail.com> Date: Fri, 9 Mar 2007 16:00:23 +0100 From: "Ulrich Spoerlein" To: stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: Weird NFS behaviour X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Mar 2007 15:00:26 -0000 Hi, we have performance problems with our FreeBSD 6.2 based NFS server. Picture the following setup: FreeBSD Client ---> Samba-Server ---> NFS-Server all three machines are running FreeBSD 6.2 (the same image). The NFS server is configured with 16 nfsd. sysctl.conf has net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 Now, what's the problem: The Samba-Server mounts shares via NFS. All servers are on Gigabit Ethernet and I get read transfer rates exceeding 50MB/s from the NFS server. This is all good and well, but if I copy a file via scp(1) (sic!) to the samba server into the NFS mounted directory, not only do I seldomly exceed 12MB/s but I also get a very strange traffic pattern on the em0 interface of the samba server. I get _twice_ as much incoming traffic on the em0 interface as outgoing traffic. systat -if on samba: em0 in 24.726 MB/s 25.905 MB/s 3.046 GB out 12.941 MB/s 13.558 MB/s 1.994 GB systat -if on nfs-server em0 in 11.497 MB/s 12.999 MB/s 3.727 GB out 11.878 MB/s 13.423 MB/s 995.485 MB To stress, this is running: gigabit-client:# scp large-file user@samba:/mnt/nfs-share/ The wicked part is this: If I copy a file from the samba server directly to the NFS share (not as a passthrough), I get these traffic patterns: systat -if on samba: em0 in 432.724 KB/s 432.724 KB/s 3.772 GB out 12.399 MB/s 12.399 MB/s 2.481 GB systat -if on nfs: em0 in 12.091 MB/s 15.791 MB/s 184.766 MB out 440.939 KB/s 562.521 KB/s 1.339 GB This is running: samba:# cp large-file /mnt/nfs-share/ What on earth is causing each received NFS packet to be _bounced_ to the samba server when using ssh, scp, smbd, etc. And not when generating the traffic locally? nfsstat -s is showing an increase in READ calls similar to WRITE calls when using the samba machine as pass-through. It is showing _no_ increase in READ calls when copying the files directly. NB: All these test were run _without_ smbd running, it's just that this server is designated to become our samba server. Setting vfs.nfsrv.async=1 doubled write performance, but the weird traffic pattern remains. (Am I asking for too much trouble by setting async NFS?) Thanks for any pointers! Uli