From owner-freebsd-stable@FreeBSD.ORG Sun Mar 11 07:37:00 2007 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D4F7216A401 for ; Sun, 11 Mar 2007 07:37:00 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.175]) by mx1.freebsd.org (Postfix) with ESMTP id 62C2113C4AA for ; Sun, 11 Mar 2007 07:37:00 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so1747171ugh for ; Sat, 10 Mar 2007 23:36:59 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:received:received:date:from:to:subject:message-id:mail-followup-to:mime-version:content-type:content-disposition; b=CAlq7p+s4mQmmtRyw69BiLLRrZfpn68VG9xNTaesZ1mZLgHNQZoKhkg7Erw3RWHZlSeGfA7EWthuI03sXAqVdJl1pnBz8qOMC92Rru+WXCp7VE3D+gvtxoOlHguafea1XfNKCJhPthX2X/hG4LT6Z/OV0iIslxLiiQlCnVH8L4s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:date:from:to:subject:message-id:mail-followup-to:mime-version:content-type:content-disposition; b=s5PvT9s0tgYs6ilfbCIOf6XZzHtUdm+XUfksFVQBWkrCDpOikExwoyV7frNoMPB6EHHRCY86W2REdBPZQjWBhfSSLH8lvzWYqIG07OQ2ozr0wVjsnc9RUzW5h4i2UCuv+UzxYTUf6qQ6H+jDlCfxNnj0iVKpXvcTCx+cHFADIp4= Received: by 10.67.106.3 with SMTP id i3mr14165883ugm.1173598619186; Sat, 10 Mar 2007 23:36:59 -0800 (PST) Received: from roadrunner.q.local ( [85.180.162.73]) by mx.google.com with ESMTP id g8sm16708092muf.2007.03.10.23.36.58; Sat, 10 Mar 2007 23:36:58 -0800 (PST) Received: from roadrunner.q.local (localhost [127.0.0.1]) by roadrunner.q.local (8.13.8/8.13.8) with ESMTP id l2B7atTG001735 for ; Sun, 11 Mar 2007 08:36:55 +0100 (CET) (envelope-from uspoerlein@gmail.com) Received: (from q@localhost) by roadrunner.q.local (8.13.8/8.13.8/Submit) id l2B7atIK001734 for stable@freebsd.org; Sun, 11 Mar 2007 08:36:55 +0100 (CET) (envelope-from uspoerlein@gmail.com) Date: Sun, 11 Mar 2007 08:36:54 +0100 From: Ulrich Spoerlein To: stable@freebsd.org Message-ID: <20070311073654.GA1543@roadrunner.q.local> Mail-Followup-To: stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Cc: Subject: Weird NFS behaviour X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Mar 2007 07:37:00 -0000 Hi, we have performance problems with our FreeBSD 6.2 based NFS server. Picture the following setup: FreeBSD Client ---> Samba-Server ---> NFS-Server all three machines are running FreeBSD 6.2 (the same image). The NFS server is configured with 16 nfsd. sysctl.conf has net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 Now, what's the problem: The Samba-Server mounts shares via NFS. All servers are on Gigabit Ethernet and I get read transfer rates exceeding 50MB/s from the NFS server. This is all good and well, but if I copy a file via scp(1) (sic!) to the samba server into the NFS mounted directory, not only do I seldomly exceed 12MB/s but I also get a very strange traffic pattern on the em0 interface of the samba server. I get _twice_ as much incoming traffic on the em0 interface as outgoing traffic. systat -if on samba: em0 in 24.726 MB/s 25.905 MB/s 3.046 GB out 12.941 MB/s 13.558 MB/s 1.994 GB systat -if on nfs-server em0 in 11.497 MB/s 12.999 MB/s 3.727 GB out 11.878 MB/s 13.423 MB/s 995.485 MB To stress, this is running: gigabit-client:# scp large-file user@samba:/mnt/nfs-share/ The wicked part is this: If I copy a file from the samba server directly to the NFS share (not as a passthrough), I get these traffic patterns: systat -if on samba: em0 in 432.724 KB/s 432.724 KB/s 3.772 GB out 12.399 MB/s 12.399 MB/s 2.481 GB systat -if on nfs: em0 in 12.091 MB/s 15.791 MB/s 184.766 MB out 440.939 KB/s 562.521 KB/s 1.339 GB This is running: samba:# cp large-file /mnt/nfs-share/ What on earth is causing each received NFS packet to be _bounced_ to the samba server when using ssh, scp, smbd, etc. And not when generating the traffic locally? nfsstat -s is showing an increase in READ calls similar to WRITE calls when using the samba machine as pass-through. It is showing _no_ increase in READ calls when copying the files directly. NB: All these test were run _without_ smbd running, it's just that this server is designated to become our samba server. Setting vfs.nfsrv.async=1 doubled write performance, but the weird traffic pattern remains. (Am I asking for too much trouble by setting async NFS?) Thanks for any pointers! Uli