From owner-freebsd-fs@FreeBSD.ORG Sat Jul 13 22:45:47 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E1556226 for ; Sat, 13 Jul 2013 22:45:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id ADB861277 for ; Sat, 13 Jul 2013 22:45:47 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.89,660,1367985600"; d="scan'208";a="40228784" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 13 Jul 2013 18:45:40 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A76E2B3F17; Sat, 13 Jul 2013 18:45:40 -0400 (EDT) Date: Sat, 13 Jul 2013 18:45:40 -0400 (EDT) From: Rick Macklem To: Berend de Boer Message-ID: <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> In-Reply-To: <8761wfvwml.wl%berend@pobox.com> Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jul 2013 22:45:47 -0000 Berend de Boer wrote: > >>>>> "Rick" == Rick Macklem writes: > > Rick> If you could do some testing where you export a UFS volume, > Rick> the results might help to isolate the issue to ZFS vs nfsd. > > Indeed! Have changed subject, as indeed ZFS is a red herring. Issue > shows up with UFS as well. Very high cpu for nfds, about the same > time > to do the operation. > > Tried it with these settings: > > vfs.nfsd.tcphighwater=5000 > vfs.nfsd.tcpcachetimeo=300 > nfs_server_flags="-u -t -n 256" > > > On nfs3 + ufs everything back to normal. I.e. nfs4 was about 15 > minutes, same operation was 241s minutes with nfs3 and nfsd using no > cpu at all basically. > All I can suggest is capturing packets and then emailing be the captured packet trace. You can use tcpdump to do the capture, since wireshark will understand it: # tcpdump -s 0 -w .pcap host and then emailing me .pcap. I can take a look at the packet capture and maybe see what is going on. I think you mentioned that you were using a Linux client, but not what version. I'd suggest a recent kernel from kernel.org. (Fedora tracks updates/fixes for NFSv4 pretty closely, so the newest Fedora release should be pretty current.) > Per other reply, tried this too: > > vfs.nfsd.issue_delegations=1 > > Locks up the client at first write access. Ctrl+C doesn't work, need > to explicitly send a KILL signal from other terminal. > > I think it locks up the server in some way as well. Doing an ls on an > exported path locks up. Ctrl+C won't work anymore. Process doesn't > react to any signal. In the end I rebooted the server to get rid of > this. > Obviously broken, but without a lot more information, I can's say anything. (Assuming you can still run commands on the server, you could start with something like "ps axl" and "procstat -kk".) rick > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/ >