From owner-freebsd-fs@FreeBSD.ORG Wed Oct 8 01:20:43 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B1F485A; Wed, 8 Oct 2014 01:20:43 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 019B8D13; Wed, 8 Oct 2014 01:20:42 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aq8EACmQNFSDaFve/2dsb2JhbABfDoQrBIMA0HICgSMBe4QEAQEEIwRSGxgCAg0ZAlkGE4g+rQqVDwEXgSyOTxU0B4J3gVQFkU6NIZBcg3+DI1whL4EGAR8EHoECAQEB X-IronPort-AV: E=Sophos;i="5.04,674,1406606400"; d="scan'208";a="160057608" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 07 Oct 2014 21:20:41 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 1977DB4081; Tue, 7 Oct 2014 21:20:41 -0400 (EDT) Date: Tue, 7 Oct 2014 21:20:41 -0400 (EDT) From: Rick Macklem To: Garrett Wollman Message-ID: <1402889337.60119616.1412731241078.JavaMail.root@uoguelph.ca> In-Reply-To: <21556.24590.269326.188103@khavrinen.csail.mit.edu> Subject: Re: 9.3 NFS client bug? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2014 01:20:43 -0000 Garrett Wollman wrote: > < said: > > >> By ay chance is your ZFS server allocating a ZFS inode (whatever > >> they > >> call it) with a fileno/inode# that doesn't fit in 32 bits? > > Nope. (And I want to emphasize that Ubuntu clients don't exhibit the > problem, so whatever it is, it doesn't appear to be related to the > backing filesystem on the server.) > > > Btw, I tried your bonnie++ command on my test machines, but they > > failed > > part way through because they couldn't allocate boundary tags. (At > > least > > now I can reproduce that easily, although it is the M_NOWAIT case > > where > > the thread just loops in the kernel.) > > You really need to get some server-class hardware. Perhaps the > Foundation can help? > > I'm currently using the following command, which seems to reproduce > the problem reliably on my client and server: > > bonnie++ -s 0 -n 144:0:0:1:16384 -u 4294967294 -D > > This minimizes the amount of temporary storage required. > > -GAWollman > I took a look at the bonnie++ sources and it does the removal of the files in a loop like while ((dp = readdir(dir)) != NULL) unlink(dp->d_name); As far as I know, this has never worked correctly for FreeBSD. The unlink() invalidates the directory offset cookies and then it has trouble finding the next entry. To make the above loop work correctly for FreeBSD, it needs to be re-written to start at the beginning of the directory after each unlink(). I can't remember why it was coded to invalidate the offset cookies, since they should still be ok after an unlink() for UFS, but maybe other file systems break. Maybe jhb@ can remember more about this? OpenBSD doesn't use the buffer cache for NFS Readdir, but I can't remember if it also suffers from this issue. Sorry, but I hadn't run into this for a while, so I didn't spot it, rick