From owner-freebsd-fs@FreeBSD.ORG Sun May 12 00:06:18 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8B9AAE16; Sun, 12 May 2013 00:06:18 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [IPv6:2001:5a8:4:7e72:4a5b:39ff:fe12:452]) by mx1.freebsd.org (Postfix) with ESMTP id 6BD1785; Sun, 12 May 2013 00:06:18 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id r4C06G3Z066231; Sat, 11 May 2013 17:06:16 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201305120006.r4C06G3Z066231@chez.mckusick.com> To: mjacob@freebsd.org Subject: Re: Tell me how to increase the virtual disk with ZFS? In-reply-to: <518EDACC.2070609@freebsd.org> Date: Sat, 11 May 2013 17:06:16 -0700 From: Kirk McKusick X-Spam-Status: No, score=0.0 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=failed version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on chez.mckusick.com Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 00:06:18 -0000 > Date: Sat, 11 May 2013 16:57:00 -0700 > From: Matthew Jacob > To: freebsd-fs@freebsd.org > Subject: Re: Tell me how to increase the virtual disk with ZFS? > > On 5/11/2013 2:31 PM, Kirk McKusick wrote: > > As of FreeBSD 9.1, the growfs(8) utility has the ability to increase > > the size of live filesystems. > > > but only for ufs, no? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Yes, growfs works only for UFS. Sorry, should have stated that. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Sun May 12 02:27:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5A353E8 for ; Sun, 12 May 2013 02:27:22 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.208.146]) by mx1.freebsd.org (Postfix) with ESMTP id 14C5362E for ; Sun, 12 May 2013 02:27:21 +0000 (UTC) Received: from maia.hub.org (unknown [200.46.151.189]) by hub.org (Postfix) with ESMTP id 7365D1FAD7DE; Sat, 11 May 2013 23:27:20 -0300 (ADT) Received: from hub.org ([200.46.208.146]) by maia.hub.org (mx1.hub.org [200.46.151.189]) (amavisd-maia, port 10024) with ESMTP id 66284-10; Sun, 12 May 2013 02:27:20 +0000 (UTC) Received: from Marc-Fourniers-Mac-mini.local (S01067cb21b2ff4ca.gv.shawcable.net [24.108.26.71]) by hub.org (Postfix) with ESMTPA id 852D51FAD7DD; Sat, 11 May 2013 23:27:18 -0300 (ADT) Message-ID: <518EFE05.8010100@hub.org> Date: Sat, 11 May 2013 19:27:17 -0700 From: "Marc G. Fournier" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Rick Macklem Subject: Re: NFS Performance issue against NetApp References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 02:27:22 -0000 With vfs.nfs.noconsist=3 ... 385595ms nfsstat -z before startup, nfsstat -c after: Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 332594 5 17238 0 224426 231137 3743 1 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 307 0 71 0 8447 Mknod Fsstat Fsinfo PathConf Commit 0 509 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 818479 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 608296 332596 526200 17245 -95425 224426 13178 231137 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 1050 55 502 7 543340 8448 ============ vfs.nfs.noconsist=2 ... 392201ms Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 332557 5 17228 0 224421 231131 3743 1 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 307 0 72 0 8430 Mknod Fsstat Fsinfo PathConf Commit 0 502 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 818395 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 607834 332557 525801 17231 -95401 224421 13178 231131 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 1028 56 502 0 542925 8431 ============ vfs.nfs.noconsist=0 ... 391622ms Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 236122 5 17221 0 230575 230823 3743 1 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 307 0 71 0 8425 Mknod Fsstat Fsinfo PathConf Commit 0 516 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 727799 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 711860 236124 526549 17225 -101525 230490 13178 230823 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 1057 55 516 0 543709 8425 I checked a second time with nonconsist=0, and the nfsstat -c values seem to come out pretty much the same ... I'm going to head down to the office and try again with Solaris (I'd have to re-install, since I used that system for the Solaris), and see what nfsstat -c results I get out of that ... will post a followup on this when completed ... On 2013-05-10 5:32 PM, Rick Macklem wrote: > Marc G. Fournier wrote: >> FYI … I just installed Solaris 11 onto the same hardware and ran the >> same test … so far, I'm seeing: >> >> Linux @ ~30s >> Solaris @ ~44s >> >> OpenBSD @ ~200s >> FreeBSD @ ~240s >> >> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' issue >> … same as 9.x … I could see Linux 'cutting corners', but >> Oracle/Solaris too … ? >> > The three client implementations (BSD, Linux, Solaris) were developed > independently and, as such, will all implement somewaht different > caching algorithms (the RFCs specify what goes on the wire, but say > little w.r.t. client side caching). > > I have a attached a patch that might be useful for determining if > the client side buffer cache consistency algorithm in FreeBSD is > causing the slow startup of jboss. Do not run this patch on a > production system, since it pretty well disables all buffer cache > coherency (ie. if another client modifies a file, the patched client > won't notice and will continue to cache stale file data). > > If the patch does speed up startup of jboss significantly, you can > use the sysctl: > vfs.nfs.noconsist > to check for which coherency check is involved by decreasing the > value for the sysctl by 1 and then trying a startup again. (When > vfs.nfs.noconsist=0, normal cache coherency will be applied.) > > I have no idea if buffer cache coherency is a factor, but trying > the attached patch might determine if it is. > > Note that you have never posted updated "nfsstat -c" values. > (Remember that what you posted indicated 88 RPCs, which seemed > bogus.) Finding out if FreeBSD does a lot more of certain RPCs > that Linux/Solaris might help isolate what is going on. > > rick > >> On 2013-05-03, at 04:50 , Mark Felder wrote: >> >>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier >>> wrote: >>> >>>> Hadn't thought to do so with Linux, but … >>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms >>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms >>> Please make sure both platforms are using similar atime settings. I >>> think most distros use ext4 with diratime by default. I'd just do >>> noatime on both platforms to be safe. >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to >>> "freebsd-fs-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sun May 12 07:13:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3E494CD0 for ; Sun, 12 May 2013 07:13:57 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.208.146]) by mx1.freebsd.org (Postfix) with ESMTP id E8128E4A for ; Sun, 12 May 2013 07:13:56 +0000 (UTC) Received: from maia.hub.org (unknown [200.46.151.188]) by hub.org (Postfix) with ESMTP id 2D11517CF18E; Sun, 12 May 2013 04:13:55 -0300 (ADT) Received: from hub.org ([200.46.208.146]) by maia.hub.org (mx1.hub.org [200.46.151.188]) (amavisd-maia, port 10024) with ESMTP id 06408-01; Sun, 12 May 2013 07:13:54 +0000 (UTC) Received: from Marc-Fourniers-Mac-mini.local (S01067cb21b2ff4ca.gv.shawcable.net [24.108.26.71]) by hub.org (Postfix) with ESMTPA id E0DED17CF18B; Sun, 12 May 2013 04:13:53 -0300 (ADT) Message-ID: <518F4130.6080201@hub.org> Date: Sun, 12 May 2013 00:13:52 -0700 From: "Marc G. Fournier" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, Rick Macklem Subject: Re: NFS Performance issue against NetApp References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> In-Reply-To: <518EFE05.8010100@hub.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 07:13:57 -0000 'k, here is on Linux ... this is right after rebooting the server, doing a mount and running the startup once: Client rpc stats: calls retrans authrefrsh 40602 0 40609 Client nfs v3: null getattr setattr lookup access readlink 0 0% 13000 32% 5 0% 6140 15% 6741 16% 0 0% read write create mkdir symlink mknod 3556 8% 6711 16% 3743 9% 307 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 1 0% 0 0% 0 0% 0 0% 16 0% 380 0% fsstat fsinfo pathconf commit 0 0% 2 0% 1 0% 0 0% One thing to note is that both Linux/FreeBSD have "rsize=65536,wsize=65536" ... but there are 63x as many reads / 34x as many writes on FreeBSD as on Linux ... ? Just noticed this on the FreeBSD stats: Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 818479 818k Retries? Is that normal ... ? Also, the NetApp volumes being used here are not shared ... there are no other clients mounting these, and the Linux/FreeBSD volumes are seperate ... same size, same jboss install, same configuration, same war file ... I could mount /vol/linux_jboss onto the FreeBSD, or /vol/freebsd_jboss onto the Linux, and they would load the same way ... in fact, the jboss install itself was done onto the FreeBSD and copied over to the Linux ... and both are using OpenJDK7 ... I tried to make it as identical as I could ... On 2013-05-11 7:27 PM, Marc G. Fournier wrote: > > With > > vfs.nfs.noconsist=3 ... 385595ms > > nfsstat -z before startup, nfsstat -c after: > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 332594 5 17238 0 224426 231137 > 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir > RdirPlus Access > 0 0 0 307 0 71 0 8447 > Mknod Fsstat Fsinfo PathConf Commit > 0 509 0 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 818479 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 608296 332596 526200 17245 -95425 224426 13178 > 231137 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 1050 55 502 7 > 543340 8448 > > > ============ > > vfs.nfs.noconsist=2 ... 392201ms > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 332557 5 17228 0 224421 231131 > 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir > RdirPlus Access > 0 0 0 307 0 72 0 8430 > Mknod Fsstat Fsinfo PathConf Commit > 0 502 0 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 818395 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 607834 332557 525801 17231 -95401 224421 13178 > 231131 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 1028 56 502 0 > 542925 8431 > > > ============ > vfs.nfs.noconsist=0 ... 391622ms > > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 236122 5 17221 0 230575 230823 > 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir > RdirPlus Access > 0 0 0 307 0 71 0 8425 > Mknod Fsstat Fsinfo PathConf Commit > 0 516 0 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 727799 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 711860 236124 526549 17225 -101525 230490 13178 > 230823 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 1057 55 516 0 > 543709 8425 > > > I checked a second time with nonconsist=0, and the nfsstat -c values > seem to come out pretty much the same ... > > I'm going to head down to the office and try again with Solaris (I'd > have to re-install, since I used that system for the Solaris), and see > what nfsstat -c results I get out of that ... will post a followup on > this when completed ... > > > > On 2013-05-10 5:32 PM, Rick Macklem wrote: >> Marc G. Fournier wrote: >>> FYI … I just installed Solaris 11 onto the same hardware and ran the >>> same test … so far, I'm seeing: >>> >>> Linux @ ~30s >>> Solaris @ ~44s >>> >>> OpenBSD @ ~200s >>> FreeBSD @ ~240s >>> >>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' issue >>> … same as 9.x … I could see Linux 'cutting corners', but >>> Oracle/Solaris too … ? >>> >> The three client implementations (BSD, Linux, Solaris) were developed >> independently and, as such, will all implement somewaht different >> caching algorithms (the RFCs specify what goes on the wire, but say >> little w.r.t. client side caching). >> >> I have a attached a patch that might be useful for determining if >> the client side buffer cache consistency algorithm in FreeBSD is >> causing the slow startup of jboss. Do not run this patch on a >> production system, since it pretty well disables all buffer cache >> coherency (ie. if another client modifies a file, the patched client >> won't notice and will continue to cache stale file data). >> >> If the patch does speed up startup of jboss significantly, you can >> use the sysctl: >> vfs.nfs.noconsist >> to check for which coherency check is involved by decreasing the >> value for the sysctl by 1 and then trying a startup again. (When >> vfs.nfs.noconsist=0, normal cache coherency will be applied.) >> >> I have no idea if buffer cache coherency is a factor, but trying >> the attached patch might determine if it is. >> >> Note that you have never posted updated "nfsstat -c" values. >> (Remember that what you posted indicated 88 RPCs, which seemed >> bogus.) Finding out if FreeBSD does a lot more of certain RPCs >> that Linux/Solaris might help isolate what is going on. >> >> rick >> >>> On 2013-05-03, at 04:50 , Mark Felder wrote: >>> >>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier >>>> wrote: >>>> >>>>> Hadn't thought to do so with Linux, but … >>>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms >>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms >>>> Please make sure both platforms are using similar atime settings. I >>>> think most distros use ext4 with diratime by default. I'd just do >>>> noatime on both platforms to be safe. >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to >>>> "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sun May 12 07:21:46 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7014FD93 for ; Sun, 12 May 2013 07:21:46 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.208.146]) by mx1.freebsd.org (Postfix) with ESMTP id 3C993E75 for ; Sun, 12 May 2013 07:21:46 +0000 (UTC) Received: from maia.hub.org (unknown [200.46.151.188]) by hub.org (Postfix) with ESMTP id 8846F17CF18E; Sun, 12 May 2013 04:21:45 -0300 (ADT) Received: from hub.org ([200.46.208.146]) by maia.hub.org (mx1.hub.org [200.46.151.188]) (amavisd-maia, port 10024) with ESMTP id 09398-01; Sun, 12 May 2013 07:21:44 +0000 (UTC) Received: from Marc-Fourniers-Mac-mini.local (S01067cb21b2ff4ca.gv.shawcable.net [24.108.26.71]) by hub.org (Postfix) with ESMTPA id 4A98C17CF18B; Sun, 12 May 2013 04:21:44 -0300 (ADT) Message-ID: <518F4307.3060908@hub.org> Date: Sun, 12 May 2013 00:21:43 -0700 From: "Marc G. Fournier" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, Rick Macklem Subject: Re: NFS Performance issue against NetApp References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> In-Reply-To: <518F4130.6080201@hub.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 07:21:46 -0000 On 2013-05-12 12:13 AM, Marc G. Fournier wrote: > > > *Just noticed this on the FreeBSD stats: ** > **** > **Rpc Info: ** > ** TimedOut Invalid X Replies Retries Requests ** > ** 0 0 0 0 818479 ** > **** > **818k Retries? Is that normal ... ? * > Ignore the above ... working too much tonight, eyes going cross eyes ... mis-read my columns :( From owner-freebsd-fs@FreeBSD.ORG Sun May 12 10:49:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0A862389 for ; Sun, 12 May 2013 10:49:06 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id 97294648 for ; Sun, 12 May 2013 10:49:05 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UbTpp-0006Yg-Hn for freebsd-fs@freebsd.org; Sun, 12 May 2013 12:48:57 +0200 Received: from dhcp-077-251-158-153.chello.nl ([77.251.158.153] helo=ronaldradial) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UbTpo-0008ON-8s for freebsd-fs@freebsd.org; Sun, 12 May 2013 12:48:56 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: Re[2]: Tell me how to increase the virtual disk with ZFS? References: <43529.1368277152.10278121996412321792@ffe11.ukr.net> <518E5973.9090003@platinum.linux.pl> <80049.1368285106.9876638397852811264@ffe6.ukr.net> Date: Sun, 12 May 2013 12:48:55 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <80049.1368285106.9876638397852811264@ffe6.ukr.net> User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: 4b8baa00aa45d625c6ee6c23f4cb0de1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 10:49:06 -0000 On Sat, 11 May 2013 17:11:46 +0200, Vladislav Prodan wrote: > > >> # zfs list md1p1 >> NAME USED AVAIL REFER MOUNTPOINT >> md1p1 126K 1.96G 31K /md1p1 >> >> -- on-line resize partition to occupy added disk space >> # sysctl kern.geom.debugflags=16 >> kern.geom.debugflags: 0 -> 16 >> # gpart recover md1 >> md1 recovered >> # gpart resize -i 1 md1 >> md1p1 resized >> # sysctl kern.geom.debugflags=0 >> kern.geom.debugflags: 16 -> 0 >> >> -- tell zfs about it >> # zpool online -e md1p1 md1p1 >> >> -- done >> # zfs list md1p1 >> NAME USED AVAIL REFER MOUNTPOINT >> md1p1 136K 9.84G 31K /md1p1 >> > > Can repeat resize the flag [-s size], so it was clearly indicate the > amount of increase in partition? > Can you putting a pair small files in /md1p1? > And compare md5 and sha256 these files before and after the resize md1? > > Thank you in advance. > > > > > I guess md1 is a memory disk. See mdconfig(8). So it is quite easy (and cheap) to experiment with it. The same example, but with the creation of a memory disk. # dd if=/dev/zero of=/data/prut/memdisk1 bs=1M count=1024 # mdconfig -f /data/prut/memdisk1 -u 0 # gpart create -s gpt /dev/md0 # gpart add -t freebsd-zfs -s 512m /dev/md0 <-- -s 512m is smaller than the virtual disk # zpool create testpool md0p1 # df -h /testpool Filesystem Size Used Avail Capacity Mounted on testpool 471M 31k 471M 0% /testpool # sysctl kern.geom.debugflags=16 # gpart resize -i 1 md0 'If -s option is omitted then new size is automatically calculated to maximum available from given geom.' # sysctl kern.geom.debugflags=0 # zpool online -e testpool md0p1 # df -h /testpool Filesystem Size Used Avail Capacity Mounted on testpool 983M 31k 983M 0% /testpool Ronald. From owner-freebsd-fs@FreeBSD.ORG Sun May 12 12:48:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 74D2E34B for ; Sun, 12 May 2013 12:48:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id F23ABA26 for ; Sun, 12 May 2013 12:48:05 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEAGiOj1GDaFvO/2dsb2JhbABSCIM+gzy8ZYEXdIIfAQEBBAEBASAEJyALGQIYERkCBCUBCSYGCAcEARwEh2sMj1WbIZAZjXIEfhkbB4JCgRMDj3SEdoJCgSaQD4MrIDKBBDU X-IronPort-AV: E=Sophos;i="4.87,657,1363147200"; d="diff'?scan'208";a="29339275" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 12 May 2013 08:48:03 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id F0051B3F4E; Sun, 12 May 2013 08:48:03 -0400 (EDT) Date: Sun, 12 May 2013 08:48:03 -0400 (EDT) From: Rick Macklem To: "Marc G. Fournier" Message-ID: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <518F4130.6080201@hub.org> Subject: Re: NFS Performance issue against NetApp MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_291492_1969524791.1368362883961" X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2013 12:48:06 -0000 ------=_Part_291492_1969524791.1368362883961 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Marc G. Fournier wrote: > 'k, here is on Linux ... this is right after rebooting the server, > doing > a mount and running the startup once: >=20 > Client rpc stats: > calls retrans authrefrsh > 40602 0 40609 >=20 > Client nfs v3: > null getattr setattr lookup access readlink > 0 0% 13000 32% 5 0% 6140 15% 6741 16% > 0 0% > read write create mkdir symlink mknod > 3556 8% 6711 16% 3743 9% 307 0% 0 0% > 0 0% > remove rmdir rename link readdir readdirplus > 1 0% 0 0% 0 0% 0 0% 16 0% > 380 0% > fsstat fsinfo pathconf commit > 0 0% 2 0% 1 0% 0 0% >=20 > One thing to note is that both Linux/FreeBSD have > "rsize=3D65536,wsize=3D65536" ... but there are 63x as many reads / 34x a= s > many writes on FreeBSD as on Linux ... ? >=20 > Just noticed this on the FreeBSD stats: >=20 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 818479 >=20 > 818k Retries? Is that normal ... ? >=20 > Also, the NetApp volumes being used here are not shared ... there are > no > other clients mounting these, and the Linux/FreeBSD volumes are > seperate > ... same size, same jboss install, same configuration, same war file > ... > I could mount /vol/linux_jboss onto the FreeBSD, or /vol/freebsd_jboss > onto the Linux, and they would load the same way ... in fact, the > jboss > install itself was done onto the FreeBSD and copied over to the Linux > ... and both are using OpenJDK7 ... I tried to make it as identical as > I > could ... >=20 >=20 > On 2013-05-11 7:27 PM, Marc G. Fournier wrote: > > > > With > > > > vfs.nfs.noconsist=3D3 ... 385595ms > > > > nfsstat -z before startup, nfsstat -c after: > > > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > 332594 5 17238 0 224426 231137 > > 3743 1 > > Rename Link Symlink Mkdir Rmdir Readdir > > RdirPlus Access > > 0 0 0 307 0 71 0 8447 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 509 0 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 818479 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > > Hits Misses > > 608296 332596 526200 17245 -95425 224426 13178 > > 231137 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > > Hits Misses > > 0 0 1050 55 502 7 > > 543340 8448 > > Ok, so disabling the mtime based cache consistency doesn't make much difference. Forget about that one. I've attached another patch (which you probably shouldn't use for a production system either) to be tried instead of the last one. (This one is basically "work in progress" by Alexander Kabaev for better performance during file linking. I hope he doesn't mind me posting it.) rick > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > vfs.nfs.noconsist=3D2 ... 392201ms > > > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > 332557 5 17228 0 224421 231131 > > 3743 1 > > Rename Link Symlink Mkdir Rmdir Readdir > > RdirPlus Access > > 0 0 0 307 0 72 0 8430 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 502 0 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 818395 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > > Hits Misses > > 607834 332557 525801 17231 -95401 224421 13178 > > 231131 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > > Hits Misses > > 0 0 1028 56 502 0 > > 542925 8431 > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > vfs.nfs.noconsist=3D0 ... 391622ms > > > > > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > 236122 5 17221 0 230575 230823 > > 3743 1 > > Rename Link Symlink Mkdir Rmdir Readdir > > RdirPlus Access > > 0 0 0 307 0 71 0 8425 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 516 0 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 727799 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > > Hits Misses > > 711860 236124 526549 17225 -101525 230490 13178 > > 230823 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > > Hits Misses > > 0 0 1057 55 516 0 > > 543709 8425 > > > > > > I checked a second time with nonconsist=3D0, and the nfsstat -c values > > seem to come out pretty much the same ... > > > > I'm going to head down to the office and try again with Solaris (I'd > > have to re-install, since I used that system for the Solaris), and > > see > > what nfsstat -c results I get out of that ... will post a followup > > on > > this when completed ... > > > > > > > > On 2013-05-10 5:32 PM, Rick Macklem wrote: > >> Marc G. Fournier wrote: > >>> FYI =E2=80=A6 I just installed Solaris 11 onto the same hardware and = ran > >>> the > >>> same test =E2=80=A6 so far, I'm seeing: > >>> > >>> Linux @ ~30s > >>> Solaris @ ~44s > >>> > >>> OpenBSD @ ~200s > >>> FreeBSD @ ~240s > >>> > >>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' > >>> issue > >>> =E2=80=A6 same as 9.x =E2=80=A6 I could see Linux 'cutting corners', = but > >>> Oracle/Solaris too =E2=80=A6 ? > >>> > >> The three client implementations (BSD, Linux, Solaris) were > >> developed > >> independently and, as such, will all implement somewaht different > >> caching algorithms (the RFCs specify what goes on the wire, but say > >> little w.r.t. client side caching). > >> > >> I have a attached a patch that might be useful for determining if > >> the client side buffer cache consistency algorithm in FreeBSD is > >> causing the slow startup of jboss. Do not run this patch on a > >> production system, since it pretty well disables all buffer cache > >> coherency (ie. if another client modifies a file, the patched > >> client > >> won't notice and will continue to cache stale file data). > >> > >> If the patch does speed up startup of jboss significantly, you can > >> use the sysctl: > >> vfs.nfs.noconsist > >> to check for which coherency check is involved by decreasing the > >> value for the sysctl by 1 and then trying a startup again. (When > >> vfs.nfs.noconsist=3D0, normal cache coherency will be applied.) > >> > >> I have no idea if buffer cache coherency is a factor, but trying > >> the attached patch might determine if it is. > >> > >> Note that you have never posted updated "nfsstat -c" values. > >> (Remember that what you posted indicated 88 RPCs, which seemed > >> bogus.) Finding out if FreeBSD does a lot more of certain RPCs > >> that Linux/Solaris might help isolate what is going on. > >> > >> rick > >> > >>> On 2013-05-03, at 04:50 , Mark Felder wrote: > >>> > >>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier > >>>> wrote: > >>>> > >>>>> Hadn't thought to do so with Linux, but =E2=80=A6 > >>>>> Linux =E2=80=A6=E2=80=A6. 20732ms, 20117ms, 20935ms, 20130ms, 20560= ms > >>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms > >>>> Please make sure both platforms are using similar atime settings. > >>>> I > >>>> think most distros use ext4 with diratime by default. I'd just do > >>>> noatime on both platforms to be safe. > >>>> _______________________________________________ > >>>> freebsd-fs@freebsd.org mailing list > >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>>> To unsubscribe, send any mail to > >>>> "freebsd-fs-unsubscribe@freebsd.org" > >>> _______________________________________________ > >>> freebsd-fs@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>> To unsubscribe, send any mail to > >>> "freebsd-fs-unsubscribe@freebsd.org" > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to > > "freebsd-fs-unsubscribe@freebsd.org" ------=_Part_291492_1969524791.1368362883961 Content-Type: text/x-patch; name=nfs.diff Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=nfs.diff ZGlmZiAtLWdpdCBhL3N5cy9mcy9uZnNjbGllbnQvbmZzX2NsYmlvLmMgYi9zeXMvZnMvbmZzY2xp ZW50L25mc19jbGJpby5jDQppbmRleCBhMGVjOGVlLi5lNmE3MjY3IDEwMDY0NA0KLS0tIGEvc3lz L2ZzL25mc2NsaWVudC9uZnNfY2xiaW8uYw0KKysrIGIvc3lzL2ZzL25mc2NsaWVudC9uZnNfY2xi aW8uYw0KQEAgLTEwMzEsMTMgKzEwMzEsMTYgQEAgZmx1c2hfYW5kX3Jlc3RhcnQ6DQogCQlsYm4g PSB1aW8tPnVpb19vZmZzZXQgLyBiaW9zaXplOw0KIAkJb24gPSB1aW8tPnVpb19vZmZzZXQgLSAo bGJuICogYmlvc2l6ZSk7DQogCQluID0gTUlOKCh1bnNpZ25lZCkoYmlvc2l6ZSAtIG9uKSwgdWlv LT51aW9fcmVzaWQpOw0KKyNpZiAwDQogYWdhaW46DQorI2VuZGlmDQogCQkvKg0KIAkJICogSGFu ZGxlIGRpcmVjdCBhcHBlbmQgYW5kIGZpbGUgZXh0ZW5zaW9uIGNhc2VzLCBjYWxjdWxhdGUNCiAJ CSAqIHVuYWxpZ25lZCBidWZmZXIgc2l6ZS4NCiAJCSAqLw0KIAkJbXR4X2xvY2soJm5wLT5uX210 eCk7DQotCQlpZiAodWlvLT51aW9fb2Zmc2V0ID09IG5wLT5uX3NpemUgJiYgbikgew0KKwkJaWYg KGxibiA9PSAobnAtPm5fc2l6ZSAvIGJpb3NpemUpICYmDQorCQkgICAgdWlvLT51aW9fb2Zmc2V0 ICsgbiA+IG5wLT5uX3NpemUgJiYgbikgew0KIAkJCW10eF91bmxvY2soJm5wLT5uX210eCk7DQog CQkJLyoNCiAJCQkgKiBHZXQgdGhlIGJ1ZmZlciAoaW4gaXRzIHByZS1hcHBlbmQgc3RhdGUgdG8g bWFpbnRhaW4NCkBAIC0xMDQ1LDcgKzEwNDgsNyBAQCBhZ2FpbjoNCiAJCQkgKiBuZnNub2RlIGFm dGVyIHdlIGhhdmUgbG9ja2VkIHRoZSBidWZmZXIgdG8gcHJldmVudA0KIAkJCSAqIHJlYWRlcnMg ZnJvbSByZWFkaW5nIGdhcmJhZ2UuDQogCQkJICovDQotCQkJYmNvdW50ID0gb247DQorCQkJYmNv dW50ID0gbnAtPm5fc2l6ZSAtIChsYm4gKiBiaW9zaXplKTsNCiAJCQlicCA9IG5mc19nZXRjYWNo ZWJsayh2cCwgbGJuLCBiY291bnQsIHRkKTsNCiANCiAJCQlpZiAoYnAgIT0gTlVMTCkgew0KQEAg LTEwNTgsNyArMTA2MSw3IEBAIGFnYWluOg0KIAkJCQltdHhfdW5sb2NrKCZucC0+bl9tdHgpOw0K IA0KIAkJCQlzYXZlID0gYnAtPmJfZmxhZ3MgJiBCX0NBQ0hFOw0KLQkJCQliY291bnQgKz0gbjsN CisJCQkJYmNvdW50ID0gb24gKyBuOw0KIAkJCQlhbGxvY2J1ZihicCwgYmNvdW50KTsNCiAJCQkJ YnAtPmJfZmxhZ3MgfD0gc2F2ZTsNCiAJCQl9DQpAQCAtMTE1NCw2ICsxMTU3LDcgQEAgYWdhaW46 DQogCQlpZiAoYnAtPmJfZGlydHlvZmYgPj0gYnAtPmJfZGlydHllbmQpDQogCQkJYnAtPmJfZGly dHlvZmYgPSBicC0+Yl9kaXJ0eWVuZCA9IDA7DQogDQorI2lmIDANCiAJCS8qDQogCQkgKiBJZiB0 aGUgbmV3IHdyaXRlIHdpbGwgbGVhdmUgYSBjb250aWd1b3VzIGRpcnR5DQogCQkgKiBhcmVhLCBq dXN0IHVwZGF0ZSB0aGUgYl9kaXJ0eW9mZiBhbmQgYl9kaXJ0eWVuZCwNCkBAIC0xMTc5LDYgKzEx ODMsMTQgQEAgYWdhaW46DQogCQkJfQ0KIAkJCWdvdG8gYWdhaW47DQogCQl9DQorI2Vsc2UNCisJ CS8qDQorCQkgKiBSZWxheCBjb2hlcmVuY3kgYSBiaXQgZm9yIHRoZSBzYWtlIG9mIHBlcmZvcm1h bmNlIGFuZA0KKwkJICogZXhwYW5kIHRoZSBjdXJyZW50IGRpcnR5IHJlZ2lvbiB0byBjb250YWlu IHRoZSBuZXcNCisJCSAqIHdyaXRlIGV2ZW4gaWYgaXQgbWVhbnMgd2UgbWFyayBzb21lIG5vbi1k aXJ0eSBkYXRhIGFzDQorCQkgKiBkaXJ0eS4gIFRoaXMgc2hvdWxkIHByb2JhYmx5IGJlIGNvbmZp Z3VyYWJsZS4NCisJCSAqLw0KKyNlbmRpZg0KIA0KIAkJbG9jYWxfcmVzaWQgPSB1aW8tPnVpb19y ZXNpZDsNCiAJCWVycm9yID0gdm5faW9fZmF1bHRfdWlvbW92ZSgoY2hhciAqKWJwLT5iX2RhdGEg KyBvbiwgbiwgdWlvKTsNCmRpZmYgLS1naXQgYS9zeXMvbmZzY2xpZW50L25mc19iaW8uYyBiL3N5 cy9uZnNjbGllbnQvbmZzX2Jpby5jDQppbmRleCA2MzBhN2ZmLi45ZDhkYzdjIDEwMDY0NA0KLS0t IGEvc3lzL25mc2NsaWVudC9uZnNfYmlvLmMNCisrKyBiL3N5cy9uZnNjbGllbnQvbmZzX2Jpby5j DQpAQCAtMTEzMyw2ICsxMTMzLDcgQEAgYWdhaW46DQogCQlpZiAoYnAtPmJfZGlydHlvZmYgPj0g YnAtPmJfZGlydHllbmQpDQogCQkJYnAtPmJfZGlydHlvZmYgPSBicC0+Yl9kaXJ0eWVuZCA9IDA7 DQogDQorI2lmIDANCiAJCS8qDQogCQkgKiBJZiB0aGUgbmV3IHdyaXRlIHdpbGwgbGVhdmUgYSBj b250aWd1b3VzIGRpcnR5DQogCQkgKiBhcmVhLCBqdXN0IHVwZGF0ZSB0aGUgYl9kaXJ0eW9mZiBh bmQgYl9kaXJ0eWVuZCwNCkBAIC0xMTU4LDYgKzExNTksMTQgQEAgYWdhaW46DQogCQkJfQ0KIAkJ CWdvdG8gYWdhaW47DQogCQl9DQorI2Vsc2UNCisJCS8qDQorCQkgKiBSZWxheCBjb2hlcmVuY3kg YSBiaXQgZm9yIHRoZSBzYWtlIG9mIHBlcmZvcm1hbmNlIGFuZA0KKwkJICogZXhwYW5kIHRoZSBj dXJyZW50IGRpcnR5IHJlZ2lvbiB0byBjb250YWluIHRoZSBuZXcNCisJCSAqIHdyaXRlIGV2ZW4g aWYgaXQgbWVhbnMgd2UgbWFyayBzb21lIG5vbi1kaXJ0eSBkYXRhIGFzDQorCQkgKiBkaXJ0eS4g IFRoaXMgc2hvdWxkIHByb2JhYmx5IGJlIGNvbmZpZ3VyYWJsZS4NCisJCSAqLw0KKyNlbmRpZg0K IA0KIAkJZXJyb3IgPSB1aW9tb3ZlKChjaGFyICopYnAtPmJfZGF0YSArIG9uLCBuLCB1aW8pOw0K IA0K ------=_Part_291492_1969524791.1368362883961-- From owner-freebsd-fs@FreeBSD.ORG Mon May 13 00:37:24 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7EE17F2A for ; Mon, 13 May 2013 00:37:24 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.208.146]) by mx1.freebsd.org (Postfix) with ESMTP id 3440E8FD for ; Mon, 13 May 2013 00:37:23 +0000 (UTC) Received: from maia.hub.org (unknown [200.46.151.188]) by hub.org (Postfix) with ESMTP id AFA2D1BCB072; Sun, 12 May 2013 21:37:15 -0300 (ADT) Received: from hub.org ([200.46.208.146]) by maia.hub.org (mx1.hub.org [200.46.151.188]) (amavisd-maia, port 10024) with ESMTP id 12506-05; Mon, 13 May 2013 00:37:13 +0000 (UTC) Received: from Marc-Fourniers-Mac-mini.local (S01067cb21b2ff4ca.gv.shawcable.net [24.108.26.71]) by hub.org (Postfix) with ESMTPA id 2D93E1BCB06E; Sun, 12 May 2013 21:27:11 -0300 (ADT) Message-ID: <5190335D.9090105@hub.org> Date: Sun, 12 May 2013 17:27:09 -0700 From: "Marc G. Fournier" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Rick Macklem , freebsd-fs@freebsd.org Subject: Re: NFS Performance issue against NetApp References: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 May 2013 00:37:24 -0000 On 2013-05-12 5:48 AM, Rick Macklem wrote: > Marc G. Fournier wrote: >> >>> With >>> >>> vfs.nfs.noconsist=3 ... 385595ms >>> >>> nfsstat -z before startup, nfsstat -c after: >>> >>> Client Info: >>> Rpc Counts: >>> Getattr Setattr Lookup Readlink Read Write Create >>> Remove >>> 332594 5 17238 0 224426 231137 >>> 3743 1 >>> Rename Link Symlink Mkdir Rmdir Readdir >>> RdirPlus Access >>> 0 0 0 307 0 71 0 8447 >>> Mknod Fsstat Fsinfo PathConf Commit >>> 0 509 0 0 0 >>> Rpc Info: >>> TimedOut Invalid X Replies Retries Requests >>> 0 0 0 0 818479 >>> Cache Info: >>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW >>> Hits Misses >>> 608296 332596 526200 17245 -95425 224426 13178 >>> 231137 >>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs >>> Hits Misses >>> 0 0 1050 55 502 7 >>> 543340 8448 With patch applied: Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 236577 5 17311 0 233269 231136 3743 1 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 307 0 391 0 8488 Mknod Fsstat Fsinfo PathConf Commit 0 543 1 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 731770 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 714778 236578 529160 17312 -104087 233068 13178 231136 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 788 375 542 0 546435 8488 RPC Info Requests appear to be down but # of read/writes/getattr haven't changed any ... Why does it take 34x as many reads on FreeBSD, where rsize on both Linux/FreeBSD are the same ... ? The amount of data to be read is the same ... shouldn't the # of reads be within the same ballpark, at least ... ? > Ok, so disabling the mtime based cache consistency doesn't make > much difference. Forget about that one. > > I've attached another patch (which you probably shouldn't use for > a production system either) to be tried instead of the last one. > (This one is basically "work in progress" by Alexander Kabaev for > better performance during file linking. I hope he doesn't mind > me posting it.) > > rick > >>> ============ >>> >>> vfs.nfs.noconsist=2 ... 392201ms >>> >>> Client Info: >>> Rpc Counts: >>> Getattr Setattr Lookup Readlink Read Write Create >>> Remove >>> 332557 5 17228 0 224421 231131 >>> 3743 1 >>> Rename Link Symlink Mkdir Rmdir Readdir >>> RdirPlus Access >>> 0 0 0 307 0 72 0 8430 >>> Mknod Fsstat Fsinfo PathConf Commit >>> 0 502 0 0 0 >>> Rpc Info: >>> TimedOut Invalid X Replies Retries Requests >>> 0 0 0 0 818395 >>> Cache Info: >>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW >>> Hits Misses >>> 607834 332557 525801 17231 -95401 224421 13178 >>> 231131 >>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs >>> Hits Misses >>> 0 0 1028 56 502 0 >>> 542925 8431 >>> >>> >>> ============ >>> vfs.nfs.noconsist=0 ... 391622ms >>> >>> >>> Client Info: >>> Rpc Counts: >>> Getattr Setattr Lookup Readlink Read Write Create >>> Remove >>> 236122 5 17221 0 230575 230823 >>> 3743 1 >>> Rename Link Symlink Mkdir Rmdir Readdir >>> RdirPlus Access >>> 0 0 0 307 0 71 0 8425 >>> Mknod Fsstat Fsinfo PathConf Commit >>> 0 516 0 0 0 >>> Rpc Info: >>> TimedOut Invalid X Replies Retries Requests >>> 0 0 0 0 727799 >>> Cache Info: >>> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW >>> Hits Misses >>> 711860 236124 526549 17225 -101525 230490 13178 >>> 230823 >>> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs >>> Hits Misses >>> 0 0 1057 55 516 0 >>> 543709 8425 >>> >>> >>> I checked a second time with nonconsist=0, and the nfsstat -c values >>> seem to come out pretty much the same ... >>> >>> I'm going to head down to the office and try again with Solaris (I'd >>> have to re-install, since I used that system for the Solaris), and >>> see >>> what nfsstat -c results I get out of that ... will post a followup >>> on >>> this when completed ... >>> >>> >>> >>> On 2013-05-10 5:32 PM, Rick Macklem wrote: >>>> Marc G. Fournier wrote: >>>>> FYI … I just installed Solaris 11 onto the same hardware and ran >>>>> the >>>>> same test … so far, I'm seeing: >>>>> >>>>> Linux @ ~30s >>>>> Solaris @ ~44s >>>>> >>>>> OpenBSD @ ~200s >>>>> FreeBSD @ ~240s >>>>> >>>>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' >>>>> issue >>>>> … same as 9.x … I could see Linux 'cutting corners', but >>>>> Oracle/Solaris too … ? >>>>> >>>> The three client implementations (BSD, Linux, Solaris) were >>>> developed >>>> independently and, as such, will all implement somewaht different >>>> caching algorithms (the RFCs specify what goes on the wire, but say >>>> little w.r.t. client side caching). >>>> >>>> I have a attached a patch that might be useful for determining if >>>> the client side buffer cache consistency algorithm in FreeBSD is >>>> causing the slow startup of jboss. Do not run this patch on a >>>> production system, since it pretty well disables all buffer cache >>>> coherency (ie. if another client modifies a file, the patched >>>> client >>>> won't notice and will continue to cache stale file data). >>>> >>>> If the patch does speed up startup of jboss significantly, you can >>>> use the sysctl: >>>> vfs.nfs.noconsist >>>> to check for which coherency check is involved by decreasing the >>>> value for the sysctl by 1 and then trying a startup again. (When >>>> vfs.nfs.noconsist=0, normal cache coherency will be applied.) >>>> >>>> I have no idea if buffer cache coherency is a factor, but trying >>>> the attached patch might determine if it is. >>>> >>>> Note that you have never posted updated "nfsstat -c" values. >>>> (Remember that what you posted indicated 88 RPCs, which seemed >>>> bogus.) Finding out if FreeBSD does a lot more of certain RPCs >>>> that Linux/Solaris might help isolate what is going on. >>>> >>>> rick >>>> >>>>> On 2013-05-03, at 04:50 , Mark Felder wrote: >>>>> >>>>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier >>>>>> wrote: >>>>>> >>>>>>> Hadn't thought to do so with Linux, but … >>>>>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms >>>>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms >>>>>> Please make sure both platforms are using similar atime settings. >>>>>> I >>>>>> think most distros use ext4 with diratime by default. I'd just do >>>>>> noatime on both platforms to be safe. >>>>>> _______________________________________________ >>>>>> freebsd-fs@freebsd.org mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>> To unsubscribe, send any mail to >>>>>> "freebsd-fs-unsubscribe@freebsd.org" >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to >>>>> "freebsd-fs-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to >>> "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon May 13 00:59:01 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 369CC2DC for ; Mon, 13 May 2013 00:59:01 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:32]) by mx1.freebsd.org (Postfix) with ESMTP id 0517796D for ; Mon, 13 May 2013 00:59:00 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta03.emeryville.ca.mail.comcast.net with comcast id bC941l0031afHeLA3Cz0rj; Mon, 13 May 2013 00:59:00 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id bCyz1l0051t3BNj8dCyz7r; Mon, 13 May 2013 00:58:59 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id EE6AA73A33; Sun, 12 May 2013 17:58:58 -0700 (PDT) Date: Sun, 12 May 2013 17:58:58 -0700 From: Jeremy Chadwick To: "Marc G. Fournier" Subject: Re: NFS Performance issue against NetApp Message-ID: <20130513005858.GA73875@icarus.home.lan> References: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> <5190335D.9090105@hub.org> MIME-Version: 1.0 Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5190335D.9090105@hub.org> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1368406740; bh=ejuZH+sbUXMh5H7LzzdE+Ubo8u292EsjkIFljP4jqmE=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=XXNNj/A20zBmHLKv02TymyWvbX3NI8QXyO71CkZscScf9S95WMZsF3TZo06BeNdjf +6v93PPBrqT5z49jEPX213EwVb+PgZlgzaluIsQUv2HXYoZaagpogWo36oVaMc7JJ7 sUfKNDkdLH2GqOPPuNvmSSnlJxIJDQUks7Q7uyPP2NS0/PEpDnB1+Dd2Nmd+Pnnyx+ cN/6QrjsdYbiuarjp7hfrApLzMPcMPo2xhBsfyvmNWnAuUrNI6509gbXoEz/HkUqAq d83fCgTeZYPhtGf4jH5qdTC6haYTva0BWCLf/LN6pN/ikSrqY6RaaUE/KyVQtusAt+ xu2zRrZfpitog== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 May 2013 00:59:01 -0000 On Sun, May 12, 2013 at 05:27:09PM -0700, Marc G. Fournier wrote: > On 2013-05-12 5:48 AM, Rick Macklem wrote: > >Marc G. Fournier wrote: > >> > >>>With > >>> > >>>vfs.nfs.noconsist=3 ... 385595ms > >>> > >>>nfsstat -z before startup, nfsstat -c after: > >>> > >>>Client Info: > >>>Rpc Counts: > >>> Getattr Setattr Lookup Readlink Read Write Create > >>>Remove > >>> 332594 5 17238 0 224426 231137 > >>>3743 1 > >>> Rename Link Symlink Mkdir Rmdir Readdir > >>>RdirPlus Access > >>> 0 0 0 307 0 71 0 8447 > >>> Mknod Fsstat Fsinfo PathConf Commit > >>> 0 509 0 0 0 > >>>Rpc Info: > >>> TimedOut Invalid X Replies Retries Requests > >>> 0 0 0 0 818479 > >>>Cache Info: > >>>Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > >>>Hits Misses > >>> 608296 332596 526200 17245 -95425 224426 13178 > >>>231137 > >>>BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > >>>Hits Misses > >>> 0 0 1050 55 502 7 > >>>543340 8448 > > With patch applied: > > Client Info: > Rpc Counts: > Getattr Setattr Lookup Readlink Read Write Create > Remove > 236577 5 17311 0 233269 231136 3743 1 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > Access > 0 0 0 307 0 391 0 8488 > Mknod Fsstat Fsinfo PathConf Commit > 0 543 1 0 0 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 731770 > Cache Info: > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > Hits Misses > 714778 236578 529160 17312 -104087 233068 13178 231136 > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > Hits Misses > 0 0 788 375 542 0 546435 > 8488 > > RPC Info Requests appear to be down but # of read/writes/getattr > haven't changed any ... > > Why does it take 34x as many reads on FreeBSD, where rsize on both > Linux/FreeBSD are the same ... ? The amount of data to be read is > the same ... shouldn't the # of reads be within the same ballpark, > at least ... ? > > > >Ok, so disabling the mtime based cache consistency doesn't make > >much difference. Forget about that one. > > > >I've attached another patch (which you probably shouldn't use for > >a production system either) to be tried instead of the last one. > >(This one is basically "work in progress" by Alexander Kabaev for > > better performance during file linking. I hope he doesn't mind > > me posting it.) > > > >rick > > > >>>============ > >>> > >>>vfs.nfs.noconsist=2 ... 392201ms > >>> > >>>Client Info: > >>>Rpc Counts: > >>> Getattr Setattr Lookup Readlink Read Write Create > >>>Remove > >>> 332557 5 17228 0 224421 231131 > >>>3743 1 > >>> Rename Link Symlink Mkdir Rmdir Readdir > >>>RdirPlus Access > >>> 0 0 0 307 0 72 0 8430 > >>> Mknod Fsstat Fsinfo PathConf Commit > >>> 0 502 0 0 0 > >>>Rpc Info: > >>> TimedOut Invalid X Replies Retries Requests > >>> 0 0 0 0 818395 > >>>Cache Info: > >>>Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > >>>Hits Misses > >>> 607834 332557 525801 17231 -95401 224421 13178 > >>>231131 > >>>BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > >>>Hits Misses > >>> 0 0 1028 56 502 0 > >>>542925 8431 > >>> > >>> > >>>============ > >>>vfs.nfs.noconsist=0 ... 391622ms > >>> > >>> > >>>Client Info: > >>>Rpc Counts: > >>> Getattr Setattr Lookup Readlink Read Write Create > >>>Remove > >>> 236122 5 17221 0 230575 230823 > >>>3743 1 > >>> Rename Link Symlink Mkdir Rmdir Readdir > >>>RdirPlus Access > >>> 0 0 0 307 0 71 0 8425 > >>> Mknod Fsstat Fsinfo PathConf Commit > >>> 0 516 0 0 0 > >>>Rpc Info: > >>> TimedOut Invalid X Replies Retries Requests > >>> 0 0 0 0 727799 > >>>Cache Info: > >>>Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > >>>Hits Misses > >>> 711860 236124 526549 17225 -101525 230490 13178 > >>>230823 > >>>BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > >>>Hits Misses > >>> 0 0 1057 55 516 0 > >>>543709 8425 > >>> > >>> > >>>I checked a second time with nonconsist=0, and the nfsstat -c values > >>>seem to come out pretty much the same ... > >>> > >>>I'm going to head down to the office and try again with Solaris (I'd > >>>have to re-install, since I used that system for the Solaris), and > >>>see > >>>what nfsstat -c results I get out of that ... will post a followup > >>>on > >>>this when completed ... > >>> > >>> > >>> > >>>On 2013-05-10 5:32 PM, Rick Macklem wrote: > >>>>Marc G. Fournier wrote: > >>>>>FYI … I just installed Solaris 11 onto the same hardware and ran > >>>>>the > >>>>>same test … so far, I'm seeing: > >>>>> > >>>>>Linux @ ~30s > >>>>>Solaris @ ~44s > >>>>> > >>>>>OpenBSD @ ~200s > >>>>>FreeBSD @ ~240s > >>>>> > >>>>>I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' > >>>>>issue > >>>>>… same as 9.x … I could see Linux 'cutting corners', but > >>>>>Oracle/Solaris too … ? > >>>>> > >>>>The three client implementations (BSD, Linux, Solaris) were > >>>>developed > >>>>independently and, as such, will all implement somewaht different > >>>>caching algorithms (the RFCs specify what goes on the wire, but say > >>>>little w.r.t. client side caching). > >>>> > >>>>I have a attached a patch that might be useful for determining if > >>>>the client side buffer cache consistency algorithm in FreeBSD is > >>>>causing the slow startup of jboss. Do not run this patch on a > >>>>production system, since it pretty well disables all buffer cache > >>>>coherency (ie. if another client modifies a file, the patched > >>>>client > >>>>won't notice and will continue to cache stale file data). > >>>> > >>>>If the patch does speed up startup of jboss significantly, you can > >>>>use the sysctl: > >>>> vfs.nfs.noconsist > >>>>to check for which coherency check is involved by decreasing the > >>>>value for the sysctl by 1 and then trying a startup again. (When > >>>>vfs.nfs.noconsist=0, normal cache coherency will be applied.) > >>>> > >>>>I have no idea if buffer cache coherency is a factor, but trying > >>>>the attached patch might determine if it is. > >>>> > >>>>Note that you have never posted updated "nfsstat -c" values. > >>>>(Remember that what you posted indicated 88 RPCs, which seemed > >>>> bogus.) Finding out if FreeBSD does a lot more of certain RPCs > >>>>that Linux/Solaris might help isolate what is going on. > >>>> > >>>>rick > >>>> > >>>>>On 2013-05-03, at 04:50 , Mark Felder wrote: > >>>>> > >>>>>>On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier > >>>>>> wrote: > >>>>>> > >>>>>>>Hadn't thought to do so with Linux, but … > >>>>>>>Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms > >>>>>>>FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms > >>>>>>Please make sure both platforms are using similar atime settings. > >>>>>>I > >>>>>>think most distros use ext4 with diratime by default. I'd just do > >>>>>>noatime on both platforms to be safe. Quoting/top-reply/Email-clients-suck-balls madness has made this thread basically impossible to follow visually at this point, so I'm replying at the bottom quoting what was said above: > Why does it take 34x as many reads on FreeBSD, where rsize on both > Linux/FreeBSD are the same ... ? The amount of data to be read is > the same ... shouldn't the # of reads be within the same ballpark, > at least ... ? Can you provide actual proof of this, re: that at the syscall level, JBoss is issuing the same nbytes count to read(2) on BSD as it is on Linux and Solaris? You can accomplish this with strace on Linux and ktrace (not truss) on FreeBSD. For all we know, JBoss could have some "OS optimisation" setting within it that says for BSDs use 512 bytes, while for Linux and/or Solaris use 16384 bytes. Nothing mandates that a framework/language/whatever use the same size across all OSes. In fact, there was already a statement made earlier in this thread about something doing file I/O with a byte size of ***1 byte*** for reads and writes: http://lists.freebsd.org/pipermail/freebsd-fs/2013-May/017166.html Welcome to why "abstracted" languages with a crap-load of "middleman" layers are very, very hard to debug/troubleshoot, particularly with regards to performance. KISS principle mandates the less crap between the application and the syscall, the easier it is to figure out. My advice at this point: take JBoss out of the picture. Use some other file-based I/O test, like benchmarks/bonnie++ or benchmarks/fio, against a file that's backed by an NFS mount. Do not ask me if these are good utilities to test such behaviour, or what arguments to use. If it turns out JBoss does not behave/play nicely on the BSDs, I won't be surprised. Probably off-topic but worth pointing out: I do not know about Solaris, but Linux has multiple layers of caching, and is well-known for doing things like caching (and aggregating!) reads/writes to **block** devices (this is why on Linux you have to make sure to avoid caching your application use O_DIRECT with open(2) or other mechanisms -- the BSDs do not do this, block devices are always non-cached). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Mon May 13 11:06:44 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 110A2989 for ; Mon, 13 May 2013 11:06:44 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id E8CE483C for ; Mon, 13 May 2013 11:06:43 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r4DB6hkW075843 for ; Mon, 13 May 2013 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r4DB6hsw075841 for freebsd-fs@FreeBSD.org; Mon, 13 May 2013 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 13 May 2013 11:06:43 GMT Message-Id: <201305131106.r4DB6hsw075841@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 May 2013 11:06:44 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/178412 fs [smbfs] Coredump when smbfs mounted o kern/178388 fs [zfs] [patch] allow up to 8MB recordsize o kern/178349 fs [zfs] zfs scrub on deduped data could be much less see o kern/178329 fs [zfs] extended attributes leak o kern/177985 fs [zfs] disk usage problem when copying from one zfs dat o kern/177971 fs [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3, o kern/177966 fs [zfs] resilver completes but subsequent scrub reports o kern/177658 fs [ufs] FreeBSD panics after get full filesystem with uf o kern/177536 fs [zfs] zfs livelock (deadlock) with high write-to-disk o kern/177445 fs [hast] HAST panic o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/174060 fs [ext2fs] Ext2FS system crashes (buffer overflow?) o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172942 fs [smbfs] Unmounting a smb mount when the server became o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic f kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 312 problems total. From owner-freebsd-fs@FreeBSD.ORG Tue May 14 17:12:49 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1BB6CBDA for ; Tue, 14 May 2013 17:12:49 +0000 (UTC) (envelope-from scrappy@hub.org) Received: from hub.org (hub.org [200.46.208.146]) by mx1.freebsd.org (Postfix) with ESMTP id DFAC3B55 for ; Tue, 14 May 2013 17:12:48 +0000 (UTC) Received: from maia.hub.org (unknown [200.46.151.188]) by hub.org (Postfix) with ESMTP id 439B5E79DD8; Tue, 14 May 2013 14:12:42 -0300 (ADT) Received: from hub.org ([200.46.208.146]) by maia.hub.org (mx1.hub.org [200.46.151.188]) (amavisd-maia, port 10024) with ESMTP id 11526-06; Tue, 14 May 2013 17:12:41 +0000 (UTC) Received: from [10.5.250.150] (remote.ilcs.sd63.bc.ca [142.31.148.2]) by hub.org (Postfix) with ESMTPA id 0C720E79DD7; Tue, 14 May 2013 14:12:39 -0300 (ADT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: NFS Performance issue against NetApp From: "Marc G. Fournier" In-Reply-To: <20130513005858.GA73875@icarus.home.lan> Date: Tue, 14 May 2013 10:12:37 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <94661399-66AC-4E83-B39B-0426442BB84C@hub.org> References: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> <5190335D.9090105@hub.org> <20130513005858.GA73875@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1503) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 17:12:49 -0000 On 2013-05-12, at 17:58 , Jeremy Chadwick wrote: >=20 >> Why does it take 34x as many reads on FreeBSD, where rsize on both >> Linux/FreeBSD are the same ... ? The amount of data to be read is >> the same ... shouldn't the # of reads be within the same ballpark, >> at least ... ? >=20 > Can you provide actual proof of this, re: that at the syscall level, > JBoss is issuing the same nbytes count to read(2) on BSD as it is on > Linux and Solaris? You can accomplish this with strace on Linux and > ktrace (not truss) on FreeBSD. >=20 > For all we know, JBoss could have some "OS optimisation" setting = within > it that says for BSDs use 512 bytes, while for Linux and/or Solaris = use > 16384 bytes. =20 That would explain a 4x difference .. not a 34x difference =85=20 As for strace, if anyone feels this has merit and can tell me what I = need to run, I'm happy to do so =85=20 > Probably off-topic but worth pointing out: I do not know about = Solaris, > but Linux has multiple layers of caching, and is well-known for doing > things like caching (and aggregating!) reads/writes to **block** = devices > (this is why on Linux you have to make sure to avoid caching your > application use O_DIRECT with open(2) or other mechanisms -- the BSDs = do > not do this, block devices are always non-cached). Caching *should* only come into play after the first run of the = application =85 the first run after a reboot of the server shouldn't = have anything in cache yet for caching to come into play =85 unless = Linux/Solaris have some way of anticipating what application I'm going = to run before me =85 ? From owner-freebsd-fs@FreeBSD.ORG Tue May 14 19:08:10 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 74F9019E for ; Tue, 14 May 2013 19:08:10 +0000 (UTC) (envelope-from henner.heck@web.de) Received: from mout.web.de (mout.web.de [212.227.17.12]) by mx1.freebsd.org (Postfix) with ESMTP id 1B1402F0 for ; Tue, 14 May 2013 19:08:09 +0000 (UTC) Received: from sender ([77.3.187.219]) by smtp.web.de (mrweb003) with ESMTPSA (Nemesis) id 0MOAmi-1UZ2PW4Ajc-006Dwu for ; Tue, 14 May 2013 20:43:23 +0200 Message-ID: <519285C9.8000306@web.de> Date: Tue, 14 May 2013 20:43:21 +0200 From: Henner Heck User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> In-Reply-To: <518F4307.3060908@hub.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:JYA8Ot6S1idQw7asAG/62L181aoKAOC7Qa1w+yHCX6s kyq5qTiOogVrWLY3AS0lQu/81rtpwZfsQ0vAZI/RwTyBTKfhVw KpgM3aK8jOZaUTwY5DQ9T3wOv8lDFp88lyNYX+kENN5Mpmq06n GMvyYdEONJ4c3XKyeTK+z8qE/g5n/oCTwLhrEMKTtf3nYpm2P/ FdmJ2AFDfNP2ad9dtY1Cw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 19:08:10 -0000 Hello all, i set up a PC with FreeBSD 9.1-Release and 2 zfs pools. One is a mirror from which FreeBSD is booting (tough enough without getting "error 2"), the other one a raidz2 for data. The disks for the raidz2 are encrypted with geli and attached manually. I noticed that a "zpool status" or a "zfs list" before attaching the encrypted disks waits for about one minute before showing output. When they finally do, the output is as expected, the raidz2 pool is shown as UNAVAIL and its datasets are not listed. When all the disks are attached with geli, the outputs are given immediately. On boot there are 2 delays. One of about 1 minute after the output "Setting hostid: 0x........" and one of 2 minutes after "Mounting local file systems:.". Both these outputs don't show up in dmesg, which ends with "Trying to mount root from zfs:zroot/ROOT []..." shortly before. I suspect that the boot delays too are because of the encrypted pool. A different machine running FreeBSD 8.3-RELEASE has a delay of only about 3 seconds on "zpool status" with an encrypted pool and also the boot shows no annoying anomalies. Any idea how to get rid of these really long delays? Regards, Henner Heck Some more info: uname -a FreeBSD titan 9.1-RELEASE-p3 FreeBSD 9.1-RELEASE-p3 #0: Mon Apr 29 18:27:25 UTC 2013 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 rc.conf hostname="titan" keymap="german.iso.kbd" ifconfig_em0=" inet 192.168.1.21 netmask 255.255.255.0" defaultrouter="192.168.1.1" sshd_enable="YES" ntpd_enable="YES" powerd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="NO" zfs_enable="YES" geli_autodetach="NO" samba_enable="YES" loader.conf zfs_load="YES" vfs.root.mountfrom="zfs:zroot/ROOT" geom_eli_load="YES" aio_load="YES" "zpool status" before attaching (1min delay): pool: titanpool state: UNAVAIL status: One or more devices could not be opened. There are insufficient replicas for the pool to continue functioning. action: Attach the missing device and online it using 'zpool online'. see: http://illumos.org/msg/ZFS-8000-3C scan: none requested config: NAME STATE READ WRITE CKSUM titanpool UNAVAIL 0 0 0 raidz2-0 UNAVAIL 0 0 0 17524249743005198207 UNAVAIL 0 0 0 was /dev/gpt/titan01.eli 12400954011483674215 UNAVAIL 0 0 0 was /dev/gpt/titan02.eli 4127776205786896399 UNAVAIL 0 0 0 was /dev/gpt/titan03.eli 13439834871331336588 UNAVAIL 0 0 0 was /dev/gpt/titan04.eli 7734910905079966692 UNAVAIL 0 0 0 was /dev/gpt/titan05.eli 17344716162596142682 UNAVAIL 0 0 0 was /dev/gpt/titan06.eli 11943961185890967830 UNAVAIL 0 0 0 was /dev/gpt/titan07.eli 13738344899380447289 UNAVAIL 0 0 0 was /dev/gpt/titan08.eli 18205195048240167252 UNAVAIL 0 0 0 was /dev/gpt/titan09.eli 12338698010126903234 UNAVAIL 0 0 0 was /dev/gpt/titan10.eli pool: zroot state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Mon May 13 04:42:28 2013 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors "zpool status" after attaching (no delay): pool: titanpool state: ONLINE scan: scrub repaired 0 in 1h35m with 0 errors on Mon May 13 06:18:22 2013 config: NAME STATE READ WRITE CKSUM titanpool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/titan01.eli ONLINE 0 0 0 gpt/titan02.eli ONLINE 0 0 0 gpt/titan03.eli ONLINE 0 0 0 gpt/titan04.eli ONLINE 0 0 0 gpt/titan05.eli ONLINE 0 0 0 gpt/titan06.eli ONLINE 0 0 0 gpt/titan07.eli ONLINE 0 0 0 gpt/titan08.eli ONLINE 0 0 0 gpt/titan09.eli ONLINE 0 0 0 gpt/titan10.eli ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Mon May 13 04:42:28 2013 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors From owner-freebsd-fs@FreeBSD.ORG Tue May 14 19:22:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 73E05CA6 for ; Tue, 14 May 2013 19:22:44 +0000 (UTC) (envelope-from prvs=18460eef42=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 03C1D5E0 for ; Tue, 14 May 2013 19:22:43 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003804937.msg for ; Tue, 14 May 2013 20:22:42 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 14 May 2013 20:22:42 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18460eef42=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Henner Heck" , References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) Date: Tue, 14 May 2013 20:22:39 +0100 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_09AF_01CE50E0.C7C3AFF0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 19:22:44 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_09AF_01CE50E0.C7C3AFF0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit ----- Original Message ----- From: "Henner Heck" > Hello all, > > i set up a PC with FreeBSD 9.1-Release and 2 zfs pools. > One is a mirror from which FreeBSD is booting > (tough enough without getting "error 2"), > the other one a raidz2 for data. > The disks for the raidz2 are encrypted with geli and attached manually. > > I noticed that a "zpool status" or a "zfs list" before attaching > the encrypted disks waits for about one minute before showing output. > When they finally do, the output is as expected, the raidz2 pool is shown as > UNAVAIL and its datasets are not listed. > When all the disks are attached with geli, the outputs are given > immediately. > > On boot there are 2 delays. > One of about 1 minute after the output > "Setting hostid: 0x........" > and one of 2 minutes after > "Mounting local file systems:.". > Both these outputs don't show up in dmesg, which ends with > "Trying to mount root from zfs:zroot/ROOT []..." shortly before. > > I suspect that the boot delays too are because of the encrypted pool. > > A different machine running FreeBSD 8.3-RELEASE has a delay of only > about 3 seconds on "zpool status" with an encrypted pool > and also the boot shows no annoying anomalies. > > Any idea how to get rid of these really long delays? Try the attached patch see if that helps? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. ------=_NextPart_000_09AF_01CE50E0.C7C3AFF0 Content-Type: application/octet-stream; name="zfs-slice-boot.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="zfs-slice-boot.patch" --- sys/boot/zfs/zfs.c.orig 2011-10-20 18:15:29.966685430 +0000=0A= +++ sys/boot/zfs/zfs.c 2011-10-20 18:18:22.291033636 +0000=0A= @@ -45,6 +45,12 @@=0A= =0A= #include "zfsimpl.c"=0A= =0A= +/*=0A= + * For GPT this should be 128 but leads to 50+ second delay in BTX = loader so=0A= + * we use the original 4 pre r198420 by default for the boot process=0A= + */=0A= +#define ZFS_MAX_SLICES 4=0A= +=0A= static int zfs_open(const char *path, struct open_file *f);=0A= static int zfs_write(struct open_file *f, void *buf, size_t size, = size_t *resid);=0A= static int zfs_close(struct open_file *f);=0A= @@ -415,7 +421,7 @@=0A= if (vdev_probe(vdev_read, (void*) (uintptr_t) fd, 0))=0A= close(fd);=0A= =0A= - for (slice =3D 1; slice <=3D 128; slice++) {=0A= + for (slice =3D 1; slice <=3D ZFS_MAX_SLICES; slice++) {=0A= sprintf(devname, "disk%dp%d:", unit, slice);=0A= fd =3D open(devname, O_RDONLY);=0A= if (fd =3D=3D -1) {=0A= ------=_NextPart_000_09AF_01CE50E0.C7C3AFF0-- From owner-freebsd-fs@FreeBSD.ORG Tue May 14 19:50:18 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 590219A9 for ; Tue, 14 May 2013 19:50:18 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta05.emeryville.ca.mail.comcast.net (qmta05.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:48]) by mx1.freebsd.org (Postfix) with ESMTP id 32E967EC for ; Tue, 14 May 2013 19:50:18 +0000 (UTC) Received: from omta04.emeryville.ca.mail.comcast.net ([76.96.30.35]) by qmta05.emeryville.ca.mail.comcast.net with comcast id buvL1l0060lTkoCA5vqHuu; Tue, 14 May 2013 19:50:17 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta04.emeryville.ca.mail.comcast.net with comcast id bvqG1l00P1t3BNj8QvqHdd; Tue, 14 May 2013 19:50:17 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id C8A3373A33; Tue, 14 May 2013 12:50:16 -0700 (PDT) Date: Tue, 14 May 2013 12:50:16 -0700 From: Jeremy Chadwick To: Henner Heck Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) Message-ID: <20130514195016.GA16117@icarus.home.lan> References: <519285C9.8000306@web.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <519285C9.8000306@web.de> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1368561017; bh=PAhhBiPiPTJu+WtBhaaxlr0szpc/UxbCbkOt2d1Z5B4=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=B/rB5T9ASX22xoIO0M6owoSb1u0G3mhbdd2q2oOAvZE8a9XHj1cDiRgQqR0sHw/ol L0AQeKj0qcuyzXUX+Nh1mTLEt6AhEDYJTvNJCq84YBx0MjRQ9kdhzlGzZB9t/Z0Vfg buJ154D39IGyydMdSHY1kQl6HmIJ/atEqV5yMNvhnmBb0EMD8Dw0B7HN1pjss1bTR/ mGgT0jBQftPXHgc3qko8d5ojiHh8E5PoL/QUjniDmkWz1Pe/lb3NX815ZAuYU/u426 fN38/j+5cS+82KWG2ngfZzp+DX7CXwkLmZIo6mifwcnmC1svbNvsnczUqrrZzI+n3A F/Pev58pYdykw== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 19:50:18 -0000 On Tue, May 14, 2013 at 08:43:21PM +0200, Henner Heck wrote: > ... > "Setting hostid: 0x........" > ... > "Mounting local file systems:.". > Both these outputs don't show up in dmesg ... This is normal (and has been normal since as long as I can remember): those messages are being output by the rc(8) subsystem and being echo'd to /dev/console. If you want to see these messages, use the -a flag with dmesg(8). P.S. -- When starting a new topic/thread about something, please do not reply to an existing thread and change the subject -- this is insufficient, and is sometimes called "thread hijacking". Some (many, and often UNIX) mail clients use a header called In-Reply-To to "cross-reference" mails to one another regardless of Subject. Proof of you doing this: 171 05/12 00:21 Marc G. Fournier (0.6K) |-> 172 N 05/14 20:43 Henner Heck (5.0K) | `->Long delays during boot and zpool/zfs commands on 9. 173 N 05/14 20:22 Steven Hartland (3.6K) | `-> And your mail client had this in it: > From: Henner Heck > To: freebsd-fs@freebsd.org > Date: Tue, 14 May 2013 20:43:21 +0200 > Subject: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) > References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> > <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> > <518F4307.3060908@hub.org> > In-Reply-To: <518F4307.3060908@hub.org> So in the future actually make a new mail (new thread). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue May 14 20:11:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 92311374 for ; Tue, 14 May 2013 20:11:54 +0000 (UTC) (envelope-from henner.heck@web.de) Received: from mout.web.de (mout.web.de [212.227.15.3]) by mx1.freebsd.org (Postfix) with ESMTP id 11ED5912 for ; Tue, 14 May 2013 20:11:53 +0000 (UTC) Received: from sender ([77.3.187.219]) by smtp.web.de (mrweb002) with ESMTPSA (Nemesis) id 0MNc1S-1UZZMK0qkc-007YYs; Tue, 14 May 2013 22:11:50 +0200 Message-ID: <51929A85.2090503@web.de> Date: Tue, 14 May 2013 22:11:49 +0200 From: Henner Heck User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) References: <519285C9.8000306@web.de> <20130514195016.GA16117@icarus.home.lan> In-Reply-To: <20130514195016.GA16117@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:p33P/JYwJYBmTt22l5y7tRP6LjTJZSvbmQKvgjSmG44 2bGv5JRcRn/r5pygd5R+Gg5ogNWyzwBeijwW3NNVdqPEr2Z+1b tgKlbJKJUj6v/euNM7h4EgVwvokk2KGIKkjsjt1/AdjwchRrit +3gSBIrBPtSddPHO8wbvT9qvSWnFq3pHSERSohBoKpCJoNh15d fxGOKYd254/55UmJhSf+Q== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 20:11:54 -0000 Am 14.05.2013 21:50, schrieb Jeremy Chadwick: > On Tue, May 14, 2013 at 08:43:21PM +0200, Henner Heck wrote: >> ... >> "Setting hostid: 0x........" >> ... >> "Mounting local file systems:.". >> Both these outputs don't show up in dmesg ... > This is normal (and has been normal since as long as I can remember): > those messages are being output by the rc(8) subsystem and being echo'd > to /dev/console. > > If you want to see these messages, use the -a flag with dmesg(8). > > > P.S. -- When starting a new topic/thread about something, please do not > reply to an existing thread and change the subject -- this is > insufficient, and is sometimes called "thread hijacking". Some (many, > and often UNIX) mail clients use a header called In-Reply-To to > "cross-reference" mails to one another regardless of Subject. Proof of > you doing this: > > 171 05/12 00:21 Marc G. Fournier (0.6K) |-> > 172 N 05/14 20:43 Henner Heck (5.0K) | `->Long delays during boot and zpool/zfs commands on 9. > 173 N 05/14 20:22 Steven Hartland (3.6K) | `-> > > And your mail client had this in it: > >> From: Henner Heck >> To: freebsd-fs@freebsd.org >> Date: Tue, 14 May 2013 20:43:21 +0200 >> Subject: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) >> References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> >> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> >> <518F4307.3060908@hub.org> >> In-Reply-To: <518F4307.3060908@hub.org> > So in the future actually make a new mail (new thread). > Sorry about that, i just took a mail from the list to get the address "freebsd-fs@freebsd.org" and cleaned everything up not knowing about Thunderbird setting these headers. From owner-freebsd-fs@FreeBSD.ORG Tue May 14 21:10:16 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EF4DF75E for ; Tue, 14 May 2013 21:10:16 +0000 (UTC) (envelope-from prvs=18460eef42=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 92B14D56 for ; Tue, 14 May 2013 21:10:16 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003806304.msg for ; Tue, 14 May 2013 22:10:12 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 14 May 2013 22:10:12 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18460eef42=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Ajit Jain" , References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> Subject: Re: seeing data corruption with zfs trim functionality Date: Tue, 14 May 2013 22:10:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 21:10:17 -0000 ----- Original Message ----- From: "Steven Hartland" >>> What version are you porting the changes to? >>> >>> What SSD are you using? >>> >>> What LSI controller are you using? >> >> I'd also like to see "zpool status" (for every pool that involves this >> SSD) and "gpart show" against the disk itself. > > Also: > 1. What FW version is your LSI? You can get this from dmesg. > 2. The exact command line your running iotest with? Any update on this? I'd like to try and replicate your test here so would appreciate as much information as possible. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Tue May 14 21:33:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3BA17465 for ; Tue, 14 May 2013 21:33:14 +0000 (UTC) (envelope-from henner.heck@web.de) Received: from mout.web.de (mout.web.de [212.227.15.4]) by mx1.freebsd.org (Postfix) with ESMTP id 838E5ECA for ; Tue, 14 May 2013 21:33:13 +0000 (UTC) Received: from sender ([77.3.187.219]) by smtp.web.de (mrweb102) with ESMTPSA (Nemesis) id 0LxODe-1UQj3E4AnK-0175SK; Tue, 14 May 2013 23:33:11 +0200 Message-ID: <5192AD94.5020707@web.de> Date: Tue, 14 May 2013 23:33:08 +0200 From: Henner Heck User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Steven Hartland Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> In-Reply-To: Content-Type: multipart/mixed; boundary="------------040909000208010909070704" X-Provags-ID: V02:K0:nAML/bvz9aYLTQ4fEcH3ffI0gB3elg9nkiA6/lEcNdE BiBj6g0AJwbeejZLwK5chfz0yc+r6mBephiM0D8MWDcwKvdesO n2V0nYIw0xzkR6iBm+OEg2W+HbBW3yz6HArpc588tK7W0E81fG m6wZElM749ZOMq+U1TuEbjDnPfR24EKJEnllZVYj28HwepG57w 44Il/yuafTuTN9BQ4uwOA== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 21:33:14 -0000 This is a multi-part message in MIME format. --------------040909000208010909070704 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Am 14.05.2013 21:22, schrieb Steven Hartland: > ----- Original Message ----- From: "Henner Heck" >> Hello all, >> >> i set up a PC with FreeBSD 9.1-Release and 2 zfs pools. >> One is a mirror from which FreeBSD is booting >> (tough enough without getting "error 2"), >> the other one a raidz2 for data. >> The disks for the raidz2 are encrypted with geli and attached manually. >> >> I noticed that a "zpool status" or a "zfs list" before attaching >> the encrypted disks waits for about one minute before showing output. >> When they finally do, the output is as expected, the raidz2 pool is >> shown as >> UNAVAIL and its datasets are not listed. >> When all the disks are attached with geli, the outputs are given >> immediately. >> >> On boot there are 2 delays. >> One of about 1 minute after the output >> "Setting hostid: 0x........" >> and one of 2 minutes after >> "Mounting local file systems:.". >> Both these outputs don't show up in dmesg, which ends with >> "Trying to mount root from zfs:zroot/ROOT []..." shortly before. >> >> I suspect that the boot delays too are because of the encrypted pool. >> >> A different machine running FreeBSD 8.3-RELEASE has a delay of only >> about 3 seconds on "zpool status" with an encrypted pool >> and also the boot shows no annoying anomalies. >> >> Any idea how to get rid of these really long delays? > > Try the attached patch see if that helps? > > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. Hello Steven, i tried to apply the patch, but patch didn't return and gave no output. I checked my the zfs.c and it doesn't seem to fit this patch. Please see attached file. Regards, Henner Heck --------------040909000208010909070704 Content-Type: text/plain; charset=windows-1252; name="zfs.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="zfs.c" /*- * Copyright (c) 2007 Doug Rabson * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD: release/9.1.0/sys/boot/zfs/zfs.c 237771 2012-06-29 10:30:59Z avg $ */ #include __FBSDID("$FreeBSD: release/9.1.0/sys/boot/zfs/zfs.c 237771 2012-06-29 10:30:59Z avg $"); /* * Stand-alone file reading package. */ #include #include #include #include #include #include #include #include #include #include "libzfs.h" #include "zfsimpl.c" static int zfs_open(const char *path, struct open_file *f); static int zfs_write(struct open_file *f, void *buf, size_t size, size_t *resid); static int zfs_close(struct open_file *f); static int zfs_read(struct open_file *f, void *buf, size_t size, size_t *resid); static off_t zfs_seek(struct open_file *f, off_t offset, int where); static int zfs_stat(struct open_file *f, struct stat *sb); static int zfs_readdir(struct open_file *f, struct dirent *d); struct devsw zfs_dev; struct fs_ops zfs_fsops = { "zfs", zfs_open, zfs_close, zfs_read, zfs_write, zfs_seek, zfs_stat, zfs_readdir }; /* * In-core open file. */ struct file { off_t f_seekp; /* seek pointer */ dnode_phys_t f_dnode; uint64_t f_zap_type; /* zap type for readdir */ uint64_t f_num_leafs; /* number of fzap leaf blocks */ zap_leaf_phys_t *f_zap_leaf; /* zap leaf buffer */ }; /* * Open a file. */ static int zfs_open(const char *upath, struct open_file *f) { struct zfsmount *mount = (struct zfsmount *)f->f_devdata; struct file *fp; int rc; if (f->f_dev != &zfs_dev) return (EINVAL); /* allocate file system specific data structure */ fp = malloc(sizeof(struct file)); bzero(fp, sizeof(struct file)); f->f_fsdata = (void *)fp; rc = zfs_lookup(mount, upath, &fp->f_dnode); fp->f_seekp = 0; if (rc) { f->f_fsdata = NULL; free(fp); } return (rc); } static int zfs_close(struct open_file *f) { struct file *fp = (struct file *)f->f_fsdata; dnode_cache_obj = 0; f->f_fsdata = (void *)0; if (fp == (struct file *)0) return (0); free(fp); return (0); } /* * Copy a portion of a file into kernel memory. * Cross block boundaries when necessary. */ static int zfs_read(struct open_file *f, void *start, size_t size, size_t *resid /* out */) { const spa_t *spa = ((struct zfsmount *)f->f_devdata)->spa; struct file *fp = (struct file *)f->f_fsdata; struct stat sb; size_t n; int rc; rc = zfs_stat(f, &sb); if (rc) return (rc); n = size; if (fp->f_seekp + n > sb.st_size) n = sb.st_size - fp->f_seekp; rc = dnode_read(spa, &fp->f_dnode, fp->f_seekp, start, n); if (rc) return (rc); if (0) { int i; for (i = 0; i < n; i++) putchar(((char*) start)[i]); } fp->f_seekp += n; if (resid) *resid = size - n; return (0); } /* * Don't be silly - the bootstrap has no business writing anything. */ static int zfs_write(struct open_file *f, void *start, size_t size, size_t *resid /* out */) { return (EROFS); } static off_t zfs_seek(struct open_file *f, off_t offset, int where) { struct file *fp = (struct file *)f->f_fsdata; switch (where) { case SEEK_SET: fp->f_seekp = offset; break; case SEEK_CUR: fp->f_seekp += offset; break; case SEEK_END: { struct stat sb; int error; error = zfs_stat(f, &sb); if (error != 0) { errno = error; return (-1); } fp->f_seekp = sb.st_size - offset; break; } default: errno = EINVAL; return (-1); } return (fp->f_seekp); } static int zfs_stat(struct open_file *f, struct stat *sb) { const spa_t *spa = ((struct zfsmount *)f->f_devdata)->spa; struct file *fp = (struct file *)f->f_fsdata; return (zfs_dnode_stat(spa, &fp->f_dnode, sb)); } static int zfs_readdir(struct open_file *f, struct dirent *d) { const spa_t *spa = ((struct zfsmount *)f->f_devdata)->spa; struct file *fp = (struct file *)f->f_fsdata; mzap_ent_phys_t mze; struct stat sb; size_t bsize = fp->f_dnode.dn_datablkszsec << SPA_MINBLOCKSHIFT; int rc; rc = zfs_stat(f, &sb); if (rc) return (rc); if (!S_ISDIR(sb.st_mode)) return (ENOTDIR); /* * If this is the first read, get the zap type. */ if (fp->f_seekp == 0) { rc = dnode_read(spa, &fp->f_dnode, 0, &fp->f_zap_type, sizeof(fp->f_zap_type)); if (rc) return (rc); if (fp->f_zap_type == ZBT_MICRO) { fp->f_seekp = offsetof(mzap_phys_t, mz_chunk); } else { rc = dnode_read(spa, &fp->f_dnode, offsetof(zap_phys_t, zap_num_leafs), &fp->f_num_leafs, sizeof(fp->f_num_leafs)); if (rc) return (rc); fp->f_seekp = bsize; fp->f_zap_leaf = (zap_leaf_phys_t *)malloc(bsize); rc = dnode_read(spa, &fp->f_dnode, fp->f_seekp, fp->f_zap_leaf, bsize); if (rc) return (rc); } } if (fp->f_zap_type == ZBT_MICRO) { mzap_next: if (fp->f_seekp >= bsize) return (ENOENT); rc = dnode_read(spa, &fp->f_dnode, fp->f_seekp, &mze, sizeof(mze)); if (rc) return (rc); fp->f_seekp += sizeof(mze); if (!mze.mze_name[0]) goto mzap_next; d->d_fileno = ZFS_DIRENT_OBJ(mze.mze_value); d->d_type = ZFS_DIRENT_TYPE(mze.mze_value); strcpy(d->d_name, mze.mze_name); d->d_namlen = strlen(d->d_name); return (0); } else { zap_leaf_t zl; zap_leaf_chunk_t *zc, *nc; int chunk; size_t namelen; char *p; uint64_t value; /* * Initialise this so we can use the ZAP size * calculating macros. */ zl.l_bs = ilog2(bsize); zl.l_phys = fp->f_zap_leaf; /* * Figure out which chunk we are currently looking at * and consider seeking to the next leaf. We use the * low bits of f_seekp as a simple chunk index. */ fzap_next: chunk = fp->f_seekp & (bsize - 1); if (chunk == ZAP_LEAF_NUMCHUNKS(&zl)) { fp->f_seekp = (fp->f_seekp & ~(bsize - 1)) + bsize; chunk = 0; /* * Check for EOF and read the new leaf. */ if (fp->f_seekp >= bsize * fp->f_num_leafs) return (ENOENT); rc = dnode_read(spa, &fp->f_dnode, fp->f_seekp, fp->f_zap_leaf, bsize); if (rc) return (rc); } zc = &ZAP_LEAF_CHUNK(&zl, chunk); fp->f_seekp++; if (zc->l_entry.le_type != ZAP_CHUNK_ENTRY) goto fzap_next; namelen = zc->l_entry.le_name_length; if (namelen > sizeof(d->d_name)) namelen = sizeof(d->d_name); /* * Paste the name back together. */ nc = &ZAP_LEAF_CHUNK(&zl, zc->l_entry.le_name_chunk); p = d->d_name; while (namelen > 0) { int len; len = namelen; if (len > ZAP_LEAF_ARRAY_BYTES) len = ZAP_LEAF_ARRAY_BYTES; memcpy(p, nc->l_array.la_array, len); p += len; namelen -= len; nc = &ZAP_LEAF_CHUNK(&zl, nc->l_array.la_next); } d->d_name[sizeof(d->d_name) - 1] = 0; /* * Assume the first eight bytes of the value are * a uint64_t. */ value = fzap_leaf_value(&zl, zc); d->d_fileno = ZFS_DIRENT_OBJ(value); d->d_type = ZFS_DIRENT_TYPE(value); d->d_namlen = strlen(d->d_name); return (0); } } static int vdev_read(vdev_t *vdev, void *priv, off_t offset, void *buf, size_t size) { int fd; fd = (uintptr_t) priv; lseek(fd, offset, SEEK_SET); if (read(fd, buf, size) == size) { return 0; } else { return (EIO); } } static int zfs_dev_init(void) { zfs_init(); if (archsw.arch_zfs_probe == NULL) return (ENXIO); archsw.arch_zfs_probe(); return (0); } int zfs_probe_dev(const char *devname, uint64_t *pool_guid) { spa_t *spa; int fd; int ret; fd = open(devname, O_RDONLY); if (fd == -1) return (ENXIO); ret = vdev_probe(vdev_read, (void *)(uintptr_t)fd, &spa); if (ret != 0) close(fd); else if (pool_guid != NULL) *pool_guid = spa->spa_guid; return (0); } /* * Print information about ZFS pools */ static void zfs_dev_print(int verbose) { spa_t *spa; char line[80]; if (verbose) { spa_all_status(); return; } STAILQ_FOREACH(spa, &zfs_pools, spa_link) { sprintf(line, " zfs:%s\n", spa->spa_name); pager_output(line); } } /* * Attempt to open the pool described by (dev) for use by (f). */ static int zfs_dev_open(struct open_file *f, ...) { va_list args; struct zfs_devdesc *dev; struct zfsmount *mount; spa_t *spa; int rv; va_start(args, f); dev = va_arg(args, struct zfs_devdesc *); va_end(args); spa = spa_find_by_guid(dev->pool_guid); if (!spa) return (ENXIO); rv = zfs_spa_init(spa); if (rv != 0) return (rv); mount = malloc(sizeof(*mount)); rv = zfs_mount(spa, dev->root_guid, mount); if (rv != 0) { free(mount); return (rv); } if (mount->objset.os_type != DMU_OST_ZFS) { printf("Unexpected object set type %ju\n", (uintmax_t)mount->objset.os_type); free(mount); return (EIO); } f->f_devdata = mount; free(dev); return (0); } static int zfs_dev_close(struct open_file *f) { free(f->f_devdata); f->f_devdata = NULL; return (0); } static int zfs_dev_strategy(void *devdata, int rw, daddr_t dblk, size_t size, char *buf, size_t *rsize) { return (ENOSYS); } struct devsw zfs_dev = { .dv_name = "zfs", .dv_type = DEVT_ZFS, .dv_init = zfs_dev_init, .dv_strategy = zfs_dev_strategy, .dv_open = zfs_dev_open, .dv_close = zfs_dev_close, .dv_ioctl = noioctl, .dv_print = zfs_dev_print, .dv_cleanup = NULL }; int zfs_parsedev(struct zfs_devdesc *dev, const char *devspec, const char **path) { static char rootname[ZFS_MAXNAMELEN]; static char poolname[ZFS_MAXNAMELEN]; spa_t *spa; const char *end; const char *np; const char *sep; int rv; np = devspec; if (*np != ':') return (EINVAL); np++; end = strchr(np, ':'); if (end == NULL) return (EINVAL); sep = strchr(np, '/'); if (sep == NULL || sep >= end) sep = end; memcpy(poolname, np, sep - np); poolname[sep - np] = '\0'; if (sep < end) { sep++; memcpy(rootname, sep, end - sep); rootname[end - sep] = '\0'; } else rootname[0] = '\0'; spa = spa_find_by_name(poolname); if (!spa) return (ENXIO); rv = zfs_spa_init(spa); if (rv != 0) return (rv); dev->pool_guid = spa->spa_guid; if (rootname[0] != '\0') { rv = zfs_lookup_dataset(spa, rootname, &dev->root_guid); if (rv != 0) return (rv); } else dev->root_guid = 0; if (path != NULL) *path = (*end == '\0') ? end : end + 1; dev->d_dev = &zfs_dev; dev->d_type = zfs_dev.dv_type; return (0); } char * zfs_fmtdev(void *vdev) { static char rootname[ZFS_MAXNAMELEN]; static char buf[2 * ZFS_MAXNAMELEN + 8]; struct zfs_devdesc *dev = (struct zfs_devdesc *)vdev; spa_t *spa; buf[0] = '\0'; if (dev->d_type != DEVT_ZFS) return (buf); spa = spa_find_by_guid(dev->pool_guid); if (spa == NULL) { printf("ZFS: can't find pool by guid\n"); return (buf); } if (zfs_spa_init(spa) != 0) { printf("ZFS: can't init pool\n"); return (buf); } if (dev->root_guid == 0 && zfs_get_root(spa, &dev->root_guid)) { printf("ZFS: can't find root filesystem\n"); return (buf); } if (zfs_rlookup(spa, dev->root_guid, rootname)) { printf("ZFS: can't find filesystem by guid\n"); return (buf); } if (rootname[0] == '\0') sprintf(buf, "%s:%s:", dev->d_dev->dv_name, spa->spa_name); else sprintf(buf, "%s:%s/%s:", dev->d_dev->dv_name, spa->spa_name, rootname); return (buf); } --------------040909000208010909070704-- From owner-freebsd-fs@FreeBSD.ORG Tue May 14 21:36:22 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2EDEEA04 for ; Tue, 14 May 2013 21:36:22 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qc0-x231.google.com (mail-qc0-x231.google.com [IPv6:2607:f8b0:400d:c01::231]) by mx1.freebsd.org (Postfix) with ESMTP id E92E9F2C for ; Tue, 14 May 2013 21:36:21 +0000 (UTC) Received: by mail-qc0-f177.google.com with SMTP id e1so60661qcy.8 for ; Tue, 14 May 2013 14:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=org/Ucf1oY6Jjbs8+xUlK4yIBTH24XVwi3zJq2J6r/o=; b=oPawMNi43S6Wru1U57r6/VAJF6sU3O/OYkfH0M2ypvqGJD1R5y19y4CJkb8+WcGar0 tLAJrCgZIh5QrKidYK9akDHMhgKRchNrZbL7Dr7n+mlr+NuglgxFEcavW3Ngetw3/frL iPuNLD7VdOhh3U7+G2PuZCOIufpENgdfdKXdgyHzoS1z/z2tdJ3F8C57qfz34CKp4dfT elXMu4KHzHoOpalSBCQZEtyIn4t76LR05f9w5oPJ+7wYQXvcO0Gl2R/27OtsslKcmSXT yK+fJ/Eme+eGq8PnKCs5f0YxzrEIj4NDYdpSaIVGcQiFis4b2X9tgIfEAp/CPEDaKstM ekEg== MIME-Version: 1.0 X-Received: by 10.224.38.133 with SMTP id b5mr10374533qae.78.1368567381136; Tue, 14 May 2013 14:36:21 -0700 (PDT) Received: by 10.49.1.44 with HTTP; Tue, 14 May 2013 14:36:21 -0700 (PDT) In-Reply-To: <519285C9.8000306@web.de> References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> Date: Tue, 14 May 2013 14:36:21 -0700 Message-ID: Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) From: Freddie Cash To: Henner Heck Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 21:36:22 -0000 On Tue, May 14, 2013 at 11:43 AM, Henner Heck wrote: > I noticed that a "zpool status" or a "zfs list" before attaching > the encrypted disks waits for about one minute before showing output. > When they finally do, the output is as expected, the raidz2 pool is shown > as > UNAVAIL and its datasets are not listed. > When all the disks are attached with geli, the outputs are given > immediately. > > On boot there are 2 delays. > One of about 1 minute after the output > "Setting hostid: 0x........" > and one of 2 minutes after > "Mounting local file systems:.". > > Both these outputs don't show up in dmesg, which ends with > "Trying to mount root from zfs:zroot/ROOT []..." shortly before. > > "dmesg -a" will show all the messages from the boot, including the ones from the RC scripts (like the ones above). -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Tue May 14 22:04:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B4454697 for ; Tue, 14 May 2013 22:04:54 +0000 (UTC) (envelope-from prvs=18460eef42=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 578F216D for ; Tue, 14 May 2013 22:04:54 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003806770.msg for ; Tue, 14 May 2013 23:04:53 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 14 May 2013 23:04:53 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18460eef42=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <53B9B2FF12A44757A8254C6DDD08829D@multiplay.co.uk> From: "Steven Hartland" To: "Henner Heck" References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> <5192AD94.5020707@web.de> Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) Date: Tue, 14 May 2013 23:04:49 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 22:04:54 -0000 ----- Original Message ----- From: "Henner Heck" To: "Steven Hartland" > Am 14.05.2013 21:22, schrieb Steven Hartland: >> ----- Original Message ----- From: "Henner Heck" >>> Hello all, >>> >>> i set up a PC with FreeBSD 9.1-Release and 2 zfs pools. >>> One is a mirror from which FreeBSD is booting >>> (tough enough without getting "error 2"), >>> the other one a raidz2 for data. >>> The disks for the raidz2 are encrypted with geli and attached manually. >>> >>> I noticed that a "zpool status" or a "zfs list" before attaching >>> the encrypted disks waits for about one minute before showing output. >>> When they finally do, the output is as expected, the raidz2 pool is >>> shown as >>> UNAVAIL and its datasets are not listed. >>> When all the disks are attached with geli, the outputs are given >>> immediately. >>> >>> On boot there are 2 delays. >>> One of about 1 minute after the output >>> "Setting hostid: 0x........" >>> and one of 2 minutes after >>> "Mounting local file systems:.". >>> Both these outputs don't show up in dmesg, which ends with >>> "Trying to mount root from zfs:zroot/ROOT []..." shortly before. >>> >>> I suspect that the boot delays too are because of the encrypted pool. >>> >>> A different machine running FreeBSD 8.3-RELEASE has a delay of only >>> about 3 seconds on "zpool status" with an encrypted pool >>> and also the boot shows no annoying anomalies. >>> >>> Any idea how to get rid of these really long delays? >> >> Try the attached patch see if that helps? Looks like 9.1 has been significantly changed from when I created that patch and isn't needed any more, so not going to be that. Should have checked, sorry for the noise. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Tue May 14 22:21:28 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 02FA7B40 for ; Tue, 14 May 2013 22:21:28 +0000 (UTC) (envelope-from prvs=18460eef42=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 9AC19251 for ; Tue, 14 May 2013 22:21:27 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003806962.msg for ; Tue, 14 May 2013 23:21:24 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 14 May 2013 23:21:24 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18460eef42=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Steven Hartland" , "Ajit Jain" , Subject: Re: seeing data corruption with zfs trim functionality Date: Tue, 14 May 2013 23:21:20 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 May 2013 22:21:28 -0000 ----- Original Message ----- From: "Steven Hartland" >>>> What version are you porting the changes to? >>>> >>>> What SSD are you using? >>>> >>>> What LSI controller are you using? >>> >>> I'd also like to see "zpool status" (for every pool that involves this >>> SSD) and "gpart show" against the disk itself. >> >> Also: >> 1. What FW version is your LSI? You can get this from dmesg. >> 2. The exact command line your running iotest with? > > Any update on this? I'd like to try and replicate your test here so > would appreciate as much information as possible. Oh and of course details on where to download your non-standard iotest including version info required :) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 00:19:32 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 23DBDEFF for ; Wed, 15 May 2013 00:19:32 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) by mx1.freebsd.org (Postfix) with ESMTP id 9E208921 for ; Wed, 15 May 2013 00:19:31 +0000 (UTC) Received: from [193.68.136.216] (digsys216-136.pip.digsys.bg [193.68.136.216]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r4F0JJPQ094360 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 15 May 2013 03:19:21 +0300 (EEST) (envelope-from daniel@digsys.bg) References: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> <5190335D.9090105@hub.org> <20130513005858.GA73875@icarus.home.lan> <94661399-66AC-4E83-B39B-0426442BB84C@hub.org> Mime-Version: 1.0 (1.0) In-Reply-To: <94661399-66AC-4E83-B39B-0426442BB84C@hub.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-Id: <12D600DE-CBAB-40C6-B166-083DE7018E7E@digsys.bg> X-Mailer: iPhone Mail (10B329) From: Daniel Kalchev Subject: Re: NFS Performance issue against NetApp Date: Wed, 15 May 2013 03:19:21 +0300 To: "Marc G. Fournier" Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 00:19:32 -0000 >=20 >> Probably off-topic but worth pointing out: I do not know about Solaris, >> but Linux has multiple layers of caching, and is well-known for doing >> things like caching (and aggregating!) reads/writes to **block** devices >> (this is why on Linux you have to make sure to avoid caching your >> application use O_DIRECT with open(2) or other mechanisms -- the BSDs do >> not do this, block devices are always non-cached). >=20 > Caching *should* only come into play after the first run of the applicatio= n =E2=80=A6 the first run after a reboot of the server shouldn't have anythi= ng in cache yet for caching to come into play=20 >=20 Or, instead of issuing 30 separate NFS calls over the network, issue just on= e. With more latency the difference will be more pronounced. I believe Jeremy was referring more to the aggregating aspect, which might p= roduce significant difference for poorly written software.=20 Daniel= From owner-freebsd-fs@FreeBSD.ORG Wed May 15 00:23:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8BB8CFEF for ; Wed, 15 May 2013 00:23:59 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:56]) by mx1.freebsd.org (Postfix) with ESMTP id 5A065960 for ; Wed, 15 May 2013 00:23:59 +0000 (UTC) Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87]) by qmta06.emeryville.ca.mail.comcast.net with comcast id byhp1l0011smiN4A60Py8v; Wed, 15 May 2013 00:23:58 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta20.emeryville.ca.mail.comcast.net with comcast id c0Px1l00K1t3BNj8g0PxwU; Wed, 15 May 2013 00:23:58 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7B63373A33; Tue, 14 May 2013 17:23:57 -0700 (PDT) Date: Tue, 14 May 2013 17:23:57 -0700 From: Jeremy Chadwick To: Daniel Kalchev Subject: Re: NFS Performance issue against NetApp Message-ID: <20130515002357.GA21753@icarus.home.lan> References: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> <5190335D.9090105@hub.org> <20130513005858.GA73875@icarus.home.lan> <94661399-66AC-4E83-B39B-0426442BB84C@hub.org> <12D600DE-CBAB-40C6-B166-083DE7018E7E@digsys.bg> MIME-Version: 1.0 Content-Type: text/plain; charset=unknown-8bit Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <12D600DE-CBAB-40C6-B166-083DE7018E7E@digsys.bg> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1368577438; bh=Pq1c3ZvKrev6bTZx37oLzldmMAP8mAGVzLcvm+g7yes=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=BDJOXXs1B1qeTAt1eWXE+CMqJEG3MBWiXQvz0ghTWfCu5ycoNocbZgy0jM3Qq54eS PxiRitGmvYlVtuQBGc4kJJXn8L+mu7Pu08LdL587GoMNRwkg/XOWsZIkrYPUTylsGw VWrHTVcD1p0LZQgHCC/L5GjFD6l9kcDNGrmvcu3HqHi44sG0U18SNu8XUd3ItgEUT2 sPBS10NahekRwlYR+36i5NcsnZ4Tw/eFVJgAxaB0vojDxh0xF/K1j72+e/khRUST1i fzpv132aGHLeAtNQXrgw+Zp1qBV4Uo4dY6CUgxbfuR4LL7M3UBZqlC4oIf0qRSbz+h AEj1x4sraxSLw== Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 00:23:59 -0000 On Wed, May 15, 2013 at 03:19:21AM +0300, Daniel Kalchev wrote: > > > > >> Probably off-topic but worth pointing out: I do not know about Solaris, > >> but Linux has multiple layers of caching, and is well-known for doing > >> things like caching (and aggregating!) reads/writes to **block** devices > >> (this is why on Linux you have to make sure to avoid caching your > >> application use O_DIRECT with open(2) or other mechanisms -- the BSDs do > >> not do this, block devices are always non-cached). > > > > Caching *should* only come into play after the first run of the application … the first run after a reboot of the server shouldn't have anything in cache yet for caching to come into play > > > > Or, instead of issuing 30 separate NFS calls over the network, issue just one. With more latency the difference will be more pronounced. > > I believe Jeremy was referring more to the aggregating aspect, which might produce significant difference for poorly written software. Thanks Daniel -- yes, correct. :-) -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed May 15 05:48:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4771D122 for ; Wed, 15 May 2013 05:48:15 +0000 (UTC) (envelope-from ajit.jain@cloudbyte.com) Received: from mail-oa0-f52.google.com (mail-oa0-f52.google.com [209.85.219.52]) by mx1.freebsd.org (Postfix) with ESMTP id 153ADF66 for ; Wed, 15 May 2013 05:48:15 +0000 (UTC) Received: by mail-oa0-f52.google.com with SMTP id h1so1632440oag.25 for ; Tue, 14 May 2013 22:48:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=RORFQExJ326HqBs6LeVZsVo+r+FlOTk5zSV/qb044LA=; b=aoZgtK6DJ7IHtiROBKWbo7S+RIeBQaP+ztJL+B71mYtc15lAZhJpBtwJEPgs5LRC6F NVPpHK6cCAhACM2+hhjWiClo4Erkax+GN0oPoz6TSOC8gGgm4GtbLN9zeNtSiD5erJaY OMd6fgz08+lRvg2Rn4+Y+4v+VvynsNerQxIV5dqIywrfFFL7kS8uy2L3S88CMV7S2yOs 7PyRacAqqgdyy0Qx7zxtB+frr/iS7KQXMplgHQ7HZdSOHwu4K+iWU6JhDaEDbpXeK5Tf lGlz05KBidr35AxEjRA7i+Qn8Ivliqu9JSDK8SlqZzae3cpqEf7DDwkN4gi+JIo6/QOS 7PCQ== X-Received: by 10.60.65.5 with SMTP id t5mr18652867oes.139.1368596894417; Tue, 14 May 2013 22:48:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.151.134 with HTTP; Tue, 14 May 2013 22:47:54 -0700 (PDT) In-Reply-To: References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> From: Ajit Jain Date: Wed, 15 May 2013 11:17:54 +0530 Message-ID: Subject: Re: seeing data corruption with zfs trim functionality To: Steven Hartland X-Gm-Message-State: ALoCoQm+L5y8fPqzzMGX/ZC+4PTJblYxSHoeMCY+1+txg6hyYLPJ2UFDRzEwMvL2mF4g1UmbW5/6 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 05:48:15 -0000 Hi Steven, Thanks for the follow-up. The code where I pulled in zfs trim patches is not updated to 9 stable specially the cam directory. I pulled in many dependent patches in order to apply the patches that you gave. After that all da devices CAM_PERIPH_INVALID in dadone() because read capability was returning a very big number (bigger than MAXPHYS) for the block size. I think this is because I have not update the code to 9 stable (only pulled in required patches and miss some patches). So, I am planning to first update my code to 9stable and then try the same test again. That might take some time. thanks again, ajit On Wed, May 15, 2013 at 2:40 AM, Steven Hartland wrote: > ----- Original Message ----- From: "Steven Hartland" > > What version are you porting the changes to? >>>> >>>> What SSD are you using? >>>> >>>> What LSI controller are you using? >>>> >>> >>> I'd also like to see "zpool status" (for every pool that involves this >>> SSD) and "gpart show" against the disk itself. >>> >> >> Also: >> 1. What FW version is your LSI? You can get this from dmesg. >> 2. The exact command line your running iotest with? >> > > Any update on this? I'd like to try and replicate your test here so > would appreciate as much information as possible. > > > Regards > Steve > > ==============================**================== > This e.mail is private and confidential between Multiplay (UK) Ltd. and > the person or entity to whom it is addressed. In the event of misdirection, > the recipient is prohibited from using, copying, printing or otherwise > disseminating it or any information contained in it. > In the event of misdirection, illegible or incomplete transmission please > telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 07:26:14 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7756140B for ; Wed, 15 May 2013 07:26:14 +0000 (UTC) (envelope-from prvs=1847987112=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 07772371 for ; Wed, 15 May 2013 07:26:13 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003814671.msg for ; Wed, 15 May 2013 08:26:11 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 15 May 2013 08:26:11 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1847987112=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> From: "Steven Hartland" To: "Ajit Jain" References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> Subject: Re: seeing data corruption with zfs trim functionality Date: Wed, 15 May 2013 08:26:09 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 07:26:14 -0000 Could you provide us with details on the tests your using so we can run them here on current sources and see if we see any issues? Regards Steve ----- Original Message ----- From: "Ajit Jain" To: "Steven Hartland" Cc: "freebsd-fs" Sent: Wednesday, May 15, 2013 6:47 AM Subject: Re: seeing data corruption with zfs trim functionality > Hi Steven, > > Thanks for the follow-up. > The code where I pulled in zfs trim patches is not updated to 9 stable > specially the cam directory. > I pulled in many dependent patches in order to apply the patches that you > gave. After that all da devices > CAM_PERIPH_INVALID in dadone() because read capability was returning a very > big number (bigger than MAXPHYS) > for the block size. I think this is because I have not update the code to 9 > stable (only pulled in required patches and miss > some patches). > > So, I am planning to first update my code to 9stable and then try the same > test again. That might take some time. > > > thanks again, > ajit > > > On Wed, May 15, 2013 at 2:40 AM, Steven Hartland wrote: > >> ----- Original Message ----- From: "Steven Hartland" >> >> What version are you porting the changes to? >>>>> >>>>> What SSD are you using? >>>>> >>>>> What LSI controller are you using? >>>>> >>>> >>>> I'd also like to see "zpool status" (for every pool that involves this >>>> SSD) and "gpart show" against the disk itself. >>>> >>> >>> Also: >>> 1. What FW version is your LSI? You can get this from dmesg. >>> 2. The exact command line your running iotest with? >>> >> >> Any update on this? I'd like to try and replicate your test here so >> would appreciate as much information as possible. >> >> >> Regards >> Steve >> >> ==============================**================== >> This e.mail is private and confidential between Multiplay (UK) Ltd. and >> the person or entity to whom it is addressed. In the event of misdirection, >> the recipient is prohibited from using, copying, printing or otherwise >> disseminating it or any information contained in it. >> In the event of misdirection, illegible or incomplete transmission please >> telephone +44 845 868 1337 >> or return the E.mail to postmaster@multiplay.co.uk. >> >> > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 08:20:16 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BBD64303 for ; Wed, 15 May 2013 08:20:16 +0000 (UTC) (envelope-from ajit.jain@cloudbyte.com) Received: from mail-oa0-f41.google.com (mail-oa0-f41.google.com [209.85.219.41]) by mx1.freebsd.org (Postfix) with ESMTP id 6C6A17F6 for ; Wed, 15 May 2013 08:20:16 +0000 (UTC) Received: by mail-oa0-f41.google.com with SMTP id n9so1836915oag.0 for ; Wed, 15 May 2013 01:20:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=X1wFbexW+oYjryaY35Q8hu+EpoHrqSUBMJ40jC5YPK0=; b=pcRKNVUbXL8YWCIjP4hxsAN2NzpjfAwp5FTGsWyP1mTfWxkVYh0+IX4a0+l+XLsTNY SUG9fK56OmoSsDfsn9gMVlj0vK0ecapgecqPqyoKXJl7w13rvRGTVgEIaNTAtMzZQWTX 6ViGPlNsHavfUcbWHgvni7J2zV0Q58v8xm5Aczm/yaK5Z1ljzv+Q8kiO7PYc87pyUM/3 XmS4l+k7rR+DyQGibrjouuZFoS2uswnmmubvz/e9vcyYFZrnYWBZP7G6DpnzStwrbMiX ypIZHySjXyk3Yp2RRUtfqqRSNhjVav7tO64xxtoipOwwtQXFRZWI/r+DEF7opRCc9Lrb 8aqQ== X-Received: by 10.60.65.5 with SMTP id t5mr18912673oes.139.1368606010204; Wed, 15 May 2013 01:20:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.151.134 with HTTP; Wed, 15 May 2013 01:19:49 -0700 (PDT) In-Reply-To: <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> From: Ajit Jain Date: Wed, 15 May 2013 13:49:49 +0530 Message-ID: Subject: Re: seeing data corruption with zfs trim functionality To: Steven Hartland Content-Type: multipart/mixed; boundary=001a11c1d6a813773304dcbd6b00 X-Gm-Message-State: ALoCoQkmX9rfPrnmn4LNKkfiyiffTJ5TEUnFsml1TYHazh2JPdjc3pSu28tzhs3SSsZw+RqXunPO X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 08:20:16 -0000 --001a11c1d6a813773304dcbd6b00 Content-Type: text/plain; charset=ISO-8859-1 Hi Steven, Please find the tar ball of src code and binary of the test utility attached with the mail. Steps of test: 1. Enable zfs trim in /boot/loader.conf (vfs.zfs.trim_disable=0) 2. Set the delete method of the ssd device as UNMAP or WS16. 3. Create a pool (and optionally dataset) on the device. 4. Run iotest utility with thread count 10 (-t option) file size as at least 5GB Second to execute as at least 500 sec (-T option) and write as 100% (W option). regards, ajit On Wed, May 15, 2013 at 12:56 PM, Steven Hartland wrote: > Could you provide us with details on the tests your using so we can > run them here on current sources and see if we see any issues? > > Regards > Steve > > ----- Original Message ----- From: "Ajit Jain" > To: "Steven Hartland" > Cc: "freebsd-fs" > Sent: Wednesday, May 15, 2013 6:47 AM > > Subject: Re: seeing data corruption with zfs trim functionality > > > Hi Steven, >> >> Thanks for the follow-up. >> The code where I pulled in zfs trim patches is not updated to 9 stable >> specially the cam directory. >> I pulled in many dependent patches in order to apply the patches that you >> gave. After that all da devices >> CAM_PERIPH_INVALID in dadone() because read capability was returning a >> very >> big number (bigger than MAXPHYS) >> for the block size. I think this is because I have not update the code to >> 9 >> stable (only pulled in required patches and miss >> some patches). >> >> So, I am planning to first update my code to 9stable and then try the same >> test again. That might take some time. >> >> >> thanks again, >> ajit >> >> >> On Wed, May 15, 2013 at 2:40 AM, Steven Hartland > >**wrote: >> >> ----- Original Message ----- From: "Steven Hartland" >>> >>> What version are you porting the changes to? >>> >>>> >>>>>> What SSD are you using? >>>>>> >>>>>> What LSI controller are you using? >>>>>> >>>>>> >>>>> I'd also like to see "zpool status" (for every pool that involves this >>>>> SSD) and "gpart show" against the disk itself. >>>>> >>>>> >>>> Also: >>>> 1. What FW version is your LSI? You can get this from dmesg. >>>> 2. The exact command line your running iotest with? >>>> >>>> >>> Any update on this? I'd like to try and replicate your test here so >>> would appreciate as much information as possible. >>> >>> >>> Regards >>> Steve >>> >>> ==============================****================== >>> >>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>> the person or entity to whom it is addressed. In the event of >>> misdirection, >>> the recipient is prohibited from using, copying, printing or otherwise >>> disseminating it or any information contained in it. >>> In the event of misdirection, illegible or incomplete transmission please >>> telephone +44 845 868 1337 >>> or return the E.mail to postmaster@multiplay.co.uk. >>> >>> >>> >> > ==============================**================== > This e.mail is private and confidential between Multiplay (UK) Ltd. and > the person or entity to whom it is addressed. In the event of misdirection, > the recipient is prohibited from using, copying, printing or otherwise > disseminating it or any information contained in it. > In the event of misdirection, illegible or incomplete transmission please > telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > --001a11c1d6a813773304dcbd6b00-- From owner-freebsd-fs@FreeBSD.ORG Wed May 15 08:30:21 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F17637D2 for ; Wed, 15 May 2013 08:30:21 +0000 (UTC) (envelope-from ajit.jain@cloudbyte.com) Received: from mail-oa0-f44.google.com (mail-oa0-f44.google.com [209.85.219.44]) by mx1.freebsd.org (Postfix) with ESMTP id BC675871 for ; Wed, 15 May 2013 08:30:21 +0000 (UTC) Received: by mail-oa0-f44.google.com with SMTP id n12so1830074oag.17 for ; Wed, 15 May 2013 01:30:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=RVhCBcsexK0FbFVgNsmt6RX8k4+fO6taGt2vr3XBXIg=; b=BPG0utleCBqD/K0fxNqIfUXB9vD52CnTjPSmuA9r4ikfGd1/Qu/lP7nBcZjzI7buh8 44tNjiXkHas3kCnOk+Q8FZfZXbZ9r6djD7Sxw8jZsL6NOYMuHrpTmEiVYAWZuVv7r4It gtyAnE0/Vql4V+RfIpWSob1oGY+/GPa3otaUnFp4fBCg31vqMCPusOYZPxmBvPDrftc5 a2yKifrXQu4vxsXUyPT6C1sQ/z7UQZ47sktiUQooVAw/Jp+8J6QLKr+8Ntj06yMNCM/2 Xv3JY2jqJDmf66g2o3Bk7SdwTvtqbHNrKuek86+yqx7QMpCa2hMEiaB0z8pn5y8vRlmj 71Bw== X-Received: by 10.182.231.197 with SMTP id ti5mr16976491obc.64.1368606620979; Wed, 15 May 2013 01:30:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.151.134 with HTTP; Wed, 15 May 2013 01:29:55 -0700 (PDT) In-Reply-To: References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> From: Ajit Jain Date: Wed, 15 May 2013 13:59:55 +0530 Message-ID: Subject: Re: seeing data corruption with zfs trim functionality To: Steven Hartland X-Gm-Message-State: ALoCoQn75pom1Og6vDGUuDCQg5Cl061kpj15FdqUfOnRy9ThTjBljbrAVNWr6t1KRq2tO5RsPjPl Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 08:30:22 -0000 Hi Steve, One more thing I am not seeing data corruption with SATA SSD (kingston). The issue was seen on SAS SSD i.e. Seagate PULSAR ST100FM0002 . regards, ajit On Wed, May 15, 2013 at 1:49 PM, Ajit Jain wrote: > Hi Steven, > > Please find the tar ball of src code and binary of the test utility > attached with the mail. > Steps of test: > 1. Enable zfs trim in /boot/loader.conf (vfs.zfs.trim_disable=0) > 2. Set the delete method of the ssd device as UNMAP or WS16. > 3. Create a pool (and optionally dataset) on the device. > 4. Run iotest utility with thread count 10 (-t option) file size as at > least 5GB > Second to execute as at least 500 sec (-T option) and write as 100% (W > option). > > regards, > ajit > > > On Wed, May 15, 2013 at 12:56 PM, Steven Hartland > wrote: > >> Could you provide us with details on the tests your using so we can >> run them here on current sources and see if we see any issues? >> >> Regards >> Steve >> >> ----- Original Message ----- From: "Ajit Jain" >> To: "Steven Hartland" >> Cc: "freebsd-fs" >> Sent: Wednesday, May 15, 2013 6:47 AM >> >> Subject: Re: seeing data corruption with zfs trim functionality >> >> >> Hi Steven, >>> >>> Thanks for the follow-up. >>> The code where I pulled in zfs trim patches is not updated to 9 stable >>> specially the cam directory. >>> I pulled in many dependent patches in order to apply the patches that you >>> gave. After that all da devices >>> CAM_PERIPH_INVALID in dadone() because read capability was returning a >>> very >>> big number (bigger than MAXPHYS) >>> for the block size. I think this is because I have not update the code >>> to 9 >>> stable (only pulled in required patches and miss >>> some patches). >>> >>> So, I am planning to first update my code to 9stable and then try the >>> same >>> test again. That might take some time. >>> >>> >>> thanks again, >>> ajit >>> >>> >>> On Wed, May 15, 2013 at 2:40 AM, Steven Hartland < >>> killing@multiplay.co.uk>**wrote: >>> >>> ----- Original Message ----- From: "Steven Hartland" >>>> >>>> What version are you porting the changes to? >>>> >>>>> >>>>>>> What SSD are you using? >>>>>>> >>>>>>> What LSI controller are you using? >>>>>>> >>>>>>> >>>>>> I'd also like to see "zpool status" (for every pool that involves this >>>>>> SSD) and "gpart show" against the disk itself. >>>>>> >>>>>> >>>>> Also: >>>>> 1. What FW version is your LSI? You can get this from dmesg. >>>>> 2. The exact command line your running iotest with? >>>>> >>>>> >>>> Any update on this? I'd like to try and replicate your test here so >>>> would appreciate as much information as possible. >>>> >>>> >>>> Regards >>>> Steve >>>> >>>> ==============================****================== >>>> >>>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>>> the person or entity to whom it is addressed. In the event of >>>> misdirection, >>>> the recipient is prohibited from using, copying, printing or otherwise >>>> disseminating it or any information contained in it. >>>> In the event of misdirection, illegible or incomplete transmission >>>> please >>>> telephone +44 845 868 1337 >>>> or return the E.mail to postmaster@multiplay.co.uk. >>>> >>>> >>>> >>> >> ==============================**================== >> This e.mail is private and confidential between Multiplay (UK) Ltd. and >> the person or entity to whom it is addressed. In the event of misdirection, >> the recipient is prohibited from using, copying, printing or otherwise >> disseminating it or any information contained in it. >> In the event of misdirection, illegible or incomplete transmission please >> telephone +44 845 868 1337 >> or return the E.mail to postmaster@multiplay.co.uk. >> >> > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 08:42:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E8962FE9 for ; Wed, 15 May 2013 08:42:43 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id 7FC6991A for ; Wed, 15 May 2013 08:42:43 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UcXIA-00078Q-DU; Wed, 15 May 2013 10:42:35 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UcXI8-000832-US; Wed, 15 May 2013 10:42:32 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Steven Hartland" , "Henner Heck" Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> <5192AD94.5020707@web.de> Date: Wed, 15 May 2013 10:42:32 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: <5192AD94.5020707@web.de> User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.0 X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled version=3.3.1 X-Scan-Signature: 6afe0509a8fa736d7bae96a824aab2db Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 08:42:44 -0000 On Tue, 14 May 2013 23:33:08 +0200, Henner Heck wrote: > Am 14.05.2013 21:22, schrieb Steven Hartland: >> ----- Original Message ----- From: "Henner Heck" >>> Hello all, >>> >>> i set up a PC with FreeBSD 9.1-Release and 2 zfs pools. >>> One is a mirror from which FreeBSD is booting >>> (tough enough without getting "error 2"), >>> the other one a raidz2 for data. >>> The disks for the raidz2 are encrypted with geli and attached manually. >>> >>> I noticed that a "zpool status" or a "zfs list" before attaching >>> the encrypted disks waits for about one minute before showing output. >>> When they finally do, the output is as expected, the raidz2 pool is >>> shown as >>> UNAVAIL and its datasets are not listed. >>> When all the disks are attached with geli, the outputs are given >>> immediately. >>> >>> On boot there are 2 delays. >>> One of about 1 minute after the output >>> "Setting hostid: 0x........" >>> and one of 2 minutes after >>> "Mounting local file systems:.". >>> Both these outputs don't show up in dmesg, which ends with >>> "Trying to mount root from zfs:zroot/ROOT []..." shortly before. >>> >>> I suspect that the boot delays too are because of the encrypted pool. >>> >>> A different machine running FreeBSD 8.3-RELEASE has a delay of only >>> about 3 seconds on "zpool status" with an encrypted pool >>> and also the boot shows no annoying anomalies. >>> >>> Any idea how to get rid of these really long delays? >> >> Try the attached patch see if that helps? >> >> Regards >> Steve >> >> ================================================ >> This e.mail is private and confidential between Multiplay (UK) Ltd. >> and the person or entity to whom it is addressed. In the event of >> misdirection, the recipient is prohibited from using, copying, >> printing or otherwise disseminating it or any information contained in >> it. >> In the event of misdirection, illegible or incomplete transmission >> please telephone +44 845 868 1337 >> or return the E.mail to postmaster@multiplay.co.uk. > > > > Hello Steven, > > i tried to apply the patch, but patch didn't return and gave no output. > I checked my the zfs.c and it doesn't seem to fit this patch. > Please see attached file. > > Regards, > Henner Heck Sounds likes my daily wtf moment. Patch likes the patch-file on stdin. That is why it waits forever and does not return. Ctrl-z will help that. Use something like this: patch < foo.diff Regards, Ronald. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 12:13:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8348DA15 for ; Wed, 15 May 2013 12:13:40 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oa0-f53.google.com (mail-oa0-f53.google.com [209.85.219.53]) by mx1.freebsd.org (Postfix) with ESMTP id 563E1841 for ; Wed, 15 May 2013 12:13:39 +0000 (UTC) Received: by mail-oa0-f53.google.com with SMTP id g12so2011406oah.40 for ; Wed, 15 May 2013 05:13:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=0u4TRuszK2tY4FcRe+vhbWVPUToLjuDDA2whP92bLoA=; b=ZhBte8IIQBnd69BpKW9YOx9AA+TkKMJb/pxFO2h0rEDVl3hhyNs04hc3nhFBDFkh/U 0bTjyx7C788Vgct3YeSZZZ7HKz15U2w9fC4Qhx6QLyEDETk7pA8bP+AFOZ1Wpvr+/AmX N23gOh1pYrUvv1ZkuTy8szuC1rofw7NoQ8mfGPI3sdVgCH43rHdFtI5gz3hiy3Ank5vD fUSnIKrwmSKJPp/vjGBO17+rEBcfBI1nANvLqgfgiAGK5U5NSbiaabxeHPMgsm4aFJDd a579HcJeYzSp5IHiqKVJs2G100Sfn4O3QTYE8DTW/7p1mXlLvaXeM+VXcYL+x4+X3iWA 2vKQ== MIME-Version: 1.0 X-Received: by 10.60.56.132 with SMTP id a4mr10183141oeq.113.1368620012965; Wed, 15 May 2013 05:13:32 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 05:13:32 -0700 (PDT) Date: Wed, 15 May 2013 08:13:32 -0400 Message-ID: Subject: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 12:13:40 -0000 So it seems a new deployment we just built with zfsonroot mirror and a 48TB master pool is already out of File Descriptors??? pool: master state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: none requested config: NAME STATE READ WRITE CKSUM master ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 multipath/SATA_LUN01 ONLINE 0 0 0 multipath/SATA_LUN03 ONLINE 0 0 0 multipath/SATA_LUN04 ONLINE 0 0 0 multipath/SATA_LUN05 ONLINE 0 0 0 multipath/SATA_LUN07 ONLINE 0 0 0 multipath/SATA_LUN08 ONLINE 0 0 0 multipath/SATA_LUN09 ONLINE 0 0 0 multipath/SATA_LUN10 ONLINE 0 0 0 raidz3-1 ONLINE 0 0 0 multipath/SATA_LUN11 ONLINE 0 0 0 multipath/SATA_LUN12 ONLINE 0 0 0 multipath/SATA_LUN13 ONLINE 0 0 0 multipath/SATA_LUN14 ONLINE 0 0 0 multipath/SATA_LUN15 ONLINE 0 0 0 multipath/SATA_LUN16 ONLINE 0 0 0 multipath/SATA_LUN17 ONLINE 0 0 0 multipath/SATA_LUN18 ONLINE 0 0 0 raidz3-2 ONLINE 0 0 0 multipath/SATA_LUN19 ONLINE 0 0 0 multipath/SATA_LUN20 ONLINE 0 0 0 multipath/SATA_LUN21 ONLINE 0 0 0 multipath/SATA_LUN22 ONLINE 0 0 0 multipath/SATA_LUN23 ONLINE 0 0 0 multipath/SATA_LUN24 ONLINE 0 0 0 multipath/SATA_LUN26 ONLINE 0 0 0 multipath/SATA_LUN27 ONLINE 0 0 0 raidz3-3 ONLINE 0 0 0 multipath/SATA_LUN29 ONLINE 0 0 0 multipath/SATA_LUN30 ONLINE 0 0 0 multipath/SATA_LUN32 ONLINE 0 0 0 multipath/SATA_LUN33 ONLINE 0 0 0 multipath/SATA_LUN35 ONLINE 0 0 0 multipath/SATA_LUN36 ONLINE 0 0 0 multipath/SATA_LUN37 ONLINE 0 0 0 multipath/SATA_LUN38 ONLINE 0 0 0 logs multipath/SATA_LUN06 ONLINE 0 0 0 cache multipath/SATA_LUN02 ONLINE 0 0 0 errors: No known data errors pool: tank state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da34p3 ONLINE 0 0 0 da35p3 ONLINE 0 0 0 errors: No known data errors while compiling a kernel, buildworld worked and installed ok lex -t /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm_scan.l > aicasm_scan.c lex -t -Pmm /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm_macro_scan.l > aicasm_macro_scan.c rm -f .depend_aicasm mkdep -f .depend_aicasm -a -I. -I/usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm -std=gnu99 /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm.c /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm_symbol.c aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c Out of file descriptors *** [.depend_aicasm] Error code 2 Stop in /usr/src/sys/modules/aic7xxx/aicasm. *** [buildkernel] Error code 1 Stop in /usr/src. *** [buildkernel] Error code 1 From owner-freebsd-fs@FreeBSD.ORG Wed May 15 12:22:47 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 72E47B84 for ; Wed, 15 May 2013 12:22:47 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id 0EE918AE for ; Wed, 15 May 2013 12:22:46 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UcajE-0002sE-6f; Wed, 15 May 2013 14:22:44 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UcajC-0006oF-Oz; Wed, 15 May 2013 14:22:42 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org, "Outback Dingo" Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? References: Date: Wed, 15 May 2013 14:22:42 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: 27b922e39ba943ef41a3b4c3f0284049 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 12:22:47 -0000 On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo wrote: > So it seems a new deployment we just built with zfsonroot mirror and a > 48TB > master pool is already out of File Descriptors??? > > pool: master > state: ONLINE > status: The pool is formatted using a legacy on-disk format. The pool > can > still be used, but some features are unavailable. > action: Upgrade the pool using 'zpool upgrade'. Once this is done, the > pool will no longer be accessible on software that does not > support > feature > flags. > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > master ONLINE 0 0 0 > raidz3-0 ONLINE 0 0 0 > multipath/SATA_LUN01 ONLINE 0 0 0 > multipath/SATA_LUN03 ONLINE 0 0 0 > multipath/SATA_LUN04 ONLINE 0 0 0 > multipath/SATA_LUN05 ONLINE 0 0 0 > multipath/SATA_LUN07 ONLINE 0 0 0 > multipath/SATA_LUN08 ONLINE 0 0 0 > multipath/SATA_LUN09 ONLINE 0 0 0 > multipath/SATA_LUN10 ONLINE 0 0 0 > raidz3-1 ONLINE 0 0 0 > multipath/SATA_LUN11 ONLINE 0 0 0 > multipath/SATA_LUN12 ONLINE 0 0 0 > multipath/SATA_LUN13 ONLINE 0 0 0 > multipath/SATA_LUN14 ONLINE 0 0 0 > multipath/SATA_LUN15 ONLINE 0 0 0 > multipath/SATA_LUN16 ONLINE 0 0 0 > multipath/SATA_LUN17 ONLINE 0 0 0 > multipath/SATA_LUN18 ONLINE 0 0 0 > raidz3-2 ONLINE 0 0 0 > multipath/SATA_LUN19 ONLINE 0 0 0 > multipath/SATA_LUN20 ONLINE 0 0 0 > multipath/SATA_LUN21 ONLINE 0 0 0 > multipath/SATA_LUN22 ONLINE 0 0 0 > multipath/SATA_LUN23 ONLINE 0 0 0 > multipath/SATA_LUN24 ONLINE 0 0 0 > multipath/SATA_LUN26 ONLINE 0 0 0 > multipath/SATA_LUN27 ONLINE 0 0 0 > raidz3-3 ONLINE 0 0 0 > multipath/SATA_LUN29 ONLINE 0 0 0 > multipath/SATA_LUN30 ONLINE 0 0 0 > multipath/SATA_LUN32 ONLINE 0 0 0 > multipath/SATA_LUN33 ONLINE 0 0 0 > multipath/SATA_LUN35 ONLINE 0 0 0 > multipath/SATA_LUN36 ONLINE 0 0 0 > multipath/SATA_LUN37 ONLINE 0 0 0 > multipath/SATA_LUN38 ONLINE 0 0 0 > logs > multipath/SATA_LUN06 ONLINE 0 0 0 > cache > multipath/SATA_LUN02 ONLINE 0 0 0 > > errors: No known data errors > pool: tank > state: ONLINE > status: The pool is formatted using a legacy on-disk format. The pool > can > still be used, but some features are unavailable. > action: Upgrade the pool using 'zpool upgrade'. Once this is done, the > pool will no longer be accessible on software that does not > support > feature > flags. > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > da34p3 ONLINE 0 0 0 > da35p3 ONLINE 0 0 0 > > errors: No known data errors > > while compiling a kernel, buildworld worked and installed ok > > lex -t > /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm_scan.l >> aicasm_scan.c > lex -t -Pmm > /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm_macro_scan.l >> aicasm_macro_scan.c > rm -f .depend_aicasm > mkdep -f .depend_aicasm -a -I. > -I/usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm > -std=gnu99 > /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm.c > /usr/src/sys/modules/aic7xxx/aicasm/../../../dev/aic7xxx/aicasm/aicasm_symbol.c > aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c > Out of file descriptors > *** [.depend_aicasm] Error code 2 > > Stop in /usr/src/sys/modules/aic7xxx/aicasm. > *** [buildkernel] Error code 1 > > Stop in /usr/src. > *** [buildkernel] Error code 1 Do you know what a file descriptor is? There is something about it here: http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html (search for 'file descriptor') Or more info in here: https://www.google.nl/search?q=freebsd+Out+of+file+descriptors It mainly says that you have more files open than your system is configured to allow. This is often produced by a bug in a program which does not close some files properly. Ronald. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 12:27:17 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D8AABCFF for ; Wed, 15 May 2013 12:27:17 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oa0-f45.google.com (mail-oa0-f45.google.com [209.85.219.45]) by mx1.freebsd.org (Postfix) with ESMTP id A70C28F7 for ; Wed, 15 May 2013 12:27:17 +0000 (UTC) Received: by mail-oa0-f45.google.com with SMTP id j6so2037003oag.4 for ; Wed, 15 May 2013 05:27:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=BFOzkAPJqDtVlyJCnIn14oD3V1Ud2SOQ26mfMVT77K8=; b=QRDJU+Tpm2uGt0OilGbNa9ZiFsJ1qDVmhlKUwhxDGSsMFt81C5wZEOlv0Jp2ijvI0F GWcfqIvuOR7syKa+ScdVzNyTQjdD5UjHjnJ7Xb+CTjG7b+LDTHZW9PTQs9xwNmhk8HXQ xrokpsfxtPReMHB7w0aXAFngxuwpos7Kd3mBBiUpJJz25hc0T1GvF0uXF3G537hNyVkO EWO2QAzsN0SHUmF9N+ACVNuAlznuvUvtzTAzeXNcFWVQcAMWosxP3H+OEw/Sx2dRPyQS +YedNTd7YQfJ+VeBz1MFzjd7DXCoWbWqooY+Um6Qeqnyce66/GryRoQ/zjoWk2jobIxS 1OAA== MIME-Version: 1.0 X-Received: by 10.60.41.232 with SMTP id i8mr18407437oel.129.1368620831041; Wed, 15 May 2013 05:27:11 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 05:27:10 -0700 (PDT) In-Reply-To: References: Date: Wed, 15 May 2013 08:27:10 -0400 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: Ronald Klop Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 12:27:17 -0000 On Wed, May 15, 2013 at 8:22 AM, Ronald Klop wrote: > On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo > wrote: > > So it seems a new deployment we just built with zfsonroot mirror and a >> 48TB >> master pool is already out of File Descriptors??? >> >> pool: master >> state: ONLINE >> status: The pool is formatted using a legacy on-disk format. The pool can >> still be used, but some features are unavailable. >> action: Upgrade the pool using 'zpool upgrade'. Once this is done, the >> pool will no longer be accessible on software that does not >> support >> feature >> flags. >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> master ONLINE 0 0 0 >> raidz3-0 ONLINE 0 0 0 >> multipath/SATA_LUN01 ONLINE 0 0 0 >> multipath/SATA_LUN03 ONLINE 0 0 0 >> multipath/SATA_LUN04 ONLINE 0 0 0 >> multipath/SATA_LUN05 ONLINE 0 0 0 >> multipath/SATA_LUN07 ONLINE 0 0 0 >> multipath/SATA_LUN08 ONLINE 0 0 0 >> multipath/SATA_LUN09 ONLINE 0 0 0 >> multipath/SATA_LUN10 ONLINE 0 0 0 >> raidz3-1 ONLINE 0 0 0 >> multipath/SATA_LUN11 ONLINE 0 0 0 >> multipath/SATA_LUN12 ONLINE 0 0 0 >> multipath/SATA_LUN13 ONLINE 0 0 0 >> multipath/SATA_LUN14 ONLINE 0 0 0 >> multipath/SATA_LUN15 ONLINE 0 0 0 >> multipath/SATA_LUN16 ONLINE 0 0 0 >> multipath/SATA_LUN17 ONLINE 0 0 0 >> multipath/SATA_LUN18 ONLINE 0 0 0 >> raidz3-2 ONLINE 0 0 0 >> multipath/SATA_LUN19 ONLINE 0 0 0 >> multipath/SATA_LUN20 ONLINE 0 0 0 >> multipath/SATA_LUN21 ONLINE 0 0 0 >> multipath/SATA_LUN22 ONLINE 0 0 0 >> multipath/SATA_LUN23 ONLINE 0 0 0 >> multipath/SATA_LUN24 ONLINE 0 0 0 >> multipath/SATA_LUN26 ONLINE 0 0 0 >> multipath/SATA_LUN27 ONLINE 0 0 0 >> raidz3-3 ONLINE 0 0 0 >> multipath/SATA_LUN29 ONLINE 0 0 0 >> multipath/SATA_LUN30 ONLINE 0 0 0 >> multipath/SATA_LUN32 ONLINE 0 0 0 >> multipath/SATA_LUN33 ONLINE 0 0 0 >> multipath/SATA_LUN35 ONLINE 0 0 0 >> multipath/SATA_LUN36 ONLINE 0 0 0 >> multipath/SATA_LUN37 ONLINE 0 0 0 >> multipath/SATA_LUN38 ONLINE 0 0 0 >> logs >> multipath/SATA_LUN06 ONLINE 0 0 0 >> cache >> multipath/SATA_LUN02 ONLINE 0 0 0 >> >> errors: No known data errors >> pool: tank >> state: ONLINE >> status: The pool is formatted using a legacy on-disk format. The pool can >> still be used, but some features are unavailable. >> action: Upgrade the pool using 'zpool upgrade'. Once this is done, the >> pool will no longer be accessible on software that does not >> support >> feature >> flags. >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> tank ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> da34p3 ONLINE 0 0 0 >> da35p3 ONLINE 0 0 0 >> >> errors: No known data errors >> >> while compiling a kernel, buildworld worked and installed ok >> >> lex -t >> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >> aicasm/aicasm_scan.l >> >>> aicasm_scan.c >>> >> lex -t -Pmm >> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >> aicasm/aicasm_macro_scan.l >> >>> aicasm_macro_scan.c >>> >> rm -f .depend_aicasm >> mkdep -f .depend_aicasm -a -I. >> -I/usr/src/sys/modules/**aic7xxx/aicasm/../../../dev/**aic7xxx/aicasm >> -std=gnu99 >> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >> aicasm/aicasm.c >> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >> aicasm/aicasm_symbol.c >> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c >> Out of file descriptors >> *** [.depend_aicasm] Error code 2 >> >> Stop in /usr/src/sys/modules/aic7xxx/**aicasm. >> *** [buildkernel] Error code 1 >> >> Stop in /usr/src. >> *** [buildkernel] Error code 1 >> > > > Do you know what a file descriptor is? > > There is something about it here: http://www.freebsd.org/doc/en/** > books/handbook/configtuning-**kernel-limits.html(search for 'file descriptor') > Or more info in here: https://www.google.nl/search?** > q=freebsd+Out+of+file+**descriptors > > It mainly says that you have more files open than your system is > configured to allow. This is often produced by a bug in a program which > does not close some files properly. > > Yes I do.... but this is a brand new zfsonroot with barely any data on it and sysctl -a | grep kern.openfiles kern.openfiles: 68 root@:/master/builder # sysctl -a | grep kern.maxfiles kern.maxfiles: 24600 kern.maxfilesperproc: 11095 the openfiles and maxfiles seems to be plenty, but im out of descriptors, i used to see this back in the 4.x days when you could format ufs with larger inodes, but zfs ??? really? > Ronald. > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 12:34:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5EDC0FEE for ; Wed, 15 May 2013 12:34:23 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id D4E0995B for ; Wed, 15 May 2013 12:34:22 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UcauS-0004Uh-7Y; Wed, 15 May 2013 14:34:20 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UcauQ-0007dv-QA; Wed, 15 May 2013 14:34:18 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Outback Dingo" Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? References: Date: Wed, 15 May 2013 14:34:18 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: 00ca572c58d2a72aaa1283be2358df10 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 12:34:23 -0000 On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo wrote: > On Wed, May 15, 2013 at 8:22 AM, Ronald Klop > wrote: > >> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo >> >> wrote: >> >> So it seems a new deployment we just built with zfsonroot mirror and a >>> 48TB >>> master pool is already out of File Descriptors??? >>> >>> pool: master >>> state: ONLINE >>> status: The pool is formatted using a legacy on-disk format. The pool >>> can >>> still be used, but some features are unavailable. >>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, the >>> pool will no longer be accessible on software that does not >>> support >>> feature >>> flags. >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> master ONLINE 0 0 0 >>> raidz3-0 ONLINE 0 0 0 >>> multipath/SATA_LUN01 ONLINE 0 0 0 >>> multipath/SATA_LUN03 ONLINE 0 0 0 >>> multipath/SATA_LUN04 ONLINE 0 0 0 >>> multipath/SATA_LUN05 ONLINE 0 0 0 >>> multipath/SATA_LUN07 ONLINE 0 0 0 >>> multipath/SATA_LUN08 ONLINE 0 0 0 >>> multipath/SATA_LUN09 ONLINE 0 0 0 >>> multipath/SATA_LUN10 ONLINE 0 0 0 >>> raidz3-1 ONLINE 0 0 0 >>> multipath/SATA_LUN11 ONLINE 0 0 0 >>> multipath/SATA_LUN12 ONLINE 0 0 0 >>> multipath/SATA_LUN13 ONLINE 0 0 0 >>> multipath/SATA_LUN14 ONLINE 0 0 0 >>> multipath/SATA_LUN15 ONLINE 0 0 0 >>> multipath/SATA_LUN16 ONLINE 0 0 0 >>> multipath/SATA_LUN17 ONLINE 0 0 0 >>> multipath/SATA_LUN18 ONLINE 0 0 0 >>> raidz3-2 ONLINE 0 0 0 >>> multipath/SATA_LUN19 ONLINE 0 0 0 >>> multipath/SATA_LUN20 ONLINE 0 0 0 >>> multipath/SATA_LUN21 ONLINE 0 0 0 >>> multipath/SATA_LUN22 ONLINE 0 0 0 >>> multipath/SATA_LUN23 ONLINE 0 0 0 >>> multipath/SATA_LUN24 ONLINE 0 0 0 >>> multipath/SATA_LUN26 ONLINE 0 0 0 >>> multipath/SATA_LUN27 ONLINE 0 0 0 >>> raidz3-3 ONLINE 0 0 0 >>> multipath/SATA_LUN29 ONLINE 0 0 0 >>> multipath/SATA_LUN30 ONLINE 0 0 0 >>> multipath/SATA_LUN32 ONLINE 0 0 0 >>> multipath/SATA_LUN33 ONLINE 0 0 0 >>> multipath/SATA_LUN35 ONLINE 0 0 0 >>> multipath/SATA_LUN36 ONLINE 0 0 0 >>> multipath/SATA_LUN37 ONLINE 0 0 0 >>> multipath/SATA_LUN38 ONLINE 0 0 0 >>> logs >>> multipath/SATA_LUN06 ONLINE 0 0 0 >>> cache >>> multipath/SATA_LUN02 ONLINE 0 0 0 >>> >>> errors: No known data errors >>> pool: tank >>> state: ONLINE >>> status: The pool is formatted using a legacy on-disk format. The pool >>> can >>> still be used, but some features are unavailable. >>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, the >>> pool will no longer be accessible on software that does not >>> support >>> feature >>> flags. >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> tank ONLINE 0 0 0 >>> mirror-0 ONLINE 0 0 0 >>> da34p3 ONLINE 0 0 0 >>> da35p3 ONLINE 0 0 0 >>> >>> errors: No known data errors >>> >>> while compiling a kernel, buildworld worked and installed ok >>> >>> lex -t >>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >>> aicasm/aicasm_scan.l >>> >>>> aicasm_scan.c >>>> >>> lex -t -Pmm >>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >>> aicasm/aicasm_macro_scan.l >>> >>>> aicasm_macro_scan.c >>>> >>> rm -f .depend_aicasm >>> mkdep -f .depend_aicasm -a -I. >>> -I/usr/src/sys/modules/**aic7xxx/aicasm/../../../dev/**aic7xxx/aicasm >>> -std=gnu99 >>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >>> aicasm/aicasm.c >>> /usr/src/sys/modules/aic7xxx/**aicasm/../../../dev/aic7xxx/** >>> aicasm/aicasm_symbol.c >>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c >>> Out of file descriptors >>> *** [.depend_aicasm] Error code 2 >>> >>> Stop in /usr/src/sys/modules/aic7xxx/**aicasm. >>> *** [buildkernel] Error code 1 >>> >>> Stop in /usr/src. >>> *** [buildkernel] Error code 1 >>> >> >> >> Do you know what a file descriptor is? >> >> There is something about it here: http://www.freebsd.org/doc/en/** >> books/handbook/configtuning-**kernel-limits.html(search >> for 'file descriptor') >> Or more info in here: https://www.google.nl/search?** >> q=freebsd+Out+of+file+**descriptors >> >> It mainly says that you have more files open than your system is >> configured to allow. This is often produced by a bug in a program which >> does not close some files properly. >> >> > Yes I do.... but this is a brand new zfsonroot with barely any data on it > and > > sysctl -a | grep kern.openfiles > kern.openfiles: 68 > root@:/master/builder # sysctl -a | grep kern.maxfiles > kern.maxfiles: 24600 > kern.maxfilesperproc: 11095 > > the openfiles and maxfiles seems to be plenty, but im out of > descriptors, i > used to see this back in the 4.x days when you could format > ufs with larger inodes, but zfs ??? really? Is this kern.openfiles when idle or during your buildkernel? Did you check 'ulimit -a'? You say installworld worked ok while you are just now doing buildkernel. Are your kernel and world out of sync? Ronald. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 12:46:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2E3A6443 for ; Wed, 15 May 2013 12:46:33 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-ob0-x22e.google.com (mail-ob0-x22e.google.com [IPv6:2607:f8b0:4003:c01::22e]) by mx1.freebsd.org (Postfix) with ESMTP id E6BD2A1B for ; Wed, 15 May 2013 12:46:32 +0000 (UTC) Received: by mail-ob0-f174.google.com with SMTP id un3so1912521obb.33 for ; Wed, 15 May 2013 05:46:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=dAr8NSyBwa4ewhD2X9ARR+HAP9erBQNU0fRpGXS2S7Y=; b=JDi9s9qgMXNGraQRWIwrODp2BW8v7IMAWSuX9szuApF+akUmh5E6WZwrcyJA36n5FI N73VRZqjNR3/i6pKqUqhkLXzlisH3erG4oMvFZkfHBUx7dWBTBERINrLf4FFRY9Det6A db2Xpzxz5ugkd0FGE7wAhdDBWa6lHzHPWsMQaZoBhBE7LOG0bcSyRpBWuPsL9471Q8+V WYIdD3L4LHGUTvXTToHjkxvFdfc54aNxuPk5XGiIcLuCkcclf1kUAldnjcN4wrZlkuOa Zzh9pG3+TejyzeYu4bm7vq5Fd4txmmAZz2JhrjqbCiOkEl2As5PFv73UEG117ChGTSUI rVTQ== MIME-Version: 1.0 X-Received: by 10.182.46.228 with SMTP id y4mr17273243obm.28.1368621992438; Wed, 15 May 2013 05:46:32 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 05:46:32 -0700 (PDT) In-Reply-To: References: Date: Wed, 15 May 2013 08:46:32 -0400 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: Ronald Klop Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 12:46:33 -0000 On Wed, May 15, 2013 at 8:34 AM, Ronald Klop wrote: > On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo > wrote: > > On Wed, May 15, 2013 at 8:22 AM, Ronald Klop > >**wrote: >> >> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < >>> outbackdingo@gmail.com> >>> wrote: >>> >>> So it seems a new deployment we just built with zfsonroot mirror and a >>> >>>> 48TB >>>> master pool is already out of File Descriptors??? >>>> >>>> pool: master >>>> state: ONLINE >>>> status: The pool is formatted using a legacy on-disk format. The pool >>>> can >>>> still be used, but some features are unavailable. >>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, the >>>> pool will no longer be accessible on software that does not >>>> support >>>> feature >>>> flags. >>>> scan: none requested >>>> config: >>>> >>>> NAME STATE READ WRITE CKSUM >>>> master ONLINE 0 0 0 >>>> raidz3-0 ONLINE 0 0 0 >>>> multipath/SATA_LUN01 ONLINE 0 0 0 >>>> multipath/SATA_LUN03 ONLINE 0 0 0 >>>> multipath/SATA_LUN04 ONLINE 0 0 0 >>>> multipath/SATA_LUN05 ONLINE 0 0 0 >>>> multipath/SATA_LUN07 ONLINE 0 0 0 >>>> multipath/SATA_LUN08 ONLINE 0 0 0 >>>> multipath/SATA_LUN09 ONLINE 0 0 0 >>>> multipath/SATA_LUN10 ONLINE 0 0 0 >>>> raidz3-1 ONLINE 0 0 0 >>>> multipath/SATA_LUN11 ONLINE 0 0 0 >>>> multipath/SATA_LUN12 ONLINE 0 0 0 >>>> multipath/SATA_LUN13 ONLINE 0 0 0 >>>> multipath/SATA_LUN14 ONLINE 0 0 0 >>>> multipath/SATA_LUN15 ONLINE 0 0 0 >>>> multipath/SATA_LUN16 ONLINE 0 0 0 >>>> multipath/SATA_LUN17 ONLINE 0 0 0 >>>> multipath/SATA_LUN18 ONLINE 0 0 0 >>>> raidz3-2 ONLINE 0 0 0 >>>> multipath/SATA_LUN19 ONLINE 0 0 0 >>>> multipath/SATA_LUN20 ONLINE 0 0 0 >>>> multipath/SATA_LUN21 ONLINE 0 0 0 >>>> multipath/SATA_LUN22 ONLINE 0 0 0 >>>> multipath/SATA_LUN23 ONLINE 0 0 0 >>>> multipath/SATA_LUN24 ONLINE 0 0 0 >>>> multipath/SATA_LUN26 ONLINE 0 0 0 >>>> multipath/SATA_LUN27 ONLINE 0 0 0 >>>> raidz3-3 ONLINE 0 0 0 >>>> multipath/SATA_LUN29 ONLINE 0 0 0 >>>> multipath/SATA_LUN30 ONLINE 0 0 0 >>>> multipath/SATA_LUN32 ONLINE 0 0 0 >>>> multipath/SATA_LUN33 ONLINE 0 0 0 >>>> multipath/SATA_LUN35 ONLINE 0 0 0 >>>> multipath/SATA_LUN36 ONLINE 0 0 0 >>>> multipath/SATA_LUN37 ONLINE 0 0 0 >>>> multipath/SATA_LUN38 ONLINE 0 0 0 >>>> logs >>>> multipath/SATA_LUN06 ONLINE 0 0 0 >>>> cache >>>> multipath/SATA_LUN02 ONLINE 0 0 0 >>>> >>>> errors: No known data errors >>>> pool: tank >>>> state: ONLINE >>>> status: The pool is formatted using a legacy on-disk format. The pool >>>> can >>>> still be used, but some features are unavailable. >>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, the >>>> pool will no longer be accessible on software that does not >>>> support >>>> feature >>>> flags. >>>> scan: none requested >>>> config: >>>> >>>> NAME STATE READ WRITE CKSUM >>>> tank ONLINE 0 0 0 >>>> mirror-0 ONLINE 0 0 0 >>>> da34p3 ONLINE 0 0 0 >>>> da35p3 ONLINE 0 0 0 >>>> >>>> errors: No known data errors >>>> >>>> while compiling a kernel, buildworld worked and installed ok >>>> >>>> lex -t >>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>> aicasm/aicasm_scan.l >>>> >>>> aicasm_scan.c >>>>> >>>>> lex -t -Pmm >>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>> >>>> aicasm/aicasm_macro_scan.l >>>> >>>> aicasm_macro_scan.c >>>>> >>>>> rm -f .depend_aicasm >>>> mkdep -f .depend_aicasm -a -I. >>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** >>>> aic7xxx/aicasm >>>> -std=gnu99 >>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>> aicasm/aicasm.c >>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>> >>>> aicasm/aicasm_symbol.c >>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c >>>> Out of file descriptors >>>> *** [.depend_aicasm] Error code 2 >>>> >>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. >>>> >>>> *** [buildkernel] Error code 1 >>>> >>>> Stop in /usr/src. >>>> *** [buildkernel] Error code 1 >>>> >>>> >>> >>> Do you know what a file descriptor is? >>> >>> There is something about it here: http://www.freebsd.org/doc/en/**** >>> books/handbook/configtuning-****kernel-limits.html>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html>(search >>> for 'file descriptor') >>> Or more info in here: https://www.google.nl/search?**** >>> q=freebsd+Out+of+file+****descriptors>> google.nl/search?q=freebsd+**Out+of+file+descriptors >>> > >>> >>> >>> It mainly says that you have more files open than your system is >>> configured to allow. This is often produced by a bug in a program which >>> does not close some files properly. >>> >>> >>> Yes I do.... but this is a brand new zfsonroot with barely any data on >> it >> and >> >> sysctl -a | grep kern.openfiles >> kern.openfiles: 68 >> root@:/master/builder # sysctl -a | grep kern.maxfiles >> kern.maxfiles: 24600 >> kern.maxfilesperproc: 11095 >> >> the openfiles and maxfiles seems to be plenty, but im out of descriptors, >> i >> used to see this back in the 4.x days when you could format >> ufs with larger inodes, but zfs ??? really? >> > > > Is this kern.openfiles when idle or during your buildkernel? > Did you check 'ulimit -a'? > You say installworld worked ok while you are just now doing buildkernel. > Are your kernel and world out of sync? > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make buildworld, then a make installworld, when i went to do a make buildkernel thats the error i got, so id say, yes it is "out of sync" my ulimits are ulimit -a cpu time (seconds, -t) unlimited file size (512-blocks, -f) unlimited data seg size (kbytes, -d) 33554432 stack size (kbytes, -s) 524288 core file size (512-blocks, -c) unlimited max memory size (kbytes, -m) unlimited locked memory (kbytes, -l) unlimited max user processes (-u) 5547 open files (-n) 11095 virtual mem size (kbytes, -v) unlimited swap limit (kbytes, -w) unlimited sbsize (bytes, -b) unlimited pseudo-terminals (-p) unlimited > Ronald. > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 13:08:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 354D7A8B for ; Wed, 15 May 2013 13:08:23 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id AB1F8B68 for ; Wed, 15 May 2013 13:08:22 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UcbRM-0000ay-3n; Wed, 15 May 2013 15:08:21 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UcbRK-0000nq-DR; Wed, 15 May 2013 15:08:18 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Outback Dingo" Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? References: Date: Wed, 15 May 2013 15:08:17 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: 0b90d2ac815c9ef578df8ccc1b50cfb0 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 13:08:23 -0000 On Wed, 15 May 2013 14:46:32 +0200, Outback Dingo wrote: > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop > wrote: > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo >> >> wrote: >> >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop >> >> >**wrote: >>> >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < >>>> outbackdingo@gmail.com> >>>> wrote: >>>> >>>> So it seems a new deployment we just built with zfsonroot mirror and >>>> a >>>> >>>>> 48TB >>>>> master pool is already out of File Descriptors??? >>>>> >>>>> pool: master >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The >>>>> pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, >>>>> the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> master ONLINE 0 0 0 >>>>> raidz3-0 ONLINE 0 0 0 >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 >>>>> raidz3-1 ONLINE 0 0 0 >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 >>>>> raidz3-2 ONLINE 0 0 0 >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 >>>>> raidz3-3 ONLINE 0 0 0 >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 >>>>> logs >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 >>>>> cache >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 >>>>> >>>>> errors: No known data errors >>>>> pool: tank >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The >>>>> pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, >>>>> the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> tank ONLINE 0 0 0 >>>>> mirror-0 ONLINE 0 0 0 >>>>> da34p3 ONLINE 0 0 0 >>>>> da35p3 ONLINE 0 0 0 >>>>> >>>>> errors: No known data errors >>>>> >>>>> while compiling a kernel, buildworld worked and installed ok >>>>> >>>>> lex -t >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm_scan.l >>>>> >>>>> aicasm_scan.c >>>>>> >>>>>> lex -t -Pmm >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> >>>>> aicasm/aicasm_macro_scan.l >>>>> >>>>> aicasm_macro_scan.c >>>>>> >>>>>> rm -f .depend_aicasm >>>>> mkdep -f .depend_aicasm -a -I. >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** >>>>> aic7xxx/aicasm >>>>> -std=gnu99 >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm.c >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> >>>>> aicasm/aicasm_symbol.c >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c >>>>> Out of file descriptors >>>>> *** [.depend_aicasm] Error code 2 >>>>> >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. >>>>> >>>>> *** [buildkernel] Error code 1 >>>>> >>>>> Stop in /usr/src. >>>>> *** [buildkernel] Error code 1 >>>>> >>>>> >>>> >>>> Do you know what a file descriptor is? >>>> >>>> There is something about it here: >>>> http://www.freebsd.org/doc/en/**** >>>> books/handbook/configtuning-****kernel-limits.html>>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html>(search >>>> for 'file descriptor') >>>> Or more info in here: >>>> https://www.google.nl/search?**** >>>> q=freebsd+Out+of+file+****descriptors>>> google.nl/search?q=freebsd+**Out+of+file+descriptors >>>> > >>>> >>>> >>>> It mainly says that you have more files open than your system is >>>> configured to allow. This is often produced by a bug in a program >>>> which >>>> does not close some files properly. >>>> >>>> >>>> Yes I do.... but this is a brand new zfsonroot with barely any data >>>> on >>> it >>> and >>> >>> sysctl -a | grep kern.openfiles >>> kern.openfiles: 68 >>> root@:/master/builder # sysctl -a | grep kern.maxfiles >>> kern.maxfiles: 24600 >>> kern.maxfilesperproc: 11095 >>> >>> the openfiles and maxfiles seems to be plenty, but im out of >>> descriptors, >>> i >>> used to see this back in the 4.x days when you could format >>> ufs with larger inodes, but zfs ??? really? >>> >> >> >> Is this kern.openfiles when idle or during your buildkernel? >> Did you check 'ulimit -a'? >> You say installworld worked ok while you are just now doing buildkernel. >> Are your kernel and world out of sync? >> >> > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make > buildworld, > then a make installworld, when i went to do a make buildkernel > thats the error i got, so id say, yes it is "out of sync" Read /usr/src/UPDATING from the line 'COMMON ITEMS:'. Especially interesting is 'To upgrade in-place from stable to current'. The order is important. See also http://www.freebsd.org/doc/en/books/handbook/updating-upgrading.html for a lot of text about the same. To fix your current situation. I'm not familiar with that situation but can imagine to try to put back a 9-RELEASE world. Do you have a snapshot of your current system with world from 9-RELEASE? Or another backup? Ronald. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 14:07:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 23B9ED46 for ; Wed, 15 May 2013 14:07:57 +0000 (UTC) (envelope-from prvs=1847987112=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id B7B00B9 for ; Wed, 15 May 2013 14:07:54 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003819864.msg for ; Wed, 15 May 2013 15:07:47 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 15 May 2013 15:07:47 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=1847987112=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <3E9CA9334E6F433A8F135ACD5C237340@multiplay.co.uk> From: "Steven Hartland" To: "Ajit Jain" References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> Subject: Re: seeing data corruption with zfs trim functionality Date: Wed, 15 May 2013 15:07:44 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 14:07:57 -0000 Unless you have the latest CAM patches, which is in current, you wont be doing TRIM on SATA disk connected to an LSI controller. I've just tested using the following cmd under 8.3 with MFC'ed changes from current, using ATA_TRIM, UNMAP & WS16 and have had no issues on a machine with Intel SSD and LSI controller. ./iotest -t 20 -s 536870912 -W 100 -T 500 /test/iotest/ I did however notice that your test is hardly doing any disk access apart from when its "Initializing test file....", instead it seems to be CPU bound, so not sure if there's a problem with the iotest code or with my command line args? Given this my current thinking is either: 1. There's a problem with your patches 2. There's a bug in the FW of the Seagate disk 3. There's a problem with the UNMAP code which is being trigged by your disk only. I think #3 is quite unlikely. If you could install a recent version of current and test with that it should rule out #1, leaving #2. Regards Steve ----- Original Message ----- From: "Ajit Jain" > Hi Steve, > > One more thing I am not seeing data corruption with SATA SSD (kingston). > The issue was seen on SAS SSD i.e. Seagate PULSAR ST100FM0002 . > > regards, > ajit > > > On Wed, May 15, 2013 at 1:49 PM, Ajit Jain wrote: > >> Hi Steven, >> >> Please find the tar ball of src code and binary of the test utility >> attached with the mail. >> Steps of test: >> 1. Enable zfs trim in /boot/loader.conf (vfs.zfs.trim_disable=0) >> 2. Set the delete method of the ssd device as UNMAP or WS16. >> 3. Create a pool (and optionally dataset) on the device. >> 4. Run iotest utility with thread count 10 (-t option) file size as at >> least 5GB >> Second to execute as at least 500 sec (-T option) and write as 100% (W >> option). >> >> regards, >> ajit >> >> >> On Wed, May 15, 2013 at 12:56 PM, Steven Hartland > > wrote: >> >>> Could you provide us with details on the tests your using so we can >>> run them here on current sources and see if we see any issues? >>> >>> Regards >>> Steve >>> >>> ----- Original Message ----- From: "Ajit Jain" >>> To: "Steven Hartland" >>> Cc: "freebsd-fs" >>> Sent: Wednesday, May 15, 2013 6:47 AM >>> >>> Subject: Re: seeing data corruption with zfs trim functionality >>> >>> >>> Hi Steven, >>>> >>>> Thanks for the follow-up. >>>> The code where I pulled in zfs trim patches is not updated to 9 stable >>>> specially the cam directory. >>>> I pulled in many dependent patches in order to apply the patches that you >>>> gave. After that all da devices >>>> CAM_PERIPH_INVALID in dadone() because read capability was returning a >>>> very >>>> big number (bigger than MAXPHYS) >>>> for the block size. I think this is because I have not update the code >>>> to 9 >>>> stable (only pulled in required patches and miss >>>> some patches). >>>> >>>> So, I am planning to first update my code to 9stable and then try the >>>> same >>>> test again. That might take some time. >>>> >>>> >>>> thanks again, >>>> ajit >>>> >>>> >>>> On Wed, May 15, 2013 at 2:40 AM, Steven Hartland < >>>> killing@multiplay.co.uk>**wrote: >>>> >>>> ----- Original Message ----- From: "Steven Hartland" >>>>> >>>>> What version are you porting the changes to? >>>>> >>>>>> >>>>>>>> What SSD are you using? >>>>>>>> >>>>>>>> What LSI controller are you using? >>>>>>>> >>>>>>>> >>>>>>> I'd also like to see "zpool status" (for every pool that involves this >>>>>>> SSD) and "gpart show" against the disk itself. >>>>>>> >>>>>>> >>>>>> Also: >>>>>> 1. What FW version is your LSI? You can get this from dmesg. >>>>>> 2. The exact command line your running iotest with? >>>>>> >>>>>> >>>>> Any update on this? I'd like to try and replicate your test here so >>>>> would appreciate as much information as possible. >>>>> >>>>> >>>>> Regards >>>>> Steve >>>>> >>>>> ==============================****================== >>>>> >>>>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>>>> the person or entity to whom it is addressed. In the event of >>>>> misdirection, >>>>> the recipient is prohibited from using, copying, printing or otherwise >>>>> disseminating it or any information contained in it. >>>>> In the event of misdirection, illegible or incomplete transmission >>>>> please >>>>> telephone +44 845 868 1337 >>>>> or return the E.mail to postmaster@multiplay.co.uk. >>>>> >>>>> >>>>> >>>> >>> ==============================**================== >>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>> the person or entity to whom it is addressed. In the event of misdirection, >>> the recipient is prohibited from using, copying, printing or otherwise >>> disseminating it or any information contained in it. >>> In the event of misdirection, illegible or incomplete transmission please >>> telephone +44 845 868 1337 >>> or return the E.mail to postmaster@multiplay.co.uk. >>> >>> >> > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 15:18:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1038217B for ; Wed, 15 May 2013 15:18:40 +0000 (UTC) (envelope-from ajit.jain@cloudbyte.com) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) by mx1.freebsd.org (Postfix) with ESMTP id CDF5D6A4 for ; Wed, 15 May 2013 15:18:39 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id va2so2140174obc.13 for ; Wed, 15 May 2013 08:18:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=57Z8h9PleFS+o3uO39AgMeT6IvZ/Flx2H4lHKFwy2to=; b=kVlnwsyfN840DgYCHJDz1FDpQZgMbWCPno7hSwkCNYNTSAnIBimcUgUJu7g7RhzxQJ q9rGiA4ZQG2RsFk04PUO2wG593Hck4hOJ727iQbfls86CqhZ7lIAiWMzW6lgKgPlQXYl nOW/cbb5F3IpgrgMWSX+a1nSiF2XkivYo4P4wmBJJn8BN7imKmPsgwKYPWnMfKWYn2V/ hNVRLhWx1a+NlXyiUXLbbUo9l/v8Re7u/bqVf/3N1sFMyfYIdNL6KuNEK0v4bw3djG7k /Gx7z1Om9k4Ti9+kQSYwKo7cK7m8cLJdpx1v7inYmgZQtdSsDGFo6MLcAKU23VDsQjdN t9Tw== X-Received: by 10.182.225.199 with SMTP id rm7mr4040607obc.20.1368631119300; Wed, 15 May 2013 08:18:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.151.134 with HTTP; Wed, 15 May 2013 08:18:19 -0700 (PDT) In-Reply-To: <3E9CA9334E6F433A8F135ACD5C237340@multiplay.co.uk> References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> <3E9CA9334E6F433A8F135ACD5C237340@multiplay.co.uk> From: Ajit Jain Date: Wed, 15 May 2013 20:48:19 +0530 Message-ID: Subject: Re: seeing data corruption with zfs trim functionality To: Steven Hartland X-Gm-Message-State: ALoCoQmW40N4Da434peRebl26MKlIPNf7834b/FcdBEEKKj2FZTIQF4GQC4abARLYhD2nn6uOW6K Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 15:18:40 -0000 Hi Steven, Thanks for the update. It is surprising that there is no/less disk activity as the command is correct. May be we need to enable sync=always on the zfs dataset. I will try again the once I update the cam code base. regards, ajit On Wed, May 15, 2013 at 7:37 PM, Steven Hartland wrote: > Unless you have the latest CAM patches, which is in current, you wont be > doing TRIM on SATA disk connected to an LSI controller. > > I've just tested using the following cmd under 8.3 with MFC'ed changes from > current, using ATA_TRIM, UNMAP & WS16 and have had no issues on a machine > with Intel SSD and LSI controller. > ./iotest -t 20 -s 536870912 -W 100 -T 500 /test/iotest/ > > I did however notice that your test is hardly doing any disk access apart > from when its "Initializing test file....", instead it seems to be CPU > bound, > so not sure if there's a problem with the iotest code or with my command > line args? > > Given this my current thinking is either: > 1. There's a problem with your patches 2. There's a bug in the FW of the > Seagate disk > 3. There's a problem with the UNMAP code which is being trigged by your > disk > only. > > I think #3 is quite unlikely. > > If you could install a recent version of current and test with that it > should rule out #1, leaving #2. > > > Regards > Steve > > ----- Original Message ----- From: "Ajit Jain" > > > Hi Steve, >> >> One more thing I am not seeing data corruption with SATA SSD (kingston). >> The issue was seen on SAS SSD i.e. Seagate PULSAR ST100FM0002 . >> >> regards, >> ajit >> >> >> On Wed, May 15, 2013 at 1:49 PM, Ajit Jain >> wrote: >> >> Hi Steven, >>> >>> Please find the tar ball of src code and binary of the test utility >>> attached with the mail. >>> Steps of test: >>> 1. Enable zfs trim in /boot/loader.conf (vfs.zfs.trim_disable=0) >>> 2. Set the delete method of the ssd device as UNMAP or WS16. >>> 3. Create a pool (and optionally dataset) on the device. >>> 4. Run iotest utility with thread count 10 (-t option) file size as at >>> least 5GB >>> Second to execute as at least 500 sec (-T option) and write as 100% >>> (W >>> option). >>> >>> regards, >>> ajit >>> >>> >>> On Wed, May 15, 2013 at 12:56 PM, Steven Hartland < >>> killing@multiplay.co.uk >>> > wrote: >>> >>> Could you provide us with details on the tests your using so we can >>>> run them here on current sources and see if we see any issues? >>>> >>>> Regards >>>> Steve >>>> >>>> ----- Original Message ----- From: "Ajit Jain" >>> > >>>> To: "Steven Hartland" >>>> Cc: "freebsd-fs" >>>> Sent: Wednesday, May 15, 2013 6:47 AM >>>> >>>> Subject: Re: seeing data corruption with zfs trim functionality >>>> >>>> >>>> Hi Steven, >>>> >>>>> >>>>> Thanks for the follow-up. >>>>> The code where I pulled in zfs trim patches is not updated to 9 stable >>>>> specially the cam directory. >>>>> I pulled in many dependent patches in order to apply the patches that >>>>> you >>>>> gave. After that all da devices >>>>> CAM_PERIPH_INVALID in dadone() because read capability was returning a >>>>> very >>>>> big number (bigger than MAXPHYS) >>>>> for the block size. I think this is because I have not update the code >>>>> to 9 >>>>> stable (only pulled in required patches and miss >>>>> some patches). >>>>> >>>>> So, I am planning to first update my code to 9stable and then try the >>>>> same >>>>> test again. That might take some time. >>>>> >>>>> >>>>> thanks again, >>>>> ajit >>>>> >>>>> >>>>> On Wed, May 15, 2013 at 2:40 AM, Steven Hartland < >>>>> killing@multiplay.co.uk>****wrote: >>>>> >>>>> >>>>> ----- Original Message ----- From: "Steven Hartland" >>>>> >>>>>> >>>>>> What version are you porting the changes to? >>>>>> >>>>>> >>>>>>> What SSD are you using? >>>>>>>>> >>>>>>>>> What LSI controller are you using? >>>>>>>>> >>>>>>>>> >>>>>>>>> I'd also like to see "zpool status" (for every pool that involves >>>>>>>> this >>>>>>>> SSD) and "gpart show" against the disk itself. >>>>>>>> >>>>>>>> >>>>>>>> Also: >>>>>>> 1. What FW version is your LSI? You can get this from dmesg. >>>>>>> 2. The exact command line your running iotest with? >>>>>>> >>>>>>> >>>>>>> Any update on this? I'd like to try and replicate your test here so >>>>>> would appreciate as much information as possible. >>>>>> >>>>>> >>>>>> Regards >>>>>> Steve >>>>>> >>>>>> ==============================******================== >>>>>> >>>>>> >>>>>> This e.mail is private and confidential between Multiplay (UK) Ltd. >>>>>> and >>>>>> the person or entity to whom it is addressed. In the event of >>>>>> misdirection, >>>>>> the recipient is prohibited from using, copying, printing or otherwise >>>>>> disseminating it or any information contained in it. >>>>>> In the event of misdirection, illegible or incomplete transmission >>>>>> please >>>>>> telephone +44 845 868 1337 >>>>>> or return the E.mail to postmaster@multiplay.co.uk. >>>>>> >>>>>> >>>>>> >>>>>> >>>>> ==============================****================== >>>> This e.mail is private and confidential between Multiplay (UK) Ltd. and >>>> the person or entity to whom it is addressed. In the event of >>>> misdirection, >>>> the recipient is prohibited from using, copying, printing or otherwise >>>> disseminating it or any information contained in it. >>>> In the event of misdirection, illegible or incomplete transmission >>>> please >>>> telephone +44 845 868 1337 >>>> or return the E.mail to postmaster@multiplay.co.uk. >>>> >>>> >>>> >>> >> > ==============================**================== > This e.mail is private and confidential between Multiplay (UK) Ltd. and > the person or entity to whom it is addressed. In the event of misdirection, > the recipient is prohibited from using, copying, printing or otherwise > disseminating it or any information contained in it. > In the event of misdirection, illegible or incomplete transmission please > telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 15:43:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3200D8A7 for ; Wed, 15 May 2013 15:43:54 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qe0-f46.google.com (mail-qe0-f46.google.com [209.85.128.46]) by mx1.freebsd.org (Postfix) with ESMTP id EB70284B for ; Wed, 15 May 2013 15:43:53 +0000 (UTC) Received: by mail-qe0-f46.google.com with SMTP id f6so1348713qej.33 for ; Wed, 15 May 2013 08:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=TzyWXaWFwMW06hVjBSCvQV4eFtDwmj/VFcGbfyHYUfM=; b=pkB528bMGKxbZ7tFmm5M9SB+ABt9z8pHHevIyaTMzv6xmUjyCkDkTMm1P7/6sd0C7V 47Upvp27QBGfC+XZfMqyLv5T+a2KHIr0+07yahYv63S9Ww922Yw9npCp/lCrLjNfPNFM stFSIACwlb9sbwf9f56MRedUiWGsxNE6aO+BNPTr4Qx1+n+WUb1XPUD7XGVbp79lxb/o gSRSlR6wCR5YdoPit8CEZ7JvUKK5EibXpVt8BH6FwqXw2t05whT2MbvRBC2EdJ6RMeVN 7xNASHVeBBxPMFNGQa3ni3wWg92AlJ7hlAYiF/IJeUyiHACMY96cSMvIpq1lfDZpXWgZ E3Hg== MIME-Version: 1.0 X-Received: by 10.229.32.2 with SMTP id a2mr4801806qcd.78.1368632633066; Wed, 15 May 2013 08:43:53 -0700 (PDT) Received: by 10.49.1.44 with HTTP; Wed, 15 May 2013 08:43:52 -0700 (PDT) In-Reply-To: References: Date: Wed, 15 May 2013 08:43:52 -0700 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Freddie Cash To: Outback Dingo Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Filesystems , Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 15:43:54 -0000 On Wed, May 15, 2013 at 5:46 AM, Outback Dingo wrote: > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make buildworld, > then a make installworld, when i went to do a make buildkernel > thats the error i got, so id say, yes it is "out of sync" > That's your issue right there. You are running a newer userland on an older kernel, which doesn't work. You can run an older userland on a newer kernel (aka installkernel for -CURRENT while running -RELEASE world). But you can't do the opposite. The order of install is important. Kernel before world. Always. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Wed May 15 15:49:19 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5C40799A for ; Wed, 15 May 2013 15:49:19 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) by mx1.freebsd.org (Postfix) with ESMTP id E5B8588F for ; Wed, 15 May 2013 15:49:18 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id hr14so1784191wib.16 for ; Wed, 15 May 2013 08:49:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=Gi4mUzBYSWZdtawt5inC+6rbYV5PqqHyGjU9JXqNkv4=; b=F8vn7N9jtb/KXRwzcb1IjSlL1bXQ97DPaKoZFksAUbfdQ6+37uD18i8C2T1+opyINn OZ5Zkzsb9BniUo4mLcQnrLZ0IYC3i2z/mJhz1cYexlKqJX3x+DO7iHgAlnQwXDU5khnE my9ydFZNgTIljle481ptHv7e6BNCmjs17H1IPHcl5+N6c+rMv3fP5IpJb3xSUdT88pjX 1kcXl59gNbdX7omnWjPbQVjPiZMiye1oDffFVBEQEeUVT5Fsd1ZV7F1u3PfJZCKpgPfC Qt8+eJ629gbi+d9lPPmlQUy4eJf9xlU0ZNwtHl41gDrk7dokNvsWe7iNZDUsP3UjP8tf U9+w== X-Received: by 10.180.108.3 with SMTP id hg3mr15672211wib.17.1368632944865; Wed, 15 May 2013 08:49:04 -0700 (PDT) Received: from dfleuriot.paris.hi-media-techno.com ([83.167.62.196]) by mx.google.com with ESMTPSA id ay7sm21450167wib.9.2013.05.15.08.49.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 15 May 2013 08:49:03 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Fleuriot Damien In-Reply-To: Date: Wed, 15 May 2013 17:49:03 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> References: To: Outback Dingo X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQleseUUzG1zOr/qUFyJFBZ0wmvXQe4gi4Z10c0ZyxPha8WzKvL7lW1nrbmQDlGIyqq8AHNS Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 15:49:19 -0000 On May 15, 2013, at 2:46 PM, Outback Dingo = wrote: > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop = wrote: >=20 >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo = >> wrote: >>=20 >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop = >>> **wrote: >>>=20 >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < >>>> outbackdingo@gmail.com> >>>> wrote: >>>>=20 >>>> So it seems a new deployment we just built with zfsonroot mirror = and a >>>>=20 >>>>> 48TB >>>>> master pool is already out of File Descriptors??? >>>>>=20 >>>>> pool: master >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The = pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is = done, the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>>=20 >>>>> NAME STATE READ WRITE CKSUM >>>>> master ONLINE 0 0 0 >>>>> raidz3-0 ONLINE 0 0 0 >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 >>>>> raidz3-1 ONLINE 0 0 0 >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 >>>>> raidz3-2 ONLINE 0 0 0 >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 >>>>> raidz3-3 ONLINE 0 0 0 >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 >>>>> logs >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 >>>>> cache >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 >>>>>=20 >>>>> errors: No known data errors >>>>> pool: tank >>>>> state: ONLINE >>>>> status: The pool is formatted using a legacy on-disk format. The = pool >>>>> can >>>>> still be used, but some features are unavailable. >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is = done, the >>>>> pool will no longer be accessible on software that does not >>>>> support >>>>> feature >>>>> flags. >>>>> scan: none requested >>>>> config: >>>>>=20 >>>>> NAME STATE READ WRITE CKSUM >>>>> tank ONLINE 0 0 0 >>>>> mirror-0 ONLINE 0 0 0 >>>>> da34p3 ONLINE 0 0 0 >>>>> da35p3 ONLINE 0 0 0 >>>>>=20 >>>>> errors: No known data errors >>>>>=20 >>>>> while compiling a kernel, buildworld worked and installed ok >>>>>=20 >>>>> lex -t >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm_scan.l >>>>>=20 >>>>> aicasm_scan.c >>>>>>=20 >>>>>> lex -t -Pmm >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>>=20 >>>>> aicasm/aicasm_macro_scan.l >>>>>=20 >>>>> aicasm_macro_scan.c >>>>>>=20 >>>>>> rm -f .depend_aicasm >>>>> mkdep -f .depend_aicasm -a -I. >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** >>>>> aic7xxx/aicasm >>>>> -std=3Dgnu99 >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>> aicasm/aicasm.c >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >>>>>=20 >>>>> aicasm/aicasm_symbol.c >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c = aicasm_macro_scan.c >>>>> Out of file descriptors >>>>> *** [.depend_aicasm] Error code 2 >>>>>=20 >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. >>>>>=20 >>>>> *** [buildkernel] Error code 1 >>>>>=20 >>>>> Stop in /usr/src. >>>>> *** [buildkernel] Error code 1 >>>>>=20 >>>>>=20 >>>>=20 >>>> Do you know what a file descriptor is? >>>>=20 >>>> There is something about it here: = http://www.freebsd.org/doc/en/**** >>>> books/handbook/configtuning-****kernel-limits.html>>> = freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html>= (search >>>> for 'file descriptor') >>>> Or more info in here: = https://www.google.nl/search?**** >>>> q=3Dfreebsd+Out+of+file+****descriptors>>> = google.nl/search?q=3Dfreebsd+**Out+of+file+descriptors >>>>>=20 >>>>=20 >>>>=20 >>>> It mainly says that you have more files open than your system is >>>> configured to allow. This is often produced by a bug in a program = which >>>> does not close some files properly. >>>>=20 >>>>=20 >>>> Yes I do.... but this is a brand new zfsonroot with barely any data = on >>> it >>> and >>>=20 >>> sysctl -a | grep kern.openfiles >>> kern.openfiles: 68 >>> root@:/master/builder # sysctl -a | grep kern.maxfiles >>> kern.maxfiles: 24600 >>> kern.maxfilesperproc: 11095 >>>=20 >>> the openfiles and maxfiles seems to be plenty, but im out of = descriptors, >>> i >>> used to see this back in the 4.x days when you could format >>> ufs with larger inodes, but zfs ??? really? >>>=20 >>=20 >>=20 >> Is this kern.openfiles when idle or during your buildkernel? >> Did you check 'ulimit -a'? >> You say installworld worked ok while you are just now doing = buildkernel. >> Are your kernel and world out of sync? >>=20 >>=20 > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make = buildworld, > then a make installworld, when i went to do a make buildkernel > thats the error i got, so id say, yes it is "out of sync" Wow wow wow, hold on a sec here. You're installing your new world before the kernel ? I've had no trouble upgrading several boxes from 8-STABLE to = 10.0-CURRENT, following the regular procedure as described at: http://www.freebsd.org/doc/handbook/makeworld.html Note that the correct order is: - buildworld (or kernel-toolchain if you only want to test the kernel = and not do the whole world) - buildkernel - installkernel - reboot - mergemaster -p (preferably in single user, I've always run it = multiuser w/o problems) - installworld - mergemaster - reboot You don't want to run the new world on the old kernel. From owner-freebsd-fs@FreeBSD.ORG Wed May 15 15:49:53 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7BEE2A23 for ; Wed, 15 May 2013 15:49:53 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oa0-f52.google.com (mail-oa0-f52.google.com [209.85.219.52]) by mx1.freebsd.org (Postfix) with ESMTP id 4807189C for ; Wed, 15 May 2013 15:49:53 +0000 (UTC) Received: by mail-oa0-f52.google.com with SMTP id h1so2413885oag.11 for ; Wed, 15 May 2013 08:49:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=0NMcd8FRWSCWpI9ByyD92C6vTNJI+OO6tCTtQ9l3pP0=; b=yobteV6+nYFk5TpuMCtwKkQU/84nCjr12UHObq9fg1gbVGcaKY2/AFaBnTZWZTEix+ Fk3038RJR7/szyhtexr3PhHkdMRUs5pBb4C08Bp26nQX/TM40PFcJZk+SNKedW5yVbZp fbyvtn7XnxqjLm5thxQWe3l1yb6HKmChxtVGYeQBG8pbEqUuQlWOsBz/D8UyaWYkx73V RTeyklnK4Y4kr6s22xfAF10fnPsjWV0pelYj+2f605QJbgTgb9Su2XTlf++o7p5ya6G2 dJ5UG3vTQL87xn6dg4TZ0lYdWQ+PkvXJJw5TDQckC0rXaNONq8r1CjyLNLFTyLpuq57f yXeQ== MIME-Version: 1.0 X-Received: by 10.182.138.4 with SMTP id qm4mr17881634obb.101.1368632992634; Wed, 15 May 2013 08:49:52 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 08:49:52 -0700 (PDT) In-Reply-To: References: Date: Wed, 15 May 2013 11:49:52 -0400 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: Freddie Cash Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Filesystems , Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 15:49:53 -0000 On Wed, May 15, 2013 at 11:43 AM, Freddie Cash wrote: > On Wed, May 15, 2013 at 5:46 AM, Outback Dingo wrote: > >> well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make buildworld, >> then a make installworld, when i went to do a make buildkernel >> thats the error i got, so id say, yes it is "out of sync" >> > > That's your issue right there. You are running a newer userland on an > older kernel, which doesn't work. > > You can run an older userland on a newer kernel (aka installkernel for > -CURRENT while running -RELEASE world). But you can't do the opposite. > > The order of install is important. Kernel before world. Always. > > on the reboot i got to Out of File Descriptors enter path to /bin/sh Yeah i noticed that when i ran the -j 8 buildworld buildkernel the world built, but the kernel failed... just my luck now the box is in single user mode with a readonly zfs tank pool on root, on kernel 9x with a CURRENT userland. > -- > Freddie Cash > fjwcash@gmail.com > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 15:56:51 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AD1A4B78 for ; Wed, 15 May 2013 15:56:51 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) by mx1.freebsd.org (Postfix) with ESMTP id 79C148FE for ; Wed, 15 May 2013 15:56:51 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id va2so2164955obc.27 for ; Wed, 15 May 2013 08:56:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=RBi9a2sPiQqz/iPGbPIJOq8IGOWOEnb5VQuU1kNjpv8=; b=YZ8DIZY4slmQT3ipJn51wIu1Nq6U+/QuuKyAuu9r7ICEUHlndz+k+yjhIuQVb4WdeA Rr7u1VrhwP4CvAwciOUFXsk7ZGrNmFeSP5IXnCCKnjpVe6THKuBHMasRvIbA/17i8i+U ol/s0eOGdnAZgNSZ9XH/ZbfblRRyOA8+7LrCA9MhBWgvwaZB77L+1/2rNoKxiLUzH1/i dDvPob/Db+F5hkcGadNRtN8Y+Gnn8LFLtXlLTpJFUotieO4XJJOYAseFJgwsKfez2d0j 8E4/RTd9YTgCxo7GOEUi8HnEDpKjsFPTOREOyTYxIplSZsvbKuGp428wJCyWXElsId3K T2Tg== MIME-Version: 1.0 X-Received: by 10.60.41.232 with SMTP id i8mr18982971oel.129.1368633411058; Wed, 15 May 2013 08:56:51 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 08:56:50 -0700 (PDT) In-Reply-To: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> References: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> Date: Wed, 15 May 2013 11:56:50 -0400 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: Fleuriot Damien Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 15:56:51 -0000 On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien wrote: > > On May 15, 2013, at 2:46 PM, Outback Dingo wrote: > > > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop < > ronald-freebsd8@klop.yi.org>wrote: > > > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo < > outbackdingo@gmail.com> > >> wrote: > >> > >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop < > ronald-freebsd8@klop.yi.org > >>>> **wrote: > >>> > >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < > >>>> outbackdingo@gmail.com> > >>>> wrote: > >>>> > >>>> So it seems a new deployment we just built with zfsonroot mirror and a > >>>> > >>>>> 48TB > >>>>> master pool is already out of File Descriptors??? > >>>>> > >>>>> pool: master > >>>>> state: ONLINE > >>>>> status: The pool is formatted using a legacy on-disk format. The > pool > >>>>> can > >>>>> still be used, but some features are unavailable. > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, > the > >>>>> pool will no longer be accessible on software that does not > >>>>> support > >>>>> feature > >>>>> flags. > >>>>> scan: none requested > >>>>> config: > >>>>> > >>>>> NAME STATE READ WRITE CKSUM > >>>>> master ONLINE 0 0 0 > >>>>> raidz3-0 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 > >>>>> raidz3-1 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 > >>>>> raidz3-2 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 > >>>>> raidz3-3 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 > >>>>> logs > >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 > >>>>> cache > >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 > >>>>> > >>>>> errors: No known data errors > >>>>> pool: tank > >>>>> state: ONLINE > >>>>> status: The pool is formatted using a legacy on-disk format. The > pool > >>>>> can > >>>>> still be used, but some features are unavailable. > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, > the > >>>>> pool will no longer be accessible on software that does not > >>>>> support > >>>>> feature > >>>>> flags. > >>>>> scan: none requested > >>>>> config: > >>>>> > >>>>> NAME STATE READ WRITE CKSUM > >>>>> tank ONLINE 0 0 0 > >>>>> mirror-0 ONLINE 0 0 0 > >>>>> da34p3 ONLINE 0 0 0 > >>>>> da35p3 ONLINE 0 0 0 > >>>>> > >>>>> errors: No known data errors > >>>>> > >>>>> while compiling a kernel, buildworld worked and installed ok > >>>>> > >>>>> lex -t > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> aicasm/aicasm_scan.l > >>>>> > >>>>> aicasm_scan.c > >>>>>> > >>>>>> lex -t -Pmm > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> > >>>>> aicasm/aicasm_macro_scan.l > >>>>> > >>>>> aicasm_macro_scan.c > >>>>>> > >>>>>> rm -f .depend_aicasm > >>>>> mkdep -f .depend_aicasm -a -I. > >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** > >>>>> aic7xxx/aicasm > >>>>> -std=gnu99 > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> aicasm/aicasm.c > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> > >>>>> aicasm/aicasm_symbol.c > >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c > >>>>> Out of file descriptors > >>>>> *** [.depend_aicasm] Error code 2 > >>>>> > >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. > >>>>> > >>>>> *** [buildkernel] Error code 1 > >>>>> > >>>>> Stop in /usr/src. > >>>>> *** [buildkernel] Error code 1 > >>>>> > >>>>> > >>>> > >>>> Do you know what a file descriptor is? > >>>> > >>>> There is something about it here: http://www.freebsd.org/doc/en/****< > http://www.freebsd.org/doc/en/**> > >>>> books/handbook/configtuning-****kernel-limits.html >>>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html > < > http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html > >>(search > >>>> for 'file descriptor') > >>>> Or more info in here: https://www.google.nl/search?****< > https://www.google.nl/search?**> > >>>> q=freebsd+Out+of+file+****descriptors >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors< > https://www.google.nl/search?q=freebsd+Out+of+file+descriptors> > >>>>> > >>>> > >>>> > >>>> It mainly says that you have more files open than your system is > >>>> configured to allow. This is often produced by a bug in a program > which > >>>> does not close some files properly. > >>>> > >>>> > >>>> Yes I do.... but this is a brand new zfsonroot with barely any data on > >>> it > >>> and > >>> > >>> sysctl -a | grep kern.openfiles > >>> kern.openfiles: 68 > >>> root@:/master/builder # sysctl -a | grep kern.maxfiles > >>> kern.maxfiles: 24600 > >>> kern.maxfilesperproc: 11095 > >>> > >>> the openfiles and maxfiles seems to be plenty, but im out of > descriptors, > >>> i > >>> used to see this back in the 4.x days when you could format > >>> ufs with larger inodes, but zfs ??? really? > >>> > >> > >> > >> Is this kern.openfiles when idle or during your buildkernel? > >> Did you check 'ulimit -a'? > >> You say installworld worked ok while you are just now doing buildkernel. > >> Are your kernel and world out of sync? > >> > >> > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make > buildworld, > > then a make installworld, when i went to do a make buildkernel > > thats the error i got, so id say, yes it is "out of sync" > > > Wow wow wow, hold on a sec here. > > You're installing your new world before the kernel ? > > > I've had no trouble upgrading several boxes from 8-STABLE to 10.0-CURRENT, > following the regular procedure as described at: > http://www.freebsd.org/doc/handbook/makeworld.html > > > Note that the correct order is: > - buildworld (or kernel-toolchain if you only want to test the kernel and > not do the whole world) > - buildkernel > - installkernel > - reboot > - mergemaster -p (preferably in single user, I've always run it multiuser > w/o problems) > - installworld > - mergemaster > - reboot > > You don't want to run the new world on the old kernel. > yeah i know better, seems when i went to install the old kernel the build had failed, i can get through it from here it was odd because the CURRENT kernel failed to build with a yyparse error, until after i had completed a buildworld the other node i did is fine.......... From owner-freebsd-fs@FreeBSD.ORG Wed May 15 16:03:44 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6885DEC2 for ; Wed, 15 May 2013 16:03:44 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) by mx1.freebsd.org (Postfix) with ESMTP id DFEAD96C for ; Wed, 15 May 2013 16:03:43 +0000 (UTC) Received: by mail-wi0-f176.google.com with SMTP id hr14so5333937wib.15 for ; Wed, 15 May 2013 09:03:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:message-id:references:to:x-mailer:x-gm-message-state; bh=pR/V/NmcH/P5rNTgJoJbecwEFyU11iOkuQnw0gkPSPc=; b=ISbwf3KoPf6M1kdqX4WGtHBXA6KSIaEVGkh/rIAMiYKLndnn3FDA2NiLvGRSuTDp+M sj+mZ/nluZja/ubElAFBtJQwG4VagdbXvSlvCI2LWBdMv1bsgmuuRiySpcgLGoTwYCi0 lqpl56VCCQ5SAvdcBCTBHdrEESA0V57mOMPeB4HqtI/GK8GTLA9VkRqr5rMAKpT+e6mJ j6EUwDJgNdI+rN6frUn0pKK7L5Ls1ub2oI04AA3GzewhF9WRXydlX+8mciAme5DYVa5X OHH3TxizBEwAvPBcVk/EDrNysiKFyn1HzjSu2ehKmow0zeMEmOUNGqfur4RQTipJIRZO XkhQ== X-Received: by 10.194.161.166 with SMTP id xt6mr5350349wjb.28.1368633817974; Wed, 15 May 2013 09:03:37 -0700 (PDT) Received: from dfleuriot.paris.hi-media-techno.com ([83.167.62.196]) by mx.google.com with ESMTPSA id ay7sm21529838wib.9.2013.05.15.09.03.35 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 15 May 2013 09:03:36 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Fleuriot Damien In-Reply-To: Date: Wed, 15 May 2013 18:03:35 +0200 Message-Id: <1CB05302-7A5C-4245-A997-65A70C3DAFB8@my.gd> References: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> To: Outback Dingo X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQk5EPEVUt0rN4/bS4vnw9PrBxZ+/ZQOuP4uMRFBcOy9Ep69+BiRJdDbyCX8Lll7t1NmXIx0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 16:03:44 -0000 On May 15, 2013, at 5:56 PM, Outback Dingo = wrote: >=20 >=20 >=20 > On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien wrote: >=20 > On May 15, 2013, at 2:46 PM, Outback Dingo = wrote: >=20 > > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop = wrote: > > > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo = > >> wrote: > >> > >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop = >>>> **wrote: > >>> > >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < > >>>> outbackdingo@gmail.com> > >>>> wrote: > >>>> > >>>> So it seems a new deployment we just built with zfsonroot mirror = and a > >>>> > >>>>> 48TB > >>>>> master pool is already out of File Descriptors??? > >>>>> > >>>>> pool: master > >>>>> state: ONLINE > >>>>> status: The pool is formatted using a legacy on-disk format. = The pool > >>>>> can > >>>>> still be used, but some features are unavailable. > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is = done, the > >>>>> pool will no longer be accessible on software that does = not > >>>>> support > >>>>> feature > >>>>> flags. > >>>>> scan: none requested > >>>>> config: > >>>>> > >>>>> NAME STATE READ WRITE CKSUM > >>>>> master ONLINE 0 0 0 > >>>>> raidz3-0 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 > >>>>> raidz3-1 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 > >>>>> raidz3-2 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 > >>>>> raidz3-3 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 > >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 > >>>>> logs > >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 > >>>>> cache > >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 > >>>>> > >>>>> errors: No known data errors > >>>>> pool: tank > >>>>> state: ONLINE > >>>>> status: The pool is formatted using a legacy on-disk format. = The pool > >>>>> can > >>>>> still be used, but some features are unavailable. > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is = done, the > >>>>> pool will no longer be accessible on software that does = not > >>>>> support > >>>>> feature > >>>>> flags. > >>>>> scan: none requested > >>>>> config: > >>>>> > >>>>> NAME STATE READ WRITE CKSUM > >>>>> tank ONLINE 0 0 0 > >>>>> mirror-0 ONLINE 0 0 0 > >>>>> da34p3 ONLINE 0 0 0 > >>>>> da35p3 ONLINE 0 0 0 > >>>>> > >>>>> errors: No known data errors > >>>>> > >>>>> while compiling a kernel, buildworld worked and installed ok > >>>>> > >>>>> lex -t > >>>>> = /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> aicasm/aicasm_scan.l > >>>>> > >>>>> aicasm_scan.c > >>>>>> > >>>>>> lex -t -Pmm > >>>>> = /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> > >>>>> aicasm/aicasm_macro_scan.l > >>>>> > >>>>> aicasm_macro_scan.c > >>>>>> > >>>>>> rm -f .depend_aicasm > >>>>> mkdep -f .depend_aicasm -a -I. > >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** > >>>>> aic7xxx/aicasm > >>>>> -std=3Dgnu99 > >>>>> = /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> aicasm/aicasm.c > >>>>> = /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > >>>>> > >>>>> aicasm/aicasm_symbol.c > >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c = aicasm_macro_scan.c > >>>>> Out of file descriptors > >>>>> *** [.depend_aicasm] Error code 2 > >>>>> > >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. > >>>>> > >>>>> *** [buildkernel] Error code 1 > >>>>> > >>>>> Stop in /usr/src. > >>>>> *** [buildkernel] Error code 1 > >>>>> > >>>>> > >>>> > >>>> Do you know what a file descriptor is? > >>>> > >>>> There is something about it here: = http://www.freebsd.org/doc/en/**** > >>>> books/handbook/configtuning-****kernel-limits.html >>>> = freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html>= (search > >>>> for 'file descriptor') > >>>> Or more info in here: = https://www.google.nl/search?**** > >>>> q=3Dfreebsd+Out+of+file+****descriptors >>>> = google.nl/search?q=3Dfreebsd+**Out+of+file+descriptors > >>>>> > >>>> > >>>> > >>>> It mainly says that you have more files open than your system is > >>>> configured to allow. This is often produced by a bug in a program = which > >>>> does not close some files properly. > >>>> > >>>> > >>>> Yes I do.... but this is a brand new zfsonroot with barely any = data on > >>> it > >>> and > >>> > >>> sysctl -a | grep kern.openfiles > >>> kern.openfiles: 68 > >>> root@:/master/builder # sysctl -a | grep kern.maxfiles > >>> kern.maxfiles: 24600 > >>> kern.maxfilesperproc: 11095 > >>> > >>> the openfiles and maxfiles seems to be plenty, but im out of = descriptors, > >>> i > >>> used to see this back in the 4.x days when you could format > >>> ufs with larger inodes, but zfs ??? really? > >>> > >> > >> > >> Is this kern.openfiles when idle or during your buildkernel? > >> Did you check 'ulimit -a'? > >> You say installworld worked ok while you are just now doing = buildkernel. > >> Are your kernel and world out of sync? > >> > >> > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make = buildworld, > > then a make installworld, when i went to do a make buildkernel > > thats the error i got, so id say, yes it is "out of sync" >=20 >=20 > Wow wow wow, hold on a sec here. >=20 > You're installing your new world before the kernel ? >=20 >=20 > I've had no trouble upgrading several boxes from 8-STABLE to = 10.0-CURRENT, following the regular procedure as described at: > http://www.freebsd.org/doc/handbook/makeworld.html >=20 >=20 > Note that the correct order is: > - buildworld (or kernel-toolchain if you only want to test the kernel = and not do the whole world) > - buildkernel > - installkernel > - reboot > - mergemaster -p (preferably in single user, I've always run it = multiuser w/o problems) > - installworld > - mergemaster > - reboot >=20 > You don't want to run the new world on the old kernel. >=20 > yeah i know better, seems when i went to install the old kernel the = build had failed, i can get through it from here >=20 > it was odd because the CURRENT kernel failed to build with a yyparse = error, until after i had completed a buildworld >=20 > the other node i did is fine.......... =20 >=20 Perhaps their sources weren't at the same revision level ? Anyway, you wanna reinstall a 9.x world from the ISO now, then update = sources and redo things in order ;) As a sidenote, if your root is on UFS, you can use nextboot to = failsafe-try your new kernel.= From owner-freebsd-fs@FreeBSD.ORG Wed May 15 16:05:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 18E79F86 for ; Wed, 15 May 2013 16:05:59 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oa0-f42.google.com (mail-oa0-f42.google.com [209.85.219.42]) by mx1.freebsd.org (Postfix) with ESMTP id D90D1993 for ; Wed, 15 May 2013 16:05:58 +0000 (UTC) Received: by mail-oa0-f42.google.com with SMTP id i10so2442287oag.29 for ; Wed, 15 May 2013 09:05:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=K93EynZg2cPMBj8OqTz7BCG//MlLno5Yhzpxm02bYNc=; b=BE4IeGLeOUoZvzIROda+Mk5WEbqTktF6mY8nwyhe08flmJooFUTcrCcqh7dvUsua7R OGIxrS+9JyqU677MNuxFGY/DG1EWual0jmqINBNqrGifHcZQZzGZsPlkv1YX4KO4vPFd v08J3jdzTkhgroydVIce2uQHdZnMqnNiVwNzRpF/UgPLAm8xPyCMURNNQ7jKBgoCxtex D5MROUcTdMkfHzz4uD+Rd6HQtZAHbCwCUZk7i1WEZsVDXWO+LLPkv0Oyyv4zq7PVj29/ hPRdRBzRjK3xY4hJyXe3HakAEWhCjm5ctjnwtgZiZXRphEBZswPtXTdmoGiAn7OzvbS8 mPRg== MIME-Version: 1.0 X-Received: by 10.60.93.67 with SMTP id cs3mr19147191oeb.88.1368633957957; Wed, 15 May 2013 09:05:57 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 09:05:57 -0700 (PDT) In-Reply-To: <1CB05302-7A5C-4245-A997-65A70C3DAFB8@my.gd> References: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> <1CB05302-7A5C-4245-A997-65A70C3DAFB8@my.gd> Date: Wed, 15 May 2013 12:05:57 -0400 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: Fleuriot Damien Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 16:05:59 -0000 On Wed, May 15, 2013 at 12:03 PM, Fleuriot Damien wrote: > > On May 15, 2013, at 5:56 PM, Outback Dingo wrote: > > > > > On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien wrote: > >> >> On May 15, 2013, at 2:46 PM, Outback Dingo >> wrote: >> >> > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop < >> ronald-freebsd8@klop.yi.org>wrote: >> > >> >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo < >> outbackdingo@gmail.com> >> >> wrote: >> >> >> >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop < >> ronald-freebsd8@klop.yi.org >> >>>> **wrote: >> >>> >> >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < >> >>>> outbackdingo@gmail.com> >> >>>> wrote: >> >>>> >> >>>> So it seems a new deployment we just built with zfsonroot mirror and >> a >> >>>> >> >>>>> 48TB >> >>>>> master pool is already out of File Descriptors??? >> >>>>> >> >>>>> pool: master >> >>>>> state: ONLINE >> >>>>> status: The pool is formatted using a legacy on-disk format. The >> pool >> >>>>> can >> >>>>> still be used, but some features are unavailable. >> >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, >> the >> >>>>> pool will no longer be accessible on software that does not >> >>>>> support >> >>>>> feature >> >>>>> flags. >> >>>>> scan: none requested >> >>>>> config: >> >>>>> >> >>>>> NAME STATE READ WRITE CKSUM >> >>>>> master ONLINE 0 0 0 >> >>>>> raidz3-0 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 >> >>>>> raidz3-1 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 >> >>>>> raidz3-2 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 >> >>>>> raidz3-3 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 >> >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 >> >>>>> logs >> >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 >> >>>>> cache >> >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 >> >>>>> >> >>>>> errors: No known data errors >> >>>>> pool: tank >> >>>>> state: ONLINE >> >>>>> status: The pool is formatted using a legacy on-disk format. The >> pool >> >>>>> can >> >>>>> still be used, but some features are unavailable. >> >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, >> the >> >>>>> pool will no longer be accessible on software that does not >> >>>>> support >> >>>>> feature >> >>>>> flags. >> >>>>> scan: none requested >> >>>>> config: >> >>>>> >> >>>>> NAME STATE READ WRITE CKSUM >> >>>>> tank ONLINE 0 0 0 >> >>>>> mirror-0 ONLINE 0 0 0 >> >>>>> da34p3 ONLINE 0 0 0 >> >>>>> da35p3 ONLINE 0 0 0 >> >>>>> >> >>>>> errors: No known data errors >> >>>>> >> >>>>> while compiling a kernel, buildworld worked and installed ok >> >>>>> >> >>>>> lex -t >> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >> >>>>> aicasm/aicasm_scan.l >> >>>>> >> >>>>> aicasm_scan.c >> >>>>>> >> >>>>>> lex -t -Pmm >> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >> >>>>> >> >>>>> aicasm/aicasm_macro_scan.l >> >>>>> >> >>>>> aicasm_macro_scan.c >> >>>>>> >> >>>>>> rm -f .depend_aicasm >> >>>>> mkdep -f .depend_aicasm -a -I. >> >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** >> >>>>> aic7xxx/aicasm >> >>>>> -std=gnu99 >> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >> >>>>> aicasm/aicasm.c >> >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** >> >>>>> >> >>>>> aicasm/aicasm_symbol.c >> >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c >> >>>>> Out of file descriptors >> >>>>> *** [.depend_aicasm] Error code 2 >> >>>>> >> >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. >> >>>>> >> >>>>> *** [buildkernel] Error code 1 >> >>>>> >> >>>>> Stop in /usr/src. >> >>>>> *** [buildkernel] Error code 1 >> >>>>> >> >>>>> >> >>>> >> >>>> Do you know what a file descriptor is? >> >>>> >> >>>> There is something about it here: http://www.freebsd.org/doc/en/**** >> >> >>>> books/handbook/configtuning-****kernel-limits.html> >>>> >> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html< >> http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html >> >>(search >> >>>> for 'file descriptor') >> >>>> Or more info in here: https://www.google.nl/search?****< >> https://www.google.nl/search?**> >> >>>> q=freebsd+Out+of+file+****descriptors> >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors< >> https://www.google.nl/search?q=freebsd+Out+of+file+descriptors> >> >>>>> >> >>>> >> >>>> >> >>>> It mainly says that you have more files open than your system is >> >>>> configured to allow. This is often produced by a bug in a program >> which >> >>>> does not close some files properly. >> >>>> >> >>>> >> >>>> Yes I do.... but this is a brand new zfsonroot with barely any data >> on >> >>> it >> >>> and >> >>> >> >>> sysctl -a | grep kern.openfiles >> >>> kern.openfiles: 68 >> >>> root@:/master/builder # sysctl -a | grep kern.maxfiles >> >>> kern.maxfiles: 24600 >> >>> kern.maxfilesperproc: 11095 >> >>> >> >>> the openfiles and maxfiles seems to be plenty, but im out of >> descriptors, >> >>> i >> >>> used to see this back in the 4.x days when you could format >> >>> ufs with larger inodes, but zfs ??? really? >> >>> >> >> >> >> >> >> Is this kern.openfiles when idle or during your buildkernel? >> >> Did you check 'ulimit -a'? >> >> You say installworld worked ok while you are just now doing >> buildkernel. >> >> Are your kernel and world out of sync? >> >> >> >> >> > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make >> buildworld, >> > then a make installworld, when i went to do a make buildkernel >> > thats the error i got, so id say, yes it is "out of sync" >> >> >> Wow wow wow, hold on a sec here. >> >> You're installing your new world before the kernel ? >> >> >> I've had no trouble upgrading several boxes from 8-STABLE to >> 10.0-CURRENT, following the regular procedure as described at: >> http://www.freebsd.org/doc/handbook/makeworld.html >> >> >> Note that the correct order is: >> - buildworld (or kernel-toolchain if you only want to test the kernel and >> not do the whole world) >> - buildkernel >> - installkernel >> - reboot >> - mergemaster -p (preferably in single user, I've always run it multiuser >> w/o problems) >> - installworld >> - mergemaster >> - reboot >> >> You don't want to run the new world on the old kernel. >> > > yeah i know better, seems when i went to install the old kernel the build > had failed, i can get through it from here > > it was odd because the CURRENT kernel failed to build with a yyparse > error, until after i had completed a buildworld > > the other node i did is fine.......... > > > > Perhaps their sources weren't at the same revision level ? > > Anyway, you wanna reinstall a 9.x world from the ISO now, then update > sources and redo things in order ;) > > As a sidenote, if your root is on UFS, you can use nextboot to > failsafe-try your new kernel. > actually there is no need to revert to 9x but thanks, ill just remount the pool read write and do a make installkernel now and reboot From owner-freebsd-fs@FreeBSD.ORG Wed May 15 16:22:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 50738204 for ; Wed, 15 May 2013 16:22:13 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 1B8D1A3A for ; Wed, 15 May 2013 16:22:13 +0000 (UTC) Received: by mail-ob0-f172.google.com with SMTP id tb18so2230889obb.3 for ; Wed, 15 May 2013 09:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=4Gt3+yCl8fodhGi0sJdC1uOf6uT3wjSvS+8TUHOt30U=; b=xUANeyYiyZGoXBmxKDGtpz1e05Rr5ZYcShfBB5PSJEw0WN4P7wrYreYmEMjaJJWAqF 5Kcts3V0gqkv8/TVeVbMjuSD8Y3bM+Q9IF2BzrXLwE/7DLpIgF0vIVu7/bwisqRDEtRu n449A2De3fukvoinWQxfvGi1oi3ObEDXq6wczIKrsniYQKo/aI3fSHVKGStSLK/IgrjF 9yNGh0xCMd4ZwAl8tqP9rYDeYjO+ibyWcjGw3YSGvxWgmNR0gs6QqFQgyWAh0rmJ4nI9 IGyEINj5AnT1wXGt48Q5ro9kcRj8QoExlGS3P/5GNvvo0DWdyxBiZIJOzDqbuuJro38h RTLw== MIME-Version: 1.0 X-Received: by 10.182.138.4 with SMTP id qm4mr17972968obb.101.1368634932697; Wed, 15 May 2013 09:22:12 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Wed, 15 May 2013 09:22:12 -0700 (PDT) In-Reply-To: <201305151617.r4FGHRGo031542@higson.cam.lispworks.com> References: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> <201305151617.r4FGHRGo031542@higson.cam.lispworks.com> Date: Wed, 15 May 2013 12:22:12 -0400 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Outback Dingo To: Martin Simmons Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 16:22:13 -0000 On Wed, May 15, 2013 at 12:17 PM, Martin Simmons wrote: > >>>>> On Wed, 15 May 2013 11:56:50 -0400, Outback Dingo said: > > > > On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien wrote: > > > > > > > > On May 15, 2013, at 2:46 PM, Outback Dingo > wrote: > > > > > > > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop < > > > ronald-freebsd8@klop.yi.org>wrote: > > > > > > > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo < > > > outbackdingo@gmail.com> > > > >> wrote: > > > >> > > > >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop < > > > ronald-freebsd8@klop.yi.org > > > >>>> **wrote: > > > >>> > > > >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < > > > >>>> outbackdingo@gmail.com> > > > >>>> wrote: > > > >>>> > > > >>>> So it seems a new deployment we just built with zfsonroot mirror > and a > > > >>>> > > > >>>>> 48TB > > > >>>>> master pool is already out of File Descriptors??? > > > >>>>> > > > >>>>> pool: master > > > >>>>> state: ONLINE > > > >>>>> status: The pool is formatted using a legacy on-disk format. The > > > pool > > > >>>>> can > > > >>>>> still be used, but some features are unavailable. > > > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is > done, > > > the > > > >>>>> pool will no longer be accessible on software that does > not > > > >>>>> support > > > >>>>> feature > > > >>>>> flags. > > > >>>>> scan: none requested > > > >>>>> config: > > > >>>>> > > > >>>>> NAME STATE READ WRITE CKSUM > > > >>>>> master ONLINE 0 0 0 > > > >>>>> raidz3-0 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 > > > >>>>> raidz3-1 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 > > > >>>>> raidz3-2 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 > > > >>>>> raidz3-3 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 > > > >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 > > > >>>>> logs > > > >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 > > > >>>>> cache > > > >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 > > > >>>>> > > > >>>>> errors: No known data errors > > > >>>>> pool: tank > > > >>>>> state: ONLINE > > > >>>>> status: The pool is formatted using a legacy on-disk format. The > > > pool > > > >>>>> can > > > >>>>> still be used, but some features are unavailable. > > > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is > done, > > > the > > > >>>>> pool will no longer be accessible on software that does > not > > > >>>>> support > > > >>>>> feature > > > >>>>> flags. > > > >>>>> scan: none requested > > > >>>>> config: > > > >>>>> > > > >>>>> NAME STATE READ WRITE CKSUM > > > >>>>> tank ONLINE 0 0 0 > > > >>>>> mirror-0 ONLINE 0 0 0 > > > >>>>> da34p3 ONLINE 0 0 0 > > > >>>>> da35p3 ONLINE 0 0 0 > > > >>>>> > > > >>>>> errors: No known data errors > > > >>>>> > > > >>>>> while compiling a kernel, buildworld worked and installed ok > > > >>>>> > > > >>>>> lex -t > > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > > >>>>> aicasm/aicasm_scan.l > > > >>>>> > > > >>>>> aicasm_scan.c > > > >>>>>> > > > >>>>>> lex -t -Pmm > > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > > >>>>> > > > >>>>> aicasm/aicasm_macro_scan.l > > > >>>>> > > > >>>>> aicasm_macro_scan.c > > > >>>>>> > > > >>>>>> rm -f .depend_aicasm > > > >>>>> mkdep -f .depend_aicasm -a -I. > > > >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** > > > >>>>> aic7xxx/aicasm > > > >>>>> -std=gnu99 > > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > > >>>>> aicasm/aicasm.c > > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > > >>>>> > > > >>>>> aicasm/aicasm_symbol.c > > > >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c > aicasm_macro_scan.c > > > >>>>> Out of file descriptors > > > >>>>> *** [.depend_aicasm] Error code 2 > > > >>>>> > > > >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. > > > >>>>> > > > >>>>> *** [buildkernel] Error code 1 > > > >>>>> > > > >>>>> Stop in /usr/src. > > > >>>>> *** [buildkernel] Error code 1 > > > >>>>> > > > >>>>> > > > >>>> > > > >>>> Do you know what a file descriptor is? > > > >>>> > > > >>>> There is something about it here: > http://www.freebsd.org/doc/en/****< > > > http://www.freebsd.org/doc/en/**> > > > >>>> books/handbook/configtuning-****kernel-limits.html > > >>>> > freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html > > > < > > > > http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html > > > >>(search > > > >>>> for 'file descriptor') > > > >>>> Or more info in here: https://www.google.nl/search?****< > > > https://www.google.nl/search?**> > > > >>>> q=freebsd+Out+of+file+****descriptors > > >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors< > > > https://www.google.nl/search?q=freebsd+Out+of+file+descriptors> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> It mainly says that you have more files open than your system is > > > >>>> configured to allow. This is often produced by a bug in a program > > > which > > > >>>> does not close some files properly. > > > >>>> > > > >>>> > > > >>>> Yes I do.... but this is a brand new zfsonroot with barely any > data on > > > >>> it > > > >>> and > > > >>> > > > >>> sysctl -a | grep kern.openfiles > > > >>> kern.openfiles: 68 > > > >>> root@:/master/builder # sysctl -a | grep kern.maxfiles > > > >>> kern.maxfiles: 24600 > > > >>> kern.maxfilesperproc: 11095 > > > >>> > > > >>> the openfiles and maxfiles seems to be plenty, but im out of > > > descriptors, > > > >>> i > > > >>> used to see this back in the 4.x days when you could format > > > >>> ufs with larger inodes, but zfs ??? really? > > > >>> > > > >> > > > >> > > > >> Is this kern.openfiles when idle or during your buildkernel? > > > >> Did you check 'ulimit -a'? > > > >> You say installworld worked ok while you are just now doing > buildkernel. > > > >> Are your kernel and world out of sync? > > > >> > > > >> > > > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make > > > buildworld, > > > > then a make installworld, when i went to do a make buildkernel > > > > thats the error i got, so id say, yes it is "out of sync" > > > > > > > > > Wow wow wow, hold on a sec here. > > > > > > You're installing your new world before the kernel ? > > > > > > > > > I've had no trouble upgrading several boxes from 8-STABLE to > 10.0-CURRENT, > > > following the regular procedure as described at: > > > http://www.freebsd.org/doc/handbook/makeworld.html > > > > > > > > > Note that the correct order is: > > > - buildworld (or kernel-toolchain if you only want to test the kernel > and > > > not do the whole world) > > > - buildkernel > > > - installkernel > > > - reboot > > > - mergemaster -p (preferably in single user, I've always run it > multiuser > > > w/o problems) > > > - installworld > > > - mergemaster > > > - reboot > > > > > > You don't want to run the new world on the old kernel. > > > > > > > yeah i know better, seems when i went to install the old kernel the build > > had failed, i can get through it from here > > > > it was odd because the CURRENT kernel failed to build with a yyparse > error, > > until after i had completed a buildworld > > Taking a zfs snapshot of the system file systems before installing > anything is > a good idea. > yeah in hindsight, it is actually > > __Martin > From owner-freebsd-fs@FreeBSD.ORG Wed May 15 16:28:33 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3D8B94A4 for ; Wed, 15 May 2013 16:28:33 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-lb0-f175.google.com (mail-lb0-f175.google.com [209.85.217.175]) by mx1.freebsd.org (Postfix) with ESMTP id BA11CA97 for ; Wed, 15 May 2013 16:28:32 +0000 (UTC) Received: by mail-lb0-f175.google.com with SMTP id v10so2071532lbd.20 for ; Wed, 15 May 2013 09:28:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=Si9su8LQjM7DH8ArpIC1eladYv39sWR6zf+CPoZeqi8=; b=CPD/j5QNeoaU7oD6bAUhOmSqxLMa1ucKBMFignNtVspjkxl4Kp06sxyb6cJ13jFheB hpgDPCjaFrfflXbmDa+6RDRE+BeANASUkBKDKonDHGwEvBSFjyVPa6QsP1Fs78Q45g/6 5MKsBP6LUlNbVdqVkbD8+d/j72GdBPlAK6z1k4nw6Ys7ksCOUUM4asdtPL3zbXA0cKzG RFJcnRVghZn9grM3k3/IsMMK1nc5gaBYWw1G32Rts6f4xgKn2VOgxFgdO9kqZlKtqteV ugl9HCniGPT8a7m5fJqV2qCiobKZ1IFHDTQQZcwjVP/GILlR5Cyaol/+t9F2mbGvfA+6 cmRg== MIME-Version: 1.0 X-Received: by 10.112.63.169 with SMTP id h9mr3232340lbs.135.1368635311223; Wed, 15 May 2013 09:28:31 -0700 (PDT) Received: by 10.112.202.137 with HTTP; Wed, 15 May 2013 09:28:31 -0700 (PDT) In-Reply-To: References: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> Date: Wed, 15 May 2013 17:28:31 +0100 Message-ID: Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? From: Tom Evans To: Outback Dingo Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 16:28:33 -0000 On Wed, May 15, 2013 at 4:56 PM, Outback Dingo wro= te: > > yeah i know better, seems when i went to install the old kernel the build > had failed, i can get through it from here > > it was odd because the CURRENT kernel failed to build with a yyparse erro= r, > until after i had completed a buildworld > > the other node i did is fine.......... The only supported way of getting to CURRENT is from STABLE, therefore if you start on 9.0-RELEASE, you should first update to 9-STABLE, and from there to CURRENT. Trying to go directly may work, it may not work=E2=80=A6 Cheers Tom From owner-freebsd-fs@FreeBSD.ORG Wed May 15 16:28:37 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 24F6B4A7 for ; Wed, 15 May 2013 16:28:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [193.34.186.230]) by mx1.freebsd.org (Postfix) with ESMTP id 9F06EA9A for ; Wed, 15 May 2013 16:28:36 +0000 (UTC) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.5/8.14.5) with ESMTP id r4FGHRJM092927; Wed, 15 May 2013 17:17:27 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id r4FGHRDH032168; Wed, 15 May 2013 17:17:27 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id r4FGHRGo031542; Wed, 15 May 2013 17:17:27 +0100 Date: Wed, 15 May 2013 17:17:27 +0100 Message-Id: <201305151617.r4FGHRGo031542@higson.cam.lispworks.com> From: Martin Simmons To: Outback Dingo In-reply-to: (message from Outback Dingo on Wed, 15 May 2013 11:56:50 -0400) Subject: Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ?? References: <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 May 2013 16:28:37 -0000 >>>>> On Wed, 15 May 2013 11:56:50 -0400, Outback Dingo said: > > On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien wrote: > > > > > On May 15, 2013, at 2:46 PM, Outback Dingo wrote: > > > > > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop < > > ronald-freebsd8@klop.yi.org>wrote: > > > > > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo < > > outbackdingo@gmail.com> > > >> wrote: > > >> > > >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop < > > ronald-freebsd8@klop.yi.org > > >>>> **wrote: > > >>> > > >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo < > > >>>> outbackdingo@gmail.com> > > >>>> wrote: > > >>>> > > >>>> So it seems a new deployment we just built with zfsonroot mirror and a > > >>>> > > >>>>> 48TB > > >>>>> master pool is already out of File Descriptors??? > > >>>>> > > >>>>> pool: master > > >>>>> state: ONLINE > > >>>>> status: The pool is formatted using a legacy on-disk format. The > > pool > > >>>>> can > > >>>>> still be used, but some features are unavailable. > > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, > > the > > >>>>> pool will no longer be accessible on software that does not > > >>>>> support > > >>>>> feature > > >>>>> flags. > > >>>>> scan: none requested > > >>>>> config: > > >>>>> > > >>>>> NAME STATE READ WRITE CKSUM > > >>>>> master ONLINE 0 0 0 > > >>>>> raidz3-0 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN01 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN03 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN04 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN05 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN07 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN08 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN09 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN10 ONLINE 0 0 0 > > >>>>> raidz3-1 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN11 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN12 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN13 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN14 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN15 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN16 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN17 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN18 ONLINE 0 0 0 > > >>>>> raidz3-2 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN19 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN20 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN21 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN22 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN23 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN24 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN26 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN27 ONLINE 0 0 0 > > >>>>> raidz3-3 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN29 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN30 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN32 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN33 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN35 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN36 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN37 ONLINE 0 0 0 > > >>>>> multipath/SATA_LUN38 ONLINE 0 0 0 > > >>>>> logs > > >>>>> multipath/SATA_LUN06 ONLINE 0 0 0 > > >>>>> cache > > >>>>> multipath/SATA_LUN02 ONLINE 0 0 0 > > >>>>> > > >>>>> errors: No known data errors > > >>>>> pool: tank > > >>>>> state: ONLINE > > >>>>> status: The pool is formatted using a legacy on-disk format. The > > pool > > >>>>> can > > >>>>> still be used, but some features are unavailable. > > >>>>> action: Upgrade the pool using 'zpool upgrade'. Once this is done, > > the > > >>>>> pool will no longer be accessible on software that does not > > >>>>> support > > >>>>> feature > > >>>>> flags. > > >>>>> scan: none requested > > >>>>> config: > > >>>>> > > >>>>> NAME STATE READ WRITE CKSUM > > >>>>> tank ONLINE 0 0 0 > > >>>>> mirror-0 ONLINE 0 0 0 > > >>>>> da34p3 ONLINE 0 0 0 > > >>>>> da35p3 ONLINE 0 0 0 > > >>>>> > > >>>>> errors: No known data errors > > >>>>> > > >>>>> while compiling a kernel, buildworld worked and installed ok > > >>>>> > > >>>>> lex -t > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> aicasm/aicasm_scan.l > > >>>>> > > >>>>> aicasm_scan.c > > >>>>>> > > >>>>>> lex -t -Pmm > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> > > >>>>> aicasm/aicasm_macro_scan.l > > >>>>> > > >>>>> aicasm_macro_scan.c > > >>>>>> > > >>>>>> rm -f .depend_aicasm > > >>>>> mkdep -f .depend_aicasm -a -I. > > >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/**** > > >>>>> aic7xxx/aicasm > > >>>>> -std=gnu99 > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> aicasm/aicasm.c > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/**** > > >>>>> > > >>>>> aicasm/aicasm_symbol.c > > >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c aicasm_macro_scan.c > > >>>>> Out of file descriptors > > >>>>> *** [.depend_aicasm] Error code 2 > > >>>>> > > >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm. > > >>>>> > > >>>>> *** [buildkernel] Error code 1 > > >>>>> > > >>>>> Stop in /usr/src. > > >>>>> *** [buildkernel] Error code 1 > > >>>>> > > >>>>> > > >>>> > > >>>> Do you know what a file descriptor is? > > >>>> > > >>>> There is something about it here: http://www.freebsd.org/doc/en/****< > > http://www.freebsd.org/doc/en/**> > > >>>> books/handbook/configtuning-****kernel-limits.html > >>>> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html > > < > > http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html > > >>(search > > >>>> for 'file descriptor') > > >>>> Or more info in here: https://www.google.nl/search?****< > > https://www.google.nl/search?**> > > >>>> q=freebsd+Out+of+file+****descriptors > >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors< > > https://www.google.nl/search?q=freebsd+Out+of+file+descriptors> > > >>>>> > > >>>> > > >>>> > > >>>> It mainly says that you have more files open than your system is > > >>>> configured to allow. This is often produced by a bug in a program > > which > > >>>> does not close some files properly. > > >>>> > > >>>> > > >>>> Yes I do.... but this is a brand new zfsonroot with barely any data on > > >>> it > > >>> and > > >>> > > >>> sysctl -a | grep kern.openfiles > > >>> kern.openfiles: 68 > > >>> root@:/master/builder # sysctl -a | grep kern.maxfiles > > >>> kern.maxfiles: 24600 > > >>> kern.maxfilesperproc: 11095 > > >>> > > >>> the openfiles and maxfiles seems to be plenty, but im out of > > descriptors, > > >>> i > > >>> used to see this back in the 4.x days when you could format > > >>> ufs with larger inodes, but zfs ??? really? > > >>> > > >> > > >> > > >> Is this kern.openfiles when idle or during your buildkernel? > > >> Did you check 'ulimit -a'? > > >> You say installworld worked ok while you are just now doing buildkernel. > > >> Are your kernel and world out of sync? > > >> > > >> > > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make > > buildworld, > > > then a make installworld, when i went to do a make buildkernel > > > thats the error i got, so id say, yes it is "out of sync" > > > > > > Wow wow wow, hold on a sec here. > > > > You're installing your new world before the kernel ? > > > > > > I've had no trouble upgrading several boxes from 8-STABLE to 10.0-CURRENT, > > following the regular procedure as described at: > > http://www.freebsd.org/doc/handbook/makeworld.html > > > > > > Note that the correct order is: > > - buildworld (or kernel-toolchain if you only want to test the kernel and > > not do the whole world) > > - buildkernel > > - installkernel > > - reboot > > - mergemaster -p (preferably in single user, I've always run it multiuser > > w/o problems) > > - installworld > > - mergemaster > > - reboot > > > > You don't want to run the new world on the old kernel. > > > > yeah i know better, seems when i went to install the old kernel the build > had failed, i can get through it from here > > it was odd because the CURRENT kernel failed to build with a yyparse error, > until after i had completed a buildworld Taking a zfs snapshot of the system file systems before installing anything is a good idea. __Martin From owner-freebsd-fs@FreeBSD.ORG Thu May 16 10:08:38 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2A8DD5A7 for ; Thu, 16 May 2013 10:08:38 +0000 (UTC) (envelope-from henner.heck@web.de) Received: from mout.web.de (mout.web.de [212.227.17.12]) by mx1.freebsd.org (Postfix) with ESMTP id CCF2CBD7 for ; Thu, 16 May 2013 10:08:37 +0000 (UTC) Received: from sender ([93.132.149.124]) by smtp.web.de (mrweb101) with ESMTPSA (Nemesis) id 0Lo0V2-1U1OME17EW-00gYTY; Thu, 16 May 2013 12:03:18 +0200 Message-ID: <5194AEDE.9040504@web.de> Date: Thu, 16 May 2013 12:03:10 +0200 From: Henner Heck User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Ronald Klop Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> <5192AD94.5020707@web.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:2BN8GfU4ZLLHd+P9BFdkbrKahik5bPIxKipf/c+OFIs 3z4mtyAq/zMO2FJ8lS0wgbP7DmuoOfYCT5dvE+FPvmUNA3IXlt dSrkzQdNp1W9zHqKrkuBUMfYoTbfAUBVB8owiRsgGReBfA/vuC p4JiuXPcfMN2PyeNcuCFcZGk1kVe9nPoaARY8GqDSLA3AYCkxo GwVhg3a41a1K9MxYK0NMg== Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 May 2013 10:08:38 -0000 Am 15.05.2013 10:42, schrieb Ronald Klop: > On Tue, 14 May 2013 23:33:08 +0200, Henner Heck > wrote: > >> Am 14.05.2013 21:22, schrieb Steven Hartland: >>> ----- Original Message ----- From: "Henner Heck" >>>> Hello all, >>>> >>>> i set up a PC with FreeBSD 9.1-Release and 2 zfs pools. >>>> One is a mirror from which FreeBSD is booting >>>> (tough enough without getting "error 2"), >>>> the other one a raidz2 for data. >>>> The disks for the raidz2 are encrypted with geli and attached >>>> manually. >>>> >>>> I noticed that a "zpool status" or a "zfs list" before attaching >>>> the encrypted disks waits for about one minute before showing output. >>>> When they finally do, the output is as expected, the raidz2 pool is >>>> shown as >>>> UNAVAIL and its datasets are not listed. >>>> When all the disks are attached with geli, the outputs are given >>>> immediately. >>>> >>>> On boot there are 2 delays. >>>> One of about 1 minute after the output >>>> "Setting hostid: 0x........" >>>> and one of 2 minutes after >>>> "Mounting local file systems:.". >>>> Both these outputs don't show up in dmesg, which ends with >>>> "Trying to mount root from zfs:zroot/ROOT []..." shortly before. >>>> >>>> I suspect that the boot delays too are because of the encrypted pool. >>>> >>>> A different machine running FreeBSD 8.3-RELEASE has a delay of only >>>> about 3 seconds on "zpool status" with an encrypted pool >>>> and also the boot shows no annoying anomalies. >>>> >>>> Any idea how to get rid of these really long delays? >>> >>> Try the attached patch see if that helps? >>> >>> Regards >>> Steve >>> >>> ================================================ >>> This e.mail is private and confidential between Multiplay (UK) Ltd. >>> and the person or entity to whom it is addressed. In the event of >>> misdirection, the recipient is prohibited from using, copying, >>> printing or otherwise disseminating it or any information contained in >>> it. >>> In the event of misdirection, illegible or incomplete transmission >>> please telephone +44 845 868 1337 >>> or return the E.mail to postmaster@multiplay.co.uk. >> >> >> >> Hello Steven, >> >> i tried to apply the patch, but patch didn't return and gave no output. >> I checked my the zfs.c and it doesn't seem to fit this patch. >> Please see attached file. >> >> Regards, >> Henner Heck > > Sounds likes my daily wtf moment. Patch likes the patch-file on stdin. > That is why it waits forever and does not return. Ctrl-z will help that. > > Use something like this: > patch < foo.diff > > Regards, > Ronald. > Always happy to give someone a special moment. Thank you, now patch did something. It confirmed that it is the wrong file version for the patch, since the replacements failed and the only changes are additions. Output: ---------------------------------------------------------------------------------------------------------- Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |--- sys/boot/zfs/zfs.c.orig 2011-10-20 18:15:29.966685430 +0000 |+++ sys/boot/zfs/zfs.c 2011-10-20 18:18:22.291033636 +0000 -------------------------- Patching file sys/boot/zfs/zfs.c using Plan A... Hunk #1 succeeded at 47 (offset 2 lines). Hunk #2 failed at 423. 1 out of 2 hunks failed--saving rejects to sys/boot/zfs/zfs.c.rej Hmm... Ignoring the trailing garbage. done ---------------------------------------------------------------------------------------------------------- I'm still hoping that someone can give the reason for the increased delays. Since nothing actually goes wrong but just takes ages, they might have been introduced on purpose, not considering the possibility of a deliberately unavailable pool.(?) Regards, Henner Heck From owner-freebsd-fs@FreeBSD.ORG Thu May 16 17:43:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3953842F for ; Thu, 16 May 2013 17:43:23 +0000 (UTC) (envelope-from prvs=18480ee867=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id D01717BA for ; Thu, 16 May 2013 17:43:22 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003839245.msg for ; Thu, 16 May 2013 18:43:13 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 16 May 2013 18:43:13 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18480ee867=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <3A4D99D3DC69494880790E7971F218B4@multiplay.co.uk> From: "Steven Hartland" To: "Henner Heck" , "Ronald Klop" References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> <5192AD94.5020707@web.de> <5194AEDE.9040504@web.de> Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) Date: Thu, 16 May 2013 18:43:11 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 May 2013 17:43:23 -0000 ----- Original Message ----- From: "Henner Heck" > Always happy to give someone a special moment. > Thank you, now patch did something. > It confirmed that it is the wrong file version for the patch, > since the replacements failed and the only changes are additions. > Output: > > ---------------------------------------------------------------------------------------------------------- > Hmm... Looks like a unified diff to me... > The text leading up to this was: > -------------------------- > |--- sys/boot/zfs/zfs.c.orig 2011-10-20 18:15:29.966685430 +0000 > |+++ sys/boot/zfs/zfs.c 2011-10-20 18:18:22.291033636 +0000 > -------------------------- > Patching file sys/boot/zfs/zfs.c using Plan A... > Hunk #1 succeeded at 47 (offset 2 lines). > Hunk #2 failed at 423. > 1 out of 2 hunks failed--saving rejects to sys/boot/zfs/zfs.c.rej > Hmm... Ignoring the trailing garbage. > done > ---------------------------------------------------------------------------------------------------------- > > I'm still hoping that someone can give the reason for the increased delays. > Since nothing actually goes wrong but just takes ages, > they might have been introduced on purpose, > not considering the possibility of a deliberately unavailable pool.(?) As I mentioned in my previous response this patch is no longer needed in 9.x. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Thu May 16 19:00:54 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AAFE1D54 for ; Thu, 16 May 2013 19:00:54 +0000 (UTC) (envelope-from henner.heck@web.de) Received: from mout.web.de (mout.web.de [212.227.15.4]) by mx1.freebsd.org (Postfix) with ESMTP id 29CBAB0F for ; Thu, 16 May 2013 19:00:53 +0000 (UTC) Received: from sender ([95.112.177.32]) by smtp.web.de (mrweb001) with ESMTPSA (Nemesis) id 0LwZA7-1URPO60t0Z-017lc7; Thu, 16 May 2013 21:00:46 +0200 Message-ID: <51952CDF.8080705@web.de> Date: Thu, 16 May 2013 21:00:47 +0200 From: Henner Heck User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Steven Hartland Subject: Re: Long delays during boot and zpool/zfs commands on 9.1-RELEASE (possibly because of unavailable pool?) References: <968416157.282645.1368232366317.JavaMail.root@erie.cs.uoguelph.ca> <518EFE05.8010100@hub.org> <518F4130.6080201@hub.org> <518F4307.3060908@hub.org> <519285C9.8000306@web.de> <5192AD94.5020707@web.de> <5194AEDE.9040504@web.de> <3A4D99D3DC69494880790E7971F218B4@multiplay.co.uk> In-Reply-To: <3A4D99D3DC69494880790E7971F218B4@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:sgGy1HuH8ipfPjelL8i93WilkF5ePGwBPcTclvAgLUp CnEcUiFwP/xOg1GvgtSdmVfSwSV3Tm9A3JiI/RFwnL37C4HWAA t3PVqbpi3anhW2ZBTsh2XlLuRKx4UyyJTml3XSklC8mVHzgkBx 2EcTNa2uobf65MaHznZUy4yVSJ8DfapBkYIfqQN1YRVxItysH2 /eWmeqEJVXWDktdIrIBAA== Cc: freebsd-fs@freebsd.org, Ronald Klop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 May 2013 19:00:54 -0000 Am 16.05.2013 19:43, schrieb Steven Hartland: > > ----- Original Message ----- From: "Henner Heck" > >> Always happy to give someone a special moment. >> Thank you, now patch did something. >> It confirmed that it is the wrong file version for the patch, >> since the replacements failed and the only changes are additions. >> Output: >> >> ---------------------------------------------------------------------------------------------------------- >> >> Hmm... Looks like a unified diff to me... >> The text leading up to this was: >> -------------------------- >> |--- sys/boot/zfs/zfs.c.orig 2011-10-20 18:15:29.966685430 +0000 >> |+++ sys/boot/zfs/zfs.c 2011-10-20 18:18:22.291033636 +0000 >> -------------------------- >> Patching file sys/boot/zfs/zfs.c using Plan A... >> Hunk #1 succeeded at 47 (offset 2 lines). >> Hunk #2 failed at 423. >> 1 out of 2 hunks failed--saving rejects to sys/boot/zfs/zfs.c.rej >> Hmm... Ignoring the trailing garbage. >> done >> ---------------------------------------------------------------------------------------------------------- >> >> >> I'm still hoping that someone can give the reason for the increased >> delays. >> Since nothing actually goes wrong but just takes ages, >> they might have been introduced on purpose, >> not considering the possibility of a deliberately unavailable pool.(?) > > As I mentioned in my previous response this patch is no longer needed in > 9.x. > > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > > I got that, and thanks again for trying to help. I just tried it out to show Roland that i learned to use patch properly and his hint was not in vain. ;) Unfortunately the initial issue remains unsolved. Regards, Henner Heck From owner-freebsd-fs@FreeBSD.ORG Thu May 16 20:20:01 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B2B152F7 for ; Thu, 16 May 2013 20:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 8AB1AEBE for ; Thu, 16 May 2013 20:20:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r4GKK1uY095173 for ; Thu, 16 May 2013 20:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r4GKK1hN095171; Thu, 16 May 2013 20:20:01 GMT (envelope-from gnats) Date: Thu, 16 May 2013 20:20:01 GMT Message-Id: <201305162020.r4GKK1hN095171@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: System Integrated Subject: Re: kern/153680: [xfs] 8.1 failing to mount XFS partitions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: System Integrated List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 May 2013 20:20:01 -0000 The following reply was made to PR kern/153680; it has been noted by GNATS. From: System Integrated To: bug-followup@FreeBSD.org, jan0sch@gmx.net Cc: Subject: Re: kern/153680: [xfs] 8.1 failing to mount XFS partitions Date: Thu, 16 May 2013 22:03:37 +0200 Hey there, if got the exact same error on FreeBSD 9.0-RELEASE and 9.1-RELEASE. The bug prevents me to migrate from linux to freebsd on some systems. Needs to be fixed, i guess. Kind regards From owner-freebsd-fs@FreeBSD.ORG Thu May 16 22:35:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 872205D8 for ; Thu, 16 May 2013 22:35:25 +0000 (UTC) (envelope-from prvs=18480ee867=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 1561D18F for ; Thu, 16 May 2013 22:35:24 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003842988.msg for ; Thu, 16 May 2013 23:35:23 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 16 May 2013 23:35:23 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18480ee867=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Ajit Jain" References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> <3E9CA9334E6F433A8F135ACD5C237340@multiplay.co.uk> Subject: Re: seeing data corruption with zfs trim functionality Date: Thu, 16 May 2013 23:35:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 May 2013 22:35:25 -0000 ----- Original Message ----- From: "Ajit Jain" To: "Steven Hartland" Cc: "freebsd-fs" Sent: Wednesday, May 15, 2013 4:18 PM Subject: Re: seeing data corruption with zfs trim functionality > Hi Steven, > > Thanks for the update. > It is surprising that there is no/less disk activity as the command is > correct. May be we need to enable > sync=always on the zfs dataset. > > I will try again the once I update the cam code base. > > regards, > ajit > > > On Wed, May 15, 2013 at 7:37 PM, Steven Hartland wrote: > >> Unless you have the latest CAM patches, which is in current, you wont be >> doing TRIM on SATA disk connected to an LSI controller. >> >> I've just tested using the following cmd under 8.3 with MFC'ed changes from >> current, using ATA_TRIM, UNMAP & WS16 and have had no issues on a machine >> with Intel SSD and LSI controller. >> ./iotest -t 20 -s 536870912 -W 100 -T 500 /test/iotest/ >> >> I did however notice that your test is hardly doing any disk access apart >> from when its "Initializing test file....", instead it seems to be CPU >> bound, >> so not sure if there's a problem with the iotest code or with my command >> line args? >> >> Given this my current thinking is either: >> 1. There's a problem with your patches 2. There's a bug in the FW of the >> Seagate disk >> 3. There's a problem with the UNMAP code which is being trigged by your >> disk >> only. >> >> I think #3 is quite unlikely. >> >> If you could install a recent version of current and test with that it >> should rule out #1, leaving #2. After initially seeing not issues, our overnight monitoring started moaning big time on the test box. So we checked and there was zpool corruption as well as a missing boot loader and a corrupt GPT, so I believe we have reproduced your issue. After recovering the machine I created 3 pools on 3 different disks each running a different delete_method. We then re-ran the tests which resulted in the pool running with delete_method WS16 being so broken it had suspended IO. A reboot resulted in it once again reporting no partition table via gpart. A third test run again produced a corrupt pool for WS16. I've conducted a preliminary review of the CAM WS16 code path along with SBC-3 spec which didn't identify any obvious issues. Given we're both using LSI 2008 based controllers it could be FW issue specific to WS16 but that's just speculation atm, so I'll continue to investigate. If you could re-test you end without using WS16 to see if you can reproduce the problem with either UNMAP or ATA_TRIM that would be a very useful data point. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Fri May 17 11:49:03 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B74449B1; Fri, 17 May 2013 11:49:03 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id 4DEFE78A; Fri, 17 May 2013 11:49:03 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UdJ9Z-0003W0-D7; Fri, 17 May 2013 13:48:54 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UdJ9X-0007oF-T7; Fri, 17 May 2013 13:48:51 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: "Boris Astardzhiev" , "Mateusz Guzik" , "Ronald Klop" Subject: Re: NANDFS eats itself up References: <20121128104218.GA17871@dft-labs.eu> <20121128151336.GC17871@dft-labs.eu> <20121129110719.GA26212@dft-labs.eu> <20130103235451.GA31491@dft-labs.eu> Date: Fri, 17 May 2013 13:48:49 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: + X-Spam-Score: 1.1 X-Spam-Status: No, score=1.1 required=5.0 tests=BAYES_40, URI_HEX autolearn=disabled version=3.3.1 X-Scan-Signature: 729ef2e9e2cd27dd49f9ca04774c95e6 Cc: freebsd-fs@freebsd.org, gjb@semihalf.com, Grzegorz Bernacki , stanislav_galabov@smartcom.bg X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2013 11:49:03 -0000 On Thu, 07 Feb 2013 10:41:11 +0100, Ronald Klop wrote: > On Fri, 04 Jan 2013 00:54:51 +0100, Mateusz Guzik > wrote: > >> On Thu, Nov 29, 2012 at 01:13:55PM +0200, Boris Astardzhiev wrote: >>> > On Wed, Nov 28, 2012 at 05:36:11PM +0200, Boris Astardzhiev wrote: >>> > > If I do that I might be unable to set back the verbose level to 0 >>> since >>> > the >>> > > output is very very NOISY. This means that I will have to start >>> again >>> > from >>> > > the beginning if I need to reproduce it. Nevertheless here you go. >>> Check >>> > > the attachment (OUTPUTNAND.txt.bz2). >>> > > >>> > >>> > I think I reproduced your problem. Possibly I'll have time to work on >>> > this next week, but no promises. >>> > >> >> So, I think I understand what is causing the problem here and will most >> likely have time to work on it this month. >> > > Hi, > > Not to push you or something like that, but did you find the time? > I'm curious to running my SHEEVAPLUG using the NAND memory. > If there is something to test or help in another way, I'm willing to do > that. > > Ronald. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > Hi, This full thread is available here. http://freebsd.1045724.n5.nabble.com/NANDFS-eats-itself-up-td5764878.html I follow the commits on FreeBSD, but did not see a fix for this. Maybe I missed it. I'm wondering if there is something news about this issue. I understand everybody is very busy, but am curious about it. Regards, Ronald. From owner-freebsd-fs@FreeBSD.ORG Fri May 17 21:56:17 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 444E14AA; Fri, 17 May 2013 21:56:17 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 1E0F8DBD; Fri, 17 May 2013 21:56:17 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r4HLuGjK020635; Fri, 17 May 2013 21:56:17 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r4HLuGqQ020634; Fri, 17 May 2013 21:56:16 GMT (envelope-from linimon) Date: Fri, 17 May 2013 21:56:16 GMT Message-Id: <201305172156.r4HLuGqQ020634@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/178713: [nfs] [patch] Correct WebNFS support in NFS server and client from sys/fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2013 21:56:17 -0000 Old Synopsis: Correct WebNFS support in NFS server and client from sys/fs New Synopsis: [nfs] [patch] Correct WebNFS support in NFS server and client from sys/fs Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Fri May 17 21:56:04 UTC 2013 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=178713 From owner-freebsd-fs@FreeBSD.ORG Fri May 17 22:46:04 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5668A307 for ; Fri, 17 May 2013 22:46:04 +0000 (UTC) (envelope-from prvs=18493e2b7e=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id D683522C for ; Fri, 17 May 2013 22:46:03 +0000 (UTC) Received: from r2d2 ([46.65.172.4]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50003858128.msg for ; Fri, 17 May 2013 23:45:57 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 17 May 2013 23:45:57 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 46.65.172.4 X-Return-Path: prvs=18493e2b7e=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <93D0677B373A452BAF58C8EA6823783D@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Ajit Jain" References: <60316751643743738AB83DABC6A5934B@multiplay.co.uk> <20130429105143.GA1492@icarus.home.lan> <3AD1AB31003D49B2BF2EA7DD411B38A2@multiplay.co.uk> <9681E07546D348168052D4FC5365B4CD@multiplay.co.uk> <3E9CA9334E6F433A8F135ACD5C237340@multiplay.co.uk> Subject: Re: seeing data corruption with zfs trim functionality Date: Fri, 17 May 2013 23:45:57 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 May 2013 22:46:04 -0000 ----- Original Message ----- From: "Steven Hartland" > > After initially seeing not issues, our overnight monitoring started moaning > big time on the test box. So we checked and there was zpool corruption as well > as a missing boot loader and a corrupt GPT, so I believe we have reproduced > your issue. > > After recovering the machine I created 3 pools on 3 different disks each > running a different delete_method. > > We then re-ran the tests which resulted in the pool running with delete_method > WS16 being so broken it had suspended IO. A reboot resulted in it once again > reporting no partition table via gpart. > > A third test run again produced a corrupt pool for WS16. > > I've conducted a preliminary review of the CAM WS16 code path along with SBC-3 > spec which didn't identify any obvious issues. > > Given we're both using LSI 2008 based controllers it could be FW issue specific > to WS16 but that's just speculation atm, so I'll continue to investigate. > > If you could re-test you end without using WS16 to see if you can reproduce the > problem with either UNMAP or ATA_TRIM that would be a very useful data point. After much playing I narrow down a test case of one delete which was causing disc corruption for us (deleted the partition table instead of data in the middle of the disk). The conclusion is LSI 2008 HBA with FW below P13 will eat the data on your SATA disks if you use WS16 due to the following bug:- SCGCQ00230159 (DFCT) - Write same command to a SATA drive that doesn't support SCT write same may write wrong region. After updating here to P16, which we would generally be running, but test box was new and hadnt updated yet the corruption issue is no longer reproducable. So Ajit please check your FW version, I'm hoping to here your on something below P13, P12 possibly? If so then this is your issue, to fix simply update to P16 and the problem should be gone. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.