From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 00:03:58 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF90710656B0; Sun, 11 Oct 2009 00:03:58 +0000 (UTC) (envelope-from sten@blinkenlights.nl) Received: from mx0.blinkenlights.nl (mx0.blinkenlights.nl [IPv6:2001:7b8:666:8000::25]) by mx1.freebsd.org (Postfix) with ESMTP id 5FDE78FC29; Sun, 11 Oct 2009 00:03:58 +0000 (UTC) Received: from bastard.snore.nl (bastard.snore.nl [IPv6:2001:7b8:666:ffff:0:42ff:fe00:4]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx0.blinkenlights.nl (Postfix) with ESMTPS id C813ECF381; Sun, 11 Oct 2009 02:03:57 +0200 (CEST) Received: by bastard.snore.nl (Postfix, from userid 1000) id 1A5B828A044; Sun, 11 Oct 2009 02:03:56 +0200 (CEST) X-Original-To: sten@blinkenlights.nl Delivered-To: sten@blinkenlights.nl Received: from mx0.blinkenlights.nl (mx0.blinkenlights.nl [IPv6:2a01:1b0:201:1::25]) by zaphod.blinkenlights.nl (Postfix) with ESMTP id A950F4AC053 for ; Mon, 11 Feb 2008 18:12:42 +0100 (CET) Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by mx0.blinkenlights.nl (Postfix) with ESMTP id 38E0B73009 for ; Mon, 11 Feb 2008 18:12:42 +0100 (CET) Received: from hub.freebsd.org (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 14DD16151C; Mon, 11 Feb 2008 17:11:23 +0000 (UTC) (envelope-from owner-freebsd-stable@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 608D816A4C8; Mon, 11 Feb 2008 17:11:23 +0000 (UTC) (envelope-from owner-freebsd-stable@freebsd.org) Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5B6216A418; Mon, 11 Feb 2008 17:11:07 +0000 (UTC) (envelope-from gavin.atkinson@ury.york.ac.uk) Received: from mail-gw1.york.ac.uk (mail-gw1.york.ac.uk [144.32.128.246]) by mx1.freebsd.org (Postfix) with ESMTP id 61D6113C4D9; Mon, 11 Feb 2008 17:11:06 +0000 (UTC) (envelope-from gavin.atkinson@ury.york.ac.uk) Received: from mail-gw7.york.ac.uk (mail-gw7.york.ac.uk [144.32.129.30]) by mail-gw1.york.ac.uk (8.13.6/8.13.6) with ESMTP id m1BGdEX8006034; Mon, 11 Feb 2008 16:39:14 GMT Received: from buffy-128.york.ac.uk ([144.32.128.160] helo=buffy.york.ac.uk) by mail-gw7.york.ac.uk with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1JObgY-0003B9-Jr; Mon, 11 Feb 2008 16:39:14 +0000 Received: from buffy.york.ac.uk (localhost [127.0.0.1]) by buffy.york.ac.uk (8.14.2/8.14.2) with ESMTP id m1BGdEfU028425; Mon, 11 Feb 2008 16:39:14 GMT (envelope-from gavin.atkinson@ury.york.ac.uk) Received: (from ga9@localhost) by buffy.york.ac.uk (8.14.2/8.14.2/Submit) id m1BGdDes028424; Mon, 11 Feb 2008 16:39:13 GMT (envelope-from gavin.atkinson@ury.york.ac.uk) X-Authentication-Warning: buffy.york.ac.uk: ga9 set sender to gavin.atkinson@ury.york.ac.uk using -f From: Gavin Atkinson To: Joe Peterson In-Reply-To: <47ACF0AE.3040802@skyrush.com> References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> <47ACF0AE.3040802@skyrush.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit Message-Id: <1202747953.27277.7.camel@buffy.york.ac.uk> Mime-Version: 1.0 X-Mailer: Evolution 2.10.1 FreeBSD GNOME Team Port X-York-MailScanner: Found to be clean X-York-MailScanner-From: gavin.atkinson@ury.york.ac.uk X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Sender: owner-freebsd-stable@freebsd.org Errors-To: owner-freebsd-stable@freebsd.org Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Sun, 11 Oct 2009 00:03:59 -0000 X-Original-Date: Mon, 11 Feb 2008 16:39:13 +0000 X-List-Received-Date: Sun, 11 Oct 2009 00:03:59 -0000 On Fri, 2008-02-08 at 17:15 -0700, Joe Peterson wrote: > Chris Dillon wrote: > > That is a chunk of a Mozilla Mork-format database. Perhaps the > > Firefox URL history or address book from Thunderbird. > > Interesting (thanks to all who recognized Mork). I do use Firefox and > Thunderbird, so it's feasible, but how the heck would a piece of one of > those files find its way into 1/2 of a ZFS block in one of my mp3 files? > I wonder if it could have been done on write when the file was copied > to the ZFS pool (maybe some write-caching issue?), but I thought ZFS > would have verified the block after write. It seems unlikely that it > would get changed later - I never rewrote that file after the original > copy... Are the datestamps (Thu Jan 24 23:20:58 2008) found within the corrupt block before or after the datestamp of the file it was found within? i.e. was the corrupt block on the disk before or after the mp3 was written there? You could possibly confirm this by grepping for that datestamp in the files in your home directory, and with the aid of http://developer.mozilla.org/en/docs/Mork_Structure#Rows, try to establish exactly what the datestamp means (ie was it the time you visited a URL, etc). Gavin _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 00:03:59 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF60B10656A9; Sun, 11 Oct 2009 00:03:59 +0000 (UTC) (envelope-from sten@blinkenlights.nl) Received: from mx1.blinkenlights.nl (mx1.blinkenlights.nl [IPv6:2a02:898:17:8000::25]) by mx1.freebsd.org (Postfix) with ESMTP id 4AAC48FC2E; Sun, 11 Oct 2009 00:03:59 +0000 (UTC) Received: from bastard.snore.nl (bastard.snore.nl [IPv6:2001:7b8:666:ffff:0:42ff:fe00:4]) by mx1.blinkenlights.nl (Postfix) with ESMTP id BEE311142A; Sun, 11 Oct 2009 02:03:58 +0200 (CEST) Received: by bastard.snore.nl (Postfix, from userid 1000) id A80C628A023; Sun, 11 Oct 2009 02:03:58 +0200 (CEST) X-Original-To: sten@blinkenlights.nl Delivered-To: sten@blinkenlights.nl Received: from mx0.blinkenlights.nl (mx0.blinkenlights.nl [IPv6:2a01:1b0:201:1::25]) by zaphod.blinkenlights.nl (Postfix) with ESMTP id E86388A049 for ; Tue, 25 Nov 2008 16:07:46 +0100 (CET) Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by mx0.blinkenlights.nl (Postfix) with ESMTP id 7C0E173009 for ; Tue, 25 Nov 2008 16:07:46 +0100 (CET) Received: from hub.freebsd.org (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 2BDD317ABF1; Tue, 25 Nov 2008 15:05:30 +0000 (UTC) (envelope-from owner-freebsd-stable@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 05F911065693; Tue, 25 Nov 2008 15:05:29 +0000 (UTC) (envelope-from owner-freebsd-stable@freebsd.org) Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B1111065670 for ; Tue, 25 Nov 2008 15:05:06 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from mail-gx0-f12.google.com (mail-gx0-f12.google.com [209.85.217.12]) by mx1.freebsd.org (Postfix) with ESMTP id CDF7F8FC1E for ; Tue, 25 Nov 2008 15:05:05 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by gxk5 with SMTP id 5so321517gxk.19 for ; Tue, 25 Nov 2008 07:05:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:reply-to :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=RRS1FXdk6KlOebjY9FSAS8fA0kl0y8YlEbfplRT45M0=; b=Jdoquum9NTRB8Y0QWe5f8nVgLb8KfQcg0Nt6AoMr9acbSrXoHEhV7gYmJrymn+7uwW Sw9ypTcOTH1ZR8JZ6e55Ii8jZ36ytrGvHULsbe9aLoYkI+CEMwR5t5stql0dU8B492Q6 Npv9G8Qdzt/ok6XTJQxB8TWm2JSo1p3K/UARI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:reply-to:to:subject:cc:in-reply-to :mime-version:content-type:content-transfer-encoding :content-disposition:references; b=lsYAyPaKcTCUAIp/Wj2A4i6S4P+wIouowlWywrqAgdKywOQ/brvAQrjy/fq6EoQdDz 0+iFAW/TN5PGzkneHzVWCxSrEm5dbniD/tsv/135HgwW9S1uLJiV5BRVrwvK4riSd8tU HkCGK/J+VNsfPKDfHslQVJytvu0VuUBTeJb5E= Received: by 10.150.49.15 with SMTP id w15mr8563645ybw.152.1227625038227; Tue, 25 Nov 2008 06:57:18 -0800 (PST) Received: by 10.150.218.5 with HTTP; Tue, 25 Nov 2008 06:57:18 -0800 (PST) Message-ID: <8cb6106e0811250657q6fdf08b0x1e94f35fd0a7ed4f@mail.gmail.com> From: "Josh Carroll" To: "Kostik Belousov" In-Reply-To: <20081125142827.GI2042@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0811241129o642dcf28re4ae177c8ccbaa25@mail.gmail.com> <20081125140601.GH2042@deviant.kiev.zoral.com.ua> <8cb6106e0811250617q5fffb41exe20dfb8314fc4a9d@mail.gmail.com> <20081125142827.GI2042@deviant.kiev.zoral.com.ua> X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Sender: owner-freebsd-stable@freebsd.org Errors-To: owner-freebsd-stable@freebsd.org Cc: freebsd-fs@freebsd.org, FreeBSD Stable Subject: Re: ext2 inode size patch - RE: PR kern/124621 X-BeenThere: freebsd-fs@freebsd.org Reply-To: josh.carroll@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Sun, 11 Oct 2009 00:03:59 -0000 X-Original-Date: Tue, 25 Nov 2008 09:57:18 -0500 X-List-Received-Date: Sun, 11 Oct 2009 00:03:59 -0000 > I do not suggest testing. I suggest understand what inode metadata is stored > in the added 128 bytes and evaluate whether this information can be ignored > without dangerous consequences for filesystem consistency or user data. > Well, to be clear I didn't just double the size of the inode table. It is dynamically determined based on the data structure. I'm not a file system expert (to call me a novice would probably be stretching it), so I'm hoping someone more versed can chime in. All the code does is query the data structure (specifically, the s_inode_size field of the structure) and use that value instead of blindly assuming an inode size of 128. I don't think it's a matter of what is done with the extra bits, since it's just querying the size of an already created filesystem. Thanks, Josh _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 04:39:13 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF1FC106568F for ; Sun, 11 Oct 2009 04:39:13 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 3B4248FC12 for ; Sun, 11 Oct 2009 04:39:12 +0000 (UTC) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id DF64719E023 for ; Sun, 11 Oct 2009 06:39:10 +0200 (CEST) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 7E8E519E019 for ; Sun, 11 Oct 2009 06:39:08 +0200 (CEST) Message-ID: <4AD1616C.8060504@quip.cz> Date: Sun, 11 Oct 2009 06:39:08 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: ZFS vs. df -h completely different size of filesystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 04:39:13 -0000 Hi, I am using ZFS on this machine for one year without any problems. The main purpose is to run jails on filesystems with ZFS quotas. The biggest jail has 360G (322GB used) running Lighttpd. I must change IP for this jail, so I stop the jail, asign new IP alias on interface, change IP in config and start the jail again. But now the jail reports only 24G quota and by another accident I deleted stored files by rsync. The strange thing is this: root@cage ~/# df -h /vol0/jail/kurt2 Filesystem Size Used Avail Capacity Mounted on tank/vol0/jail/kurt2 24G 19G 4.9G 80% /vol0/jail/kurt2 root@cage ~/# zfs list | grep tank/vol0/jail/kurt2 tank/vol0/jail/kurt2 355G 4.89G 19.2G /vol0/jail/kurt2 tank/vol0/jail/kurt2@manual-2009-08-18_11-04 7.29G - 299G - tank/vol0/jail/kurt2@manual-2009-09-04_10-24 2.24G - 308G - tank/vol0/jail/kurt2@manual-2009-09-28_11-41 21.4G - 317G - root@cage ~/# zfs get -r quota tank/vol0/jail/kurt2 NAME PROPERTY VALUE SOURCE tank/vol0/jail/kurt2 quota 360G local As you can see, df reports 24G size and zfs list shows 355G. What is wrong? I tired to write some date on to this filesystem and it stoped after 24G were filled. Here are more details: root@cage ~/# zfs get all tank/vol0/jail/kurt2 NAME PROPERTY VALUE SOURCE tank/vol0/jail/kurt2 type filesystem - tank/vol0/jail/kurt2 creation Sat Feb 28 0:52 2009 - tank/vol0/jail/kurt2 used 355G - tank/vol0/jail/kurt2 available 4.89G - tank/vol0/jail/kurt2 referenced 19.2G - tank/vol0/jail/kurt2 compressratio 1.02x - tank/vol0/jail/kurt2 mounted yes - tank/vol0/jail/kurt2 quota 360G local tank/vol0/jail/kurt2 reservation none default tank/vol0/jail/kurt2 recordsize 128K default tank/vol0/jail/kurt2 mountpoint /vol0/jail/kurt2 inherited from tank/vol0 tank/vol0/jail/kurt2 sharenfs off default tank/vol0/jail/kurt2 checksum on default tank/vol0/jail/kurt2 compression gzip inherited from tank/vol0 tank/vol0/jail/kurt2 atime off inherited from tank tank/vol0/jail/kurt2 devices on default tank/vol0/jail/kurt2 exec on inherited from tank/vol0/jail tank/vol0/jail/kurt2 setuid on inherited from tank/vol0/jail tank/vol0/jail/kurt2 readonly off default tank/vol0/jail/kurt2 jailed off default tank/vol0/jail/kurt2 snapdir hidden default tank/vol0/jail/kurt2 aclmode groupmask default tank/vol0/jail/kurt2 aclinherit secure default tank/vol0/jail/kurt2 canmount on default tank/vol0/jail/kurt2 shareiscsi off default tank/vol0/jail/kurt2 xattr off temporary tank/vol0/jail/kurt2 copies 1 default root@cage ~/# df -ht zfs Filesystem Size Used Avail Capacity Mounted on tank 76G 16G 59G 22% /tank tank/vol0 59G 128K 59G 0% /vol0 tank/vol0/jail 60G 605M 59G 1% /vol0/jail tank/vol0/jail/china 20G 786M 19G 4% /vol0/jail/china tank/vol0/jail/costa 40G 527M 39G 1% /vol0/jail/costa tank/vol0/jail/kurt2 24G 19G 4.9G 80% /vol0/jail/kurt2 tank/vol0/jail/tester1 20G 874M 19G 4% /vol0/jail/tester1 tank/vol0/jail/tvujweb 25G 494M 24G 2% /vol0/jail/tvujweb tank/vol0/jail/yamaha 19G 1.2G 18G 6% /vol0/jail/yamaha tank/vol0/jail/kurt2_clone 376G 317G 59G 84% /vol0/jail/kurt2_clone root@cage ~/# zfs list -t filesystem NAME USED AVAIL REFER MOUNTPOINT tank 378G 59.2G 16.4G /tank tank/vol0 361G 59.2G 19K /vol0 tank/vol0/jail 361G 59.2G 605M /vol0/jail tank/vol0/jail/china 996M 19.0G 786M /vol0/jail/china tank/vol0/jail/costa 938M 39.1G 527M /vol0/jail/costa tank/vol0/jail/kurt2 355G 4.89G 19.2G /vol0/jail/kurt2 tank/vol0/jail/kurt2_clone 0 59.2G 317G /vol0/jail/kurt2_clone tank/vol0/jail/tester1 1.23G 18.8G 874M /vol0/jail/tester1 tank/vol0/jail/tvujweb 691M 24.3G 494M /vol0/jail/tvujweb tank/vol0/jail/yamaha 1.72G 18.3G 1.18G /vol0/jail/yamaha kurt2_clone was created as clone of snapshot kurt2@manual-2009-09-28_11-41 after this crazyness Did somebody seen this error behavior? I can get all the data from master server by rsync, but 320GB of data, it will take many hours. Is there any way to move the data from kurt2_clone to kurt2? (after I find a reason, why kurt2 is so small now) I cannot use promote: root@cage ~/# zfs promote tank/vol0/jail/kurt2_clone cannot promote 'tank/vol0/jail/kurt2_clone': out of space system: FreeBSD 7.2-RELEASE #1: Tue May 5 16:17:50 CEST 2009 root@cage ~/# zdb tank version=6 name='tank' state=0 txg=4 pool_guid=15882109565948440736 hostid=2083894993 hostname='cage.xxxxx.yyy' vdev_tree type='root' id=0 guid=15882109565948440736 children[0] type='mirror' id=0 guid=15485397551239445092 metaslab_array=14 metaslab_shift=32 ashift=9 asize=478648991744 children[0] type='disk' id=0 guid=7324230677133648509 path='/dev/ad4s2' devid='ad:GEB531RE0B213Fs1' whole_disk=0 children[1] type='disk' id=1 guid=16876684646990874563 path='/dev/ad6s2' devid='ad:GEB531RE0BJ72Fs1' whole_disk=0 Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 07:04:20 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 335741065695; Sun, 11 Oct 2009 07:04:20 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CB7428FC25; Sun, 11 Oct 2009 07:04:19 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9B74JAV011813; Sun, 11 Oct 2009 07:04:19 GMT (envelope-from delphij@freefall.freebsd.org) Received: (from delphij@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9B74JdG011809; Sun, 11 Oct 2009 07:04:19 GMT (envelope-from delphij) Date: Sun, 11 Oct 2009 07:04:19 GMT Message-Id: <200910110704.n9B74JdG011809@freefall.freebsd.org> To: delphij@FreeBSD.org, freebsd-fs@FreeBSD.org, delphij@FreeBSD.org From: delphij@FreeBSD.org Cc: Subject: Re: kern/122038: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc7d2fab0 0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 07:04:20 -0000 Synopsis: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc7d2fab0 0 Responsible-Changed-From-To: freebsd-fs->delphij Responsible-Changed-By: delphij Responsible-Changed-When: Sun Oct 11 07:04:00 UTC 2009 Responsible-Changed-Why: Take since I have committed a patch from gk@ http://www.freebsd.org/cgi/query-pr.cgi?pr=122038 From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 15:02:14 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A66F7106568B for ; Sun, 11 Oct 2009 15:02:14 +0000 (UTC) (envelope-from alextzfs@googlemail.com) Received: from mail-fx0-f222.google.com (mail-fx0-f222.google.com [209.85.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id 057C08FC17 for ; Sun, 11 Oct 2009 15:02:13 +0000 (UTC) Received: by fxm22 with SMTP id 22so7614204fxm.36 for ; Sun, 11 Oct 2009 08:02:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:cc:content-type; bh=saoYg6NmPJVBFxE6PWECNo4OA7DWZ3dhyQgha1CDf38=; b=TFHSY8oXhqNyut7cQupQuDLwYcyTFMCwLJdKSkT2DI6/TcNgNJVlXkweK0CLX+0BDN 7/eHA9yx71AmtR16f0KXsNwVicxZ4ZdisqorI8jc5ibduIAK+iAZcBZzTogkXCCsvbv4 3Ou8nGWZ3qv2EZWVTitzoDsy93a1X9OsGUhuM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=lVvgdYwBbwmnf1YBKD1Z7aOgFrb3wIFv/OmmCEmWR/mUh+DT2pmwd/OEhFY0MKeniF IGNW9bjt2R6woKcSMwTT2lkLK1U+A0Euq6X9I3S8PWh0HHTRaIt78Wg3nOfKp3JEj2XU gN4wPPS87P2UI3bcNQUgs6HIAVyiHkT5S/f+Y= MIME-Version: 1.0 Received: by 10.204.23.77 with SMTP id q13mr4184666bkb.14.1255272111747; Sun, 11 Oct 2009 07:41:51 -0700 (PDT) Date: Sun, 11 Oct 2009 15:41:51 +0100 Message-ID: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com> From: Alex Trull To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pjd@freebsd.org Subject: zraid2 loses a single disk and becomes difficult to recover X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 15:02:14 -0000 Hi All, My zraid2 has broken this morning on releng_7 zfs13. System failed this morning and came back without pool - having lost a disk. This is how I found the system: pool: fatman state: FAULTED status: One or more devices could not be used because the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. see: http://www.sun.com/msg/ZFS-8000-5E scrub: none requested config: NAME STATE READ WRITE CKSUM fatman FAULTED 0 0 1 corrupted data raidz2 DEGRADED 0 0 6 da2 FAULTED 0 0 0 corrupted data ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad20 ONLINE 0 0 0 ad22 ONLINE 0 0 0 ad17 ONLINE 0 0 0 da2 ONLINE 0 0 0 ad10 ONLINE 0 0 0 ad16 ONLINE 0 0 0 Initialy it complained that da3 had gone to da2 (da2 had failed and was no longer seen) I replaced the original da2 and bumped what was originaly da3 back up to da3 using the controllers ordering. [root@potjie /dev]# zpool status pool: fatman state: FAULTED status: One or more devices could not be used because the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. see: http://www.sun.com/msg/ZFS-8000-5E scrub: none requested config: NAME STATE READ WRITE CKSUM fatman FAULTED 0 0 1 corrupted data raidz2 ONLINE 0 0 6 da2 UNAVAIL 0 0 0 corrupted data ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 ad20 ONLINE 0 0 0 ad22 ONLINE 0 0 0 ad17 ONLINE 0 0 0 da3 ONLINE 0 0 0 ad10 ONLINE 0 0 0 ad16 ONLINE 0 0 0 Issue looks very similar to this (JMR's issue) : http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html I've tried the methods there without much result. Using JMR's patches/debugs to see what is going on, this is what I got: JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834 JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780 JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834 JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750 JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834 JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263 JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834 JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263 JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834 But JMR's patch still doesn't let me import even with a decremented txg I then had a look around the drives using zdb and some dirty script: [root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo "$1";zdb -l "$1" |grep txg"}' | sh /dev/ad10 txg=46488654 txg=46488654 txg=46488654 txg=46488654 /dev/ad16 txg=46408223 <- old TXGid ? txg=46408223 txg=46408223 txg=46408223 /dev/ad17 txg=46408223 <- old TXGid ? txg=46408223 txg=46408223 txg=46408223 /dev/ad18 (ssd) /dev/ad19 (spare drive, removed from pool some time ago) txg=0 create_txg=0 txg=0 create_txg=0 txg=0 create_txg=0 txg=0 create_txg=0 /dev/ad20 txg=46488654 txg=46488654 txg=46488654 txg=46488654 /dev/ad22 txg=46488654 txg=46488654 txg=46488654 txg=46488654 /dev/ad4 txg=46488654 txg=46488654 txg=46488654 txg=46488654 /dev/ad6 txg=46488654 txg=46488654 txg=46488654 txg=46488654 /dev/da2 < new drive replaced broken da2 /dev/da3 txg=46488654 txg=46488654 txg=46488654 txg=46488654 I did not see any checksums or other issues on ad16 and ad17 previously, and I do check regularly. Any thoughts on what to try next ? Regards, Alex From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 16:27:22 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04863106566B; Sun, 11 Oct 2009 16:27:22 +0000 (UTC) (envelope-from alextzfs@googlemail.com) Received: from mail-bw0-f223.google.com (mail-bw0-f223.google.com [209.85.218.223]) by mx1.freebsd.org (Postfix) with ESMTP id E9BA68FC15; Sun, 11 Oct 2009 16:27:20 +0000 (UTC) Received: by bwz23 with SMTP id 23so1517223bwz.43 for ; Sun, 11 Oct 2009 09:27:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=54iHu6qyajsa8eQt5hhkD4EVYCy6appOunjA2TyocNU=; b=r2iB5su0l0mYNT+OXTd5PxUFr6r+6LKngts9TXRwu3Wv8bEuSnbj8uWQDhFuOtKxMx 81CJkPdp0fhQGTfZIdy+IJsy1ZKfZnCGyzD36Hp9cO7KjO1tXJsYY1+Ol4hq7bOk4nFM SSToQNR0dtX8KTNc+RvYgGE19DK3dO4S96NW0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=xTVer/TeSHHH22O5PMYphE0nRBgcONR/30wXaFSirKVTlbkRgxuAgEjgWBstsldW+R 0VRBSnKHucQ6gS60GuP8tb6AnralZ6ZKNHme6IiTAG5PIQkY8mZxlGcU/hPYHDF85kOL 01MESDufdLHRGDvybCnWc0awXx97EXgTDFFaQ= MIME-Version: 1.0 Received: by 10.204.10.6 with SMTP id n6mr4276078bkn.27.1255278438632; Sun, 11 Oct 2009 09:27:18 -0700 (PDT) In-Reply-To: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com> References: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com> Date: Sun, 11 Oct 2009 17:27:18 +0100 Message-ID: <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com> From: Alex Trull To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pjd@freebsd.org Subject: Re: zraid2 loses a single disk and becomes difficult to recover X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 16:27:22 -0000 Well after trying alot of things (zpool import with or without cache file in place, etc), it randomly managed to mount the pool up, atleast, with errors - : zfs list output: cannot iterate filesystems: I/O error NAME USED AVAIL REFER MOUNTPOINT fatman 1.40T 1.70T 51.2K /fatman fatman/backup 100G 99.5G 95.5G /fatman/backup fatman/jail 422G 1.70T 60.5K /fatman/jail fatman/jail/havnor 198G 51.7G 112G /fatman/jail/havnor fatman/jail/mail 19.4G 30.6G 13.0G /fatman/jail/mail fatman/jail/syndicate 16.6G 103G 10.5G /fatman/jail/syndicate fatman/jail/thirdforces 159G 41.4G 78.1G /fatman/jail/thirdforces fatman/jail/web 24.8G 25.2G 22.3G /fatman/jail/web fatman/stash 913G 1.70T 913G /fatman/stash (end of the dmesg) JMR: vdev_uberblock_load_done ubbest ub_txg=46475461 ub_timestamp=1255231841 JMR: vdev_uberblock_load_done ub_txg=46481476 ub_timestamp=1255234263 JMR: vdev_uberblock_load_done ubbest ub_txg=46481476 ub_timestamp=1255234263 JMR: vdev_uberblock_load_done ubbest ub_txg=46475459 ub_timestamp=1255231780 JMR: vdev_uberblock_load_done ubbest ub_txg=46475458 ub_timestamp=1255231750 JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263 JMR: vdev_uberblock_load_done ubbest ub_txg=46481473 ub_timestamp=1255234263 JMR: vdev_uberblock_load_done ubbest ub_txg=46481472 ub_timestamp=1255234263 Solaris: WARNING: can't open objset for fatman/jail/margaret Solaris: WARNING: can't open objset for fatman/jail/margaret Solaris: WARNING: ZFS replay transaction error 86, dataset fatman/jail/havnor, seq 0x25442, txtype 9 Solaris: WARNING: ZFS replay transaction error 86, dataset fatman/jail/mail, seq 0x1e200, txtype 9 Solaris: WARNING: ZFS replay transaction error 86, dataset fatman/jail/thirdforces, seq 0x732e3, txtype 9 [root@potjie /fatman/jail/mail]# zpool status -v pool: fatman state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver in progress for 0h4m, 0.83% done, 8h21m to go config: NAME STATE READ WRITE CKSUM fatman DEGRADED 0 0 34 raidz2 DEGRADED 0 0 384 replacing DEGRADED 0 0 0 da2/old REMOVED 0 24 0 da2 ONLINE 0 0 0 1.71G resilvered ad4 ONLINE 0 0 0 21.3M resilvered ad6 ONLINE 0 0 0 21.4M resilvered ad20 ONLINE 0 0 0 21.3M resilvered ad22 ONLINE 0 0 0 21.3M resilvered ad17 ONLINE 0 0 0 21.3M resilvered da3 ONLINE 0 0 0 21.3M resilvered ad10 ONLINE 0 0 1 21.4M resilvered ad16 ONLINE 0 0 0 21.2M resilvered cache ad18 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: fatman/jail/margaret:<0x0> fatman/jail/syndicate:<0x0> fatman/jail/mail:<0x0> /fatman/jail/mail/tmp fatman/jail/havnor:<0x0> fatman/jail/thirdforces:<0x0> fatman/backup:<0x0> jail/margaret & backup isn't showing up in zfs list jail/syndicate is showing up but isn't viewable It seems the latest content on the better-looking zfs filesystems are quite recent. Any thoughts about what is going on ? I have snapshots for africa on these zfs filesystems - any suggestions on trying to get them back ? -- Alex 2009/10/11 Alex Trull > Hi All, > > My zraid2 has broken this morning on releng_7 zfs13. > > System failed this morning and came back without pool - having lost a disk. > > This is how I found the system: > > pool: fatman > state: FAULTED > status: One or more devices could not be used because the label is missing > or invalid. There are insufficient replicas for the pool to continue > functioning. > action: Destroy and re-create the pool from a backup source. > see: http://www.sun.com/msg/ZFS-8000-5E > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > fatman FAULTED 0 0 1 corrupted data > raidz2 DEGRADED 0 0 6 > da2 FAULTED 0 0 0 corrupted data > ad4 ONLINE 0 0 0 > ad6 ONLINE 0 0 0 > ad20 ONLINE 0 0 0 > ad22 ONLINE 0 0 0 > ad17 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > ad10 ONLINE 0 0 0 > ad16 ONLINE 0 0 0 > > Initialy it complained that da3 had gone to da2 (da2 had failed and was no > longer seen) > > I replaced the original da2 and bumped what was originaly da3 back up to > da3 using the controllers ordering. > > [root@potjie /dev]# zpool status > pool: fatman > state: FAULTED > status: One or more devices could not be used because the label is missing > or invalid. There are insufficient replicas for the pool to continue > functioning. > action: Destroy and re-create the pool from a backup source. > see: http://www.sun.com/msg/ZFS-8000-5E > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > fatman FAULTED 0 0 1 corrupted data > raidz2 ONLINE 0 0 6 > da2 UNAVAIL 0 0 0 corrupted data > ad4 ONLINE 0 0 0 > ad6 ONLINE 0 0 0 > ad20 ONLINE 0 0 0 > ad22 ONLINE 0 0 0 > ad17 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > ad10 ONLINE 0 0 0 > ad16 ONLINE 0 0 0 > > Issue looks very similar to this (JMR's issue) : > http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html > > I've tried the methods there without much result. > > Using JMR's patches/debugs to see what is going on, this is what I got: > > JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 > ub_timestamp=1255246834 > JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780 > JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 > ub_timestamp=1255246834 > JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750 > JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 > ub_timestamp=1255246834 > JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263 > JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 > ub_timestamp=1255246834 > JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263 > JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 > ub_timestamp=1255246834 > > But JMR's patch still doesn't let me import even with a decremented txg > > I then had a look around the drives using zdb and some dirty script: > > [root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo > "$1";zdb -l "$1" |grep txg"}' | sh > /dev/ad10 > txg=46488654 > txg=46488654 > txg=46488654 > txg=46488654 > /dev/ad16 > txg=46408223 <- old TXGid ? > txg=46408223 > txg=46408223 > txg=46408223 > /dev/ad17 > txg=46408223 <- old TXGid ? > txg=46408223 > txg=46408223 > txg=46408223 > /dev/ad18 (ssd) > /dev/ad19 (spare drive, removed from pool some time ago) > txg=0 > create_txg=0 > txg=0 > create_txg=0 > txg=0 > create_txg=0 > txg=0 > create_txg=0 > /dev/ad20 > txg=46488654 > txg=46488654 > txg=46488654 > txg=46488654 > /dev/ad22 > txg=46488654 > txg=46488654 > txg=46488654 > txg=46488654 > /dev/ad4 > txg=46488654 > txg=46488654 > txg=46488654 > txg=46488654 > /dev/ad6 > txg=46488654 > txg=46488654 > txg=46488654 > txg=46488654 > /dev/da2 < new drive replaced broken da2 > /dev/da3 > txg=46488654 > txg=46488654 > txg=46488654 > txg=46488654 > > I did not see any checksums or other issues on ad16 and ad17 previously, > and I do check regularly. > > Any thoughts on what to try next ? > > Regards, > > Alex > > From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 16:43:50 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9E331065672 for ; Sun, 11 Oct 2009 16:43:50 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 98FDE8FC0A for ; Sun, 11 Oct 2009 16:43:50 +0000 (UTC) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 9750B19E023 for ; Sun, 11 Oct 2009 18:43:48 +0200 (CEST) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 75E3219E019 for ; Sun, 11 Oct 2009 18:43:46 +0200 (CEST) Message-ID: <4AD20B41.3070405@quip.cz> Date: Sun, 11 Oct 2009 18:43:45 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4AD1616C.8060504@quip.cz> In-Reply-To: <4AD1616C.8060504@quip.cz> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: ZFS vs. df -h completely different size of filesystem [solved] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 16:43:50 -0000 Solved after few hours of sleep - main problem was ... me. (working for more than 18 hours is not a good idea) Miroslav Lachman wrote: [...] > The strange thing is this: > > root@cage ~/# df -h /vol0/jail/kurt2 > Filesystem Size Used Avail Capacity Mounted on > tank/vol0/jail/kurt2 24G 19G 4.9G 80% /vol0/jail/kurt2 > > > root@cage ~/# zfs list | grep tank/vol0/jail/kurt2 > tank/vol0/jail/kurt2 355G 4.89G 19.2G > /vol0/jail/kurt2 > tank/vol0/jail/kurt2@manual-2009-08-18_11-04 7.29G - 299G - > tank/vol0/jail/kurt2@manual-2009-09-04_10-24 2.24G - 308G - > tank/vol0/jail/kurt2@manual-2009-09-28_11-41 21.4G - 317G - > > > root@cage ~/# zfs get -r quota tank/vol0/jail/kurt2 > NAME PROPERTY VALUE SOURCE > tank/vol0/jail/kurt2 quota 360G local Directory on source server was moved and rsync started from cron accidentaly deleted all data. But space remains occupied by data of snapshots! Thats why df showed just 24G size and not 360G. Df knows nothing about snapshots. The fix was relatively easy. 1] make a backup of a new files (logs, configs, etc.) 2] destroy clone: zfs destroy tank/vol0/jail/kurt2_clone 3] rollback last snapshot zfs rollback -f tank/vol0/jail/kurt2@manual-2009-09-28_11-41 4] restore backup from step 1) 5] fix rsync settings 6] run rsync as usual ;) Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Sun Oct 11 18:18:44 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4A7E106566B for ; Sun, 11 Oct 2009 18:18:44 +0000 (UTC) (envelope-from josh@multipart-mixed.com) Received: from joshcarter.com (67-207-137-80.slicehost.net [67.207.137.80]) by mx1.freebsd.org (Postfix) with ESMTP id B65BF8FC1E for ; Sun, 11 Oct 2009 18:18:44 +0000 (UTC) Received: from [192.168.1.122] (dsl081-096-235.den1.dsl.speakeasy.net [64.81.96.235]) by joshcarter.com (Postfix) with ESMTPSA id 9D0511C003 for ; Sun, 11 Oct 2009 18:00:39 +0000 (UTC) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Mime-Version: 1.0 (Apple Message framework v1076) From: Josh Carter In-Reply-To: <4AD20B41.3070405@quip.cz> Date: Sun, 11 Oct 2009 12:00:36 -0600 Content-Transfer-Encoding: 7bit Message-Id: References: <4AD1616C.8060504@quip.cz> <4AD20B41.3070405@quip.cz> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1076) Subject: Re: ZFS vs. df -h completely different size of filesystem [solved] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2009 18:18:44 -0000 Miroslav, > But space remains occupied by data of snapshots! Thats why df showed > just 24G size and not 360G. Df knows nothing about snapshots. Some additional factors to keep in mind when looking at ZFS and used/ available disk space: - Snapshots (as you discovered). - Compression: when compression is turned on (as you have), you can't know exactly how much more data will fit into the filesystem because it depends on how well the data compresses. - Sparse files: "ls -h" will show you how large a file says it is; "du -h" and "zfs list" should show how much space is actually used. These will disagree on sparse files. - Space reserved for copy-on-write: "zpool list" and "zfs list" will differ on available space because ZFS reserves some amount of slop space; truly running out of blocks is disastrous in a COW system. In general, learn to trust "zfs list" because the traditional tools don't know the full story. Best regards, Josh From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 07:56:12 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 859B3106566B; Mon, 12 Oct 2009 07:56:12 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 9191F8FC15; Mon, 12 Oct 2009 07:56:11 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 7FE48183654; Mon, 12 Oct 2009 09:56:09 +0200 (CEST) X-CRM114-Version: 20090423-BlameSteveJobs ( TRE 0.7.6 (BSD) ) MF-ACE0E1EA [pR: 21.9356] X-CRM114-CacheID: sfid-20091012_09560_374ADE94 X-CRM114-Status: Good ( pR: 21.9356 ) Message-ID: <4AD2E118.2050202@fsn.hu> Date: Mon, 12 Oct 2009 09:56:08 +0200 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu> <20091008160718.GB2134@garage.freebsd.pl> In-Reply-To: <20091008160718.GB2134@garage.freebsd.pl> X-Stationery: 0.4.10 X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (people.fsn.hu); Mon, 12 Oct 2009 09:56:08 +0200 (CEST) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@FreeBSD.org Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 07:56:12 -0000 Pawel Jakub Dawidek wrote: > On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote: > >> Attila Nagy wrote: >> >>> Hello, >>> >>> Pawel Jakub Dawidek wrote: >>> >>>> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote: >>>> >>>> >>>>> Backing out this change from the 8-STABLE kernel: >>>>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 >>>>> >>>>> >>>>> makes it survive about half and hour of IMAP searching. Of course >>>>> only time will tell whether this helps in the long run, but so far >>>>> 10/10 tries succeeded to kill the machine with this method... >>>>> >>>>> >>>> Could you try this patch: >>>> >>>> http://people.freebsd.org/~pjd/patches/arc.c.4.patch >>>> >>>> >>> It seems (after running for two days) that this fixes my problem. And >>> I see that Kip has came out with a similar version (which I couldn't >>> yet test, but hope that will also do). >>> >> It seems that I was a little bit quick regarding this. >> The machine just stopped with this: >> last pid: 32358; load averages: 0.01, 0.04, 0.12 up 2+06:33:56 >> 14:36:25 >> 114 processes: 1 running, 112 sleeping, 1 zombie >> CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf >> Swap: 4096M Total, 15M Used, 4081M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 24025 root 1 44 0 3932K 992K vmwait 0 6:06 0.00% zpool >> 84190 root 1 44 0 4700K 1592K CPU1 1 4:17 0.00% top >> 99029 root 1 44 0 4132K 1212K nanslp 1 3:53 0.00% gstat >> 26317 root 1 44 0 1528K 352K piperd 1 3:38 0.00% >> readproctitl >> 49143 125 4 45 0 12248K 3788K sigwai 0 2:50 0.00% >> milter-greyl >> 39969 root 1 44 0 1536K 516K vmwait 0 2:50 0.00% supervise >> 40241 root 1 44 0 1536K 516K vmwait 0 2:47 0.00% supervise >> 44633 root 1 44 0 1536K 512K vmwait 0 2:43 0.00% supervise >> 43434 root 1 44 0 1536K 516K vmwait 0 2:43 0.00% supervise >> 50575 root 1 44 0 1536K 516K vmwait 0 2:42 0.00% supervise >> 45510 root 1 44 0 1536K 512K vmwait 0 2:42 0.00% supervise >> 58146 60 1 44 0 264M 8828K pfault 0 2:32 0.00% imapd >> 47526 389 6 44 0 92688K 2296K ucond 1 1:29 0.00% slapd >> 5417 root 1 44 0 9396K 1680K pfault 1 1:26 0.00% sshd >> 13147 root 1 44 0 3340K 860K vmwait 1 0:45 0.00% syslogd >> 92597 root 1 44 0 9396K 1676K pfault 1 0:39 0.00% sshd >> 26437 125 1 44 0 6924K 1700K vmwait 0 0:33 0.00% qmgr >> >> The above top was refreshing, but every other stuff on different ssh >> consoles (like a running zpool iostat and gstat) was frozen. >> Even top stopped when I have resized the window. >> > > Please try Kip's patch that was committed, it changes priorities a bit, > which should help. > My i386 machine is still alive after two days of uptime (with your patch, it lived for about two days, so I can't say -at least now- that it's OK). The amd64 machine started to loose ARC memory again. See these: http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem-week.png http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png Your patch was active between 7 and 9. You can see that the ARC size was somewhat constant. On october 9, I installed Kip's modification, and ARC size started to decrease. BTW, previously (before october 7) I set the arc min size to 10-15GB (can't remember the exact value), but now it runs with the defaults (only the max size is set): vfs.zfs.arc_min: 3623878656 vfs.zfs.arc_max: 28991029248 As you can see, there are plenty of memory. This machine uses UFS as well (and writes it heavily), maybe that's what affects ZFS size, by caching a lot of stuff? From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 08:32:29 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35BD4106568B; Mon, 12 Oct 2009 08:32:29 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 87BCB8FC2A; Mon, 12 Oct 2009 08:32:27 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 929E4183787; Mon, 12 Oct 2009 10:32:25 +0200 (CEST) X-CRM114-Version: 20090423-BlameSteveJobs ( TRE 0.7.6 (BSD) ) MF-ACE0E1EA [pR: 20.9900] X-CRM114-CacheID: sfid-20091012_10322_3B663058 X-CRM114-Status: Good ( pR: 20.9900 ) Message-ID: <4AD2E998.8090409@fsn.hu> Date: Mon, 12 Oct 2009 10:32:24 +0200 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: "K. Macy" References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu> <20091008160718.GB2134@garage.freebsd.pl> <4AD2E118.2050202@fsn.hu> <47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org> In-Reply-To: <47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org> X-Stationery: 0.4.10 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (people.fsn.hu); Mon, 12 Oct 2009 10:32:24 +0200 (CEST) Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 08:32:29 -0000 K. Macy wrote: >> >> The amd64 machine started to loose ARC memory again. See these: >> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem-week.png >> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png >> >> Your patch was active between 7 and 9. You can see that the ARC size >> was somewhat constant. >> On october 9, I installed Kip's modification, and ARC size started to >> decrease. >> BTW, previously (before october 7) I set the arc min size to 10-15GB >> (can't remember the exact value), but now it runs with the defaults >> (only the max size is set): >> vfs.zfs.arc_min: 3623878656 >> vfs.zfs.arc_max: 28991029248 >> >> As you can see, there are plenty of memory. This machine uses UFS as >> well (and writes it heavily), maybe that's what affects ZFS size, by >> caching a lot of stuff? >> > > Currently, the inactive page queue will grow until ARC is shrunk to > arc_min. > > > I think I'll probably spend some time making the ARC play better with > the page cache this week. Unfortunately, under heavy memory pressure > when competing with UFS the ARC will degrade to LRU, but I think that > is still an improvement over the current static sizing with low and > high water marks. Will setting ARC's minimum size help here? (I will try) For me, it's OK, and I think it's generally not a problem, if somebody uses UFS as well. Is it possible to merge the memory management of the two, or they are completely different beasts? Thanks, From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 08:55:18 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9519106568F for ; Mon, 12 Oct 2009 08:55:18 +0000 (UTC) (envelope-from kmatthew.macy@gmail.com) Received: from mail-yw0-f197.google.com (mail-yw0-f197.google.com [209.85.211.197]) by mx1.freebsd.org (Postfix) with ESMTP id 4E8E58FC13 for ; Mon, 12 Oct 2009 08:55:18 +0000 (UTC) Received: by ywh35 with SMTP id 35so28664097ywh.7 for ; Mon, 12 Oct 2009 01:55:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:subject:mime-version :content-type:from:in-reply-to:date:cc:message-id:references:to :x-mailer; bh=a2u5Ixb1owJ0ErCPIGKIKLX4nAjDlKMgGSm5bDYpvv4=; b=eYnacw389WQzmHettyLnm4/TgAz+mk1wQ11xJhmi2FB/1FCoO/97ZRc8LDoN6sma2y oOk4pwF2uybvoQSPa0PMorzRN7Oyelx/UXxCU6sxcwjCpvTDQZNtHZx7rKho7MhdybcJ 23b8sXaU+xqgxx7UfRkMbzhnIgGzkQ0V0hy5w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=DmsNggXP8pfvU5dTD5rqaX0MKHv/QBNDbBmVc8mnKoKnx7o3h+RiYTJnT89nE8iuTl Z/Zj77YCmXYhcR/EwRuzHJHXif76CMTzSwZsaBqFGvFMv+oUB2pkFHeg2H73fmmLgMRD 2iJvpaUEfeTqXLYsY0hlCDzUxZ+Zt0P4FZKxY= Received: by 10.101.34.10 with SMTP id m10mr5176023anj.32.1255335808555; Mon, 12 Oct 2009 01:23:28 -0700 (PDT) Received: from ?192.168.1.113? ([76.102.48.254]) by mx.google.com with ESMTPS id 22sm2189262yxe.3.2009.10.12.01.23.26 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 12 Oct 2009 01:23:27 -0700 (PDT) Sender: Matthew Macy Mime-Version: 1.0 (Apple Message framework v1076) From: "K. Macy" In-Reply-To: <4AD2E118.2050202@fsn.hu> Date: Mon, 12 Oct 2009 01:23:25 -0700 Message-Id: <47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org> References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu> <20091008160718.GB2134@garage.freebsd.pl> <4AD2E118.2050202@fsn.hu> To: Attila Nagy X-Mailer: Apple Mail (2.1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 08:55:18 -0000 > > The amd64 machine started to loose ARC memory again. See these: > http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem- > week.png > http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png > > Your patch was active between 7 and 9. You can see that the ARC size > was somewhat constant. > On october 9, I installed Kip's modification, and ARC size started > to decrease. > BTW, previously (before october 7) I set the arc min size to 10-15GB > (can't remember the exact value), but now it runs with the defaults > (only the max size is set): > vfs.zfs.arc_min: 3623878656 > vfs.zfs.arc_max: 28991029248 > > As you can see, there are plenty of memory. This machine uses UFS as > well (and writes it heavily), maybe that's what affects ZFS size, by > caching a lot of stuff? > Currently, the inactive page queue will grow until ARC is shrunk to arc_min. I think I'll probably spend some time making the ARC play better with the page cache this week. Unfortunately, under heavy memory pressure when competing with UFS the ARC will degrade to LRU, but I think that is still an improvement over the current static sizing with low and high water marks. -Kip From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 09:12:19 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECDA4106566B; Mon, 12 Oct 2009 09:12:19 +0000 (UTC) (envelope-from kmatthew.macy@gmail.com) Received: from mail-gx0-f214.google.com (mail-gx0-f214.google.com [209.85.217.214]) by mx1.freebsd.org (Postfix) with ESMTP id 596868FC15; Mon, 12 Oct 2009 09:12:19 +0000 (UTC) Received: by gxk6 with SMTP id 6so8242788gxk.13 for ; Mon, 12 Oct 2009 02:12:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:subject:mime-version :content-type:from:in-reply-to:date:cc:message-id:references:to :x-mailer; bh=qPAtTmvWEnXUYXeap5SZYKOzu4wzfajcEikLxQPF0Ok=; b=xua7Z9Iy9G9Zn1Fc9u8u7ivqF4hyRyyH6rIroVlBMp8QX4V2w08nbwJhf7OeFow7dv DLxMlKGR4yQcMesXchl9OSfan2wCj3IkGj9FSDV+gvAzkD/tnWi1GAphmU7PCEViDT2c HTBTxCTH4u8gYNQLhusvTsY/DcDHam4piwkOk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=S21lTLNghfIuzW+IrAULqtq6PDFOpc74jF0bEa5EyJ9Tb8d1Al05euLTiMQWHYbYJh Kl/0z2MrtlCgalODtKE2xhlMH4Lxn5iRiEKW857xdfmdpuPIKAP/cKjFfOQNnZSKGyFi GtwCwtDPCkEU4tkAu+YvW8Z5zoC+S7CyXWe2I= Received: by 10.101.146.11 with SMTP id y11mr5193477ann.33.1255338738759; Mon, 12 Oct 2009 02:12:18 -0700 (PDT) Received: from ?192.168.1.113? ([76.102.48.254]) by mx.google.com with ESMTPS id 23sm2205558yxe.12.2009.10.12.02.12.17 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 12 Oct 2009 02:12:18 -0700 (PDT) Sender: Matthew Macy Mime-Version: 1.0 (Apple Message framework v1076) From: "K. Macy" In-Reply-To: <4AD2E998.8090409@fsn.hu> Date: Mon, 12 Oct 2009 02:12:15 -0700 Message-Id: References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu> <20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu> <20091008160718.GB2134@garage.freebsd.pl> <4AD2E118.2050202@fsn.hu> <47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org> <4AD2E998.8090409@fsn.hu> To: Attila Nagy X-Mailer: Apple Mail (2.1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: ARC size constantly shrinks, then ZFS slows down extremely X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 09:12:20 -0000 >> >> Currently, the inactive page queue will grow until ARC is shrunk to >> arc_min. >> >> I think I'll probably spend some time making the ARC play better >> with the page cache this week. Unfortunately, under heavy memory >> pressure when competing with UFS the ARC will degrade to LRU, but I >> think that is still an improvement over the current static sizing >> with low and high water marks. > Will setting ARC's minimum size help here? (I will try) > > For me, it's OK, and I think it's generally not a problem, if > somebody uses UFS as well. > Is it possible to merge the memory management of the two, or they > are completely different beasts? To some degree it is possible to merge them by partly backing the arc from the page cache. This would allow for a fair amount of auto- tuning. However, it isn't possibly to completely merge them - the ARC is a virtual device block cache, UFS caches pages in the vm object based on their offset in the file. Thus it would never be possible to use blocks in the ARC for mmap - for applications that dirty file backed mmaped memory it will always be necessary to have two copies of the page, one in the vm object for the file and one in ZFS that maps to a block offset. It all makes a bit more sense if you understand that ZFS is a transactional object store with a posix file system interface. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 11:06:52 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 481F91065693 for ; Mon, 12 Oct 2009 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1269C8FC27 for ; Mon, 12 Oct 2009 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9CB6pR3036376 for ; Mon, 12 Oct 2009 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9CB6pkQ036372 for freebsd-fs@FreeBSD.org; Mon, 12 Oct 2009 11:06:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 12 Oct 2009 11:06:51 GMT Message-Id: <200910121106.n9CB6pkQ036372@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138524 fs [msdosfs] disks and usb flashes/cards with Russian lab o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138367 fs [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/138109 fs [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133980 fs [panic] [ffs] panic: ffs_valloc: dup alloc o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [smbfs] [panic] panic: ffs_truncate: read-only filesys o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132597 fs [tmpfs] [panic] tmpfs-related panic while interrupting o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/131086 fs [ext2fs] [patch] mkfs.ext2 creates rotten partition o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE f kern/128173 fs [ext2fs] ls gives "Input/output error" on mounted ext3 o kern/127659 fs [tmpfs] tmpfs memory leak o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS f kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li f kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122047 fs [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b f usb/112640 fs [ext2fs] [hang] Kernel freezes when writing a file to o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/105093 fs [ext2fs] [patch] ext2fs on read-only media cannot be m o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/89991 fs [ufs] softupdates with mount -ur causes fs UNREFS o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/77826 fs [ext2fs] ext2fs usb filesystem will not mount RW o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 135 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 12:33:18 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64301106568F; Mon, 12 Oct 2009 12:33:18 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7826F8FC0A; Mon, 12 Oct 2009 12:33:17 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA16105; Mon, 12 Oct 2009 15:33:15 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4AD3220B.5030502@icyb.net.ua> Date: Mon, 12 Oct 2009 15:33:15 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20090825) MIME-Version: 1.0 To: freebsd-current@FreeBSD.ORG, freebsd-fs@FreeBSD.ORG X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: zfs boot vs loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 12:33:18 -0000 I am using ZFS for boot-and-root filesystem. gptzfsboot is used for bootstrapping, if that matters. Maybe the following is some kind of a pilot error, but I think that I see a problem with selecting a different kernel using loader prompt. That is, I escape to a loader prompt from boot menu. There I do 'unload'. Then I 'load' a different kernel, opensolaris.ko and zfs.ko. Then I 'boot'. Then root mounting inevitably fails because ZFS root filesystem can not be found. This is an unexpected result for me. I wonder if something important for ZFS get unloaded when I unload preloaded kernel and modules. Perhaps I have to manually re-load zpool.cache? This happens with recent CURRENT, amd64. What's strange is that I think that I can switch kernels from loader prompt without any problems when using RELENG_7 and UFS-boot-plus-ZFS-root approach. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 15:16:16 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01510106566B for ; Mon, 12 Oct 2009 15:16:16 +0000 (UTC) (envelope-from kraduk@googlemail.com) Received: from mail-bw0-f223.google.com (mail-bw0-f223.google.com [209.85.218.223]) by mx1.freebsd.org (Postfix) with ESMTP id 73CA98FC0C for ; Mon, 12 Oct 2009 15:16:15 +0000 (UTC) Received: by bwz23 with SMTP id 23so1997045bwz.43 for ; Mon, 12 Oct 2009 08:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=4ih3QCVWx9Doe1BaEE8PK4OYt4APD28RwTYrBjjFqKA=; b=X1dLA1Hg0W+q5XnApi7or6amBVl8eZ7mxh5EICcsXAIjkcsmztn9d7XGX+QoLRybjq eEZO5Q/1p+FlqrprD2Ar5PKJfoibgJhn1auLpjN8WeIzOAPRi8zfAXtorgB3e+dvlOUj MC0R5SI8a1Hu6glGyKbS8yHY9VRhcUqoYE+uw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=jiEy2ESHfw337qALL91qh6CI4FxOAjbCn5LXLJ5U+rkB7HO4aqrTPlXhEYUxfR5C8X x1CnQt2m4dLTiVBXg08+DiMcz/FWThEXOjRSZ7R6W+aUtgHOXrTKP9vvOkbfI0KJLaCe gBXNQjMw7oKZ+YP6W+kPzQNwIw6PJ7u00PrDU= MIME-Version: 1.0 Received: by 10.239.145.142 with SMTP id s14mr368689hba.144.1255359300585; Mon, 12 Oct 2009 07:55:00 -0700 (PDT) In-Reply-To: <4AD3220B.5030502@icyb.net.ua> References: <4AD3220B.5030502@icyb.net.ua> Date: Mon, 12 Oct 2009 15:55:00 +0100 Message-ID: From: krad To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: zfs boot vs loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 15:16:16 -0000 2009/10/12 Andriy Gapon > > I am using ZFS for boot-and-root filesystem. gptzfsboot is used for > bootstrapping, > if that matters. > Maybe the following is some kind of a pilot error, but I think that I see a > problem with selecting a different kernel using loader prompt. > That is, I escape to a loader prompt from boot menu. There I do 'unload'. > Then I > 'load' a different kernel, opensolaris.ko and zfs.ko. Then I 'boot'. Then > root > mounting inevitably fails because ZFS root filesystem can not be found. > > This is an unexpected result for me. > I wonder if something important for ZFS get unloaded when I unload > preloaded > kernel and modules. Perhaps I have to manually re-load zpool.cache? > > This happens with recent CURRENT, amd64. > What's strange is that I think that I can switch kernels from loader prompt > without any problems when using RELENG_7 and UFS-boot-plus-ZFS-root > approach. > > -- > Andriy Gapon > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > before you boot the kernels check the values of the following are still set eg vfs.root.mountfrom="zfs:system/root" vfs.root.mountfrom.options=rw From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 16:20:02 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB1D4106568B for ; Mon, 12 Oct 2009 16:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B09318FC15 for ; Mon, 12 Oct 2009 16:20:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9CGK2cO020293 for ; Mon, 12 Oct 2009 16:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9CGK2Q8020292; Mon, 12 Oct 2009 16:20:02 GMT (envelope-from gnats) Date: Mon, 12 Oct 2009 16:20:02 GMT Message-Id: <200910121620.n9CGK2Q8020292@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Andrey Simonenko Cc: Subject: Re: kern/136865: NFS exports atomic and on-the-fly atomic updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Andrey Simonenko List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 16:20:02 -0000 The following reply was made to PR kern/136865; it has been noted by GNATS. From: Andrey Simonenko To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/136865: NFS exports atomic and on-the-fly atomic updates Date: Mon, 12 Oct 2009 19:19:15 +0300 Updated version nfse-20091012 can be used on 9.0-CURRENT and 8.0-RC1. with experimental and regular NFS server. Can be downloaded here http://comsys.ntu-kpi.kiev.ua/~simon/nfse/ MD5 (nfse-20091012.tar.bz2) = 3562d449406ac0a728928de5d0583884 List of changes: * Command "flush" can be combined with "add" commands. * Added new command "clear" -- clear configuration for the given pathname. * Added new command "show" -- show current configuration. * Now nfse recognizes shadowed mount points. Manual page nfs.exports(5) was updated and contains information how nfse(8) works with mount points. * Now all settings (export specifications for file systems, export specifications for NFSv4 root directory and WebNFS settings) are updated atomically. There is one exception for WebNFS settings for new exported file system, but this exception can be suppressed by the administrator. * Added support for NFSv4 root directory configuration (atomic and on-the-fly atomic updates are supported). * Added support for experimental NFS server. * Several mistakes were corrected, some parts were optimized and/or simplified. * Support for obsolete options was removed. * mountd(8) was renamed to nfse(8) (actually nfse uses only modified RPC and XDR related code from mountd), exports(5) was renamed to nfs.exports(5). * This distribution contains changes for FreeBSD 9.0-CURRENT and 8.0-RC1. List of open-questions: 1. WebNFS settings cannot be updated by "-c command", it is necessary to discuss semantics for experimental NFS server. 2. WebNFS settings are not protected by any lock in NFS server. 3. NFSv4 root directory settings are not protected by any lock in NFS server. 4. It is possible to make better integration of nfs_export.c with NFS server (see nfs_export.c:nfse_fs_check() and nfs_export.c:nfse_rd_check()), it is necessary to discuss this. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 17:36:50 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D448E106568B for ; Mon, 12 Oct 2009 17:36:50 +0000 (UTC) (envelope-from swell.k@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id 5DB968FC18 for ; Mon, 12 Oct 2009 17:36:50 +0000 (UTC) Received: by ewy18 with SMTP id 18so3008451ewy.43 for ; Mon, 12 Oct 2009 10:36:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :date:in-reply-to:message-id:user-agent:mime-version:content-type; bh=43/D0hXcMuoGrKXXiNkcWmcWSxNRrHXGcDT0t5gB040=; b=GG+qXFkU6jhGgcuFRsvjmMo2J1EukVatJrSyew/WOuN4q9zbT9GfdeeEq0A3cc0dsr geFa+WTCy0eLBi7Yi3I5zanvhgjI3XiAjrHRiABQ2fnD2Cuwd5D+Tg4UK2C7eKjtGuyh MrVuf24jtKqvRte/YligHXzIrXZlHf/jdVv4k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-type; b=UMcyZKQr6BT+BZfCQcW0gMEK69n4KIw4Pfa6xlmmEw4ydk+f9tfZYRB3N+pxdP6zq2 Lw2zlKNCJ1W1m7dKiQLCqzbjOqxeGIk03l37zaevy6c7oPHZeNPnikQ1W7PEkcURqosI 8M9oGrugy8H1RsFr8nYPHrWLyCsZub+nKqwWw= Received: by 10.211.130.13 with SMTP id h13mr4439144ebn.13.1255369009372; Mon, 12 Oct 2009 10:36:49 -0700 (PDT) Received: from localhost (95-24-212-202.broadband.corbina.ru [95.24.212.202]) by mx.google.com with ESMTPS id 5sm806931eyh.4.2009.10.12.10.36.48 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 12 Oct 2009 10:36:48 -0700 (PDT) From: Anonymous To: Andriy Gapon References: <4AD3220B.5030502@icyb.net.ua> Date: Mon, 12 Oct 2009 21:36:03 +0400 In-Reply-To: <4AD3220B.5030502@icyb.net.ua> (Andriy Gapon's message of "Mon, 12 Oct 2009 15:33:15 +0300") Message-ID: <86ws30ikfw.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-fs@FreeBSD.ORG Subject: Re: zfs boot vs loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 17:36:50 -0000 Andriy Gapon writes: > I am using ZFS for boot-and-root filesystem. gptzfsboot is used for bootstrapping, > if that matters. > Maybe the following is some kind of a pilot error, but I think that I see a > problem with selecting a different kernel using loader prompt. > That is, I escape to a loader prompt from boot menu. There I do 'unload'. Then I > 'load' a different kernel, opensolaris.ko and zfs.ko. Then I 'boot'. Then root > mounting inevitably fails because ZFS root filesystem can not be found. > You can just type OK boot kernel.old here to switch to kernel.old (or any other one). No need to manually load modules if you specified them in loader.conf. > This is an unexpected result for me. > I wonder if something important for ZFS get unloaded when I unload preloaded > kernel and modules. Perhaps I have to manually re-load zpool.cache? Does following help? OK load -t blah /boot/zfs/zpool.cache > > This happens with recent CURRENT, amd64. > What's strange is that I think that I can switch kernels from loader prompt > without any problems when using RELENG_7 and UFS-boot-plus-ZFS-root approach. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 12 19:49:40 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 280361065672 for ; Mon, 12 Oct 2009 19:49:40 +0000 (UTC) (envelope-from alextzfs@googlemail.com) Received: from mail-fx0-f222.google.com (mail-fx0-f222.google.com [209.85.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id 56CBF8FC13 for ; Mon, 12 Oct 2009 19:49:38 +0000 (UTC) Received: by fxm22 with SMTP id 22so8428659fxm.36 for ; Mon, 12 Oct 2009 12:49:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=ctHl8nrk2CvjOq+KhD1cyqPrUd0IfVRmuN1bdVWr1JY=; b=FGIoNHYUmThqd1wXWfeBd0OVJCquFVoSPX+/XzuZNCt1hmLz9Xgwf+0MqM6Q9AVzwR igQHhRqB+NOoUq+YpDPkOAa9JVLJ5v5ti2go6e2+ZFFHBtU80jhw+PAejb1e2TypsWRX CN1Pn8/1Xo/TH6fKN7VoHfKJXmZlilYsb5pGg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=R49kd2iwxsvIyjkGzDp01JkSEgc2SzyCpQRbY6k207DpxjjMcTOVBKjIvnh/9Qvy2s K9u19mgcJx3U2btsboOkHuxRSCSs8Ycu9SO5l+hOf4oFj6b1kIBey0q1/JhrOqbydjSg EtPn8xWznl4HrwDfiO5izZiWzno7IJMsU0QCI= MIME-Version: 1.0 Received: by 10.204.150.81 with SMTP id x17mr5332393bkv.73.1255376977838; Mon, 12 Oct 2009 12:49:37 -0700 (PDT) In-Reply-To: <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com> References: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com> <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com> Date: Mon, 12 Oct 2009 20:49:37 +0100 Message-ID: <4d98b5320910121249q36c68b8vf63ec27cf4bb94c9@mail.gmail.com> From: Alex Trull To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: zraid2 loses a single disk and becomes difficult to recover X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2009 19:49:40 -0000 I managed to cleanly recover all critical data by cloning the most recent snapshots of all my filesystems (which worked even for those filesystems that had disappeared from 'zfs list') - and moving back to ufs2 The 'live' filesystems since the snapshots had pretty much gone corrupt. Intereresting note is that even if I promoted those clones - if the system was rebooted the contents of the snapshots became gobbledygooked (invalid byte sequence errors on numerous files). As it stands I managed to recover 100% of the data, so I'm out the woods. How does a dual-parity array lose its mind when only one disk is lost ? Might it have been related to the old TXGid I found on ad16 and ad17 ? -- Alex 2009/10/11 Alex Trull > Well after trying alot of things (zpool import with or without cache file > in place, etc), it randomly managed to mount the pool up, atleast, with > errors - : > > zfs list output: > cannot iterate filesystems: I/O error > NAME USED AVAIL REFER MOUNTPOINT > fatman 1.40T 1.70T 51.2K /fatman > fatman/backup 100G 99.5G 95.5G /fatman/backup > fatman/jail 422G 1.70T 60.5K /fatman/jail > fatman/jail/havnor 198G 51.7G 112G /fatman/jail/havnor > fatman/jail/mail 19.4G 30.6G 13.0G /fatman/jail/mail > fatman/jail/syndicate 16.6G 103G 10.5G /fatman/jail/syndicate > fatman/jail/thirdforces 159G 41.4G 78.1G /fatman/jail/thirdforces > fatman/jail/web 24.8G 25.2G 22.3G /fatman/jail/web > fatman/stash 913G 1.70T 913G /fatman/stash > > (end of the dmesg) > JMR: vdev_uberblock_load_done ubbest ub_txg=46475461 > ub_timestamp=1255231841 > JMR: vdev_uberblock_load_done ub_txg=46481476 ub_timestamp=1255234263 > JMR: vdev_uberblock_load_done ubbest ub_txg=46481476 > ub_timestamp=1255234263 > JMR: vdev_uberblock_load_done ubbest ub_txg=46475459 > ub_timestamp=1255231780 > JMR: vdev_uberblock_load_done ubbest ub_txg=46475458 > ub_timestamp=1255231750 > JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263 > JMR: vdev_uberblock_load_done ubbest ub_txg=46481473 > ub_timestamp=1255234263 > JMR: vdev_uberblock_load_done ubbest ub_txg=46481472 > ub_timestamp=1255234263 > Solaris: WARNING: can't open objset for fatman/jail/margaret > Solaris: WARNING: can't open objset for fatman/jail/margaret > Solaris: WARNING: ZFS replay transaction error 86, dataset > fatman/jail/havnor, seq 0x25442, txtype 9 > > Solaris: WARNING: ZFS replay transaction error 86, dataset > fatman/jail/mail, seq 0x1e200, txtype 9 > > Solaris: WARNING: ZFS replay transaction error 86, dataset > fatman/jail/thirdforces, seq 0x732e3, txtype 9 > > [root@potjie /fatman/jail/mail]# zpool status -v > pool: fatman > state: DEGRADED > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scrub: resilver in progress for 0h4m, 0.83% done, 8h21m to go > config: > > NAME STATE READ WRITE CKSUM > fatman DEGRADED 0 0 34 > raidz2 DEGRADED 0 0 384 > replacing DEGRADED 0 0 0 > da2/old REMOVED 0 24 0 > da2 ONLINE 0 0 0 1.71G resilvered > ad4 ONLINE 0 0 0 21.3M resilvered > ad6 ONLINE 0 0 0 21.4M resilvered > ad20 ONLINE 0 0 0 21.3M resilvered > ad22 ONLINE 0 0 0 21.3M resilvered > ad17 ONLINE 0 0 0 21.3M resilvered > da3 ONLINE 0 0 0 21.3M resilvered > ad10 ONLINE 0 0 1 21.4M resilvered > ad16 ONLINE 0 0 0 21.2M resilvered > cache > ad18 ONLINE 0 0 0 > > errors: Permanent errors have been detected in the following files: > > fatman/jail/margaret:<0x0> > fatman/jail/syndicate:<0x0> > fatman/jail/mail:<0x0> > /fatman/jail/mail/tmp > fatman/jail/havnor:<0x0> > fatman/jail/thirdforces:<0x0> > fatman/backup:<0x0> > > jail/margaret & backup isn't showing up in zfs list > jail/syndicate is showing up but isn't viewable > > It seems the latest content on the better-looking zfs filesystems are quite > recent. > > Any thoughts about what is going on ? > > I have snapshots for africa on these zfs filesystems - any suggestions on > trying to get them back ? > > -- > Alex > > 2009/10/11 Alex Trull > > Hi All, >> >> My zraid2 has broken this morning on releng_7 zfs13. >> >> System failed this morning and came back without pool - having lost a >> disk. >> >> This is how I found the system: >> >> pool: fatman >> state: FAULTED >> status: One or more devices could not be used because the label is missing >> >> or invalid. There are insufficient replicas for the pool to continue >> functioning. >> action: Destroy and re-create the pool from a backup source. >> see: http://www.sun.com/msg/ZFS-8000-5E >> scrub: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> fatman FAULTED 0 0 1 corrupted data >> raidz2 DEGRADED 0 0 6 >> da2 FAULTED 0 0 0 corrupted data >> ad4 ONLINE 0 0 0 >> ad6 ONLINE 0 0 0 >> ad20 ONLINE 0 0 0 >> ad22 ONLINE 0 0 0 >> ad17 ONLINE 0 0 0 >> da2 ONLINE 0 0 0 >> ad10 ONLINE 0 0 0 >> ad16 ONLINE 0 0 0 >> >> Initialy it complained that da3 had gone to da2 (da2 had failed and was no >> longer seen) >> >> I replaced the original da2 and bumped what was originaly da3 back up to >> da3 using the controllers ordering. >> >> [root@potjie /dev]# zpool status >> pool: fatman >> state: FAULTED >> status: One or more devices could not be used because the label is missing >> >> or invalid. There are insufficient replicas for the pool to continue >> functioning. >> action: Destroy and re-create the pool from a backup source. >> see: http://www.sun.com/msg/ZFS-8000-5E >> scrub: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> fatman FAULTED 0 0 1 corrupted data >> raidz2 ONLINE 0 0 6 >> da2 UNAVAIL 0 0 0 corrupted data >> ad4 ONLINE 0 0 0 >> ad6 ONLINE 0 0 0 >> ad20 ONLINE 0 0 0 >> ad22 ONLINE 0 0 0 >> ad17 ONLINE 0 0 0 >> da3 ONLINE 0 0 0 >> ad10 ONLINE 0 0 0 >> ad16 ONLINE 0 0 0 >> >> Issue looks very similar to this (JMR's issue) : >> http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html >> >> I've tried the methods there without much result. >> >> Using JMR's patches/debugs to see what is going on, this is what I got: >> >> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 >> ub_timestamp=1255246834 >> JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780 >> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 >> ub_timestamp=1255246834 >> JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750 >> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 >> ub_timestamp=1255246834 >> JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263 >> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 >> ub_timestamp=1255246834 >> JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263 >> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 >> ub_timestamp=1255246834 >> >> But JMR's patch still doesn't let me import even with a decremented txg >> >> I then had a look around the drives using zdb and some dirty script: >> >> [root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo >> "$1";zdb -l "$1" |grep txg"}' | sh >> /dev/ad10 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> /dev/ad16 >> txg=46408223 <- old TXGid ? >> txg=46408223 >> txg=46408223 >> txg=46408223 >> /dev/ad17 >> txg=46408223 <- old TXGid ? >> txg=46408223 >> txg=46408223 >> txg=46408223 >> /dev/ad18 (ssd) >> /dev/ad19 (spare drive, removed from pool some time ago) >> txg=0 >> create_txg=0 >> txg=0 >> create_txg=0 >> txg=0 >> create_txg=0 >> txg=0 >> create_txg=0 >> /dev/ad20 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> /dev/ad22 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> /dev/ad4 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> /dev/ad6 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> /dev/da2 < new drive replaced broken da2 >> /dev/da3 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> txg=46488654 >> >> I did not see any checksums or other issues on ad16 and ad17 previously, >> and I do check regularly. >> >> Any thoughts on what to try next ? >> >> Regards, >> >> Alex >> >> > From owner-freebsd-fs@FreeBSD.ORG Tue Oct 13 04:17:07 2009 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E87B10656A8 for ; Tue, 13 Oct 2009 04:17:07 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 668938FC18 for ; Tue, 13 Oct 2009 04:17:05 +0000 (UTC) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id n9D3t4SP072829; Tue, 13 Oct 2009 11:55:04 +0800 (KRAST) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id n9D3t47O072828; Tue, 13 Oct 2009 11:55:04 +0800 (KRAST) (envelope-from eugen) Date: Tue, 13 Oct 2009 11:55:04 +0800 From: Eugene Grosbein To: stable@freebsd.org Message-ID: <20091013035504.GA72620@svzserv.kemerovo.su> References: <20091011175428.GA3626@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20091011175428.GA3626@freefall.freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: fs@freebsd.org Subject: Re: FreeBSD Status Reports April - September, 2009 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2009 04:17:07 -0000 On Sun, Oct 11, 2009 at 05:54:29PM +0000, Daniel Gerzo wrote: > FreeBSD/ZFS > > Contact: Pawel Dawidek > > We believe that the ZFS file system is now production-ready in FreeBSD > 8.0. Most (if not all) reported bugs were fixed and ZFS is no longer > tagged as experimental. There is also ongoing work in Perforce to bring > the latest ZFS version (v19) to FreeBSD. That's great news. However, my experience says me not place dot-zero relese under business-critical tasks and load. What about status of ZFS in 7.2? Does 7.2 contain the same ZFS code? Eugene Grosbein From owner-freebsd-fs@FreeBSD.ORG Tue Oct 13 10:46:40 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC8EE106568F; Tue, 13 Oct 2009 10:46:40 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C61008FC12; Tue, 13 Oct 2009 10:46:39 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA07724; Tue, 13 Oct 2009 13:46:38 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4AD45A8D.4070605@icyb.net.ua> Date: Tue, 13 Oct 2009 13:46:37 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20090825) MIME-Version: 1.0 To: freebsd-fs@FreeBSD.ORG, freebsd-current@FreeBSD.ORG References: <4AD3220B.5030502@icyb.net.ua> <86ws30ikfw.fsf@gmail.com> In-Reply-To: <86ws30ikfw.fsf@gmail.com> X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Anonymous Subject: Re: zfs boot vs loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2009 10:46:40 -0000 Thanks to all who replied! I think that I figured it out with your help. 'unload' does indeed unload zpool.cache and the latter is needed for proper zfs root mounting. 'boot /other/kernel' executes some loader scripts and zpool.cache gets loaded in this case (because of the settings in defaults/loader.conf). When doing manual 'load xxx' and then just 'boot' no loader scripts are executed and zpool.cache does not get loaded. In this case one has to load it manually using the suggested 'load -t xxx' approach. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Oct 13 13:40:25 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F02771065696; Tue, 13 Oct 2009 13:40:25 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C74918FC1D; Tue, 13 Oct 2009 13:40:25 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9DDePIH063211; Tue, 13 Oct 2009 13:40:25 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9DDePKE063200; Tue, 13 Oct 2009 13:40:25 GMT (envelope-from linimon) Date: Tue, 13 Oct 2009 13:40:25 GMT Message-Id: <200910131340.n9DDePKE063200@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/139564: [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2009 13:40:26 -0000 Old Synopsis: zfs - 8.0-RC1 - Fatal trap 12 at end of shutdown New Synopsis: [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdown Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Tue Oct 13 13:40:02 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=139564 From owner-freebsd-fs@FreeBSD.ORG Tue Oct 13 15:57:07 2009 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED1891065672; Tue, 13 Oct 2009 15:57:06 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-yx0-f184.google.com (mail-yx0-f184.google.com [209.85.210.184]) by mx1.freebsd.org (Postfix) with ESMTP id 91B278FC22; Tue, 13 Oct 2009 15:57:06 +0000 (UTC) Received: by yxe14 with SMTP id 14so21285990yxe.7 for ; Tue, 13 Oct 2009 08:57:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=BOCDw0YPyUacEfOROerZ3aElAsd9M4L760LaLHLrhGA=; b=c3ZakIC8uZz2HPbeJsD9kXd2qP0XDs8eSvKH29nZH8RBIJfefGwuDc/3Y+eC1vEq4R mGValAvqGBic844mY/ZRNnuUattdORMf7WJ/qm4jitXMjQ2oQE7GWo7CfvyMhrzaDr3p +NXU0EQFQjvB1V/v2ZrlytYM0GcifqBXEldHw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=cuOUG0pYvYTLbDaYmzgndWW6IEAckg7zJxkfNev0J6upzd9RCOxRvVMMrUMLom5Xzt L2SlxD6KddG1tUoeNg9FFewRy+2PBpwV1CGSP3XOqi8XG4hzuGX2O6cY4F/70sfmdaGk mu7k7BVRCKhX9pFdqK7/9GWMadX+oNvNIkp0Q= MIME-Version: 1.0 Received: by 10.150.130.38 with SMTP id c38mr12776141ybd.213.1255447934616; Tue, 13 Oct 2009 08:32:14 -0700 (PDT) In-Reply-To: <20091013035504.GA72620@svzserv.kemerovo.su> References: <20091011175428.GA3626@freefall.freebsd.org> <20091013035504.GA72620@svzserv.kemerovo.su> Date: Tue, 13 Oct 2009 08:32:14 -0700 Message-ID: From: Freddie Cash To: stable@freebsd.org, fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: FreeBSD Status Reports April - September, 2009 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2009 15:57:07 -0000 On Mon, Oct 12, 2009 at 8:55 PM, Eugene Grosbein wrote: > On Sun, Oct 11, 2009 at 05:54:29PM +0000, Daniel Gerzo wrote: > > FreeBSD/ZFS > > > > Contact: Pawel Dawidek > > > > We believe that the ZFS file system is now production-ready in FreeBSD > > 8.0. Most (if not all) reported bugs were fixed and ZFS is no longer > > tagged as experimental. There is also ongoing work in Perforce to > bring > > the latest ZFS version (v19) to FreeBSD. > > That's great news. However, my experience says me not place dot-zero > relese under business-critical tasks and load. > > What about status of ZFS in 7.2? Does 7.2 contain the same ZFS code? > > 7.2-RELEASE includes ZFSv6. 7-STABLE (after the release of 7.2) includes ZFSv13. 8.0-RELEASE will include ZFSv13. IOW, 8.0 and 7.3 will have roughly the same ZFS code, although I believe the plan is to leave it marked as "experimental" in 7.x and only remove that warning in 8.x. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 06:21:16 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6BC4106566B for ; Wed, 14 Oct 2009 06:21:16 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206049004.chello.pl [87.206.49.4]) by mx1.freebsd.org (Postfix) with ESMTP id 25A378FC0A for ; Wed, 14 Oct 2009 06:21:15 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id E0EBA45E94; Wed, 14 Oct 2009 08:21:13 +0200 (CEST) Received: from localhost (chello087206049004.chello.pl [87.206.49.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id B2B1445B36; Wed, 14 Oct 2009 08:21:08 +0200 (CEST) Date: Wed, 14 Oct 2009 08:21:07 +0200 From: Pawel Jakub Dawidek To: Alex Trull Message-ID: <20091014062107.GB1696@garage.freebsd.pl> References: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com> <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com> <4d98b5320910121249q36c68b8vf63ec27cf4bb94c9@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XF85m9dhOBO43t/C" Content-Disposition: inline In-Reply-To: <4d98b5320910121249q36c68b8vf63ec27cf4bb94c9@mail.gmail.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: zraid2 loses a single disk and becomes difficult to recover X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 06:21:16 -0000 --XF85m9dhOBO43t/C Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 12, 2009 at 08:49:37PM +0100, Alex Trull wrote: > I managed to cleanly recover all critical data by cloning the most recent > snapshots of all my filesystems (which worked even for those filesystems > that had disappeared from 'zfs list') - and moving back to ufs2 >=20 > The 'live' filesystems since the snapshots had pretty much gone corrupt. >=20 > Intereresting note is that even if I promoted those clones - if the system > was rebooted the contents of the snapshots became gobbledygooked (invalid > byte sequence errors on numerous files). >=20 > As it stands I managed to recover 100% of the data, so I'm out the woods. I'm glad to hear that. > How does a dual-parity array lose its mind when only one disk is lost ? > Might it have been related to the old TXGid I found on ad16 and ad17 ? Yes, definiately. For some reason ZFS didn't update txg on those two disks, so at this point you were running without parity. The problem is that ZFS didn't start resilver automatically and also didn't report this situation properly. I think I saw this in the past. Running 'zpool scrub' on this pool will trigger resilver. There must be a bug. I tried to reproduce it by modifying the code not to update txg on one of the components. There are three places where this can happen on sytem crash/power failure and I tried all of them - no luck, ZFS was able to recover properly. It would be good idea to run 'zpool scrub' on regular basis, even if only to see if it won't trigger resilver (it can be stopped after few minutes with 'zpool scrub -s'). Of course it is adviced to run full scrub from time to time. Do you have this pool around still? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --XF85m9dhOBO43t/C Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFK1W3SForvXbEpPzQRAntPAKDRIJlRaFazDnVyQ836Zgksdeg7+wCgzV+Z 3+DBuZkOEgeihv4p3OXMyYI= =JN8d -----END PGP SIGNATURE----- --XF85m9dhOBO43t/C-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 06:28:35 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F967106568D; Wed, 14 Oct 2009 06:28:35 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206049004.chello.pl [87.206.49.4]) by mx1.freebsd.org (Postfix) with ESMTP id 9316E8FC1F; Wed, 14 Oct 2009 06:28:34 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 0A9ED45CAC; Wed, 14 Oct 2009 08:28:33 +0200 (CEST) Received: from localhost (chello087206049004.chello.pl [87.206.49.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id ED95F45B36; Wed, 14 Oct 2009 08:28:27 +0200 (CEST) Date: Wed, 14 Oct 2009 08:28:27 +0200 From: Pawel Jakub Dawidek To: Andriy Gapon Message-ID: <20091014062827.GC1696@garage.freebsd.pl> References: <4AD3220B.5030502@icyb.net.ua> <86ws30ikfw.fsf@gmail.com> <4AD45A8D.4070605@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Pk6IbRAofICFmK5e" Content-Disposition: inline In-Reply-To: <4AD45A8D.4070605@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@FreeBSD.ORG, Anonymous , freebsd-current@FreeBSD.ORG Subject: Re: zfs boot vs loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 06:28:35 -0000 --Pk6IbRAofICFmK5e Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 13, 2009 at 01:46:37PM +0300, Andriy Gapon wrote: >=20 > Thanks to all who replied! > I think that I figured it out with your help. >=20 > 'unload' does indeed unload zpool.cache and the latter is needed for prop= er zfs > root mounting. > 'boot /other/kernel' executes some loader scripts and zpool.cache gets lo= aded in > this case (because of the settings in defaults/loader.conf). > When doing manual 'load xxx' and then just 'boot' no loader scripts are e= xecuted > and zpool.cache does not get loaded. In this case one has to load it manu= ally > using the suggested 'load -t xxx' approach. Could you repeat what you were doing but with vfs.zfs.debug set to 1 from the loader? You should see some messages about missing file, maybe we should do them visible always and not after increasing debug level? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Pk6IbRAofICFmK5e Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFK1W+KForvXbEpPzQRAsAXAJ9LWdtKzOwrlZAfq6yVOZCnGWB8JgCfWSUr tqaZYC/lZtc5Do9/Yf9z6M8= =A5Tp -----END PGP SIGNATURE----- --Pk6IbRAofICFmK5e-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 16:47:39 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 806301065670; Wed, 14 Oct 2009 16:47:39 +0000 (UTC) (envelope-from solon@pyro.de) Received: from srv23.fsb.echelon.bnd.org (mail.pyro.de [83.137.99.96]) by mx1.freebsd.org (Postfix) with ESMTP id 8F8E38FC15; Wed, 14 Oct 2009 16:47:37 +0000 (UTC) Received: from port-87-193-183-44.static.qsc.de ([87.193.183.44] helo=flash.home) by srv23.fsb.echelon.bnd.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1My70e-00041m-Fe; Wed, 14 Oct 2009 18:47:36 +0200 Date: Wed, 14 Oct 2009 18:47:31 +0200 From: Solon Lutz X-Mailer: The Bat! (v3.99.25) Professional Organization: pyro.labs berlin X-Priority: 3 (Normal) Message-ID: <473227349.20091014184731@pyro.de> To: Pawel Jakub Dawidek , freebsd-fs@FreeBSD.ORG In-Reply-To: <673550066.20091013133544@pyro.de> References: <20091004185738.GI1660@garage.freebsd.pl> <165582258.20091012155536@pyro.de> <20091012193636.GC1762@garage.freebsd.pl> <756365088.20091013005202@pyro.de> <20091013055754.GA3197@garage.freebsd.pl> <90685589.20091013092418@pyro.de> <20091013072712.GA1597@garage.freebsd.pl> <12910471099.20091013095322@pyro.de> <20091013075511.GC1597@garage.freebsd.pl> <1433853337.20091013100348@pyro.de> <20091013082116.GE1597@garage.freebsd.pl> <673550066.20091013133544@pyro.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Score: -0.1 (/) X-Spam-Report: Spam detection software, running on the system "srv23.fsb.echelon.bnd.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see The administrator of that system for details. Content preview: >>> >> > As I understand the values 13462283 and 13462284 were showing errors? >>> >> > Yes, just try to import the pool and set vfs.zfs.maxtxg back to zero >>> >> > afterwards. >>> >> 13462283 and 13462284 were showing errors. >>> >> vfs.zfs.maxtxg was at -1 before I changed it. [...] Content analysis details: (-0.1 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.4 ALL_TRUSTED Passed through trusted hosts only via SMTP 1.3 PLING_QUERY Subject has exclamation mark and question mark X-Spam-Flag: NO Cc: Subject: Re: Help needed! ZFS I/O error recovery? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 16:47:39 -0000 >>> >> > As I understand the values 13462283 and 13462284 were showing errors? >>> >> > Yes, just try to import the pool and set vfs.zfs.maxtxg back to zero >>> >> > afterwards. >>> >> 13462283 and 13462284 were showing errors. >>> >> vfs.zfs.maxtxg was at -1 before I changed it. >>> > So does it work now? >>> radium# zpool import -f temp >>> cannot iterate filesystems: I/O error >>> radium# ll /temp/ >>> ls: backup: Input/output error >>> total 9 >>> drwxr-xr-x 2 root wheel 2 May 16 2007 audio >>> drwxr-xr-x 2 root wheel 2 May 16 2007 misc >>> drwxr-xr-x 2 root wheel 2 May 16 2007 video >>> drwxr-xr-x 8 dhcpd dhcpd 11 Jul 8 10:36 www >>> radium# ll /temp/audio/ >>> total 0 >>> radium# ll /temp/misc/ >>> total 0 >>> radium# zfs unmount temp >>> cannot iterate filesystems: I/O error >> Can you show 'zpool status'? >> It might be worth trying to go even more in the past with maxtxg. > sysctl vfs.zfs.maxtxg=13462260 > pool: temp > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scrub: none requested > config: > NAME STATE READ WRITE CKSUM > temp ONLINE 0 0 9 > da0 ONLINE 0 0 36 > errors: Permanent errors have been detected in the following files: > temp:<0x0> > temp:<0x49681> > temp:<0x499a4> > temp:<0x495fd> > temp/space1:<0x0> > temp/space2:<0x0> > temp/space3:<0x0> > temp/space4:<0x0> > temp/space5:<0x0> I just tried it with more TXGs, even with a jump of -300, but it always gives an "cannot iterate filesystems: I/O error" error if I try to import the pool. Also because of mounting the pool, the TXg has gone up to 13445935 from initially 13462284. solon From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 20:10:56 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 852861065697; Wed, 14 Oct 2009 20:10:56 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5BE288FC27; Wed, 14 Oct 2009 20:10:56 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9EKAu1F076105; Wed, 14 Oct 2009 20:10:56 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9EKAute076100; Wed, 14 Oct 2009 20:10:56 GMT (envelope-from linimon) Date: Wed, 14 Oct 2009 20:10:56 GMT Message-Id: <200910142010.n9EKAute076100@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/139597: [patch] [tmpfs] tmpfs initializes va_gen but doesn't update it on vnode change X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 20:10:56 -0000 Synopsis: [patch] [tmpfs] tmpfs initializes va_gen but doesn't update it on vnode change Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Wed Oct 14 20:10:47 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=139597 From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 21:05:41 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48168106568B for ; Wed, 14 Oct 2009 21:05:41 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206049004.chello.pl [87.206.49.4]) by mx1.freebsd.org (Postfix) with ESMTP id 8116B8FC20 for ; Wed, 14 Oct 2009 21:05:39 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id B202045E9C; Wed, 14 Oct 2009 23:05:38 +0200 (CEST) Received: from localhost (chello087206049004.chello.pl [87.206.49.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id EA5D445B36; Wed, 14 Oct 2009 23:05:30 +0200 (CEST) Date: Wed, 14 Oct 2009 23:05:28 +0200 From: Pawel Jakub Dawidek To: Solon Lutz Message-ID: <20091014210528.GC1727@garage.freebsd.pl> References: <756365088.20091013005202@pyro.de> <20091013055754.GA3197@garage.freebsd.pl> <90685589.20091013092418@pyro.de> <20091013072712.GA1597@garage.freebsd.pl> <12910471099.20091013095322@pyro.de> <20091013075511.GC1597@garage.freebsd.pl> <1433853337.20091013100348@pyro.de> <20091013082116.GE1597@garage.freebsd.pl> <673550066.20091013133544@pyro.de> <473227349.20091014184731@pyro.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="da4uJneut+ArUgXk" Content-Disposition: inline In-Reply-To: <473227349.20091014184731@pyro.de> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.2 required=4.5 tests=BAYES_00,PLING_QUERY, RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@FreeBSD.ORG Subject: Re: Help needed! ZFS I/O error recovery? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 21:05:41 -0000 --da4uJneut+ArUgXk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 14, 2009 at 06:47:31PM +0200, Solon Lutz wrote: > > pool: temp > > state: ONLINE > > status: One or more devices has experienced an error resulting in data > > corruption. Applications may be affected. > > action: Restore the file in question if possible. Otherwise restore the > > entire pool from backup. > > see: http://www.sun.com/msg/ZFS-8000-8A > > scrub: none requested > > config: >=20 > > NAME STATE READ WRITE CKSUM > > temp ONLINE 0 0 9 > > da0 ONLINE 0 0 36 >=20 > > errors: Permanent errors have been detected in the following files: >=20 > > temp:<0x0> > > temp:<0x49681> > > temp:<0x499a4> > > temp:<0x495fd> > > temp/space1:<0x0> > > temp/space2:<0x0> > > temp/space3:<0x0> > > temp/space4:<0x0> > > temp/space5:<0x0> >=20 > I just tried it with more TXGs, even with a jump of -300, but it always g= ives > an "cannot iterate filesystems: I/O error" error if I try to import the p= ool. >=20 > Also because of mounting the pool, the TXg has gone up to 13445935 from > initially 13462284. We can try to turn off checksum verification entirely, but it will most likely just panic your system. If you want to do this, edit sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h file and change ZIO_CHECKSUM_EQUAL() macro to something like this: #define ZIO_CHECKSUM_EQUAL(zc1, zc2) (1) --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --da4uJneut+ArUgXk Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFK1j0VForvXbEpPzQRAu1TAJ9nax0CU+vG+dSJKDJWYQk8SWYLtACfWZcR IdJpAo2bIRyHNiN1ZYDNpJU= =IDpH -----END PGP SIGNATURE----- --da4uJneut+ArUgXk-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 22:02:34 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1FD2106568F; Wed, 14 Oct 2009 22:02:34 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id 404BD8FC0A; Wed, 14 Oct 2009 22:02:34 +0000 (UTC) Received: by ewy18 with SMTP id 18so262535ewy.43 for ; Wed, 14 Oct 2009 15:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:cc:content-type; bh=bOj3B2qWYfS9Qksn0n6pTLKZ3eMU71mmDxcbec7hk3A=; b=VdzmpkITIVLBfPUl9XTSI8zC8N+8wXIFUKuqb2HYb2RlCFdiV2hPD3/yGSuAS+U8SX 4Xuu3xURzfKobMFm3xe3Sff3JptB7Z/9+TROr2xYzEQF8tiPakH1qu/oTKs4lP+ThBNU bC90w3cLNNyzb19+te2UEjjiov3H0vwcEVB4E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=d7fcLNedH3tGM3/m9lw/DydRPnMF9at0qAVtlaM0tKgyhBP8tPxHoisNgkbYyE7q1T gd3a/UMQuNrMpaGOoCu8PSBv6qOOaudiszsODIer8ZzwCc0LQtu+1hjJAWkT1xeFWbrm 451oZmJLihlnl+pQrWd7pkN2KpnkmXSSze1Q4= MIME-Version: 1.0 Received: by 10.211.130.15 with SMTP id h15mr8050293ebn.82.1255557753166; Wed, 14 Oct 2009 15:02:33 -0700 (PDT) Date: Wed, 14 Oct 2009 18:02:32 -0400 Message-ID: From: grarpamp To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: ZFS repeatable reboot 8.0-RC1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 22:02:34 -0000 Hi. I'm running i386 on i386, single P4 cpu, 1GiB RAM. SiI 3114 -> SATA [single disk] -> GELI [AES-128] -> ZFS [sha256] Straight RELENG_8 as of cvsup Oct 12 14:49:00 aka 8.0-RC1 plus. ZFS pool is at v13, ZFS fs is at v3. Hardware seems stable. The only modification to config defaults is: loader.conf.local: vfs.zfs.arc_max=100663296 After boot -v, geli, zpool import, xf86, browser, etc my mem looks like: Mem: 33M Active, 22M Inact, 105M Wired, 676K Cache, 37M Buf, 827M Free When putting load on ZFS it usually grows to about: Mem: 95M Active, 22M Inact, 302M Wired, 468K Cache, 37M Buf, 569M Free Ls -l in one of the dirs takes 10min plus and I get: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 11 root 171 ki31 0K 8K RUN 21:24 47.27% idle 1092 user 76 0 77328K 76116K zio->i 3:25 37.89% ls 802 root -8 - 0K 8K geli:w 1:42 8.98% g_eli[0] ad6 9 root -8 - 0K 128K arc_re 0:23 4.88% {arc_reclaim_thre} I did not watch these during rm. I have 1 parent dir holding 4 subdirs. The file count in each subdir is respectively: 256363, 254086, 256017, 178054 Two thirds of files are about 14KiB in size, not many are more than a few MiB nor less than 1KiB though a third are 1 byte. I issue rm -r and after maybe 30 seconds the machine reboots. No syslog, panic or console messages. Dmesg from the prior boot is still present in ram to prove kernel didn't emit any message. memtest86 passes. There are maybe 10 seconds of complete GUI hangup before the reboot occurs. I also see it when make release'ing. Usually during what I _think_ is distributeworld or rolling up the tarballs under /R. This is a big repeatable problem. How can I debug or fix it? Can someone else create some mega sized dirs as above and replicate? Thanks. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 22:55:54 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F05601065672; Wed, 14 Oct 2009 22:55:54 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.26]) by mx1.freebsd.org (Postfix) with ESMTP id 66D898FC15; Wed, 14 Oct 2009 22:55:54 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 9so88014eyd.9 for ; Wed, 14 Oct 2009 15:55:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:cc:content-type; bh=i4hDRf+guR0N1BeNCKYiabtliNB6SJkVsj7JX8fAgh0=; b=G5teCFu/W9VUGYKkCtmFQvx8jdInWmWjPlSbeQkz88Xev/FGKex8c2PTC7k1s17vPB bHVp3uIUBrzqgNHgzcWKD+dzgMScHGHlTvJGH7Gzzcq7YnHan62D6+1aKDztiuxO5MDu AkCBKuKBrz/DVu8Et3d6mMaGZIXl1/qkjbQdI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=xzLMYj6BApPFUDPRzbsRwJcc8eJ38ukMdtqqHiHqNaJxg9efEBwke2qMxUtmDkVdG0 V7d8MnJG2J8DRM4sdBv4fkoqQDkPPN0Nx6YyODIIyAJ9Wor6CqHYEsVYgwcy+78Id4ic I3FpMcMM6bengheFK+lgXrvRjK+ujAqQppPjc= MIME-Version: 1.0 Received: by 10.211.128.15 with SMTP id f15mr4367366ebn.84.1255560953602; Wed, 14 Oct 2009 15:55:53 -0700 (PDT) Date: Wed, 14 Oct 2009 18:55:53 -0400 Message-ID: From: grarpamp To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: re: ZFS repeatable reboot 8.0-RC1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 22:55:55 -0000 Happened again :) So some more notes... Watched it this time, it jumped from about 29xMiB straight to 366MiB, then hung and rebooted itself. The first zpool import after reboot takes about a minute more than the usual ten seconds to import. And uses up to maybe 125MiB instead of maybe 40MiB. Could be ZFS fscking itself? Second and further manual reboots are all normal speed and mem use. The fs is fine, I can do all sorts of normal ops on it. Even rm -r , ^C after a few seconds, and repeat ad nauseum until the tree is removed. Only continuous rm -r reboots. I have 1 more GiB RAM I can put in. Sorry to break threads, I'm not on list. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 14 23:11:44 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 741581065679; Wed, 14 Oct 2009 23:11:43 +0000 (UTC) (envelope-from solon@pyro.de) Received: from srv23.fsb.echelon.bnd.org (mail.pyro.de [83.137.99.96]) by mx1.freebsd.org (Postfix) with ESMTP id 0B81A8FC12; Wed, 14 Oct 2009 23:11:43 +0000 (UTC) Received: from port-87-193-183-44.static.qsc.de ([87.193.183.44] helo=flash.home) by srv23.fsb.echelon.bnd.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1MyD0N-0004mU-1U; Thu, 15 Oct 2009 01:11:42 +0200 Date: Thu, 15 Oct 2009 01:11:37 +0200 From: Solon Lutz X-Mailer: The Bat! (v3.99.25) Professional Organization: pyro.labs berlin X-Priority: 3 (Normal) Message-ID: <572088116.20091015011137@pyro.de> To: Pawel Jakub Dawidek , freebsd-fs@FreeBSD.ORG In-Reply-To: <20091014210528.GC1727@garage.freebsd.pl> References: <756365088.20091013005202@pyro.de> <20091013055754.GA3197@garage.freebsd.pl> <90685589.20091013092418@pyro.de> <20091013072712.GA1597@garage.freebsd.pl> <12910471099.20091013095322@pyro.de> <20091013075511.GC1597@garage.freebsd.pl> <1433853337.20091013100348@pyro.de> <20091013082116.GE1597@garage.freebsd.pl> <673550066.20091013133544@pyro.de> <473227349.20091014184731@pyro.de> <20091014210528.GC1727@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Score: -0.1 (/) X-Spam-Report: Spam detection software, running on the system "srv23.fsb.echelon.bnd.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see The administrator of that system for details. Content preview: >> I just tried it with more TXGs, even with a jump of -300, but it always gives >> an "cannot iterate filesystems: I/O error" error if I try to import the pool. >> Also because of mounting the pool, the TXg has gone up to 13445935 from >> initially 13462284. [...] Content analysis details: (-0.1 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.4 ALL_TRUSTED Passed through trusted hosts only via SMTP 1.3 PLING_QUERY Subject has exclamation mark and question mark X-Spam-Flag: NO Cc: Subject: Re: Help needed! ZFS I/O error recovery? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2009 23:11:44 -0000 >> I just tried it with more TXGs, even with a jump of -300, but it always gives >> an "cannot iterate filesystems: I/O error" error if I try to import the pool. >> Also because of mounting the pool, the TXg has gone up to 13445935 from >> initially 13462284. > We can try to turn off checksum verification entirely, but it will most > likely just panic your system. > If you want to do this, edit > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h file and change > ZIO_CHECKSUM_EQUAL() macro to something like this: > #define ZIO_CHECKSUM_EQUAL(zc1, zc2) (1) Yes, it paniced as soon as I tried zpool import. Can you tell where ZFS is taking the information from, that there are I/O errors? Is this based on checksums or is there some kind of I/O-error-flag? solon From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 01:35:59 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44312106568D for ; Thu, 15 Oct 2009 01:35:59 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-qy0-f203.google.com (mail-qy0-f203.google.com [209.85.221.203]) by mx1.freebsd.org (Postfix) with ESMTP id E82288FC0A for ; Thu, 15 Oct 2009 01:35:58 +0000 (UTC) Received: by qyk42 with SMTP id 42so323366qyk.28 for ; Wed, 14 Oct 2009 18:35:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=zQ1chPmXiWhcnxkRCkYFa/FFGhseZ2c37OJh3QgweOA=; b=sjHGzuVI0G6T8X5x2zFjCFyrRLViBDcTFQ5w9GvxWdmpF/PKMLGWUvDlVOIiLzLzzC i6lJrsZtXSEnWcJPP1R+9q+wZHSVC8iSBpOpx/+4enqNkOjl4tnd1dECgEBQH3qN0mi5 rbjGECBdDVYegl3twJyMEtgrKRa+d3CKWwiTg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=WkPE75CD48AVEajnar8/eBruQjCXSiBlMOCdI7noeV1ftbCZ1F1I0dHUyvcbWKUWsw adJM7pZTL5zEUvJYsxzjiIlVBma2xkthOQiizPJr7Zi7U3wqfV2EgI1q7/XeHf2LceNK rYvukdEd4Kj2SyIlr4AuXJH5L87l/WSx1nI6w= Received: by 10.224.100.13 with SMTP id w13mr7706460qan.292.1255569313522; Wed, 14 Oct 2009 18:15:13 -0700 (PDT) Received: from dimension.5p.local (adsl-99-19-46-209.dsl.klmzmi.sbcglobal.net [99.19.46.209]) by mx.google.com with ESMTPS id 6sm7812506qwd.45.2009.10.14.18.15.11 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 14 Oct 2009 18:15:11 -0700 (PDT) Sender: "J. Hellenthal" Date: Wed, 14 Oct 2009 21:15:09 -0400 From: jhell To: grarpamp In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: re: ZFS repeatable reboot 8.0-RC1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 01:35:59 -0000 On Wed, 14 Oct 2009 18:55, grarpamp@ wrote: > Happened again :) So some more notes... > > Watched it this time, it jumped from about 29xMiB straight to 366MiB, > then hung and rebooted itself. > > The first zpool import after reboot takes about a minute more than > the usual ten seconds to import. And uses up to maybe 125MiB instead > of maybe 40MiB. Could be ZFS fscking itself? Second and further > manual reboots are all normal speed and mem use. > > The fs is fine, I can do all sorts of normal ops on it. Even rm -r > , ^C after a few seconds, and repeat ad nauseum until the > tree is removed. Only continuous rm -r reboots. > > I have 1 more GiB RAM I can put in. > > Sorry to break threads, I'm not on list. > Your machine is starving! what you can do to improve upon performance of the pool is add a separate disk to the machine in which you can configure your ZIL(ZFS Intent Log) and a Cache. Those two things can improve greatly upon performance and reduce eating up so much ram that your system starts to starve itself. Add that other 1G of RAM if you have it just sitting around then put it to good use it will certainly help. The disk that your doing your remove operation on ? is that being done on a ZFS GELI ? PS: You can use thumb drives as caches and intent logs but beware I have had them stop and disappear in my system to only appear again on reboot. -- ;; dataix.net!jhell 2048R/89D8547E 2009-09-30 ;; BSD since FreeBSD 4.2 Linux since Slackware 2.1 ;; 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 06:47:02 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F33B106566C; Thu, 15 Oct 2009 06:47:02 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mx1.freebsd.org (Postfix) with ESMTP id 8A7778FC17; Thu, 15 Oct 2009 06:47:01 +0000 (UTC) Received: by ewy18 with SMTP id 18so531096ewy.43 for ; Wed, 14 Oct 2009 23:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=cwSpgvA4HMkQKW/3/NVKxzSgO/3R0Uzvsfkh3Yq64Nk=; b=JD5QKMXEPYWxt4Q/8jRGL9NSbSLHW/JK/czXtwGIBPJt+QWGbU3bC5v21tCJt4ByPq 1jOCmYYZxuCDFXmh4z8Mrb2Utb1IYJ29jxp32WyvGYK+xxlggg3IQUMmSrkPuAuSVYXB W7dAClA41OW9pwdknLgEW+YVdkL+gtjz/tEJ8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=fPqG8WLQZjZ8DCXbp8bQ9DFZHTKM1ZN1KeTjxfRdgTTW22oVkGs+mqbZBmiBPl8EJl NarGB9mY48svls/EEj9hdLbg0Kv8LikWl2CIXTEfkQgrFXvw/G2Hg3+FV9JwkMWX3Hmn xki2qSeOPwUlKJdi+E6L323xYm78L0C5SxklE= MIME-Version: 1.0 Received: by 10.211.132.28 with SMTP id j28mr11565532ebn.95.1255589220505; Wed, 14 Oct 2009 23:47:00 -0700 (PDT) In-Reply-To: References: Date: Thu, 15 Oct 2009 02:47:00 -0400 Message-ID: From: grarpamp To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS repeatable reboot 8.0-RC1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 06:47:02 -0000 > Your machine is starving! How can this be, there is over 500MiB free ram at all times? I'm running literally no userland apps other than X and xterms when it reboots. I think I may be hitting some limit with this 366MiB and reboot bit. How can I tell what my kernel limits are on this platform? Don't I have to limit ARC to ( kern + kern headroom + ARC = kern addressablility limit ) or something like that? Anything I should use for that besides sysctl -a and vmstat -z? I thought I looked at my wired history before using zfs and set arc_max to 96MiB so wired wouldn't even get close 512. > what you can do to improve upon performance of the pool Performance is not the problem. Yes, it's dog slow, but it's usable, sortof :) The issue is that it's rebooting spontaneously. No OS should do that. Though unlike a userland process that the kernel kills when out of ram, I don't know how the kernel would recover when its own processes bloat up. I expect slow performance with this setup. Especially if I'm blowing out some cache somewhere. Take UFS2 with dirhash for example. If the size of the directory inode is much bigger than vfs.ufs.dirhash_maxmem, it just slows down to spindle speed ... not reboot, no big deal. > is add a separate disk to the machine in which you can configure > your ZIL(ZFS Intent Log) and a Cache. Those two things can ... > reduce eating up so much ram that your system starts to starve > itself. Reduce ram?, how so? I already have a ZIL in the main pool by default, presumably using just as much ram as a separate one would, so no need for a separate log. Similarly for cache, which is in core in the default case. They just help speed if on 'faster than spindles' devices. I suppose I could as easily set vfs.zfs.zil_disable=0 as a test if it wouldn't risk loss of the entire pool. > Add that other 1G of RAM Isn't that a game of whack a mole? What happens when I rm a dir with 1M files in it? Add more ram? 2M files? ... > The disk that your doing your remove operation on ? is that being > done on a ZFS GELI ? As mentioned, yes. > PS: You can use thumb drives as caches and intent logs I would presume their bit error rate is higher than platters. There was a time when SSD's meant RAM drives, not flash drives. # vmstat -m | egrep -i 'requests|zfs|zil|zio|arc|solaris|geom|eli' Type InUse MemUse HighUse Requests Size(s) GEOM 208 27K - 1708 16,32,64,128,512,1024,2048,4096 solaris 145673 135033K - 3413390 16,32,64,128,256,512,1024,2048,4096 eli data 8 5K - 23628 32,256,512,1024,2048,4096 # vmstat -z | egrep -i 'requests|zfs|zil|zio|arc|solaris|geom|eli' ITEM SIZE LIMIT USED FREE REQUESTS FAILURES zio_cache: 596, 0, 0, 4596, 489861, 0 arc_buf_hdr_t: 136, 0, 9367, 29, 15769, 0 arc_buf_t: 40, 0, 2128, 264, 19641, 0 zil_lwb_cache: 176, 0, 3, 85, 488, 0 zfs_znode_cache: 232, 0, 19298, 473, 62000, 0 # sysctl -a vfs.zfs kstat.zfs vfs.zfs.arc_meta_limit: 25165824 vfs.zfs.arc_meta_used: 39459076 vfs.zfs.mdcomp_disable: 0 vfs.zfs.arc_min: 16777216 vfs.zfs.arc_max: 100663296 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 1 vfs.zfs.recover: 0 vfs.zfs.txg.synctime: 5 vfs.zfs.txg.timeout: 30 vfs.zfs.scrub_limit: 10 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 35 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.version.zpl: 3 vfs.zfs.version.vdev_boot: 1 vfs.zfs.version.spa: 13 vfs.zfs.version.dmu_backup_stream: 1 vfs.zfs.version.dmu_backup_header: 2 vfs.zfs.version.acl: 1 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 kstat.zfs.misc.arcstats.hits: 102514 kstat.zfs.misc.arcstats.misses: 12662 kstat.zfs.misc.arcstats.demand_data_hits: 8150 kstat.zfs.misc.arcstats.demand_data_misses: 741 kstat.zfs.misc.arcstats.demand_metadata_hits: 94364 kstat.zfs.misc.arcstats.demand_metadata_misses: 11921 kstat.zfs.misc.arcstats.prefetch_data_hits: 0 kstat.zfs.misc.arcstats.prefetch_data_misses: 0 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0 kstat.zfs.misc.arcstats.mru_hits: 49617 kstat.zfs.misc.arcstats.mru_ghost_hits: 2511 kstat.zfs.misc.arcstats.mfu_hits: 52897 kstat.zfs.misc.arcstats.mfu_ghost_hits: 1193 kstat.zfs.misc.arcstats.deleted: 1429 kstat.zfs.misc.arcstats.recycle_miss: 5314 kstat.zfs.misc.arcstats.mutex_miss: 0 kstat.zfs.misc.arcstats.evict_skip: 3645 kstat.zfs.misc.arcstats.hash_elements: 9362 kstat.zfs.misc.arcstats.hash_elements_max: 9363 kstat.zfs.misc.arcstats.hash_collisions: 8135 kstat.zfs.misc.arcstats.hash_chains: 2042 kstat.zfs.misc.arcstats.hash_chain_max: 5 kstat.zfs.misc.arcstats.p: 57449472 kstat.zfs.misc.arcstats.c: 100663296 kstat.zfs.misc.arcstats.c_min: 16777216 kstat.zfs.misc.arcstats.c_max: 100663296 kstat.zfs.misc.arcstats.size: 99728132 kstat.zfs.misc.arcstats.hdr_size: 1273504 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 0 kstat.zfs.misc.vdev_cache_stats.delegations: 961 kstat.zfs.misc.vdev_cache_stats.hits: 8593 kstat.zfs.misc.vdev_cache_stats.misses: 3377 From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 12:28:20 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24F9D1065679; Thu, 15 Oct 2009 12:28:20 +0000 (UTC) (envelope-from valin@buchlovice.org) Received: from smtp-sfn.sitkom.cz (smtp-sfn.sitkom.cz [88.146.175.4]) by mx1.freebsd.org (Postfix) with ESMTP id DD0148FC0C; Thu, 15 Oct 2009 12:28:19 +0000 (UTC) Received: from osiris.buchlovice.sfn (valin-osiris-lan.brestek.sfn [10.8.20.3]) by smtp-sfn.sitkom.cz (Postfix) with ESMTP id 96F851FA69E; Thu, 15 Oct 2009 14:08:55 +0200 (CEST) Message-ID: <4AD710D6.70404@buchlovice.org> Date: Thu, 15 Oct 2009 14:08:54 +0200 From: =?UTF-8?B?UmFkZWsgVmFsw6HFoWVr?= User-Agent: Thunderbird 2.0.0.23 (X11/20091015) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 12:28:20 -0000 Hi, I want to ask if there is something new in adding support to gptzfsboot/zfsboot for reading gang-blocks? From Sun's docs: Gang blocks When there is not enough contiguous space to write a complete block, the ZIO pipeline will break the I/O up into smaller 'gang blocks' which can later be assembled transparently to appear as complete blocks. Everything works fine for me, until I rewrite kernel/world after system upgrade to latest one (releng_8). After this am I no longer able to boot from zfs raidz1 pool with following messages: >/ ZFS: i/o error - all block copies unavailable />/ ZFS: can't read MOS />/ ZFS: unexpected object set type lld />/ ZFS: unexpected object set type lld />/ />/ FreeBSD/i386 boot />/ Default: z:/boot/kernel/kernel />/ boot: />/ ZFS: unexpected object set type lld />/ />/ FreeBSD/i386 boot />/ Default: tank:/boot/kernel/kernel />/ boot: // /I presume it's the same issue as talked in june-2009 current mailing list http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html Any success in that matter? Thnx for answer. vaLin From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 13:19:54 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 672CA1065676 for ; Thu, 15 Oct 2009 13:19:54 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id E8F958FC18 for ; Thu, 15 Oct 2009 13:19:53 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1MyQEz-0002Mg-5O for freebsd-fs@freebsd.org; Thu, 15 Oct 2009 15:19:37 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 15 Oct 2009 15:19:37 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 15 Oct 2009 15:19:37 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Thu, 15 Oct 2009 15:19:17 +0200 Lines: 46 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.23 (X11/20090928) Sender: news Subject: ZFS threads? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 13:19:54 -0000 A VM running 8.0-RC1 with ZFS has load average of 18 while doing rsync to it. I notice there are two sets of 8 ZFS kernel threads present on the system, which look like might be the cause of the high load. Is there a way to reduce the number of these worker threads? 0 root -16 0 0K 280K - 11:47 3.96% {spa_zio_6} 0 root -16 0 0K 280K - 12:34 0.00% {spa_zio_5} 0 root -16 0 0K 280K - 12:11 0.00% {spa_zio_3} 0 root -16 0 0K 280K - 11:43 0.00% {spa_zio_1} 0 root -16 0 0K 280K - 11:24 0.00% {spa_zio_0} 0 root -16 0 0K 280K - 11:20 0.00% {spa_zio_7} 0 root -16 0 0K 280K - 11:19 0.00% {spa_zio_2} 0 root -16 0 0K 280K - 10:54 0.00% {spa_zio_4} 4 root -8 - 0K 8K - 3:10 0.00% [g_down] 0 root -16 0 0K 280K - 2:49 0.00% {spa_zio} 12 root -40 - 0K 128K WAIT 1:31 0.00% {swi2: cambio} 12 root -64 - 0K 128K WAIT 0:42 0.00% {irq17: mpt0} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_7} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_3} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_6} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_1} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_5} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_0} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_2} 0 root -16 0 0K 280K - 0:38 0.00% {spa_zio_4} From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 13:53:34 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB473106566B for ; Thu, 15 Oct 2009 13:53:34 +0000 (UTC) (envelope-from rnoland@FreeBSD.org) Received: from gizmo.2hip.net (gizmo.2hip.net [64.74.207.195]) by mx1.freebsd.org (Postfix) with ESMTP id 9EA158FC2A for ; Thu, 15 Oct 2009 13:53:34 +0000 (UTC) Received: from [192.168.1.4] (adsl-154-199-198.ard.bellsouth.net [72.154.199.198]) (authenticated bits=0) by gizmo.2hip.net (8.14.3/8.14.3) with ESMTP id n9FDEU3o031872 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 15 Oct 2009 09:14:32 -0400 (EDT) (envelope-from rnoland@FreeBSD.org) From: Robert Noland To: Radek =?iso-8859-2?Q?Val=E1=B9ek?= In-Reply-To: <4AD710D6.70404@buchlovice.org> References: <4AD710D6.70404@buchlovice.org> Content-Type: multipart/mixed; boundary="=-fMS+j1G984RTPd8ikE1K" Organization: FreeBSD Date: Thu, 15 Oct 2009 08:14:25 -0500 Message-Id: <1255612465.2356.808.camel@balrog.2hip.net> Mime-Version: 1.0 X-Mailer: Evolution 2.26.3 FreeBSD GNOME Team Port X-Spam-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00, RCVD_IN_SORBS_DUL, RDNS_DYNAMIC, SPF_SOFTFAIL autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gizmo.2hip.net Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 13:53:35 -0000 --=-fMS+j1G984RTPd8ikE1K Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: 8bit On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote: > Hi, > > I want to ask if there is something new in adding support to > gptzfsboot/zfsboot for reading gang-blocks? I've been thinking of trying to fix this, but haven't really come up with a repeatable way to test it. I might be able to come up with at least a hack to allow booting in the short term, but if you can try this patch so that we can verify that the issue is indeed gang blocks. This doesn't fix anything yet, but it should report when it finds a gang block. I know that it is tricky to test when you can't boot, but if you can apply this patch and reinstall gptzfsboot, it should tell us for sure that gang blocks are the issue. I assume that you have a partition layout something like mine: balrog% gpart show => 34 1953525101 ada0 GPT (932G) 34 128 1 freebsd-boot (64K) 162 8388608 2 freebsd-swap (4.0G) 8388770 1945136365 3 freebsd-zfs (928G) If so, all you should need to do is get this built and then: #gpart bootcode -p /boot/gptzfsboot -i 1 ada0 substituting appropriate partition index and device info obviously. robert. > From Sun's docs: > > Gang blocks > > When there is not enough contiguous space to write a complete block, the ZIO > pipeline will break the I/O up into smaller 'gang blocks' which can later be > assembled transparently to appear as complete blocks. > > Everything works fine for me, until I rewrite kernel/world after system > upgrade to latest one (releng_8). After this am I no longer able to boot > from zfs raidz1 pool with following messages: > > >/ ZFS: i/o error - all block copies unavailable > />/ ZFS: can't read MOS > />/ ZFS: unexpected object set type lld > />/ ZFS: unexpected object set type lld > />/ > />/ FreeBSD/i386 boot > />/ Default: z:/boot/kernel/kernel > />/ boot: > />/ ZFS: unexpected object set type lld > />/ > />/ FreeBSD/i386 boot > />/ Default: tank:/boot/kernel/kernel > />/ boot: > // > /I presume it's the same issue as talked in june-2009 current mailing > list > http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html > > Any success in that matter? > > Thnx for answer. > > vaLin > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- Robert Noland FreeBSD --=-fMS+j1G984RTPd8ikE1K Content-Disposition: attachment; filename="zfs-report-gb.patch" Content-Type: text/x-patch; name="zfs-report-gb.patch"; charset="us-ascii" Content-Transfer-Encoding: 7bit diff --git a/sys/boot/zfs/zfsimpl.c b/sys/boot/zfs/zfsimpl.c index ff567a4..a2893bf 100644 --- a/sys/boot/zfs/zfsimpl.c +++ b/sys/boot/zfs/zfsimpl.c @@ -920,6 +920,11 @@ zio_read(spa_t *spa, const blkptr_t *bp, void *buf) if (!dva->dva_word[0] && !dva->dva_word[1]) continue; + if (DVA_GET_GANG(dva)) { + printf("ZFS: i/o error - gang block unimplemented!\n"); + continue; + } + vdevid = DVA_GET_VDEV(dva); offset = DVA_GET_OFFSET(dva); STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink) --=-fMS+j1G984RTPd8ikE1K-- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 17:42:07 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B421106566C for ; Thu, 15 Oct 2009 17:42:07 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206049004.chello.pl [87.206.49.4]) by mx1.freebsd.org (Postfix) with ESMTP id C60718FC39 for ; Thu, 15 Oct 2009 17:42:06 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 4ECDF45F89; Thu, 15 Oct 2009 19:42:05 +0200 (CEST) Received: from localhost (chello087206049004.chello.pl [87.206.49.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id EA7B445EE6; Thu, 15 Oct 2009 19:41:59 +0200 (CEST) Date: Thu, 15 Oct 2009 19:42:00 +0200 From: Pawel Jakub Dawidek To: Solon Lutz Message-ID: <20091015174200.GB1880@garage.freebsd.pl> References: <90685589.20091013092418@pyro.de> <20091013072712.GA1597@garage.freebsd.pl> <12910471099.20091013095322@pyro.de> <20091013075511.GC1597@garage.freebsd.pl> <1433853337.20091013100348@pyro.de> <20091013082116.GE1597@garage.freebsd.pl> <673550066.20091013133544@pyro.de> <473227349.20091014184731@pyro.de> <20091014210528.GC1727@garage.freebsd.pl> <572088116.20091015011137@pyro.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="mxv5cy4qt+RJ9ypb" Content-Disposition: inline In-Reply-To: <572088116.20091015011137@pyro.de> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.2 required=4.5 tests=BAYES_00,PLING_QUERY, RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@FreeBSD.ORG Subject: Re: Help needed! ZFS I/O error recovery? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 17:42:07 -0000 --mxv5cy4qt+RJ9ypb Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 15, 2009 at 01:11:37AM +0200, Solon Lutz wrote: > >> I just tried it with more TXGs, even with a jump of -300, but it alway= s gives > >> an "cannot iterate filesystems: I/O error" error if I try to import th= e pool. >=20 > >> Also because of mounting the pool, the TXg has gone up to 13445935 from > >> initially 13462284. >=20 > > We can try to turn off checksum verification entirely, but it will most > > likely just panic your system. >=20 > > If you want to do this, edit > > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h file and change > > ZIO_CHECKSUM_EQUAL() macro to something like this: >=20 > > #define ZIO_CHECKSUM_EQUAL(zc1, zc2) (1) >=20 > Yes, it paniced as soon as I tried zpool import. Can you tell where ZFS i= s taking > the information from, that there are I/O errors? Is this based on checksu= ms or is > there some kind of I/O-error-flag? Checksum mismatch is reported as EIO error. Not sure what else we can try. I guess you are not able to snapshot problematic datasets? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --mxv5cy4qt+RJ9ypb Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFK117nForvXbEpPzQRAvYhAJkBAzP9CCdc+vxSNROLLHCwELyIlwCgh0Tk qCjt7yqW/WRDs6OyDi5uf10= =YbqB -----END PGP SIGNATURE----- --mxv5cy4qt+RJ9ypb-- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 19:04:00 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51CAD1065672; Thu, 15 Oct 2009 19:04:00 +0000 (UTC) (envelope-from rnoland@FreeBSD.org) Received: from gizmo.2hip.net (gizmo.2hip.net [64.74.207.195]) by mx1.freebsd.org (Postfix) with ESMTP id 1895D8FC20; Thu, 15 Oct 2009 19:04:00 +0000 (UTC) Received: from [192.168.1.4] (adsl-154-199-198.ard.bellsouth.net [72.154.199.198]) (authenticated bits=0) by gizmo.2hip.net (8.14.3/8.14.3) with ESMTP id n9FJ3uDU033665 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 15 Oct 2009 15:03:56 -0400 (EDT) (envelope-from rnoland@FreeBSD.org) From: Robert Noland To: Radek =?iso-8859-2?Q?Val=E1=B9ek?= In-Reply-To: <4AD710D6.70404@buchlovice.org> References: <4AD710D6.70404@buchlovice.org> Content-Type: multipart/mixed; boundary="=-a2tiU1GGu1eRIpNGphrs" Organization: FreeBSD Date: Thu, 15 Oct 2009 14:03:50 -0500 Message-Id: <1255633430.2175.12.camel@balrog.2hip.net> Mime-Version: 1.0 X-Mailer: Evolution 2.26.3 FreeBSD GNOME Team Port X-Spam-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00, RCVD_IN_SORBS_DUL, RDNS_DYNAMIC, SPF_SOFTFAIL autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gizmo.2hip.net Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 19:04:00 -0000 --=-a2tiU1GGu1eRIpNGphrs Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: 8bit On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote: > Hi, > > I want to ask if there is something new in adding support to > gptzfsboot/zfsboot for reading gang-blocks? Ok, I can't figure out any way to test this... beyond the fact that it builds and doesn't break my currently working setup. Can you give this a try? It should still report if it finds gang blocks, but hopefully now will read them as well. robert. > From Sun's docs: > > Gang blocks > > When there is not enough contiguous space to write a complete block, the ZIO > pipeline will break the I/O up into smaller 'gang blocks' which can later be > assembled transparently to appear as complete blocks. > > Everything works fine for me, until I rewrite kernel/world after system > upgrade to latest one (releng_8). After this am I no longer able to boot > from zfs raidz1 pool with following messages: > > >/ ZFS: i/o error - all block copies unavailable > />/ ZFS: can't read MOS > />/ ZFS: unexpected object set type lld > />/ ZFS: unexpected object set type lld > />/ > />/ FreeBSD/i386 boot > />/ Default: z:/boot/kernel/kernel > />/ boot: > />/ ZFS: unexpected object set type lld > />/ > />/ FreeBSD/i386 boot > />/ Default: tank:/boot/kernel/kernel > />/ boot: > // > /I presume it's the same issue as talked in june-2009 current mailing > list > http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html > > Any success in that matter? > > Thnx for answer. > > vaLin > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" -- Robert Noland FreeBSD --=-a2tiU1GGu1eRIpNGphrs Content-Disposition: attachment; filename="zfs-gang-block.patch" Content-Type: text/x-patch; name="zfs-gang-block.patch"; charset="us-ascii" Content-Transfer-Encoding: 7bit diff --git a/sys/boot/zfs/zfsimpl.c b/sys/boot/zfs/zfsimpl.c index ff567a4..6a18b44 100644 --- a/sys/boot/zfs/zfsimpl.c +++ b/sys/boot/zfs/zfsimpl.c @@ -53,6 +53,8 @@ static char *zfs_temp_buf, *zfs_temp_end, *zfs_temp_ptr; #define TEMP_SIZE (1*SPA_MAXBLOCKSIZE) +static int zio_read(spa_t *spa, const blkptr_t *bp, void *buf); + static void zfs_init(void) { @@ -897,6 +899,33 @@ ilog2(int n) } static int +zio_read_gang(spa_t *spa, const blkptr_t *bp, const dva_t *dva, void *buf) +{ + zio_gbh_phys_t zio_gb; + vdev_t *vdev; + int vdevid; + off_t offset; + int i; + + vdevid = DVA_GET_VDEV(dva); + offset = DVA_GET_OFFSET(dva); + STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink) + if (vdev->v_id == vdevid) + break; + if (!vdev || !vdev->v_read) + return (EIO); + if (vdev->v_read(vdev, bp, &zio_gb, offset, SPA_GANGBLOCKSIZE)) + return (EIO); + + for (i = 0; i < SPA_GBH_NBLKPTRS; i++) { + if (zio_read(spa, &zio_gb.zg_blkptr[i], buf)) + return (EIO); + } + + return (0); +} + +static int zio_read(spa_t *spa, const blkptr_t *bp, void *buf) { int cpfunc = BP_GET_COMPRESS(bp); @@ -920,20 +949,26 @@ zio_read(spa_t *spa, const blkptr_t *bp, void *buf) if (!dva->dva_word[0] && !dva->dva_word[1]) continue; - vdevid = DVA_GET_VDEV(dva); - offset = DVA_GET_OFFSET(dva); - STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink) - if (vdev->v_id == vdevid) - break; - if (!vdev || !vdev->v_read) - continue; - if (vdev->v_read(vdev, bp, pbuf, offset, psize)) - continue; + if (DVA_GET_GANG(dva)) { + printf("ZFS: gang block detected!\n"); + if (zio_read_gang(spa, bp, dva, buf)) + return (EIO); + } else { + vdevid = DVA_GET_VDEV(dva); + offset = DVA_GET_OFFSET(dva); + STAILQ_FOREACH(vdev, &spa->spa_vdevs, v_childlink) + if (vdev->v_id == vdevid) + break; + if (!vdev || !vdev->v_read) + continue; + if (vdev->v_read(vdev, bp, pbuf, offset, psize)) + continue; - if (cpfunc != ZIO_COMPRESS_OFF) { - if (zio_decompress_data(cpfunc, pbuf, psize, - buf, lsize)) - return (EIO); + if (cpfunc != ZIO_COMPRESS_OFF) { + if (zio_decompress_data(cpfunc, pbuf, psize, + buf, lsize)) + return (EIO); + } } return (0); diff --git a/sys/cddl/boot/zfs/zfsimpl.h b/sys/cddl/boot/zfs/zfsimpl.h index a0b7b72..688bb5c 100644 --- a/sys/cddl/boot/zfs/zfsimpl.h +++ b/sys/cddl/boot/zfs/zfsimpl.h @@ -374,6 +374,24 @@ typedef struct vdev_label { #define VDEV_LABEL_END_SIZE (2 * sizeof (vdev_label_t)) #define VDEV_LABELS 4 +/* + * Gang block headers are self-checksumming and contain an array + * of block pointers. + */ +#define SPA_GANGBLOCKSIZE SPA_MINBLOCKSIZE +#define SPA_GBH_NBLKPTRS ((SPA_GANGBLOCKSIZE - \ + sizeof (zio_block_tail_t)) / sizeof (blkptr_t)) +#define SPA_GBH_FILLER ((SPA_GANGBLOCKSIZE - \ + sizeof (zio_block_tail_t) - \ + (SPA_GBH_NBLKPTRS * sizeof (blkptr_t))) /\ + sizeof (uint64_t)) + +typedef struct zio_gbh { + blkptr_t zg_blkptr[SPA_GBH_NBLKPTRS]; + uint64_t zg_filler[SPA_GBH_FILLER]; + zio_block_tail_t zg_tail; +} zio_gbh_phys_t; + enum zio_checksum { ZIO_CHECKSUM_INHERIT = 0, ZIO_CHECKSUM_ON, --=-a2tiU1GGu1eRIpNGphrs-- From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 19:10:04 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96BC11065670 for ; Thu, 15 Oct 2009 19:10:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6D9598FC12 for ; Thu, 15 Oct 2009 19:10:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9FJA4ha094517 for ; Thu, 15 Oct 2009 19:10:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9FJA4Ot094516; Thu, 15 Oct 2009 19:10:04 GMT (envelope-from gnats) Date: Thu, 15 Oct 2009 19:10:04 GMT Message-Id: <200910151910.n9FJA4Ot094516@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Alexander Haderer Cc: Subject: Re: kern/136470: [nfs] Cannot mount / in read-only, over NFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Alexander Haderer List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 19:10:04 -0000 The following reply was made to PR kern/136470; it has been noted by GNATS. From: Alexander Haderer To: bug-followup@FreeBSD.org, admin@lissyara.su Cc: Subject: Re: kern/136470: [nfs] Cannot mount / in read-only, over NFS Date: Thu, 15 Oct 2009 20:53:12 +0200 hello, I submitted a new PR because this problem comes from a bug in the 'mount' command, see PR bin/139651 with best regards, Alexander From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 19:17:46 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0674C106568B for ; Thu, 15 Oct 2009 19:17:46 +0000 (UTC) (envelope-from tzim@tzim.net) Received: from golanth.tzim.net (unknown [IPv6:2001:41d0:1:d91f:21c:c0ff:fe4b:cf32]) by mx1.freebsd.org (Postfix) with ESMTP id 958848FC13 for ; Thu, 15 Oct 2009 19:17:45 +0000 (UTC) Received: from carenath.tzim.net ([82.67.108.3] helo=[192.168.0.200]) by golanth.tzim.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1MyVpY-00013b-HQ for freebsd-fs@freebsd.org; Thu, 15 Oct 2009 21:17:44 +0200 Message-ID: <4AD77556.7050607@tzim.net> Date: Thu, 15 Oct 2009 21:17:42 +0200 From: Arnaud Houdelette User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Authenticated-User: tzim@tzim.net X-Authenticator: plain Subject: ZFS send from FreeBSD 7.1 (v6) to 8.0 (v13). X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 19:17:46 -0000 Hi. I use ZFS send/receive to do daily (or nightly) remote replication from a dedicated server (at ISP's ) to my Home server. Dedicated server use 7.1-RELEASE. Home server is 7.2-RELEASE. Each night, a snapshot is taken on the dedicated server and an incremental send used to replicate the content to my Home server. No problem so far. With incoming 8.0, I'd like to upgrade my home server (and maybe upgrade pool to v13) before even thinking to do anything to the dedicated server. Will the send/receive scheme continue to work ? Regards, Arnaud From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 19:18:02 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1959110656C1 for ; Thu, 15 Oct 2009 19:18:02 +0000 (UTC) (envelope-from stef-list@memberwebs.com) Received: from memberwebs.com (memberwebs.com [94.75.203.95]) by mx1.freebsd.org (Postfix) with ESMTP id DA94B8FC23 for ; Thu, 15 Oct 2009 19:18:01 +0000 (UTC) Received: from [172.27.5.159] (unknown [172.27.5.159]) by memberwebs.com (Postfix) with ESMTP id 7761B83E4C8 for ; Thu, 15 Oct 2009 19:04:21 +0000 (UTC) Message-ID: <4AD77230.3030803@memberwebs.com> Date: Thu, 15 Oct 2009 14:04:16 -0500 From: Stef Walter User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Deadlock after canceled zfs recv X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: stef@memberwebs.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 19:18:02 -0000 I'm running the latest RELENG_8, and been doing some pre-production stress testing. I can deadlock (reproduceable) after a canceled ssh + zfs recv. Here's how I reproduce the problem: Do this in a new tank without any data in it. Reboot the system, and make this the first zfs operations done. Files available here: http://memberwebs.com/stef/misc/recv-snapshots-zfs-hang.tbz Receive new file system, and then incremental snapshot: # cat step-one | zfs recv tank/received # cat step-two | zfs recv tank/received At this point should look like: # zfs list -t snapshot,filesystem | grep received tank 2.35G 16.4G 22K /tank tank/received 491M 16.4G 489M /tank/received tank/received@justnow 1.32M - 160M - tank/received@later 0 - 489M - The third one goes through ssh. Count about three to five seconds (one one thousand, two one thousand, three one thousand) and press Ctrl-C # cat step-three | ssh localhost zfs recv tank/received Execute the above 'zfs list' command, and more often than not, parts of the zfs system are hung, and remain deadlocked until reboot. If it doesn't happen the first time, try the step three + ctrl-c again. When run through ssh, and ctrl-c cancelled it seems like 'zfs recv' doesn't have time to do the cleanup that it normally does if run directly. FreeBSD zfs8.ws.local 8.0-RC1 FreeBSD 8.0-RC1 #0: Wed Oct 14 16:04:50 UTC 2009 root@zfs8.ws.local:/usr/obj/usr/src/sys/GENERIC i386 I'm available for any further information and want to help nail down this bug. Cheers, Stef From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 19:31:45 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1658106566C; Thu, 15 Oct 2009 19:31:45 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 89A5D8FC21; Thu, 15 Oct 2009 19:31:45 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9FJVjkn020998; Thu, 15 Oct 2009 19:31:45 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9FJVjU9020994; Thu, 15 Oct 2009 19:31:45 GMT (envelope-from linimon) Date: Thu, 15 Oct 2009 19:31:45 GMT Message-Id: <200910151931.n9FJVjU9020994@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: bin/139651: [nfs] mount(8): read-only remount of NFS volume does not work X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 19:31:45 -0000 Old Synopsis: mount: read-only remount of NFS volume does not work New Synopsis: [nfs] mount(8): read-only remount of NFS volume does not work Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Thu Oct 15 19:30:48 UTC 2009 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=139651 From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 19:37:35 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A53BE106566C; Thu, 15 Oct 2009 19:37:35 +0000 (UTC) (envelope-from valin@buchlovice.org) Received: from smtp-sfn.sitkom.cz (smtp-sfn.sitkom.cz [88.146.175.4]) by mx1.freebsd.org (Postfix) with ESMTP id 63EEE8FC08; Thu, 15 Oct 2009 19:37:35 +0000 (UTC) Received: from osiris.buchlovice.sfn (osiris.buchlovice.sfn [10.6.193.10]) by smtp-sfn.sitkom.cz (Postfix) with ESMTP id 4F99D1FA69D; Thu, 15 Oct 2009 21:37:34 +0200 (CEST) Message-ID: <4AD779FC.1070204@buchlovice.org> Date: Thu, 15 Oct 2009 21:37:32 +0200 From: =?ISO-8859-2?Q?Radek_Val=E1=B9ek?= User-Agent: Thunderbird 2.0.0.23 (X11/20091015) MIME-Version: 1.0 To: Robert Noland References: <4AD710D6.70404@buchlovice.org> <1255633430.2175.12.camel@balrog.2hip.net> In-Reply-To: <1255633430.2175.12.camel@balrog.2hip.net> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 19:37:35 -0000 Robert Noland napsal(a): > On Thu, 2009-10-15 at 14:08 +0200, Radek Valášek wrote: > >> Hi, >> >> I want to ask if there is something new in adding support to >> gptzfsboot/zfsboot for reading gang-blocks? >> > > Ok, I can't figure out any way to test this... beyond the fact that it > builds and doesn't break my currently working setup. Can you give this > a try? It should still report if it finds gang blocks, but hopefully > now will read them as well. > > robert. > > Big thanks for the patches Robert, I will definitely test them as soon as possible (tomorrow) and report the results immediately to list. I can repeat this issue probably at any time (up to cca 30 times tested with the same result), so don't bother about the broken booting, I'm prepared for it... vaLin >> From Sun's docs: >> >> Gang blocks >> >> When there is not enough contiguous space to write a complete block, the ZIO >> pipeline will break the I/O up into smaller 'gang blocks' which can later be >> assembled transparently to appear as complete blocks. >> >> Everything works fine for me, until I rewrite kernel/world after system >> upgrade to latest one (releng_8). After this am I no longer able to boot >> from zfs raidz1 pool with following messages: >> >> >/ ZFS: i/o error - all block copies unavailable >> />/ ZFS: can't read MOS >> />/ ZFS: unexpected object set type lld >> />/ ZFS: unexpected object set type lld >> />/ >> />/ FreeBSD/i386 boot >> />/ Default: z:/boot/kernel/kernel >> />/ boot: >> />/ ZFS: unexpected object set type lld >> />/ >> />/ FreeBSD/i386 boot >> />/ Default: tank:/boot/kernel/kernel >> />/ boot: >> // >> /I presume it's the same issue as talked in june-2009 current mailing >> list >> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html >> >> Any success in that matter? >> >> Thnx for answer. >> >> vaLin >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >> From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 22:15:20 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8B2D106566B; Thu, 15 Oct 2009 22:15:19 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.27]) by mx1.freebsd.org (Postfix) with ESMTP id 4C31E8FC15; Thu, 15 Oct 2009 22:15:19 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 9so325417eyd.9 for ; Thu, 15 Oct 2009 15:15:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=IPGsFErhvgZy9TM194bFffvNcHDTuiLXBgTvndWkfJk=; b=gwBGWyB31QjVz3AMM4aeabtC/92l5Lc5jAqtK+LxctB6u6Kn+8FJDK+l9+z82zusfT FuP8vfBMh9/MDkDtR0hq5Kf+1o7f0IJ2+6vyZtgsTIZQSOQKbASKEe8aiUtCcC3LY97S KavdJcH0gKuDgBC8bHAsTULC9IAXszp0UkJUA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Nht2h/WJt7TqvnPo6CwPQKRO+0E1OcB1USTXI70dCnWjcFlGmDV7Hb/JQz332ZVVH9 2FpKm5LpEFs4u2CihkO3D4kvE33ue/b1QG/cfmUPMaQ1wgtv8hNIawBt3vEtqtjJb7v5 hoqLAexK3g1QRYMGFRYjHVDQAnMW87qBRwT5o= MIME-Version: 1.0 Received: by 10.216.88.212 with SMTP id a62mr286083wef.72.1255644918374; Thu, 15 Oct 2009 15:15:18 -0700 (PDT) In-Reply-To: <43425103-6DAE-43FD-82E7-9B71BAB862B0@gothic.net.au> References: <43425103-6DAE-43FD-82E7-9B71BAB862B0@gothic.net.au> Date: Thu, 15 Oct 2009 18:15:18 -0400 Message-ID: From: grarpamp To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS repeatable reboot 8.0-RC1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 22:15:20 -0000 Note: I tried setting vfs.zfs.arc_max=32M and zfs mem usage still grew past its limits and the machine rebooted. Forwarding a note I received... """"" >> Your machine is starving! > How can this be, there is over 500MiB free ram at all times? I'm sysctl vm.kmem_size_max I've got the following in my kernel configuration file options KVA_PAGES=512 and in /boot/loader.conf vm.kmem_size_max=1536M vm.kmem_size=1536M On two machines with 2G of RAM....both 8.0-RC1/i386 the ZFS tuning guide gives a better idea of how to play with things like that http://wiki.freebsd.org/ZFSTuningGuide """"" Some questions... Am I reading correctly that vm.kmem_size is how much ram the kernel initially allocates for itself on boot? And that vm.kmem_size_min and vm.kmem_size_max are the range that vm.kmem_size is allowed to float naturally within at runtime? Is KVA_PAGES the hard max space the kern is allowed/capable of addressing/using at runtime? Such that I could set kmem_size_max to the KVA_PAGES limit and then vm.kmem_size will grow into it as needed? With the caveat of course that with the below defaults and hardware, if I just bumped vm.kmem_size_max to 1GiB [as per KVA_PAGES limit] I'd have to add maybe another 1GiB ram so that this new vm.kmem_size_max kernel reservation wouldn't conflict with userland memory needs when vm.kmem_size grows to it? And KVA_PAGES is typically say 1/3 of installed ram? If vm.kmem_size starts out being under vm.kmem_size_max, can user apps use the unused space (vm.kmem_size_max - vm.kmem_size) until vm.kmem_size grows to vm.kmem_size_max and the kernel kills them off? Or can user apps only use (ram = user apps + [KVA_PAGES hard limit and/or vm.kmem_size_max])? What is the idea behind setting vm.kmem_size = vm.kmem_size_max? Should not just vm.kmem_size_max be set and allow vm.kmem_size [unset] to grow up to vm.kmem_size_max instead of allocating it all at boot with vm.kmem_size? Maybe someone can wikify these answers? I think I need to find more to read and then test one by one to see what changes. With untuned defaults and 1GiB ram I have: #define KVA_PAGES 256 # gives 1GiB kern address space vm.kmem_size_max: 335544320 # auto calculated by the kernel at boot? Less than KVA_PAGES? vm.kmem_size_min: 0 vm.kmem_size: 335544320 # amount in use at runtime? I'm still figuring out how to find and add all the kernel memory. Here's zfs: vfs.zfs.arc_meta_limit: 52428800 vfs.zfs.arc_meta_used: 56241732 # greater than meta_limit? vfs.zfs.arc_min: 26214400 vfs.zfs.arc_max: 209715200 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 kstat.zfs.misc.arcstats.p: 20589785 # was 104857600 on boot kstat.zfs.misc.arcstats.c: 128292242 # was 209715200 on boot kstat.zfs.misc.arcstats.c_min: 26214400 kstat.zfs.misc.arcstats.c_max: 209715200 loader(8) vm.kmem_size Sets the size of kernel memory (bytes). This overrides the value determined when the kernel was compiled. Modifies VM_KMEM_SIZE. vm.kmem_size_min vm.kmem_size_max Sets the minimum and maximum (respectively) amount of ker- nel memory that will be automatically allocated by the ker- nel. These override the values determined when the kernel was compiled. Modifies VM_KMEM_SIZE_MIN and VM_KMEM_SIZE_MAX. sys/i386/include/pmap.h /* * Size of Kernel address space. This is the number of page table pages * (4MB each) to use for the kernel. 256 pages == 1 Gigabyte. * This **MUST** be a multiple of 4 (eg: 252, 256, 260, etc). * For PAE, the page table page unit size is 2MB. This means that 512 pages * is 1 Gigabyte. Double everything. It must be a multiple of 8 for PAE. */ #ifndef KVA_PAGES #ifdef PAE #define KVA_PAGES 512 #else #define KVA_PAGES 256 #endif #endif sys/i386/conf/NOTES # Change the size of the kernel virtual address space. Due to # constraints in loader(8) on i386, this must be a multiple of 4. # 256 = 1 GB of kernel address space. Increasing this also causes # a reduction of the address space in user processes. 512 splits # the 4GB cpu address space in half (2GB user, 2GB kernel). For PAE # kernels, the value will need to be double non-PAE. A value of 1024 # for PAE kernels is necessary to split the address space in half. # This will likely need to be increased to handle memory sizes >4GB. # PAE kernels default to a value of 512. # options KVA_PAGES=260 From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 22:24:04 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68B2C10656A3 for ; Thu, 15 Oct 2009 22:24:04 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.26]) by mx1.freebsd.org (Postfix) with ESMTP id 02C398FC20 for ; Thu, 15 Oct 2009 22:24:03 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 9so326897eyd.9 for ; Thu, 15 Oct 2009 15:24:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=7Xyl4BI+bzxiKUCngoKrVmypjxaaLugnnFlflmggl9Y=; b=acQvZbTK90oAHT0svEP/XAGvNgwf6E18N0EEEnwnc0PEmI+VcZC3ZyRFfA33iuyrow q31lFtYJgtzENFEe4XNE2byD8WVKZdk4aSvorSliSG6wXjVqir1dALH8MBziJvOL5NKI FnSiOwcSuhVKBTDfw7cHtJ7PFS09Bl43rIfOk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=b+JBiS77pa6R9XRywKZMO6cqrZ0ktmwf3ykfTluoFcvx9aOWT+qdA0BErpR+9zvyeq ArcdGSEOCWWWODrSu1sn5xF2H6hPUNEO+hgHAiem2znFbgin/4FHUSDgSAStwS+I4tES BpyoJ82l/a9/v5KHMZj9lt+lThVkowvx7kfAY= MIME-Version: 1.0 Received: by 10.216.86.4 with SMTP id v4mr293723wee.200.1255645442895; Thu, 15 Oct 2009 15:24:02 -0700 (PDT) Date: Thu, 15 Oct 2009 18:24:02 -0400 Message-ID: From: grarpamp To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: ZFS threads? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 22:24:04 -0000 I don't know about reducing them directly, maybe vfs.zfs.zfetch.max_streams. On mine it's 8 which might be that 0 through 7 you see. This system has 9 zfs mountpoints and 104 spa_zio* threads. So you might try cutting mountpoints if possible. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 22:30:08 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EE72106566B for ; Thu, 15 Oct 2009 22:30:08 +0000 (UTC) (envelope-from ml@infosec.pl) Received: from v027580.home.net.pl (v027580.home.net.pl [89.161.156.148]) by mx1.freebsd.org (Postfix) with SMTP id B5FD18FC15 for ; Thu, 15 Oct 2009 22:30:07 +0000 (UTC) Received: from localhost (HELO ?192.168.1.67?) (ml.freeside@home@127.0.0.1) by m094.home.net.pl with SMTP; Thu, 15 Oct 2009 22:03:35 -0000 Message-ID: <4AD7AA19.8050802@infosec.pl> Date: Thu, 15 Oct 2009 23:02:49 +0000 From: Michal User-Agent: Thunderbird 2.0.0.23 (X11/20091003) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: root on ZFS and zpool.cache importance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 22:30:08 -0000 Hello. I'm a bit confused about zpool.cache file. I've got a configuration with /boot sitting on a usb dongle (UFS) and everything else on internal ZFS hard drive. I'm booting my system off the usb drive so zpool.cache file is there (it was generated when I created zpool). Basic zfs root + ufs boot setup. Everything works like a charm no problems so far and I would like to keep it that way hence my question. I've noticed that zpool.cache on ZFS drive is being created and updated from time to time and it is different from zpool.cache file on usb drive. Even when I remove zpool.cache file on hard disk then it gets recreated automatically and system still boots and works fine since it starts with zpool.cache on usb drive which is intact. Now how important it is to keep them in sync and what I'm risking by not doing that? Should I copy zpool.cache from hard drive to usb boot disk every day or should I leave it how it is and don't even bother? Michal -- "Never forget what a man says to you when he is angry." -Henry Ward Beecher From owner-freebsd-fs@FreeBSD.ORG Thu Oct 15 23:32:42 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5069C1065672 for ; Thu, 15 Oct 2009 23:32:42 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) by mx1.freebsd.org (Postfix) with ESMTP id 10CAD8FC13 for ; Thu, 15 Oct 2009 23:32:41 +0000 (UTC) Received: from localhost (localhost.codelab.cz [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D4BCB19E027; Fri, 16 Oct 2009 01:32:40 +0200 (CEST) Received: from [192.168.1.2] (r5bb235.net.upc.cz [86.49.61.235]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 919F319E019; Fri, 16 Oct 2009 01:32:38 +0200 (CEST) Message-ID: <4AD7B115.2080104@quip.cz> Date: Fri, 16 Oct 2009 01:32:37 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: grarpamp References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS threads? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2009 23:32:42 -0000 grarpamp wrote: > I don't know about reducing them directly, > maybe vfs.zfs.zfetch.max_streams. On mine > it's 8 which might be that 0 through 7 you see. > > This system has 9 zfs mountpoints and > 104 spa_zio* threads. So you might try > cutting mountpoints if possible. I don't know how it is related, but I have system with 17 ZFS mountpoints and just 25 spa_zio threads (named as spa_zio_issue_2, spa_zio_intr_2 etc). This is on 7.2-RELEASE-p2 amd64 Miroslav Lachman From owner-freebsd-fs@FreeBSD.ORG Fri Oct 16 08:28:21 2009 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 445A1106568D; Fri, 16 Oct 2009 08:28:21 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 210578FC17; Fri, 16 Oct 2009 08:28:19 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA09373; Fri, 16 Oct 2009 11:28:17 +0300 (EEST) (envelope-from avg@icyb.net.ua) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1MyiAb-0006oi-EE; Fri, 16 Oct 2009 11:28:17 +0300 Message-ID: <4AD82E91.3030703@icyb.net.ua> Date: Fri, 16 Oct 2009 11:28:01 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20090823) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4AD3220B.5030502@icyb.net.ua> <86ws30ikfw.fsf@gmail.com> <4AD45A8D.4070605@icyb.net.ua> <20091014062827.GC1696@garage.freebsd.pl> In-Reply-To: <20091014062827.GC1696@garage.freebsd.pl> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=KOI8-U Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, Anonymous , freebsd-current@FreeBSD.org Subject: Re: zfs boot vs loader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2009 08:28:21 -0000 on 14/10/2009 09:28 Pawel Jakub Dawidek said the following: > On Tue, Oct 13, 2009 at 01:46:37PM +0300, Andriy Gapon wrote: >> Thanks to all who replied! >> I think that I figured it out with your help. >> >> 'unload' does indeed unload zpool.cache and the latter is needed for proper zfs >> root mounting. >> 'boot /other/kernel' executes some loader scripts and zpool.cache gets loaded in >> this case (because of the settings in defaults/loader.conf). >> When doing manual 'load xxx' and then just 'boot' no loader scripts are executed >> and zpool.cache does not get loaded. In this case one has to load it manually >> using the suggested 'load -t xxx' approach. > > Could you repeat what you were doing but with vfs.zfs.debug set to 1 > from the loader? You should see some messages about missing file, maybe > we should do them visible always and not after increasing debug level? Yes, I tried this and got the following message: spa_config_load:92[1]: Cannot open /boot/zfs/zpool.cache Perhaps it's a good idea indeed to be more verbose about this. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Oct 16 09:19:25 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 927811065676 for ; Fri, 16 Oct 2009 09:19:25 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 4FD908FC19 for ; Fri, 16 Oct 2009 09:19:25 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1Myiy2-0000rb-Iw for freebsd-fs@freebsd.org; Fri, 16 Oct 2009 11:19:22 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 16 Oct 2009 11:19:22 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 16 Oct 2009 11:19:22 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Fri, 16 Oct 2009 11:19:03 +0200 Lines: 23 Message-ID: References: <4AD7B115.2080104@quip.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.23 (X11/20090928) In-Reply-To: <4AD7B115.2080104@quip.cz> Sender: news Subject: Re: ZFS threads? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2009 09:19:25 -0000 Miroslav Lachman wrote: > grarpamp wrote: > >> I don't know about reducing them directly, >> maybe vfs.zfs.zfetch.max_streams. On mine >> it's 8 which might be that 0 through 7 you see. >> >> This system has 9 zfs mountpoints and >> 104 spa_zio* threads. So you might try >> cutting mountpoints if possible. > > I don't know how it is related, but I have system with 17 ZFS > mountpoints and just 25 spa_zio threads (named as spa_zio_issue_2, > spa_zio_intr_2 etc). > This is on 7.2-RELEASE-p2 amd64 Hmm, this is interesting. On a quad-core 7.2-R box amd64 with 11 mount points I have 6 sets of 4 zio threads (spa_zio_intr_[0-5] and spa_zio_issue_[0-5] in each). The original machine I wrote about is 8.0-RC1 i386 with a single CPU. Maybe CPU count detection is turned off in 8? From owner-freebsd-fs@FreeBSD.ORG Fri Oct 16 17:00:09 2009 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF66810656A4 for ; Fri, 16 Oct 2009 17:00:09 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7A61A8FC13 for ; Fri, 16 Oct 2009 17:00:09 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id n9GH096T034487 for ; Fri, 16 Oct 2009 17:00:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id n9GH09dR034486; Fri, 16 Oct 2009 17:00:09 GMT (envelope-from gnats) Date: Fri, 16 Oct 2009 17:00:09 GMT Message-Id: <200910161700.n9GH09dR034486@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Andrey Simonenko Cc: Subject: Re: kern/136865: NFS exports atomic and on-the-fly atomic updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Andrey Simonenko List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2009 17:00:09 -0000 The following reply was made to PR kern/136865; it has been noted by GNATS. From: Andrey Simonenko To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/136865: NFS exports atomic and on-the-fly atomic updates Date: Fri, 16 Oct 2009 19:51:22 +0300 New version nfse-20091016 does not have limitation for number of groups: nfse(8) gets number of groups via sysconf(_SC_NGROUPS_MAX) and nfsserver/nfs_exports.c allows up to NGROUPS groups. URL: http://comsys.ntu-kpi.kiev.ua/~simon/nfse/ MD5 (nfse-20091016.tar.bz2) = 49328852374275763876a03f666e016a From owner-freebsd-fs@FreeBSD.ORG Fri Oct 16 22:44:07 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 139B3106568B for ; Fri, 16 Oct 2009 22:44:07 +0000 (UTC) (envelope-from torbjoern@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id 953818FC0A for ; Fri, 16 Oct 2009 22:44:06 +0000 (UTC) Received: by fxm6 with SMTP id 6so2902433fxm.43 for ; Fri, 16 Oct 2009 15:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=+RTh/bNJU/z5n1+/LdqsOoiCzL9eU5wS0keg8Uy7iG4=; b=wDFzx1/pUVH2amNeySoa24K7wicWe3gm2v/vhdolvXKrD3j+mfAl4n8Rf1QJVXLq5u fBEYzNOs0fLEtSXLfhOD07702Y/o4vYKllI1N50BraMtPemOKQ2kDR3C+XQ70AG7dBjo m0/zCj+STIBQn5kBcSSU8ToUufY6bjTvDdkvk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=EEU2cB0tHsHV0WwczYLYthV5pFJLjhqDgL2TLRNpKY9ldowwilBfkW5abIfAvsl1Cf Xt7Ncq6N4dqpq6T2Hs0U2rGuFx8uzXZ4hsLmJSH8LJFW53Zc35zLSyfZfefTzFPSDamy /9UERLJ2zvJCbaghkVtbm5yIduscAx18CiEW4= MIME-Version: 1.0 Received: by 10.204.153.220 with SMTP id l28mr1864123bkw.86.1255731506226; Fri, 16 Oct 2009 15:18:26 -0700 (PDT) Date: Sat, 17 Oct 2009 00:18:26 +0200 Message-ID: <103e3fa20910161518t57e2c3afkddf160cb1f4ecfdb@mail.gmail.com> From: Torbjorn Kristoffersen To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Advice needed with ZFS on root filesystem (installing remotely via mfsBSD) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2009 22:44:07 -0000 I am trying to install FreeBSD 8.0-RC1 on amd64, and I plan to use ZFS (mirrored) as the root filesystem. I have been struggling with this for many hours -- it just won't boot up and I can't understand what I'm doing wrong. Before I describe my steps, keep in mind that I am doing this using the Remote Install method of mfsBSD. My mfsBSD system boots up just fine and I log in via SSH. However, seeing that it's a remote installation I'm unable to see any error message if the system does not boot up correctly. A sad situation, indeed. What is wrong with my approach ? Please see below. -- First I create a GPL label on my two disks % gpart create -s gpt ad4 % gpart create -s gpt ad6 Then I add a boot-partition that will hold the ZFS bootcode % gpart add -b 34 -s 128 -t freebsd-boot ad4 % gpart add -b 34 -s 128 -t freebsd-boot ad6 I then add the freebsd-zfs partition on each disk % gpart add -b 162 -s 1465148973 -t freebsd-zfs ad4 % gpart add -b 162 -s 1465148973 -t freebsd-zfs ad6 The results: % gpart show ad4 => 34 1465149101 ad4 GPT (699G) 34 128 1 freebsd-boot (64K) 162 1465148973 2 freebsd-zfs (699G) % gpart show ad6 => 34 1465149101 ad6 GPT (699G) 34 128 1 freebsd-boot (64K) 162 1465148973 2 freebsd-zfs (699G) I add GPT bootcode to the MBR and add the ZFS bootcode: % gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot -i 1 ad4 % gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot -i 1 ad6 I make sure that the first partition is set to active % fdisk -a /dev/ad4 ... % fdisk -a /dev/ad6 I create zpool on ad4p2 and ad6p2 (mirrored) % zpool create zroot mirror ad4p2 ad6p2 Create filesystem % zfs create -p zroot Make sure that I enable the bootfs % zpool set bootfs=zroot zroot At this point, /zroot is mounted on my filesystem, so I go ahead and do a sysinstall and install FreeBSD. I make sure that /zroot is set as "root folder" in sysinstall's Options menu. I add zfs_load="YES" to /zroot/boot/loader.conf and vfs.root.mountfrom="zfs:zroot" to /zroot/boot/loader.conf And I also copy /boot/zfs/zpool.cache to /zroot/boot/zfs/zpool.cache Then I do % chroot /zroot % sysinstall ... I go into configuration, set the proper network settings, make sure that sshd will get started. I change root's password and create a user. My /etc/rc.conf (on zroot) looks like this: ifconfig_re0="inet xxx.xxx.121.11 netmask 255.255.255.192" defaultrouter="xxx.xxx.121.1" sshd_enable="YES" hostname="grim" At this point I have also tried compiling a kernel, making sure that GEOM_PART_BSD, GEOM_PART_MBR and other required options are included. I also installed a ZFS aware /boot/loader (**) (Note: I will create a swap partition later once I get the installation booted.) At this point % exit (from the chroot) % zfs unmount -a % reboot At this point the computer never comes back and I have to use my colo's rescue system to put mfsBSD back on. One thing I noticed at that point, is that the freebsd-boot on ad4 is gone and the freebsd-zfs partition has been replaced with a smaller freebsd-ufs. I did not touch "Partitions" or "Labels" in the sysinstall installation. What could possibly cause this? (In case it matters; ad6 is unchanged.) Kind Regards, Torbjorn Kristoffersen (**) http://wiki.freebsd.org/ZFSOnRootWithZFSboot#installLoader From owner-freebsd-fs@FreeBSD.ORG Sat Oct 17 04:16:09 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11D41106566C for ; Sat, 17 Oct 2009 04:16:09 +0000 (UTC) (envelope-from kickbsd@ya.ru) Received: from forward7.yandex.ru (forward7.yandex.ru [77.88.61.37]) by mx1.freebsd.org (Postfix) with ESMTP id BF5E28FC1D for ; Sat, 17 Oct 2009 04:16:08 +0000 (UTC) Received: from webmail71.yandex.ru (webmail71.yandex.ru [77.88.60.175]) by forward7.yandex.ru (Yandex) with ESMTP id B9E6D31057D for ; Sat, 17 Oct 2009 07:56:48 +0400 (MSD) Received: from localhost (localhost [127.0.0.1]) by webmail71.yandex.ru (Yandex) with ESMTP id A1CE79C4479 for ; Sat, 17 Oct 2009 07:56:48 +0400 (MSD) X-Yandex-Spam: 1 X-Yandex-Front: webmail71 X-Yandex-TimeMark: 1255751808 Received: from modemcable177.130-37-24.mc.videotron.ca (modemcable177.130-37-24.mc.videotron.ca [24.37.130.177]) by mail.yandex.ru with HTTP; Sat, 17 Oct 2009 07:56:48 +0400 From: kickbsd kickbsd To: freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: <66051255751808@webmail71.yandex.ru> Date: Sat, 17 Oct 2009 07:56:48 +0400 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain Subject: zfs and vfs.numvnodes issue X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2009 04:16:09 -0000 Hi! I have a strange behavior on zfs filesystem. vfs.numvnodes tends to grow and when reach kern.maxvnodes no new files can be created or modified. System AMD64 8.0-RC1 FreeBSD 8.0-RC1 CVS from Oct 13 2009. I have increased kern.maxvnodes but vfs.numvnodes grows slowly. Any suggestions ?