From owner-freebsd-fs@FreeBSD.ORG Sun Jul 28 15:36:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0483AC56 for ; Sun, 28 Jul 2013 15:36:05 +0000 (UTC) (envelope-from Albert.Shih@obspm.fr) Received: from smtp-int-m.obspm.fr (smtp-int-m.obspm.fr [145.238.187.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 936322011 for ; Sun, 28 Jul 2013 15:36:04 +0000 (UTC) Received: from pcjas.obspm.fr (pcjas.obspm.fr [145.238.184.233]) by smtp-int-m.obspm.fr (8.14.3/8.14.3/SIO Observatoire de Paris - 07/2009) with ESMTP id r6SFXG1G007735 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sun, 28 Jul 2013 17:33:17 +0200 Date: Sun, 28 Jul 2013 17:33:16 +0200 From: Albert Shih To: freebsd-fs@freebsd.org Subject: Big problem with zfs....or not Message-ID: <20130728153316.GA94100@pcjas.obspm.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.5.21 (2010-09-15) X-Miltered: at smtp-int-m.obspm.fr with ID 51F539BC.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 51F539BC.000/145.238.184.233/pcjas.obspm.fr/pcjas.obspm.fr/ X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Jul 2013 15:36:05 -0000 Hi I've running a server Dell R610 (with LSI-9200-8E) + 5 x MD1200 during almost 2 years without any problem (under FreeBSD 9.0). Without any change on the software and the hardware, zfs status hang. And when I try to reboot the server the kernel stuck in run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config (and 120 sec etc.) What do you think ? Because I didn't change anything I think it's a hardware problem, but I don't see what kind of problem and how to track the problem If someone think it's software (or both) problem have you any idea how I can fix it. Regards. JAS -- Albert SHIH DIO btiment 15 Observatoire de Paris 5 Place Jules Janssen 92195 Meudon Cedex France Tlphone : +33 1 45 07 76 26/+33 6 86 69 95 71 xmpp: jas@obspm.fr Heure local/Local time: dim 28 jul 2013 17:27:28 CEST From owner-freebsd-fs@FreeBSD.ORG Sun Jul 28 18:30:01 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 85CC357E for ; Sun, 28 Jul 2013 18:30:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 730E824E0 for ; Sun, 28 Jul 2013 18:30:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6SIU1qE044204 for ; Sun, 28 Jul 2013 18:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6SIU1ZI044203; Sun, 28 Jul 2013 18:30:01 GMT (envelope-from gnats) Date: Sun, 28 Jul 2013 18:30:01 GMT Message-Id: <201307281830.r6SIU1ZI044203@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Mikolaj Golub Subject: Re: kern/180876: [zfs] [hast] ZFS with trim,bio_flush or bio_delete locks hast device write operations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Mikolaj Golub List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Jul 2013 18:30:01 -0000 The following reply was made to PR kern/180876; it has been noted by GNATS. From: Mikolaj Golub To: bug-followup@FreeBSD.org, ivk@kristal.ru Cc: Subject: Re: kern/180876: [zfs] [hast] ZFS with trim,bio_flush or bio_delete locks hast device write operations Date: Sun, 28 Jul 2013 21:25:19 +0300 I have failed to reproduce the issue. So could you give more details? It would be nice to see your configs and logs from both primary and secondary, and the sequence of commands that led to the hang. Also, it would be very nice to find out exactly what option (or combination) caused the issue. -- Mikolaj Golub From owner-freebsd-fs@FreeBSD.ORG Sun Jul 28 20:12:02 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E60EC107 for ; Sun, 28 Jul 2013 20:12:02 +0000 (UTC) (envelope-from prvs=19216b46f0=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 86B0E28CA for ; Sun, 28 Jul 2013 20:12:02 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50005195999.msg for ; Sun, 28 Jul 2013 21:11:54 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sun, 28 Jul 2013 21:11:54 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=19216b46f0=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <33A524DDB4E84BE5A4F9E3BEA0ACE125@multiplay.co.uk> From: "Steven Hartland" To: "Albert Shih" , References: <20130728153316.GA94100@pcjas.obspm.fr> Subject: Re: Big problem with zfs....or not Date: Sun, 28 Jul 2013 21:12:27 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Jul 2013 20:12:03 -0000 Sounds like you may either have a failing disk (failing to return) or failing controller. Have you tried installing new disks and booting from that? ----- Original Message ----- From: "Albert Shih" To: Sent: Sunday, July 28, 2013 4:33 PM Subject: Big problem with zfs....or not Hi I've running a server Dell R610 (with LSI-9200-8E) + 5 x MD1200 during almost 2 years without any problem (under FreeBSD 9.0). Without any change on the software and the hardware, zfs status hang. And when I try to reboot the server the kernel stuck in run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config (and 120 sec etc.) What do you think ? Because I didn't change anything I think it's a hardware problem, but I don't see what kind of problem and how to track the problem If someone think it's software (or both) problem have you any idea how I can fix it. Regards. JAS -- Albert SHIH DIO btiment 15 Observatoire de Paris 5 Place Jules Janssen 92195 Meudon Cedex France Tlphone : +33 1 45 07 76 26/+33 6 86 69 95 71 xmpp: jas@obspm.fr Heure local/Local time: dim 28 jul 2013 17:27:28 CEST _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Sun Jul 28 23:41:39 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A69D63DF for ; Sun, 28 Jul 2013 23:41:39 +0000 (UTC) (envelope-from krh@kirkholz.com.au) Received: from baha.lunarbreeze.com (baha.lunarbreeze.com [67.210.123.5]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8855D205E for ; Sun, 28 Jul 2013 23:41:39 +0000 (UTC) Received: from [203.23.208.206] (port=8417 helo=saga-wired.splurben.localdomain) by baha.lunarbreeze.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1V3aaf-0003qC-EL; Sun, 28 Jul 2013 16:41:29 -0700 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Trying to recover 2-element zfs striped (raid0) filesystem From: Kirk Richard Holz In-Reply-To: <20130726220159.GI86483@server.rulingia.com> Date: Mon, 29 Jul 2013 07:41:26 +0800 Content-Transfer-Encoding: quoted-printable Message-Id: <0DAA1527-B338-4232-84E7-D138EBA61EAC@kirkholz.com.au> References: <1b756c89576eb509d1197c4d9ab66fea@kirkholz.com> <20130726220159.GI86483@server.rulingia.com> To: Peter Jeremy X-Mailer: Apple Mail (2.1508) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - baha.lunarbreeze.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - kirkholz.com.au X-Get-Message-Sender-Via: baha.lunarbreeze.com: authenticated_id: krh@kirkholz.com X-Source: X-Source-Args: X-Source-Dir: Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Jul 2013 23:41:39 -0000 Thanks for the help, just a query here, what does the = 8683733800792668130 number signify in the zpool / zshare on the UNAVAIL = line of the list, I=92ve quoted below? It doesn=92t look like a UUID as = it appears to be decimal not hex. I haven=92t written to the drives and = I=92m cloning them now. > NAME STATE READ WRITE CKSUM > zShare UNAVAIL 0 0 0 > ada1 ONLINE 0 0 0 > 8683733800792668130 UNAVAIL 0 0 0 was = /dev/ada3s1 >=20 On 27/07/2013, at 06:01 , Peter Jeremy wrote: > On 2013-Jul-26 01:33:01 -0700, Kirk Richard Holz = wrote: >> The partition table of one of the two disks in a zfs striped (raid0)=20= >> array has been corrupted. >=20 > Once you recover the partition table for ada3, ZFS should be OK (as > long as you haven't written too much to the pool). If zShare is your > boot device, I strongly recommend booting off alternative media. > Ideally, you should take full physical copies of both disks. >=20 > If you are unable to remember the partition layout, you should be able > to recover it by looking for the ZFS vdev labels: Each ZFS vdev has > 4 256KiB labels - 2 at the start of the partition and 2 at the end of > the partition so locating those will identify the start and end of the > partition - if you used the entire disk for ZFS, they will be close > to the start & end of the disk and "hd /dev/ada3|less" should be > enough to find the first label. The first label on all my vdevs = begins: >=20 > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 00003fd0 00 00 00 00 00 00 00 00 11 7a 0c b1 7a da 10 02 = |.........z..z...| > 00003fe0 3f 2a 6e 7f 80 8f f4 97 fc ce aa 58 16 9f 90 af = |?*n........X....| > 00003ff0 8b b4 6d ff 57 ea d1 cb ab 5f 46 0d db 92 c6 6e = |..m.W...._F....n| > 00004000 01 01 00 00 00 00 00 00 00 00 00 01 00 00 00 24 = |...............$| > 00004010 00 00 00 20 00 00 00 07 76 65 72 73 69 6f 6e 00 |... = ....version.| > 00004020 00 00 00 08 00 00 00 01 00 00 00 00 00 00 13 88 = |................| > 00004030 00 00 00 24 00 00 00 20 00 00 00 04 6e 61 6d 65 |...$... = ....name| >=20 > If you look at the output of "zdb -C zShare", the 'asize' value is the > usable size of the vdev in bytes - the physical size is a slightly > larger (~4.5MB for me) but the labels are at the end of the physical > partition. >=20 > For more details, see the ZFS On-Disk Specification = (ondiskformat0822.pdf) >=20 > --=20 > Peter Jeremy From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 02:50:02 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BC989917 for ; Mon, 29 Jul 2013 02:50:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A8C5F267B for ; Mon, 29 Jul 2013 02:50:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6T2o1mT050552 for ; Mon, 29 Jul 2013 02:50:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6T2o1pj050551; Mon, 29 Jul 2013 02:50:01 GMT (envelope-from gnats) Date: Mon, 29 Jul 2013 02:50:01 GMT Message-Id: <201307290250.r6T2o1pj050551@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: =?UTF-8?B?ItCa0YPQu9C10LzQt9C40L0g0Jgu0JIuIg==?= Subject: Re: kern/180876: [zfs] [hast] ZFS with trim,bio_flush or bio_delete locks hast device write operations X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: =?UTF-8?B?ItCa0YPQu9C10LzQt9C40L0g0Jgu0JIuIg==?= List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 02:50:02 -0000 The following reply was made to PR kern/180876; it has been noted by GNATS. From: =?UTF-8?B?ItCa0YPQu9C10LzQt9C40L0g0Jgu0JIuIg==?= To: Mikolaj Golub , bug-followup@FreeBSD.org Cc: Subject: Re: kern/180876: [zfs] [hast] ZFS with trim,bio_flush or bio_delete locks hast device write operations Date: Mon, 29 Jul 2013 12:36:49 +1000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2ILQJRLSXWOBQSQFDETSN Content-Type: multipart/mixed; boundary="------------000409060006060202080400" This is a multi-part message in MIME format. --------------000409060006060202080400 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 CiAgaGFzdC5jb25mIGF0dGFjaGVkCgogIE9ubHkgdGhhdCBzdGVwczoKICAtIGJvb3QgcHJp bWFyeQogIC0gYm9vdCBzZWNvbmRhcnkKICAtIG9uIHByaW1hcnk6IGhhc3RjdGwgcm9sZSBw cmltYXJ5IGhhc3QwCiAgLSBvbiBzZWNvbmRhcnk6IGhhc3RjdGwgcm9sZSBzZWNvbmRhcnkg aGFzdDAKICAtIHdhaXQgc29tZSBzZWNvbmRzOiAxMC0yMC4gImhhc3RjdGwgbGlzdCIgc2hv d3MgYWN0aXZlIHJlcGxpY2F0aW9uCiAgLSBvbiBwcmltYXJ5IHpwb29sIGltcG9ydCB6cm9v dC4genBvb2wgaW1wb3J0ZWQgYW5kIHN0YXJ0IHJlc2lsdmVyaW5nLgpBbmQgYWZ0ZXIgc29t ZSBzZWNvbmRzIGRpc2tzIGFjdGl2aXR5IHN0b3BwZWQuIEkgY2FuJ3QgbW91bnQgWkZTCmRh dGFzZXRzLiBBbHNvIEkndmUgdHJpZWQgYXR0YWNoaW5nIGRpc2sgdG8genBvb2wgd2l0aCBy ZXNpbHZlcmluZyB0b28uCkdvdCBzYW1lIHJlc3VsdHMuIEluIGZpcnN0IHBvc3QgSSBkaWRu J3Qgc2F5cyBhYm91dCByZXNpbHZlcmluZy4gTWF5IGJlCndpdGhvdXQgdGhlbSBzaG91bGQg YmUgd3JpdHRlbi9kZWxldGVkIG1vcmUgZmlsZXMgYW5kIGRhdGE/CgoyOS4wNy4yMDEzIDA0 OjI1LCBNaWtvbGFqIEdvbHViINC/0LjRiNC10YI6Cj4gSSBoYXZlIGZhaWxlZCB0byByZXBy b2R1Y2UgdGhlIGlzc3VlLiBTbyBjb3VsZCB5b3UgZ2l2ZSBtb3JlIGRldGFpbHM/Cj4KPiBJ dCB3b3VsZCBiZSBuaWNlIHRvIHNlZSB5b3VyIGNvbmZpZ3MgYW5kIGxvZ3MgZnJvbSBib3Ro IHByaW1hcnkgYW5kCj4gc2Vjb25kYXJ5LCBhbmQgdGhlIHNlcXVlbmNlIG9mIGNvbW1hbmRz IHRoYXQgbGVkIHRvIHRoZSBoYW5nLgo+Cj4gQWxzbywgaXQgd291bGQgYmUgdmVyeSBuaWNl IHRvIGZpbmQgb3V0IGV4YWN0bHkgd2hhdCBvcHRpb24gKG9yCj4gY29tYmluYXRpb24pIGNh dXNlZCB0aGUgaXNzdWUuCj4KCi0tIArQmtGD0LvQtdC80LfQuNC9INCYLtCSLgrQvdCw0YfQ sNC70YzQvdC40Log0L7RgtC00LXQu9CwINCQ0KHQowrQl9CQ0J4gItCa0KDQmNCh0KLQkNCb 0Jst0JDQnNCj0KAiCijQvdCw0LjQvNC10L3QvtCy0LDQvdC40LUg0L7RgNCz0LDQvdC40LfQ sNGG0LjQuCDRgdC70LXQtNGD0LXRgiDQuNGB0L/QvtC70YzQt9C+0LLQsNGC0Ywg0LIg0YLQ vtGH0L3QvtGB0YLQuCwg0LrQsNC6INGD0LrQsNC30LDQvdC+LCDRgSDRgdC+0LHQu9GO0LTQ tdC90LjQtdC8INGA0LXQs9C40YHRgtGA0LAg0YHQuNC80LLQvtC70L7QsiEpCgo= --------------000409060006060202080400 Content-Type: text/plain; charset=UTF-8; name="hast.conf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="hast.conf" cmVzb3VyY2UgaGFzdDAgewoKICBvbiBteDIgewogICAgbG9jYWwgL2Rldi9taXJyb3IvaGFz dDAKICAgIHJlbW90ZSAxOTIuMTY4LjI1NC4yCiAgfQogIG9uIG14IHsKICAgIGxvY2FsIC9k ZXYvbWlycm9yL2hhc3QwCiAgICByZW1vdGUgMTkyLjE2OC4yNTQuMQogIH0KfQoK --------------000409060006060202080400 Content-Type: text/x-vcard; charset=utf-8; name="ivk.vcf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="ivk.vcf" YmVnaW46dmNhcmQNCmZuO3F1b3RlZC1wcmludGFibGU6PUQwPTlBPUQxPTgzPUQwPUJCPUQw PUI1PUQwPUJDPUQwPUI3PUQwPUI4PUQwPUJEID1EMD05OC49RDA9OTIuDQpuO3F1b3RlZC1w cmludGFibGU7cXVvdGVkLXByaW50YWJsZTo9RDA9OUE9RDE9ODM9RDA9QkI9RDA9QjU9RDA9 QkM9RDA9Qjc9RDA9Qjg9RDA9QkQ7PUQwPTk4PUQwPUIzPUQwPUJFPUQxPTgwPUQxPThDDQpv cmc7cXVvdGVkLXByaW50YWJsZTtxdW90ZWQtcHJpbnRhYmxlOj1EMD05Nz1EMD05MD1EMD05 RSAiPUQwPTlBPUQwPUEwPUQwPTk4PUQwPUExPUQwPUEyPUQwPTkwPUQwPTlCPUQwPTlCLT1E MD05MD0NCgk9RDA9OUM9RDA9QTM9RDA9QTAiOz1EMD1CRT1EMT04Mj1EMD1CND1EMD1CNT1E MD1CQiA9RDA9OTA9RDA9QTE9RDA9QTMNCmFkcjtxdW90ZWQtcHJpbnRhYmxlO3F1b3RlZC1w cmludGFibGU7cXVvdGVkLXByaW50YWJsZTo7Ozs9RDA9OTE9RDA9QkI9RDA9QjA9RDA9QjM9 RDA9QkU9RDA9QjI9RDA9QjU9RDE9ODk9RDA9QjU9RDA9QkQ9RDE9ODE9RDA9QkE7PUQwPTkw PUQwPUJDPUQxPTgzPUQxPTgwPUQxPTgxPUQwPUJBPUQwPUIwPUQxPThGID1EMD1CRT1EMD1C MT1EMD1CQi47Njc1MDAwOz1EMD1BMD1EMD1CRT1EMT04MT1EMT04MT1EMD1COD1EMT04Rg0K ZW1haWw7aW50ZXJuZXQ6aXZrQGtyaXN0YWwucnUNCnRpdGxlO3F1b3RlZC1wcmludGFibGU6 PUQwPTlEPUQwPUIwPUQxPTg3Lj1EMD1CRT1EMT04Mj1EMD1CND1EMD1CNT1EMD1CQj1EMD1C MCA9RDA9OTA9RDA9QTE9RDA9QTM9DQoJDQp0ZWw7Y2VsbDo4LTkxNC01NTAtNTMtOTkNCngt bW96aWxsYS1odG1sOkZBTFNFDQp2ZXJzaW9uOjIuMQ0KZW5kOnZjYXJkDQoNCg== --------------000409060006060202080400-- ------enig2ILQJRLSXWOBQSQFDETSN Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlH11UQACgkQtbH1YPYdaOdv6ACbBNmJ7AldAtM02jtoBNTz/KmJ A2EAoLp2shpagQCZ4TgvGSseVcOeohhe =jJHb -----END PGP SIGNATURE----- ------enig2ILQJRLSXWOBQSQFDETSN-- From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 07:31:50 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EA02D274; Mon, 29 Jul 2013 07:31:50 +0000 (UTC) (envelope-from varanasisai@gmail.com) Received: from mail-vb0-x229.google.com (mail-vb0-x229.google.com [IPv6:2607:f8b0:400c:c02::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8635922CD; Mon, 29 Jul 2013 07:31:50 +0000 (UTC) Received: by mail-vb0-f41.google.com with SMTP id g17so2580173vbg.28 for ; Mon, 29 Jul 2013 00:31:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=x/Wa0liEFAFo/KSVMEpO9U/ohxK5wos95gGaxjysA6w=; b=QRJ52eSnFFrMYWT77cX3arbKaHx/4Hw0sfJJ146WS9eyki5hQ3UoDOWmEOhhB0JBOi 6b/aL0r3HrwWo63yFZVfq1Mh/RBDyZgIYcrYSdSUQ6fxDAb5Zolm7JA2kD5IsSec/HqT DSRPrII2koarJarQA9I0hZ9LfFADD8xITfrxfjrsnXV5hhdNU3Sg2YnS3HKSLofxo1YU nyjHh9ou6szpql3aDVS+dEiq3k45HtTgSWoyaPHZwTs6i9hzh61iUIisAa6BPUpu5WsZ iNwAEKUHU0kzVgNP3dbrWJQMEGySRP09NZccqxYH1Agn/Xzj2dpspf0sKXxdIE+V+c+8 yu8A== MIME-Version: 1.0 X-Received: by 10.52.64.243 with SMTP id r19mr1670441vds.76.1375083109553; Mon, 29 Jul 2013 00:31:49 -0700 (PDT) Received: by 10.52.233.9 with HTTP; Mon, 29 Jul 2013 00:31:49 -0700 (PDT) Date: Mon, 29 Jul 2013 13:01:49 +0530 Message-ID: Subject: Kernel Panic - Unix socket communication in kernel module From: varanasi sainath To: freebsd-drivers@freebsd.org, freebsd-fs@freebsd.org, freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: abgupta@microsoft.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 07:31:51 -0000 Hello, I am writing a kernel module in which I am trying to connect to a UNIX socket (UNIX domain sockets use the file system as their address name space). Kernel module (loadable) acts as a client and User mode program acts as server, I have loaded the module using kldload and communication between user and kernel module works fine, when I try to load the kernel module from loader.conf - auto load the kernel module at boot up leads to kernel panic as the file system is not ready and kern_connect fails. How to notify kernel module that File system is ready? (any specific event flags) Is there any specific location for Unix domain socket files? (currently created it under /root/soc/socket ) Using "MODULE_DEPEND" Can I make the module dependent of file system? Thanks. * * From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 10:57:29 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C7F48B8D; Mon, 29 Jul 2013 10:57:29 +0000 (UTC) (envelope-from bofh@terranova.net) Received: from tog.net (tog.net [IPv6:2605:5a00::5]) by mx1.freebsd.org (Postfix) with ESMTP id 9A7202CFB; Mon, 29 Jul 2013 10:57:29 +0000 (UTC) Received: from [IPv6:2605:5a00:ffff::face] (unknown [IPv6:2605:5a00:ffff::face]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tog.net (Postfix) with ESMTPSA id 3c3dC60WwHz29B; Mon, 29 Jul 2013 06:57:22 -0400 (EDT) Message-ID: <51F64A81.5010404@terranova.net> Date: Mon, 29 Jul 2013 06:57:05 -0400 From: Travis Mikalson Organization: TerraNovaNet Internet Services User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: d@delphij.net Subject: Re: Report: ZFS deadlock in 9-STABLE References: <51D45401.5050801@terranova.net> <51D47A5F.3030501@delphij.net> In-Reply-To: <51D47A5F.3030501@delphij.net> X-Enigmail-Version: 0.96.0 OpenPGP: url=http://www.terranova.net/pgp/bofh Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, kib@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 10:57:29 -0000 Xin Li wrote: > Hi, > > Sorry for the top posting but I am quite convinced that this is a > known issue that we have seen with our customer. Please try applying > this patch [1] and please report back if that fixes your problem. It has been 21 days since I booted a kernel with your patch applied and so far so good. This is the longest this system has gone without livelocking before. I'll report back one last time if this system makes it another few weeks without a livelock, since that will be an extremely strong indication that I was having the problem that you seem to have resolved with this patch. > Note that if you would like to provide more help, we would appreciate > that you test Konstantin's patch as well, at: > > http://lists.freebsd.org/pipermail/freebsd-hackers/2013-May/042876.html > > [1] See attachment; the commit is > https://github.com/trueos/trueos/commit/f678ae7c7f72fba577b00e3d0c237c4f297575c6 > > Cheers, > > On 07/03/13 09:40, Travis Mikalson wrote: >> Hello, > >> To cut to the chase, I have a procstat -kk -a captured during a >> livelock for you here: >> http://tog.net/freebsd/zfsdeadlock-storage1-20130703 > >> The other relevant configurations I could think of to show you are >> available within that http://tog.net/freebsd/ directory. > >> If you want any additional information that I haven't given here >> please let me know! > >> This is a FreeBSD 9-STABLE AMD64 system currently at: r250777: Sat >> May 18 17:41:39 EDT 2013 > >> I didn't see too many relevant ZFS-related fixes after that date so >> am waiting for another round of interesting commits to update >> again. > >> Unfortunately, this system has been livelocking on average about >> once every 7-14 days. Its lot in life is a ZFS storage server >> serving NFS and istgt traffic. > >> It has 32GB of RAM and is an 8-core 2.6GHz Opteron 6212. The zpool >> looks like this, it has eight 1TB SAS drives and two SSDs being >> used for log and cache. > >> pool: storage1 state: ONLINE status: The pool is formatted using a >> legacy on-disk format. The pool can still be used, but some >> features are unavailable. action: Upgrade the pool using 'zpool >> upgrade'. Once this is done, the pool will no longer be accessible >> on software that does not support feature flags. scan: scrub >> repaired 0 in 6h4m with 0 errors on Sun Jan 6 06:39:38 2013 >> config: > >> NAME STATE READ WRITE CKSUM storage1 ONLINE 0 >> 0 0 raidz1-0 ONLINE 0 0 0 da0 ONLINE 0 >> 0 0 da2 ONLINE 0 0 0 da4 ONLINE 0 >> 0 0 da6 ONLINE 0 0 0 raidz1-1 ONLINE 0 >> 0 0 da1 ONLINE 0 0 0 da3 ONLINE 0 >> 0 0 da5 ONLINE 0 0 0 da7 ONLINE 0 >> 0 0 logs mirror-2 ONLINE 0 0 0 da8p2 ONLINE >> 0 0 0 da9p2 ONLINE 0 0 0 cache da8p3 >> ONLINE 0 0 0 da9p3 ONLINE 0 0 0 > >> errors: No known data errors From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 11:06:44 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 42D3F1A2 for ; Mon, 29 Jul 2013 11:06:44 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2F8A92DCC for ; Mon, 29 Jul 2013 11:06:44 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6TB6hdH061746 for ; Mon, 29 Jul 2013 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6TB6hrI061744 for freebsd-fs@FreeBSD.org; Mon, 29 Jul 2013 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 29 Jul 2013 11:06:43 GMT Message-Id: <201307291106.r6TB6hrI061744@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 11:06:44 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/180876 fs [zfs] [hast] ZFS with trim,bio_flush or bio_delete loc o kern/180678 fs [NFS] succesfully exported filesystems being reported o kern/180438 fs [smbfs] [patch] mount_smbfs fails on arm because of wr p kern/180236 fs [zfs] [nullfs] Leakage free space using ZFS with nullf o kern/178854 fs [ufs] FreeBSD kernel crash in UFS o kern/178713 fs [nfs] [patch] Correct WebNFS support in NFS server and o kern/178412 fs [smbfs] Coredump when smbfs mounted o kern/178388 fs [zfs] [patch] allow up to 8MB recordsize o kern/178349 fs [zfs] zfs scrub on deduped data could be much less see o kern/178329 fs [zfs] extended attributes leak o kern/178238 fs [nullfs] nullfs don't release i-nodes on unlink. f kern/178231 fs [nfs] 8.3 nfsv4 client reports "nfsv4 client/server pr o kern/178103 fs [kernel] [nfs] [patch] Correct support of index files o kern/177985 fs [zfs] disk usage problem when copying from one zfs dat o kern/177971 fs [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3, o kern/177966 fs [zfs] resilver completes but subsequent scrub reports o kern/177658 fs [ufs] FreeBSD panics after get full filesystem with uf o kern/177536 fs [zfs] zfs livelock (deadlock) with high write-to-disk o kern/177445 fs [hast] HAST panic o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175449 fs [unionfs] unionfs and devfs misbehaviour o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172942 fs [smbfs] Unmounting a smb mount when the server became o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic f kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/145750 fs [unionfs] [hang] unionfs locks the machine s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs/ufs Lock order reversal o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/137588 fs [unionfs] [lor] LOR nfs/ufs/nfs o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126973 fs [unionfs] [hang] System hang with unionfs and init chr o kern/126553 fs [unionfs] unionfs move directory problem 2 (files appe o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o bin/123574 fs [unionfs] df(1) -t option destroys info for unionfs (a o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o kern/121385 fs [unionfs] unionfs cross mount -> kernel panic o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/67326 fs [msdosfs] crash after attempt to mount write protected o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t o kern/9619 fs [nfs] Restarting mountd kills existing mounts 328 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 12:37:43 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 96EBDACF; Mon, 29 Jul 2013 12:37:43 +0000 (UTC) (envelope-from ali@transip.nl) Received: from mailwww.transip.nl (mailwww.transip.nl [80.69.67.60]) by mx1.freebsd.org (Postfix) with ESMTP id 5957C240E; Mon, 29 Jul 2013 12:37:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailwww.transip.nl (Postfix) with ESMTP id BE9A33458437; Mon, 29 Jul 2013 14:37:42 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailwww.transip.nl Received: from mailwww.transip.nl ([127.0.0.1]) by localhost (mailwww.transip.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mc73Kh7PJoFM; Mon, 29 Jul 2013 14:37:42 +0200 (CEST) Received: from [10.2.6.4] (kantoor.transip.nl [80.69.69.100]) by mailwww.transip.nl (Postfix) with ESMTP id 9A0D53458422; Mon, 29 Jul 2013 14:37:42 +0200 (CEST) Message-ID: <51F66217.5080505@transip.nl> Date: Mon, 29 Jul 2013 14:37:43 +0200 From: Ali Niknam Organization: Transip BV User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Subject: nfsclient: incorrect st_blksize (bug?) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: rmacklem@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 12:37:43 -0000 Hi All, I've come across a problem that has proven to be unsolvable for me so far. It might be a bug in the NFS Client code, it also be my general lack of knowledge :). Can someone please give me a hint in the right direction? This is the case: mount_nfs -o rsize=32768 -o wsize=32768 -o nfsv4 -o tcp host:/path /mnt/nfs stat /mnt/nfs gives st_blksize of 4096 bytes. statfs /mnt/nfs gives an iosize of 4096 bytes. Mounting with nfsv3 gives the same results, regardless of udp or tcp protocol. NFSv2 however seems to give a st_blksize of 128k, with an iosize of 8192 bytes. In short: it seems that with BSD 9.1 the rsize/wsize's arent passed along correctly. I tried to debug it by looking in the kernel code but I got lost unfortunately in the abstraction layers (everything seems to set NFS_FABLKSIZE). Mounting the same host on a linux machine gives the correct st_blksize (32k). The disadvantage is ofcourse that apache/etc adhere to the 4k st_blksize by only reading 4k chunks so that nfs io slows down substantially. Ofcourse I'm also prepared to do further testing if that's needed. Kind Regards, Ali -- Transip BV | http://www.transip.nl/ From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 14:40:27 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5BEF8FD3 for ; Mon, 29 Jul 2013 14:40:27 +0000 (UTC) (envelope-from nobody@opus30.register.it) Received: from hostingsmtp.register.it (hostingsmtp07.register.it [81.88.50.248]) by mx1.freebsd.org (Postfix) with ESMTP id 9CCCB2AA5 for ; Mon, 29 Jul 2013 14:40:26 +0000 (UTC) Received: (qmail 2057 invoked from network); 29 Jul 2013 14:40:22 -0000 Received: from unknown (HELO opus30.register.it) by hostingsmtp.register.it with ESMTP; 29 Jul 2013 14:40:22 -0000 Received: (from nobody@localhost) by opus30.register.it (8.14.3/8.12.11/Submit) id r6TEeM3k031625; Mon, 29 Jul 2013 16:40:22 +0200 Date: Mon, 29 Jul 2013 16:40:22 +0200 X-RID: 7OnsJ3NuKzt0JW1yK2ZuOy9zbjtn6CjsL3RuI3RuK2Mt7FsvYiNbL3NjK2R06S8K|dDsnbSVjJW0jZCdbW1sK|MCwnLPkn4F0nXeDgCg==|WE5MR05JVFNPSAo= To: freebsd-fs@freebsd.org Subject: =?utf-8?B?UmVbMl06?= X-PHP-Originating-Script: 99:informations.php From: =?utf-8?B?0JvRg9C40LfQsCDQkNCy0LvRg9C60L7QstCw?= Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 14:40:27 -0000 Добрый день! Предлагаю сим карту для отдыхающих за границей звонки в Россию от 4,8 рублей. Срок действия карты неограничен, без абонентской платы. Подскажите, какой у Вас сотовый оператор, чтобы я могла уточнить сможете ли Вы привязать свой мобильный номер к новой сим карте? PS Могу сделать не только для Турции. с уважением, Людмила From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 14:59:06 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B723BACD; Mon, 29 Jul 2013 14:59:06 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 7C0922C35; Mon, 29 Jul 2013 14:59:05 +0000 (UTC) Received: from c122-106-156-23.carlnfd1.nsw.optusnet.com.au (c122-106-156-23.carlnfd1.nsw.optusnet.com.au [122.106.156.23]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id 545A610437FA; Tue, 30 Jul 2013 00:59:01 +1000 (EST) Date: Tue, 30 Jul 2013 00:59:01 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Ali Niknam Subject: Re: nfsclient: incorrect st_blksize (bug?) In-Reply-To: <51F66217.5080505@transip.nl> Message-ID: <20130729235447.S1849@besplex.bde.org> References: <51F66217.5080505@transip.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=YYGEuWhf c=1 sm=1 tr=0 a=ebeQFi2P/qHVC0Yw9JDJ4g==:117 a=PO7r1zJSAAAA:8 a=scOl8eOnVhUA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=P39JI5hRqiwA:10 a=wFYBqoTo3KADuSv-IfUA:9 a=CjuIK1q_8ugA:10 Cc: freebsd-fs@FreeBSD.org, rmacklem@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 14:59:06 -0000 On Mon, 29 Jul 2013, Ali Niknam wrote: > I've come across a problem that has proven to be unsolvable for me so far. It > might be a bug in the NFS Client code, it also be my general lack of > knowledge :). Can someone please give me a hint in the right direction? > > This is the case: > > mount_nfs -o rsize=32768 -o wsize=32768 -o nfsv4 -o tcp host:/path /mnt/nfs > > stat /mnt/nfs gives st_blksize of 4096 bytes. > statfs /mnt/nfs gives an iosize of 4096 bytes. > > Mounting with nfsv3 gives the same results, regardless of udp or tcp > protocol. NFSv2 however seems to give a st_blksize of 128k, with an iosize of > 8192 bytes. > > In short: it seems that with BSD 9.1 the rsize/wsize's arent passed along > correctly. I tried to debug it by looking in the kernel code but I got lost > unfortunately in the abstraction layers (everything seems to set > NFS_FABLKSIZE). > > Mounting the same host on a linux machine gives the correct st_blksize (32k). > > The disadvantage is ofcourse that apache/etc adhere to the 4k st_blksize by > only reading 4k chunks so that nfs io slows down substantially. nfs still seems to seems to ask for a blocksize of NFS_FABLKSIZE = 512. Old versions of FreeBSD honored the leaf file system's idea of the best block size and gave this 512. After many intermediate broken versions, vn_stat() now has a hack that involves it using PAGE_SIZE iff the leaf file system prefers a smaller size, so 512 becomes 4096 on x86. 4096 is not as bad as 512, but still too small for most purposes. OTOH, 512 works quite well for nfs over local networks with low latency. 512 fits in a 1500-byte packet but 4096 doesn't, so latency can be better with small block sizes and lower latency also gives higher throughput provided everything can keep up with the small blocks. A workaround might by to use statfs() instead of stat(). st_blksize can vary within a file system in theory, but usually doesn't, and can't be trusted anyway. struct statfs has fields f_bsize ("fragment" size) and f_iosize (optimal transfer size). These seem to be set better by leaf file systems, and are certainly never frobbed by upper layers (except to translate to old statfs()). nfs still seems to set f_bsize to NFS_FABSLKSIZE, but it sets f_iosize to its i/o size. ffs sets f_bsize to its fragment size (not so good. statfs() can't even respresent ffs's 2 types of block size. Neither can stat(), but st_blksize is initialized with the other one, so unportable code can determine both). ffs sets f_iosize to a disk-specific size. There are many bugs in the setting of the latter too, and it now almost always reduces to a hard-coded setting of MAXPHYS that has nothing to do with disks' preferred sizes. Hard-coding of MAXPHYS everywhere would be OK for throughput but not so good for latency. To optimize for latency, there seems to be nothing better than using statfs()'s f_bsize, but we know that that reduces to a hard-coded 512 for nfs and to the not-necessarily best fragment size for ffs. Bruce From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 14:25:50 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 29707A96; Mon, 29 Jul 2013 14:25:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CD6D329B8; Mon, 29 Jul 2013 14:25:48 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA06580; Mon, 29 Jul 2013 17:25:40 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1V3oOJ-00072l-ND; Mon, 29 Jul 2013 17:25:39 +0300 Message-ID: <51F67B2A.3040302@FreeBSD.org> Date: Mon, 29 Jul 2013 17:24:42 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130708 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-arch@FreeBSD.org Subject: translate INVARIANTS to DEBUG for code from OpenSolaris X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=X-VIET-VPS Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 29 Jul 2013 17:39:40 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 14:25:50 -0000 [zfs-devel@, fs@, dtrace@ are Bcc-ed] In OpenSolaris and its descendants DEBUG is used in a fashion similar to our INVARIANTS. For example, ASSERT macros are enabled by it. In our kernel code DEBUG has a different meaning and enables far too verbose or far too obscure code and, as such, it is very rarely enabled. The idea of a change that I would like to propose is to translate INVARIANTS kernel option into DEBUG for the files that originated from OpenSolaris (and hopefully only for them). The change: opensolaris code: translate INVARIANTS to DEBUG and ZFS_DEBUG do this by forcing inclusion of sys/cddl/compat/opensolaris/sys/debug_compat.h via -include option into all source files from OpenSolaris. Note that this -include option must always be after -include opt_global.h. Additionally, remove forced definition of DEBUG for some modules and fix their build without DEBUG. Also, meaning of DEBUG was overloaded to enable WITNESS support for some OpenSolaris (primarily ZFS) locks. Now this overloading is removed and that use of DEBUG is replaced with a new option OPENSOLARIS_WITNESS. http://people.freebsd.org/~avg/osol-invariants-debug.diff I would like to ask for your feedback on the soundness of the whole idea. Also on the name, location and style of inclusion for sys/cddl/compat/opensolaris/sys/debug_compat.h. And on any other details of the proposed change. Testing is also welcome, of course. Thank you very much. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Jul 29 22:48:45 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3D3AE7D; Mon, 29 Jul 2013 22:48:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E3ED92547; Mon, 29 Jul 2013 22:48:44 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqMEALbw9lGDaFve/2dsb2JhbABbgztQgxC6WYEydIIkAQEBAwEBAQEgKyALGw4KAgINGQIpAQkmBggCBQQBHASHaQYMpwiRX4EojSZ7ATMHgmWBIgOVGINwiHiHK4MwIDKBAzk X-IronPort-AV: E=Sophos;i="4.89,773,1367985600"; d="scan'208";a="42603739" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 29 Jul 2013 18:48:37 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id AD5B8B403E; Mon, 29 Jul 2013 18:48:37 -0400 (EDT) Date: Mon, 29 Jul 2013 18:48:37 -0400 (EDT) From: Rick Macklem To: Bruce Evans Message-ID: <728189143.3592552.1375138117697.JavaMail.root@uoguelph.ca> In-Reply-To: <20130729235447.S1849@besplex.bde.org> Subject: Re: nfsclient: incorrect st_blksize (bug?) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs@FreeBSD.org, rmacklem@FreeBSD.org, Ali Niknam X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 22:48:45 -0000 Bruce Evans wrote: > On Mon, 29 Jul 2013, Ali Niknam wrote: > > > I've come across a problem that has proven to be unsolvable for me > > so far. It > > might be a bug in the NFS Client code, it also be my general lack > > of > > knowledge :). Can someone please give me a hint in the right > > direction? > > > > This is the case: > > > > mount_nfs -o rsize=32768 -o wsize=32768 -o nfsv4 -o tcp host:/path > > /mnt/nfs > > > > stat /mnt/nfs gives st_blksize of 4096 bytes. > > statfs /mnt/nfs gives an iosize of 4096 bytes. > > This value is not the rsize, wsize being used for I/O RPCs on the wire. For recent systems, you can find out what is being used on the wire via: "nfsstat -m" executed as root. For older FreeBSD systems, you can look at the read and write RPCs via wireshark. If you have specified 32768, that is what you should get unless the NFS server has specified a smaller value. The 4096 value is just the page size and it is set that way (rather bogusly) to make the NFS paging code work correctly. (The only time this affects what happens on the wire is when mmap'd files are read/written, since the NFS VOP_GETPAGES/VOP_PUTPAGES() get 4096 byte pages to read/write. This needs to be fixed someday by making the NFS paging VOPs do scatter/gather I/O.) At least that is how I believe it works, although I am not familiar with the vm side of things. rick > > Mounting with nfsv3 gives the same results, regardless of udp or > > tcp > > protocol. NFSv2 however seems to give a st_blksize of 128k, with an > > iosize of > > 8192 bytes. > > > > In short: it seems that with BSD 9.1 the rsize/wsize's arent passed > > along > > correctly. I tried to debug it by looking in the kernel code but I > > got lost > > unfortunately in the abstraction layers (everything seems to set > > NFS_FABLKSIZE). > > > > Mounting the same host on a linux machine gives the correct > > st_blksize (32k). > > > > The disadvantage is ofcourse that apache/etc adhere to the 4k > > st_blksize by > > only reading 4k chunks so that nfs io slows down substantially. > > nfs still seems to seems to ask for a blocksize of NFS_FABLKSIZE = > 512. Old > versions of FreeBSD honored the leaf file system's idea of the best > block > size and gave this 512. After many intermediate broken versions, > vn_stat() > now has a hack that involves it using PAGE_SIZE iff the leaf file > system > prefers a smaller size, so 512 becomes 4096 on x86. 4096 is not as > bad as > 512, but still too small for most purposes. OTOH, 512 works quite > well for > nfs over local networks with low latency. 512 fits in a 1500-byte > packet > but 4096 doesn't, so latency can be better with small block sizes and > lower latency also gives higher throughput provided everything can > keep > up with the small blocks. > > A workaround might by to use statfs() instead of stat(). st_blksize > can vary within a file system in theory, but usually doesn't, and > can't > be trusted anyway. struct statfs has fields f_bsize ("fragment" > size) > and f_iosize (optimal transfer size). These seem to be set better by > leaf file systems, and are certainly never frobbed by upper layers > (except to translate to old statfs()). nfs still seems to set > f_bsize > to NFS_FABSLKSIZE, but it sets f_iosize to its i/o size. ffs sets > f_bsize to its fragment size (not so good. statfs() can't even > respresent ffs's 2 types of block size. Neither can stat(), but > st_blksize is initialized with the other one, so unportable code can > determine both). ffs sets f_iosize to a disk-specific size. There > are many bugs in the setting of the latter too, and it now almost > always reduces to a hard-coded setting of MAXPHYS that has nothing > to do with disks' preferred sizes. Hard-coding of MAXPHYS everywhere > would be OK for throughput but not so good for latency. To optimize > for latency, there seems to be nothing better than using statfs()'s > f_bsize, but we know that that reduces to a hard-coded 512 for nfs > and to the not-necessarily best fragment size for ffs. > > Bruce > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 01:08:58 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 043DF7C2 for ; Wed, 31 Jul 2013 01:08:58 +0000 (UTC) (envelope-from berend@pobox.com) Received: from smtp.pobox.com (b-pb-sasl-quonix.pobox.com [208.72.237.35]) by mx1.freebsd.org (Postfix) with ESMTP id B29F3229B for ; Wed, 31 Jul 2013 01:08:57 +0000 (UTC) Received: from smtp.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 74F91292F2; Wed, 31 Jul 2013 01:08:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date :message-id:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=sasl; bh=tOKbqFCvyu8I0gHJ/C7YQKsqlR8=; b=yb5vUgbO5Kdxd8TSb/W8w++18cLE QGRDmNhz07RyiROul30r+qXe3EAvg8G8/jcBi58XZUPrRnRt+43lc64z7EjVg6Gf L3znje0MNItfMqSnGvaXgbmCJ19IrLZts7sDMZLKeNo+s98ZdcEW8Zbs1zcRSizw 2RMPunXj1Zsy09U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:message-id :from:to:cc:subject:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=f+yWfY YIY7C/oVXH0elruTAJJWKH5yw0PIrTjHHREo4TbLyLNZLHnyAfY9f8eVyUYDcb/x N3uMnM6/ruaaAc2nQGbVYb4DljJry9OD26kfEnW9EfVixv6wATv2Obg+uTTdZhGX oA5wsFMVgkcqYfDgn++fLlSAyYSYFdupApFbY= Received: from b-pb-sasl-quonix.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 67240292F1; Wed, 31 Jul 2013 01:08:49 +0000 (UTC) Received: from bmach.nederware.nl (unknown [27.252.146.198]) by b-sasl-quonix.pobox.com (Postfix) with ESMTPA id 57972292EF; Wed, 31 Jul 2013 01:08:48 +0000 (UTC) Received: from quadrio.nederware.nl (quadrio.nederware.nl [192.168.33.13]) by bmach.nederware.nl (Postfix) with ESMTP id 73F795C80; Wed, 31 Jul 2013 13:07:20 +1200 (NZST) Received: from quadrio.nederware.nl (quadrio.nederware.nl [127.0.0.1]) by quadrio.nederware.nl (Postfix) with ESMTP id 09202404431E; Wed, 31 Jul 2013 13:08:44 +1200 (NZST) Date: Wed, 31 Jul 2013 13:08:43 +1200 Message-ID: <874nbb7cys.wl%berend@pobox.com> From: Berend de Boer To: Rick Macklem Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 In-Reply-To: <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/24.3 (i686-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Xplain Technology Ltd MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: multipart/signed; boundary="pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1"; micalg=pgp-sha256; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit X-Pobox-Relay-ID: C14AD838-F97D-11E2-99C0-E84251E3A03C-48001098!b-pb-sasl-quonix.pobox.com Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 01:08:58 -0000 --pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1 Content-Type: text/plain; charset=US-ASCII >>>>> "Rick" == Rick Macklem writes: Rick> I think you mentioned that you were using a Linux client, Rick> but not what version. I'd suggest a recent kernel from Rick> kernel.org. (Fedora tracks updates/fixes for NFSv4 pretty Rick> closely, so the newest Fedora release should be pretty Rick> current.) This was Ubuntu 10.04 LTS. Have just tried a FreeBSD 9.1 client. Similar numbers. NFSv3 is about 30% slower on FreeBSD than Linux: 3m30s versus 2m10s. NFSv4 has the same terribly slow performance, i.e. 21m56s for the same test. Interestingly, the nfsd cpu usage doesn't rise as high as with Linux. But goes up to 20% (instead of over 50%). I had a look at collectd measurements as well, one cpu on the FreeBSD server is spending a lot of time in IRQ (whatever that means). BTS, this was a FreeBSD NFS4 out-of-the-box server, not with the patch (as the patch didn't do that much for me, it did some, but performance was still 8 times slower than nfs3). Rick> All I can suggest is capturing packets and then emailing be Rick> the captured packet trace. You can use tcpdump to do the Rick> capture, since wireshark will understand it: # tcpdump -s 0 Rick> -w .pcap host and then emailing me Rick> .pcap. Rick> I can take a look at the packet capture and maybe see what Rick> is going on. Will email them shortly. -- All the best, Berend de Boer ------------------------------------------------------ Awesome Drupal hosting: https://www.xplainhosting.com/ --pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1 Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit Content-Description: OpenPGP Digital Signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABCAAGBQJR+GObAAoJEKOfeD48G3g52tUQANGXsgDpVeOJaTCdCrL87Lfv JHU7KDVfHF2EsODVw0zV0nO/nd+iyggzjAO8sc4EUDXU9HljLOsZccmu9mUr1X/Q ytwuLN57ZL+RbQab9Mu7U/+NeGu0qhT1Y+RPAL4zeNYBRSXECAkqwD9z2pPmA/ca D7MAXRqssU6toPIb44xoTMyfxIPEWEjFRk5dgSWdJHXrsUmFgrfEQhAc4gU/1tS0 nH5Y/zEGi3Tp99pmZczncSrDIk6SkzA7ZxKyIfDSpZfdcd9cqgm8olZPXXy+OYHu o1PnFMHcCb5e77+ALlfExXvN+AQHfoA+VXXTK/XOeyPY9LH/lo6v6OlgnzEjcEd4 tZ9SSYTFF6UyDZc6fatzLLJwhcqRCZPwWdXKtPCdEGorUBONVZ626b304eUc4iG4 0CBIPU5Ziang+8+2PDnAxyDzJmUmHljWu8Cr54+hZjxs0m0T07hNHLaoDpqtLHB3 VMy1BSg2ABZ5+oOtBx5IS1uBDVqXMUGbxJfhvHxI/iyAdOhz18wgiv5e+HlVkvLn aOgCUgFdILBh8OquA1uUzfTWkadndcHHSpF7mN9OzM+Vc1i6KJIDyGpKmCIHLbVm yreJGrTDM/gS2WjqXpyuzf3mqFzVP5bwLsvstRIU//2zEGDqy8WLhLgvxvRFm9zv YaIA98hQsJKA6H2WqtzD =2Alm -----END PGP SIGNATURE----- --pgp-sign-Multipart_Wed_Jul_31_13:08:43_2013-1-- From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 04:38:10 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4BFEFA8F for ; Wed, 31 Jul 2013 04:38:10 +0000 (UTC) (envelope-from boydjd@jbip.net) Received: from mail-oa0-x231.google.com (mail-oa0-x231.google.com [IPv6:2607:f8b0:4003:c02::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 10DE428ED for ; Wed, 31 Jul 2013 04:38:09 +0000 (UTC) Received: by mail-oa0-f49.google.com with SMTP id n16so525093oag.22 for ; Tue, 30 Jul 2013 21:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jbip.net; s=google; h=mime-version:x-originating-ip:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=dC5swr3+6Gw/VXMUTGkkXQ1hAZ16Ondf554NYlmymz0=; b=GXZ/MqUo3oVmcX3fOaLSROjUWeBXZ/1RtUfHnwocdJcb4ZVDmLopqnCyt/iEdX3r3d C6hCNItHqTDxXysLllQIVh6W8SXtBeIXFxMx/NgBd5SWlFk/EvPPZbfQhFA5g+aQ4Mhp 6vg6rJdvzRieNjLotxza6m8yBz3vf/1NLRs4s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:x-gm-message-state; bh=dC5swr3+6Gw/VXMUTGkkXQ1hAZ16Ondf554NYlmymz0=; b=HY3g5iF/q6cT/EWttUoAy7jXg1IbzxvyXXGbUbm1r5g+3UMgLnoLH/FwzVBVKOXcZs mbhMhJKF0JOiyB25DBsDkpkW0h8Kl/m31N4r3MxQg8GeWVQGHxpkxRMEMaFxlA2a/1BM 7IgVXh++UinmuOghHO12EFZDhEjPBROqr6gJyQV/bXuIoqypiPSHS7gSP4/hVHNwvUww vNEn/9WsrlMYWUeyTfT1UzuKhv33bUwLFapXi1E08WUxEGBfW/9YVlK4m73S4TQZ/kBN hWiPRUcshQ591sfK209VOhUljIkFn48LaGsQ5REcOoYU+k064FXBRa5CmpC/7zQ9WUHr 67/Q== X-Received: by 10.42.228.68 with SMTP id jd4mr22599902icb.44.1375245489086; Tue, 30 Jul 2013 21:38:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.100.234 with HTTP; Tue, 30 Jul 2013 21:37:49 -0700 (PDT) X-Originating-IP: [76.120.245.171] In-Reply-To: <874nbb7cys.wl%berend@pobox.com> References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> <874nbb7cys.wl%berend@pobox.com> From: Joshua Boyd Date: Wed, 31 Jul 2013 00:37:49 -0400 Message-ID: Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 To: Berend de Boer X-Gm-Message-State: ALoCoQmUC5nsUtxT4xsSlPCCtbB/iCugyjZaFfceUciqOmZcuSomCJ2aqhFgJq9NAA1/OU+GI3Nw Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 04:38:10 -0000 Are you using EBS or ephemeral storage? If you are using EBS, how many provisioned IOPS and how many volumes are you using? In what configuration? What ec2 instance size are you using? Are you running your tests against the NFS server from within EC2, or from a host outside of EC2? On Tue, Jul 30, 2013 at 9:08 PM, Berend de Boer wrote: > >>>>> "Rick" == Rick Macklem writes: > > Rick> I think you mentioned that you were using a Linux client, > Rick> but not what version. I'd suggest a recent kernel from > Rick> kernel.org. (Fedora tracks updates/fixes for NFSv4 pretty > Rick> closely, so the newest Fedora release should be pretty > Rick> current.) > > This was Ubuntu 10.04 LTS. > > Have just tried a FreeBSD 9.1 client. Similar numbers. NFSv3 is about > 30% slower on FreeBSD than Linux: 3m30s versus 2m10s. NFSv4 has the > same terribly slow performance, i.e. 21m56s for the same test. > > Interestingly, the nfsd cpu usage doesn't rise as high as with > Linux. But goes up to 20% (instead of over 50%). > > I had a look at collectd measurements as well, one cpu on the FreeBSD > server is spending a lot of time in IRQ (whatever that means). > > BTS, this was a FreeBSD NFS4 out-of-the-box server, not with the patch > (as the patch didn't do that much for me, it did some, but performance > was still 8 times slower than nfs3). > > > Rick> All I can suggest is capturing packets and then emailing be > Rick> the captured packet trace. You can use tcpdump to do the > Rick> capture, since wireshark will understand it: # tcpdump -s 0 > Rick> -w .pcap host and then emailing me > Rick> .pcap. > > Rick> I can take a look at the packet capture and maybe see what > Rick> is going on. > > Will email them shortly. > > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/ > -- Joshua Boyd E-mail: boydjd@jbip.net http://www.jbip.net From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 06:37:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 89E8A8FA for ; Wed, 31 Jul 2013 06:37:34 +0000 (UTC) (envelope-from berend@pobox.com) Received: from smtp.pobox.com (b-pb-sasl-quonix.pobox.com [208.72.237.35]) by mx1.freebsd.org (Postfix) with ESMTP id 43ED62C8C for ; Wed, 31 Jul 2013 06:37:33 +0000 (UTC) Received: from smtp.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 2F8612C9CB; Wed, 31 Jul 2013 06:37:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date :message-id:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=sasl; bh=K3y8RFm6UUMamkFWnsCYfPV0KdU=; b=SGimK2ioNNgNRlzcPQYFWqUO3HHY JAIK4ZzWoVgLCMV+2aI+adPh3660nqdjLZhwjpMsgynspcRf3gIjjG/OjkKbATim W/EgV0TmjjMkAI8ufumIaPAKbVvxFiB9jvFXQKIanAGCugTLxAhT9c5JBzvJV+se iMydv6vIIT49NtY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:message-id :from:to:cc:subject:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=Upcy9K 1v7dFGf5FKuVogPdzsemRK/NtbhbWoJrPQVdMlldukkZWK+bcffIjrSLdhWG4JjB 9dGGfZeIBhCLGoagfnMY7CLWdqbkbzv6CJ9bNje3ZU0jM3ASOoSC6pF8pbJ0dcAh c6349olEe2MGZwdMQcwLtjQ9XGvRKVs09pr2k= Received: from b-pb-sasl-quonix.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 2438C2C9C8; Wed, 31 Jul 2013 06:37:33 +0000 (UTC) Received: from bmach.nederware.nl (unknown [27.252.146.198]) by b-sasl-quonix.pobox.com (Postfix) with ESMTPA id 7CA862C9C3; Wed, 31 Jul 2013 06:37:32 +0000 (UTC) Received: from quadrio.nederware.nl (quadrio.nederware.nl [192.168.33.13]) by bmach.nederware.nl (Postfix) with ESMTP id 9D5645C80; Wed, 31 Jul 2013 18:36:04 +1200 (NZST) Received: from quadrio.nederware.nl (quadrio.nederware.nl [127.0.0.1]) by quadrio.nederware.nl (Postfix) with ESMTP id 3D915404432C; Wed, 31 Jul 2013 18:37:28 +1200 (NZST) Date: Wed, 31 Jul 2013 18:37:28 +1200 Message-ID: <87ob9j5j6f.wl%berend@pobox.com> From: Berend de Boer To: Joshua Boyd Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 In-Reply-To: References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> <874nbb7cys.wl%berend@pobox.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/24.3 (i686-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Xplain Technology Ltd MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: multipart/signed; boundary="pgp-sign-Multipart_Wed_Jul_31_18:37:27_2013-1"; micalg=pgp-sha256; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit X-Pobox-Relay-ID: ADCDB1F8-F9AB-11E2-9A1A-E84251E3A03C-48001098!b-pb-sasl-quonix.pobox.com Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 06:37:34 -0000 --pgp-sign-Multipart_Wed_Jul_31_18:37:27_2013-1 Content-Type: text/plain; charset=US-ASCII >>>>> "Joshua" == Joshua Boyd writes: Joshua> Are you using EBS or ephemeral storage? Both. Joshua> If you are using EBS, how many provisioned IOPS and how Joshua> many volumes are you using? In what configuration? It doesn't matter a iota what you use. It's the NFS4 server in FreeBSD. Not using any provisioned EBS. Have tested against ephemeral zfs mirror, ebs root disk ufs, and raidz2 zfs with 4 EBS disks. Joshua> What ec2 instance size are you using? m1.large. Joshua> Are you running your tests against the NFS server from Joshua> within EC2, or from a host outside of EC2? within same zone. All these details are irrelevant. See my test nfs3 versus nfs4. We have a factor 10 difference here. Any hardware/disk factor is irrelevant here. It's the FreeBSD nfs4 implementation. I wish I had the time to solve it, sounds like an interesting problem, but a lot of work I'm afraid. -- All the best, Berend de Boer ------------------------------------------------------ Awesome Drupal hosting: https://www.xplainhosting.com/ --pgp-sign-Multipart_Wed_Jul_31_18:37:27_2013-1 Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit Content-Description: OpenPGP Digital Signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABCAAGBQJR+LCnAAoJEKOfeD48G3g5WzoQALOFFHdMcEzpZehWHGTaM5BI xaTaSFFI101WP8Z13rZEU4gffXXOyAHRWvpQACBhn5rF5BgiDxkLsRbNjhodBGVm 2yML8IJU+p3oNdTQDxMWRHfpYfvyBVNa1ifo3ki4xCzkgtE4ccb/HiGGE9C+JMKR IlfIeSpy0zBgFJnFO39bnCYgCWql+AveQ1TqunZevSZ+F18JEWHmoN5cmTAulJRm TcJ9yH4IzgUetS2KNh9PiLDp3mWnDA/IKtW3WXX43baeUb7AqvsPNf7iWhMpOiYd YQR7sNl2b1gXO+s4glRGDC4idTdqIs5S+Q6mOQESPVxgPNrLsQ19SYG6gNQ34Xz4 Xl0g/c0fxDiAA1skHseEP4Cuwo003msvEupxf/dgjY1+9jNq4lfQ/3sphalamPYc JdWYqLc77kyv8zkb4fn5nOSVBJCjrbq6eh5hLYoUYgY8j2Jwl5F9hH9H8aCdU/sM s2TqMWNWpPMjKrPq5mDAd+Ws7sVl2wTm4+bQJ207Pv6D42dCtNKpt78ovZeAhqG5 /rColvWPg4UuuAiz9qnwJTEfIn7Dg8oCLEISzXnZ1CnS+DoO8IJ4HUEqPQKvzDXz bIYZN6fpEftCUSSh9Nahy9WlgWOnw7d/rODwaok5G/cEgLCvEuZDoFjSYGWs8vtm fwLLXLi8Ypp0ASjaPuFo =7Lbs -----END PGP SIGNATURE----- --pgp-sign-Multipart_Wed_Jul_31_18:37:27_2013-1-- From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 07:22:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 39664B0E for ; Wed, 31 Jul 2013 07:22:40 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E7CEB2E44 for ; Wed, 31 Jul 2013 07:22:39 +0000 (UTC) Received: from Julian-MBP3.local (etroy.elischer.org [121.45.226.51]) (authenticated bits=0) by vps1.elischer.org (8.14.7/8.14.6) with ESMTP id r6V7MX1n032062 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 31 Jul 2013 00:22:35 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <51F8BB33.2060901@freebsd.org> Date: Wed, 31 Jul 2013 15:22:27 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Berend de Boer Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> <874nbb7cys.wl%berend@pobox.com> In-Reply-To: <874nbb7cys.wl%berend@pobox.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 07:22:40 -0000 On 7/31/13 9:08 AM, Berend de Boer wrote: >>>>>> "Rick" == Rick Macklem writes: > Rick> I think you mentioned that you were using a Linux client, > Rick> but not what version. I'd suggest a recent kernel from > Rick> kernel.org. (Fedora tracks updates/fixes for NFSv4 pretty > Rick> closely, so the newest Fedora release should be pretty > Rick> current.) > > This was Ubuntu 10.04 LTS. > > Have just tried a FreeBSD 9.1 client. Similar numbers. NFSv3 is about > 30% slower on FreeBSD than Linux: 3m30s versus 2m10s. NFSv4 has the > same terribly slow performance, i.e. 21m56s for the same test. > > Interestingly, the nfsd cpu usage doesn't rise as high as with > Linux. But goes up to 20% (instead of over 50%). > > I had a look at collectd measurements as well, one cpu on the FreeBSD > server is spending a lot of time in IRQ (whatever that means). > > BTS, this was a FreeBSD NFS4 out-of-the-box server, not with the patch > (as the patch didn't do that much for me, it did some, but performance > was still 8 times slower than nfs3). > > > Rick> All I can suggest is capturing packets and then emailing be > Rick> the captured packet trace. You can use tcpdump to do the > Rick> capture, since wireshark will understand it: # tcpdump -s 0 > Rick> -w .pcap host and then emailing me > Rick> .pcap. > > Rick> I can take a look at the packet capture and maybe see what > Rick> is going on. > > Will email them shortly. Recent evidence with AWS is suggesting that the NOADAPTIVE_XXX options in the XENHVM kernel are now seriously hindering AWS performance all over the place. make sure you have tried with these options removed. > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/ From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 09:12:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6C2BFF53; Wed, 31 Jul 2013 09:12:35 +0000 (UTC) (envelope-from berend@pobox.com) Received: from smtp.pobox.com (b-pb-sasl-quonix.pobox.com [208.72.237.35]) by mx1.freebsd.org (Postfix) with ESMTP id 25769241F; Wed, 31 Jul 2013 09:12:34 +0000 (UTC) Received: from smtp.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 9B4F531352; Wed, 31 Jul 2013 09:12:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date :message-id:from:to:cc:subject:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=sasl; bh=0v4gq8I96HsiyeCZ1Tz9h+64Aw0=; b=NiVwqTKQfUWutbcYcCpBWHo0rACe IIecE/9+IHimmDIdYzoY9rfKPSWUPKj6ah2THJaaebyf7E6pjryDjXVO28q6OC3o 5/AAh9BKn/2gJovtdufMSutz+3IpZqHbG3zEDaNEdIS5kzOPRaBeus8qmADiLp3E gZdwEsodjTW2UTo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:message-id :from:to:cc:subject:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=sasl; b=IGDe+a CQQ9uMxyX1M9EXGwxoRUyaj6z1i6npszIH++hQnUxIUQBxmLaywMOAskYDQuoebH kwYgOARYNO/Sy6tfSNFUUEl0XX4vHbCxNo97r1XJ8gZbIiE/BiNiukzrqU2MJnVV w7I5PaAVlRsLuixyhnrouoPheuVyRRJ+xkht8= Received: from b-pb-sasl-quonix.pobox.com (unknown [127.0.0.1]) by b-sasl-quonix.pobox.com (Postfix) with ESMTP id 907FD31351; Wed, 31 Jul 2013 09:12:33 +0000 (UTC) Received: from bmach.nederware.nl (unknown [27.252.146.198]) by b-sasl-quonix.pobox.com (Postfix) with ESMTPA id D68EA3134F; Wed, 31 Jul 2013 09:12:32 +0000 (UTC) Received: from quadrio.nederware.nl (quadrio.nederware.nl [192.168.33.13]) by bmach.nederware.nl (Postfix) with ESMTP id 0973B5D58; Wed, 31 Jul 2013 21:11:05 +1200 (NZST) Received: from quadrio.nederware.nl (quadrio.nederware.nl [127.0.0.1]) by quadrio.nederware.nl (Postfix) with ESMTP id A9D75404432D; Wed, 31 Jul 2013 21:12:28 +1200 (NZST) Date: Wed, 31 Jul 2013 21:12:28 +1200 Message-ID: <87iozr5c03.wl%berend@pobox.com> From: Berend de Boer To: Julian Elischer Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + UFS/ZFS + AWS EC2 In-Reply-To: <51F8BB33.2060901@freebsd.org> References: <8761wfvwml.wl%berend@pobox.com> <153512858.1034456.1373755540674.JavaMail.root@uoguelph.ca> <874nbb7cys.wl%berend@pobox.com> <51F8BB33.2060901@freebsd.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/24.3 (i686-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Xplain Technology Ltd MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: multipart/signed; boundary="pgp-sign-Multipart_Wed_Jul_31_21:12:28_2013-1"; micalg=pgp-sha256; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit X-Pobox-Relay-ID: 55418170-F9C1-11E2-806A-E84251E3A03C-48001098!b-pb-sasl-quonix.pobox.com Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 09:12:35 -0000 --pgp-sign-Multipart_Wed_Jul_31_21:12:28_2013-1 Content-Type: text/plain; charset=US-ASCII >>>>> "Julian" == Julian Elischer writes: Julian> Recent evidence with AWS is suggesting that the Julian> NOADAPTIVE_XXX options in the XENHVM kernel are now Julian> seriously hindering AWS performance all over the place. Julian> make sure you have tried with these options removed. Julian, on the same machine nfs3 is better than Linux, and nfs4 is terrible. -- All the best, Berend de Boer ------------------------------------------------------ Awesome Drupal hosting: https://www.xplainhosting.com/ --pgp-sign-Multipart_Wed_Jul_31_21:12:28_2013-1 Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit Content-Description: OpenPGP Digital Signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABCAAGBQJR+NT8AAoJEKOfeD48G3g5HQ4QAKKakcHpiWbPF8kiSI6XiyFQ yt6U62goFcQ8Uj0er4yKx3eu2W+yd7e/9zWFNl2q1zhISze7zHW4mN/DK8g0g5oT R9QuFahYBlb+ZGowVGx59BCwxAiOw2We+I5IA+PUI0+CAXbDXNe2+622UOVAXSB4 YROulVzyGGpgazw3h6rH/HC3U32BTqTuI6br/hTz00VXEcf4jUBvhl6MJBDM196J pBaxRSrb16usUNH3uimPGOpgePNDU0yhbqLbGZPWM7sD0B95D7lYHAZtdkU+bGGX VDDEecI6gNrrGviSSEl2C42liN1ihtrQBDItPFrrEwL9WQ9/VhIqFcdcC3rCCUJ0 xCsq9dVfQxtV2y+C0UbeW3moQSN8YpYObnzUTiYLXIRTaSMPWA5h5dYcQkBp4ZZY zeXC006eeEVgGRl8jMpVj3BSsSbAblT13ac94YeEWBsv9phMTQ0eJWGcIxH5iQDZ c5xrZlfhUxqOfn5qer6QyqIikfdmetPO8yIMQr7UApGW1FRNoPwOJt6C++RhYxBv 1fGyHztr538MvY+qzl0chcNtza3+/oMQJXj1KaDXUyPjveFo9JDQiTrpInIallr/ sGNDj1sy6aj5OcW6eC2/ppBl3wB7lPl4Hiq9HKjzxT4O47lIIqt24gjpLZr7fYqK JABKfM+Sp4EkFovPYM6F =f093 -----END PGP SIGNATURE----- --pgp-sign-Multipart_Wed_Jul_31_21:12:28_2013-1-- From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 16:40:01 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DB55EE10 for ; Wed, 31 Jul 2013 16:40:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C96FB2B05 for ; Wed, 31 Jul 2013 16:40:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6VGe1XS028178 for ; Wed, 31 Jul 2013 16:40:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6VGe1BT028177; Wed, 31 Jul 2013 16:40:01 GMT (envelope-from gnats) Date: Wed, 31 Jul 2013 16:40:01 GMT Message-Id: <201307311640.r6VGe1BT028177@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: "Christopher D. Harrison" Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: "Christopher D. Harrison" List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 16:40:01 -0000 The following reply was made to PR kern/177536; it has been noted by GNATS. From: "Christopher D. Harrison" To: bug-followup@FreeBSD.org, Martin.Birgmeier@aon.at Cc: Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load Date: Wed, 31 Jul 2013 11:36:26 -0500 I have also been hitting this bug with 9.1-RELEASE. My debugging has got me to the point where I am almost certain the problem is in the MPS driver in 9.1-RELEASE. 9.0-RELEASE would not deadlock under high I/O but 9.1-RELEASE running the mps driver seems too. Martin: What device are your drives connected too? Is it an LSI HBA? My device is: mps0@pci0:5:0:0: class=0x010700 card=0x30c01000 chip=0x00641000 rev=0x02 hdr=0x00 vendor = 'LSI Logic / Symbios Logic' device = 'SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor]' class = mass storage subclass = SAS igb2@pci0:4:0:0: class=0x020000 card=0x10c915d9 chip=0x10c98086 rev=0x01 hdr=0x00 I have 6 systems in production which experience this deadlock issue. The deadlock appears most frequently under high load but I have experienced it on systems which are idle too. -C From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 20:10:02 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id DFA55F80 for ; Wed, 31 Jul 2013 20:10:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CA98E259B for ; Wed, 31 Jul 2013 20:10:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6VKA1L7070420 for ; Wed, 31 Jul 2013 20:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6VKA1jK070419; Wed, 31 Jul 2013 20:10:01 GMT (envelope-from gnats) Date: Wed, 31 Jul 2013 20:10:01 GMT Message-Id: <201307312010.r6VKA1jK070419@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Martin Birgmeier Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 20:10:02 -0000 The following reply was made to PR kern/177536; it has been noted by GNATS. From: Martin Birgmeier To: "Christopher D. Harrison" Cc: bug-followup@FreeBSD.org Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load Date: Wed, 31 Jul 2013 21:56:20 +0200 No, this is not an LSI HBA, but rather atapci0: port 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3 atapci1: at channel -1 on atapci0 atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported ata2: on atapci1 ata3: on atapci1 ata4: at channel 0 on atapci0 atapci2: port 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0 atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported ata5: at channel 0 on atapci2 ata6: at channel 1 on atapci2 ata7: at channel 2 on atapci2 ata8: at channel 3 on atapci2 ata9: at channel 4 on atapci2 ata10: at channel 5 on atapci2 ada0 at ata5 bus 0 scbus3 target 0 lun 0 ada1 at ata6 bus 0 scbus4 target 0 lun 0 ada2 at ata7 bus 0 scbus5 target 0 lun 0 ada3 at ata8 bus 0 scbus6 target 0 lun 0 ada4 at ata9 bus 0 scbus7 target 0 lun 0 ada5 at ata10 bus 0 scbus8 target 0 lun 0 (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.) Regards, Martin From owner-freebsd-fs@FreeBSD.ORG Wed Jul 31 21:40:02 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2B4E86B8 for ; Wed, 31 Jul 2013 21:40:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 180A628BB for ; Wed, 31 Jul 2013 21:40:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6VLe12u087728 for ; Wed, 31 Jul 2013 21:40:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6VLe1PS087727; Wed, 31 Jul 2013 21:40:01 GMT (envelope-from gnats) Date: Wed, 31 Jul 2013 21:40:01 GMT Message-Id: <201307312140.r6VLe1PS087727@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: "Christopher D. Harrison" Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: "Christopher D. Harrison" List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Jul 2013 21:40:02 -0000 The following reply was made to PR kern/177536; it has been noted by GNATS. From: "Christopher D. Harrison" To: Martin Birgmeier Cc: bug-followup@FreeBSD.org Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load Date: Wed, 31 Jul 2013 15:32:02 -0500 what raidz level are you using: raidz2, raid0, raid1 also what are the pool config (see below, my pool is called z): MOS Configuration: version: 28 name: 'z' state: 0 txg: 214028175 pool_guid: 5334451366918939808 hostid: 4266313884 hostname: 'pisces.biostat.wisc.edu' hole_array[0]: 3 vdev_children: 4 vdev_tree: type: 'root' id: 0 guid: 5334451366918939808 children[0]: type: 'raidz' id: 0 guid: 10192098197416954465 nparity: 2 metaslab_array: 34 metaslab_shift: 37 ashift: 9 asize: 16003083796480 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 5442684633350386182 path: '/dev/da3p1' phys_path: '/dev/da3p1' whole_disk: 1 DTL: 2318341 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 4184769488943604229 path: '/dev/da2p1' phys_path: '/dev/da2p1' whole_disk: 1 DTL: 2318340 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 6673578809124996458 path: '/dev/da1p1' phys_path: '/dev/da1p1' whole_disk: 1 DTL: 2318358 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 14565784372994613264 path: '/dev/da0p1' phys_path: '/dev/da0p1' whole_disk: 1 DTL: 2318357 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 17127372890035360647 path: '/dev/da6p1' phys_path: '/dev/da6p1' whole_disk: 1 DTL: 2318356 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 5667576937780231702 path: '/dev/label/disk05' phys_path: '/dev/label/disk05' whole_disk: 1 DTL: 26389621 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 16545540667878636920 path: '/dev/da5p1' phys_path: '/dev/da5p1' whole_disk: 1 DTL: 2318354 create_txg: 4 children[7]: type: 'disk' id: 7 guid: 16180883375304336701 path: '/dev/da4p1' phys_path: '/dev/da4p1' whole_disk: 1 DTL: 2318353 create_txg: 4 children[1]: type: 'raidz' id: 1 guid: 7000405778472969514 nparity: 2 metaslab_array: 32 metaslab_shift: 37 ashift: 9 asize: 16003083796480 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 11848332905648146086 path: '/dev/da10p1' phys_path: '/dev/da10p1' whole_disk: 1 DTL: 2318352 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 17180048053682905420 path: '/dev/da9p1' phys_path: '/dev/da9p1' whole_disk: 1 DTL: 2318351 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 14648362260613207535 path: '/dev/da8p1' phys_path: '/dev/da8p1' whole_disk: 1 DTL: 2318350 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 2512073543886070959 path: '/dev/da7p1' phys_path: '/dev/da7p1' whole_disk: 1 DTL: 2318349 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 9213113982598250625 path: '/dev/da14p1' phys_path: '/dev/da14p1' whole_disk: 1 DTL: 2318348 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 14965308070554000358 path: '/dev/da13p1' phys_path: '/dev/da13p1' whole_disk: 1 DTL: 2318347 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 8812761598924675500 path: '/dev/da12p1' phys_path: '/dev/da12p1' whole_disk: 1 DTL: 2318346 create_txg: 4 children[7]: type: 'disk' id: 7 guid: 15539718945154864246 path: '/dev/da11p1' phys_path: '/dev/da11p1' whole_disk: 1 DTL: 2318345 create_txg: 4 children[2]: type: 'raidz' id: 2 guid: 12675549095184402399 nparity: 2 metaslab_array: 30 metaslab_shift: 37 ashift: 9 asize: 16003083796480 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 9157198169193782917 path: '/dev/da16p1' phys_path: '/dev/da16p1' whole_disk: 1 DTL: 35652327 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 14494767130322696735 path: '/dev/da17p1' phys_path: '/dev/da17p1' whole_disk: 1 DTL: 2318365 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 17589735305722725561 path: '/dev/da18p1' phys_path: '/dev/da18p1' whole_disk: 1 DTL: 2318364 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 14307956723966398718 path: '/dev/da19p1' phys_path: '/dev/da19p1' whole_disk: 1 DTL: 2318363 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 15965579855708087029 path: '/dev/da20p1' phys_path: '/dev/da20p1' whole_disk: 1 DTL: 2318362 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 760237787742339099 path: '/dev/da21p1' phys_path: '/dev/da21p1' whole_disk: 1 DTL: 2318361 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 2500372020986714069 path: '/dev/da22p1' phys_path: '/dev/da22p1' whole_disk: 1 DTL: 2318360 create_txg: 4 children[7]: type: 'disk' id: 7 guid: 4544065732274116075 path: '/dev/label/disk24' phys_path: '/dev/label/disk24' whole_disk: 1 DTL: 5572568 create_txg: 4 children[3]: type: 'hole' id: 3 guid: 0 metaslab_array: 0 metaslab_shift: 17 ashift: 0 asize: 0 is_log: 0 is_hole: 1 On 07/31/13 14:56, Martin Birgmeier wrote: > No, this is not an LSI HBA, but rather > > atapci0: port > 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f > mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3 > atapci1: at channel -1 on atapci0 > atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata2: on atapci1 > ata3: on atapci1 > ata4: at channel 0 on atapci0 > atapci2: port > 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f > mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0 > atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported > ata5: at channel 0 on atapci2 > ata6: at channel 1 on atapci2 > ata7: at channel 2 on atapci2 > ata8: at channel 3 on atapci2 > ata9: at channel 4 on atapci2 > ata10: at channel 5 on atapci2 > ada0 at ata5 bus 0 scbus3 target 0 lun 0 > ada1 at ata6 bus 0 scbus4 target 0 lun 0 > ada2 at ata7 bus 0 scbus5 target 0 lun 0 > ada3 at ata8 bus 0 scbus6 target 0 lun 0 > ada4 at ata9 bus 0 scbus7 target 0 lun 0 > ada5 at ata10 bus 0 scbus8 target 0 lun 0 > > (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.) > > Regards, > > Martin > From owner-freebsd-fs@FreeBSD.ORG Thu Aug 1 05:30:03 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 10025D1D for ; Thu, 1 Aug 2013 05:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F26D22987 for ; Thu, 1 Aug 2013 05:30:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r715U1JS081411 for ; Thu, 1 Aug 2013 05:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r715U1vD081410; Thu, 1 Aug 2013 05:30:01 GMT (envelope-from gnats) Date: Thu, 1 Aug 2013 05:30:01 GMT Message-Id: <201308010530.r715U1vD081410@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Martin Birgmeier Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Aug 2013 05:30:03 -0000 The following reply was made to PR kern/177536; it has been noted by GNATS. From: Martin Birgmeier To: "Christopher D. Harrison" Cc: bug-followup@FreeBSD.org Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load Date: Thu, 01 Aug 2013 07:20:50 +0200 It is raidz2, using gpt partitions each covering half of six 2 TB disks. The remainder of these disks are mostly unused, with one containing a small UFS root for booting, and the others some small partitions which are used for various VirtualBox instances. Regards, Martin On 07/31/13 22:32, Christopher D. Harrison wrote: > what raidz level are you using: raidz2, raid0, raid1 > also what are the pool config (see below, my pool is called z): > > MOS Configuration: > version: 28 > name: 'z' > state: 0 > txg: 214028175 > pool_guid: 5334451366918939808 > hostid: 4266313884 > hostname: 'pisces.biostat.wisc.edu' > hole_array[0]: 3 > vdev_children: 4 > vdev_tree: > type: 'root' > id: 0 > guid: 5334451366918939808 > children[0]: > type: 'raidz' > id: 0 > guid: 10192098197416954465 > nparity: 2 > metaslab_array: 34 > metaslab_shift: 37 > ashift: 9 > asize: 16003083796480 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 5442684633350386182 > path: '/dev/da3p1' > phys_path: '/dev/da3p1' > whole_disk: 1 > DTL: 2318341 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 4184769488943604229 > path: '/dev/da2p1' > phys_path: '/dev/da2p1' > whole_disk: 1 > DTL: 2318340 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 6673578809124996458 > path: '/dev/da1p1' > phys_path: '/dev/da1p1' > whole_disk: 1 > DTL: 2318358 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 14565784372994613264 > path: '/dev/da0p1' > phys_path: '/dev/da0p1' > whole_disk: 1 > DTL: 2318357 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 17127372890035360647 > path: '/dev/da6p1' > phys_path: '/dev/da6p1' > whole_disk: 1 > DTL: 2318356 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 5667576937780231702 > path: '/dev/label/disk05' > phys_path: '/dev/label/disk05' > whole_disk: 1 > DTL: 26389621 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 16545540667878636920 > path: '/dev/da5p1' > phys_path: '/dev/da5p1' > whole_disk: 1 > DTL: 2318354 > create_txg: 4 > children[7]: > type: 'disk' > id: 7 > guid: 16180883375304336701 > path: '/dev/da4p1' > phys_path: '/dev/da4p1' > whole_disk: 1 > DTL: 2318353 > create_txg: 4 > children[1]: > type: 'raidz' > id: 1 > guid: 7000405778472969514 > nparity: 2 > metaslab_array: 32 > metaslab_shift: 37 > ashift: 9 > asize: 16003083796480 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 11848332905648146086 > path: '/dev/da10p1' > phys_path: '/dev/da10p1' > whole_disk: 1 > DTL: 2318352 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 17180048053682905420 > path: '/dev/da9p1' > phys_path: '/dev/da9p1' > whole_disk: 1 > DTL: 2318351 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 14648362260613207535 > path: '/dev/da8p1' > phys_path: '/dev/da8p1' > whole_disk: 1 > DTL: 2318350 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 2512073543886070959 > path: '/dev/da7p1' > phys_path: '/dev/da7p1' > whole_disk: 1 > DTL: 2318349 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 9213113982598250625 > path: '/dev/da14p1' > phys_path: '/dev/da14p1' > whole_disk: 1 > DTL: 2318348 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 14965308070554000358 > path: '/dev/da13p1' > phys_path: '/dev/da13p1' > whole_disk: 1 > DTL: 2318347 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 8812761598924675500 > path: '/dev/da12p1' > phys_path: '/dev/da12p1' > whole_disk: 1 > DTL: 2318346 > create_txg: 4 > children[7]: > type: 'disk' > id: 7 > guid: 15539718945154864246 > path: '/dev/da11p1' > phys_path: '/dev/da11p1' > whole_disk: 1 > DTL: 2318345 > create_txg: 4 > children[2]: > type: 'raidz' > id: 2 > guid: 12675549095184402399 > nparity: 2 > metaslab_array: 30 > metaslab_shift: 37 > ashift: 9 > asize: 16003083796480 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 9157198169193782917 > path: '/dev/da16p1' > phys_path: '/dev/da16p1' > whole_disk: 1 > DTL: 35652327 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 14494767130322696735 > path: '/dev/da17p1' > phys_path: '/dev/da17p1' > whole_disk: 1 > DTL: 2318365 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 17589735305722725561 > path: '/dev/da18p1' > phys_path: '/dev/da18p1' > whole_disk: 1 > DTL: 2318364 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 14307956723966398718 > path: '/dev/da19p1' > phys_path: '/dev/da19p1' > whole_disk: 1 > DTL: 2318363 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 15965579855708087029 > path: '/dev/da20p1' > phys_path: '/dev/da20p1' > whole_disk: 1 > DTL: 2318362 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 760237787742339099 > path: '/dev/da21p1' > phys_path: '/dev/da21p1' > whole_disk: 1 > DTL: 2318361 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 2500372020986714069 > path: '/dev/da22p1' > phys_path: '/dev/da22p1' > whole_disk: 1 > DTL: 2318360 > create_txg: 4 > children[7]: > type: 'disk' > id: 7 > guid: 4544065732274116075 > path: '/dev/label/disk24' > phys_path: '/dev/label/disk24' > whole_disk: 1 > DTL: 5572568 > create_txg: 4 > children[3]: > type: 'hole' > id: 3 > guid: 0 > metaslab_array: 0 > metaslab_shift: 17 > ashift: 0 > asize: 0 > is_log: 0 > is_hole: 1 > > On 07/31/13 14:56, Martin Birgmeier wrote: >> No, this is not an LSI HBA, but rather >> >> atapci0: port >> 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f >> mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3 >> atapci1: at channel -1 on atapci0 >> atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported >> ata2: on atapci1 >> ata3: on atapci1 >> ata4: at channel 0 on atapci0 >> atapci2: port >> 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f >> mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0 >> atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported >> ata5: at channel 0 on atapci2 >> ata6: at channel 1 on atapci2 >> ata7: at channel 2 on atapci2 >> ata8: at channel 3 on atapci2 >> ata9: at channel 4 on atapci2 >> ata10: at channel 5 on atapci2 >> ada0 at ata5 bus 0 scbus3 target 0 lun 0 >> ada1 at ata6 bus 0 scbus4 target 0 lun 0 >> ada2 at ata7 bus 0 scbus5 target 0 lun 0 >> ada3 at ata8 bus 0 scbus6 target 0 lun 0 >> ada4 at ata9 bus 0 scbus7 target 0 lun 0 >> ada5 at ata10 bus 0 scbus8 target 0 lun 0 >> >> (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.) >> >> Regards, >> >> Martin >> > > > From owner-freebsd-fs@FreeBSD.ORG Thu Aug 1 14:20:02 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EF1229DD for ; Thu, 1 Aug 2013 14:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CBF8525BE for ; Thu, 1 Aug 2013 14:20:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r71EK11E099622 for ; Thu, 1 Aug 2013 14:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r71EK1qv099621; Thu, 1 Aug 2013 14:20:01 GMT (envelope-from gnats) Date: Thu, 1 Aug 2013 14:20:01 GMT Message-Id: <201308011420.r71EK1qv099621@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Christopher Harrison Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Christopher Harrison List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Aug 2013 14:20:02 -0000 The following reply was made to PR kern/177536; it has been noted by GNATS. From: Christopher Harrison To: Martin Birgmeier Cc: bug-followup@FreeBSD.org Subject: Re: kern/177536: [zfs] zfs livelock (deadlock) with high write-to-disk load Date: Thu, 01 Aug 2013 09:11:30 -0500 Just last night I got the system to deadlock again. I grabbed a kernel bt but I realized I do no have full debug enabled. I think our problems are to different ones. As my bt points directly to an LSI adapter or kernel driver issue. -C On 8/1/13 12:20 AM, Martin Birgmeier wrote: > It is raidz2, using gpt partitions each covering half of six 2 TB disks. > The remainder of these disks are mostly unused, with one containing a > small UFS root for booting, and the others some small partitions which > are used for various VirtualBox instances. > > Regards, > > Martin > > On 07/31/13 22:32, Christopher D. Harrison wrote: >> what raidz level are you using: raidz2, raid0, raid1 >> also what are the pool config (see below, my pool is called z): >> >> MOS Configuration: >> version: 28 >> name: 'z' >> state: 0 >> txg: 214028175 >> pool_guid: 5334451366918939808 >> hostid: 4266313884 >> hostname: 'pisces.biostat.wisc.edu' >> hole_array[0]: 3 >> vdev_children: 4 >> vdev_tree: >> type: 'root' >> id: 0 >> guid: 5334451366918939808 >> children[0]: >> type: 'raidz' >> id: 0 >> guid: 10192098197416954465 >> nparity: 2 >> metaslab_array: 34 >> metaslab_shift: 37 >> ashift: 9 >> asize: 16003083796480 >> is_log: 0 >> create_txg: 4 >> children[0]: >> type: 'disk' >> id: 0 >> guid: 5442684633350386182 >> path: '/dev/da3p1' >> phys_path: '/dev/da3p1' >> whole_disk: 1 >> DTL: 2318341 >> create_txg: 4 >> children[1]: >> type: 'disk' >> id: 1 >> guid: 4184769488943604229 >> path: '/dev/da2p1' >> phys_path: '/dev/da2p1' >> whole_disk: 1 >> DTL: 2318340 >> create_txg: 4 >> children[2]: >> type: 'disk' >> id: 2 >> guid: 6673578809124996458 >> path: '/dev/da1p1' >> phys_path: '/dev/da1p1' >> whole_disk: 1 >> DTL: 2318358 >> create_txg: 4 >> children[3]: >> type: 'disk' >> id: 3 >> guid: 14565784372994613264 >> path: '/dev/da0p1' >> phys_path: '/dev/da0p1' >> whole_disk: 1 >> DTL: 2318357 >> create_txg: 4 >> children[4]: >> type: 'disk' >> id: 4 >> guid: 17127372890035360647 >> path: '/dev/da6p1' >> phys_path: '/dev/da6p1' >> whole_disk: 1 >> DTL: 2318356 >> create_txg: 4 >> children[5]: >> type: 'disk' >> id: 5 >> guid: 5667576937780231702 >> path: '/dev/label/disk05' >> phys_path: '/dev/label/disk05' >> whole_disk: 1 >> DTL: 26389621 >> create_txg: 4 >> children[6]: >> type: 'disk' >> id: 6 >> guid: 16545540667878636920 >> path: '/dev/da5p1' >> phys_path: '/dev/da5p1' >> whole_disk: 1 >> DTL: 2318354 >> create_txg: 4 >> children[7]: >> type: 'disk' >> id: 7 >> guid: 16180883375304336701 >> path: '/dev/da4p1' >> phys_path: '/dev/da4p1' >> whole_disk: 1 >> DTL: 2318353 >> create_txg: 4 >> children[1]: >> type: 'raidz' >> id: 1 >> guid: 7000405778472969514 >> nparity: 2 >> metaslab_array: 32 >> metaslab_shift: 37 >> ashift: 9 >> asize: 16003083796480 >> is_log: 0 >> create_txg: 4 >> children[0]: >> type: 'disk' >> id: 0 >> guid: 11848332905648146086 >> path: '/dev/da10p1' >> phys_path: '/dev/da10p1' >> whole_disk: 1 >> DTL: 2318352 >> create_txg: 4 >> children[1]: >> type: 'disk' >> id: 1 >> guid: 17180048053682905420 >> path: '/dev/da9p1' >> phys_path: '/dev/da9p1' >> whole_disk: 1 >> DTL: 2318351 >> create_txg: 4 >> children[2]: >> type: 'disk' >> id: 2 >> guid: 14648362260613207535 >> path: '/dev/da8p1' >> phys_path: '/dev/da8p1' >> whole_disk: 1 >> DTL: 2318350 >> create_txg: 4 >> children[3]: >> type: 'disk' >> id: 3 >> guid: 2512073543886070959 >> path: '/dev/da7p1' >> phys_path: '/dev/da7p1' >> whole_disk: 1 >> DTL: 2318349 >> create_txg: 4 >> children[4]: >> type: 'disk' >> id: 4 >> guid: 9213113982598250625 >> path: '/dev/da14p1' >> phys_path: '/dev/da14p1' >> whole_disk: 1 >> DTL: 2318348 >> create_txg: 4 >> children[5]: >> type: 'disk' >> id: 5 >> guid: 14965308070554000358 >> path: '/dev/da13p1' >> phys_path: '/dev/da13p1' >> whole_disk: 1 >> DTL: 2318347 >> create_txg: 4 >> children[6]: >> type: 'disk' >> id: 6 >> guid: 8812761598924675500 >> path: '/dev/da12p1' >> phys_path: '/dev/da12p1' >> whole_disk: 1 >> DTL: 2318346 >> create_txg: 4 >> children[7]: >> type: 'disk' >> id: 7 >> guid: 15539718945154864246 >> path: '/dev/da11p1' >> phys_path: '/dev/da11p1' >> whole_disk: 1 >> DTL: 2318345 >> create_txg: 4 >> children[2]: >> type: 'raidz' >> id: 2 >> guid: 12675549095184402399 >> nparity: 2 >> metaslab_array: 30 >> metaslab_shift: 37 >> ashift: 9 >> asize: 16003083796480 >> is_log: 0 >> create_txg: 4 >> children[0]: >> type: 'disk' >> id: 0 >> guid: 9157198169193782917 >> path: '/dev/da16p1' >> phys_path: '/dev/da16p1' >> whole_disk: 1 >> DTL: 35652327 >> create_txg: 4 >> children[1]: >> type: 'disk' >> id: 1 >> guid: 14494767130322696735 >> path: '/dev/da17p1' >> phys_path: '/dev/da17p1' >> whole_disk: 1 >> DTL: 2318365 >> create_txg: 4 >> children[2]: >> type: 'disk' >> id: 2 >> guid: 17589735305722725561 >> path: '/dev/da18p1' >> phys_path: '/dev/da18p1' >> whole_disk: 1 >> DTL: 2318364 >> create_txg: 4 >> children[3]: >> type: 'disk' >> id: 3 >> guid: 14307956723966398718 >> path: '/dev/da19p1' >> phys_path: '/dev/da19p1' >> whole_disk: 1 >> DTL: 2318363 >> create_txg: 4 >> children[4]: >> type: 'disk' >> id: 4 >> guid: 15965579855708087029 >> path: '/dev/da20p1' >> phys_path: '/dev/da20p1' >> whole_disk: 1 >> DTL: 2318362 >> create_txg: 4 >> children[5]: >> type: 'disk' >> id: 5 >> guid: 760237787742339099 >> path: '/dev/da21p1' >> phys_path: '/dev/da21p1' >> whole_disk: 1 >> DTL: 2318361 >> create_txg: 4 >> children[6]: >> type: 'disk' >> id: 6 >> guid: 2500372020986714069 >> path: '/dev/da22p1' >> phys_path: '/dev/da22p1' >> whole_disk: 1 >> DTL: 2318360 >> create_txg: 4 >> children[7]: >> type: 'disk' >> id: 7 >> guid: 4544065732274116075 >> path: '/dev/label/disk24' >> phys_path: '/dev/label/disk24' >> whole_disk: 1 >> DTL: 5572568 >> create_txg: 4 >> children[3]: >> type: 'hole' >> id: 3 >> guid: 0 >> metaslab_array: 0 >> metaslab_shift: 17 >> ashift: 0 >> asize: 0 >> is_log: 0 >> is_hole: 1 >> >> On 07/31/13 14:56, Martin Birgmeier wrote: >>> No, this is not an LSI HBA, but rather >>> >>> atapci0: port >>> 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f >>> mem 0xfe8fe000-0xfe8fffff irq 18 at device 0.0 on pci3 >>> atapci1: at channel -1 on atapci0 >>> atapci1: AHCI v1.00 controller with 2 3Gbps ports, PM supported >>> ata2: on atapci1 >>> ata3: on atapci1 >>> ata4: at channel 0 on atapci0 >>> atapci2: port >>> 0xa000-0xa007,0x9000-0x9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f >>> mem 0xfe4ffc00-0xfe4fffff irq 19 at device 17.0 on pci0 >>> atapci2: AHCI v1.20 controller with 6 3Gbps ports, PM supported >>> ata5: at channel 0 on atapci2 >>> ata6: at channel 1 on atapci2 >>> ata7: at channel 2 on atapci2 >>> ata8: at channel 3 on atapci2 >>> ata9: at channel 4 on atapci2 >>> ata10: at channel 5 on atapci2 >>> ada0 at ata5 bus 0 scbus3 target 0 lun 0 >>> ada1 at ata6 bus 0 scbus4 target 0 lun 0 >>> ada2 at ata7 bus 0 scbus5 target 0 lun 0 >>> ada3 at ata8 bus 0 scbus6 target 0 lun 0 >>> ada4 at ata9 bus 0 scbus7 target 0 lun 0 >>> ada5 at ata10 bus 0 scbus8 target 0 lun 0 >>> >>> (This is just a simple PC motherboard - ASUS M4A89GTD PRO USB3.) >>> >>> Regards, >>> >>> Martin >>> >> >> From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 06:21:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5B8A8881 for ; Fri, 2 Aug 2013 06:21:15 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-oa0-x22f.google.com (mail-oa0-x22f.google.com [IPv6:2607:f8b0:4003:c02::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 26954264B for ; Fri, 2 Aug 2013 06:21:15 +0000 (UTC) Received: by mail-oa0-f47.google.com with SMTP id g12so576070oah.34 for ; Thu, 01 Aug 2013 23:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:date:message-id:subject:from:to:content-type; bh=SenOEBeLXAHz7ccV5/r2fx8X3Ogw8mVDjwVWJzkovmY=; b=D+iWyWUuAsu7mhF5I6DscjNuPqwsmjoSHtOmN7Sss7ILrk0eGXuz0SEM+DId4TpLSY MYdmA/BXYflFPhMkJVDyuoYUDdWhAlfACTdjfxcJg63Ej+hf042IljADv2ONwgkAqe4U mvSLDl3JXUZbvq20DzGaRawjQVXEnwkkBNI2E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :x-gm-message-state; bh=SenOEBeLXAHz7ccV5/r2fx8X3Ogw8mVDjwVWJzkovmY=; b=PHWC7kdaBDoThOEy0cATNXGsMFfuppa7uJh7WtthVx/aXZoIsYUzp2CRFaRNKGZCpe PYaUkxLz0d+ZQZwU/yGlSWLzeh7W5ykz8ipqXO0wh9OOx3z8gNkR3U/hUpD1kiLU7kZ+ F9IPwJ3vUL+twINgxLfsG6+2ijNPLdHRCs9FYeOxFOkjwrsCsJeFi8INbzyZUvg7noBb vEL4C4BD05gOK3QiIwYFim7my2RbBm41ALbg/ZrU16klNdMWNEzqKAB2Fdx8+C/IXZ14 S7Hqn8JBXCdee+/J4tlMqNWtcP04r/OJhBfzqbtljoKFEwuaWzBG5IwrIfxp9AAPYoP2 OPjg== MIME-Version: 1.0 X-Received: by 10.42.250.148 with SMTP id mo20mr488961icb.34.1375424473612; Thu, 01 Aug 2013 23:21:13 -0700 (PDT) Received: by 10.42.233.143 with HTTP; Thu, 1 Aug 2013 23:21:13 -0700 (PDT) Date: Thu, 1 Aug 2013 23:21:13 -0700 Message-ID: Subject: ZFS: unsupported ZFS version 5000 (should be 28) From: Tim Gustafson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQl0GfC7kiaSaia8EltkVr+61JqqVpv1L3i+K2BZAFFsKoAwzKANa7fyUgkZyDJFYADNvYO3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 06:21:15 -0000 I recently rebuilt my FreeBSD 8.4 OS and rebooted, and when the system came back up, it refused to mount the zpool because: ZFS: unsupported ZFS version 5000 (should be 28) I didn't upgrade my zpool as far as I know. I tried booting off the latest 9.1 snapshot, but it also says that the zpool has a too-new version number and won't import it. I Google around and found a post about using gpart to update the boot code, which I tried with both the newest 8.4 release and also the newest 9.1 snapshot, just to make sure something wonky didn't happen with the disks. The good news is that I have relatively recent backups (yesterday evening) for this system, but I'm still totally confused as to what the heck happened. Is there some magic that I can type into a live boot CD to fix this problem without rebuilding the whole machine? -- Tim Gustafson tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 06:47:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A4F67EBD for ; Fri, 2 Aug 2013 06:47:06 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-we0-x234.google.com (mail-we0-x234.google.com [IPv6:2a00:1450:400c:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 384602713 for ; Fri, 2 Aug 2013 06:47:06 +0000 (UTC) Received: by mail-we0-f180.google.com with SMTP id p61so193714wes.25 for ; Thu, 01 Aug 2013 23:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P1tHNZZ4L1gfhvQVg9Rr3HGl2adtg9Zu5nLeUd9bsGc=; b=tmBcRK/XBi28EToJD7Y1KBDEJU6XyD74PmibJHIyuZm+KXAKj4T5P7VhT74XGeonOS rrHHzXZTYBCalGEaRAgISHEUhj5tqrjMdKwtVDRvMlZQyHvKSn6SpUPfu3aSTQ1tw1/J FXjqeCiOfYCs9+uDlqMXapfSriI9TyQcdlG1Yk4B/+NpSJR3K4zCBzk7VMeGJ4byLvro ke6mKvXJzXJY7pYsGLlb194fQ/BIkuL3Zngobw17MM6Sqj54iMQpumbsw0LZGqfWA2Ev q849FvHN4a4IIvQR+EB67XTnPDCMZRdF0HEDe/rbIZsDX3/XmZsmdI4DZTIbqixHdjNb F8mg== MIME-Version: 1.0 X-Received: by 10.180.11.146 with SMTP id q18mr753294wib.50.1375426024315; Thu, 01 Aug 2013 23:47:04 -0700 (PDT) Received: by 10.216.67.193 with HTTP; Thu, 1 Aug 2013 23:47:04 -0700 (PDT) In-Reply-To: References: Date: Fri, 2 Aug 2013 07:47:04 +0100 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: krad To: Tim Gustafson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 06:47:06 -0000 5000 sounds like a linuxzfs pool number to me, could that be a possibility? On 2 August 2013 07:21, Tim Gustafson wrote: > I recently rebuilt my FreeBSD 8.4 OS and rebooted, and when the system > came back up, it refused to mount the zpool because: > > ZFS: unsupported ZFS version 5000 (should be 28) > > I didn't upgrade my zpool as far as I know. I tried booting off the > latest 9.1 snapshot, but it also says that the zpool has a too-new > version number and won't import it. > > I Google around and found a post about using gpart to update the boot > code, which I tried with both the newest 8.4 release and also the > newest 9.1 snapshot, just to make sure something wonky didn't happen > with the disks. > > The good news is that I have relatively recent backups (yesterday > evening) for this system, but I'm still totally confused as to what > the heck happened. > > Is there some magic that I can type into a live boot CD to fix this > problem without rebuilding the whole machine? > > -- > > Tim Gustafson > tjg@ucsc.edu > 831-459-5354 > Baskin Engineering, Room 313A > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 06:51:00 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2A8A5F4B for ; Fri, 2 Aug 2013 06:51:00 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-vb0-x22b.google.com (mail-vb0-x22b.google.com [IPv6:2607:f8b0:400c:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C7E642729 for ; Fri, 2 Aug 2013 06:50:59 +0000 (UTC) Received: by mail-vb0-f43.google.com with SMTP id h11so255814vbh.16 for ; Thu, 01 Aug 2013 23:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=NfS1NOZFPp3mGwDUsnc9XtftU4ohcsaTZ+U35HzCuIM=; b=M0wecewAFl3whJUYCsnEoe/iyejY4PIB37qxn5zst7ZJF+Q2FfgJqoigrzKrglVLd4 cCKxVXBGV+HCNrT66vSRGd+ECzehzpvx6Iwz7LuwLtgNYGavyeDTAZa7XX2EYDFW1LyW CIU3wxb4amq9f2Xx16EqqoeHa2LfCZmEAXTjfQXCQZyCBIc+KYRHAZ3cjTfRNx3BA0yy d9CR9anUZoiWfrduUeFX98WokwYA4I5peSfW5B3SalXQyTt92kxiXj+3rFt6xToAUTqN onhTyonAWe0zkGkJlB6Beg48i+eHfvaQV25RYjMo9JdejPFijF7ZBU0uu1LQaAr+uZSR vcXw== MIME-Version: 1.0 X-Received: by 10.220.101.81 with SMTP id b17mr1558265vco.79.1375426258802; Thu, 01 Aug 2013 23:50:58 -0700 (PDT) Received: by 10.220.96.78 with HTTP; Thu, 1 Aug 2013 23:50:58 -0700 (PDT) In-Reply-To: References: Date: Fri, 2 Aug 2013 02:50:58 -0400 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: "Sam Fourman Jr." To: krad Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 06:51:00 -0000 On Fri, Aug 2, 2013 at 2:47 AM, krad wrote: > 5000 sounds like a linuxzfs pool number to me, could that be a possibility? > > No this has NOTHING at all to do with Linux, 5000 means it supports Feature flags -HEAD has had support for months, I believe 9.x has support as well or it should... there was a CFT last year http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015290.html -- Sam Fourman Jr. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 06:55:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6594F90; Fri, 2 Aug 2013 06:55:06 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from mail0.glenbarber.us (mail0.glenbarber.us [IPv6:2607:fc50:1:2300:1001:1001:1001:face]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3B8442745; Fri, 2 Aug 2013 06:55:06 +0000 (UTC) Received: from glenbarber.us (70.15.88.86.res-cmts.sewb.ptd.net [70.15.88.86]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: gjb) by mail0.glenbarber.us (Postfix) with ESMTPSA id 438217545; Fri, 2 Aug 2013 06:55:04 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.8.3 mail0.glenbarber.us 438217545 Authentication-Results: mail0.glenbarber.us; dkim=none reason="no signature"; dkim-adsp=none Date: Fri, 2 Aug 2013 02:55:02 -0400 From: Glen Barber To: krad Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) Message-ID: <20130802065502.GD2132@glenbarber.us> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="9UV9rz0O2dU/yYYn" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 10.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 06:55:06 -0000 --9UV9rz0O2dU/yYYn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 02, 2013 at 07:47:04AM +0100, krad wrote: > 5000 sounds like a linuxzfs pool number to me, could that be a possibilit= y? >=20 No. >=20 > On 2 August 2013 07:21, Tim Gustafson wrote: >=20 > > I recently rebuilt my FreeBSD 8.4 OS and rebooted, and when the system > > came back up, it refused to mount the zpool because: > > > > ZFS: unsupported ZFS version 5000 (should be 28) > > > > I didn't upgrade my zpool as far as I know. I tried booting off the > > latest 9.1 snapshot, but it also says that the zpool has a too-new > > version number and won't import it. > > Try the 9.2-BETA2 snapshot. The 9.1-RELEASE (you did not provide any clear indication *which* snapshot you tried) does not support zpool version >28. 9.2-BETA1 and 9.2-BETA2 snapshots do. If you did update your zpool but not the zfs boot loader, you can fix the loader by following the instructions in /usr/src/UPDATING, specifically the "ZFS notes" section. Glen --9UV9rz0O2dU/yYYn Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQEcBAEBCAAGBQJR+1fGAAoJEFJPDDeguUajxkIH/ib/Z9CfkzwxvgTgKNQ0R9c9 v2qof3Ml3293C98PpcDUii40Fp+zNR8ZV5HzqoAFLTF4DFrzz2IoWUqV1+wfuPJV 9Vp3WbWy6et02aPlfdqT+H4awwSWxcjLXYtkNIxhbncZizft3lQf36Y0716Ah6x3 E0DRVSGrE96dxU3qzPhYTg423YD9qFilYQ8va11AQxp6eWm6JvPWKRaJmH+7hRz5 zzCafTutS4QRw5ghIEV4I+/FW32FQNGNUZW/U3p9vq/JFitJjoR8ztZG2mSYVw8a yauR/1ZvmwvNhEaCrQL59ZZs2DhaZZ1C1aQMMW8RAcNRT+FB5OBPRs7PbCAMVZk= =bUdG -----END PGP SIGNATURE----- --9UV9rz0O2dU/yYYn-- From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 06:55:29 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2721D117 for ; Fri, 2 Aug 2013 06:55:29 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp1.bway.net (smtp1.bway.net [216.220.96.27]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 015AE274D for ; Fri, 2 Aug 2013 06:55:28 +0000 (UTC) Received: from toasty.sporklab.com (foon.sporktines.com [96.57.144.66]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp1.bway.net (Postfix) with ESMTPSA id D2B9895855; Fri, 2 Aug 2013 02:55:21 -0400 (EDT) References: In-Reply-To: Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=windows-1252 Message-Id: <1C40D1C7-245E-4B2E-A7B4-0590BBEA34EA@bway.net> Content-Transfer-Encoding: quoted-printable From: Charles Sprickman Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) Date: Fri, 2 Aug 2013 02:55:21 -0400 To: "Sam Fourman Jr." X-Mailer: Apple Mail (2.1085) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 06:55:29 -0000 On Aug 2, 2013, at 2:50 AM, Sam Fourman Jr. wrote: > On Fri, Aug 2, 2013 at 2:47 AM, krad wrote: >=20 >> 5000 sounds like a linuxzfs pool number to me, could that be a = possibility? >>=20 >>=20 > No this has NOTHING at all to do with Linux, 5000 means it supports = Feature > flags > -HEAD has had support for months, I believe 9.x has support as well = or it > should... there was a CFT last year > http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015290.html 8.4 supports feature flags, I would assume the loader does as well. OP should probably share the commands used to put new loader code on the = drives=85 Charles >=20 >=20 > --=20 >=20 > Sam Fourman Jr. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 07:04:24 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E2C0A38B; Fri, 2 Aug 2013 07:04:24 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B7A26278B; Fri, 2 Aug 2013 07:04:24 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r7274OZY001684; Fri, 2 Aug 2013 07:04:24 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r7274OjC001683; Fri, 2 Aug 2013 07:04:24 GMT (envelope-from remko) Date: Fri, 2 Aug 2013 07:04:24 GMT Message-Id: <201308020704.r7274OjC001683@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: remko@FreeBSD.org Subject: Re: kern/180979: [netsmb][patch]: Fix large files handling X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 07:04:25 -0000 Synopsis: [netsmb][patch]: Fix large files handling Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: remko Responsible-Changed-When: Fri Aug 2 07:04:08 UTC 2013 Responsible-Changed-Why: Filesystem related -> -fs http://www.freebsd.org/cgi/query-pr.cgi?pr=180979 From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 07:20:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E9CF58DB for ; Fri, 2 Aug 2013 07:20:08 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-oa0-x22e.google.com (mail-oa0-x22e.google.com [IPv6:2607:f8b0:4003:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 985D32886 for ; Fri, 2 Aug 2013 07:20:08 +0000 (UTC) Received: by mail-oa0-f46.google.com with SMTP id l10so664080oag.33 for ; Fri, 02 Aug 2013 00:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BojTg52/JEZpM6/RDB4GuNDcx1uxpTpwp5ZddvCHBCg=; b=efbD6YwvhqKd++TAU41D/lQfWY4olYxaOS4drfrFiVaK9VT/PwIB62hN/OQC4xbshj d5ZHVmmmn/Yk8Sp0GPXzqQPm16G9GTVtIOn3LCfqVdPaIruPU5TCRaMtMpHGDw62lSig OgJhEuDiNjBtVajAOjpkhYFTBvcfb11Fy66Q4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=BojTg52/JEZpM6/RDB4GuNDcx1uxpTpwp5ZddvCHBCg=; b=Vz+yecGL80apXWsDJt05KOlZ1k1R5PyuI3AFPOxgZyaihDmTOsIdbeSbib5YemBHMl 63oE/XDVg7TMnDTw0727DcKuxvuhopde7oHaBsc8dghMoxII+PEDAi1HJq0kmNaelKHR rY5KXeVIZUIDI+a5sPbL4RhsC9xnI3XjDg3VHBd816qPRDrz9Jvr+3ixI9Cql/bvldo4 d/fyxv7AsV7kS5eOik3Tv7T1wuE4257aACztknCKO1jUzI6WXJDw05lTLIOMUOPQnNGj VBZim00dZO2gACZQQXKyUvKwoO75XdU/YetzLKiMxhhnZ7OoUO07dWIdsloTgtiOkZJl Qvjw== MIME-Version: 1.0 X-Received: by 10.42.63.207 with SMTP id d15mr469947ici.21.1375428007703; Fri, 02 Aug 2013 00:20:07 -0700 (PDT) Received: by 10.42.233.143 with HTTP; Fri, 2 Aug 2013 00:20:07 -0700 (PDT) In-Reply-To: References: Date: Fri, 2 Aug 2013 00:20:07 -0700 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Tim Gustafson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQleZiT+rfAal7iuv66RbFkqNiKkyVO1ptJIZ0p1KWgnCRhcQZwoWxlG24R4XRCKSWpovpAX X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 07:20:09 -0000 > the other option you have is 9.2 BETA, I think that has feature flags > support I downloaded a 9.2-BETA ISO and booted the Live CD. That CD recognized the zpool and I was able to access the data. Then I did a "gpart bootcode" using the pmbr and gptzfsboot from the 9.2-BETA disk and rebooted, but it still wouldn't boot - same error about incorrect zpool version. I rebooted using the Live CD again into single user mode, and then imported the zpool int and mounted the root partition, and then did a "gpart bootcode" using pmbr and gptzfsroot from the OS that exists on the zpool and rebooted again, and that didn't work - same error message again. I'm starting to think that my "gpart bootcode" command is incorrect. I've been using: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 And then re-run that for each of da1 through da5, to install the boot code on each. (I don't know if that helps or not, but it's a habit of mine, in case a drive dies in the future). Is that the right command to run for this? -- Tim Gustafson tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 07:48:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 567664DE for ; Fri, 2 Aug 2013 07:48:05 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-ob0-x236.google.com (mail-ob0-x236.google.com [IPv6:2607:f8b0:4003:c01::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1C9212961 for ; Fri, 2 Aug 2013 07:48:04 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id wo10so614983obc.13 for ; Fri, 02 Aug 2013 00:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=GMXTCXY/gq4nsp8ZS7e+KGwnG0EAWAaNNehDdyKCoVE=; b=HLvll7Eu9UNhhJm29dQzF8lb+iTneWgGkwhBxcRw3ePWJ8eTw2/iRak2EQplZHJn7c AAr/Qmmc4psjhE5wxj442B0bqWczEkSQ7f81/JrmrC7q3euhAidFh68kwy825m9OjRO3 vgvQ0h3rQGyTjc6nGECHXZpPC7ADnOZo5QCyI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=GMXTCXY/gq4nsp8ZS7e+KGwnG0EAWAaNNehDdyKCoVE=; b=mu0FLPnsAydJSQd+4UBz9SDR9gsQzNslOoQ7wLwwqiwJe/vOM+nUJQGab1L/JzTK4E 7v7eFyfwrF5BCutpE81hl7Eehp9AeXq1OWLB1BmBSsAbIosm/vhUJ8AUybDLwNyauwOn jyutzf5dJ5CkHsnfPohDyVU8prgFZ8gB3oBIxIMuZIiw0NDzzD/AgG4aNmsEXULanx2u jaV1+7T0zvYXwOsT+4tcnSkwZaxEGqWrqbVOt1n6DfJGD3V6ei8ACo5CFx8iwugdDIYj ie+IflHL1HOjVH05+B/RHEicpVt7W6OZ0lbMGM/UgYDc5SgdJuFdDcKA6p0ZplWIxEvq SkLg== MIME-Version: 1.0 X-Received: by 10.50.20.195 with SMTP id p3mr148759ige.26.1375429683357; Fri, 02 Aug 2013 00:48:03 -0700 (PDT) Received: by 10.42.233.143 with HTTP; Fri, 2 Aug 2013 00:48:03 -0700 (PDT) In-Reply-To: References: Date: Fri, 2 Aug 2013 00:48:03 -0700 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Tim Gustafson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQnoLSXS46dUnvdqdYoPdu7xOEZouUXMqRU/0sVr15WBayydibRm9SXeA4ChiyEJBqr5a7mN X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 07:48:05 -0000 Ok, so I think I have a little more of an idea as to what happened. Apparently, when I did an "svn checkout" this evening, I grabbed releng/9.1 rather than releng/8.4. Apparently, that make buildworld/buildkernel/installkernel/installworld also upgraded the zpool for me, or something...I'm not sure what happened, but I *am* sure I didn't type "zpool upgrade" at any point. Is an automatic zpool upgrade included in installkernel now? So now I've got base/releng/9.1 revision number 253878 installed on that machine, which apparently does support the 5000 zpool version, but seems to somehow not have a compatible boot loader. I'm feeling like "gpart bootcode" ought to fix this problem for me, but I've tried several iterations of that without success. For now, I'm booted off the 9.2-BETA CD in single user mode, and then mounted my zpool and zfs file systems, and then ran "sh /etc/rc" by hand, which started (almost) everything up for me. My jails are running, so my services are up, so I'm going to sleep on it. I don't have some commands like "ps" and "top" right now in the root OS (/dev seems to be wedged), but at least I can poke around and look at some files and see what happened, and then maybe fix it. -- Tim Gustafson tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 07:58:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E989988A for ; Fri, 2 Aug 2013 07:58:12 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-vb0-x22a.google.com (mail-vb0-x22a.google.com [IPv6:2607:f8b0:400c:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A53C029BC for ; Fri, 2 Aug 2013 07:58:12 +0000 (UTC) Received: by mail-vb0-f42.google.com with SMTP id e12so300311vbg.15 for ; Fri, 02 Aug 2013 00:58:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2YwklSQMxD0ZYxBhqAI/CPV2inN+zxhGKzcSerVEFZY=; b=GiED9qkAM+soxVz+JjifoLKOXwwixnoLGPFuM+qxpuggOHO2lrh9cECFGdMHmE2W/o EHMn1YjnNEL+6bPQjtAXshyB5FDHCAVCYGTZYvs8m5IQPUm1Tq/+zvNxXgbQOWSR/gLW OPc4kgBaSwmdYsL84oYnl2fjiINRBTdoGVUB7FufSy2C6TkwThDrbU0Y7g9TwWPesK1n e8Ilccs38gYl0bf/aLquAOTrXryGGl9sWJ3HejIeTSXXUm1SmTnVCJWBffMencTeFLvq PEgVALNTzk7SKLBiLzPvJEzKcJoeMBtjS8QTMjN0sAVp7xhoZ0sINSzRvPoj2SZ93Jhf 91IQ== MIME-Version: 1.0 X-Received: by 10.52.176.106 with SMTP id ch10mr260689vdc.41.1375430291646; Fri, 02 Aug 2013 00:58:11 -0700 (PDT) Received: by 10.220.96.78 with HTTP; Fri, 2 Aug 2013 00:58:11 -0700 (PDT) In-Reply-To: References: Date: Fri, 2 Aug 2013 03:58:11 -0400 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: "Sam Fourman Jr." To: Tim Gustafson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 07:58:13 -0000 > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 > > And then re-run that for each of da1 through da5, to install the boot > code on each. (I don't know if that helps or not, but it's a habit of > mine, in case a drive dies in the future). > > Is that the right command to run for this? > Yes, that SHOULD have worked, I just re looked at my script I used to setup my zpool for reference here is what I did from a -HEAD snapshot image gpart create -s gpt ada0 > /dev/null 2>&1 gpart add -b 34 -s 64k -t freebsd-boot ada0 > /dev/null 2>&1 gpart add -t freebsd-zfs -a 4k -l tank-disk0 ada0 > /dev/null 2>&1 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 > /dev/null 2>&1 gnop create -S 4096 /dev/gpt/tank-disk0 > /dev/null 2>&1 Sam Fourman Jr. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 07:58:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 70469901 for ; Fri, 2 Aug 2013 07:58:43 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 307E829C4 for ; Fri, 2 Aug 2013 07:58:42 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1V5AFs-0007EC-O8 for freebsd-fs@freebsd.org; Fri, 02 Aug 2013 09:58:33 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1V5AFt-0005uV-5Q for freebsd-fs@freebsd.org; Fri, 02 Aug 2013 09:58:33 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) References: Date: Fri, 02 Aug 2013 09:58:32 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.16 (Win32) X-Authenticated-As-Hash: 5a5bc696c05b24d66fef48d694aeed0652e57d03 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: - X-Spam-Score: -1.9 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.1 X-Scan-Signature: 2ecd0b53b7de9511489f92806276a3d7 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 07:58:43 -0000 On Fri, 02 Aug 2013 09:48:03 +0200, Tim Gustafson wrote: > Ok, so I think I have a little more of an idea as to what happened. > > Apparently, when I did an "svn checkout" this evening, I grabbed > releng/9.1 rather than releng/8.4. Apparently, that make > buildworld/buildkernel/installkernel/installworld also upgraded the > zpool for me, or something...I'm not sure what happened, but I *am* > sure I didn't type "zpool upgrade" at any point. Is an automatic > zpool upgrade included in installkernel now? No, it isn't. > So now I've got base/releng/9.1 revision number 253878 installed on > that machine, which apparently does support the 5000 zpool version, > but seems to somehow not have a compatible boot loader. > > I'm feeling like "gpart bootcode" ought to fix this problem for me, > but I've tried several iterations of that without success. > > For now, I'm booted off the 9.2-BETA CD in single user mode, and then > mounted my zpool and zfs file systems, and then ran "sh /etc/rc" by > hand, which started (almost) everything up for me. My jails are > running, so my services are up, so I'm going to sleep on it. I don't > have some commands like "ps" and "top" right now in the root OS (/dev > seems to be wedged), but at least I can poke around and look at some > files and see what happened, and then maybe fix it. Maybe post the setup of your disks. So people can take a look if your commands are right or wrong. The output of 'gpart show' and 'gpart list' are a good starting point. Ronald. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 08:38:04 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0F465636 for ; Fri, 2 Aug 2013 08:38:04 +0000 (UTC) (envelope-from leaf-sapporo@www2865.sakura.ne.jp) Received: from www2865.sakura.ne.jp (www2865.sakura.ne.jp [IPv6:2403:3a00:201:1c:49:212:198:75]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B51862B4D for ; Fri, 2 Aug 2013 08:38:03 +0000 (UTC) Received: from www2865.sakura.ne.jp (localhost [127.0.0.1]) by www2865.sakura.ne.jp (8.14.4/8.14.4) with ESMTP id r728c2g0034086 for ; Fri, 2 Aug 2013 17:38:02 +0900 (JST) (envelope-from leaf-sapporo@www2865.sakura.ne.jp) Received: (from leaf-sapporo@localhost) by www2865.sakura.ne.jp (8.14.4/8.14.4/Submit) id r728c2rK034085; Fri, 2 Aug 2013 17:38:02 +0900 (JST) (envelope-from leaf-sapporo) Date: Fri, 2 Aug 2013 17:38:02 +0900 (JST) To: freebsd-fs@freebsd.org Subject: =?utf-8?B?0J/QviDQv9C+0LLQvtC00YMg0JLQsNGI0LXQs9C+INGB0LDQudGC0LA=?= From: =?utf-8?B?0JDQu9C10LrRgdC10Lk=?= Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 08:38:04 -0000 Здравствуйте! Предлагаю для Вашего сайта целевые переходы. Источником являются почтовые рассылки. Наши преимущества: - Таргетинг по любому городу РФ; - Статистика в личном кабинете; - Цена клика во много раз меньше "Яндекс.Директа"; - Объем рекламного объявления до 200 символов без темы и ссылки. Обращайтесь по любым вопросам по телефону: ( 4 9 5 ) 5 02 -61 -8 5 From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 10:02:32 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7A14876A for ; Fri, 2 Aug 2013 10:02:32 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 005AA2EBC for ; Fri, 2 Aug 2013 10:02:31 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [193.68.6.1]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r72A2M3R057777 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Fri, 2 Aug 2013 13:02:22 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <51FB83AE.4020202@digsys.bg> Date: Fri, 02 Aug 2013 13:02:22 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130731 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 10:02:32 -0000 On 02.08.13 10:48, Tim Gustafson wrote: > So now I've got base/releng/9.1 revision number 253878 installed on > that machine, which apparently does support the 5000 zpool version, > but seems to somehow not have a compatible boot loader. As it was already indicated, there is no support for zpool version 5000 (feature flags) in base/releng/9.1. There is however in base/stable/9 and in base/stable/8 what you obviously has before. I believe here the unfortunate naming of 9-stable as RELENG_9 in CVS has confused you. SVN layout is different now, more logical. In general, it is good idea if you migrate from 8-stable, to migrate to 9-stable etc, because chances are 8-stable contains "newer" code than 9-release -- except if the 9-release just happened. Apparently, at some point you, or someone who has root on that server did zpool upgrade. ZFS does not auto upgrade. Daniel From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 14:12:45 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 240023B8 for ; Fri, 2 Aug 2013 14:12:45 +0000 (UTC) (envelope-from wblock@wonkity.com) Received: from wonkity.com (wonkity.com [67.158.26.137]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 55CCB2887 for ; Fri, 2 Aug 2013 14:12:44 +0000 (UTC) Received: from wonkity.com (localhost [127.0.0.1]) by wonkity.com (8.14.7/8.14.7) with ESMTP id r72EChUM098193; Fri, 2 Aug 2013 08:12:43 -0600 (MDT) (envelope-from wblock@wonkity.com) Received: from localhost (wblock@localhost) by wonkity.com (8.14.7/8.14.7/Submit) with ESMTP id r72EChSF098190; Fri, 2 Aug 2013 08:12:43 -0600 (MDT) (envelope-from wblock@wonkity.com) Date: Fri, 2 Aug 2013 08:12:43 -0600 (MDT) From: Warren Block To: Ronald Klop Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (wonkity.com [127.0.0.1]); Fri, 02 Aug 2013 08:12:43 -0600 (MDT) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 14:12:45 -0000 On Fri, 2 Aug 2013, Ronald Klop wrote: > On Fri, 02 Aug 2013 09:48:03 +0200, Tim Gustafson wrote: > >> Ok, so I think I have a little more of an idea as to what happened. >> >> Apparently, when I did an "svn checkout" this evening, I grabbed >> releng/9.1 rather than releng/8.4. Apparently, that make >> buildworld/buildkernel/installkernel/installworld also upgraded the >> zpool for me, or something...I'm not sure what happened, but I *am* >> sure I didn't type "zpool upgrade" at any point. Is an automatic >> zpool upgrade included in installkernel now? > > No, it isn't. > >> So now I've got base/releng/9.1 revision number 253878 installed on >> that machine, which apparently does support the 5000 zpool version, >> but seems to somehow not have a compatible boot loader. >> >> I'm feeling like "gpart bootcode" ought to fix this problem for me, >> but I've tried several iterations of that without success. >> >> For now, I'm booted off the 9.2-BETA CD in single user mode, and then >> mounted my zpool and zfs file systems, and then ran "sh /etc/rc" by >> hand, which started (almost) everything up for me. My jails are >> running, so my services are up, so I'm going to sleep on it. I don't >> have some commands like "ps" and "top" right now in the root OS (/dev >> seems to be wedged), but at least I can poke around and look at some >> files and see what happened, and then maybe fix it. > > Maybe post the setup of your disks. So people can take a look if your > commands are right or wrong. > The output of 'gpart show' and 'gpart list' are a good starting point. Agreed. One thing that comes to mind is some older setups may only have 32K freebsd-boot partitions, and gptzfsboot is 41K at present. I'd expect an error on trying to install the bootcode, but have not tested it. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 14:29:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CAD52782; Fri, 2 Aug 2013 14:29:35 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-la0-x22a.google.com (mail-la0-x22a.google.com [IPv6:2a00:1450:4010:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1FA19294E; Fri, 2 Aug 2013 14:29:34 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id mf11so482676lab.29 for ; Fri, 02 Aug 2013 07:29:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=GPjFkPFuqcyyofH7B/DVcb4qspV8bAD+S/kNzJ+ebsU=; b=H4B4Yhq4MzYNSQJo8oHxJ6kl6qTGcGpu5m1HwjJqMMjE6qFPPUe94OHsa34U0AwY9D /pTvxjE8BaG9FwOscOBsjEsTwSLV0/gfR0F0UdvmXYUSsiJeIevIwVQbTcRMZtTG6RnG +tzmGyuAzdXH/88DFTWMZNBNiSRYGLkqV12MzImc7caVJ+cLpchL/U+eHiDk69bR8mwD b6ZuoveVt0z074WZyFWKcBqCUd8bPrk7YYUSVgxsFYVBMLcMBf8m1qHOX9ylDoNOO7zU H2+pRLJlHfbEi6cIJNEdOCd4J4MAtqgfnJCGiUvQPYA+UPgXIv3cR6azNO7MwNsoAfcG jhbQ== X-Received: by 10.112.219.102 with SMTP id pn6mr3685320lbc.18.1375453772938; Fri, 02 Aug 2013 07:29:32 -0700 (PDT) Received: from [192.168.1.125] (mau.donbass.com. [92.242.127.250]) by mx.google.com with ESMTPSA id m1sm3443967lag.3.2013.08.02.07.29.31 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 02 Aug 2013 07:29:32 -0700 (PDT) Message-ID: <51FBC24B.5030609@gmail.com> Date: Fri, 02 Aug 2013 17:29:31 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130802 Thunderbird/17.0.7 MIME-Version: 1.0 To: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> In-Reply-To: <51ED5B69.8050200@wasikowski.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 14:29:35 -0000 22.07.2013 19:18, Łukasz Wąsikowski wrote: > I've got a problem with booting zfs-on-root FreeBSD 9.2-PRERELEASE. I'm > getting: > > ZFS: i/o error - all block copies unavailable > ZFS: can't read MOS of pool klawisz > gptzfsboot: failed to mount default pool klawisz > > Machine is VM running under KVM on Proxmox 2.3-13. VM has 8 GB of RAM, > 400 GB of local storage with SCSI Controller type: Default (lsi). > > I'm not sure what I did to make this VM unbootable. I've installed > 9.2-PRERELEASE, did source based upgrade to r253470, mergemaster, > reinstalled bootcode and rebooted. To this point VM was bootable. > > Then I did installworld from /usr/src to ezjail's basejail (ezjail-admin > update -i), did mergemaster for jails, install some ports - none of this > should mess with booting. I rebooted VM and got unbootable system. > > When I boot from liveCD I can import this pool (scrub shows no errors), > mount it, chroot to it, and work with it. I just can't get it to boot. > > Some information about the system: > http://pastie.org/private/mtfhkx0wx0vve29xn0plw > > I've tried to downgrade to r252316 - no luck, system is still unbootable. > > Any hints how to go from here? First, how did you update bootcode? `ls -la /boot` also wood help. Second, what is your /etc/make.conf and /etc/src.conf? -- Sphinx of black quartz, judge my vow. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 14:40:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AB40FD03; Fri, 2 Aug 2013 14:40:36 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.wasikowski.net (mail.wasikowski.net [IPv6:2001:6a0:1cb::b]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 674B12A1B; Fri, 2 Aug 2013 14:40:36 +0000 (UTC) Received: from mail.wasikowski.net (mail.wasikowski.net [IPv6:2001:6a0:1cb::b]) by mail.wasikowski.net (Postfix) with ESMTP id 63CE12834; Fri, 2 Aug 2013 16:40:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at wasikowski.net Received: from mail.wasikowski.net ([IPv6:2001:6a0:1cb::b]) by mail.wasikowski.net (scan.wasikowski.net [IPv6:2001:6a0:1cb::b]) (amavisd-new, port 10026) with ESMTP id uCk0dhE-sdeN; Fri, 2 Aug 2013 16:40:33 +0200 (CEST) Received: from [192.168.138.150] (83-144-115-210.static.chello.pl [83.144.115.210]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.wasikowski.net (Postfix) with ESMTPSA id 34AF42830; Fri, 2 Aug 2013 16:40:32 +0200 (CEST) Message-ID: <51FBC4DC.4090506@wasikowski.net> Date: Fri, 02 Aug 2013 16:40:28 +0200 From: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Volodymyr Kostyrko Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> <51FBC24B.5030609@gmail.com> In-Reply-To: <51FBC24B.5030609@gmail.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 14:40:36 -0000 W dniu 2013-08-02 16:29, Volodymyr Kostyrko pisze: > 22.07.2013 19:18, Łukasz Wąsikowski wrote: >> I've got a problem with booting zfs-on-root FreeBSD 9.2-PRERELEASE. I'm >> getting: >> >> ZFS: i/o error - all block copies unavailable >> ZFS: can't read MOS of pool klawisz >> gptzfsboot: failed to mount default pool klawisz >> >> Machine is VM running under KVM on Proxmox 2.3-13. VM has 8 GB of RAM, >> 400 GB of local storage with SCSI Controller type: Default (lsi). >> >> I'm not sure what I did to make this VM unbootable. I've installed >> 9.2-PRERELEASE, did source based upgrade to r253470, mergemaster, >> reinstalled bootcode and rebooted. To this point VM was bootable. >> >> Then I did installworld from /usr/src to ezjail's basejail (ezjail-admin >> update -i), did mergemaster for jails, install some ports - none of this >> should mess with booting. I rebooted VM and got unbootable system. >> >> When I boot from liveCD I can import this pool (scrub shows no errors), >> mount it, chroot to it, and work with it. I just can't get it to boot. >> >> Some information about the system: >> http://pastie.org/private/mtfhkx0wx0vve29xn0plw >> >> I've tried to downgrade to r252316 - no luck, system is still unbootable. >> >> Any hints how to go from here? > > First, how did you update bootcode? `ls -la /boot` also wood help. > > Second, what is your /etc/make.conf and /etc/src.conf? I'm updating bootcode with: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 /etc/src.conf doesn't exist (I'm using GENERIC kernel on this VM) /etc/make.conf - http://pastebin.com/QapEWzfJ -- best regards, Lukasz Wasikowski From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 14:49:26 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0DB5B18B; Fri, 2 Aug 2013 14:49:26 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 579CA2A88; Fri, 2 Aug 2013 14:49:25 +0000 (UTC) Received: by mail-la0-f41.google.com with SMTP id ec20so511192lab.28 for ; Fri, 02 Aug 2013 07:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=nvTLD1eQb2Xc2MWScbYWZtcKq17Q8s9UYWN0WKDD/Do=; b=TrS70D0qOj+gxc4OFFCwNhLwFnoa8j4HiSRHkHbakr+Qp6furHq5rHlTzZo0onm8JV YOrWKftjhzqdggZZtuhbjqkllViXOBQknshZGZLgcE/ZHZ38JxNEWaEIARwl4jr4UMZI zsplU2qbUie+wbGE7VTV0PyMfhcZjXTi3fpIPbEDZ66q+7b/J1H6Vt7MH7Y9Q3LwHu3l jSrAAjKeBDw1PMWsJ9baBpJ50JRZ2jV4JymYifESUMsSumYPwdZgjv/BOBCjshBgOWvh XNWfuv+tJXBPLJgIL2UnWG3OKSQher3RnFATnXiupG+ZtGOhoN2nlJgy1N0ouZ86w+AR VqbQ== X-Received: by 10.152.29.161 with SMTP id l1mr3222059lah.17.1375454963185; Fri, 02 Aug 2013 07:49:23 -0700 (PDT) Received: from [192.168.1.125] (mau.donbass.com. [92.242.127.250]) by mx.google.com with ESMTPSA id j1sm3466250lag.4.2013.08.02.07.49.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 02 Aug 2013 07:49:22 -0700 (PDT) Message-ID: <51FBC6F1.9030408@gmail.com> Date: Fri, 02 Aug 2013 17:49:21 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130802 Thunderbird/17.0.7 MIME-Version: 1.0 To: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> <51FBC24B.5030609@gmail.com> <51FBC4DC.4090506@wasikowski.net> In-Reply-To: <51FBC4DC.4090506@wasikowski.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 14:49:26 -0000 02.08.2013 17:40, Łukasz Wąsikowski wrote: >>> Any hints how to go from here? >> >> First, how did you update bootcode? `ls -la /boot` also wood help. >> >> Second, what is your /etc/make.conf and /etc/src.conf? > > I'm updating bootcode with: > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 > > /etc/src.conf doesn't exist (I'm using GENERIC kernel on this VM) > /etc/make.conf - http://pastebin.com/QapEWzfJ > Looks good. Can you also try what Trond suggests about boot order? You can also list your boot fs in /boot/loader.conf like vfs.root.mountfrom=zfs:klawisz/ROOTFS. Or you can just add this at loader prompt. There's also a ${SRC}/tools/tools/zfsboottest script that can tell you something about booting from your pool. -- Sphinx of black quartz, judge my vow. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 14:54:05 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EF7E4397 for ; Fri, 2 Aug 2013 14:54:05 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (hill.egr.msu.edu [35.9.37.162]) by mx1.freebsd.org (Postfix) with ESMTP id C4C7C2ACF for ; Fri, 2 Aug 2013 14:54:05 +0000 (UTC) Received: from hill (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 7A6EA15530 for ; Fri, 2 Aug 2013 10:53:58 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by hill (hill.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 74Vf8EK-TIsF for ; Fri, 2 Aug 2013 10:53:58 -0400 (EDT) Received: from EGR authenticated sender Message-ID: <51FBC806.3070500@egr.msu.edu> Date: Fri, 02 Aug 2013 10:53:58 -0400 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130708 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 14:54:06 -0000 On 08/02/13 03:48, Tim Gustafson wrote: > Ok, so I think I have a little more of an idea as to what happened. > > Apparently, when I did an "svn checkout" this evening, I grabbed > releng/9.1 rather than releng/8.4. Apparently, that make > buildworld/buildkernel/installkernel/installworld also upgraded the > zpool for me, or something...I'm not sure what happened, but I *am* > sure I didn't type "zpool upgrade" at any point. Is an automatic > zpool upgrade included in installkernel now? can you run 'zpool history | grep upgrade' and see if it recalls any upgrade commands? From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 14:56:57 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id AC961438; Fri, 2 Aug 2013 14:56:57 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.wasikowski.net (mail.wasikowski.net [IPv6:2001:6a0:1cb::b]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5037B2AED; Fri, 2 Aug 2013 14:56:57 +0000 (UTC) Received: from mail.wasikowski.net (mail.wasikowski.net [IPv6:2001:6a0:1cb::b]) by mail.wasikowski.net (Postfix) with ESMTP id C857628B4; Fri, 2 Aug 2013 16:56:55 +0200 (CEST) X-Virus-Scanned: amavisd-new at wasikowski.net Received: from mail.wasikowski.net ([91.204.91.44]) by mail.wasikowski.net (scan.wasikowski.net [91.204.91.44]) (amavisd-new, port 10026) with ESMTP id SFfjPrAVN6Mz; Fri, 2 Aug 2013 16:56:55 +0200 (CEST) Received: from [192.168.138.150] (83-144-115-210.static.chello.pl [83.144.115.210]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.wasikowski.net (Postfix) with ESMTPSA id 629D828B1; Fri, 2 Aug 2013 16:56:55 +0200 (CEST) Message-ID: <51FBC8B3.4010304@wasikowski.net> Date: Fri, 02 Aug 2013 16:56:51 +0200 From: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Volodymyr Kostyrko Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> <51FBC24B.5030609@gmail.com> <51FBC4DC.4090506@wasikowski.net> <51FBC6F1.9030408@gmail.com> In-Reply-To: <51FBC6F1.9030408@gmail.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 14:56:57 -0000 W dniu 2013-08-02 16:49, Volodymyr Kostyrko pisze: > 02.08.2013 17:40, Łukasz Wąsikowski wrote: >>>> Any hints how to go from here? >>> >>> First, how did you update bootcode? `ls -la /boot` also wood help. >>> >>> Second, what is your /etc/make.conf and /etc/src.conf? >> >> I'm updating bootcode with: >> >> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 >> >> /etc/src.conf doesn't exist (I'm using GENERIC kernel on this VM) >> /etc/make.conf - http://pastebin.com/QapEWzfJ >> > > Looks good. > > Can you also try what Trond suggests about boot order? You can also list > your boot fs in /boot/loader.conf like I listed ROOTFS in /boot/loader.conf, didn't helped. I'm using this boot order on 20+ boxes and never had any issues with it, but I'll check it as I don't have better idea what to do next. > vfs.root.mountfrom=zfs:klawisz/ROOTFS. Or you can just add this at > loader prompt. > > There's also a ${SRC}/tools/tools/zfsboottest script that can tell you > something about booting from your pool. This tools doesn't compile on 9.2-BETA2 r253884 # cd /usr/src/tools/tools/zfsboottest/ && make Warning: Object directory not changed from original /usr/src/tools/tools/zfsboottest ln -sf /usr/src/tools/tools/zfsboottest/../../../sys/i386/include machine cc -O1 -I/usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs -I/usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs -I. -fdiagnostics-show-option -W -Wextra -Wno-sign-compare -Wno-unused-parameter -Werror -std=gnu99 -fstack-protector -c zfsboottest.c cc1: warnings being treated as errors In file included from /usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs/zfssubr.c:122, from /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:38, from zfsboottest.c:55: /usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs/lz4.c: In function 'lz4_decompress': /usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs/lz4.c:45: warning: implicit declaration of function 'htonl' In file included from zfsboottest.c:55: /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c: In function 'spa_status': /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:817: error: 'ZFS_MAXNAMELEN' undeclared (first use in this function) /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:817: error: (Each undeclared identifier is reported only once /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:817: error: for each function it appears in.) *** [zfsboottest.o] Error code 1 Stop in /usr/src/tools/tools/zfsboottest. -- best regards, Lukasz Wasikowski From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 15:14:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4E38D8DA; Fri, 2 Aug 2013 15:14:23 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-lb0-x22f.google.com (mail-lb0-x22f.google.com [IPv6:2a00:1450:4010:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9764F2BC1; Fri, 2 Aug 2013 15:14:22 +0000 (UTC) Received: by mail-lb0-f175.google.com with SMTP id 13so528483lba.6 for ; Fri, 02 Aug 2013 08:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=f87Ssaw4vUZNQ1c/QNhfKW/3i+XAftoH0pURdRdDKSs=; b=APtol2X2GjRMEbwoap0eKSypiAuSWw47XXOf9KEIxIcXD29BdaqH9dNXCmqw3c5heF QoqGJZBcI6z5ERfO8wi8t3iRASCTnUFDHZnclOGh8HHhnymvrYGOztFdjvo1eOI5pL82 17yuZZg9ldY4OC8LOSTCn5GUQMLigMZlDhy9wgYYsrnWfe2iaA5+nh+NXEziTOp6L5Ub 2lOk9Qlv0hUd+GFOBY3Oy9UCooCjOB6WS3tPpGbGegxfVa47lap568qwwNp7iGBhXquw 7VSWpq/MkzmGFAdEyk2WZYJr0F2EpUIBMXeNm+N+KuaC1YswJCPgXAHpWa5h8E7sdUcN Y6Mg== X-Received: by 10.152.26.104 with SMTP id k8mr3072431lag.85.1375456458562; Fri, 02 Aug 2013 08:14:18 -0700 (PDT) Received: from [192.168.1.125] (mau.donbass.com. [92.242.127.250]) by mx.google.com with ESMTPSA id 8sm3485149lbq.4.2013.08.02.08.14.17 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 02 Aug 2013 08:14:18 -0700 (PDT) Message-ID: <51FBCCC8.6040805@gmail.com> Date: Fri, 02 Aug 2013 18:14:16 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130802 Thunderbird/17.0.7 MIME-Version: 1.0 To: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> <51FBC24B.5030609@gmail.com> <51FBC4DC.4090506@wasikowski.net> <51FBC6F1.9030408@gmail.com> <51FBC8B3.4010304@wasikowski.net> In-Reply-To: <51FBC8B3.4010304@wasikowski.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 15:14:23 -0000 02.08.2013 17:56, Łukasz Wąsikowski wrote: >> Can you also try what Trond suggests about boot order? You can also list >> your boot fs in /boot/loader.conf like > > I listed ROOTFS in /boot/loader.conf, didn't helped. I'm using this boot > order on 20+ boxes and never had any issues with it, but I'll check it > as I don't have better idea what to do next. I'm almost out of suggestions... 1. Can you try to removing 'canmount' property from klawisz fs and 'bootfs' property from pool? 2. Can you try fetching late nexenta or illumos and give your pool a full scrub? 3. Can you try disablng/enabling features? I assume you haven't used compression and have no snapshots. >> vfs.root.mountfrom=zfs:klawisz/ROOTFS. Or you can just add this at >> loader prompt. >> >> There's also a ${SRC}/tools/tools/zfsboottest script that can tell you >> something about booting from your pool. > > This tools doesn't compile on 9.2-BETA2 r253884 > > # cd /usr/src/tools/tools/zfsboottest/ && make > > Warning: Object directory not changed from original > /usr/src/tools/tools/zfsboottest > ln -sf /usr/src/tools/tools/zfsboottest/../../../sys/i386/include machine > cc -O1 -I/usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs > -I/usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs -I. > -fdiagnostics-show-option -W -Wextra -Wno-sign-compare > -Wno-unused-parameter -Werror -std=gnu99 -fstack-protector -c > zfsboottest.c > cc1: warnings being treated as errors > In file included from > /usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs/zfssubr.c:122, > from > /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:38, > from zfsboottest.c:55: > /usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs/lz4.c: In > function 'lz4_decompress': > /usr/src/tools/tools/zfsboottest/../../../sys/cddl/boot/zfs/lz4.c:45: > warning: implicit declaration of function 'htonl' > In file included from zfsboottest.c:55: > /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c: In > function 'spa_status': > /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:817: > error: 'ZFS_MAXNAMELEN' undeclared (first use in this function) > /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:817: > error: (Each undeclared identifier is reported only once > /usr/src/tools/tools/zfsboottest/../../../sys/boot/zfs/zfsimpl.c:817: > error: for each function it appears in.) > *** [zfsboottest.o] Error code 1 > > Stop in /usr/src/tools/tools/zfsboottest. Ahem, sorry. Never used that. -- Sphinx of black quartz, judge my vow. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 15:43:08 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BFEC442F for ; Fri, 2 Aug 2013 15:43:08 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-oa0-x234.google.com (mail-oa0-x234.google.com [IPv6:2607:f8b0:4003:c02::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 824102D27 for ; Fri, 2 Aug 2013 15:43:08 +0000 (UTC) Received: by mail-oa0-f52.google.com with SMTP id n12so1607016oag.25 for ; Fri, 02 Aug 2013 08:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=h/DyHHa4l6iM5YIh6orNw7nnyYY61o01RBbOzkWBIfA=; b=D4N+4ULKOI2PvB69T87UIpA8sxCCX2MDuIb8f4UNczS/o8J261RWxTIA2hm+f9suPL Us3jXg9V0JB+ePogDeSnOCTimY4yltdfVDhXGePFtIq0T1MJ0Z7foQ0V58UkmR6xSHJ4 XuIM8cdfbeG+fmHTiz3QB+GQbGLjWhHOTjeYY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=h/DyHHa4l6iM5YIh6orNw7nnyYY61o01RBbOzkWBIfA=; b=RQ+drUh2zPy0Lsj43Lc4y74xpOx0ak57mHizvqOFhPEDZGzXW172oazzOeMpc9kX4e xQtW+mKL2qvL6qxvZq/u49xkRGom4rMDUXQ0Bt6Na6GJYjYUAL6zCOkzdM5cn+8Lh2lN oy11iG8OkLZBT+6W2nRjD7dO5j9flbomixe6T8xvaGkw6Q7dlCLvKh0x9eXDtUPHIKwZ z86SRC8B6pepD8uJLrEaJfXobJvSXQ8AvGT5WM0aT/6B9GIsDDALIb+FucFD9SIDlLzf q0z06UjiyLlgkpung0owWBdKztTmsdi/uc48rynsulBTck3qBQ+wDQsbCK9YPYSSovP1 pWjg== MIME-Version: 1.0 X-Received: by 10.50.82.104 with SMTP id h8mr352443igy.17.1375458187747; Fri, 02 Aug 2013 08:43:07 -0700 (PDT) Received: by 10.42.233.143 with HTTP; Fri, 2 Aug 2013 08:43:07 -0700 (PDT) In-Reply-To: <51FBC806.3070500@egr.msu.edu> References: <51FBC806.3070500@egr.msu.edu> Date: Fri, 2 Aug 2013 08:43:07 -0700 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Tim Gustafson To: freebsd-fs@freebsd.org Content-Type: multipart/mixed; boundary=e89a8f503b16af3a8304e2f8d00a X-Gm-Message-State: ALoCoQnuzg4YN//eV03SOI9nh4TKoPWzXswpRrJ3fzUUapFWaCJiKZ1y9QWcy4FAVq2nwLm+htVK X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 15:43:08 -0000 --e89a8f503b16af3a8304e2f8d00a Content-Type: text/plain; charset=ISO-8859-1 First, let me thank you all for offering help. I really appreciate it. I'm going to try to respond to all your comments and questions in-line here: > Maybe post the setup of your disks. So people can take a look if your > commands are right or wrong. The output of 'gpart show' and 'gpart list' > are a good starting point. I've attached the output of both commands to this e-mail. Incidentally, this zpool was built using mfsBSD (http://mfsbsd.vx.sk/) originally as a FreeBSD 8.2 system, if that matters. > As it was already indicated, there is no support for zpool version 5000 > (feature flags) in base/releng/9.1. There is however in base/stable/9 and > in base/stable/8 what you obviously has before. I'm quite certain I've never tracked anything other than the following two Subversion URLs for my /usr/src: http://svn.freebsd.org/base/releng/8.2 http://svn.freebsd.org/base/releng/8.4 http://svn.freebsd.org/base/releng/9.1 As I understand it, those are supposed to be only "release" versions; they aren't "stable" versions. Or am I using the incorrect URLs? > One thing that comes to mind is some older setups may only have 32K > freebsd-boot partitions, and gptzfsboot is 41K at present. I'd expect an > error on trying to install the bootcode, but have not tested it. This has never been anything other than a 64-bit machine. > can you run 'zpool history | grep upgrade' and see if it recalls any > upgrade commands? Actually, I can't right now. All zfs-related commands fail with: internal error: failed to initialized ZFS library I suspect that's because I don't have a proper /dev and because my boot environment is now on an inaccessible mount point, as my boot root partition has been over-mounted with my zfs root partition: /dev/iso9660/FREEBSD_INSTALL on / (cd9660, local, read-only) devfs on /dev (devfs, local, multilabel) tank/root on / (zfs, local, noatime, nfsv4acls) tank/root/jails on /jails (zfs, local, noatime, nfsv4acls) tank/root/jails/db-01 on /jails/db-01 (zfs, local, noatime, nfsv4acls) tank/root/jails/www-01 on /jails/www-01 (zfs, local, noatime, nfsv4acls) devfs on /jails/db-01/dev (devfs, local, multilabel) devfs on /jails/www-01/dev (devfs, local, multilabel) So, I was thinking about possible recovery modes now, and I wonder: it seems that the zpool version has been upgraded to something that's not supported by releng/9.1, but the ZFS file systems themselves are still at version 5, as far as I can tell. So, wouldn't it be possible to "zfs send" my file systems to another box that is running releng/9.1 so that the snapshots are preserved, rebuild this box as a fresh releng/9.1 install and then "zfs send" the file systems back again? Or does upgrading the zpool also affect the zfs file systems, even if the zfs file system version hasn't changed? -- Tim Gustafson tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A --e89a8f503b16af3a8304e2f8d00a Content-Type: text/plain; charset=US-ASCII; name="gpart-list.txt" Content-Disposition: attachment; filename="gpart-list.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hjvjqlgi0 R2VvbSBuYW1lOiBkYTANCm1vZGlmaWVkOiBmYWxzZQ0Kc3RhdGU6IE9LDQpmd2hlYWRzOiAyNTUN CmZ3c2VjdG9yczogNjMNCmxhc3Q6IDE5NTM1MjUxMzQNCmZpcnN0OiAzNA0KZW50cmllczogMTI4 DQpzY2hlbWU6IEdQVA0KUHJvdmlkZXJzOg0KMS4gTmFtZTogZGEwcDENCiAgIE1lZGlhc2l6ZTog NjU1MzYgKDY0aykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgU3RyaXBlc2l6ZTogMA0KICAgU3Ry aXBlb2Zmc2V0OiAxNzQwOA0KICAgTW9kZTogcjB3MGUwDQogICByYXd1dWlkOiA1NjgwYmQzMi0w OTRjLTExZTAtOTZlMS04NDJiMmI1YTllZTANCiAgIHJhd3R5cGU6IDgzYmQ2YjlkLTdmNDEtMTFk Yy1iZTBiLTAwMTU2MGI4NGYwZg0KICAgbGFiZWw6IChudWxsKQ0KICAgbGVuZ3RoOiA2NTUzNg0K ICAgb2Zmc2V0OiAxNzQwOA0KICAgdHlwZTogZnJlZWJzZC1ib290DQogICBpbmRleDogMQ0KICAg ZW5kOiAxNjENCiAgIHN0YXJ0OiAzNA0KMi4gTmFtZTogZGEwcDINCiAgIE1lZGlhc2l6ZTogMjU3 Njk4MDM3NzYgKDI0RykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgU3RyaXBlc2l6ZTogMA0KICAg U3RyaXBlb2Zmc2V0OiA4Mjk0NA0KICAgTW9kZTogcjB3MGUwDQogICByYXd1dWlkOiA1NjhjMDFl MS0wOTRjLTExZTAtOTZlMS04NDJiMmI1YTllZTANCiAgIHJhd3R5cGU6IDUxNmU3Y2I1LTZlY2Yt MTFkNi04ZmY4LTAwMDIyZDA5NzEyYg0KICAgbGFiZWw6IChudWxsKQ0KICAgbGVuZ3RoOiAyNTc2 OTgwMzc3Ng0KICAgb2Zmc2V0OiA4Mjk0NA0KICAgdHlwZTogZnJlZWJzZC1zd2FwDQogICBpbmRl eDogMg0KICAgZW5kOiA1MDMzMTgwOQ0KICAgc3RhcnQ6IDE2Mg0KMy4gTmFtZTogZGEwcDMNCiAg IE1lZGlhc2l6ZTogOTc0NDM0OTgyNDAwICg5MDdHKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBT dHJpcGVzaXplOiAwDQogICBTdHJpcGVvZmZzZXQ6IDgyOTQ0DQogICBNb2RlOiByMXcxZTINCiAg IHJhd3V1aWQ6IDU2OTkyNWJiLTA5NGMtMTFlMC05NmUxLTg0MmIyYjVhOWVlMA0KICAgcmF3dHlw ZTogNTE2ZTdjYmEtNmVjZi0xMWQ2LThmZjgtMDAwMjJkMDk3MTJiDQogICBsYWJlbDogKG51bGwp DQogICBsZW5ndGg6IDk3NDQzNDk4MjQwMA0KICAgb2Zmc2V0OiAyNTc2OTg4NjcyMA0KICAgdHlw ZTogZnJlZWJzZC16ZnMNCiAgIGluZGV4OiAzDQogICBlbmQ6IDE5NTM1MjUxMzQNCiAgIHN0YXJ0 OiA1MDMzMTgxMA0KQ29uc3VtZXJzOg0KMS4gTmFtZTogZGEwDQogICBNZWRpYXNpemU6IDEwMDAy MDQ4ODYwMTYgKDkzMUcpDQogICBTZWN0b3JzaXplOiA1MTINCiAgIE1vZGU6IHIxdzFlMw0KDQpH ZW9tIG5hbWU6IGRhMQ0KbW9kaWZpZWQ6IGZhbHNlDQpzdGF0ZTogT0sNCmZ3aGVhZHM6IDI1NQ0K ZndzZWN0b3JzOiA2Mw0KbGFzdDogMTk1MzUyNTEzNA0KZmlyc3Q6IDM0DQplbnRyaWVzOiAxMjgN CnNjaGVtZTogR1BUDQpQcm92aWRlcnM6DQoxLiBOYW1lOiBkYTFwMQ0KICAgTWVkaWFzaXplOiA2 NTUzNiAoNjRrKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBTdHJpcGVzaXplOiAwDQogICBTdHJp cGVvZmZzZXQ6IDE3NDA4DQogICBNb2RlOiByMHcwZTANCiAgIHJhd3V1aWQ6IDU3NzVmZDcwLTA5 NGMtMTFlMC05NmUxLTg0MmIyYjVhOWVlMA0KICAgcmF3dHlwZTogODNiZDZiOWQtN2Y0MS0xMWRj LWJlMGItMDAxNTYwYjg0ZjBmDQogICBsYWJlbDogKG51bGwpDQogICBsZW5ndGg6IDY1NTM2DQog ICBvZmZzZXQ6IDE3NDA4DQogICB0eXBlOiBmcmVlYnNkLWJvb3QNCiAgIGluZGV4OiAxDQogICBl bmQ6IDE2MQ0KICAgc3RhcnQ6IDM0DQoyLiBOYW1lOiBkYTFwMg0KICAgTWVkaWFzaXplOiAyNTc2 OTgwMzc3NiAoMjRHKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBTdHJpcGVzaXplOiAwDQogICBT dHJpcGVvZmZzZXQ6IDgyOTQ0DQogICBNb2RlOiByMHcwZTANCiAgIHJhd3V1aWQ6IDU3ODAwM2M0 LTA5NGMtMTFlMC05NmUxLTg0MmIyYjVhOWVlMA0KICAgcmF3dHlwZTogNTE2ZTdjYjUtNmVjZi0x MWQ2LThmZjgtMDAwMjJkMDk3MTJiDQogICBsYWJlbDogKG51bGwpDQogICBsZW5ndGg6IDI1NzY5 ODAzNzc2DQogICBvZmZzZXQ6IDgyOTQ0DQogICB0eXBlOiBmcmVlYnNkLXN3YXANCiAgIGluZGV4 OiAyDQogICBlbmQ6IDUwMzMxODA5DQogICBzdGFydDogMTYyDQozLiBOYW1lOiBkYTFwMw0KICAg TWVkaWFzaXplOiA5NzQ0MzQ5ODI0MDAgKDkwN0cpDQogICBTZWN0b3JzaXplOiA1MTINCiAgIFN0 cmlwZXNpemU6IDANCiAgIFN0cmlwZW9mZnNldDogODI5NDQNCiAgIE1vZGU6IHIxdzFlMg0KICAg cmF3dXVpZDogNTc4YWE4NzktMDk0Yy0xMWUwLTk2ZTEtODQyYjJiNWE5ZWUwDQogICByYXd0eXBl OiA1MTZlN2NiYS02ZWNmLTExZDYtOGZmOC0wMDAyMmQwOTcxMmINCiAgIGxhYmVsOiAobnVsbCkN CiAgIGxlbmd0aDogOTc0NDM0OTgyNDAwDQogICBvZmZzZXQ6IDI1NzY5ODg2NzIwDQogICB0eXBl OiBmcmVlYnNkLXpmcw0KICAgaW5kZXg6IDMNCiAgIGVuZDogMTk1MzUyNTEzNA0KICAgc3RhcnQ6 IDUwMzMxODEwDQpDb25zdW1lcnM6DQoxLiBOYW1lOiBkYTENCiAgIE1lZGlhc2l6ZTogMTAwMDIw NDg4NjAxNiAoOTMxRykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgTW9kZTogcjF3MWUzDQoNCkdl b20gbmFtZTogZGEyDQptb2RpZmllZDogZmFsc2UNCnN0YXRlOiBPSw0KZndoZWFkczogMjU1DQpm d3NlY3RvcnM6IDYzDQpsYXN0OiAxOTUzNTI1MTM0DQpmaXJzdDogMzQNCmVudHJpZXM6IDEyOA0K c2NoZW1lOiBHUFQNClByb3ZpZGVyczoNCjEuIE5hbWU6IGRhMnAxDQogICBNZWRpYXNpemU6IDY1 NTM2ICg2NGspDQogICBTZWN0b3JzaXplOiA1MTINCiAgIFN0cmlwZXNpemU6IDANCiAgIFN0cmlw ZW9mZnNldDogMTc0MDgNCiAgIE1vZGU6IHIwdzBlMA0KICAgcmF3dXVpZDogNTg2OTNhN2UtMDk0 Yy0xMWUwLTk2ZTEtODQyYjJiNWE5ZWUwDQogICByYXd0eXBlOiA4M2JkNmI5ZC03ZjQxLTExZGMt YmUwYi0wMDE1NjBiODRmMGYNCiAgIGxhYmVsOiAobnVsbCkNCiAgIGxlbmd0aDogNjU1MzYNCiAg IG9mZnNldDogMTc0MDgNCiAgIHR5cGU6IGZyZWVic2QtYm9vdA0KICAgaW5kZXg6IDENCiAgIGVu ZDogMTYxDQogICBzdGFydDogMzQNCjIuIE5hbWU6IGRhMnAyDQogICBNZWRpYXNpemU6IDI1NzY5 ODAzNzc2ICgyNEcpDQogICBTZWN0b3JzaXplOiA1MTINCiAgIFN0cmlwZXNpemU6IDANCiAgIFN0 cmlwZW9mZnNldDogODI5NDQNCiAgIE1vZGU6IHIwdzBlMA0KICAgcmF3dXVpZDogNTg3NDRjZTIt MDk0Yy0xMWUwLTk2ZTEtODQyYjJiNWE5ZWUwDQogICByYXd0eXBlOiA1MTZlN2NiNS02ZWNmLTEx ZDYtOGZmOC0wMDAyMmQwOTcxMmINCiAgIGxhYmVsOiAobnVsbCkNCiAgIGxlbmd0aDogMjU3Njk4 MDM3NzYNCiAgIG9mZnNldDogODI5NDQNCiAgIHR5cGU6IGZyZWVic2Qtc3dhcA0KICAgaW5kZXg6 IDINCiAgIGVuZDogNTAzMzE4MDkNCiAgIHN0YXJ0OiAxNjINCjMuIE5hbWU6IGRhMnAzDQogICBN ZWRpYXNpemU6IDk3NDQzNDk4MjQwMCAoOTA3RykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgU3Ry aXBlc2l6ZTogMA0KICAgU3RyaXBlb2Zmc2V0OiA4Mjk0NA0KICAgTW9kZTogcjF3MWUyDQogICBy YXd1dWlkOiA1ODdlYTEyNi0wOTRjLTExZTAtOTZlMS04NDJiMmI1YTllZTANCiAgIHJhd3R5cGU6 IDUxNmU3Y2JhLTZlY2YtMTFkNi04ZmY4LTAwMDIyZDA5NzEyYg0KICAgbGFiZWw6IChudWxsKQ0K ICAgbGVuZ3RoOiA5NzQ0MzQ5ODI0MDANCiAgIG9mZnNldDogMjU3Njk4ODY3MjANCiAgIHR5cGU6 IGZyZWVic2QtemZzDQogICBpbmRleDogMw0KICAgZW5kOiAxOTUzNTI1MTM0DQogICBzdGFydDog NTAzMzE4MTANCkNvbnN1bWVyczoNCjEuIE5hbWU6IGRhMg0KICAgTWVkaWFzaXplOiAxMDAwMjA0 ODg2MDE2ICg5MzFHKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBNb2RlOiByMXcxZTMNCg0KR2Vv bSBuYW1lOiBkYTMNCm1vZGlmaWVkOiBmYWxzZQ0Kc3RhdGU6IE9LDQpmd2hlYWRzOiAyNTUNCmZ3 c2VjdG9yczogNjMNCmxhc3Q6IDE5NTM1MjUxMzQNCmZpcnN0OiAzNA0KZW50cmllczogMTI4DQpz Y2hlbWU6IEdQVA0KUHJvdmlkZXJzOg0KMS4gTmFtZTogZGEzcDENCiAgIE1lZGlhc2l6ZTogNjU1 MzYgKDY0aykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgU3RyaXBlc2l6ZTogMA0KICAgU3RyaXBl b2Zmc2V0OiAxNzQwOA0KICAgTW9kZTogcjB3MGUwDQogICByYXd1dWlkOiA1OTU5OWVlMC0wOTRj LTExZTAtOTZlMS04NDJiMmI1YTllZTANCiAgIHJhd3R5cGU6IDgzYmQ2YjlkLTdmNDEtMTFkYy1i ZTBiLTAwMTU2MGI4NGYwZg0KICAgbGFiZWw6IChudWxsKQ0KICAgbGVuZ3RoOiA2NTUzNg0KICAg b2Zmc2V0OiAxNzQwOA0KICAgdHlwZTogZnJlZWJzZC1ib290DQogICBpbmRleDogMQ0KICAgZW5k OiAxNjENCiAgIHN0YXJ0OiAzNA0KMi4gTmFtZTogZGEzcDINCiAgIE1lZGlhc2l6ZTogMjU3Njk4 MDM3NzYgKDI0RykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgU3RyaXBlc2l6ZTogMA0KICAgU3Ry aXBlb2Zmc2V0OiA4Mjk0NA0KICAgTW9kZTogcjB3MGUwDQogICByYXd1dWlkOiA1OTYzYTc4ZC0w OTRjLTExZTAtOTZlMS04NDJiMmI1YTllZTANCiAgIHJhd3R5cGU6IDUxNmU3Y2I1LTZlY2YtMTFk Ni04ZmY4LTAwMDIyZDA5NzEyYg0KICAgbGFiZWw6IChudWxsKQ0KICAgbGVuZ3RoOiAyNTc2OTgw Mzc3Ng0KICAgb2Zmc2V0OiA4Mjk0NA0KICAgdHlwZTogZnJlZWJzZC1zd2FwDQogICBpbmRleDog Mg0KICAgZW5kOiA1MDMzMTgwOQ0KICAgc3RhcnQ6IDE2Mg0KMy4gTmFtZTogZGEzcDMNCiAgIE1l ZGlhc2l6ZTogOTc0NDM0OTgyNDAwICg5MDdHKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBTdHJp cGVzaXplOiAwDQogICBTdHJpcGVvZmZzZXQ6IDgyOTQ0DQogICBNb2RlOiByMXcxZTINCiAgIHJh d3V1aWQ6IDU5NmY2YTE0LTA5NGMtMTFlMC05NmUxLTg0MmIyYjVhOWVlMA0KICAgcmF3dHlwZTog NTE2ZTdjYmEtNmVjZi0xMWQ2LThmZjgtMDAwMjJkMDk3MTJiDQogICBsYWJlbDogKG51bGwpDQog ICBsZW5ndGg6IDk3NDQzNDk4MjQwMA0KICAgb2Zmc2V0OiAyNTc2OTg4NjcyMA0KICAgdHlwZTog ZnJlZWJzZC16ZnMNCiAgIGluZGV4OiAzDQogICBlbmQ6IDE5NTM1MjUxMzQNCiAgIHN0YXJ0OiA1 MDMzMTgxMA0KQ29uc3VtZXJzOg0KMS4gTmFtZTogZGEzDQogICBNZWRpYXNpemU6IDEwMDAyMDQ4 ODYwMTYgKDkzMUcpDQogICBTZWN0b3JzaXplOiA1MTINCiAgIE1vZGU6IHIxdzFlMw0KDQpHZW9t IG5hbWU6IGRhNA0KbW9kaWZpZWQ6IGZhbHNlDQpzdGF0ZTogT0sNCmZ3aGVhZHM6IDI1NQ0KZndz ZWN0b3JzOiA2Mw0KbGFzdDogMTk1MzUyNTEzNA0KZmlyc3Q6IDM0DQplbnRyaWVzOiAxMjgNCnNj aGVtZTogR1BUDQpQcm92aWRlcnM6DQoxLiBOYW1lOiBkYTRwMQ0KICAgTWVkaWFzaXplOiA2NTUz NiAoNjRrKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBTdHJpcGVzaXplOiAwDQogICBTdHJpcGVv ZmZzZXQ6IDE3NDA4DQogICBNb2RlOiByMHcwZTANCiAgIHJhd3V1aWQ6IDVhNGU4MTdhLTA5NGMt MTFlMC05NmUxLTg0MmIyYjVhOWVlMA0KICAgcmF3dHlwZTogODNiZDZiOWQtN2Y0MS0xMWRjLWJl MGItMDAxNTYwYjg0ZjBmDQogICBsYWJlbDogKG51bGwpDQogICBsZW5ndGg6IDY1NTM2DQogICBv ZmZzZXQ6IDE3NDA4DQogICB0eXBlOiBmcmVlYnNkLWJvb3QNCiAgIGluZGV4OiAxDQogICBlbmQ6 IDE2MQ0KICAgc3RhcnQ6IDM0DQoyLiBOYW1lOiBkYTRwMg0KICAgTWVkaWFzaXplOiAyNTc2OTgw Mzc3NiAoMjRHKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBTdHJpcGVzaXplOiAwDQogICBTdHJp cGVvZmZzZXQ6IDgyOTQ0DQogICBNb2RlOiByMHcwZTANCiAgIHJhd3V1aWQ6IDVhNTg4MzVhLTA5 NGMtMTFlMC05NmUxLTg0MmIyYjVhOWVlMA0KICAgcmF3dHlwZTogNTE2ZTdjYjUtNmVjZi0xMWQ2 LThmZjgtMDAwMjJkMDk3MTJiDQogICBsYWJlbDogKG51bGwpDQogICBsZW5ndGg6IDI1NzY5ODAz Nzc2DQogICBvZmZzZXQ6IDgyOTQ0DQogICB0eXBlOiBmcmVlYnNkLXN3YXANCiAgIGluZGV4OiAy DQogICBlbmQ6IDUwMzMxODA5DQogICBzdGFydDogMTYyDQozLiBOYW1lOiBkYTRwMw0KICAgTWVk aWFzaXplOiA5NzQ0MzQ5ODI0MDAgKDkwN0cpDQogICBTZWN0b3JzaXplOiA1MTINCiAgIFN0cmlw ZXNpemU6IDANCiAgIFN0cmlwZW9mZnNldDogODI5NDQNCiAgIE1vZGU6IHIxdzFlMg0KICAgcmF3 dXVpZDogNWE2NGEzNzMtMDk0Yy0xMWUwLTk2ZTEtODQyYjJiNWE5ZWUwDQogICByYXd0eXBlOiA1 MTZlN2NiYS02ZWNmLTExZDYtOGZmOC0wMDAyMmQwOTcxMmINCiAgIGxhYmVsOiAobnVsbCkNCiAg IGxlbmd0aDogOTc0NDM0OTgyNDAwDQogICBvZmZzZXQ6IDI1NzY5ODg2NzIwDQogICB0eXBlOiBm cmVlYnNkLXpmcw0KICAgaW5kZXg6IDMNCiAgIGVuZDogMTk1MzUyNTEzNA0KICAgc3RhcnQ6IDUw MzMxODEwDQpDb25zdW1lcnM6DQoxLiBOYW1lOiBkYTQNCiAgIE1lZGlhc2l6ZTogMTAwMDIwNDg4 NjAxNiAoOTMxRykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgTW9kZTogcjF3MWUzDQoNCkdlb20g bmFtZTogZGE1DQptb2RpZmllZDogZmFsc2UNCnN0YXRlOiBPSw0KZndoZWFkczogMjU1DQpmd3Nl Y3RvcnM6IDYzDQpsYXN0OiAxOTUzNTI1MTM0DQpmaXJzdDogMzQNCmVudHJpZXM6IDEyOA0Kc2No ZW1lOiBHUFQNClByb3ZpZGVyczoNCjEuIE5hbWU6IGRhNXAxDQogICBNZWRpYXNpemU6IDY1NTM2 ICg2NGspDQogICBTZWN0b3JzaXplOiA1MTINCiAgIFN0cmlwZXNpemU6IDANCiAgIFN0cmlwZW9m ZnNldDogMTc0MDgNCiAgIE1vZGU6IHIwdzBlMA0KICAgcmF3dXVpZDogNWI0NTgzODctMDk0Yy0x MWUwLTk2ZTEtODQyYjJiNWE5ZWUwDQogICByYXd0eXBlOiA4M2JkNmI5ZC03ZjQxLTExZGMtYmUw Yi0wMDE1NjBiODRmMGYNCiAgIGxhYmVsOiAobnVsbCkNCiAgIGxlbmd0aDogNjU1MzYNCiAgIG9m ZnNldDogMTc0MDgNCiAgIHR5cGU6IGZyZWVic2QtYm9vdA0KICAgaW5kZXg6IDENCiAgIGVuZDog MTYxDQogICBzdGFydDogMzQNCjIuIE5hbWU6IGRhNXAyDQogICBNZWRpYXNpemU6IDI1NzY5ODAz Nzc2ICgyNEcpDQogICBTZWN0b3JzaXplOiA1MTINCiAgIFN0cmlwZXNpemU6IDANCiAgIFN0cmlw ZW9mZnNldDogODI5NDQNCiAgIE1vZGU6IHIwdzBlMA0KICAgcmF3dXVpZDogNWI0ZjgxZTQtMDk0 Yy0xMWUwLTk2ZTEtODQyYjJiNWE5ZWUwDQogICByYXd0eXBlOiA1MTZlN2NiNS02ZWNmLTExZDYt OGZmOC0wMDAyMmQwOTcxMmINCiAgIGxhYmVsOiAobnVsbCkNCiAgIGxlbmd0aDogMjU3Njk4MDM3 NzYNCiAgIG9mZnNldDogODI5NDQNCiAgIHR5cGU6IGZyZWVic2Qtc3dhcA0KICAgaW5kZXg6IDIN CiAgIGVuZDogNTAzMzE4MDkNCiAgIHN0YXJ0OiAxNjINCjMuIE5hbWU6IGRhNXAzDQogICBNZWRp YXNpemU6IDk3NDQzNDk4MjQwMCAoOTA3RykNCiAgIFNlY3RvcnNpemU6IDUxMg0KICAgU3RyaXBl c2l6ZTogMA0KICAgU3RyaXBlb2Zmc2V0OiA4Mjk0NA0KICAgTW9kZTogcjF3MWUyDQogICByYXd1 dWlkOiA1YjVhZTY5ZS0wOTRjLTExZTAtOTZlMS04NDJiMmI1YTllZTANCiAgIHJhd3R5cGU6IDUx NmU3Y2JhLTZlY2YtMTFkNi04ZmY4LTAwMDIyZDA5NzEyYg0KICAgbGFiZWw6IChudWxsKQ0KICAg bGVuZ3RoOiA5NzQ0MzQ5ODI0MDANCiAgIG9mZnNldDogMjU3Njk4ODY3MjANCiAgIHR5cGU6IGZy ZWVic2QtemZzDQogICBpbmRleDogMw0KICAgZW5kOiAxOTUzNTI1MTM0DQogICBzdGFydDogNTAz MzE4MTANCkNvbnN1bWVyczoNCjEuIE5hbWU6IGRhNQ0KICAgTWVkaWFzaXplOiAxMDAwMjA0ODg2 MDE2ICg5MzFHKQ0KICAgU2VjdG9yc2l6ZTogNTEyDQogICBNb2RlOiByMXcxZTM= --e89a8f503b16af3a8304e2f8d00a Content-Type: text/plain; charset=US-ASCII; name="gpart-show.txt" Content-Disposition: attachment; filename="gpart-show.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hjvjqli11 PT4gICAgICAgIDM0ICAxOTUzNTI1MTAxICBkYTAgIEdQVCAgKDkzMUcpDQogICAgICAgICAgMzQg ICAgICAgICAxMjggICAgMSAgZnJlZWJzZC1ib290ICAoNjRrKQ0KICAgICAgICAgMTYyICAgIDUw MzMxNjQ4ICAgIDIgIGZyZWVic2Qtc3dhcCAgKDI0RykNCiAgICA1MDMzMTgxMCAgMTkwMzE5MzMy NSAgICAzICBmcmVlYnNkLXpmcyAgKDkwN0cpDQoNCj0+ICAgICAgICAzNCAgMTk1MzUyNTEwMSAg ZGExICBHUFQgICg5MzFHKQ0KICAgICAgICAgIDM0ICAgICAgICAgMTI4ICAgIDEgIGZyZWVic2Qt Ym9vdCAgKDY0aykNCiAgICAgICAgIDE2MiAgICA1MDMzMTY0OCAgICAyICBmcmVlYnNkLXN3YXAg ICgyNEcpDQogICAgNTAzMzE4MTAgIDE5MDMxOTMzMjUgICAgMyAgZnJlZWJzZC16ZnMgICg5MDdH KQ0KDQo9PiAgICAgICAgMzQgIDE5NTM1MjUxMDEgIGRhMiAgR1BUICAoOTMxRykNCiAgICAgICAg ICAzNCAgICAgICAgIDEyOCAgICAxICBmcmVlYnNkLWJvb3QgICg2NGspDQogICAgICAgICAxNjIg ICAgNTAzMzE2NDggICAgMiAgZnJlZWJzZC1zd2FwICAoMjRHKQ0KICAgIDUwMzMxODEwICAxOTAz MTkzMzI1ICAgIDMgIGZyZWVic2QtemZzICAoOTA3RykNCg0KPT4gICAgICAgIDM0ICAxOTUzNTI1 MTAxICBkYTMgIEdQVCAgKDkzMUcpDQogICAgICAgICAgMzQgICAgICAgICAxMjggICAgMSAgZnJl ZWJzZC1ib290ICAoNjRrKQ0KICAgICAgICAgMTYyICAgIDUwMzMxNjQ4ICAgIDIgIGZyZWVic2Qt c3dhcCAgKDI0RykNCiAgICA1MDMzMTgxMCAgMTkwMzE5MzMyNSAgICAzICBmcmVlYnNkLXpmcyAg KDkwN0cpDQoNCj0+ICAgICAgICAzNCAgMTk1MzUyNTEwMSAgZGE0ICBHUFQgICg5MzFHKQ0KICAg ICAgICAgIDM0ICAgICAgICAgMTI4ICAgIDEgIGZyZWVic2QtYm9vdCAgKDY0aykNCiAgICAgICAg IDE2MiAgICA1MDMzMTY0OCAgICAyICBmcmVlYnNkLXN3YXAgICgyNEcpDQogICAgNTAzMzE4MTAg IDE5MDMxOTMzMjUgICAgMyAgZnJlZWJzZC16ZnMgICg5MDdHKQ0KDQo9PiAgICAgICAgMzQgIDE5 NTM1MjUxMDEgIGRhNSAgR1BUICAoOTMxRykNCiAgICAgICAgICAzNCAgICAgICAgIDEyOCAgICAx ICBmcmVlYnNkLWJvb3QgICg2NGspDQogICAgICAgICAxNjIgICAgNTAzMzE2NDggICAgMiAgZnJl ZWJzZC1zd2FwICAoMjRHKQ0KICAgIDUwMzMxODEwICAxOTAzMTkzMzI1ICAgIDMgIGZyZWVic2Qt emZzICAoOTA3Ryk= --e89a8f503b16af3a8304e2f8d00a-- From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 15:52:23 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4E3576C1 for ; Fri, 2 Aug 2013 15:52:23 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C83C02D94 for ; Fri, 2 Aug 2013 15:52:22 +0000 (UTC) Received: by mail-la0-f50.google.com with SMTP id fn20so553994lab.9 for ; Fri, 02 Aug 2013 08:52:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=YWzFUQBxZc8zKCX9eOr0YLDm/Rf5MN+/0St24ABnuyc=; b=aju9F/+mvWE67DdUgWCL7c85tL70zUdJXYiCeVtQWGxQFMObalFzQo97TXlovCQBjC pn3+Oj6ZMkkhAEbzxAyXsObGqN60H0MnyTNmt69aLX9N1U9DV5PTy6q2bo73yBu2CW1W 5PuLDkaEVx9G1XQoSlkq7AfqsVQ1AY/6L9FoKPl3XZ1XP3CtHPhqE5qCO7ZfDlegv4dK LbCKAD5ga1GP7blxRK+x+QaQL7q6V+ZO3tQGGCPB0Q/+RC6AIVpO7OIKR1CgWDTg3c/j 66YsJbMtaV4T00n4ffpWKGMzQ6356+26YkBh9SWqwnLtUMpgvaGUAVt2M9UfA9VYFFD/ baHA== MIME-Version: 1.0 X-Received: by 10.152.21.131 with SMTP id v3mr3304111lae.50.1375458740609; Fri, 02 Aug 2013 08:52:20 -0700 (PDT) Received: by 10.112.201.41 with HTTP; Fri, 2 Aug 2013 08:52:20 -0700 (PDT) In-Reply-To: References: <51FBC806.3070500@egr.msu.edu> Date: Fri, 2 Aug 2013 16:52:20 +0100 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Tom Evans To: Tim Gustafson Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 15:52:23 -0000 On Fri, Aug 2, 2013 at 4:43 PM, Tim Gustafson wrote: > http://svn.freebsd.org/base/releng/8.2 > http://svn.freebsd.org/base/releng/8.4 > http://svn.freebsd.org/base/releng/9.1 > > As I understand it, those are supposed to be only "release" versions; > they aren't "stable" versions. Or am I using the incorrect URLs? No, those are release engineering branches - which receive updates - rather than release tags. The release tags are of this sort of variety: http://svn.freebsd.org/base/release/9.1.0 The other version is the stable, which have this structure: http://svn.freebsd.org/base/stable/9 Cheers Tom From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 16:18:56 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D8C52E6B for ; Fri, 2 Aug 2013 16:18:56 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-oa0-x232.google.com (mail-oa0-x232.google.com [IPv6:2607:f8b0:4003:c02::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9D3AF2F04 for ; Fri, 2 Aug 2013 16:18:56 +0000 (UTC) Received: by mail-oa0-f50.google.com with SMTP id i4so1731942oah.37 for ; Fri, 02 Aug 2013 09:18:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vHgJQr3QxB56AKTpTceA3/X+KUt2Cw6Lcyq93Q+P2bI=; b=mOPu/5dpuRQKE7iPCdW108NHXYkl4yRErkLmNJLF71e0MdICxDNlMCJOn6PuFE37I5 qzgI9gRoIv0QAFA81JfkAAuaEZGSvADv97ONqp6RVZyB2mEPeTZ+9ikGgxuyiwY4+8sH /AnO/0pjCI/OYBZe3nenqBvKXYssAhzA4IEic= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vHgJQr3QxB56AKTpTceA3/X+KUt2Cw6Lcyq93Q+P2bI=; b=Y6T9/8KrsnZv6+0hNMf59x5W27r0PnZvFC56YuzMiKp0SePS8KdmLJ32Z5qxJuw00H 9l1WkMY5ryLCHmxvdNpHTWNzGFNzByLsC7euU7eoXicvFy2fwLKPfz8RyEENr9IVsvES HG2iXV6mrCwRew7ydj3quB5jqi+3xIJlvHVCLl27ZzmTdEfOZA321N9YJMTCn58xOkes +mC2VmE8FV6u1gfMKi3NTMwF8pO6Cuq5Up06OQgf/mlpDPUKOvqZlPk62UIwzvWqj+s2 ZmuRD37Y7Lnb5B6VdufW4bUp+S/9tXYUeCOjr3lF1241CtxemWZyxsvcdw4ueNjC35uw DRKg== X-Gm-Message-State: ALoCoQkXXZB484x/I0LhXpoe+1teCno7VK77f7IB4iqJFwtvUc8Uo8+2mbHXtgegrCeYIvRkmhtF MIME-Version: 1.0 X-Received: by 10.50.106.48 with SMTP id gr16mr371350igb.26.1375460335816; Fri, 02 Aug 2013 09:18:55 -0700 (PDT) Received: by 10.42.233.143 with HTTP; Fri, 2 Aug 2013 09:18:55 -0700 (PDT) In-Reply-To: References: <51FBC806.3070500@egr.msu.edu> Date: Fri, 2 Aug 2013 09:18:55 -0700 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Tim Gustafson To: Tom Evans Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 16:18:56 -0000 >> http://svn.freebsd.org/base/releng/8.2 >> http://svn.freebsd.org/base/releng/8.4 >> http://svn.freebsd.org/base/releng/9.1 > No, those are release engineering branches - which receive updates - > rather than release tags. The release tags are of this sort of > variety: Sorry, I think I was using incorrect labels. The bottom line is: are the "releng" branches the ones that I should be tracking for production servers, or should I stick to "release"? Is it correct that if I stick to "release", I won't get any post-release updates at all? -- Tim Gustafson tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 16:25:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9F823FED for ; Fri, 2 Aug 2013 16:25:06 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qc0-x22b.google.com (mail-qc0-x22b.google.com [IPv6:2607:f8b0:400d:c01::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5FBDA2F51 for ; Fri, 2 Aug 2013 16:25:06 +0000 (UTC) Received: by mail-qc0-f171.google.com with SMTP id n10so429879qcw.2 for ; Fri, 02 Aug 2013 09:25:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6MW2wf2r0K77NJbNQzzp1VX1h/uH/dS3nRngEy251UE=; b=j+f0u59qTkNhr7gkCtJZTIrbcNc3CMI0vKh6sI/0AP7IK2rmmP/wO/ACP7jmD6yg3G BZTpitnlr9wbzTfb5BdKuL0uVZQezIGDGPmsqwy1dkPvW4+iFugo7XS6JSbCQQhsytA7 OEDv6bqAQ1VX5gT9YudjOjrCgShl9lFMDQY3tEYUs9M3onKjEQqZSQdubSjYi53ue04l L2/m3hzGqu3oUwVPEey2nFvxhls4UcGF7CdmmZf1d0wEeBhUdF7K/BBTSJQOtBRBL0E1 Z321OlaI/ogINZqSvo+0qFcyPMiJ94kDEzJw+LPn5+MTvUsb11Cd/aRu0T5aqFtxS0zR UbiQ== MIME-Version: 1.0 X-Received: by 10.224.3.197 with SMTP id 5mr1078422qao.25.1375460705547; Fri, 02 Aug 2013 09:25:05 -0700 (PDT) Received: by 10.224.78.194 with HTTP; Fri, 2 Aug 2013 09:25:05 -0700 (PDT) In-Reply-To: References: <51FBC806.3070500@egr.msu.edu> Date: Fri, 2 Aug 2013 19:25:05 +0300 Message-ID: Subject: Re: ZFS: unsupported ZFS version 5000 (should be 28) From: Kimmo Paasiala To: Tim Gustafson Content-Type: text/plain; charset=UTF-8 Cc: Tom Evans , freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 16:25:06 -0000 On Fri, Aug 2, 2013 at 7:18 PM, Tim Gustafson wrote: >>> http://svn.freebsd.org/base/releng/8.2 >>> http://svn.freebsd.org/base/releng/8.4 >>> http://svn.freebsd.org/base/releng/9.1 > >> No, those are release engineering branches - which receive updates - >> rather than release tags. The release tags are of this sort of >> variety: > > Sorry, I think I was using incorrect labels. The bottom line is: are > the "releng" branches the ones that I should be tracking for > production servers, or should I stick to "release"? Is it correct > that if I stick to "release", I won't get any post-release updates at > all? > > -- > > Tim Gustafson > tjg@ucsc.edu > 831-459-5354 > Baskin Engineering, Room 313A > _______________________________________________ Tracking release/x.y branches is incorrect because they are frozen in time snapshots that are never updated. Use the releng/x.y branches for production systems to get errata and security fixes. -Kimmo From owner-freebsd-fs@FreeBSD.ORG Fri Aug 2 19:21:45 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C627DE72; Fri, 2 Aug 2013 19:21:45 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.wasikowski.net (mail.wasikowski.net [IPv6:2001:6a0:1cb::b]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7FC5926DD; Fri, 2 Aug 2013 19:21:45 +0000 (UTC) Received: from mail.wasikowski.net (mail.wasikowski.net [IPv6:2001:6a0:1cb::b]) by mail.wasikowski.net (Postfix) with ESMTP id 100AC2A29; Fri, 2 Aug 2013 21:21:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at wasikowski.net Received: from mail.wasikowski.net ([IPv6:2001:6a0:1cb::b]) by mail.wasikowski.net (scan.wasikowski.net [IPv6:2001:6a0:1cb::b]) (amavisd-new, port 10026) with ESMTP id 3eIiiJp7hlcq; Fri, 2 Aug 2013 21:21:43 +0200 (CEST) Received: from [192.168.168.1] (89-71-136-148.dynamic.chello.pl [89.71.136.148]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.wasikowski.net (Postfix) with ESMTPSA id 976322A24; Fri, 2 Aug 2013 21:21:43 +0200 (CEST) Message-ID: <51FC06C8.50906@wasikowski.net> Date: Fri, 02 Aug 2013 21:21:44 +0200 From: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Volodymyr Kostyrko Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> <51FBC24B.5030609@gmail.com> <51FBC4DC.4090506@wasikowski.net> <51FBC6F1.9030408@gmail.com> <51FBC8B3.4010304@wasikowski.net> <51FBCCC8.6040805@gmail.com> In-Reply-To: <51FBCCC8.6040805@gmail.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Aug 2013 19:21:45 -0000 W dniu 2013-08-02 17:14, Volodymyr Kostyrko pisze: > 02.08.2013 17:56, Łukasz Wąsikowski wrote: >>> Can you also try what Trond suggests about boot order? You can also list >>> your boot fs in /boot/loader.conf like >> >> I listed ROOTFS in /boot/loader.conf, didn't helped. I'm using this boot >> order on 20+ boxes and never had any issues with it, but I'll check it >> as I don't have better idea what to do next. > > I'm almost out of suggestions... > > 1. Can you try to removing 'canmount' property from klawisz fs and > 'bootfs' property from pool? Property canmount is off for klawisz already. It's enabled for klawisz/ROOTFS though. I removed bootfs from the pool, didn't worked - can't read MOS. > 2. Can you try fetching late nexenta or illumos and give your pool a > full scrub? OpenIndiana's 151a7 scrub shows no error. > 3. Can you try disablng/enabling features? I assume you haven't used > compression and have no snapshots. I'm using lzjb compression on some not important for booting datasets (not on klawisz/ROOTFS) and I created some snapshots. -- best regards, Lukasz Wasikowski From owner-freebsd-fs@FreeBSD.ORG Sat Aug 3 10:54:14 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E4F9FFE2; Sat, 3 Aug 2013 10:54:14 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B82B32EDB; Sat, 3 Aug 2013 10:54:14 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r73AsEVJ021904; Sat, 3 Aug 2013 10:54:14 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r73AsEmx021903; Sat, 3 Aug 2013 10:54:14 GMT (envelope-from remko) Date: Sat, 3 Aug 2013 10:54:14 GMT Message-Id: <201308031054.r73AsEmx021903@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-fs@FreeBSD.org From: remko@FreeBSD.org Subject: Re: kern/181002: zpool import fails after export X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Aug 2013 10:54:15 -0000 Synopsis: zpool import fails after export Responsible-Changed-From-To: freebsd-i386->freebsd-fs Responsible-Changed-By: remko Responsible-Changed-When: Sat Aug 3 10:53:58 UTC 2013 Responsible-Changed-Why: Reassign to the fs team http://www.freebsd.org/cgi/query-pr.cgi?pr=181002 From owner-freebsd-fs@FreeBSD.ORG Sat Aug 3 15:43:53 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F215F118; Sat, 3 Aug 2013 15:43:52 +0000 (UTC) (envelope-from smh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C673B25F2; Sat, 3 Aug 2013 15:43:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r73Fhqvw077687; Sat, 3 Aug 2013 15:43:52 GMT (envelope-from smh@freefall.freebsd.org) Received: (from smh@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r73FhqXu077686; Sat, 3 Aug 2013 15:43:52 GMT (envelope-from smh) Date: Sat, 3 Aug 2013 15:43:52 GMT Message-Id: <201308031543.r73FhqXu077686@freefall.freebsd.org> To: harrison@biostat.wisc.edu, smh@FreeBSD.org, freebsd-fs@FreeBSD.org From: smh@FreeBSD.org Subject: Re: kern/181002: zpool import fails after export X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Aug 2013 15:43:53 -0000 Synopsis: zpool import fails after export State-Changed-From-To: open->closed State-Changed-By: smh State-Changed-When: Sat Aug 3 15:43:52 UTC 2013 State-Changed-Why: System only had 9.2 kernel not world http://www.freebsd.org/cgi/query-pr.cgi?pr=181002 From owner-freebsd-fs@FreeBSD.ORG Sat Aug 3 19:00:46 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 732A7ED6; Sat, 3 Aug 2013 19:00:46 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 8C9FF2A8E; Sat, 3 Aug 2013 19:00:45 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id WAA02519; Sat, 03 Aug 2013 22:00:37 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1V5h48-0006AL-PU; Sat, 03 Aug 2013 22:00:36 +0300 Message-ID: <51FD5303.30507@FreeBSD.org> Date: Sat, 03 Aug 2013 21:59:15 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130708 Thunderbird/17.0.7 MIME-Version: 1.0 To: =?UTF-8?B?xYF1a2FzeiBXxIVzaWtvd3NraQ==?= Subject: Re: ZFS: can't read MOS of pool References: <51ED5B69.8050200@wasikowski.net> <51FBC24B.5030609@gmail.com> <51FBC4DC.4090506@wasikowski.net> <51FBC6F1.9030408@gmail.com> <51FBC8B3.4010304@wasikowski.net> In-Reply-To: <51FBC8B3.4010304@wasikowski.net> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Aug 2013 19:00:46 -0000 on 02/08/2013 17:56 Łukasz Wąsikowski said the following: > This tools doesn't compile on 9.2-BETA2 r253884 There are fixes for this in head that haven't been MFC-ed yet: http://svnweb.freebsd.org/base/head/tools/tools/zfsboottest/zfsboottest.c?view=patch&r1=253067&r2=253066&pathrev=253067 http://svnweb.freebsd.org/base/head/tools/tools/zfsboottest/zfsboottest.sh?view=patch&r1=253068&r2=253067&pathrev=253068 http://svnweb.freebsd.org/base/head/tools/tools/zfsboottest/Makefile?view=patch&r1=253605&r2=253604&pathrev=253605 It would be very interesting to see what zfsboottest says about the pool. Especially interesting it would be to see the error in a debugger. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Aug 3 22:56:56 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EF030411 for ; Sat, 3 Aug 2013 22:56:56 +0000 (UTC) (envelope-from karl@denninger.net) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AE8E920C4 for ; Sat, 3 Aug 2013 22:56:56 +0000 (UTC) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.14.6/8.13.1) with ESMTP id r73McudU091458 for ; Sat, 3 Aug 2013 17:38:56 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (TLS/SSL) [192.168.1.40] by Spamblock-sys (LOCAL/AUTH); Sat Aug 3 17:38:56 2013 Message-ID: <51FD867B.8070803@denninger.net> Date: Sat, 03 Aug 2013 17:38:51 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130215 Thunderbird/17.0.3 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: ZFS option vfs.zfs.cache_flush_disable=1 X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Aug 2013 22:56:57 -0000 Hi folks; I'm trying to run down a rather odd bit of ugly interaction between an ARECA 1680-IX adapter (in JBOD mode) and ZFS. The adapter has both a fair bit of cache memory on it and a BBU unit. If I turn off write-back caching ZFS performance goes straight into the toilet. With it on, however, it appears that cache flushes are being honored and, what's worse, when they come down the pipe they force all further I/O to the adapter to cease until the ENTIRE cache is flushed. This can and occasionally does lead to degenerate cases where very severe performance problems ensue. I have no meaningful way to adjust behavior on the ARECA adapter that appears to matter. I could go to an LSI 2008-based HBA (no cache memory at all, just an adapter that will do SATA-3) and have one in-building, but it appears that while it's quite fast when left "alone" I am using GELI to encrypt the packs. With the ARECA adapter the read-ahead on the adapter keeps the pipeline full and GELI whallops the CPU -- but this box has a lot of CPU to burn, and so performance remains excellent. With the HBA and no cache memory, and thus no read-ahead, that's not true and the performance is impacted fairly-severely. In addition the LSI HBA has a habit of "jumping" disk attachment points around -- that is, a disk may come up as "da1" on one boot and then "da2" on another. I label all my drives but this still can bite me if for some reason I have a failure and have to swap one -- and pull the wrong one. Turning on vfs.zfs.cache_flush_disable=1 results in a _*very*_ significant difference in performance on this adapter and appears to stop the bad interaction between ZFS and the adapter's cache buffer RAM. So this leads to the question -- if I enable the option vfs.zfs.cache_flush_disable=1, /*what am I risking*/ -- if in fact I have a BBU on the adapter? It is not at all clear whether this is a safe option to turn on _*if*_ the adapter has battery-backed memory, or if a crash could leave me with a corrupt and unmountable pool. Thanks in advance! -- Karl Denninger karl@denninger.net /Cuda Systems LLC/ From owner-freebsd-fs@FreeBSD.ORG Sat Aug 3 23:50:56 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E41069B6 for ; Sat, 3 Aug 2013 23:50:56 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ye0-x22c.google.com (mail-ye0-x22c.google.com [IPv6:2607:f8b0:4002:c04::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9FB232202 for ; Sat, 3 Aug 2013 23:50:56 +0000 (UTC) Received: by mail-ye0-f172.google.com with SMTP id m2so760171yen.31 for ; Sat, 03 Aug 2013 16:50:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to:x-mailer; bh=JXTl+8ajAjJpRH11Nbm10ZfVpiK5ohcNQh0HaRxqD+8=; b=J9tNTRdFMof1BEjWnQdE+yysreeWyLvRuJEc53DAeorKDKNhNDUvbl3oaGjlPZ6Yg0 zqnzjD+aQGccfGNe8TF7NMohGegAmThBBePvAv01/VZA++s6MrY4X/MqzFryN4AwQcfv tM9d5y6lqcIlJG+OHbh9ycbALfX0ZYhQcmhWY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to:x-mailer:x-gm-message-state; bh=JXTl+8ajAjJpRH11Nbm10ZfVpiK5ohcNQh0HaRxqD+8=; b=pF6/J3iCqhwVLqx5yoJXVxF+4fsX1lhRcH7KwaJi7wGN+LydJsHihgCqV6snBJir3f IKOiR4JKmPc9hjlIc4ADHOSJLTtkpXdo8MyoxTAyLm+wTnJfXJG0SWmV5ollBVIAHhpf 0Wk6Q71hUn/lsjSV3rhC92972OfUewrP+wPpu6HUSmv0TBF50z3ZaX+NFIssDdZecxSC hdTW/4HCJe4iT+r4kNojvCBVdNYlj3Z8oEL4TlVzhK1PCiF432x2WFPEMp3q/RBxC5dv PkhwXvhnKXojOfdVW/A0bvwFqla1Tl7TvAovCConp45ZfRYQGGYywe9h+wkbYSdD79NN 8J4Q== X-Received: by 10.236.125.97 with SMTP id y61mr7795337yhh.160.1375573855702; Sat, 03 Aug 2013 16:50:55 -0700 (PDT) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPSA id j63sm24038252yhh.17.2013.08.03.16.50.53 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 03 Aug 2013 16:50:54 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_D25F7325-C1EB-4946-B43E-0C1087E85020"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: ZFS option vfs.zfs.cache_flush_disable=1 From: Kevin Day In-Reply-To: <51FD867B.8070803@denninger.net> Date: Sat, 3 Aug 2013 18:50:52 -0500 Message-Id: <9F85D42E-D431-455E-8DFC-CEA7AC53F4BE@dragondata.com> References: <51FD867B.8070803@denninger.net> To: Karl Denninger X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQmKheC3+cdsjEbfFuZsMLpNDyCRClB2hrMxkNXJAGCtW0gXYn1i2uOmkUVuevSZ0mRPmYJA Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Aug 2013 23:50:57 -0000 --Apple-Mail=_D25F7325-C1EB-4946-B43E-0C1087E85020 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Aug 3, 2013, at 5:38 PM, Karl Denninger wrote: > Hi folks; >=20 > I'm trying to run down a rather odd bit of ugly interaction between an > ARECA 1680-IX adapter (in JBOD mode) and ZFS. >=20 > The adapter has both a fair bit of cache memory on it and a BBU unit.=20= > If I turn off write-back caching ZFS performance goes straight into = the > toilet. With it on, however, it appears that cache flushes are being > honored and, what's worse, when they come down the pipe they force all > further I/O to the adapter to cease until the ENTIRE cache is flushed. >=20 > This can and occasionally does lead to degenerate cases where very > severe performance problems ensue. Hey, Karl! We had similar problems with the 3ware 9650SE cards. Cache flush = operations would just kill performance to the entire array, even if ZFS = was only being used on one volume on that card. We've got vfs.zfs.cache_flush_disable=3D1 set on piles of servers, and = so far we've only had one bit of weird ZFS corruption that I'm not even = sure was related to that setting. While this isn't answering your = question, I can say that having that flag enabled isn't causing a ton of = corruption for us even with random hard reboots happening sometimes. I will say though that the LSI 2008 based cards are substantially faster = and less problematic for ZFS than anything else we've used. The "drives = getting reordered" thing seems to only be happening on a few systems = with a specific SATA/SAS backplane that doesn't appear to = deterministically initialize things. gpart labels make that far less of = a problem though, if you can use them. Failing that, it is possible to = use "camcontrol inquiry daXX" to see the serial number of the drives, = which you can then label the fronts of the drives with. Not as pretty, = but being able to have a naked JBOD interface without controllers = treating every drive as it's own 1-drive RAID0 volume is so much easier = that I'll put up with a lot to have it. -- Kevin --Apple-Mail=_D25F7325-C1EB-4946-B43E-0C1087E85020 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPLzCCBN0w ggPFoAMCAQICEHGS++YZX6xNEoV0cTSiGKcwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0Ix GzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwR Q29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0w NDAxMDEwMDAwMDBaFw0yODEyMzEyMzU5NTlaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQx FzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsx ITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJz dC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIx B8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8 om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHG TPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7Nl yP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4IBJzCCASMwHwYDVR0j BBgwFoAUoBEKIz6W8Qfs4q8p74Klf9AwpLQwHQYDVR0OBBYEFImCZ33EnSZwAEu0UEh83j2uBG59 MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDARBgNVHSAECjAIMAYGBFUdIAAwewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5j b21vZG9jYS5jb20vQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwu Y29tb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDARBglghkgBhvhCAQEEBAMCAQYw DQYJKoZIhvcNAQEFBQADggEBAJ2Vyzy4fqUJxB6/C8LHdo45PJTGEKpPDMngq4RdiVTgZTvzbRx8 NywlVF+WIfw3hJGdFdwUT4HPVB1rbEVgxy35l1FM+WbKPKCCjKbI8OLp1Er57D9Wyd12jMOCAU9s APMeGmF0BEcDqcZAV5G8ZSLFJ2dPV9tkWtmNH7qGL/QGrpxp7en0zykX2OBKnxogL5dMUbtGB8SK N04g4wkxaMeexIud6H4RvDJoEJYRmETYKlFgTYjrdDrfQwYyyDlWjDoRUtNBpEMD9O3vMyfbOeAU TibJ2PU54om4k123KSZB6rObroP8d3XK6Mq1/uJlSmM+RMTQw16Hc6mYHK9/FX8wggUaMIIEAqAD AgECAhBtGeqnGU9qMyLmIjJ6qnHeMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQGEwJVUzELMAkG A1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNU IE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTExMDQyODAwMDAw MFoXDTIwMDUzMDEwNDgzOFowgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNo ZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTkwNwYD VQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCShIRbS1eY1F4vi6ThQMijU1hfZmXxMk73nzJ9 VdB4TFW3QpTg+SdxB8XGaaS5MsTxQBqQzCdWYn8XtXFpruUgG+TLY15gyqJB9mrho/+43x9IbWVD jCouK2M4d9+xF6zC2oIC1tQyatRnbyATj1w1+uVUgK/YcQodNwoCUFNslR2pEBS0mZVZEjH/CaLS TNxS297iQAFbSGjdxUq04O0kHzqvcV8H46y/FDuwJXFoPfQP1hdYRhWBPGiLi4MPbXohV+Y0sNsy fuNK4aVScmQmkU6lkg//4LFg/RpvaFGZY40ai6XMQpubfSJj06mg/M6ekN9EGfRcWzW6FvOnm//B AgMBAAGjggFLMIIBRzAfBgNVHSMEGDAWgBSJgmd9xJ0mcABLtFBIfN49rgRufTAdBgNVHQ4EFgQU ehNOAHRbxnhjZCfBL+KgW7x5xXswDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAw EQYDVR0gBAowCDAGBgRVHSAAMFgGA1UdHwRRME8wTaBLoEmGR2h0dHA6Ly9jcmwudXNlcnRydXN0 LmNvbS9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwuY3JsMHQGCCsG AQUFBwEBBGgwZjA9BggrBgEFBQcwAoYxaHR0cDovL2NydC51c2VydHJ1c3QuY29tL1VUTkFkZFRy dXN0Q2xpZW50X0NBLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAN BgkqhkiG9w0BAQUFAAOCAQEAhda+eFdVbTN/RFL+QtUGqAEDgIr7DbL9Sr/2r0FJ9RtaxdKtG3Nu PukmfOZMmMEwKN/L+0I8oSU+CnXW0D05hmbRoZu1TZtvryhsHa/l6nRaqNqxwPF1ei+eupN5yv7i kR5WdLL4jdPgQ3Ib7Y/9YDkgR/uLrzplSDyYPaUlv73vYOBJ5RbI6z9Dg/Dg7g3B080zX5vQvWBq szv++tTJOjwf7Zv/m0kzvkIpOYPuM2kugp1FTahp2oAbHj3SGl18R5mlmwhtEpmG1l1XBxunML5L SUS4kH7K0Xk467Qz+qA6XSZYnmFVGLQh1ZnV4ENAQjC+6qXnlNKw/vN1+X9u5zCCBSwwggQUoAMC AQICEQDbETdDYf7wYKjx8ymk38yAMA0GCSqGSIb3DQEBBQUAMIGTMQswCQYDVQQGEwJHQjEbMBkG A1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01P RE8gQ0EgTGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQg U2VjdXJlIEVtYWlsIENBMB4XDTEzMDYxNjAwMDAwMFoXDTE0MDYxNjIzNTk1OVowJjEkMCIGCSqG SIb3DQEJARYVdG9hc3R5QGRyYWdvbmRhdGEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAvoIO+cLWLe7YYAGV/WdoWC85K8uIgstlYMg/bC8eGbC7AY/nuQXpRV5+xlTXgN7qry/m 6XErlaw1U3rmwlNyjMhJdYaPZclywBKKpYnc3sp0q2A6naeVmOF/t4QDImtfc3sV7SaEkIr7zssK MFTnkOX57g1r3MuiYoHBx1cMaWXYCJ5LDzsynwHGAExYuziRzXcu4sRZ1HBJlQ8hM3yhTTGGOQv1 H1ky13a1RxXC+uoTtYFyrxdBgPUd4eGF1tILHtK9NXnU6lhey90wDa2jmQOJQErgYuYPZriSuBXz QobK7tGcjMBgBQ1U+gxaTyThbXgxfb1MTjDx46hSl8Z35wIDAQABo4IB5TCCAeEwHwYDVR0jBBgw FoAUehNOAHRbxnhjZCfBL+KgW7x5xXswHQYDVR0OBBYEFO9wHp89I1B980w64KR38bmtuHFYMA4G A1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIx AQMFAjARBglghkgBhvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggr BgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwVwYDVR0fBFAwTjBMoEqgSIZG aHR0cDovL2NybC5jb21vZG9jYS5jb20vQ09NT0RPQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1 cmVFbWFpbENBLmNybDCBiAYIKwYBBQUHAQEEfDB6MFIGCCsGAQUFBzAChkZodHRwOi8vY3J0LmNv bW9kb2NhLmNvbS9DT01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3VyZUVtYWlsQ0EuY3J0 MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wIAYDVR0RBBkwF4EVdG9hc3R5 QGRyYWdvbmRhdGEuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQCBaQ8dcaprzzREiMtsc2UtOPSHFiCy dcd5OjE6BN+pkcQozhx3nol9dFKJ+YfGvIxIjHmDGFTOgJgJvjRZ0D1Hw2WJCEtyD+U6yi/cnDFu Ksl039qafzbah6ft2r+GM0QufuFmrBi/bTWU3lGuhL8TKOvsWeLFkyGqtv9AJz2vg7j7dpxutLQY NWnrt7nS2x6p4f1LXu3iwczefyNNFUYwE9zXAT0Uwn48g2iijuf9vekfpqtHBmfSu0tSfd3FS3JC hmFp1fMxnWOnuZ529HFtGeYzr1K8Tp+JEVPjzPCxymVFsZ945Vzj0kc0DT3f9N5Gdw6uybrUwupM NHJJCB9VMYIDrjCCA6oCAQEwgakwgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1h bmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTkw NwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EC EQDbETdDYf7wYKjx8ymk38yAMAkGBSsOAwIaBQCgggHZMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0B BwEwHAYJKoZIhvcNAQkFMQ8XDTEzMDgwMzIzNTA1M1owIwYJKoZIhvcNAQkEMRYEFIjZCjQk6HPt JQZYya5Rhy37dL0jMIG6BgkrBgEEAYI3EAQxgawwgakwgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQI ExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBD QSBMaW1pdGVkMTkwNwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1 cmUgRW1haWwgQ0ECEQDbETdDYf7wYKjx8ymk38yAMIG8BgsqhkiG9w0BCRACCzGBrKCBqTCBkzEL MAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9y ZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0 aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIRANsRN0Nh/vBgqPHzKaTfzIAwDQYJKoZI hvcNAQEBBQAEggEAgCMeNwg1kOFh1u3vjyBJpidAqANwIIiDSurxvCVaBoExhOVECGu2g9HebD8t Vln1lIJkB73efXZcP3NpbFoYYTgYU/2QoAj21CZ7aWQa/g0CRAILeOkS9Ccf+MPABHHfYv7b7F1m 0rlVgaXkgqWy9IRBA46QdkcJ3M953u4VwVIhBB6ltvX029JGh8Z5SbSEKTygamyary46wr/OJhPX O5x4W08b5fsJcthSuj89h93Khjg6XyPNXR7do16B7nWfXT2a9i02tcgtW5K95U3hWcr6kVkJFXnS 7ciUVOJ0Q2bH7fBeOdXHTH83C4frzYk2eEgNiAyXNd4cIdtt3hzoLAAAAAAAAA== --Apple-Mail=_D25F7325-C1EB-4946-B43E-0C1087E85020--