From owner-freebsd-fs@FreeBSD.ORG Sun Aug 3 15:11:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 506B510656CC for ; Sun, 3 Aug 2008 15:11:06 +0000 (UTC) (envelope-from grafan@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.184]) by mx1.freebsd.org (Postfix) with ESMTP id 6E0628FC08 for ; Sun, 3 Aug 2008 15:11:06 +0000 (UTC) (envelope-from grafan@gmail.com) Received: by fk-out-0910.google.com with SMTP id k31so2420727fkk.11 for ; Sun, 03 Aug 2008 08:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=HhtmYsnWULT11C8USHqbJwJ4I8kCUQWrbEIS0mOA838=; b=YORROfYJHWT0gf+vPn3VPST14ytriNzbVmIDFlUwpxgc46bBziKMM7hfvJhR0cgeS+ VQi9tkUA5mmlx88BT5uLseA1VxJRg8bNG2ISGXPk3EFeg3on3HzCbSRvvR6+Ziuoic5N CItfOVdPfDBLRYaCcgmlMuWNDvwN42jtQLrCU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=frXqP1EhQnewCA/unV0+yVbV/Mkjz1SiAtLb7rb7NUa1JpFsU8HDoZLbNGd0VDu7IR dCjL7Fria96wA+WnxYJIKwKpnRJywlMWw2HBtrEKuCD3oBz8HjZds2vwTdg1l7/3t8kU gN/E5sy5AWqLrVGJi0/HFP75mNYat+FlItOfc= Received: by 10.180.213.14 with SMTP id l14mr4734994bkg.55.1217774581936; Sun, 03 Aug 2008 07:43:01 -0700 (PDT) Received: by 10.180.222.6 with HTTP; Sun, 3 Aug 2008 07:43:01 -0700 (PDT) Message-ID: <6eb82e0808030743hc41c68bgd0c5121daba95d42@mail.gmail.com> Date: Sun, 3 Aug 2008 22:43:01 +0800 From: "Rong-en Fan" To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: journaling filesystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Aug 2008 15:11:07 -0000 In NetBSD, they now have metadata journaling support, see http://www.netbsd.org/changes/#wapbl I'm not a fs guru, I just want to know what are the status of BluFFS and UFS journaling support which were mentioned in recent years. Thanks, Rong-En Fan From owner-freebsd-fs@FreeBSD.ORG Sun Aug 3 20:59:24 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 878021065672; Sun, 3 Aug 2008 20:59:24 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 70E828FC08; Sun, 3 Aug 2008 20:59:24 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id BEFDA33C62; Sun, 3 Aug 2008 13:59:43 -0700 (PDT) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 3E33F33C5B; Sun, 3 Aug 2008 13:59:43 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18582.7211.285531.190516@almost.alerce.com> Date: Sun, 3 Aug 2008 13:59:23 -0700 To: Pawel Jakub Dawidek In-Reply-To: <20080727125413.GG1345@garage.freebsd.pl> References: <20080727125413.GG1345@garage.freebsd.pl> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Aug 2008 20:59:24 -0000 Pawel Jakub Dawidek writes: > Hi. > > http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2 > > The patch above contains the most recent ZFS version that could be found > in OpenSolaris as of today. Apart for large amount of new functionality, > I belive there are many stability (and also performance) improvements > compared to the version from the base system. > > Check out OpenSolaris website to find out the differences between base > system version and patch version. > > Please test, test, test. If I get enough positive feedback, I may be > able to squeeze it into 7.1-RELEASE, but this might be hard. > [...] Thanks Pawel! I have a version of -CURRENT from 7-29-08 running on an AMD 3800+ with 2GB of RAM and two Seagate ST3300622AS disks. This machine was previously running zfs filesystems using -STABLE. It uses a root on zfs configuration based on http://yds.coolrat.org/zfsboot.shtml I tested for stablilty by firing up a bunch of xterms on my desktop workstation, ssh'ing into the system and running a bunch of stuff in parallel. The list of stuff settled down to be: two while(1) loops that did "dd if=/dev/random of=FILENAME bs=1M count=1000" (w/ distinct filenames) then removed the file; rsyncing a large directory of mp3's from another local machine; make -j 4 buildworld; and several windows with top, systat -vmstat 1, etc.... With the -STABLE zfs I ended up with the following settings to achieve stability: vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="192M" I started off with no tuning, added the kmem_size{,_max} settings, then added the arc_max setting and tuned it down until it didn't wedge. With the experimental code on -CURRENT I needed the following settings: vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="128M" I don't know if I just got lucky with the larger arc_max under -STABLE or if something's changed. Otherwise the new code seems to be behaving nicely. g. From owner-freebsd-fs@FreeBSD.ORG Mon Aug 4 11:06:54 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8CC31065671 for ; Mon, 4 Aug 2008 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D98C18FC18 for ; Mon, 4 Aug 2008 11:06:54 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m74B6sqA082053 for ; Mon, 4 Aug 2008 11:06:54 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m74B6sEY082049 for freebsd-fs@FreeBSD.org; Mon, 4 Aug 2008 11:06:54 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 4 Aug 2008 11:06:54 GMT Message-Id: <200808041106.m74B6sEY082049@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Aug 2008 11:06:55 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o kern/122888 fs [zfs] zfs hang w/ prefetch on, zil off while running t 7 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o bin/118249 fs mv(1): moving a directory changes its mtime o kern/124621 fs [ext3] Cannot mount ext2fs partition o kern/125536 fs [ext2fs] ext 2 mounts cleanly but fails on commands li 8 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Aug 4 13:25:23 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF7B01065670 for ; Mon, 4 Aug 2008 13:25:23 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from itchy.rabson.org (unknown [IPv6:2002:50b1:e8f2:1::143]) by mx1.freebsd.org (Postfix) with ESMTP id 8997A8FC0C for ; Mon, 4 Aug 2008 13:25:23 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc] (unknown [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc]) by itchy.rabson.org (Postfix) with ESMTP id C7B8A3FA4; Mon, 4 Aug 2008 14:25:22 +0100 (BST) Message-Id: <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> From: Doug Rabson To: Rick Macklem In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v928.1) Date: Mon, 4 Aug 2008 14:25:22 +0100 References: <86myk06e18.fsf@ds4.des.no> X-Mailer: Apple Mail (2.928.1) Cc: freebsd-fs@freebsd.org, =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= Subject: Re: Which GSSAPI library does FreeBSD use? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Aug 2008 13:25:23 -0000 On 29 Jul 2008, at 15:27, Rick Macklem wrote: > > > On Tue, 29 Jul 2008, Dag-Erling Sm=F8rgrav wrote: > >> Rick Macklem writes: >>> Hope this isn't too simplistic for this list, but I need to know =20 >>> which >>> GSSAPI library sources are being used. They don't appear to be =20 >>> either >>> vanilla MIT nor Heimdal. >> >> Homegrown (by Doug Rabson, dfr@) with portions borrowed from Heimdal. >> > Ok, thanks. I was able to work around my problem by statically linking > my gssd against libraries built from vanilla Heimdal sources. It looks > like it inherited the heimdal-0.6 bug, which ignores the lack of the > GSS_C_SEQUENCE_FLAG and checks it even if it wasn't specified. This > breaks the client side of RPCSEC_GSS, since somewhat out-of-order > Sun RPCs, is normal. (RPCSEC_GSS uses a window of recent seq#s to > protect against replay attempts.) > > Should I email Doug or submit a bug report, to see if someone is =20 > willing > to work on fixing this? Try using current - I updated heimdal to 1.1 in current. The GSS-API implementation in 7.x and current is a plugin system which =20= heimdal's krb5 code plugs into as a GSS-API mechanism provider. With =20 heimdal 1.1, it also supports spnego and ntlm as plugins. From owner-freebsd-fs@FreeBSD.ORG Mon Aug 4 15:44:47 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2DFF61065683 for ; Mon, 4 Aug 2008 15:44:47 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [212.17.241.230]) by mx1.freebsd.org (Postfix) with ESMTP id 784068FC08 for ; Mon, 4 Aug 2008 15:44:46 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m74FIIGA080415; Mon, 4 Aug 2008 17:18:29 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m74FID4b080414; Mon, 4 Aug 2008 17:18:13 +0200 (CEST) (envelope-from olli) Date: Mon, 4 Aug 2008 17:18:13 +0200 (CEST) Message-Id: <200808041518.m74FID4b080414@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, pfgshield-freebsd@yahoo.com In-Reply-To: <809288.56058.qm@web32703.mail.mud.yahoo.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Mon, 04 Aug 2008 17:18:32 +0200 (CEST) Cc: Subject: Re: Should we change dirent for 64 bit directory cookies ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, pfgshield-freebsd@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Aug 2008 15:44:47 -0000 Pedro Giffuni wrote: > I've been sort of following the DragonFly list wrt to the changes Matt made for his HAMMER fs. > I don't know if anyone is considering a port: he added a lot of stuff to the base system that will be a pain to port, but he also triggered some bugs in the old BSD code that would be nice to fix on FreeBSD too. > > One of the not-*too*-tough things to consider adopting would be 64 directory cookies: > > Main commit: > http://leaf.dragonflybsd.org/mailarchive/commits/2007-11/msg00151.html > Follow up for the linuxulator: > http://leaf.dragonflybsd.org/mailarchive/commits/2007-11/msg00153.html > > Here is a excerpt of a discussion from the DragonFly Kernel ML (Re: [Tux3] Comparison to Hammer fs design), that pretty much sums up the issues: > [...] > > I'd recommend dropping support for NFSv2. It is not really worth > supporting any more. Does it even support 64 bit inodes? (I don't > remember), or 64 bit file offsets? NFSv2 is garbage. One of the problems with that is that our PXE boot loader requires NFSv2 support, AFAIK. If you drop NFSv2 server support from the kernel, you can't boot PXE clients from it anymore, unless someone adds NFSv3 support to the boot loader. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "[...] one observation we can make here is that Python makes an excellent pseudocoding language, with the wonderful attribute that it can actually be executed." -- Bruce Eckel From owner-freebsd-fs@FreeBSD.ORG Mon Aug 4 16:34:33 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51E1B1065671 for ; Mon, 4 Aug 2008 16:34:33 +0000 (UTC) (envelope-from pfgshield-freebsd@yahoo.com) Received: from web32704.mail.mud.yahoo.com (web32704.mail.mud.yahoo.com [68.142.207.248]) by mx1.freebsd.org (Postfix) with SMTP id 02B408FC19 for ; Mon, 4 Aug 2008 16:34:32 +0000 (UTC) (envelope-from pfgshield-freebsd@yahoo.com) Received: (qmail 32901 invoked by uid 60001); 4 Aug 2008 16:34:32 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=CQUuMy5WM7bmxnVohNIDl5gjsQNZLynVKWE/oYmaCSP3fgcEjFpNujCAiyMz7hcNr8PgqbxomMPIYTaA04WLVpNkYAdF4ES1y5hwMFeMCOZF7nUkvzbzOApuAPg504pwRrqdT/OCTdyOen/MuscGMcaRo0JYEDHUKGcECGnM8Ok=; Received: from [190.156.48.247] by web32704.mail.mud.yahoo.com via HTTP; Mon, 04 Aug 2008 09:34:30 PDT X-Mailer: YahooMailWebService/0.7.218 Date: Mon, 4 Aug 2008 09:34:30 -0700 (PDT) From: Pedro Giffuni To: freebsd-fs@FreeBSD.ORG In-Reply-To: <200808041518.m74FID4b080414@lurza.secnetix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <370602.32048.qm@web32704.mail.mud.yahoo.com> X-Mailman-Approved-At: Mon, 04 Aug 2008 17:17:20 +0000 Cc: Subject: Re: Should we change dirent for 64 bit directory cookies ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pfgshield-freebsd@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Aug 2008 16:34:33 -0000 --- Lun 4/8/08, Oliver Fromme ha scritto: ... One of the problems with that is that our PXE boot loader requires NFSv2 support, AFAIK. If you drop NFSv2 server support from the kernel, you can't boot PXE clients from it anymore, unless someone adds NFSv3 support to the boot loader. Best regards Oliver ____ I see, thanks for the explanation... yes, it sounds like this will have to = wait a while then. I am looking at the Intel PXE spec and there is no menti= on about NFS. I will investigate a bit more but I guess there is a new wish= for the PXE guys ;-). cheers, Pedro.=0A=0A=0A Posta, news, sport, oroscopo: tutto in una sola pa= gina. =0ACrea l'home page che piace a te!=0Awww.yahoo.it/latuapagina From owner-freebsd-fs@FreeBSD.ORG Mon Aug 4 20:53:46 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CE621065679 for ; Mon, 4 Aug 2008 20:53:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from moe.cs.uoguelph.ca (moe.cs.uoguelph.ca [131.104.94.198]) by mx1.freebsd.org (Postfix) with ESMTP id C93C98FC27 for ; Mon, 4 Aug 2008 20:53:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by moe.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m74KriEN028491 for ; Mon, 4 Aug 2008 16:53:44 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m74L4wP04895 for ; Mon, 4 Aug 2008 17:04:58 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Mon, 4 Aug 2008 17:04:58 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: freebsd-fs@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.198 Subject: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Aug 2008 20:53:46 -0000 There's a place in my nfsv4 client where I need to vfs_hash_get() when another blocked thread may be holding a lock on the vnode. (It's during a recovery case where the other threads are blocked, so there isn't a race problem, as far as I understand it.) For FreeBSD7, all I did was call vfs_hash_get() with flags == 0 and it gave me what I wanted (the vnode for the file handle with a reference count, but no lock). For FreeBSD-CURRENT/8, this no longer works, because vfs_hash_get() calls vget(), which calls _vn_lock() and _vn_lock() now complains if the lock type field of "flags" is 0. I came up with a really ugly workaround, by setting the flags arg. to LK_EXCLOTHER for vfs_hash_get() and then providing my own VOP_LOCK1() which just VI_UNLOCK()s and returns 0 for this case (doing the same as vop_stdlock() for other cases). Yuck!! Is it possible to re-enable the case of _vn_lock() getting a locktype field == 0 (or defining one that says "just return 0 unless VI_DOOMED"), so I don't need the dirty hack? Have a good week, rick From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 08:32:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FD961065677 for ; Tue, 5 Aug 2008 08:32:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id F27638FC1E for ; Tue, 5 Aug 2008 08:32:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m758WUco066705 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 5 Aug 2008 11:32:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m758WUBn097207; Tue, 5 Aug 2008 11:32:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m758WTMj097200; Tue, 5 Aug 2008 11:32:29 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 5 Aug 2008 11:32:29 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qnuS/wU1MXEWeKjo" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 08:32:35 -0000 --qnuS/wU1MXEWeKjo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Aug 04, 2008 at 05:04:58PM -0400, Rick Macklem wrote: > There's a place in my nfsv4 client where I need to vfs_hash_get() when > another blocked thread may be holding a lock on the vnode. (It's during > a recovery case where the other threads are blocked, so there isn't a > race problem, as far as I understand it.) >=20 > For FreeBSD7, all I did was call vfs_hash_get() with flags =3D=3D 0 and it > gave me what I wanted (the vnode for the file handle with a reference > count, but no lock). >=20 > For FreeBSD-CURRENT/8, this no longer works, because vfs_hash_get() calls > vget(), which calls _vn_lock() and _vn_lock() now complains if the lock > type field of "flags" is 0. I came up with a really ugly workaround, by > setting the flags arg. to LK_EXCLOTHER for vfs_hash_get() and then > providing my own VOP_LOCK1() which just VI_UNLOCK()s and returns 0 for > this case (doing the same as vop_stdlock() for other cases). Yuck!! >=20 > Is it possible to re-enable the case of _vn_lock() getting a locktype > field =3D=3D 0 (or defining one that says "just return 0 unless VI_DOOMED= "), > so I don't need the dirty hack? I do not quite understand what you really need there. The non-locked vnode may be reclaimed at any moment, so the check for !VI_DOOMED returns no useful information. --qnuS/wU1MXEWeKjo Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiYEB0ACgkQC3+MBN1Mb4gt+gCfSgADScqDKFM/6MZ+P3odCjXZ l6QAoL9x78alOWSpFGCzB+CT70FB2JAM =x5zz -----END PGP SIGNATURE----- --qnuS/wU1MXEWeKjo-- From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 14:30:50 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4888106572F for ; Tue, 5 Aug 2008 14:30:50 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [212.17.241.230]) by mx1.freebsd.org (Postfix) with ESMTP id D10218FC17 for ; Tue, 5 Aug 2008 14:30:49 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m75EUR3K037118; Tue, 5 Aug 2008 16:30:32 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m75EUQkK037117; Tue, 5 Aug 2008 16:30:26 +0200 (CEST) (envelope-from olli) Date: Tue, 5 Aug 2008 16:30:26 +0200 (CEST) Message-Id: <200808051430.m75EUQkK037117@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG, pfgshield-freebsd@yahoo.com In-Reply-To: <370602.32048.qm@web32704.mail.mud.yahoo.com> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Tue, 05 Aug 2008 16:30:35 +0200 (CEST) Cc: Subject: Re: Should we change dirent for 64 bit directory cookies ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG, pfgshield-freebsd@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 14:30:50 -0000 Pedro Giffuni wrote: > Oliver Fromme wrote: > > One of the problems with that is that our PXE boot loader > > requires NFSv2 support, AFAIK. If you drop NFSv2 server > > support from the kernel, you can't boot PXE clients from > > it anymore, unless someone adds NFSv3 support to the boot > > loader. > > I see, thanks for the explanation... yes, it sounds like this will > have to wait a while then. I am looking at the Intel PXE spec and > there is no mention about NFS. I will investigate a bit more but I > guess there is a new wish for the PXE guys ;-). The PXE spec is rather low-level, i.e. bascially it allows for handling raw packets only. It does not implement a real TCP/IP stack. So if you want to use any higher-level protocols, you have to implement them yourself on top of the PXE interface. Note that our bootlaoder uses libstand for the actual protocol implementations that are needed (BOOTP, NFS, RPC, TFTP). You'll find the sources in /usr/src/lib/libstand. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "If Java had true garbage collection, most programs would delete themselves upon execution." -- Robert Sewell From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 14:57:46 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B76B71065672 for ; Tue, 5 Aug 2008 14:57:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from dargo.cs.uoguelph.ca (dargo.cs.uoguelph.ca [131.104.94.197]) by mx1.freebsd.org (Postfix) with ESMTP id 7400C8FC23 for ; Tue, 5 Aug 2008 14:57:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by dargo.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m75Evjog025267; Tue, 5 Aug 2008 10:57:45 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m75F90l11124; Tue, 5 Aug 2008 11:09:00 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 5 Aug 2008 11:09:00 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Kostik Belousov In-Reply-To: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> Message-ID: References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.197 Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 14:57:46 -0000 On Tue, 5 Aug 2008, Kostik Belousov wrote: > On Mon, Aug 04, 2008 at 05:04:58PM -0400, Rick Macklem wrote: >> There's a place in my nfsv4 client where I need to vfs_hash_get() when >> another blocked thread may be holding a lock on the vnode. (It's during >> a recovery case where the other threads are blocked, so there isn't a >> race problem, as far as I understand it.) >> >> For FreeBSD7, all I did was call vfs_hash_get() with flags == 0 and it >> gave me what I wanted (the vnode for the file handle with a reference >> count, but no lock). >> >> For FreeBSD-CURRENT/8, this no longer works, because vfs_hash_get() calls >> vget(), which calls _vn_lock() and _vn_lock() now complains if the lock >> type field of "flags" is 0. I came up with a really ugly workaround, by >> setting the flags arg. to LK_EXCLOTHER for vfs_hash_get() and then >> providing my own VOP_LOCK1() which just VI_UNLOCK()s and returns 0 for >> this case (doing the same as vop_stdlock() for other cases). Yuck!! >> >> Is it possible to re-enable the case of _vn_lock() getting a locktype >> field == 0 (or defining one that says "just return 0 unless VI_DOOMED"), >> so I don't need the dirty hack? > > I do not quite understand what you really need there. The non-locked > vnode may be reclaimed at any moment, so the check for !VI_DOOMED > returns no useful information. > I need a referenced vnode (v_usecount incremented, which I thought would avoid it being recycled) when another blocked thread in the kernel has the vnode locked. (This thread is doing recovery while the other blocked thread is waiting for that recovery to complete.) For FreeBSD7, I call vfs_hash_get(mntp, hash, 0, td, &nvp, mycmpfh, nfhp) for this case. It: - calls vget() with (0 | LK_INTERLOCK) for flags - it calls vn_lock() with the flags as above - _vn_lock() simply checks for VI_DOOMED and then, otherwise, returns 0 because (flags & LK_TYPE_MASK) == 0 - vget then calls v_upgrade_usecount(), incrementing the use count (so it won't be recycled, as I understand it) --> and I get the vnode with an incremented usecount that I need. When this thread is done with it, it simply vrele(vp)'s it. For FreeBSD-CURRENT, this panics because there is now a check at the beginning of _vn_lock() requiring an non-zero LK_TYPE_MASK field. (I worked around it by using LK_EXCLOTHER instead of 0 as the argument to vfs_hash_get() and then implemented by own nfs_lock1() VOP, that handles the bogus case of the LK_TYPE_MASK being set to LK_EXCLOTHER by just returning 0 after VI_UNLOCK(vp) if LK_INTERLOCK is set. For all other cases, it calls _lockmgr_args() just like vop_stdlock().) Did this help or just muddy the waters? rick From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 15:32:29 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06EAE10656E4 for ; Tue, 5 Aug 2008 15:32:29 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id A1D088FC1C for ; Tue, 5 Aug 2008 15:32:26 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.2/8.14.2) with ESMTP id m75FX5Pk062580; Tue, 5 Aug 2008 10:33:05 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.2/8.14.2/Submit) id m75FX5Tg062579; Tue, 5 Aug 2008 10:33:05 -0500 (CDT) (envelope-from brooks) Date: Tue, 5 Aug 2008 10:33:05 -0500 From: Brooks Davis To: freebsd-fs@freebsd.org, pfgshield-freebsd@yahoo.com Message-ID: <20080805153305.GB55735@lor.one-eyed-alien.net> References: <370602.32048.qm@web32704.mail.mud.yahoo.com> <200808051430.m75EUQkK037117@lurza.secnetix.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/NkBOFFp2J2Af1nK" Content-Disposition: inline In-Reply-To: <200808051430.m75EUQkK037117@lurza.secnetix.de> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Subject: Re: Should we change dirent for 64 bit directory cookies ? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 15:32:29 -0000 --/NkBOFFp2J2Af1nK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 05, 2008 at 04:30:26PM +0200, Oliver Fromme wrote: > Pedro Giffuni wrote: > > Oliver Fromme wrote: > > > One of the problems with that is that our PXE boot loader > > > requires NFSv2 support, AFAIK. If you drop NFSv2 server > > > support from the kernel, you can't boot PXE clients from > > > it anymore, unless someone adds NFSv3 support to the boot > > > loader. > >=20 > > I see, thanks for the explanation... yes, it sounds like this will > > have to wait a while then. I am looking at the Intel PXE spec and > > there is no mention about NFS. I will investigate a bit more but I > > guess there is a new wish for the PXE guys ;-). >=20 > The PXE spec is rather low-level, i.e. bascially it allows > for handling raw packets only. It does not implement a > real TCP/IP stack. So if you want to use any higher-level > protocols, you have to implement them yourself on top of > the PXE interface. >=20 > Note that our bootlaoder uses libstand for the actual > protocol implementations that are needed (BOOTP, NFS, RPC, > TFTP). You'll find the sources in /usr/src/lib/libstand. A summer of code project last year implemented TCP and HTTP support. -- Brooks --/NkBOFFp2J2Af1nK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iD8DBQFImHKxXY6L6fI4GtQRAgJHAKDZulAEa5vyykEZ+NcDBXeAeXFwGwCgh6+f q9eH+ADikrcS8gOdQKiA4eA= =N+x5 -----END PGP SIGNATURE----- --/NkBOFFp2J2Af1nK-- From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 15:32:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24847106569A for ; Tue, 5 Aug 2008 15:32:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id AF9BA8FC12 for ; Tue, 5 Aug 2008 15:32:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m75FWL6j086297 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 5 Aug 2008 18:32:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m75FWLIt014149; Tue, 5 Aug 2008 18:32:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m75FWLxr014148; Tue, 5 Aug 2008 18:32:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 5 Aug 2008 18:32:21 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20080805153221.GG97161@deviant.kiev.zoral.com.ua> References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vt4ed5MNefdcgji4" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 15:32:35 -0000 --vt4ed5MNefdcgji4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 05, 2008 at 11:09:00AM -0400, Rick Macklem wrote: >=20 >=20 > On Tue, 5 Aug 2008, Kostik Belousov wrote: >=20 > >On Mon, Aug 04, 2008 at 05:04:58PM -0400, Rick Macklem wrote: > >>There's a place in my nfsv4 client where I need to vfs_hash_get() when > >>another blocked thread may be holding a lock on the vnode. (It's during > >>a recovery case where the other threads are blocked, so there isn't a > >>race problem, as far as I understand it.) > >> > >>For FreeBSD7, all I did was call vfs_hash_get() with flags =3D=3D 0 and= it > >>gave me what I wanted (the vnode for the file handle with a reference > >>count, but no lock). > >> > >>For FreeBSD-CURRENT/8, this no longer works, because vfs_hash_get() cal= ls > >>vget(), which calls _vn_lock() and _vn_lock() now complains if the lock > >>type field of "flags" is 0. I came up with a really ugly workaround, by > >>setting the flags arg. to LK_EXCLOTHER for vfs_hash_get() and then > >>providing my own VOP_LOCK1() which just VI_UNLOCK()s and returns 0 for > >>this case (doing the same as vop_stdlock() for other cases). Yuck!! > >> > >>Is it possible to re-enable the case of _vn_lock() getting a locktype > >>field =3D=3D 0 (or defining one that says "just return 0 unless VI_DOOM= ED"), > >>so I don't need the dirty hack? > > > >I do not quite understand what you really need there. The non-locked > >vnode may be reclaimed at any moment, so the check for !VI_DOOMED > >returns no useful information. > > > I need a referenced vnode (v_usecount incremented, which I thought would > avoid it being recycled) when another blocked thread in the kernel has No, this is a wrong assumption. Use count does not prevent the vnode from being reclaimed. Unless you held the vnode lock, it may be reclaimed. To set the VI_DOOMED flag, both exclusive vnode lock and vnode interlock must be held. > the vnode locked. (This thread is doing recovery while the other blocked > thread is waiting for that recovery to complete.) If you can guarantee that the other thread does not relinquish the vnode lock while curthread operates on the vnode, you may use vref() and direct check on VI_DOOMED. I shall admit that this is quite perversive and fragile. >=20 > For FreeBSD7, I call vfs_hash_get(mntp, hash, 0, td, &nvp, mycmpfh, nfhp) > for this case. It: > - calls vget() with (0 | LK_INTERLOCK) for flags > - it calls vn_lock() with the flags as above > - _vn_lock() simply checks for VI_DOOMED and then, otherwise, > returns 0 because (flags & LK_TYPE_MASK) =3D=3D 0 > - vget then calls v_upgrade_usecount(), incrementing the use count > (so it won't be recycled, as I understand it) > --> and I get the vnode with an incremented usecount that I need. When > this thread is done with it, it simply vrele(vp)'s it. >=20 > For FreeBSD-CURRENT, this panics because there is now a check at the > beginning of _vn_lock() requiring an non-zero LK_TYPE_MASK field. > (I worked around it by using LK_EXCLOTHER instead of 0 as the argument to > vfs_hash_get() and then implemented by own nfs_lock1() VOP, that handles > the bogus case of the LK_TYPE_MASK being set to LK_EXCLOTHER by just > returning 0 after VI_UNLOCK(vp) if LK_INTERLOCK is set. For all other > cases, it calls _lockmgr_args() just like vop_stdlock().) >=20 > Did this help or just muddy the waters? rick --vt4ed5MNefdcgji4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiYcoQACgkQC3+MBN1Mb4h/YgCfYezNLwy1TYRk9F9Q65+ZkfUq u7cAnA7Gz1log01tAtCfserHWUqQkHqU =tQt9 -----END PGP SIGNATURE----- --vt4ed5MNefdcgji4-- From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 16:43:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D9811065674 for ; Tue, 5 Aug 2008 16:43:11 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from phoenix.cs.uoguelph.ca (phoenix.cs.uoguelph.ca [131.104.94.216]) by mx1.freebsd.org (Postfix) with ESMTP id 184558FC1C for ; Tue, 5 Aug 2008 16:43:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by phoenix.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m75GhAFH009687; Tue, 5 Aug 2008 12:43:10 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m75GsQD28307; Tue, 5 Aug 2008 12:54:26 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 5 Aug 2008 12:54:26 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Kostik Belousov In-Reply-To: <20080805153221.GG97161@deviant.kiev.zoral.com.ua> Message-ID: References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> <20080805153221.GG97161@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.216 Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 16:43:11 -0000 On Tue, 5 Aug 2008, Kostik Belousov wrote: [stuff snipped] >>> >> I need a referenced vnode (v_usecount incremented, which I thought would >> avoid it being recycled) when another blocked thread in the kernel has > No, this is a wrong assumption. Use count does not prevent the vnode > from being reclaimed. > What does v_usecount mean then, if it doesn't say "I have it in use, so you can't recycle it until I vrele() it"? I suppose I can test for the lock and grab it, if no other thread already has it locked. > Unless you held the vnode lock, it may be reclaimed. To set the > VI_DOOMED flag, both exclusive vnode lock and vnode interlock must be > held. > I don't care about VI_DOOMED nor want to set it. It is just what vget() checked for the case of LK_TYPE_MASK == 0 under FreeBSD7. > If you can guarantee that the other thread does not relinquish the vnode > lock while curthread operates on the vnode, you may use vref() and > direct check on VI_DOOMED. I shall admit that this is quite perversive > and fragile. > I'll have to think about it but, yes, I think I can guarantee that if another thread holds the vnode lock then it is blocked waiting for this thread to complete recovery. (The only other way to do this recovery is without the vnode and that means I have to do a lot of coding. I'm pretty sure holding a v_usecount works for OpenBSD and Mac OS X. I've done quite a bit of testing on both and not had a problem.) rick From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 15:46:04 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D61F1106570A; Tue, 5 Aug 2008 15:46:04 +0000 (UTC) (envelope-from nork@FreeBSD.org) Received: from sakura.ninth-nine.com (unknown [IPv6:2001:2f0:104:80a0:230:48ff:fe41:2455]) by mx1.freebsd.org (Postfix) with ESMTP id 437628FC17; Tue, 5 Aug 2008 15:46:04 +0000 (UTC) (envelope-from nork@FreeBSD.org) Received: from nadesico.ninth-nine.com (nadesico.ninth-nine.com [219.127.74.122]) by sakura.ninth-nine.com (8.14.1/8.14.1/NinthNine) with SMTP id m75FjvYI054299; Wed, 6 Aug 2008 00:45:57 +0900 (JST) (envelope-from nork@FreeBSD.org) Date: Wed, 6 Aug 2008 00:45:57 +0900 From: Norikatsu Shigemura To: Pawel Jakub Dawidek Message-Id: <20080806004557.6e538e5c.nork@FreeBSD.org> In-Reply-To: <20080731013229.9d342ee5.nork@FreeBSD.org> References: <20080727125413.GG1345@garage.freebsd.pl> <488F0C71.9010902@moneybookers.com> <20080729125551.GA70379@eos.sc1.parodius.com> <1217338852.10413.1.camel@dingo-laptop> <488F2078.708@psg.com> <1217347882.10413.5.camel@dingo-laptop> <20080729211137.GA52154@nobby.studby.ntnu.no> <20080731013229.9d342ee5.nork@FreeBSD.org> X-Mailer: Sylpheed 2.5.0 (GTK+ 2.12.11; i386-portbld-freebsd8.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (sakura.ninth-nine.com [219.127.74.121]); Wed, 06 Aug 2008 00:45:58 +0900 (JST) X-Mailman-Approved-At: Tue, 05 Aug 2008 16:47:47 +0000 Cc: Randy Bush , Ulf Lilleengen , Jeremy Chadwick , Stefan, Norikatsu Shigemura , Lambrev , freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org, OutBackdingo Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 15:46:05 -0000 On Thu, 31 Jul 2008 01:32:29 +0900 Norikatsu Shigemura wrote: > > However, this feature is a bit undocumented yet, and it didn't work correctly > > for me. But you can always test it out. > I'm using zfsboot on my note PC, and not using UFS. I know many > problems about it:-). > 1. zpool configuration is too limited, only single and mirror > usable. If you want to zfsboot, you can't use RAIDZ, striping > and cache(zpool add ... cache ...):-(. I missed. zfsboot is disregarded zpool cache rather than supports it. > SEE ALSO: > http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004895.html > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/125878 I found some zfsboot issues, please apply following patches: 1. zfsboot2 (boot2) doesn't %d (printf), so change %d to %u. 2. chase new zpool versioning as SPA_VERSION. Obtained from: sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- sys/boot/zfs/zfsimpl.c.orig 2008-07-28 01:54:49.194419000 +0900 +++ sys/boot/zfs/zfsimpl.c 2008-08-05 23:48:12.035247220 +0900 @@ -656,8 +656,8 @@ return (EIO); } - if (val != ZFS_VERSION) { - printf("ZFS: unsupported ZFS version %d\n", (int) val); + if (val > SPA_VERSION) { + printf("ZFS: unsupported ZFS version %u (should be %u)\n", (int) val, (int) SPA_VERSION); return (EIO); } --- sys/cddl/boot/zfs/zfsimpl.h.orig 2008-07-28 01:54:49.296418000 +0900 +++ sys/cddl/boot/zfs/zfsimpl.h 2008-08-06 00:07:41.871760182 +0900 @@ -448,19 +448,24 @@ /* * On-disk version number. */ -#define ZFS_VERSION_1 1ULL -#define ZFS_VERSION_2 2ULL -#define ZFS_VERSION_3 3ULL -#define ZFS_VERSION_4 4ULL -#define ZFS_VERSION_5 5ULL -#define ZFS_VERSION_6 6ULL +#define SPA_VERSION_1 1ULL +#define SPA_VERSION_2 2ULL +#define SPA_VERSION_3 3ULL +#define SPA_VERSION_4 4ULL +#define SPA_VERSION_5 5ULL +#define SPA_VERSION_6 6ULL +#define SPA_VERSION_7 7ULL +#define SPA_VERSION_8 8ULL +#define SPA_VERSION_9 9ULL +#define SPA_VERSION_10 10ULL +#define SPA_VERSION_11 11ULL /* * When bumping up ZFS_VERSION, make sure GRUB ZFS understand the on-disk * format change. Go to usr/src/grub/grub-0.95/stage2/{zfs-include/, fsys_zfs*}, * and do the appropriate changes. */ -#define ZFS_VERSION ZFS_VERSION_6 -#define ZFS_VERSION_STRING "6" +#define SPA_VERSION SPA_VERSION_11 +#define SPA_VERSION_STRING "11" /* * Symbolic names for the changes that caused a ZFS_VERSION switch. @@ -473,16 +478,26 @@ * last synced uberblock. Checking the in-flight version can * be dangerous in some cases. */ -#define ZFS_VERSION_INITIAL ZFS_VERSION_1 -#define ZFS_VERSION_DITTO_BLOCKS ZFS_VERSION_2 -#define ZFS_VERSION_SPARES ZFS_VERSION_3 -#define ZFS_VERSION_RAID6 ZFS_VERSION_3 -#define ZFS_VERSION_BPLIST_ACCOUNT ZFS_VERSION_3 -#define ZFS_VERSION_RAIDZ_DEFLATE ZFS_VERSION_3 -#define ZFS_VERSION_DNODE_BYTES ZFS_VERSION_3 -#define ZFS_VERSION_ZPOOL_HISTORY ZFS_VERSION_4 -#define ZFS_VERSION_GZIP_COMPRESSION ZFS_VERSION_5 -#define ZFS_VERSION_BOOTFS ZFS_VERSION_6 +#define SPA_VERSION_INITIAL SPA_VERSION_1 +#define SPA_VERSION_DITTO_BLOCKS SPA_VERSION_2 +#define SPA_VERSION_SPARES SPA_VERSION_3 +#define SPA_VERSION_RAID6 SPA_VERSION_3 +#define SPA_VERSION_BPLIST_ACCOUNT SPA_VERSION_3 +#define SPA_VERSION_RAIDZ_DEFLATE SPA_VERSION_3 +#define SPA_VERSION_DNODE_BYTES SPA_VERSION_3 +#define SPA_VERSION_ZPOOL_HISTORY SPA_VERSION_4 +#define SPA_VERSION_GZIP_COMPRESSION SPA_VERSION_5 +#define SPA_VERSION_BOOTFS SPA_VERSION_6 +#define SPA_VERSION_SLOGS SPA_VERSION_7 +#define SPA_VERSION_DELEGATED_PERMS SPA_VERSION_8 +#define SPA_VERSION_FUID SPA_VERSION_9 +#define SPA_VERSION_REFRESERVATION SPA_VERSION_9 +#define SPA_VERSION_REFQUOTA SPA_VERSION_9 +#define SPA_VERSION_UNIQUE_ACCURATE SPA_VERSION_9 +#define SPA_VERSION_L2CACHE SPA_VERSION_10 +#define SPA_VERSION_NEXT_CLONES SPA_VERSION_11 +#define SPA_VERSION_ORIGIN SPA_VERSION_11 +#define SPA_VERSION_DSL_SCRUB SPA_VERSION_11 /* * The following are configuration names used in the nvlist describing a pool's --- sys/cddl/boot/zfs/zfssubr.c.orig 2008-07-28 01:54:49.297420000 +0900 +++ sys/cddl/boot/zfs/zfssubr.c 2008-08-06 00:19:29.665026084 +0900 @@ -162,7 +162,7 @@ /* ASSERT((uint_t)cpfunc < ZIO_COMPRESS_FUNCTIONS); */ if (!ci->ci_decompress) { - printf("ZFS: unsupported compression algorithm %d\n", cpfunc); + printf("ZFS: unsupported compression algorithm %u\n", cpfunc); return (EIO); } - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 16:51:19 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E48CF106566B for ; Tue, 5 Aug 2008 16:51:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 7BEBD8FC1E for ; Tue, 5 Aug 2008 16:51:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m75GpFAl089731 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 5 Aug 2008 19:51:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m75GpFUA071454; Tue, 5 Aug 2008 19:51:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m75GpFsP071447; Tue, 5 Aug 2008 19:51:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 5 Aug 2008 19:51:14 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20080805165114.GH97161@deviant.kiev.zoral.com.ua> References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> <20080805153221.GG97161@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aRuUY7KzmetiZjaI" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 16:51:20 -0000 --aRuUY7KzmetiZjaI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 05, 2008 at 12:54:26PM -0400, Rick Macklem wrote: >=20 >=20 > On Tue, 5 Aug 2008, Kostik Belousov wrote: >=20 > [stuff snipped] > >>> > >>I need a referenced vnode (v_usecount incremented, which I thought would > >>avoid it being recycled) when another blocked thread in the kernel has > >No, this is a wrong assumption. Use count does not prevent the vnode > >from being reclaimed. > > > What does v_usecount mean then, if it doesn't say "I have it in use, so > you can't recycle it until I vrele() it"? It means that the vnode memory will not be freed until vrele(). But the VOP_RECLAIM may be called any time, and it requires exclusive lock. After vnode is reclaimed, it is reassigned to the deadfs. In particular, VOP_RECLAIM implementation must clear v_data. For the reclaimed vnode you still hold a reference to, you can reliably obtain the vnode lock. >=20 > I suppose I can test for the lock and grab it, if no other thread already > has it locked. >=20 > >Unless you held the vnode lock, it may be reclaimed. To set the > >VI_DOOMED flag, both exclusive vnode lock and vnode interlock must be > >held. > > > I don't care about VI_DOOMED nor want to set it. It is just what vget() > checked for the case of LK_TYPE_MASK =3D=3D 0 under FreeBSD7. >=20 > >If you can guarantee that the other thread does not relinquish the vnode > >lock while curthread operates on the vnode, you may use vref() and > >direct check on VI_DOOMED. I shall admit that this is quite perversive > >and fragile. > > > I'll have to think about it but, yes, I think I can guarantee that if > another thread holds the vnode lock then it is blocked waiting for this > thread to complete recovery. (The only other way to do this recovery is > without the vnode and that means I have to do a lot of coding. I'm > pretty sure holding a v_usecount works for OpenBSD and Mac OS X. I've > done quite a bit of testing on both and not had a problem.) I do not know about these systems, esp. whether and how they implement a forced unmount. --aRuUY7KzmetiZjaI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiYhQIACgkQC3+MBN1Mb4hX1wCgjLV/Tr/QaTG+1hiMbVzDifOA 0bYAn0MDZtboyjCEBXxBU4QTSyJIQU8t =W7MY -----END PGP SIGNATURE----- --aRuUY7KzmetiZjaI-- From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 17:40:26 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F9E01065681 for ; Tue, 5 Aug 2008 17:40:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from moe.cs.uoguelph.ca (moe.cs.uoguelph.ca [131.104.94.198]) by mx1.freebsd.org (Postfix) with ESMTP id 5A5A48FC1E for ; Tue, 5 Aug 2008 17:40:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by moe.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m75HeOcH017156; Tue, 5 Aug 2008 13:40:24 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m75Hpf706810; Tue, 5 Aug 2008 13:51:41 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 5 Aug 2008 13:51:40 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Kostik Belousov In-Reply-To: <20080805165114.GH97161@deviant.kiev.zoral.com.ua> Message-ID: References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> <20080805153221.GG97161@deviant.kiev.zoral.com.ua> <20080805165114.GH97161@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.198 Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 17:40:26 -0000 On Tue, 5 Aug 2008, Kostik Belousov wrote: [stuff snipped] >> What does v_usecount mean then, if it doesn't say "I have it in use, so >> you can't recycle it until I vrele() it"? > It means that the vnode memory will not be freed until vrele(). > > But the VOP_RECLAIM may be called any time, and it requires exclusive lock. > After vnode is reclaimed, it is reassigned to the deadfs. In particular, > VOP_RECLAIM implementation must clear v_data. > > For the reclaimed vnode you still hold a reference to, you can reliably > obtain the vnode lock. > [stuff snipped] > I do not know about these systems, esp. whether and how they implement > a forced unmount. > Ok, I just spent a few minutes snooping around in vfs_subr.c and I think I see the problem. vget() has called vholdl() and then v_upgrade_usecount(), which has incremented the usecount and taken the vnode off the free list. This appears to prevent vgonel() from being called on it for most cases, but there is still the case in vflush() where the FORCECLOSE flag is set. But, it seems that it is my nfs_unmount() that calls this, so I can just delay the FORCECLOSE for this weird case. In fact, it looks like vgonel() would call VOP_CLOSE() because v_usecount is still non-zero (active) and that would block during the recovery in my code, anyhow. Thanks for clarifying it, rick From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 19:43:47 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7D321065676 for ; Tue, 5 Aug 2008 19:43:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 367DB8FC17 for ; Tue, 5 Aug 2008 19:43:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m75JhfWc096131 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 5 Aug 2008 22:43:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m75JhfTo070127; Tue, 5 Aug 2008 22:43:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m75Jhfdj070126; Tue, 5 Aug 2008 22:43:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 5 Aug 2008 22:43:41 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20080805194341.GI97161@deviant.kiev.zoral.com.ua> References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> <20080805153221.GG97161@deviant.kiev.zoral.com.ua> <20080805165114.GH97161@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1BXV+/FYeXhtv2WT" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 19:43:47 -0000 --1BXV+/FYeXhtv2WT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 05, 2008 at 01:51:40PM -0400, Rick Macklem wrote: >=20 >=20 > On Tue, 5 Aug 2008, Kostik Belousov wrote: >=20 > [stuff snipped] > >>What does v_usecount mean then, if it doesn't say "I have it in use, so > >>you can't recycle it until I vrele() it"? > >It means that the vnode memory will not be freed until vrele(). > > > >But the VOP_RECLAIM may be called any time, and it requires exclusive lo= ck. > >After vnode is reclaimed, it is reassigned to the deadfs. In particular, > >VOP_RECLAIM implementation must clear v_data. > > > >For the reclaimed vnode you still hold a reference to, you can reliably > >obtain the vnode lock. > > > [stuff snipped] > >I do not know about these systems, esp. whether and how they implement > >a forced unmount. > > > Ok, I just spent a few minutes snooping around in vfs_subr.c and I think > I see the problem. vget() has called vholdl() and then=20 > v_upgrade_usecount(), which has incremented the usecount and taken the > vnode off the free list. This appears to prevent vgonel() from being > called on it for most cases, but there is still the case in vflush() > where the FORCECLOSE flag is set. Yes, exactly. >=20 > But, it seems that it is my nfs_unmount() that calls this, so I can just > delay the FORCECLOSE for this weird case. >=20 > In fact, it looks like vgonel() would call VOP_CLOSE() because v_usecount > is still non-zero (active) and that would block during the recovery in my > code, anyhow. But, what guarantees that the vnode would not be reclaimed before/under your vref() it ? For instance, what if the vnode is locked due to reclaim being in progress ? --1BXV+/FYeXhtv2WT Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEUEARECAAYFAkiYrW0ACgkQC3+MBN1Mb4i3gwCXdpeU8gvT1AGkKgT2Eh2lGHcQ gACg7k7+/cXzve/72UvEDJlIfCy1ENs= =asfp -----END PGP SIGNATURE----- --1BXV+/FYeXhtv2WT-- From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 20:47:14 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 979AF106566C for ; Tue, 5 Aug 2008 20:47:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from phoenix.cs.uoguelph.ca (phoenix.cs.uoguelph.ca [131.104.94.216]) by mx1.freebsd.org (Postfix) with ESMTP id 5099E8FC15 for ; Tue, 5 Aug 2008 20:47:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by phoenix.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m75KlDf3001479; Tue, 5 Aug 2008 16:47:13 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m75KwUP03929; Tue, 5 Aug 2008 16:58:30 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 5 Aug 2008 16:58:30 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Kostik Belousov In-Reply-To: <20080805194341.GI97161@deviant.kiev.zoral.com.ua> Message-ID: References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> <20080805153221.GG97161@deviant.kiev.zoral.com.ua> <20080805165114.GH97161@deviant.kiev.zoral.com.ua> <20080805194341.GI97161@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.216 Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 20:47:14 -0000 On Tue, 5 Aug 2008, Kostik Belousov wrote: [stuff snipped] >> Ok, I just spent a few minutes snooping around in vfs_subr.c and I think >> I see the problem. vget() has called vholdl() and then >> v_upgrade_usecount(), which has incremented the usecount and taken the >> vnode off the free list. This appears to prevent vgonel() from being >> called on it for most cases, but there is still the case in vflush() >> where the FORCECLOSE flag is set. > Yes, exactly. > [more stuff snipped] > > But, what guarantees that the vnode would not be reclaimed before/under > your vref() it ? For instance, what if the vnode is locked due to reclaim > being in progress ? > So long as I never do a vflush() with FORCECLOSE, I can't see anywhere that will vgonel() it once I have gotten it via vget(). (v_usecount incremented and not on the vnode freelist) The way I just coded it is: - the function that does the vfs_hash_get() without LK_EXCLUSIVE just fails if MNTK_UNMOUNTF is set. - my nfs_close just returns when MNTK_UNMOUNTF is set. - my nfs_unmount() doesn't set FORCECLOSE on the vflush() but instead sleeps and retries a bunch of times if vflush() fails for MNT_FORCE. - my nfs_unmount() and other code (mostly based on the vanilla FreeBSD client makes requests all fail with EINTR when MNTK_UNMOUNTF is set). I think this should work for a forced unmount, since once requests all fail and the recovery also fails, I think vflush() will work without the FORCECLOSE flag. As far as I can see, since I'm not vflush()'ng with FORCECLOSE, then nothing will vgonel() the vnode until it has been vrele()'d. (If there is a case other than vflush() with FORCECLOSE that will vgone() it when it is not on the freelist and has a v_usecount > 0, then I'll need to handle that as well, but I can't see one.) rick From owner-freebsd-fs@FreeBSD.ORG Tue Aug 5 18:58:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B42610658BB; Tue, 5 Aug 2008 18:58:45 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 9F46B8FC0A; Tue, 5 Aug 2008 18:58:44 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m75IwXKs094564; Tue, 5 Aug 2008 14:58:34 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-current@freebsd.org Date: Tue, 5 Aug 2008 13:28:17 -0400 User-Agent: KMail/1.9.7 References: <20080727125413.GG1345@garage.freebsd.pl> <20080731013229.9d342ee5.nork@FreeBSD.org> <20080806004557.6e538e5c.nork@FreeBSD.org> In-Reply-To: <20080806004557.6e538e5c.nork@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200808051328.18308.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Tue, 05 Aug 2008 14:58:34 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/7946/Tue Aug 5 13:44:22 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx X-Mailman-Approved-At: Tue, 05 Aug 2008 20:49:27 +0000 Cc: freebsd-fs@freebsd.org, Ulf Lilleengen , Pawel Jakub Dawidek , Norikatsu Shigemura , Lambrev , Randy Bush , Stefan@freebsd.org, OutBackdingo Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2008 18:58:45 -0000 On Tuesday 05 August 2008 11:45:57 am Norikatsu Shigemura wrote: > On Thu, 31 Jul 2008 01:32:29 +0900 > Norikatsu Shigemura wrote: > > > However, this feature is a bit undocumented yet, and it didn't work correctly > > > for me. But you can always test it out. > > I'm using zfsboot on my note PC, and not using UFS. I know many > > problems about it:-). > > 1. zpool configuration is too limited, only single and mirror > > usable. If you want to zfsboot, you can't use RAIDZ, striping > > and cache(zpool add ... cache ...):-(. > > I missed. zfsboot is disregarded zpool cache rather than supports it. > > > SEE ALSO: > > http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004895.html > > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/125878 > > I found some zfsboot issues, please apply following patches: > 1. zfsboot2 (boot2) doesn't %d (printf), so change %d to %u. > 2. chase new zpool versioning as SPA_VERSION. > Obtained from: sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > --- sys/boot/zfs/zfsimpl.c.orig 2008-07-28 01:54:49.194419000 +0900 > +++ sys/boot/zfs/zfsimpl.c 2008-08-05 23:48:12.035247220 +0900 > @@ -656,8 +656,8 @@ > return (EIO); > } > > - if (val != ZFS_VERSION) { > - printf("ZFS: unsupported ZFS version %d\n", (int) val); > + if (val > SPA_VERSION) { > + printf("ZFS: unsupported ZFS version %u (should be %u)\n", (int) val, (int) SPA_VERSION); > return (EIO); > } > > --- sys/cddl/boot/zfs/zfsimpl.h.orig 2008-07-28 01:54:49.296418000 +0900 > +++ sys/cddl/boot/zfs/zfsimpl.h 2008-08-06 00:07:41.871760182 +0900 > @@ -448,19 +448,24 @@ > /* > * On-disk version number. > */ > -#define ZFS_VERSION_1 1ULL > -#define ZFS_VERSION_2 2ULL > -#define ZFS_VERSION_3 3ULL > -#define ZFS_VERSION_4 4ULL > -#define ZFS_VERSION_5 5ULL > -#define ZFS_VERSION_6 6ULL > +#define SPA_VERSION_1 1ULL > +#define SPA_VERSION_2 2ULL > +#define SPA_VERSION_3 3ULL > +#define SPA_VERSION_4 4ULL > +#define SPA_VERSION_5 5ULL > +#define SPA_VERSION_6 6ULL FYI, style(9) prefers '#define' to '#define'. Keeping with the existing style would likely shorten the diffs. -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 00:03:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAAAE106566C for ; Wed, 6 Aug 2008 00:03:54 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.123]) by mx1.freebsd.org (Postfix) with ESMTP id A75AD8FC15 for ; Wed, 6 Aug 2008 00:03:54 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from shop.chemikals.org ([75.182.7.127]) by cdptpa-omta05.mail.rr.com with ESMTP id <20080806000353.PXZQ18001.cdptpa-omta05.mail.rr.com@shop.chemikals.org> for ; Wed, 6 Aug 2008 00:03:53 +0000 Received: from volatile.chemikals.org (root@r74-193-170-223.bssrcmta01.bscyla.by.dh.suddenlink.net [74.193.170.223] (may be forged)) by shop.chemikals.org (8.14.2/8.14.2) with ESMTP id m7603qn6052673 for ; Tue, 5 Aug 2008 20:03:52 -0400 (EDT) (envelope-from morganw@chemikals.org) Received: from localhost (morganw@localhost [127.0.0.1]) by volatile.chemikals.org (8.14.2/8.14.2) with ESMTP id m7603nW5025177 for ; Tue, 5 Aug 2008 19:03:51 -0500 (CDT) (envelope-from morganw@chemikals.org) Date: Tue, 5 Aug 2008 19:03:49 -0500 (CDT) From: Wes Morgan To: freebsd-fs@freebsd.org Message-ID: User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 00:03:55 -0000 I'm looking for information and advice from those experienced in building storage arrays with good performance. Thus far, I've simply been using a motherboard with a lot of built-in SATA ports. I've concentrated on making most of the investment in high quality storage rather than controllers, cases etc. It's just a 4U chassis (I don't even have a rack for it, too much $$$) with 16 hot-swap bays, for use as a media server. However, I've reached the point where I have a 8-drive raidz2. Any additional storage would need to be another independent raidz2 set, and there are not a lot of inexpensive options for go to 16 ports. So this brings up a few questions: - Has anyone looked at what kind of workloads tend to perform best with prefetch enabled or disabled? - Would I have better performance from a dedicated controller, and would the improvement be worth the cost? As it stands now, heavy read/write activity definitely interferes with both streaming and rtorrent. - The 16-port controllers tend to have a lot of fancy "Intel RAID chips" etc, which is simply a waste of money when using zfs, right? - Is one 16-port controller better than 2 8-port? Assuming two 8-device arrays, which will perform better? - Which brand of controllers are best supported by FreeBSD? I've seen 3Ware, Areca and LSI mentioned, and the prices are all pretty much the same. Can anyone share some of their experiences with these vendors? Thanks, WM From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 00:15:04 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E3DD1065686; Wed, 6 Aug 2008 00:15:04 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 133BF8FC1A; Wed, 6 Aug 2008 00:15:04 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m760F3Xl023896; Wed, 6 Aug 2008 00:15:03 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m760F389023892; Wed, 6 Aug 2008 00:15:03 GMT (envelope-from linimon) Date: Wed, 6 Aug 2008 00:15:03 GMT Message-Id: <200808060015.m760F389023892@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 00:15:04 -0000 Synopsis: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Wed Aug 6 00:14:52 UTC 2008 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=126287 From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 02:41:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D93DF1065672 for ; Wed, 6 Aug 2008 02:41:05 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id C4A5E8FC17 for ; Wed, 6 Aug 2008 02:41:05 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 4A15233C62; Tue, 5 Aug 2008 19:41:25 -0700 (PDT) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id E0CE733C5B; Tue, 5 Aug 2008 19:41:24 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18585.3903.895425.122613@almost.alerce.com> Date: Tue, 5 Aug 2008 19:41:03 -0700 To: Wes Morgan In-Reply-To: References: X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 02:41:05 -0000 Wes Morgan writes: > I'm looking for information and advice from those experienced in building > storage arrays with good performance. > [...] I don't have any experimental data to contribute, but there are some interesting hardware discussions on the opensolaris ZFS web site and blog entries. It does seem to be true that ZFS does best when used with simple controllers, and a lot of the opensolaris community seems to like this relatively inexpensive 8-port card (<$100 at newegg), it's apparently what SUN ships in their "Thumper": http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm One of the opensolaris threads links off to this this blog entry which discusses experiences with PCI-X and PCI-E cards, it might be useful. http://jmlittle.blogspot.com/2008/06/recommended-disk-controllers-for-zfs.html The AoC-SAT2-MV8 is based on the "Marvell Hercules-2 Rev. C0 SATA host controller", which seems to be AKA 88SX6081, which is listed as supported by the ata driver in 7.0-RELEASE. Has anyone had any ZFS experience with it? g. From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 07:00:07 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5BC521065679 for ; Wed, 6 Aug 2008 07:00:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3D65F8FC13 for ; Wed, 6 Aug 2008 07:00:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m76706bg066674 for ; Wed, 6 Aug 2008 07:00:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m76706oq066673; Wed, 6 Aug 2008 07:00:06 GMT (envelope-from gnats) Date: Wed, 6 Aug 2008 07:00:06 GMT Message-Id: <200808060700.m76706oq066673@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: "Remko Lodder" Cc: Subject: Re: kern/126287: Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Remko Lodder List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 07:00:07 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: "Remko Lodder" To: "Carel Braam" Cc: freebsd-gnats-submit@freebsd.org Subject: Re: kern/126287: Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Wed, 6 Aug 2008 08:54:14 +0200 (CEST) On Tue, August 5, 2008 11:36 pm, Carel Braam wrote: Hello Carel, This information is a bit narrow. If the kernel panics, you should be able to get a kernelcoredump. Please follow the procedure at http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html. Thanks, Remko -- /"\ Best regards, | remko@FreeBSD.org \ / Remko Lodder | remko@EFnet X http://www.evilcoder.org/ | / \ ASCII Ribbon Campaign | Against HTML Mail and News From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 07:32:47 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1C83106567F for ; Wed, 6 Aug 2008 07:32:47 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.233]) by mx1.freebsd.org (Postfix) with ESMTP id 8DFD58FC1C for ; Wed, 6 Aug 2008 07:32:47 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: by rv-out-0506.google.com with SMTP id b25so4334470rvf.43 for ; Wed, 06 Aug 2008 00:32:46 -0700 (PDT) Received: by 10.140.125.1 with SMTP id x1mr8698723rvc.287.1218007966635; Wed, 06 Aug 2008 00:32:46 -0700 (PDT) Received: from imac24.simerson.net ( [24.19.46.17]) by mx.google.com with ESMTPS id b39sm612260rvf.9.2008.08.06.00.32.44 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 06 Aug 2008 00:32:45 -0700 (PDT) Message-Id: <46EF69A7-349C-4E15-928C-D305E67273CA@spry.com> To: freebsd-fs@freebsd.org In-Reply-To: <18585.3903.895425.122613@almost.alerce.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v928.1) Date: Wed, 6 Aug 2008 00:32:43 -0700 References: <18585.3903.895425.122613@almost.alerce.com> X-Mailer: Apple Mail (2.928.1) From: Matt Simerson Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 07:32:47 -0000 On Aug 5, 2008, at 7:41 PM, George Hartzell wrote: > Wes Morgan writes: >> I'm looking for information and advice from those experienced in >> building >> storage arrays with good performance. >> [...] > > I don't have any experimental data to contribute, but there are some > interesting hardware discussions on the opensolaris ZFS web site and > blog entries. > > It does seem to be true that ZFS does best when used with simple > controllers, and a lot of the opensolaris community seems to like this > relatively inexpensive 8-port card (<$100 at newegg), it's apparently > what SUN ships in their "Thumper": > > http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm > > One of the opensolaris threads links off to this this blog entry which > discusses experiences with PCI-X and PCI-E cards, it might be useful. > > http://jmlittle.blogspot.com/2008/06/recommended-disk-controllers-for-zfs.html > > The AoC-SAT2-MV8 is based on the "Marvell Hercules-2 Rev. C0 SATA host > controller", which seems to be AKA 88SX6081, which is listed as > supported by the ata driver in 7.0-RELEASE. Has anyone had any ZFS > experience with it? I had 3 of them inside a SuperMicro 24 disk chassis with 16GB RAM, 8 core Xeon, and 24 1TB disks each. The other 24 disk chassis just like it I build with two Areca 1231ML SATA RAID controllers. I first tested UFS performance with a single disk on FreeBSD (Areca and Marvell) and OpenSolaris (Marvell). Write performance heavily favored the Areca card with the optional BBWC. The read performance was close enough to call even. Then I tested ZFS with RAIDZ in various configs (raidz, raidz2, 4,6, and 8 disk arrays) on FreeBSD. When using raidz and FreeBSD, the difference in performance of the controllers is much smaller. It's bad with the Areca controller and worse with the Marvell. My overall impression is that ZFS performance under FreeBSD is poor. I say this because I also tested one of the systems with OpenSolaris on the Marvell card (OpenSolaris doesn't support the Areca). Read performance with ZFS and RAIDZ on Solaris was not just 2-3 but 10-12x faster on Solaris. OpenSolaris write performance was about 50% faster than FreeBSD on the Areca controller and 100% faster than FreeBSD on the Marvell. The only way I could get decent performance out of FreeBSD and ZFS was to use the Areca as a RAID controller and then ZFS stripe the data across the two RAID arrays. I haven't tried it but I'm willing to bet that if I used UFS and geom_stripe to do the same thing, I'd get better performance with UFS. If you are looking for performance, then raidz and ZFS is not where you want to be looking. I use ZFS because these are backup servers and without the file system compression, I'd be using 16TB of storage instead of 11. As far as workload with prefetch: under my workloads (heavy network & file system I/O) prefetch=almost instant crash and burn. As soon as I put any heavy load on it, it hangs (as I've described previously on this list). Because I need the performance and prefer FreeBSD, the Areca cards with BBWC are well worth it. But if you need serious performance on a shoestring budget, consider running OpenSolaris with the Marvell cards. As to whether an 8 or 16 port will perform better, it depends on the bus and the cards. As long as you are using them on a PCIe multilane bus, you'll likely be hitting your disks I/O limits long before you read the bus limits. So it won't matter much. 3ware controllers = cheap and you get what you pay for. At my last job we had thousands of 3ware cards deployed because they were so inexpensive and RAID = RAID, right? Well, they were the controllers most likely to result in catastrophic data loss for our clients. Maybe it's because the interface is confusing the NOC technicians, maybe it's because their recovery tools suck, or because when the controller fails it hoses the disks in interesting ways. For various reasons, our luck at recovering failed RAID arrays on 3ware cards was poor. I've used a lot of LSI in the past. They work well but they aren't performance stars. The Areca is going to be a faster card than the others and it comes with a built-in Ethernet jack. Plug that sucker into your private network and use a web server to remotely manage the card. That's a nice feature. :) Several weeks after deploying both systems, we took down the AoC based one and retrofitted it with another pair of Areca controllers. Publishing the benchmarks is on my TODO list. Matt From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 09:22:27 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F089106566B for ; Wed, 6 Aug 2008 09:22:27 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id F0D758FC22 for ; Wed, 6 Aug 2008 09:22:26 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 09A5E1CC0B0; Wed, 6 Aug 2008 02:22:26 -0700 (PDT) Date: Wed, 6 Aug 2008 02:22:26 -0700 From: Jeremy Chadwick To: Matt Simerson Message-ID: <20080806092226.GA49691@eos.sc1.parodius.com> References: <18585.3903.895425.122613@almost.alerce.com> <46EF69A7-349C-4E15-928C-D305E67273CA@spry.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46EF69A7-349C-4E15-928C-D305E67273CA@spry.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 09:22:27 -0000 On Wed, Aug 06, 2008 at 12:32:43AM -0700, Matt Simerson wrote: > Then I tested ZFS with RAIDZ in various configs (raidz, raidz2, 4,6, and > 8 disk arrays) on FreeBSD. When using raidz and FreeBSD, the difference > in performance of the controllers is much smaller. It's bad with the > Areca controller and worse with the Marvell. My overall impression is > that ZFS performance under FreeBSD is poor. Performance has been significantly improved with a patch from pjd@ provided about a week ago. I have not tested it personally, but there have been a couple reports so far that the performance improvement is significant. I recommend you read the *entire thread*, and yes, it is very long. Subject is "ZFS patches". http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/thread.html It continues into August, with some people using mail clients that don't properly utilise mail reference IDs, so their replies are scattered. Again, look for "ZFS patches". http://lists.freebsd.org/pipermail/freebsd-fs/2008-August/thread.html > I say this because I also tested one of the systems with OpenSolaris on > the Marvell card (OpenSolaris doesn't support the Areca). Read > performance with ZFS and RAIDZ on Solaris was not just 2-3 but 10-12x > faster on Solaris. OpenSolaris write performance was about 50% faster > than FreeBSD on the Areca controller and 100% faster than FreeBSD on the > Marvell. > > The only way I could get decent performance out of FreeBSD and ZFS was > to use the Areca as a RAID controller and then ZFS stripe the data > across the two RAID arrays. I haven't tried it but I'm willing to bet > that if I used UFS and geom_stripe to do the same thing, I'd get better > performance with UFS. If you are looking for performance, then raidz and > ZFS is not where you want to be looking. Do you have any actual numbers showing performance differential? If so, how did you obtain them under FreeBSD? Also, what tuning parameters did you use on FreeBSD (specifically kernel/loader.conf stuff)? It seems that using "zpool iostat" is only useful if one wishes to see the amount of I/O that ends up hitting the physical disks in the pool; if the data is cached in memory, "zpool iostat" won't show any I/O. I can get performance data from gstat(8), but that won't tell me how ZFS itself is actually performing, only at what rate the kernel read/write data from the physical disks. On my system (a 3-disk raidz spool, using WDC WD5000AAKS disks, SATA300, on an Intel ICH7 controller), I can get about 70MB/sec from each disk when reading, and somewhere around 55-60MB/sec when writing. But again, this is for actual disk I/O and isn't testing ZFS performance. > As far as workload with prefetch: under my workloads (heavy network & > file system I/O) prefetch=almost instant crash and burn. As soon as I > put any heavy load on it, it hangs (as I've described previously on this > list). I assume by "hangs" you mean the system becomes unresponsive while disk reads/writes are being performed, then recovers, then stalls again, recovers, rinse lather repeat? If so -- yes, that's the exact behaviour others and myself have reported. Disabling prefetch makes the system much more usable during heavy I/O. > 3ware controllers = cheap and you get what you pay for. At my last job > we had thousands of 3ware cards deployed because they were so > inexpensive and RAID = RAID, right? Well, they were the controllers > most likely to result in catastrophic data loss for our clients. Maybe > it's because the interface is confusing the NOC technicians, maybe it's > because their recovery tools suck, or because when the controller fails > it hoses the disks in interesting ways. For various reasons, our luck at > recovering failed RAID arrays on 3ware cards was poor. This story is pretty much the norm. Once in a while you'll find someone praising 3ware controllers, but I often wonder just what kind of workload and failure testing they've done prior to their praise. A friend of at Rackable told me horror stories involving 3ware controllers. I'm thankful 3ware cares about FreeBSD (most vendors do not), but with a history of firmware/BIOS bugs and the sensitive nature of their cards, I choose to stay away from them. I've heard nothing but praise when it comes to Areca controllers. All that said -- have you actually performed a hard failure with an Areca controller on FreeBSD (using both UFS and ZFS)? Assuming you have hot-swap enclosures/carriers, what happens if you yank a disk on the Areca controller? How does FreeBSD behave in this case? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 09:29:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C53F1065671 for ; Wed, 6 Aug 2008 09:29:48 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from marvin.harmless.hu (marvin.harmless.hu [195.56.55.204]) by mx1.freebsd.org (Postfix) with ESMTP id AED068FC21 for ; Wed, 6 Aug 2008 09:29:47 +0000 (UTC) (envelope-from phoemix@harmless.hu) Received: from fw.publishing.hu ([82.131.181.62] helo=twoflower.in.publishing.hu) by marvin.harmless.hu with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KQfKz-000KoZ-TI; Wed, 06 Aug 2008 11:29:46 +0200 Date: Wed, 6 Aug 2008 11:29:44 +0200 From: CZUCZY Gergely To: Matt Simerson Message-ID: <20080806112944.6793fc11@twoflower.in.publishing.hu> In-Reply-To: <62D3072A-E41A-4CFC-971D-9924958F38C7@corp.spry.com> References: <20253C48-38CB-4A77-9C59-B993E7E5D78A@corp.spry.com> <62D3072A-E41A-4CFC-971D-9924958F38C7@corp.spry.com> Organization: Harmless Digital X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.11; i386-portbld-freebsd6.3) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_//.JVBhPRnbKd9=4bfY14EBp"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Sender: Czuczy Gergely Cc: freebsd-fs@freebsd.org Subject: Re: ZFS hang issue and prefetch_disable - UPDATE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 09:29:48 -0000 --Sig_//.JVBhPRnbKd9=4bfY14EBp Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, A few weeks ago, i was exactly referring to this. Somewhere around here: http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004796.html The thing, that it works on pointyhat, and it works on kris@'s box, is just= an IWFM-level, not the proof of any stability, reliability. FreeBSD is a quite stable OS, the code has a relatively good quality as far= as I've seen it, and it's quite stable. Somewhy the ZFS port seems to be an exception, it's refused to be merged properly and the issues to be solved. No matter how much someone tunes ZFS, no matter what you disable, it's not garanteed, not even on the tiniest level ever, to not to freeze your box, n= ot to throw a panic, to keep your data and everything. Many of us has reported this, bot noone looked into it. I know, we're free = to use something else, but that's not the point. The point is, I don't see a meaning of a port of this quality. I know it's quite complex and whatnot, b= ut at this level, it cannot be run in a production environment. It's missing reliability. No matter how much you hack it, there's always a not-so-impossible chance, = that it will shot you in your back, when you're not watching. I hope the latest ZFS patches will solve a lot of issues, and we won't see problems like this anymore. On Thu, 31 Jul 2008 13:58:26 -0700 Matt Simerson wrote: >=20 > My announcement that vfs.zfs.prefetch_disable=3D1 resulted in a stable =20 > system was premature. >=20 > One of my backup servers (see specs below) hung. When I got onto the =20 > console via KVM, it looked normal with no errors but didn't respond to =20 > Control-Alt-Delete. After a power cycle, zpool status showed 8 disks =20 > FAULTED and the action state was: http://www.sun.com/msg/ZFS-8000-5E >=20 > Basically, that meant my ZFS file system and 7.5TB of data was gone. =20 > Ouch. >=20 > I'm using a pair of ARECA 1231ML RAID controllers. Previously, I had =20 > them configured in JBOD with raidz2. This time around, I configured =20 > both controllers with one 12 disk RAID 6 volume. Now FreeBSD just sees =20 > two 10TB disks which I stripe with ZFS: zpool create back01 /dev/=20 > da0 /dev/da1 >=20 > I also did a bit more fiddling with /boot/loader.conf. Previous I had: >=20 > vm.kmem_size=3D"1536M" > vm.kmem_size_max=3D"1536M" > vfs.zfs.prefetch_disable=3D1 >=20 > This resulted in ZFS using 1.1GB of RAM (as measured using the =20 > technique described on the wiki) during normal use. The system in =20 > question hung during the nightly processing (which backs up some other =20 > systems via rsync) and my suspicions are that when I/O load picked up, =20 > it exhausted the available kernel memory and hung the system. So now I =20 > have these settings on one system: >=20 > vm.kmem_size=3D"1536M" > vm.kmem_size_max=3D"1536M" > vfs.zfs.arc_min=3D"16M" > vfs.zfs.arc_max=3D"64M" > vfs.zfs.prefetch_disable=3D1 >=20 > and the same except vfs.zfs.arc_max=3D"256M" on the other. The one with = =20 > 64M uses 256MB of RAM for ZFS and the one set at 256M uses 600MB of =20 > RAM. These are measured under heavy network and disk IO load being =20 > generated by multiple rsync processes pulling backups from remote =20 > nodes and storing it on ZFS. I am using ZFS compression. >=20 > I get much better performance now with RAID 6 on the controller and =20 > ZFS striping than using raidz2. >=20 > Unless tuning the arc_ settings made the difference. Either way, the =20 > system I just rebuilt is now quite a bit faster with RAID 6 than JBOD =20 > + raidz2. >=20 > Hopefully tuning vfs.zfs.arc_max will result in stability. If it =20 > doesn't, my next choice is upgrading to -HEAD with the recent ZFS =20 > patch or ditching ZFS entirely and using geom_stripe. I don't like =20 > either option. >=20 > Matt >=20 >=20 > > From: Matt Simerson > > Date: July 22, 2008 1:25:42 PM PDT > > To: freebsd-fs@freebsd.org > > Subject: ZFS hang issue and prefetch_disable > > > > Symptoms > > > > Deadlocks under heavy IO load on the ZFS file system with =20 > > prefetch_disable=3D0. Setting vfs.zfs.prefetch_disable=3D1 results in = a =20 > > stable system. > > > > Configuration > > > > Two machines. Identically built. Both exhibit identical behavior. > > 8 cores (2 x E5420) x 2.5GHz, 16 GB RAM, 24 x 1TB disks. > > FreeBSD 7.0 amd64 > > dmesg: http://matt.simerson.net/computing/zfs/dmesg.txt > > > > Boot disk is a read only 1GB compact flash > > # cat /etc/fstab > > /dev/ad0s1a / ufs ro,noatime 2 2 > > > > # df -h / > > Filesystem 1K-blocks Used Avail Capacity Mounted on > > /dev/ad0s1a 939M 555M 309M 64% / > > > > RAM has been boosted as suggested in ZFS Tuning Guide > > # cat /boot/loader.conf > > vm.kmem_size=3D 1610612736 > > vm.kmem_size_max=3D 1610612736 > > vfs.zfs.prefetch_disable=3D1 > > > > I haven't mucked much with the other memory settings as I'm using =20 > > amd64 and according to the FreeBSD ZFS wiki, that isn't necessary. =20 > > I've tried higher settings for kmem but that resulted in a failed =20 > > boot. I have ample RAM And would love to use as much as possible for =20 > > network and disk I/O buffers as that's principally all this system =20 > > does. > > > > Disks & ZFS options > > > > Sun's "Best Practices" suggests limiting the number of disks in a =20 > > raidz pool to no more than 6-10, IIRC. ZFS is configured as shown: > > http://matt.simerson.net/computing/zfs/zpool.txt > > > > I'm using all of the ZFS default properties except: atime=3Doff, =20 > > compression=3Don. > > > > Environment > > > > I'm using these machines as backup servers. I wrote an application =20 > > that generates a list of the thousands of VPS accounts we host. For =20 > > each host, it generates a rsnapshot configuration file and backs up =20 > > up their VPS to these systems via rsync. The application manages =20 > > concurrency and will spawn additional rsync processes if system i/o =20 > > load is below a defined threshhold. Which is to say, I can crank up =20 > > or down the amount of disk IO the system sees. > > > > With vfs.zfs.prefetch_disable=3D0, I can trigger a hang within a few =20 > > hours (no more than a day). If I keep the i/o load (measured via =20 > > iostat) down to a low level (< 200 iops) then I still get hangs but =20 > > less frequently (1-6 days). The only way I have found to prevent =20 > > the hangs is by setting vfs.zfs.prefetch_disable=3D1. >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 =C3=9Cdv=C3=B6lettel, Czuczy Gergely Harmless Digital Bt mailto: gergely.czuczy@harmless.hu Tel: +36-30-9702963 --Sig_//.JVBhPRnbKd9=4bfY14EBp Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (FreeBSD) iD8DBQFImW8IzrC0WyuMkpsRAke8AJ0V3tIeTpnA5POMJmWXxb0DW2sjrwCbB5aG s3DJQYYnSDAPXKN4qHeXBns= =1kIB -----END PGP SIGNATURE----- --Sig_//.JVBhPRnbKd9=4bfY14EBp-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 09:30:05 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D9811065677 for ; Wed, 6 Aug 2008 09:30:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6AD398FC1F for ; Wed, 6 Aug 2008 09:30:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m769U5m3008915 for ; Wed, 6 Aug 2008 09:30:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m769U5Yf008912; Wed, 6 Aug 2008 09:30:05 GMT (envelope-from gnats) Date: Wed, 6 Aug 2008 09:30:05 GMT Message-Id: <200808060930.m769U5Yf008912@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: "Mateusz Guzik" Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Mateusz Guzik List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 09:30:05 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: "Mateusz Guzik" To: bug-followup@freebsd.org Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Wed, 6 Aug 2008 11:24:16 +0200 ------=_Part_20104_30199813.1218014656704 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi, function vfs_deleteopt() was called with NULL pointer (opts) used in TAILQ_FOREACH_SAFE macro -- I believe that simple `if (opts == NULL) return; ' in that function is ok to fix this. (Take a look at attachment.) At least the kernel does not panic. ;) Thanks, -- Mateusz Guzik ------=_Part_20104_30199813.1218014656704 Content-Type: application/octet-stream; name=vfs_mount.diff Content-Transfer-Encoding: base64 X-Attachment-Id: f_fjjqf5in0 Content-Disposition: attachment; filename=vfs_mount.diff LS0tIHN5cy9rZXJuL3Zmc19tb3VudC5jLm9yaWcJMjAwOC0wOC0wNiAxMToxNDoxNi4wMDAwMDAw MDAgKzAyMDAKKysrIHN5cy9rZXJuL3Zmc19tb3VudC5jCTIwMDgtMDgtMDYgMTE6MTQ6MzIuMDAw MDAwMDAwICswMjAwCkBAIC0xOTYsMTAgKzE5NiwxMyBAQAogdm9pZAogdmZzX2RlbGV0ZW9wdChz dHJ1Y3QgdmZzb3B0bGlzdCAqb3B0cywgY29uc3QgY2hhciAqbmFtZSkKIHsKIAlzdHJ1Y3QgdmZz b3B0ICpvcHQsICp0ZW1wOwogCisJaWYgKG9wdHMgPT0gTlVMTCkKKwkJcmV0dXJuOworCiAJVEFJ TFFfRk9SRUFDSF9TQUZFKG9wdCwgb3B0cywgbGluaywgdGVtcCkgIHsKIAkJaWYgKHN0cmNtcChv cHQtPm5hbWUsIG5hbWUpID09IDApCiAJCQl2ZnNfZnJlZW9wdChvcHRzLCBvcHQpOwogCX0KIH0K ------=_Part_20104_30199813.1218014656704-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 10:15:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A2181065670 for ; Wed, 6 Aug 2008 10:15:07 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.154]) by mx1.freebsd.org (Postfix) with ESMTP id 4C4BF8FC23 for ; Wed, 6 Aug 2008 10:15:05 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so4000487fgb.35 for ; Wed, 06 Aug 2008 03:15:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=DC2rACN/hHk6BF4FgPRIpgpvYDH8anAnROPr8Yzcujc=; b=j0JeP2it3gS2eVsySmPhJvlapqvf1YZNE4CDvDKpGAyx07NIN4v7wd5UTjWiOKdr+K eH3Sgy2g2N/qi5SGDzkXeDtEi/dYYlqMNXhoMsHmO2qIaAB5ZxsjML57PCkJJFnjRnNT 1DZAulBjf7SwxpmvPaoWzDx/A2hSuJYz5Mzrk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=eDNjji53Jb+016G7Ceod2y2bJt5CbFsfcrgvP8CwXrFTzFKxh0ndDsgQGUTNoXnFpp J2Nj6ljJCFYh551Snjt9jVlaJxfDVjnDDjmWMM4z3ppRWltzphXB1fUQWHhe3A2X0MeN wUBcKy/7ejcDEWySWVpA/acXVes3zT4Uwijkg= Received: by 10.86.73.3 with SMTP id v3mr688789fga.17.1218017704671; Wed, 06 Aug 2008 03:15:04 -0700 (PDT) Received: by 10.86.35.9 with HTTP; Wed, 6 Aug 2008 03:15:04 -0700 (PDT) Message-ID: Date: Wed, 6 Aug 2008 12:15:04 +0200 From: "Claus Guttesen" To: "Pawel Jakub Dawidek" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080727125413.GG1345@garage.freebsd.pl> Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 10:15:07 -0000 >>> I have a areca arc-1680 sas-card and an external sas-cabinet with 16 >>> sas-drives each 1 TB (931 binary GB). They have been setup in three >>> raidz-partitions with five disks each in one zpool and one spare. >>> >> My conclusion about it's stability was a bit hasty. I was copying >> approx. 400 GB from a nfs-share mounted from a solaris 9 on sparc >> using tcp and read- and write-size of 32768. The files are images >> slightly less than 1 MB and a thumbnail (approx. 983000 files). > > There seems to be a hardware-related problem to my setup. I'm getting > some 'arcmsr0: scsi id=1 lun=4 ccb='0xffffff02d5cc8e00' outstanding > command timeout' (in solaris). I'll check with my vendor. I did not > see such errors in FreeBSD. I changed the configration on the areca arc-1680-card and put all disks in throughput-mode and is now able to copy large amounts of data (more than 1 TB) without problems. I have 'zpool offline'd a disk and 'zpool replace'd it with a spare. The resilver is progressing as normal and the zpool is accessible. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 10:20:05 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B2B9106566C for ; Wed, 6 Aug 2008 10:20:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 793F98FC22 for ; Wed, 6 Aug 2008 10:20:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m76AK5Tv013324 for ; Wed, 6 Aug 2008 10:20:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m76AK5NI013323; Wed, 6 Aug 2008 10:20:05 GMT (envelope-from gnats) Date: Wed, 6 Aug 2008 10:20:05 GMT Message-Id: <200808061020.m76AK5NI013323@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: "Mateusz Guzik" Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Mateusz Guzik List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 10:20:05 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: "Mateusz Guzik" To: bug-followup@freebsd.org Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Wed, 6 Aug 2008 12:15:00 +0200 Something weird happened to my attachment, I'll paste it here: --- sys/kern/vfs_mount.c.orig 2008-08-06 11:14:16.000000000 +0200 +++ sys/kern/vfs_mount.c 2008-08-06 11:14:32.000000000 +0200 @@ -196,10 +196,13 @@ void vfs_deleteopt(struct vfsoptlist *opts, const char *name) { struct vfsopt *opt, *temp; + if (opts == NULL) + return; + TAILQ_FOREACH_SAFE(opt, opts, link, temp) { if (strcmp(opt->name, name) == 0) vfs_freeopt(opts, opt); } } Again, it should work fine ;) Thanks, -- Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 13:34:47 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C8DB1065671; Wed, 6 Aug 2008 13:34:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 8F6228FC1E; Wed, 6 Aug 2008 13:34:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m76DYgAk046040 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 6 Aug 2008 16:34:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m76DYfNj089938; Wed, 6 Aug 2008 16:34:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m76DYf3N089937; Wed, 6 Aug 2008 16:34:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 6 Aug 2008 16:34:41 +0300 From: Kostik Belousov To: Mateusz Guzik Message-ID: <20080806133441.GM97161@deviant.kiev.zoral.com.ua> References: <200808061020.m76AK5NI013323@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="11IAkegDWp8TRrA/" Content-Disposition: inline In-Reply-To: <200808061020.m76AK5NI013323@freefall.freebsd.org> User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 13:34:47 -0000 --11IAkegDWp8TRrA/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 06, 2008 at 10:20:05AM +0000, Mateusz Guzik wrote: > The following reply was made to PR kern/126287; it has been noted by GNAT= S. >=20 > From: "Mateusz Guzik" > To: bug-followup@freebsd.org > Cc: =20 > Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an U= FS filesystem with snapshot enabled > Date: Wed, 6 Aug 2008 12:15:00 +0200 >=20 > Something weird happened to my attachment, I'll paste it here: > =20 > --- sys/kern/vfs_mount.c.orig 2008-08-06 11:14:16.000000000 +0200 > +++ sys/kern/vfs_mount.c 2008-08-06 11:14:32.000000000 +0200 > @@ -196,10 +196,13 @@ > void > vfs_deleteopt(struct vfsoptlist *opts, const char *name) > { > struct vfsopt *opt, *temp; > =20 > + if (opts =3D=3D NULL) > + return; > + > TAILQ_FOREACH_SAFE(opt, opts, link, temp) { > if (strcmp(opt->name, name) =3D=3D 0) > vfs_freeopt(opts, opt); > } > } > =20 > Again, it should work fine ;) > =20 > Thanks, > -- > Mateusz Guzik The PR lacks the backtrace (preferrable the ddb output or "bt full" from kgdb) for the panic. Please, show me the backtrace. --11IAkegDWp8TRrA/ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiZqHAACgkQC3+MBN1Mb4jCYQCg8Zuw0keIHdOXrkv9Q5yK8M6r tEkAn3GayEaX5S9xQqiqDRBTooAe8ggD =zxuG -----END PGP SIGNATURE----- --11IAkegDWp8TRrA/-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 13:39:55 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8ED6D1065687 for ; Wed, 6 Aug 2008 13:39:55 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 233868FC0C for ; Wed, 6 Aug 2008 13:39:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m76DdpSq046188 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 6 Aug 2008 16:39:51 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m76DdpMv099960; Wed, 6 Aug 2008 16:39:51 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m76Ddo96099678; Wed, 6 Aug 2008 16:39:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 6 Aug 2008 16:39:50 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20080806133950.GN97161@deviant.kiev.zoral.com.ua> References: <20080805083229.GB97161@deviant.kiev.zoral.com.ua> <20080805153221.GG97161@deviant.kiev.zoral.com.ua> <20080805165114.GH97161@deviant.kiev.zoral.com.ua> <20080805194341.GI97161@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IiH5iyEqjK6f5lUg" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 13:39:55 -0000 --IiH5iyEqjK6f5lUg Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 05, 2008 at 04:58:30PM -0400, Rick Macklem wrote: >=20 >=20 > On Tue, 5 Aug 2008, Kostik Belousov wrote: >=20 > [stuff snipped] > >>Ok, I just spent a few minutes snooping around in vfs_subr.c and I think > >>I see the problem. vget() has called vholdl() and then > >>v_upgrade_usecount(), which has incremented the usecount and taken the > >>vnode off the free list. This appears to prevent vgonel() from being > >>called on it for most cases, but there is still the case in vflush() > >>where the FORCECLOSE flag is set. > >Yes, exactly. > > > [more stuff snipped] > > > >But, what guarantees that the vnode would not be reclaimed before/under > >your vref() it ? For instance, what if the vnode is locked due to reclaim > >being in progress ? > > > So long as I never do a vflush() with FORCECLOSE, I can't see anywhere=20 > that will vgonel() it once I have gotten it via vget(). (v_usecount > incremented and not on the vnode freelist) >=20 > The way I just coded it is: > - the function that does the vfs_hash_get() without LK_EXCLUSIVE just > fails if MNTK_UNMOUNTF is set. > - my nfs_close just returns when MNTK_UNMOUNTF is set. > - my nfs_unmount() doesn't set FORCECLOSE on the vflush() but instead > sleeps and retries a bunch of times if vflush() fails for MNT_FORCE. > - my nfs_unmount() and other code (mostly based on the vanilla FreeBSD > client makes requests all fail with EINTR when MNTK_UNMOUNTF is set). You still has the race where the MNTK_UNMOUNTF is set after you check returned false, isn't it ? BTW, is your fs marked as mpsafe ? >=20 > I think this should work for a forced unmount, since once requests all > fail and the recovery also fails, I think vflush() will work without > the FORCECLOSE flag. >=20 > As far as I can see, since I'm not vflush()'ng with FORCECLOSE, then > nothing will vgonel() the vnode until it has been vrele()'d. (If there > is a case other than vflush() with FORCECLOSE that will vgone() it when > it is not on the freelist and has a v_usecount > 0, then I'll need to > handle that as well, but I can't see one.) Yes, ATM it should be safe, since only vflush() does reclamation for the vnodes with usecount > 0. On the other hand, I believe our VFS never makes a guarantee that this is the only location of the call. --IiH5iyEqjK6f5lUg Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiZqaYACgkQC3+MBN1Mb4hdHACdFRKyFhrhmRqHxBWYnk3Ka2id s7MAoNsuFDa+MYMXykCQ8APpEDsrX4A/ =FMcN -----END PGP SIGNATURE----- --IiH5iyEqjK6f5lUg-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 13:40:04 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8945106566B for ; Wed, 6 Aug 2008 13:40:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B79058FC16 for ; Wed, 6 Aug 2008 13:40:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m76De4JT033943 for ; Wed, 6 Aug 2008 13:40:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m76De44w033942; Wed, 6 Aug 2008 13:40:04 GMT (envelope-from gnats) Date: Wed, 6 Aug 2008 13:40:04 GMT Message-Id: <200808061340.m76De44w033942@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Kostik Belousov Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Kostik Belousov List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 13:40:04 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: Kostik Belousov To: Mateusz Guzik Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Wed, 6 Aug 2008 16:34:41 +0300 --11IAkegDWp8TRrA/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 06, 2008 at 10:20:05AM +0000, Mateusz Guzik wrote: > The following reply was made to PR kern/126287; it has been noted by GNAT= S. >=20 > From: "Mateusz Guzik" > To: bug-followup@freebsd.org > Cc: =20 > Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an U= FS filesystem with snapshot enabled > Date: Wed, 6 Aug 2008 12:15:00 +0200 >=20 > Something weird happened to my attachment, I'll paste it here: > =20 > --- sys/kern/vfs_mount.c.orig 2008-08-06 11:14:16.000000000 +0200 > +++ sys/kern/vfs_mount.c 2008-08-06 11:14:32.000000000 +0200 > @@ -196,10 +196,13 @@ > void > vfs_deleteopt(struct vfsoptlist *opts, const char *name) > { > struct vfsopt *opt, *temp; > =20 > + if (opts =3D=3D NULL) > + return; > + > TAILQ_FOREACH_SAFE(opt, opts, link, temp) { > if (strcmp(opt->name, name) =3D=3D 0) > vfs_freeopt(opts, opt); > } > } > =20 > Again, it should work fine ;) > =20 > Thanks, > -- > Mateusz Guzik The PR lacks the backtrace (preferrable the ddb output or "bt full" from kgdb) for the panic. Please, show me the backtrace. --11IAkegDWp8TRrA/ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiZqHAACgkQC3+MBN1Mb4jCYQCg8Zuw0keIHdOXrkv9Q5yK8M6r tEkAn3GayEaX5S9xQqiqDRBTooAe8ggD =zxuG -----END PGP SIGNATURE----- --11IAkegDWp8TRrA/-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 14:00:03 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2BC621065671 for ; Wed, 6 Aug 2008 14:00:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 16EEF8FC15 for ; Wed, 6 Aug 2008 14:00:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m76E02Vg034823 for ; Wed, 6 Aug 2008 14:00:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m76E028t034822; Wed, 6 Aug 2008 14:00:02 GMT (envelope-from gnats) Date: Wed, 6 Aug 2008 14:00:02 GMT Message-Id: <200808061400.m76E028t034822@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: "Mateusz Guzik" Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Mateusz Guzik List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 14:00:03 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: "Mateusz Guzik" To: "Kostik Belousov" Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Wed, 6 Aug 2008 15:52:24 +0200 2008/8/6 Kostik Belousov : > On Wed, Aug 06, 2008 at 10:20:05AM +0000, Mateusz Guzik wrote: >> The following reply was made to PR kern/126287; it has been noted by GNATS. >> >> From: "Mateusz Guzik" >> To: bug-followup@freebsd.org >> Cc: >> Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled >> Date: Wed, 6 Aug 2008 12:15:00 +0200 >> >> Something weird happened to my attachment, I'll paste it here: >> >> --- sys/kern/vfs_mount.c.orig 2008-08-06 11:14:16.000000000 +0200 >> +++ sys/kern/vfs_mount.c 2008-08-06 11:14:32.000000000 +0200 >> @@ -196,10 +196,13 @@ >> void >> vfs_deleteopt(struct vfsoptlist *opts, const char *name) >> { >> struct vfsopt *opt, *temp; >> >> + if (opts == NULL) >> + return; >> + >> TAILQ_FOREACH_SAFE(opt, opts, link, temp) { >> if (strcmp(opt->name, name) == 0) >> vfs_freeopt(opts, opt); >> } >> } >> >> Again, it should work fine ;) > > The PR lacks the backtrace (preferrable the ddb output or "bt full" from > kgdb) for the panic. Please, show me the backtrace. > Sorry, I don't have currently access to fbsd 7, so here is backtrace from CURRENT(crashed by mount -o snapshot /somefilesystem): [..] #11 0xc06e1e5b in calltrap () at /srv/build/CURRENT/src/sys/i386/i386/exception.s:165 #12 0xc05c86d4 in vfs_deleteopt (opts=0x0, name=0xc074ef52 "snapshot") at /srv/build/CURRENT/src/sys/kern/vfs_mount.c:195 #13 0xc068d689 in ffs_mount (mp=0xc29f52a0, td=0xc2875af0) at /srv/build/CURRENT/src/sys/ufs/ffs/ffs_vfsops.c:172 #14 0xc05cb1d8 in vfs_donmount (td=0xc2875af0, fsflags=0, fsoptions=0xc261db80) at /srv/build/CURRENT/src/sys/kern/vfs_mount.c:1010 #15 0xc05cc3bb in nmount (td=0xc2875af0, uap=0xcd3a7cf8) at /srv/build/CURRENT/src/sys/kern/vfs_mount.c:417 #16 0xc06f9157 in syscall (frame=0xcd3a7d38) at /srv/build/CURRENT/src/sys/i386/i386/trap.c:1081 #17 0xc06e1ef0 in Xint0x80_syscall () at /srv/build/CURRENT/src/sys/i386/i386/exception.s:261 Thanks, -- Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 14:18:00 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37BF7106566B for ; Wed, 6 Aug 2008 14:18:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.226]) by mx1.freebsd.org (Postfix) with ESMTP id E24088FC22 for ; Wed, 6 Aug 2008 14:17:59 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by wx-out-0506.google.com with SMTP id h27so488191wxd.7 for ; Wed, 06 Aug 2008 07:17:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=G9l3rIqpM0DaYGqBrkTGbWwfyDI//9C4113QPsIiWEc=; b=XEuGm8gNAvtmljSEbb8AmtxKnWNQtvCOI6d+mXaDiH9RNEI933YxrQviYrceMf5NAL r4Bwfj4s3+0AAd0mXC9L3Z9buybT4Eit9ybZTBD++x9h/hiCYqMhXo/f9hKTOLIm0fP/ RwA1xNL3OM8VH//1CfIZyaX6a3XCdY/kFaOro= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=NjSG/LS9m5aUfF4nPBkrDBEgoAh1k6X7pAG8jCHvhsjbe7ZGJzSiyrTfa3gvR4bGSk AuTGeEGlIZ0uZwW/StyfMRCRVRzsqonxNQizTW/Kdt0GKJCzCydQGoFjOfjyTMHhT/Us DfkonSepImO8+z97xJZQKXpMn0gtZpHAuvNbc= Received: by 10.100.247.14 with SMTP id u14mr1805565anh.89.1218030744867; Wed, 06 Aug 2008 06:52:24 -0700 (PDT) Received: by 10.100.42.12 with HTTP; Wed, 6 Aug 2008 06:52:24 -0700 (PDT) Message-ID: Date: Wed, 6 Aug 2008 15:52:24 +0200 From: "Mateusz Guzik" To: "Kostik Belousov" In-Reply-To: <20080806133441.GM97161@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200808061020.m76AK5NI013323@freefall.freebsd.org> <20080806133441.GM97161@deviant.kiev.zoral.com.ua> Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 14:18:00 -0000 2008/8/6 Kostik Belousov : > On Wed, Aug 06, 2008 at 10:20:05AM +0000, Mateusz Guzik wrote: >> The following reply was made to PR kern/126287; it has been noted by GNATS. >> >> From: "Mateusz Guzik" >> To: bug-followup@freebsd.org >> Cc: >> Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled >> Date: Wed, 6 Aug 2008 12:15:00 +0200 >> >> Something weird happened to my attachment, I'll paste it here: >> >> --- sys/kern/vfs_mount.c.orig 2008-08-06 11:14:16.000000000 +0200 >> +++ sys/kern/vfs_mount.c 2008-08-06 11:14:32.000000000 +0200 >> @@ -196,10 +196,13 @@ >> void >> vfs_deleteopt(struct vfsoptlist *opts, const char *name) >> { >> struct vfsopt *opt, *temp; >> >> + if (opts == NULL) >> + return; >> + >> TAILQ_FOREACH_SAFE(opt, opts, link, temp) { >> if (strcmp(opt->name, name) == 0) >> vfs_freeopt(opts, opt); >> } >> } >> >> Again, it should work fine ;) > > The PR lacks the backtrace (preferrable the ddb output or "bt full" from > kgdb) for the panic. Please, show me the backtrace. > Sorry, I don't have currently access to fbsd 7, so here is backtrace from CURRENT(crashed by mount -o snapshot /somefilesystem): [..] #11 0xc06e1e5b in calltrap () at /srv/build/CURRENT/src/sys/i386/i386/exception.s:165 #12 0xc05c86d4 in vfs_deleteopt (opts=0x0, name=0xc074ef52 "snapshot") at /srv/build/CURRENT/src/sys/kern/vfs_mount.c:195 #13 0xc068d689 in ffs_mount (mp=0xc29f52a0, td=0xc2875af0) at /srv/build/CURRENT/src/sys/ufs/ffs/ffs_vfsops.c:172 #14 0xc05cb1d8 in vfs_donmount (td=0xc2875af0, fsflags=0, fsoptions=0xc261db80) at /srv/build/CURRENT/src/sys/kern/vfs_mount.c:1010 #15 0xc05cc3bb in nmount (td=0xc2875af0, uap=0xcd3a7cf8) at /srv/build/CURRENT/src/sys/kern/vfs_mount.c:417 #16 0xc06f9157 in syscall (frame=0xcd3a7d38) at /srv/build/CURRENT/src/sys/i386/i386/trap.c:1081 #17 0xc06e1ef0 in Xint0x80_syscall () at /srv/build/CURRENT/src/sys/i386/i386/exception.s:261 Thanks, -- Mateusz Guzik From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 14:48:25 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B01D91065672; Wed, 6 Aug 2008 14:48:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (skuns.zoral.com.ua [91.193.166.194]) by mx1.freebsd.org (Postfix) with ESMTP id 0C57C8FC1C; Wed, 6 Aug 2008 14:48:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m76EmLwG049121 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 6 Aug 2008 17:48:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m76EmK96053958; Wed, 6 Aug 2008 17:48:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m76EmKVs053957; Wed, 6 Aug 2008 17:48:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 6 Aug 2008 17:48:20 +0300 From: Kostik Belousov To: Mateusz Guzik Message-ID: <20080806144820.GO97161@deviant.kiev.zoral.com.ua> References: <200808061020.m76AK5NI013323@freefall.freebsd.org> <20080806133441.GM97161@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ltihE5wS63FR6l1A" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 14:48:25 -0000 --ltihE5wS63FR6l1A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 06, 2008 at 03:52:24PM +0200, Mateusz Guzik wrote: > Sorry, I don't have currently access to fbsd 7, so here is backtrace > from CURRENT(crashed by mount -o snapshot /somefilesystem): I very much doubt that original submitter has mean this problem. But thanks for noting the issue. I prefer the following change, committed as r181345: diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c index 5ee123a..4d9754e 100644 --- a/sys/ufs/ffs/ffs_vfsops.c +++ b/sys/ufs/ffs/ffs_vfsops.c @@ -169,7 +169,8 @@ ffs_mount(struct mount *mp, struct thread *td) * persist "snapshot" in the options list. */ vfs_deleteopt(mp->mnt_optnew, "snapshot"); - vfs_deleteopt(mp->mnt_opt, "snapshot"); + if (mp->mnt_opt !=3D NULL) + vfs_deleteopt(mp->mnt_opt, "snapshot"); } =20 MNT_ILOCK(mp); --ltihE5wS63FR6l1A Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiZubQACgkQC3+MBN1Mb4jkfwCgg+sP0nON+SXsND0/W1nlJU79 aD4AoOOETOGS1Jf5rv4NWLY0cukGLNeR =Cx00 -----END PGP SIGNATURE----- --ltihE5wS63FR6l1A-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 14:50:05 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4308106564A for ; Wed, 6 Aug 2008 14:50:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D39988FC1D for ; Wed, 6 Aug 2008 14:50:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m76Eo5gY039895 for ; Wed, 6 Aug 2008 14:50:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m76Eo59Y039894; Wed, 6 Aug 2008 14:50:05 GMT (envelope-from gnats) Date: Wed, 6 Aug 2008 14:50:05 GMT Message-Id: <200808061450.m76Eo59Y039894@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Kostik Belousov Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Kostik Belousov List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 14:50:06 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: Kostik Belousov To: Mateusz Guzik Cc: freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Wed, 6 Aug 2008 17:48:20 +0300 --ltihE5wS63FR6l1A Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 06, 2008 at 03:52:24PM +0200, Mateusz Guzik wrote: > Sorry, I don't have currently access to fbsd 7, so here is backtrace > from CURRENT(crashed by mount -o snapshot /somefilesystem): I very much doubt that original submitter has mean this problem. But thanks for noting the issue. I prefer the following change, committed as r181345: diff --git a/sys/ufs/ffs/ffs_vfsops.c b/sys/ufs/ffs/ffs_vfsops.c index 5ee123a..4d9754e 100644 --- a/sys/ufs/ffs/ffs_vfsops.c +++ b/sys/ufs/ffs/ffs_vfsops.c @@ -169,7 +169,8 @@ ffs_mount(struct mount *mp, struct thread *td) * persist "snapshot" in the options list. */ vfs_deleteopt(mp->mnt_optnew, "snapshot"); - vfs_deleteopt(mp->mnt_opt, "snapshot"); + if (mp->mnt_opt !=3D NULL) + vfs_deleteopt(mp->mnt_opt, "snapshot"); } =20 MNT_ILOCK(mp); --ltihE5wS63FR6l1A Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEARECAAYFAkiZubQACgkQC3+MBN1Mb4jkfwCgg+sP0nON+SXsND0/W1nlJU79 aD4AoOOETOGS1Jf5rv4NWLY0cukGLNeR =Cx00 -----END PGP SIGNATURE----- --ltihE5wS63FR6l1A-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 15:29:01 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70808106566C for ; Wed, 6 Aug 2008 15:29:01 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: from mx0.excelsus.net (emmett.excelsus.com [74.93.113.252]) by mx1.freebsd.org (Postfix) with ESMTP id 1EBCF8FC14 for ; Wed, 6 Aug 2008 15:29:00 +0000 (UTC) (envelope-from weldon@excelsus.com) Received: (qmail 21635 invoked by uid 89); 6 Aug 2008 15:00:57 -0000 Received: from unknown (HELO localhost) (127.0.0.1) by localhost.excelsus.com with SMTP; 6 Aug 2008 15:00:57 -0000 Date: Wed, 6 Aug 2008 11:00:57 -0400 (EDT) From: Weldon S Godfrey 3 To: freebsd-fs@freebsd.org Message-ID: <20080806101621.H24586@emmett.excelsus.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 15:29:01 -0000 Hello, Please forgive me, I didn't really see this discussed in the archives but I am wondering if anyone has seen this issue. I can replicate this issue under FreeBSD amd64 7.0-RELEASE and the latest -STABLE (RELENG_7). I do not replicate any problems running 9 instances of postmark on the machine directly, so the issue appears to be isolated with NFS. There are backtraces and more information in ticket kern/124280 I am experiencing random kernel panics while running postmark benchmark from 9 NFS clients (clients on RedHat) to a 3TB ZFS filesystem exported with NFS. The panics happen as soon as 5 mins from starting the benchmark or may take hours before it panics and reboots. It doesn't correspond to a time a cron job is going on. I am using the following settings in postmark: set number 20000 set transactions 10000000 set subdirectories 1000 set size 10000 15000 set report verbose set location /var/mail/store1/X (where X is a number 1-9 so each is operating in its own tree) The problem happens if I run 1 postmark on 9 NFS clients at the same time (each client is its own server) or if I run 9 postmarks on one NFS client. commands used to create filesystem: zpool create tank mirror da0 da12 mirror da1 da13 mirror da2 da14 mirror da3 da15\ mirror da4 da16 mirror da5 da17 mirror da6 da18 mirror da7 da19 mirror da8 da20 \ mirror da9 da21 mirror da10 da22 spare da11 da23 zfs set atime=off tank zfs create tank/mail zfs set mountpoint=/var/mail tank/mail zfs set sharenfs="-maproot=root -network 192.168.2.0 -mask 255.255.255.0" tank/mail I am using a 3ware 9690 SAS controller. I have 2 IBM EXP3000 enclosures, each drive is shown as single disk by the controller. this is my loader.conf: vm.kmem_size_max="1073741824" vm.kmem_size="1073741824" kern.maxvnodes="800000" vfs.zfs.prefetch_disable="1" vfs.zfs.cache_flush_disable="1" (I should note that kern.maxnodes in loader.conf does not appear to do anything, after boot, it is shown to be at 100000 with sysctl. It does change to 800000 if I manually set it with sysctl. However it appears my vnode usage sits at around 25-26K and is near that within 5s of the panic. The server has 16GB of RAM, and 2 quad core XEON processors. This server is only a NFS fileserver. The only non-default daemon running is sshd. It is running the GENERIC kernel, right now, unmodified. I am using two NICs. NFS is exported only on the secondary NIC. Each NIC is in it's own subnet. nothing in /var/log/messages near time of panic except: Aug 6 08:45:30 store1 savecore: reboot after panic: page fault Aug 6 08:45:30 store1 savecore: writing core to vmcore.2 I can provide cores if needed. Thank you for your time! Weldon kgdb with backtrace: store1# kgdb kernel.debug /var/crash/vmcore.2 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 5; apic id = 05 fault virtual address = 0xdc fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff8063b3d8 stack pointer = 0x10:0xffffffffdfbc5720 frame pointer = 0x10:0xffffff00543ed000 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 839 (nfsd) trap number = 12 panic: page fault cpuid = 5 Uptime: 18m53s Physical memory: 16366 MB Dumping 1991 MB: 1976 1960 1944 1928 1912 1896 1880 1864 1848 1832 1816 1800 1784 1768 1752 1736 1720 1704 1688 1672 1656 1640 1624 1608 1592 1576 1560 1544 1528 1512 1496 1480 1464 1448 1432 1416 1400 1384 1368 1352 1336 1320 1304 1288 1272 1256 1240 1224 1208 1192 1176 1160 1144 1128 1112 1096 1080 1064 1048 1032 1016 1000 984 968 952 936 920 904 888 872 856 840 824 808 792 776 760 744 728 712 696 680 664 648 632 616 600 584 568 552 536 520 504 488 472 456 440 424 408 392 376 360 344 328 312 296 280 264 248 232 216 200 184 168 152 136 120 104 88 72 56 40 24 8 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko #0 doadump () at pcpu.h:194 194 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff804a7049 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #3 0xffffffff804a744d in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:572 #4 0xffffffff807780e4 in trap_fatal (frame=0xffffff000bce26c0, eva=18446742974395967712) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff807784b5 in trap_pfault (frame=0xffffffffdfbc5670, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641 #6 0xffffffff80778de8 in trap (frame=0xffffffffdfbc5670) at /usr/src/sys/amd64/amd64/trap.c:410 #7 0xffffffff8075e7ce in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #8 0xffffffff8063b3d8 in nfsrv_access (vp=0xffffff00207d7dc8, flags=128, cred=0xffffff00403d4800, rdonly=0, td=0xffffff000bce26c0, override=0) at /usr/src/sys/nfsserver/nfs_serv.c:4284 #9 0xffffffff8063c4f1 in nfsrv3_access (nfsd=0xffffff00543ed000, slp=0xffffff0006396d00, td=0xffffff000bce26c0, mrq=0xffffffffdfbc5af0) at /usr/src/sys/nfsserver/nfs_serv.c:234 #10 0xffffffff8064cd1d in nfssvc (td=Variable "td" is not available. ) at /usr/src/sys/nfsserver/nfs_syscalls.c:456 #11 0xffffffff80778737 in syscall (frame=0xffffffffdfbc5c70) at /usr/src/sys/amd64/amd64/trap.c:852 #12 0xffffffff8075e9db in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:290 #13 0x0000000800687acc in ?? () Previous frame inner to this frame (corrupt stack?) From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 17:03:18 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F40441065676 for ; Wed, 6 Aug 2008 17:03:17 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.191]) by mx1.freebsd.org (Postfix) with ESMTP id 7F46D8FC1B for ; Wed, 6 Aug 2008 17:03:17 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: by nf-out-0910.google.com with SMTP id h3so6842nfh.33 for ; Wed, 06 Aug 2008 10:03:15 -0700 (PDT) Received: by 10.210.18.18 with SMTP id 18mr2828197ebr.95.1218042195276; Wed, 06 Aug 2008 10:03:15 -0700 (PDT) Received: from matts.spry.com ( [64.79.222.10]) by mx.google.com with ESMTPS id j8sm3165184gvb.1.2008.08.06.10.03.13 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 06 Aug 2008 10:03:14 -0700 (PDT) Message-Id: From: Matt Simerson To: freebsd-fs@freebsd.org In-Reply-To: <20080806112944.6793fc11@twoflower.in.publishing.hu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v928.1) Date: Wed, 6 Aug 2008 10:03:08 -0700 References: <20253C48-38CB-4A77-9C59-B993E7E5D78A@corp.spry.com> <62D3072A-E41A-4CFC-971D-9924958F38C7@corp.spry.com> <20080806112944.6793fc11@twoflower.in.publishing.hu> X-Mailer: Apple Mail (2.928.1) Subject: Re: ZFS hang issue and prefetch_disable - UPDATE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 17:03:18 -0000 On Aug 6, 2008, at 2:29 AM, CZUCZY Gergely wrote: > A few weeks ago, i was exactly referring to this. Somewhere around =20 > here: > http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004796.html > > The thing, that it works on pointyhat, and it works on kris@'s box, =20= > is just an > IWFM-level, not the proof of any stability, reliability. > > FreeBSD is a quite stable OS, the code has a relatively good quality =20= > as far as > I've seen it, and it's quite stable. Somewhy the ZFS port seems to =20 > be an > exception, it's refused to be merged properly and the issues to be =20 > solved. > > No matter how much someone tunes ZFS, no matter what you disable, =20 > it's not > garanteed, not even on the tiniest level ever, to not to freeze your =20= > box, not > to throw a panic, to keep your data and everything. You want/expect guarantees of stability with experimental features? I =20= think someone needs their expectations calibrated. > Many of us has reported this, bot noone looked into it. Because you haven't seen proof that someone looked into doesn't mean =20 nobody has. You are not being fair nor respectful to the time that =20 others are investing in ZFS. > use something else, but that's not the point. The point is, I don't =20= > see a > meaning of a port of this quality. I know it's quite complex and =20 > whatnot, but > at this level, it cannot be run in a production environment. It's =20 > missing > reliability. If you don't see the value of ZFS, don't use it. I'm not complaining =20 because ZFS isn't stable. I'd like it to be, but the best way I can =20 help is provide detailed information about my setup and under what =20 conditions the feature has problems. By doing so, I'm providing useful =20= data. Denigrating the authors because ZFS doesn't meet your =20 expectations doesn't help anybody, so please don't do that. Matt > No matter how much you hack it, there's always a not-so-impossible =20 > chance, that > it will shot you in your back, when you're not watching. > > I hope the latest ZFS patches will solve a lot of issues, and we =20 > won't see > problems like this anymore. > > On Thu, 31 Jul 2008 13:58:26 -0700 > Matt Simerson wrote: > >> >> My announcement that vfs.zfs.prefetch_disable=3D1 resulted in a = stable >> system was premature. >> >> One of my backup servers (see specs below) hung. When I got onto the >> console via KVM, it looked normal with no errors but didn't respond =20= >> to >> Control-Alt-Delete. After a power cycle, zpool status showed 8 =20 >> disks >> FAULTED and the action state was: http://www.sun.com/msg/ZFS-8000-5E >> >> Basically, that meant my ZFS file system and 7.5TB of data was gone. >> Ouch. >> >> I'm using a pair of ARECA 1231ML RAID controllers. Previously, I had >> them configured in JBOD with raidz2. This time around, I configured >> both controllers with one 12 disk RAID 6 volume. Now FreeBSD just =20 >> sees >> two 10TB disks which I stripe with ZFS: zpool create back01 /dev/ >> da0 /dev/da1 >> >> I also did a bit more fiddling with /boot/loader.conf. Previous I =20 >> had: >> >> vm.kmem_size=3D"1536M" >> vm.kmem_size_max=3D"1536M" >> vfs.zfs.prefetch_disable=3D1 >> >> This resulted in ZFS using 1.1GB of RAM (as measured using the >> technique described on the wiki) during normal use. The system in >> question hung during the nightly processing (which backs up some =20 >> other >> systems via rsync) and my suspicions are that when I/O load picked =20= >> up, >> it exhausted the available kernel memory and hung the system. So =20 >> now I >> have these settings on one system: >> >> vm.kmem_size=3D"1536M" >> vm.kmem_size_max=3D"1536M" >> vfs.zfs.arc_min=3D"16M" >> vfs.zfs.arc_max=3D"64M" >> vfs.zfs.prefetch_disable=3D1 >> >> and the same except vfs.zfs.arc_max=3D"256M" on the other. The one = with >> 64M uses 256MB of RAM for ZFS and the one set at 256M uses 600MB of >> RAM. These are measured under heavy network and disk IO load being >> generated by multiple rsync processes pulling backups from remote >> nodes and storing it on ZFS. I am using ZFS compression. >> >> I get much better performance now with RAID 6 on the controller and >> ZFS striping than using raidz2. >> >> Unless tuning the arc_ settings made the difference. Either way, the >> system I just rebuilt is now quite a bit faster with RAID 6 than JBOD >> + raidz2. >> >> Hopefully tuning vfs.zfs.arc_max will result in stability. If it >> doesn't, my next choice is upgrading to -HEAD with the recent ZFS >> patch or ditching ZFS entirely and using geom_stripe. I don't like >> either option. >> >> Matt >> >> >>> From: Matt Simerson >>> Date: July 22, 2008 1:25:42 PM PDT >>> To: freebsd-fs@freebsd.org >>> Subject: ZFS hang issue and prefetch_disable >>> >>> Symptoms >>> >>> Deadlocks under heavy IO load on the ZFS file system with >>> prefetch_disable=3D0. Setting vfs.zfs.prefetch_disable=3D1 results = in a >>> stable system. >>> >>> Configuration >>> >>> Two machines. Identically built. Both exhibit identical behavior. >>> 8 cores (2 x E5420) x 2.5GHz, 16 GB RAM, 24 x 1TB disks. >>> FreeBSD 7.0 amd64 >>> dmesg: http://matt.simerson.net/computing/zfs/dmesg.txt >>> >>> Boot disk is a read only 1GB compact flash >>> # cat /etc/fstab >>> /dev/ad0s1a / ufs ro,noatime 2 2 >>> >>> # df -h / >>> Filesystem 1K-blocks Used Avail Capacity Mounted on >>> /dev/ad0s1a 939M 555M 309M 64% / >>> >>> RAM has been boosted as suggested in ZFS Tuning Guide >>> # cat /boot/loader.conf >>> vm.kmem_size=3D 1610612736 >>> vm.kmem_size_max=3D 1610612736 >>> vfs.zfs.prefetch_disable=3D1 >>> >>> I haven't mucked much with the other memory settings as I'm using >>> amd64 and according to the FreeBSD ZFS wiki, that isn't necessary. >>> I've tried higher settings for kmem but that resulted in a failed >>> boot. I have ample RAM And would love to use as much as possible for >>> network and disk I/O buffers as that's principally all this system >>> does. >>> >>> Disks & ZFS options >>> >>> Sun's "Best Practices" suggests limiting the number of disks in a >>> raidz pool to no more than 6-10, IIRC. ZFS is configured as shown: >>> http://matt.simerson.net/computing/zfs/zpool.txt >>> >>> I'm using all of the ZFS default properties except: atime=3Doff, >>> compression=3Don. >>> >>> Environment >>> >>> I'm using these machines as backup servers. I wrote an application >>> that generates a list of the thousands of VPS accounts we host. For >>> each host, it generates a rsnapshot configuration file and backs up >>> up their VPS to these systems via rsync. The application manages >>> concurrency and will spawn additional rsync processes if system i/o >>> load is below a defined threshhold. Which is to say, I can crank up >>> or down the amount of disk IO the system sees. >>> >>> With vfs.zfs.prefetch_disable=3D0, I can trigger a hang within a few >>> hours (no more than a day). If I keep the i/o load (measured via >>> iostat) down to a low level (< 200 iops) then I still get hangs but >>> less frequently (1-6 days). The only way I have found to prevent >>> the hangs is by setting vfs.zfs.prefetch_disable=3D1. >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > --=20 > =DCdv=F6lettel, > > Czuczy Gergely > Harmless Digital Bt > mailto: gergely.czuczy@harmless.hu > Tel: +36-30-9702963 From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 17:21:33 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1CA841065693 for ; Wed, 6 Aug 2008 17:21:33 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from smtp.infidyne.com (ds9.infidyne.com [88.80.6.206]) by mx1.freebsd.org (Postfix) with ESMTP id CED358FC25 for ; Wed, 6 Aug 2008 17:21:32 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se (c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se [85.229.22.169]) by smtp.infidyne.com (Postfix) with ESMTPSA id 49B7F7ADC8; Wed, 6 Aug 2008 19:21:31 +0200 (CEST) From: Peter Schuller To: freebsd-fs@freebsd.org Date: Wed, 6 Aug 2008 19:22:45 +0200 User-Agent: KMail/1.9.7 References: In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart16988886.SpBpzAnUjj"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200808061922.55472.peter.schuller@infidyne.com> Cc: Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 17:21:33 -0000 --nextPart16988886.SpBpzAnUjj Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline > - Which brand of controllers are best supported by FreeBSD? I've seen > 3Ware, Areca and LSI mentioned, and the prices are all pretty much the > same. Can anyone share some of their experiences with these vendors? I realize I am ignoring the most interesting bits of your post, but I recen= tly=20 found indications that the HighPoint driver in FreeBSD supports port=20 multipliers. I found someone claiming that it is supported on a mailinglist= =2E=20 Checking HighPoint does list multiplier support as an "official" card featu= re=20 for those cards that support it, and they are maintaining FreeBSD has a=20 supported platform AFAIK. I sent a request to them to confirm whether this= =20 was indeed the case, but haven't received a response (yet?). Since you mention you're on a budget, I thought I'd mention this since, ri= ght=20 now, the most likely future path for me will be either HighPoint if I can=20 confirm multiplier support, or future FreeBSD releases given that multiplie= r=20 support seems to be on the way in for various controllers. =2D-=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --nextPart16988886.SpBpzAnUjj Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEABECAAYFAkiZ3e8ACgkQDNor2+l1i30T+wCfVMIs4YlYDxcnw2zhM3/HEFLR nGkAoMjTCSTDOUYVmDS3wHOnVBmdKtpp =Qcom -----END PGP SIGNATURE----- --nextPart16988886.SpBpzAnUjj-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 17:27:12 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BAE791065681 for ; Wed, 6 Aug 2008 17:27:12 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from smtp.infidyne.com (ds9.infidyne.com [88.80.6.206]) by mx1.freebsd.org (Postfix) with ESMTP id 77C4A8FC1E for ; Wed, 6 Aug 2008 17:27:12 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se (c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se [85.229.22.169]) by smtp.infidyne.com (Postfix) with ESMTPSA id C05857ADE2; Wed, 6 Aug 2008 19:27:11 +0200 (CEST) From: Peter Schuller To: freebsd-fs@freebsd.org, hartzell@alerce.com Date: Wed, 6 Aug 2008 19:28:36 +0200 User-Agent: KMail/1.9.7 References: <18585.3903.895425.122613@almost.alerce.com> In-Reply-To: <18585.3903.895425.122613@almost.alerce.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart29072657.oddkThEjFf"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200808061928.37001.peter.schuller@infidyne.com> Cc: Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 17:27:12 -0000 --nextPart29072657.oddkThEjFf Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline > The AoC-SAT2-MV8 is based on the "Marvell Hercules-2 Rev. C0 SATA host > controller", which seems to be AKA 88SX6081, which is listed as > supported by the ata driver in 7.0-RELEASE. Has anyone had any ZFS > experience with it? Yes; it has been working quite fine for me with 7 up to a release-candidate= =2E=20 In 7.0-RELEASE you must disable the hptrr driver because it eats the device= =2E=20 See: http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/120615 http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/120842 When I say "working fine", this is with 8 SATA drives (6 of them in a raidz= 2,=20 two of them for other stuff) and not having any issues like corruption,=20 timeouts or whatever else people have had with shaky controllers. However, I cannot speak to performance because it's a PCI-X card that I've= =20 plugged into a PCI slow, so throughput is limited by the PCI bus (and the=20 machine is otherwise not the fastest to begin with). Note that this is on 32 bit; haven't been able to try it on 64 bit because = I=20 the PCI-X card wouldn't work on the motherboard (again PCI, so it's=20 hit-and-miss) where I would otherwise have tried it. I'd love to find a buyable PCI-E version that also worked in FreeBSD... I'l= l=20 see if the link in your post contains any such hints. =2D-=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --nextPart29072657.oddkThEjFf Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEABECAAYFAkiZ30QACgkQDNor2+l1i31dpACgnSgvtHU/zkIi9sJl4irUGcaa 5s4AoK/nkNVJqsgIGd7Wzxhj97mydh3p =EEas -----END PGP SIGNATURE----- --nextPart29072657.oddkThEjFf-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 17:36:06 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A14991065681; Wed, 6 Aug 2008 17:36:06 +0000 (UTC) (envelope-from nork@FreeBSD.org) Received: from sakura.ninth-nine.com (unknown [IPv6:2001:2f0:104:80a0:230:48ff:fe41:2455]) by mx1.freebsd.org (Postfix) with ESMTP id 197FF8FC12; Wed, 6 Aug 2008 17:36:05 +0000 (UTC) (envelope-from nork@FreeBSD.org) Received: from nadesico.ninth-nine.com (nadesico.ninth-nine.com [219.127.74.122]) by sakura.ninth-nine.com (8.14.1/8.14.1/NinthNine) with SMTP id m76HZxNa006311; Thu, 7 Aug 2008 02:35:59 +0900 (JST) (envelope-from nork@FreeBSD.org) Date: Thu, 7 Aug 2008 02:35:58 +0900 From: Norikatsu Shigemura To: John Baldwin Message-Id: <20080807023558.39ca4ffa.nork@FreeBSD.org> In-Reply-To: <200808051328.18308.jhb@freebsd.org> References: <20080727125413.GG1345@garage.freebsd.pl> <20080731013229.9d342ee5.nork@FreeBSD.org> <20080806004557.6e538e5c.nork@FreeBSD.org> <200808051328.18308.jhb@freebsd.org> X-Mailer: Sylpheed 2.5.0 (GTK+ 2.12.11; i386-portbld-freebsd8.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (sakura.ninth-nine.com [219.127.74.121]); Thu, 07 Aug 2008 02:36:00 +0900 (JST) X-Mailman-Approved-At: Wed, 06 Aug 2008 18:06:47 +0000 Cc: freebsd-fs@FreeBSD.org, Lilleengen , Pawel Jakub Dawidek , Jeremy Chadwick , Stefan, Norikatsu Shigemura , Lambrev , Randy Bush , freebsd-current@FreeBSD.org, OutBackdingo , Ulf Subject: Re: ZFS patches. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 17:36:06 -0000 On Tue, 5 Aug 2008 13:28:17 -0400 John Baldwin wrote: > On Tuesday 05 August 2008 11:45:57 am Norikatsu Shigemura wrote: > > On Thu, 31 Jul 2008 01:32:29 +0900 > > Norikatsu Shigemura wrote: > > > > However, this feature is a bit undocumented yet, and it didn't work > correctly > > > > for me. But you can always test it out. > > > I'm using zfsboot on my note PC, and not using UFS. I know many > > > problems about it:-). > > > 1. zpool configuration is too limited, only single and mirror > > > usable. If you want to zfsboot, you can't use RAIDZ, striping > > > and cache(zpool add ... cache ...):-(. > > I missed. zfsboot is disregarded zpool cache rather than supports it. Grrr my broken English! "rather than doesn't support it."... > > > SEE ALSO: > > > http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004895.html > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/125878 > > I found some zfsboot issues, please apply following patches: > > 1. zfsboot2 (boot2) doesn't %d (printf), so change %d to %u. Grrr my broken English! "doesn't support %d(printf)"... > > +#define SPA_VERSION_6 6ULL > FYI, style(9) prefers '#define' to '#define'. Keeping with the > existing style would likely shorten the diffs. Yes, thanks for your pointed out. From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 18:07:36 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDBC21065672 for ; Wed, 6 Aug 2008 18:07:36 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [212.17.241.230]) by mx1.freebsd.org (Postfix) with ESMTP id 2D7138FC24 for ; Wed, 6 Aug 2008 18:07:35 +0000 (UTC) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (localhost [127.0.0.1]) by lurza.secnetix.de (8.14.1/8.14.1) with ESMTP id m76I7BUv004738; Wed, 6 Aug 2008 20:07:21 +0200 (CEST) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.14.1/8.14.1/Submit) id m76I7BUj004737; Wed, 6 Aug 2008 20:07:11 +0200 (CEST) (envelope-from olli) Date: Wed, 6 Aug 2008 20:07:11 +0200 (CEST) Message-Id: <200808061807.m76I7BUj004737@lurza.secnetix.de> From: Oliver Fromme To: freebsd-fs@FreeBSD.ORG In-Reply-To: <20080806112944.6793fc11@twoflower.in.publishing.hu> X-Newsgroups: list.freebsd-fs User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.2-STABLE-20070808 (i386)) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.2 (lurza.secnetix.de [127.0.0.1]); Wed, 06 Aug 2008 20:07:22 +0200 (CEST) Cc: Subject: Re: ZFS hang issue and prefetch_disable - UPDATE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-fs@FreeBSD.ORG List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 18:07:36 -0000 CZUCZY Gergely wrote: > The thing, that it works on pointyhat, and it works on kris@'s box, is just an > IWFM-level, not the proof of any stability, reliability. You cannot "prove" stability or reliability. If you think you can, please tell me how. > No matter how much someone tunes ZFS, no matter what you disable, it's not > garanteed, [...] You want a guarantee? There is none. Not for ZFS, not for UFS, not for any other file system on any operating system, be it commercial or open-source. I agree that ZFS still has problems, especially related to kernel memory. AFAIK this is being worked on, and Pawel's latest patches seem to improve things a lot. Note that the ZFS code is still considered experimental (you've seen the fat warning, I assume), so it's reasonable to expect that it doesn't provide production-quality yet. It is rather totally unreasonable to expect a port of ZFS to appear in FreeBSD and be bug-free and without problems from day one. Also note that, in earlier days, certain UFS-features such as soft-updates and dirhash were also known to need a lot of memory (well, "a lot" by standards as of those days), and it's still wise to disable them on small embedded boxes with limited RAM. The memory requirements of ZFS aren't that much different, although on a larger scale. Alan Cox has worked on the problem and lifted the existing kmem limit for amd64 in FreeBSD -current. (I'm not sure if he will MFC that to 7-stable.) If you run with that code *and* Pawels latest ZFS patches, you should be a lot less likely to see the dreaded kmem panics. And that's without any tuning. Of course it might still make sense to tune ZFS for your workload in order to get better performance (e.g. disable prefetch etc.). Just the same as UFS. > Many of us has reported this, bot noone looked into it. That's completely untrue. Some people put a lot of time and efforts in FreeBSD's ZFS port. Please don't insult them. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "C++ is over-complicated nonsense. And Bjorn Shoestrap's book a danger to public health. I tried reading it once, I was in recovery for months." -- Cliff Sarginson From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 18:59:16 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BED7106566C for ; Wed, 6 Aug 2008 18:59:16 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206045140.chello.pl [87.206.45.140]) by mx1.freebsd.org (Postfix) with ESMTP id F06038FC0A for ; Wed, 6 Aug 2008 18:59:15 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 2079045C89; Wed, 6 Aug 2008 20:59:15 +0200 (CEST) Received: from localhost (chello087206045140.chello.pl [87.206.45.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 45AA645685; Wed, 6 Aug 2008 20:59:07 +0200 (CEST) Date: Wed, 6 Aug 2008 20:59:09 +0200 From: Pawel Jakub Dawidek To: Peter Schuller Message-ID: <20080806185909.GC2580@garage.freebsd.pl> References: <200807262005.54235.peter.schuller@infidyne.com> <20080726205118.GB1345@garage.freebsd.pl> <200807272026.54907.peter.schuller@infidyne.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="R+My9LyyhiUvIEro" Content-Disposition: inline In-Reply-To: <200807272026.54907.peter.schuller@infidyne.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Asynchronous writing to zvols (ZFS) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 18:59:16 -0000 --R+My9LyyhiUvIEro Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jul 27, 2008 at 08:26:46PM +0200, Peter Schuller wrote: > Hello, >=20 > > The problem is that we don't between async and sync I/O request on GEOM > > level, that's why I decided to commit a ZIL log after each write, which > > wasn't very smart it seems. This is handled differently in version I've > > in perforce. Could you try the below patch and see how it performs now? > > > > http://people.freebsd.org/~pjd/patches/zvol.c.patch >=20 > The above (though the files has moved, for anyone else reading wanting to= =20 > apply) does eliminate the synchronicity problem. I am now seeing 5-15=20 > MB/second write speeds to the zvol, with 100% constituent disk utilizatio= n. >=20 > I am not sure why I don't see faster writes; I get more like 40-60 when= =20 > writing to a file in a ZFS file system on the same pool. But regardless, = the=20 > synchronisity issue is gone. Not sure why's that, I spent no time on optimizing ZVOL yet, sorry. > Does your comment above regarding distinguishing bewteen sync and asynch = apply=20 > to the section of code affected by the above patch, or did you mean there= is=20 > some other place above the zvol handling where there is lack of distincti= on? >=20 > That is, is the end-effect of the above change that we *never* do synchro= nous=20 > writes (because the fact that a write is supposed to be synchronous is=20 > somehow lost before it reaches that point)? >=20 > I understand a zil_commit is only required on BIO_FLUSH requests, which i= s=20 > what the patch fixes. But I get the impression from your phrasing above t= hat=20 > the reason that a zil_commit was done on every I/O from the get go was in= an=20 > effort to honor actual synchronous writes by conservatively *always* doin= g=20 > synchronous writes, because the synchronicity of synchronous writes would= not=20 > be propagated down to the zvol class. I wouldn't want to sacrifice=20 > correctness just to get the speed ;) With the patch above we synchoronize in-memory transactions every 5 seconds or when queue is full or when we receive BIO_FLUSH. Of course the previous behaviour was more conservative, but sending writes down doesn't mean they will reach disk platters. There is still disk cache in the way. If we really want to be sure that data is safe on the disk, we should send BIO_FLUSH. In other words if you use UFS on raw disk, sync writes can still be delayed by disk's cache. When you use UFS on top of ZVOL, writes can be delayed by ZFS cache. I think the way to go is to pass sync/async property of I/O request down to the GEOM stack. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --R+My9LyyhiUvIEro Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFImfR9ForvXbEpPzQRAmFpAKDWDG6e500u/ENxVw+gFw5K8DL/fACfVgdL cr+CDy5pzlEYToSBnFgKBZU= =fGxK -----END PGP SIGNATURE----- --R+My9LyyhiUvIEro-- From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 19:15:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1E98106566B for ; Wed, 6 Aug 2008 19:15:07 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.121]) by mx1.freebsd.org (Postfix) with ESMTP id 7DE638FC1E for ; Wed, 6 Aug 2008 19:15:07 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from shop.chemikals.org ([75.182.7.127]) by cdptpa-omta01.mail.rr.com with ESMTP id <20080806191506.SUXT14026.cdptpa-omta01.mail.rr.com@shop.chemikals.org>; Wed, 6 Aug 2008 19:15:06 +0000 Received: from volatile.chemikals.org (root@r74-193-170-223.bssrcmta01.bscyla.by.dh.suddenlink.net [74.193.170.223] (may be forged)) by shop.chemikals.org (8.14.2/8.14.2) with ESMTP id m76JF4u9077484; Wed, 6 Aug 2008 15:15:05 -0400 (EDT) (envelope-from morganw@chemikals.org) Received: from localhost (morganw@localhost [127.0.0.1]) by volatile.chemikals.org (8.14.2/8.14.2) with ESMTP id m76JF2Dp036582; Wed, 6 Aug 2008 14:15:03 -0500 (CDT) (envelope-from morganw@chemikals.org) Date: Wed, 6 Aug 2008 14:15:02 -0500 (CDT) From: Wes Morgan To: Peter Schuller In-Reply-To: <200808061928.37001.peter.schuller@infidyne.com> Message-ID: References: <18585.3903.895425.122613@almost.alerce.com> <200808061928.37001.peter.schuller@infidyne.com> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 19:15:07 -0000 On Wed, 6 Aug 2008, Peter Schuller wrote: >> The AoC-SAT2-MV8 is based on the "Marvell Hercules-2 Rev. C0 SATA host >> controller", which seems to be AKA 88SX6081, which is listed as >> supported by the ata driver in 7.0-RELEASE. Has anyone had any ZFS >> experience with it? > > Yes; it has been working quite fine for me with 7 up to a release-candidate. > In 7.0-RELEASE you must disable the hptrr driver because it eats the device. > See: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120615 > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120842 > > When I say "working fine", this is with 8 SATA drives (6 of them in a raidz2, > two of them for other stuff) and not having any issues like corruption, > timeouts or whatever else people have had with shaky controllers. > > However, I cannot speak to performance because it's a PCI-X card that I've > plugged into a PCI slow, so throughput is limited by the PCI bus (and the > machine is otherwise not the fastest to begin with). > > Note that this is on 32 bit; haven't been able to try it on 64 bit because I > the PCI-X card wouldn't work on the motherboard (again PCI, so it's > hit-and-miss) where I would otherwise have tried it. > > I'd love to find a buyable PCI-E version that also worked in FreeBSD... I'll > see if the link in your post contains any such hints. Hmmm... That PCI-X card is interesting. Supermicro also lists this: http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ics/lsisas1068e/index.html Not much onboard ram, but it's PCI-E and even SAS. CDW lists it for $155. That would be cheaper than buying a new board with a PCI-X slot or two, and would even handle SAS drives. Claims to be based on the "LSISAS 1068E SAS controller". Any idea if that is supported? I don't see it listed in the mfi man page. LSI has a Linux driver for download. That card looks like it would be just what I need. From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 20:36:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F1B2106567F for ; Wed, 6 Aug 2008 20:36:54 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from ccshst09.cs.uoguelph.ca (ccshst09.cs.uoguelph.ca [131.104.94.206]) by mx1.freebsd.org (Postfix) with ESMTP id 4C2848FC67 for ; Wed, 6 Aug 2008 20:36:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by ccshst09.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m76Kaqfm022777; Wed, 6 Aug 2008 16:36:52 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m76KmAQ01189; Wed, 6 Aug 2008 16:48:10 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Wed, 6 Aug 2008 16:48:10 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: kostikbel@gmail.com Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.206 Cc: freebsd-fs@freebsd.org Subject: Re: doing vfs_hash_get when vnode locked X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 20:36:54 -0000 On Tue, Aug 05, 2008 at 04:58:30PM -0400, Rick Macklem wrote: [stuff snipped] >> The way I just coded it is: >> - the function that does the vfs_hash_get() without LK_EXCLUSIVE just >> fails if MNTK_UNMOUNTF is set. >> - my nfs_close just returns when MNTK_UNMOUNTF is set. >> - my nfs_unmount() doesn't set FORCECLOSE on the vflush() but instead >> sleeps and retries a bunch of times if vflush() fails for MNT_FORCE. >> - my nfs_unmount() and other code (mostly based on the vanilla FreeBSD >> client makes requests all fail with EINTR when MNTK_UNMOUNTF is set). > You still has the race where the MNTK_UNMOUNTF is set after you check > returned false, isn't it ? I don't think it will be an issue, but I haven't tested the forced unmount without FORCECLOSE yet. > BTW, is your fs marked as mpsafe ? Yep. Except for the low level code doing the RPCs, the client side is basically a clone of the regular nfsclient. > Yes, ATM it should be safe, since only vflush() does reclamation for the > vnodes with usecount > 0. On the other hand, I believe our VFS never > makes a guarantee that this is the only location of the call. Well, at this point, vflush() only seems to be called inside the file systems for a mount point of that file system type. If someone adds a vflush() with FORCECLOSE outside of my code that acts on a mountpoint for my nfs client, it could break horribly. That's life in this game;-) I'm thinking about how to avoid this, but it ain't gonna be trivial and it could be ugly. (The short version is that, for nfsv4, the nfs part of the vnode must keep the directory fh and component name of the file in it, so that Opens can be done. That's ugly enough, given renames and multiple hard links to a file. If I can't get at that nfs vnode, I'll have to keep a copy of the directory fh and component name in the Open data structure, which wouldn't be too bad until a rename occurs.) I might try and do this in the next month or so... rick From owner-freebsd-fs@FreeBSD.ORG Wed Aug 6 23:05:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18C27106564A for ; Wed, 6 Aug 2008 23:05:05 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id CD88B8FC0C for ; Wed, 6 Aug 2008 23:05:04 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from localhost (localhost [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id CBAFB71F079; Wed, 6 Aug 2008 19:05:03 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N91o1wJigyCw; Wed, 6 Aug 2008 19:05:03 -0400 (EDT) Received: from [35.9.44.65] (daemon.egr.msu.edu [35.9.44.65]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mcdouga9) by mx.egr.msu.edu (Postfix) with ESMTPSA id A6B4F71F059; Wed, 6 Aug 2008 19:05:03 -0400 (EDT) Message-ID: <489A2E1F.6000405@egr.msu.edu> Date: Wed, 06 Aug 2008 19:05:03 -0400 From: Adam McDougall User-Agent: Thunderbird 2.0.0.16 (X11/20080727) MIME-Version: 1.0 To: Wes Morgan References: <18585.3903.895425.122613@almost.alerce.com> <200808061928.37001.peter.schuller@infidyne.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Aug 2008 23:05:05 -0000 Wes Morgan wrote: > On Wed, 6 Aug 2008, Peter Schuller wrote: > >>> The AoC-SAT2-MV8 is based on the "Marvell Hercules-2 Rev. C0 SATA host >>> controller", which seems to be AKA 88SX6081, which is listed as >>> supported by the ata driver in 7.0-RELEASE. Has anyone had any ZFS >>> experience with it? >> >> Yes; it has been working quite fine for me with 7 up to a >> release-candidate. >> In 7.0-RELEASE you must disable the hptrr driver because it eats the >> device. >> See: >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120615 >> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120842 >> >> When I say "working fine", this is with 8 SATA drives (6 of them in a >> raidz2, >> two of them for other stuff) and not having any issues like corruption, >> timeouts or whatever else people have had with shaky controllers. >> >> However, I cannot speak to performance because it's a PCI-X card that >> I've >> plugged into a PCI slow, so throughput is limited by the PCI bus (and >> the >> machine is otherwise not the fastest to begin with). >> >> Note that this is on 32 bit; haven't been able to try it on 64 bit >> because I >> the PCI-X card wouldn't work on the motherboard (again PCI, so it's >> hit-and-miss) where I would otherwise have tried it. >> >> I'd love to find a buyable PCI-E version that also worked in >> FreeBSD... I'll >> see if the link in your post contains any such hints. > > Hmmm... That PCI-X card is interesting. Supermicro also lists this: > > http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm > > http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ics/lsisas1068e/index.html > > > Not much onboard ram, but it's PCI-E and even SAS. CDW lists it for > $155. That would be cheaper than buying a new board with a PCI-X slot > or two, and would even handle SAS drives. Claims to be based on the > "LSISAS 1068E SAS controller". Any idea if that is supported? I don't > see it listed in the mfi man page. LSI has a Linux driver for > download. That card looks like it would be just what I need. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > mpt2@pci0:8:0:0: class=0x010000 card=0x31501000 chip=0x00581000 rev=0x04 hdr=0x00 vendor = 'LSI Logic (Was: Symbios Logic, NCR)' device = 'SAS 3000 series, 8-port with 1068E -StorPort' class = mass storage subclass = SCSI mpt2: port 0xa800-0xa8ff mem 0xfcbfc000-0xfcbfffff,0xfcbe0000-0xfcbeffff irq 17 at device 0.0 on pci8 da1 at mpt2 bus 0 target 0 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing Enabled da1: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da2 at mpt2 bus 0 target 1 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 300.000MB/s transfers da2: Command Queueing Enabled da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da3 at mpt2 bus 0 target 2 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 300.000MB/s transfers da3: Command Queueing Enabled da3: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da4 at mpt2 bus 0 target 3 lun 0 da4: Fixed Direct Access SCSI-5 device da4: 300.000MB/s transfers da4: Command Queueing Enabled da4: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da5 at mpt2 bus 0 target 4 lun 0 da5: Fixed Direct Access SCSI-5 device da5: 300.000MB/s transfers da5: Command Queueing Enabled da5: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da6 at mpt2 bus 0 target 5 lun 0 da6: Fixed Direct Access SCSI-5 device da6: 300.000MB/s transfers da6: Command Queueing Enabled da6: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da7 at mpt2 bus 0 target 6 lun 0 da7: Fixed Direct Access SCSI-5 device da7: 300.000MB/s transfers da7: Command Queueing Enabled da7: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) da8 at mpt2 bus 0 target 7 lun 0 da8: Fixed Direct Access SCSI-5 device da8: 300.000MB/s transfers da8: Command Queueing Enabled da8: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 da1s1d ONLINE 0 0 0 da2s1d ONLINE 0 0 0 da3s1d ONLINE 0 0 0 da4s1d ONLINE 0 0 0 da5s1d ONLINE 0 0 0 da6s1d ONLINE 0 0 0 da7s1d ONLINE 0 0 0 da8s1d ONLINE 0 0 0 From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 04:35:14 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDA27106579C for ; Thu, 7 Aug 2008 04:35:14 +0000 (UTC) (envelope-from vyeperman@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.171]) by mx1.freebsd.org (Postfix) with ESMTP id 8D8728FC12 for ; Thu, 7 Aug 2008 04:35:14 +0000 (UTC) (envelope-from vyeperman@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so203819wfg.7 for ; Wed, 06 Aug 2008 21:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=PyslW6ZmGXZUyGkdvxEKJfIKFUHKNiw2np2ANU/elGU=; b=ZG/Lyd8UVnGkB56CYFmvD9DQ8NeCIiNFbWdM9br+AfxkS/IDHIR2cupH0E9Z9FazCc w/1sTR4LwEps5eknB4DSTWMN2nimNCjxDzLz0Efe5GL1jUM5YtlQdBke4s1mS2HVIMYQ Es24CmYybm7/pT/foRHiz/M+VBFy0wWn4bX3Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=qSWiMctKLhG1BbcAEiflrYxXGUhOtE80M4TO6AyF/U24HayrfrF6oH/JJGZnpcYdNH QRgO+fS0e5nWdc16hqkPrm+jK3xfx7U6OQR3fzlWmovvgB7rhK0E4OokHD5//hDLoC5Z klr1E03KvwAPu9NGM5sqNVxyn+w7CwAaFN7lc= Received: by 10.142.126.17 with SMTP id y17mr288197wfc.189.1218082142787; Wed, 06 Aug 2008 21:09:02 -0700 (PDT) Received: by 10.143.7.6 with HTTP; Wed, 6 Aug 2008 21:09:02 -0700 (PDT) Message-ID: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> Date: Wed, 6 Aug 2008 21:09:02 -0700 From: "Vye Wilson" To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_28177_14563589.1218082142782" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 04:35:14 -0000 ------=_Part_28177_14563589.1218082142782 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, I setup a raidz1 zpool to test ZFS with a device failure and to see how quickly the zpool could be resilvered. The system I'm using has a backplane that all the drives are connected to, so everything is hotswappable. I created the raidz1 zpool and then removed one of the drives. zpool status showed that the pool was degraded but online. Ok great, so lets bring the now functioning drive back online. [root@Touzyoh /home/vye]# zpool online ztemp ad18 Bringing device ad18 online Everything looks good... lets check the zpool status pool: ztemp state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-D3 scrub: resilver completed with 0 errors on Wed Aug 6 20:59:54 2008 config: NAME STATE READ WRITE CKSUM ztemp DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad10 ONLINE 0 0 0 ad14 ONLINE 0 0 0 ad18 UNAVAIL 0 0 0 cannot open errors: No known data errors Doh! still degraded. It shows 'UNAVAIL cannot open' I've tried rebooting but it will not open that drive at all. According to dmesg the drive is functional, and if I destroy the pool and recreate it the drive works fine. I wasn't able to find any similar issues on this mailing list or in google. Does anyone have any ideas? I've attached my dmesg output. Thanks. -- --Vye ------=_Part_28177_14563589.1218082142782 Content-Type: text/plain; name=dmesg.txt Content-Transfer-Encoding: base64 X-Attachment-Id: f_fjkuk93d0 Content-Disposition: attachment; filename=dmesg.txt Q29weXJpZ2h0IChjKSAxOTkyLTIwMDggVGhlIEZyZWVCU0QgUHJvamVjdC4KQ29weXJpZ2h0IChj KSAxOTc5LCAxOTgwLCAxOTgzLCAxOTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAx OTk0CglUaGUgUmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBDYWxpZm9ybmlhLiBBbGwgcmln aHRzIHJlc2VydmVkLgpGcmVlQlNEIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhlIEZy ZWVCU0QgRm91bmRhdGlvbi4KRnJlZUJTRCA3LjAtU1RBQkxFICMxOiBXZWQgSnVsIDMwIDEwOjI5 OjQzIFBEVCAyMDA4CiAgICB2eWVAVG91enlvaC53b3cuY29tOi91c3Ivb2JqL3Vzci9zcmMvc3lz L0dFTkVSSUMKVGltZWNvdW50ZXIgImk4MjU0IiBmcmVxdWVuY3kgMTE5MzE4MiBIeiBxdWFsaXR5 IDAKQ1BVOiBBTUQgT3B0ZXJvbih0bSkgUHJvY2Vzc29yIDI0OCAoMjIwMC4wMS1NSHogSzgtY2xh c3MgQ1BVKQogIE9yaWdpbiA9ICJBdXRoZW50aWNBTUQiICBJZCA9IDB4ZjVhICBTdGVwcGluZyA9 IDEwCiAgRmVhdHVyZXM9MHg3OGJmYmZmPEZQVSxWTUUsREUsUFNFLFRTQyxNU1IsUEFFLE1DRSxD WDgsQVBJQyxTRVAsTVRSUixQR0UsTUNBLENNT1YsUEFULFBTRTM2LENMRkxVU0gsTU1YLEZYU1Is U1NFLFNTRTI+CiAgQU1EIEZlYXR1cmVzPTB4ZTA1MDA4MDA8U1lTQ0FMTCxOWCxNTVgrLExNLDNE Tm93ISssM0ROb3chPgp1c2FibGUgbWVtb3J5ID0gMzc0NDcyNzA0MCAoMzU3MSBNQikKYXZhaWwg bWVtb3J5ICA9IDM2MjEzODQxOTIgKDM0NTMgTUIpCkFDUEkgQVBJQyBUYWJsZTogPEEgTSBJICBP RU1BUElDID4KRnJlZUJTRC9TTVA6IE11bHRpcHJvY2Vzc29yIFN5c3RlbSBEZXRlY3RlZDogMiBD UFVzCiBjcHUwIChCU1ApOiBBUElDIElEOiAgMAogY3B1MSAoQVApOiBBUElDIElEOiAgMQppb2Fw aWMwIDxWZXJzaW9uIDEuMT4gaXJxcyAwLTIzIG9uIG1vdGhlcmJvYXJkCmlvYXBpYzEgPFZlcnNp b24gMS4xPiBpcnFzIDI0LTQ3IG9uIG1vdGhlcmJvYXJkCmtiZDEgYXQga2JkbXV4MAphdGhfaGFs OiAwLjkuMjAuMyAoQVI1MjEwLCBBUjUyMTEsIEFSNTIxMiwgUkY1MTExLCBSRjUxMTIsIFJGMjQx MywgUkY1NDEzKQphY3BpMDogPEEgTSBJIE9FTVJTRFQ+IG9uIG1vdGhlcmJvYXJkCmFjcGkwOiBb SVRIUkVBRF0KYWNwaTA6IFBvd2VyIEJ1dHRvbiAoZml4ZWQpCmFjcGkwOiByZXNlcnZhdGlvbiBv ZiAxMDAwMDAsIGRmZjAwMDAwICgzKSBmYWlsZWQKYWNwaTA6IHJlc2VydmF0aW9uIG9mIGRmZWZk MDAwLCA0MDAgKDMpIGZhaWxlZAphY3BpMDogcmVzZXJ2YXRpb24gb2YgZGZlZmUwMDAsIDQwMCAo MykgZmFpbGVkCmFjcGkwOiByZXNlcnZhdGlvbiBvZiBkZmVmZjAwMCwgMTAwMCAoMykgZmFpbGVk CmFjcGkwOiByZXNlcnZhdGlvbiBvZiAwLCBhMDAwMCAoMykgZmFpbGVkClRpbWVjb3VudGVyICJB Q1BJLXNhZmUiIGZyZXF1ZW5jeSAzNTc5NTQ1IEh6IHF1YWxpdHkgODUwCmFjcGlfdGltZXIwOiA8 MjQtYml0IHRpbWVyIGF0IDMuNTc5NTQ1TUh6PiBwb3J0IDB4NDAwOC0weDQwMGIgb24gYWNwaTAK cGNpYjA6IDxBQ1BJIEhvc3QtUENJIGJyaWRnZT4gcG9ydCAweGNmOC0weGNmZiBvbiBhY3BpMApw Y2kwOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMApwY2kwOiA8bWVtb3J5PiBhdCBkZXZpY2UgMC4w IChubyBkcml2ZXIgYXR0YWNoZWQpCmlzYWIwOiA8UENJLUlTQSBicmlkZ2U+IGF0IGRldmljZSAx LjAgb24gcGNpMAppc2EwOiA8SVNBIGJ1cz4gb24gaXNhYjAKcGNpMDogPHNlcmlhbCBidXMsIFNN QnVzPiBhdCBkZXZpY2UgMS4xIChubyBkcml2ZXIgYXR0YWNoZWQpCm9oY2kwOiA8T0hDSSAoZ2Vu ZXJpYykgVVNCIGNvbnRyb2xsZXI+IG1lbSAweGZkYmZjMDAwLTB4ZmRiZmNmZmYgaXJxIDIwIGF0 IGRldmljZSAyLjAgb24gcGNpMApvaGNpMDogW0dJQU5ULUxPQ0tFRF0Kb2hjaTA6IFtJVEhSRUFE XQp1c2IwOiBPSENJIHZlcnNpb24gMS4wLCBsZWdhY3kgc3VwcG9ydAp1c2IwOiBTTU0gZG9lcyBu b3QgcmVzcG9uZCwgcmVzZXR0aW5nCnVzYjA6IDxPSENJIChnZW5lcmljKSBVU0IgY29udHJvbGxl cj4gb24gb2hjaTAKdXNiMDogVVNCIHJldmlzaW9uIDEuMAp1aHViMDogPG5WaWRpYSBPSENJIHJv b3QgaHViLCBjbGFzcyA5LzAsIHJldiAxLjAwLzEuMDAsIGFkZHIgMT4gb24gdXNiMAp1aHViMDog MTAgcG9ydHMgd2l0aCAxMCByZW1vdmFibGUsIHNlbGYgcG93ZXJlZAplaGNpMDogPE5WSURJQSBu Rm9yY2U0IFVTQiAyLjAgY29udHJvbGxlcj4gbWVtIDB4ZmRiZTAwMDAtMHhmZGJlMDBmZiBpcnEg MjEgYXQgZGV2aWNlIDIuMSBvbiBwY2kwCmVoY2kwOiBbR0lBTlQtTE9DS0VEXQplaGNpMDogW0lU SFJFQURdCnVzYjE6IEVIQ0kgdmVyc2lvbiAxLjAKdXNiMTogY29tcGFuaW9uIGNvbnRyb2xsZXIs IDQgcG9ydHMgZWFjaDogdXNiMAp1c2IxOiA8TlZJRElBIG5Gb3JjZTQgVVNCIDIuMCBjb250cm9s bGVyPiBvbiBlaGNpMAp1c2IxOiBVU0IgcmV2aXNpb24gMi4wCnVodWIxOiA8blZpZGlhIEVIQ0kg cm9vdCBodWIsIGNsYXNzIDkvMCwgcmV2IDIuMDAvMS4wMCwgYWRkciAxPiBvbiB1c2IxCnVodWIx OiAxMCBwb3J0cyB3aXRoIDEwIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkCnBjaTA6IDxtdWx0aW1l ZGlhLCBhdWRpbz4gYXQgZGV2aWNlIDQuMCAobm8gZHJpdmVyIGF0dGFjaGVkKQphdGFwY2kwOiA8 blZpZGlhIG5Gb3JjZSBDSzgwNCBVRE1BMTMzIGNvbnRyb2xsZXI+IHBvcnQgMHgxZjAtMHgxZjcs MHgzZjYsMHgxNzAtMHgxNzcsMHgzNzYsMHgzMDAwLTB4MzAwZiBhdCBkZXZpY2UgNi4wIG9uIHBj aTAKYXRhMDogPEFUQSBjaGFubmVsIDA+IG9uIGF0YXBjaTAKYXRhMDogW0lUSFJFQURdCmF0YTE6 IDxBVEEgY2hhbm5lbCAxPiBvbiBhdGFwY2kwCmF0YTE6IFtJVEhSRUFEXQphdGFwY2kxOiA8blZp ZGlhIG5Gb3JjZSBDSzgwNCBTQVRBMzAwIGNvbnRyb2xsZXI+IHBvcnQgMHhjYzAwLTB4Y2MwNyww eGM4MDAtMHhjODAzLDB4YzQwMC0weGM0MDcsMHhjMDAwLTB4YzAwMywweGJjMDAtMHhiYzBmIG1l bSAweGZkYmZmMDAwLTB4ZmRiZmZmZmYgaXJxIDIwIGF0IGRldmljZSA3LjAgb24gcGNpMAphdGFw Y2kxOiBbSVRIUkVBRF0KYXRhMjogPEFUQSBjaGFubmVsIDA+IG9uIGF0YXBjaTEKYXRhMjogW0lU SFJFQURdCmF0YTM6IDxBVEEgY2hhbm5lbCAxPiBvbiBhdGFwY2kxCmF0YTM6IFtJVEhSRUFEXQph dGFwY2kyOiA8blZpZGlhIG5Gb3JjZSBDSzgwNCBTQVRBMzAwIGNvbnRyb2xsZXI+IHBvcnQgMHhi ODAwLTB4YjgwNywweGI0MDAtMHhiNDAzLDB4YjAwMC0weGIwMDcsMHhhYzAwLTB4YWMwMywweGE4 MDAtMHhhODBmIG1lbSAweGZkYmZlMDAwLTB4ZmRiZmVmZmYgaXJxIDIxIGF0IGRldmljZSA4LjAg b24gcGNpMAphdGFwY2kyOiBbSVRIUkVBRF0KYXRhNDogPEFUQSBjaGFubmVsIDA+IG9uIGF0YXBj aTIKYXRhNDogW0lUSFJFQURdCmF0YTU6IDxBVEEgY2hhbm5lbCAxPiBvbiBhdGFwY2kyCmF0YTU6 IFtJVEhSRUFEXQpwY2liMTogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSA5LjAgb24g cGNpMApwY2k1OiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMQp2Z2FwY2kwOiA8VkdBLWNvbXBhdGli bGUgZGlzcGxheT4gcG9ydCAweDk4MDAtMHg5OGZmIG1lbSAweGZjMDAwMDAwLTB4ZmNmZmZmZmYs MHhmZGFmZjAwMC0weGZkYWZmZmZmIGlycSAxNiBhdCBkZXZpY2UgNC4wIG9uIHBjaTUKcGNpYjI6 IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMTEuMCBvbiBwY2kwCnBjaTQ6IDxBQ1BJ IFBDSSBidXM+IG9uIHBjaWIyCnBjaWIzOiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNl IDEyLjAgb24gcGNpMApwY2kyOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMwpiZ2UwOiA8QnJvYWRj b20gTmV0WHRyZW1lIEdpZ2FiaXQgRXRoZXJuZXQgQ29udHJvbGxlciwgQVNJQyByZXYuIDB4NDEw MT4gbWVtIDB4ZmI4ZjAwMDAtMHhmYjhmZmZmZiBpcnEgMTcgYXQgZGV2aWNlIDAuMCBvbiBwY2ky Cm1paWJ1czA6IDxNSUkgYnVzPiBvbiBiZ2UwCmJyZ3BoeTA6IDxCQ001NzUwIDEwLzEwMC8xMDAw YmFzZVRYIFBIWT4gUEhZIDEgb24gbWlpYnVzMApicmdwaHkwOiAgMTBiYXNlVCwgMTBiYXNlVC1G RFgsIDEwMGJhc2VUWCwgMTAwYmFzZVRYLUZEWCwgMTAwMGJhc2VULCAxMDAwYmFzZVQtRkRYLCBh dXRvCmJnZTA6IEV0aGVybmV0IGFkZHJlc3M6IDAwOjMwOjQ4OjU3OjVhOmRlCmJnZTA6IFtJVEhS RUFEXQpwY2liNDogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSAxMy4wIG9uIHBjaTAK cGNpMzogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjQKYmdlMTogPEJyb2FkY29tIE5ldFh0cmVtZSBH aWdhYml0IEV0aGVybmV0IENvbnRyb2xsZXIsIEFTSUMgcmV2LiAweDQxMDE+IG1lbSAweGZiOWYw MDAwLTB4ZmI5ZmZmZmYgaXJxIDE4IGF0IGRldmljZSAwLjAgb24gcGNpMwptaWlidXMxOiA8TUlJ IGJ1cz4gb24gYmdlMQpicmdwaHkxOiA8QkNNNTc1MCAxMC8xMDAvMTAwMGJhc2VUWCBQSFk+IFBI WSAxIG9uIG1paWJ1czEKYnJncGh5MTogIDEwYmFzZVQsIDEwYmFzZVQtRkRYLCAxMDBiYXNlVFgs IDEwMGJhc2VUWC1GRFgsIDEwMDBiYXNlVCwgMTAwMGJhc2VULUZEWCwgYXV0bwpiZ2UxOiBFdGhl cm5ldCBhZGRyZXNzOiAwMDozMDo0ODo1Nzo1YTpkZgpiZ2UxOiBbSVRIUkVBRF0KcGNpYjU6IDxB Q1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMTQuMCBvbiBwY2kwCnBjaTE6IDxBQ1BJIFBD SSBidXM+IG9uIHBjaWI1CnBjaWI2OiA8QUNQSSBIb3N0LVBDSSBicmlkZ2U+IG9uIGFjcGkwCnBj aTEyODogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjYKcGNpMTI4OiA8bWVtb3J5PiBhdCBkZXZpY2Ug MC4wIChubyBkcml2ZXIgYXR0YWNoZWQpCnBjaTEyODogPG1lbW9yeT4gYXQgZGV2aWNlIDEuMCAo bm8gZHJpdmVyIGF0dGFjaGVkKQphdGFwY2kzOiA8blZpZGlhIG5Gb3JjZSBDSzgwNCBTQVRBMzAw IGNvbnRyb2xsZXI+IHBvcnQgMHhmYzAwLTB4ZmMwNywweGY4MDAtMHhmODAzLDB4ZjQwMC0weGY0 MDcsMHhmMDAwLTB4ZjAwMywweGVjMDAtMHhlYzBmIG1lbSAweGZkZWZmMDAwLTB4ZmRlZmZmZmYg aXJxIDQ0IGF0IGRldmljZSA3LjAgb24gcGNpMTI4CmF0YXBjaTM6IFtJVEhSRUFEXQphdGE2OiA8 QVRBIGNoYW5uZWwgMD4gb24gYXRhcGNpMwphdGE2OiBbSVRIUkVBRF0KYXRhNzogPEFUQSBjaGFu bmVsIDE+IG9uIGF0YXBjaTMKYXRhNzogW0lUSFJFQURdCmF0YXBjaTQ6IDxuVmlkaWEgbkZvcmNl IENLODA0IFNBVEEzMDAgY29udHJvbGxlcj4gcG9ydCAweGU4MDAtMHhlODA3LDB4ZTQwMC0weGU0 MDMsMHhlMDAwLTB4ZTAwNywweGRjMDAtMHhkYzAzLDB4ZDgwMC0weGQ4MGYgbWVtIDB4ZmRlZmUw MDAtMHhmZGVmZWZmZiBpcnEgNDUgYXQgZGV2aWNlIDguMCBvbiBwY2kxMjgKYXRhcGNpNDogW0lU SFJFQURdCmF0YTg6IDxBVEEgY2hhbm5lbCAwPiBvbiBhdGFwY2k0CmF0YTg6IFtJVEhSRUFEXQph dGE5OiA8QVRBIGNoYW5uZWwgMT4gb24gYXRhcGNpNAphdGE5OiBbSVRIUkVBRF0KcGNpYjc6IDxB Q1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMTMuMCBvbiBwY2kxMjgKcGNpMTMyOiA8QUNQ SSBQQ0kgYnVzPiBvbiBwY2liNwpwY2liODogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmlj ZSAxNC4wIG9uIHBjaTEyOApwY2kxMjk6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWI4CnBjaWI5OiA8 UENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSAwLjAgb24gcGNpMTI5CnBjaTEzMTogPFBDSSBidXM+ IG9uIHBjaWI5CmFyY21zcjA6IDxBcmVjYSBTQVRBIEhvc3QgQWRhcHRlciBSQUlEIENvbnRyb2xs ZXIgKFJBSUQ2IGNhcGFibGUpCj4gbWVtIDB4ZmRkZmYwMDAtMHhmZGRmZmZmZiwweGZlMDAwMDAw LTB4ZmUzZmZmZmYgaXJxIDQwIGF0IGRldmljZSAxNC4wIG9uIHBjaTEzMQpBUkVDQSBSQUlEIEFE QVBURVIwOiBEcml2ZXIgVmVyc2lvbiAxLjIwLjAwLjE1IDIwMDctMTAtMDcgCkFSRUNBIFJBSUQg QURBUFRFUjA6IEZJUk1XQVJFIFZFUlNJT04gVjEuNDEgMjAwNi01LTI0ICAKYXJjbXNyMDogW0lU SFJFQURdCnBjaWIxMDogPFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMC4yIG9uIHBjaTEyOQpw Y2kxMzA6IDxQQ0kgYnVzPiBvbiBwY2liMTAKY3B1MDogPEFDUEkgQ1BVPiBvbiBhY3BpMApwb3dl cm5vdzA6IDxDb29sYG4nUXVpZXQgSzg+IG9uIGNwdTAKZGV2aWNlX2F0dGFjaDogcG93ZXJub3cw IGF0dGFjaCByZXR1cm5lZCA2CmNwdTE6IDxBQ1BJIENQVT4gb24gYWNwaTAKcG93ZXJub3cxOiA8 Q29vbGBuJ1F1aWV0IEs4PiBvbiBjcHUxCmRldmljZV9hdHRhY2g6IHBvd2Vybm93MSBhdHRhY2gg cmV0dXJuZWQgNgphY3BpX2J1dHRvbjA6IDxQb3dlciBCdXR0b24+IG9uIGFjcGkwCnNpbzA6IGNv bmZpZ3VyZWQgaXJxIDQgbm90IGluIGJpdG1hcCBvZiBwcm9iZWQgaXJxcyAwCnNpbzA6IHBvcnQg bWF5IG5vdCBiZSBlbmFibGVkCnNpbzA6IGNvbmZpZ3VyZWQgaXJxIDQgbm90IGluIGJpdG1hcCBv ZiBwcm9iZWQgaXJxcyAwCnNpbzA6IHBvcnQgbWF5IG5vdCBiZSBlbmFibGVkCnNpbzA6IDwxNjU1 MEEtY29tcGF0aWJsZSBDT00gcG9ydD4gcG9ydCAweDNmOC0weDNmZiBpcnEgNCBmbGFncyAweDEw IG9uIGFjcGkwCnNpbzA6IHR5cGUgMTY1NTBBCnNpbzA6IFtGSUxURVJdCnNpbzE6IGNvbmZpZ3Vy ZWQgaXJxIDMgbm90IGluIGJpdG1hcCBvZiBwcm9iZWQgaXJxcyAwCnNpbzE6IHBvcnQgbWF5IG5v dCBiZSBlbmFibGVkCnNpbzE6IGNvbmZpZ3VyZWQgaXJxIDMgbm90IGluIGJpdG1hcCBvZiBwcm9i ZWQgaXJxcyAwCnNpbzE6IHBvcnQgbWF5IG5vdCBiZSBlbmFibGVkCnNpbzE6IDwxNjU1MEEtY29t cGF0aWJsZSBDT00gcG9ydD4gcG9ydCAweDJmOC0weDJmZiBpcnEgMyBvbiBhY3BpMApzaW8xOiB0 eXBlIDE2NTUwQQpzaW8xOiBbRklMVEVSXQpmZGMwOiA8ZmxvcHB5IGRyaXZlIGNvbnRyb2xsZXIg KEZERSk+IHBvcnQgMHgzZjAtMHgzZjUsMHgzZjcgaXJxIDYgZHJxIDIgb24gYWNwaTAKZmRjMDog W0ZJTFRFUl0KcHBjMDogPFBhcmFsbGVsIHBvcnQ+IHBvcnQgMHgzNzgtMHgzN2YgaXJxIDcgb24g YWNwaTAKcHBjMDogR2VuZXJpYyBjaGlwc2V0IChOSUJCTEUtb25seSkgaW4gQ09NUEFUSUJMRSBt b2RlCnBwYnVzMDogPFBhcmFsbGVsIHBvcnQgYnVzPiBvbiBwcGMwCnBwYnVzMDogW0lUSFJFQURd CnBsaXAwOiA8UExJUCBuZXR3b3JrIGludGVyZmFjZT4gb24gcHBidXMwCmxwdDA6IDxQcmludGVy PiBvbiBwcGJ1czAKbHB0MDogSW50ZXJydXB0LWRyaXZlbiBwb3J0CnBwaTA6IDxQYXJhbGxlbCBJ L08+IG9uIHBwYnVzMApwcGMwOiBbR0lBTlQtTE9DS0VEXQpwcGMwOiBbSVRIUkVBRF0KYXRrYmRj MDogPEtleWJvYXJkIGNvbnRyb2xsZXIgKGk4MDQyKT4gcG9ydCAweDYwLDB4NjQgaXJxIDEgb24g YWNwaTAKYXRrYmQwOiA8QVQgS2V5Ym9hcmQ+IGlycSAxIG9uIGF0a2JkYzAKa2JkMCBhdCBhdGti ZDAKYXRrYmQwOiBbR0lBTlQtTE9DS0VEXQphdGtiZDA6IFtJVEhSRUFEXQpvcm0wOiA8SVNBIE9w dGlvbiBST01zPiBhdCBpb21lbSAweGMwMDAwLTB4YzdmZmYsMHhjODAwMC0weGM5N2ZmLDB4Yzk4 MDAtMHhjYWZmZiwweGNiMDAwLTB4Y2JmZmYgb24gaXNhMApzYzA6IDxTeXN0ZW0gY29uc29sZT4g YXQgZmxhZ3MgMHgxMDAgb24gaXNhMApzYzA6IFZHQSA8MTYgdmlydHVhbCBjb25zb2xlcywgZmxh Z3M9MHgzMDA+CnZnYTA6IDxHZW5lcmljIElTQSBWR0E+IGF0IHBvcnQgMHgzYzAtMHgzZGYgaW9t ZW0gMHhhMDAwMC0weGJmZmZmIG9uIGlzYTAKVGltZWNvdW50ZXJzIHRpY2sgZXZlcnkgMS4wMDAg bXNlYwphZDY6IDE1MjYyN01CIDxTZWFnYXRlIFNUMzE2MDgxMUFTIDMuQUFFPiBhdCBhdGEzLW1h c3RlciBTQVRBMTUwCmFkMTA6IDk1Mzg2OU1CIDxTQU1TVU5HIEhEMTAzVUogMUFBMDExMDk+IGF0 IGF0YTUtbWFzdGVyIFNBVEEzMDAKYWQxNDogMjg2MTY4TUIgPFNlYWdhdGUgU1QzMzAwNjIwQVMg My5BQUU+IGF0IGF0YTctbWFzdGVyIFNBVEExNTAKYWQxODogMjg2MTY4TUIgPFNlYWdhdGUgU1Qz MzAwNjIwQVMgMy5BQUU+IGF0IGF0YTktbWFzdGVyIFNBVEExNTAKV2FpdGluZyA1IHNlY29uZHMg Zm9yIFNDU0kgZGV2aWNlcyB0byBzZXR0bGUKKHByb2JlMTY6YXJjbXNyMDowOjE2OjApOiBpbnF1 aXJ5IGRhdGEgZmFpbHMgY29tcGFyaXNvbiBhdCBEVjEgc3RlcApkYTAgYXQgYXJjbXNyMCBidXMg MCB0YXJnZXQgMCBsdW4gMApkYTA6IDxBcmVjYSBBUkMtMTI2MC1WT0wjMDAgUjAwMT4gRml4ZWQg RGlyZWN0IEFjY2VzcyBTQ1NJLTMgZGV2aWNlIApkYTA6IDE2Ni42NjZNQi9zIHRyYW5zZmVycyAo ODMuMzMzTUh6IERULCBvZmZzZXQgMzIsIDE2Yml0KQpkYTA6IDI4NjEwMk1CICg1ODU5Mzc0MDgg NTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAzNjQ3MkMpCmRhMSBhdCBhcmNtc3IwIGJ1cyAw IHRhcmdldCAwIGx1biAxCmRhMTogPEFyZWNhIEFSQy0xMjYwLVZPTCMwMSBSMDAxPiBGaXhlZCBE aXJlY3QgQWNjZXNzIFNDU0ktMyBkZXZpY2UgCmRhMTogMTY2LjY2Nk1CL3MgdHJhbnNmZXJzICg4 My4zMzNNSHogRFQsIG9mZnNldCAzMiwgMTZiaXQpCmRhMTogOTUzNjc0TUIgKDE5NTMxMjQzNTIg NTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCAxMjE1NzZDKQpkYTIgYXQgYXJjbXNyMCBidXMg MCB0YXJnZXQgMCBsdW4gMgpkYTI6IDxBcmVjYSBBUkMtMTI2MC1WT0wjMDIgUjAwMT4gRml4ZWQg RGlyZWN0IEFjY2VzcyBTQ1NJLTMgZGV2aWNlIApkYTI6IDE2Ni42NjZNQi9zIHRyYW5zZmVycyAo ODMuMzMzTUh6IERULCBvZmZzZXQgMzIsIDE2Yml0KQpkYTI6IDU3MjIwNE1CICgxMTcxODc0ODE2 IDUxMiBieXRlIHNlY3RvcnM6IDI1NUggNjNTL1QgNzI5NDVDKQpTTVA6IEFQIENQVSAjMSBMYXVu Y2hlZCEKVHJ5aW5nIHRvIG1vdW50IHJvb3QgZnJvbSB1ZnM6L2Rldi9kYTBzMWEKV0FSTklORzog WkZTIGlzIGNvbnNpZGVyZWQgdG8gYmUgYW4gZXhwZXJpbWVudGFsIGZlYXR1cmUgaW4gRnJlZUJT RC4KWkZTIGZpbGVzeXN0ZW0gdmVyc2lvbiA2ClpGUyBzdG9yYWdlIHBvb2wgdmVyc2lvbiA2Cgo= ------=_Part_28177_14563589.1218082142782-- From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 04:47:59 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B774A106567D for ; Thu, 7 Aug 2008 04:47:59 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id A64F38FC2E for ; Thu, 7 Aug 2008 04:47:59 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 88A931CC0B0; Wed, 6 Aug 2008 21:47:59 -0700 (PDT) Date: Wed, 6 Aug 2008 21:47:59 -0700 From: Jeremy Chadwick To: Vye Wilson Message-ID: <20080807044759.GA7505@eos.sc1.parodius.com> References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 04:47:59 -0000 On Wed, Aug 06, 2008 at 09:09:02PM -0700, Vye Wilson wrote: > Hello, > > I setup a raidz1 zpool to test ZFS with a device failure and to see how > quickly the zpool could be resilvered. The system I'm using has a backplane > that all the drives are connected to, so everything is hotswappable. I > created the raidz1 zpool and then removed one of the drives. zpool status > showed that the pool was degraded but online. Ok great, so lets bring the > now functioning drive back online. > > [root@Touzyoh /home/vye]# zpool online ztemp ad18 > Bringing device ad18 online > > Everything looks good... lets check the zpool status > > pool: ztemp > state: DEGRADED > status: One or more devices could not be opened. Sufficient replicas exist > for > the pool to continue functioning in a degraded state. > action: Attach the missing device and online it using 'zpool online'. > see: http://www.sun.com/msg/ZFS-8000-D3 > scrub: resilver completed with 0 errors on Wed Aug 6 20:59:54 2008 > config: > > NAME STATE READ WRITE CKSUM > ztemp DEGRADED 0 0 0 > raidz1 DEGRADED 0 0 0 > ad10 ONLINE 0 0 0 > ad14 ONLINE 0 0 0 > ad18 UNAVAIL 0 0 0 cannot open > > errors: No known data errors > > Doh! still degraded. It shows 'UNAVAIL cannot open' I've tried rebooting but > it will not open that drive at all. According to dmesg the drive is > functional, and if I destroy the pool and recreate it the drive works fine. > I wasn't able to find any similar issues on this mailing list or in google. > Does anyone have any ideas? I've attached my dmesg output. What was in your dmesg when you yanked the disk? What was in your dmesg when you re-inserted the disk? Did you try detaching it administratively using "atacontrol detach" first, then retaching it using "atacontrol attach"? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 05:12:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A6A01065679 for ; Thu, 7 Aug 2008 05:12:02 +0000 (UTC) (envelope-from vyeperman@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.168]) by mx1.freebsd.org (Postfix) with ESMTP id 498FB8FC0A for ; Thu, 7 Aug 2008 05:12:02 +0000 (UTC) (envelope-from vyeperman@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so213657wfg.7 for ; Wed, 06 Aug 2008 22:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=YFEdZAVZCTjxR/5V/JMMV1E+hKSFU4vFXmLQS5dsRbE=; b=rkvmdI8WYVFXVnsc4qKKsnY3xzHnjUd869o+1c2p/EKxgOZyuVFZ0aCjj5+nQE48/k 4ZHy1/Fj8sJvS63kgeIoPvUMeMsOLg3SNICbN2GjVzgJHo2+fyVCrkBeGQBl5j1hN8C8 K18BuJcAgiIZmjVkwfFLUOaSDcUOI2j2oRKh0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=D5IyMWCK2o+/HASO/LaU3kwOno2XahkUktKmp8Hu11oxgd7jJkemf4V2nyA7/sDgWx OdVGqHO5C5feSyucb8nuU5qZ8roqllqFA4DWZ/EfczcwTjz7lCWvTY6LgxgXDf5Dqh9P aR+Y4wobAdpMkynGGcIXrWtZzorfxNSYTpEGY= Received: by 10.142.212.19 with SMTP id k19mr311311wfg.142.1218085921917; Wed, 06 Aug 2008 22:12:01 -0700 (PDT) Received: by 10.143.7.6 with HTTP; Wed, 6 Aug 2008 22:12:01 -0700 (PDT) Message-ID: <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> Date: Wed, 6 Aug 2008 22:12:01 -0700 From: "Vye Wilson" To: "Jeremy Chadwick" In-Reply-To: <20080807044759.GA7505@eos.sc1.parodius.com> MIME-Version: 1.0 References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 05:12:02 -0000 When I physically disconnected the disk it showed: subdisk18: detached ad18: detached There was nothing in dmesg after plugging the disk back in but atacontrol showed it on channel 9: ATA channel 9: Master: ad18 Serial ATA v1.0 Slave: no device present After detaching and reattaching the device with atacontrol this is what dmesg had to say: subdisk18: detached ad18: detached ata9: [ITHREAD] ad18: 286168MB at ata9-master SATA150 zpool is still saying it cannot read from ad18 after detaching and reattaching with atacontrol. However, I remade the zpool and instead of physically removing the drive I just used the atacontrol detatch/attach and it was able to resilver without any issues. That helps but what if I do have a drive failure? I shouldn't need to halt the system to switch out the drive. Thanks. On Wed, Aug 6, 2008 at 9:47 PM, Jeremy Chadwick wrote: > On Wed, Aug 06, 2008 at 09:09:02PM -0700, Vye Wilson wrote: > > Hello, > > > > I setup a raidz1 zpool to test ZFS with a device failure and to see how > > quickly the zpool could be resilvered. The system I'm using has a > backplane > > that all the drives are connected to, so everything is hotswappable. I > > created the raidz1 zpool and then removed one of the drives. zpool status > > showed that the pool was degraded but online. Ok great, so lets bring the > > now functioning drive back online. > > > > [root@Touzyoh /home/vye]# zpool online ztemp ad18 > > Bringing device ad18 online > > > > Everything looks good... lets check the zpool status > > > > pool: ztemp > > state: DEGRADED > > status: One or more devices could not be opened. Sufficient replicas > exist > > for > > the pool to continue functioning in a degraded state. > > action: Attach the missing device and online it using 'zpool online'. > > see: http://www.sun.com/msg/ZFS-8000-D3 > > scrub: resilver completed with 0 errors on Wed Aug 6 20:59:54 2008 > > config: > > > > NAME STATE READ WRITE CKSUM > > ztemp DEGRADED 0 0 0 > > raidz1 DEGRADED 0 0 0 > > ad10 ONLINE 0 0 0 > > ad14 ONLINE 0 0 0 > > ad18 UNAVAIL 0 0 0 cannot open > > > > errors: No known data errors > > > > Doh! still degraded. It shows 'UNAVAIL cannot open' I've tried rebooting > but > > it will not open that drive at all. According to dmesg the drive is > > functional, and if I destroy the pool and recreate it the drive works > fine. > > I wasn't able to find any similar issues on this mailing list or in > google. > > Does anyone have any ideas? I've attached my dmesg output. > > What was in your dmesg when you yanked the disk? What was in your dmesg > when you re-inserted the disk? > > Did you try detaching it administratively using "atacontrol detach" > first, then retaching it using "atacontrol attach"? > > -- > | Jeremy Chadwick jdc at parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP: 4BD6C0CB | > > -- --Vye From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 05:35:55 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A39E110656CC for ; Thu, 7 Aug 2008 05:35:55 +0000 (UTC) (envelope-from vyeperman@gmail.com) Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.170]) by mx1.freebsd.org (Postfix) with ESMTP id 70DFE8FC34 for ; Thu, 7 Aug 2008 05:35:55 +0000 (UTC) (envelope-from vyeperman@gmail.com) Received: by wf-out-1314.google.com with SMTP id 24so220331wfg.7 for ; Wed, 06 Aug 2008 22:35:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=g8LUAIC9Hs4lF8W/MqSP3hAKtA6w59NOaGYQoorTw08=; b=chAd+gqibkdjlnAxX8F2b7xy9K3FxOZmuOZmSoAx0STUsPC7gW8vJZ5U1DCFU1sCwl hND69auNQzPmQxNFkicxvCk2+h5imM8mgXxpwGAsQCWubJgLsmM/Qa5A7Z88OSnoJOrv luPMSDOlurqerIQvXNqVCnjQz/FBXsC/Gh4B0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=V+5bKB4IQQcaaYLGdcmuEx+RDBdrcs5FOYGVyzYcdz3+/0tWFXLYZ9MWd+NqGH/2uu bKO6dLV4Y2TBmtyjQYAuYQjCw4C/sWnyarGMV/t9QbcUc6IymZPX+BgZy8usBMosHET9 OvcGe/e11jAlgjjx+LMjZZm15OPw39DpO84QE= Received: by 10.142.105.13 with SMTP id d13mr311830wfc.275.1218087354870; Wed, 06 Aug 2008 22:35:54 -0700 (PDT) Received: by 10.143.7.6 with HTTP; Wed, 6 Aug 2008 22:35:54 -0700 (PDT) Message-ID: <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> Date: Wed, 6 Aug 2008 22:35:54 -0700 From: "Vye Wilson" To: "Jeremy Chadwick" In-Reply-To: <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> MIME-Version: 1.0 References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 05:35:55 -0000 I tested it again and it seems I was mistaken in my last email. If the drive is removed manually it will _not_ show up in atacontrol. atacontrol attach ata3 atacontrol: ioctl(IOCATAATTACH): File exists [root@Touzyoh /home/vye]# atacontrol list ATA channel 3: Master: no device present Slave: no device present [root@Touzyoh /home/vye]# atacontrol detach ata6 [root@Touzyoh /home/vye]# atacontrol attach ata6 Master: no device present Slave: no device present I'm not sure what happened the first few times but if I reboot the server the device will show up in atacontrol and the zpool will resilver. This doesn't seem to be a ZFS problem like I originally thought. Thanks. On Wed, Aug 6, 2008 at 10:12 PM, Vye Wilson wrote: > When I physically disconnected the disk it showed: > > subdisk18: detached > ad18: detached > > There was nothing in dmesg after plugging the disk back in but atacontrol > showed it on channel 9: > > ATA channel 9: > Master: ad18 Serial ATA v1.0 > Slave: no device present > > After detaching and reattaching the device with atacontrol this is what > dmesg had to say: > subdisk18: detached > ad18: detached > ata9: [ITHREAD] > ad18: 286168MB at ata9-master SATA150 > > zpool is still saying it cannot read from ad18 after detaching and > reattaching with atacontrol. However, I remade the zpool and instead of > physically removing the drive I just used the atacontrol detatch/attach and > it was able to resilver without any issues. That helps but what if I do have > a drive failure? I shouldn't need to halt the system to switch out the > drive. > > Thanks. > > > On Wed, Aug 6, 2008 at 9:47 PM, Jeremy Chadwick wrote: > >> On Wed, Aug 06, 2008 at 09:09:02PM -0700, Vye Wilson wrote: >> > Hello, >> > >> > I setup a raidz1 zpool to test ZFS with a device failure and to see how >> > quickly the zpool could be resilvered. The system I'm using has a >> backplane >> > that all the drives are connected to, so everything is hotswappable. I >> > created the raidz1 zpool and then removed one of the drives. zpool >> status >> > showed that the pool was degraded but online. Ok great, so lets bring >> the >> > now functioning drive back online. >> > >> > [root@Touzyoh /home/vye]# zpool online ztemp ad18 >> > Bringing device ad18 online >> > >> > Everything looks good... lets check the zpool status >> > >> > pool: ztemp >> > state: DEGRADED >> > status: One or more devices could not be opened. Sufficient replicas >> exist >> > for >> > the pool to continue functioning in a degraded state. >> > action: Attach the missing device and online it using 'zpool online'. >> > see: http://www.sun.com/msg/ZFS-8000-D3 >> > scrub: resilver completed with 0 errors on Wed Aug 6 20:59:54 2008 >> > config: >> > >> > NAME STATE READ WRITE CKSUM >> > ztemp DEGRADED 0 0 0 >> > raidz1 DEGRADED 0 0 0 >> > ad10 ONLINE 0 0 0 >> > ad14 ONLINE 0 0 0 >> > ad18 UNAVAIL 0 0 0 cannot open >> > >> > errors: No known data errors >> > >> > Doh! still degraded. It shows 'UNAVAIL cannot open' I've tried rebooting >> but >> > it will not open that drive at all. According to dmesg the drive is >> > functional, and if I destroy the pool and recreate it the drive works >> fine. >> > I wasn't able to find any similar issues on this mailing list or in >> google. >> > Does anyone have any ideas? I've attached my dmesg output. >> >> What was in your dmesg when you yanked the disk? What was in your dmesg >> when you re-inserted the disk? >> >> Did you try detaching it administratively using "atacontrol detach" >> first, then retaching it using "atacontrol attach"? >> >> -- >> | Jeremy Chadwick jdc at parodius.com | >> | Parodius Networking http://www.parodius.com/ | >> | UNIX Systems Administrator Mountain View, CA, USA | >> | Making life hard for others since 1977. PGP: 4BD6C0CB | >> >> > > > -- > --Vye > -- --Vye From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 05:58:41 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 772C5106566B for ; Thu, 7 Aug 2008 05:58:41 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 63F608FC12 for ; Thu, 7 Aug 2008 05:58:41 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 453F81CC0B1; Wed, 6 Aug 2008 22:58:41 -0700 (PDT) Date: Wed, 6 Aug 2008 22:58:41 -0700 From: Jeremy Chadwick To: Vye Wilson Message-ID: <20080807055841.GB9735@eos.sc1.parodius.com> References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 05:58:41 -0000 On Wed, Aug 06, 2008 at 10:35:54PM -0700, Vye Wilson wrote: > I tested it again and it seems I was mistaken in my last email. If the drive > is removed manually it will _not_ show up in atacontrol. > > atacontrol attach ata3 > atacontrol: ioctl(IOCATAATTACH): File exists > [root@Touzyoh /home/vye]# atacontrol list > ATA channel 3: > Master: no device present > Slave: no device present > [root@Touzyoh /home/vye]# atacontrol detach ata6 > [root@Touzyoh /home/vye]# atacontrol attach ata6 > Master: no device present > Slave: no device present > > I'm not sure what happened the first few times but if I reboot the server > the device will show up in atacontrol and the zpool will resilver. This > doesn't seem to be a ZFS problem like I originally thought. Correct, it's a FreeBSD ATA subsystem/driver problem. I'm actually amazed your kernel didn't panic when you did the 2nd attach. The first attach returning "File exists" doesn't surprise me either. I've been following these kinds of issues with ATA for many months now, so I want you to know that you're not alone. It is NOT specific to your nVidia controller either; others see the same thing on Intel ICH, and VIA. If you attach those SATA disks to your Areca controller and perform the exact same tests (you'll need to use camcontrol, of course), I can almost guarantee you things will behave 100% correctly. My advice at this point in time, because as of today I have officially lost faith in it: avoid ata(4) at all costs. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 06:44:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A457106566B for ; Thu, 7 Aug 2008 06:44:45 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp15.yandex.ru (smtp15.yandex.ru [77.88.32.85]) by mx1.freebsd.org (Postfix) with ESMTP id BC5568FC1C for ; Thu, 7 Aug 2008 06:44:44 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([77.72.136.145]:48377 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S1377654AbYHGGdj (ORCPT ); Thu, 7 Aug 2008 10:33:39 +0400 X-Yandex-Spam: 1 X-Yandex-Front: smtp15 X-Yandex-TimeMark: 1218090819 X-MsgDayCount: 4 X-Comment: RFC 2476 MSA function at smtp15.yandex.ru logged sender identity as: bu7cher Message-ID: <489A9739.20707@yandex.ru> Date: Thu, 07 Aug 2008 10:33:29 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Jeremy Chadwick References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> In-Reply-To: <20080807055841.GB9735@eos.sc1.parodius.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 06:44:45 -0000 Jeremy Chadwick wrote: > Correct, it's a FreeBSD ATA subsystem/driver problem. I tried 8.0-CURRENT on marvell's, nvida's and intel's controllers. Hot plug and attach/detach works on any of these controllers without any problems.. What i should to do to get similar problems? :) > My advice at this point in time, because as of today I have officially > lost faith in it: avoid ata(4) at all costs. I tried to contact you some time ago, but didn't receive any answers.. Do you still want to resolve your problems with ATA? -- WBR, Andrey V. Elsukov From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 07:14:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28D97106564A for ; Thu, 7 Aug 2008 07:14:35 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 0B1538FC0A for ; Thu, 7 Aug 2008 07:14:34 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id BBEC51CC0AD; Thu, 7 Aug 2008 00:14:34 -0700 (PDT) Date: Thu, 7 Aug 2008 00:14:34 -0700 From: Jeremy Chadwick To: "Andrey V. Elsukov" Message-ID: <20080807071434.GA15465@eos.sc1.parodius.com> References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <489A9739.20707@yandex.ru> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 07:14:35 -0000 On Thu, Aug 07, 2008 at 10:33:29AM +0400, Andrey V. Elsukov wrote: > Jeremy Chadwick wrote: >> Correct, it's a FreeBSD ATA subsystem/driver problem. > > I tried 8.0-CURRENT on marvell's, nvida's and intel's controllers. > Hot plug and attach/detach works on any of these controllers without > any problems.. What i should to do to get similar problems? :) I haven't tried CURRENT; I don't track HEAD. I will work on setting up another testbed environment at home and repeating my tests on HEAD. That will take me some time, however. My test method is very simple, at least in regards to disk removal. Here's the step-by-step I've used to hit the bugs in question: http://lists.freebsd.org/pipermail/freebsd-stable/2008-February/040534.html >> My advice at this point in time, because as of today I have officially >> lost faith in it: avoid ata(4) at all costs. > > I tried to contact you some time ago, but didn't receive any > answers.. Do you still want to resolve your problems with ATA? Yes, I did receive your mails, but you just wanted to know "if I was still having problems". I should have replied, but I did not. That is my fault, and for that I apologise. The issues aren't problems specific to me -- they are affecting a significant userbase, specifically folks who use servers in production environments. But maybe I've misunderstood what you meant by "your problems" -- my apologies if I have. But have you looked at my Wiki page, documenting most (but not all) of the issues? http://wiki.freebsd.org/JeremyChadwick/ATA_issues_and_troubleshooting We still don't have an answer to the famous "DMA timeout issue", which continues to haunt many. I provided a small analysis in my Wiki, but the technical justification is over my head -- it needs review from someone who is familiar with the ATA protocol. I inteprete the NID_NOT_FOUND error to mean FreeBSD is asking the disk to r/w to/from an invalid LBA. I received one mail from a user (I forget if a mailing list was CC'd or not -- I need to dig up the mail) who said that in some cases NID_NOT_FOUND is normal. The FreeNAS folks reported that increasing the internal ATA command timeout from 5 seconds to 10 or 15 has helped (FreeNAS users), but those on FreeBSD who suffer from said timeouts and have tried the patches said they have made no difference. That said, I have some questions: 1) Are you trying to tell me that individuals running commercial services in production environments should run CURRENT? I don't think many are willing to do this; I know I'm not, and I can probably speak for Randy Bush. ;-) 2) If the issues above were fixed in HEAD, why were none of the PRs listed in my Wiki updated to reflect that? 3) If the above issues were fixed in HEAD, can you point me to the CVS commits for them? Any time I see ATA commits happen in RELENG_7, I immediately use cvsweb to look at the changes and commit message -- that means I look at HEAD, RELENG_7, and any other branchpoint. I haven't seen anything committed for these issues. 4) If the above issues were actually fixed in HEAD, are there scheduled plans to MFC the fixes? I appreciate you taking the time to help track these down and investigate them, but I feel like you, myself, Scott Long, and the users are the only ones who care about these issues. The maintainer is alive and active, but hasn't said a word, and some of those PRs go untouched for 2+ years... -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 07:28:22 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4E071065673; Thu, 7 Aug 2008 07:28:22 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id A8A098FC0A; Thu, 7 Aug 2008 07:28:22 +0000 (UTC) (envelope-from randy@psg.com) Received: from 50.216.138.210.bn.2iij.net ([210.138.216.50] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KQzv3-000LN4-Vw; Thu, 07 Aug 2008 07:28:22 +0000 Message-ID: <489AA414.5080709@psg.com> Date: Thu, 07 Aug 2008 16:28:20 +0900 From: Randy Bush User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: Jeremy Chadwick References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> <20080807071434.GA15465@eos.sc1.parodius.com> In-Reply-To: <20080807071434.GA15465@eos.sc1.parodius.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 07:28:22 -0000 > 1) Are you trying to tell me that individuals running commercial > services in production environments should run CURRENT? I don't > think many are willing to do this; I know I'm not, and I can probably > speak for Randy Bush. ;-) depends on what you say. :) i am personally (not day job) running current with zfs on one production server and contemplating more[0]. i am running current ufs on a number of other servers. i figure someone has to put the stuff under load. and it does not bite me too often. of course, when it does, i kick myself. :) > I feel like you, myself, Scott Long, and the users are the only ones > who care about these issues. i imagine some others may be like i. as i have no time to code, i try not to demand a lot unless something is really likely to bite someone. i just report bugs and don't expect a refund. randy --- [0] need to build a 20tb raidz2 system, kind of an nfs store to serve some data collertor(s) and compute egine(s). trying to sort out disk controller and 10ge card decisions and the hardware / freebsd support space is complex. From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 08:51:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8A65106566C for ; Thu, 7 Aug 2008 08:51:06 +0000 (UTC) (envelope-from tamaru@myn.rcast.u-tokyo.ac.jp) Received: from mail4.ecc.u-tokyo.ac.jp (mail3.ecc.u-tokyo.ac.jp [133.11.205.99]) by mx1.freebsd.org (Postfix) with ESMTP id 9BBFF8FC33 for ; Thu, 7 Aug 2008 08:51:06 +0000 (UTC) (envelope-from tamaru@myn.rcast.u-tokyo.ac.jp) Received: from mail0.ecc.u-tokyo.ac.jp (mail0.ecc.u-tokyo.ac.jp [133.11.45.132]) by mail4.ecc.u-tokyo.ac.jp (Postfix) with ESMTP id 23FE55B0C9F for ; Thu, 7 Aug 2008 17:05:47 +0900 (JST) Received: from mhs002.ecc.u-tokyo.ac.jp (mhs002.ecc.u-tokyo.ac.jp [133.11.70.162]) by mail0.ecc.u-tokyo.ac.jp (Postfix) with ESMTP id 3435F1BE801B for ; Thu, 7 Aug 2008 17:05:46 +0900 (JST) Received: from gin.myn.rcast.u-tokyo.ac.jp (157.82.72.158 [157.82.72.158]) by mhs002.ecc.u-tokyo.ac.jp (SpamBlock.pstn.b 3.4.102) with ESMTP id for ; Thu, 7 Aug 2008 17:05:40 +0900 Date: Thu, 07 Aug 2008 17:05:39 +0900 Message-ID: From: Hiroharu Tamaru To: "Andrey V. Elsukov" In-Reply-To: <489A9739.20707@yandex.ru> References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.6 Emacs/21.3 (i386--freebsd) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-IP: 157.82.72.158 X-FROM-DOMAIN: myn.rcast.u-tokyo.ac.jp X-FROM-EMAIL: tamaru@myn.rcast.u-tokyo.ac.jp Cc: freebsd-fs@freebsd.org, Jeremy Chadwick Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 08:51:06 -0000 Hi, At Thu, 07 Aug 2008 10:33:29 +0400, Andrey V. Elsukov wrote: > Jeremy Chadwick wrote: > > Correct, it's a FreeBSD ATA subsystem/driver problem. > > I tried 8.0-CURRENT on marvell's, nvida's and intel's controllers. > Hot plug and attach/detach works on any of these controllers without > any problems.. What i should to do to get similar problems? :) Since the topic is here: Can you boot up the machine with that ata HDD offline (unplugged), and then insert it and atthach it? On my 6.2-STABLE machine, I can detach-replace-attach ata disks (most of the time), but only if I have that slot active at boot time. And yes, though I haven't come up with an absolute sequence to reproduce it, I occasionary have the same problem that hot replaced disks are not recognized by 'atacontrol attach', be it the same physical disk or another. I feel there is a 'right sequence' of atacontrol commands (or the lack of), and some 'wrong sequences'. The only way to reattach it then is to reboot the machine. Thanks -- Hiroharu Tamaru From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 11:30:10 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3771C1065763 for ; Thu, 7 Aug 2008 11:30:10 +0000 (UTC) (envelope-from fbsd-fs@mawer.org) Received: from outbound.icp-qv1-irony-out1.iinet.net.au (outbound.icp-qv1-irony-out1.iinet.net.au [203.59.1.108]) by mx1.freebsd.org (Postfix) with ESMTP id AA4FB8FC22 for ; Thu, 7 Aug 2008 11:30:09 +0000 (UTC) (envelope-from fbsd-fs@mawer.org) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkQCAOh5mkh8qj0p/2dsb2JhbAAIrG4 X-IronPort-AV: E=Sophos;i="4.31,319,1215360000"; d="scan'208";a="366964412" Received: from unknown (HELO [10.4.1.1]) ([124.170.61.41]) by outbound.icp-qv1-irony-out1.iinet.net.au with ESMTP; 07 Aug 2008 19:30:06 +0800 Message-ID: <489ADD89.8070809@mawer.org> Date: Thu, 07 Aug 2008 21:33:29 +1000 From: Antony Mawer User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) MIME-Version: 1.0 To: Jeremy Chadwick References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> <20080807071434.GA15465@eos.sc1.parodius.com> In-Reply-To: <20080807071434.GA15465@eos.sc1.parodius.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 11:30:10 -0000 On 7/08/2008 5:14 PM, Jeremy Chadwick wrote: >>> My advice at this point in time, because as of today I have officially >>> lost faith in it: avoid ata(4) at all costs. >> I tried to contact you some time ago, but didn't receive any >> answers.. Do you still want to resolve your problems with ATA? > > Yes, I did receive your mails, but you just wanted to know "if I was > still having problems". I should have replied, but I did not. That is > my fault, and for that I apologise. > > The issues aren't problems specific to me -- they are affecting a > significant userbase, specifically folks who use servers in production > environments. But maybe I've misunderstood what you meant by "your > problems" -- my apologies if I have. ... > We still don't have an answer to the famous "DMA timeout issue", which > continues to haunt many. I provided a small analysis in my Wiki, but > the technical justification is over my head -- it needs review from > someone who is familiar with the ATA protocol. I inteprete the > NID_NOT_FOUND error to mean FreeBSD is asking the disk to r/w to/from an > invalid LBA. I received one mail from a user (I forget if a mailing > list was CC'd or not -- I need to dig up the mail) who said that in some > cases NID_NOT_FOUND is normal. Do you know if most people found these are something that go away, at least temporarily, when the server is rebooted? Or do they persist across reboots? I'm going to do some analysis and find out whether I can find any of our systems that may be experiencing ATA errors that don't correlate with what their SMART data is saying. To date I haven't caught any, but that's not to say they may not be happening... just that all of the ones I have caught to date do appear to have been hardware-related issues... It seems a shame that, at one point, bits of FreeBSD ATA code wound up in Linux (the controversy when parts were lifted but Soren's copyright removed)... now FreeBSD's ata subsystem is left languishing with no immediate solution in sight :-( Is there any means or interest for the FreeBSD Foundation to find someone experienced enough with ATA to spend some time reviewing/testing it? Or does that sort of thing just "not happen"? --Antony From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 12:12:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49DAC106567B for ; Thu, 7 Aug 2008 12:12:45 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 33C278FC21 for ; Thu, 7 Aug 2008 12:12:45 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 1AD081CC0B3; Thu, 7 Aug 2008 05:12:45 -0700 (PDT) Date: Thu, 7 Aug 2008 05:12:45 -0700 From: Jeremy Chadwick To: Antony Mawer Message-ID: <20080807121245.GA26629@eos.sc1.parodius.com> References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> <20080807071434.GA15465@eos.sc1.parodius.com> <489ADD89.8070809@mawer.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <489ADD89.8070809@mawer.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 12:12:45 -0000 On Thu, Aug 07, 2008 at 09:33:29PM +1000, Antony Mawer wrote: > On 7/08/2008 5:14 PM, Jeremy Chadwick wrote: >>>> My advice at this point in time, because as of today I have officially >>>> lost faith in it: avoid ata(4) at all costs. >>> I tried to contact you some time ago, but didn't receive any >>> answers.. Do you still want to resolve your problems with ATA? >> >> Yes, I did receive your mails, but you just wanted to know "if I was >> still having problems". I should have replied, but I did not. That is >> my fault, and for that I apologise. >> >> The issues aren't problems specific to me -- they are affecting a >> significant userbase, specifically folks who use servers in production >> environments. But maybe I've misunderstood what you meant by "your >> problems" -- my apologies if I have. > ... >> We still don't have an answer to the famous "DMA timeout issue", which >> continues to haunt many. I provided a small analysis in my Wiki, but >> the technical justification is over my head -- it needs review from >> someone who is familiar with the ATA protocol. I inteprete the >> NID_NOT_FOUND error to mean FreeBSD is asking the disk to r/w to/from an >> invalid LBA. I received one mail from a user (I forget if a mailing >> list was CC'd or not -- I need to dig up the mail) who said that in some >> cases NID_NOT_FOUND is normal. > > Do you know if most people found these are something that go away, at > least temporarily, when the server is rebooted? Or do they persist > across reboots? For some people, the problems are permanent. Those people have been referred to talk to Scott Long, who offered to help look into the issue, specifically those who can reproduce the errors. For others, the problems eventually disappear, or possibly disappear after "messing around" with their systems. Some people swapped SATA cables, others replaced their entire motherboard; "things work fine" was the result. The problem is that these were considered solutions, and I'm not so sure the problem really was with their hardware, cables, or anything else. For example, the FreeNAS folks found that for some users, increasing the ATA command timeout in the code from an arbitrary 5 seconds to something larger (10-15 seconds) fixed the problem. I don't know why an ATA transaction would take 10-15 seconds, but I suppose it's possible if the disk is doing something internal. I have personal experience with an example of drives doing such. Some older (circa late 90s/early 2000) IBM ATA disks have a feature called ADM, where almost like clockwork, the drive would spin down and start doing some sort of internal maintenance. Upon recieving an ATA command, it would abort ADM, spin up, then complete the request. The result in FreeBSD (4.x) was a slew of ATA timeout/DMA errors. I eventually found this feature mentioned in the disk specification PDF, and contacted IBM about it. IBM confirmed the feature, confirmed it was responsible, and stated there was no way to disable it on ATA disks. (On SCSI, the feature defaulted to off, but could be toggled on via a custom vendor-specific SCSI command). I still have the mail from IBM if anyone wants to see it. A few months later, when IBM released their next generation version of their ATA disks, I found the ADM feature completely gone (from the disk and the disk specification PDF). In almost every case I've looked at so far, the individuals' chipsets, disks, and overall setup are different. SMART statistics on the drives show absolutely no sign of errors, or anything that indicates a hardware failure. Many of the users are using AHCI as well (myself included, and I have seen the DMA error issue myself), which is more reliable than classic IDE. Even in the case of temporary failures ("I saw those DMA errors once, but they haven't returned"), it would be benefitial if users could submit that data to me, so I can put more example cases in the Wiki. The NID_NOT_FOUND error bother me, because the ATA specification implies the OS is asking for an invalid LBA, which would likely be caused by a bug in FreeBSD. That's why that error condition needs to be analysed more. I will point people to the libata FAQ on errors, too -- look at the definition of error type IDNF (this is what FreeBSD calls NID_NOT_FOUND): http://ata.wiki.kernel.org/index.php/Libata_error_messages For SATA, FreeBSD does not appear to support printing SATA SError codes, so for those of us with SATA disks, we're actually missing some verbosity on what the errors could be caused by. It would be benefitial if there was some form of sysctl to increase the verbosity from the ATA subsystem when an error happens. The existing data we get back is terse, and barely useful. I know for a fact there's more debug information that could be output in such scenarios. And please do not reply with "good idea, send patches" unless you're wanting to be chewed out. :-) > I'm going to do some analysis and find out whether I can find any of our > systems that may be experiencing ATA errors that don't correlate with > what their SMART data is saying. To date I haven't caught any, but > that's not to say they may not be happening... just that all of the ones > I have caught to date do appear to have been hardware-related issues... > > It seems a shame that, at one point, bits of FreeBSD ATA code wound up > in Linux (the controversy when parts were lifted but Soren's copyright > removed)... now FreeBSD's ata subsystem is left languishing with no > immediate solution in sight :-( I had forgotten all about that piece of history -- thanks for reminding me. The only solutions/options that are on the horizon: * Recommending people buy SATA controllers that utilise CAM and da(4), thus avoiding ata(4) entirely. Areca makes such controllers, but it's impractical to ask users to buy one, since they're expensive; end-users should be able to use their onboard SATA controllers like the Intel ICH series and nVidia nForce and MCP with reliability. * Implement a form of ATA-to-SCSI translation, similar to what Linux libata does; this would make ATA/SATA disks utilise CAM and da(4) through a translation layer. Scott Long has told me that he is actually in the process of writing such, but I know Scott is also *incredibly* busy, so that project may take a very long time. * If you use ATA (read: PATA) disks, disabling DMA and forcing PIO does apparently work. The performance hit is quite substantial, however, and this isn't practical for servers. Disabling DMA is not possible with SATA. * Contact Jeff Garzik (Linux libata maintainer) and ask for help. This might upset some people, especially considering the history item you brought up earlier. By "help" I don't mean "Hey man, write the code", I mean "can you help expand on what this error means, and have you folks seen it on Linux?" > Is there any means or interest for the FreeBSD Foundation to find > someone experienced enough with ATA to spend some time reviewing/testing > it? Or does that sort of thing just "not happen"? That's a very good question. I don't know how the politics of the FreeBSD Foundation work. But I would be more than happy to donate money (read: a couple thousand US dollars out of my pocket) to the Foundation assuming the proceeds went to getting someone very familiar with ATA to look at the problem, look at the code, address existing PRs, and fix things. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 15:46:22 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DFD3106567F for ; Thu, 7 Aug 2008 15:46:22 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.120]) by mx1.freebsd.org (Postfix) with ESMTP id F209A8FC1E for ; Thu, 7 Aug 2008 15:46:21 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from shop.chemikals.org ([75.182.7.127]) by cdptpa-omta02.mail.rr.com with ESMTP id <20080807154621.DRHV15817.cdptpa-omta02.mail.rr.com@shop.chemikals.org>; Thu, 7 Aug 2008 15:46:21 +0000 Received: from volatile.chemikals.org (root@r74-193-170-223.bssrcmta01.bscyla.by.dh.suddenlink.net [74.193.170.223] (may be forged)) by shop.chemikals.org (8.14.2/8.14.2) with ESMTP id m77FkJEP003687; Thu, 7 Aug 2008 11:46:20 -0400 (EDT) (envelope-from morganw@chemikals.org) Received: from localhost (morganw@localhost [127.0.0.1]) by volatile.chemikals.org (8.14.2/8.14.2) with ESMTP id m77FkCVG063821; Thu, 7 Aug 2008 10:46:18 -0500 (CDT) (envelope-from morganw@chemikals.org) Date: Thu, 7 Aug 2008 10:46:12 -0500 (CDT) From: Wes Morgan To: Adam McDougall In-Reply-To: <489A2E1F.6000405@egr.msu.edu> Message-ID: References: <18585.3903.895425.122613@almost.alerce.com> <200808061928.37001.peter.schuller@infidyne.com> <489A2E1F.6000405@egr.msu.edu> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 15:46:22 -0000 On Wed, 6 Aug 2008, Adam McDougall wrote: > Wes Morgan wrote: >> On Wed, 6 Aug 2008, Peter Schuller wrote: >> >>>> The AoC-SAT2-MV8 is based on the "Marvell Hercules-2 Rev. C0 SATA host >>>> controller", which seems to be AKA 88SX6081, which is listed as >>>> supported by the ata driver in 7.0-RELEASE. Has anyone had any ZFS >>>> experience with it? >>> >>> Yes; it has been working quite fine for me with 7 up to a >>> release-candidate. >>> In 7.0-RELEASE you must disable the hptrr driver because it eats the >>> device. >>> See: >>> >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120615 >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120842 >>> >>> When I say "working fine", this is with 8 SATA drives (6 of them in a >>> raidz2, >>> two of them for other stuff) and not having any issues like corruption, >>> timeouts or whatever else people have had with shaky controllers. >>> >>> However, I cannot speak to performance because it's a PCI-X card that I've >>> plugged into a PCI slow, so throughput is limited by the PCI bus (and the >>> machine is otherwise not the fastest to begin with). >>> >>> Note that this is on 32 bit; haven't been able to try it on 64 bit because >>> I >>> the PCI-X card wouldn't work on the motherboard (again PCI, so it's >>> hit-and-miss) where I would otherwise have tried it. >>> >>> I'd love to find a buyable PCI-E version that also worked in FreeBSD... >>> I'll >>> see if the link in your post contains any such hints. >> >> Hmmm... That PCI-X card is interesting. Supermicro also lists this: >> >> http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm >> >> http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ics/lsisas1068e/index.html >> >> Not much onboard ram, but it's PCI-E and even SAS. CDW lists it for $155. >> That would be cheaper than buying a new board with a PCI-X slot or two, and >> would even handle SAS drives. Claims to be based on the "LSISAS 1068E SAS >> controller". Any idea if that is supported? I don't see it listed in the >> mfi man page. LSI has a Linux driver for download. That card looks like it >> would be just what I need. >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > mpt2@pci0:8:0:0: class=0x010000 card=0x31501000 chip=0x00581000 > rev=0x04 hdr=0x00 > vendor = 'LSI Logic (Was: Symbios Logic, NCR)' > device = 'SAS 3000 series, 8-port with 1068E -StorPort' > class = mass storage > subclass = SCSI > mpt2: port 0xa800-0xa8ff mem > 0xfcbfc000-0xfcbfffff,0xfcbe0000-0xfcbeffff irq 17 at device 0.0 on pci8 > > da1 at mpt2 bus 0 target 0 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing Enabled > da1: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da2 at mpt2 bus 0 target 1 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing Enabled > da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da3 at mpt2 bus 0 target 2 lun 0 > da3: Fixed Direct Access SCSI-5 device > da3: 300.000MB/s transfers > da3: Command Queueing Enabled > da3: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da4 at mpt2 bus 0 target 3 lun 0 > da4: Fixed Direct Access SCSI-5 device > da4: 300.000MB/s transfers > da4: Command Queueing Enabled > da4: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da5 at mpt2 bus 0 target 4 lun 0 > da5: Fixed Direct Access SCSI-5 device > da5: 300.000MB/s transfers > da5: Command Queueing Enabled > da5: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da6 at mpt2 bus 0 target 5 lun 0 > da6: Fixed Direct Access SCSI-5 device > da6: 300.000MB/s transfers > da6: Command Queueing Enabled > da6: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da7 at mpt2 bus 0 target 6 lun 0 > da7: Fixed Direct Access SCSI-5 device > da7: 300.000MB/s transfers > da7: Command Queueing Enabled > da7: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) > da8 at mpt2 bus 0 target 7 lun 0 > da8: Fixed Direct Access SCSI-5 device > da8: 300.000MB/s transfers > da8: Command Queueing Enabled > da8: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) Excellent! How is your experience with the performance and reliability of the card? From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 16:50:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AA9C1065678 for ; Thu, 7 Aug 2008 16:50:07 +0000 (UTC) (envelope-from boris.kotzev@gmail.com) Received: from gv-out-0910.google.com (gv-out-0910.google.com [216.239.58.190]) by mx1.freebsd.org (Postfix) with ESMTP id BD7C58FC08 for ; Thu, 7 Aug 2008 16:50:06 +0000 (UTC) (envelope-from boris.kotzev@gmail.com) Received: by gv-out-0910.google.com with SMTP id n8so12407gve.39 for ; Thu, 07 Aug 2008 09:50:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:subject:date :user-agent:mime-version:content-type:content-transfer-encoding :content-disposition:message-id; bh=ti8MSRstW0qwvhVTZeBQKtaCNd9q5C1h7xAiZTV5g5U=; b=tj4WbGsVx6vRHMaZIC0vEAuJ2BEEm1KHmnS3l+B8vYYUJEXHaasDYARA74YyOjFaVE zg1h35crRSTBkC9C3QSuPWA33pd9kFUMLGK7IZiVSiG4bqUzHhQ+7Rt1J+gWnHNB5XN7 XOSglBF4+EMGCQSGm5dDVAOnfLsWJVftSQioo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:mime-version:content-type :content-transfer-encoding:content-disposition:message-id; b=iy2V1go4w+CRJbajTOC7x5UA+vHFLflBx2zFZLVAvacTfcPVScfW1pzI9X5PPgBVLI rlckIc15mgWZxuhn1T2qnXsPkJqDto/BmkhEXTZ4EqqrPAjeziAAJCs/gjkDPN+6Wqgx KjbwNjRo+wmKA/oNo0/BGam3ZQnRpruG38WlE= Received: by 10.103.211.3 with SMTP id n3mr2397654muq.43.1218126349171; Thu, 07 Aug 2008 09:25:49 -0700 (PDT) Received: from zembla.universe ( [213.169.62.7]) by mx.google.com with ESMTPS id b9sm10506500mug.13.2008.08.07.09.25.47 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 07 Aug 2008 09:25:48 -0700 (PDT) From: Boris Kotzev To: freebsd-fs@freebsd.org Date: Thu, 7 Aug 2008 19:25:45 +0300 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: 8bit Content-Disposition: inline Message-Id: <200808071925.45786.boris.kotzev@gmail.com> Subject: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 16:50:07 -0000 Hello, I used the zfs port to Mac OS X (http://zfs.macosforge.org) to create a storage pool under Mac OS X. The pool can be imported successfully under FreeBSD: root:~-114# zpool import macpool root:~-115# zpool list macpool NAME SIZE USED AVAIL CAP HEALTH ALTROOT macpool 6,94G 510K 6,94G 0% ONLINE - root:~-116# zfs list macpool NAME USED AVAIL REFER MOUNTPOINT macpool 474K 6,83G 308K /macpool and is fully accessible to the root user: root:~-118# id uid=0(root) gid=0(wheel) groups=0(wheel),5(operator) root:~-119# ls -ld /macpool drwxr-xr-x 7 root wheel 8 7 Àâã 16:59 /macpool root:~-120# ls -l /macpool total 43 drwx------ 3 root wheel 3 7 Àâã 16:31 .Spotlight-V100 -rw-r--r-- 1 root wheel 35014 7 Àâã 16:31 .VolumeIcon.icns drwx------ 2 root wheel 4 7 Àâã 16:32 .fseventsd drwxr-xr-x 2 root wheel 2 7 Àâã 16:59 backup drwxr-xr-x 2 root wheel 2 7 Àâã 16:59 downloads drwxr-xr-x 2 root wheel 2 7 Àâã 16:58 music According to the file permissions on /macpool (drwxr-xr-x), anyone should have read access to it. This is not the case though: root:~-121# su user % id uid=1003(user) gid=1003(user) groups=1003(user),0(wheel),5(operator) % ls -l /macpool ls: /macpool: Permission denied % cd /macpool /macpool: Permission denied. Is this a bug, or is there some way to get access to /macpool as an ordinary user? The pool was created under version zfs-119 of the Mac OS X port; the FreeBSD version is: root:~-122# uname -a FreeBSD xxxx 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Aug 2 14:19:33 EEST 2008 root@xxxx:/usr/obj/usr/src/sys/MACBOOK amd64 with the latest zfs patch, but the problem was also present before applying the patch. Thanks, Boris Kotzev From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 16:55:03 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2073E1065690 for ; Thu, 7 Aug 2008 16:55:03 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 0F3628FC0A for ; Thu, 7 Aug 2008 16:55:02 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id C35181CC0B1; Thu, 7 Aug 2008 09:55:02 -0700 (PDT) Date: Thu, 7 Aug 2008 09:55:02 -0700 From: Jeremy Chadwick To: Boris Kotzev Message-ID: <20080807165502.GA39420@eos.sc1.parodius.com> References: <200808071925.45786.boris.kotzev@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200808071925.45786.boris.kotzev@gmail.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 16:55:03 -0000 On Thu, Aug 07, 2008 at 07:25:45PM +0300, Boris Kotzev wrote: > Hello, > > I used the zfs port to Mac OS X (http://zfs.macosforge.org) to create > a storage pool under Mac OS X. The pool can be imported successfully > under FreeBSD: > > root:~-114# zpool import macpool > root:~-115# zpool list macpool > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > macpool 6,94G 510K 6,94G 0% ONLINE - > root:~-116# zfs list macpool > NAME USED AVAIL REFER MOUNTPOINT > macpool 474K 6,83G 308K /macpool > > and is fully accessible to the root user: > > root:~-118# id > uid=0(root) gid=0(wheel) groups=0(wheel),5(operator) > root:~-119# ls -ld /macpool > drwxr-xr-x 7 root wheel 8 7 ??? 16:59 /macpool > root:~-120# ls -l /macpool > total 43 > drwx------ 3 root wheel 3 7 ??? 16:31 .Spotlight-V100 > -rw-r--r-- 1 root wheel 35014 7 ??? 16:31 .VolumeIcon.icns > drwx------ 2 root wheel 4 7 ??? 16:32 .fseventsd > drwxr-xr-x 2 root wheel 2 7 ??? 16:59 backup > drwxr-xr-x 2 root wheel 2 7 ??? 16:59 downloads > drwxr-xr-x 2 root wheel 2 7 ??? 16:58 music > > According to the file permissions on /macpool (drwxr-xr-x), anyone > should have read access to it. This is not the case though: > > root:~-121# su user > % id > uid=1003(user) gid=1003(user) groups=1003(user),0(wheel),5(operator) > % ls -l /macpool > ls: /macpool: Permission denied > % cd /macpool > /macpool: Permission denied. > > Is this a bug, or is there some way to get access to /macpool as an > ordinary user? > > The pool was created under version zfs-119 of the Mac OS X port; the > FreeBSD version is: > > root:~-122# uname -a > FreeBSD xxxx 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Aug 2 14:19:33 > EEST 2008 root@xxxx:/usr/obj/usr/src/sys/MACBOOK amd64 > > with the latest zfs patch, but the problem was also present before > applying the patch. As root, what does "zfs get all macpool" return on FreeBSD? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 17:41:22 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18D401065670 for ; Thu, 7 Aug 2008 17:41:22 +0000 (UTC) (envelope-from boris.kotzev@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.188]) by mx1.freebsd.org (Postfix) with ESMTP id 87E7E8FC23 for ; Thu, 7 Aug 2008 17:41:21 +0000 (UTC) (envelope-from boris.kotzev@gmail.com) Received: by mu-out-0910.google.com with SMTP id i2so367767mue.3 for ; Thu, 07 Aug 2008 10:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:subject:date :user-agent:cc:references:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:message-id; bh=32rwrqR2bKrv/Q/Aw8tdsLqj9cV7YyHLWnoBREt5w24=; b=nX+xXkHlByaN9tdd9dVWi1XaPQdWIoBi0idfwrj3SE3HtFMEwGN6oK/rhS3e9zREmf EkCvp5lbsjVEpAjNh/O4s6U1L8iVFuI+QDD/eZLwPyAE6CgH1O79XeAP053Cvz/6FU4r cw2VJFQq/ZMFJaTKA30/pZYbvrQbx7akKnLLM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:cc:references:in-reply-to :mime-version:content-type:content-transfer-encoding :content-disposition:message-id; b=u7Qmit4ZuT++gufWuAWbuf6JcQtGuyeCJNW7Uit8SNrNcEXtumq9Mi+0oBy9MSzsXf kDUcjBYtW9wLPsWxETd1gobwrF80j6RB+6rfa2yzXffxDFzzRRlfJRkYiriqAVkzs6FP Ew3j6GQb9mylAgfjRw2CeNjuc3UMjteBMPwZ4= Received: by 10.103.20.7 with SMTP id x7mr2443871mui.75.1218130879752; Thu, 07 Aug 2008 10:41:19 -0700 (PDT) Received: from zembla.universe ( [213.169.62.7]) by mx.google.com with ESMTPS id j10sm36330mue.6.2008.08.07.10.40.56 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 07 Aug 2008 10:40:57 -0700 (PDT) From: Boris Kotzev To: Jeremy Chadwick Date: Thu, 7 Aug 2008 20:40:55 +0300 User-Agent: KMail/1.9.7 References: <200808071925.45786.boris.kotzev@gmail.com> <20080807165502.GA39420@eos.sc1.parodius.com> In-Reply-To: <20080807165502.GA39420@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: 8bit Content-Disposition: inline Message-Id: <200808072040.55571.boris.kotzev@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 17:41:22 -0000 Íà Thursday 07 August 2008 19:55:02 Jeremy Chadwick íàïèñà: > On Thu, Aug 07, 2008 at 07:25:45PM +0300, Boris Kotzev wrote: > > Hello, > > > > I used the zfs port to Mac OS X (http://zfs.macosforge.org) to > > create a storage pool under Mac OS X. The pool can be imported > > successfully under FreeBSD: > > > > root:~-114# zpool import macpool > > root:~-115# zpool list macpool > > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > > macpool 6,94G 510K 6,94G 0% ONLINE - > > root:~-116# zfs list macpool > > NAME USED AVAIL REFER MOUNTPOINT > > macpool 474K 6,83G 308K /macpool > > > > and is fully accessible to the root user: > > > > root:~-118# id > > uid=0(root) gid=0(wheel) groups=0(wheel),5(operator) > > root:~-119# ls -ld /macpool > > drwxr-xr-x 7 root wheel 8 7 ??? 16:59 /macpool > > root:~-120# ls -l /macpool > > total 43 > > drwx------ 3 root wheel 3 7 ??? 16:31 .Spotlight-V100 > > -rw-r--r-- 1 root wheel 35014 7 ??? 16:31 .VolumeIcon.icns > > drwx------ 2 root wheel 4 7 ??? 16:32 .fseventsd > > drwxr-xr-x 2 root wheel 2 7 ??? 16:59 backup > > drwxr-xr-x 2 root wheel 2 7 ??? 16:59 downloads > > drwxr-xr-x 2 root wheel 2 7 ??? 16:58 music > > > > According to the file permissions on /macpool (drwxr-xr-x), > > anyone should have read access to it. This is not the case > > though: > > > > root:~-121# su user > > % id > > uid=1003(user) gid=1003(user) > > groups=1003(user),0(wheel),5(operator) % ls -l /macpool > > ls: /macpool: Permission denied > > % cd /macpool > > /macpool: Permission denied. > > > > Is this a bug, or is there some way to get access to /macpool as > > an ordinary user? > > > > The pool was created under version zfs-119 of the Mac OS X port; > > the FreeBSD version is: > > > > root:~-122# uname -a > > FreeBSD xxxx 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Aug 2 > > 14:19:33 EEST 2008 root@xxxx:/usr/obj/usr/src/sys/MACBOOK amd64 > > > > with the latest zfs patch, but the problem was also present > > before applying the patch. > > As root, what does "zfs get all macpool" return on FreeBSD? root@:~-116# zfs get all macpool NAME PROPERTY VALUE SOURCE macpool type filesystem - macpool creation ×ò Àâã 7 16:31 2008 - macpool used 474K - macpool available 6,83G - macpool referenced 308K - macpool compressratio 1.00x - macpool mounted yes - macpool quota none default macpool reservation none default macpool recordsize 128K default macpool mountpoint /macpool default macpool sharenfs off default macpool checksum on default macpool compression off default macpool atime on default macpool devices on default macpool exec on default macpool setuid on default macpool readonly off default macpool jailed off default macpool snapdir hidden default macpool aclmode groupmask default macpool aclinherit restricted default macpool canmount on default macpool shareiscsi off default macpool xattr off temporary macpool copies 1 default macpool version 1 - macpool utf8only off - macpool normalization none - macpool casesensitivity sensitive - macpool vscan off default macpool nbmand off default macpool sharesmb off default macpool refquota none default macpool refreservation none default Thanks! Boris Kotzev From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 17:59:30 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 817681065671 for ; Thu, 7 Aug 2008 17:59:30 +0000 (UTC) (envelope-from bp@barryp.org) Received: from itasca.hexavalent.net (itasca.hexavalent.net [67.207.138.180]) by mx1.freebsd.org (Postfix) with ESMTP id 59BB88FC08 for ; Thu, 7 Aug 2008 17:59:30 +0000 (UTC) (envelope-from bp@barryp.org) Received: from host-42-60-230-24.midco.net ([24.230.60.42] helo=eden.barryp.org) by itasca.hexavalent.net with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.67) (envelope-from ) id 1KR9Ew-0003pn-LE for freebsd-fs@freebsd.org; Thu, 07 Aug 2008 12:25:30 -0500 Received: from geo.med.und.nodak.edu ([134.129.166.11]) by eden.barryp.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67 (FreeBSD)) (envelope-from ) id 1KR9Ek-00036D-EV; Thu, 07 Aug 2008 12:25:18 -0500 Message-ID: <489B2FFE.5050406@barryp.org> Date: Thu, 07 Aug 2008 12:25:18 -0500 From: Barry Pederson User-Agent: Thunderbird 2.0.0.14 (X11/20080421) MIME-Version: 1.0 To: Wes Morgan References: <18585.3903.895425.122613@almost.alerce.com> <200808061928.37001.peter.schuller@infidyne.com> <489A2E1F.6000405@egr.msu.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 17:59:30 -0000 Wes Morgan wrote: > On Wed, 6 Aug 2008, Adam McDougall wrote: > >> Wes Morgan wrote: >>> On Wed, 6 Aug 2008, Peter Schuller wrote: >>> Hmmm... That PCI-X card is interesting. Supermicro also lists this: >>> >>> http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm >>> >>> http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ics/lsisas1068e/index.html >>> >>> Not much onboard ram, but it's PCI-E and even SAS. CDW lists it for >>> $155. I think CDW is mistaken in saying it's a PCI-E card, UIO is a proprietary Supermicro bus that some of their motherboards support. http://www.supermicro.com/products/nfo/UIO.cfm Too bad, that's a good price for an 8-port SAS card. I've got some LSI-branded PCI-X and PCI-E SAS cards that have been working very well with ZFS and SATA drives, but the 8-port card was more like $288. Barry From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 18:47:51 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5ABFE106567F for ; Thu, 7 Aug 2008 18:47:51 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id 2CF098FC12 for ; Thu, 7 Aug 2008 18:47:51 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from localhost (localhost [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id A72E471F08A; Thu, 7 Aug 2008 14:47:50 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b+eGJT0eOxqt; Thu, 7 Aug 2008 14:47:50 -0400 (EDT) Received: from [35.9.44.65] (daemon.egr.msu.edu [35.9.44.65]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mcdouga9) by mx.egr.msu.edu (Postfix) with ESMTPSA id 843FA71EDA3; Thu, 7 Aug 2008 14:47:50 -0400 (EDT) Message-ID: <489B4356.8060702@egr.msu.edu> Date: Thu, 07 Aug 2008 14:47:50 -0400 From: Adam McDougall User-Agent: Thunderbird 2.0.0.16 (X11/20080727) MIME-Version: 1.0 To: Wes Morgan References: <18585.3903.895425.122613@almost.alerce.com> <200808061928.37001.peter.schuller@infidyne.com> <489A2E1F.6000405@egr.msu.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 18:47:51 -0000 Wes Morgan wrote: > On Wed, 6 Aug 2008, Adam McDougall wrote: > >> Wes Morgan wrote: >>> >>> http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ics/lsisas1068e/index.html >>> >>> Not much onboard ram, but it's PCI-E and even SAS. CDW lists it for >>> $155. That would be cheaper than buying a new board with a PCI-X >>> slot or two, and would even handle SAS drives. Claims to be based on >>> the "LSISAS 1068E SAS controller". Any idea if that is supported? I >>> don't see it listed in the mfi man page. LSI has a Linux driver for >>> download. That card looks like it would be just what I need. >>> _______________________________________________ >>> >> mpt2@pci0:8:0:0: class=0x010000 card=0x31501000 >> chip=0x00581000 rev=0x04 hdr=0x00 >> vendor = 'LSI Logic (Was: Symbios Logic, NCR)' >> device = 'SAS 3000 series, 8-port with 1068E -StorPort' >> class = mass storage >> subclass = SCSI >> mpt2: port 0xa800-0xa8ff mem >> 0xfcbfc000-0xfcbfffff,0xfcbe0000-0xfcbeffff irq 17 at device 0.0 on pci8 >> >> da1 at mpt2 bus 0 target 0 lun 0 >> da1: Fixed Direct Access SCSI-5 device >> da1: 300.000MB/s transfers >> da1: Command Queueing Enabled >> da1: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C) >> > > Excellent! How is your experience with the performance and reliability > of the card? > The system is in production right now with a smallish amount of load, so the quick test I did to see 85MB/sec write to a raidz1 and 60M/sec read is probably a bit unfair. Each set of 4 disks is attached to a SAS expander which is attached to the card internally in a sun x4150. I didn't try to tune it for performance before I started using that volume for storage in a manner where size and realiability were more important than speed. But on a whole it seems reliable. I won't have any time soon to get it into a better state for testing, but I may test the new ZFS patches on it before that. I've been pretty happy with all other MPT family chips so I would doubt it has any crippling speed problems. From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 21:12:57 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0D621065673 for ; Thu, 7 Aug 2008 21:12:57 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.28]) by mx1.freebsd.org (Postfix) with ESMTP id 9D5478FC12 for ; Thu, 7 Aug 2008 21:12:57 +0000 (UTC) (envelope-from matt@corp.spry.com) Received: by yw-out-2324.google.com with SMTP id 9so339333ywe.13 for ; Thu, 07 Aug 2008 14:12:57 -0700 (PDT) Received: by 10.143.5.21 with SMTP id h21mr640848wfi.175.1218143575929; Thu, 07 Aug 2008 14:12:55 -0700 (PDT) Received: from matts.spry.com ( [207.178.4.6]) by mx.google.com with ESMTPS id 22sm394167wfi.14.2008.08.07.14.12.53 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 07 Aug 2008 14:12:54 -0700 (PDT) Message-Id: <135AC59E-45A2-4254-AC2C-43D3BB60EE39@corp.spry.com> From: Matt Simerson To: freebsd-fs@freebsd.org In-Reply-To: <489AA414.5080709@psg.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v928.1) Date: Thu, 7 Aug 2008 14:12:48 -0700 References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> <20080807071434.GA15465@eos.sc1.parodius.com> <489AA414.5080709@psg.com> X-Mailer: Apple Mail (2.928.1) Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 21:12:58 -0000 On Aug 7, 2008, at 12:28 AM, Randy Bush wrote: >> 1) Are you trying to tell me that individuals running commercial >> services in production environments should run CURRENT? I don't >> think many are willing to do this; I know I'm not, and I can probably >> speak for Randy Bush. ;-) > > depends on what you say. :) > > i am personally (not day job) running current with zfs on one > production > server and contemplating more[0]. i am running current ufs on a > number of > other servers. > > --- > > [0] need to build a 20tb raidz2 system, kind of an nfs store to serve > some data collertor(s) and compute egine(s). trying to sort out disk > controller and 10ge card decisions and the hardware / freebsd support > space is complex. SuperMicro 24 disk chassis (CSE-846TQ-R900B) 24 1TB SATA disks (2) Areca 1231ML controllers with BBWC Two RAID 5 partitions with no hot-spare disks will net you 19TB of formatted storage. If you ZFS stripe across the two RAID arrays, you'll get decent performance. If your data is compressible, enabling file system compression will easily store the 20TB you need. If you have heavy I/O load, do not use raidz2 in 7.0 lest you experience the instability and hangs that many of us are seeing. Use 8.0-HEAD with the latest patches and with a little luck, you'll have lots of storage and stability. Matt From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 21:32:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A86FA106568A for ; Thu, 7 Aug 2008 21:32:02 +0000 (UTC) (envelope-from randy@psg.com) Received: from rip.psg.com (rip.psg.com [IPv6:2001:418:1::39]) by mx1.freebsd.org (Postfix) with ESMTP id 8504D8FC30 for ; Thu, 7 Aug 2008 21:32:02 +0000 (UTC) (envelope-from randy@psg.com) Received: from 50.216.138.210.bn.2iij.net ([210.138.216.50] helo=rmac.psg.com) by rip.psg.com with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1KRD5V-000NEJ-G7; Thu, 07 Aug 2008 21:32:01 +0000 Message-ID: <489B69D0.4050303@psg.com> Date: Fri, 08 Aug 2008 06:32:00 +0900 From: Randy Bush User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: Matt Simerson References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> <20080807071434.GA15465@eos.sc1.parodius.com> <489AA414.5080709@psg.com> <135AC59E-45A2-4254-AC2C-43D3BB60EE39@corp.spry.com> In-Reply-To: <135AC59E-45A2-4254-AC2C-43D3BB60EE39@corp.spry.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: medium sized zfs nfs server (was: zpool degraded - 'UNAVAIL cannot open' functioning drive) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 21:32:02 -0000 [ i am a internet / research / software guy, and this pee cee hardware stuff scares the outta me. so i am seriously desperate with these questions and *highly* appreciative of the responses ] >> [0] need to build a 20tb raidz2 system, kind of an nfs store to serve >> some data collertor(s) and compute engine(s). trying to sort out disk >> controller and 10ge card decisions and the hardware / freebsd support >> space is complex. > > SuperMicro 24 disk chassis (CSE-846TQ-R900B) > 24 1TB SATA disks > (2) Areca 1231ML controllers with BBWC thank you. and the 10gige card? thinking copper cx as it's just between two or three hosts in the rack. > Two RAID 5 partitions with no hot-spare disks will net you 19TB of > formatted storage. If you ZFS stripe across the two RAID arrays, you'll > get decent performance. If your data is compressible, enabling file > system compression will easily store the 20TB you need. actually, i was thinking what i hope is a bit more conservatively (except for the last line) o exactly that chassis with a mobo with 8gb ram and > 2 cores o the goal is 20tb of spindles to get 10tb of data store [ may even wait some weeks for the 1.5tb drives, oink oink ] o two hot spare drives as this is in a remote rack o no hardware raid, have been burned by 3ware, ... o zfs's raidz2 o (gasp!) -current for the zfs fixes whack me harder with the clue bat, please randy From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 22:21:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 969FF106568D for ; Thu, 7 Aug 2008 22:21:24 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from kuber.nabble.com (kuber.nabble.com [216.139.236.158]) by mx1.freebsd.org (Postfix) with ESMTP id 63DC08FC16 for ; Thu, 7 Aug 2008 22:21:24 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1KRDby-00006E-Nj for freebsd-fs@freebsd.org; Thu, 07 Aug 2008 15:05:34 -0700 Message-ID: <18880785.post@talk.nabble.com> Date: Thu, 7 Aug 2008 15:05:34 -0700 (PDT) From: Maximillian Dornseif To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: md@hudora.de Subject: Strange (?) hangs with ZFS/rsync. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 22:21:24 -0000 I have a ZFS based backup server which rsyncs from a dozen other machines. There is only a single rsync active for any point in time and the system is used for nothing else. It has an amd64 kernel and 4GB RAM. I tried to follow the ZFS tuning guide. For a few weeks the machine works well. Since monday it rsync always hangs when fetching a big logfile from a remote machine. The two rsync processes are in state "zfs:&b" and "zfs:lo" and can't be killed. The other strange thing is that the system is reporting 3367M (!) wired memory. I have uploaded dmesg output and a some additional information at http://static.23.nu/md/Pictures/freebsd+zfs+issue.txt Any suggestions on how to debug this issue? Regards Maximillian Dornseif -- View this message in context: http://www.nabble.com/Strange-%28-%29-hangs-with-ZFS-rsync.-tp18880785p18880785.html Sent from the freebsd-fs mailing list archive at Nabble.com. From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 22:33:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 341011065674 for ; Thu, 7 Aug 2008 22:33:28 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from kuber.nabble.com (kuber.nabble.com [216.139.236.158]) by mx1.freebsd.org (Postfix) with ESMTP id 053688FC19 for ; Thu, 7 Aug 2008 22:33:27 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1KRE2x-00015M-3V for freebsd-fs@freebsd.org; Thu, 07 Aug 2008 15:33:27 -0700 Message-ID: <18881178.post@talk.nabble.com> Date: Thu, 7 Aug 2008 15:33:27 -0700 (PDT) From: Maximillian Dornseif To: freebsd-fs@freebsd.org In-Reply-To: <18880785.post@talk.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: md@hudora.de References: <18880785.post@talk.nabble.com> Subject: Re: Strange (?) hangs with ZFS/rsync. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 22:33:28 -0000 Maximillian Dornseif wrote: > > For a few weeks the machine works well. Since monday it rsync always hangs > when fetching a big logfile from a remote machine. > I forget to mention: other processes still work and the ZFS filesystem is still accessible to casual inspection by ls and cat. The machine does not reboot but hangs during the "syncing" phase of shutdown. It has to be powercycled to restart. --md -- View this message in context: http://www.nabble.com/Strange-%28-%29-hangs-with-ZFS-rsync.-tp18880785p18881178.html Sent from the freebsd-fs mailing list archive at Nabble.com. From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 23:09:13 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13B2E1065677 for ; Thu, 7 Aug 2008 23:09:13 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id BCB4B8FC1D for ; Thu, 7 Aug 2008 23:09:12 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 6A8FB2089; Fri, 8 Aug 2008 01:09:11 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 59D548449F; Fri, 8 Aug 2008 01:09:11 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Kostik Belousov References: <200808061020.m76AK5NI013323@freefall.freebsd.org> <20080806133441.GM97161@deviant.kiev.zoral.com.ua> <20080806144820.GO97161@deviant.kiev.zoral.com.ua> Date: Fri, 08 Aug 2008 01:09:11 +0200 In-Reply-To: <20080806144820.GO97161@deviant.kiev.zoral.com.ua> (Kostik Belousov's message of "Wed, 6 Aug 2008 17:48:20 +0300") Message-ID: <86zlnoe82w.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Mateusz Guzik , bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 23:09:13 -0000 Kostik Belousov writes: > @@ -169,7 +169,8 @@ ffs_mount(struct mount *mp, struct thread *td) > * persist "snapshot" in the options list. > */ > vfs_deleteopt(mp->mnt_optnew, "snapshot"); > - vfs_deleteopt(mp->mnt_opt, "snapshot"); > + if (mp->mnt_opt !=3D NULL) > + vfs_deleteopt(mp->mnt_opt, "snapshot"); > } >=20=20 > MNT_ILOCK(mp); I would suggest also adding a KASSERT to vfs_deleteopt(). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 23:30:03 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E108B106566B for ; Thu, 7 Aug 2008 23:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CDEED8FC0A for ; Thu, 7 Aug 2008 23:30:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m77NU3kg052288 for ; Thu, 7 Aug 2008 23:30:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m77NU3GP052285; Thu, 7 Aug 2008 23:30:03 GMT (envelope-from gnats) Date: Thu, 7 Aug 2008 23:30:03 GMT Message-Id: <200808072330.m77NU3GP052285@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= Cc: Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 23:30:04 -0000 The following reply was made to PR kern/126287; it has been noted by GNATS. From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Kostik Belousov Cc: Mateusz Guzik , freebsd-fs@freebsd.org, bug-followup@freebsd.org Subject: Re: kern/126287: [ufs] [panic] Kernel panics while mounting an UFS filesystem with snapshot enabled Date: Fri, 08 Aug 2008 01:09:11 +0200 Kostik Belousov writes: > @@ -169,7 +169,8 @@ ffs_mount(struct mount *mp, struct thread *td) > * persist "snapshot" in the options list. > */ > vfs_deleteopt(mp->mnt_optnew, "snapshot"); > - vfs_deleteopt(mp->mnt_opt, "snapshot"); > + if (mp->mnt_opt !=3D NULL) > + vfs_deleteopt(mp->mnt_opt, "snapshot"); > } >=20=20 > MNT_ILOCK(mp); I would suggest also adding a KASSERT to vfs_deleteopt(). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Thu Aug 7 23:53:13 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07B36106566B for ; Thu, 7 Aug 2008 23:53:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from gigi.cs.uoguelph.ca (gigi.cs.uoguelph.ca [131.104.94.210]) by mx1.freebsd.org (Postfix) with ESMTP id 9C0568FC16 for ; Thu, 7 Aug 2008 23:53:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by gigi.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m77Nr8Z6023527; Thu, 7 Aug 2008 19:53:08 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m7804TM08475; Thu, 7 Aug 2008 20:04:29 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Thu, 7 Aug 2008 20:04:29 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Doug Rabson In-Reply-To: <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> Message-ID: References: <86myk06e18.fsf@ds4.des.no> <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.94.210 Cc: freebsd-fs@freebsd.org, =?utf-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= Subject: Re: Which GSSAPI library does FreeBSD use? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Aug 2008 23:53:13 -0000 On Mon, 4 Aug 2008, Doug Rabson wrote: > > Try using current - I updated heimdal to 1.1 in current. > > The GSS-API implementation in 7.x and current is a plugin system which > heimdal's krb5 code plugs into as a GSS-API mechanism provider. With heimdal > 1.1, it also supports spnego and ntlm as plugins. > Well, vanilla Heimdal-1.1 seems to work fine. However, when I try to link to the libraries in FreeBSD-CURRENT, I get a bunch of multiply defined globals, because it gets both external.o and gss_names.o, out of libgssapi.a and libgssapi_krb5.a respectively. Btw, I was able to use gss_inquire_sec_context_by_oid() to get the session key out of the security context. rick From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 03:39:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE0A71065670 for ; Fri, 8 Aug 2008 03:39:02 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id BBD318FC16 for ; Fri, 8 Aug 2008 03:39:02 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 609B71CC0B7; Thu, 7 Aug 2008 20:39:02 -0700 (PDT) Date: Thu, 7 Aug 2008 20:39:02 -0700 From: Jeremy Chadwick To: Boris Kotzev Message-ID: <20080808033902.GA72860@eos.sc1.parodius.com> References: <200808071925.45786.boris.kotzev@gmail.com> <20080807165502.GA39420@eos.sc1.parodius.com> <200808072040.55571.boris.kotzev@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200808072040.55571.boris.kotzev@gmail.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 03:39:02 -0000 On Thu, Aug 07, 2008 at 08:40:55PM +0300, Boris Kotzev wrote: > ?? Thursday 07 August 2008 19:55:02 Jeremy Chadwick ??????: > > On Thu, Aug 07, 2008 at 07:25:45PM +0300, Boris Kotzev wrote: > > > Hello, > > > > > > I used the zfs port to Mac OS X (http://zfs.macosforge.org) to > > > create a storage pool under Mac OS X. The pool can be imported > > > successfully under FreeBSD: > > > > > > root:~-114# zpool import macpool > > > root:~-115# zpool list macpool > > > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > > > macpool 6,94G 510K 6,94G 0% ONLINE - > > > root:~-116# zfs list macpool > > > NAME USED AVAIL REFER MOUNTPOINT > > > macpool 474K 6,83G 308K /macpool > > > > > > and is fully accessible to the root user: > > > > > > root:~-118# id > > > uid=0(root) gid=0(wheel) groups=0(wheel),5(operator) > > > root:~-119# ls -ld /macpool > > > drwxr-xr-x 7 root wheel 8 7 ??? 16:59 /macpool > > > root:~-120# ls -l /macpool > > > total 43 > > > drwx------ 3 root wheel 3 7 ??? 16:31 .Spotlight-V100 > > > -rw-r--r-- 1 root wheel 35014 7 ??? 16:31 .VolumeIcon.icns > > > drwx------ 2 root wheel 4 7 ??? 16:32 .fseventsd > > > drwxr-xr-x 2 root wheel 2 7 ??? 16:59 backup > > > drwxr-xr-x 2 root wheel 2 7 ??? 16:59 downloads > > > drwxr-xr-x 2 root wheel 2 7 ??? 16:58 music > > > > > > According to the file permissions on /macpool (drwxr-xr-x), > > > anyone should have read access to it. This is not the case > > > though: > > > > > > root:~-121# su user > > > % id > > > uid=1003(user) gid=1003(user) > > > groups=1003(user),0(wheel),5(operator) % ls -l /macpool > > > ls: /macpool: Permission denied > > > % cd /macpool > > > /macpool: Permission denied. > > > > > > Is this a bug, or is there some way to get access to /macpool as > > > an ordinary user? > > > > > > The pool was created under version zfs-119 of the Mac OS X port; > > > the FreeBSD version is: > > > > > > root:~-122# uname -a > > > FreeBSD xxxx 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Aug 2 > > > 14:19:33 EEST 2008 root@xxxx:/usr/obj/usr/src/sys/MACBOOK amd64 > > > > > > with the latest zfs patch, but the problem was also present > > > before applying the patch. > > > > As root, what does "zfs get all macpool" return on FreeBSD? > > root@:~-116# zfs get all macpool > NAME PROPERTY VALUE SOURCE > macpool type filesystem - > macpool creation ?? ??? 7 16:31 2008 - > macpool used 474K - > macpool available 6,83G - > macpool referenced 308K - > macpool compressratio 1.00x - > macpool mounted yes - > macpool quota none default > macpool reservation none default > macpool recordsize 128K default > macpool mountpoint /macpool default > macpool sharenfs off default > macpool checksum on default > macpool compression off default > macpool atime on default > macpool devices on default > macpool exec on default > macpool setuid on default > macpool readonly off default > macpool jailed off default > macpool snapdir hidden default > macpool aclmode groupmask default > macpool aclinherit restricted default > macpool canmount on default > macpool shareiscsi off default > macpool xattr off temporary > macpool copies 1 default > macpool version 1 - > macpool utf8only off - > macpool normalization none - > macpool casesensitivity sensitive - > macpool vscan off default > macpool nbmand off default > macpool sharesmb off default > macpool refquota none default > macpool refreservation none default It's interesting to note that your filesystem has a significantly larger number of properties returned than mine. I wonder if the ZFS code has support for those properties on FreeBSD, but they simply aren't listed. Or maybe the patch you're using adds all of them? I don't know. Anyway, the property that may be relevant is aclinherit. The zfs(1) manpage on FreeBSD makes no mention of what "restricted" means for property "aclinherit". I believe it may be the source of the problem. A ZFS filesystem made on FreeBSD has a different value for that property. I explicitly enabled compression on the below fs, BTW, which is why that value is not the default value: NAME PROPERTY VALUE SOURCE storage type filesystem - storage creation Sun May 25 19:33 2008 - storage used 183G - storage available 730G - storage referenced 183G - storage compressratio 1.02x - storage mounted yes - storage quota none default storage reservation none default storage recordsize 128K default storage mountpoint /storage default storage sharenfs off default storage checksum on default storage compression on local storage atime off local storage devices on default storage exec on default storage setuid on default storage readonly off default storage jailed off default storage snapdir hidden default storage aclmode groupmask default storage aclinherit secure default storage canmount on default storage shareiscsi off default storage xattr off temporary storage copies 1 default -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 04:24:08 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31710106566C; Fri, 8 Aug 2008 04:24:08 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp12.yandex.ru (smtp12.yandex.ru [77.88.32.82]) by mx1.freebsd.org (Postfix) with ESMTP id 373B18FC13; Fri, 8 Aug 2008 04:24:06 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from ns.kirov.so-cdu.ru ([77.72.136.145]:22005 "EHLO [127.0.0.1]" smtp-auth: "bu7cher" TLS-CIPHER: "DHE-RSA-AES256-SHA keybits 256/256 version TLSv1/SSLv3" TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S1525076AbYHHEYF (ORCPT + 2 others); Fri, 8 Aug 2008 08:24:05 +0400 X-Yandex-Spam: 1 X-Yandex-Front: smtp12 X-Yandex-TimeMark: 1218169445 X-MsgDayCount: 4 X-Comment: RFC 2476 MSA function at smtp12.yandex.ru logged sender identity as: bu7cher Message-ID: <489BCA4D.3050704@yandex.ru> Date: Fri, 08 Aug 2008 08:23:41 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla Thunderbird 1.5 (FreeBSD/20051231) MIME-Version: 1.0 To: Jeremy Chadwick References: <6c3c36d00808062109y6ae176a0ha055129392b00542@mail.gmail.com> <20080807044759.GA7505@eos.sc1.parodius.com> <6c3c36d00808062212y4e9a1464i48e146e84725a36e@mail.gmail.com> <6c3c36d00808062235v5cbb4470v990b76d569f85614@mail.gmail.com> <20080807055841.GB9735@eos.sc1.parodius.com> <489A9739.20707@yandex.ru> <20080807071434.GA15465@eos.sc1.parodius.com> <489ADD89.8070809@mawer.org> <20080807121245.GA26629@eos.sc1.parodius.com> In-Reply-To: <20080807121245.GA26629@eos.sc1.parodius.com> Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Scott Long Subject: Re: zpool degraded - 'UNAVAIL cannot open' functioning drive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 04:24:08 -0000 Jeremy Chadwick wrote: > In almost every case I've looked at so far, the individuals' chipsets, > disks, and overall setup are different. SMART statistics on the drives > show absolutely no sign of errors, or anything that indicates a hardware > failure. Many of the users are using AHCI as well (myself included, and > I have seen the DMA error issue myself), which is more reliable than > classic IDE. I have done some work on AHCI part of ATA driver and I am looking for testers... http://perforce.freebsd.org/changeList.cgi?CMD=changes&FSPC=//depot/user/butcher/src/... > It would be benefitial if there was some form of sysctl to increase the > verbosity from the ATA subsystem when an error happens. The existing > data we get back is terse, and barely useful. I know for a fact there's > more debug information that could be output in such scenarios. And > please do not reply with "good idea, send patches" unless you're wanting > to be chewed out. :-) Ok, I'll try to add some verbose 'printfs' in my branch in perforce :) >> I'm going to do some analysis and find out whether I can find any of our >> systems that may be experiencing ATA errors that don't correlate with >> what their SMART data is saying. To date I haven't caught any, but >> that's not to say they may not be happening... just that all of the ones >> I have caught to date do appear to have been hardware-related issues... IMHO. Today we have many hardware versions and revisions and some of them are buggy. But another OSes (windows, linux) work with buggy hardware without big problems. Yes, some developers have docs and can make workarounds.. I think our ata driver needs new error handling subsystem, which can correctly handle errors. -- WBR, Andrey V. Elsukov From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 06:37:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27D5A106564A for ; Fri, 8 Aug 2008 06:37:02 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from kuber.nabble.com (kuber.nabble.com [216.139.236.158]) by mx1.freebsd.org (Postfix) with ESMTP id ED5298FC08 for ; Fri, 8 Aug 2008 06:37:01 +0000 (UTC) (envelope-from bounces@nabble.com) Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1KRLau-0000b4-Rl for freebsd-fs@freebsd.org; Thu, 07 Aug 2008 23:37:00 -0700 Message-ID: <18886364.post@talk.nabble.com> Date: Thu, 7 Aug 2008 23:37:00 -0700 (PDT) From: Maximillian Dornseif To: freebsd-fs@freebsd.org In-Reply-To: <18881178.post@talk.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: md@hudora.de References: <18880785.post@talk.nabble.com> <18881178.post@talk.nabble.com> Subject: Re: Strange (?) hangs with ZFS/rsync. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 06:37:02 -0000 Maximillian Dornseif wrote: > > The machine does not reboot but hangs during the "syncing" phase of > shutdown. It has to be powercycled to restart. > after a shutdown -r now the box complains that it couldn't terminate vnlru, bufdaemon and syncer. -- View this message in context: http://www.nabble.com/Strange-%28-%29-hangs-with-ZFS-rsync.-tp18880785p18886364.html Sent from the freebsd-fs mailing list archive at Nabble.com. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 08:08:01 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74023106567B for ; Fri, 8 Aug 2008 08:08:01 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from itchy.rabson.org (unknown [IPv6:2002:50b1:e8f2:1::143]) by mx1.freebsd.org (Postfix) with ESMTP id 299608FC1A for ; Fri, 8 Aug 2008 08:08:01 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc] (unknown [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc]) by itchy.rabson.org (Postfix) with ESMTP id 1F2B13FB6; Fri, 8 Aug 2008 09:07:18 +0100 (BST) Message-Id: From: Doug Rabson To: Rick Macklem In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v928.1) Date: Fri, 8 Aug 2008 09:07:59 +0100 References: <86myk06e18.fsf@ds4.des.no> <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> X-Mailer: Apple Mail (2.928.1) Cc: freebsd-fs@freebsd.org, =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= Subject: Re: Which GSSAPI library does FreeBSD use? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 08:08:01 -0000 On 8 Aug 2008, at 01:04, Rick Macklem wrote: > > > On Mon, 4 Aug 2008, Doug Rabson wrote: >> >> Try using current - I updated heimdal to 1.1 in current. >> >> The GSS-API implementation in 7.x and current is a plugin system >> which heimdal's krb5 code plugs into as a GSS-API mechanism >> provider. With heimdal 1.1, it also supports spnego and ntlm as >> plugins. >> > Well, vanilla Heimdal-1.1 seems to work fine. However, when I try to > link > to the libraries in FreeBSD-CURRENT, I get a bunch of multiply defined > globals, because it gets both external.o and gss_names.o, out of > libgssapi.a and libgssapi_krb5.a respectively. Don't use static linking? > > Btw, I was able to use gss_inquire_sec_context_by_oid() to get the > session key out of the security context. Excellent. From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 11:01:30 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C1101065671 for ; Fri, 8 Aug 2008 11:01:30 +0000 (UTC) (envelope-from gary.jennejohn@freenet.de) Received: from mout0.freenet.de (mout0.freenet.de [IPv6:2001:748:100:40::2:2]) by mx1.freebsd.org (Postfix) with ESMTP id 9D7E08FC17 for ; Fri, 8 Aug 2008 11:01:29 +0000 (UTC) (envelope-from gary.jennejohn@freenet.de) Received: from [195.4.92.22] (helo=12.mx.freenet.de) by mout0.freenet.de with esmtpa (ID gary.jennejohn@freenet.de) (port 25) (Exim 4.69 #65) id 1KRPiq-0005SO-CR for freebsd-fs@freebsd.org; Fri, 08 Aug 2008 13:01:28 +0200 Received: from ma651.m.pppool.de ([89.49.166.81]:50105 helo=peedub.jennejohn.org) by 12.mx.freenet.de with esmtpa (ID gary.jennejohn@freenet.de) (port 25) (Exim 4.69 #12) id 1KRPiq-0005aQ-2O for freebsd-fs@freebsd.org; Fri, 08 Aug 2008 13:01:28 +0200 Date: Fri, 8 Aug 2008 13:01:27 +0200 From: Gary Jennejohn To: freebsd-fs@freebsd.org Message-ID: <20080808130127.4cc71ac9@peedub.jennejohn.org> In-Reply-To: <20080808033902.GA72860@eos.sc1.parodius.com> References: <200808071925.45786.boris.kotzev@gmail.com> <20080807165502.GA39420@eos.sc1.parodius.com> <200808072040.55571.boris.kotzev@gmail.com> <20080808033902.GA72860@eos.sc1.parodius.com> X-Mailer: Claws Mail 3.5.0 (GTK+ 2.10.14; amd64-portbld-freebsd8.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gary.jennejohn@freenet.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 11:01:30 -0000 On Thu, 7 Aug 2008 20:39:02 -0700 Jeremy Chadwick wrote: > On Thu, Aug 07, 2008 at 08:40:55PM +0300, Boris Kotzev wrote: [snip] > > macpool aclinherit restricted default > > macpool canmount on default > > macpool shareiscsi off default > > macpool xattr off temporary > > macpool copies 1 default > > macpool version 1 - > > macpool utf8only off - > > macpool normalization none - > > macpool casesensitivity sensitive - > > macpool vscan off default > > macpool nbmand off default > > macpool sharesmb off default > > macpool refquota none default > > macpool refreservation none default > > It's interesting to note that your filesystem has a significantly larger > number of properties returned than mine. I wonder if the ZFS code has > support for those properties on FreeBSD, but they simply aren't listed. > Or maybe the patch you're using adds all of them? I don't know. > > Anyway, the property that may be relevant is aclinherit. The zfs(1) > manpage on FreeBSD makes no mention of what "restricted" means for > property "aclinherit". I believe it may be the source of the problem. > > A ZFS filesystem made on FreeBSD has a different value for that > property. I explicitly enabled compression on the below fs, BTW, which > is why that value is not the default value: > No, it doesn't necessarily. Here the output from a ZFS FS made with FreeBSD but using the old version 6 ZFS: root:peedub:~:bash:1> zfs get all mirpool NAME PROPERTY VALUE SOURCE mirpool type filesystem - mirpool creation Sat Nov 24 17:53 2007 - mirpool used 141G - mirpool available 316G - mirpool referenced 18K - mirpool compressratio 1.00x - mirpool mounted yes - mirpool quota none default mirpool reservation none default mirpool recordsize 128K default mirpool mountpoint /mirpool default mirpool sharenfs off local mirpool checksum on default mirpool compression off default mirpool atime on default mirpool devices on default mirpool exec on default mirpool setuid on default mirpool readonly off default mirpool jailed off default mirpool snapdir hidden default mirpool aclmode groupmask default mirpool aclinherit restricted default <== mirpool canmount on default mirpool shareiscsi off default mirpool xattr off temporary mirpool copies 1 default mirpool version 1 - mirpool utf8only off - mirpool normalization none - mirpool casesensitivity sensitive - mirpool vscan off default mirpool nbmand off default mirpool sharesmb off default mirpool refquota none default mirpool refreservation none default root:peedub:~:bash:2> zfs set aclinherit=secure mirpool property 'aclinherit' not supported on FreeBSD: permission denied Apparently it's not really used. > NAME PROPERTY VALUE SOURCE > storage type filesystem - > storage creation Sun May 25 19:33 2008 - > storage used 183G - > storage available 730G - > storage referenced 183G - > storage compressratio 1.02x - > storage mounted yes - > storage quota none default > storage reservation none default > storage recordsize 128K default > storage mountpoint /storage default > storage sharenfs off default > storage checksum on default > storage compression on local > storage atime off local > storage devices on default > storage exec on default > storage setuid on default > storage readonly off default > storage jailed off default > storage snapdir hidden default > storage aclmode groupmask default > storage aclinherit secure default <== > storage canmount on default > storage shareiscsi off default > storage xattr off temporary > storage copies 1 default > --- Gary Jennejohn From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 11:26:13 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6436106566B for ; Fri, 8 Aug 2008 11:26:13 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id C242A8FC17 for ; Fri, 8 Aug 2008 11:26:13 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 890C91CC0B0; Fri, 8 Aug 2008 04:26:13 -0700 (PDT) Date: Fri, 8 Aug 2008 04:26:13 -0700 From: Jeremy Chadwick To: Gary Jennejohn Message-ID: <20080808112613.GA91032@eos.sc1.parodius.com> References: <200808071925.45786.boris.kotzev@gmail.com> <20080807165502.GA39420@eos.sc1.parodius.com> <200808072040.55571.boris.kotzev@gmail.com> <20080808033902.GA72860@eos.sc1.parodius.com> <20080808130127.4cc71ac9@peedub.jennejohn.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080808130127.4cc71ac9@peedub.jennejohn.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 11:26:13 -0000 On Fri, Aug 08, 2008 at 01:01:27PM +0200, Gary Jennejohn wrote: > On Thu, 7 Aug 2008 20:39:02 -0700 > Jeremy Chadwick wrote: > > > On Thu, Aug 07, 2008 at 08:40:55PM +0300, Boris Kotzev wrote: > [snip] > > > macpool aclinherit restricted default > > > macpool canmount on default > > > macpool shareiscsi off default > > > macpool xattr off temporary > > > macpool copies 1 default > > > macpool version 1 - > > > macpool utf8only off - > > > macpool normalization none - > > > macpool casesensitivity sensitive - > > > macpool vscan off default > > > macpool nbmand off default > > > macpool sharesmb off default > > > macpool refquota none default > > > macpool refreservation none default > > > > It's interesting to note that your filesystem has a significantly larger > > number of properties returned than mine. I wonder if the ZFS code has > > support for those properties on FreeBSD, but they simply aren't listed. > > Or maybe the patch you're using adds all of them? I don't know. > > > > Anyway, the property that may be relevant is aclinherit. The zfs(1) > > manpage on FreeBSD makes no mention of what "restricted" means for > > property "aclinherit". I believe it may be the source of the problem. > > > > A ZFS filesystem made on FreeBSD has a different value for that > > property. I explicitly enabled compression on the below fs, BTW, which > > is why that value is not the default value: > > No, it doesn't necessarily. Here the output from a ZFS FS made with > FreeBSD but using the old version 6 ZFS: > > root:peedub:~:bash:1> zfs get all mirpool > NAME PROPERTY VALUE SOURCE > mirpool type filesystem - > mirpool creation Sat Nov 24 17:53 2007 - > mirpool used 141G - > mirpool available 316G - > mirpool referenced 18K - > mirpool compressratio 1.00x - > mirpool mounted yes - > mirpool quota none default > mirpool reservation none default > mirpool recordsize 128K default > mirpool mountpoint /mirpool default > mirpool sharenfs off local > mirpool checksum on default > mirpool compression off default > mirpool atime on default > mirpool devices on default > mirpool exec on default > mirpool setuid on default > mirpool readonly off default > mirpool jailed off default > mirpool snapdir hidden default > mirpool aclmode groupmask default > mirpool aclinherit restricted default <== > mirpool canmount on default > mirpool shareiscsi off default > mirpool xattr off temporary > mirpool copies 1 default > mirpool version 1 - > mirpool utf8only off - > mirpool normalization none - > mirpool casesensitivity sensitive - > mirpool vscan off default > mirpool nbmand off default > mirpool sharesmb off default > mirpool refquota none default > mirpool refreservation none default > > root:peedub:~:bash:2> zfs set aclinherit=secure mirpool > property 'aclinherit' not supported on FreeBSD: permission denied > > Apparently it's not really used. You need to remember the individual is using the patch on CURRENT provided by pjd, which bring ZFS up to the latest OpenSolaris version. It's possible on that version it *is* implemented; I do not know. Based on the manpage description for aclinherit, that option could definitely cause what he's seeing. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 11:51:15 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29DA71065670 for ; Fri, 8 Aug 2008 11:51:15 +0000 (UTC) (envelope-from gary.jennejohn@freenet.de) Received: from mout3.freenet.de (mout3.freenet.de [IPv6:2001:748:100:40::2:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8F6B38FC15 for ; Fri, 8 Aug 2008 11:51:14 +0000 (UTC) (envelope-from gary.jennejohn@freenet.de) Received: from [195.4.92.24] (helo=14.mx.freenet.de) by mout3.freenet.de with esmtpa (ID gary.jennejohn@freenet.de) (port 25) (Exim 4.69 #19) id 1KRQUz-0003T9-3p for freebsd-fs@freebsd.org; Fri, 08 Aug 2008 13:51:13 +0200 Received: from ma651.m.pppool.de ([89.49.166.81]:51757 helo=peedub.jennejohn.org) by 14.mx.freenet.de with esmtpa (ID gary.jennejohn@freenet.de) (port 25) (Exim 4.69 #12) id 1KRQUy-0007aC-SP for freebsd-fs@freebsd.org; Fri, 08 Aug 2008 13:51:13 +0200 Date: Fri, 8 Aug 2008 13:51:12 +0200 From: Gary Jennejohn To: freebsd-fs@freebsd.org Message-ID: <20080808135112.7bf59d83@peedub.jennejohn.org> X-Mailer: Claws Mail 3.5.0 (GTK+ 2.10.14; amd64-portbld-freebsd8.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Fw: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gary.jennejohn@freenet.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 11:51:15 -0000 Oops, forgot to include the ML. Begin forwarded message: Date: Fri, 8 Aug 2008 13:49:38 +0200 From: Gary Jennejohn To: Jeremy Chadwick Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges On Fri, 8 Aug 2008 04:26:13 -0700 Jeremy Chadwick wrote: > On Fri, Aug 08, 2008 at 01:01:27PM +0200, Gary Jennejohn wrote: [BIG snip] > > mirpool aclinherit restricted default <== > > mirpool canmount on default > > mirpool shareiscsi off default > > mirpool xattr off temporary > > mirpool copies 1 default > > mirpool version 1 - > > mirpool utf8only off - > > mirpool normalization none - > > mirpool casesensitivity sensitive - > > mirpool vscan off default > > mirpool nbmand off default > > mirpool sharesmb off default > > mirpool refquota none default > > mirpool refreservation none default > > > > root:peedub:~:bash:2> zfs set aclinherit=secure mirpool > > property 'aclinherit' not supported on FreeBSD: permission denied > > > > Apparently it's not really used. > > You need to remember the individual is using the patch on CURRENT > provided by pjd, which bring ZFS up to the latest OpenSolaris version. > It's possible on that version it *is* implemented; I do not know. > > Based on the manpage description for aclinherit, that option could > definitely cause what he's seeing. > I _am_ using the patched version from pjd@. garyj:peedub:freebsd:-bash:11> grep ZFS /var/run/dmesg.boot WARNING: ZFS is considered to be an experimental feature in FreeBSD. ZFS filesystem version 11 ZFS storage pool version 11 --- Gary Jennejohn From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 13:00:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB0E91065673 for ; Fri, 8 Aug 2008 13:00:54 +0000 (UTC) (envelope-from boris.kotzev@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.191]) by mx1.freebsd.org (Postfix) with ESMTP id 22A838FC1F for ; Fri, 8 Aug 2008 13:00:53 +0000 (UTC) (envelope-from boris.kotzev@gmail.com) Received: by mu-out-0910.google.com with SMTP id i2so802540mue.3 for ; Fri, 08 Aug 2008 06:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:subject:date :user-agent:references:in-reply-to:cc:mime-version:content-type :content-transfer-encoding:content-disposition:message-id; bh=cpEMvDPooophpBUYQzhBCmp0F6nLAFt2mLCPvekQ518=; b=oFfJef/u/aMiNwfZMFvsDQBskToCcheMxdjdRQS5Fc+45lVK6osz4M4zdl7NNfYwdz VP2cxoPmujA7uuf2bOCxNSt/AsUmlt/mdfR8RyhAiJLzlZvNGP062SV8RvPQoPIRae/e af+7TyAjoQHesmxX8dxeOqtOjQjXP5RmMxZUU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:references:in-reply-to:cc :mime-version:content-type:content-transfer-encoding :content-disposition:message-id; b=N2XI8LPW0GkymIyU33SPbCldI2uVu4hXYgQMwgizjxkCyD6VogmKbTeAxnICFAiDXz J/DAiMv2UOaTHhKelJJqppDt1ex8aM3ZuTq8loVuOkqMN6IPwRDHq2u/HKkhZdEWlj/k 0hdmN4WTmmxLEaXTZWntPYTECbfoo4E5C28pE= Received: by 10.103.5.4 with SMTP id h4mr3089499mui.17.1218200452459; Fri, 08 Aug 2008 06:00:52 -0700 (PDT) Received: from zembla.universe ( [213.169.62.7]) by mx.google.com with ESMTPS id j10sm855026muh.18.2008.08.08.06.00.49 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 08 Aug 2008 06:00:50 -0700 (PDT) From: Boris Kotzev To: Jeremy Chadwick Date: Fri, 8 Aug 2008 16:00:47 +0300 User-Agent: KMail/1.9.7 References: <200808071925.45786.boris.kotzev@gmail.com> <200808072040.55571.boris.kotzev@gmail.com> <20080808033902.GA72860@eos.sc1.parodius.com> In-Reply-To: <20080808033902.GA72860@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: 8bit Content-Disposition: inline Message-Id: <200808081600.47603.boris.kotzev@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 13:00:55 -0000 Íà Friday 08 August 2008 06:39:02 íàïèñàõòå: > On Thu, Aug 07, 2008 at 08:40:55PM +0300, Boris Kotzev wrote: > > ?? Thursday 07 August 2008 19:55:02 Jeremy Chadwick ??????: > > > On Thu, Aug 07, 2008 at 07:25:45PM +0300, Boris Kotzev wrote: > > > > Hello, > > > > > > > > I used the zfs port to Mac OS X (http://zfs.macosforge.org) > > > > to create a storage pool under Mac OS X. The pool can be > > > > imported successfully under FreeBSD: > > > > > > > > root:~-114# zpool import macpool > > > > root:~-115# zpool list macpool > > > > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > > > > macpool 6,94G 510K 6,94G 0% ONLINE - > > > > root:~-116# zfs list macpool > > > > NAME USED AVAIL REFER MOUNTPOINT > > > > macpool 474K 6,83G 308K /macpool > > > > > > > > and is fully accessible to the root user: > > > > > > > > root:~-118# id > > > > uid=0(root) gid=0(wheel) groups=0(wheel),5(operator) > > > > root:~-119# ls -ld /macpool > > > > drwxr-xr-x 7 root wheel 8 7 ??? 16:59 /macpool > > > > root:~-120# ls -l /macpool > > > > total 43 > > > > drwx------ 3 root wheel 3 7 ??? 16:31 .Spotlight-V100 > > > > -rw-r--r-- 1 root wheel 35014 7 ??? 16:31 > > > > .VolumeIcon.icns drwx------ 2 root wheel 4 7 ??? > > > > 16:32 .fseventsd drwxr-xr-x 2 root wheel 2 7 ??? > > > > 16:59 backup drwxr-xr-x 2 root wheel 2 7 ??? 16:59 > > > > downloads drwxr-xr-x 2 root wheel 2 7 ??? 16:58 music > > > > > > > > According to the file permissions on /macpool (drwxr-xr-x), > > > > anyone should have read access to it. This is not the case > > > > though: > > > > > > > > root:~-121# su user > > > > % id > > > > uid=1003(user) gid=1003(user) > > > > groups=1003(user),0(wheel),5(operator) % ls -l /macpool > > > > ls: /macpool: Permission denied > > > > % cd /macpool > > > > /macpool: Permission denied. > > > > > > > > Is this a bug, or is there some way to get access to /macpool > > > > as an ordinary user? > > > > > > > > The pool was created under version zfs-119 of the Mac OS X > > > > port; the FreeBSD version is: > > > > > > > > root:~-122# uname -a > > > > FreeBSD xxxx 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Aug 2 > > > > 14:19:33 EEST 2008 root@xxxx:/usr/obj/usr/src/sys/MACBOOK > > > > amd64 > > > > > > > > with the latest zfs patch, but the problem was also present > > > > before applying the patch. > > > > > > As root, what does "zfs get all macpool" return on FreeBSD? > > > > root@:~-116# zfs get all macpool > > NAME PROPERTY VALUE SOURCE > > macpool type filesystem - > > macpool creation ?? ??? 7 16:31 2008 - > > macpool used 474K - > > macpool available 6,83G - > > macpool referenced 308K - > > macpool compressratio 1.00x - > > macpool mounted yes - > > macpool quota none default > > macpool reservation none default > > macpool recordsize 128K default > > macpool mountpoint /macpool default > > macpool sharenfs off default > > macpool checksum on default > > macpool compression off default > > macpool atime on default > > macpool devices on default > > macpool exec on default > > macpool setuid on default > > macpool readonly off default > > macpool jailed off default > > macpool snapdir hidden default > > macpool aclmode groupmask default > > macpool aclinherit restricted default > > macpool canmount on default > > macpool shareiscsi off default > > macpool xattr off temporary > > macpool copies 1 default > > macpool version 1 - > > macpool utf8only off - > > macpool normalization none - > > macpool casesensitivity sensitive - > > macpool vscan off default > > macpool nbmand off default > > macpool sharesmb off default > > macpool refquota none default > > macpool refreservation none default > > It's interesting to note that your filesystem has a significantly > larger number of properties returned than mine. I wonder if the > ZFS code has support for those properties on FreeBSD, but they > simply aren't listed. Or maybe the patch you're using adds all of > them? I don't know. > The extra properties appeared after applying the ZFS patches. The newer versions of zfs and zpool exhibit more poperties than zpool version 6 and zfs version 1: % zpool upgrade -v This system is currently running ZFS pool version 11. The following versions are supported: VER DESCRIPTION --- -------------------------------------------------------- 1 Initial ZFS version 2 Ditto blocks (replicated metadata) 3 Hot spares and double parity RAID-Z 4 zpool history 5 Compression using the gzip algorithm 6 bootfs pool property 7 Separate intent log devices 8 Delegated administration 9 refquota and refreservation properties 10 Cache devices 11 Improved scrub performance For more information on a particular version, including supported releases, see: http://www.opensolaris.org/os/community/zfs/version/N Where 'N' is the version number. % zfs upgrade -v The following filesystem versions are supported: VER DESCRIPTION --- -------------------------------------------------------- 1 Initial ZFS filesystem version 2 Enhanced directory entries 3 Case insensitive and File system unique identifer (FUID) For more information on a particular version, including supported releases, see: http://www.opensolaris.org/os/community/zfs/version/zpl/N Where 'N' is the version number. > Anyway, the property that may be relevant is aclinherit. The > zfs(1) manpage on FreeBSD makes no mention of what "restricted" > means for property "aclinherit". I believe it may be the source of > the problem. This property has different values under FreeBSD and Mac OS X. It is shown as "secure" in Mac OS X: sh-3.2# zfs get aclinherit macpool NAME PROPERTY VALUE SOURCE macpool aclinherit secure default It is not possible to change the value inder FreeBSD: root@:/-112# zfs set aclinherit=discard macpool property 'aclinherit' not supported on FreeBSD: permission denied I set the value under Mac OS X to "discard" but the change did not seem to make any difference. > > A ZFS filesystem made on FreeBSD has a different value for that > property. I explicitly enabled compression on the below fs, BTW, > which is why that value is not the default value: > > NAME PROPERTY VALUE SOURCE > storage type filesystem - > storage creation Sun May 25 19:33 2008 - > storage used 183G - > storage available 730G - > storage referenced 183G - > storage compressratio 1.02x - > storage mounted yes - > storage quota none default > storage reservation none default > storage recordsize 128K default > storage mountpoint /storage default > storage sharenfs off default > storage checksum on default > storage compression on local > storage atime off local > storage devices on default > storage exec on default > storage setuid on default > storage readonly off default > storage jailed off default > storage snapdir hidden default > storage aclmode groupmask default > storage aclinherit secure default > storage canmount on default > storage shareiscsi off default > storage xattr off temporary > storage copies 1 default It is also possible to import a pool created under FreeBSD to Mac OS X but whenever I write to the pool in Mac OS X and then try to read the entries in FreeBSD, I encounter the same problem: the entries created under Mac OS X are accessible by the root user only. I also noticed that all entries in a FreeBSD pool acquired ACL's in Mac OS X. For example the etc directory of FreeBSD has the following ACL in MAC OS X: sh-3.2# ls -lde etc drwxr-xr-x+ 19 root wheel 122 7 Àâã 18:39 etc 0: group:nogroup deny This ACL looks suspicious to me though when I compare it to the ACL's on the Mac OS X hfs+ volume: sh-3.2# ls -lde /Applications drwxrwxr-x+ 49 root admin 1666 6 Àâã 21:27 /Applications 0: group:everyone deny delete Can the problem be related to the fact that I run the AMD 64 version of FreeBSD? Thanks, Boris Kotzev From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 13:12:41 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B1D21065675 for ; Fri, 8 Aug 2008 13:12:41 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 69D558FC13 for ; Fri, 8 Aug 2008 13:12:41 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 280D61CC0B1; Fri, 8 Aug 2008 06:12:41 -0700 (PDT) Date: Fri, 8 Aug 2008 06:12:41 -0700 From: Jeremy Chadwick To: Gary Jennejohn Message-ID: <20080808131241.GA94716@eos.sc1.parodius.com> References: <20080808135112.7bf59d83@peedub.jennejohn.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080808135112.7bf59d83@peedub.jennejohn.org> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-fs@freebsd.org Subject: Re: Fw: zfs - no access to a Mac OS X zfs pool without root privileges X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 13:12:41 -0000 On Fri, Aug 08, 2008 at 01:51:12PM +0200, Gary Jennejohn wrote: > Oops, forgot to include the ML. > > Begin forwarded message: > > Date: Fri, 8 Aug 2008 13:49:38 +0200 > From: Gary Jennejohn > To: Jeremy Chadwick > Subject: Re: zfs - no access to a Mac OS X zfs pool without root privileges > > > On Fri, 8 Aug 2008 04:26:13 -0700 > Jeremy Chadwick wrote: > > > On Fri, Aug 08, 2008 at 01:01:27PM +0200, Gary Jennejohn wrote: > [BIG snip] > > > mirpool aclinherit restricted default <== > > > mirpool canmount on default > > > mirpool shareiscsi off default > > > mirpool xattr off temporary > > > mirpool copies 1 default > > > mirpool version 1 - > > > mirpool utf8only off - > > > mirpool normalization none - > > > mirpool casesensitivity sensitive - > > > mirpool vscan off default > > > mirpool nbmand off default > > > mirpool sharesmb off default > > > mirpool refquota none default > > > mirpool refreservation none default > > > > > > root:peedub:~:bash:2> zfs set aclinherit=secure mirpool > > > property 'aclinherit' not supported on FreeBSD: permission denied > > > > > > Apparently it's not really used. > > > > You need to remember the individual is using the patch on CURRENT > > provided by pjd, which bring ZFS up to the latest OpenSolaris version. > > It's possible on that version it *is* implemented; I do not know. > > > > Based on the manpage description for aclinherit, that option could > > definitely cause what he's seeing. > > > > I _am_ using the patched version from pjd@. > > garyj:peedub:freebsd:-bash:11> grep ZFS /var/run/dmesg.boot > WARNING: ZFS is considered to be an experimental feature in FreeBSD. > ZFS filesystem version 11 > ZFS storage pool version 11 Then I don't have an explanation. The only thing different I see about the filesystem is that aclinherit option is "restricted" (and the value of that option, I believe, could cause what you're seeing), while on ZFS fs/pool version 6, the default appears to be "secure". Possibly they're the same, just renamed? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 14:18:08 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02DDB1065670 for ; Fri, 8 Aug 2008 14:18:08 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from aeryn.cs.uoguelph.ca (aeryn.cs.uoguelph.ca [131.104.20.160]) by mx1.freebsd.org (Postfix) with ESMTP id B04FB8FC13 for ; Fri, 8 Aug 2008 14:18:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by aeryn.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m78EI4sp027966; Fri, 8 Aug 2008 10:18:04 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m78ETPG17936; Fri, 8 Aug 2008 10:29:25 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 8 Aug 2008 10:29:25 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Doug Rabson In-Reply-To: Message-ID: References: <86myk06e18.fsf@ds4.des.no> <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.20.161 Cc: freebsd-fs@freebsd.org, =?utf-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= Subject: Re: Which GSSAPI library does FreeBSD use? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 14:18:08 -0000 On Thu, 7 Aug 2008, Rick Macklem wrote: > > > On Mon, 4 Aug 2008, Doug Rabson wrote: >> >> Try using current - I updated heimdal to 1.1 in current. >> >> The GSS-API implementation in 7.x and current is a plugin system which >> heimdal's krb5 code plugs into as a GSS-API mechanism provider. With >> heimdal 1.1, it also supports spnego and ntlm as plugins. >> > Well, vanilla Heimdal-1.1 seems to work fine. However, when I try to link > to the libraries in FreeBSD-CURRENT, I get a bunch of multiply defined > globals, because it gets both external.o and gss_names.o, out of > libgssapi.a and libgssapi_krb5.a respectively. > Oops, spoke too soon. It worked for a mount last night, but couldn't re-acquire fresh credentials this morning. (There are slightly different problems with Heimdal-0.8 and Heimdal-1.1, but they both seem related to getting a TGT via the keytab entry.) I'm going to try contacting the Heimdal folks. (In the meantime, I'm back to Heimdal-0.7 which works fine.) If you're doing RPCSEC_GSS for the NLM, you are probably going to want this to work too. (Solaris uses a keytab entry with root/.@ in it for root accesse.) rick From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 14:24:31 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 533581065692 for ; Fri, 8 Aug 2008 14:24:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from aeryn.cs.uoguelph.ca (aeryn.cs.uoguelph.ca [131.104.20.160]) by mx1.freebsd.org (Postfix) with ESMTP id E6DD38FC19 for ; Fri, 8 Aug 2008 14:24:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by aeryn.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id m78EOSqm030144; Fri, 8 Aug 2008 10:24:28 -0400 Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id m78EZp719243; Fri, 8 Aug 2008 10:35:51 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 8 Aug 2008 10:35:51 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Doug Rabson In-Reply-To: Message-ID: References: <86myk06e18.fsf@ds4.des.no> <326AF658-D96D-4410-9E32-0001FF8264AA@rabson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Scanned-By: MIMEDefang 2.63 on 131.104.20.161 Cc: freebsd-fs@freebsd.org, =?utf-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= Subject: Re: Which GSSAPI library does FreeBSD use? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 14:24:31 -0000 On Fri, 8 Aug 2008, Doug Rabson wrote: > > Don't use static linking? > When I linked without "-static", it would crash in the gss_acquire_cred() call. (For some reason, I couldn't find a core dump left anywhere.) rick From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 20:40:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04CB2106566B for ; Fri, 8 Aug 2008 20:40:02 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id AC9F98FC13 for ; Fri, 8 Aug 2008 20:40:01 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1KRYkg-00063I-9d for freebsd-fs@freebsd.org; Fri, 08 Aug 2008 20:39:58 +0000 Received: from 89-172-47-83.adsl.net.t-com.hr ([89.172.47.83]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Aug 2008 20:39:58 +0000 Received: from ivoras by 89-172-47-83.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 08 Aug 2008 20:39:58 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Fri, 08 Aug 2008 22:39:39 +0200 Lines: 50 Message-ID: References: <18880785.post@talk.nabble.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB3EC48105323958CF540777B" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-47-83.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) In-Reply-To: <18880785.post@talk.nabble.com> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: Strange (?) hangs with ZFS/rsync. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 20:40:02 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB3EC48105323958CF540777B Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Maximillian Dornseif wrote: > I have a ZFS based backup server which rsyncs from a dozen other machin= es. > There is only a single rsync active for any point in time and the syste= m is > used for nothing else. It has an amd64 kernel and 4GB RAM. I tried to f= ollow > the ZFS tuning guide. >=20 > For a few weeks the machine works well. Since monday it rsync always ha= ngs > when fetching a big logfile from a remote machine.=20 >=20 > The two rsync processes are in state "zfs:&b" and "zfs:lo" and can't be= > killed. > The other strange thing is that the system is reporting 3367M (!) wired= > memory. These look like well known problems with ZFS (see=20 http://wiki.freebsd.org/ZFSKnownProblems). A new version of the ZFS port = is currently under development, if you want to try it, you need a=20 8-CURRENT machine and this patch: http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004887.html People have reported that the new patch resolves hangs such as yours. --------------enigB3EC48105323958CF540777B Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFInK8LldnAQVacBcgRAmczAJ44+ltgRwBflcOnzs4t6MTlXGzZiACdGy0I wd1mTI2sxnh9mV9f0gdTrs0= =QQv+ -----END PGP SIGNATURE----- --------------enigB3EC48105323958CF540777B-- From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 22:05:59 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF0451065670; Fri, 8 Aug 2008 22:05:59 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from smtp.infidyne.com (ds9.infidyne.com [88.80.6.206]) by mx1.freebsd.org (Postfix) with ESMTP id ABE058FC17; Fri, 8 Aug 2008 22:05:59 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se (c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se [85.229.22.169]) by smtp.infidyne.com (Postfix) with ESMTPSA id 07A5D8472D; Sat, 9 Aug 2008 00:05:58 +0200 (CEST) From: Peter Schuller To: Pawel Jakub Dawidek Date: Sat, 9 Aug 2008 00:07:15 +0200 User-Agent: KMail/1.9.7 References: <200807262005.54235.peter.schuller@infidyne.com> <200807272026.54907.peter.schuller@infidyne.com> <20080806185909.GC2580@garage.freebsd.pl> In-Reply-To: <20080806185909.GC2580@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart46921207.vcrJ7sQpCh"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200808090007.25865.peter.schuller@infidyne.com> Cc: freebsd-fs@freebsd.org Subject: Re: Asynchronous writing to zvols (ZFS) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 22:06:00 -0000 --nextPart46921207.vcrJ7sQpCh Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline [zvol write performance less than file-on-zfs] > Not sure why's that, I spent no time on optimizing ZVOL yet, sorry. Absolutely, I was not complaining! I just felt it was worth mentioning. It = was=20 not meant to be negative criticism. > With the patch above we synchoronize in-memory transactions every 5 > seconds or when queue is full or when we receive BIO_FLUSH.=20 That was my understanding. Sorry, I probably wasn't being clear. I thought= =20 your original comment ("The problem is that we don't between async and sync= =20 I/O request on GEOM level") implied that there was some reason one could no= t=20 trust bio_cmd to be preserved correctly (somehow being downgraded from=20 BIO_FLUSH before it reaches the zvol class). Because if this was the case = it=20 would make perfect sense under the circumstances to treat all writes as=20 flushes, in order to achieve correct semantics for *actual* flushes. So my question was meant to confirm that this was not the case, because if = it=20 were, the patch would mean that *actual* flushes would not get treated as=20 such. Thanks, =2D-=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --nextPart46921207.vcrJ7sQpCh Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEABECAAYFAkicw50ACgkQDNor2+l1i30KQQCg2rGWUR8BACnZ9Yu7x6VpSWhG zdIAoKMXIoljbq6lhcaE5ChgUOA3CpVF =HrQV -----END PGP SIGNATURE----- --nextPart46921207.vcrJ7sQpCh-- From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 22:18:38 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 623D31065687 for ; Fri, 8 Aug 2008 22:18:38 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from smtp.infidyne.com (ds9.infidyne.com [88.80.6.206]) by mx1.freebsd.org (Postfix) with ESMTP id 1E2788FC2F for ; Fri, 8 Aug 2008 22:18:37 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se (c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se [85.229.22.169]) by smtp.infidyne.com (Postfix) with ESMTPSA id 70A9984766; Sat, 9 Aug 2008 00:18:36 +0200 (CEST) From: Peter Schuller To: freebsd-fs@freebsd.org Date: Sat, 9 Aug 2008 00:19:54 +0200 User-Agent: KMail/1.9.7 References: <489B2FFE.5050406@barryp.org> In-Reply-To: <489B2FFE.5050406@barryp.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1719046.TdaERK221C"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200808090020.04315.peter.schuller@infidyne.com> Cc: Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 22:18:38 -0000 --nextPart1719046.TdaERK221C Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline > >>> http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm > > I think CDW is mistaken in saying it's a PCI-E card, UIO is a > proprietary Supermicro bus that some of their motherboards support. > > http://www.supermicro.com/products/nfo/UIO.cfm As far as I can tell the USAS-L8i on the supermicro page is claimed by them= to=20 be PCI-E. Or are you saying the one on CDW is actually not the same card? =2D-=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --nextPart1719046.TdaERK221C Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEABECAAYFAkicxpQACgkQDNor2+l1i30CPQCgrevDEMujQNMXy7iycYJ+ZGXp zLEAn1kH1uQLeHwGRDLKyhR0V9HZJ2+W =AIpS -----END PGP SIGNATURE----- --nextPart1719046.TdaERK221C-- From owner-freebsd-fs@FreeBSD.ORG Fri Aug 8 22:27:13 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EEFA1065671 for ; Fri, 8 Aug 2008 22:27:13 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from smtp.infidyne.com (ds9.infidyne.com [88.80.6.206]) by mx1.freebsd.org (Postfix) with ESMTP id 1A1538FC1D for ; Fri, 8 Aug 2008 22:27:12 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se (c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se [85.229.22.169]) by smtp.infidyne.com (Postfix) with ESMTPSA id AB4A684785; Sat, 9 Aug 2008 00:27:11 +0200 (CEST) From: Peter Schuller To: Wes Morgan Date: Sat, 9 Aug 2008 00:28:38 +0200 User-Agent: KMail/1.9.7 References: <200808061928.37001.peter.schuller@infidyne.com> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2116982.ssuW4itSP7"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200808090028.39729.peter.schuller@infidyne.com> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Aug 2008 22:27:13 -0000 --nextPart2116982.ssuW4itSP7 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline > http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm > > http://www.lsi.com/storage_home/products_home/standard_product_ics/sas_ic= s/ >lsisas1068e/index.html > > Not much onboard ram, but it's PCI-E and even SAS. CDW lists it for $155. > That would be cheaper than buying a new board with a PCI-X slot or two, > and would even handle SAS drives. Claims to be based on the "LSISAS 1068E > SAS controller". Any idea if that is supported? I don't see it listed in > the mfi man page. LSI has a Linux driver for download. That card looks > like it would be just what I need. Well: =2E/dev/mpt/mpilib/mpi_cnfg.h:#define MPI_MANUFACTPAGE_DEVID_SAS1068E = =20 (0x0058) =2E/dev/mpt/mpilib/mpi_ioc.h: * 03-11-05 01.05.08 Added family code for = 1068E=20 family. =2E/dev/mpt/mpilib/mpi_ioc.h:#define MPI_FW_HEADER_PID_FAMILY_106xE_SAS = =20 (0x0004) /* 1068E, 1066E, and 1064E */ And mpt(4) has "LSI Logic AS1064, LSI Logic AS1068 (SAS/SATA)" listed as=20 supported. So as far as I can tell it should be supported, assuming it truly works in= =20 practice. Given past history in trying to find suitable controllers I have to be=20 paranoid and wonder what I am missing... surely I cannot have found the=20 perfect controller. Does anyone have hands-on experience? I especially like the SAS support; particularly since I have recently notic= ed=20 the availability of low-cost 7k2rpm SAS drives. =2D-=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --nextPart2116982.ssuW4itSP7 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEABECAAYFAkicyJcACgkQDNor2+l1i314dQCg2/QkT/TOg35FzF9MzS7+SedM k/oAn2gLVK3g28Tqyx+xLX/YN44pU+sk =T5aR -----END PGP SIGNATURE----- --nextPart2116982.ssuW4itSP7-- From owner-freebsd-fs@FreeBSD.ORG Sat Aug 9 02:03:03 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 951EF106567F for ; Sat, 9 Aug 2008 02:03:03 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id 7CCC68FC17 for ; Sat, 9 Aug 2008 02:03:03 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id C147F33C62; Fri, 8 Aug 2008 19:03:22 -0700 (PDT) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 4AC0733C5B; Fri, 8 Aug 2008 19:03:22 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18588.64214.354495.804458@almost.alerce.com> Date: Fri, 8 Aug 2008 19:03:02 -0700 To: Peter Schuller In-Reply-To: <200808090020.04315.peter.schuller@infidyne.com> References: <489B2FFE.5050406@barryp.org> <200808090020.04315.peter.schuller@infidyne.com> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 02:03:03 -0000 Peter Schuller writes: > > >>> http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm > > > > I think CDW is mistaken in saying it's a PCI-E card, UIO is a > > proprietary Supermicro bus that some of their motherboards support. > > > > http://www.supermicro.com/products/nfo/UIO.cfm > > As far as I can tell the USAS-L8i on the supermicro page is claimed by them to > be PCI-E. Or are you saying the one on CDW is actually not the same card? > I've been googling around trying to figure out if UIO is the same thing as PCI-E. Supermicro has a bunch of motherboards for which the description explicitly points out PCI-E and UIO slots, which makes it sounds like they're different beasts. Or, it could be that a UIO slot is specifically a PCI-Ex8 slot? You can buy risers that convert "1U PCI-E (x16) to 1 UIO and 1 PCI-E " Supermicro's description of the card does explicitly say that it "uses a PCI Express host interface", but maybe they mean that they're using it in some nonstandard fashion? It's confusing... g. From owner-freebsd-fs@FreeBSD.ORG Sat Aug 9 02:08:59 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6EEC1065675 for ; Sat, 9 Aug 2008 02:08:59 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (merlin.alerce.com [64.62.142.94]) by mx1.freebsd.org (Postfix) with ESMTP id AB53A8FC1A for ; Sat, 9 Aug 2008 02:08:59 +0000 (UTC) (envelope-from hartzell@alerce.com) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id 39C1F33C6C; Fri, 8 Aug 2008 19:09:19 -0700 (PDT) Received: from merlin.alerce.com (localhost [127.0.0.1]) by merlin.alerce.com (Postfix) with ESMTP id CFC5B33C64; Fri, 8 Aug 2008 19:09:18 -0700 (PDT) From: George Hartzell MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18588.64570.786730.812483@almost.alerce.com> Date: Fri, 8 Aug 2008 19:08:58 -0700 To: Ivan Voras In-Reply-To: References: <18880785.post@talk.nabble.com> X-Mailer: VM 7.19 under Emacs 22.1.50.1 X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-fs@freebsd.org Subject: Re: Strange (?) hangs with ZFS/rsync. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: hartzell@alerce.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 02:08:59 -0000 Ivan Voras writes: > Maximillian Dornseif wrote: > > I have a ZFS based backup server which rsyncs from a dozen other machines. > > There is only a single rsync active for any point in time and the system is > > used for nothing else. It has an amd64 kernel and 4GB RAM. I tried to follow > > the ZFS tuning guide. > > > > For a few weeks the machine works well. Since monday it rsync always hangs > > when fetching a big logfile from a remote machine. > > > > The two rsync processes are in state "zfs:&b" and "zfs:lo" and can't be > > killed. > > The other strange thing is that the system is reporting 3367M (!) wired > > memory. > > These look like well known problems with ZFS (see > http://wiki.freebsd.org/ZFSKnownProblems). A new version of the ZFS port > is currently under development, if you want to try it, you need a > 8-CURRENT machine and this patch: > http://lists.freebsd.org/pipermail/freebsd-fs/2008-July/004887.html > > People have reported that the new patch resolves hangs such as yours. Not to be a party pooper, but on an arguably underconfigured server (2GB RAM, dual core AMD, amd64), moving to -CURRENT and applying the new patch actually required that I tune it with an even lower arc_max than I had been getting away with on the -STABLE zfs with the same hardware. My testing (described on this list a week or two back) is admittedly unscientific and it may be that the -STABLE configuration was already skating on thin ice but it's certainly not the case that the subsystem now autotunes itself on limited hardware. But, with kernel mem and arc configured as many folks have described, both versions run nicely in spite of anything I've thrown at them and the zfs feature set is great. Thanks again to Pawel and all involved! g. From owner-freebsd-fs@FreeBSD.ORG Sat Aug 9 07:16:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 753691065676 for ; Sat, 9 Aug 2008 07:16:07 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from smtp.infidyne.com (ds9.infidyne.com [88.80.6.206]) by mx1.freebsd.org (Postfix) with ESMTP id 340538FC21 for ; Sat, 9 Aug 2008 07:16:06 +0000 (UTC) (envelope-from peter.schuller@infidyne.com) Received: from c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se (c-a916e555.03-51-73746f3.cust.bredbandsbolaget.se [85.229.22.169]) by smtp.infidyne.com (Postfix) with ESMTPSA id 4497484B6B; Sat, 9 Aug 2008 09:16:05 +0200 (CEST) From: Peter Schuller To: hartzell@alerce.com Date: Sat, 9 Aug 2008 09:17:23 +0200 User-Agent: KMail/1.9.7 References: <200808090020.04315.peter.schuller@infidyne.com> <18588.64214.354495.804458@almost.alerce.com> In-Reply-To: <18588.64214.354495.804458@almost.alerce.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2552340.nhHxSF47av"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200808090917.34149.peter.schuller@infidyne.com> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 07:16:07 -0000 --nextPart2552340.nhHxSF47av Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline > Or, it could be that a UIO slot is specifically a PCI-Ex8 slot? You > can buy risers that convert "1U PCI-E (x16) to 1 UIO and 1 PCI-E " http://www.supermicro.com/products/nfo/UIO_cards.cfm This one actually says in the clear: "8-Lane PCI-Express interface (Supermicro UIO slot)" My guess is one of the following: * It means nothing but "PCI-E", and the UIO stuff is just marketing BS. * It means "extra PCI-E", and the UIO stuff is just marketing BS. * (Based on the grapical animation on the UIO page) They actually do have a= =20 special slot on their motherboard for use with their special UIO card which= =20 then provides a few extra PCI-E slots. The mentioning of UIO on pages=20 describing standard PCI-E cards is just marketing BS resulting from the=20 technically correct fact that they can be used with their UIO card. If I don't find specific information to the contrary I'll probably chance i= t=20 and see. =2D-=20 / Peter Schuller PGP userID: 0xE9758B7D or 'Peter Schuller ' Key retrieval: Send an E-Mail to getpgpkey@scode.org E-Mail: peter.schuller@infidyne.com Web: http://www.scode.org --nextPart2552340.nhHxSF47av Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEABECAAYFAkidRI4ACgkQDNor2+l1i33dUwCgkjdJj3SJPBpBaoFDq9OekHRX uyQAnAyZy5m0k/+d5xAgP+roFyzqns56 =TEVQ -----END PGP SIGNATURE----- --nextPart2552340.nhHxSF47av-- From owner-freebsd-fs@FreeBSD.ORG Sat Aug 9 09:03:38 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F7B11065671 for ; Sat, 9 Aug 2008 09:03:38 +0000 (UTC) (envelope-from dimitar.vassilev@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.155]) by mx1.freebsd.org (Postfix) with ESMTP id D2EFF8FC18 for ; Sat, 9 Aug 2008 09:03:37 +0000 (UTC) (envelope-from dimitar.vassilev@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so1652456fgb.35 for ; Sat, 09 Aug 2008 02:03:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:mime-version:content-type; bh=O/7l5HXZ/pebJnT8Q/Tthb9PWzyzor9112WUetXak+Y=; b=CTz3M5VrracpHtKm+QDRsdCu0WqxkoUXYrC+9REIbdmDxwl/WPJSRg7MSlt/SkCZst iZkt/Sxv51iz6KDtAX+ld9fHpk6HMcsNAcigM9TGsuxWFTiVJyFgMP9uW3kivQ/6GtFc MoF9iO4xKtFwMWy/kDLoZcJldQI1b7n4F27hQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:mime-version:content-type; b=unqK4xzXy+kqNmiUw7XQVrnQXXaknMzFRmLVerkJlpXpDGld1iAAYY3V0l4wESxyaC dF2JHlACCAHJOZk0lAq9Ja89PojvgVrsUZwIqCy3za1YkQovhssQQPxCVTllSG1NEQsT PBl7tYWBrcvuKGHmpSWTtKCs9PaTJTwgbqbKU= Received: by 10.86.31.18 with SMTP id e18mr4315688fge.52.1218271123955; Sat, 09 Aug 2008 01:38:43 -0700 (PDT) Received: by 10.86.100.10 with HTTP; Sat, 9 Aug 2008 01:38:43 -0700 (PDT) Message-ID: <59adc1a0808090138t7ab9913bmc08d42fb56801e0d@mail.gmail.com> Date: Sat, 9 Aug 2008 11:38:43 +0300 From: "Dimitar Vasilev" To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pjd@freebsd.org Subject: zfs snapshot panic problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 09:03:38 -0000 Hi all, I'm having a problem with a 7-stable SMP amd64 machine running zfs snapshots for backups. It starts to complain about bad file descriptors after running 8 days without a problem. Then we try to unmount the problem fs and we got a system crash. Kernel config ident FOO include GENERIC nooptions SCHED_4BSD options SCHED_ULE options GEOM_JOURNAL options ALTQ options ALTQ_CBQ options ALTQ_RED options ALTQ_RIO options ALTQ_HFSC options ALTQ_PRIQ options ALTQ_NOPCC options DEVICE_POLLING options ZERO_COPY_SOCKETS options HZ=2000 # device pf device pflog #logging support interface for PF device pfsync #synchronization interface for PF device carp #Common Address Redundancy Protocol Here is output of backtrace: kgdb -c /var/crash/vmcore.7 /boot/kernel/kernel GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xc0 fault code = supervisor write data, page not present instruction pointer = 0x8:0xffffffff804c2515 stack pointer = 0x10:0xffffffffd77e4980 frame pointer = 0x10:0xffffffffd77e4a20 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 19469 (zfs) trap number = 12 panic: page fault cpuid = 0 Uptime: 8d12h7m45s Physical memory: 2034 MB Dumping 1417 MB: 1402 1386 1370 1354 1338 1322 1306 1290 1274 1258 1242 1226 1210 1194 1178 1162 1146 1130 1114 1098 1082 1066 1050 1034 1018 1002 986 970 954 938 922 906 890 874 858 842 826 810 794 778 762 746 730 714 698 682 666 650 634 618 602 586 570 554 538 522 506 490 474 458 442 426 410 394 378 362 346 330 314 298 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_journal.ko Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/fdescfs.ko Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /boot/kernel/pflog.ko.symbols...done. done. Loaded symbols for /boot/kernel/pflog.ko Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done. done. Loaded symbols for /boot/kernel/pf.ko Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done. done. Loaded symbols for /boot/kernel/accf_http.ko #0 doadump () at pcpu.h:194 194 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff804ba839 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #3 0xffffffff804bac3d in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:572 #4 0xffffffff807871c4 in trap_fatal (frame=0xffffff00015c7360, eva=18446742974224558304) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff80787595 in trap_pfault (frame=0xffffffffd77e48d0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641 #6 0xffffffff80787ed8 in trap (frame=0xffffffffd77e48d0) at /usr/src/sys/amd64/amd64/trap.c:410 #7 0xffffffff8076d8ae in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #8 0xffffffff804c2515 in _sx_xlock (sx=0xa0, opts=0, file=0xffffffff80cb88e0 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c", line=1069) at atomic.h:142 #9 0xffffffff80c9fb3a in zfsctl_umount_snapshots (vfsp=Variable "vfsp" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c:1069 #10 0xffffffff80ca6988 in zfs_umount (vfsp=0xffffff0001560a68, fflag=0, td=0xffffff00015c7360) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:692 #11 0xffffffff80533dbe in dounmount (mp=0xffffff0001560a68, flags=0, td=0xffffff00015c7360) at /usr/src/sys/kern/vfs_mount.c:1286 #12 0xffffffff8053458e in unmount (td=0xffffff00015c7360, uap=0xffffffffd77e4be0) at /usr/src/sys/kern/vfs_mount.c:1182 #13 0xffffffff80787817 in syscall (frame=0xffffffffd77e4c70) at /usr/src/sys/amd64/amd64/trap.c:852 #14 0xffffffff8076dabb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:290 #15 0x0000000800f1514c in ?? () It's the same machine mentioned in: http://lists.freebsd.org/pipermail/freebsd-fs/2008-February/004377.html http://lists.freebsd.org/pipermail/freebsd-fs/2008-February/004418.html Any ideas how to fix? Can compile kernel with debug if needed. Best regards, Dimitar Vassilev From owner-freebsd-fs@FreeBSD.ORG Sat Aug 9 12:56:09 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC0A41065671 for ; Sat, 9 Aug 2008 12:56:09 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from warped.bluecherry.net (unknown [IPv6:2001:440:eeee:fffb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1F5EC8FC14 for ; Sat, 9 Aug 2008 12:56:09 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from volatile.chemikals.org (unknown [74.193.170.223]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by warped.bluecherry.net (Postfix) with ESMTPSA id 28B7F86A2975; Sat, 9 Aug 2008 07:56:06 -0500 (CDT) Received: from localhost (morganw@localhost [127.0.0.1]) by volatile.chemikals.org (8.14.2/8.14.2) with ESMTP id m79Cu0n3001553; Sat, 9 Aug 2008 07:56:01 -0500 (CDT) (envelope-from morganw@chemikals.org) Date: Sat, 9 Aug 2008 07:56:00 -0500 (CDT) From: Wes Morgan To: Peter Schuller In-Reply-To: <200808090917.34149.peter.schuller@infidyne.com> Message-ID: References: <200808090020.04315.peter.schuller@infidyne.com> <18588.64214.354495.804458@almost.alerce.com> <200808090917.34149.peter.schuller@infidyne.com> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 12:56:09 -0000 On Sat, 9 Aug 2008, Peter Schuller wrote: >> Or, it could be that a UIO slot is specifically a PCI-Ex8 slot? You >> can buy risers that convert "1U PCI-E (x16) to 1 UIO and 1 PCI-E " > > http://www.supermicro.com/products/nfo/UIO_cards.cfm > > This one actually says in the clear: > > "8-Lane PCI-Express interface (Supermicro UIO slot)" > > My guess is one of the following: > > * It means nothing but "PCI-E", and the UIO stuff is just marketing BS. > > * It means "extra PCI-E", and the UIO stuff is just marketing BS. > > * (Based on the grapical animation on the UIO page) They actually do have a > special slot on their motherboard for use with their special UIO card which > then provides a few extra PCI-E slots. The mentioning of UIO on pages > describing standard PCI-E cards is just marketing BS resulting from the > technically correct fact that they can be used with their UIO card. > > If I don't find specific information to the contrary I'll probably chance it > and see. As you say, it's hard to tell from the SuperMicro page. LSI has this card: http://lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3081er/index.html http://www.newegg.com/Product/Product.aspx?Item=N82E16816118092 Which is their official card with the same chipset. The slots on both the SuperMicro and the LSI cards look mighty similar. The only obvious difference is the little right-angle thing next to the PCI-E interface. I'm very interested to see if it works out. If I wasn't going to be away for a few weeks I'd try to abuse CDW's return policy to give it a test. From owner-freebsd-fs@FreeBSD.ORG Sat Aug 9 20:58:33 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B182D1065672 for ; Sat, 9 Aug 2008 20:58:33 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello087206045140.chello.pl [87.206.45.140]) by mx1.freebsd.org (Postfix) with ESMTP id 584E28FC23 for ; Sat, 9 Aug 2008 20:58:33 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id E353045C8C; Sat, 9 Aug 2008 22:58:31 +0200 (CEST) Received: from localhost (chello087206045140.chello.pl [87.206.45.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 35700456AB; Sat, 9 Aug 2008 22:58:27 +0200 (CEST) Date: Sat, 9 Aug 2008 22:58:35 +0200 From: Pawel Jakub Dawidek To: Weldon S Godfrey 3 Message-ID: <20080809205835.GE1363@garage.freebsd.pl> References: <20080806101621.H24586@emmett.excelsus.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZInfyf7laFu/Kiw7" Content-Disposition: inline In-Reply-To: <20080806101621.H24586@emmett.excelsus.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS-NFS kernel panic under load X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Aug 2008 20:58:33 -0000 --ZInfyf7laFu/Kiw7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 06, 2008 at 11:00:57AM -0400, Weldon S Godfrey 3 wrote: >=20 > Hello, >=20 > Please forgive me, I didn't really see this discussed in the archives but= =20 > I am wondering if anyone has seen this issue. I can replicate this issue= =20 > under FreeBSD amd64 7.0-RELEASE and the latest -STABLE (RELENG_7). I do= =20 > not replicate any problems running 9 instances of postmark on the machine= =20 > directly, so the issue appears to be isolated with NFS. The backtrace you posted doesn't suggest it is ZFS problem, although it can of course be. Can you try to reproduce it with NFS-exported UFS file system? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --ZInfyf7laFu/Kiw7 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFIngT7ForvXbEpPzQRAhyCAKC9PRNZ+k8O+3lIhqsM8EoQ+ZV0NACfbM3X IDKxgoxi7pufjA0DKj1XFnU= =I7jC -----END PGP SIGNATURE----- --ZInfyf7laFu/Kiw7--