From owner-freebsd-fs@FreeBSD.ORG Sun Mar 14 01:50:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A67AB1065709; Sun, 14 Mar 2010 01:50:39 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.24]) by mx1.freebsd.org (Postfix) with ESMTP id 4432E8FC14; Sun, 14 Mar 2010 01:50:38 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 5so702612qwi.7 for ; Sat, 13 Mar 2010 17:50:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=N0HzVyndFN1w7vjzBRUOdjZ/xWdseQ4lmHQVwrIlTuY=; b=OrMnlj68LR5wMHDb4o7WI4rDX6QH2niqtbOp1kSxrRMuQYaSoEC7yiek9VLMK3Td/S N/lXb8rdY2S2QKmtBU63rCdYogi0gAsRGsK0uSqB/qYzzrkmgt1P7bK9ev/qeObqTYLN vg/ah5DlupH4yXz1ZDnXaqMWmlkAinXLKqLME= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=qbPTseKr6Ce1wRZJxe3Euve+VB2ShcsREZLiv2pngSLuJFTWdDNh4qWQJvgydjP0m6 ac/nrZ2Z1PGA+1ttAmXB2WPJah7RkK+mALbZzsSFme+F38ooFge1Pyo6Q7Mn5rnuvoEq +uAqcomrRfMwtE7iHOG0FIwloZjH9u5uiFgno= Received: by 10.229.251.69 with SMTP id mr5mr1546426qcb.91.1268531438526; Sat, 13 Mar 2010 17:50:38 -0800 (PST) Received: from ppp-21.234.dialinfree.com (ppp-21.234.dialinfree.com [209.172.21.234]) by mx.google.com with ESMTPS id 21sm2251474qyk.1.2010.03.13.17.50.35 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 13 Mar 2010 17:50:37 -0800 (PST) Sender: "J. Hellenthal" Date: Sat, 13 Mar 2010 20:40:57 -0500 From: jhell To: Alexander Zagrebin In-Reply-To: <3A28259E0677447BBFDECFCCDBD97FD5@vosz.local> Message-ID: References: <3A28259E0677447BBFDECFCCDBD97FD5@vosz.local> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS allows deletion of files in a sticky directory X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2010 01:50:39 -0000 On Fri, 19 Feb 2010 18:23, alexz@ wrote: > I have found that directory entry may be deleted from a ZFS directory > with the sticky bit, if "the entry is a plain file and you have write > access" > (this is citation from a comments in zfs_dir.c) > But this behavior isn't described in the sticky(8) and isn't allowed on a > UFS. > The attached patch provides the UFS-like behavior of a sticky directories on > a ZFS. > Is this bug or feature? > > Perhaps you have removed a directory on a share that is managed through Samba and somehow you have had a ACL entry that allowed you to remove that directory ?. This patch is unsuitable for implementation. It effectively removes ACL access for determining writes to a object that you have ACL write access to. Regards, -- jhell From owner-freebsd-fs@FreeBSD.ORG Sun Mar 14 03:26:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3CAC106566C for ; Sun, 14 Mar 2010 03:26:47 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id DC0AE8FC19 for ; Sun, 14 Mar 2010 03:26:47 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rmac.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.71 (FreeBSD)) (envelope-from ) id 1NqeTW-0006wZ-CF for freebsd-fs@freebsd.org; Sun, 14 Mar 2010 03:26:46 +0000 Date: Sun, 14 Mar 2010 12:26:45 +0900 Message-ID: From: Randy Bush To: freebsd-fs User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Subject: degraded zfs slowdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2010 03:26:48 -0000 i lost a drive on a remote server. i had to use tw_cli from single user at boot time to remove it from the controller as it was making the controller unusable. so now, while waiting for the replacement drive to ship in, i have # df Filesystem 1024-blocks Used Avail Capacity Mounted on /dev/twed0s1a 253678 198102 35282 85% / /dev/twed0s1h 63254 2414 55780 4% /root tank 154191872 16256 154175616 0% /tank tank/usr 173331328 19155712 154175616 11% /usr tank/usr/home 213014784 58839168 154175616 28% /usr/home tank/var 157336192 3160576 154175616 2% /var tank/var/spool 154475392 299776 154175616 0% /var/spool /dev/md0 126702 156 116410 0% /tmp devfs 1 1 0 100% /dev procfs 4 4 0 100% /proc and # zpool status pool: tank state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 twed1 REMOVED 0 2 0 twed2 ONLINE 0 0 0 errors: No known data errors but the system is extremely soggy and hard to light. do i need to do some sort of remove at the zfs layer? randy From owner-freebsd-fs@FreeBSD.ORG Sun Mar 14 15:09:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79CE2106566C for ; Sun, 14 Mar 2010 15:09:26 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id 61AD18FC15 for ; Sun, 14 Mar 2010 15:09:26 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rmac.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.71 (FreeBSD)) (envelope-from ) id 1NqpRV-0008l4-Ti for freebsd-fs@freebsd.org; Sun, 14 Mar 2010 15:09:26 +0000 Date: Mon, 15 Mar 2010 00:09:24 +0900 Message-ID: From: Randy Bush To: freebsd-fs In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Subject: Re: degraded zfs slowdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2010 15:09:26 -0000 i managed to tell the controller to ignore the bad drive. now i have # zpool status pool: tank state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 twed1 FAULTED 0 6 0 corrupted data twed1 ONLINE 0 0 0 errors: No known data errors with zfs whining Mar 14 15:05:20 psg root: ZFS: vdev failure, zpool=tank type=vdev.corrupt_data Mar 14 15:05:50 psg root: ZFS: vdev failure, zpool=tank type=vdev.corrupt_data Mar 14 15:07:50 psg last message repeated 4 times as the ghostly bad drive does not have a unique name, [how] can i tell zfs to remove it for the moment? randy From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 06:20:06 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DF1E106566B; Mon, 15 Mar 2010 06:20:06 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 23B3A8FC15; Mon, 15 Mar 2010 06:20:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2F6K6jT048129; Mon, 15 Mar 2010 06:20:06 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2F6K5Pc048125; Mon, 15 Mar 2010 06:20:06 GMT (envelope-from linimon) Date: Mon, 15 Mar 2010 06:20:06 GMT Message-Id: <201003150620.o2F6K5Pc048125@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144755: [iwi] [panic] iwi panic when issuing /etc/rc.d/netif restart on 8-STABLE r205159 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 06:20:06 -0000 Old Synopsis: [iwi] iwi panic when issuing /etc/rc.d/netif restart on 8-STABLE r205159 New Synopsis: [iwi] [panic] iwi panic when issuing /etc/rc.d/netif restart on 8-STABLE r205159 Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Mon Mar 15 06:19:52 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=144755 From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 08:31:45 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F1C4106564A; Mon, 15 Mar 2010 08:31:45 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 25ABC8FC12; Mon, 15 Mar 2010 08:31:45 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2F8Vjqe097991; Mon, 15 Mar 2010 08:31:45 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2F8VjTT097987; Mon, 15 Mar 2010 08:31:45 GMT (envelope-from linimon) Date: Mon, 15 Mar 2010 08:31:45 GMT Message-Id: <201003150831.o2F8VjTT097987@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-fs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144755: [iwi] [panic] iwi panic when issuing /etc/rc.d/netif restart on 8-STABLE r205159 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 08:31:45 -0000 Synopsis: [iwi] [panic] iwi panic when issuing /etc/rc.d/netif restart on 8-STABLE r205159 Responsible-Changed-From-To: freebsd-fs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Mon Mar 15 08:31:21 UTC 2010 Responsible-Changed-Why: oops. pointed out by: brucec http://www.freebsd.org/cgi/query-pr.cgi?pr=144755 From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 08:32:13 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DE81106567B; Mon, 15 Mar 2010 08:32:13 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id C53318FC26; Mon, 15 Mar 2010 08:32:12 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop2.sarenet.es (Postfix) with ESMTP id 2867073646; Mon, 15 Mar 2010 09:32:11 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20100313152429.GH3209@garage.freebsd.pl> Date: Mon, 15 Mar 2010 09:32:10 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <27A6DF40-2EA8-4CFC-9E42-DD995E2F9342@sarenet.es> <20100313152429.GH3209@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@freebsd.org Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 08:32:13 -0000 On Mar 13, 2010, at 4:24 PM, Pawel Jakub Dawidek wrote: > On Fri, Mar 12, 2010 at 11:12:49AM +0100, Borja Marcos wrote: >>=20 >> On Mar 10, 2010, at 6:31 PM, Pawel Jakub Dawidek wrote: >>=20 >>> Hmm, interesting. Especially those two traces: >>>=20 >>> Tracing command zfs pid 1820 tid 100105 td 0xffffff0002ca4000 >>=20 >> Just in case something was wrong, I wiped the two virtual machines = and started everything again. The two virtual systems were acting a bit = weird, giving lots of LORs. I don't know why, maybe something was = corrupted, these virtual machines have been panicked and restarted many = times. >>=20 >> So, experiment repeated again, fresh start. Still deadlocking pretty = soon if I start, say, 5 or 6 tar processes. Traces follow. >=20 > Are you able to dump the kernel memory and make it available for me > somewhere? I'd need to look at it wit GDB, as DDB is not enough here, > I'm afraid. Sure. Do you have access to VMWare Fusion? I can create a snapshot of = the deadlocked system and make it available to you. Borja. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 10:10:00 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 14E25106566C; Mon, 15 Mar 2010 10:10:00 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id DF3988FC0A; Mon, 15 Mar 2010 10:09:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2FA9xY2074064; Mon, 15 Mar 2010 10:09:59 GMT (envelope-from delphij@freefall.freebsd.org) Received: (from delphij@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2FA9x6s074060; Mon, 15 Mar 2010 10:09:59 GMT (envelope-from delphij) Date: Mon, 15 Mar 2010 10:09:59 GMT Message-Id: <201003151009.o2FA9x6s074060@freefall.freebsd.org> To: delphij@FreeBSD.org, freebsd-fs@FreeBSD.org, delphij@FreeBSD.org From: delphij@FreeBSD.org Cc: Subject: Re: kern/144720: [zfs][patch] zfs list improvements (vendor patches) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 10:10:00 -0000 Synopsis: [zfs][patch] zfs list improvements (vendor patches) Responsible-Changed-From-To: freebsd-fs->delphij Responsible-Changed-By: delphij Responsible-Changed-When: Mon Mar 15 10:09:39 UTC 2010 Responsible-Changed-Why: Take, I'll work on this. http://www.freebsd.org/cgi/query-pr.cgi?pr=144720 From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 11:07:11 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C86F11065670 for ; Mon, 15 Mar 2010 11:07:11 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9B6378FC1B for ; Mon, 15 Mar 2010 11:07:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2FB7B8a026852 for ; Mon, 15 Mar 2010 11:07:11 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2FB7AtH026850 for freebsd-fs@FreeBSD.org; Mon, 15 Mar 2010 11:07:10 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 15 Mar 2010 11:07:10 GMT Message-Id: <201003151107.o2FB7AtH026850@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 11:07:11 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/144458 fs [nfs] [patch] nfsd fails as a kld o kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization o kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144402 fs [zfs] [panic] panic at zfs_znode_dmu_init: existing zn o kern/144330 fs [nfs] mbuf leakage in nfsd with zfs o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o bin/144214 fs zfsboot fails on gang block after upgrade to zfs v14 o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o kern/143345 fs [ext2fs] [patch] extfs minor header cleanups to better o kern/143343 fs [zfs] bug in sunlink flag on directories o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142924 fs [ext2fs] [patch] Small cleanup for the inode struct in o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142872 fs [zfs] ZFS ZVOL Lockmgr Deadlock o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142594 fs [zfs] Modification time reset to 1 Jan 1970 after fsyn o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs/ufs Lock order reversal o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141718 fs [zfs] [panic] kernel panic when 'zfs rename' is used o o kern/141685 fs [zfs] zfs corruption on adaptec 5805 raid controller o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141177 fs [zfs] fsync() on FIFO causes panic() on zfs o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140682 fs [netgraph] [panic] random panic in netgraph o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140433 fs [zfs] [panic] panic while replaying ZIL after crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138524 fs [msdosfs] disks and usb flashes/cards with Russian lab o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 168 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 11:10:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D326106566C for ; Mon, 15 Mar 2010 11:10:01 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id E07A78FC1B for ; Mon, 15 Mar 2010 11:10:00 +0000 (UTC) Received: by gwj15 with SMTP id 15so1694757gwj.13 for ; Mon, 15 Mar 2010 04:10:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=mjgNAhkgGIds/bsK6BgjKOjfNvAA53uK7y26CGG0lGY=; b=I63dWcu7C+k2R6Q4rmsYlXNBDPGkKgFGwABohkExSR5uJMN51vs5kvVaKv+xM0Lmxt 62ILss8aEvo6EiUvuy5cMz14gcspBUJWreehcjsRlROVYtrN52Ehoc9jgM+SSCOxXB9m 1m1iDEfR2usdZcrY3ItTmDhS/987gdXJYfP0o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=U23P/xCCE6sdzNbjyrW4SIGW+fKfG0M9XcIHZFEng4FX6uVAkZpeswf7mhQOrHq4wY LJDDgaEyJmcLcSothubvEElqf65sTj+4oAXWdxY9wN8hTwmPfgonbHUZd/R4vj8umRCf H42OBDHxSS7+7ZOZ0dZXBs5B57f8QloxkbrYg= MIME-Version: 1.0 Received: by 10.101.202.12 with SMTP id e12mr940579anq.132.1268651399938; Mon, 15 Mar 2010 04:09:59 -0700 (PDT) Date: Mon, 15 Mar 2010 13:09:59 +0200 Message-ID: From: Dan Naumov To: Randy Bush , freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: RE: degraded zfs slowdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 11:10:01 -0000 >as the ghostly bad drive does not have a unique name, [how] can i tell >zfs to remove it for the moment? Have you tried running a scrub on the pool and THEN trying to offline and detach (zpool commands, not physical actions) the ghost drive? Note: when adding the replacement drive, remember to ATTACH the drives together. If you ADD the new drive, you will end up with a stripe, not a mirror. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 15:59:08 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 149A01065675 for ; Mon, 15 Mar 2010 15:59:08 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id A7BF28FC15 for ; Mon, 15 Mar 2010 15:59:07 +0000 (UTC) Received: by wyb33 with SMTP id 33so1783199wyb.13 for ; Mon, 15 Mar 2010 08:59:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=Xb11NFlyFRsL2h1lkHwq/N1eqgyY6dhKzgvqWE4fUZs=; b=N3G+pN4OMtSq2syLwhY3cVYnn3c5VXJiFrffMN3Lc+tFiIhiO3vflDOeg6mXUHx6mp ZxfnvUvxXNamJMv0B/PlDXRXcvio4osXRrf6uIslicH5NdPk7fmeIM77oS7SIrpx65zi MPsZks/LLtvX1Zs88H5D0QVa+xnNKPUCwvRsY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Px5VwMurQzRETR4gWcZt4UyY5wHbfUIY4IjUqiNsbOWJdowdGxkGhwQnwIt9rOVrTs wtGyusuXNwn5EcmdcUTIT0G9LUdjwk7wlsdp1bobSPLxjRHiveoVN7MpDL7HYLhKIXrS fUeW5GBV5BNPOuKpBBh7GAHwQYVANJODe4pGg= MIME-Version: 1.0 Received: by 10.216.85.132 with SMTP id u4mr950050wee.191.1268668746410; Mon, 15 Mar 2010 08:59:06 -0700 (PDT) Date: Mon, 15 Mar 2010 10:59:06 -0500 Message-ID: <5f67a8c41003150859s550a63s32a9ab070ac514c1@mail.gmail.com> From: Zaphod Beeblebrox To: freebsd-fs Content-Type: text/plain; charset=ISO-8859-1 Subject: A completely strange ZFS hang. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 15:59:08 -0000 I recently copied my entire ZFS pool off into a temporary ZFS pool with "zfs send -R". Marvelous way to do things. Anyways... I reformatted my ZFS pool (more RAIDZ1 disks) and started a copy back... something along the lines of: zfs send -R zroot/vr2 | ssh otherhost "zfs receive -Fd vr2" ... turns out this puts the stuff from zroot/vr2 into vr2/vr2 --- but I can rename it later. While this is ongoing (5T of data at about 2T per day transfer --- probably could go faster, but I havn't tried), I needed to download some stuff over my 50 megabit link. So sitting in an NFS mounted directory (mounted -3 -T), I: mv somedir1 /vr2/new/ Where /vr2/new is a new ZFS filesystem not related to the heirachy that the "zfs receive" is creating in vr2/vr2. At some point I realize I need something before I need somedir1, so I CTRL-Z that command and: mv somedir2 /vr2/new/ ... sometime after this command starts, ZFS locks hard. zfs list or even an ls -l inside the zfs dir is locked hard. The zfs receive process has gone idle too. Because I don't want to restart everything, I try backgrounding the first mv. Surprisingly, this works ?!? Keep in mind while thinking about this problem that the locked condition occurred for about 20 minutes while I tried to work it out. The machine running ZFS was not locked (it has a UFS root and standard commands were working fine) From owner-freebsd-fs@FreeBSD.ORG Mon Mar 15 23:40:28 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E421C1065672; Mon, 15 Mar 2010 23:40:27 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f228.google.com (mail-bw0-f228.google.com [209.85.218.228]) by mx1.freebsd.org (Postfix) with ESMTP id 5254B8FC15; Mon, 15 Mar 2010 23:40:26 +0000 (UTC) Received: by bwz28 with SMTP id 28so3492174bwz.14 for ; Mon, 15 Mar 2010 16:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=kTezEwnvGUoxTWT6csQjE9+sfxcsj4bUFE4BKr/SOJA=; b=G9qUDKl+RNf6gE12OzMuHmrrp5q2mzDj9bg6RiFJTOSMsIqO6FJUzj+g2dHNN7OdLH 4AABzaZao7VtAHG8IQuo8UMbdv7P/Os0YtnMK3dcW7SjmXKW3cmdFfM+5I1HzrsrxV8n 5iVouwAfYLciOkbqzOSeowfD2HRk1cuPJMlG8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=VXeFv0eZhdwFfc8AFtUi6yDzWUNqMnrMVC7Wiag/SSwRXc6AcTaE1/1XaRa3+d7D00 YCHgCf9qkQ/NYjHwzkeDc0kW6pq8dGWa12CKMBCHI3rNrGIMmneJ1LcOChYuRg979bsw TdDEiXsccs+lQUDRg8wFajMSsUCcx2WxMOeRs= MIME-Version: 1.0 Received: by 10.204.141.9 with SMTP id k9mr747685bku.73.1268696425833; Mon, 15 Mar 2010 16:40:25 -0700 (PDT) Date: Tue, 16 Mar 2010 01:40:25 +0200 Message-ID: From: Dan Naumov To: freebsd-questions@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Some questions about vfs.zfs.prefetch_disable=1 and ZFS filesystem versions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Mar 2010 23:40:28 -0000 After looking at the arc_summary.pl script (found at http://jhell.googlecode.com/files/arc_summary.pl), I have realized that my system has set vfs.zfs.prefetch_disable=1 by default, looking at dmesg, I see: ========================= ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ========================= ...except I do have 4gb of RAM. Is this caused by integrated GPU snatching some of my memory at boot? From dmesg: ========================= real memory = 4294967296 (4096 MB) avail memory = 4088082432 (3898 MB) ========================= What kind of things does this tunable affect and how much of a performance impact does enabling / disabling it have? Should I manually enable it? I've also noticed a really weird inconsistency, my dmesg says the following: ========================= ZFS filesystem version 13 ZFS storage pool version 13 ========================= Yet: ========================= zfs get version NAME PROPERTY VALUE SOURCE cerberus version 3 - cerberus/DATA version 3 - cerberus/ROOT version 3 - cerberus/ROOT/cerberus version 3 - cerberus/home version 3 - cerberus/home/atombsd version 3 - cerberus/home/friction version 3 - cerberus/home/jago version 3 - cerberus/home/karni version 3 - cerberus/tmp version 3 - cerberus/usr-local version 3 - cerberus/usr-obj version 3 - cerberus/usr-ports version 3 - cerberus/usr-ports-distfiles version 3 - cerberus/usr-src version 3 - cerberus/var version 3 - cerberus/var-db version 3 - cerberus/var-log version 3 - cerberus/var-tmp version 3 - ========================= Is this normal or should "zfs get version" also show version 13? This is on a system with the pool and filesystems created with 8.0-RELEASE, by the way. Thanks! - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 00:06:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 654A2106564A; Tue, 16 Mar 2010 00:06:37 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mx1.freebsd.org (Postfix) with ESMTP id C3D308FC13; Tue, 16 Mar 2010 00:06:36 +0000 (UTC) Received: by bwz8 with SMTP id 8so3493825bwz.3 for ; Mon, 15 Mar 2010 17:06:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=RmfBIh6EOAqwR7O9L8KH5lqSDnWvYinTjgaw6e89zG0=; b=rmxA/r9MXXMfOLd87QrXf2ESSwzXVk5EXubb9aBoJh9kZvHxELyhPB8HOKGS+vlI/E 7hQ/Xtgrt5joSYVEVnjpvpyOnEQ4Ix6DrPt+1CGLdFb0tmhMX31S8CR64hYE/gV01j67 cU9XQdZYwn80aHeTyA1qFspaI21sMLaQva3ss= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=DK3y/2dF8w48ccvkQVqFelMAcn6Zga2Q7Eu4z4VHvNsBuWaT0ino17yVeD1mMv5lSi sWR5UQ/dW4+QK7RUGPvjN+XOdUShlYlH8mDGBYbHSmxUQ855jYrZ5//+DXVX1qOfujCN 0q0iYrkN1NU6euIvcOLNENP7UxpZtmK3SVsas= MIME-Version: 1.0 Received: by 10.204.141.9 with SMTP id k9mr771834bku.73.1268697995076; Mon, 15 Mar 2010 17:06:35 -0700 (PDT) In-Reply-To: References: Date: Tue, 16 Mar 2010 02:06:35 +0200 Message-ID: From: Dan Naumov To: freebsd-questions@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Re: Some questions about vfs.zfs.prefetch_disable=1 and ZFS filesystem versions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 00:06:37 -0000 Nevermind the question about ZFS filesystem versions, I should've Googled more throughly and read Pawel's responce to this question before (answer: dmesg picks the filesystem version wrong, it IS and supposed to be v3). I am still curious about prefetch though. - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 05:11:50 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C556106564A for ; Tue, 16 Mar 2010 05:11:50 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta11.emeryville.ca.mail.comcast.net (qmta11.emeryville.ca.mail.comcast.net [76.96.27.211]) by mx1.freebsd.org (Postfix) with ESMTP id 8390C8FC15 for ; Tue, 16 Mar 2010 05:11:50 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta11.emeryville.ca.mail.comcast.net with comcast id tfwe1d00617UAYkABh7C1y; Tue, 16 Mar 2010 05:07:12 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta13.emeryville.ca.mail.comcast.net with comcast id thBq1d0013S48mS8ZhBq3v; Tue, 16 Mar 2010 05:11:50 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id BEB499B436; Mon, 15 Mar 2010 22:11:48 -0700 (PDT) Date: Mon, 15 Mar 2010 22:11:48 -0700 From: Jeremy Chadwick To: Dan Naumov Message-ID: <20100316051148.GA73270@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, freebsd-questions@freebsd.org Subject: Re: Some questions about vfs.zfs.prefetch_disable=1 and ZFS filesystem versions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 05:11:50 -0000 On Tue, Mar 16, 2010 at 01:40:25AM +0200, Dan Naumov wrote: > After looking at the arc_summary.pl script (found at > http://jhell.googlecode.com/files/arc_summary.pl), I have realized > that my system has set vfs.zfs.prefetch_disable=1 by default, looking > at dmesg, I see: > > ========================= > ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; > to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. > ========================= > > ...except I do have 4gb of RAM. Is this caused by integrated GPU > snatching some of my memory at boot? From dmesg: > > ========================= > real memory = 4294967296 (4096 MB) > avail memory = 4088082432 (3898 MB) > ========================= I've blogged about this problem when testing out 8.0-RC1. See the bottom third of my post for an explanation: http://koitsu.wordpress.com/2009/10/12/testing-out-freebsd-8-0-rc1/ The message is confusing/badly worded, despite having gone through numerous commits to change its wording. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 05:16:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 828CB1065672 for ; Tue, 16 Mar 2010 05:16:39 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.emeryville.ca.mail.comcast.net (qmta02.emeryville.ca.mail.comcast.net [76.96.30.24]) by mx1.freebsd.org (Postfix) with ESMTP id 68F858FC12 for ; Tue, 16 Mar 2010 05:16:39 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta02.emeryville.ca.mail.comcast.net with comcast id tfbt1d00C1vN32cA2hGg6H; Tue, 16 Mar 2010 05:16:40 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta22.emeryville.ca.mail.comcast.net with comcast id thL91d00H3S48mS8ihL9K7; Tue, 16 Mar 2010 05:20:10 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id C09EC9B436; Mon, 15 Mar 2010 22:16:37 -0700 (PDT) Date: Mon, 15 Mar 2010 22:16:37 -0700 From: Jeremy Chadwick To: Dan Naumov Message-ID: <20100316051637.GB73270@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, freebsd-questions@freebsd.org Subject: Re: Some questions about vfs.zfs.prefetch_disable=1 and ZFS filesystem versions X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 05:16:39 -0000 On Tue, Mar 16, 2010 at 02:06:35AM +0200, Dan Naumov wrote: > Nevermind the question about ZFS filesystem versions, I should've > Googled more throughly and read Pawel's responce to this question > before (answer: dmesg picks the filesystem version wrong, it IS and > supposed to be v3). The printing of the incorrect version number was fixed in RELENG_7 and RELENG_8 approx. 8 weeks ago. See commit revs 1.14.2.8 (RELENG_7) and 1.18.2.5 (RELENG_8) below: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 08:23:08 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B5A4106564A for ; Tue, 16 Mar 2010 08:23:08 +0000 (UTC) (envelope-from randy@psg.com) Received: from ran.psg.com (ran.psg.com [IPv6:2001:418:1::36]) by mx1.freebsd.org (Postfix) with ESMTP id 20D9E8FC0A for ; Tue, 16 Mar 2010 08:23:08 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=rmac.psg.com.psg.com) by ran.psg.com with esmtp (Exim 4.71 (FreeBSD)) (envelope-from ) id 1NrS3P-000Ejj-C8; Tue, 16 Mar 2010 08:23:07 +0000 Date: Tue, 16 Mar 2010 01:23:07 -0700 Message-ID: From: Randy Bush To: Dan Naumov In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/22.3 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: freebsd-fs@freebsd.org Subject: Re: degraded zfs slowdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 08:23:08 -0000 >> as the ghostly bad drive does not have a unique name, [how] can i tell >> zfs to remove it for the moment? > Have you tried running a scrub on the pool and THEN trying to offline > and detach (zpool commands, not physical actions) the ghost drive? psg.com:/root# zpool scrub tank psg.com:/root# zpool status pool: tank state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: scrub in progress for 0h0m, 0.00% done, 169h12m to go config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror DEGRADED 0 0 0 twed1 FAULTED 0 3 0 corrupted data twed1 ONLINE 0 0 0 errors: No known data errors still do not know how to detach bad drive as it does not have a unique name randy > Note: when adding the replacement drive, remember to ATTACH the drives > together. If you ADD the new drive, you will end up with a stripe, not > a mirror. ack. thanks. randy From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 09:07:38 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EDE4A106566C for ; Tue, 16 Mar 2010 09:07:38 +0000 (UTC) (envelope-from dan.naumov@gmail.com) Received: from mail-bw0-f228.google.com (mail-bw0-f228.google.com [209.85.218.228]) by mx1.freebsd.org (Postfix) with ESMTP id 804AD8FC16 for ; Tue, 16 Mar 2010 09:07:38 +0000 (UTC) Received: by bwz28 with SMTP id 28so3713458bwz.14 for ; Tue, 16 Mar 2010 02:07:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ZFoW4MK7OoKccvnukhucrmEmGR5yfGRpQ9x1kLSB74s=; b=Cx6pxJ+vBLXkSvzc3auj6az9ABCFjmg+AwQ7rjlbVFC2U3SJ+hu/ZjRZYRtCmRMXhC wBnGi2kWFQdr8bMhDqH2E9aKm77BkMZBV42kT428+NtiR+et7Rm1u29qEaoLdA4ekpRO mHOsJULCYlgtUpsxAeg6IsFtigGbuKFtQHqZU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=LCr2TvOT908/FI3lwXd7a+OtEbMEm7K0pFkQL5/j7Eu3UwXvcjWEf8XBOzxBMm0ukt 1STYUhmZ7WkQtyrFNjK0RRK0aS/vrkVV78oC1NoN87i5UsGlbjFtors2RjyHVi86rPuC AsIHcz36PVRwxDNedr0ZNwMusUIUlcXt5eSJs= MIME-Version: 1.0 Received: by 10.204.25.70 with SMTP id y6mr320759bkb.130.1268730451105; Tue, 16 Mar 2010 02:07:31 -0700 (PDT) In-Reply-To: References: Date: Tue, 16 Mar 2010 11:07:31 +0200 Message-ID: From: Dan Naumov To: Randy Bush Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: degraded zfs slowdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 09:07:39 -0000 On Tue, Mar 16, 2010 at 10:23 AM, Randy Bush wrote: >>> as the ghostly bad drive does not have a unique name, [how] can i tell >>> zfs to remove it for the moment? >> Have you tried running a scrub on the pool and THEN trying to offline >> and detach (zpool commands, not physical actions) the ghost drive? > > =A0 =A0psg.com:/root# zpool scrub tank > =A0 =A0psg.com:/root# zpool status > =A0 =A0 =A0pool: tank > =A0 =A0 state: DEGRADED > =A0 =A0status: One or more devices could not be used because the label is= missing or > =A0 =A0 =A0 =A0 =A0 =A0invalid. =A0Sufficient replicas exist for the pool= to continue > =A0 =A0 =A0 =A0 =A0 =A0functioning in a degraded state. > =A0 =A0action: Replace the device using 'zpool replace'. > =A0 =A0 =A0 see: http://www.sun.com/msg/ZFS-8000-4J > =A0 =A0 scrub: scrub in progress for 0h0m, 0.00% done, 169h12m to go > =A0 =A0config: > > =A0 =A0 =A0 =A0 =A0 =A0NAME =A0 =A0 =A0 =A0STATE =A0 =A0 READ WRITE CKSUM > =A0 =A0 =A0 =A0 =A0 =A0tank =A0 =A0 =A0 =A0DEGRADED =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0mirror =A0 =A0DEGRADED =A0 =A0 0 =A0 =A0 0 =A0= =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0twed1 =A0 FAULTED =A0 =A0 =A00 =A0 =A0 3 = =A0 =A0 0 =A0corrupted data > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0twed1 =A0 ONLINE =A0 =A0 =A0 0 =A0 =A0 0 = =A0 =A0 0 > > =A0 =A0errors: No known data errors > > still do not know how to detach bad drive as it does not have a unique > name As you can see from your own message, the scrub hasn't finished, it has just started. What happens after the scrub is finished? If both twed1's are still visible at this point (after a finished scrub), what happens if you attempt to "zfs detach tank twed1" the ghost drive, do you get an error or does it recognize and correctly remove the ghost twed1? - Sincerely, Dan Naumov From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 13:24:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF868106566C for ; Tue, 16 Mar 2010 13:24:24 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-qy0-f198.google.com (mail-qy0-f198.google.com [209.85.221.198]) by mx1.freebsd.org (Postfix) with ESMTP id 8324B8FC0C for ; Tue, 16 Mar 2010 13:24:24 +0000 (UTC) Received: by qyk36 with SMTP id 36so330362qyk.30 for ; Tue, 16 Mar 2010 06:24:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=kn35p/g3qP2tNopeQm5G5Fd96iNWBUvlyH/8IDzfAZg=; b=n8LvhsVa3ZLG1oAXJ7N/B6mSXtVREH38l1vqDaxivb4i1MvBPtpN2dTdn6jARA6GPb yNnAhmi7JlWOsMMw+nljnIwfAKEEC+KBYTyCXZvJaydoPuRyqt7NTq6QeLRqtdILfpFy MyHGOflYAS669AdDwN7659ILhal0S3GP50vRg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=FyvJpTAgIHxNJEN7v5j5UgSObKAh2G08jJCENSubTHo3S17FIxyN+wsobtCyPPJOLe oEnsV0F7OsEjOHDJ0gEo4BE77pf+oT3DV/KOSX23ihheiQ+jhEa65uyelwrPAOh7tbNE 63Ocz6uQwFIKankMrOCrpHfVfLJm28g+OhDuU= Received: by 10.229.216.21 with SMTP id hg21mr7067590qcb.14.1268745862434; Tue, 16 Mar 2010 06:24:22 -0700 (PDT) Received: from centel.dataix.local (ppp-22.211.dialinfree.com [209.172.22.211]) by mx.google.com with ESMTPS id 20sm3893902qyk.4.2010.03.16.06.24.18 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 16 Mar 2010 06:24:21 -0700 (PDT) Sender: "J. Hellenthal" Date: Tue, 16 Mar 2010 09:12:39 -0400 From: jhell To: Randy Bush In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, Dan Naumov Subject: Re: degraded zfs slowdown X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 13:24:25 -0000 On Tue, 16 Mar 2010 04:23, Randy Bush wrote: In Message-Id: >>> as the ghostly bad drive does not have a unique name, [how] can i tell >>> zfs to remove it for the moment? >> Have you tried running a scrub on the pool and THEN trying to offline >> and detach (zpool commands, not physical actions) the ghost drive? > > psg.com:/root# zpool scrub tank > psg.com:/root# zpool status > pool: tank > state: DEGRADED > status: One or more devices could not be used because the label is missing or > invalid. Sufficient replicas exist for the pool to continue > functioning in a degraded state. > action: Replace the device using 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-4J > scrub: scrub in progress for 0h0m, 0.00% done, 169h12m to go > config: > > NAME STATE READ WRITE CKSUM > tank DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > twed1 FAULTED 0 3 0 corrupted data > twed1 ONLINE 0 0 0 > > errors: No known data errors > > still do not know how to detach bad drive as it does not have a unique > name > Have you, Can you ? (zpool export tank) and then (zpool import tank) and then followup with the results ? You might need some force flags with this so don't be surprised. When importing the pool it should search out the GUID for the badly named drive and replace it with its proper drive. I believe if at all possible you should wait for the scrub to finish or properly stop the scrub with (zpool scrub -s) Good Luck. -- jhell From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 16:51:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BADF9106566B for ; Tue, 16 Mar 2010 16:51:22 +0000 (UTC) (envelope-from kloderik@gmail.com) Received: from mail-bw0-f228.google.com (mail-bw0-f228.google.com [209.85.218.228]) by mx1.freebsd.org (Postfix) with ESMTP id 51B8F8FC17 for ; Tue, 16 Mar 2010 16:51:21 +0000 (UTC) Received: by bwz28 with SMTP id 28so147190bwz.14 for ; Tue, 16 Mar 2010 09:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=J2q8XjbiLH4zBN2U5mXcfJJP99T+Uj7sIlxJSCVWP/k=; b=fXdBqdwz7Zb8Zha5FCW3tbrZIZyPQ+OHtEaAqkZTBiguWfZSyBsOBllJCInHcEAaN+ NDrtdZPcx8KlQAzipjhT/HlEGfB98wWpjl271etSsew7DwNMSaDj9qMu9lpj1psL+wxN M+U3L2y4nPxylOyMoct1uq1kXEBq3Q7y5H2yY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=qm7AL+nQ3Xyyp/xm9rrcxACK5zghtSBnNxUlEsQsw3sm4RtaWUIpBB0+zeCE/XF8Px IlSuZVCpnciVMn105mKat71IJbx3m0WDOrz2IayTg44Z4jUfAmO9ZRJhDC9IVm5Q/9sE cav7Pv1tdrH2+MKXPQqb8kxofpMAIjlnst1W8= MIME-Version: 1.0 Received: by 10.204.33.132 with SMTP id h4mr24990bkd.103.1268756496030; Tue, 16 Mar 2010 09:21:36 -0700 (PDT) Date: Tue, 16 Mar 2010 18:21:36 +0200 Message-ID: <881b8a361003160921s246c7a0u912099f56fcb0fcd@mail.gmail.com> From: "Victor A. Savinoff" To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: growfs(8) trouble X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 16:51:22 -0000 In growfs.c: static void get_dev_size(int fd, int *size) int size (which, actually, is device length measured in blocks) is limited to largest 32-bit integer. If device block size is 512 bytes (which is usual by now), this limits new filesystem size to 2TiB. Without changes, growfs refuses to grow a filesystem beyond 2TiB with message "like we are not growing (old_size->new_size)", where new_size < old_size. I tried to change both its and p_size (in main()) type to u_int64_t. That worked for growing filesystem from 2TiB to 3.62TiB with block size 64KiB and fragment size 8KiB, but failed with default block and fragment sizes. As I am not an experienced developer, I would't even try to post patches here, so I leave the solution for gurus. -- WBR, Victor A. "Claus" Savinoff From owner-freebsd-fs@FreeBSD.ORG Tue Mar 16 19:07:16 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 304D4106566C for ; Tue, 16 Mar 2010 19:07:16 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7A54A8FC21 for ; Tue, 16 Mar 2010 19:07:15 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id VAA27570 for ; Tue, 16 Mar 2010 21:07:13 +0200 (EET) (envelope-from avg@icyb.net.ua) Message-ID: <4B9FD6E0.5000303@icyb.net.ua> Date: Tue, 16 Mar 2010 21:07:12 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: few ideas for zfsloader X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2010 19:07:16 -0000 I have three crazy-ish ideas for zfsloader. Unfortunately, right now I am short on time and I am very short on ZFS knowledge to go for implementing them myself. 1. If vfs.root.mountfrom is not already set and there is no root mount entry in fstab then set vfs.root.mountfrom to "zfs:". This obviously requires being able to get properties of the current dataset (filesystem) and get the name from them. 2. Currently nextboot functionality doesn't work properly with ZFS because there is no RW support in zfsloader. Adding that support seems to be quite hard given the transactional nature of ZFS, checksums, compression and what not. Here's an alternative idea: honor nextboot.conf only if boot filesystem has a a special property set on it, for example org.freebsd:nextboot. Then all we need is to flip the property off. Although doing this is still not trivial it should be simpler than filesystem RW support. 3. Currently ZFS boot chain lacks an ability to switch bootfs property of a pool. So, if a (new) boot environment is not quite bootable and bootfs points to it, then an alternative boot media (e.g. livecd) is needed to correct the situation. Implementing selection of a boot filesystem in ZFS boot chain seems like a hard task. Alternative idea: a new FreeBSD-specific pool property, nextbootfs. This property would designate a boot filesystem for the next boot and would be automatically reset by a boot loader at sufficiently early stage. If the next boot doesn't succeed, then we are back to the regular bootfs property, if it does succeed, then bootfs can be safely changed to the new value. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Mar 17 09:52:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E51C1065672 for ; Wed, 17 Mar 2010 09:52:37 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id D32AA8FC17 for ; Wed, 17 Mar 2010 09:52:36 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA11411; Wed, 17 Mar 2010 11:52:33 +0200 (EET) (envelope-from avg@freebsd.org) Message-ID: <4BA0A660.3000902@freebsd.org> Date: Wed, 17 Mar 2010 11:52:32 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-geom@freebsd.org X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Bruce Evans Subject: g_vfs_open and bread(devvp, ...) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2010 09:52:37 -0000 I've given a fresh look to the issue of g_vfs_open and bread(devvp, ...) in filesystem code. This time I hope to present my reasoning more clearly than I did in my previous attempts. For this reason I am omitting historical references and dramatic examples/demonstrations but they are still available upon request (and in archives). I hope that my shortened notation in references to the code and data structures will not be confusing. bread() and the API family it belongs to is an interface to buffer cache system. Buffer cache system can be roughly divided into two parts: cache part and I/O path part. I think that it is understood that both parts must have the same notion of a block size when translating a block number to a byte offset. (If this point is not clear, I can follow up with another essay). In the case of cache code the translation is explicit and simple: A) offset = blkno * bo_bsize In the case of I/O code the translation is not that straightforward, because it can be altered/overridden by bop_strategy which can in turn hook vop_strategy, etc. Let's consider a simple case of a filesystem that: a) connects to a geom provider via g_vfs_open b) doesn't modify anything in devvp->v_bufobj (in particular bo_ops and bo_bsize) c) uses bread(devvp) (e.g. to access an equivalent of superblock, etc) Short overview of geom_vfs glue: 1) g_vfs_open sets devvp->v_bufobj.bo_ops to g_vfs_bufops, where bop_strategy is g_vfs_strategy 2) bo_bsize is set to pp->sectorsize 3) g_vfs_strategy doesn't perform any block-to-offset translation of its own, it expects b_iooffset to be correctly set and passes its value to bio_offset When a filesystem issues bread(devvp) the following happens in the I/O path: I) bread() calls breadn() II) in breadn(): bp->b_iooffset = dbtob(bp->b_blkno), that is b_iooffset is set to blkno * DEV_BSIZE (where DEV_BSIZE is 512) III) breadn() then calls bstrategy() which is a simple wrapper around BO_STRATEGY IV) g_vfs_strategy gets called and, as described in (3) above, it simply passes on b_iooffset value to bio_offset V) thus, a block size used for I/O operation is 512 (DEV_BSIZE) VI) on the other hand, as stated in (A) above, block size used in caching code is bo_bsize Thus, if bo_bsize != DEV_BSIZE, or alternatively said, pp->sectorsize != 512, we have a trouble of data getting cached with incorrect offsets. Additionally, from (V) above we must conclude that a filesystem must specify block numbers in DEV_BSIZE units to bread(devvp, blkno, ...) if the conditions (a), (b), (c) are met. In fact, all such filesystems already do that, because otherwise they would read incorrect data from the media. So, the problem is only with the caching of the data. As such, this issue has little practical effects, because only a small number of reads is done via devvp and only for sufficiently small chunks of data (I hope). fs/udf used to be greatly affected by this issue when it was reading directory nodes via devvp, but that was in the past (prior to 189082). Still I think that we should fix this issue for general code correctness/quality reasons. And also to avoid possible future bugs. As demonstrated by (V) and (VI) above, obvious and easiest fix is to (always) set bo_bsize to DEV_BSIZE in g_vfs_open(): --- a/sys/geom/geom_vfs.c +++ b/sys/geom/geom_vfs.c @@ -179,7 +179,7 @@ g_vfs_open(struct vnode *vp, struct g_consumer **cpp, const char *fsname, int wr bo = &vp->v_bufobj; bo->bo_ops = g_vfs_bufops; bo->bo_private = cp; - bo->bo_bsize = pp->sectorsize; + bo->bo_bsize = DEV_BSIZE; gp->softc = bo; return (error); I tested this change with ufs, udf and cd9660 and haven't observed any regressions. P.S. There might something to changing bread(devvp) convention, so that blkno is expected in sectorsize units. But setting bo_bsize to sectorsize is only a tiny portion of what needs to be changed to make it actually work. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Mar 17 11:20:04 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11F11106564A for ; Wed, 17 Mar 2010 11:20:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 017D58FC1C for ; Wed, 17 Mar 2010 11:20:04 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2HBK3DX082091 for ; Wed, 17 Mar 2010 11:20:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2HBK3CV082081; Wed, 17 Mar 2010 11:20:03 GMT (envelope-from gnats) Date: Wed, 17 Mar 2010 11:20:03 GMT Message-Id: <201003171120.o2HBK3CV082081@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Kai Kockro Cc: Subject: Re: kern/144330: [nfs] mbuf leakage in nfsd with zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Kai Kockro List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2010 11:20:04 -0000 The following reply was made to PR kern/144330; it has been noted by GNATS. From: Kai Kockro To: bug-followup@freebsd.org, gerrit@pmp.uni-hannover.de Cc: Subject: Re: kern/144330: [nfs] mbuf leakage in nfsd with zfs Date: Wed, 17 Mar 2010 11:43:51 +0100 Hi, same problem here. netstat -m ( after 6 days ) 140889/35376/176265 mbufs in use (current/cache/total) 139381/4689/144070/262144 mbuf clusters in use (current/cache/total/max) 137206/3594 mbuf+clusters out of packet secondary zone in use (current/cache) 2/245/247/131072 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/65536 9k jumbo clusters in use (current/cache/total/max) 0/0/0/32768 16k jumbo clusters in use (current/cache/total/max) 313992K/19202K/333194K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines Its a NFS exportet ZPOOL ( 14TB, 40 clients, no UDP mounts ). #uname -a FreeBSD xxxx 8.0-STABLE FreeBSD 8.0-STABLE #0: Fri Feb 12 03:27:54 CET 2010 root@xxxx:/usr/obj/usr/src/sys/GENERIC amd64 #cat /boot/loader.conf kern.maxdsiz="3G" kern.maxssiz="2G" vfs.zfs.prefetch_disable="1" kern.ipc.nmbclusters="262144" From owner-freebsd-fs@FreeBSD.ORG Wed Mar 17 11:39:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 471851065670 for ; Wed, 17 Mar 2010 11:39:55 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 2EC738FC13 for ; Wed, 17 Mar 2010 11:39:54 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta01.emeryville.ca.mail.comcast.net with comcast id uBWQ1d00317UAYkA1BfvkQ; Wed, 17 Mar 2010 11:39:55 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta13.emeryville.ca.mail.comcast.net with comcast id uBfu1d0063S48mS8ZBfu2X; Wed, 17 Mar 2010 11:39:55 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5AA2D9B436; Wed, 17 Mar 2010 04:39:53 -0700 (PDT) Date: Wed, 17 Mar 2010 04:39:53 -0700 From: Jeremy Chadwick To: Kai Kockro Message-ID: <20100317113953.GA14582@icarus.home.lan> References: <201003171120.o2HBK3CV082081@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201003171120.o2HBK3CV082081@freefall.freebsd.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@FreeBSD.org Subject: Re: kern/144330: [nfs] mbuf leakage in nfsd with zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2010 11:39:55 -0000 On Wed, Mar 17, 2010 at 11:20:03AM +0000, Kai Kockro wrote: > The following reply was made to PR kern/144330; it has been noted by GNATS. The problem appears to be specific to UDP NFS and not TCP. The following is a statement from the PR originator: http://lists.freebsd.org/pipermail/freebsd-stable/2010-March/055543.html And the remaining part of the thread which includes comments from the NFS maintainer (who will look into this when he gets time): http://lists.freebsd.org/pipermail/freebsd-stable/2010-March/055552.html These details (particularly that it happens with UDP NFS but not TCP) need to be put into the PR. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed Mar 17 12:31:07 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C1501065675 for ; Wed, 17 Mar 2010 12:31:07 +0000 (UTC) (envelope-from gallasch@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) by mx1.freebsd.org (Postfix) with ESMTP id D79DF8FC19 for ; Wed, 17 Mar 2010 12:31:06 +0000 (UTC) Received: (qmail 7215 invoked from network); 17 Mar 2010 13:31:04 +0100 Received: from smtp.free.de (HELO orwell.free.de) (gallasch@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 17 Mar 2010 13:31:04 +0100 Date: Wed, 17 Mar 2010 13:30:59 +0100 From: Kai Gallasch To: Alexander Motin Message-ID: <20100317133059.51245776@orwell.free.de> In-Reply-To: <20100312222659.0198dd03@orwell.free.de> References: <20100311133916.42ba69b0@orwell.free.de> <20100312115028.GG1819@garage.freebsd.pl> <4B9A8A27.8050608@FreeBSD.org> <20100312222659.0198dd03@orwell.free.de> X-Mailer: Claws Mail 3.7.3 (GTK+ 2.18.7; powerpc-apple-darwin9.8.0) X-Face: 7"x0zA5=*cXGZw-xjU<">'+!3(KXTUXZVLD42KVN{'go[UQr"Mc.e(XW92N8plZ(9x.{x; I<|95e+b&GH-36\15F~L$YD*Y +u}o&KV?6.%"mJIkaY3G>BKNt`1|Y+%K1P4t; 47D65&(Y7h5Ll-[ltkhamx.-; ,jggK'}oMpUgEHFG YQ"9oXKAl>!d,J}T{)@uxvfu?YFWC*\~h+,^f Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2010 12:31:07 -0000 > Am Fri, 12 Mar 2010 20:38:31 +0200 > schrieb Alexander Motin : > > > Pawel Jakub Dawidek wrote: > > > On Thu, Mar 11, 2010 at 01:39:16PM +0100, Kai Gallasch wrote: > > >> I have some trouble with an opteron server locking up > > >> spontaneously. It looses all networks connectivity and even > > >> through console I can get no shell. > > >> > > >> Lockups occur mostly under disk load (periodic daily, bacula > > >> backup running, make buildworld/buildkernel) and I can provoke > > >> them easily. > > > [...] > > >> 4 0 0 0 LL *cissmtx 0xffffff04ed820c00 > > >> [g_down] > > > [...] > > >> 100046 L *cissmtx 0xffffff04ed820c00 > > >> [irq257: ciss0] > > > [...] > > > > > > I was analizing similar problem as potential ZFS bug. It turned > > > out to be bug in ciss(4) and I believe mav@ (CCed) has fix for > > > that. > > > > That my patch is already at 8-STABLE since r204873 of 2010-03-08. > > Make sure you have it. Rebuilding the kernel with your 8-STABLE commited ciss patch seems to have resolved this issue. Server now has an uptime of 5 days and survives under high filesystem load. Alexander, thanks for fixing ciss. Kai. -- Da das Pferd pfluegt, lasst uns den Esel satteln. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 17 21:42:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B6E3106566B for ; Wed, 17 Mar 2010 21:42:36 +0000 (UTC) (envelope-from baldur@foo.is) Received: from gremlin.foo.is (gremlin.foo.is [194.105.250.10]) by mx1.freebsd.org (Postfix) with ESMTP id E7C768FC12 for ; Wed, 17 Mar 2010 21:42:35 +0000 (UTC) Received: by gremlin.foo.is (Postfix, from userid 1000) id 011F5DA855; Wed, 17 Mar 2010 21:42:34 +0000 (GMT) Date: Wed, 17 Mar 2010 21:42:34 +0000 From: Baldur Gislason To: freebsd-fs@freebsd.org Message-ID: <20100317214234.GF63370@gremlin.foo.is> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Subject: Frustration: replace not doing what I expected. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2010 21:42:36 -0000 A drive failed in a pool and I had to replace it. I did zpool replace ad18 ad18, the pool resilvered for 5 hours and finished but did not return from degraded mode. I tried removing the cache file and reimporting the pool, no change, it hasn't gotten rid of the old drive which does not exist anymore. What to do? Baldur pool: zirconium state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM zirconium DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 replacing DEGRADED 0 0 0 2614810928866691230 UNAVAIL 0 962 0 was /dev/ad18/old ad18 ONLINE 0 0 0 ad20 ONLINE 0 0 0 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 17 22:33:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF1521065675 for ; Wed, 17 Mar 2010 22:33:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 854AA8FC1A for ; Wed, 17 Mar 2010 22:33:42 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEALb1oEuDaFvI/2dsb2JhbACbH3O5J4R2BA X-IronPort-AV: E=Sophos;i="4.51,261,1267419600"; d="scan'208";a="69386192" Received: from darling.cs.uoguelph.ca ([131.104.91.200]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 17 Mar 2010 18:33:41 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by darling.cs.uoguelph.ca (Postfix) with ESMTP id 8F18D940119; Wed, 17 Mar 2010 18:33:41 -0400 (EDT) X-Virus-Scanned: amavisd-new at darling.cs.uoguelph.ca Received: from darling.cs.uoguelph.ca ([127.0.0.1]) by localhost (darling.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7PcJ6gSLRSRj; Wed, 17 Mar 2010 18:33:41 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by darling.cs.uoguelph.ca (Postfix) with ESMTP id 2AA1A940099; Wed, 17 Mar 2010 18:33:41 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o2HMkRx21456; Wed, 17 Mar 2010 18:46:27 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Wed, 17 Mar 2010 18:46:27 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Jeremy Chadwick In-Reply-To: <20100317113953.GA14582@icarus.home.lan> Message-ID: References: <201003171120.o2HBK3CV082081@freefall.freebsd.org> <20100317113953.GA14582@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org, Kai Kockro Subject: Re: kern/144330: [nfs] mbuf leakage in nfsd with zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Mar 2010 22:33:42 -0000 On Wed, 17 Mar 2010, Jeremy Chadwick wrote: > On Wed, Mar 17, 2010 at 11:20:03AM +0000, Kai Kockro wrote: >> The following reply was made to PR kern/144330; it has been noted by GNATS. > > The problem appears to be specific to UDP NFS and not TCP. The > following is a statement from the PR originator: > Yes. Using TCP mounts or switching to the experimental server ("-e" option on both of mountd and nfsd) makes the problem go away. It also goes away if you disable the reply cache in the regular server, but I wouldn't recommend that. Sofar, i've had no luck coming up with a fix, so it could be a while... rick From owner-freebsd-fs@FreeBSD.ORG Thu Mar 18 04:23:36 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B521106566C; Thu, 18 Mar 2010 04:23:36 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 52A0D8FC17; Thu, 18 Mar 2010 04:23:36 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2I4Na4Z063684; Thu, 18 Mar 2010 04:23:36 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2I4Na2d063680; Thu, 18 Mar 2010 04:23:36 GMT (envelope-from linimon) Date: Thu, 18 Mar 2010 04:23:36 GMT Message-Id: <201003180423.o2I4Na2d063680@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-fs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/140682: [netgraph] [panic] random panic in netgraph X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Mar 2010 04:23:36 -0000 Synopsis: [netgraph] [panic] random panic in netgraph Responsible-Changed-From-To: freebsd-fs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Thu Mar 18 04:23:21 UTC 2010 Responsible-Changed-Why: Fix assignment brain-o. http://www.freebsd.org/cgi/query-pr.cgi?pr=140682 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 18 15:11:28 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D85F106564A for ; Thu, 18 Mar 2010 15:11:28 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 9E3408FC1E for ; Thu, 18 Mar 2010 15:11:27 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA08117 for ; Thu, 18 Mar 2010 17:11:25 +0200 (EET) (envelope-from avg@icyb.net.ua) Message-ID: <4BA2429D.5080808@icyb.net.ua> Date: Thu, 18 Mar 2010 17:11:25 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: ffs_mountfs micro-change X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Mar 2010 15:11:28 -0000 Remove redundant assignment of devvp->v_bufobj.bo_private. This assignment is already done in g_vfs_open. --- a/sys/ufs/ffs/ffs_vfsops.c +++ b/sys/ufs/ffs/ffs_vfsops.c @@ -669,7 +669,6 @@ ffs_mountfs(devvp, mp, td) if (mp->mnt_iosize_max > MAXPHYS) mp->mnt_iosize_max = MAXPHYS; - devvp->v_bufobj.bo_private = cp; devvp->v_bufobj.bo_ops = &ffs_ops; fs = NULL; -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 11:34:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBE94106566B for ; Fri, 19 Mar 2010 11:34:24 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id 899E88FC28 for ; Fri, 19 Mar 2010 11:34:24 +0000 (UTC) Received: from [192.168.2.164] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Fri, 19 Mar 2010 07:49:59 -0400 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::25 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-fs@freebsd.org X-SMFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw== Message-ID: <4BA3613F.4070606@comcast.net> Date: Fri, 19 Mar 2010 07:34:23 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100311 Thunderbird/3.0.1 MIME-Version: 1.0 To: User Questions , freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: bseklecki@noc.cfi.pgh.pa.us Subject: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 11:34:24 -0000 Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an NFS server to provide user home directories which get mounted across a few machines (all 6.3-RELEASE). For the past few weeks we have been running into problems where one particular client will go into an infinite loop where it is repeatedly trying to write data which causes the NFS server to return "reply ok 40 write ERROR: Input/output error PRE: POST:". This retry loop can cause between 20mbps and 500mbps of constant traffic on our network, depending on the size of the data associated with the failed write. We spent some time on the issue and determined that something on one of the clients is deleting a file as it is being written to by another NFS client. We were able to enable the NFS lockmgr and use lockf(1) to fix most of these conditions, and the frequency of this problem has dropped from once a night to once a week. However, it's still a problem and we can't necessarily force all of our users to "play nice" and use lockf/flock. Has anyone seen this before? No errors are being logged on the NFS server itself, but the "Server Ret-Failed" counter begins to increase rapidly whenever a client gets stuck in this infinite retry loop: Server Ret-Failed 224768961 I have a feeling that using NFS in such a matter may simply be prone to such problems, but what confuses me is why the NFS client system is infinitely retrying the write operation and causing itself so much grief. Thanks for any suggestions anyone can provide, Steve Polyack From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 13:19:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2792B1065675; Fri, 19 Mar 2010 13:19:29 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id ECA4B8FC14; Fri, 19 Mar 2010 13:19:28 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 9A84246B1A; Fri, 19 Mar 2010 09:19:28 -0400 (EDT) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPA id D91F28A025; Fri, 19 Mar 2010 09:19:27 -0400 (EDT) From: John Baldwin To: freebsd-fs@freebsd.org Date: Fri, 19 Mar 2010 08:31:00 -0400 User-Agent: KMail/1.12.1 (FreeBSD/7.3-CBSD-20100217; KDE/4.3.1; amd64; ; ) References: <4BA3613F.4070606@comcast.net> In-Reply-To: <4BA3613F.4070606@comcast.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201003190831.00950.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Fri, 19 Mar 2010 09:19:27 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.6 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: bseklecki@noc.cfi.pgh.pa.us, User Questions Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 13:19:29 -0000 On Friday 19 March 2010 7:34:23 am Steve Polyack wrote: > Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an > NFS server to provide user home directories which get mounted across a > few machines (all 6.3-RELEASE). For the past few weeks we have been > running into problems where one particular client will go into an > infinite loop where it is repeatedly trying to write data which causes > the NFS server to return "reply ok 40 write ERROR: Input/output error > PRE: POST:". This retry loop can cause between 20mbps and 500mbps of > constant traffic on our network, depending on the size of the data > associated with the failed write. > > We spent some time on the issue and determined that something on one of > the clients is deleting a file as it is being written to by another NFS > client. We were able to enable the NFS lockmgr and use lockf(1) to fix > most of these conditions, and the frequency of this problem has dropped > from once a night to once a week. However, it's still a problem and we > can't necessarily force all of our users to "play nice" and use lockf/flock. > > Has anyone seen this before? No errors are being logged on the NFS > server itself, but the "Server Ret-Failed" counter begins to increase > rapidly whenever a client gets stuck in this infinite retry loop: > Server Ret-Failed > 224768961 > > I have a feeling that using NFS in such a matter may simply be prone to > such problems, but what confuses me is why the NFS client system is > infinitely retrying the write operation and causing itself so much grief. Yes, your feeling is correct. This sort of race is inherent to NFS if you do not use some sort of locking protocol to resolve the race. The infinite retries sound like a client-side issue. Have you been able to try a newer OS version on a client to see if it still causes the same behavior? -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 13:23:56 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 451B01065676 for ; Fri, 19 Mar 2010 13:23:56 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id 001818FC15 for ; Fri, 19 Mar 2010 13:23:55 +0000 (UTC) Received: from [192.168.2.164] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Fri, 19 Mar 2010 09:39:29 -0400 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::530 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-fs@freebsd.org X-SMFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw== Message-ID: <4BA37AE9.4060806@comcast.net> Date: Fri, 19 Mar 2010 09:23:53 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100311 Thunderbird/3.0.1 MIME-Version: 1.0 To: John Baldwin References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> In-Reply-To: <201003190831.00950.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, User Questions , bseklecki@noc.cfi.pgh.pa.us Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 13:23:56 -0000 On 03/19/10 08:31, John Baldwin wrote: > On Friday 19 March 2010 7:34:23 am Steve Polyack wrote: > >> Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an >> NFS server to provide user home directories which get mounted across a >> few machines (all 6.3-RELEASE). For the past few weeks we have been >> running into problems where one particular client will go into an >> infinite loop where it is repeatedly trying to write data which causes >> the NFS server to return "reply ok 40 write ERROR: Input/output error >> PRE: POST:". This retry loop can cause between 20mbps and 500mbps of >> constant traffic on our network, depending on the size of the data >> associated with the failed write. >> >> We spent some time on the issue and determined that something on one of >> the clients is deleting a file as it is being written to by another NFS >> client. We were able to enable the NFS lockmgr and use lockf(1) to fix >> most of these conditions, and the frequency of this problem has dropped >> from once a night to once a week. However, it's still a problem and we >> can't necessarily force all of our users to "play nice" and use lockf/flock. >> >> Has anyone seen this before? No errors are being logged on the NFS >> server itself, but the "Server Ret-Failed" counter begins to increase >> rapidly whenever a client gets stuck in this infinite retry loop: >> Server Ret-Failed >> 224768961 >> >> I have a feeling that using NFS in such a matter may simply be prone to >> such problems, but what confuses me is why the NFS client system is >> infinitely retrying the write operation and causing itself so much grief. >> > Yes, your feeling is correct. This sort of race is inherent to NFS if you do > not use some sort of locking protocol to resolve the race. The infinite > retries sound like a client-side issue. Have you been able to try a newer OS > version on a client to see if it still causes the same behavior? > > I can't try a newer FBSD version on the client where we are seeing the problems, but I can recreate the problem fairly easily. Perhaps I'll try it with an 8.0 client. If I remember correctly, one of the strange things is that it doesn't seem to hit "critical mass" until a few hours after the operation first fails. I may be wrong, but I'll double check that when I check vs. 8.0-release. I forgot to add this in the first post, but these are all TCP NFS v3 mounts. Thanks for the response. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 15:05:23 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EAAF106564A for ; Fri, 19 Mar 2010 15:05:23 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id 095BE8FC1E for ; Fri, 19 Mar 2010 15:05:22 +0000 (UTC) Received: from [192.168.2.164] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Fri, 19 Mar 2010 11:20:56 -0400 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::599 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-fs@freebsd.org X-SMFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw== Message-ID: <4BA392B1.4050107@comcast.net> Date: Fri, 19 Mar 2010 11:05:21 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100311 Thunderbird/3.0.1 MIME-Version: 1.0 To: John Baldwin References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> <4BA37AE9.4060806@comcast.net> In-Reply-To: <4BA37AE9.4060806@comcast.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, User Questions , bseklecki@noc.cfi.pgh.pa.us Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 15:05:23 -0000 On 03/19/10 09:23, Steve Polyack wrote: > On 03/19/10 08:31, John Baldwin wrote: >> On Friday 19 March 2010 7:34:23 am Steve Polyack wrote: >>> Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an >>> NFS server to provide user home directories which get mounted across a >>> few machines (all 6.3-RELEASE). For the past few weeks we have been >>> running into problems where one particular client will go into an >>> infinite loop where it is repeatedly trying to write data which causes >>> the NFS server to return "reply ok 40 write ERROR: Input/output error >>> PRE: POST:". This retry loop can cause between 20mbps and 500mbps of >>> constant traffic on our network, depending on the size of the data >>> associated with the failed write. >>> >>> We spent some time on the issue and determined that something on one of >>> the clients is deleting a file as it is being written to by another NFS >>> client. We were able to enable the NFS lockmgr and use lockf(1) to fix >>> most of these conditions, and the frequency of this problem has dropped >>> from once a night to once a week. However, it's still a problem and we >>> can't necessarily force all of our users to "play nice" and use >>> lockf/flock. >>> >>> Has anyone seen this before? No errors are being logged on the NFS >>> server itself, but the "Server Ret-Failed" counter begins to increase >>> rapidly whenever a client gets stuck in this infinite retry loop: >>> Server Ret-Failed >>> 224768961 >>> >>> I have a feeling that using NFS in such a matter may simply be prone to >>> such problems, but what confuses me is why the NFS client system is >>> infinitely retrying the write operation and causing itself so much >>> grief. >> Yes, your feeling is correct. This sort of race is inherent to NFS >> if you do >> not use some sort of locking protocol to resolve the race. The infinite >> retries sound like a client-side issue. Have you been able to try a >> newer OS >> version on a client to see if it still causes the same behavior? >> > I can't try a newer FBSD version on the client where we are seeing the > problems, but I can recreate the problem fairly easily. Perhaps I'll > try it with an 8.0 client. If I remember correctly, one of the > strange things is that it doesn't seem to hit "critical mass" until a > few hours after the operation first fails. I may be wrong, but I'll > double check that when I check vs. 8.0-release. > > I forgot to add this in the first post, but these are all TCP NFS v3 > mounts. > > Thanks for the response. Ok, so I'm still able to trigger what appears to be the same retry loop with an 8.0-RELEASE nfsv3 client (going on 1.5 hours now): $ cat nfs.sh client#!/usr/local/bin/bash for a in {1..15} ; do sleep 1; echo "$a$a$"; done client$ ./nfs.sh >~/output the on the server while the above is running: server$ rm ~/output What happens is that you will see 3-4 of the same write attempts happen per minute via tcpdump. Our previous logs show that this is how it starts, and then ~4 hours later it begins to spiral out of control, throwing out up to 3,000 of the same failed write requests per second. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 20:20:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD50A1065672 for ; Fri, 19 Mar 2010 20:20:55 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id EB6B58FC13 for ; Fri, 19 Mar 2010 20:20:54 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id CCB1445E35; Fri, 19 Mar 2010 21:20:52 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id A2EA645C9C; Fri, 19 Mar 2010 21:20:47 +0100 (CET) Date: Fri, 19 Mar 2010 21:20:47 +0100 From: Pawel Jakub Dawidek To: Alexander Leidinger Message-ID: <20100319202047.GA1733@garage.freebsd.pl> References: <4B8B5780.2050601@jrv.org> <20100302103826.14273mzlwp38550k@webmail.leidinger.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vtzGhvizbBRQ85DL" Content-Disposition: inline In-Reply-To: <20100302103826.14273mzlwp38550k@webmail.leidinger.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs Subject: Re: [zfs] attach by name/uuid still attaches wrong device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 20:20:55 -0000 --vtzGhvizbBRQ85DL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 02, 2010 at 10:38:26AM +0100, Alexander Leidinger wrote: > Quoting "James R. Van Artsdalen" (from =20 > Sun, 28 Feb 2010 23:58:24 -0600): >=20 > >I don't think it's possible to do this right in vdev_geom.c: there's no > >way to guess what is intended without a hint from higher ZFS layers as > >to which drives should be found and which are new. >=20 > There is a way: do not attach blindly. The same code was used for 'zpool import' and 'zpool create', so ignoring guids was needed to make creation work. Fortunately I found solution today by looking at different part of ZFS. I can distinguish creation from import by looking at vd->vdev_spa->spa_load_state. When it is equal to SPA_LOAD_NONE we're creating a pool and it is not, we're importing. This should be fixed now, see @205346. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --vtzGhvizbBRQ85DL Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuj3J8ACgkQForvXbEpPzQSHQCg3XJGn31yFsURDGGFG7Rq7N9y WHUAmgNO/O9zI3EyR5Uxb6ODOdg2jkGE =RiCK -----END PGP SIGNATURE----- --vtzGhvizbBRQ85DL-- From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 20:29:50 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 165851065675 for ; Fri, 19 Mar 2010 20:29:50 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id C3E858FC1C for ; Fri, 19 Mar 2010 20:29:49 +0000 (UTC) Received: from [192.168.2.164] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Fri, 19 Mar 2010 16:45:20 -0400 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::1300 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-fs@freebsd.org X-SMFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw== Message-ID: <4BA3DEBC.2000608@comcast.net> Date: Fri, 19 Mar 2010 16:29:48 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100311 Thunderbird/3.0.1 MIME-Version: 1.0 To: John Baldwin References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> <4BA37AE9.4060806@comcast.net> <4BA392B1.4050107@comcast.net> In-Reply-To: <4BA392B1.4050107@comcast.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, User Questions , bseklecki@noc.cfi.pgh.pa.us Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 20:29:50 -0000 On 03/19/10 11:05, Steve Polyack wrote: > On 03/19/10 09:23, Steve Polyack wrote: >> On 03/19/10 08:31, John Baldwin wrote: >>> On Friday 19 March 2010 7:34:23 am Steve Polyack wrote: >>>> Hi, we use a FreeBSD 8-STABLE (from shortly after release) system >>>> as an >>>> NFS server to provide user home directories which get mounted across a >>>> few machines (all 6.3-RELEASE). For the past few weeks we have been >>>> running into problems where one particular client will go into an >>>> infinite loop where it is repeatedly trying to write data which causes >>>> the NFS server to return "reply ok 40 write ERROR: Input/output error >>>> PRE: POST:". This retry loop can cause between 20mbps and 500mbps of >>>> constant traffic on our network, depending on the size of the data >>>> associated with the failed write. >>>> >>> Yes, your feeling is correct. This sort of race is inherent to NFS >>> if you do >>> not use some sort of locking protocol to resolve the race. The >>> infinite >>> retries sound like a client-side issue. Have you been able to try a >>> newer OS >>> version on a client to see if it still causes the same behavior? >>> >> I can't try a newer FBSD version on the client where we are seeing >> the problems, but I can recreate the problem fairly easily. Perhaps >> I'll try it with an 8.0 client. If I remember correctly, one of the >> strange things is that it doesn't seem to hit "critical mass" until a >> few hours after the operation first fails. I may be wrong, but I'll >> double check that when I check vs. 8.0-release. >> >> I forgot to add this in the first post, but these are all TCP NFS v3 >> mounts. >> >> Thanks for the response. > > Ok, so I'm still able to trigger what appears to be the same retry > loop with an 8.0-RELEASE nfsv3 client (going on 1.5 hours now): > $ cat nfs.sh > client#!/usr/local/bin/bash > for a in {1..15} ; do > sleep 1; > echo "$a$a$"; > done > client$ ./nfs.sh >~/output > > the on the server while the above is running: > server$ rm ~/output > > What happens is that you will see 3-4 of the same write attempts > happen per minute via tcpdump. Our previous logs show that this is > how it starts, and then ~4 hours later it begins to spiral out of > control, throwing out up to 3,000 of the same failed write requests > per second. To anyone who is interested: I did some poking around with DTrace, which led me to the nfsiod client code. In src/sys/nfsclient/nfs_nfsiod.c: } else { if (bp->b_iocmd == BIO_READ) (void) nfs_doio(bp->b_vp, bp, bp->b_rcred, NULL); else (void) nfs_doio(bp->b_vp, bp, bp->b_wcred, NULL); } These two calls to nfs_doio trash the return codes (which are errors cascading up from various other NFS write-related functions). I'm not entirely familiar with the way nfsiod works, but if nfs_doio() or other subsequent functions are supposed to be removing the current async NFS operation from a queue which nfsiod handles, they are not doing so when they encounter an error. They simply report the error back to the caller, who in this case is not even looking at the value. I've tested this by pushing the return code into a new int, errno, and adding: if (errno) { NFS_DPF(ASYNCIO, ("nfssvc_iod: iod %d nfs_doio returned errno: %d\n", myiod, errno)); } The result is that my problematic repeatable circumstance begins logging "nfssvc_iod: iod 0 nfs_doio returned errno: 5" (corresponding to NFSERR_INVAL?) for each repetition of the failed write. The only things triggering this are my failed writes. I can also see the nfsiod0 process waking up each iteration. Do we need some kind of "retry x times then abort" logic within nfsiod_iod(), or does this belong in the subsequent functions, such as nfs_doio()? I think it's best to avoid these sorts of infinite loops which have the potential to take out the system or overload the network due to dumb decisions made by unprivileged users. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 22:49:46 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 106861065744; Fri, 19 Mar 2010 22:49:46 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D955D8FC16; Fri, 19 Mar 2010 22:49:45 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2JMnjDV089758; Fri, 19 Mar 2010 22:49:45 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2JMnjC8089752; Fri, 19 Mar 2010 22:49:45 GMT (envelope-from pjd) Date: Fri, 19 Mar 2010 22:49:45 GMT Message-Id: <201003192249.o2JMnjC8089752@freefall.freebsd.org> To: jhell@DataIX.net, pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/144447: [zfs] sharenfs fsunshare() & fsshare_main() non functiional. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 22:49:46 -0000 Synopsis: [zfs] sharenfs fsunshare() & fsshare_main() non functiional. State-Changed-From-To: open->feedback State-Changed-By: pjd State-Changed-When: ptk 19 mar 2010 22:48:50 UTC State-Changed-Why: I believe this is fixed in FreeBSD 8.x. Could you confirm this? Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: ptk 19 mar 2010 22:48:50 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=144447 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 22:51:41 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CCA7A106567D; Fri, 19 Mar 2010 22:51:41 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A3B0C8FC1A; Fri, 19 Mar 2010 22:51:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2JMpfCH097900; Fri, 19 Mar 2010 22:51:41 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2JMpfMc097896; Fri, 19 Mar 2010 22:51:41 GMT (envelope-from pjd) Date: Fri, 19 Mar 2010 22:51:41 GMT Message-Id: <201003192251.o2JMpfMc097896@freefall.freebsd.org> To: roddi@me.com, pjd@FreeBSD.org, freebsd-fs@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/144415: [zfs] [panic] kernel panics on boot after zfs crash X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 22:51:41 -0000 Synopsis: [zfs] [panic] kernel panics on boot after zfs crash State-Changed-From-To: open->suspended State-Changed-By: pjd State-Changed-When: ptk 19 mar 2010 22:51:02 UTC State-Changed-Why: I don't believe we will be able to help without any debug info. http://www.freebsd.org/cgi/query-pr.cgi?pr=144415 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 22:53:15 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6529B1065672; Fri, 19 Mar 2010 22:53:15 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3BB9C8FC0A; Fri, 19 Mar 2010 22:53:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2JMrFSE098059; Fri, 19 Mar 2010 22:53:15 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2JMrFf2098055; Fri, 19 Mar 2010 22:53:15 GMT (envelope-from pjd) Date: Fri, 19 Mar 2010 22:53:15 GMT Message-Id: <201003192253.o2JMrFf2098055@freefall.freebsd.org> To: pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/144402: [zfs] [panic] panic at zfs_znode_dmu_init: existing znode for dbuf X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 22:53:15 -0000 Synopsis: [zfs] [panic] panic at zfs_znode_dmu_init: existing znode for dbuf Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: ptk 19 mar 2010 22:51:58 UTC Responsible-Changed-Why: I'll look into this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=144402 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 22:57:10 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E027D106566B; Fri, 19 Mar 2010 22:57:10 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B71868FC1A; Fri, 19 Mar 2010 22:57:10 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2JMvAtm098432; Fri, 19 Mar 2010 22:57:10 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2JMvAxh098427; Fri, 19 Mar 2010 22:57:10 GMT (envelope-from pjd) Date: Fri, 19 Mar 2010 22:57:10 GMT Message-Id: <201003192257.o2JMvAxh098427@freefall.freebsd.org> To: pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/143343: [zfs] bug in sunlink flag on directories X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 22:57:11 -0000 Synopsis: [zfs] bug in sunlink flag on directories Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: ptk 19 mar 2010 22:56:35 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=143343 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 23:01:51 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4685E106566B; Fri, 19 Mar 2010 23:01:51 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1CE5B8FC12; Fri, 19 Mar 2010 23:01:51 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2JN1pp7007179; Fri, 19 Mar 2010 23:01:51 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2JN1ovH007175; Fri, 19 Mar 2010 23:01:50 GMT (envelope-from pjd) Date: Fri, 19 Mar 2010 23:01:50 GMT Message-Id: <201003192301.o2JN1ovH007175@freefall.freebsd.org> To: pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/142872: [zfs] ZFS ZVOL Lockmgr Deadlock X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 23:01:51 -0000 Synopsis: [zfs] ZFS ZVOL Lockmgr Deadlock Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: ptk 19 mar 2010 22:59:56 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=142872 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 19 23:15:07 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FCB2106564A; Fri, 19 Mar 2010 23:15:07 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 762A78FC13; Fri, 19 Mar 2010 23:15:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2JNF7ha017574; Fri, 19 Mar 2010 23:15:07 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2JNF7He017570; Fri, 19 Mar 2010 23:15:07 GMT (envelope-from pjd) Date: Fri, 19 Mar 2010 23:15:07 GMT Message-Id: <201003192315.o2JNF7He017570@freefall.freebsd.org> To: pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/142594: [zfs] Modification time reset to 1 Jan 1970 after fsync+crash on ZFS root volume ('legacy' mount) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Mar 2010 23:15:07 -0000 Synopsis: [zfs] Modification time reset to 1 Jan 1970 after fsync+crash on ZFS root volume ('legacy' mount) Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: ptk 19 mar 2010 23:02:05 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=142594 From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 00:23:02 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69DFC106564A; Sat, 20 Mar 2010 00:23:02 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 40CA38FC12; Sat, 20 Mar 2010 00:23:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2K0N21S077380; Sat, 20 Mar 2010 00:23:02 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2K0N2lO077376; Sat, 20 Mar 2010 00:23:02 GMT (envelope-from pjd) Date: Sat, 20 Mar 2010 00:23:02 GMT Message-Id: <201003200023.o2K0N2lO077376@freefall.freebsd.org> To: pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/141718: [zfs] [panic] kernel panic when 'zfs rename' is used on mounted snapshot X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 00:23:02 -0000 Synopsis: [zfs] [panic] kernel panic when 'zfs rename' is used on mounted snapshot Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: ptk 19 mar 2010 23:15:20 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=141718 From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 00:24:41 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A01931065672; Sat, 20 Mar 2010 00:24:41 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7723A8FC15; Sat, 20 Mar 2010 00:24:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2K0Of8M077439; Sat, 20 Mar 2010 00:24:41 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2K0OeEe077435; Sat, 20 Mar 2010 00:24:40 GMT (envelope-from pjd) Date: Sat, 20 Mar 2010 00:24:40 GMT Message-Id: <201003200024.o2K0OeEe077435@freefall.freebsd.org> To: dernst@gmx.de, pjd@FreeBSD.org, freebsd-fs@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/141177: [zfs] fsync() on FIFO causes panic() on zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 00:24:41 -0000 Synopsis: [zfs] fsync() on FIFO causes panic() on zfs State-Changed-From-To: open->closed State-Changed-By: pjd State-Changed-When: sob 20 mar 2010 00:24:09 UTC State-Changed-Why: Problem fixed. http://www.freebsd.org/cgi/query-pr.cgi?pr=141177 From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 00:29:38 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D528A1065670; Sat, 20 Mar 2010 00:29:38 +0000 (UTC) (envelope-from pjd@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AB89B8FC08; Sat, 20 Mar 2010 00:29:38 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2K0TccH077525; Sat, 20 Mar 2010 00:29:38 GMT (envelope-from pjd@freefall.freebsd.org) Received: (from pjd@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2K0Tc09077521; Sat, 20 Mar 2010 00:29:38 GMT (envelope-from pjd) Date: Sat, 20 Mar 2010 00:29:38 GMT Message-Id: <201003200029.o2K0Tc09077521@freefall.freebsd.org> To: Tom.Payne@unige.ch, pjd@FreeBSD.org, freebsd-fs@FreeBSD.org, pjd@FreeBSD.org From: pjd@FreeBSD.org Cc: Subject: Re: kern/141685: [zfs] zfs corruption on adaptec 5805 raid controller X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 00:29:38 -0000 Synopsis: [zfs] zfs corruption on adaptec 5805 raid controller State-Changed-From-To: open->feedback State-Changed-By: pjd State-Changed-When: sob 20 mar 2010 00:27:29 UTC State-Changed-Why: This is unlikely ZFS bug. It s hard to tell if the problem is in the driver, controller, cabels, disks, etc. You could try to configure geli(8) with data integrity verification on top of this array and see if geli will also report problems. Responsible-Changed-From-To: freebsd-fs->pjd Responsible-Changed-By: pjd Responsible-Changed-When: sob 20 mar 2010 00:27:29 UTC Responsible-Changed-Why: I'll take this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=141685 From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 00:58:16 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5E4D106564A; Sat, 20 Mar 2010 00:58:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 2B9AB8FC19; Sat, 20 Mar 2010 00:58:14 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAEW6o0uDaFvH/2dsb2JhbACbPnO7O4R8BA X-IronPort-AV: E=Sophos;i="4.51,277,1267419600"; d="scan'208";a="69311447" Received: from danube.cs.uoguelph.ca ([131.104.91.199]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 19 Mar 2010 20:58:14 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id D69A410842BD; Fri, 19 Mar 2010 20:58:13 -0400 (EDT) X-Virus-Scanned: amavisd-new at danube.cs.uoguelph.ca Received: from danube.cs.uoguelph.ca ([127.0.0.1]) by localhost (danube.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eQWt1OGBglIs; Fri, 19 Mar 2010 20:58:13 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 635CB108402D; Fri, 19 Mar 2010 20:58:13 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o2K1B3S21356; Fri, 19 Mar 2010 21:11:03 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 19 Mar 2010 21:11:03 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: John Baldwin In-Reply-To: <201003190831.00950.jhb@freebsd.org> Message-ID: References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, User Questions , bseklecki@noc.cfi.pgh.pa.us Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 00:58:16 -0000 On Fri, 19 Mar 2010, John Baldwin wrote: > On Friday 19 March 2010 7:34:23 am Steve Polyack wrote: >> Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an >> NFS server to provide user home directories which get mounted across a >> few machines (all 6.3-RELEASE). For the past few weeks we have been >> running into problems where one particular client will go into an >> infinite loop where it is repeatedly trying to write data which causes >> the NFS server to return "reply ok 40 write ERROR: Input/output error >> PRE: POST:". This retry loop can cause between 20mbps and 500mbps of I'm afraid I don't quite understand what you mean by "causes the NFS server to return "reply ok 40 write ERROR..."". Is this something logged by syslog (I can't find a printf like this in the kernel sources) or is this something that tcpdump is giving you or ??? Why I ask is that it seems to say that the server is returning EIO (or maybe 40 == EMSGSIZE). The server should return ESTALE (NFSERR_STALE) after a file has been deleted. If it is returning EIO, then that will cause the client to keep trying to write the dirty block to the server. (EIO is interpreted by the client as a "transient error".) [good stuff snipped] >> >> I have a feeling that using NFS in such a matter may simply be prone to >> such problems, but what confuses me is why the NFS client system is >> infinitely retrying the write operation and causing itself so much grief. > > Yes, your feeling is correct. This sort of race is inherent to NFS if you do > not use some sort of locking protocol to resolve the race. The infinite > retries sound like a client-side issue. Have you been able to try a newer OS > version on a client to see if it still causes the same behavior? > As John notes, having one client delete a file while another is trying to write it, is not a good thing. However, the server should return ESTALE after the file is deleted and that tells the client that the write can never succeed, so it marks the buffer cache block invalid and returns the error to the app. (The app. may not see it, if it doesn't check for error returns upon close as well as write, but that's another story...) If you could look at a packet trace via wireshark when the problem occurs, it would be nice to see what the server is returning. (If it isn't ESTALE and the file no longer exists on the server, then thats a server problem.) If it is returning ESTALE, then the client is busted. (At a glance, the client code looks like it would handle ESTALE as a fatal error for the buffer cache, but that doesn't mean it isn't broken, just that it doesn't appear wrong. Also, it looks like mmap'd writes won't recognize a fatal write error and will just keep trying to write the dirty page back to the server. Take this with a big grain of salt, since I just took a quick look at the sources. FreeBSD6->8 appear to be pretty much the same as far as this goes, in the client. Please let us know if you can see the server's error reply code. Good luck with it, rick ps: If the server isn't returning ESTALE, you could try switching to the experimental nfs server and see if it exhibits the same behaviour? ("-e" option on both mountd and nfsd, assuming the server is FreeBSD8.) From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 01:20:02 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0AC9106566C; Sat, 20 Mar 2010 01:20:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 54A2D8FC0C; Sat, 20 Mar 2010 01:20:02 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAGO/o0uDaFvJ/2dsb2JhbACbPnO7OoR8BA X-IronPort-AV: E=Sophos;i="4.51,277,1267419600"; d="scan'208";a="69651639" Received: from ganges.cs.uoguelph.ca ([131.104.91.201]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 19 Mar 2010 21:20:01 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by ganges.cs.uoguelph.ca (Postfix) with ESMTP id C5821FB809B; Fri, 19 Mar 2010 21:20:00 -0400 (EDT) X-Virus-Scanned: amavisd-new at ganges.cs.uoguelph.ca Received: from ganges.cs.uoguelph.ca ([127.0.0.1]) by localhost (ganges.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CrkY7SyDEoUh; Fri, 19 Mar 2010 21:20:00 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by ganges.cs.uoguelph.ca (Postfix) with ESMTP id 5766FFB8066; Fri, 19 Mar 2010 21:20:00 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o2K1WoZ23611; Fri, 19 Mar 2010 21:32:50 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 19 Mar 2010 21:32:50 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Steve Polyack In-Reply-To: <4BA3DEBC.2000608@comcast.net> Message-ID: References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> <4BA37AE9.4060806@comcast.net> <4BA392B1.4050107@comcast.net> <4BA3DEBC.2000608@comcast.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, bseklecki@noc.cfi.pgh.pa.us, User Questions Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 01:20:03 -0000 On Fri, 19 Mar 2010, Steve Polyack wrote: > > To anyone who is interested: I did some poking around with DTrace, which led > me to the nfsiod client code. > In src/sys/nfsclient/nfs_nfsiod.c: > } else { > if (bp->b_iocmd == BIO_READ) > (void) nfs_doio(bp->b_vp, bp, bp->b_rcred, NULL); > else > (void) nfs_doio(bp->b_vp, bp, bp->b_wcred, NULL); > } > If you look t nfs_doio(), it decides whether or not to mark the buffer invalid, based on the return value it gets. Some (EINTR, ETIMEDOUT, EIO) are not considered fatal, but the others are. (When the async I/O daemons call nfs_doio(), they are threads that couldn't care less if the underlying I/O op succeeded. The outcome of the I/O operation determines what nfs_doio() does with the buffer cache block.) > > The result is that my problematic repeatable circumstance begins logging > "nfssvc_iod: iod 0 nfs_doio returned errno: 5" (corresponding to > NFSERR_INVAL?) for each repetition of the failed write. The only things > triggering this are my failed writes. I can also see the nfsiod0 process > waking up each iteration. > Nope, errno 5 is EIO and that's where the problem is. I don't know why the server is returning EIO after the file has been deleted on the server (I assume you did that when running your little shell script?). > Do we need some kind of "retry x times then abort" logic within nfsiod_iod(), > or does this belong in the subsequent functions, such as nfs_doio()? I think > it's best to avoid these sorts of infinite loops which have the potential to > take out the system or overload the network due to dumb decisions made by > unprivileged users. > Nope, people don't like data not getting written back to a server when it is slow or temporarily network partitioned. The only thing that should stop a client from retrying a write back to the server is a fatal error from the server that says "this won't ever succeed". I think we need to figure out if the EIO (NFS3ERR_IO in wireshark) or if the server is sending NFS3ERR_STALE and the client is somehow munging that into EIO, causing the confusion. rick From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 02:27:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A54A7106566C for ; Sat, 20 Mar 2010 02:27:47 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from qmta04.westchester.pa.mail.comcast.net (qmta04.westchester.pa.mail.comcast.net [76.96.62.40]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5458FC17 for ; Sat, 20 Mar 2010 02:27:46 +0000 (UTC) Received: from omta04.westchester.pa.mail.comcast.net ([76.96.62.35]) by qmta04.westchester.pa.mail.comcast.net with comcast id vECC1d0020ldTLk54ETnKc; Sat, 20 Mar 2010 02:27:47 +0000 Received: from [10.0.0.51] ([71.199.122.142]) by omta04.westchester.pa.mail.comcast.net with comcast id vETm1d00334Sj4f3QETn5K; Sat, 20 Mar 2010 02:27:47 +0000 Message-ID: <4BA432C8.4040707@comcast.net> Date: Fri, 19 Mar 2010 22:28:24 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.8) Gecko/20100227 Lightning/1.0b1 Thunderbird/3.0.3 MIME-Version: 1.0 To: Rick Macklem References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> <4BA37AE9.4060806@comcast.net> <4BA392B1.4050107@comcast.net> <4BA3DEBC.2000608@comcast.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, bseklecki@noc.cfi.pgh.pa.us, User Questions Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 02:27:47 -0000 On 3/19/2010 9:32 PM, Rick Macklem wrote: > > On Fri, 19 Mar 2010, Steve Polyack wrote: > >> >> To anyone who is interested: I did some poking around with DTrace, >> which led me to the nfsiod client code. >> In src/sys/nfsclient/nfs_nfsiod.c: >> } else { >> if (bp->b_iocmd == BIO_READ) >> (void) nfs_doio(bp->b_vp, bp, bp->b_rcred, NULL); >> else >> (void) nfs_doio(bp->b_vp, bp, bp->b_wcred, NULL); >> } >> > > If you look t nfs_doio(), it decides whether or not to mark the buffer > invalid, based on the return value it gets. Some (EINTR, ETIMEDOUT, EIO) > are not considered fatal, but the others are. (When the async I/O > daemons call nfs_doio(), they are threads that couldn't care less if > the underlying I/O op succeeded. The outcome of the I/O operation > determines what nfs_doio() does with the buffer cache block.) I was looking at this and noticed the above after my last post. >> >> The result is that my problematic repeatable circumstance begins >> logging "nfssvc_iod: iod 0 nfs_doio returned errno: 5" (corresponding >> to NFSERR_INVAL?) for each repetition of the failed write. The only >> things triggering this are my failed writes. I can also see the >> nfsiod0 process waking up each iteration. >> > > Nope, errno 5 is EIO and that's where the problem is. I don't know why > the server is returning EIO after the file has been deleted on the > server (I assume you did that when running your little shell script?). Yes, while running the simple shell script I simply deleted the file on the NFS server itself. >> Do we need some kind of "retry x times then abort" logic within >> nfsiod_iod(), or does this belong in the subsequent functions, such >> as nfs_doio()? I think it's best to avoid these sorts of infinite >> loops which have the potential to take out the system or overload the >> network due to dumb decisions made by unprivileged users. >> > Nope, people don't like data not getting written back to a server when > it is slow or temporarily network partitioned. The only thing that should > stop a client from retrying a write back to the server is a fatal error > from the server that says "this won't ever succeed". > > I think we need to figure out if the EIO (NFS3ERR_IO in wireshark) or > if the server is sending NFS3ERR_STALE and the client is somehow munging > that into EIO, causing the confusion. This makes sense. According to wireshark, the server is indeed transmitting "Status: NFS3ERR_IO (5)". Perhaps this should be STALE instead; it sounds more correct than marking it a general IO error. Also, the NFS server is serving its share off of a ZFS filesystem, if it makes any difference. I suppose ZFS could be talking to the NFS server threads with some mismatched language, but I doubt it. Thanks for the informative response, Steve From owner-freebsd-fs@FreeBSD.ORG Sat Mar 20 03:14:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCC65106566C; Sat, 20 Mar 2010 03:14:24 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5E9F28FC1D; Sat, 20 Mar 2010 03:14:24 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAFbao0uDaFvH/2dsb2JhbACbPnO7EYR8BA X-IronPort-AV: E=Sophos;i="4.51,278,1267419600"; d="scan'208";a="69657474" Received: from danube.cs.uoguelph.ca ([131.104.91.199]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 19 Mar 2010 23:14:23 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 6E2F2108427E; Fri, 19 Mar 2010 23:14:23 -0400 (EDT) X-Virus-Scanned: amavisd-new at danube.cs.uoguelph.ca Received: from danube.cs.uoguelph.ca ([127.0.0.1]) by localhost (danube.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p0DKoEaI02CN; Fri, 19 Mar 2010 23:14:23 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id E190410841FA; Fri, 19 Mar 2010 23:14:22 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o2K3RDc06841; Fri, 19 Mar 2010 23:27:13 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Fri, 19 Mar 2010 23:27:13 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Steve Polyack In-Reply-To: <4BA432C8.4040707@comcast.net> Message-ID: References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> <4BA37AE9.4060806@comcast.net> <4BA392B1.4050107@comcast.net> <4BA3DEBC.2000608@comcast.net> <4BA432C8.4040707@comcast.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, bseklecki@noc.cfi.pgh.pa.us, User Questions Subject: Re: FreeBSD NFS client goes into infinite retry loop X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 03:14:25 -0000 On Fri, 19 Mar 2010, Steve Polyack wrote: [good stuff snipped] > > This makes sense. According to wireshark, the server is indeed transmitting > "Status: NFS3ERR_IO (5)". Perhaps this should be STALE instead; it sounds > more correct than marking it a general IO error. Also, the NFS server is > serving its share off of a ZFS filesystem, if it makes any difference. I > suppose ZFS could be talking to the NFS server threads with some mismatched > language, but I doubt it. > Ok, now I think we're making progress. If VFS_FHTOVP() doesn't return ESTALE when the file no longer exists, the NFS server returns whatever error it has returned. So, either VFS_FHTOVP() succeeds after the file has been deleted, which would be a problem that needs to be fixed within ZFS OR ZFS returns an error other than ESTALE when it doesn't exist. Try the following patch on the server (which just makes any error returned by VFS_FHTOVP() into ESTALE) and see if that helps. --- nfsserver/nfs_srvsubs.c.sav 2010-03-19 22:06:43.000000000 -0400 +++ nfsserver/nfs_srvsubs.c 2010-03-19 22:07:22.000000000 -0400 @@ -1127,6 +1127,8 @@ } } error = VFS_FHTOVP(mp, &fhp->fh_fid, vpp); + if (error != 0) + error = ESTALE; vfs_unbusy(mp); if (error) goto out; Please let me know if the patch helps, rick