From owner-freebsd-current@FreeBSD.ORG Fri Nov 14 10:36:53 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBFDD1065678; Fri, 14 Nov 2008 10:36:53 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from itchy.rabson.org (unknown [IPv6:2002:50b1:e8f2:1::143]) by mx1.freebsd.org (Postfix) with ESMTP id A84A48FC08; Fri, 14 Nov 2008 10:36:53 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc] (unknown [IPv6:2001:470:909f:1:21b:63ff:feb8:5abc]) by itchy.rabson.org (Postfix) with ESMTP id 5293A3FB8; Fri, 14 Nov 2008 10:36:09 +0000 (GMT) Message-Id: <6EC5D6EA-8864-43C9-A76C-115AAA50F573@rabson.org> From: Doug Rabson To: Ivan Voras In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Fri, 14 Nov 2008 10:36:51 +0000 References: <20081112235903.GA19865@troutmask.apl.washington.edu> X-Mailer: Apple Mail (2.929.2) Cc: freebsd-current@freebsd.org Subject: Re: GEOM hangover disables NFS X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Nov 2008 10:36:54 -0000 On 14 Nov 2008, at 10:05, Ivan Voras wrote: > Steve Kargl wrote: >> On three nodes in my cluster (nodes n17, n18, and n19), I had >> GEOM use /dev/ad4s1e for tests with gmirror and ggated/ggatec. >> I found that GEOM was insufficient for my needs and decided >> to return the 3 partitions to NFS-exported partitions. It seems >> that once GEOM touches a partition, the partition can no longer >> be used by NFS. >> >> I'll illustrute the problem with n17:/dev/ad4s1e. In what follows, >> n10 is the master node. Both n10 and n17 have brand new worlds >> and kernels from about 45 minutes ago. >> >> n10:kargl[203] ssh n17 >> n17:kargl[201] df >> Filesystem 1M-blocks Used Avail Capacity Mounted on >> /dev/ad4s1a 247 104 123 46% / >> devfs 0 0 0 100% /dev >> /dev/ad4s1e 222780 0 204958 0% /data >> /dev/ad4s1d 3962 182 3463 5% /usr >> n10:/home 193947 92855 85576 52% /home >> n10:/usr/local 19832 10494 7750 58% /usr/local >> >> n17:kargl[202] tail -1 /etc/exports >> /data -alldirs node10 node21 >> >> The above is after a 'newfs -U /dev/ad4s1e' and a reboot. >> >> n10:root[244] ls / | grep -E ^n >> n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/ >> >> n10:root[245] mount_nfs -o tcp n17:/data /n17 >> n10:root[246] mount -v | grep n17 >> n17:/data on /n17 (nfs, fsid 0eff000303000000) >> n10:root[247] ls /n17 >> ls: /n17: Input/output error >> n10:root[248] ls / | grep -E ^n >> ls: n17: Input/output error >> n11/ n12/ n13/ n14/ n15/ n16/ n18/ n19/ n20/ n21/ >> >> n10:root[251] umount /n17 >> n10:root[252] ls / | grep -E ^n >> n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/ >> >> So, how does one exorcise GEOM from /dev/ad4s1e? > > Your message is very confusing. Here's some information that could > help you: > > * "GEOM" is just a name for "system of handling disk-like devices in > FreeBSD". If you "remove GEOM" (which would probably be hard to do and > would involve heavily modifying the kernel source) you will have no > access to your disk drives and other similar devices. > > * Using ggated/ggatec you can either: > - mount the file system exactly once as rw and none as ro (doesn't > matter if the file system is on local node or exported, the total is > important) > - mount the file system as ro any number of times and none as rw. > (In other words, only one machine can have the file system mounted for > reading and writing. Once a file system has been mounted as rw, no > other > machines can do anything with it. This has nothing to do with NFS as > NFS > operates on top of a mounted file system and inherits its rules.) > > * You can remove gmirror metadata on devices with "gmirror clean". > ggated and ggatec don't store metadata on devices. This was a bug in the NFS code which is now fixed. The bug had nothing to do with GEOM.