From owner-freebsd-current@FreeBSD.ORG Thu Nov 13 01:14:21 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D938B106568D for ; Thu, 13 Nov 2008 01:14:21 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 73BFE8FC14 for ; Thu, 13 Nov 2008 01:14:21 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.3/8.14.2) with ESMTP id mAD1F6ED008926; Wed, 12 Nov 2008 19:15:07 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.3/8.14.3/Submit) id mAD1F66Z008925; Wed, 12 Nov 2008 19:15:06 -0600 (CST) (envelope-from brooks) Date: Wed, 12 Nov 2008 19:15:06 -0600 From: Brooks Davis To: Steve Kargl Message-ID: <20081113011506.GA6719@lor.one-eyed-alien.net> References: <20081112235903.GA19865@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UlVJffcvxoiEqYs2" Content-Disposition: inline In-Reply-To: <20081112235903.GA19865@troutmask.apl.washington.edu> User-Agent: Mutt/1.5.17 (2007-11-01) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Wed, 12 Nov 2008 19:15:07 -0600 (CST) Cc: freebsd-current@freebsd.org Subject: Re: GEOM hangover disables NFS X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2008 01:14:21 -0000 --UlVJffcvxoiEqYs2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 12, 2008 at 03:59:03PM -0800, Steve Kargl wrote: > On three nodes in my cluster (nodes n17, n18, and n19), I had > GEOM use /dev/ad4s1e for tests with gmirror and ggated/ggatec. > I found that GEOM was insufficient for my needs and decided=20 > to return the 3 partitions to NFS-exported partitions. It seems > that once GEOM touches a partition, the partition can no longer > be used by NFS. > > I'll illustrute the problem with n17:/dev/ad4s1e. In what follows, > n10 is the master node. Both n10 and n17 have brand new worlds > and kernels from about 45 minutes ago. >=20 > n10:kargl[203] ssh n17 > n17:kargl[201] df > Filesystem 1M-blocks Used Avail Capacity Mounted on > /dev/ad4s1a 247 104 123 46% / > devfs 0 0 0 100% /dev > /dev/ad4s1e 222780 0 204958 0% /data > /dev/ad4s1d 3962 182 3463 5% /usr > n10:/home 193947 92855 85576 52% /home > n10:/usr/local 19832 10494 7750 58% /usr/local >=20 > n17:kargl[202] tail -1 /etc/exports > /data -alldirs node10 node21 >=20 > The above is after a 'newfs -U /dev/ad4s1e' and a reboot. >=20 > n10:root[244] ls / | grep -E ^n > n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/ >=20 > n10:root[245] mount_nfs -o tcp n17:/data /n17 > n10:root[246] mount -v | grep n17 > n17:/data on /n17 (nfs, fsid 0eff000303000000) > n10:root[247] ls /n17 > ls: /n17: Input/output error > n10:root[248] ls / | grep -E ^n > ls: n17: Input/output error > n11/ n12/ n13/ n14/ n15/ n16/ n18/ n19/ n20/ n21/ >=20 > n10:root[251] umount /n17 > n10:root[252] ls / | grep -E ^n > n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/ >=20 > So, how does one exorcise GEOM from /dev/ad4s1e? All disks and partitions are always represented as GEOM devices. This is the only way to access storage so I'm pretty sure you don't actually want to get rid of GEOM. :) The output of "sysctl -b kern.geom.conftxt" would show if there were bits that were being picked up by a stray geom consumer. A dmesg from the problem nodes might also be helpful in determining what's wrong. The best guess would be left over state, probably at the end of the volumes. If local access to the file system actually works, NFS really shouldn't care what's been done to the disk below the file system. -- Brooks --UlVJffcvxoiEqYs2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iD8DBQFJG3+ZXY6L6fI4GtQRAlRpAKDC8lAETXUD46pXwg0ok5YDj4g8EgCeMKYP uVNqjVu+Ms5cRQiccHhATU0= =sfSz -----END PGP SIGNATURE----- --UlVJffcvxoiEqYs2--