From owner-freebsd-fs@FreeBSD.ORG Sun Jun 26 18:20:36 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 03DA516A41C for ; Sun, 26 Jun 2005 18:20:36 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (cs.earlham.edu [159.28.230.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8D6843D1F for ; Sun, 26 Jun 2005 18:20:35 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (localhost [127.0.0.1]) by quark.cs.earlham.edu (8.13.4/8.13.3) with ESMTP id j5QIKVmr005401 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 26 Jun 2005 13:20:31 -0500 (EST) (envelope-from skylar@cs.earlham.edu) Received: (from skylar@localhost) by quark.cs.earlham.edu (8.13.4/8.13.3/Submit) id j5QIKV1o005400 for fs@freebsd.org; Sun, 26 Jun 2005 13:20:31 -0500 (EST) (envelope-from skylar@cs.earlham.edu) X-Authentication-Warning: quark.cs.earlham.edu: skylar set sender to skylar@quark.cs.earlham.edu using -f Date: Sun, 26 Jun 2005 13:20:31 -0500 From: Skylar Thompson To: fs@freebsd.org Message-ID: <20050626182031.GA5268@quark.cs.earlham.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7AUc2qLy4jB3hD7Z" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Sender: "Skylar Thompson" X-Accept-Primary-Language: en X-Accept-Secondary-Language: es SMTP-Mailing-Host: quark.cs.earlham.edu X-Operating-System: FreeBSD 5.4-RELEASE-p1 X-Uptime: 1:17PM up 20 days, 19:04, 16 users, load averages: 0.29, 0.19, 0.17 X-Editor: VIM - Vi IMproved 6.3 (2004 June 7, compiled May 14 2005 15:15:17) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (quark.cs.earlham.edu [127.0.0.1]); Sun, 26 Jun 2005 13:20:32 -0500 (EST) X-Virus-Scanned: ClamAV 0.85.1/957/Fri Jun 24 20:17:57 2005 on quark.cs.earlham.edu X-Virus-Status: Clean X-Sanitizer: This message has passed the MIMEDefang sanitizer. X-Sanitizer-URL: http://www.cs.earlham.edu/applied-groups/admin/ X-Sanitizer-Version: MIMEDefang/ECSanitizer $Revision: 1.18 $ X-Sanitizer-Config-Version: $Revision: 1.180 $ X-Scanned-By: MIMEDefang 2.51 on 192.168.0.3 X-Spam-Status: No, score=-2.8 required=8.0 tests=ALL_TRUSTED autolearn=failed version=3.0.3 X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on quark.cs.earlham.edu Cc: Subject: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Skylar Thompson List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jun 2005 18:20:36 -0000 --7AUc2qLy4jB3hD7Z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I've discovered a repeatable problem with FreeBSD's UFS2 snapshots. If I create several snapshots, and then do heavy disk I/O on the original filesystem (deletions, creations, simple touches, etc.) I can cause the I/O system to crash. There is no kernel panic, and the machine still answers pings, but no disk I/O occurs. I can replicate this on a dual-processor beige-box system with a Mylex RAID controller and a RAID-5 set, and also on a dual-processor Dell Poweredge 2650 with a PERC 3/i RAID controller and a RAID-5 set and RAID-1 set. FreeBSD 5.4-RELEASE is installed on both systems, and SMP is enabled as well, with HTT disabled on the Poweredge. I have DDB compiled in, so I can get debug information but I don't know what to look for. --=20 -- Skylar Thompson (skylar@cs.earlham.edu) -- http://www.cs.earlham.edu/~skylar/ --7AUc2qLy4jB3hD7Z Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCvvHvsc4yyULgN4YRAp9YAKCpPZsj14OGuuvinJgbKnN7H6EDcQCfWEWZ 5kmdDFdtkqlqLrE9VIzCiP4= =eauT -----END PGP SIGNATURE----- --7AUc2qLy4jB3hD7Z-- From owner-freebsd-fs@FreeBSD.ORG Mon Jun 27 13:18:42 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2574F16A41C for ; Mon, 27 Jun 2005 13:18:42 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id CF7AC43D49 for ; Mon, 27 Jun 2005 13:18:40 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j5RDIWXC024850; Mon, 27 Jun 2005 08:18:32 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42BFFCA2.1060206@centtech.com> Date: Mon, 27 Jun 2005 08:18:26 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Skylar Thompson References: <20050626182031.GA5268@quark.cs.earlham.edu> In-Reply-To: <20050626182031.GA5268@quark.cs.earlham.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jun 2005 13:18:42 -0000 Skylar Thompson wrote: > I've discovered a repeatable problem with FreeBSD's UFS2 snapshots. If I > create several snapshots, and then do heavy disk I/O on the original > filesystem (deletions, creations, simple touches, etc.) I can cause the I/O > system to crash. There is no kernel panic, and the machine still answers > pings, but no disk I/O occurs. I can replicate this on a dual-processor > beige-box system with a Mylex RAID controller and a RAID-5 set, and also on > a dual-processor Dell Poweredge 2650 with a PERC 3/i RAID controller and a > RAID-5 set and RAID-1 set. FreeBSD 5.4-RELEASE is installed on both > systems, and SMP is enabled as well, with HTT disabled on the Poweredge. I > have DDB compiled in, so I can get debug information but I don't know what > to look for. If you are able to get into the debugger, can you get a backtrace? Also - you might want to send this to freebsd-stable@, try upgrading to -stable, or try installing 6-CURRENT and see if the problem exists there (if that's an option). Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Mon Jun 27 13:40:21 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4A85116A41C for ; Mon, 27 Jun 2005 13:40:21 +0000 (GMT) (envelope-from delphij@frontfree.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [210.51.165.229]) by mx1.FreeBSD.org (Postfix) with ESMTP id C122743D62 for ; Mon, 27 Jun 2005 13:40:20 +0000 (GMT) (envelope-from delphij@frontfree.net) Received: from beastie.frontfree.net (unknown [219.239.99.7]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id CAF9FEB249F for ; Mon, 27 Jun 2005 21:40:17 +0800 (CST) Received: from localhost (localhost.frontfree.net [127.0.0.1]) by beastie.frontfree.net (Postfix) with ESMTP id 7BF7E1320CD; Mon, 27 Jun 2005 21:40:15 +0800 (CST) Received: from beastie.frontfree.net ([127.0.0.1]) by localhost (beastie.frontfree.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 05105-11; Mon, 27 Jun 2005 21:40:09 +0800 (CST) Received: by beastie.frontfree.net (Postfix, from userid 1001) id 80D3D13205E; Mon, 27 Jun 2005 21:40:08 +0800 (CST) Date: Mon, 27 Jun 2005 21:40:08 +0800 From: Xin LI To: Skylar Thompson Message-ID: <20050627134008.GA5764@frontfree.net> References: <20050626182031.GA5268@quark.cs.earlham.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="k+w/mQv8wyuph6w0" Content-Disposition: inline In-Reply-To: <20050626182031.GA5268@quark.cs.earlham.edu> User-Agent: Mutt/1.4.2.1i X-GPG-key-ID/Fingerprint: 0xCAEEB8C0 / 43B8 B703 B8DD 0231 B333 DC28 39FB 93A0 CAEE B8C0 X-GPG-Public-Key: http://www.delphij.net/delphij.asc X-Operating-System: FreeBSD beastie.frontfree.net 5.4-RELEASE-p1 FreeBSD 5.4-RELEASE-p1 #1: Fri May 27 00:47:15 CST 2005 delphij@beastie.frontfree.net:/usr/obj/usr/src/sys/BEASTIE i386 X-URL: http://www.delphij.net X-By: delphij@beastie.frontfree.net X-Location: Beijing, China X-Virus-Scanned: amavisd-new at frontfree.net Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jun 2005 13:40:21 -0000 --k+w/mQv8wyuph6w0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jun 26, 2005 at 01:20:31PM -0500, Skylar Thompson wrote: > I've discovered a repeatable problem with FreeBSD's UFS2 snapshots. If I > create several snapshots, and then do heavy disk I/O on the original > filesystem (deletions, creations, simple touches, etc.) I can cause the I= /O > system to crash. There is no kernel panic, and the machine still answers > pings, but no disk I/O occurs. I can replicate this on a dual-processor > beige-box system with a Mylex RAID controller and a RAID-5 set, and also = on > a dual-processor Dell Poweredge 2650 with a PERC 3/i RAID controller and a > RAID-5 set and RAID-1 set. FreeBSD 5.4-RELEASE is installed on both > systems, and SMP is enabled as well, with HTT disabled on the Poweredge. I > have DDB compiled in, so I can get debug information but I don't know what > to look for. I think a script that can reliably trigger the "crash" would be helpful. What do you mean by "IO system crash", BTW? I got confused since it does not cause kernel panic and stop ping responses. Do you mean that the I/O system was stalled/suspended when there is heavy disk operations? My guess is that there is some underlying deadlock(s) present. Would you mind compiling WITESS/WITESS_SUPPORT into your kernel and give it a try? This will reduce performance, but would also be helpful for picking locking bugs. Thanks in advance! Cheers, --=20 Xin LI http://www.delphij.net/ See complete headers for GPG key and other information. --k+w/mQv8wyuph6w0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCwAG4/cVsHxFZiIoRAsDvAKCIIHcGqSx3btWHVgfqE2s9HmMNFACfYRdh TTmCWZEvoMVYBK1buCScNDY= =Iqkn -----END PGP SIGNATURE----- --k+w/mQv8wyuph6w0-- From owner-freebsd-fs@FreeBSD.ORG Tue Jun 28 16:47:11 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5D0F916A46D for ; Tue, 28 Jun 2005 16:47:11 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (cs.earlham.edu [159.28.230.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8772543D53 for ; Tue, 28 Jun 2005 16:47:08 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from [159.28.57.98] (wir057098.rpa.earlham.edu [159.28.57.98]) (authenticated bits=0) by quark.cs.earlham.edu (8.13.4/8.13.3) with ESMTP id j5SGkwRw091814 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 28 Jun 2005 11:47:04 -0500 (EST) (envelope-from skylar@cs.earlham.edu) Message-ID: <42C18544.4000909@cs.earlham.edu> Date: Tue, 28 Jun 2005 12:13:40 -0500 From: Skylar Thompson Organization: Earlham College Computer Science Department User-Agent: Mozilla Thunderbird 1.0 (X11/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Xin LI References: <20050626182031.GA5268@quark.cs.earlham.edu> <20050627134008.GA5764@frontfree.net> In-Reply-To: <20050627134008.GA5764@frontfree.net> X-Enigmail-Version: 0.89.5.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3FF10CF33F5EBE6F7C45F97B" X-Scanned-By: MIMEDefang 2.51 on 159.28.230.3 X-Scanned-By: milter-bcc/0.8.72 (cs.earlham.edu [159.28.230.3]); Tue, 28 Jun 2005 11:47:04 -0500 X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-1.6 (quark.cs.earlham.edu [159.28.230.3]); Tue, 28 Jun 2005 11:47:04 -0500 (EST) X-Virus-Scanned: ClamAV 0.85.1/959/Mon Jun 27 18:00:06 2005 on quark.cs.earlham.edu X-Virus-Status: Clean X-Sanitizer: This message has passed the MIMEDefang sanitizer. X-Sanitizer-URL: http://www.cs.earlham.edu/applied-groups/admin/ X-Sanitizer-Version: MIMEDefang/ECSanitizer $Revision: 1.18 $ X-Sanitizer-Config-Version: $Revision: 1.180 $ X-Spam-Status: No, score=0.0 required=8.0 tests=none autolearn=failed version=3.0.3 X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on quark.cs.earlham.edu Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jun 2005 16:47:11 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3FF10CF33F5EBE6F7C45F97B Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Xin LI wrote: >On Sun, Jun 26, 2005 at 01:20:31PM -0500, Skylar Thompson wrote: > > >>I've discovered a repeatable problem with FreeBSD's UFS2 snapshots. If I >>create several snapshots, and then do heavy disk I/O on the original >>filesystem (deletions, creations, simple touches, etc.) I can cause the I/O >>system to crash. There is no kernel panic, and the machine still answers >>pings, but no disk I/O occurs. I can replicate this on a dual-processor >>beige-box system with a Mylex RAID controller and a RAID-5 set, and also on >>a dual-processor Dell Poweredge 2650 with a PERC 3/i RAID controller and a >>RAID-5 set and RAID-1 set. FreeBSD 5.4-RELEASE is installed on both >>systems, and SMP is enabled as well, with HTT disabled on the Poweredge. I >>have DDB compiled in, so I can get debug information but I don't know what >>to look for. >> >> > >I think a script that can reliably trigger the "crash" would be helpful. > > I was using this script to take the snapshots: #!/bin/sh if [ -f /var/run/hourly_snap ]; then echo "Lock file exists. Exiting...." exit 1 else HOUR=`date "+%H"` touch /var/run/hourly_snap for f in / /usr /var /clients; do if [ -f $f/snapshots/hourly_snap.$HOUR ]; then rm -f $f/snapshots/hourly_snap.$HOUR fi mksnap_ffs $f $f/snapshots/hourly_snap.$HOUR; done rm /var/run/hourly_snap fi I ran this once every other hour, so I had 12 snapshots in circulation at any given time. The number of snapshots seemed to exacerbate the problem; just having one or two around rarely (although sometimes) caused a crash. >What do you mean by "IO system crash", BTW? I got confused since it does >not cause kernel panic and stop ping responses. Do you mean that the >I/O system was stalled/suspended when there is heavy disk operations? > > Yes. The kernel still responds and I can get into DDB just fine, but there's no disk activity, at least on the affected filesystem. Usually it's /usr, which has many used inodes on account of ports and src. >My guess is that there is some underlying deadlock(s) present. Would you >mind compiling WITESS/WITESS_SUPPORT into your kernel and give it a try? >This will reduce performance, but would also be helpful for picking locking >bugs. > > > Sure. I've got the 2650 booted up with WITNESS support in addition to DDB. Where should I go from here? -- -- Skylar Thompson (skylar@cs.earlham.edu) -- http://www.cs.earlham.edu/~skylar/ --------------enig3FF10CF33F5EBE6F7C45F97B Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCwYVIsc4yyULgN4YRAoNUAKCM08ndP7Rx/gBBOvLdktRmSu/z0QCeMEDj 036FSKdyLFjEELNwkz3WSZI= =15PY -----END PGP SIGNATURE----- --------------enig3FF10CF33F5EBE6F7C45F97B-- From owner-freebsd-fs@FreeBSD.ORG Wed Jun 29 19:38:55 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E3CE916A41F for ; Wed, 29 Jun 2005 19:38:54 +0000 (GMT) (envelope-from emartinez@crockettint.com) Received: from mxc1.crockettint.com (chameleon1.crockettint.com [206.224.72.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 920D243D55 for ; Wed, 29 Jun 2005 19:38:52 +0000 (GMT) (envelope-from emartinez@crockettint.com) Received: from localhost (localhost.crockettint.com [127.0.0.1]) by mxc1.crockettint.com (Postfix) with ESMTP id 3D89E33F68; Wed, 29 Jun 2005 14:38:51 -0500 (CDT) Received: from mxc1.crockettint.com ([127.0.0.1]) by localhost (mxc1.crockettint.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 76757-08; Wed, 29 Jun 2005 14:38:49 -0500 (CDT) Received: from memnoch (rrcs-24-153-230-83.sw.biz.rr.com [24.153.230.83]) by mxc1.crockettint.com (Postfix) with ESMTP id 884AD33F35; Wed, 29 Jun 2005 14:38:49 -0500 (CDT) From: "Edgar Martinez" To: "'Scott Long'" Date: Wed, 29 Jun 2005 14:39:10 -0500 Organization: Crockett International MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 Thread-Index: AcV1OJ87POHBOngqT2OFUxxIBzpcMQHqQEmQ In-Reply-To: <42B61DCD.4030307@samsco.org> Message-Id: <20050629193849.884AD33F35@mxc1.crockettint.com> X-Virus-Scanned: amavisd-new at crockettint.com Cc: freebsd-fs@freebsd.org Subject: RE: UFS2+Softupdates Corruption Regardless on Seven various systems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: emartinez@crockettint.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jun 2005 19:38:55 -0000 Again sorry for the delay. I am back and ready to help as much as possible on this issue. When and how do you notice the corruption? I found the corruption during a decompression of a large tarball. During the untarring, the process stopped with an error. Subsequent attempts would stop randomly in the process. I then did a simple fsck on the fs to see if anything was amiss...which is when I discovered the problem...This occurred after 2 months of runtime... Does it have a particular pattern? None that I have found...other then long uptimes... Would it be possible to try a different brand of disk controller in order to rule out the driver being buggy? It is a TX2 which has had some pretty nifty support for a while. However from my understanding of the TX2, it really is not a true hardware controller...in fact, I would have used atacontrol if I could go back in time... -----Original Message----- From: Scott Long [mailto:scottl@samsco.org] Sent: Sunday, June 19, 2005 8:37 PM To: emartinez@crockettint.com Cc: freebsd-fs@freebsd.org Subject: Re: UFS2+Softupdates Corruption Regardless on Seven various systems Edgar Martinez wrote: > All, > > > > I have a network of FBSD boxen running 5.3 w/ 2x PATA WD1200JB Drives and a > Promise Fastrack TX2 controller in mirror. The systems mainly just pass > internet traffic and rarely ever touch the disks. After running for a few > weeks -> months.the disks become corrupted forcing a manual fsck from single > user mode. And since the system is thousands of miles away, it can become > painful to walk someone with a language barrier thru that. > > > > Question is WHY does this occur? > > How can you avoid this? > > What can you do to remotely fix the issue? > > Any proactive maintenance I need to be doing? > > Did I mention I would like to know WHY? > This certainly sounds like a bug, and is not something that people normally see. When and how do you notice the corruption? Does it have a particular pattern? Would it be possible to try a different brand of disk controller in order to rule out the driver being buggy? Scott From owner-freebsd-fs@FreeBSD.ORG Thu Jun 30 00:10:54 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0021116A41C for ; Thu, 30 Jun 2005 00:10:53 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (cs.earlham.edu [159.28.230.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id A504243D1D for ; Thu, 30 Jun 2005 00:10:51 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (localhost [127.0.0.1]) by quark.cs.earlham.edu (8.13.4/8.13.3) with ESMTP id j5U0AloI053067 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 29 Jun 2005 19:10:48 -0500 (EST) (envelope-from skylar@cs.earlham.edu) Received: (from skylar@localhost) by quark.cs.earlham.edu (8.13.4/8.13.3/Submit) id j5U0AhoQ053061; Wed, 29 Jun 2005 19:10:43 -0500 (EST) (envelope-from skylar@cs.earlham.edu) X-Authentication-Warning: quark.cs.earlham.edu: skylar set sender to skylar@quark.cs.earlham.edu using -f Date: Wed, 29 Jun 2005 19:10:43 -0500 From: Skylar Thompson To: Xin LI Message-ID: <20050630001043.GA52960@quark.cs.earlham.edu> References: <20050626182031.GA5268@quark.cs.earlham.edu> <20050627134008.GA5764@frontfree.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="envbJBWh7q8WU6mo" Content-Disposition: inline In-Reply-To: <20050627134008.GA5764@frontfree.net> User-Agent: Mutt/1.4.2.1i X-Sender: "Skylar Thompson" X-Accept-Primary-Language: en X-Accept-Secondary-Language: es SMTP-Mailing-Host: quark.cs.earlham.edu X-Operating-System: FreeBSD 5.4-RELEASE-p1 X-Uptime: 7:09PM up 2 days, 23:56, 14 users, load averages: 0.33, 0.74, 0.66 X-Editor: VIM - Vi IMproved 6.3 (2004 June 7, compiled May 14 2005 15:15:17) X-Scanned-By: MIMEDefang 2.51 on 192.168.0.3 X-Scanned-By: milter-bcc/0.8.72 (localhost [127.0.0.1]); Wed, 29 Jun 2005 19:10:48 -0500 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (quark.cs.earlham.edu [127.0.0.1]); Wed, 29 Jun 2005 19:10:48 -0500 (EST) X-Virus-Scanned: ClamAV 0.86.1/960/Tue Jun 28 23:31:06 2005 on quark.cs.earlham.edu X-Virus-Status: Clean X-Sanitizer: This message has passed the MIMEDefang sanitizer. X-Sanitizer-URL: http://www.cs.earlham.edu/applied-groups/admin/ X-Sanitizer-Version: MIMEDefang/ECSanitizer $Revision: 1.18 $ X-Sanitizer-Config-Version: $Revision: 1.180 $ X-Spam-Status: No, score=-2.8 required=8.0 tests=ALL_TRUSTED autolearn=failed version=3.0.4 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on quark.cs.earlham.edu Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Skylar Thompson List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jun 2005 00:10:54 -0000 --envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 27, 2005 at 09:40:08PM +0800, Xin LI wrote: > My guess is that there is some underlying deadlock(s) present. Would you > mind compiling WITESS/WITESS_SUPPORT into your kernel and give it a try? > This will reduce performance, but would also be helpful for picking locki= ng > bugs. I've replicated the crash again on the 2650, and the kernel logged this error to the console: handle workitem freeblocks: block count aa addsingleroute: error 17 --=20 -- Skylar Thompson (skylar@cs.earlham.edu) -- http://www.cs.earlham.edu/~skylar/ --envbJBWh7q8WU6mo Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCwziDsc4yyULgN4YRAn97AKCDCWBKWLBbrntODCOJDXheMff5MgCgicT4 ofpdKq1Ue3MtiKMpxWmX0Bs= =HQB2 -----END PGP SIGNATURE----- --envbJBWh7q8WU6mo-- From owner-freebsd-fs@FreeBSD.ORG Thu Jun 30 13:49:25 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E322C16A41C for ; Thu, 30 Jun 2005 13:49:25 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (cs.earlham.edu [159.28.230.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8EA2743D1D for ; Thu, 30 Jun 2005 13:49:25 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from [192.168.1.10] (12-222-56-168.client.insightBB.com [12.222.56.168]) (authenticated bits=0) by quark.cs.earlham.edu (8.13.4/8.13.3) with ESMTP id j5UDnIqJ079693 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 30 Jun 2005 08:49:21 -0500 (EST) (envelope-from skylar@cs.earlham.edu) Message-ID: <42C47C86.2020306@cs.earlham.edu> Date: Thu, 30 Jun 2005 18:13:10 -0500 From: Skylar Thompson Organization: Earlham College Computer Science Department User-Agent: Mozilla Thunderbird 1.0 (X11/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Xin LI References: <20050626182031.GA5268@quark.cs.earlham.edu> <20050627134008.GA5764@frontfree.net> In-Reply-To: <20050627134008.GA5764@frontfree.net> X-Enigmail-Version: 0.89.5.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig8DA748B6E41D7684263296AA" X-Scanned-By: MIMEDefang 2.51 on 159.28.230.3 X-Scanned-By: milter-bcc/0.8.72 (cs.earlham.edu [159.28.230.3]); Thu, 30 Jun 2005 08:49:21 -0500 X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-1.6 (quark.cs.earlham.edu [159.28.230.3]); Thu, 30 Jun 2005 08:49:21 -0500 (EST) X-Virus-Scanned: ClamAV 0.86.1/960/Tue Jun 28 23:31:06 2005 on quark.cs.earlham.edu X-Virus-Status: Clean X-Sanitizer: This message has passed the MIMEDefang sanitizer. X-Sanitizer-URL: http://www.cs.earlham.edu/applied-groups/admin/ X-Sanitizer-Version: MIMEDefang/ECSanitizer $Revision: 1.18 $ X-Sanitizer-Config-Version: $Revision: 1.180 $ X-Spam-Status: No, score=1.8 required=8.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on quark.cs.earlham.edu Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jun 2005 13:49:26 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig8DA748B6E41D7684263296AA Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Xin LI wrote: >My guess is that there is some underlying deadlock(s) present. Would you >mind compiling WITESS/WITESS_SUPPORT into your kernel and give it a try? >This will reduce performance, but would also be helpful for picking locking >bugs. > > I've managed to crash the machine again, this time with this error message coming up on the console every few seconds: kern.maxpipekva exceeded -- -- Skylar Thompson (skylar@cs.earlham.edu) -- http://www.cs.earlham.edu/~skylar/ --------------enig8DA748B6E41D7684263296AA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCxHyJsc4yyULgN4YRAuilAJ9C6FfMVoMxP7Q0wdQBftEUlq/mlwCeK0OE fsy0Zi7+Ksj6tpLeLOZT//o= =cxgS -----END PGP SIGNATURE----- --------------enig8DA748B6E41D7684263296AA-- From owner-freebsd-fs@FreeBSD.ORG Fri Jul 1 12:32:39 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3FCBF16A41C for ; Fri, 1 Jul 2005 12:32:39 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id EC25543D1F for ; Fri, 1 Jul 2005 12:32:38 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j61CWZAt077267; Fri, 1 Jul 2005 07:32:35 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42C537D8.2000403@centtech.com> Date: Fri, 01 Jul 2005 07:32:24 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Bakul Shah References: <200506221445.j5MEje5P097719@gate.bitblocks.com> In-Reply-To: <200506221445.j5MEje5P097719@gate.bitblocks.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Cluster Filesystem for FreeBSD - any interest? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jul 2005 12:32:39 -0000 Bakul Shah wrote: >>Hmm. I'm not sure if it can or not. I'll try to explain what I'm >>dreaming of. I currently have about 1000 clients needing access to the >>same pools of data (read/write) all the time. The data changes >>constantly. There is a lot of this data. We use NFS currently. > > > Sounds like you want SGI's clustered xfs.... Maybe so - also, GFS looks perfect as well, only side effect is license, but that isn't *that* big of an issue. Porting may be easier? >>I'll be honest here - I'm not a code developer. I would love to learn >>some C here, and 'just do it', but filesystems aren't exactly simple, so >>I'm looking for a group of people that would love to code up something >>amazing like this - I'll support the developers and hopefully learn >>something in the process. My goal personally would be to do anything I >>could to make the developers work most productively, and do testing. I >>can probably provide equipment, and a good testbed for it. > > > If you are not a seasoned programmer in _some_ language, this > will not be easy at all. Well, depends on if you call perl programming 'real' programming. :) Perl just isn't quite the tool for this job (although I'm sure there are some out there that might actually argue). > One suggestion is to develop an abstract model of what a CFS > is. Coming up with a clear detailed precise specification is > not an easy task either but it has to be done and if you can > do it, it will be immensely helpful all around. You will > truly understand what you are doing, you have a basis for > evaluating design choices, you will have made choices before > writing any code, you can write test cases, writing code is > far easier etc. etc. Google for clustered filesystems. > The citeseer site has some papers as well. Thanks - this is a great suggestion. I'll try to come up with something. Really, the truth is (now that I have read even more docs), RedHat's GFS is exactly what I would like for FreeBSD. They already have all the components, etc. I would prefer a BSD licensed piece of software, but I just want something that works on FreeBSD mostly. > A couple FS specific suggestions: > - perhaps clustering can be built on top of existing > filesystems. Each machine's local filesystem is considered > a cache and you use some sort of cache coherency protocol. > That way you don't have to deal with filesystem allocation > and layout issues. I see - that's an interesting idea. Almost like each machine could mount the shared version read-only, then slap a layer on top that is connected to a cache coherency manager (maybe there is a daemon on each node, and the nodes sync their caches via the network) to keep the filesystems 'in sync'. Then maybe only one elected node actually writes the data to the disk. If that node dies, then another node is elected. > - a network wide stable storage `disk' may be easier to do > given GEOM. There are atleast N copies of each data block. > Data may be cached locally at any site but writing data is > done as a distributed transaction. So again cache > coherency is needed. A network RAID if you will! I'm not sure how this would work. A network RAID with geom+ggate is simple (I've done this a couple times - cool!), but how does that get me shared read-write access to the same data? > But again, let me stress that one must have a clear *model* > of the problem being solved. Getting distributed programs > right is very hard even at an abstract model level. > Debugging a distributed program that doesn't have a clear > model is, well, for masochists (nothing against them -- I > bet even they'd rather get their pain some other way:-) :) I understand. Any nudging in the right direction here would be appreciated. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Jul 1 12:33:53 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D1CA16A41F for ; Fri, 1 Jul 2005 12:33:53 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id C1AE243D53 for ; Fri, 1 Jul 2005 12:33:52 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j61CXjPu077286; Fri, 1 Jul 2005 07:33:48 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42C5381E.9080708@centtech.com> Date: Fri, 01 Jul 2005 07:33:34 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: yfyoufeng@263.net References: <42B825CC.806@centtech.com> <20050622023727.GA19408@afields.ca> <42B954FE.2070406@centtech.com> <1119490133.2298.14.camel@localhost.localdomain> In-Reply-To: <1119490133.2298.14.camel@localhost.localdomain> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Allan Fields Subject: Re: Cluster Filesystem for FreeBSD - any interest? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jul 2005 12:33:53 -0000 yf-263 wrote: > =E5=9C=A8 2005-06-22=E4=B8=89=E7=9A=84 07:09 -0500=EF=BC=8CEric Anderso= n=E5=86=99=E9=81=93=EF=BC=9A >=20 >>Allan Fields wrote: >> >>>On Tue, Jun 21, 2005 at 09:35:56AM -0500, Eric Anderson wrote: >>> >>> >>>>This is something I've brought up before on other lists, but I'm curi= ous=20 >>>>if anyone is interested in developing a BSD licensed clustered=20 >>>>filesystem for FreeBSD (and anyone else)? >>> >>> >>>A few questions: >>> >>>Could this be done as a stackable file system (vnode layer distributed= >>>file system) or did you have something else in mind (i.e. specifically= >>>a full implementation of a network filesystem including storage >>>layer)? >> >>Hmm. I'm not sure if it can or not. I'll try to explain what I'm=20 >>dreaming of. I currently have about 1000 clients needing access to the= =20 >=20 >=20 > This is exactly what we are doing now :) but only a proof-of-concept > toy. >=20 > Now it has a asymmetric arch, i.e. a client, some metadata server, and > some filedata server. >=20 > thanks for your professional description as what we are target to ;) >=20 >=20 >>same pools of data (read/write) all the time. The data changes=20 >>constantly. There is a lot of this data. We use NFS currently.=20 >>FreeBSD is *very* fast and stable at serving NFS data. The problem is,= =20 >>that even though it is very fast and stable, I still cannot pump out=20 >>enough bits fast enough with one machine, and if that one machine fails= =20 >>(hardware problems, etc), then all my machines are hung waiting for me = >>to bring it back online. >> >>So, what I would love to have, is this kind of setup: shared media=20 >>storage (fibre channel SAN, iscsi, or something like ggated possibly), = >>connected up to a cluster of hosts running FreeBSD. Each FreeBSD serve= r=20 >>has access to the logical disks, same partitions, and can mount them al= l=20 >>r/w. Now, I can kind of do this now, however there are obviously some = >>issues with this currently. I want all machines in this cluster to be = >>able to serve the data via NFS (or http, or anything else for that=20 >>matter really - if you can make NFS work, anything will pretty much=20 >>work) simultaneously from the same partitions, and see writes=20 >>immediately as the other hosts in the cluster commit them. >> >>I currently have a solution just like this for Linux - Polyserve=20 >>(http://www.polyserve.com) has a clustered filesystem for linux, that=20 >>works very well. I've even tried to convince them to port it to=20 >>FreeBSD, but it falls on deaf ears, so it's time to make our own. >> >> >> >>>Why not a port of an existing network filesystem say from Linux? >>>(A BSD rewrite could be done, if the code was GPLed.) Would >>>cross-platform capabilities make sense? >> >>That would work fine I'm sure - but I have found some similar threads i= n=20 >>the past that claim it would be just as hard and time consuming to port= =20 >>one as it would be to create one from scratch. Cross platform=20 >>capabilities would be great, but I'm mostly interested in getting=20 >>FreeBSD into this arena (as it will soon be an extremely important one = >>to be in). >> >> >> >>>How do you see this comparing to device-level solutions? I know >>>the argument can be made to implement file systems/storage >>>abstractions at multiple layers, but I thought I might ask. >> >>I'm not sure of a device level solution that does this. I think the OS= =20 >>has to know to commit the meta-data to a journal, or otherwise let the = >>other machines know about locking, etc, in order for this to work. >> >> >> >>>The other thing is there a wealth of filesystem papers out there, >>>any in specific caught your eye? >> >>No - can you point me to some? >> >>I'll be honest here - I'm not a code developer. I would love to learn = >>some C here, and 'just do it', but filesystems aren't exactly simple, s= o=20 >>I'm looking for a group of people that would love to code up something = >>amazing like this - I'll support the developers and hopefully learn=20 >>something in the process. My goal personally would be to do anything I= =20 >>could to make the developers work most productively, and do testing. I= =20 >>can probably provide equipment, and a good testbed for it. You've mentioned this a few times before - is this something you will be = offering publicly sometime soon? Eric --=20 ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Jul 1 18:18:48 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 15D8E16A422 for ; Fri, 1 Jul 2005 18:18:48 +0000 (GMT) (envelope-from cdillon@wolves.k12.mo.us) Received: from mail.wolves.k12.mo.us (mail.wolves.k12.mo.us [207.160.214.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id D633443D66 for ; Fri, 1 Jul 2005 18:18:47 +0000 (GMT) (envelope-from cdillon@wolves.k12.mo.us) Received: from localhost (localhost [127.0.0.1]) by mail.wolves.k12.mo.us (Postfix) with ESMTP id 249761FEAA; Fri, 1 Jul 2005 13:18:47 -0500 (CDT) Received: from mail.wolves.k12.mo.us ([127.0.0.1]) by localhost (mail.wolves.k12.mo.us [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 52867-02-6; Fri, 1 Jul 2005 13:18:44 -0500 (CDT) Received: by mail.wolves.k12.mo.us (Postfix, from userid 1001) id 6639F1FE5B; Fri, 1 Jul 2005 13:18:44 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mail.wolves.k12.mo.us (Postfix) with ESMTP id 64BBE1A902; Fri, 1 Jul 2005 13:18:44 -0500 (CDT) Date: Fri, 1 Jul 2005 13:18:44 -0500 (CDT) From: Chris Dillon To: Skylar Thompson In-Reply-To: <20050626182031.GA5268@quark.cs.earlham.edu> Message-ID: <20050701130315.C52686@duey.wolves.k12.mo.us> References: <20050626182031.GA5268@quark.cs.earlham.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: amavisd-new at wolves.k12.mo.us Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jul 2005 18:18:48 -0000 On Sun, 26 Jun 2005, Skylar Thompson wrote: > I've discovered a repeatable problem with FreeBSD's UFS2 snapshots. > If I create several snapshots, and then do heavy disk I/O on the > original filesystem (deletions, creations, simple touches, etc.) I > can cause the I/O system to crash. There is no kernel panic, and the > machine still answers pings, but no disk I/O occurs. I see a similar problem (I think stuck processes show they are in the 'disk wait' state in top/ps), but my problem occurs on a production box so I've been unable to debug it. I've taken to rebooting the box automatically every day about 15 minutes before the snapshots are scheduled to be made using Ralf S. Engelschall's snapshot scripts (http://people.freebsd.org/~rse/snapshot/). The daily reboot seems to prevent the problem from happening. If I don't reboot the system daily, I can only go one to three days without a problem. > I can replicate this on a dual-processor beige-box system with a > Mylex RAID controller and a RAID-5 set, and also on a dual-processor > Dell Poweredge 2650 with a PERC 3/i RAID controller and a RAID-5 set > and RAID-1 set. FreeBSD 5.4-RELEASE is installed on both systems, > and SMP is enabled as well, with HTT disabled on the Poweredge. I > have DDB compiled in, so I can get debug information but I don't > know what to look for. I'm using FreeBSD 5.4-STABLE on a relatively new dual-processor HP DL380 G3 with integrated SmartArray 5i+ and a 7-disk RAID5 array. Given the range of hardware we are seeing the problem on, it doesn't seem to be hardware or driver related. In the meantime, try rebooting the box at a scheduled time every day to see if that helps alleviate your problem. -- Chris Dillon - cdillon(at)wolves.k12.mo.us FreeBSD: The fastest, most open, and most stable OS on the planet - Available for IA32, IA64, AMD64, PC98, Alpha, and UltraSPARC architectures - PowerPC, ARM, MIPS, and S/390 under development - http://www.freebsd.org Q: Because it reverses the logical flow of conversation. A: Why is putting a reply at the top of the message frowned upon? From owner-freebsd-fs@FreeBSD.ORG Sat Jul 2 00:38:26 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E138A16A41C for ; Sat, 2 Jul 2005 00:38:26 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from gate.bitblocks.com (bitblocks.com [209.204.185.216]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8FE8843D1F for ; Sat, 2 Jul 2005 00:38:26 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost [127.0.0.1]) by gate.bitblocks.com (8.13.3/8.13.1) with ESMTP id j620cO7F071025; Fri, 1 Jul 2005 17:38:25 -0700 (PDT) (envelope-from bakul@bitblocks.com) Message-Id: <200507020038.j620cO7F071025@gate.bitblocks.com> To: Eric Anderson In-reply-to: Your message of "Fri, 01 Jul 2005 07:32:24 CDT." <42C537D8.2000403@centtech.com> Date: Fri, 01 Jul 2005 17:38:24 -0700 From: Bakul Shah Cc: freebsd-fs@freebsd.org Subject: Re: Cluster Filesystem for FreeBSD - any interest? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jul 2005 00:38:27 -0000 > > A couple FS specific suggestions: > > - perhaps clustering can be built on top of existing > > filesystems. Each machine's local filesystem is considered > > a cache and you use some sort of cache coherency protocol. > > That way you don't have to deal with filesystem allocation > > and layout issues. > > I see - that's an interesting idea. Almost like each machine could > mount the shared version read-only, then slap a layer on top that is > connected to a cache coherency manager (maybe there is a daemon on each > node, and the nodes sync their caches via the network) to keep the > filesystems 'in sync'. Then maybe only one elected node actually writes > the data to the disk. If that node dies, then another node is elected. \begin{handwaving} What I was thinking of: - The cluster system assures that there are atleast N copies of every file at N+ separate locations. - More than N copies may be cached dependign on usage pattern. - any node can write. The system takes care of replication and placement. - meta data, directories are implemented *above* this level. - more likely you'd want to map file *fragments* to local files so that a file can grow beyond one disk and smaller fragements mean you don't have to cache an entire file. - you still need to mediate access at file level but this is no different from two+ processes accessing a local file. Of course, the devil is in the details! > > - a network wide stable storage `disk' may be easier to do > > given GEOM. There are atleast N copies of each data block. > > Data may be cached locally at any site but writing data is > > done as a distributed transaction. So again cache > > coherency is needed. A network RAID if you will! > > I'm not sure how this would work. A network RAID with geom+ggate is > simple (I've done this a couple times - cool!), but how does that get me > shared read-write access to the same data? What I had in mind something like this: Each logical block is backed by N physical blocks at N sites. Individual filesystems live in partitions of this space. So in effect you have a single NFS server per filesystem that deals with all metadata+dir lookup but due to caching read access should be faster. When a server goes down, another server can be elected. > :) I understand. Any nudging in the right direction here would be > appreciated. I'd probably start with modelling a single filesystem and how it maps to a sequence of disk blocks (*without* using any code or worrying about details of formats but capturing the essential elements). I'd describe various operations in terms of preconditions and postconditions. Then, I'd extend the model to deal with redundancy and so on. Then I'd model various failure modes. etc. If you are interested _enough_ we can take this offline and try to work something out. You may even be able to use perl to create an `executable' specification:-) \end{handwaving} From owner-freebsd-fs@FreeBSD.ORG Sat Jul 2 17:02:10 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1698416A41C for ; Sat, 2 Jul 2005 17:02:10 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from quark.cs.earlham.edu (cs.earlham.edu [159.28.230.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id A3F6043D1D for ; Sat, 2 Jul 2005 17:02:05 +0000 (GMT) (envelope-from skylar@cs.earlham.edu) Received: from [192.168.1.10] (12-222-56-168.client.insightBB.com [12.222.56.168]) (authenticated bits=0) by quark.cs.earlham.edu (8.13.4/8.13.3) with ESMTP id j62H20xF008939 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 2 Jul 2005 12:02:01 -0500 (EST) (envelope-from skylar@cs.earlham.edu) Message-ID: <42C6C873.8050808@cs.earlham.edu> Date: Sat, 02 Jul 2005 12:01:39 -0500 From: Skylar Thompson Organization: Earlham College Computer Science Department User-Agent: Mozilla Thunderbird 1.0 (X11/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Chris Dillon References: <20050626182031.GA5268@quark.cs.earlham.edu> <20050701130315.C52686@duey.wolves.k12.mo.us> In-Reply-To: <20050701130315.C52686@duey.wolves.k12.mo.us> X-Enigmail-Version: 0.89.5.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigFBF234F2B53AC36A838B70FB" X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-1.6 (quark.cs.earlham.edu [159.28.230.3]); Sat, 02 Jul 2005 12:02:02 -0500 (EST) X-Virus-Scanned: ClamAV 0.86.1/963/Fri Jul 1 08:27:29 2005 on quark.cs.earlham.edu X-Virus-Status: Clean X-Sanitizer: This message has passed the MIMEDefang sanitizer. X-Sanitizer-URL: http://www.cs.earlham.edu/applied-groups/admin/ X-Sanitizer-Version: MIMEDefang/ECSanitizer $Revision: 1.18 $ X-Sanitizer-Config-Version: $Revision: 1.180 $ X-Scanned-By: MIMEDefang 2.51 on 159.28.230.3 X-Spam-Status: No, score=1.8 required=8.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on quark.cs.earlham.edu Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jul 2005 17:02:10 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigFBF234F2B53AC36A838B70FB Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Chris Dillon wrote: > > I'm using FreeBSD 5.4-STABLE on a relatively new dual-processor HP > DL380 G3 with integrated SmartArray 5i+ and a 7-disk RAID5 array. > Given the range of hardware we are seeing the problem on, it doesn't > seem to be hardware or driver related. > > In the meantime, try rebooting the box at a scheduled time every day > to see if that helps alleviate your problem. > No joy. The machine hung again yesterday afternoon, with the error "kern.ipc.maxpipekva exceeded" repeated over and over again on the console. -- -- Skylar Thompson (skylar@cs.earlham.edu) -- http://www.cs.earlham.edu/~skylar/ --------------enigFBF234F2B53AC36A838B70FB Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCxsh5sc4yyULgN4YRAsxuAJsF/elkpAcyP5hj4sdF6Vexw7SZgQCgmvFq LNA07phE+eqdBvpSVf33SBU= =adVW -----END PGP SIGNATURE----- --------------enigFBF234F2B53AC36A838B70FB-- From owner-freebsd-fs@FreeBSD.ORG Sat Jul 2 18:31:25 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4309516A41C for ; Sat, 2 Jul 2005 18:31:25 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.232]) by mx1.FreeBSD.org (Postfix) with ESMTP id DC80043D1F for ; Sat, 2 Jul 2005 18:31:23 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 1A5A351326; Sat, 2 Jul 2005 14:31:18 -0400 (EDT) Date: Sat, 2 Jul 2005 14:31:17 -0400 From: Kris Kennaway To: Skylar Thompson Message-ID: <20050702183117.GA4513@xor.obsecurity.org> References: <20050626182031.GA5268@quark.cs.earlham.edu> <20050701130315.C52686@duey.wolves.k12.mo.us> <42C6C873.8050808@cs.earlham.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="82I3+IH0IqGh5yIs" Content-Disposition: inline In-Reply-To: <42C6C873.8050808@cs.earlham.edu> User-Agent: Mutt/1.4.2.1i Cc: fs@freebsd.org Subject: Re: Snapshot problems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jul 2005 18:31:25 -0000 --82I3+IH0IqGh5yIs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 02, 2005 at 12:01:39PM -0500, Skylar Thompson wrote: > Chris Dillon wrote: >=20 > > > >I'm using FreeBSD 5.4-STABLE on a relatively new dual-processor HP=20 > >DL380 G3 with integrated SmartArray 5i+ and a 7-disk RAID5 array.=20 > >Given the range of hardware we are seeing the problem on, it doesn't=20 > >seem to be hardware or driver related. > > > >In the meantime, try rebooting the box at a scheduled time every day=20 > >to see if that helps alleviate your problem. > > >=20 > No joy. The machine hung again yesterday afternoon, with the error=20 > "kern.ipc.maxpipekva exceeded" repeated over and over again on the consol= e. So increase kern.ipc.maxpipekva? Kris --82I3+IH0IqGh5yIs Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCxt11Wry0BWjoQKURApC2AKDVH7+K8LLlcHV38jIiSrkmbTs/GQCdEj8p aTQDsnKkCtD929mTfDfA1/E= =ssZT -----END PGP SIGNATURE----- --82I3+IH0IqGh5yIs--