From owner-freebsd-stable@FreeBSD.ORG Sat Sep 9 17:38:15 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 08B4116A403 for ; Sat, 9 Sep 2006 17:38:15 +0000 (UTC) (envelope-from karl@FS.denninger.net) Received: from FS.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 67CC243D46 for ; Sat, 9 Sep 2006 17:38:14 +0000 (GMT) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net (localhost [127.0.0.1]) by FS.denninger.net (8.13.6/8.13.1) with SMTP id k89HcDPg001493 for ; Sat, 9 Sep 2006 12:38:13 -0500 (CDT) (envelope-from karl@FS.denninger.net) Received: from fs.denninger.net [127.0.0.1] by Spamblock-sys (LOCAL); Sat Sep 9 12:38:13 2006 Received: (from karl@localhost) by FS.denninger.net (8.13.6/8.13.1/Submit) id k89HcDNx001490 for freebsd-stable@freebsd.org; Sat, 9 Sep 2006 12:38:13 -0500 (CDT) (envelope-from karl) Date: Sat, 9 Sep 2006 12:38:13 -0500 From: Karl Denninger To: freebsd-stable@freebsd.org Message-ID: <20060909173813.GA1388@FS.denninger.net> Mail-Followup-To: freebsd-stable@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Organization: Karl's Sushi and Packet Smashers X-Die-Spammers: Spammers cheerfully broiled for supper and served with ketchup! Subject: ARRRRGH! Guys, who's breaking -STABLE's GMIRROR code?! X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Sep 2006 17:38:15 -0000 This is not cool folks. Anyone know what I have to roll back to - and what files I have to roll back - to stop this cluster-##@kery? tty ad4 ad6 twed0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 224 453 0.61 0 0.00 120.16 427 50.06 0.61 0 0.00 2 0 4 2 92 See that? There's nothing really running. What I tried to do was "gmirror insert b500 ad4s1" The command took, but NO IO WAS TAKEN TO THE TARGET DRIVE FOR REBUILDING; the SOURCE disk was locked in a 100% I/O run, and after stopping the rebuild THE I/O INFINITE LOOP IS STILL GOING ON! I had a PRODUCTION MACHINE go down on my last night over this when it attempted to run its backup process and wedged due to process table overflow; the first attempt apparently never finished the day before and the second, to a SECOND backup disk (I have a rolling disk backup system using GMIRROR's resync) caused the system to wedge in an I/O wait. This was also not cleanly restartable, as the root partition had multiple error on it that fsck -p couldn't fix. This is a SEVERE emergency in that anyone who has a disk that has to be rebuilt under -STABLE right now (sources as of 7 September) is screwed, blued and tattooed. That PRODUCTION machine is running UNPROTECTED right now (no mirroring) as a consequence of this, and I can neither back it up using the usual mirror NOR restore its redundancy! I see only one comment about GMIRROR changes in the commitlogs since 9/1, and it claims to be (mostly) cosmetic. Obviously not! -- -- Karl Denninger (karl@denninger.net) Internet Consultant & Kids Rights Activist http://www.denninger.net My home on the net - links to everything I do! http://scubaforum.org Your UNCENSORED place to talk about DIVING! http://genesis3.blogspot.com Musings Of A Sentient Mind