From owner-freebsd-stable@FreeBSD.ORG Mon Nov 24 21:50:47 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62C4F1065674; Mon, 24 Nov 2008 21:50:47 +0000 (UTC) (envelope-from jrhett@netconsonance.com) Received: from mail.netconsonance.com (mail.netconsonance.com [198.207.204.4]) by mx1.freebsd.org (Postfix) with ESMTP id 48C3F8FC25; Mon, 24 Nov 2008 21:50:47 +0000 (UTC) (envelope-from jrhett@netconsonance.com) Received: from [10.66.240.106] (public-wireless.sv.svcolo.com [64.13.135.30]) (authenticated bits=0) by mail.netconsonance.com (8.14.1/8.14.1) with ESMTP id mAOLoiok088507; Mon, 24 Nov 2008 13:50:44 -0800 (PST) (envelope-from jrhett@netconsonance.com) X-Virus-Scanned: amavisd-new at netconsonance.com X-Spam-Flag: NO X-Spam-Score: -1.054 X-Spam-Level: X-Spam-Status: No, score=-1.054 tagged_above=-999 required=3.5 tests=[ALL_TRUSTED=-1.44, AWL=0.386] Message-Id: <1766C532-64AB-400F-8383-2DBE6BF51D9B@netconsonance.com> From: Jo Rhett To: freebsd-stable Stable In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Mon, 24 Nov 2008 13:50:38 -0800 References: X-Mailer: Apple Mail (2.929.2) Cc: Jeremy Chadwick Subject: Re: smartd long self-test causes drives to hang X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2008 21:50:47 -0000 On re-reading the message I realized that my message was in danger of being content-free. gmirror whole-disk mirror of seagate 300gb drives $ atacontrol list ATA channel 0: Master: ad0 ATA/ATAPI revision 7 Slave: ad1 ATA/ATAPI revision 7 $ gmirror list Geom name: gm0 State: COMPLETE Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 575427344 Providers: 1. Name: mirror/gm0 Mediasize: 300069051904 (279G) Sectorsize: 512 Mode: r5w5e6 Consumers: 1. Name: ad0 Mediasize: 300069052416 (279G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: DIRTY GenID: 0 SyncID: 1 ID: 3917165570 2. Name: ad1 Mediasize: 300069052416 (279G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: DIRTY GenID: 0 SyncID: 1 ID: 3874187635 On Nov 24, 2008, at 12:48 PM, Jo Rhett wrote: > I've spent about 3 months tracing down what was causing my personal > colo box to start getting "sluggish" right around dawn every > Saturday morning. It took so long because some mornings I simply > couldn't pull my head out of my tail enough to do proper debugging. > > The cause was *really slow* filesystem response time. No cron jobs > in that period. No specific process ran any slower than another, > although I eventually learned that ones which did no file i/o were > fine. And finally I realized that just "ls -la" was very slow (~1 > minute) even after I had killed off every disk-using process in the > system. SMTP and HTTP in particular were basically fubar. > > No data loss, just *real slow*. Nothing other than a soft reboot > ever solved the problem. Even leaving it running only minimal > processes for 24 hours didn't bring it back to normal. > > Finally I was browsing through Jeremy Chadwick's list of known ATA > problems and spotted his comments about smartd self-tests causing > problems. Sure enough, my long self test was scheduled for 5am on > Saturday mornings. Rechecking the observed slow-down periods > confirmed that the problem never became visible before 5am. > (sometimes it took up to 45 minutes before things slowed down enough > to set off monitoring alarms) > > So, long story short, if you're having weirdness in system time > response - check the smartd configuration, and try disabling the > self tests. The short self test I was running daily didn't appear > to affect anything, but the long test was just bringing the system > to just shuddering and limping at best. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > "