From owner-freebsd-fs@FreeBSD.ORG Tue Jan 22 18:19:09 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C9D847B; Tue, 22 Jan 2013 18:19:09 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-la0-f46.google.com (mail-la0-f46.google.com [209.85.215.46]) by mx1.freebsd.org (Postfix) with ESMTP id 2A8D4ABF; Tue, 22 Jan 2013 18:19:08 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id fq12so5380274lab.19 for ; Tue, 22 Jan 2013 10:19:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=2Ky5WXRbp7Q12poesWHeD5DZjJ4qtFD4L0BXepddLBU=; b=jgUIXrYUm1EXQIu3fKJz0HusWe1pdCfp31qvDVulAhl7xsTsGu5WVZVm+IsTzmUW0T Qv4v/9h+JFijYsZj9Cd4AoP4V1fW6SPaZhP8aga4O4OfglGEx7Ohz7Zfxd6XG+u5CSV+ 8wCZZNNlJ5QzTUWLkKWw0IupwgvEZkltVMRw2tqvuJ884nJBclrKbQINfFaO76OaUggR t3uIk9NfvcPLzSHb4CBzUu2ZWsbTgdmhhV7pIh6Py5wAsZTTdR5EO/SUNW0sg2L+Pz20 opIhReRWtf9FVG/lmY8N4IW73JYuNK58GT4O8HmthLeYWpxwS9C7bpx0UMJbKtdVU2ZQ AF4Q== X-Received: by 10.112.88.7 with SMTP id bc7mr9741645lbb.108.1358878747794; Tue, 22 Jan 2013 10:19:07 -0800 (PST) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id ml1sm7357808lab.15.2013.01.22.10.19.06 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 22 Jan 2013 10:19:07 -0800 (PST) Sender: Alexander Motin Message-ID: <50FED818.7070704@FreeBSD.org> Date: Tue, 22 Jan 2013 20:19:04 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Jeremy Chadwick Subject: Re: disk "flipped" - a known problem? References: <20130121221617.GA23909@icarus.home.lan> In-Reply-To: <20130121221617.GA23909@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, avg@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2013 18:19:09 -0000 On 22.01.2013 00:16, Jeremy Chadwick wrote: > (Please keep me CC'd as I am not subscribed) > > WRT this: > > http://lists.freebsd.org/pipermail/freebsd-fs/2013-January/016197.html > > I can reproduce the first problem 100% of the time on my home system > here. I can provide hardware specs if needed, but the important part is > that I'm using RELENG_9 / r245697, the controller is an ICH9R in AHCI > mode (and does not share an IRQ), hot-swap bays are in use, and I'm > using ahci.ko. > > I also want to make this clear to Andriy: I'm not saying "there's a > problem with your disk". In my case, I KNOW there's a problem with the > disk (that's the entire point to my tests! :-) ). > > In my case the disk is a WD Raptor (150GB, circa 2006) that has a very > badly-designed firmware that goes completely catatonic when encountering > certain sector-level conditions. That's not the problem though -- the > problem is with FreeBSD apparently getting confused as to the internal > state of its devices after a device falls off the bus and comes back. > Explanation: > > 1. System powered off; disk is attached; system powered on, shows up as > ada5. Can communicate with device in every way (the way I tend to test > simple I/O is to use "smartctl -a /dev/ada5"). This disk has no > filesystems or other "stuff" on it -- it's just a raw disk, so I believe > the g_wither_washer oddity does not apply in this situation. > > 2. "dd if=/dev/zero of=/dev/ada5 bs=64k" > > 3. Drive hits a bad sector which it cannot remap/deal with. Drive > firmware design flaw results in drive becoming 100% stuck trying to > re-read the sector and work out internal decisions to do remapping or > not. Drive audibly clicking during this time (not actuator arm being > reset to track 0 noise; some other mechanical issue). Due to firmware > issue, drive remains in this state indefinitely. > > 4. FreeBSD CAM reports repeated WRITE_FPDMA_QUEUED (i.e. writes using NCQ) > errors every 30 seconds (kern.cam.ada.default_timeout), for a total of 5 > times (kern.cam.da.retry_count+1). > > 5. FreeBSD spits out similar messages you see; retries exhausted, > cam_periph_alloc error, and devfs claims device removal. > > 6. Drive is still catatonic of course. Only way to reset the drive is > to power-cycle it. Drive removed from hot-swap bay, let sit for 20 > seconds, then is reinserted. > > 7. FreeBSD sees the disk reappear, shows up much like it did during #1, > except... > > 8. "smartctl -a /dev/ada5" claims no such device or unknown device type > (I forget which). "ls -l /dev/ada5" shows an entry. "camcontrol > devlist" shows the disk on the bus, yet I/O does not work. If I > remember right, re-attempting the dd command returns some error (I > forget which). > > 9. "camcontrol rescan all" stalls for quite some time when trying to > communicate with entry 5, but eventually does return (I think with some > error). camcontrol reset all" works without a hitch. "camcontrol > devlist" during this time shows the same disk on ada5 (which to me means > ATA IDENTIFY, i.e. vendor strings, etc. are reobtained somehow, meaning > I/O works at some level). > > 10. System otherwise works fine, but the only way to bring back > usability of ada5 is to reboot ("shutdown -r now"). > > To me, this looks like FreeBSD at some layer within the kernel (or some > driver (I don't know which)) is internally confused about the true state > of things. > > Alexander, do you have any ideas? > > I can enable CAM debugging (I do use options CAMDEBUG so I can toggle > this with camcontrol) as well as take notes and do a full step-by-step > diagnosis (along with relevant kernel output seen during each phase) if > that would help you. And I can test patches but not against -CURRENT > (will be a cold day in hell before I run that, sorry). Command timeout itself is not a reason for AHCI driver to drop the disk, neither it is for CAM in case of payload requests. Disk can be dropped if controller report device absence detected by SATA PHY, or by errors during device reinitialization after reset by CAM SATA XPT. What is interesting, is what exactly goes on after disk got stuck and you have removed it. In normal case controller should immediately report PHY status change, driver should run PHY reset and see that link is lost. It should trigger bus rescan for CAM, that should invalidate device. That should make dd abort with error. After dd gone, device should be destroyed and ready for reattachment. So it should be great if you start with the full verbose dmesg from the boot up to the moment when system becomes stable after disk removal. If it won't be enough, we can enable some more debugging with `camcontrol debug -IPXp BUS`, where BUS is the bus number from `camcontrol devlist`. -- Alexander Motin