From owner-freebsd-fs@freebsd.org Thu Dec 14 09:41:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BE2CE9E30F; Thu, 14 Dec 2017 09:41:01 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C9D937CC46; Thu, 14 Dec 2017 09:41:00 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f51.google.com with SMTP id f20so5882672lfe.3; Thu, 14 Dec 2017 01:41:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=nNTnuQswS1AYZc2U/wyOpdL6RvGXoegKvnU8CBYS+MI=; b=Hgjxq/feua8HLMHSadq6aN9sHLgNhaCUtBYGZIYsqk2xrTCbVBGB5ngR2kfdFowKMo SSshsj3Kab2nNU4j/ogv1dLNpFeYdPdMX/KW1tQOXFRLtoWNnRagJMjyhx7KSaif959c rPhV7pQ/dL+Qd/l9KasWZeMnttJUZVD26KXzFhoM4KkO7gQdeEUKZ7i60cSBsK4V49VV QzYPxsFXURjqsyi8jwQJlptpQ+1GLLCrJbCJfOOo/Mj7HqaNSONha6PPFmnA5FfpYrbO hqRiyzewEEgTMRsYVIgZuzqgzvFC5mLNRnwHI8JNEYhG1xHqCBxAW5+hxW06NtaEaVgc ZlsA== X-Gm-Message-State: AKGB3mLSOrrBMkT0zcQMtHGn7xEhupWTDzFd3derrgmb9L8ryhJZGcmq a/P8P8gU0wWij1kQwYw+6N4UOUeM X-Google-Smtp-Source: ACJfBosUjL00d1OG34TV0tBHXHmAxQMGU7GzUz2h1q7g0H3NhwA2SOI0hvH2bzAYqRNQgtiSaUIJvQ== X-Received: by 10.46.55.20 with SMTP id e20mr3482902lja.118.1513244452499; Thu, 14 Dec 2017 01:40:52 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id h142sm711545lfh.37.2017.12.14.01.40.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Dec 2017 01:40:51 -0800 (PST) From: Andriy Gapon Subject: Re: add BIO_NORETRY flag, implement support in ata_da, use in ZFS vdev_geom To: Scott Long Cc: Warner Losh , FreeBSD FS , freebsd-geom@freebsd.org References: <391f2cc7-0036-06ec-b6c9-e56681114eeb@FreeBSD.org> <64f37301-a3d8-5ac4-a25f-4f6e4254ffe9@FreeBSD.org> <39E8D9C4-6BF3-4844-85AD-3568A6D16E64@samsco.org> <33101e6c-0c74-34b7-ee92-f9c4a11685d5@FreeBSD.org> <783CA790-C0D3-442D-A5A2-4CB424FCBD62@samsco.org> Message-ID: <59654c50-b24e-ac87-2154-681cc57627de@FreeBSD.org> Date: Thu, 14 Dec 2017 11:40:49 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <783CA790-C0D3-442D-A5A2-4CB424FCBD62@samsco.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 09:41:01 -0000 This email is rather large, with a lot of contexts. Replying just to a single piece here. On 27/11/2017 17:29, Scott Long wrote: > > >> On Nov 25, 2017, at 10:36 AM, Andriy Gapon wrote: >> Let's assume that I am talking about the case of not being able to read an HDD >> sector that is gone bad. >> Here is a real world example: >> >> Jun 16 10:40:18 trant kernel: ahcich0: NCQ error, slot = 20, port = -1 >> Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 >> 00 00 58 62 40 2c 00 00 08 00 00 >> Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): CAM status: ATA Status Error >> Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), >> error: 40 (UNC ) >> Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): RES: 41 40 68 58 62 40 2c 00 >> 00 00 00 >> Jun 16 10:40:18 trant kernel: (ada0:ahcich0:0:0:0): Retrying command >> Jun 16 10:40:20 trant kernel: ahcich0: NCQ error, slot = 22, port = -1 > ... >> I do not see anything wrong in what CAM / ahci /ata_da did here. >> They did what I would expect them to do. They tried very hard to get data that >> I told them I need. > > Two things I see here. The first is that the drive is trying for 2 seconds to get good > data off of the media. The second is that it’s failing and reporting the error as > uncorrectable. I think that retries at the OS/driver > layer are therefore useless; the drive is already doing a bunch of its own retries and > is failing, and is telling you that it’s failing. In the past, the conventional wisdom has > been to do retries, because 30 years ago drives had minimal firmware and didn’t do > a good job at managing data integrity. Now they do an extensive amount of > analysis-driven error correction and retries, so I think it’s time to change the > conventional wisdom. I’d propose that for direct-attach SSDs and HDDs we treat this > error as non-retriable. Normally this would be a one-line change, but I think it needs > to be nuanced to distinguish between optical drives, simple flash media drives, and > regular HDDs and SSDs. > > An interim solution would be to just set the kern.cam.ada.retry_count to 0. I went through some ada errors that I have in my logs and I think that there can be a difference between HDDs and SSDs too. I thought that the HDD internal retry mechanism would be thorough enough, but, to my surprise, I see that majority of read failures are recovered by the first retry. Sometimes it's the second retry that's successful, in all other cases the standard four retries do not help. So, it may be too early to set ada.retry_count to 0 for all types of supported disks. -- Andriy Gapon