From nobody Sun Jul 9 21:00:22 2023 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qzffb1QWDz4mZtn for ; Sun, 9 Jul 2023 21:00:23 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QzffZ6VS7z3KdM for ; Sun, 9 Jul 2023 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688936422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=G4dyEVEBs8QVeV/Z1wrzd75KFGpGdR2dBHWFLKDLJqM=; b=AWUTG0q0KBo5QYoSgyEDXXqgQmpZE5pa7TJG4hh+j3wPiVwQqZMkTuXptfaElRZxOmIJTn RTN3nfdpYZGP3RS+X8zvPqwVT3ksV4nXH5l1H+SP8gsLt0KighQ6zD/WM95EumB1NI5toX Bb3ABpAJVXhS3bhVG/gKzvzcAK1bGQ/XlW7H9Mgltt4rqrZAR0rLnyBGHHCb1jRKA8ruHC LJPArY8VVv6xuFeJZOUpYh7r6VnSRmOtMgMiZRy+2SJ+m0L/RKxL7+Lq/xs+6KdESdPJqO EatRjSYmDcNUodt0AjmkbXZtVE5j8KBTSRWED2bnwz8rsTvT0Jr4EqOHxsaV+A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688936422; a=rsa-sha256; cv=none; b=BjCHxtajnYEVbDGZJpR7kmQRH24ru79Zg/nVOxrC5xZAnTEXQngd2ZNhaervLIeH4ocgB6 aX6DcggB+sg1hENqwphgUbFE/DS7uQgwLwg9n528Z7vsWaF26/6MZzX8/5XJ2DPtRBv5vj Jh7UANriRqeTV6j2kOmNH18O++V6T9LXLWIjdE97HUtdpMFbHiX3+r2n2xp0nhH7mq9Bmh EMgjZi52dA3jG6qFwLVGe6DogxpX8H5VkjdS7A8II85/dNHMezgJl/akEhdisL8JIp/NV/ /QJIj2ItEnhb/5K6OpXObiqMPWUbhFQDhzjLS4kCDWPgqH+T7lR0lvQJQ4IBwQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QzffZ594SzHwW for ; Sun, 9 Jul 2023 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 369L0MNB093593 for ; Sun, 9 Jul 2023 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 369L0Mfl093592 for scsi@FreeBSD.org; Sun, 9 Jul 2023 21:00:22 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202307092100.369L0Mfl093592@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: scsi@FreeBSD.org Subject: Problem reports for scsi@FreeBSD.org that need special attention Date: Sun, 9 Jul 2023 21:00:22 +0000 List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16889364227.4a97BeA.91508" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N --16889364227.4a97BeA.91508 Date: Sun, 9 Jul 2023 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 221952 | cam iosched: Fix trim statistics 1 problems total for which you should take action. --16889364227.4a97BeA.91508 Date: Sun, 9 Jul 2023 21:00:22 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    221952 | cam iosched: Fix trim statistics

1 problems total for which you should take action.
--16889364227.4a97BeA.91508-- From nobody Thu Jul 13 19:14:20 2023 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R246j2Hqrz4mgYj for ; Thu, 13 Jul 2023 19:14:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R246h32khz4PSg for ; Thu, 13 Jul 2023 19:14:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20221208.gappssmtp.com header.s=20221208 header.b=MvNMH3eA; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::136) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-4fb7b2e3dacso1959295e87.0 for ; Thu, 13 Jul 2023 12:14:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1689275671; x=1691867671; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=ipzusbKCjrDC33CNQKHma4ECs1xEzioMNlWZSaTapDw=; b=MvNMH3eAu4aKz405vxU5hjkzbhxsx4Y6oclGPZv868aXcRIxLfMwTs6q9bJNi9lU6v vPI2bs7Z/i8bTNuIVwsLdtuCVsIdzx+O9ShR3TbPZAkIeOEXCDEwmp4eyV8WIgphuMzu KHxcCFq9DxZSwxL9o4sgGPqFwED8C8WUDvpv+Sl5LQM5+I4MIhkQ/vewbMpfEUDvsrrB UCDH0iXgZzYIOyYRtBy/Gha/6duphsSkU8hTHVkW/KbgL/skQ84l+uH/3g/BsW5KJ1r7 IT603j8MSECye5cMJ7bTpzn8HaRNMZbyIT87vUJ1UzWxu9aK3ly0zK3vV1k6N3BtVyGo U5rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689275671; x=1691867671; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ipzusbKCjrDC33CNQKHma4ECs1xEzioMNlWZSaTapDw=; b=PqbWsLs34v78xVWt2R+JyNQLnChxstllFJio1DfUiGeggILhS/Ko0Lrme398kMAnBw CF34zlo9W3bsNayWA8Q/dzdyD15jv7yZ30SBTMiUe4Lqr4k566LBqY4uibtEZp+CdA74 MG1QY5KkaqQMbLx6lAWA9qdrlqhEI1s77opOGQZwkNYKXXuc+j5GOuJKy8cBdZA2dbH7 FwU6cLQRCBj6wjggM2o4KBqyRUODCWuv54FD3Nex9PJRyywKtjKxmu4zmaPeoIE5W7G4 Eois30F0A3nNC4/hhhmvTVMGp4OwGPzplCMgss/dF6KFx7QbUbKbYVJB2rmYO6YqFPpn VeEA== X-Gm-Message-State: ABy/qLYKnBMcEFE5JE02teQPRbqaNDs3dKrbFAcZE4KcY031n6UogVFV Sj/rJLBM8htv7AeGlB+OG2DU2sPCNM584JO0Qef+YBPqxOruD0mv X-Google-Smtp-Source: APBJJlEjrB8cInRRGpj4jPglrjLv0c3yprcaJsd4CHFyPYcX51/18No+kBEh+nNKC222+ypFGV30vGsDdM48lmndVs4= X-Received: by 2002:a05:6512:2313:b0:4f9:51b7:a19c with SMTP id o19-20020a056512231300b004f951b7a19cmr2271819lfu.19.1689275671126; Thu, 13 Jul 2023 12:14:31 -0700 (PDT) List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 From: Warner Losh Date: Thu, 13 Jul 2023 13:14:20 -0600 Message-ID: Subject: ASC/ASCQ Review To: scsi@freebsd.org Content-Type: multipart/alternative; boundary="0000000000004e42d20600632415" X-Spamd-Result: default: False [-2.96 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.964]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20221208.gappssmtp.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::136:from]; MLMMJ_DEST(0.00)[scsi@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_SPF_NA(0.00)[no SPF record]; ARC_NA(0.00)[]; DMARC_NA(0.00)[bsdimp.com]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20221208.gappssmtp.com:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[scsi@freebsd.org]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4R246h32khz4PSg X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --0000000000004e42d20600632415 Content-Type: text/plain; charset="UTF-8" Greetings, i've been looking closely at failed drives for $WORK lately. I've noticed that a lot of errors that kinda sound like fatal errors have SS_RDEF set on them. What's the process for evaluating whether those error codes are worth retrying. There are several errors that we seem to be seeing (preliminary read of the data) before the drive gives up the ghost altogether. For those cases, I'd like to post more specific lists. Should I do that here? Independent of that, I may want to have a more aggressive 'fail fast' policy than is appropriate for my work load (we have a lot of data that's a copy of a copy of a copy, so if we lose it, we don't care: we'll just delete any files we can't read and get on with life, though I know others will have a more conservative attitude towards data that might be precious and unique). I can set the number of retries lower, I can do some other hacks for disks that tell the disk to fail faster, but I think part of the solution is going to have to be failing for some sense-code/ASC/ASCQ tuples that we don't want to fail in upstream or the general case. I was thinking of identifying those and creating a 'global quirk table' that gets applied after the drive-specific quirk table that would let $WORK override the defaults, while letting others keep the current behavior. IMHO, it would be better to have these separate rather than in the global data for tracking upstream... Is that clear, or should I give concrete examples? Comments? Warner --0000000000004e42d20600632415 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Greetings,

i've been looking closel= y at failed drives for $WORK lately. I've noticed that a lot of errors = that kinda sound like fatal errors have SS_RDEF set on them.

=
What's the process for evaluating=C2=A0whether those error c= odes are worth retrying.=C2=A0There are several errors that we seem to be s= eeing (preliminary read of the data) before the drive gives up the ghost al= together. For those cases, I'd like to post more specific lists. Should= I do that here?

Independent of that, I may want t= o have a more aggressive=C2=A0'fail fast' policy than is appropriat= e for my work load (we have a lot of data that's a copy of a copy of a = copy, so if we lose it, we don't care: we'll just delete any files = we can't read and get on with life, though I know others will have a mo= re conservative attitude towards data that might be precious and unique). I= can set the number of retries lower, I can do some other hacks for disks t= hat tell the disk to fail faster, but I think part of the solution is going= to have to be failing for some sense-code/ASC/ASCQ tuples that we don'= t want to fail in upstream or the general case. I was thinking of identifyi= ng those and creating a 'global quirk table' that gets applied afte= r the drive-specific quirk table that would let $WORK override the defaults= , while letting others keep the current behavior. IMHO, it would be better = to have these separate rather than in the global data for tracking upstream= ...

Is that clear, or should I give concrete examp= les?

Comments?

Warner
--0000000000004e42d20600632415-- From nobody Fri Jul 14 17:12:34 2023 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2dMh1x9Gz2tkFQ for ; Fri, 14 Jul 2023 17:12:48 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f53.google.com (mail-vs1-f53.google.com [209.85.217.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2dMh00mVz40JJ for ; Fri, 14 Jul 2023 17:12:48 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vs1-f53.google.com with SMTP id ada2fe7eead31-44357f34e2dso817271137.3 for ; Fri, 14 Jul 2023 10:12:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689354767; x=1691946767; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=skIivI5EqhX6uxQ32Jju7CkIGdoD8umVxH/Slh8Y228=; b=Uj1/HtRCv7TMdCr878+caWAJ2SPboOkJG3aDYwiU/YSlkjs+FSh+m6JuOQnRfIfXmQ qSlb6Es4g+v6EWQFTfd9lKQ1KUbvukbrRkRQ01WbTBaTRAQznQiqTS1Ft56a4Mt2jsaJ KcTY2itFw/r3Wkr5xF1PIy5nJ51NC+IcUZuoO4ZPf2YXhGumG6rAxNsJCR6l1cjzIzM5 Jd1lz6odf28OH53Q0rVSsAsZiwglO9NGLHMDzaic/MsYeKxp9rruxSwEKy+Hpi1bvb++ u/Otq1u26y/DJN01TBYvZaHgSEMTSyL+7Fqge6pImgPouHOx4ecOCX6eQ1VYNyOvWcBt gvvw== X-Gm-Message-State: ABy/qLbdevlkXyQYJCO6mT/KZgiYOjhLHJmeJMWqF8YDYgUZ9oZiV/Qk g5YpubpWSIPoLJ9RGAA4iRsBEWtdcCA92i1gGGrgEMFV X-Google-Smtp-Source: APBJJlFHiXsrZUCCe4G5Kgigr3fQ2hWOF5zizL8mAXetUIg1WCRYT5w6HayLQgtFPRhxFbqGCjcupSZykr0z0wBrGt0= X-Received: by 2002:a67:f5ce:0:b0:443:7bbc:e397 with SMTP id t14-20020a67f5ce000000b004437bbce397mr3939647vso.26.1689354767004; Fri, 14 Jul 2023 10:12:47 -0700 (PDT) List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Fri, 14 Jul 2023 10:12:34 -0700 Message-ID: Subject: Re: ASC/ASCQ Review To: Warner Losh Cc: scsi@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4R2dMh00mVz40JJ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Jul 13, 2023 at 12:14=E2=80=AFPM Warner Losh wrote= : > > Greetings, > > i've been looking closely at failed drives for $WORK lately. I've noticed= that a lot of errors that kinda sound like fatal errors have SS_RDEF set o= n them. > > What's the process for evaluating whether those error codes are worth ret= rying. There are several errors that we seem to be seeing (preliminary read= of the data) before the drive gives up the ghost altogether. For those cas= es, I'd like to post more specific lists. Should I do that here? > > Independent of that, I may want to have a more aggressive 'fail fast' pol= icy than is appropriate for my work load (we have a lot of data that's a co= py of a copy of a copy, so if we lose it, we don't care: we'll just delete = any files we can't read and get on with life, though I know others will hav= e a more conservative attitude towards data that might be precious and uniq= ue). I can set the number of retries lower, I can do some other hacks for d= isks that tell the disk to fail faster, but I think part of the solution is= going to have to be failing for some sense-code/ASC/ASCQ tuples that we do= n't want to fail in upstream or the general case. I was thinking of identif= ying those and creating a 'global quirk table' that gets applied after the = drive-specific quirk table that would let $WORK override the defaults, whil= e letting others keep the current behavior. IMHO, it would be better to hav= e these separate rather than in the global data for tracking upstream... > > Is that clear, or should I give concrete examples? > > Comments? > > Warner Basically, you want to change the retry counts for certain ASC/ASCQ codes only, on a site-by-site basis? That sounds reasonable. Would it be configurable at runtime or only at build time? Also, I've been thinking lately that it would be real nice if READ UNRECOVERABLE could be translated to EINTEGRITY instead of EIO. That would let consumers know that retries are pointless, but that the data is probably healable. From nobody Fri Jul 14 18:05:38 2023 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2fXx5Wcxz4mZyv for ; Fri, 14 Jul 2023 18:05:53 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2fXw6sSvz49v8 for ; Fri, 14 Jul 2023 18:05:52 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-4fb960b7c9dso3794168e87.0 for ; Fri, 14 Jul 2023 11:05:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1689357950; x=1691949950; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WrAJ8V53r2lXNjuUSwDNUzfBirrAnaeDrAzrEjiwPNc=; b=CSjDU6W9jFv2Je1rJbUTZ57TlapXzbGdww1/7QQNc0KeRQV4BZJzeNaPTjPbhvJilX hIKFyWKJyTl9siBL9tgNC6d0N/P6MS25Nc35eYrOIsIv04RKzQEFPlPzHkyHn5fU73ZV V6mVIHso/FkYOtHobThFhj5oV/HaqHshLnmdTqHYCzVnjlGeIfC/8x7VGYFXWbJLW8z1 noRaRB3H7cjzy1TMCd9xY6roDraYeVKE14JU/GGuZgbLFcjePEWniYBp1Apatac5IZI3 j6vRK7VS4pFyommow/vpJyvEaTM+JF5oTvKXxgcRX+xRc3wyBIzEfC5t3yVC1p1U8rcG NK/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689357950; x=1691949950; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WrAJ8V53r2lXNjuUSwDNUzfBirrAnaeDrAzrEjiwPNc=; b=dQzowmEyxSQ6pYq9nB4vkybkLP8Rfhnv8ATeRvRNpBJulNcLLhDiqD7Okyf+V//4ws sjQL1ZVYUTm7hB+ZNyXNI468tkSFPg3Eh8YzMlqWiEibMIl6ZfZU4kRf6y4MYHt3UuIk 53odZq+Dg8AeQ96Zqg9Y9b/6ej0QYHBPKscoZ0w1p6B6FfY/DFUmbvZXLbT9jFF+LLrp ztYnMaHTjNnU9wNISxPQ0jBxXBaeXqX0RznwAFjQQSgiQb5uEtHgxykVjhb2CAPFaA1G sc3GCjvSmDEtyiKsiIzJ7zmoVd0lhYFrnGc+xyGRy5MBMLWmdon592ISvZrXvORkoenm S8xg== X-Gm-Message-State: ABy/qLYQFmmA5ldaIqXszxZxfhRi8tspNPsnzgHUly/16Jbdbu1c6Ltw If2nNU91p1Jjup/8QhE1zgLoYZkEPIQVmQZSNYktXw== X-Google-Smtp-Source: APBJJlEo7TJUBEZ8v8Q7wVtPn1mflp+VhhVlcu0YC0YATRGAvrOzOg3dcGaaS2DySYgwelM52zko3TxsTy4MAoN213U= X-Received: by 2002:a05:6512:e9c:b0:4f9:740e:ca38 with SMTP id bi28-20020a0565120e9c00b004f9740eca38mr4472510lfb.53.1689357949503; Fri, 14 Jul 2023 11:05:49 -0700 (PDT) List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 14 Jul 2023 12:05:38 -0600 Message-ID: Subject: Re: ASC/ASCQ Review To: Alan Somers Cc: scsi@freebsd.org Content-Type: multipart/alternative; boundary="0000000000007aa3d30600764c4a" X-Rspamd-Queue-Id: 4R2fXw6sSvz49v8 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000007aa3d30600764c4a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Jul 14, 2023, 11:12 AM Alan Somers wrote: > On Thu, Jul 13, 2023 at 12:14=E2=80=AFPM Warner Losh wro= te: > > > > Greetings, > > > > i've been looking closely at failed drives for $WORK lately. I've > noticed that a lot of errors that kinda sound like fatal errors have > SS_RDEF set on them. > > > > What's the process for evaluating whether those error codes are worth > retrying. There are several errors that we seem to be seeing (preliminary > read of the data) before the drive gives up the ghost altogether. For tho= se > cases, I'd like to post more specific lists. Should I do that here? > > > > Independent of that, I may want to have a more aggressive 'fail fast' > policy than is appropriate for my work load (we have a lot of data that's= a > copy of a copy of a copy, so if we lose it, we don't care: we'll just > delete any files we can't read and get on with life, though I know others > will have a more conservative attitude towards data that might be preciou= s > and unique). I can set the number of retries lower, I can do some other > hacks for disks that tell the disk to fail faster, but I think part of th= e > solution is going to have to be failing for some sense-code/ASC/ASCQ tupl= es > that we don't want to fail in upstream or the general case. I was thinkin= g > of identifying those and creating a 'global quirk table' that gets applie= d > after the drive-specific quirk table that would let $WORK override the > defaults, while letting others keep the current behavior. IMHO, it would = be > better to have these separate rather than in the global data for tracking > upstream... > > > > Is that clear, or should I give concrete examples? > > > > Comments? > > > > Warner > > Basically, you want to change the retry counts for certain ASC/ASCQ > codes only, on a site-by-site basis? That sounds reasonable. Would > it be configurable at runtime or only at build time? > I'd like to change the default actions. But maybe we just do that for everyone and assume modern drives... Also, I've been thinking lately that it would be real nice if READ > UNRECOVERABLE could be translated to EINTEGRITY instead of EIO. That > would let consumers know that retries are pointless, but that the data > is probably healable. > Unlikely, unless you've tuned things to not try for long at recovery... But regardless... do you have a concrete example of a use case? There's a number of places that map any error to EIO. And I'd like a use case before we expand the errors the lower layers return... Warner > --0000000000007aa3d30600764c4a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Jul 14, 2023, 11:12 AM Alan Somers <asomers@freebsd.org> wrote:
On Thu, Jul 13, 2023 at 12:14=E2=80=AFPM Wa= rner Losh <imp@bsdimp.com> wrote:
>
> Greetings,
>
> i've been looking closely at failed drives for $WORK lately. I'= ;ve noticed that a lot of errors that kinda sound like fatal errors have SS= _RDEF set on them.
>
> What's the process for evaluating whether those error codes are wo= rth retrying. There are several errors that we seem to be seeing (prelimina= ry read of the data) before the drive gives up the ghost altogether. For th= ose cases, I'd like to post more specific lists. Should I do that here?=
>
> Independent of that, I may want to have a more aggressive 'fail fa= st' policy than is appropriate for my work load (we have a lot of data = that's a copy of a copy of a copy, so if we lose it, we don't care:= we'll just delete any files we can't read and get on with life, th= ough I know others will have a more conservative attitude towards data that= might be precious and unique). I can set the number of retries lower, I ca= n do some other hacks for disks that tell the disk to fail faster, but I th= ink part of the solution is going to have to be failing for some sense-code= /ASC/ASCQ tuples that we don't want to fail in upstream or the general = case. I was thinking of identifying those and creating a 'global quirk = table' that gets applied after the drive-specific quirk table that woul= d let $WORK override the defaults, while letting others keep the current be= havior. IMHO, it would be better to have these separate rather than in the = global data for tracking upstream...
>
> Is that clear, or should I give concrete examples?
>
> Comments?
>
> Warner

Basically, you want to change the retry counts for certain ASC/ASCQ
codes only, on a site-by-site basis?=C2=A0 That sounds reasonable.=C2=A0 Wo= uld
it be configurable at runtime or only at build time?
=

I'd like to change = the default actions. But maybe we just do that for everyone and assume mode= rn drives...

Also, I've been thinking lately that it would be real nice if READ
UNRECOVERABLE could be translated to EINTEGRITY instead of EIO.=C2=A0 That<= br> would let consumers know that retries are pointless, but that the data
is probably healable.

Unlikely, unless you've tuned things to not try fo= r long at recovery...=C2=A0

But regardless... do you have a concrete example of a use case? There&#= 39;s a number of places that map any error to EIO. And I'd like a use c= ase before we expand the errors the lower layers return...

Warner
--0000000000007aa3d30600764c4a-- From nobody Fri Jul 14 18:30:51 2023 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2g614YYHz4mlFf for ; Fri, 14 Jul 2023 18:31:05 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vk1-f182.google.com (mail-vk1-f182.google.com [209.85.221.182]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2g611r2Wz4HZ4 for ; Fri, 14 Jul 2023 18:31:05 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vk1-f182.google.com with SMTP id 71dfb90a1353d-4813422f311so882915e0c.2 for ; Fri, 14 Jul 2023 11:31:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689359464; x=1691951464; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=09uR28IFKkwaSwgVuP9u782VjNZ4Uh/UDLHu/cmNkBU=; b=aJIRPFVarx0tYiXeQgayBQ3EpWBe8XVO9elgxEMKhOBUGClMshKDzecZV2Eaw89Tvo mmjdtJn70L3rKyZ7XIibHpUGanDpthXEXywVVngaHrdKwShRF1KNVve4bsrVkoJz/Oqe 45p9UN5qNC8QVtJorC2QGqhkE5IzlV1MIDJrrLAtdcqo2Qpl2GK1CWInBOx+gu3tVBic FxbGuiokE7S9rjO1nUXS//mT0HSJzG4iGTcyMmNzHjjtIFED7MJwNjH8Q8n13EUHD8tQ 1ENPDZkGljveqtd8oXt4dYUlLTMG4puHk2AeCB480VL++ZTvrzRSFLXUz8xucdnYSCjO 0png== X-Gm-Message-State: ABy/qLYFM73iCeGYkZGoMG1lbNgHkw0LNtPx9ODvSQyzgTez40iVRsJ+ ucnkf/wIIisfp63M9quj+3jmd2iAApgC14Tdaqvo/A71 X-Google-Smtp-Source: APBJJlFO0YYWI9KDT8L2R0WroC0xTCqrHw8xHLyPIznwqopOW6EFcI/fVS+HbDBmO81qV739A7xUAT0W6Z9qPLvtp8E= X-Received: by 2002:a1f:4315:0:b0:481:2ea9:979e with SMTP id q21-20020a1f4315000000b004812ea9979emr3859827vka.5.1689359463628; Fri, 14 Jul 2023 11:31:03 -0700 (PDT) List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Fri, 14 Jul 2023 11:30:51 -0700 Message-ID: Subject: Re: ASC/ASCQ Review To: Warner Losh Cc: scsi@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4R2g611r2Wz4HZ4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Fri, Jul 14, 2023 at 11:05=E2=80=AFAM Warner Losh wrote= : > > > > On Fri, Jul 14, 2023, 11:12 AM Alan Somers wrote: >> >> On Thu, Jul 13, 2023 at 12:14=E2=80=AFPM Warner Losh wr= ote: >> > >> > Greetings, >> > >> > i've been looking closely at failed drives for $WORK lately. I've noti= ced that a lot of errors that kinda sound like fatal errors have SS_RDEF se= t on them. >> > >> > What's the process for evaluating whether those error codes are worth = retrying. There are several errors that we seem to be seeing (preliminary r= ead of the data) before the drive gives up the ghost altogether. For those = cases, I'd like to post more specific lists. Should I do that here? >> > >> > Independent of that, I may want to have a more aggressive 'fail fast' = policy than is appropriate for my work load (we have a lot of data that's a= copy of a copy of a copy, so if we lose it, we don't care: we'll just dele= te any files we can't read and get on with life, though I know others will = have a more conservative attitude towards data that might be precious and u= nique). I can set the number of retries lower, I can do some other hacks fo= r disks that tell the disk to fail faster, but I think part of the solution= is going to have to be failing for some sense-code/ASC/ASCQ tuples that we= don't want to fail in upstream or the general case. I was thinking of iden= tifying those and creating a 'global quirk table' that gets applied after t= he drive-specific quirk table that would let $WORK override the defaults, w= hile letting others keep the current behavior. IMHO, it would be better to = have these separate rather than in the global data for tracking upstream... >> > >> > Is that clear, or should I give concrete examples? >> > >> > Comments? >> > >> > Warner >> >> Basically, you want to change the retry counts for certain ASC/ASCQ >> codes only, on a site-by-site basis? That sounds reasonable. Would >> it be configurable at runtime or only at build time? > > > I'd like to change the default actions. But maybe we just do that for eve= ryone and assume modern drives... > >> Also, I've been thinking lately that it would be real nice if READ >> UNRECOVERABLE could be translated to EINTEGRITY instead of EIO. That >> would let consumers know that retries are pointless, but that the data >> is probably healable. > > > Unlikely, unless you've tuned things to not try for long at recovery... > > But regardless... do you have a concrete example of a use case? There's a= number of places that map any error to EIO. And I'd like a use case before= we expand the errors the lower layers return... > > Warner My first use-case is a user-space FUSE file system. It only has access to errnos, not ASC/ASCQ codes. If we do as I suggest, then it could heal a READ UNRECOVERABLE by rewriting the sector, whereas other EIO errors aren't likely to be healed that way. My second use-case is ZFS. zfsd treats checksum errors differently from I/O errors. A checksum error normally means that a read returned wrong data. But I think that READ UNRECOVERABLE should also count. After all, that means that the disk's media returned wrong data which was detected by the disk's own EDC/ECC. I've noticed that zfsd seems to fault disks too eagerly when their only problem is READ UNRECOVERABLE errors. Mapping it to EINTEGRITY, or even a new error code, would let zfsd be tuned better. From nobody Fri Jul 14 23:34:36 2023 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R2nrT3nT5z4mvBd for ; Fri, 14 Jul 2023 23:34:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R2nrT2PK5z49mP for ; Fri, 14 Jul 2023 23:34:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52c.google.com with SMTP id 4fb4d7f45d1cf-5216f44d881so663777a12.1 for ; Fri, 14 Jul 2023 16:34:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1689377688; x=1691969688; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=en0mBBm5nahTH7miK6RfzWLn7vNznv7fWamf7QMwvdw=; b=uozwTbUxbZ2tm/MLV1A/ZuVn1BaqEmKTZuhzXlmF7zWosB7damyqwPe+n8GWYFh6p4 4uNTLwxAZB+tyNgP4D4trc+KqxA+JiBRlx8el8Zku4BJ7fo+fEec61FHU6p26z/wH5jL PqA1QNyvilq9zz1CUWKbSFmdb5tFEVn4Df9qYccBuMa9hW809XF/Cr/PkhTw2FSb1c+d 5AxvARgm9OOuEV4qf9F6ZB/fKRUcQV3uQbpibCtaALV++c5EugVAXrHxuzGOZ25ubI44 XTFLhdI6Ngjb67H9t0G5gpVeiSDMCdQuLgKfAt1rnZ3aK4IjDehs1MbVBP9dcA7GQ7kv fqxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689377688; x=1691969688; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=en0mBBm5nahTH7miK6RfzWLn7vNznv7fWamf7QMwvdw=; b=A8peRJNFiEQMy6BR2ipnv2gyvtKviHF5bhsjRhwCJeThpw0hdNj8bWqX5N7trBMwxC H6EAa0JPDuOQND5/lu9etf3GEud5I/2Pk4GhH1SB+3oKpi8/PLaLs8Otih16ml7CP/h3 h60v8MagM6ZN6cFB4TJW7xIEoF1/0y2hqaPlUp3cf1v1gfuLGSacuCcmdMbc9XQmp5Wi HHNKKfJzEoolhEB912xYSYvMI2bTlXMTUhS4uO9z2J2siBzry7CfPuUve52cS51E1lJg R8TJSwVuZFSRl8SI3tjc+hmKoK4xudxFEEQ5R99eVisRi6Vq9hc658Qm4wdbt9EarmE7 WB+Q== X-Gm-Message-State: ABy/qLaCbo+t00TyrIlbhXoY4qP9pWvH4lbnGEY4+/jEcsg8nvnyjSOu lVFUfbuP31HU1B2SebnvQR3aU9209DpHxQJdWV2O4zBXHZqJwbo5W5o= X-Google-Smtp-Source: APBJJlHMtNg2Zg/u+purZ+Ej0NsefUfpvLzzZdAgZfqP6baFMlSFwI3OkTYs+aa2/IeZmVelL+ZsKmAHXrUYhg1yDAI= X-Received: by 2002:a50:ed94:0:b0:51d:95f2:ee76 with SMTP id h20-20020a50ed94000000b0051d95f2ee76mr5144790edr.27.1689377687784; Fri, 14 Jul 2023 16:34:47 -0700 (PDT) List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Fri, 14 Jul 2023 17:34:36 -0600 Message-ID: Subject: Re: ASC/ASCQ Review To: Alan Somers Cc: scsi@freebsd.org Content-Type: multipart/alternative; boundary="000000000000f8e4d106007ae405" X-Rspamd-Queue-Id: 4R2nrT2PK5z49mP X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000f8e4d106007ae405 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Jul 14, 2023 at 12:31=E2=80=AFPM Alan Somers = wrote: > On Fri, Jul 14, 2023 at 11:05=E2=80=AFAM Warner Losh wro= te: > > > > > > > > On Fri, Jul 14, 2023, 11:12 AM Alan Somers wrote: > >> > >> On Thu, Jul 13, 2023 at 12:14=E2=80=AFPM Warner Losh = wrote: > >> > > >> > Greetings, > >> > > >> > i've been looking closely at failed drives for $WORK lately. I've > noticed that a lot of errors that kinda sound like fatal errors have > SS_RDEF set on them. > >> > > >> > What's the process for evaluating whether those error codes are wort= h > retrying. There are several errors that we seem to be seeing (preliminary > read of the data) before the drive gives up the ghost altogether. For tho= se > cases, I'd like to post more specific lists. Should I do that here? > >> > > >> > Independent of that, I may want to have a more aggressive 'fail fast= ' > policy than is appropriate for my work load (we have a lot of data that's= a > copy of a copy of a copy, so if we lose it, we don't care: we'll just > delete any files we can't read and get on with life, though I know others > will have a more conservative attitude towards data that might be preciou= s > and unique). I can set the number of retries lower, I can do some other > hacks for disks that tell the disk to fail faster, but I think part of th= e > solution is going to have to be failing for some sense-code/ASC/ASCQ tupl= es > that we don't want to fail in upstream or the general case. I was thinkin= g > of identifying those and creating a 'global quirk table' that gets applie= d > after the drive-specific quirk table that would let $WORK override the > defaults, while letting others keep the current behavior. IMHO, it would = be > better to have these separate rather than in the global data for tracking > upstream... > >> > > >> > Is that clear, or should I give concrete examples? > >> > > >> > Comments? > >> > > >> > Warner > >> > >> Basically, you want to change the retry counts for certain ASC/ASCQ > >> codes only, on a site-by-site basis? That sounds reasonable. Would > >> it be configurable at runtime or only at build time? > > > > > > I'd like to change the default actions. But maybe we just do that for > everyone and assume modern drives... > > > >> Also, I've been thinking lately that it would be real nice if READ > >> UNRECOVERABLE could be translated to EINTEGRITY instead of EIO. That > >> would let consumers know that retries are pointless, but that the data > >> is probably healable. > > > > > > Unlikely, unless you've tuned things to not try for long at recovery... > > > > But regardless... do you have a concrete example of a use case? There's > a number of places that map any error to EIO. And I'd like a use case > before we expand the errors the lower layers return... > > > > Warner > > My first use-case is a user-space FUSE file system. It only has > access to errnos, not ASC/ASCQ codes. If we do as I suggest, then it > could heal a READ UNRECOVERABLE by rewriting the sector, whereas other > EIO errors aren't likely to be healed that way. > Yea... but READ UNRECOVERABLE is kinda hit or miss... > My second use-case is ZFS. zfsd treats checksum errors differently > from I/O errors. A checksum error normally means that a read returned > wrong data. But I think that READ UNRECOVERABLE should also count. > After all, that means that the disk's media returned wrong data which > was detected by the disk's own EDC/ECC. I've noticed that zfsd seems > to fault disks too eagerly when their only problem is READ > UNRECOVERABLE errors. Mapping it to EINTEGRITY, or even a new error > code, would let zfsd be tuned better. > EINTEGRITY would then mean two different things. UFS returns in when checksums fail for critical filesystem errors. I'm not saying no, per se, just that it conflates two different errors. I think both of these use cases would be better served by CAM's publishing of the errors to devctl today. Here's some example data from a system I'm looking at: system=3DCAM subsystem=3Dperiph type=3Dtimeout device=3Dda36 serial=3D"1234= 5" cam_status=3D"0x44b" timeout=3D30000 CDB=3D"28 00 4e b7 cb a3 00 04 cc 00 " timestamp=3D1634739729.312068 system=3DCAM subsystem=3Dperiph type=3Dtimeout device=3Dda36 serial=3D"1234= 5" cam_status=3D"0x44b" timeout=3D30000 CDB=3D"28 00 20 6b d5 56 00 00 c0 00 " timestamp=3D1634739729.585541 system=3DCAM subsystem=3Dperiph type=3Derror device=3Dda36 serial=3D"12345" cam_status=3D"0x4cc" scsi_status=3D2 scsi_sense=3D"72 03 11 00" CDB=3D"28 0= 0 ad 1a 35 96 00 00 56 00 " timestamp=3D1641979267.469064 system=3DCAM subsystem=3Dperiph type=3Derror device=3Dda36 serial=3D"12345" cam_status=3D"0x4cc" scsi_status=3D2 scsi_sense=3D"72 03 11 00" CDB=3D"28 0= 0 ad 1a 35 96 00 01 5e 00 " timestamp=3D1642252539.693699 system=3DCAM subsystem=3Dperiph type=3Derror device=3Dda39 serial=3D"12346" cam_status=3D"0x4cc" scsi_status=3D2 scsi_sense=3D"72 04 02 00" CDB=3D"2a 0= 0 01 2b c8 f6 00 07 81 00 " timestamp=3D1669603144.090835 Here we get the sense key, the asc and the ascq in the scsi_sense data (I'm currently looking at expanding this to the entire sense buffer, since it includes how hard the drive tried to read the data on media and hardware errors). It doesn't include nvme data, but does include ata data (I'll have to add that data, now that I've noticed it is missing). With the sense data and the CDB you know what kind of error you got, plus what block didn't read/write correctly. With the extended sense data, you can find out even more details that are sense-key dependent... So I'm unsure that trying to shoehorn our imperfect knowledge of what's retriable, fixable, should be written with zeros into the kernel and converting that to a separate errno would give good results, and tapping into this stream daemons that want to make more nuanced calls about disks might be the better way to go. One of the things I'm planning for $WORK is to enable the retry time limit of one of the mode pages so that we fail faster and can just delete the file with the 'bad' block that we'd get eventually if we allowed the full, default error processing to run, but that 'slow path' processing kills performance for all other users of the drive... I'm unsure how well that will work out (and I know I'm lucky that I can always recover any data for my application since it's just a cache). I'd be interested to hear what others have to say here thought, since my focus on this data is through the lense of my rather specialized application... Warner P.S. That was generated with this rule if you wanted to play with it... You'd have to translate absolute disk blocks to a partition and an offset into the filesystem, then give the filesystem a chance to tell you what of its data/metadata that block is used for... # Disk errors notify 10 { match "system" "CAM"; match "subsystem" "periph"; match "device" "[an]?da[0-9]+"; action "logger -t diskerr -p daemon.info $_ timestamp=3D$timestamp"= ; }; --000000000000f8e4d106007ae405 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Jul 14, 2023 at 12:31=E2=80= =AFPM Alan Somers <asomers@freebs= d.org> wrote:
On Fri, Jul 14, 2023 at 11:05=E2=80=AFAM Warner Losh <imp@bsdimp.com> wrote:
>
>
>
> On Fri, Jul 14, 2023, 11:12 AM Alan Somers <asomers@freebsd.org> wrote:
>>
>> On Thu, Jul 13, 2023 at 12:14=E2=80=AFPM Warner Losh <imp@bsdimp.com> wrote:<= br> >> >
>> > Greetings,
>> >
>> > i've been looking closely at failed drives for $WORK late= ly. I've noticed that a lot of errors that kinda sound like fatal error= s have SS_RDEF set on them.
>> >
>> > What's the process for evaluating whether those error cod= es are worth retrying. There are several errors that we seem to be seeing (= preliminary read of the data) before the drive gives up the ghost altogethe= r. For those cases, I'd like to post more specific lists. Should I do t= hat here?
>> >
>> > Independent of that, I may want to have a more aggressive = 9;fail fast' policy than is appropriate for my work load (we have a lot= of data that's a copy of a copy of a copy, so if we lose it, we don= 9;t care: we'll just delete any files we can't read and get on with= life, though I know others will have a more conservative attitude towards = data that might be precious and unique). I can set the number of retries lo= wer, I can do some other hacks for disks that tell the disk to fail faster,= but I think part of the solution is going to have to be failing for some s= ense-code/ASC/ASCQ tuples that we don't want to fail in upstream or the= general case. I was thinking of identifying those and creating a 'glob= al quirk table' that gets applied after the drive-specific quirk table = that would let $WORK override the defaults, while letting others keep the c= urrent behavior. IMHO, it would be better to have these separate rather tha= n in the global data for tracking upstream...
>> >
>> > Is that clear, or should I give concrete examples?
>> >
>> > Comments?
>> >
>> > Warner
>>
>> Basically, you want to change the retry counts for certain ASC/ASC= Q
>> codes only, on a site-by-site basis?=C2=A0 That sounds reasonable.= =C2=A0 Would
>> it be configurable at runtime or only at build time?
>
>
> I'd like to change the default actions. But maybe we just do that = for everyone and assume modern drives...
>
>> Also, I've been thinking lately that it would be real nice if = READ
>> UNRECOVERABLE could be translated to EINTEGRITY instead of EIO.=C2= =A0 That
>> would let consumers know that retries are pointless, but that the = data
>> is probably healable.
>
>
> Unlikely, unless you've tuned things to not try for long at recove= ry...
>
> But regardless... do you have a concrete example of a use case? There&= #39;s a number of places that map any error to EIO. And I'd like a use = case before we expand the errors the lower layers return...
>
> Warner

My first use-case is a user-space FUSE file system.=C2=A0 It only has
access to errnos, not ASC/ASCQ codes.=C2=A0 If we do as I suggest, then it<= br> could heal a READ UNRECOVERABLE by rewriting the sector, whereas other
EIO errors aren't likely to be healed that way.
Yea... but READ UNRECOVERABLE is kinda hit or miss...
=C2=A0
My second use-case is ZFS.=C2=A0 zfsd treats checksum errors differently from I/O errors.=C2=A0 A checksum error normally means that a read returned=
wrong data.=C2=A0 But I think that READ UNRECOVERABLE should also count. After all, that means that the disk's media returned wrong data which was detected by the disk's own EDC/ECC.=C2=A0 I've noticed that zfs= d seems
to fault disks too eagerly when their only problem is READ
UNRECOVERABLE errors.=C2=A0 Mapping it to EINTEGRITY, or even a new error code, would let zfsd be tuned better.

E= INTEGRITY would then mean two different things. UFS returns in when checksu= ms fail for critical=C2=A0filesystem errors. I'm not saying no, per se,= just that it conflates two different errors.

I th= ink both of these use cases would be better served by CAM's publishing = of the errors to devctl today. Here's some example data from a system I= 'm looking at:

system=3DCAM subsystem=3Dperiph= type=3Dtimeout device=3Dda36 serial=3D"12345" cam_status=3D"= ;0x44b" timeout=3D30000 CDB=3D"28 00 4e b7 cb a3 00 04 cc 00 &quo= t; =C2=A0timestamp=3D1634739729.312068
system=3DCAM subsystem=3Dperiph t= ype=3Dtimeout device=3Dda36 serial=3D"12345" cam_status=3D"0= x44b" timeout=3D30000 CDB=3D"28 00 20 6b d5 56 00 00 c0 00 "= =C2=A0timestamp=3D1634739729.585541
system=3DCAM subsystem=3Dperiph typ= e=3Derror device=3Dda36 serial=3D"12345" cam_status=3D"0x4cc= " scsi_status=3D2 scsi_sense=3D"72 03 11 00" CDB=3D"28 = 00 ad 1a 35 96 00 00 56 00 " timestamp=3D1641979267.469064
system= =3DCAM subsystem=3Dperiph type=3Derror device=3Dda36 serial=3D"12345&q= uot; cam_status=3D"0x4cc" scsi_status=3D2 scsi_sense=3D"72 0= 3 11 00" CDB=3D"28 00 ad 1a 35 96 00 01 5e 00 " =C2=A0timest= amp=3D1642252539.693699
system=3DCAM subsystem=3Dperiph type= =3Derror device=3Dda39 serial=3D"12346" cam_status=3D"0x4cc&= quot; scsi_status=3D2 scsi_sense=3D"72 04 02 00" CDB=3D"2a 0= 0 01 2b c8 f6 00 07 81 00 " =C2=A0timestamp=3D1669603144.090835

Here we get the sense key, the asc and the ascq in t= he scsi_sense data (I'm currently looking at expanding this to the enti= re sense buffer, since it includes how hard the drive tried to read the dat= a on media and hardware errors).=C2=A0 It doesn't include nvme data, bu= t does include ata data (I'll have to add that data, now that I've = noticed it is missing).=C2=A0 With the sense data and the CDB you know what= kind of error you got, plus what block didn't read/write correctly. Wi= th the extended sense data, you can find out even more details that are sen= se-key dependent...

So I'm unsure that trying = to shoehorn our imperfect knowledge of what's retriable, fixable, shoul= d be written with zeros into the kernel and converting that to a separate e= rrno would give good results, and tapping into this stream daemons that wan= t to make more nuanced calls about disks might be the better way to go. One= of the things I'm planning for $WORK is to enable the retry time limit= of one of the mode pages so that we fail faster and can just delete the fi= le with the 'bad' block that we'd get eventually if we allowed = the full, default error processing to run, but that 'slow path' pro= cessing kills performance for all other users of the drive...=C2=A0 I'm= unsure how well that will work out (and I know I'm lucky that I can al= ways recover any data for my application since it's just a cache).

I'd be interested to hear what others have to say = here thought, since my focus on this data is through the lense of my rather= specialized application...

Warner

<= /div>
P.S. That was generated with this rule if you wanted to play with= it... You'd have to translate absolute disk blocks to a partition and = an offset into the filesystem, then give the filesystem a chance to tell yo= u what of its data/metadata that block is used for...

<= div># Disk errors
notify 10 {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 match "= ;system" =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"CAM";
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 match "subsystem" =C2=A0 =C2=A0 =C2=A0 "= ;periph";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 match "device" =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"[an]?da[0-9]+";
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 action "logger -t diskerr -p daemon.info $_ timestamp=3D$timestamp";
};
--000000000000f8e4d106007ae405--