From nobody Wed Dec 1 20:47:01 2021 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AD06318BF8BF for ; Wed, 1 Dec 2021 20:47:19 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J4B3W4Q81z4nSD for ; Wed, 1 Dec 2021 20:47:19 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f179.google.com with SMTP id bf8so51169389oib.6 for ; Wed, 01 Dec 2021 12:47:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zNQlH2oS2bKZlKX14dqHJz1ibYd4zSgkk2GleEU+CoQ=; b=Eng971scrHs1DsAl7VlR+xI57uVvUFGt5R7oXqTMlgZEdLoKIYJJJ9nnujjoK9VUqH G0aUs0EfLthn+tqL42h+zXpGKFNxVebJrqx+EyL3wpAdsH4e4gKpOy3jJ/ei/AOgzDos d8wr4WpwJv+Sqc3LvzJ1QurvXUs77dREr1IJvum6Wtd2y8RkT4FIt/7GfkGOMyqE/+tr vsy7AMGpY1ec+09OfsMRU0WD/4HBNy1JmhIt3HNvKSlS0hBUWar6fO/Zue4W8DLABP0U WBGHgHqtRPeW8Aew0BuJ7KMb/4NNEtL1h8v2ChIdIBeynDon2j/W/ITSQ6KMClx12wqg 7EyQ== X-Gm-Message-State: AOAM533qX7/6UKiy9sUX4LzTqBh1LEsBZSyXB3m3136tiaJmy8leDdh+ x6Q1iqlP0Rx4Hw62TZHVQMyKxX8x0eoX4a4ChXHgp5CH X-Google-Smtp-Source: ABdhPJyVkHQ1cXYsh8YylRCrL8PmcgblSRWUSXf2HOLsOyHW40uhYKo6lOGT4MTkUXT/Zr4wfPtgAFSsrsj8ng+5E2E= X-Received: by 2002:a05:6808:ec3:: with SMTP id q3mr604148oiv.57.1638391633119; Wed, 01 Dec 2021 12:47:13 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Wed, 1 Dec 2021 13:47:01 -0700 Message-ID: Subject: Re: ZFS deadlocks triggered by HDD timeouts To: Warner Losh Cc: FreeBSD Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4J4B3W4Q81z4nSD X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Wed, Dec 1, 2021 at 1:37 PM Warner Losh wrote: > > > > On Wed, Dec 1, 2021 at 1:28 PM Alan Somers wrote: >> >> On Wed, Dec 1, 2021 at 11:25 AM Warner Losh wrote: >> > >> > >> > >> > On Wed, Dec 1, 2021, 11:16 AM Alan Somers wrote: >> >> >> >> On a stable/13 build from 16-Sep-2021 I see frequent ZFS deadlocks >> >> triggered by HDD timeouts. The timeouts are probably caused by >> >> genuine hardware faults, but they didn't lead to deadlocks in >> >> 12.2-RELEASE or 13.0-RELEASE. Unfortunately I don't have much >> >> additional information. ZFS's stack traces aren't very informative, >> >> and dmesg doesn't show anything besides the usual information about >> >> the disk timeout. I don't see anything obviously related in the >> >> commit history for that time range, either. >> >> >> >> Has anybody else observed this phenomenon? Or does anybody have a >> >> good way to deliberately inject timeouts? CAM makes it easy enough to >> >> inject an error, but not a timeout. If it did, then I could bisect >> >> the problem. As it is I can only reproduce it on production servers. >> > >> > >> > What SIM? Timeouts are tricky because they have many sources, some of which are nonlocal... >> > >> > Warner >> >> mpr(4) > > > Is this just a single drive that's acting up, or is the controller initialized as part of the error recovery? I'm not doing anything fancy with mprutil or sas3flash, if that's what you're asking. > If a single drive, > are there multiple timeouts that happen at the same time such that we timeout a request while we're waiting for > the abort command we send to the firmware to be acknowledged? I don't know. > Would you be able to run a kgdb script to see > if you're hitting a situation that I fixed in mpr that would cause I/O to never complete in this rather odd circumstance? > If you can, and if it is, then there's a change I can MFC :). Possibly. When would I run this kgdb script? Before ZFS locks up, after, or while the problematic timeout happens? > > Warner