Date: Wed, 1 Dec 2021 13:56:33 -0700 From: Warner Losh <imp@bsdimp.com> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD <freebsd-stable@freebsd.org> Subject: Re: ZFS deadlocks triggered by HDD timeouts Message-ID: <CANCZdfr_s_10zePSWoaVyi7ExcG9yqK=v5oDjLnVCVZ05hDJAw@mail.gmail.com> In-Reply-To: <CAOtMX2gnEgGn-h16UJHhrS79ypH357=r2R0DaYAa1J-TOGAKCQ@mail.gmail.com> References: <CAOtMX2hMu7qXqHt5rhi9CBNDRERpWshcF%2BR9N_VQOrYvYFERQg@mail.gmail.com> <CANCZdfo7W-eFoQ6X4y0rY=k5in6T7Ledjhes39ToO9ZXLXyVbw@mail.gmail.com> <CAOtMX2jmppMTwnK_g4OiWSnGu=Vwxm1FMa-_izdNPTYaJPyiDA@mail.gmail.com> <CANCZdfqfcbObUUonrEdNViJ-5xvU%2BFeYT%2BapHwmTpiHmfBVaXg@mail.gmail.com> <CAOtMX2gnEgGn-h16UJHhrS79ypH357=r2R0DaYAa1J-TOGAKCQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000005b14f905d21be9fc
Content-Type: text/plain; charset="UTF-8"
On Wed, Dec 1, 2021 at 1:47 PM Alan Somers <asomers@freebsd.org> wrote:
> On Wed, Dec 1, 2021 at 1:37 PM Warner Losh <imp@bsdimp.com> wrote:
> >
> >
> >
> > On Wed, Dec 1, 2021 at 1:28 PM Alan Somers <asomers@freebsd.org> wrote:
> >>
> >> On Wed, Dec 1, 2021 at 11:25 AM Warner Losh <imp@bsdimp.com> wrote:
> >> >
> >> >
> >> >
> >> > On Wed, Dec 1, 2021, 11:16 AM Alan Somers <asomers@freebsd.org>
> wrote:
> >> >>
> >> >> On a stable/13 build from 16-Sep-2021 I see frequent ZFS deadlocks
> >> >> triggered by HDD timeouts. The timeouts are probably caused by
> >> >> genuine hardware faults, but they didn't lead to deadlocks in
> >> >> 12.2-RELEASE or 13.0-RELEASE. Unfortunately I don't have much
> >> >> additional information. ZFS's stack traces aren't very informative,
> >> >> and dmesg doesn't show anything besides the usual information about
> >> >> the disk timeout. I don't see anything obviously related in the
> >> >> commit history for that time range, either.
> >> >>
> >> >> Has anybody else observed this phenomenon? Or does anybody have a
> >> >> good way to deliberately inject timeouts? CAM makes it easy enough
> to
> >> >> inject an error, but not a timeout. If it did, then I could bisect
> >> >> the problem. As it is I can only reproduce it on production servers.
> >> >
> >> >
> >> > What SIM? Timeouts are tricky because they have many sources, some of
> which are nonlocal...
> >> >
> >> > Warner
> >>
> >> mpr(4)
> >
> >
> > Is this just a single drive that's acting up, or is the controller
> initialized as part of the error recovery?
>
> I'm not doing anything fancy with mprutil or sas3flash, if that's what
> you're asking.
>
No. I'm asking if you've enabled debugging on the recovery messages and see
that we enter any kind of
controller reset when the timeouts occur.
> > If a single drive,
> > are there multiple timeouts that happen at the same time such that we
> timeout a request while we're waiting for
> > the abort command we send to the firmware to be acknowledged?
>
> I don't know.
>
OK.
> > Would you be able to run a kgdb script to see
> > if you're hitting a situation that I fixed in mpr that would cause I/O
> to never complete in this rather odd circumstance?
> > If you can, and if it is, then there's a change I can MFC :).
>
> Possibly. When would I run this kgdb script? Before ZFS locks up,
> after, or while the problematic timeout happens?
>
After the timeouts. I've been doing 'kgdb' followed by 'source
mpr-hang.gdb' to run this.
What you are looking for is anything with a qfrozen_cnt > 0.. The script is
imperfect and racy
with normal operations (but not in a bad way), so you may need to run it a
couple of times
to get consistent data. On my systems, there'd be one or two devices with a
frozen count > 1
and no I/O happened on those drives and processes hung. That might not be
any different than
a deadlock :)
Warner
P.S. here's the mpr-hang.gdb script. Not sure if I can make an attachment
survive the mailing lists :)
define cam_path
set $path=(struct cam_path *)$arg0
printf " Periph: %p\n", $path->periph
printf " Bus: %p\n", $path->bus
printf " Target: %p\n", $path->target
printf " Device: %p\n", $path->device
end
define periph
set $periph = (struct cam_periph *)$arg0
printf "%s%d:\n", $periph->periph_name, $periph->unit_number
printf "softc: %p\n", $periph->softc
printf "sim: %p\n", $periph->sim
printf "flags: 0x%x\n", $periph->flags
cam_path $periph->path
printf "priority: sched %d immed %d\n", $periph->scheduled_priority,
$periph->immediate_priority
printf "allocated %d allocating %d\n", $periph->periph_allocated,
$periph->periph_allocating
printf "refcount: %d\n", $periph->refcount
printf "qfrozen_cnt: %d\n",
$periph->path->device->ccbq.queue.qfrozen_cnt
end
define periphunits
set $count = 0
set $driver = $arg0
set $periph = $driver.units.tqh_first
while ($periph != 0)
if $periph->periph_allocated != 0 || $periph->periph_allocating !=
0 || $periph->path->device->ccbq.queue.qfrozen_cnt != 0
periph $periph
set $count = $count + 1
end
set $periph = $periph->unit_links.tqe_next
end
if ($count == 0)
printf "No problems found for periph %s\n", $driver->driver_name
end
end
define periphs
set $i = 0
while (periph_drivers[$i] != 0)
set $p = periph_drivers[$i]
periphunits $p
set $i = $i + 1
end
end
periphs
Warner
--0000000000005b14f905d21be9fc--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfr_s_10zePSWoaVyi7ExcG9yqK=v5oDjLnVCVZ05hDJAw>