Date: Mon, 24 Dec 2012 21:42:42 -0800 From: olivier <olivier777a7@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: freebsd-fs@freebsd.org, ken@freebsd.org, freebsd-stable@freebsd.org Subject: CAM hangs in 9-STABLE? [Was: NFS/ZFS hangs after upgrading from 9.0-RELEASE to -STABLE] Message-ID: <CALC5%2B1MbmG8xyKmr6LVYvrFOOWrv-v=BR6JkyG4jW_d3-Js7GA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Dear All
It turns out that reverting to an older version of the mps driver did not
fix the ZFS hangs I've been struggling with in 9.1 and 9-STABLE after all
(they just took a bit longer to occur again, possibly just by chance). I
followed steps along lines suggested by Andriy to collect more information
when the problem occurs. Hopefully this will help figure out what's going
on.
As far as I can tell, what happens is that at some point IO operations to a
bunch of drives that belong to different pools get stuck. For these drives,
gstat shows no activity but 1 pending operation, as such:
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d
%busy Name
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da1
I've been running gstat in a loop (every 100s) to monitor the machine. Just
before the hang occurs, everything seems fine (see full gstat output
below). Right after the hang occurs a number of drives seem stuck (see full
gstat output below). Notably, some stuck drives are seen through the mps
driver and others through the mpt driver. So the problem doesn't seem to be
driver-specific. I have had the problem occur (at a lower frequency) on
similar machines that don't use the mpt driver (and only have 1 disk
provided through mps), so the problem doesn't seem to be caused by the mpt
driver (and is likely not caused by defective hardware). Since based on the
information I provided earlier Andriy thinks the problem might not
originate in ZFS, perhaps that means that the problem is in the CAM layer?
camcontrol tags -v (as suggested by Andriy) in the hung state shows for
example
(pass56:mpt1:0:8:20): dev_openings 254
(pass56:mpt1:0:8:20): dev_active 1
(pass56:mpt1:0:8:20): devq_openings 254
(pass56:mpt1:0:8:20): devq_queued 0
(pass56:mpt1:0:8:20): held 0
(pass56:mpt1:0:8:20): mintags 2
(pass56:mpt1:0:8:20): maxtags 255
(I'm not providing full camcontrol tags output below because I couldn't get
it to run during the specific hang I documented most thoroughly; the
example above is from a different occurrence of the hang).
The buses don't seem completely frozen: if I manually remove drives while
the machine is hanging, that's picked up by the mpt driver, which prints
out corresponding messages to the console. But camcontrol reset all or
rescan all don't seem to do anything.
I've tried reducing vfs.zfs.vdev.min_pending and vfs.zfs.vdev.max_pending
to 1, to no avail.
Any suggestions to resolve this problem, work around it, or further
investigate it would be greatly appreciated!
Thanks a lot
Olivier
Detailed information:
Output of procstat -a -kk when the machine is hanging is available at
http://pastebin.com/7D2KtT35 (not putting it here because it's pretty long)
dmesg is available at http://pastebin.com/9zJQwWJG . Note that I'm using
LUN masking, so the "illegal requests" reported aren't really errors. Maybe
one day if I get my problems sorted out I'll use geom multipathing instead.
My kernel config is
include GENERIC
ident MYKERNEL
options IPSEC
device crypto
options OFED # Infiniband protocol
device mlx4ib # ConnectX Infiniband support
device mlxen # ConnectX Ethernet support
device mthca # Infinihost cards
device ipoib # IP over IB devices
options ATA_CAM # Handle legacy controllers with CAM
options ATA_STATIC_ID # Static device numbering
options KDB
options DDB
Full output of gstat just before the hang (at most 100s before the hang):
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d
%busy Name
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da0
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da2/da2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da0/da0
1 85 48 79 4.7 35 84 0.5 0 0 0.0
24.3 da1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da1/da1
1 83 47 77 4.3 34 79 0.5 0 0 0.0
22.1 da4
1 1324 1303 21433 0.6 19 42 0.7 0 0 0.0
79.8 da3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da5
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da6
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da7
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da8
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da9
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da10
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da11
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da12
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da13
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da14
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da15
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da16
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da17
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da19
0 97 57 93 3.5 38 84 0.3 0 0 0.0
21.3 da20
0 85 47 69 3.3 36 86 0.4 0 0 0.0
16.8 da21
0 1666 1641 18992 0.3 23 43 0.4 0 0 0.0
57.9 da22
0 93 55 98 3.5 36 87 0.4 0 0 0.0
20.6 da23
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da24
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da25
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da27
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da28
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da29
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da30
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da31
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da32
0 1200 0 0 0.0 1198 11751 0.6 0 0 0.0
67.3 da33
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da34
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da35
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da36
0 81 44 67 2.0 35 84 0.3 0 0 0.0
10.1 da37
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da38
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da39
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da40
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da42
1 1020 999 22028 0.8 19 42 0.7 0 0 0.0
84.8 da43
0 1050 1029 23479 0.8 19 47 0.7 0 0 0.0
83.3 da44
1 1006 984 22758 0.8 21 46 0.6 0 0 0.0
84.8 da45
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da46
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da47
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da48
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da49
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da50
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 cd0
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da4/da4
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da3/da3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da5/da5
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da6/da6
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da7/da7
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da8/da8
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da9/da9
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da10/da10
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da11/da11
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da12/da12
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da13/da13
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da14/da14
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da15/da15
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da16/da16
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da17/da17
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da18/da18
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da19/da19
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da20/da20
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da21/da21
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da22/da22
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da23/da23
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da24/da24
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da25/da25
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26/da26
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 PART/da26/da26
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26p1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26p2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26p3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da27/da27
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da28/da28
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da29/da29
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da30/da30
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da31/da31
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da32/da32
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da33/da33
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da34/da34
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da35/da35
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da36/da36
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da37/da37
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da38/da38
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da39/da39
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da40/da40
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da41/da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da42/da42
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da43/da43
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da44/da44
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da45/da45
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da46/da46
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da47/da47
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da48/da48
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da49/da49
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da50/da50
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/cd0/cd0
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26p1/da26p1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26p2/da26p2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 LABEL/da26p1/da26p1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 gptid/84d4487b-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26p3/da26p3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 LABEL/da26p2/da26p2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 gptid/b4255780-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0
DEV/gptid/84d4487b-34e3-11e2-b773-00259058949a/gptid/84d4487b-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da25
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0
DEV/gptid/b4255780-34e3-11e2-b773-00259058949a/gptid/b4255780-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da40
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da26p3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da29
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da30
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da24
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da6
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da7
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da16
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da17
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da20
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da21
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da37
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da23
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da4
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da43
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da44
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da22
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da33
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da45
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da3
Full output of gstat just after the hang (at most 100s after the hang):
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d
%busy Name
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da0
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da2/da2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da0/da0
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da1/da1
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da4
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da5
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da6
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da7
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da8
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da9
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da10
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da11
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da12
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da13
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da14
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da15
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da16
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da17
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da18
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da19
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da20
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da21
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da22
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da23
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da24
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da25
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da27
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da28
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da29
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da30
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da31
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da32
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da33
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da34
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da35
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da36
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da37
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da38
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da39
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da40
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da42
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da43
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da44
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da45
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da46
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da47
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da48
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da49
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da50
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 cd0
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da4/da4
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da3/da3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da5/da5
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da6/da6
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da7/da7
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da8/da8
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da9/da9
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da10/da10
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da11/da11
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da12/da12
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da13/da13
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da14/da14
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da15/da15
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da16/da16
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da17/da17
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da18/da18
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da19/da19
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da20/da20
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da21/da21
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da22/da22
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da23/da23
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da24/da24
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da25/da25
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26/da26
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 PART/da26/da26
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26p1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26p2
1 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 da26p3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da27/da27
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da28/da28
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da29/da29
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da30/da30
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da31/da31
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da32/da32
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da33/da33
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da34/da34
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da35/da35
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da36/da36
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da37/da37
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da38/da38
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da39/da39
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da40/da40
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da41/da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da42/da42
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da43/da43
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da44/da44
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da45/da45
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da46/da46
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da47/da47
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da48/da48
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da49/da49
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da50/da50
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/cd0/cd0
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26p1/da26p1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26p2/da26p2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 LABEL/da26p1/da26p1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 gptid/84d4487b-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 DEV/da26p3/da26p3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 LABEL/da26p2/da26p2
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 gptid/b4255780-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0
DEV/gptid/84d4487b-34e3-11e2-b773-00259058949a/gptid/84d4487b-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da25
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0
DEV/gptid/b4255780-34e3-11e2-b773-00259058949a/gptid/b4255780-34e3-11e2-b773-00259058949a
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da40
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da41
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da26p3
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da29
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da30
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da24
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da6
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da7
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da16
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da17
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da20
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da21
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da37
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da23
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da1
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da4
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da43
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da44
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da22
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da33
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da45
0 0 0 0 0.0 0 0 0.0 0 0 0.0
0.0 ZFS::VDEV/zfs::vdev/da3
On Thu, Dec 13, 2012 at 10:14 PM, olivier <olivier777a7@gmail.com> wrote:
> For what it's worth, I think I might have solved my problem by reverting
> to an older version of the mps driver. I checked out a recent version of
> 9-STABLE and reversed the changes in
> http://svnweb.freebsd.org/base?view=revision&revision=230592 (perhaps
> there was a simpler way of reverting to the older mps driver). So far so
> good, no hang even when hammering the file system.
>
> This does not conclusively prove that the new LSI mps driver is at fault,
> but that seems to be a likely explanation.
>
> Thanks to everybody who pointed me in the right direction. Hope this helps
> others who run into similar problems with 9.1
> Olivier
>
>
> On Thu, Dec 13, 2012 at 10:14 AM, olivier <olivier777a7@gmail.com> wrote:
>
>>
>>
>> On Thu, Dec 13, 2012 at 9:54 AM, Andriy Gapon <avg@freebsd.org> wrote:
>>
>>> Google for "zfs deadman". This is already committed upstream and I
>>> think that it
>>> is imported into FreeBSD, but I am not sure... Maybe it's imported just
>>> into the
>>> vendor area and is not merged yet.
>>>
>>
>> Yes, that's exactly what I had in mind. The logic for panicking makes
>> sense.
>> As far as I can tell you're correct that deadman is in the vendor area
>> but not merged. Any idea when it might make it into 9-STABLE?
>> Thanks
>> Olivier
>>
>>
>>
>>
>>> So, when enabled this logic would panic a system as a way of letting
>>> know that
>>> something is wrong. You can read in the links why panic was selected
>>> for this job.
>>>
>>> And speaking FreeBSD-centric - I think that our CAM layer would be a
>>> perfect place
>>> to detect such issues in non-ZFS-specific way.
>>>
>>> --
>>> Andriy Gapon
>>>
>>
>>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALC5%2B1MbmG8xyKmr6LVYvrFOOWrv-v=BR6JkyG4jW_d3-Js7GA>