Date: Wed, 03 Aug 2011 07:14:30 -0700 From: Matthew Jacob <mj@feral.com> To: freebsd-geom@freebsd.org Subject: Re: Poor interaction between gmultipath(8), ZFS and isp(4) Message-ID: <4E3957C6.1000406@feral.com> In-Reply-To: <4E394269.3090208@darkbsd.org> References: <4E394269.3090208@darkbsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Known problem. Or rather, one of a long set of known problems. Most of these were addressed at Panasas under RELENG_7, but I have not had the time to redo them for RELENG_8 and later. Nor was I really happy with a lot of the results. At least from my perspective, due to work commitments, I'm unlikely to get to this very soon. Regrets. On 8/3/2011 5:43 AM, Stephane LAPIE wrote: > Hello list, > > (Not 100% sure the bug is in GEOM_MULTIPATH or in another driver.) > > I am running a FreeBSD 8.2-RELEASE server with ZFSv15, with the > following hardware : > > http://www.darkbsd.org/~darksoul/server_dmesg.txt > > I have a dual fibre-channel controller (isp(4) driver), and I am > accessing 16 RAID0 logical drives on a Promise vTrak E630fD (1 volume / > physical disk) > > Since both controllers are plugged to the same storage unit with no LUN > masking, both controllers end up seeing the same devices. Which is what > made me combine these devices using geom_multipath. > > Here is my zpool structure : > config: > > NAME STATE READ WRITE CKSUM > data ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > multipath/disk0 ONLINE 0 0 0 > multipath/disk1 ONLINE 0 0 0 > multipath/disk2 ONLINE 0 0 0 > multipath/disk3 ONLINE 0 0 0 > multipath/disk4 ONLINE 0 0 0 > multipath/disk5 ONLINE 0 0 0 > multipath/disk6 ONLINE 0 0 0 > multipath/disk7 ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > multipath/disk8 ONLINE 0 0 0 > multipath/disk9 ONLINE 0 0 0 > multipath/disk10 ONLINE 0 0 0 > multipath/disk11 ONLINE 0 0 0 > multipath/disk12 ONLINE 0 0 0 > multipath/disk13 ONLINE 0 0 0 > multipath/disk14 ONLINE 0 0 0 > multipath/disk15 ONLINE 0 0 0 > > errors: No known data errors > > > Using gmultipath, I eventually want to have disk{1,3,5,7,9,11,13,15} use > the second controller, while the rest uses the first. The idea was that > if anyone removed the fiber, it would switch everything over to the > remaining fiber. > > For the sake of testing, I put every multipath device on the same > controller, isp1. > > Here is the kernel log fragment I could acquire from my test (removing a > fiber on which transfers are actively running), however since I don't > have serial console access, I couldn't acquire the relevant kernel panic > trace (it simply mentions a kernel trap during a page fault in g_mp_kt > in the last readable section displayed, but I reckon it's like every CPU > raises the panic message) > > http://www.darkbsd.org/~darksoul/server_lastlog_before_kernelpanic.txt > > After that, I get the aforementioned kernel panic. I can consistently > reproduce it, and will try to acquire serial console output to get more > detailed kernel panic trace, but it feels like everything is occuring at > the same time without proper locking, or confirming relevant structures > are still allocated. This looks like a race condition between isp(4) > loopdown provoking da(4) destruction, and gmultipath(8) failover. > (Therefore having g_mp_kt accessing a da(4) structure that is being > destroyed, or already destroyed, and accessing unallocated memory) > > Maybe this is similar to this issue : > http://freebsd.1045724.n5.nabble.com/Kernel-panic-with-gmultipath-td4204700.html > > > Could this be tuned so that : > 1) initially, on isp(4) loopdown -> da(4) devices depending on it return > SCSI errors, provoking clean failover of gmultipath > 2) afterwards, on isp(4) timeout -> da(4) devices are destroyed > > Is this a case for using the following boot hints ? > - "hint.isp.0.loop_down_limit" and "hint.isp.0.gone_device_time" (though > I am not quite sure what the difference is between the two ... Which one > does the actual deallocation of underlying devices ?) > > Thanks in advance for your time,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E3957C6.1000406>