Date: Thu, 13 Aug 2009 08:50:08 -0600 From: "Kenneth D. Merry" <ken@kdm.org> To: Panagiotis Christias <p.christias@noc.ntua.gr> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, "Hearn, Trevor" <trevor.hearn@Vanderbilt.Edu> Subject: Re: UFS Filesystem issues, and the loss of my hair... Message-ID: <20090813145008.GA39384@nargothrond.kdm.org> In-Reply-To: <20090812124721.GA71441@noc.ntua.gr> References: <8E9591D8BCB72D4C8DE0884D9A2932DC35BD34C3@ITS-HCWNEM03.ds.Vanderbilt.edu> <200908101605.12332.jhb@freebsd.org> <200908101707.49526.jhb@freebsd.org> <8E9591D8BCB72D4C8DE0884D9A2932DC6D2EDF21@ITS-HCWNEM03.ds.Vanderbilt.edu> <20090812124721.GA71441@noc.ntua.gr>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Aug 12, 2009 at 15:47:21 +0300, Panagiotis Christias wrote: > On Mon, Aug 10, 2009 at 05:20:44PM -0500, Hearn, Trevor wrote: > > Yes, it does seem like it was part of one of the other messages. The isp(4) > > driver was just recently updated in HEAD by mjacob@ who has maintained that > > driver in the past. He may have some insight if there is an isp(4)-specific > > problem. > > > > -- > > John Baldwin > > > > Heh. Ok, I just watched the same error message scroll across the screen > > for about 5 minutes now, with a different offset, same length. The fun > > part is that it is not touching the device, /dev/da1p7 at all. From the > > systat -vmstat display, I see all of the traffic coming from the > > /dev/mfid0 drives. It ran for a while, then stopped. So, no access to > > the drive in question, da1p7, but on the root drive, mfid0. Odd. The > > partition is mapped to the root drive. I wonder if the driver lost > > itself, and it tried to access the file on the empty folder on the root > > drive. Sigh. Anyone? > > Hello, > > we faced a similar problem here (major greek university) about a year ago > [1]. Our setup consists of Dell 2950 servers, QLogic 2462 HBAs (PCI-E) > and an EMC CLARiiON CX3-40. As soon as we tried to do a simple "tar zxf > ports.tgz" on a SAN volume the system would freeze or/and panic (same error > messages as yours). Oleg Sharoiko suggested that we could decrease the > number of tag openings (tag queue depth). Decreasing it would make the > system a bit more stable but did not eliminate the problem. > > Then, I contacted Matthew Jacob and tested his latest isp code [2] along > with alternative solutions like zfs and gjournal. Matthew was kind enough > to offer his support but eventually I ran out time and patience, so I moved > a couple of servers to centos in order to put the storage into production. > That was around December last year. > > About a month ago Kenneth Merry announced that a new version of isp was > available [3] which corrected bugs and added new functionality. I thought > it was worth trying so I set up FreeBSD 7-stable in two Dell boxes, added > the isp patches, recompiled the kernel and started the stress tests. I > also looked around for more info and hints regarding qlogic hbas. The > Linux driver (ql2xxx) has a 32 max queue depth by default (see > ql2xmaxqdepth) which is also the recommended value by EMC. There are also > similar references for Solaris (see sd:sd_max_throttle). Some mention > even smaller values depending the storage. > > Currently, I am running stress tests, using fsx, ffsb, postmark, iozone, > bonnie++, blogbench and other home-made scripts (any other suggestion?) on > two 7-stable-amd64 + isp_diffs.releng7.20090629 boxes. So far, at 32 maximum > tag openings, everything looks good, I have not seen any panics and the > following fsck run cleanly. I will keep running more tests for a week or two > hoping that they will help draw a conclusion. Thanks for the report! I'm glad to hear it is working for you. The driver has gone into -current, and will be in 8.0-RELEASE. Hopefully it'll get propagated back into RELENG_7 before too long. Ken -- Kenneth Merry ken@kdm.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090813145008.GA39384>