From owner-freebsd-fs@FreeBSD.ORG Thu Aug 13 15:23:10 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EDA6106567B for ; Thu, 13 Aug 2009 15:23:10 +0000 (UTC) (envelope-from ken@nargothrond.kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id 0314F8FC1F for ; Thu, 13 Aug 2009 15:23:09 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.13.6) with ESMTP id n7DEo9Fd039518; Thu, 13 Aug 2009 08:50:09 -0600 (MDT) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id n7DEo9Sa039517; Thu, 13 Aug 2009 08:50:09 -0600 (MDT) (envelope-from ken) Date: Thu, 13 Aug 2009 08:50:08 -0600 From: "Kenneth D. Merry" To: Panagiotis Christias Message-ID: <20090813145008.GA39384@nargothrond.kdm.org> References: <8E9591D8BCB72D4C8DE0884D9A2932DC35BD34C3@ITS-HCWNEM03.ds.Vanderbilt.edu> <200908101605.12332.jhb@freebsd.org> <200908101707.49526.jhb@freebsd.org> <8E9591D8BCB72D4C8DE0884D9A2932DC6D2EDF21@ITS-HCWNEM03.ds.Vanderbilt.edu> <20090812124721.GA71441@noc.ntua.gr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090812124721.GA71441@noc.ntua.gr> User-Agent: Mutt/1.4.2i X-Virus-Scanned: ClamAV 0.92.1/9689/Thu Aug 13 06:32:09 2009 on nargothrond.kdm.org X-Virus-Status: Clean Cc: "freebsd-fs@freebsd.org" , "Hearn, Trevor" Subject: Re: UFS Filesystem issues, and the loss of my hair... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2009 15:23:10 -0000 On Wed, Aug 12, 2009 at 15:47:21 +0300, Panagiotis Christias wrote: > On Mon, Aug 10, 2009 at 05:20:44PM -0500, Hearn, Trevor wrote: > > Yes, it does seem like it was part of one of the other messages. The isp(4) > > driver was just recently updated in HEAD by mjacob@ who has maintained that > > driver in the past. He may have some insight if there is an isp(4)-specific > > problem. > > > > -- > > John Baldwin > > > > Heh. Ok, I just watched the same error message scroll across the screen > > for about 5 minutes now, with a different offset, same length. The fun > > part is that it is not touching the device, /dev/da1p7 at all. From the > > systat -vmstat display, I see all of the traffic coming from the > > /dev/mfid0 drives. It ran for a while, then stopped. So, no access to > > the drive in question, da1p7, but on the root drive, mfid0. Odd. The > > partition is mapped to the root drive. I wonder if the driver lost > > itself, and it tried to access the file on the empty folder on the root > > drive. Sigh. Anyone? > > Hello, > > we faced a similar problem here (major greek university) about a year ago > [1]. Our setup consists of Dell 2950 servers, QLogic 2462 HBAs (PCI-E) > and an EMC CLARiiON CX3-40. As soon as we tried to do a simple "tar zxf > ports.tgz" on a SAN volume the system would freeze or/and panic (same error > messages as yours). Oleg Sharoiko suggested that we could decrease the > number of tag openings (tag queue depth). Decreasing it would make the > system a bit more stable but did not eliminate the problem. > > Then, I contacted Matthew Jacob and tested his latest isp code [2] along > with alternative solutions like zfs and gjournal. Matthew was kind enough > to offer his support but eventually I ran out time and patience, so I moved > a couple of servers to centos in order to put the storage into production. > That was around December last year. > > About a month ago Kenneth Merry announced that a new version of isp was > available [3] which corrected bugs and added new functionality. I thought > it was worth trying so I set up FreeBSD 7-stable in two Dell boxes, added > the isp patches, recompiled the kernel and started the stress tests. I > also looked around for more info and hints regarding qlogic hbas. The > Linux driver (ql2xxx) has a 32 max queue depth by default (see > ql2xmaxqdepth) which is also the recommended value by EMC. There are also > similar references for Solaris (see sd:sd_max_throttle). Some mention > even smaller values depending the storage. > > Currently, I am running stress tests, using fsx, ffsb, postmark, iozone, > bonnie++, blogbench and other home-made scripts (any other suggestion?) on > two 7-stable-amd64 + isp_diffs.releng7.20090629 boxes. So far, at 32 maximum > tag openings, everything looks good, I have not seen any panics and the > following fsck run cleanly. I will keep running more tests for a week or two > hoping that they will help draw a conclusion. Thanks for the report! I'm glad to hear it is working for you. The driver has gone into -current, and will be in 8.0-RELEASE. Hopefully it'll get propagated back into RELENG_7 before too long. Ken -- Kenneth Merry ken@kdm.org