From owner-freebsd-stable@FreeBSD.ORG Thu Sep 10 14:51:49 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C0B11065670 for ; Thu, 10 Sep 2009 14:51:49 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 9F4B48FC0A for ; Thu, 10 Sep 2009 14:51:48 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1Mlkzx-0002aS-6X for freebsd-stable@freebsd.org; Thu, 10 Sep 2009 16:51:45 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Sep 2009 16:51:45 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 10 Sep 2009 16:51:45 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Thu, 10 Sep 2009 16:51:28 +0200 Lines: 57 Message-ID: References: <4AA90D88.4010004@eng.auth.gr> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.22 (X11/20090817) In-Reply-To: <4AA90D88.4010004@eng.auth.gr> Sender: news Subject: Re: FreeBSD 8.0-BETA4 IBM ServerRaid 8k issues X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2009 14:51:49 -0000 George Mamalakis wrote: > Hello everybody, > > Yesterday I installed FreeBSD 8.0-BETA4 on an IBM 3650, having a 3650 M1 or M2? > ServerRaid 8k adapter, and 6 sata disks on raid-6. The raid-6 volume was > "synchronizing" for a day, so this syncing process was happening while I > was installing fbsd on the server. During the installation I was I can give you only some generic information. > lock order reversal: > 1st 0xffffff807c133540 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559 > 2nd 0xffffff0003deb200 dirhash (dirhash) @ > /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 You have received this message because you are running a debug kernel; a normal kernel with debugging disabled would not have shown it. By the fact that you did received it and the system managed to recover from it it is likely that the issue is harmless. It's likely related to the following timeouts. > aac0: COMMAND 0xffffff80003e08a0 TIMEOUT AFTER 40 SECONDS > aac0: COMMAND 0xffffff80003d5070 TIMEOUT AFTER 40 SECONDS > aac0: COMMAND 0xffffff80003e0d00 TIMEOUT AFTER 40 SECONDS > aac0: COMMAND 0xffffff80003d9440 TIMEOUT AFTER 40 SECONDS > .... > > ...and kept on like that, for many many lines, with decreasing timeouts. It looks like the controller was too busy rebuilding to take any new requests. It is possible you have filled the controller's write cache and that is why the lag happened at this point. You can easily test this theory. > Once the syncing process stopped, everything came back to normal (not > that I have stress-tested the machine, to be honest...). But since it > happened once, during this specific procedure, then maybe it could also > happen when the raid controller is reconstructing its volumes; and this > would be very annoying, as far as the server's efficiency (and/or maybe > stability) is concerned. Yes, you are right. But if the controller is the issue here, there is not much you can do about it. If it has a "priority" setting between normal usage and rebuilding/resyncing you might alter it to favour normal usage. Initial rebuild of RAID 5/6 is also a bit specific as it touches all drives and, in some instances (don't know specifically about ServeRAID), it means all drives are rewritten in their entirety. If you determine rebuild/resync is problematic, you might consider using a RAID mode that doesn't require it to be so extensive, like RAID 10 with 4+ drives, or software RAID with ZFS.