From owner-freebsd-scsi@FreeBSD.ORG Wed Sep 10 11:52:28 2008 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64B531065679 for ; Wed, 10 Sep 2008 11:52:28 +0000 (UTC) (envelope-from kolpanen@qads.kearfott.com) Received: from mailhost.kearfott.com (mailhost.kearfott.com [192.77.142.195]) by mx1.freebsd.org (Postfix) with ESMTP id 308928FC14 for ; Wed, 10 Sep 2008 11:52:28 +0000 (UTC) (envelope-from kolpanen@qads.kearfott.com) Received: from qads.kearfott.com (qads.kearfott.com [172.18.1.1]) by mailhost.kearfott.com (8.13.1/8.13.1) with ESMTP id m8ABq02I010285; Wed, 10 Sep 2008 07:52:00 -0400 Received: from qads.kearfott.com (localhost [127.0.0.1]) by qads.kearfott.com (8.13.1/8.13.1) with ESMTP id m8ABq0P0000598; Wed, 10 Sep 2008 07:52:00 -0400 (EDT) Received: (from kolpanen@localhost) by qads.kearfott.com (8.13.1/8.13.1/Submit) id m8ABq0kL018683; Wed, 10 Sep 2008 07:52:00 -0400 (EDT) Date: Wed, 10 Sep 2008 07:51:59 -0400 From: "Dennis R. Kolpanen" To: Danny Braniss Message-ID: <20080910115159.GA28900@qads.kearfott.com> References: <20080909211532.GA16009@qads.kearfott.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i Cc: freebsd-scsi@freebsd.org Subject: Re: iscsi initiator - system hang X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Sep 2008 11:52:28 -0000 On Wed, Sep 10, 2008 at 09:32:01AM +0300, Danny Braniss wrote: > > On a FreeBSD 7.0 system, certain commands issued against any of the > > three mounted iscsi drives causes the system to "hang". In this > > context, hang means: > > > > the system continues to respond to pings > > sendmail stops accepting connections > > imapd stops accepting connections > > sshd stops accepting connections > > shell sessions already established stop accepting commands > > login at the system console is not possible > > > > The problem has been caused, so far, by dump, restore, and pax. These > > commands work perfectly if they are directed against one of the > > internal drives and not the iscsi drives. The failures noted above do > > not normally happen immediately after issuing one of these commands. > > The problems seem to build over a period of minutes or tens of > > minutes. Note that the dump/restore/pax commands can take hours to > > run. > > > > Nothing is written to the system console or any of the log files > > indicating a problem. The only way to recover from the hang is by > > means of the hardware reset button. > > > > When the system was first being set up and no other users were on it, > > shutting down all but one of the CPUs by means of sysctl and > > "machdep.hlt_cpus" allowed restoring about 150gb to the three iscsi > > drives. Once the machine was placed into production, massive hardware > > problems on an old server required this to be done immediately, this > > trick no longer works. > > > > An overview of the hardware involved: > > > > dual, quad-core Intel Xeon processors > > 16 gb RAM FreeBSD 7.0 amd64 release > > NetworkAppliance FAS2020 SAN > > generic kernel > > > > A complete dmesg output can be provided if desired. > > > > By default, iscontrol creates the iscsi drives with the number of tags > > set to one. The performance of the iscsi drives with this default > > setting was quite poor. Based on a recommendation made on a mailing > > list some time ago, /etc/iscsi.conf was changed to set the tags to > > 128. This had a dramatic improvement on the iscsi performance. > > > > Testing on the system that was rushed into production is not really > > possible. However, within the next week or so, a nearly identical > > system should become available and this one could be used for testing. > > > > Any ideas on what could be wrong? Any solutions? > > > > maybe, but first, can you send me the output of 'sysctl net.iscsi'? > > danny > Danny, No problem. Here it is: net.iscsi.driver_version: 2.0.99 net.iscsi.isid: DIB00 net.iscsi.sessions: 1 net.iscsi.0.targetname: iqn.1992-08.com.netapp:sn.135029528 net.iscsi.0.targeaddress: 10.172.172.1 net.iscsi.0.stats: recv=1288005 sent=1287999 flags=0x0000011b pdus-alloc=0 pdus-max=220 cws=64 cmd=afa32 exp=afa32 max=afa71 stat=afa32 itt=afa32 net.iscsi.0.pid: 916 Thanks. Dennis