From owner-freebsd-hackers@FreeBSD.ORG Sat Jan 13 17:42:56 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 993FD16A412; Sat, 13 Jan 2007 17:42:56 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 200CC13C459; Sat, 13 Jan 2007 17:42:56 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id l0DHgSel038730; Sat, 13 Jan 2007 10:42:33 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <45A91A02.906@samsco.org> Date: Sat, 13 Jan 2007 10:42:26 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.2pre) Gecko/20061227 SeaMonkey/1.1 MIME-Version: 1.0 To: Danny Braniss References: <20070112195549.GA77181@freebie.xs4all.nl> <45A7F6A4.4030707@samsco.org> In-Reply-To: X-Enigmail-Version: 0.94.1.2 X-Enigmail-Version: 0.94.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (pooker.samsco.org [168.103.85.57]); Sat, 13 Jan 2007 10:42:33 -0700 (MST) X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org X-Mailman-Approved-At: Mon, 15 Jan 2007 00:52:57 +0000 Cc: Wilko Bulte , Pawel Jakub Dawidek , freebsd-hackers@freebsd.org, freebsd-scsi@freebsd.org Subject: Re: iSCSI disconnects dilema X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2007 17:42:56 -0000 Danny Braniss wrote: >> I forget, does iSCSI have a concept of a flush_cache command, or the >> equivalent of what parallel SCSI does with ordered tags? > > not realy - or I can't find it. iSCSI is mainly and envelope for > scsi commands, so whatever the CAM does, it will pass it on. > There are some managemenet commands, so the target can tell the initiator > that it's going down for example (and what should the driver > do in such a case in freebsd?) > If the periph is open (i.e. mounted), I'd just ignore this and have the stack go through a normal retry timeout cycle to see if the device comes back. If it's closed, then I'd remove the periph. Knowing if it's opened or closed is likely hard to do from the iSCSI driver, which is one reason why iSCSI knowledge needs to eventually be moved upwards in CAM. >> If so, then >> that's how your app or OS knows that the transaction got committed to >> stable storage. It's been long assumed in the external storage world >> that you are at the mercy of the external storage cache, so the problem >> that Danny is referring to is nothing new. The real question is how >> to implement the equivalent mechanism that iSCSI provides in a way that >> the OS/app can make use of it. For example, CAM issues an ordered tag >> periodically to flush the disk cache to stable storage. > nice, (or wishful thinking :-), the scsi part of iSCSI is/can be > software/virtual. > If the target device returns a successful completion from a command, the initiator must assume that it's not lying. You could do a flush/sync cache command after every I/O, but then you'd have a completely unacceptable level of performance. But again, this is not a new problem specific to iSCSI. It's long been a design consideration of external storage, and is why external storage 1) carries a high price tag to accompany good engineering and testing, and 2) comes with some form of battery backup, to prevent data loss in case of power loss. >> Most storage >> drivers, including CAM, will issue some sort of a flush_cache command to >> the controller and media during system shutdown. > > this took me a long time to fix! the userland program got killed at shutdown, > the link was lost, and so there was no way to flush buffers, fixed by calling > fget(...) too. > > I guess I can summarize: (and use the 3 monkey law :-) > 1- assume the target is 'well behaved' and will flush cache. > 2- there is - currently - no way to tell the OS that not all > seems to be as expected. > 3- keep quiet and hope for the best. > danny > > So you had a scenario where a program was doing I/O right up to system (initiator) shutdown, and some of those I/O's got lost in the process? I guess I don't understand why the OS didn't flush all outstanding I/O buffers after terminating the program and before finishing the shutdown. Maybe you are doing something illegal in your driver, or maybe you need to implement a kernel shutdown hook that will allow you to block the shutdown until everything is flushed. Scott