Date: Sat, 13 Jan 2007 10:42:26 -0700 From: Scott Long <scottl@samsco.org> To: Danny Braniss <danny@cs.huji.ac.il> Cc: Wilko Bulte <wb@freebie.xs4all.nl>, Pawel Jakub Dawidek <pjd@freebsd.org>, freebsd-hackers@freebsd.org, freebsd-scsi@freebsd.org Subject: Re: iSCSI disconnects dilema Message-ID: <45A91A02.906@samsco.org> In-Reply-To: <E1H5ftb-000Okd-FO@cs1.cs.huji.ac.il> References: <E1H5S7E-000BS0-RR@cs1.cs.huji.ac.il> <20070112195549.GA77181@freebie.xs4all.nl> <45A7F6A4.4030707@samsco.org> <E1H5ftb-000Okd-FO@cs1.cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
Danny Braniss wrote: >> I forget, does iSCSI have a concept of a flush_cache command, or the >> equivalent of what parallel SCSI does with ordered tags? > > not realy - or I can't find it. iSCSI is mainly and envelope for > scsi commands, so whatever the CAM does, it will pass it on. > There are some managemenet commands, so the target can tell the initiator > that it's going down for example (and what should the driver > do in such a case in freebsd?) > If the periph is open (i.e. mounted), I'd just ignore this and have the stack go through a normal retry timeout cycle to see if the device comes back. If it's closed, then I'd remove the periph. Knowing if it's opened or closed is likely hard to do from the iSCSI driver, which is one reason why iSCSI knowledge needs to eventually be moved upwards in CAM. >> If so, then >> that's how your app or OS knows that the transaction got committed to >> stable storage. It's been long assumed in the external storage world >> that you are at the mercy of the external storage cache, so the problem >> that Danny is referring to is nothing new. The real question is how >> to implement the equivalent mechanism that iSCSI provides in a way that >> the OS/app can make use of it. For example, CAM issues an ordered tag >> periodically to flush the disk cache to stable storage. > nice, (or wishful thinking :-), the scsi part of iSCSI is/can be > software/virtual. > If the target device returns a successful completion from a command, the initiator must assume that it's not lying. You could do a flush/sync cache command after every I/O, but then you'd have a completely unacceptable level of performance. But again, this is not a new problem specific to iSCSI. It's long been a design consideration of external storage, and is why external storage 1) carries a high price tag to accompany good engineering and testing, and 2) comes with some form of battery backup, to prevent data loss in case of power loss. >> Most storage >> drivers, including CAM, will issue some sort of a flush_cache command to >> the controller and media during system shutdown. > > this took me a long time to fix! the userland program got killed at shutdown, > the link was lost, and so there was no way to flush buffers, fixed by calling > fget(...) too. > > I guess I can summarize: (and use the 3 monkey law :-) > 1- assume the target is 'well behaved' and will flush cache. > 2- there is - currently - no way to tell the OS that not all > seems to be as expected. > 3- keep quiet and hope for the best. > danny > > So you had a scenario where a program was doing I/O right up to system (initiator) shutdown, and some of those I/O's got lost in the process? I guess I don't understand why the OS didn't flush all outstanding I/O buffers after terminating the program and before finishing the shutdown. Maybe you are doing something illegal in your driver, or maybe you need to implement a kernel shutdown hook that will allow you to block the shutdown until everything is flushed. Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45A91A02.906>