From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 8 04:32:34 2005 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DEFC516A41F for ; Thu, 8 Dec 2005 04:32:33 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 85F3743D5C for ; Thu, 8 Dec 2005 04:32:33 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [192.168.42.22] (andersonbox2.centtech.com [192.168.42.22]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id jB84WFZa007079; Wed, 7 Dec 2005 22:32:15 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <4397B731.6010308@centtech.com> Date: Wed, 07 Dec 2005 22:31:45 -0600 From: Eric Anderson User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051204) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Nate Lawson References: <4395BF04.50101@centtech.com> <43960F55.3010508@root.org> <43975926.1010302@centtech.com> <43975F5F.5080901@samsco.org> <439782AA.6000408@root.org> In-Reply-To: <439782AA.6000408@root.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1205/Wed Dec 7 08:00:48 2005 on mh2.centtech.com X-Virus-Status: Clean Cc: Scott Long , hackers@freebsd.org Subject: Re: scsi-target and the buffer cache X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2005 04:32:34 -0000 Nate Lawson wrote: > Scott Long wrote: > >> Eric Anderson wrote: >> >>> Nate Lawson wrote: >>> >>>> Eric Anderson wrote: >>>> >>>>> I'm curious about whether a target mode device would use the >>>>> buffer cache or not. Here's a scenario: >>>>> >>>>> Host A: has fibre channel host adapter, in target mode, large >>>>> memory pool, and another fiber channel host adapter connecting to >>>>> fibre channel block device. >>>>> Host B: Fibre channel host adapter, connecting to Host A. 'sees' >>>>> the target mode block device created by Host A. >>>>> >>>>> Will Host A use the buffer cache to cache blocks between the real >>>>> block device, and the shared target mode device? >>>>> What about if Host A put a filesystem on the block device, created >>>>> a single file the size of the filesystem, and shared that >>>>> filesystem via a target mode device to Host B? >>>>> What I'm wanting is a box (FreeBSD?) that can be placed between a >>>>> fibre channel block device (like a RAID array), and a fibre >>>>> channel host using that block device, and act as a block cache for >>>>> that device, using the FreeBSD's memory. If it had a significant >>>>> amount of memory, this could be very useful. >>>> >>>> >>>> >>>> >>>> >>>> If you use the example scsi_target usermode >>>> (usr/share/examples/scsi_target), then the buffer cache will be >>>> used since its reads/writes are from usermode like normal. If you >>>> don't want that behavior, you can set O_DIRECT in the open() call >>>> of the backing store file. >>>> >>>> If you chose to modify the kernel side, you'd have to make sure >>>> your accesses were through the VOP layer and then it would be cached. >>>> >>>> You should check to be sure the target mode performance meets your >>>> expectations also. >>>> >>> >>> I guess I would be using the user mode tool, unless there's another >>> way? Your comment on performance also makes me a little worried >>> about that now - do you think I would see a large performance hit? >>> Thanks! >>> Eric >>> >>> >> >> The way the target mode stack works in FreeBSD is that the kernel >> provides some of the basic services, but the actual target emulator >> is meant to live in userland. The userland program responds to >> events from the kernel via the select interface. This generally >> works pretty well. However, it does mean that control has to >> cross the kernel-userland boundary at least once for every event. >> >> What I'd suggest doing is prototyping your target emulator in userland >> and evaluating the performance there, and then moving it to the kernel >> if you _really_ need more performance. > > > Agree 100%. While having it in usermode means there are boundary > crossings that increase per-transaction latency, the actual bulk data > transfer is via zero-copy IO and you should be able to exceed the data > transfer rates of several 10K RPM drives on decent hardware. > Ok, great.. Now, will scsi_target work ok with raw devices, or only files? (although I'm not sure theres all that much difference really). Thanks!! Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------