From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 8 04:36:10 2005 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1622E16A41F for ; Thu, 8 Dec 2005 04:36:10 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id D216F43D90 for ; Thu, 8 Dec 2005 04:35:57 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id jB84ZjKU062536; Wed, 7 Dec 2005 21:35:45 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <4397B82C.5020004@samsco.org> Date: Wed, 07 Dec 2005 21:35:56 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050615 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Eric Anderson References: <4395BF04.50101@centtech.com> <43960F55.3010508@root.org> <43975926.1010302@centtech.com> <43975F5F.5080901@samsco.org> <439782AA.6000408@root.org> <4397B731.6010308@centtech.com> In-Reply-To: <4397B731.6010308@centtech.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on pooker.samsco.org Cc: hackers@freebsd.org, Nate Lawson Subject: Re: scsi-target and the buffer cache X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2005 04:36:10 -0000 Eric Anderson wrote: > Nate Lawson wrote: > >> Scott Long wrote: >> >>> Eric Anderson wrote: >>> >>>> Nate Lawson wrote: >>>> >>>>> Eric Anderson wrote: >>>>> >>>>>> I'm curious about whether a target mode device would use the >>>>>> buffer cache or not. Here's a scenario: >>>>>> >>>>>> Host A: has fibre channel host adapter, in target mode, large >>>>>> memory pool, and another fiber channel host adapter connecting to >>>>>> fibre channel block device. >>>>>> Host B: Fibre channel host adapter, connecting to Host A. 'sees' >>>>>> the target mode block device created by Host A. >>>>>> >>>>>> Will Host A use the buffer cache to cache blocks between the real >>>>>> block device, and the shared target mode device? >>>>>> What about if Host A put a filesystem on the block device, created >>>>>> a single file the size of the filesystem, and shared that >>>>>> filesystem via a target mode device to Host B? >>>>>> What I'm wanting is a box (FreeBSD?) that can be placed between a >>>>>> fibre channel block device (like a RAID array), and a fibre >>>>>> channel host using that block device, and act as a block cache for >>>>>> that device, using the FreeBSD's memory. If it had a significant >>>>>> amount of memory, this could be very useful. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> If you use the example scsi_target usermode >>>>> (usr/share/examples/scsi_target), then the buffer cache will be >>>>> used since its reads/writes are from usermode like normal. If you >>>>> don't want that behavior, you can set O_DIRECT in the open() call >>>>> of the backing store file. >>>>> >>>>> If you chose to modify the kernel side, you'd have to make sure >>>>> your accesses were through the VOP layer and then it would be cached. >>>>> >>>>> You should check to be sure the target mode performance meets your >>>>> expectations also. >>>>> >>>> >>>> I guess I would be using the user mode tool, unless there's another >>>> way? Your comment on performance also makes me a little worried >>>> about that now - do you think I would see a large performance hit? >>>> Thanks! >>>> Eric >>>> >>>> >>> >>> The way the target mode stack works in FreeBSD is that the kernel >>> provides some of the basic services, but the actual target emulator >>> is meant to live in userland. The userland program responds to >>> events from the kernel via the select interface. This generally >>> works pretty well. However, it does mean that control has to >>> cross the kernel-userland boundary at least once for every event. >>> >>> What I'd suggest doing is prototyping your target emulator in userland >>> and evaluating the performance there, and then moving it to the kernel >>> if you _really_ need more performance. >> >> >> >> Agree 100%. While having it in usermode means there are boundary >> crossings that increase per-transaction latency, the actual bulk data >> transfer is via zero-copy IO and you should be able to exceed the data >> transfer rates of several 10K RPM drives on decent hardware. >> > > > Ok, great.. Now, will scsi_target work ok with raw devices, or only > files? (although I'm not sure theres all that much difference really). > > Thanks!! > Eric > > You can write your userland code to use whatever files or devices you want. Are you talking about the scs_target.c code in /usr/share/examples? That's just a skeletal example that you can use as a starting point for your own work. Scott Scott