From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 8 00:48:06 2005 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1BF5D16A41F for ; Thu, 8 Dec 2005 00:48:06 +0000 (GMT) (envelope-from nate@root.org) Received: from www.cryptography.com (li-22.members.linode.com [64.5.53.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id 73E6243D69 for ; Thu, 8 Dec 2005 00:48:05 +0000 (GMT) (envelope-from nate@root.org) Received: from [172.16.0.224] (fi01-84CBdd.tokyo.flets.isao.net [211.132.203.221]) by www.cryptography.com (8.12.8/8.12.8) with ESMTP id jB80lodu021436 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 7 Dec 2005 16:47:58 -0800 Message-ID: <439782AA.6000408@root.org> Date: Thu, 08 Dec 2005 09:47:38 +0900 From: Nate Lawson User-Agent: Mozilla Thunderbird 1.0.6 (Windows/20050716) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Scott Long References: <4395BF04.50101@centtech.com> <43960F55.3010508@root.org> <43975926.1010302@centtech.com> <43975F5F.5080901@samsco.org> In-Reply-To: <43975F5F.5080901@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 08 Dec 2005 00:58:25 +0000 Cc: hackers@freebsd.org, Eric Anderson Subject: Re: scsi-target and the buffer cache X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2005 00:48:06 -0000 Scott Long wrote: > Eric Anderson wrote: > >> Nate Lawson wrote: >> >>> Eric Anderson wrote: >>> >>>> I'm curious about whether a target mode device would use the buffer >>>> cache or not. Here's a scenario: >>>> >>>> Host A: has fibre channel host adapter, in target mode, large memory >>>> pool, and another fiber channel host adapter connecting to fibre >>>> channel block device. >>>> Host B: Fibre channel host adapter, connecting to Host A. 'sees' >>>> the target mode block device created by Host A. >>>> >>>> Will Host A use the buffer cache to cache blocks between the real >>>> block device, and the shared target mode device? >>>> What about if Host A put a filesystem on the block device, created a >>>> single file the size of the filesystem, and shared that filesystem >>>> via a target mode device to Host B? >>>> What I'm wanting is a box (FreeBSD?) that can be placed between a >>>> fibre channel block device (like a RAID array), and a fibre channel >>>> host using that block device, and act as a block cache for that >>>> device, using the FreeBSD's memory. If it had a significant amount >>>> of memory, this could be very useful. >>> >>> >>> >>> >>> If you use the example scsi_target usermode >>> (usr/share/examples/scsi_target), then the buffer cache will be used >>> since its reads/writes are from usermode like normal. If you don't >>> want that behavior, you can set O_DIRECT in the open() call of the >>> backing store file. >>> >>> If you chose to modify the kernel side, you'd have to make sure your >>> accesses were through the VOP layer and then it would be cached. >>> >>> You should check to be sure the target mode performance meets your >>> expectations also. >>> >> >> I guess I would be using the user mode tool, unless there's another >> way? Your comment on performance also makes me a little worried about >> that now - do you think I would see a large performance hit? >> Thanks! >> Eric >> >> > > The way the target mode stack works in FreeBSD is that the kernel > provides some of the basic services, but the actual target emulator > is meant to live in userland. The userland program responds to > events from the kernel via the select interface. This generally > works pretty well. However, it does mean that control has to > cross the kernel-userland boundary at least once for every event. > > What I'd suggest doing is prototyping your target emulator in userland > and evaluating the performance there, and then moving it to the kernel > if you _really_ need more performance. Agree 100%. While having it in usermode means there are boundary crossings that increase per-transaction latency, the actual bulk data transfer is via zero-copy IO and you should be able to exceed the data transfer rates of several 10K RPM drives on decent hardware. -- Nate