Date: Wed, 07 Dec 2005 22:45:03 -0600 From: Eric Anderson <anderson@centtech.com> To: Scott Long <scottl@samsco.org> Cc: hackers@freebsd.org, Nate Lawson <nate@root.org> Subject: Re: scsi-target and the buffer cache Message-ID: <4397BA4F.8060708@centtech.com> In-Reply-To: <4397B82C.5020004@samsco.org> References: <4395BF04.50101@centtech.com> <43960F55.3010508@root.org> <43975926.1010302@centtech.com> <43975F5F.5080901@samsco.org> <439782AA.6000408@root.org> <4397B731.6010308@centtech.com> <4397B82C.5020004@samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Scott Long wrote: > Eric Anderson wrote: > >> Nate Lawson wrote: >> >>> Scott Long wrote: >>> >>>> Eric Anderson wrote: >>>> >>>>> Nate Lawson wrote: >>>>> >>>>>> Eric Anderson wrote: >>>>>> >>>>>>> I'm curious about whether a target mode device would use the >>>>>>> buffer cache or not. Here's a scenario: >>>>>>> >>>>>>> Host A: has fibre channel host adapter, in target mode, large >>>>>>> memory pool, and another fiber channel host adapter connecting >>>>>>> to fibre channel block device. >>>>>>> Host B: Fibre channel host adapter, connecting to Host A. >>>>>>> 'sees' the target mode block device created by Host A. >>>>>>> >>>>>>> Will Host A use the buffer cache to cache blocks between the >>>>>>> real block device, and the shared target mode device? >>>>>>> What about if Host A put a filesystem on the block device, >>>>>>> created a single file the size of the filesystem, and shared >>>>>>> that filesystem via a target mode device to Host B? >>>>>>> What I'm wanting is a box (FreeBSD?) that can be placed between >>>>>>> a fibre channel block device (like a RAID array), and a fibre >>>>>>> channel host using that block device, and act as a block cache >>>>>>> for that device, using the FreeBSD's memory. If it had a >>>>>>> significant amount of memory, this could be very useful. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> If you use the example scsi_target usermode >>>>>> (usr/share/examples/scsi_target), then the buffer cache will be >>>>>> used since its reads/writes are from usermode like normal. If >>>>>> you don't want that behavior, you can set O_DIRECT in the open() >>>>>> call of the backing store file. >>>>>> >>>>>> If you chose to modify the kernel side, you'd have to make sure >>>>>> your accesses were through the VOP layer and then it would be >>>>>> cached. >>>>>> >>>>>> You should check to be sure the target mode performance meets >>>>>> your expectations also. >>>>>> >>>>> >>>>> I guess I would be using the user mode tool, unless there's >>>>> another way? Your comment on performance also makes me a little >>>>> worried about that now - do you think I would see a large >>>>> performance hit? >>>>> Thanks! >>>>> Eric >>>>> >>>>> >>>> >>>> The way the target mode stack works in FreeBSD is that the kernel >>>> provides some of the basic services, but the actual target emulator >>>> is meant to live in userland. The userland program responds to >>>> events from the kernel via the select interface. This generally >>>> works pretty well. However, it does mean that control has to >>>> cross the kernel-userland boundary at least once for every event. >>>> >>>> What I'd suggest doing is prototyping your target emulator in userland >>>> and evaluating the performance there, and then moving it to the kernel >>>> if you _really_ need more performance. >>> >>> >>> >>> >>> Agree 100%. While having it in usermode means there are boundary >>> crossings that increase per-transaction latency, the actual bulk >>> data transfer is via zero-copy IO and you should be able to exceed >>> the data transfer rates of several 10K RPM drives on decent hardware. >>> >> >> >> Ok, great.. Now, will scsi_target work ok with raw devices, or only >> files? (although I'm not sure theres all that much difference really). >> >> Thanks!! >> Eric >> >> > > You can write your userland code to use whatever files or devices you > want. Are you talking about the scs_target.c code in > /usr/share/examples? That's just a skeletal example that you can use > as a starting point for your own work. Alright.. I was indeed talking about the example code, but I suppose it wouldn't be too hard to make it work with raw devices. Thanks for the help! Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4397BA4F.8060708>