Date: Wed, 19 Sep 2012 16:44:54 +0300 From: Alexander Motin <mav@FreeBSD.org> To: jack sparrow <jacks.1785@gmail.com> Cc: freebsd-drivers@freebsd.org Subject: Re: Exclusive access of SCSI/ATA devices from user space Message-ID: <5059CC56.8040705@FreeBSD.org> In-Reply-To: <201209190825.07384.jhb@freebsd.org> References: <CACmXQA09W56rSvVrprD8cuAbZ3T7hFmFKmfREiyXGOZwfpH5=g@mail.gmail.com> <201209190825.07384.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 19.09.2012 15:25, John Baldwin wrote: > On Wednesday, September 19, 2012 8:08:41 am jack sparrow wrote: >> Hi all, >> >> I'm developing a userland/usermode utility for FreeBSD. The utility >> description follows. >> >> The utility have options to select the hard drives and tape drives >> attached to the system. >> Once the user selects a tape drive or hard drive, he/she can read, >> write, reset, standby(in case of ATA), idle(in case of ATA), start, >> stop, issue identify device(in case of ATA), and inquiry in case of >> SCSI device. >> >> Now the point is, I need to make sure that this device must not be >> mounted at all, so that user can't directly read/write to the device. >> >> Another constraint is that there must not be sorting/scheduling of >> disk/drive I/O requests, not even at the kernel level. >> e.g. If user read 10 LBAs beginning from LBA 5000 say, and the LBA >> 1000, and then LBA 6000, then the i/o requests must be passed in this >> same order to the disk/drive - ie there must not be reordering of disk >> i/o requests, and the kernel must not cache/buffer the read/write data >> . >> >> Another constraint is that only the utility's process/thread must be >> able to access that drive, and no other userland process. >> >> >> I do not need filesystem access, just raw sector/byte read/write, for >> the selected drive. >> >> >> Till now after reading docs and books related to FreeBSD, it seems >> that I need to write a kernel mode driver(in form of kernel module) >> for the purpose, that will communicate with userland utility via a >> custom protocol. This driver will attach itself to the drive that is >> selected by the user from utility. This driver need not concerned for >> other devices except the selected one, as kernel has already >> drivers/modules for this. >> >> This driver will issue appropriate commands to the device selected, >> and read/write the LBAs directly from/to userland utility buffer. >> >> It seems I may need to write a custom system call too. Am I right? >> >> The point is, I need direct control of the device selected. The other >> devices can be read/write normally as they would be. >> For e.g. say the utility reads LBA 5000 from the selected device, and >> somehow device failed to respond after a timeout, then the drive will >> issue reset command to the device. >> The LBA read will be passed to userland process, and no disk i/o >> scheduling or sorting must be done on the i/o requests made by >> userland process. >> >> In case of ATA disk drives, I also need to control that whether the >> data transfer will be via PIO or DMA transfers. >> >> Can someone enlighten me how could I begin with? I don't know at which >> I/O layer such a driver should sit - just above CAM peripheral layer, >> or at CAM peripheral layer or CAM transport layer, or, at just above >> raw device layer, or at raw device layer itself, or something else. >> >> Also I wanna know is there other way that goes w/o writing such a >> driver, so that only userland code will do the work effectively. >> >> I'm targetting FreeBSD 9.0-RELEASE and onwards. > > My first take would be to use a custom GEOM that claims exclusive access to > the disk (that prevents mounting, etc.). However, that still sits above > the driver layer, and some of the things you want to do really depend on > the controller's driver (e.g. I/O sorting is typically done in the controller > driver via bioq_disksort(), as are decisions like PIO vs DMA). > > I don't think you need a custom system call btw. The easiest thing to do > is to export a new file in /dev/ for each disk and userland applications can > just use read/write against that directly. This should be easy to do with > a GEOM module. > > It may be that you can even get the desired semantics you want if you hack > on each controller driver to accept custom GEOM control requests (that would > come from your module) to do things like disable any sorting and toggle > PIO vs DMA. You might have to hack on the CAM da/ada drivers to pass those > requests down to the underlying sim as well as the controller drivers. I'm > not sure if we have any similar control requests like that in place already > that you could use as a reference. Jack. You've started your mail speaking about SCSI/ATA layer access, but finished closer to block device layer. Each device can be accessed both ways, depending on your needs. SCSI/ATA access you may get through the CAM pass driver. It allows you to run any SCSI or ATA commands, it does not modify them and does not reorder them during normal operation. But CAM does not restrict simultaneous access to the devices, so the same device can be accessed through both pass and da/ada drivers (or any other if they happen) at the same time. Usually it is not a problem because pass interface is used only by short list of tools, like ones for CD/DVD recording. Block access you may get through da/ada CAM drivers and respectfully through GEOM. GEOM does controls concurrent device access. If you open some device from user-level for writing, nothing else will be able to access it at the same time on a block layer (raw access through the pass driver is still possible). But before device can be opened, GEOM will do its usual job, tasting device for some metadata, partitions, file systems, etc. I don't know whether it is a problem, but the only way to prevent that is hack CAM transport or da/ada drivers to not attach to specific devices at all, leaving only pass interface. Also, as John told, there is a request sorter on a top side of ada and da drivers. So if several request sent simultaneously, they may be reordered. There is special BIO request flag BIO_ORDERED to force original request order, but I don't know way to send it from user-level. If you need absolute guarantee of order, you should wait for request completion. Neither CAM nor GEOM are caching requests, so that will help with ordering, but limit performance. CAM provides control over ATA PIO/DMA modes via camcontrol tool or respective calls via the pass interface. It works for all supported ATA/SATA controllers. For SAS controllers it can be problematic, as these controllers implement some translation and may not provide such interfaces. -- Alexander Motin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5059CC56.8040705>