Date: Thu, 20 Sep 2012 17:48:34 +0530 From: Jack <jacks.1785@gmail.com> To: freebsd-drivers@freebsd.org Subject: Re: Exclusive access of SCSI/ATA devices from user space Message-ID: <CACmXQA3Ek5Qk94L=D8zGWjeLOaTzaM04MTzU6KRtF1ANPwUuvg@mail.gmail.com> In-Reply-To: <CACmXQA3sq4ydM7iEN4pKBOhRyJkufyB4BQntAZ6G2apNvv-2GQ@mail.gmail.com> References: <CACmXQA09W56rSvVrprD8cuAbZ3T7hFmFKmfREiyXGOZwfpH5=g@mail.gmail.com> <201209190825.07384.jhb@freebsd.org> <5059CC56.8040705@FreeBSD.org> <CACmXQA3Rg3QAjLHQOd1_mY=8nXEonTuGnUCpZ=WutwpLNRBLmg@mail.gmail.com> <CACmXQA3sq4ydM7iEN4pKBOhRyJkufyB4BQntAZ6G2apNvv-2GQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 9/20/12, Jack <jacks.1785@gmail.com> wrote: > On Wed, Sep 19, 2012 at 8:53 PM, Jack <jacks.1785@gmail.com> wrote: >> On Wed, Sep 19, 2012 at 7:14 PM, Alexander Motin <mav@freebsd.org> wrote: >>> On 19.09.2012 15:25, John Baldwin wrote: >>>> >>>> On Wednesday, September 19, 2012 8:08:41 am jack sparrow wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I'm developing a userland/usermode utility for FreeBSD. The utility >>>>> description follows. >>>>> >>>>> The utility have options to select the hard drives and tape drives >>>>> attached to the system. >>>>> Once the user selects a tape drive or hard drive, he/she can read, >>>>> write, reset, standby(in case of ATA), idle(in case of ATA), start, >>>>> stop, issue identify device(in case of ATA), and inquiry in case of >>>>> SCSI device. >>>>> >>>>> Now the point is, I need to make sure that this device must not be >>>>> mounted at all, so that user can't directly read/write to the device. >>>>> >>>>> Another constraint is that there must not be sorting/scheduling of >>>>> disk/drive I/O requests, not even at the kernel level. >>>>> e.g. If user read 10 LBAs beginning from LBA 5000 say, and the LBA >>>>> 1000, and then LBA 6000, then the i/o requests must be passed in this >>>>> same order to the disk/drive - ie there must not be reordering of disk >>>>> i/o requests, and the kernel must not cache/buffer the read/write data >>>>> . >>>>> >>>>> Another constraint is that only the utility's process/thread must be >>>>> able to access that drive, and no other userland process. >>>>> >>>>> >>>>> I do not need filesystem access, just raw sector/byte read/write, for >>>>> the selected drive. >>>>> >>>>> >>>>> Till now after reading docs and books related to FreeBSD, it seems >>>>> that I need to write a kernel mode driver(in form of kernel module) >>>>> for the purpose, that will communicate with userland utility via a >>>>> custom protocol. This driver will attach itself to the drive that is >>>>> selected by the user from utility. This driver need not concerned for >>>>> other devices except the selected one, as kernel has already >>>>> drivers/modules for this. >>>>> >>>>> This driver will issue appropriate commands to the device selected, >>>>> and read/write the LBAs directly from/to userland utility buffer. >>>>> >>>>> It seems I may need to write a custom system call too. Am I right? >>>>> >>>>> The point is, I need direct control of the device selected. The other >>>>> devices can be read/write normally as they would be. >>>>> For e.g. say the utility reads LBA 5000 from the selected device, and >>>>> somehow device failed to respond after a timeout, then the drive will >>>>> issue reset command to the device. >>>>> The LBA read will be passed to userland process, and no disk i/o >>>>> scheduling or sorting must be done on the i/o requests made by >>>>> userland process. >>>>> >>>>> In case of ATA disk drives, I also need to control that whether the >>>>> data transfer will be via PIO or DMA transfers. >>>>> >>>>> Can someone enlighten me how could I begin with? I don't know at which >>>>> I/O layer such a driver should sit - just above CAM peripheral layer, >>>>> or at CAM peripheral layer or CAM transport layer, or, at just above >>>>> raw device layer, or at raw device layer itself, or something else. >>>>> >>>>> Also I wanna know is there other way that goes w/o writing such a >>>>> driver, so that only userland code will do the work effectively. >>>>> >>>>> I'm targetting FreeBSD 9.0-RELEASE and onwards. >>>> >>>> >>>> My first take would be to use a custom GEOM that claims exclusive >>>> access >>>> to >>>> the disk (that prevents mounting, etc.). However, that still sits >>>> above >>>> the driver layer, and some of the things you want to do really depend >>>> on >>>> the controller's driver (e.g. I/O sorting is typically done in the >>>> controller >>>> driver via bioq_disksort(), as are decisions like PIO vs DMA). >>>> >>>> I don't think you need a custom system call btw. The easiest thing to >>>> do >>>> is to export a new file in /dev/ for each disk and userland >>>> applications >>>> can >>>> just use read/write against that directly. This should be easy to do >>>> with >>>> a GEOM module. >>>> >>>> It may be that you can even get the desired semantics you want if you >>>> hack >>>> on each controller driver to accept custom GEOM control requests (that >>>> would >>>> come from your module) to do things like disable any sorting and toggle >>>> PIO vs DMA. You might have to hack on the CAM da/ada drivers to pass >>>> those >>>> requests down to the underlying sim as well as the controller drivers. >>>> I'm >>>> not sure if we have any similar control requests like that in place >>>> already >>>> that you could use as a reference. >>> >>> >>> Jack. You've started your mail speaking about SCSI/ATA layer access, but >>> finished closer to block device layer. Each device can be accessed both >>> ways, depending on your needs. >>> >>> SCSI/ATA access you may get through the CAM pass driver. It allows you >>> to >>> run any SCSI or ATA commands, it does not modify them and does not >>> reorder >>> them during normal operation. But CAM does not restrict simultaneous >>> access >>> to the devices, so the same device can be accessed through both pass and >>> da/ada drivers (or any other if they happen) at the same time. Usually it >>> is >>> not a problem because pass interface is used only by short list of >>> tools, >>> like ones for CD/DVD recording. >>> >>> Block access you may get through da/ada CAM drivers and respectfully >>> through >>> GEOM. GEOM does controls concurrent device access. If you open some >>> device >>> from user-level for writing, nothing else will be able to access it at >>> the >>> same time on a block layer (raw access through the pass driver is still >>> possible). But before device can be opened, GEOM will do its usual job, >>> tasting device for some metadata, partitions, file systems, etc. I don't >>> know whether it is a problem, but the only way to prevent that is hack >>> CAM >>> transport or da/ada drivers to not attach to specific devices at all, >>> leaving only pass interface. Also, as John told, there is a request >>> sorter >>> on a top side of ada and da drivers. So if several request sent >>> simultaneously, they may be reordered. There is special BIO request flag >>> BIO_ORDERED to force original request order, but I don't know way to send >>> it >>> from user-level. If you need absolute guarantee of order, you should >>> wait >>> for request completion. Neither CAM nor GEOM are caching requests, so >>> that >>> will help with ordering, but limit performance. >>> >>> CAM provides control over ATA PIO/DMA modes via camcontrol tool or >>> respective calls via the pass interface. It works for all supported >>> ATA/SATA >>> controllers. For SAS controllers it can be problematic, as these >>> controllers >>> implement some translation and may not provide such interfaces. >>> >>> -- >>> Alexander Motin >> >> Thank you both, for your responses, I'll need to grasp the approaches >> you discussed. I'll post the update. >> Thanks. > > So, basically to follow these requirements in utility: > > 1. No other userland process is allowed to access the selected device > while utility is accessing the device. > 2. No disk I/O sorting on selected device, must be done. > 3. Utility can specify whether to transfer data via PIO or DMA(in case > of ata devices). > 4. issue IDLE or Standby, reset device comand, and similar commands to > drive. > 5. I also need to disable(if enabled) SATA NCQ, and Command tag > queing( in SCSI drives), on the selected device. > 6. No kernel level buffering, for selected device's I/O request data > transfer. > > it seems that I need to modify da/ada/sa drivers, so that they do not > get attach themselves to the selected device. > > (The above requirements/contraints only apply to the device that is > selected, they do not apply on other scsi/ata devices > attached to the system. The other device will be accessed by the > utility in normal manner.) > > After going through the approaches mentioned, following issues arose > to my understanding. > > 1. It seems that raw device(device = tape or hard drive, here) layer > is different from CAM passthrough driver. ie raw device > access do go though GEOM layer, while CAM passthrough accesses go > directly to da/sda/sa drivers of CAM sub-system, bypassing GEOM layer > completely. Am I right? > If it is true then I dont' need to modify GEOM layer - just modify > the default da/ada/sa driver and access the device > through cam pass driver. > > > 2. Will only modifying the default da/ada/sa drivers(so that they do > not get attach themselves to selected device) will do > the work? > I mean do I have to also modify default GEOM layer, or add a new > customized GEOM module, or, do I need to modify layers above or below > the da/sda/sa drivers in CAM subsystem? > Do I need to modify on each controller(host bus adapter) driver? > Since AFAIK, decisions like I/O request sorting and PIO/DMA in case of > ata devices, are done at the CAM peripheral layer, or CAM transport > layer. Please correct me if I'm wrong. > Actually I'm asking this so that rather than making modifications at > many layers/modules, it would be nice to modify a > single layer or module, so that we will not compromise the kernel > stability, and consistency. > > > > 3. Will modifying default da/sda/sa drivers(so that they do not get > attach themselves to selected device) affect other > device's access, e.g. sorting of I/O requests, etc.? > > Since there is a request sorter on a top side of default ada and da > drivers, I need to disable sorting of I/O requests on > selected device, right from the beginning, ie as soon as device is > selected. > Is there a way to disable I/O sorting only for a particular device, or > will I need to modify default da/ada/sa drivers such > that they don't do I/O request sorting at all. In this case, of course > I/O requests for other devices, will also not get > sorted, but that is ok, if it is the only solution. > > In short, the purpose of modifying default da/ada/sa drivers is : > a) so that they do not attach themselves to the seleted device, or if > already attached then they must detach themselves from the selected > device, as soon as the device is selected. > b) so that no sorting of I/O request for selected device, is done. > > 4. Two more things I would like to stress, are that the data transfer > from selected device via utility, can be many > gigabytes, and that other devices will be accessed by the utility in > normal manner, like via read(), write(), ioctl(), calls. > Will modifying default da/ada/sa, can have any negative effect on > accesses made for other devices in normal way? > > > > I tried my best to clarify the issues, but if more clarification is > needed about any of the above issue, please let me know. > > Thanks > hello again, Okay, this time I'm more clear. Below is the ascii artwork to show the flow of control that will happen when a I/O request is issued to the selected device via CAM pass through driver. All the I/O layers involved, are shown. Assume that we modified da/ada/sa drivers so they will not attach to this selected device. Nothing else is modified. _________________________________ | | | utility code | |________________________________| | | | read / write / ioctl system calls on | | pass device created | |________________________________| | | | CAM passthrough driver | | Send CAM control Blocks to XPT | |_________________________________| | | | CAM Transport Layer(XPT) | | communication to upper layers is via | | CAM Control Blocks (CCB). | |_________________________________| | | | SCSI Interface Module(SIM), or | | HBA driver module | | Parallel SCSI(SPI), Fibre channel(FC), | | USB Mass Storage(UMASS), | | FireWire(IEEE 1394), ATAPI. | | SAS and SATA/ATA | |_________________________________| Please feel free to correct me if I am wrong in the control flow in above diagram. Is there really no sorting of I/O requests inside cam passthrough driver, or cam transport layer? Say if I send one i/o request to selected device via cam passthrough driver and then send other one, before previous one is completed. Do I need to modify cam transport layer also, to switch ata device from PIO to DMA and vice versa for data transfer and to disable SATA NCQ, and Command tag queing ; or CAM passthrough driver will be able to do that. It seems that all this should be suffice for following the below requirements: 1. No other userland process is allowed to access the selected device while utility is accessing the device. 2. No disk I/O sorting on selected device, must be done. 3. Utility can specify whether to transfer data via PIO or DMA(in case of ata devices). 4. issue IDLE or Standby, reset device comand, and similar commands to drive. 5. I also need to disable(if enabled) SATA NCQ, and Command tag queing( in SCSI drives), on the selected device. 6. No kernel level buffering, for selected device's I/O request data transfer. Thanks
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACmXQA3Ek5Qk94L=D8zGWjeLOaTzaM04MTzU6KRtF1ANPwUuvg>