Date: Wed, 13 Oct 1999 13:55:20 -0700 (PDT) From: Matthew Jacob <mjacob@feral.com> To: Randell Jesup <rjesup@wgate.com> Cc: scsi@FreeBSD.ORG Subject: Re: Driver for GDT6517RD RAID controller Message-ID: <Pine.BSF.4.10.9910131334430.69867-100000@beppo.feral.com> In-Reply-To: <ybuu2nvxn2f.fsf@jesup.eng.tvol.net.jesup.eng.tvol.net>
next in thread | previous in thread | raw e-mail | index | archive | help
> > I'll need to look at the FreeBSD-CAM/XPT. None of the driver > designs I've ever done have had any inherent limitations on the number of > active requests (other than physical memory). I wonder if the XPT (or SIM) > could reserve resources for target-mode driver responses... Part of the > problem is not understanding FreeBSD's XPT's resource-allocation code. I > think I really need to start looking at the code; perhaps this weekend. > The CAM target mode model is in fact also what the Qlogic f/w uses for target mode. You in fact instruct the f/w about the number of both immediate notification as well as command resources to reserve- usually on a per-lun basis. This allows the f/w to handle flow control for you without even being interrupted. But this isn't always enabled- for example the SCCLUN (16 bit first level lun) mode for the Fibre Channel card just punts this all to the driver level- the f/w is not going to attempt to resource manage 65K worth of luns... I may have misunderstood Justin in some email about 2-3 weeks ago- but if I didn't, I don't agree with him. I also feel that for the amount of infrastructure resources (CCBs) you *should* in fact pre-allocate at least for the case of being able to say "no, I'm busy...". I believe that the current CAM implementation for FreeBSD attempts to size the CCB free pool size based upon the HBA's 'max openings' quantity- but this is probably wrong in three respects. First, it can't account for race conditions between being done with a CCB but the CCB isn't yet freed (because the thread that does completions hasn't run yet), so the pool should be a bit larger to account for this. Second, this doesn't account for the non-command specific CCBs that are always being used like PATH_INQUIRY and so on. Third, it also doesn't account for the load imposed by handling *both* target and initiator mode which may have the same 'max openings' limit, or twice that, or something completely different. In any case, there are two major strategic choices for this kind of resource allocation. One is a general pool (which is what is there now) that attempts to (re)size itself based upon expected load based upon the number of HBAs attached. If you support Target Mode as well, then I suppose you could call directly into the XPT framework and ask it to resize itself upwards as needed. The other choice is to make the HBAs (sim layer) responsible for CCB allocation for both initiator and target commands. This is a mechanism akin to streams head or mbuf head allocators that have been used very effectively for years in some platforms- the actual code that drives the device is the one most likely to know the 'right' amount to allocate. On the other hand, this makes it difficult for CAM which does all the queue management and scheduling. For either case, you still cannot entirely guarantee that you won't run out of some kind of system resource at an inconvenient time, so you still have to design for that eventuality. My own choice in these matters (for a Solaris implementation that is now being moved into my *BSD/Linux Qlogic drivers) is reserve enough *local* resource for the driver so that if the requisite platform resources (CCBs for FreeBSD, DDI dma mapping resources in Solaris) aren't available, I can turn around and complete an incoming command right away with QFull. It does make some of the code a bit ugly because a check has to be made as to whether this is a private or a public command, but it means I'll never get caught short. -matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-scsi" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.10.9910131334430.69867-100000>