From owner-freebsd-amd64@FreeBSD.ORG Wed Oct 26 16:57:29 2005 Return-Path: X-Original-To: freebsd-amd64@FreeBSD.ORG Delivered-To: freebsd-amd64@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 08EF616A41F; Wed, 26 Oct 2005 16:57:29 +0000 (GMT) (envelope-from jc@oxado.com) Received: from mars.interactivemediafactory.net (mars.imfeurope.net [194.2.222.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5CA0F43D45; Wed, 26 Oct 2005 16:57:28 +0000 (GMT) (envelope-from jc@oxado.com) Received: from JC-8600.oxado.com (localhost [127.0.0.1]) by mars.interactivemediafactory.net (8.12.11/8.12.11) with ESMTP id j9QGvKR0040469; Wed, 26 Oct 2005 18:57:26 +0200 (CEST) (envelope-from jc@oxado.com) Message-Id: <6.2.3.4.0.20051026183714.03b48e20@wheresmymailserver.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Wed, 26 Oct 2005 18:56:28 +0200 To: Scott Long From: Jacques Caron In-Reply-To: <435FAEE1.6000506@samsco.org> References: <6.2.3.4.0.20051025171333.03a15490@pop.interactivemediafactory.net> <6.2.3.4.0.20051026131012.03a80a20@pop.interactivemediafactory.net> <435F8E06.9060507@samsco.org> <6.2.3.4.0.20051026163501.03b7d3e8@wheresmymailserver.com> <435FAEE1.6000506@samsco.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-amd64@FreeBSD.ORG, sos@FreeBSD.ORG Subject: Re: busdma dflt_lock on amd64 > 4 GB X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2005 16:57:29 -0000 Hi Scott, Thank you for your time on this issue... At 18:29 26/10/2005, Scott Long wrote: >I sent a long email on this on Dec 14, 2004. I'll pull it up and >forward it out. What I really should do is publish a definitive article >on the whole topic. Definitely! And add a link in the bus_dma man page, or expand its contents... >As for 'limitations as to what can happen in the callback', there are >none if you use the correct code structure. Isn't the callback called from an interrupt context of some kind? (if you think I don't know a thing about kernel internals you've guessed right!) Would it be possible, for instance, to move this whole bunch of code: /* start ATAPI operation */ if (ch->hw.command(request->device, ATA_PACKET_CMD, 0, 0, ATA_F_DMA)) { ata_prtdev(request->device, "error issuing ATAPI packet command\n"); request->result = EIO; break; } /* wait for ready to write ATAPI command block */ { int timeout = 5000; /* might be less for fast devices */ while (timeout--) { int reason = ATA_IDX_INB(ch, ATA_IREASON); int status = ATA_IDX_INB(ch, ATA_STATUS); if (((reason & (ATA_I_CMD | ATA_I_IN)) | (status & (ATA_S_DRQ | ATA_S_BUSY))) == ATAPI_P_CMDOUT) break; DELAY(20); } if (timeout <= 0) { ata_prtdev(request->device,"timeout waiting for ATAPI ready\n"); request->result = EIO; break; } } /* this seems to be needed for some (slow) devices */ DELAY(10); /* output actual command block */ ATA_IDX_OUTSW_STRM(ch, ATA_DATA, (int16_t *)request->u.atapi.ccb, (request->device->param->config & ATA_PROTO_MASK) == ATA_PROTO_ATAPI_12 ? 6 : 8); /* start DMA engine */ if (ch->dma->start(ch)) { request->result = EIO; break; } into a callback? It's mainly all the DELAY's that seem out of place in a callback to me... Note for Soren: there is a good explanation at the end of http://freebsd.active-venture.com/arch-handbook/isa-driver-busmem.html of callback integration strategies... >>>No. Some tags specifically should not permit deferals. >> >>How do they do that? Setting BUS_DMA_ALLOCNOW in the tag, or >>BUS_DMA_NOWAIT in the map_load, or both, or something else? > >They set it by using NULL as the lockfunc. Errrr... I'm lost here. I understood that you should use NULL as the lockfunc if and only if you know there won't be any deferrals (and I know by experience now that not having a lockfunc and having a deferral cause a panic!). So having NULL as the lockfunc is a consequence of not having any deferrals, it won't prevent them. >Actually, one map per tag is not common. If the ATA driver supported >tagged queuing (which I assume that it will someday for SATAII, yes?) I'm waiting for it eagerly, I have quite a bunch of hardware all set for it :-) >If a map is being created for every drive in the system, and the result >is that not enough bounce pages are being reserved for all three drives >to operate concurrently, then there might be a bug in busdma. We should >discuss this offline. Maps are created for every busy drive in the system, but I believe they each belong to a separate tag (all tags and maps are created on the fly for each I/O operation, as I understand it). If three drives are (really) busy at the same time, a panic is guaranteed if bounce buffers are needed. I'm trying to figure out since yesterday where the problem is (busdma or ata), but I think it's "somewhere in between", with each part assuming that the other does things some way or another. >>In this case 32 bounce pages (out of 8 GB RAM) for 6 disks seems >>like a very tight bottleneck to me. > >If that's all that is needed to saturate non-tagged ATA, then there is >nothing wrong with that. It might be enough to saturate one or two non-tagged ATA drives, but certainly not 6... Anyway, right now, it's not even saturation I have to deal with, but just a brutal panic :-( Jacques.