From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 18 04:40:23 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 352EF16A4B3 for ; Thu, 18 Sep 2003 04:40:23 -0700 (PDT) Received: from sirppi.helsinki.fi (sirppi.helsinki.fi [128.214.205.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9BB7A43FBF for ; Thu, 18 Sep 2003 04:40:21 -0700 (PDT) (envelope-from akoskine@cc.helsinki.fi) Received: from sirppi.helsinki.fi (localhost [127.0.0.1]) by sirppi.helsinki.fi (8.12.10/8.12.10) with ESMTP id h8IBeHKN005332; Thu, 18 Sep 2003 14:40:17 +0300 (EET DST) Received: from localhost (akoskine@localhost)h8IBeHwL025098; Thu, 18 Sep 2003 14:40:17 +0300 (EET DST) X-Authentication-Warning: sirppi.helsinki.fi: akoskine owned process doing -bs Date: Thu, 18 Sep 2003 14:40:17 +0300 (EET DST) From: Aaro Koskinen Sender: akoskine@cc.helsinki.fi To: gallatin@cs.duke.edu Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-hackers@freebsd.org Subject: Re: PCI interrupts passing DMA X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: aaro@iki.fi List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2003 11:40:23 -0000 Hello, > I was toying with a programmable PCI card and wrote some code > which DMAs a small block of data to the host, and then interrupts the > host. The host checks the end of the block, and sees if it gets the > value it expects. > On an SMP P4 (hyperthreaded, with ServerWorks chipset) FreeBSD 4.8 UP, > and on Linux 2.4.18, there is a huge delay between the interrupt being > handled, and the DMA finally completing (from the host's perspective). > Time enough for the interrupt handler to be triggered 3 or 4 times, > and to print "foo" to a serial console line each time it notices > that the DMA has not completed. > > The interesting thing is that on FreeBSD 4.8SMP, and FreeBSD > 5.1-current (SMP), the data has arrived by the time the interrupt > handler is called. [...] > My question is: What the heck could the SMP kernel be doing which > causes the DMA to "complete" faster? The chipset probably uses PCI bus (MSI-like mechanism) to deliver the interrupt from the IO APIC to the local APIC, which means that the PCI bridge(s) must complete the DMA transfer before the interrupt is delivered to preserve the write order. In PIC mode, the interrupt is delivered by the wire and it has no effect on pending writes. A common solution is that the interrupt handler must perform a read from the device to the force flushing of buffers. A. -- Aaro Koskinen E-mail: aaro@iki.fi "I'm the ocean, I'm the giant undertow." http://www.iki.fi/aaro