From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 18 07:20:45 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CB6DE16A4B3 for ; Thu, 18 Sep 2003 07:20:45 -0700 (PDT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9DB743F3F for ; Thu, 18 Sep 2003 07:20:44 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id h8IEKi5J011187 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Thu, 18 Sep 2003 10:20:44 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id h8IEKci60859; Thu, 18 Sep 2003 10:20:38 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16233.48950.348205.388570@grasshopper.cs.duke.edu> Date: Thu, 18 Sep 2003 10:20:38 -0400 (EDT) To: aaro@iki.fi In-Reply-To: References: X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-hackers@freebsd.org Subject: Re: PCI interrupts passing DMA X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2003 14:20:45 -0000 Aaro Koskinen writes: > > My question is: What the heck could the SMP kernel be doing which > > causes the DMA to "complete" faster? > > The chipset probably uses PCI bus (MSI-like mechanism) to deliver the > interrupt from the IO APIC to the local APIC, which means that the PCI > bridge(s) must complete the DMA transfer before the interrupt is > delivered to preserve the write order. AHA! I think you hit it on the nose. It turns out that the FreeBSD SMP kernel sets up all IOAPIC interrupts as IOART_DELLOPRI. But linux doesn't set the IOART_DELLOPRI bit. This seems account for the difference in behaviour between FreeBSD & linux. The following diff seems to make SMP FreeBSD behave the same as linux, and the same as UP FreeBSD: Index: i386/i386/mpapic.c =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/mpapic.c,v retrieving revision 1.63 diff -u -r1.63 mpapic.c --- i386/i386/mpapic.c 23 Jul 2003 18:59:38 -0000 1.63 +++ i386/i386/mpapic.c 18 Sep 2003 14:07:38 -0000 @@ -134,7 +134,7 @@ ((u_int32_t) \ (IOART_INTMSET | \ IOART_DESTPHY | \ - IOART_DELLOPRI)) + IOART_DELFIXED)) #define DEFAULT_ISA_FLAGS \ ((u_int32_t) \ > In PIC mode, the interrupt is delivered by the wire and it has no > effect on pending writes. A common solution is that the interrupt > handler must perform a read from the device to the force flushing of > buffers. Yep. I was trying to avoid that because PIO reads are so horribly expensive.. I guess I'll have to do it after all. I wish MSIs had been around from the beginning & were more widely used. Thanks for your help, Drew