From owner-freebsd-current@FreeBSD.ORG Sun Apr 10 22:56:16 2005 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1F0D416A4CE; Sun, 10 Apr 2005 22:56:16 +0000 (GMT) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD25943D3F; Sun, 10 Apr 2005 22:56:15 +0000 (GMT) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id B291C72DE5; Sun, 10 Apr 2005 15:56:15 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id AD3AA72DE4; Sun, 10 Apr 2005 15:56:15 -0700 (PDT) Date: Sun, 10 Apr 2005 15:56:15 -0700 (PDT) From: Doug White To: John Baldwin In-Reply-To: <200504101523.56787.jhb@FreeBSD.org> Message-ID: <20050410152946.W82708@carver.gumbysoft.com> References: <20050406233405.O47071@carver.gumbysoft.com> <200504081656.51917.jhb@FreeBSD.org> <200504101523.56787.jhb@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@FreeBSD.org Subject: Re: Potential source of interrupt aliasing X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Apr 2005 22:56:16 -0000 On Sun, 10 Apr 2005, John Baldwin wrote: > > For fun, I put together this one-liner to at least test the theory: > > > > http://people.freebsd.org/~dwhite/intr_machdep.c.20050409.patch > > Doing this should result in an instant interrupt storm since your PCI > interrupts (which are level triggered) should now keep firing over and over > and the machine will spend all it's time rescheduling the ithread. If it's > not doing that then this is very curious. Causing the storm was partly the point. I wanted to find a way to weather it as cheaply as possible until the ithread would get run. > > This makes the aliased interrupts go away at the cost of hitting > > sched_lock for each ithread interrupt. I'm trying to dream up a way of > > telling if the ithread is active or scheduled so we don't need to bump it > > again with setrunqueue(). I think I can use the it_need sparkplug to at > > least save mutex grabs on interrupts that fire much faster than their > > ithread handlers cycle. > > > > I'd like better granularity, though, but I'm not sure that setting a flag > > that is cleared just before mi_switch() in ithread_loop() and set just > > after won't race against an interrupt checking it to see if it needs to > > setrunqueue() the ithread. Maybe clear the flag before trying to grab > > sched_lock? > > it_need already does this. You haven't added any extra sched_locks. What've > you done is change the interrupt code to never mask interrupt sources. If > you aren't getting an interrupt storm on your first PCI interrupt then that > is very curious indeed. Have you tried this with SMP disabled so that you > only have one CPU running? It may be that due to SMP your ithread still gets > a chance to run on one CPU while the other CPU is stormed with interrupt > requests. Note that since storm detection is done in the ithread, this type > of storm won't be detected and throttled. It also might not show up in the > vmstats since we don't use atomic ops there anymore depending on whether or > not the interrupt bounces between CPUs that keep overwriting the count on top > of each other. I think you're right in the SMP case -- if one CPU gets bogged down another can get the ithread to bail it out. For the UP case, turning on INVARIANTS+WITNESS+!WITNESS_SKIPSPIN causes a lockup as the IOAPIC vectors are unmasked before the other CPUs are started on Scott's WV2. A traceback shows softclock's ithread running and taking an interrupt from em1, which is assumably triggering over and over since softclock can't seem to get out of its ithread. The debugging options and the constant interrupts slow it down so much that it isn't cycling fully before softclock fires again (or thats what it looks like). Adding a mi_switch() in the callout_lock drop/pickup didn't seem to help. Adding a bypass of setrunqueue() in ithread_schedule() if it_need is set also had no impact. I haven't tried things with PREEMPTION. I'm also taking suggestions for other places to try to mask interrupts. I pondered pulling the vector from the IDT, but that generates a trap. I don't think there's anything we can do in the LAPIC thats anything less than disabling interrupts entirely. -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org