Date: Mon, 9 Feb 2009 10:11:20 -0600 From: Jason Harmening <jason.harmening@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-drivers@freebsd.org Subject: Re: PCI-Express interrupt issues Message-ID: <2d1264630902090811p761b0a14w997985779aad8748@mail.gmail.com> In-Reply-To: <200902090946.46917.jhb@freebsd.org> References: <20090208113202.6d591589@CORONA> <200902090946.46917.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 9, 2009 at 8:46 AM, John Baldwin <jhb@freebsd.org> wrote: > On Sunday 08 February 2009 12:32:02 pm Jason Harmening wrote: >> Hi All, >> >> I'm the maintainer for the FreeBSD cx88 driver (multimedia/cx88). I've >> recently encountered some issues w/ interrupt handling, specifically on >> PCI-Express systems, and I was hoping someone would be able to help me. >> >> Issue #1: >> >> The cx88 driver has split interrupt handling between filters and >> ithreads since filters became available w/ 7.0. For the past year, the >> filters have been set up to return FILTER_SCHEDULE_THREAD if the status >> register indicates an interrupt, FILTER_STRAY otherwise. Everything >> has worked fine. >> >> However, I recently stumbled across some documentation indicating that >> FILTER_SCHEDULE_THREAD shouldn't be returned alone--instead, the filter >> should return FILTER_HANDLED | FILTER_SCHEDULE_THREAD. So I modified >> the cx88 driver to do that instead, and that's where things turned >> strange: >> >> On machine #1, which uses a VIA K8T800 chipset, everything still worked >> fine. >> >> On machine #2, which uses a VIA K8T890 chipset, the threaded interrupt >> handlers were no longer invoked. It's as though the bus driver saw >> FILTER_HANDLED in the bitmask and assumed the interrupt was already >> processed without checking to see if an ithread should be scheduled. >> What's interesting is that the only significant difference between the >> K8T800 in machine #1 and the K8T890 in machine #2 is that the K8T890 >> supports PCI-Express, while the K8T800 is PCI-only. > > Are you seeing this only on 7.0? Also, do you have 'INTR_FILTER' enabled in > the kernel? If you don't, then your ithread will never be called if you have > a filter (actually, the bus_setup_intr() should fail in that case if you > specify both). All machines are running 7.1-STABLE. Filters are enabled across the board, and reverting to just returning FILTER_SCHEDULE_THREAD fixes the issue. > >> Issue #2: >> >> We're working on adding support for a newer generation of PCI-Express >> cx88 devices, which support message signaled interrupts. >> When MSIs are enabled for these boards, the interrupts seem to simply >> stop firing after the first several. During transfers across the >> cx88's onboard I2C bus, the first several (e.g. 8-10) bytes will be >> transferred successfully, which means the I2C transfer interrupts are >> working for those bytes. But after that the passive thread will timeout >> waiting to be signaled by the interrupt handler. The I2C interrupts >> are handled entirely in their own filter (no ithread here), and they can >> occur with high frequency (e.g. several hundred per second for large >> firmware loads). We don't see anything in the system log indicating >> that an interrupt storm is detected or that throttling is coming into >> play here. >> >> If we revert to legacy INTx interrupts, everything works fine. All >> other transfer logic is identical between the legacy and MSI cases. >> The MSI behavior is identical between machine #2 w/ the K8T890 and a >> third machine w/ an nForce4. We haven't been able to try it anywhere >> else yet. >> >> >> I'm hoping someone will be able to shed some light on these >> issues--neither one is a real emergency, but they're both kind of >> troubling. > > The only thing I can think of is perhaps a hardware bug. MSI interrupts are > edge triggered while INTx are level. However, if a device implements MSI > by "listening" to the level-triggered interrupt and sending a message when > the level changes and it ever has a bug where it fails to send a message, > then it won't recover. FreeBSD doesn't do any masking of interrupts for MSI, > it just installs the handler in the IDT and calls it for every interrupt. > Thus, if you are not seeing interrupts it is likely not an issue with FreeBSD > itself, but an issue in the hardware (or possibly the driver if your > interrupt handler doesn't fully ACK something perhaps). With this hardware, failing to ACK would just cause a storm, which we'd see w/ INTx as well. But you are right about the possibility of a hardware issue--the particular piece we're testing right now has some other known issues as well. There's a linux driver that seems to work OK w/ MSIs, but they don't actually use hardware I2C and don't use interrupts there. This issue just needs more testing w/ more hardware, and we have a device hint to enable/disable MSIs anyway. > > -- > John Baldwin >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2d1264630902090811p761b0a14w997985779aad8748>