From owner-freebsd-drivers@FreeBSD.ORG Mon Feb 9 16:11:27 2009 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A391D106566B for ; Mon, 9 Feb 2009 16:11:27 +0000 (UTC) (envelope-from jason.harmening@gmail.com) Received: from mail-qy0-f17.google.com (mail-qy0-f17.google.com [209.85.221.17]) by mx1.freebsd.org (Postfix) with ESMTP id 33C7C8FC0A for ; Mon, 9 Feb 2009 16:11:27 +0000 (UTC) (envelope-from jason.harmening@gmail.com) Received: by qyk10 with SMTP id 10so3637892qyk.19 for ; Mon, 09 Feb 2009 08:11:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=CEnrp+OsScjpqngqP9eu2sEFEjG5DFJfL1cLtJIbY5I=; b=kGYzT4IrWaZ9L6TSrLtczwCBETWI6TMGC6Z3nVExgfN7WSkeEmetdaCW8uIKOk9GPF Kr+dGT8EDFlt15FpbAXvcMG5JOkO9a0H6JwwbucACf1PSF6i2nYemIQTWRndICNPtnNn xV9JeuzHmDZTRLsojzkn3DXrc25W7QCDYyk1Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=bDKHsCi/EwEMVi+5Aurd7p6NvHDYuhp3g89PzEUEuW0v65mp17+b0R+PnXH/1PQFoS YST1motc/bE7aztvpbS2AAXs63tvaxtlU0gdnPsqs1kY8CuoBq0m+3iNNVltZX4BIYuX 1lcFeFRyGq5sTwhF+1byI8n2c4rIc9c1lZLzU= MIME-Version: 1.0 Received: by 10.229.85.12 with SMTP id m12mr362049qcl.98.1234195880568; Mon, 09 Feb 2009 08:11:20 -0800 (PST) In-Reply-To: <200902090946.46917.jhb@freebsd.org> References: <20090208113202.6d591589@CORONA> <200902090946.46917.jhb@freebsd.org> Date: Mon, 9 Feb 2009 10:11:20 -0600 Message-ID: <2d1264630902090811p761b0a14w997985779aad8748@mail.gmail.com> From: Jason Harmening To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-drivers@freebsd.org Subject: Re: PCI-Express interrupt issues X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Feb 2009 16:11:28 -0000 On Mon, Feb 9, 2009 at 8:46 AM, John Baldwin wrote: > On Sunday 08 February 2009 12:32:02 pm Jason Harmening wrote: >> Hi All, >> >> I'm the maintainer for the FreeBSD cx88 driver (multimedia/cx88). I've >> recently encountered some issues w/ interrupt handling, specifically on >> PCI-Express systems, and I was hoping someone would be able to help me. >> >> Issue #1: >> >> The cx88 driver has split interrupt handling between filters and >> ithreads since filters became available w/ 7.0. For the past year, the >> filters have been set up to return FILTER_SCHEDULE_THREAD if the status >> register indicates an interrupt, FILTER_STRAY otherwise. Everything >> has worked fine. >> >> However, I recently stumbled across some documentation indicating that >> FILTER_SCHEDULE_THREAD shouldn't be returned alone--instead, the filter >> should return FILTER_HANDLED | FILTER_SCHEDULE_THREAD. So I modified >> the cx88 driver to do that instead, and that's where things turned >> strange: >> >> On machine #1, which uses a VIA K8T800 chipset, everything still worked >> fine. >> >> On machine #2, which uses a VIA K8T890 chipset, the threaded interrupt >> handlers were no longer invoked. It's as though the bus driver saw >> FILTER_HANDLED in the bitmask and assumed the interrupt was already >> processed without checking to see if an ithread should be scheduled. >> What's interesting is that the only significant difference between the >> K8T800 in machine #1 and the K8T890 in machine #2 is that the K8T890 >> supports PCI-Express, while the K8T800 is PCI-only. > > Are you seeing this only on 7.0? Also, do you have 'INTR_FILTER' enabled in > the kernel? If you don't, then your ithread will never be called if you have > a filter (actually, the bus_setup_intr() should fail in that case if you > specify both). All machines are running 7.1-STABLE. Filters are enabled across the board, and reverting to just returning FILTER_SCHEDULE_THREAD fixes the issue. > >> Issue #2: >> >> We're working on adding support for a newer generation of PCI-Express >> cx88 devices, which support message signaled interrupts. >> When MSIs are enabled for these boards, the interrupts seem to simply >> stop firing after the first several. During transfers across the >> cx88's onboard I2C bus, the first several (e.g. 8-10) bytes will be >> transferred successfully, which means the I2C transfer interrupts are >> working for those bytes. But after that the passive thread will timeout >> waiting to be signaled by the interrupt handler. The I2C interrupts >> are handled entirely in their own filter (no ithread here), and they can >> occur with high frequency (e.g. several hundred per second for large >> firmware loads). We don't see anything in the system log indicating >> that an interrupt storm is detected or that throttling is coming into >> play here. >> >> If we revert to legacy INTx interrupts, everything works fine. All >> other transfer logic is identical between the legacy and MSI cases. >> The MSI behavior is identical between machine #2 w/ the K8T890 and a >> third machine w/ an nForce4. We haven't been able to try it anywhere >> else yet. >> >> >> I'm hoping someone will be able to shed some light on these >> issues--neither one is a real emergency, but they're both kind of >> troubling. > > The only thing I can think of is perhaps a hardware bug. MSI interrupts are > edge triggered while INTx are level. However, if a device implements MSI > by "listening" to the level-triggered interrupt and sending a message when > the level changes and it ever has a bug where it fails to send a message, > then it won't recover. FreeBSD doesn't do any masking of interrupts for MSI, > it just installs the handler in the IDT and calls it for every interrupt. > Thus, if you are not seeing interrupts it is likely not an issue with FreeBSD > itself, but an issue in the hardware (or possibly the driver if your > interrupt handler doesn't fully ACK something perhaps). With this hardware, failing to ACK would just cause a storm, which we'd see w/ INTx as well. But you are right about the possibility of a hardware issue--the particular piece we're testing right now has some other known issues as well. There's a linux driver that seems to work OK w/ MSIs, but they don't actually use hardware I2C and don't use interrupts there. This issue just needs more testing w/ more hardware, and we have a device hint to enable/disable MSIs anyway. > > -- > John Baldwin >