From owner-freebsd-arch@FreeBSD.ORG Thu Oct 1 18:27:11 2009 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D5BD106568F for ; Thu, 1 Oct 2009 18:27:11 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id ED72B8FC1E for ; Thu, 1 Oct 2009 18:27:10 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 85AF646B1A; Thu, 1 Oct 2009 14:27:10 -0400 (EDT) Received: from jhbbsd.hudson-trading.com (unknown [209.249.190.8]) by bigwig.baldwin.cx (Postfix) with ESMTPA id A5D408A020; Thu, 1 Oct 2009 14:27:09 -0400 (EDT) From: John Baldwin To: Bruce Evans Date: Thu, 1 Oct 2009 14:26:55 -0400 User-Agent: KMail/1.9.7 References: <200909301732.20589.jhb@freebsd.org> <200910010949.20849.jhb@freebsd.org> <20091002024550.X21519@delplex.bde.org> In-Reply-To: <20091002024550.X21519@delplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200910011426.56047.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Thu, 01 Oct 2009 14:27:09 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: arch@freebsd.org Subject: Re: Interrupt Descriptions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Oct 2009 18:27:11 -0000 On Thursday 01 October 2009 1:14:01 pm Bruce Evans wrote: > On Thu, 1 Oct 2009, John Baldwin wrote: > > > On Wednesday 30 September 2009 8:02:57 pm Bruce Evans wrote: > >> On Wed, 30 Sep 2009, John Baldwin wrote: > >> > >>> A few folks have asked recently for the ability to add descriptive strings to > >>> registered interrupt handlers. This is especially true since the advent of > >>> MSI with multiple interrupts per device. I hacked up a prototype today that > >> > >> Interrupt names should be no longer than 4 (5 works sometimes) characters so > >> that they can be displayed by systat -v. > > > > For the machines this is targeted at (i.e. ones with devices using MSI/MSI-X) > > the number of interrupts is already well beyond what systat -v can currently > > display anyway. > > This should be fixed before expanding the names into descriptions. To me this change doesn't make systat any more or less broken than it was before, so I consider systat an orthogonal problem. > I wonder what uses all these interrupts. Most FreeBSD cluster machines use > a whole 3 interrupts apart from acpi_timer, with 1 barely active (1 disk, > 1 net, and 1 serial i/o probably only used for a serial console). My > workstations use about 10 plus timers, but most are usually inactive > except for net, keyboard and sometimes disk. Does anyone have systems > more than about 16 disk controllers and 16 network cards so as to need > more than 32 interrupts? As Jack mentioned, many newer devices are starting to take advantage of MSI to use multiple interrupts for a single device. A typical cxgb(4) adapter will use up to 9 interrupts (1 for link/misc, and 1 per-CPU for a RX/TX queueset). The Intel 10gb driver takes this even farther by giving each queueset separate RX/TX interrupts and it supports up to 16 queues giving up to 33 total interrupts per device. Even the igb(4) driver currently uses 1 interrupt for link, 1 for each RX queue, and 1 for each TX queue and it supports having at least 4 RX queues (possibly as many as 8?). Devices of this nature using lots of interrupts is also what drove the change to use per-CPU IDTs in 8.0 to increase the number of available IDT vectors for these types of devices that want MSI interrupts tied to every CPU. > > I believe most folks use vmstat -i or ithread stats in top > > to measure real-time interrupts at this point. Possibly systat could grow a > > new interrupt-only mode that has sufficient room for full interrupt names. > > I don't find static info and thus vmstat -i very useful. vmstat with a > repeat interval is useful, but for some reason vmstat -i never supported > the repeat interval (its multi-line display is certainly unsuitable for > repetitions being readable), and with systat -v repetition in vmstat -i > is rarely missed. A repeating vmstat -i would have even more space > problems than systat -v, since it would want to repeat single line output. For certain tests simply doing a before and after of vmstat -i and figuring out the diff manually (sometimes just having a diff at all is important rather than the number (e.g. did some action I just did generate an interrupt?)) is sufficient. > systat and most things just want a unique tag, not a description or even > a full name. The full name could be looked up from a tag by looking at > vmstat -i output which could show both. This could be interesting perhaps. I would probably want to change my proposed intrcnt structure to contain three things then: a unique tag, a full name, and a count. One possible idea for the unique "tag" would be to use the interrupt handler name if there is only a single handler or to use the 'irqXXX' name if the interrupt is shared. -- John Baldwin