From owner-freebsd-net@FreeBSD.ORG Thu Jan 17 17:45:01 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 01AC3C8B; Thu, 17 Jan 2013 17:45:01 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id B5CEC66B; Thu, 17 Jan 2013 17:45:00 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id BA2B97300A; Thu, 17 Jan 2013 18:44:27 +0100 (CET) Date: Thu, 17 Jan 2013 18:44:27 +0100 From: Luigi Rizzo To: Adrian Chadd Subject: Re: two problems in dev/e1000/if_lem.c::lem_handle_rxtx() Message-ID: <20130117174427.GA65218@onelab2.iet.unipi.it> References: <20130117025502.GA57613@onelab2.iet.unipi.it> <1358438932.56236.YahooMailClassic@web121601.mail.ne1.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: Barney Cordoba , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jan 2013 17:45:01 -0000 On Thu, Jan 17, 2013 at 08:48:25AM -0800, Adrian Chadd wrote: > There's also the subtle race condition in TX and RX handling that > re-queuing the taskqueue gets around. > > Which is: > > * The hardware is constantly receiving frames , right until you blow > the FIFO away by filling it up; > * The RX thread receives a bunch of frames; > * .. and processes them; > * .. once it's done processing, the hardware may have read some more > frames in the meantime; (in the "lem" driver this cannot happen until you release some rx slots, which only happens once at the end of the lem_rxeof() routine, not long before re-enabling interrupts) > * .. and the hardware may have generated a mitigated interrupt which > you're ignoring, since you're processing frames; > * So if your architecture isn't 100% paranoid, you may end up having > to wait for the next interrupt to handle what's currently in the > queue. > > Now if things are done correct: > > * The hardware generates a mitigated interrupt > * The mask register has that bit disabled, so you don't end up receiving it; > * You finish your RX queue processing, and there's more stuff that's > appeared in the FIFO (hence why the hardware has generated another > mitigated interrupt); > * You unmask the interrupt; > * .. and the hardware immediately sends you the MSI or signals an interrupt; > * .. thus you re-enter the RX processing thread almost(!) immediately. the problem i was actually seeing are slightly different, namely: - once the driver lags behind, it does not have a chance to recover even if there are CPU cycles available, because both interrupt rate and packets per interrupt are capped. - much worse, once the input stream stops, you have a huge backlog that is not drained. And if, say, you try to ping the machine, the incoming packet is behind another 3900 packets, so the first interrupt drains 100 (but not the ping request, so no response), you keep going for a while, eventually the external world sees the machine as not responding and stops even trying to talk to it. cheers luigi