From owner-freebsd-arch Thu Jun 20 20: 7:58 2002 Delivered-To: freebsd-arch@freebsd.org Received: from prism.flugsvamp.com (66-191-112-47.mad.wi.charter.com [66.191.112.47]) by hub.freebsd.org (Postfix) with ESMTP id 30A0E37B429 for ; Thu, 20 Jun 2002 20:07:28 -0700 (PDT) Received: (from jlemon@localhost) by prism.flugsvamp.com (8.11.6/8.11.6) id g5L37CK24875; Thu, 20 Jun 2002 22:07:12 -0500 (CDT) (envelope-from jlemon) Date: Thu, 20 Jun 2002 22:07:12 -0500 From: Jonathan Lemon To: Terry Lambert Cc: Gary Thorpe , freebsd-arch@FreeBSD.ORG Subject: Re: multiple threads for interrupts Message-ID: <20020620220712.E70387@prism.flugsvamp.com> References: <3D1293AE.FDEC441D@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre2i In-Reply-To: <3D1293AE.FDEC441D@mindspring.com> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, Jun 20, 2002 at 07:47:10PM -0700, Terry Lambert wrote: > The way I would suggest doing this is to run the protocol processing > up to user space at interrupt time (LRP). This gets rid of NETISR. > > A lot of people complain that this won't allow you to receive as > many packets in a given period of time. They are missing the fact > that this only affect the burst rate until poll saturation occurs, > at which point in time the number of packets that you receive is in > fact clocked by buffer availability, and buffer availability is > clocked by the ability of NETISR to process the packets up to the > user space boundary, and that in turn is clocked by the ability to > process the packets out to the user space programs on the other end > of the sockets. > > What this all boils down to is that you should only permit receive > data interrupts to occur at the rate that you can move the data from > the wire, all the way through the system, to completion. > > The feedback process in the absence of available mbufs is to take > the interrupt, and then replace the contents of the mubuf receive > buffer ring with the new contents. The mbuf's only ever get pushed > up the stack if there is a replacement mbuf allocable from the > system in order to put on the ring in place of the received mbuf. > Effectively, we are talking about receive ring overflow here. > > If you trace the dependency graph on mbuf availability all the > way to user space, you will see that if you are receiving packets > faster than you can process them, then you end up spending all > your time servicing interrupts, and that takes away from your > time to actually push data through. > > Jeff Mogul of DEC Western Research Laboratories described this as > "receiver livelock" back early in the last decade. Yes, I believe anyone who has done work in this area is familiar with that technical report. > Luigi's and Jon Lemon's work only partially mitigates the problem. > Turning off interrupts doesn't deal with the NETISR triggering, > which only occurs when you splx() down from a hardware interrupt > level so that the SWL list is run. Running the packets on partially > up the stack doesn't resolve the problems up to the user/kernel > barrier. So both are only partial solutions. Yes. The patches (which I should really make an effort to find out where I put them; maybe to my left under some strawberry jam) were for -stable, and delivered the packet all the way to the socket buffer in interrupt context. They were more intended as a proof of concept than anything else, although I wouldn't have any qualms about committing them to -stable; the corresponding work has to be done in -current first though. I think that an interesting next step(s) would be to start collapsing some of the stack layering a bit. The design notes that VJ outlined for 4.4BSD (which were never completed) look interesting. -- Jonathan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message