Date: Thu, 20 Jun 2002 23:05:14 -0400 From: Bosko Milekic <bmilekic@unixdaemons.com> To: Terry Lambert <tlambert2@mindspring.com> Cc: Gary Thorpe <gat7634@hotmail.com>, freebsd-arch@FreeBSD.ORG Subject: Re: multiple threads for interrupts Message-ID: <20020620230514.B38506@unixdaemons.com> In-Reply-To: <3D1293AE.FDEC441D@mindspring.com>; from tlambert2@mindspring.com on Thu, Jun 20, 2002 at 07:47:10PM -0700 References: <F112l4rhYQYx3G5aYY6000252ae@hotmail.com> <3D1293AE.FDEC441D@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jun 20, 2002 at 07:47:10PM -0700, Terry Lambert wrote: > Gary Thorpe wrote: > > >Seigo Tanimura wrote: > > > > One solution is to run multiple threads for each of the interrupt > > > > types. Since I noticed this issue first during my work of network > > > > locking, I have been tweaking the swi subsystem so that it runs > > > > multiple threads for an swi type. For those who are interested, the > > > > patch can be found at: > > > > > > > > http://people.FreeBSD.org/~tanimura/patches/swipool.diff.gz > > > > > >Benchmarks before and after, demonstrating an improvement? > > > > > >-- Terry > > > > I am not a kernel programmer, but I have read a paper which concludes that > > making threads have an "affinity" or "stickiness" to the last CPU it was run > > on is benifical because it leads to less cache flushing/refilling. Maybe > > this will be a factor in having multiple threads for interrupt handling? > > THat's a general scheduling problem. The solution is well known, > and implemented in Dynix, then IRIX, now Linux. Alfred Perlstein > has some patches that take it most of the way there, but leave an > interlock, by maintaining a global queue. > > The solution I'm talking about is per CPU scheduling queues, where > threads are only migrated between scheduling queues under extraordinary > conditiona, so most of the scheduling never requires the locking the > FreeBSD-current has today. > > This solves the affinity problem. > > I'm not sure the affinity fix solves the NETISR problem, because I > think that the issue there is that the affinity you want in that > case is mbuf<->cpu affinity. Basically, if you take the network > interrupt on CPU 3, then you want to run the NETISR code that is > associated with the protocol processing on CPU 3, as well, to avoid > cache busting. > > The way I would suggest doing this is to run the protocol processing > up to user space at interrupt time (LRP). This gets rid of NETISR. > > A lot of people complain that this won't allow you to receive as > many packets in a given period of time. They are missing the fact > that this only affect the burst rate until poll saturation occurs, > at which point in time the number of packets that you receive is in > fact clocked by buffer availability, and buffer availability is > clocked by the ability of NETISR to process the packets up to the > user space boundary, and that in turn is clocked by the ability to > process the packets out to the user space programs on the other end > of the sockets. > > What this all boils down to is that you should only permit receive > data interrupts to occur at the rate that you can move the data from > the wire, all the way through the system, to completion. > > The feedback process in the absence of available mbufs is to take > the interrupt, and then replace the contents of the mubuf receive > buffer ring with the new contents. The mbuf's only ever get pushed > up the stack if there is a replacement mbuf allocable from the > system in order to put on the ring in place of the received mbuf. > Effectively, we are talking about receive ring overflow here. > > If you trace the dependency graph on mbuf availability all the > way to user space, you will see that if you are receiving packets > faster than you can process them, then you end up spending all > your time servicing interrupts, and that takes away from your > time to actually push data through. > > Jeff Mogul of DEC Western Research Laboratories described this as > "receiver livelock" back early in the last decade. > > Luigi's and Jon Lemon's work only partially mitigates the problem. > Turning off interrupts doesn't deal with the NETISR triggering, > which only occurs when you splx() down from a hardware interrupt > level so that the SWL list is run. Running the packets on partially > up the stack doesn't resolve the problems up to the user/kernel > barrier. So both are only partial solutions. > > > I'm convinced that CPU affinity needs to happen. > > I'm also convinced that, for the most part, running NETISR in > kernel threads, rather than to completion at interrupt, is the > wrong way to go. > > I'm currently agnostic on the idea of whether interrupt threads > will help in areas outside of networking. My instinct is that > the added contention will mean that they will not. I'm reserving > judgement pending seeing real benchmarks. > > To me, it looks like a lot of people are believing something is > better because they are being told that it is better, not because > they have personally measured it, gotten better numbers, and have > proven to themselves that those better numbers were a result of > what they thought they were measuring, rather than an artifact > that could be exploited in the old code, as well. Terry, dude, I can't believe I'm reading this. A lot of the stuff above makes sense but you're telling us to believe you just because you're saying it's better. And then to top it all off, you mention how we shouldn't believe things that people tell us simply because they claim that they are better. That's totally awesome! Have you ever read "Godel, Escher, Bach" by Hofstadter (sp?)? He discusses these self-referring systems in which he detects things he calls "Strange Loops" (well, that's only part of what he discusses, actually). I think I've found a "Strange Loop" in your Email: - Believe me because I'm telling you this is better. - Don't believe people who tell you to believe them because they say that it is better. If I accept the first point, I cannot accept the second without re-defining the meaning of "belief," at the very least. Similarly, if I accept the second point, I cannot accept the first without, again, somehow re-defining the meaning of "belief," at least. > -- Terry Cheers, -- Bosko Milekic bmilekic@unixdaemons.com bmilekic@FreeBSD.org P.S.: If you haven't read "GEB," I strongly recommend it... it's really a good book. :-) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020620230514.B38506>