Date: Tue, 29 Apr 2003 12:41:29 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: John Baldwin <jhb@FreeBSD.org> Cc: freebsd-arch@FreeBSD.ORG Subject: Re: lots of malloc(M_WAITOK)'s in interrupt context from camisr Message-ID: <3EAED569.193E333E@mindspring.com> References: <XFMail.20030429140451.jhb@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
John Baldwin wrote: > On 29-Apr-2003 Archie Cobbs wrote: > > [ moving this followup to -arch ] > > > > Random thought.. it's always seemed unnatural to me to say that > > interrupt threads can't sleep. > > > > Why couldn't the system be designed so that if an interrupt thread > > tried to sleep, it would actually sleep but atomically (a) detach > > itself from the interrupt and (b) spawn a new thread to handle future > > interrupts. I.e., sleep with "on demand" additional interrupt thread > > creation. [ ... ] > If you need to do more work in your interrupt routine than just wakeups > and dinking with registers, you can always wake up a software interrupt > handler or some other random kthread to do things that take a long amount > of time. Sleeping in an interrupt thread would destroy interrupt latency > far worse than it is now. I'm sure we can all agree that that would be > unacceptable. Rather than making the interrupt thread implementation > very complex with magical spawning kthreads and such, I would prefer that > driver authors kick up software interrupt threads and the like on their > own and keep the ithread implementation simple. Raising a software interrupt to handle the work presents its own set of problems; for example, one of the biggest factors in TCP performance is the latency between the interrupt for the packets, and the running of the NETISR code. There are knobs to get rid of this, but they are not on by default. Adding soft interrupts also presents the possibility that the soft interrupt thread will be scheduled on a CPU other than the one that took the hard interrupt. If this occurs, then every interrupt is going to result in a cache-bust and a TLB hit, so unless you can guarantee affinity between the hard interrupt CPU and the soft interrupt thread, you are pretty screwed. This is currently also a factor in TCP performance on SMP machines, when you are using NETISR, which can effectively cause the mbufs with the data to CPU-hop during protocol processing. IMO, it's much better to run interrupts as far to completion as possible. The Jeff Mogul paper is instructive here (seperating between hard and soft interrupt processing is the main recipe for receiver livelock); so are the Peter Druschel/Mohit Aron/Rice University papers on the SCALA Server Project. I think that to effectively handle, for example, disk interrupts for an NFS server the way you are suggesting would require that a lot of the disk I/O subsystem be changed to support the moral equivalent of the DEVICE_POLLING interface in the network code; that's not really worth it. Better to do the work when the hardware asks you to do it, than to pray for rain. You are still screwed on user space process CPU starvation, no matter what you do, unless you go to something like weighted fair share queueing, and the kernel threads participate in the scheduling process as priority-lending peers for the user space code. That's really complicated to implement, although QLinux seems to have been able to do it with a fairly small team (who are, admittedly, professor-level professional OS researchers). -- Terry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3EAED569.193E333E>