Date: Thu, 1 Jul 1999 01:33:12 +0000 (GMT) From: Terry Lambert <tlambert@primenet.com> To: dillon@apollo.backplane.com (Matthew Dillon) Cc: adsharma@home.com, freebsd-smp@FreeBSD.ORG Subject: Re: BSDI Lazy threading Message-ID: <199907010133.SAA14411@usr09.primenet.com> In-Reply-To: <199907010051.RAA41944@apollo.backplane.com> from "Matthew Dillon" at Jun 30, 99 05:51:06 pm
next in thread | previous in thread | raw e-mail | index | archive | help
> Proposed (w/ kernel threads): > > Yes, they will be allowed to sleep but the idea is for it to only > occur in special (and not oft-occuring) situations. I would strongly recommend _not_ implementing the BSDI soloution, at least not without further investigation of alternatives. I'm actually rather taken with the NT approach to achieving better network performance; those guys are sneaky. What they do is to, with multiple CPU's and multiple Ethernet cards, bind the interrupts for each card to a different processor. > What we do currently is attempt to disable interrupts in mainline code > to avoid contention. Theoretically such disablements are not supposed > to occur for long periods of time but the reality is that they often do, > especially for network-related things. One advantage of moving interrupts > to kernel threads is that the latency issue can be more easily managed. > Rather then disable an interrupt entirely we instead allow the interrupt > thread to preempt the kernel and then block if necessary. The advantage to this approach, which makes it appear attractive, is the ability to have multiple simultaneous interrupt stacks; however, I think that it's possible to have an implementation that has this capability, _without_ needing kernel threads to implement it. I think that what you are wanting to fix is actually the NETISR concurrency and latency issues. I think the threads approach addresses concurrency, but fails to address latency. Again, per my previous posting in the async call gates thread, one could imagine a trivial implementation using a 32 bit service mask, with one bit per CPU, in order to hand out interrupts. A per CPU interrupt stack would be enough to deal with the issue, without needing kernel threads. One real problem is loss of determinism that occurs with kernel thread scheduling, unless you greatly complicate the implementation of the scheduler. I think kernel threads are a really bad idea. People use threads in user space programs, either because they lack the asynchronous primitives necessary to get concurrency, or because they are lazy thinkers, and don't want to deal with the issue of organizing their data structures so that there can be multiple instances of them (i.e. migrating their context out of global variables and off the stack). I think that there is sufficient brain-power available that it's not necessary to do this to the kernel to achieve the performance goals. In the limit, multiple simultaneous asynchronous operations beat kernel threads, due to allocation, deallocation, context, and scheduling overhead that exist with threads, but not with asynchronous operations. In this case, because you *are* the kernel, there's not the same SMP scalability issue -- that issue only exists when migrating CPU's from kernel to user space. Give fault or interrupt based kernel entry, these issues don't exist for kernel code. > The manageability here is that the interrupt thread can now > explicitly check to see if it would block and, if it is a > really critical interrupt, can do something about it. I see > this as a big advantage because it would allow us to run the > most critical interrupts without any real latency at all(1). > The serial and keyboard interrupts come to mind. > > note(1): cavet: interrupts must often be truely disabled when I/O to > non-DMA IDE drives is occuring due to bugs in many IDE controllers. > > The other advantage of running interrupts in threads is that you can > run several interrupts simultaniously in an SMP system. For example, > a gigabit ethernet device internet would be able to run concurrently > with the TCP stack. You should be able to do this anyway, without resorting to threads. The only real issue that has prevented the (trivial) code necessary to implement this is the potential for inter-handler resource contention, which is a pending problem with the more complicated threads approach as well. This is mostly a driver architecture problem for individual drivers servicing multiple cards from a common (instead of seperately allocated) driver-global context. I also _seriously_ dislike the idea that it's premissible to block in an interrupt handler under any circumstances, especially if it isn't a panic condition to block before ack'ing the interrupt that got you there. 8-(. You need to ask Julian about the Cyrix 55xx chips, and what you have to do to make sure you don't lose an interrupt under some perverse circumstances. Blocking prior to ack'ing the interrupt would _greatly_ exacerbate the problem. I don't know about the Alpha (I haven't cracked my architecture manual for a year), but long experience with the VAX tells me that it would be Bad(tm) to leave a high priority interrupt un-ack'ed on the theory that you could service a lower priority interrupt (i.e. you're not going to get one). The BSDI approach precludes operation on some architectures (I assume that you will want to use the same implementation in UP kernels for the same concurrency reasons; I think a better approach to UP concurrency issues is to -- finally -- resolve the kernel preeemption problem, once and for all). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199907010133.SAA14411>