Date: Mon, 07 Oct 2002 14:14:07 -0700 From: Terry Lambert <tlambert2@mindspring.com> To: Gary Thorpe <gathorpe79@yahoo.com> Cc: freebsd-arch@freebsd.org Subject: Re: Running independent kernel instances on dual-Xeon/E7500 system Message-ID: <3DA1F91F.F707826E@mindspring.com> References: <20021007173913.50425.qmail@web11207.mail.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Gary Thorpe wrote: > --- Terry Lambert <tlambert2@mindspring.com> wrote: > > Nate Lawson wrote: > > > My dismissiveness was due to anticipating the direction this was going, > > > which is nicely shown by the response below. In short, dedicated > > > processors for IO were used in the minicomputer days but are wasteful > > > nowadays when you have lightweight interrupts and/or polling when > > > appropriate. > > > > Yet, I keep running into employers who want to pay people to do > > exactly that, particularly for offloading network processing to > > one processor, and running applications on the other. > > Wouldn't this be "solved" by using thread affinity? No. As I pointed out, "netperf" is a single server process, and will not gain advantage from multiple CPUs, under any circumstances, and will only suffer performance degradation attributable to SMP related overhead, when running on an SMP system. Some of this overhead is the lack of process affinity for a given CPU; some of it is the network interrupts going to whichever CPU holds "giant", instead of going to a particular CPU; some of it is the use of interrupt threads, which can cause the processing to occur on a CPU other than the one the interrupt came in on; some of it is the NETISR processing; etc.. Basically, it's possible at this point to: 1) Receive packets on CPU #1 2) Run the interrupt thread processing on CPU #2 3) Run the NETISR on CPU #3 4) Run the user application on CPU #4 Even if you dealt with the CPU affinity issue for the process, and even if you dealt with it for the interrupt thread, there are still two (potentially) disjoint processing boundaries to cross, which *will* be statistically disjoint. Running the protocol processing from interrupt through completion (queueing of data on the sb_rcv) on a signel CPU is a significant win. > > Actually, NT proved that wiring particular interrupts to particular > > processors was the way to go -- that was one of the things they did > > to beat the Linux numbers in both the Netcraft and Ziff-Davis > > benchmarks... perfect symmetry isn't all that it's promised. > > I remember when I mentioned that some time ago and got > the general response that this setup is highly > specialized, inflexible, and probably not very useful > for a real-world server. The NT setup had multiple interfaces on the same wire, which is what makes this assessment accurate: it's not intrinsically correct to claim this, it's only correct to claim it on the Netcraft benchmark hardware configuration. For the Ziff-Davis configuration, this is an inaccurate claim. > People did point of that with > MORE cpus and/or MORE network adapters or some > combinations that is not n:n ratio, NT would not have > scaled well at all. How would NT compare to Tru64, > Solaris, AIX, or IRIX in a similar test? If there are more adapters than CPUs, and the load is not constant, you will get better performance by sharing the load, particularly if you are compute bound; though that load sharing would be best done by moving the unbalanced network interfaces around. If you have more CPUs than network interfaces, then you are in fact better off handling the interrupts from a particular adapter on a particular CPU, each time, instead of moving it around, to avoid destroying cache locality. > Do any of > these "hardwire" interrupts to particular cpus? I know that IRIX does, and that AIX does. I can't speak for Solaris in general, but in general, all NUMA machines process interrupts local to the processor cluster in which they occur. 8-). > I think what the original poster would want is > something like user-mode linux or vmware. I have had extensive off-list communications with the original poster. Without revealing anything not posted to the list, I can say that you are wrong: neither one of those would solve the problem. Some of this might be my assumption. It's true that seperating the hardware into logically seperate machines is one approach to the problem, and not the only possible approach, so there is some "If I have a hammer, it must be a nail" here. IMO, adding CPUs is a solution looking for a problem to be able to solve, in this case. > Aside from > machine emulation (via bochs and simular simulators), > does anything exist for FreeBSD which would allow you > to run seperate, independent environments? No. That's kind of what we were talking about implementing, with this thread. Actually, I was much more conservative with my own suggestions, in this regard: I think it would be useful for debugging purposes, and it would be less difficult than trying to implement something like "WINICE". It could *also* be useful for satisfying bosses who are under the impression that you should be able to multiply the performance of a box with one CPU by the number of CPUs you have in it after you upgrade it, and come out with a rough approximation of the performance. In a general sense, though, the idea of being able to run certain higher priority tasks on one CPU, and lower priority work on another, so that if the lower priority work gets overloaded, at least the high priority work does not end up with degraded performance, is a good one. Normally, you would handle this by having hard RT. PC hardware, unfortunately, is incapable of handling hard RT tasks, other than a single hard RT task per platform, due to its design; but you should be able to have at least *one* hard RT task. LinuxRT deals with this by having a hard RT executive, that runs the Linux kernel as one of its tasks, and assigns resources. Doing something similar on a two processor box in FreeBSD, without needing the RT executive because you have an extra CPU, is not that much of an intuitive leap, I think. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3DA1F91F.F707826E>