Date: Wed, 02 Apr 2003 06:05:18 -0800 From: Terry Lambert <tlambert2@mindspring.com> To: Alexander Leidinger <Alexander@Leidinger.net> Cc: current@freebsd.org Subject: Re: libthr and 1:1 threading. Message-ID: <3E8AEE1E.AB0583B0@mindspring.com> References: <20030402070515.40396.qmail@web41803.mail.yahoo.com> <20030402132252.23f4e6f3.Alexander@Leidinger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Alexander Leidinger wrote: > On Tue, 01 Apr 2003 23:28:01 -0800 > Terry Lambert <tlambert2@mindspring.com> wrote: > > The primary performance reasoning behind a 1:1 kernel threading > > implementation, relative to the user space single kernel entry > > scheduler in the libc_r implementation is SMP scalability for > > threaded applications. > > I think Jeff (or someone else?) said, that some web browsers gain > "something" too (serialization issues with libc_r)? I had the impression > that this also applies to UP systems. > > Do I misremember this? If not, does it not apply to UP systems as well? FWIW: the libc_r reentrancy isn't fixed by a 1:1 model for anything but calls for which there are no non-blocking alternative kernel APIs. That means you are saved on things like System V IPC interfaces (which you were saved on before, if you used my kqueue patch for System V IPC to turn it into events), but the file I/O, which is mostly what a browser is doing, is not significantly improved. And for calls for which entrancy must be serialized, like the resolver, you are stuck serializing in user space. There is potential for some additional interleaved I/O occurring, when the kernel is multiply entrant, rather than EWOULDBLOCK plus a call conversion using a non-blocking file descriptor, that's true. I'm not sure that anything is gained by this in the UP case, however. Oh, and your process gets to compete for quantum as if it were "number of threads" processes. This is actually wrong, if you are PTHREAD_SCOPE_PROCESS, and setting the other option, PTHREAD_SCOPE_SYSTEM, requires root priviledges (but is on by default in the libthr implementation, because a 1:1 model requires more sophisticated scheduler support). You can get this same "cheat for extra quantum" effect with libc_r by using the same root priviledge to "nice" your process to a higher priority relative to the default. The other issue with going from the user space model to a kernel 1:1 model is that you are trading additional context switch overhead, by way of not fully utilizing your quantum, when you go into the kernel to block, instead of doing the call conversion: when you go to sleep, you hand the remainder of your quantum back to the system in 1:1, whereas with the libc_r, you hand it to another of your own threads. This is also what the N:M addresses. This also makes you less efficient when it comes to context switch overhead, since without thread-in-group affinity, when you give up the quantum, you are likely to take a full context switch (%cr3 reload for a different process; for example, cron runs once a second, and there are other "standard" system processes which are far from idle). In other words, you only see a benefit for a lone threaded application on a relatively quiecent system, which is OK, I guess, for a microbenchmark, but pretty poor when it comes to real world systems, which tend to be more heavily loaded. The last time and of this difference in overhead between the models was measured in any meaningful way (AFAIK) was in an article in (November?) 2001 in: Parallel and Distributed Systems, IEEE Trans. on (T-PDS) Which is where most of the academic work on scheduling and load balancing for parallel and distributed systems takes place. Unfortunately, these papers are not available online, unless you are an IEEE member, who also subscribes to T-PDS. Maybe you can find them, if you work at IBM and have access to a good technical library (e.g. Almaden), or if you have access to the Berkeley or Stanford libraries, or some other good technical library. There could have been a more recent paper, starting about 8 months back, which I don't know about (I need to take a trip to the library myself ;^)). The bottom line is that the 1:1 model is primarily useful for improving SMP scalability at this time, and that the overhead tradeoffs that are there right now don't really favor it, IMO, in a UP system. FreeBSD's libc_r has been in developement for nearly a decade; it is a *very good* pthreads implementation, and the only places it falls down, really, are in SMP scalability, and some blocking system calls that aren't amenable to conversion. You'll note that the System V IPC interfaces are not supported in a threaded environment by the POSIX standard; most of these blocking cases aren't really a problem. User space serialization in things like the resolver, which only opens one socket to do lookups, and can only deal with a single response context are a cross everyone bears. Mostly this bothers no one, because all browsers cache responses, there is response locality, and it's not a problem unless you compile in support for multiple protocol families (IPv4 + IPv6 support in the same browser usually means waiting for an IPv6 lookup timeout, if the remote host is not an RFC compliant DNS server, and fails to return an immediate reject response, like it's supposed to do). If you "fix" that for libthr, you also "fix" it for libc_r. -- Terry
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E8AEE1E.AB0583B0>