From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 09:30:07 2006 Return-Path: X-Original-To: threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BA65116A407 for ; Mon, 3 Jul 2006 09:30:07 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5605443D45 for ; Mon, 3 Jul 2006 09:30:07 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id DE26D46C8F for ; Mon, 3 Jul 2006 05:30:06 -0400 (EDT) Date: Mon, 3 Jul 2006 10:30:06 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: threads@FreeBSD.org Message-ID: <20060703101554.Q26325@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 09:30:07 -0000 I know this has been discussed in the past, but I figured with 7.x trundling forward, it was time to think about it again. In benchmarks for many common applications and scenarios, libthr demonstrates significantly better performance over libpthread -- this is not a coincidence, as David Xu has invested a lot of time in improving libthr and application performance, and libthr has benefitted significantly from the last ten years of threading work on FreeBSD. libthr is also implemented across a larger number of our platforms, and is already libpthread on several. The first recommendation we make to MySQL and other heavy thread users is "Switch to libthr", which is suggestive, also! Likewise, I know David has worked hard to eliminate technical and standards obstacles that have been raised in the past. This e-mail is a strawman proposal, intended to raise discussion, and possibly lead to action. So the strawman proposal is: make libthr the default threading library on 7.x. A few questions given the proposal: - Are there technical features present in libpthread that aren't yet in libthr, and are required? In the past system/local thread support has been the complaint, but I believe that is now long fixed. This is useful regardless of a switch. - A further thought: it would be really nice to be able to benchmark the behavior of applications like Mozilla, Tor, KDE, etc, which also depend on threads, but which don't come with easy-to-use potted benchmarks (that I know of). Can we get some hands to help explore this issue? This is also useful regardless of a switch. - What mechanism should be used? Should libthr simply be compiled as libpthread, and libpthread as libkse, so that they can both be around and libkse can be used for benchmarking still? - Are there significant objections to doing this? BTW, the one technical concern I have with libthr is that under high contention kernel activity involving a small number of kernel locks, libkse implicitly reduces kernel lock contention by bounding the number of threads that can be in the run state in the kernel. If, for example, you have 32 threads, all performing accept() and then file descriptor I/O in a tight loop, libpthread will often perform significantly better, as libthr allows them all into the kernel at once. This is most likely a scheduler issue, but it's significantly mitigated by M:N threading. To experiment with this problem, which I may not have characterized, you can try out the src/tools/tools/netrate/{http,httpd} micro-benchmarks using the -t argument to force threaded instead of process-based behavior. This effect seems not to come out as a practical issue in macro-benchmarks I've tried, perhaps as they are generally tuned for Linux threading, which presumably faces the same problem as our 1:1 package. Exploring this issue is a strong argument to continue to support multiple thread libraries in the development branch, as you can't do a comparative evaluation if you don't have choices to compare! If we're going to think about this seriously for 7.0, increasing libthr exposure sooner rather than later is probably a good idea. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 11:03:17 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CB38516A70F for ; Mon, 3 Jul 2006 11:03:17 +0000 (UTC) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5006243D45 for ; Mon, 3 Jul 2006 11:03:17 +0000 (GMT) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (peter@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k63B3H2C070013 for ; Mon, 3 Jul 2006 11:03:17 GMT (envelope-from owner-bugmaster@freebsd.org) Received: (from peter@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k63B3FsU070008 for freebsd-threads@freebsd.org; Mon, 3 Jul 2006 11:03:16 GMT (envelope-from owner-bugmaster@freebsd.org) Date: Mon, 3 Jul 2006 11:03:16 GMT Message-Id: <200607031103.k63B3FsU070008@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: peter set sender to owner-bugmaster@freebsd.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 11:03:18 -0000 Current FreeBSD problem reports Critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- s [2005/01/26] threads/76690threads fork hang in child for (-lc_r & -lthr) 1 problem total. Serious problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/07/18] kern/20016 threads pthreads: Cannot set scheduling timer/Can s [2001/01/20] threads/24472threads libc_r does not honor SO_SNDTIMEO/SO_RCVT s [2001/01/25] threads/24632threads libc_r delicate deviation from libc in ha s [2001/11/26] bin/32295 threads pthread dont dequeue signals s [2002/02/01] threads/34536threads accept() blocks other threads o [2002/05/25] kern/38549 threads the procces compiled whith pthread stoppe s [2002/06/27] threads/39922threads [threads] [patch] Threaded applications e s [2003/03/02] threads/48856threads Setting SIGCHLD to SIG_IGN still leaves z s [2003/03/10] threads/49087threads Signals lost in programs linked with libc s [2004/03/15] kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/un o [2004/08/26] threads/70975threads unexpected and unreliable behaviour when o [2004/10/05] threads/72353threads Assertion fails in /usr/src/lib/libpthrea o [2004/10/07] threads/72429threads threads blocked in stdio (fgets, etc) are o [2004/10/21] threads/72953threads fork() unblocks blocked signals w/o PTHRE o [2004/12/19] threads/75273threads FBSD 5.3 libpthread (KSE) bug o [2004/12/21] threads/75374threads pthread_kill() ignores SA_SIGINFO flag s [2005/01/26] threads/76694threads fork cause hang in dup()/close() function o [2005/04/08] threads/79683threads svctcp_create() fails if multiple threads o [2005/04/28] threads/80435threads panic on high loads o [2005/05/19] threads/81258threads Thread specific data is sometimes assigne o [2005/07/22] threads/83914threads [libc] popen() doesn't work in static thr s [2005/08/02] threads/84483threads problems with devel/nspr and -lc_r on 4.x o [2005/08/20] threads/85160threads [libthr] [patch] libobjc + libpthread/lib p [2005/11/19] threads/89262threads [kernel] [patch] multi-threaded process h o [2005/12/12] threads/90278threads libthr, ULE and -current produces >100% W o [2006/01/03] kern/91266 threads [threads] Trying sleep, but thread marked s [2006/03/15] threads/94467threads send(), sendto() and sendmsg() are not co o [2006/06/01] threads/98256threads gnome-system-monitor core dumps from pthr 28 problems total. Non-critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- s [2000/06/13] kern/19247 threads uthread_sigaction.c does not do anything s [2000/10/21] kern/22190 threads A threaded read(2) from a socketpair(2) f s [2001/09/09] threads/30464threads pthread mutex attributes -- pshared s [2002/05/02] threads/37676threads libc_r: msgsnd(), msgrcv(), pread(), pwri s [2002/07/16] threads/40671threads pthread_cancel doesn't remove thread from s [2004/07/13] threads/69020threads pthreads library leaks _gc_mutex o [2004/09/21] threads/71966threads Mlnet Core Dumped : Fatal error '_pq_inse o [2004/11/21] threads/74180threads KSE problem. Applications those riched ma o [2005/04/13] threads/79887threads [patch] freopen() isn't thread-safe o [2005/05/13] threads/80992threads abort() sometimes not caught by gdb depen o [2005/05/26] threads/81534threads [libc_r] [patch] libc_r close() will fail 11 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 11:48:45 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7052116A4A0; Mon, 3 Jul 2006 11:48:45 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0CD6444368; Mon, 3 Jul 2006 11:48:44 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63Bmh5F026626; Mon, 3 Jul 2006 07:48:44 -0400 (EDT) Date: Mon, 3 Jul 2006 07:48:43 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Robert Watson In-Reply-To: <20060703101554.Q26325@fledge.watson.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 11:48:45 -0000 On Mon, 3 Jul 2006, Robert Watson wrote: > > I know this has been discussed in the past, but I figured with 7.x trundling > forward, it was time to think about it again. In benchmarks for many common > applications and scenarios, libthr demonstrates significantly better > performance over libpthread -- this is not a coincidence, as David Xu has > invested a lot of time in improving libthr and application performance, and > libthr has benefitted significantly from the last ten years of threading work > on FreeBSD. libthr is also implemented across a larger number of our > platforms, and is already libpthread on several. The first recommendation we > make to MySQL and other heavy thread users is "Switch to libthr", which is > suggestive, also! Likewise, I know David has worked hard to eliminate > technical and standards obstacles that have been raised in the past. This > e-mail is a strawman proposal, intended to raise discussion, and possibly > lead to action. > > So the strawman proposal is: make libthr the default threading library on > 7.x. A few questions given the proposal: > > - Are there technical features present in libpthread that aren't yet in > libthr, and are required? In the past system/local thread support has been > the complaint, but I believe that is now long fixed. This is useful > regardless of a switch. Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling (hopefully not under the restriction that you are a privileged user). If you can those in libthr, I have no objection. However, these are not as easy to do in 1:1. -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:01:18 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4131C16A412; Mon, 3 Jul 2006 12:01:18 +0000 (UTC) (envelope-from jb@what-creek.com) Received: from what-creek.com (what-creek.com [66.111.37.70]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF4C843D5A; Mon, 3 Jul 2006 12:01:16 +0000 (GMT) (envelope-from jb@what-creek.com) Received: by what-creek.com (Postfix, from userid 102) id E6CED78C1D; Mon, 3 Jul 2006 12:01:13 +0000 (GMT) Date: Mon, 3 Jul 2006 12:01:13 +0000 From: John Birrell To: Daniel Eischen Message-ID: <20060703120113.GA24614@what-creek.com> References: <20060703101554.Q26325@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:01:18 -0000 On Mon, Jul 03, 2006 at 07:48:43AM -0400, Daniel Eischen wrote: > Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT > mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > (hopefully not under the restriction that you are a privileged user). How important are those relative to having libpthread work on other architectures? Are there any plans to get libpthread working on the other architectures? -- John Birrell From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:08:58 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DB47B16A416; Mon, 3 Jul 2006 12:08:58 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id B75CE43D5E; Mon, 3 Jul 2006 12:08:57 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63C8uVZ014631; Mon, 3 Jul 2006 08:08:56 -0400 (EDT) Date: Mon, 3 Jul 2006 08:08:56 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: John Birrell In-Reply-To: <20060703120113.GA24614@what-creek.com> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <20060703120113.GA24614@what-creek.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:08:59 -0000 On Mon, 3 Jul 2006, John Birrell wrote: > On Mon, Jul 03, 2006 at 07:48:43AM -0400, Daniel Eischen wrote: >> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT >> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling >> (hopefully not under the restriction that you are a privileged user). > > How important are those relative to having libpthread work on other > architectures? It's not so much getting libpthread working on the other architectures (sparc64 is the only Tier 1 (is Sparc64 Tier 1 yet?) that libpthread doesn't work on). It's being able to support the POSIX standard. That was the goal years ago when we started the process of designing a new thread library. I maintain that we have to be able to support the standard, if we can't then it's a not a good design. > Are there any plans to get libpthread working on the other architectures? Someone with sparc64-fu will have to do some grunt work, but probably some can be grok'd from NetBSD (which also uses an SA approach). -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:13:04 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 2507416A416; Mon, 3 Jul 2006 12:13:04 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org Date: Mon, 3 Jul 2006 20:12:40 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> In-Reply-To: <20060703101554.Q26325@fledge.watson.org> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200607032012.40374.davidxu@freebsd.org> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:13:04 -0000 On Monday 03 July 2006 17:30, Robert Watson wrote: > If we're going to think about this seriously for 7.0, increasing libthr > exposure sooner rather than later is probably a good idea. > > Robert N M Watson > Computer Laboratory > University of Cambridge I think libthr should be default thread library. David Xu From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:13:04 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 2507416A416; Mon, 3 Jul 2006 12:13:04 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org Date: Mon, 3 Jul 2006 20:12:40 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> In-Reply-To: <20060703101554.Q26325@fledge.watson.org> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200607032012.40374.davidxu@freebsd.org> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:13:04 -0000 On Monday 03 July 2006 17:30, Robert Watson wrote: > If we're going to think about this seriously for 7.0, increasing libthr > exposure sooner rather than later is probably a good idea. > > Robert N M Watson > Computer Laboratory > University of Cambridge I think libthr should be default thread library. David Xu From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:16:53 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D1C7E16A40F; Mon, 3 Jul 2006 12:16:53 +0000 (UTC) (envelope-from jb@what-creek.com) Received: from what-creek.com (what-creek.com [66.111.37.70]) by mx1.FreeBSD.org (Postfix) with ESMTP id A01E143E1A; Mon, 3 Jul 2006 12:16:49 +0000 (GMT) (envelope-from jb@what-creek.com) Received: by what-creek.com (Postfix, from userid 102) id 4F77578C1D; Mon, 3 Jul 2006 12:16:47 +0000 (GMT) Date: Mon, 3 Jul 2006 12:16:47 +0000 From: John Birrell To: Daniel Eischen Message-ID: <20060703121647.GA24781@what-creek.com> References: <20060703101554.Q26325@fledge.watson.org> <20060703120113.GA24614@what-creek.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:16:53 -0000 On Mon, Jul 03, 2006 at 08:08:56AM -0400, Daniel Eischen wrote: > It's not so much getting libpthread working on the other architectures > (sparc64 is the only Tier 1 (is Sparc64 Tier 1 yet?) that libpthread > doesn't work on). It's being able to support the POSIX standard. > That was the goal years ago when we started the process of designing > a new thread library. I maintain that we have to be able to support > the standard, if we can't then it's a not a good design. Right now I would settle for performance matching Linux before trecking to the end of the earth to comply with POSIX. Most Linux developers probably don't even know what POSIX is. -- John Birrell From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:20:34 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id E860016A49E; Mon, 3 Jul 2006 12:20:33 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Mon, 3 Jul 2006 20:20:10 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607032020.10993.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:20:34 -0000 On Monday 03 July 2006 19:48, Daniel Eischen wrote: > Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT > mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > (hopefully not under the restriction that you are a privileged user). > I would tell you don't implement system scope thread in libpthread, because system scope thread does not work in the way you said here, it seems you are telling user that the libpthread is fully working in the way, but the reality is not, without a correct kernel support, I don't think you should introduce system scope thread into libpthread, please remove this feautre if you think libpthread should work in the way. but reality is, mysql is using system scope in libpthread to get better performance but not process scope thread. in the real world, these nominal features are also not used widely, using it can only cause serious performance problem in most applications, until you need hard realtime, but why do you use FreeBSD if you need hard realtime ? there are dedicated systems which are designed for this requirement. I still does not find an application really need this feature. Also I don't think one can easily implement a modern CPU friendly scheduler in userland, HTT, shared L2 dual-core, NUMA scheduling, doing it in kernel is already extreme diffcult, putting lots of effort in single library but other single threaded application still gain nothing is also not worthy to work on M:N thread library. > If you can those in libthr, I have no objection. However, these > are not as easy to do in 1:1. it depends on whether one has interest to implement it, nothing is impossible here, kernel has already turnstile code, this already made a bit forward. David Xu From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:20:34 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id E860016A49E; Mon, 3 Jul 2006 12:20:33 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Mon, 3 Jul 2006 20:20:10 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607032020.10993.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:20:34 -0000 On Monday 03 July 2006 19:48, Daniel Eischen wrote: > Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT > mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > (hopefully not under the restriction that you are a privileged user). > I would tell you don't implement system scope thread in libpthread, because system scope thread does not work in the way you said here, it seems you are telling user that the libpthread is fully working in the way, but the reality is not, without a correct kernel support, I don't think you should introduce system scope thread into libpthread, please remove this feautre if you think libpthread should work in the way. but reality is, mysql is using system scope in libpthread to get better performance but not process scope thread. in the real world, these nominal features are also not used widely, using it can only cause serious performance problem in most applications, until you need hard realtime, but why do you use FreeBSD if you need hard realtime ? there are dedicated systems which are designed for this requirement. I still does not find an application really need this feature. Also I don't think one can easily implement a modern CPU friendly scheduler in userland, HTT, shared L2 dual-core, NUMA scheduling, doing it in kernel is already extreme diffcult, putting lots of effort in single library but other single threaded application still gain nothing is also not worthy to work on M:N thread library. > If you can those in libthr, I have no objection. However, these > are not as easy to do in 1:1. it depends on whether one has interest to implement it, nothing is impossible here, kernel has already turnstile code, this already made a bit forward. David Xu From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:29:22 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF99216A407; Mon, 3 Jul 2006 12:29:22 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6055643DE5; Mon, 3 Jul 2006 12:29:18 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63CTHnN003380; Mon, 3 Jul 2006 08:29:17 -0400 (EDT) Date: Mon, 3 Jul 2006 08:29:17 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607032020.10993.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:29:22 -0000 On Mon, 3 Jul 2006, David Xu wrote: > On Monday 03 July 2006 19:48, Daniel Eischen wrote: > >> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT >> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling >> (hopefully not under the restriction that you are a privileged user). >> > > I would tell you don't implement system scope thread in libpthread, > because system scope thread does not work in the way you said here, > it seems you are telling user that the libpthread is fully working in > the way, but the reality is not, without a correct kernel support, > I don't think you should introduce system scope thread into libpthread, > please remove this feautre if you think libpthread should work in the way. I don't believe that system scope threads have to abide by SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling since their contention scope is different. > but reality is, mysql is using system scope in libpthread to get better > performance but not process scope thread. in the real world, these > nominal features are also not used widely, using it can only cause > serious performance problem in most applications, until you need hard > realtime, but why do you use FreeBSD if you need hard realtime ? > there are dedicated systems which are designed for this requirement. > I still does not find an application really need this feature. Also > I don't think one can easily implement a modern CPU friendly scheduler > in userland, HTT, shared L2 dual-core, NUMA scheduling, doing it in kernel is > already extreme diffcult, putting lots of effort in single library but other > single threaded application still gain nothing is also not worthy to work > on M:N thread library. This has no impact on libpthread. The KSE is the thing that the kernel knows about and it can keep threads running on the same KSE and assign KSEs to different CPUs, etc. All the threads library has to do is keep threads running on the same KSE. I've said this before, but you keep making the same argument ;-) >> If you can those in libthr, I have no objection. However, these >> are not as easy to do in 1:1. > > it depends on whether one has interest to implement it, nothing > is impossible here, kernel has already turnstile code, this already > made a bit forward. Are you going to do it? -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:29:22 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF99216A407; Mon, 3 Jul 2006 12:29:22 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6055643DE5; Mon, 3 Jul 2006 12:29:18 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63CTHnN003380; Mon, 3 Jul 2006 08:29:17 -0400 (EDT) Date: Mon, 3 Jul 2006 08:29:17 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607032020.10993.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:29:22 -0000 On Mon, 3 Jul 2006, David Xu wrote: > On Monday 03 July 2006 19:48, Daniel Eischen wrote: > >> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT >> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling >> (hopefully not under the restriction that you are a privileged user). >> > > I would tell you don't implement system scope thread in libpthread, > because system scope thread does not work in the way you said here, > it seems you are telling user that the libpthread is fully working in > the way, but the reality is not, without a correct kernel support, > I don't think you should introduce system scope thread into libpthread, > please remove this feautre if you think libpthread should work in the way. I don't believe that system scope threads have to abide by SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling since their contention scope is different. > but reality is, mysql is using system scope in libpthread to get better > performance but not process scope thread. in the real world, these > nominal features are also not used widely, using it can only cause > serious performance problem in most applications, until you need hard > realtime, but why do you use FreeBSD if you need hard realtime ? > there are dedicated systems which are designed for this requirement. > I still does not find an application really need this feature. Also > I don't think one can easily implement a modern CPU friendly scheduler > in userland, HTT, shared L2 dual-core, NUMA scheduling, doing it in kernel is > already extreme diffcult, putting lots of effort in single library but other > single threaded application still gain nothing is also not worthy to work > on M:N thread library. This has no impact on libpthread. The KSE is the thing that the kernel knows about and it can keep threads running on the same KSE and assign KSEs to different CPUs, etc. All the threads library has to do is keep threads running on the same KSE. I've said this before, but you keep making the same argument ;-) >> If you can those in libthr, I have no objection. However, these >> are not as easy to do in 1:1. > > it depends on whether one has interest to implement it, nothing > is impossible here, kernel has already turnstile code, this already > made a bit forward. Are you going to do it? -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:36:42 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 300A316A412; Mon, 3 Jul 2006 12:36:42 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9141C43E32; Mon, 3 Jul 2006 12:36:24 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 17BE246C61; Mon, 3 Jul 2006 08:36:23 -0400 (EDT) Date: Mon, 3 Jul 2006 13:36:23 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Daniel Eischen In-Reply-To: Message-ID: <20060703133454.L57091@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:36:42 -0000 On Mon, 3 Jul 2006, Daniel Eischen wrote: >> - Are there technical features present in libpthread that aren't yet in >> libthr, and are required? In the past system/local thread support has >> been >> the complaint, but I believe that is now long fixed. This is useful >> regardless of a switch. > > Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT mutexes, > and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling (hopefully not under > the restriction that you are a privileged user). > > If you can those in libthr, I have no objection. However, these are not as > easy to do in 1:1. Thanks for leeting me know. Other than thee above missing scheduling functionality, are you aware of any other missing or substantially non-functional features in libthr that are important to this discussion? Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 12:41:12 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 00D7816A407; Mon, 3 Jul 2006 12:41:12 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 363F643E6D; Mon, 3 Jul 2006 12:40:37 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63Cebj9014470; Mon, 3 Jul 2006 08:40:37 -0400 (EDT) Date: Mon, 3 Jul 2006 08:40:37 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Robert Watson In-Reply-To: <20060703133454.L57091@fledge.watson.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <20060703133454.L57091@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 12:41:12 -0000 On Mon, 3 Jul 2006, Robert Watson wrote: > > On Mon, 3 Jul 2006, Daniel Eischen wrote: > >>> - Are there technical features present in libpthread that aren't yet in >>> libthr, and are required? In the past system/local thread support has >>> been >>> the complaint, but I believe that is now long fixed. This is useful >>> regardless of a switch. >> >> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT >> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling (hopefully >> not under the restriction that you are a privileged user). >> >> If you can those in libthr, I have no objection. However, these are not as >> easy to do in 1:1. > > Thanks for leeting me know. Other than thee above missing scheduling > functionality, are you aware of any other missing or substantially > non-functional features in libthr that are important to this discussion? No, I think those are what libthr lacks in supporting POSIX. I think the problem will be getting our 3 kernel schedulers to support them. -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 13:25:48 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 420D416A415; Mon, 3 Jul 2006 13:25:48 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Mon, 3 Jul 2006 21:25:25 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <20060703133454.L57091@fledge.watson.org> In-Reply-To: MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200607032125.26156.davidxu@freebsd.org> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 13:25:48 -0000 On Monday 03 July 2006 20:40, Daniel Eischen wrote: > No, I think those are what libthr lacks in supporting POSIX. > I think the problem will be getting our 3 kernel schedulers to > support them. it is mutex code and priority propagating which is already supported by turnstile code, in theory, it is not depended on scheduler. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 13:25:48 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 420D416A415; Mon, 3 Jul 2006 13:25:48 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Mon, 3 Jul 2006 21:25:25 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <20060703133454.L57091@fledge.watson.org> In-Reply-To: MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200607032125.26156.davidxu@freebsd.org> Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 13:25:48 -0000 On Monday 03 July 2006 20:40, Daniel Eischen wrote: > No, I think those are what libthr lacks in supporting POSIX. > I think the problem will be getting our 3 kernel schedulers to > support them. it is mutex code and priority propagating which is already supported by turnstile code, in theory, it is not depended on scheduler. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 13:29:46 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 9A6EE16A403; Mon, 3 Jul 2006 13:29:46 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Mon, 3 Jul 2006 21:29:23 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607032129.23754.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 13:29:47 -0000 On Monday 03 July 2006 20:29, Daniel Eischen wrote: > On Mon, 3 Jul 2006, David Xu wrote: > > On Monday 03 July 2006 19:48, Daniel Eischen wrote: > >> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT > >> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > >> (hopefully not under the restriction that you are a privileged user). > > > > I would tell you don't implement system scope thread in libpthread, > > because system scope thread does not work in the way you said here, > > it seems you are telling user that the libpthread is fully working in > > the way, but the reality is not, without a correct kernel support, > > I don't think you should introduce system scope thread into libpthread, > > please remove this feautre if you think libpthread should work in the > > way. > > I don't believe that system scope threads have to abide > by SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > since their contention scope is different. > using system native thread to get performance, using a second level scheduler hides and eliminates improvment made in kernel and throw away all hardwork done by kernel hacker. > This has no impact on libpthread. The KSE is the thing > that the kernel knows about and it can keep threads running > on the same KSE and assign KSEs to different CPUs, etc. > All the threads library has to do is keep threads running > on the same KSE. I've said this before, but you keep > making the same argument ;-) > you don't know, cpu soft affinity is not that simple, only kernel may guess what should be done, there is lots of detail, e.g, under heavy threaded IPC load, wake up a thread to local cpu will win, moving it to the original cpu the thread run will lost fresh cached data generated by current thread, in that load, every cpu has the application's hot code and static data structure cached, but new data is only available on the local cpu, I suggest you don't do that work in userland thread, it is a sort of time waste to repeat kernel scheduler function. > >> If you can those in libthr, I have no objection. However, these > >> are not as easy to do in 1:1. > > > > it depends on whether one has interest to implement it, nothing > > is impossible here, kernel has already turnstile code, this already > > made a bit forward. > > Are you going to do it? yes, but it is not that urgent. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 13:29:46 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 9A6EE16A403; Mon, 3 Jul 2006 13:29:46 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Mon, 3 Jul 2006 21:29:23 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607032129.23754.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 13:29:47 -0000 On Monday 03 July 2006 20:29, Daniel Eischen wrote: > On Mon, 3 Jul 2006, David Xu wrote: > > On Monday 03 July 2006 19:48, Daniel Eischen wrote: > >> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT > >> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > >> (hopefully not under the restriction that you are a privileged user). > > > > I would tell you don't implement system scope thread in libpthread, > > because system scope thread does not work in the way you said here, > > it seems you are telling user that the libpthread is fully working in > > the way, but the reality is not, without a correct kernel support, > > I don't think you should introduce system scope thread into libpthread, > > please remove this feautre if you think libpthread should work in the > > way. > > I don't believe that system scope threads have to abide > by SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > since their contention scope is different. > using system native thread to get performance, using a second level scheduler hides and eliminates improvment made in kernel and throw away all hardwork done by kernel hacker. > This has no impact on libpthread. The KSE is the thing > that the kernel knows about and it can keep threads running > on the same KSE and assign KSEs to different CPUs, etc. > All the threads library has to do is keep threads running > on the same KSE. I've said this before, but you keep > making the same argument ;-) > you don't know, cpu soft affinity is not that simple, only kernel may guess what should be done, there is lots of detail, e.g, under heavy threaded IPC load, wake up a thread to local cpu will win, moving it to the original cpu the thread run will lost fresh cached data generated by current thread, in that load, every cpu has the application's hot code and static data structure cached, but new data is only available on the local cpu, I suggest you don't do that work in userland thread, it is a sort of time waste to repeat kernel scheduler function. > >> If you can those in libthr, I have no objection. However, these > >> are not as easy to do in 1:1. > > > > it depends on whether one has interest to implement it, nothing > > is impossible here, kernel has already turnstile code, this already > > made a bit forward. > > Are you going to do it? yes, but it is not that urgent. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 13:44:13 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7FD4E16A407; Mon, 3 Jul 2006 13:44:13 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id B9FBA43D6E; Mon, 3 Jul 2006 13:44:12 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63DiBfp029948; Mon, 3 Jul 2006 09:44:11 -0400 (EDT) Date: Mon, 3 Jul 2006 09:44:11 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607032125.26156.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <20060703133454.L57091@fledge.watson.org> <200607032125.26156.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 13:44:13 -0000 On Mon, 3 Jul 2006, David Xu wrote: > On Monday 03 July 2006 20:40, Daniel Eischen wrote: > >> No, I think those are what libthr lacks in supporting POSIX. >> I think the problem will be getting our 3 kernel schedulers to >> support them. > > it is mutex code and priority propagating which is already > supported by turnstile code, in theory, it is not depended > on scheduler. Sure it is. SCHED_FIFO and SCHED_RR are scheduling attributes. Mutex code and priority propagation have nothing to do with this. -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 13:44:13 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7FD4E16A407; Mon, 3 Jul 2006 13:44:13 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id B9FBA43D6E; Mon, 3 Jul 2006 13:44:12 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k63DiBfp029948; Mon, 3 Jul 2006 09:44:11 -0400 (EDT) Date: Mon, 3 Jul 2006 09:44:11 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607032125.26156.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <20060703133454.L57091@fledge.watson.org> <200607032125.26156.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 13:44:13 -0000 On Mon, 3 Jul 2006, David Xu wrote: > On Monday 03 July 2006 20:40, Daniel Eischen wrote: > >> No, I think those are what libthr lacks in supporting POSIX. >> I think the problem will be getting our 3 kernel schedulers to >> support them. > > it is mutex code and priority propagating which is already > supported by turnstile code, in theory, it is not depended > on scheduler. Sure it is. SCHED_FIFO and SCHED_RR are scheduling attributes. Mutex code and priority propagation have nothing to do with this. -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 22:12:46 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id DFE7E16A403; Mon, 3 Jul 2006 22:12:45 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Daniel Eischen Date: Tue, 4 Jul 2006 06:12:23 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607032125.26156.davidxu@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607040612.23493.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 22:12:46 -0000 On Monday 03 July 2006 21:44, Daniel Eischen wrote: > On Mon, 3 Jul 2006, David Xu wrote: > > On Monday 03 July 2006 20:40, Daniel Eischen wrote: > >> No, I think those are what libthr lacks in supporting POSIX. > >> I think the problem will be getting our 3 kernel schedulers to > >> support them. > > > > it is mutex code and priority propagating which is already > > supported by turnstile code, in theory, it is not depended > > on scheduler. > > Sure it is. SCHED_FIFO and SCHED_RR are scheduling attributes. > Mutex code and priority propagation have nothing to do with > this. I have never said SCHED_FIFO and SCHED_RR is related to mutex, in fact, I am confused that you always said them at same time. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 3 22:12:46 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id DFE7E16A403; Mon, 3 Jul 2006 22:12:45 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Daniel Eischen Date: Tue, 4 Jul 2006 06:12:23 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607032125.26156.davidxu@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607040612.23493.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2006 22:12:46 -0000 On Monday 03 July 2006 21:44, Daniel Eischen wrote: > On Mon, 3 Jul 2006, David Xu wrote: > > On Monday 03 July 2006 20:40, Daniel Eischen wrote: > >> No, I think those are what libthr lacks in supporting POSIX. > >> I think the problem will be getting our 3 kernel schedulers to > >> support them. > > > > it is mutex code and priority propagating which is already > > supported by turnstile code, in theory, it is not depended > > on scheduler. > > Sure it is. SCHED_FIFO and SCHED_RR are scheduling attributes. > Mutex code and priority propagation have nothing to do with > this. I have never said SCHED_FIFO and SCHED_RR is related to mutex, in fact, I am confused that you always said them at same time. From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 05:53:37 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D005416A4DA; Tue, 4 Jul 2006 05:53:37 +0000 (UTC) (envelope-from peter@wemm.org) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60DFB43D46; Tue, 4 Jul 2006 05:53:37 +0000 (GMT) (envelope-from peter@wemm.org) Received: from fw.wemm.org (canning.wemm.org [192.203.228.65]) by canning.wemm.org (Postfix) with ESMTP id 3D8D52A8DF; Mon, 3 Jul 2006 22:53:37 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (overcee.wemm.org [10.0.0.3]) by fw.wemm.org (Postfix) with ESMTP id D3B78E2B3; Mon, 3 Jul 2006 22:53:36 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (localhost [127.0.0.1]) by overcee.wemm.org (8.13.4/8.13.4) with ESMTP id k645raih070401; Mon, 3 Jul 2006 22:53:36 -0700 (PDT) (envelope-from peter@wemm.org) Received: from localhost (localhost [[UNIX: localhost]]) by overcee.wemm.org (8.13.4/8.13.1/Submit) id k645ra39070400; Mon, 3 Jul 2006 22:53:36 -0700 (PDT) (envelope-from peter@wemm.org) X-Authentication-Warning: overcee.wemm.org: peter set sender to peter@wemm.org using -f From: Peter Wemm To: Daniel Eischen Date: Mon, 3 Jul 2006 22:53:35 -0700 User-Agent: KMail/1.8.1 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607032253.35982.peter@wemm.org> Cc: threads@freebsd.org, Robert Watson , davidxu@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 05:53:37 -0000 On Mon, 3 Jul 2006, Daniel Eischen wrote: > On Mon, 3 Jul 2006, David Xu wrote: > >> On Monday 03 July 2006 19:48, Daniel Eischen wrote: >> >>> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT >>> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling >>> (hopefully not under the restriction that you are a privileged >>> user). >>> >> >> I would tell you don't implement system scope thread in libpthread, >> because system scope thread does not work in the way you said here, >> it seems you are telling user that the libpthread is fully working in >> the way, but the reality is not, without a correct kernel support, >> I don't think you should introduce system scope thread into >> libpthread, please remove this feautre if you think libpthread should >> work in the way. > > I don't believe that system scope threads have to abide > by SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling > since their contention scope is different. It sounds like (by your definition) that switching to a libthr that only has system scope threads means we don't have to implement those modes, right? My interest is reducing the performance cost with critical applications that are optimized for the assumption of a linux-like system-scope-only threading model. -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 06:58:33 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8490B16A4E1 for ; Tue, 4 Jul 2006 06:58:33 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from nz-out-0102.google.com (nz-out-0102.google.com [64.233.162.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1CB6843D53 for ; Tue, 4 Jul 2006 06:58:33 +0000 (GMT) (envelope-from kip.macy@gmail.com) Received: by nz-out-0102.google.com with SMTP id 34so601204nzf for ; Mon, 03 Jul 2006 23:58:32 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=ESD9SYVYZuude274C6aFyKi6XJMy8qPaB2fclTi0PmVwUmMz6ufFLcffJs1TY69X6RzIqf5gkVsEsW0GpxwWoEpkLshQLD7BpziQcsrWkQXvWP6uU/ahIWCjCh++VNnclE/bQXrwVQeeWnxy2RYuI4cB5cGTLzDq+8k5baQoB2Y= Received: by 10.65.210.4 with SMTP id m4mr3874565qbq; Mon, 03 Jul 2006 23:58:32 -0700 (PDT) Received: by 10.65.225.9 with HTTP; Mon, 3 Jul 2006 23:58:32 -0700 (PDT) Message-ID: Date: Mon, 3 Jul 2006 23:58:32 -0700 From: "Kip Macy" To: freebsd-threads@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@fsmware.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 06:58:33 -0000 Hi, as some of you may know, I've been working on a port to the sun4v. This has motivated me to take a closer look at some of our locking. One glaring item is the existence of sched_lock that serializes context switches and thread state changes across all 32 logical cpus. I'm interested in adding somewhat finer-grained locking, but this is made more complicated by the kernel-side implementation of KSE. I find it worthy of note, that Sun dropped support for scheduler activations in Solaris because their engineers recognized that most significant applications are written with the assumption of linux-style system scope threads. Can someone please point me at real-world applications that are used to compare FreeBSD and Linux that rely on PROCESS_SCOPE and or general scheduler activation semantics? -Kip From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 13:08:40 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE50116A4DF; Tue, 4 Jul 2006 13:08:40 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 63DA343D45; Tue, 4 Jul 2006 13:08:40 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k64D8d2J011922; Tue, 4 Jul 2006 09:08:39 -0400 (EDT) Date: Tue, 4 Jul 2006 09:08:38 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607040612.23493.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <200607032125.26156.davidxu@freebsd.org> <200607040612.23493.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 13:08:40 -0000 On Tue, 4 Jul 2006, David Xu wrote: > On Monday 03 July 2006 21:44, Daniel Eischen wrote: >> On Mon, 3 Jul 2006, David Xu wrote: >>> On Monday 03 July 2006 20:40, Daniel Eischen wrote: >>>> No, I think those are what libthr lacks in supporting POSIX. >>>> I think the problem will be getting our 3 kernel schedulers to >>>> support them. >>> >>> it is mutex code and priority propagating which is already >>> supported by turnstile code, in theory, it is not depended >>> on scheduler. >> >> Sure it is. SCHED_FIFO and SCHED_RR are scheduling attributes. >> Mutex code and priority propagation have nothing to do with >> this. > > I have never said SCHED_FIFO and SCHED_RR is related to mutex, > in fact, I am confused that you always said them at same time. The question was what does libthr lack. The answer is priority inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and (in the future) SCHED_SPORADIC scheduling. That is what I stated earlier in this thread. -- DE From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 13:08:40 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE50116A4DF; Tue, 4 Jul 2006 13:08:40 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 63DA343D45; Tue, 4 Jul 2006 13:08:40 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k64D8d2J011922; Tue, 4 Jul 2006 09:08:39 -0400 (EDT) Date: Tue, 4 Jul 2006 09:08:38 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607040612.23493.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <200607032125.26156.davidxu@freebsd.org> <200607040612.23493.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 13:08:40 -0000 On Tue, 4 Jul 2006, David Xu wrote: > On Monday 03 July 2006 21:44, Daniel Eischen wrote: >> On Mon, 3 Jul 2006, David Xu wrote: >>> On Monday 03 July 2006 20:40, Daniel Eischen wrote: >>>> No, I think those are what libthr lacks in supporting POSIX. >>>> I think the problem will be getting our 3 kernel schedulers to >>>> support them. >>> >>> it is mutex code and priority propagating which is already >>> supported by turnstile code, in theory, it is not depended >>> on scheduler. >> >> Sure it is. SCHED_FIFO and SCHED_RR are scheduling attributes. >> Mutex code and priority propagation have nothing to do with >> this. > > I have never said SCHED_FIFO and SCHED_RR is related to mutex, > in fact, I am confused that you always said them at same time. The question was what does libthr lack. The answer is priority inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and (in the future) SCHED_SPORADIC scheduling. That is what I stated earlier in this thread. -- DE From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 13:20:13 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC4C216A4E0; Tue, 4 Jul 2006 13:20:12 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D50943D53; Tue, 4 Jul 2006 13:20:12 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k64DKAoZ024102; Tue, 4 Jul 2006 09:20:11 -0400 (EDT) Date: Tue, 4 Jul 2006 09:20:10 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Peter Wemm In-Reply-To: <200607032253.35982.peter@wemm.org> Message-ID: References: <200607032253.35982.peter@wemm.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , davidxu@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 13:20:13 -0000 On Mon, 3 Jul 2006, Peter Wemm wrote: > On Mon, 3 Jul 2006, Daniel Eischen wrote: >> On Mon, 3 Jul 2006, David Xu wrote: >> >>> On Monday 03 July 2006 19:48, Daniel Eischen wrote: >>> >>>> Yes, you have to support PTHREAD_PRIO_PROTECT, PTHREAD_PRIO_INHERIT >>>> mutexes, and SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling >>>> (hopefully not under the restriction that you are a privileged >>>> user). >>>> >>> >>> I would tell you don't implement system scope thread in libpthread, >>> because system scope thread does not work in the way you said here, >>> it seems you are telling user that the libpthread is fully working in >>> the way, but the reality is not, without a correct kernel support, >>> I don't think you should introduce system scope thread into >>> libpthread, please remove this feautre if you think libpthread should >>> work in the way. >> >> I don't believe that system scope threads have to abide >> by SCHED_RR, SCHED_FIFO, and SCHED_SPORADIC scheduling >> since their contention scope is different. > > It sounds like (by your definition) that switching to a libthr that only > has system scope threads means we don't have to implement those modes, > right? Actually, I confused contention scope with allocation domain. Threads with scope system contention scope have to compete with all other system scope threads regardless of the process' priority. On a single CPU system (allocation domain = 1), system scope threads with SCHED_RR, SCHED_FIFO, and (if supported) SCHED_SPORADIC scheduling attributes need to be scheduled accordingly. -- DE From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 14:05:17 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 530EE16A4E1; Tue, 4 Jul 2006 14:05:16 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Tue, 4 Jul 2006 22:04:52 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607040612.23493.davidxu@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607042204.52572.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 14:05:17 -0000 On Tuesday 04 July 2006 21:08, Daniel Eischen wrote: > The question was what does libthr lack. The answer is priority > inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and > (in the future) SCHED_SPORADIC scheduling. That is what I stated > earlier in this thread. As other people said, we need performance, these features, as you said, in the future, but I don't think it is more important than performance problem. you have to answer people what they should do when they bought two cpus but works like they only have one, as the major author of libpthread, in the past, you decided to keep silent, ignoring such requirement. also, the signal queue may not work reliably with libpthread, this nightmare appears again. From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 14:05:17 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 530EE16A4E1; Tue, 4 Jul 2006 14:05:16 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org, Daniel Eischen Date: Tue, 4 Jul 2006 22:04:52 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607040612.23493.davidxu@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607042204.52572.davidxu@freebsd.org> Cc: threads@freebsd.org, Robert Watson Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 14:05:17 -0000 On Tuesday 04 July 2006 21:08, Daniel Eischen wrote: > The question was what does libthr lack. The answer is priority > inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and > (in the future) SCHED_SPORADIC scheduling. That is what I stated > earlier in this thread. As other people said, we need performance, these features, as you said, in the future, but I don't think it is more important than performance problem. you have to answer people what they should do when they bought two cpus but works like they only have one, as the major author of libpthread, in the past, you decided to keep silent, ignoring such requirement. also, the signal queue may not work reliably with libpthread, this nightmare appears again. From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 19:42:13 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1002B16A4DD; Tue, 4 Jul 2006 19:42:13 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3A80643D58; Tue, 4 Jul 2006 19:42:08 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.37]) by a50.ironport.com with ESMTP; 04 Jul 2006 12:41:58 -0700 Message-ID: <44AAC47F.2040508@elischer.org> Date: Tue, 04 Jul 2006 12:41:51 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Xu References: <20060703101554.Q26325@fledge.watson.org> <200607040612.23493.davidxu@freebsd.org> <200607042204.52572.davidxu@freebsd.org> In-Reply-To: <200607042204.52572.davidxu@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 19:42:13 -0000 David Xu wrote: >On Tuesday 04 July 2006 21:08, Daniel Eischen wrote: > > > >>The question was what does libthr lack. The answer is priority >>inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and >>(in the future) SCHED_SPORADIC scheduling. That is what I stated >>earlier in this thread. >> >> > >As other people said, we need performance, these features, as you >said, in the future, but I don't think it is more important than performance >problem. you have to answer people what they should do when they bought >two cpus but works like they only have one, as the major author of libpthread, >in the past, you decided to keep silent, ignoring such requirement. >also, the signal queue may not work reliably with libpthread, this >nightmare appears again. > > As much as it pains me to say it, we could do with looking at using the simpler mode of 1:1 as the default. M:N does work but it turns out that many of the promissed advantages turn out to be phantoms due to the complexities of actually implementing it. The main problem that occurs again and again is that the kernel has no history of how a particular thread ran, which is important for our scheduer, and that there are other places where it would be advantageous for the kernel to have visibility into the whole pool of threads. There is actually an answer for the 1st problem that I have played with but there continue to be complexities introduced that make it difficult. For example, keeping statistics alligned woth threads and reporting them for the correct thread is exteremely difficult. THIS MUST NOT be confused with the complexities introduced by trying to make threads run fairly. Some of the percieved advantages of going to 1:1 are in fact only achievable if we abandon fair scheduling, which is a completely orthogonal issue. The complexity that introduce the extra locking in the schedular that Kip refers to, is the attempt to do fair scheduling and is independent of 1:1 or M:N. If someone could work out a way that fair scheduling could be achieved without the extra locking then that would be valid for both M:N and 1:1 THREADING, and would solve Kip's problem. > > > > > >_______________________________________________ >freebsd-threads@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-threads >To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > > From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 19:42:13 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1002B16A4DD; Tue, 4 Jul 2006 19:42:13 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3A80643D58; Tue, 4 Jul 2006 19:42:08 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.37]) by a50.ironport.com with ESMTP; 04 Jul 2006 12:41:58 -0700 Message-ID: <44AAC47F.2040508@elischer.org> Date: Tue, 04 Jul 2006 12:41:51 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Xu References: <20060703101554.Q26325@fledge.watson.org> <200607040612.23493.davidxu@freebsd.org> <200607042204.52572.davidxu@freebsd.org> In-Reply-To: <200607042204.52572.davidxu@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 19:42:13 -0000 David Xu wrote: >On Tuesday 04 July 2006 21:08, Daniel Eischen wrote: > > > >>The question was what does libthr lack. The answer is priority >>inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and >>(in the future) SCHED_SPORADIC scheduling. That is what I stated >>earlier in this thread. >> >> > >As other people said, we need performance, these features, as you >said, in the future, but I don't think it is more important than performance >problem. you have to answer people what they should do when they bought >two cpus but works like they only have one, as the major author of libpthread, >in the past, you decided to keep silent, ignoring such requirement. >also, the signal queue may not work reliably with libpthread, this >nightmare appears again. > > As much as it pains me to say it, we could do with looking at using the simpler mode of 1:1 as the default. M:N does work but it turns out that many of the promissed advantages turn out to be phantoms due to the complexities of actually implementing it. The main problem that occurs again and again is that the kernel has no history of how a particular thread ran, which is important for our scheduer, and that there are other places where it would be advantageous for the kernel to have visibility into the whole pool of threads. There is actually an answer for the 1st problem that I have played with but there continue to be complexities introduced that make it difficult. For example, keeping statistics alligned woth threads and reporting them for the correct thread is exteremely difficult. THIS MUST NOT be confused with the complexities introduced by trying to make threads run fairly. Some of the percieved advantages of going to 1:1 are in fact only achievable if we abandon fair scheduling, which is a completely orthogonal issue. The complexity that introduce the extra locking in the schedular that Kip refers to, is the attempt to do fair scheduling and is independent of 1:1 or M:N. If someone could work out a way that fair scheduling could be achieved without the extra locking then that would be valid for both M:N and 1:1 THREADING, and would solve Kip's problem. > > > > > >_______________________________________________ >freebsd-threads@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-threads >To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > > From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 23:06:57 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id E502A16A4DF; Tue, 4 Jul 2006 23:06:56 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Julian Elischer Date: Wed, 5 Jul 2006 07:06:33 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> In-Reply-To: <44AAC47F.2040508@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607050706.33502.davidxu@freebsd.org> Cc: Daniel Eischen , threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 23:06:57 -0000 On Wednesday 05 July 2006 03:41, Julian Elischer wrote: > As much as it pains me to say it, we could do with looking at using the > simpler mode of 1:1 > as the default. M:N does work but it turns out that many of the > promissed advantages turn out to be > phantoms due to the complexities of actually implementing it. The main > problem that occurs > again and again is that the kernel has no history of how a particular > thread ran, which is important > for our scheduer, and that there are other places where it would be > advantageous for the kernel > to have visibility into the whole pool of threads. There is actually an > answer for the 1st problem that > I have played with but there continue to be complexities introduced that > make it difficult. For example, > keeping statistics alligned woth threads and reporting them for the > correct thread is exteremely difficult. > This painful and complexies thing really should be avoided, the two level scheduler will always lose some cached information. > THIS MUST NOT be confused with the complexities introduced by trying to > make threads run fairly. > Some of the percieved advantages of going to 1:1 are in fact only > achievable if we abandon fair > scheduling, which is a completely orthogonal issue. > > > The complexity that introduce the extra locking in the schedular that > Kip refers to, > is the attempt to do fair scheduling and is independent of 1:1 or M:N. > If someone could work out a way that fair scheduling could be achieved > without > the extra locking then that would be valid for both M:N and 1:1 THREADING, > and would solve Kip's problem. > Cann't solaris's user usage scheduler will solve the problem ? if I recall it correctly, the decay factor in 4bsd scheduler can be changed dynamically for a thread, the decay factor can be stored in thread structure, and timeslice can be dynamically changed, also can be in thread structure, unlike current implementation, timeslice is fixed, this may make fair scheduling possible without ksegrp. Also will the roundrobin callout preempt a thread ? if that's true, then when every timeslice tick, a thread exhausted its timeslice is put at head of runqueue, isn't it a bug ? the callout is somewhat random for a thread, might not be a problem... David Xu From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 23:06:57 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id E502A16A4DF; Tue, 4 Jul 2006 23:06:56 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Julian Elischer Date: Wed, 5 Jul 2006 07:06:33 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> In-Reply-To: <44AAC47F.2040508@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607050706.33502.davidxu@freebsd.org> Cc: Daniel Eischen , threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 23:06:57 -0000 On Wednesday 05 July 2006 03:41, Julian Elischer wrote: > As much as it pains me to say it, we could do with looking at using the > simpler mode of 1:1 > as the default. M:N does work but it turns out that many of the > promissed advantages turn out to be > phantoms due to the complexities of actually implementing it. The main > problem that occurs > again and again is that the kernel has no history of how a particular > thread ran, which is important > for our scheduer, and that there are other places where it would be > advantageous for the kernel > to have visibility into the whole pool of threads. There is actually an > answer for the 1st problem that > I have played with but there continue to be complexities introduced that > make it difficult. For example, > keeping statistics alligned woth threads and reporting them for the > correct thread is exteremely difficult. > This painful and complexies thing really should be avoided, the two level scheduler will always lose some cached information. > THIS MUST NOT be confused with the complexities introduced by trying to > make threads run fairly. > Some of the percieved advantages of going to 1:1 are in fact only > achievable if we abandon fair > scheduling, which is a completely orthogonal issue. > > > The complexity that introduce the extra locking in the schedular that > Kip refers to, > is the attempt to do fair scheduling and is independent of 1:1 or M:N. > If someone could work out a way that fair scheduling could be achieved > without > the extra locking then that would be valid for both M:N and 1:1 THREADING, > and would solve Kip's problem. > Cann't solaris's user usage scheduler will solve the problem ? if I recall it correctly, the decay factor in 4bsd scheduler can be changed dynamically for a thread, the decay factor can be stored in thread structure, and timeslice can be dynamically changed, also can be in thread structure, unlike current implementation, timeslice is fixed, this may make fair scheduling possible without ksegrp. Also will the roundrobin callout preempt a thread ? if that's true, then when every timeslice tick, a thread exhausted its timeslice is put at head of runqueue, isn't it a bug ? the callout is somewhat random for a thread, might not be a problem... David Xu From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 23:24:42 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 22DED16A4DD; Tue, 4 Jul 2006 23:24:42 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Julian Elischer Date: Wed, 5 Jul 2006 07:24:18 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <44AAC47F.2040508@elischer.org> <200607050706.33502.davidxu@freebsd.org> In-Reply-To: <200607050706.33502.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607050724.18740.davidxu@freebsd.org> Cc: Daniel Eischen , Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 23:24:42 -0000 On Wednesday 05 July 2006 07:06, David Xu wrote: > Cann't solaris's user usage scheduler will solve the problem ? if I > recall it correctly, the decay factor in 4bsd scheduler can be changed > dynamically for a thread, the decay factor can be stored in thread > structure, and timeslice can be dynamically changed, also can be in thread > structure, unlike current implementation, timeslice is fixed, this may > make fair scheduling possible without ksegrp. > > Also will the roundrobin callout preempt a thread ? if that's true, then > when every timeslice tick, a thread exhausted its timeslice is put at head > of runqueue, isn't it a bug ? the callout is somewhat random for a thread, > might not be a problem... OK, I just digged out the paper in my disks, here is the reference: http://people.freebsd.org/~davidxu/doc/solaris2.pdf [the address thread@freebsd.org is removed] I think we really should do more work in kernel scheduler, and remove ksegrp. David Xu From owner-freebsd-threads@FreeBSD.ORG Tue Jul 4 23:28:20 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id CBFEB16A4E0; Tue, 4 Jul 2006 23:28:19 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-threads@freebsd.org Date: Wed, 5 Jul 2006 07:27:56 +0800 User-Agent: KMail/1.8.2 References: <20060703101554.Q26325@fledge.watson.org> <200607050706.33502.davidxu@freebsd.org> <200607050724.18740.davidxu@freebsd.org> In-Reply-To: <200607050724.18740.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607050727.57049.davidxu@freebsd.org> Cc: Daniel Eischen , Robert Watson , Julian Elischer Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2006 23:28:20 -0000 On Wednesday 05 July 2006 07:24, David Xu wrote: > OK, I just digged out the paper in my disks, here is the reference: > http://people.freebsd.org/~davidxu/doc/solaris2.pdf > > [the address thread@freebsd.org is removed] > sorry, it is still here, I post too many noise mails in the list already. :-( > I think we really should do more work in kernel scheduler, and remove > ksegrp. > > David Xu From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 01:19:08 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7BD0B16A4DD; Wed, 5 Jul 2006 01:19:08 +0000 (UTC) (envelope-from peter@wemm.org) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id D308043D4C; Wed, 5 Jul 2006 01:19:07 +0000 (GMT) (envelope-from peter@wemm.org) Received: from fw.wemm.org (canning.wemm.org [192.203.228.65]) by canning.wemm.org (Postfix) with ESMTP id 840C52A8E7; Tue, 4 Jul 2006 18:19:07 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (overcee.wemm.org [10.0.0.3]) by fw.wemm.org (Postfix) with ESMTP id 01C7CE2B5; Tue, 4 Jul 2006 18:19:06 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (localhost [127.0.0.1]) by overcee.wemm.org (8.13.4/8.13.4) with ESMTP id k651J6D0004330; Tue, 4 Jul 2006 18:19:06 -0700 (PDT) (envelope-from peter@wemm.org) Received: from localhost (localhost [[UNIX: localhost]]) by overcee.wemm.org (8.13.4/8.13.1/Submit) id k651J5DR004329; Tue, 4 Jul 2006 18:19:05 -0700 (PDT) (envelope-from peter@wemm.org) X-Authentication-Warning: overcee.wemm.org: peter set sender to peter@wemm.org using -f From: Peter Wemm To: freebsd-threads@freebsd.org Date: Tue, 4 Jul 2006 18:19:04 -0700 User-Agent: KMail/1.8.1 References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> In-Reply-To: <44AAC47F.2040508@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607041819.05510.peter@wemm.org> Cc: Daniel Eischen , threads@freebsd.org, Robert Watson , Julian Elischer , David Xu Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 01:19:08 -0000 On Tuesday 04 July 2006 12:41 pm, Julian Elischer wrote: > David Xu wrote: > >On Tuesday 04 July 2006 21:08, Daniel Eischen wrote: > >>The question was what does libthr lack. The answer is priority > >>inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and > >>(in the future) SCHED_SPORADIC scheduling. That is what I stated > >>earlier in this thread. > > > >As other people said, we need performance, these features, as you > >said, in the future, but I don't think it is more important than > > performance problem. you have to answer people what they should do > > when they bought two cpus but works like they only have one, as the > > major author of libpthread, in the past, you decided to keep > > silent, ignoring such requirement. also, the signal queue may not > > work reliably with libpthread, this nightmare appears again. > > As much as it pains me to say it, we could do with looking at using > the simpler mode of 1:1 > as the default. M:N does work but it turns out that many of the > promissed advantages turn out to be > phantoms due to the complexities of actually implementing it. At BSDCan, I tinkered with a checkout of the cvs tree, to see what the kernel side of things would look like if M:N support came out. The result is an amazing code clarity improvement and it enables a bunch of other optimizations to be done with greater ease. Things happen like being able to easily reduce the C code executed between an interrupt and ithread dispatch by about 75%. This simplification enabled Kip to do a bunch of scalability work as well (per-cpu scheduling locks, per-cpu process lists, etc). However, my objectives there were quite different to what Robert has raised. My objectives were a 'what if?'. People have complained in the past that the complexity that KSE adds to the kernel context switching code gets in the way of other optimizations that they'd like to try, so I figured that this would be a good way to call them on that and see if it really does help or not. I was hoping to be able to present a list of things that we'd gain as a result, but unfortunately the cat is out of a bag a bit earlier than I'd have liked. I never really intended to bring it up until there was something to show for it. I know Kip has done some amazing work already but I was hoping for other things as well before going public. FWIW, My skunkworks project is in perforce: //depot/projects/bike_sched/... and there is a live diff: http://people.freebsd.org/~peter/bike_sched.diff (Yes, the name was picked long before this thread started) It does NOT have any of Kip's optimization work in it. It was just meant as a baseline for other people to experiment with. I've tested it with 4bsd as the scheduler. ULE might work, but I have not tried it. SCHED_CORE will not compile in that tree because I haven't yet gone over diffs from David Xu yet. I run this code on my laptop with libmap.conf redirecting libpthread to libthr. It works very well for me, even threaded apps like firefox etc. Anyway, back to the subject at hand. The basic problem with the KSE/SA model as I see it (besides the kernel code complexity) is that it doesn't really seem to suit the kind of threaded applications that people seem to want to run on unix boxes. In a traditional 1:1 threading system, eg: linuxthreads/nptl, libthr, etc, mutex blocking is expensive, but system calls and blocking in kernel mode is the same cost as a regular process making system calls or blocking in kernel mode. Because Linux was the most widely and massively deployed threading system out there, people tended to write (or modify) their applications to work best with those assumptions. ie: keep pthread mutex blocking to an absolute minimum, and not care about kernel blocking. However, with the SA/KSE model, our tradeoffs are different. We implement pthread mutex blocking more quickly (except for UTS bugs that can make it far slower), but we make blocking in kernel context significantly higher cost than the 1:1 case, probably as much as double the cost. For applications that block in the kernel a lot instead of on mutexes, this is a big source of pain. When most of the applications that we're called to run are written with the linux behavior in mind, when our performance is compared against linux we're the ones that usually come off the worst. I'm sure that there are threaded applications that benefit from cheap mutex operations, but I'm not personally aware of them. I do know that the ones that we get regularly compared to linux with are the likes of mysql, squid and threaded http servers. All of those depend on kernel blocking being as fast as possible. Faster mutexes doesn't seem to compensate for the extra costs of kernel blocking. I don't know where java fits into this picture. We've proven that we can make KSE work, but it was far harder than we imagined, and unfortunately, the real-world apps that matter the most just don't seem to take advantage of it. Not to mention the complexity that we have to work around for scalability work. Speaking of scalability, 16 and 32 way systems are here already and will be common within 7.0's lifetime. If we don't scale, we're sunk. My gut tells me that we HAVE to address the complexity that the KSE kernel code adds in order to improve this. We can barely work well on 4-cpu systems, let alone 32 cpu systems. PS: I think it would be interesting to see a hybrid user level M:N system. Even if it was as simple as multiplexing user threads onto a group of kernel threads (without M:N kernel support) and doing libc_r style syscall wrappers for intercepting long-term blockable operations like socket/pipe IO etc. For short term blocking (disk IO), just wear the cost of letting one thread block for a moment. I suspect that large parts of libpthread could be reused and some bits brought back from libc_r. I think this would do a fairly decent job for things like computational threaded apps because mutexes would be really fast. PPS: My opinions are not meant as a criticism of the massive amount of work that has gone into making KSE work. It is more an attempt to step back and take an objective look at the ever-changing big picture. -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 01:19:08 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7BD0B16A4DD; Wed, 5 Jul 2006 01:19:08 +0000 (UTC) (envelope-from peter@wemm.org) Received: from canning.wemm.org (canning.wemm.org [192.203.228.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id D308043D4C; Wed, 5 Jul 2006 01:19:07 +0000 (GMT) (envelope-from peter@wemm.org) Received: from fw.wemm.org (canning.wemm.org [192.203.228.65]) by canning.wemm.org (Postfix) with ESMTP id 840C52A8E7; Tue, 4 Jul 2006 18:19:07 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (overcee.wemm.org [10.0.0.3]) by fw.wemm.org (Postfix) with ESMTP id 01C7CE2B5; Tue, 4 Jul 2006 18:19:06 -0700 (PDT) (envelope-from peter@wemm.org) Received: from overcee.wemm.org (localhost [127.0.0.1]) by overcee.wemm.org (8.13.4/8.13.4) with ESMTP id k651J6D0004330; Tue, 4 Jul 2006 18:19:06 -0700 (PDT) (envelope-from peter@wemm.org) Received: from localhost (localhost [[UNIX: localhost]]) by overcee.wemm.org (8.13.4/8.13.1/Submit) id k651J5DR004329; Tue, 4 Jul 2006 18:19:05 -0700 (PDT) (envelope-from peter@wemm.org) X-Authentication-Warning: overcee.wemm.org: peter set sender to peter@wemm.org using -f From: Peter Wemm To: freebsd-threads@freebsd.org Date: Tue, 4 Jul 2006 18:19:04 -0700 User-Agent: KMail/1.8.1 References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> In-Reply-To: <44AAC47F.2040508@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200607041819.05510.peter@wemm.org> Cc: Daniel Eischen , threads@freebsd.org, Robert Watson , Julian Elischer , David Xu Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 01:19:08 -0000 On Tuesday 04 July 2006 12:41 pm, Julian Elischer wrote: > David Xu wrote: > >On Tuesday 04 July 2006 21:08, Daniel Eischen wrote: > >>The question was what does libthr lack. The answer is priority > >>inheritence & protect mutexes, and also SCHED_FIFO, SCHED_RR, and > >>(in the future) SCHED_SPORADIC scheduling. That is what I stated > >>earlier in this thread. > > > >As other people said, we need performance, these features, as you > >said, in the future, but I don't think it is more important than > > performance problem. you have to answer people what they should do > > when they bought two cpus but works like they only have one, as the > > major author of libpthread, in the past, you decided to keep > > silent, ignoring such requirement. also, the signal queue may not > > work reliably with libpthread, this nightmare appears again. > > As much as it pains me to say it, we could do with looking at using > the simpler mode of 1:1 > as the default. M:N does work but it turns out that many of the > promissed advantages turn out to be > phantoms due to the complexities of actually implementing it. At BSDCan, I tinkered with a checkout of the cvs tree, to see what the kernel side of things would look like if M:N support came out. The result is an amazing code clarity improvement and it enables a bunch of other optimizations to be done with greater ease. Things happen like being able to easily reduce the C code executed between an interrupt and ithread dispatch by about 75%. This simplification enabled Kip to do a bunch of scalability work as well (per-cpu scheduling locks, per-cpu process lists, etc). However, my objectives there were quite different to what Robert has raised. My objectives were a 'what if?'. People have complained in the past that the complexity that KSE adds to the kernel context switching code gets in the way of other optimizations that they'd like to try, so I figured that this would be a good way to call them on that and see if it really does help or not. I was hoping to be able to present a list of things that we'd gain as a result, but unfortunately the cat is out of a bag a bit earlier than I'd have liked. I never really intended to bring it up until there was something to show for it. I know Kip has done some amazing work already but I was hoping for other things as well before going public. FWIW, My skunkworks project is in perforce: //depot/projects/bike_sched/... and there is a live diff: http://people.freebsd.org/~peter/bike_sched.diff (Yes, the name was picked long before this thread started) It does NOT have any of Kip's optimization work in it. It was just meant as a baseline for other people to experiment with. I've tested it with 4bsd as the scheduler. ULE might work, but I have not tried it. SCHED_CORE will not compile in that tree because I haven't yet gone over diffs from David Xu yet. I run this code on my laptop with libmap.conf redirecting libpthread to libthr. It works very well for me, even threaded apps like firefox etc. Anyway, back to the subject at hand. The basic problem with the KSE/SA model as I see it (besides the kernel code complexity) is that it doesn't really seem to suit the kind of threaded applications that people seem to want to run on unix boxes. In a traditional 1:1 threading system, eg: linuxthreads/nptl, libthr, etc, mutex blocking is expensive, but system calls and blocking in kernel mode is the same cost as a regular process making system calls or blocking in kernel mode. Because Linux was the most widely and massively deployed threading system out there, people tended to write (or modify) their applications to work best with those assumptions. ie: keep pthread mutex blocking to an absolute minimum, and not care about kernel blocking. However, with the SA/KSE model, our tradeoffs are different. We implement pthread mutex blocking more quickly (except for UTS bugs that can make it far slower), but we make blocking in kernel context significantly higher cost than the 1:1 case, probably as much as double the cost. For applications that block in the kernel a lot instead of on mutexes, this is a big source of pain. When most of the applications that we're called to run are written with the linux behavior in mind, when our performance is compared against linux we're the ones that usually come off the worst. I'm sure that there are threaded applications that benefit from cheap mutex operations, but I'm not personally aware of them. I do know that the ones that we get regularly compared to linux with are the likes of mysql, squid and threaded http servers. All of those depend on kernel blocking being as fast as possible. Faster mutexes doesn't seem to compensate for the extra costs of kernel blocking. I don't know where java fits into this picture. We've proven that we can make KSE work, but it was far harder than we imagined, and unfortunately, the real-world apps that matter the most just don't seem to take advantage of it. Not to mention the complexity that we have to work around for scalability work. Speaking of scalability, 16 and 32 way systems are here already and will be common within 7.0's lifetime. If we don't scale, we're sunk. My gut tells me that we HAVE to address the complexity that the KSE kernel code adds in order to improve this. We can barely work well on 4-cpu systems, let alone 32 cpu systems. PS: I think it would be interesting to see a hybrid user level M:N system. Even if it was as simple as multiplexing user threads onto a group of kernel threads (without M:N kernel support) and doing libc_r style syscall wrappers for intercepting long-term blockable operations like socket/pipe IO etc. For short term blocking (disk IO), just wear the cost of letting one thread block for a moment. I suspect that large parts of libpthread could be reused and some bits brought back from libc_r. I think this would do a fairly decent job for things like computational threaded apps because mutexes would be really fast. PPS: My opinions are not meant as a criticism of the massive amount of work that has gone into making KSE work. It is more an attempt to step back and take an objective look at the ever-changing big picture. -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 04:01:31 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6570616A4E1 for ; Wed, 5 Jul 2006 04:01:31 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.234]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E4B143D46 for ; Wed, 5 Jul 2006 04:01:26 +0000 (GMT) (envelope-from kip.macy@gmail.com) Received: by wr-out-0506.google.com with SMTP id 68so978669wra for ; Tue, 04 Jul 2006 21:01:26 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=gpKtYR9hK92Y0qv4iz4pwHzvHkoRJzgV8/m1J071pWksDNbH071EGoGRuUXzorzJyoU/je/AQ1Z+kr0Zjkm+T+leDznoM/yC9tii0r0N80siuaEOjaPNGxik3sekbTPesGASm1ZrCH3gwl6ziBPqAi2hGeCyV4I5h1wnUO/2s8g= Received: by 10.65.234.20 with SMTP id l20mr2589840qbr; Tue, 04 Jul 2006 21:01:26 -0700 (PDT) Received: by 10.65.225.9 with HTTP; Tue, 4 Jul 2006 21:01:26 -0700 (PDT) Message-ID: Date: Tue, 4 Jul 2006 21:01:26 -0700 From: "Kip Macy" To: "Peter Wemm" In-Reply-To: <200607041819.05510.peter@wemm.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> Cc: freebsd-threads@freebsd.org, Daniel Eischen , Robert Watson , David Xu , threads@freebsd.org, Julian Elischer Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@fsmware.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 04:01:31 -0000 I believe that the views that Peter has expressed are held by quite a few. I initially integrated bike_sched in my development branch for the purpose of playing with different locking strategies. More recently I've integrated it into my stable branch after discovering that it greatly improved the stability of threaded applications. As a consequence of it being in my stable branch it has been integrated into a widely watched development project. I'll leave it to the developer on that project to discuss it on -current. As someone who has yet to make substantial contributions to FreeBSD it is not my place to advocate for or against KSE. However, I will say, without equivocation, that KSE needs a fair amount of TLC in the form of re-factoring and bug fixes for it to have a place on future hardware. -Kip > At BSDCan, I tinkered with a checkout of the cvs tree, to see what the > kernel side of things would look like if M:N support came out. The > result is an amazing code clarity improvement and it enables a bunch of > other optimizations to be done with greater ease. Things happen like > being able to easily reduce the C code executed between an interrupt > and ithread dispatch by about 75%. This simplification enabled Kip to > do a bunch of scalability work as well (per-cpu scheduling locks, > per-cpu process lists, etc). > > However, my objectives there were quite different to what Robert has > raised. My objectives were a 'what if?'. People have complained in > the past that the complexity that KSE adds to the kernel context > switching code gets in the way of other optimizations that they'd like > to try, so I figured that this would be a good way to call them on that > and see if it really does help or not. I was hoping to be able to > present a list of things that we'd gain as a result, but unfortunately > the cat is out of a bag a bit earlier than I'd have liked. I never > really intended to bring it up until there was something to show for > it. I know Kip has done some amazing work already but I was hoping for > other things as well before going public. > > FWIW, My skunkworks project is in perforce: > //depot/projects/bike_sched/... > and there is a live diff: > http://people.freebsd.org/~peter/bike_sched.diff > (Yes, the name was picked long before this thread started) > > It does NOT have any of Kip's optimization work in it. It was just > meant as a baseline for other people to experiment with. I've tested > it with 4bsd as the scheduler. ULE might work, but I have not tried > it. SCHED_CORE will not compile in that tree because I haven't yet > gone over diffs from David Xu yet. I run this code on my laptop with > libmap.conf redirecting libpthread to libthr. It works very well for > me, even threaded apps like firefox etc. > > > Anyway, back to the subject at hand. The basic problem with the KSE/SA > model as I see it (besides the kernel code complexity) is that it > doesn't really seem to suit the kind of threaded applications that > people seem to want to run on unix boxes. > > In a traditional 1:1 threading system, eg: linuxthreads/nptl, libthr, > etc, mutex blocking is expensive, but system calls and blocking in > kernel mode is the same cost as a regular process making system calls > or blocking in kernel mode. > > Because Linux was the most widely and massively deployed threading > system out there, people tended to write (or modify) their applications > to work best with those assumptions. ie: keep pthread mutex blocking > to an absolute minimum, and not care about kernel blocking. > > However, with the SA/KSE model, our tradeoffs are different. We > implement pthread mutex blocking more quickly (except for UTS bugs that > can make it far slower), but we make blocking in kernel context > significantly higher cost than the 1:1 case, probably as much as double > the cost. For applications that block in the kernel a lot instead of on > mutexes, this is a big source of pain. > > When most of the applications that we're called to run are written with > the linux behavior in mind, when our performance is compared against > linux we're the ones that usually come off the worst. > > I'm sure that there are threaded applications that benefit from cheap > mutex operations, but I'm not personally aware of them. I do know that > the ones that we get regularly compared to linux with are the likes of > mysql, squid and threaded http servers. All of those depend on kernel > blocking being as fast as possible. Faster mutexes doesn't seem to > compensate for the extra costs of kernel blocking. I don't know where > java fits into this picture. > > We've proven that we can make KSE work, but it was far harder than we > imagined, and unfortunately, the real-world apps that matter the most > just don't seem to take advantage of it. Not to mention the complexity > that we have to work around for scalability work. > > Speaking of scalability, 16 and 32 way systems are here already and will > be common within 7.0's lifetime. If we don't scale, we're sunk. My > gut tells me that we HAVE to address the complexity that the KSE kernel > code adds in order to improve this. We can barely work well on 4-cpu > systems, let alone 32 cpu systems. > > PS: I think it would be interesting to see a hybrid user level M:N > system. Even if it was as simple as multiplexing user threads onto a > group of kernel threads (without M:N kernel support) and doing libc_r > style syscall wrappers for intercepting long-term blockable operations > like socket/pipe IO etc. For short term blocking (disk IO), just wear > the cost of letting one thread block for a moment. I suspect that > large parts of libpthread could be reused and some bits brought back > from libc_r. I think this would do a fairly decent job for things like > computational threaded apps because mutexes would be really fast. > > PPS: My opinions are not meant as a criticism of the massive amount of > work that has gone into making KSE work. It is more an attempt to step > back and take an objective look at the ever-changing big picture. > -- > Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com > "All of this is for nothing if we don't go to the stars" - JMS/B5 > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 04:01:32 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2196E16A4EA for ; Wed, 5 Jul 2006 04:01:32 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.236]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D66743D53 for ; Wed, 5 Jul 2006 04:01:26 +0000 (GMT) (envelope-from kip.macy@gmail.com) Received: by wr-out-0506.google.com with SMTP id 68so978671wra for ; Tue, 04 Jul 2006 21:01:26 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=gpKtYR9hK92Y0qv4iz4pwHzvHkoRJzgV8/m1J071pWksDNbH071EGoGRuUXzorzJyoU/je/AQ1Z+kr0Zjkm+T+leDznoM/yC9tii0r0N80siuaEOjaPNGxik3sekbTPesGASm1ZrCH3gwl6ziBPqAi2hGeCyV4I5h1wnUO/2s8g= Received: by 10.65.234.20 with SMTP id l20mr2589840qbr; Tue, 04 Jul 2006 21:01:26 -0700 (PDT) Received: by 10.65.225.9 with HTTP; Tue, 4 Jul 2006 21:01:26 -0700 (PDT) Message-ID: Date: Tue, 4 Jul 2006 21:01:26 -0700 From: "Kip Macy" To: "Peter Wemm" In-Reply-To: <200607041819.05510.peter@wemm.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> Cc: freebsd-threads@freebsd.org, Daniel Eischen , Robert Watson , David Xu , threads@freebsd.org, Julian Elischer Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@fsmware.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 04:01:32 -0000 I believe that the views that Peter has expressed are held by quite a few. I initially integrated bike_sched in my development branch for the purpose of playing with different locking strategies. More recently I've integrated it into my stable branch after discovering that it greatly improved the stability of threaded applications. As a consequence of it being in my stable branch it has been integrated into a widely watched development project. I'll leave it to the developer on that project to discuss it on -current. As someone who has yet to make substantial contributions to FreeBSD it is not my place to advocate for or against KSE. However, I will say, without equivocation, that KSE needs a fair amount of TLC in the form of re-factoring and bug fixes for it to have a place on future hardware. -Kip > At BSDCan, I tinkered with a checkout of the cvs tree, to see what the > kernel side of things would look like if M:N support came out. The > result is an amazing code clarity improvement and it enables a bunch of > other optimizations to be done with greater ease. Things happen like > being able to easily reduce the C code executed between an interrupt > and ithread dispatch by about 75%. This simplification enabled Kip to > do a bunch of scalability work as well (per-cpu scheduling locks, > per-cpu process lists, etc). > > However, my objectives there were quite different to what Robert has > raised. My objectives were a 'what if?'. People have complained in > the past that the complexity that KSE adds to the kernel context > switching code gets in the way of other optimizations that they'd like > to try, so I figured that this would be a good way to call them on that > and see if it really does help or not. I was hoping to be able to > present a list of things that we'd gain as a result, but unfortunately > the cat is out of a bag a bit earlier than I'd have liked. I never > really intended to bring it up until there was something to show for > it. I know Kip has done some amazing work already but I was hoping for > other things as well before going public. > > FWIW, My skunkworks project is in perforce: > //depot/projects/bike_sched/... > and there is a live diff: > http://people.freebsd.org/~peter/bike_sched.diff > (Yes, the name was picked long before this thread started) > > It does NOT have any of Kip's optimization work in it. It was just > meant as a baseline for other people to experiment with. I've tested > it with 4bsd as the scheduler. ULE might work, but I have not tried > it. SCHED_CORE will not compile in that tree because I haven't yet > gone over diffs from David Xu yet. I run this code on my laptop with > libmap.conf redirecting libpthread to libthr. It works very well for > me, even threaded apps like firefox etc. > > > Anyway, back to the subject at hand. The basic problem with the KSE/SA > model as I see it (besides the kernel code complexity) is that it > doesn't really seem to suit the kind of threaded applications that > people seem to want to run on unix boxes. > > In a traditional 1:1 threading system, eg: linuxthreads/nptl, libthr, > etc, mutex blocking is expensive, but system calls and blocking in > kernel mode is the same cost as a regular process making system calls > or blocking in kernel mode. > > Because Linux was the most widely and massively deployed threading > system out there, people tended to write (or modify) their applications > to work best with those assumptions. ie: keep pthread mutex blocking > to an absolute minimum, and not care about kernel blocking. > > However, with the SA/KSE model, our tradeoffs are different. We > implement pthread mutex blocking more quickly (except for UTS bugs that > can make it far slower), but we make blocking in kernel context > significantly higher cost than the 1:1 case, probably as much as double > the cost. For applications that block in the kernel a lot instead of on > mutexes, this is a big source of pain. > > When most of the applications that we're called to run are written with > the linux behavior in mind, when our performance is compared against > linux we're the ones that usually come off the worst. > > I'm sure that there are threaded applications that benefit from cheap > mutex operations, but I'm not personally aware of them. I do know that > the ones that we get regularly compared to linux with are the likes of > mysql, squid and threaded http servers. All of those depend on kernel > blocking being as fast as possible. Faster mutexes doesn't seem to > compensate for the extra costs of kernel blocking. I don't know where > java fits into this picture. > > We've proven that we can make KSE work, but it was far harder than we > imagined, and unfortunately, the real-world apps that matter the most > just don't seem to take advantage of it. Not to mention the complexity > that we have to work around for scalability work. > > Speaking of scalability, 16 and 32 way systems are here already and will > be common within 7.0's lifetime. If we don't scale, we're sunk. My > gut tells me that we HAVE to address the complexity that the KSE kernel > code adds in order to improve this. We can barely work well on 4-cpu > systems, let alone 32 cpu systems. > > PS: I think it would be interesting to see a hybrid user level M:N > system. Even if it was as simple as multiplexing user threads onto a > group of kernel threads (without M:N kernel support) and doing libc_r > style syscall wrappers for intercepting long-term blockable operations > like socket/pipe IO etc. For short term blocking (disk IO), just wear > the cost of letting one thread block for a moment. I suspect that > large parts of libpthread could be reused and some bits brought back > from libc_r. I think this would do a fairly decent job for things like > computational threaded apps because mutexes would be really fast. > > PPS: My opinions are not meant as a criticism of the massive amount of > work that has gone into making KSE work. It is more an attempt to step > back and take an objective look at the ever-changing big picture. > -- > Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com > "All of this is for nothing if we don't go to the stars" - JMS/B5 > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 04:49:12 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from [127.0.0.1] (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 1EB7C16A4DE; Wed, 5 Jul 2006 04:49:08 +0000 (UTC) (envelope-from davidxu@freebsd.org) Message-ID: <44AB44C7.7040008@freebsd.org> Date: Wed, 05 Jul 2006 12:49:11 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060519 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kmacy@fsmware.com References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org, Daniel Eischen , Robert Watson , Julian Elischer , threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 04:49:12 -0000 Kip Macy wrote: > > I believe that the views that Peter has expressed are held by quite a > few. I initially integrated bike_sched in my development branch for > the purpose of playing with different locking strategies. More > recently I've integrated it into my stable branch after discovering > that it greatly improved the stability of threaded applications. As a > consequence of it being in my stable branch it has been integrated > into a widely watched development project. I'll leave it to the > developer on that project to discuss it on -current. As someone who > has yet to make substantial contributions to FreeBSD it is not my > place to advocate for or against KSE. However, I will say, without > equivocation, that KSE needs a fair amount of TLC in the form of > re-factoring and bug fixes for it to have a place on future hardware. > > -Kip > By removing M:N code, the kernel code looks pretty clean. however, I will not agree Peter's hybrid M:N thread library idea, remember I wrote gdb support code for libpthread and libthr, writting debugger code for libpthread is a nightmare, the hybrid M:N will has this problem too. David Xu From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 04:49:12 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from [127.0.0.1] (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 1EB7C16A4DE; Wed, 5 Jul 2006 04:49:08 +0000 (UTC) (envelope-from davidxu@freebsd.org) Message-ID: <44AB44C7.7040008@freebsd.org> Date: Wed, 05 Jul 2006 12:49:11 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060519 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kmacy@fsmware.com References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org, Daniel Eischen , Robert Watson , Julian Elischer , threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 04:49:12 -0000 Kip Macy wrote: > > I believe that the views that Peter has expressed are held by quite a > few. I initially integrated bike_sched in my development branch for > the purpose of playing with different locking strategies. More > recently I've integrated it into my stable branch after discovering > that it greatly improved the stability of threaded applications. As a > consequence of it being in my stable branch it has been integrated > into a widely watched development project. I'll leave it to the > developer on that project to discuss it on -current. As someone who > has yet to make substantial contributions to FreeBSD it is not my > place to advocate for or against KSE. However, I will say, without > equivocation, that KSE needs a fair amount of TLC in the form of > re-factoring and bug fixes for it to have a place on future hardware. > > -Kip > By removing M:N code, the kernel code looks pretty clean. however, I will not agree Peter's hybrid M:N thread library idea, remember I wrote gdb support code for libpthread and libthr, writting debugger code for libpthread is a nightmare, the hybrid M:N will has this problem too. David Xu From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 04:57:56 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2BE1616A4E6; Wed, 5 Jul 2006 04:57:56 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5A03E43D46; Wed, 5 Jul 2006 04:57:55 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.25]) by a50.ironport.com with ESMTP; 04 Jul 2006 21:57:55 -0700 Message-ID: <44AB46CF.9030503@elischer.org> Date: Tue, 04 Jul 2006 21:57:51 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Xu References: <20060703101554.Q26325@fledge.watson.org> <44AAC47F.2040508@elischer.org> <200607050706.33502.davidxu@freebsd.org> <200607050724.18740.davidxu@freebsd.org> In-Reply-To: <200607050724.18740.davidxu@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 04:57:56 -0000 David Xu wrote: >On Wednesday 05 July 2006 07:06, David Xu wrote: > > > >>Cann't solaris's user usage scheduler will solve the problem ? if I >>recall it correctly, the decay factor in 4bsd scheduler can be changed >>dynamically for a thread, the decay factor can be stored in thread >>structure, and timeslice can be dynamically changed, also can be in thread >>structure, unlike current implementation, timeslice is fixed, this may >>make fair scheduling possible without ksegrp. >> >>Also will the roundrobin callout preempt a thread ? if that's true, then >>when every timeslice tick, a thread exhausted its timeslice is put at head >>of runqueue, isn't it a bug ? the callout is somewhat random for a thread, >>might not be a problem... >> >> > >OK, I just digged out the paper in my disks, here is the reference: >http://people.freebsd.org/~davidxu/doc/solaris2.pdf > >[the address thread@freebsd.org is removed] > >I think we really should do more work in kernel scheduler, and remove >ksegrp. > > the original aim for the ksegrp was not for fair scheduling but to allow libraries to instanciate their own threads in a way that they did not impact the application's threads in any way. Effectively, running them in their own process.. Fair scheduling just made use of it as a convenient abstraction. Removing the ksegrp entirely just reduces everything down to being boring threads with no possibility of ever doing anything different. KSEs have already gone, except as a scheduler abstraction. and having KSEGROUPs shouldn't affect anything negatively from no on. though they could be renamed threadgroups now. The scheduler complexities we see now do not come from having ksegrps, but rather, from trying to do fair scheduling in the rather simplistic "obvious" manner that I did. I am truely surprised that someone has not come up with a better way to do it by now. I was expecting that the current approach would have been replaced by something more sophisticated by now. >David Xu > > From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 05:20:36 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5153916A4E0; Wed, 5 Jul 2006 05:20:36 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id E5E9A43D55; Wed, 5 Jul 2006 05:20:34 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.25]) by a50.ironport.com with ESMTP; 04 Jul 2006 22:20:35 -0700 Message-ID: <44AB4C22.3030109@elischer.org> Date: Tue, 04 Jul 2006 22:20:34 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kmacy@fsmware.com References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org, Daniel Eischen , Robert Watson , David Xu , threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 05:20:36 -0000 Kip Macy wrote: > I believe that the views that Peter has expressed are held by quite a > few. I initially integrated bike_sched in my development branch for > the purpose of playing with different locking strategies. More > recently I've integrated it into my stable branch after discovering > that it greatly improved the stability of threaded applications. As a > consequence of it being in my stable branch it has been integrated > into a widely watched development project. I'll leave it to the > developer on that project to discuss it on -current. As someone who > has yet to make substantial contributions to FreeBSD it is not my > place to advocate for or against KSE. However, I will say, without > equivocation, that KSE needs a fair amount of TLC in the form of > re-factoring and bug fixes for it to have a place on future hardware. > > -Kip > > I wrote the threading code into the system with the aim of supporting as many threading models as I could. In this light, you will notice that 1:1 threads is fully supported. It was always imagined that as time went on, various models would fall by the wayside. The first arrow in the came when it was found that KSE for some of the newer architectures (alpha and ia64) would be far harder than expected due to the fact that thread state is not always stored where you would expect it to be. This happenned fairly early on and necessitated a change in course part way through. it never really recoverred from this. The code to store a blocking thread and jump to another (newly created) one was never able to be really optimised, but it worked (kinda). The big problems are that a lot of code assumes things that are just not true when some of the threads are not resident in the kernel and have no actual trace in the kernel at all. It would have been good to see an app that could use M:N affectively, and there is some indication that some Java VMs may come under this category as I have heard of java VMs with 10,000 threads of which most are not involved with I/O. However I have never seen any sign of such a beast with my own eyes. If it come to pass that M:N threads are judged to be "unsuitable" then that is a decision that I can live with, but never be let it be said that I walled FreeBSD in to only having the option of 1:1 threads. I might add, as I did before that the double level scheduer with threads hanging off the KSEGRP until they can be run by the main scheduler when there are enough "opennings" for that process/ksegrp is orthogonal to this discussion and if that is all removed in Peter's skunkworks project then that is a separate issue that must be addressed under a different banner. Simplifying the scheduler by removing that just allows a process to overwhelm the scheduler with its own threads but it could be allowed to do that equally for both M:N and 1:1 threads., (Peter is this part of what you did?) >> > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to > "freebsd-threads-unsubscribe@freebsd.org" From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 05:20:36 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5153916A4E0; Wed, 5 Jul 2006 05:20:36 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id E5E9A43D55; Wed, 5 Jul 2006 05:20:34 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.25]) by a50.ironport.com with ESMTP; 04 Jul 2006 22:20:35 -0700 Message-ID: <44AB4C22.3030109@elischer.org> Date: Tue, 04 Jul 2006 22:20:34 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: kmacy@fsmware.com References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org, Daniel Eischen , Robert Watson , David Xu , threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 05:20:36 -0000 Kip Macy wrote: > I believe that the views that Peter has expressed are held by quite a > few. I initially integrated bike_sched in my development branch for > the purpose of playing with different locking strategies. More > recently I've integrated it into my stable branch after discovering > that it greatly improved the stability of threaded applications. As a > consequence of it being in my stable branch it has been integrated > into a widely watched development project. I'll leave it to the > developer on that project to discuss it on -current. As someone who > has yet to make substantial contributions to FreeBSD it is not my > place to advocate for or against KSE. However, I will say, without > equivocation, that KSE needs a fair amount of TLC in the form of > re-factoring and bug fixes for it to have a place on future hardware. > > -Kip > > I wrote the threading code into the system with the aim of supporting as many threading models as I could. In this light, you will notice that 1:1 threads is fully supported. It was always imagined that as time went on, various models would fall by the wayside. The first arrow in the came when it was found that KSE for some of the newer architectures (alpha and ia64) would be far harder than expected due to the fact that thread state is not always stored where you would expect it to be. This happenned fairly early on and necessitated a change in course part way through. it never really recoverred from this. The code to store a blocking thread and jump to another (newly created) one was never able to be really optimised, but it worked (kinda). The big problems are that a lot of code assumes things that are just not true when some of the threads are not resident in the kernel and have no actual trace in the kernel at all. It would have been good to see an app that could use M:N affectively, and there is some indication that some Java VMs may come under this category as I have heard of java VMs with 10,000 threads of which most are not involved with I/O. However I have never seen any sign of such a beast with my own eyes. If it come to pass that M:N threads are judged to be "unsuitable" then that is a decision that I can live with, but never be let it be said that I walled FreeBSD in to only having the option of 1:1 threads. I might add, as I did before that the double level scheduer with threads hanging off the KSEGRP until they can be run by the main scheduler when there are enough "opennings" for that process/ksegrp is orthogonal to this discussion and if that is all removed in Peter's skunkworks project then that is a separate issue that must be addressed under a different banner. Simplifying the scheduler by removing that just allows a process to overwhelm the scheduler with its own threads but it could be allowed to do that equally for both M:N and 1:1 threads., (Peter is this part of what you did?) >> > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to > "freebsd-threads-unsubscribe@freebsd.org" From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 05:20:49 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from [127.0.0.1] (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 0052F16A4E1; Wed, 5 Jul 2006 05:20:47 +0000 (UTC) (envelope-from davidxu@freebsd.org) Message-ID: <44AB4C31.5090402@freebsd.org> Date: Wed, 05 Jul 2006 13:20:49 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060519 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Julian Elischer References: <20060703101554.Q26325@fledge.watson.org> <44AAC47F.2040508@elischer.org> <200607050706.33502.davidxu@freebsd.org> <200607050724.18740.davidxu@freebsd.org> <44AB46CF.9030503@elischer.org> In-Reply-To: <44AB46CF.9030503@elischer.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 05:20:49 -0000 Julian Elischer wrote: > the original aim for the ksegrp was not for fair scheduling but to allow > libraries to instanciate > their own threads in a way that they did not impact the application's > threads in any way. > Effectively, running them in their own process.. Fair scheduling just > made use of it as a convenient > abstraction. > Removing the ksegrp entirely just reduces everything down to being > boring threads with no possibility of > ever doing anything different. > > KSEs have already gone, except as a scheduler abstraction. > and having KSEGROUPs shouldn't affect anything negatively from no on. > though they could be renamed threadgroups now. > but it still has bias to M:N, the kg_user_pri member which is not right for 1:1 thread, this also makes user interaction detection algorithm impossible, e.g one thread is cpu-bound while another thread is user-interaction thread, this model will make it impossible, and two threads will be treated as cpu-bound, the concurrency limit in ksegrp(4bsd/ule scheduler) also makes cpu-affinity algorithm to be more diffcult, as kern_switch.c will withdraw a thread from scheduler runqueue by a very simple reason - concurrency level. > The scheduler complexities we see now do not come from having ksegrps, > but rather, from trying to > do fair scheduling in the rather simplistic "obvious" manner that I > did. I am truely > surprised that someone has not come up with a better way to do it by > now. I was expecting that > the current approach would have been replaced by something more > sophisticated by now. > I would like to see a top level long term scheduler to adjust decay factor as Solaris did, this would make fairness better than ksegrp, ksegrp mechanism still can only manage a process but has above issues. From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 05:46:47 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 32A5016A4DE; Wed, 5 Jul 2006 05:46:47 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id DBEEC43D55; Wed, 5 Jul 2006 05:46:46 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [192.168.2.4]) ([10.251.60.25]) by a50.ironport.com with ESMTP; 04 Jul 2006 22:46:46 -0700 Message-ID: <44AB5243.6050905@elischer.org> Date: Tue, 04 Jul 2006 22:46:43 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Xu References: <20060703101554.Q26325@fledge.watson.org> <44AAC47F.2040508@elischer.org> <200607050706.33502.davidxu@freebsd.org> <200607050724.18740.davidxu@freebsd.org> <44AB46CF.9030503@elischer.org> <44AB4C31.5090402@freebsd.org> In-Reply-To: <44AB4C31.5090402@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 05:46:47 -0000 David Xu wrote: > Julian Elischer wrote: > >> the original aim for the ksegrp was not for fair scheduling but to >> allow libraries to instanciate >> their own threads in a way that they did not impact the application's >> threads in any way. >> Effectively, running them in their own process.. Fair scheduling just >> made use of it as a convenient >> abstraction. >> Removing the ksegrp entirely just reduces everything down to being >> boring threads with no possibility of >> ever doing anything different. >> >> KSEs have already gone, except as a scheduler abstraction. >> and having KSEGROUPs shouldn't affect anything negatively from no on. >> though they could be renamed threadgroups now. >> > > but it still has bias to M:N, the kg_user_pri member which is not right > for 1:1 thread, this also makes user interaction detection algorithm > impossible, e.g one thread is cpu-bound while another thread is > user-interaction thread, this model will make it impossible, > and two threads will be treated as cpu-bound, the concurrency limit > in ksegrp(4bsd/ule scheduler) also makes cpu-affinity algorithm to > be more diffcult, as kern_switch.c will withdraw a thread from > scheduler runqueue by a very simple reason - concurrency level. well then have a separate KSEG for each thread.. > >> The scheduler complexities we see now do not come from having >> ksegrps, but rather, from trying to >> do fair scheduling in the rather simplistic "obvious" manner that I >> did. I am truely >> surprised that someone has not come up with a better way to do it by >> now. I was expecting that >> the current approach would have been replaced by something more >> sophisticated by now. >> > > I would like to see a top level long term scheduler to adjust > decay factor as Solaris did, this would make fairness better than > ksegrp, ksegrp mechanism still can only manage a process but has > above issues. if you have a single KSEG for each thread, then you are welcome to have another method of producing thread fairness. if it can be shown to work then we can make ithte default way to run.. As I said, I did this work kbecause FreeBSD NEEDED threads and no-one was doing it, but I am pretty agnostic on how they should be implemented for the acerage thread user. From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 08:48:18 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 38D5516A4DE; Wed, 5 Jul 2006 08:48:18 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id BC0BC43D49; Wed, 5 Jul 2006 08:48:17 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 101C146CA7; Wed, 5 Jul 2006 04:48:17 -0400 (EDT) Date: Wed, 5 Jul 2006 09:48:16 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Peter Wemm In-Reply-To: <200607041819.05510.peter@wemm.org> Message-ID: <20060705092048.P70011@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Daniel Eischen , threads@freebsd.org, David Xu , Julian Elischer , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 08:48:18 -0000 On Tue, 4 Jul 2006, Peter Wemm wrote: > Because Linux was the most widely and massively deployed threading system > out there, people tended to write (or modify) their applications to work > best with those assumptions. ie: keep pthread mutex blocking to an absolute > minimum, and not care about kernel blocking. > > However, with the SA/KSE model, our tradeoffs are different. We implement > pthread mutex blocking more quickly (except for UTS bugs that can make it > far slower), but we make blocking in kernel context significantly higher > cost than the 1:1 case, probably as much as double the cost. For > applications that block in the kernel a lot instead of on mutexes, this is a > big source of pain. > > When most of the applications that we're called to run are written with the > linux behavior in mind, when our performance is compared against linux we're > the ones that usually come off the worst. The problem I've been running into is similar but different. The reason for my asking about libthr being the default is that, in practice, our performance optimization advice for a host of threaded applications has been "Switch to libthr". This causes quite a bit of complexity from a network stack optimization perspective, because the behavior of threading in threaded network/IPC applications changes enormously if the threading model is changed. As a result, the optimization strategies differ greatly. To motivate this, let me give you an example. Widely distributed MySQL benchmarks are basically kernel IPC benchmarks, and on multi-processor systems, this means they basically benchmark context switch, scheduling, network stack overhead, and network stack parallelism. However, the locking hot spots differ significantly based on the threading model used. There are two easily identified reasons for this: - Libpthread "rate limits" threads entering the kernel in the run/running state, resulting in less contention on per-process sleep mutexes. - Libthr has greater locality of behavior in that the mapping of thread activities to kernel-visible threads is greater. Consider the case of an application that makes frequent short accesses to file descriptors -- for example, by sending lots of short I/Os on a set of UNIX domain sockets from various worker threads, each performing transactions on behalf of a client via IPC. This is, FYI, a widely deployed programming approach, and is not limited to MySQL. The various user threads will be constantly looking up file descriptor numbers in the file descriptor array; often, the same thread will look up the same number several times (accept, i/o, i/o, i/o, ..., close). This results in very high contention on the file descriptor array mutex, even though individual uses are short. In practice, libpthread sees somewhat lower contention, because in the presence of adaptive mutexes, kernel threads spin rather than blocking, causing libpthread to not push further threads in to contend on the lock. However, one of the more interesting optimizations to explore involves "loaning" file descriptors to threads, in order to take advantage of locality of reference, where repeated access to the same fd is cheaper, but revocation of the loan for use by another thread is more expensive. In libthr, we have lots of locality of reference, because user threads map 1:1 to kernel threads; in libpthread, this is not the case, as user threads float across pthreads, and even if they do get mapped to the same kernel thread repeatedly, their execution in the presence of blocking is discontinuous in the same kernel thread. This makes things tricky for someone working on reducing contention in the kernel as the number of threads increases: do I optimize for libpthread, which offers little or no locality of reference with respect to mapping user thread behavior to kernel threads, or do I optimize for libthr, which offers high locality of reference? Since our stock advice is to run libthr for high performance applications, the design choice should be clear: I should optimize for libthr. However, in doing so, I would likely heavily pessimize libpthread performance, as I would basically guarantee that heuristics based on user thread locality would fail with moderate frequency, as the per-kernel thread working set for kernel objects is significantly greater. FWIW, you can quite clearly measure the difference in file descriptor array lock contention using the http/httpd micro-benchmarks in src/tools/tools/netrate. If you run without threading, performance is better, in significant part because there is much less contention. This is an interesting, and apparently counter-intuitive observation: many people believe that the reduced context switch and greater cache locality of threaded applications always results in improved performance. This is not true for a number of important workloads -- by operating with more shared data structures, contention on those shared data structures is increased, reducing performance. Moving to the two threading models, you see markedly better libpthread performance under extremely high load involving many threads with small transactions, as libpthread provides heuristically better management of kernel load. This advantage does not carry over to real-world application loads, however, which tend to use a smaller thread worker pools with sequences of locality-rich transaction, which is why libthr performs btter as the workload approaches real-world conditions. This micro-benchmark makes for quite an interesting study piece, as you can easily vary the thread/proc model, the number of workers, and the transaction size, giving pretty clear performance curves to compare. Anyhow, my main point in raising this thread was actually oriented entirely on the initial observation, which is that in practice, we find ourselves telling people who care about performance to use libthr. If our advice is always "use libthr instead of the default", that suggests we have a problem with the default. Switching the default requires an informed decision: what do we lose, not just what do we gain. Dan has now answered this question -- we lose support for a number of realtime scheduling primitives if we switch today without further work. I think the discussion of the future of M:N support is also critical, though, as it has an immediate impact on kernel optimization strategies, especially as number of CPUs grows. In case anyone failed to notice, it's now possible to buy hardware with 32 "threads" for <$10,000, and the future appears relatively clear -- parallelism isn't just for high-end servers, it now appears in off-the-shelf notebook hardware, and appears to be the way that vendors are going to continue to improve performance. Having spent the last five years working on threading and SMP, we're well-placed to be to support this hardware, but it requires us to start consolidating our gains now, which means deciding what the baseline is for optimization when it comes to threaded applications. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 08:48:18 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 38D5516A4DE; Wed, 5 Jul 2006 08:48:18 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id BC0BC43D49; Wed, 5 Jul 2006 08:48:17 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 101C146CA7; Wed, 5 Jul 2006 04:48:17 -0400 (EDT) Date: Wed, 5 Jul 2006 09:48:16 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Peter Wemm In-Reply-To: <200607041819.05510.peter@wemm.org> Message-ID: <20060705092048.P70011@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Daniel Eischen , threads@freebsd.org, David Xu , Julian Elischer , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 08:48:18 -0000 On Tue, 4 Jul 2006, Peter Wemm wrote: > Because Linux was the most widely and massively deployed threading system > out there, people tended to write (or modify) their applications to work > best with those assumptions. ie: keep pthread mutex blocking to an absolute > minimum, and not care about kernel blocking. > > However, with the SA/KSE model, our tradeoffs are different. We implement > pthread mutex blocking more quickly (except for UTS bugs that can make it > far slower), but we make blocking in kernel context significantly higher > cost than the 1:1 case, probably as much as double the cost. For > applications that block in the kernel a lot instead of on mutexes, this is a > big source of pain. > > When most of the applications that we're called to run are written with the > linux behavior in mind, when our performance is compared against linux we're > the ones that usually come off the worst. The problem I've been running into is similar but different. The reason for my asking about libthr being the default is that, in practice, our performance optimization advice for a host of threaded applications has been "Switch to libthr". This causes quite a bit of complexity from a network stack optimization perspective, because the behavior of threading in threaded network/IPC applications changes enormously if the threading model is changed. As a result, the optimization strategies differ greatly. To motivate this, let me give you an example. Widely distributed MySQL benchmarks are basically kernel IPC benchmarks, and on multi-processor systems, this means they basically benchmark context switch, scheduling, network stack overhead, and network stack parallelism. However, the locking hot spots differ significantly based on the threading model used. There are two easily identified reasons for this: - Libpthread "rate limits" threads entering the kernel in the run/running state, resulting in less contention on per-process sleep mutexes. - Libthr has greater locality of behavior in that the mapping of thread activities to kernel-visible threads is greater. Consider the case of an application that makes frequent short accesses to file descriptors -- for example, by sending lots of short I/Os on a set of UNIX domain sockets from various worker threads, each performing transactions on behalf of a client via IPC. This is, FYI, a widely deployed programming approach, and is not limited to MySQL. The various user threads will be constantly looking up file descriptor numbers in the file descriptor array; often, the same thread will look up the same number several times (accept, i/o, i/o, i/o, ..., close). This results in very high contention on the file descriptor array mutex, even though individual uses are short. In practice, libpthread sees somewhat lower contention, because in the presence of adaptive mutexes, kernel threads spin rather than blocking, causing libpthread to not push further threads in to contend on the lock. However, one of the more interesting optimizations to explore involves "loaning" file descriptors to threads, in order to take advantage of locality of reference, where repeated access to the same fd is cheaper, but revocation of the loan for use by another thread is more expensive. In libthr, we have lots of locality of reference, because user threads map 1:1 to kernel threads; in libpthread, this is not the case, as user threads float across pthreads, and even if they do get mapped to the same kernel thread repeatedly, their execution in the presence of blocking is discontinuous in the same kernel thread. This makes things tricky for someone working on reducing contention in the kernel as the number of threads increases: do I optimize for libpthread, which offers little or no locality of reference with respect to mapping user thread behavior to kernel threads, or do I optimize for libthr, which offers high locality of reference? Since our stock advice is to run libthr for high performance applications, the design choice should be clear: I should optimize for libthr. However, in doing so, I would likely heavily pessimize libpthread performance, as I would basically guarantee that heuristics based on user thread locality would fail with moderate frequency, as the per-kernel thread working set for kernel objects is significantly greater. FWIW, you can quite clearly measure the difference in file descriptor array lock contention using the http/httpd micro-benchmarks in src/tools/tools/netrate. If you run without threading, performance is better, in significant part because there is much less contention. This is an interesting, and apparently counter-intuitive observation: many people believe that the reduced context switch and greater cache locality of threaded applications always results in improved performance. This is not true for a number of important workloads -- by operating with more shared data structures, contention on those shared data structures is increased, reducing performance. Moving to the two threading models, you see markedly better libpthread performance under extremely high load involving many threads with small transactions, as libpthread provides heuristically better management of kernel load. This advantage does not carry over to real-world application loads, however, which tend to use a smaller thread worker pools with sequences of locality-rich transaction, which is why libthr performs btter as the workload approaches real-world conditions. This micro-benchmark makes for quite an interesting study piece, as you can easily vary the thread/proc model, the number of workers, and the transaction size, giving pretty clear performance curves to compare. Anyhow, my main point in raising this thread was actually oriented entirely on the initial observation, which is that in practice, we find ourselves telling people who care about performance to use libthr. If our advice is always "use libthr instead of the default", that suggests we have a problem with the default. Switching the default requires an informed decision: what do we lose, not just what do we gain. Dan has now answered this question -- we lose support for a number of realtime scheduling primitives if we switch today without further work. I think the discussion of the future of M:N support is also critical, though, as it has an immediate impact on kernel optimization strategies, especially as number of CPUs grows. In case anyone failed to notice, it's now possible to buy hardware with 32 "threads" for <$10,000, and the future appears relatively clear -- parallelism isn't just for high-end servers, it now appears in off-the-shelf notebook hardware, and appears to be the way that vendors are going to continue to improve performance. Having spent the last five years working on threading and SMP, we're well-placed to be to support this hardware, but it requires us to start consolidating our gains now, which means deciding what the baseline is for optimization when it comes to threaded applications. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 08:54:21 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E4B0616A4DF; Wed, 5 Jul 2006 08:54:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 426B843D55; Wed, 5 Jul 2006 08:54:21 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id B727046C9D; Wed, 5 Jul 2006 04:54:20 -0400 (EDT) Date: Wed, 5 Jul 2006 09:54:20 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Peter Wemm In-Reply-To: <20060705092048.P70011@fledge.watson.org> Message-ID: <20060705095144.S64340@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> <20060705092048.P70011@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Daniel Eischen , threads@freebsd.org, freebsd-threads@freebsd.org, David Xu , Julian Elischer Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 08:54:22 -0000 On Wed, 5 Jul 2006, Robert Watson wrote: > management of kernel load. This advantage does not carry over to real-world > application loads, however, which tend to use a smaller thread worker pools > with sequences of locality-rich transaction, which is why libthr performs > btter as the workload approaches real-world conditions. This > micro-benchmark makes for quite an interesting study piece, as you can > easily vary the thread/proc model, the number of workers, and the > transaction size, giving pretty clear performance curves to compare. BTW, it would be really helpful if we had more (and perhaps better) potted benchmarks for threaded applications. Supersmack has made benchmarking MySQL easy, even though it turns out to be a rather un-representative workload (better workloads actually appear to shed better light on FreeBSD performance, FWIW). What I'm utterly unable to benchmark now are threaded UI applications, such as Mozilla, KDE, etc, which use threads quite a bit, but don't come with ways to capture their performance behavior reproduceably. I would really like to see a tool for measuring the perceivable latency impact of kernel changes on user applications. "It feels more snappy" is valuable but entirely anecdotal... Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 08:54:21 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E4B0616A4DF; Wed, 5 Jul 2006 08:54:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 426B843D55; Wed, 5 Jul 2006 08:54:21 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id B727046C9D; Wed, 5 Jul 2006 04:54:20 -0400 (EDT) Date: Wed, 5 Jul 2006 09:54:20 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Peter Wemm In-Reply-To: <20060705092048.P70011@fledge.watson.org> Message-ID: <20060705095144.S64340@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> <20060705092048.P70011@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Daniel Eischen , threads@freebsd.org, freebsd-threads@freebsd.org, David Xu , Julian Elischer Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 08:54:22 -0000 On Wed, 5 Jul 2006, Robert Watson wrote: > management of kernel load. This advantage does not carry over to real-world > application loads, however, which tend to use a smaller thread worker pools > with sequences of locality-rich transaction, which is why libthr performs > btter as the workload approaches real-world conditions. This > micro-benchmark makes for quite an interesting study piece, as you can > easily vary the thread/proc model, the number of workers, and the > transaction size, giving pretty clear performance curves to compare. BTW, it would be really helpful if we had more (and perhaps better) potted benchmarks for threaded applications. Supersmack has made benchmarking MySQL easy, even though it turns out to be a rather un-representative workload (better workloads actually appear to shed better light on FreeBSD performance, FWIW). What I'm utterly unable to benchmark now are threaded UI applications, such as Mozilla, KDE, etc, which use threads quite a bit, but don't come with ways to capture their performance behavior reproduceably. I would really like to see a tool for measuring the perceivable latency impact of kernel changes on user applications. "It feels more snappy" is valuable but entirely anecdotal... Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 09:02:40 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AA9DA16A4E2; Wed, 5 Jul 2006 09:02:40 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 32BD743D58; Wed, 5 Jul 2006 09:02:40 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 2DC7D46C82; Wed, 5 Jul 2006 05:02:39 -0400 (EDT) Date: Wed, 5 Jul 2006 10:02:39 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: <44AB4C22.3030109@elischer.org> Message-ID: <20060705095450.O64340@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> <44AB4C22.3030109@elischer.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-threads@freebsd.org, Daniel Eischen , kmacy@fsmware.com, David Xu , threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 09:02:40 -0000 On Tue, 4 Jul 2006, Julian Elischer wrote: > If it come to pass that M:N threads are judged to be "unsuitable" then that > is a decision that I can live with, but never be let it be said that I > walled FreeBSD in to only having the option of 1:1 threads. Given that I have serious hindsight accuracy problems, I won't even talk about my foresight issues. :-) I think we shouldn't nail the KSE coffin closed just yet -- as Peter has already said, it's one thing to do a skunkworks hack of what might eventually be used, but it's quite another to show that the perceived benefit maps into real world benefit. I'd like to see that clearly shown before we do anything drastic. In particular, we need to show that the benefit is there for more than just a few poorly written benchmarks, but also for a range of real-world workloads, and that the scheduler simplifications pay off in terms of both code complexity and our ability to optimize. I also want to make sure we understand some of the artifacts better: the benefit of reducing per-process lock contention is significant, and I'm interested in understanding where on the workload spectrum the benefits of 1:1 outweight the benefits of M:N a bit better. Is my interpretation that this is M:N providing better kernel workload management correct, or is it an artifact of scheduler behavior? And, where future hardware and application trends will push workload behavior with respect to the optimizations we already have, not to mention the ones we are considering. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Wed Jul 5 09:02:40 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AA9DA16A4E2; Wed, 5 Jul 2006 09:02:40 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 32BD743D58; Wed, 5 Jul 2006 09:02:40 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 2DC7D46C82; Wed, 5 Jul 2006 05:02:39 -0400 (EDT) Date: Wed, 5 Jul 2006 10:02:39 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: <44AB4C22.3030109@elischer.org> Message-ID: <20060705095450.O64340@fledge.watson.org> References: <20060703101554.Q26325@fledge.watson.org> <200607042204.52572.davidxu@freebsd.org> <44AAC47F.2040508@elischer.org> <200607041819.05510.peter@wemm.org> <44AB4C22.3030109@elischer.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-threads@freebsd.org, Daniel Eischen , kmacy@fsmware.com, David Xu , threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jul 2006 09:02:40 -0000 On Tue, 4 Jul 2006, Julian Elischer wrote: > If it come to pass that M:N threads are judged to be "unsuitable" then that > is a decision that I can live with, but never be let it be said that I > walled FreeBSD in to only having the option of 1:1 threads. Given that I have serious hindsight accuracy problems, I won't even talk about my foresight issues. :-) I think we shouldn't nail the KSE coffin closed just yet -- as Peter has already said, it's one thing to do a skunkworks hack of what might eventually be used, but it's quite another to show that the perceived benefit maps into real world benefit. I'd like to see that clearly shown before we do anything drastic. In particular, we need to show that the benefit is there for more than just a few poorly written benchmarks, but also for a range of real-world workloads, and that the scheduler simplifications pay off in terms of both code complexity and our ability to optimize. I also want to make sure we understand some of the artifacts better: the benefit of reducing per-process lock contention is significant, and I'm interested in understanding where on the workload spectrum the benefits of 1:1 outweight the benefits of M:N a bit better. Is my interpretation that this is M:N providing better kernel workload management correct, or is it an artifact of scheduler behavior? And, where future hardware and application trends will push workload behavior with respect to the optimizations we already have, not to mention the ones we are considering. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-threads@FreeBSD.ORG Thu Jul 6 12:10:22 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D0AD216A4FB; Thu, 6 Jul 2006 12:10:22 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 44C2143D45; Thu, 6 Jul 2006 12:10:22 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k66CALDF000371; Thu, 6 Jul 2006 08:10:21 -0400 (EDT) Date: Thu, 6 Jul 2006 08:10:21 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607032020.10993.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jul 2006 12:10:22 -0000 On Mon, 3 Jul 2006, David Xu wrote: > but reality is, mysql is using system scope in libpthread to get better > performance but not process scope thread. in the real world, these > nominal features are also not used widely, using it can only cause > serious performance problem in most applications, until you need hard > realtime, but why do you use FreeBSD if you need hard realtime ? > there are dedicated systems which are designed for this requirement. > I still does not find an application really need this feature. Also > I don't think one can easily implement a modern CPU friendly scheduler > in userland, HTT, shared L2 dual-core, NUMA scheduling, doing it in kernel is > already extreme diffcult, putting lots of effort in single library but other > single threaded application still gain nothing is also not worthy to work > on M:N thread library. I don't need hard real-time, but would like to be able to use FreeBSD for soft real-time. And note that if an application specifies SCHED_RR, SCHED_FIFO, or SCHED_SPORADIC, it doesn't matter if you have HTT, L2 dual-core, NUMA, whatever. Optimizing for those is just fine (for SCHED_OTHER), but in the presence of one of the above scheduling attributes it means _nothing_. libpthread uses SCHED_RR by default (SCHED_OTHER = SCHED_RR), but there is no reason that it couldn't just let the kernel schedule threads any way it wants (even without upcalls) by default (for SCHED_OTHER). -- DE From owner-freebsd-threads@FreeBSD.ORG Thu Jul 6 12:10:22 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D0AD216A4FB; Thu, 6 Jul 2006 12:10:22 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 44C2143D45; Thu, 6 Jul 2006 12:10:22 +0000 (GMT) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.13.6/8.13.6/NETPLEX) with ESMTP id k66CALDF000371; Thu, 6 Jul 2006 08:10:21 -0400 (EDT) Date: Thu, 6 Jul 2006 08:10:21 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Xu In-Reply-To: <200607032020.10993.davidxu@freebsd.org> Message-ID: References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) Cc: threads@freebsd.org, Robert Watson , freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jul 2006 12:10:22 -0000 On Mon, 3 Jul 2006, David Xu wrote: > but reality is, mysql is using system scope in libpthread to get better > performance but not process scope thread. in the real world, these > nominal features are also not used widely, using it can only cause > serious performance problem in most applications, until you need hard > realtime, but why do you use FreeBSD if you need hard realtime ? > there are dedicated systems which are designed for this requirement. > I still does not find an application really need this feature. Also > I don't think one can easily implement a modern CPU friendly scheduler > in userland, HTT, shared L2 dual-core, NUMA scheduling, doing it in kernel is > already extreme diffcult, putting lots of effort in single library but other > single threaded application still gain nothing is also not worthy to work > on M:N thread library. I don't need hard real-time, but would like to be able to use FreeBSD for soft real-time. And note that if an application specifies SCHED_RR, SCHED_FIFO, or SCHED_SPORADIC, it doesn't matter if you have HTT, L2 dual-core, NUMA, whatever. Optimizing for those is just fine (for SCHED_OTHER), but in the presence of one of the above scheduling attributes it means _nothing_. libpthread uses SCHED_RR by default (SCHED_OTHER = SCHED_RR), but there is no reason that it couldn't just let the kernel schedule threads any way it wants (even without upcalls) by default (for SCHED_OTHER). -- DE From owner-freebsd-threads@FreeBSD.ORG Thu Jul 6 12:30:33 2006 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 06DEF16A4E1; Thu, 6 Jul 2006 12:30:33 +0000 (UTC) (envelope-from nkoch@demig.de) Received: from server.absolute-media.de (server.absolute-media.de [213.239.231.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6112A43D67; Thu, 6 Jul 2006 12:30:32 +0000 (GMT) (envelope-from nkoch@demig.de) Received: from localhost (unknown [127.0.0.1]) by server.absolute-media.de (Postfix) with ESMTP id 3F975DDA05; Thu, 6 Jul 2006 14:30:29 +0200 (CEST) Received: from server.absolute-media.de ([127.0.0.1]) by localhost (server [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 15167-07; Thu, 6 Jul 2006 14:30:23 +0200 (CEST) Received: from firewall.demig (p50839DAD.dip0.t-ipconnect.de [80.131.157.173]) by server.absolute-media.de (Postfix) with ESMTP id AFF9FDDFF2; Thu, 6 Jul 2006 14:30:23 +0200 (CEST) Received: from [192.168.1.72] (ws-ew-3.demig.intra [192.168.1.72]) by firewall.demig (8.13.7/8.13.6) with ESMTP id k66CRqL5064854; Thu, 6 Jul 2006 14:27:52 +0200 (CEST) (envelope-from nkoch@demig.de) Message-ID: <44AD01C8.2040704@demig.de> Date: Thu, 06 Jul 2006 12:27:52 +0000 From: Norbert Koch User-Agent: Thunderbird 1.5.0.4 (Windows/20060516) MIME-Version: 1.0 References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at absolute-media.de Cc: threads@freebsd.org, freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jul 2006 12:30:33 -0000 > I don't need hard real-time, but would like to be able to use > FreeBSD for soft real-time. me too. Norbert Koch From owner-freebsd-threads@FreeBSD.ORG Thu Jul 6 12:30:33 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 06DEF16A4E1; Thu, 6 Jul 2006 12:30:33 +0000 (UTC) (envelope-from nkoch@demig.de) Received: from server.absolute-media.de (server.absolute-media.de [213.239.231.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6112A43D67; Thu, 6 Jul 2006 12:30:32 +0000 (GMT) (envelope-from nkoch@demig.de) Received: from localhost (unknown [127.0.0.1]) by server.absolute-media.de (Postfix) with ESMTP id 3F975DDA05; Thu, 6 Jul 2006 14:30:29 +0200 (CEST) Received: from server.absolute-media.de ([127.0.0.1]) by localhost (server [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 15167-07; Thu, 6 Jul 2006 14:30:23 +0200 (CEST) Received: from firewall.demig (p50839DAD.dip0.t-ipconnect.de [80.131.157.173]) by server.absolute-media.de (Postfix) with ESMTP id AFF9FDDFF2; Thu, 6 Jul 2006 14:30:23 +0200 (CEST) Received: from [192.168.1.72] (ws-ew-3.demig.intra [192.168.1.72]) by firewall.demig (8.13.7/8.13.6) with ESMTP id k66CRqL5064854; Thu, 6 Jul 2006 14:27:52 +0200 (CEST) (envelope-from nkoch@demig.de) Message-ID: <44AD01C8.2040704@demig.de> Date: Thu, 06 Jul 2006 12:27:52 +0000 From: Norbert Koch User-Agent: Thunderbird 1.5.0.4 (Windows/20060516) MIME-Version: 1.0 References: <20060703101554.Q26325@fledge.watson.org> <200607032020.10993.davidxu@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at absolute-media.de Cc: threads@freebsd.org, freebsd-threads@freebsd.org Subject: Re: Strawman proposal: making libthr default thread implementation? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jul 2006 12:30:33 -0000 > I don't need hard real-time, but would like to be able to use > FreeBSD for soft real-time. me too. Norbert Koch From owner-freebsd-threads@FreeBSD.ORG Fri Jul 7 09:43:26 2006 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC8E016A4E9 for ; Fri, 7 Jul 2006 09:43:26 +0000 (UTC) (envelope-from nora226@hotmail.co.jp) Received: from 218-170-23-247.dynamic.hinet.net (218-170-23-247.dynamic.hinet.net [218.170.23.247]) by mx1.FreeBSD.org (Postfix) with SMTP id 5A01F43D5E for ; Fri, 7 Jul 2006 09:43:25 +0000 (GMT) (envelope-from nora226@hotmail.co.jp) Received: from 68.187.222.238 by 218.170.23.247; Fri, 07 Jul 2006 05:45:03 -0400 Message-ID: From: "‘O“c ”üŒ‹" To: freebsd-threads@freebsd.org Date: Fri, 07 Jul 2006 06:36:03 -0300 X-Mailer: AOL 93.0 for Windows US sub 060 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-IP: 66.240.6.208 Content-Type: text/plain; Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: =?iso-8859-1?q?=82=A8=8Bv=82=B5=82=D4=82=E8=82=C5=82=B7?= X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ‘O“c ”üŒ‹ List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jul 2006 09:43:26 -0000 i+CI5JdEjnGCxpBcgrWC3IK3gUIgDQoNCpPLkVKCzIOBgVuDi5GXkE2BQZBcgrWW84KygrSC ooLcgrmC8YFCDQoNCoNYg3yDk4NUgVuDVINDg2eXbILmguiT8ZBsgsyPl5CrgsyP0InuiMuX ioLwjvOCr4LcgrWCvYLMgsWBQQ0KDQqCsojEk+CCs4K5gsSCrYK+grOCooFCgUIgDQoNCojq kNiXv4vglrOCtYLMgrKP0InugsaCyILogtyCt4FCDQoNCoOBgVuDi4LMkZeO85BNgvCK3ILf gUGCx4Lqgr6Cr5eYl3CCtYLEkriCooLEguCWs5e/gsWCt4FCDQoNCoqukVOWs5e/gsWCzIKy j9CJ7oLFgreCzILFkYGK+pL3gt+Q2ILogsyP6o2HgqqCsoK0gqKC3IK3gUINCg0KgtyCvYFB kveC35DYguiCxoLIgsGCvYLGgquCzZdcjZCCyIKtj5eQq4LMlc+NWILwgreC6Y/qjYeC4IKy grSCooLcgreCqoFBDQoNCoKyl2WOzYKtgr6Cs4KigUINCg0Kg06DioNig06By4FAaHR0cDov L3Nhd2ZseS5zY3JhbWJsZWtvc2F0ZW4uY29tL2ExMi8=•