From owner-freebsd-hackers Sun Jun 27 0:34: 7 1999 Delivered-To: freebsd-hackers@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [209.157.86.2]) by hub.freebsd.org (Postfix) with ESMTP id 2B1D714E57 for ; Sun, 27 Jun 1999 00:34:04 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id AAA10635; Sun, 27 Jun 1999 00:33:35 -0700 (PDT) (envelope-from dillon) Date: Sun, 27 Jun 1999 00:33:35 -0700 (PDT) From: Matthew Dillon Message-Id: <199906270733.AAA10635@apollo.backplane.com> To: "Daniel J. O'Connor" Cc: Jesus Monroy , Ville-Pertti Keinonen , hackers@FreeBSD.ORG, "Daniel J.OConnor" Subject: Re: [Re: [Re: coarse vs fine-grained locking in SMP systems]] References: Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Here's the basic problem: The kernel is currently designed for single-threaded operation plus interrupt handling. A piece of code in the kernel can temporarily disable certain interrupts with the spl*() codes to cover situations where a race on some system resource might occur. But with SMP, several cpu's may be running in supervisor mode simultaniously. The spl*() model breaks down because while one can block interrupts, one cannot easily block another cpu that might be running conflicting code. Resource races can now occur between mainline code running on several cpu's simultaniously as well as between mainline code and interrupt code. The traditional BSD kernel code cannot deal with this new type of race. At the moment every entry into supervisor mode is being governed by a "big giant lock" which only allows one cpu to run mainline code in supervisor mode at any given moment. Both cpu's can run usermode code simultaniously just fine, but only one can run supervisor code. In order to make SMP operation work better, pieces of the kernel are slowly being moved outside the "big giant lock". Linux developers, in fact, have already moved their core data copying code and their TCP stack outside the lock. At the moment the FreeBSD-current kernel has not moved anything outside the lock, but John Dyson has shown that it is fairly easy to move certain specific pieces such as the uiomove() code outside the lock, though inefficiencies from side-effects currently make the improvement in performance less then steller. The real question is how to manage concurrency as pieces get moved outside the lock. There are lots of ways to do it. One can use spin locks to protect resources or, as someone pointed out earlier, to protect sections of code. I don't know which is better myself, it probably depends on the situation so a hybrid will probably be the end result. One can also use kernel threads to simplify resource management. The advantage of a kernel thread verses a normal process is in the ability to switch between kernel threads very quickly, allowing the time normally wasted spining in certain types of locks to be used more efficiently. The problem that generally needs to be solved is the problem of stalling on a resource. For example, if you have several threads running simultaniously and they all need access to the same resource, serialization of the threads occurs due to the 'blockage' on access to the resource. (serialization means that only one thread can run at a time within the resource, which means your efficiency drops to the efficiency of only a single cpu). There are lots of other issues (such as cache efficiency), but that is the big one. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message