From owner-freebsd-smp@FreeBSD.ORG Tue Sep 16 19:48:07 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B2DF616A4B3 for ; Tue, 16 Sep 2003 19:48:07 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4F99743FBD for ; Tue, 16 Sep 2003 19:48:06 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from gamplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3p2/8.8.7) with ESMTP id MAA17388 for ; Wed, 17 Sep 2003 12:48:03 +1000 Date: Wed, 17 Sep 2003 12:46:41 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: smp@freebsd.org Message-ID: <20030917122543.O8168@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 02:48:07 -0000 What guarantees, if any, are there that an unlocked read provides a valid value (either the current value or a previous value)? Obviously there are no guarantees if the size of the object being read is different from the natural memory access size. I'm mainly interested in atomic reads of pointers in circular buffers. The read pointer is only written to by one thread and is locked by lock R. The write pointer is only written to by another thread and is locked by a different lock W. Each thread needs to read but not write the other thread's pointer but doesn't care if it sees a stale value (it will see an up to date value later), but does care if it sees a garbage value. It would be nice if each thread doesn't have to use the other thread's lock, especially when one of the threads is actually a fast interrupt handler so it can't use the other thread's lock unless it is a spinlock but wants to be a sleep lock. Sometimes even a garbage value from reading an object non-atomically might not matter. E.g., in sigpending() there seems to be no point in locking the read, since a snapshot that is inconsistent due to not locking during the read is little different from a snapshot that is inconsistent due to the object changing after it is read. Bruce From owner-freebsd-smp@FreeBSD.ORG Tue Sep 16 20:03:34 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 240DE16A4BF for ; Tue, 16 Sep 2003 20:03:34 -0700 (PDT) Received: from tinker.exit.com (tinker.exit.com [206.223.0.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id DF8D043FB1 for ; Tue, 16 Sep 2003 20:03:30 -0700 (PDT) (envelope-from frank@exit.com) Received: from realtime.exit.com (realtime [206.223.0.5]) by tinker.exit.com (8.12.9/8.12.6) with ESMTP id h8H33NPk002287; Tue, 16 Sep 2003 20:03:26 -0700 (PDT) (envelope-from frank@exit.com) Received: from realtime.exit.com (localhost [127.0.0.1]) by realtime.exit.com (8.12.9/8.12.6) with ESMTP id h8H33MYo083419; Tue, 16 Sep 2003 20:03:22 -0700 (PDT) (envelope-from frank@realtime.exit.com) Received: (from frank@localhost) by realtime.exit.com (8.12.9/8.12.9/Submit) id h8H33HtU083418; Tue, 16 Sep 2003 20:03:17 -0700 (PDT) From: Frank Mayhar Message-Id: <200309170303.h8H33HtU083418@realtime.exit.com> In-Reply-To: <20030917122543.O8168@gamplex.bde.org> To: Bruce Evans Date: Tue, 16 Sep 2003 20:03:17 -0700 (PDT) X-Copyright0: Copyright 2003 Frank Mayhar. All Rights Reserved. X-Copyright1: Permission granted for electronic reproduction as Usenet News or email only. X-Mailer: ELM [version 2.4ME+ PL99f (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: frank@exit.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 03:03:34 -0000 Bruce Evans wrote: > What guarantees, if any, are there that an unlocked read provides a > valid value (either the current value or a previous value)? Obviously > there are no guarantees if the size of the object being read is different > from the natural memory access size. I think that this may depend on the architecture of the target, but as I understand it, in general an entity that is read in one (bus) operation is guaranteed to be valid. These days that entity is generally 32 or 64 bits. The key is that the bus is locked during the entire read or write cycle so that reads and writes are serialized by the hardware. So the answer is that yes, there are guarantees, but you still have to be careful and aware of the semantics of the target hardware. -- Frank Mayhar frank@exit.com http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ From owner-freebsd-smp@FreeBSD.ORG Tue Sep 16 23:26:51 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2587816A4B3 for ; Tue, 16 Sep 2003 23:26:51 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2DB4C43FBF for ; Tue, 16 Sep 2003 23:25:45 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from gamplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3p2/8.8.7) with ESMTP id QAA21425; Wed, 17 Sep 2003 16:25:32 +1000 Date: Wed, 17 Sep 2003 16:24:10 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Frank Mayhar In-Reply-To: <200309170303.h8H33HtU083418@realtime.exit.com> Message-ID: <20030917161747.Q587@gamplex.bde.org> References: <200309170303.h8H33HtU083418@realtime.exit.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 06:26:51 -0000 On Tue, 16 Sep 2003, Frank Mayhar wrote: > Bruce Evans wrote: > > What guarantees, if any, are there that an unlocked read provides a > > valid value (either the current value or a previous value)? Obviously > > there are no guarantees if the size of the object being read is different > > from the natural memory access size. > > I think that this may depend on the architecture of the target, but as I > understand it, in general an entity that is read in one (bus) operation > is guaranteed to be valid. These days that entity is generally 32 or 64 > bits. The key is that the bus is locked during the entire read or write > cycle so that reads and writes are serialized by the hardware. > > So the answer is that yes, there are guarantees, but you still have to > be careful and aware of the semantics of the target hardware. I'll give up micro-optimizing this and use an atomic instruction since I want it to work in MI code without ifdefs for the target hardware. atomic_load_ptr() could hide the details, but we currently only have atomic_load_acq_ptr() which gives more than I need. Bruce From owner-freebsd-smp@FreeBSD.ORG Wed Sep 17 08:45:46 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 69C8F16A4B3 for ; Wed, 17 Sep 2003 08:45:46 -0700 (PDT) Received: from mail.speakeasy.net (mail7.speakeasy.net [216.254.0.207]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA3E243FBF for ; Wed, 17 Sep 2003 08:45:45 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: (qmail 22461 invoked from network); 17 Sep 2003 15:45:43 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 17 Sep 2003 15:45:43 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.9/8.12.9) with ESMTP id h8HFjZ6Y082459; Wed, 17 Sep 2003 11:45:35 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20030917122543.O8168@gamplex.bde.org> Date: Wed, 17 Sep 2003 11:45:37 -0400 (EDT) From: John Baldwin To: Bruce Evans X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) cc: smp@freebsd.org Subject: RE: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 15:45:46 -0000 On 17-Sep-2003 Bruce Evans wrote: > What guarantees, if any, are there that an unlocked read provides a > valid value (either the current value or a previous value)? Obviously > there are no guarantees if the size of the object being read is different > from the natural memory access size. > > I'm mainly interested in atomic reads of pointers in circular buffers. > The read pointer is only written to by one thread and is locked by > lock R. The write pointer is only written to by another thread and > is locked by a different lock W. Each thread needs to read but not > write the other thread's pointer but doesn't care if it sees a stale > value (it will see an up to date value later), but does care if it > sees a garbage value. It would be nice if each thread doesn't have > to use the other thread's lock, especially when one of the threads is > actually a fast interrupt handler so it can't use the other thread's > lock unless it is a spinlock but wants to be a sleep lock. > > Sometimes even a garbage value from reading an object non-atomically > might not matter. E.g., in sigpending() there seems to be no point > in locking the read, since a snapshot that is inconsistent due to not > locking during the read is little different from a snapshot that is > inconsistent due to the object changing after it is read. I think you can assume that the read will be atomic. I don't think FreeBSD would work very well on a machine where aligned pointer reads/writes weren't atomic. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ From owner-freebsd-smp@FreeBSD.ORG Wed Sep 17 10:51:02 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D280616A4B3; Wed, 17 Sep 2003 10:51:02 -0700 (PDT) Received: from tinker.exit.com (tinker.exit.com [206.223.0.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5AA7743F85; Wed, 17 Sep 2003 10:51:01 -0700 (PDT) (envelope-from frank@exit.com) Received: from realtime.exit.com (realtime [206.223.0.5]) by tinker.exit.com (8.12.9/8.12.6) with ESMTP id h8HHoxPk006730; Wed, 17 Sep 2003 10:51:00 -0700 (PDT) (envelope-from frank@exit.com) Received: from realtime.exit.com (localhost [127.0.0.1]) by realtime.exit.com (8.12.9/8.12.6) with ESMTP id h8HHoxYo092628; Wed, 17 Sep 2003 10:50:59 -0700 (PDT) (envelope-from frank@realtime.exit.com) Received: (from frank@localhost) by realtime.exit.com (8.12.9/8.12.9/Submit) id h8HHowEf092627; Wed, 17 Sep 2003 10:50:58 -0700 (PDT) From: Frank Mayhar Message-Id: <200309171750.h8HHowEf092627@realtime.exit.com> In-Reply-To: To: John Baldwin Date: Wed, 17 Sep 2003 10:50:58 -0700 (PDT) X-Copyright0: Copyright 2003 Frank Mayhar. All Rights Reserved. X-Copyright1: Permission granted for electronic reproduction as Usenet News or email only. X-Mailer: ELM [version 2.4ME+ PL99f (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: frank@exit.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 17:51:02 -0000 John Baldwin wrote: > I think you can assume that the read will be atomic. I don't think FreeBSD > would work very well on a machine where aligned pointer reads/writes weren't > atomic. I dunno, I tend to think that making such assumptions may well come back to bite me in the ass when I least expect it. In such situation, I invariably use some kind of "atomic_xxx_load/store" primitive that does the job safely and in a machine-dependent way while hiding the details from the MI code. Just call me paranoid. -- Frank Mayhar frank@exit.com http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ From owner-freebsd-smp@FreeBSD.ORG Wed Sep 17 11:22:42 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 16C9116A4B3; Wed, 17 Sep 2003 11:22:42 -0700 (PDT) Received: from cs.rice.edu (cs.rice.edu [128.42.1.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8B08E43FBD; Wed, 17 Sep 2003 11:22:40 -0700 (PDT) (envelope-from alc@cs.rice.edu) Received: from localhost (localhost [127.0.0.1]) by localhost.cs.rice.edu (Postfix) with ESMTP id EA2284AA38; Wed, 17 Sep 2003 13:22:39 -0500 (CDT) Received: from cs.rice.edu ([127.0.0.1]) by localhost (cs.rice.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 24502-05; Wed, 17 Sep 2003 13:22:37 -0500 (CDT) Received: by cs.rice.edu (Postfix, from userid 19572) id B0AC34A9D7; Wed, 17 Sep 2003 13:22:37 -0500 (CDT) Date: Wed, 17 Sep 2003 13:22:37 -0500 From: Alan Cox To: Frank Mayhar Message-ID: <20030917182237.GM12711@cs.rice.edu> References: <200309171750.h8HHowEf092627@realtime.exit.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200309171750.h8HHowEf092627@realtime.exit.com> User-Agent: Mutt/1.3.28i X-Virus-Scanned: by amavis-20030314-p2 at cs.rice.edu cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 18:22:42 -0000 On Wed, Sep 17, 2003 at 10:50:58AM -0700, Frank Mayhar wrote: > John Baldwin wrote: > > I think you can assume that the read will be atomic. I don't think FreeBSD > > would work very well on a machine where aligned pointer reads/writes weren't > > atomic. > > I dunno, I tend to think that making such assumptions may well come back to > bite me in the ass when I least expect it. In such situation, I invariably > use some kind of "atomic_xxx_load/store" primitive that does the job safely > and in a machine-dependent way while hiding the details from the MI code. > Indeed. Atomicity of the read is not the only issue. Unless I misunderstood, Bruce is proposing to perform an unsynchronized read of a location that is updated by a different processor. This unsynchronized read is to obtain a pointer. In weaker memory consistency models (than that of the x86), there is no guarantee that the data that the pointer refers to will be consistent. In other words, updates to the data that the pointer refers to may not have completed, even though the update to the memory location containing the pointer has completed. This is what an "acquire load" is for. Roughly speaking, it should guarantee that any updates prior to a corresponding "release store" (by a different processor) will be seen. If we don't use "acquire loads" and "release stores" under such circumstances, the poor guy who ports FreeBSD to a processor that, for example, reorders loads, will be an unhappy camper. Alan From owner-freebsd-smp@FreeBSD.ORG Wed Sep 17 12:58:03 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 83FAE16A4B3 for ; Wed, 17 Sep 2003 12:58:03 -0700 (PDT) Received: from linda-1.paradise.net.nz (bm-1a.paradise.net.nz [202.0.58.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9170F43FBF for ; Wed, 17 Sep 2003 12:58:02 -0700 (PDT) (envelope-from trademe@vidaicsystems.com) Received: from smtp-3.paradise.net.nz (smtp-3b.paradise.net.nz [202.0.32.212]) by linda-1.paradise.net.nz (Paradise.net.nz) with ESMTP id <0HLD00CIZJGPR0@linda-1.paradise.net.nz> for freebsd-smp@freebsd.org; Thu, 18 Sep 2003 07:58:01 +1200 (NZST) Received: from big (218-101-44-36.paradise.net.nz [218.101.44.36]) by smtp-3.paradise.net.nz (Postfix) with ESMTP id C0B42ADF17 for ; Thu, 18 Sep 2003 07:58:00 +1200 (NZST) Date: Thu, 18 Sep 2003 07:56:00 +1200 From: Patrick Brennan To: freebsd-smp@freebsd.org Message-id: <3F696510.2995.E433456@localhost> X-Mailer: Pegasus Mail for Windows (v4.02a) Priority: normal Subject: 6 x PPro X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: trademe@vidaicsystems.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Sep 2003 19:58:03 -0000 Hi, I have just aquired an ALR 6 x PPro 200 machine. I know there is a lot of development work in the SMP area, so I have a few questions: Will the 4-Series SMP kernel utilise more than 4 processors? How well does the 5-Series SMP kernel work? Is it stable? I understand there are at least two different implementations that you can specify in the kernel_build.conf? Is there anything else I should know?! Thanks, Patrick Brennan From owner-freebsd-smp@FreeBSD.ORG Wed Sep 17 21:45:30 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8D43516A4B3 for ; Wed, 17 Sep 2003 21:45:30 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id E950B43FAF for ; Wed, 17 Sep 2003 21:45:28 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from gamplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3p2/8.8.7) with ESMTP id OAA22359; Thu, 18 Sep 2003 14:45:03 +1000 Date: Thu, 18 Sep 2003 14:43:40 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Alan Cox In-Reply-To: <20030917182237.GM12711@cs.rice.edu> Message-ID: <20030918134424.N1635@gamplex.bde.org> References: <20030917182237.GM12711@cs.rice.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2003 04:45:30 -0000 On Wed, 17 Sep 2003, Alan Cox wrote: > On Wed, Sep 17, 2003 at 10:50:58AM -0700, Frank Mayhar wrote: > > John Baldwin wrote: > > > I think you can assume that the read will be atomic. I don't think FreeBSD > > > would work very well on a machine where aligned pointer reads/writes weren't > > > atomic. > > > > I dunno, I tend to think that making such assumptions may well come back to > > bite me in the ass when I least expect it. In such situation, I invariably > > use some kind of "atomic_xxx_load/store" primitive that does the job safely > > and in a machine-dependent way while hiding the details from the MI code. > > Indeed. Atomicity of the read is not the only issue. Unless I > misunderstood, Bruce is proposing to perform an unsynchronized read of > a location that is updated by a different processor. This > unsynchronized read is to obtain a pointer. In weaker memory > consistency models (than that of the x86), there is no guarantee that > the data that the pointer refers to will be consistent. In other > words, updates to the data that the pointer refers to may not have > completed, even though the update to the memory location containing > the pointer has completed. The data is known to be consistent in my application. I'm just implementing a circular buffer with essentially 1 writer and 1 reader (they can be on different CPUs and the writer can be different, but locking for each gives coherency. Only the writer writes to the buffer, so the data is always consistent. The reader advances the read pointer (normally without telling the writer), and the writer only needs the read pointer to determine the space available for writing. If it sees an old value then it underestimates the space, but there is no a problem because the writer will get woken up by the reader when the space goes below a watermark. If the writer sees a garbage value then it will compute a garbage amount of space and may write more than fits. Reading the read pointer without locking is only a micro-optimization here, but I believe there are some lengthy list traversals that would benefit from similar optimizations. (Don't lock each element or the whole list, but somehow ensure that the elements or the list pointers in them are not garbage even if they are stale.) > This is what an "acquire load" is for. Roughly speaking, it should > guarantee that any updates prior to a corresponding "release store" > (by a different processor) will be seen. For the optimized circular buffer, stricter locking of the write pointer is required for essentially this reason. The situation with the read and write pointers is not symmetrical like I said when I started this thread. Fortunately, stricter locking for the write pointer is more natural and I already did it without thinking much about it. Bruce From owner-freebsd-smp@FreeBSD.ORG Wed Sep 17 22:57:58 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BAC0C16A4B3 for ; Wed, 17 Sep 2003 22:57:58 -0700 (PDT) Received: from tinker.exit.com (tinker.exit.com [206.223.0.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id B483E43F85 for ; Wed, 17 Sep 2003 22:57:57 -0700 (PDT) (envelope-from frank@exit.com) Received: from realtime.exit.com (realtime [206.223.0.5]) by tinker.exit.com (8.12.9/8.12.6) with ESMTP id h8I5vt6m001806; Wed, 17 Sep 2003 22:57:55 -0700 (PDT) (envelope-from frank@exit.com) Received: from realtime.exit.com (localhost [127.0.0.1]) by realtime.exit.com (8.12.9/8.12.6) with ESMTP id h8I5vtYo003135; Wed, 17 Sep 2003 22:57:55 -0700 (PDT) (envelope-from frank@realtime.exit.com) Received: (from frank@localhost) by realtime.exit.com (8.12.9/8.12.9/Submit) id h8I5vouP003134; Wed, 17 Sep 2003 22:57:50 -0700 (PDT) From: Frank Mayhar Message-Id: <200309180557.h8I5vouP003134@realtime.exit.com> In-Reply-To: <20030918134424.N1635@gamplex.bde.org> To: Bruce Evans Date: Wed, 17 Sep 2003 22:57:50 -0700 (PDT) X-Copyright0: Copyright 2003 Frank Mayhar. All Rights Reserved. X-Copyright1: Permission granted for electronic reproduction as Usenet News or email only. X-Mailer: ELM [version 2.4ME+ PL99f (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: frank@exit.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2003 05:57:58 -0000 To throw another $.02 into the fray while I kill time between job inquiries (sigh), Intel does have a few guaranteed-atomic operations. From the IA-32 Software Developer's Manual Vol 3, "Guaranteed Atomic Operations," page 7-3: The Pentium 4, Intel Xeon, P6 family, Pentium, and Intel486 processors guarantee that the following basic memory operations will always be carried out atomically: o Reading or writing a byte. o Reading or writing a word aligned on a 16-bit boundary. o Reading or writing a doubleword aligned on a 32-bit boundary. The Pentium 4, Intel Xeon, and P6 family, and Pentium processors guarantee that the following additional memory operations will always be carried out atomically: o Reading or writing a quadword aligned on a 64-bit boundary. o 16-bit accesses to uncached memory locations that fit within a 32-bit data bus. The P6 family processors guarantee that the following additional memory operation will always be carried out atomically: o Unaligned 16-, 32-, and 64-bit accesses to cached memory that fit within a 32-byte cache line. (I wasn't consciously aware of these last two but they had occurred to me when I wrote my previous email on this topic. I'm certainly not surprised; I would have been surprised if this _hadn't_ been the case.) I don't have the docs for Alpha or PowerPC so I can't speak for those architectures. Bruce, you can certainly get away with doing your reads unprotected, but I would still urge you to hide this inside an "atomic_xxx_load" macro, just so it's caught properly by the ports to the other architectures. On the Intel architecture it's a cheap and easy optimization. On another architecture it may not be so cheap and easy, but at least it will _work_. -- Frank Mayhar frank@exit.com http://www.exit.com/ Exit Consulting http://www.gpsclock.com/ http://www.exit.com/blog/frank/ From owner-freebsd-smp@FreeBSD.ORG Thu Sep 18 01:24:00 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1135A16A4B3; Thu, 18 Sep 2003 01:24:00 -0700 (PDT) Received: from stork.mail.pas.earthlink.net (stork.mail.pas.earthlink.net [207.217.120.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4AAD343FBF; Thu, 18 Sep 2003 01:23:59 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from user-2ivfk1k.dialup.mindspring.com ([165.247.208.52] helo=mindspring.com) by stork.mail.pas.earthlink.net with asmtp (SSLv3:RC4-MD5:128) (Exim 3.33 #1) id 19zu4z-0003Bd-00; Thu, 18 Sep 2003 01:23:57 -0700 Message-ID: <3F696B63.A5F36511@mindspring.com> Date: Thu, 18 Sep 2003 01:22:59 -0700 From: Terry Lambert X-Mailer: Mozilla 4.79 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: frank@exit.com References: <200309171750.h8HHowEf092627@realtime.exit.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a413764d539f1029f7c37c712b5a7f44f5667c3043c0873f7e350badd9bab72f9c350badd9bab72f9c cc: smp@freebsd.org Subject: Re: atomicity of unlocked reads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2003 08:24:00 -0000 Frank Mayhar wrote: > John Baldwin wrote: > > I think you can assume that the read will be atomic. I don't think FreeBSD > > would work very well on a machine where aligned pointer reads/writes weren't > > atomic. > > I dunno, I tend to think that making such assumptions may well come back to > bite me in the ass when I least expect it. In such situation, I invariably > use some kind of "atomic_xxx_load/store" primitive that does the job safely > and in a machine-dependent way while hiding the details from the MI code. > > Just call me paranoid. It would definitely bite you on an MP PPC machine, which can do the operations non-atomically, branch-predictively, and doesn't support a "lock" prefix to its instructions. One of the side effects of doing a lock is that you will get an idync and dsync that flush the data and instruction caches, and ensure cache line coherency between processors. It would probably bite you on a SPARC as well, unless you explicitly flushed your register windows before and after the operation (and even then, you would likely need additional voodoo for anything over 4 CPUs, the way they do things). -- Terry From owner-freebsd-smp@FreeBSD.ORG Thu Sep 18 19:31:08 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9FABB16A4B3 for ; Thu, 18 Sep 2003 19:31:08 -0700 (PDT) Received: from smtpout-3104.bay.webtv.net (smtpout-3104.bay.webtv.net [209.240.204.195]) by mx1.FreeBSD.org (Postfix) with ESMTP id 22F9043FAF for ; Thu, 18 Sep 2003 19:31:08 -0700 (PDT) (envelope-from Dumb_Reminders@webtv.net) Received: from storefull-2214.public.lawson.webtv.net (storefull-2214.public.lawson.webtv.net [209.240.213.76])E67A510666 for ; Thu, 18 Sep 2003 19:31:07 -0700 (PDT) Received: (from production@localhost) by storefull-2214.public.lawson.webtv.net (8.8.8-wtv-f/mt.gso.26Feb98) id TAA23598; Thu, 18 Sep 2003 19:31:07 -0700 (PDT) X-WebTV-Signature: 1 ETAtAhQis7NlgWU1Gf+T7SHlYf6/gmL0YgIVAIQAv1WwBbvYaQpUp9ZkjhhDN2oV From: Dumb_Reminders@webtv.net (No_Use For_A_Name) Date: Thu, 18 Sep 2003 16:31:07 -1000 (HST) To: freebsd-smp@FreeBSD.org Message-ID: <15238-3F6A6A6B-2413@storefull-2214.public.lawson.webtv.net> Content-Type: Text/Plain; Charset=US-ASCII Content-Transfer-Encoding: 7Bit MIME-Version: 1.0 (WebTV) Subject: X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2003 02:31:08 -0000 From owner-freebsd-smp@FreeBSD.ORG Fri Sep 19 06:01:14 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B0A6716A4B3 for ; Fri, 19 Sep 2003 06:01:14 -0700 (PDT) Received: from sferics.mongueurs.net (sferics.mongueurs.net [81.80.147.197]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D24A43FBD for ; Fri, 19 Sep 2003 06:01:13 -0700 (PDT) (envelope-from david@landgren.net) Received: from landgren.net (81-80-147-206.bpinet.com [81.80.147.206]) by sferics.mongueurs.net (Postfix) with ESMTP id C3BF0A990 for ; Fri, 19 Sep 2003 15:01:09 +0200 (CEST) Message-ID: <3F6AFE29.4080502@landgren.net> Date: Fri, 19 Sep 2003 15:01:29 +0200 From: David Landgren Organization: A thousand golden eyes are watching User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5a) Gecko/20030718 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-smp@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: During boot: "Programming 16 pins in IOAPIC" ... and then hangs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2003 13:01:14 -0000 Greetings list, I have an HP DL380-G3 with a P4 processor. I upgraded the source tree last night to 4.9-PRERELEASE. This is the last server I have that is running a GENERIC kernel, two others are successfully running SMP kernels (insofar as hw.ncpus = 2). I copied over a successful kernel configuration file and tried to build an SMP kernel. The new kernel boots with the following: Programming 16 pins in IOAPIc #0 IOAPIC #0 intpin 2 -> irq 0 Programming 16 pins in IOAPIc #1 Programming 16 pins in IOAPIc #2 Programming 16 pins in IOAPIc #3 ... and then hangs. If I comment out options SMP options APIC_IO options HTT in the kernel configuration file it boots correctly. I've searched around, and found that someone had a similar problem [1]. The suggestion there was to increase "your NINTR to 32", and also talks about APIC_INTMAPSIZE and ICU_LEN. These don't appear to be kernel options, at least, grep APIC LINT doesn't return anything else. I'm now worried that if I upgrade the other two servers they'll die on reboot. Included below is the kernel configuration file used Thanks for any clues I can use, David References: 1. http://www.geocrawler.com/archives/3/171/2000/2/0/3274582/ Kernel configuration file: # BEGIN # machine i386 cpu I686_CPU ident DL380G3B maxusers 0 options MAXDSIZ="(1024*1024*1024)" options MAXSSIZ="(256*1024*1024)" options DFLDSIZ="(256*1024*1024)" options NSWAPDEV=1 options MATH_EMULATE options INET options INET6 options FFS options FFS_ROOT options SOFTUPDATES options UFS_DIRHASH options MFS options MD_ROOT options MSDOSFS options CD9660 options CD9660_ROOT options PROCFS options COMPAT_43 options SCSI_DELAY=15000 options UCONSOLE options USERCONFIG options VISUAL_USERCONFIG options KTRACE options SYSVSHM options SYSVMSG options SYSVSEM options P1003_1B options _KPOSIX_PRIORITY_SCHEDULING options ICMP_BANDLIM options KBD_INSTALL_CDEV options AHC_REG_PRETTY_PRINT options AHD_REG_PRETTY_PRINT options SMP options APIC_IO options HTT device isa device eisa device pci device fdc0 at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 device fd1 at fdc0 drive 1 device ata0 at isa? port IO_WD1 irq 14 device ata1 at isa? port IO_WD2 irq 15 device ata device atadisk device atapicd device atapifd options ATA_STATIC_ID device ahb device ahc device ahd device adv0 at isa? device adw device bt0 at isa? device aha0 at isa? device aic0 at isa? device scbus device da device pass device ciss device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 flags 0x1 device psm0 at atkbdc? irq 12 device vga0 at isa? device sc0 at isa? flags 0x100 device agp device npx0 at nexus? port IO_NPX irq 13 device miibus device bge pseudo-device splash pseudo-device loop pseudo-device ether pseudo-device tun pseudo-device pty pseudo-device md pseudo-device gif pseudo-device faith 1 pseudo-device bpf options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_FORWARD options IPFIREWALL_VERBOSE_LIMIT=500 options IPDIVERT # END # From owner-freebsd-smp@FreeBSD.ORG Fri Sep 19 09:56:12 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 047A016A4B3 for ; Fri, 19 Sep 2003 09:56:12 -0700 (PDT) Received: from mail.speakeasy.net (mail14.speakeasy.net [216.254.0.214]) by mx1.FreeBSD.org (Postfix) with ESMTP id 65D1243FAF for ; Fri, 19 Sep 2003 09:56:10 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: (qmail 22786 invoked from network); 19 Sep 2003 16:56:09 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender )encrypted SMTP for ; 19 Sep 2003 16:56:09 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.9/8.12.9) with ESMTP id h8JGtr6Y093894; Fri, 19 Sep 2003 12:55:54 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.4 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3F6AFE29.4080502@landgren.net> Date: Fri, 19 Sep 2003 12:55:55 -0400 (EDT) From: John Baldwin To: David Landgren X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) cc: freebsd-smp@freebsd.org Subject: RE: During boot: "Programming 16 pins in IOAPIC" ... and then hangs X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2003 16:56:12 -0000 On 19-Sep-2003 David Landgren wrote: > Greetings list, > > I have an HP DL380-G3 with a P4 processor. I upgraded the source tree > last night to 4.9-PRERELEASE. This is the last server I have that is > running a GENERIC kernel, two others are successfully running SMP > kernels (insofar as hw.ncpus = 2). > > I copied over a successful kernel configuration file and tried to > build an SMP kernel. The new kernel boots with the following: > > Programming 16 pins in IOAPIc #0 > IOAPIC #0 intpin 2 -> irq 0 > Programming 16 pins in IOAPIc #1 > Programming 16 pins in IOAPIc #2 > Programming 16 pins in IOAPIc #3 > > ... and then hangs. If I comment out > > options SMP > options APIC_IO > options HTT Look in your BIOS and see if there is an option to make it create a "full mptable". If so, turn that on and try again. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/