From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 00:52:09 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 544CA16A41A for ; Sun, 30 Dec 2007 00:52:09 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id CEAA013C46E for ; Sun, 30 Dec 2007 00:52:08 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1J8mPN-0006xI-NP for freebsd-hackers@freebsd.org; Sun, 30 Dec 2007 00:52:05 +0000 Received: from 78-1-83-158.adsl.net.t-com.hr ([78.1.83.158]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 30 Dec 2007 00:52:05 +0000 Received: from ivoras by 78-1-83-158.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 30 Dec 2007 00:52:05 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Sun, 30 Dec 2007 01:51:54 +0100 Lines: 43 Message-ID: References: <20071229.122221.-432830441.imp@bsdimp.com> <4776d1d7.zI7kRv9uFoaBNKnQ%perryh@pluto.rain.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig80BE01B2E3758187A5AE944C" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-83-158.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: <4776d1d7.zI7kRv9uFoaBNKnQ%perryh@pluto.rain.com> X-Enigmail-Version: 0.95.5 Sender: news Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 00:52:09 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig80BE01B2E3758187A5AE944C Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable perryh@pluto.rain.com wrote: > The degree to which a PowerPC imposes a strict alignment requirement > depends on both the particular processor model and the operation > being performed. >=20 > For ordinary integer arithmetic and logical operations, newer > PPC processors tend to be more tolerant (although misalignment > will typically carry a performance penalty). For the semaphore > primitives (lwarx/stwcx.) most PPC will require proper alignment > and some will fault if the operand address is cache-inhibited > (even though correctly aligned). How would it behave in operations like x =3D x + 1 where x is unaligned in memory? A RISC would have to load the value from memory, increment it and store it. I'm not particularly interested in slowdowns, just hard faults. --------------enig80BE01B2E3758187A5AE944C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHduuqldnAQVacBcgRAo1aAJ9JcQ32lVk0QfJL7kmPWXQjFO0/5QCaA+Z4 1h8NbZU3wcv8rs36rE3lhp8= =Cq8a -----END PGP SIGNATURE----- --------------enig80BE01B2E3758187A5AE944C-- From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 03:23:05 2007 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DD9816A41A for ; Sun, 30 Dec 2007 03:23:05 +0000 (UTC) (envelope-from eddy+public+spam@noc.everquick.net) Received: from a.mx.ict1.everquick.net (a.mx.everquick.net [204.10.191.21]) by mx1.freebsd.org (Postfix) with ESMTP id 34FB613C448 for ; Sun, 30 Dec 2007 03:23:05 +0000 (UTC) (envelope-from eddy+public+spam@noc.everquick.net) Received: from pop.ict1.everquick.net (localhost [127.0.0.1]) by a.mx.ict1.everquick.net (8.12.10/8.12.10) with ESMTP id lBU3Lcmp004064; Sun, 30 Dec 2007 03:21:38 GMT X-Everquick-No-Abuse-1: Report any email abuse to or X-Everquick-No-Abuse-2: call +1 (785) 865-5885. Please be sure to reference X-Everquick-No-Abuse-3: the Message-Id and include GMT timestamps. Received: from localhost (eddy@localhost) by pop.ict1.everquick.net (8.13.3/8.13.3/Submit) with ESMTP id lBU3LaYX004058; Sun, 30 Dec 2007 03:21:38 GMT X-Authentication-Warning: pop.ict1.everquick.net: eddy owned process doing -bs Date: Sun, 30 Dec 2007 03:21:35 +0000 (GMT) From: "Edward B. DREGER" X-X-Sender: eddy@pop.ict1.everquick.net To: Peter Jeremy In-Reply-To: <20071229213521.GW40785@server.vk2pj.dyndns.org> Message-ID: References: <5950EE0C-383D-4D6B-9991-A0DEABD2ADE4@u.washington.edu> <7F9D2F63-B5E6-41DE-843A-8D673C2DC88E@u.washington.edu> <20071229213521.GW40785@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: hackers@freebsd.org Subject: Re: BSD license compatible hash algorithm? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 03:23:05 -0000 PJ> Date: Sun, 30 Dec 2007 08:35:21 +1100 PJ> From: Peter Jeremy PJ> On Sat, Dec 29, 2007 at 08:50:14PM +0000, Edward B. DREGER wrote: PJ> > PJ> >perfect_hash = ( hash1[x] + hash2[x] ) % entry_count ; PJ> This relies on pre-knowledge of all possible entries. It's PJ> excellent for (eg) keyword lookups in a compiler (and gcc uses gperf PJ> for that reason) but no good where the input can be arbitrary. IIUC, Garrett wanted to eliminate duplicates. When an item is found, check against the current [OP]MPH table; either make new reference or follow existing, as appropriate. IIRC, CHM92 had to precompute the entire table. However, one can implement OPMPHashing in a way that supports dynamic deletions and incremental additions. Unfortunately, I know of no implementation that's either BSD-licensed or in C, let alone both. It's been nearly a couple years since I wrote a [proprietary] C++ implementation, but I recall that it wasn't terribly difficult. The trick for incremental additions is to deal with hash collisions, drastically reducing the probability of a graph cycle. Thus, one rarely must discard the entire OPMPH table in favor of one built from new hash functions. (Note that the OPMPH table must still be "large enough" to deal with the additional items. I forget the exact numbers, but space requirements were notably less than CHM92.) Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter. From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 06:14:36 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C30316A419; Sun, 30 Dec 2007 06:14:36 +0000 (UTC) (envelope-from perryh@pluto.rain.com) Received: from agora.rdrop.com (agora.rdrop.com [199.26.172.34]) by mx1.freebsd.org (Postfix) with ESMTP id 4D14713C442; Sun, 30 Dec 2007 06:14:36 +0000 (UTC) (envelope-from perryh@pluto.rain.com) Received: from agora.rdrop.com (66@localhost [127.0.0.1]) by agora.rdrop.com (8.13.1/8.12.7) with ESMTP id lBU6EZUN029989 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 29 Dec 2007 22:14:35 -0800 (PST) (envelope-from perryh@pluto.rain.com) Received: (from uucp@localhost) by agora.rdrop.com (8.13.1/8.12.9/Submit) with UUCP id lBU6EZ64029988; Sat, 29 Dec 2007 22:14:35 -0800 (PST) Received: from fbsd61 by pluto.rain.com (4.1/SMI-4.1-pluto-M2060407) id AA22826; Sat, 29 Dec 07 22:11:13 PST Date: Sat, 29 Dec 2007 22:09:44 -0800 From: perryh@pluto.rain.com To: ivoras@freebsd.org Message-Id: <47773628.CaSnQ8XhoTY1UQz3%perryh@pluto.rain.com> References: <20071229.122221.-432830441.imp@bsdimp.com> <4776d1d7.zI7kRv9uFoaBNKnQ%perryh@pluto.rain.com> In-Reply-To: User-Agent: nail 11.25 7/29/05 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 06:14:36 -0000 Ivan Voras wrote: > perryh@pluto.rain.com wrote: > > > The degree to which a PowerPC imposes a strict alignment > > requirement depends on both the particular processor model > > and the operation being performed. > > > > For ordinary integer arithmetic and logical operations, newer > > PPC processors tend to be more tolerant (although misalignment > > will typically carry a performance penalty) ... > > How would it behave in operations like > > x = x + 1 > > where x is unaligned in memory? A RISC would have to load the > value from memory, increment it and store it. It depends on the processor type. IIRC most of the recent ones (7xx, 74xx) will handle it in hardware, taking a few extra cycles to do the extra memory accesses. Some of the older ones (403) will take an alignment exception, which can potentially be fixed up by a handler but at a cost of, at least, hundreds of cycles -- and only if the OS or the application has provided the handler. I recently ran into a case where a PPC970 took an alignment exception on something resembling this, but I think it was because the data cache was disabled (the 970's hardware misalignment handler being part of the cache logic). From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 07:59:40 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E3A616A420 for ; Sun, 30 Dec 2007 07:59:40 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail04.syd.optusnet.com.au (mail04.syd.optusnet.com.au [211.29.132.185]) by mx1.freebsd.org (Postfix) with ESMTP id C836113C442 for ; Sun, 30 Dec 2007 07:59:39 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mail04.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lBU7xTVW018651 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 30 Dec 2007 18:59:34 +1100 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.1) with ESMTP id lBU7xTtc076417; Sun, 30 Dec 2007 18:59:29 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id lBU7xR5C076416; Sun, 30 Dec 2007 18:59:27 +1100 (EST) (envelope-from peter) Date: Sun, 30 Dec 2007 18:59:27 +1100 From: Peter Jeremy To: Bakul Shah Message-ID: <20071230075927.GJ40785@server.vk2pj.dyndns.org> References: <20071229214644.GY40785@server.vk2pj.dyndns.org> <20071229220628.0E9CD5B2E@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Sk71+Upln2BLuDmg" Content-Disposition: inline In-Reply-To: <20071229220628.0E9CD5B2E@mail.bitblocks.com> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 07:59:40 -0000 --Sk71+Upln2BLuDmg Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 29, 2007 at 02:06:27PM -0800, Bakul Shah wrote: >> (though the AMD29K could apparently generate >> dummy bus cycles to limit the number of bit transitions on any cycle >> to reduce the I/O load). > >Are you sure it was the amd29k? I don't recall anything like >that (and am too lazy to dig out its datasheets!). Maybe I'm mis-remembering - it was a long time ago. Other possibility is the M88K. I can't quickly find anything to backup my memory. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --Sk71+Upln2BLuDmg Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHd0/f/opHv/APuIcRAqYKAKCkrvJr8BMopDswvDKFL3Acl5MfsgCghI7k EOR1w/RGtwailtywgUxmIw8= =vyBw -----END PGP SIGNATURE----- --Sk71+Upln2BLuDmg-- From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 09:34:59 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E7A0016A417; Sun, 30 Dec 2007 09:34:59 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 5932713C46A; Sun, 30 Dec 2007 09:34:58 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id lBU9YlvX003415; Sun, 30 Dec 2007 10:34:47 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [10.1.1.14]) by cicely5.cicely.de (8.13.4/8.13.4) with ESMTP id lBU9YYP0086900 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 30 Dec 2007 10:34:35 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.13.4/8.13.3) with ESMTP id lBU9YY2b045870; Sun, 30 Dec 2007 10:34:34 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.13.4/8.13.3/Submit) id lBU9YXSV045869; Sun, 30 Dec 2007 10:34:33 +0100 (CET) (envelope-from ticso) Date: Sun, 30 Dec 2007 10:34:33 +0100 From: Bernd Walter To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20071230093432.GF31522@cicely12.cicely.de> References: <20071229.122221.-432830441.imp@bsdimp.com> <20071229193034.GA73845@freebie.xs4all.nl> <86prwp161k.fsf@ds4.des.no> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86prwp161k.fsf@ds4.des.no> X-Operating-System: FreeBSD cicely12.cicely.de 5.4-STABLE alpha User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on cicely12.cicely.de Cc: Wilko Bulte , ivoras@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 09:35:00 -0000 On Sat, Dec 29, 2007 at 11:37:27PM +0100, Dag-Erling Smørgrav wrote: > Wilko Bulte writes: > > In the past the alpha port had it too. > > No, it was optional and defaulted to off. It was never optional, since no alpha CPU can do missaligned access Alphas can fix missaligned access by software trap handlers in the same way as most other strong alignment architectures can and we did this for userland, but not in kernel. Nevertheless it is horribly slow to do this, but was required since there was so many crappy software that days - fortunately this has changed over time, although I still see aligment traps on new software as well. Sadly said we never implemented missaligment traps for x86 so developers without alphas could see their faults and made us alpha people a hard live by introducing new missalignments bugs on a regular basis so that many finaly gave up on that loop. On the other hand people should keep in mind that even modern x86 are not very good when it comes to missaligned access. They handle it in hardware, but it is not that optimized as handling alligned access, so you still see a major performance penalty. -- B.Walter http://www.bwct.de http://www.fizon.de bernd@bwct.de info@bwct.de support@fizon.de From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 13:11:02 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AA2F16A419; Sun, 30 Dec 2007 13:11:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id 4206313C4F2; Sun, 30 Dec 2007 13:11:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=deviant.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1J8xwQ-0000g3-Eh; Sun, 30 Dec 2007 15:11:01 +0200 Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id lBUDAvPc022796; Sun, 30 Dec 2007 15:10:57 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id lBUDAuNh022795; Sun, 30 Dec 2007 15:10:56 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 30 Dec 2007 15:10:56 +0200 From: Kostik Belousov To: Kostik Belousov Message-ID: <20071230131056.GG57756@deviant.kiev.zoral.com.ua> References: <47760132.5040306@pacific.net.sg> <20071229111204.GX57756@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="y3pDWE12QfphwjBl" Content-Disposition: inline In-Reply-To: <20071229111204.GX57756@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.4.2.3i X-Scanner-Signature: 33ca3efb02d57e4381ef3dfbedf9ea6a X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1976 [Dec 29 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: Kip Macy , Ivan Voras , freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 13:11:02 -0000 --y3pDWE12QfphwjBl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 29, 2007 at 01:12:04PM +0200, Kostik Belousov wrote: > On Sat, Dec 29, 2007 at 12:14:11AM -0800, Kip Macy wrote: > > Isn't it everything except x86? > >=20 > > -Kip > x86 has the AC bit in the eflags. The AM bit in cr0 is enabled by the > kernel, and AC could be switched on by LD_PRELOADed shared object. >=20 > Last time I checked, our libc caused unaligned access in the locale > initialization code. Out of curiosity, I wrote the following simple LD_PRELOADable shared object. /* $Id: align.c,v 1.2 2007/12/30 13:06:32 kostik Exp $ */ #define AC "(1 << 18)" static void enable_AC() { __asm volatile("pushfl\n\t" "orl\t$" AC ", (%%esp)\n\t" "popfl\n" : : : "cc"); } static void disable_AC(void) { __asm volatile("pushfl\n\t" "andl\t$~" AC ", (%%esp)\n\t" "popfl\n" : : : "cc"); } static void set_AC(void) __attribute__ ((constructor)); void set_AC(void) { enable_AC(); } cc -O2 -shared -o align.so align.c=20 Doing LD_PRELOAD=3D./align.so /bin/ls results in the [1] 12032 bus error (core dumped) LD_PRELOAD=3D./align.so /bin/ls gdb session: Program terminated with signal 10, Bus error. #0 0x2816ee8d in __collate_load_tables (encoding=3D0x281c1280 "ru_RU.KOI8-= R") at /usr/home/kostik/work/MY/align/src/lib/libc/locale/collate.c:87 87 (void)strcat(buf, "/"); (gdb) disassemble 0x2816ee8d 0x2816ee8d+10 Dump of assembler code from 0x2816ee8d to 0x2816ee97: 0x2816ee8d <__collate_load_tables+205>: movw $0x2f,-0x1(%esi,%ecx,1) 0x2816ee94 <__collate_load_tables+212>: mov 0x8(%ebp),%eax (half-word)0x2f =3D=3D asciz '/' I.e., it seems that gcc does not feel too guilty generating unaligned half-word writes on i386. :( --y3pDWE12QfphwjBl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHd5jfC3+MBN1Mb4gRAj1NAKDOZTEYMYTI92RbwFDNK9tAaLg0NQCg8JyO rDe05y4DpR/3m1fWVfoG7l4= =+P9Y -----END PGP SIGNATURE----- --y3pDWE12QfphwjBl-- From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 18:55:15 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6D9216A418 for ; Sun, 30 Dec 2007 18:55:15 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (bhuda.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id 0BC1613C468 for ; Sun, 30 Dec 2007 18:55:14 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 60261 invoked by uid 1001); 30 Dec 2007 18:55:07 -0000 Received: from bhuda.mired.org (localhost.localdomain [127.0.0.1]) by bhuda.mired.org (tmda-ofmipd) with ESMTP; Sun, 30 Dec 2007 13:55:07 -0500 Date: Sun, 30 Dec 2007 13:55:06 -0500 To: freebsd-hackers@freebsd.org Message-ID: <20071230135506.47beac58@bhuda.mired.org> In-Reply-To: <20071230093432.GF31522@cicely12.cicely.de> References: <20071229.122221.-432830441.imp@bsdimp.com> <20071229193034.GA73845@freebie.xs4all.nl> <86prwp161k.fsf@ds4.des.no> <20071230093432.GF31522@cicely12.cicely.de> Organization: Meyer Consulting X-Mailer: Claws Mail 2.9.1 (GTK+ 2.10.12; amd64-portbld-freebsd6.2) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: ticso@cicely.de Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 18:55:15 -0000 On Sun, 30 Dec 2007 10:34:33 +0100 Bernd Walter = wrote: > On Sat, Dec 29, 2007 at 11:37:27PM +0100, Dag-Erling Sm=C3=B8rgrav wrote: > > Wilko Bulte writes: > > > In the past the alpha port had it too. > >=20 > > No, it was optional and defaulted to off. >=20 > It was never optional, since no alpha CPU can do missaligned access > Alphas can fix missaligned access by software trap handlers in the same > way as most other strong alignment architectures can and we did this > for userland, but not in kernel. > Nevertheless it is horribly slow to do this, but was required since > there was so many crappy software that days - fortunately this has > changed over time, although I still see aligment traps on new software > as well. > Sadly said we never implemented missaligment traps for x86 so [...] Ok, I'm a bit confused. Since you're talking about moving code from the x86 to the alpha, I'm assuming you're talking about C code. Isn't it the *compilers* job to enforce alignment issues, unless the programmer specifically asks for byte-specific control of the layout of a set of variables? Or are these the issue, and the problem is that people do that and then don't use the appropriate APIs to pull data from them, thus causing you headaches? Thanks, http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 19:37:33 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 849E116A481 for ; Sun, 30 Dec 2007 19:37:33 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 2D09D13C474 for ; Sun, 30 Dec 2007 19:37:32 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id lBUJbUXC025869; Sun, 30 Dec 2007 20:37:31 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [10.1.1.14]) by cicely5.cicely.de (8.13.4/8.13.4) with ESMTP id lBUJbK5k091030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 30 Dec 2007 20:37:21 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.13.4/8.13.3) with ESMTP id lBUJbKnP047185; Sun, 30 Dec 2007 20:37:20 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.13.4/8.13.3/Submit) id lBUJbIZ0047184; Sun, 30 Dec 2007 20:37:18 +0100 (CET) (envelope-from ticso) Date: Sun, 30 Dec 2007 20:37:18 +0100 From: Bernd Walter To: Mike Meyer Message-ID: <20071230193718.GH31522@cicely12.cicely.de> References: <20071229.122221.-432830441.imp@bsdimp.com> <20071229193034.GA73845@freebie.xs4all.nl> <86prwp161k.fsf@ds4.des.no> <20071230093432.GF31522@cicely12.cicely.de> <20071230135506.47beac58@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20071230135506.47beac58@bhuda.mired.org> X-Operating-System: FreeBSD cicely12.cicely.de 5.4-STABLE alpha User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on cicely12.cicely.de Cc: freebsd-hackers@freebsd.org, ticso@cicely.de Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 19:37:33 -0000 On Sun, Dec 30, 2007 at 01:55:06PM -0500, Mike Meyer wrote: > On Sun, 30 Dec 2007 10:34:33 +0100 Bernd Walter wrote: > > On Sat, Dec 29, 2007 at 11:37:27PM +0100, Dag-Erling Smørgrav wrote: > > > Wilko Bulte writes: > > > > In the past the alpha port had it too. > > > > > > No, it was optional and defaulted to off. > > > > It was never optional, since no alpha CPU can do missaligned access > > Alphas can fix missaligned access by software trap handlers in the same > > way as most other strong alignment architectures can and we did this > > for userland, but not in kernel. > > > Nevertheless it is horribly slow to do this, but was required since > > there was so many crappy software that days - fortunately this has > > changed over time, although I still see aligment traps on new software > > as well. > > Sadly said we never implemented missaligment traps for x86 so > [...] > > Ok, I'm a bit confused. Since you're talking about moving code from > the x86 to the alpha, I'm assuming you're talking about C code. Isn't > it the *compilers* job to enforce alignment issues, unless the > programmer specifically asks for byte-specific control of the layout > of a set of variables? Not in every case. It is not unusual to read raw bytes from e.g. network and use a struct pointer to it. But in case of network data it may be missaligned and if the struct contains anything else then bytes you may be doing missaligned accesses. > Or are these the issue, and the problem is that people do that and > then don't use the appropriate APIs to pull data from them, thus > causing you headaches? No - they just don't think about alignment when parsing data from random source. A very tricky case are IP headers in ethernet bytes - the ethernet header is not divideable by 4, but the IPs inside are 32 bit. Not to speak about the payload, which might contain 32 bit values. We have seen lots of missalignment bugs every now and then in ipfw, but now ipfw is aligment save since years now. I'm still not 100% sure if TCP NFS clients are save. At least in FreeBSD 5.x it is not. -- B.Walter http://www.bwct.de http://www.fizon.de bernd@bwct.de info@bwct.de support@fizon.de From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 21:37:17 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4C4716A41B for ; Sun, 30 Dec 2007 21:37:17 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: from mail6.sea5.speakeasy.net (mail6.sea5.speakeasy.net [69.17.117.8]) by mx1.freebsd.org (Postfix) with ESMTP id 8011513C469 for ; Sun, 30 Dec 2007 21:37:17 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: (qmail 3562 invoked from network); 30 Dec 2007 21:37:16 -0000 Received: from april.chuckr.org (chuckr@[66.92.151.30]) (envelope-sender ) by mail6.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 30 Dec 2007 21:37:16 -0000 Message-ID: <47780ECD.5080603@chuckr.org> Date: Sun, 30 Dec 2007 16:34:05 -0500 From: Chuck Robey User-Agent: Thunderbird 2.0.0.6 (X11/20071107) MIME-Version: 1.0 To: Jeremy Chadwick References: <476AF132.4080304@chuckr.org> <86abo1g2pu.fsf@ds4.des.no> <476EA521.9020707@chuckr.org> <20071224131425.GF23337@cicely12.cicely.de> <47715E1F.9080502@chuckr.org> <86sl1qikam.fsf@ds4.des.no> <477440A2.4070601@chuckr.org> <86sl1n14vv.fsf@ds4.des.no> <477698D8.2040802@chuckr.org> <20071229204238.GA45376@owl.midgard.homeip.net> <20071229205647.GA51467@eos.sc1.parodius.com> In-Reply-To: <20071229205647.GA51467@eos.sc1.parodius.com> X-Enigmail-Version: 0.95.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: =?ISO-8859-1?Q?Dag-Erling_Sm=F8rgrav?= , ticso@cicely.de, FreeBSD-Hackers Subject: Re: printing boot probe messages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 21:37:17 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jeremy Chadwick wrote: > On Sat, Dec 29, 2007 at 09:42:38PM +0100, Erik Trulsson wrote: >> If you do not see any boot messages at all, then my guess is that you >> probably have messed around with /boot/device.hints (or compiled in a hints >> file in the kernel) and removed or disabled the hints for the video >> adapter/system console. > > We've confirmed already in this thread that that's exactly what he's > done. But he's still yet to include any details about what all he > changed and the APRIL.hints file he's using his in kernel configuration > (rather than using /boot/devices.hints, which is what he should be > using). > > This thread is running in circles. > A large part of the reason that this thread seems to be running in circles is because the names I have for things are extremely poor. In my previous posts, I thought to provoke someone into telling me the correct names for things by using names more reminiscent of grade school, but to my astonishment (and embarrassment) others have replied using that ridiculous nomenclature "Print#1 and Print#2". Doesn't that seem more than a little ridiculous to you? To stop me from using words out of kindergarten, you merely need to supply me with better ones (heck, point me at a proper man page that uses them, I can read) and I will immediately begin using the better names. I mean, isn't this a bit ridiculous? OK, I'm going to reply to one of the other posts, regarding the actual topic, but I wanted this plea not to become submerged in other concerns. The only reason to reply to this post would be to give me better names to use, as described in my other posts. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHeA7Nz62J6PPcoOkRAtLOAJ9IBmbdX8oAv95fA+jtYT+rnPxrogCfRtb0 Y3tafYbMjBFXQbtSLWaFnQI= =XwMF -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 22:46:05 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BDAF16A468 for ; Sun, 30 Dec 2007 22:46:05 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail15.syd.optusnet.com.au (mail15.syd.optusnet.com.au [211.29.132.196]) by mx1.freebsd.org (Postfix) with ESMTP id C612B13C467 for ; Sun, 30 Dec 2007 22:45:58 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mail15.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id lBUMjf7C006598 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Dec 2007 09:45:49 +1100 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.1) with ESMTP id lBUMjdha069589; Mon, 31 Dec 2007 09:45:39 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id lBUMjZM6069588; Mon, 31 Dec 2007 09:45:35 +1100 (EST) (envelope-from peter) Date: Mon, 31 Dec 2007 09:45:35 +1100 From: Peter Jeremy To: Chuck Robey Message-ID: <20071230224535.GZ40785@server.vk2pj.dyndns.org> References: <476EA521.9020707@chuckr.org> <20071224131425.GF23337@cicely12.cicely.de> <47715E1F.9080502@chuckr.org> <86sl1qikam.fsf@ds4.des.no> <477440A2.4070601@chuckr.org> <86sl1n14vv.fsf@ds4.des.no> <477698D8.2040802@chuckr.org> <20071229204238.GA45376@owl.midgard.homeip.net> <20071229205647.GA51467@eos.sc1.parodius.com> <47780ECD.5080603@chuckr.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="r7tUYVWcdYzDJoZW" Content-Disposition: inline In-Reply-To: <47780ECD.5080603@chuckr.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-hackers@freebsd.org Subject: Re: printing boot probe messages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 22:46:05 -0000 --r7tUYVWcdYzDJoZW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Dec 30, 2007 at 04:34:05PM -0500, Chuck Robey wrote: >astonishment (and embarrassment) others have replied using that ridiculous >nomenclature "Print#1 and Print#2". Doesn't that seem more than a little >ridiculous to you? The only distinction the kernel makes is between "verbose boot" and "non-verbose boot". Based on your description of what you did, both "Print#1" and "Print#2" are "verbose boot". If you are seeing a difference then it is something local to your system. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --r7tUYVWcdYzDJoZW Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHeB+P/opHv/APuIcRAq1RAKCpsGJ+3myHhJeNj5nHJp245BqLOgCdHIuC hqh578cihU4P4Mt780cQyQ8= =EB/X -----END PGP SIGNATURE----- --r7tUYVWcdYzDJoZW-- From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 23:17:29 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F93A16A417 for ; Sun, 30 Dec 2007 23:17:29 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: from mail6.sea5.speakeasy.net (mail6.sea5.speakeasy.net [69.17.117.8]) by mx1.freebsd.org (Postfix) with ESMTP id DFC6C13C459 for ; Sun, 30 Dec 2007 23:17:28 +0000 (UTC) (envelope-from chuckr@chuckr.org) Received: (qmail 30973 invoked from network); 30 Dec 2007 23:17:28 -0000 Received: from april.chuckr.org (chuckr@[66.92.151.30]) (envelope-sender ) by mail6.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 30 Dec 2007 23:17:28 -0000 Message-ID: <47782631.6070305@chuckr.org> Date: Sun, 30 Dec 2007 18:13:53 -0500 From: Chuck Robey User-Agent: Thunderbird 2.0.0.6 (X11/20071107) MIME-Version: 1.0 To: Peter Jeremy References: <476AF132.4080304@chuckr.org> <86abo1g2pu.fsf@ds4.des.no> <476EA521.9020707@chuckr.org> <20071224131425.GF23337@cicely12.cicely.de> <47715E1F.9080502@chuckr.org> <86sl1qikam.fsf@ds4.des.no> <477440A2.4070601@chuckr.org> <86sl1n14vv.fsf@ds4.des.no> <477698D8.2040802@chuckr.org> <20071229213216.GV40785@server.vk2pj.dyndns.org> In-Reply-To: <20071229213216.GV40785@server.vk2pj.dyndns.org> X-Enigmail-Version: 0.95.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: FreeBSD-Hackers Subject: Re: printing boot probe messages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 23:17:29 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Jeremy wrote: > On Sat, Dec 29, 2007 at 01:58:32PM -0500, Chuck Robey wrote: >> booting. If I stick "-v" in /boot.config, then when the kernel probes, all >> the probes are verbose. Stuff like my HDaudio card print incredibly >> verbose listings. OK, that's what I will call here Print#1 >> >> The other thing is what I can see if I see the ascii-graphical loader (the >> picture, in ascii-graphics, either of a BSDaemon, or of the letters >> "FreeBSD", and a list of about 9 options for booting. If I select item #5, >> then I get a listing. The listing is quite distinct from what I identify >> as Print#1, so I'll call this Print#2. If I either hit return at that > > There are no distinct names for what you call Print#1 and Print#2 > because these outputs should be identical. Within the kernel, the > verbosity is controlled by a boolean flag "bootverbose" - which is > visible via sysctl as "debug.bootverbose". I just went thru a long series of reboots, so I can make really certain I have this correct. I had some stuff wrong before, I can see that. The state really is, the only way I can provoke printing of boot messages during booting is to call out a verbose boot. If I do that, then the boot messages DO print during booting, and examination afterwards shows a big file (~ 60K in size). This happens if I use /boot.config==-v, or if I enter option #5 to the beastie menu, or even if I call out a manual loader run, and enter boot -v at the prompt. If I do none of those things, it will not print boot messages. After booting, if I do a "dmesg > dm" then the resulting file is only about 10k in size, and things like the enormous pci probing don't show up. At this moment, I'm in a boot that wasn't printed, and I just got a 0 back from a "sysctl debug.bootverbose" > > What differences are you seeing between the two outputs? > What is the content of your /boot.config and /boot/loader.conf (and > any other files you have changed in /boot)? Right now, I moved my /boot.config to /notboot.config. My loader.conf is boot_mute="NO" verbose_loading="YES" linux_load="YES" #snd_hda_load="YES" #beastie_disable="NO" #loader_logo="beastiebw" nvidia_load="YES" The nvidia thing is because I brought to sources for the nvidia graphics card I use to the point where it compiles for current, and I have that working ok (although I haven't found any way to test it yet). I do notice that I have verbose_loading set to YES, but that never seemed to show any effects. I have experimented (not recently, but after I lost the boot messages) and I never saw any differences from it. The boot_mute line was another of my efforts to get printing to start up, it didn't have any effect either. > > Can you please provide the output from and "kenv" > in the the above two cases as well as a default boot. Sure. "sysctl debug.boothowto" returns -2147418112, any idea what that means? The kenv stuff is long, but I'll stick it at the end, here. > I presume you aren't seeing any of this odd behaviour when you boot > from an install CD. Never tried it. I have a trash ide disk that I initially booted from, before I got my raid up, it's still available to boot from, and yes, it prints it's boot messages during boot, just fine. When I switch to that, it's the whole / including /boot and /usr, it's 80G big. It would really help if you reverted back to a > stock GENERIC kernel, with no local mods in / or /boot and then > started re-applying your changes one at a time to see what has gone wrong. Hoping not to have to do that. but if this here doesn't seem tog et anywhere with it, that's the only choice I will have if I want to see things fixed. Luckily for me, that trash ide drive is available to me still, so I'd have to do that one. Just didn't want to do that, I have the audio still to work at, and then some others things fro friends, but if there's no choice, then I guess I will have to. > >> I'm beginning, right now, to wonder: I increased the dmesg-buffer to 64K, I >> wonder if maybe that might possibly cause the bug? I know it shouldn't, >> but it wouldn't be the first time this week that I found weird behaviour in >> the kernel: if you set the number of vtys from the default 16 down to 8, >> that caused me to lose keyboard input to my X11. > > None of this should happen either. > Well, Des explained the x11 thing, it turns out that you must turn "off" the highest numbered vty, so that X11 has a place to land. That nearly fooled me, because it's set in /etc/ttys to xdm, and I never use xdm, so I thought I didn't need that, but it turns out that it's the fact that the top vty is turned off which is the magic in that case, not the setting it to xdm. I just proved that here. HERE'S THE KENV LISTING I still can't provoke the non-verbose boot messges to print during boot (I can get them via dmesg later on, though). TCSH-april:chuckr:~:#105-17:43>kenv LINES="24" acpi_load="YES" boot_mute="NO" bootfile="kernel" comconsole_speed="9600" console="vidconsole" currdev="disk2s1a:" hint.acpi.0.oem="Nvidia" hint.acpi.0.revision="2" hint.acpi.0.rsdp="0xf7bc0" hint.acpi.0.rsdt="0xcfef3040" hint.acpi.0.xsdt="0x00000000cfef30c0" hint.acpi.0.xsdt_length="36" hint.apm.0.disabled="1" hint.apm.0.flags="0x20" hint.ata.0.at="isa" hint.ata.0.irq="14" hint.ata.0.port="0x1F0" hint.ata.1.at="isa" hint.ata.1.irq="15" hint.ata.1.port="0x170" hint.atkbd.0.at="atkbdc" hint.atkbd.0.irq="1" hint.atkbdc.0.at="isa" hint.atkbdc.0.port="0x060" hint.fd.0.at="fdc0" hint.fd.0.drive="0" hint.fd.1.at="fdc0" hint.fd.1.drive="1" hint.fdc.0.at="isa" hint.fdc.0.drq="2" hint.fdc.0.irq="6" hint.fdc.0.port="0x3F0" hint.psm.0.at="atkbdc" hint.psm.0.irq="12" hint.sc.0.at="isa" hint.sc.0.flags="0x100" hint.vga.0.at="isa" hint.vt.0.at="isa" hint.vt.0.disabled="1" interpret="OK" kernel="kernel" kernel_options="" kernelname="/boot/kernel/kernel" loaddev="disk2s1a:" mac_ifoff="NO" module_path="/boot/kernel;/boot/modules" smbios.bios.reldate="03/28/2007" smbios.bios.vendor="Phoenix Technologies, LTD" smbios.bios.version="ASUS StrikerExtreme ACPI BIOS Revision 1102" smbios.chassis.maker="Chassis Manufacture" smbios.chassis.serial="EVAL " smbios.chassis.tag="123456789000" smbios.chassis.version="Chassis Version" smbios.planar.maker="ASUSTeK Computer INC." smbios.planar.product="StrikerExtreme" smbios.planar.serial="123456789000" smbios.planar.version="1.XX " smbios.socket.enabled="1" smbios.socket.populated="1" smbios.system.maker="System manufacturer" smbios.system.product="System Product Name" smbios.system.serial="System Serial Number" smbios.system.uuid="6451934a-5599-db11-875b-460d620676ae" smbios.system.version="System Version" vfs.root.mountfrom="ufs:/dev/da0s1a" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHeCYxz62J6PPcoOkRAt8TAJwLftRbTY4Fh8tW9P9Dkg1GfPVLZwCgmY0T NcR5r3BC5WTpuLE7A654Di8= =MxUr -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 30 23:27:26 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57D9616A417; Sun, 30 Dec 2007 23:27:26 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from mta5.srv.hcvlny.cv.net (mta5.srv.hcvlny.cv.net [167.206.4.200]) by mx1.freebsd.org (Postfix) with ESMTP id 2D9FD13C457; Sun, 30 Dec 2007 23:27:25 +0000 (UTC) (envelope-from aryeh.friedman@gmail.com) Received: from flosoft.no-ip.biz (ool-435559b8.dyn.optonline.net [67.85.89.184]) by mta5.srv.hcvlny.cv.net (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTP id <0JTV00M73YH90LE0@mta5.srv.hcvlny.cv.net>; Sun, 30 Dec 2007 18:27:10 -0500 (EST) Received: from flosoft.no-ip.biz (localhost [IPv6:::1]) by flosoft.no-ip.biz (8.14.2/8.14.2) with ESMTP id lBUNR8PJ042312; Sun, 30 Dec 2007 18:27:09 -0500 Date: Sun, 30 Dec 2007 18:27:08 -0500 From: "Aryeh M. Friedman" In-reply-to: <86r6h5zpr3.fsf@ds4.des.no> To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= Message-id: <4778294C.3030905@gmail.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8BIT X-Enigmail-Version: 0.95.5 References: <5950EE0C-383D-4D6B-9991-A0DEABD2ADE4@u.washington.edu> <20071228003716.GB48997@lor.one-eyed-alien.net> <4774EF27.90307@gmail.com> <86r6h5zpr3.fsf@ds4.des.no> User-Agent: Thunderbird 2.0.0.9 (X11/20071228) Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: BSD license compatible hash algorithm? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2007 23:27:26 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dag-Erling Smørgrav wrote: > "Aryeh M. Friedman" writes: >> All hashs have issues with pooling.... see >> http://www.burtleburtle.net/bob/hash/index.html... btw it is a >> old wives tale that the number of buckets should be prime (mostly >> based on the very weak implementation Knuth offered) > > Not an "old wives' tale", but rather an easy way to implement a > hash algorithm that is good enough for most simple uses: metric > modulo table size, where metric is a number derived from the item > in such a manner as to give a good spread. Sorry for taking a while to reply.... but the above only applies if your using a very primitive hash like Knuth's multiplication one.... every modern hash I know of should have 2^k buckets actually for some k<2^32 [in almost all cases <2^16 except for algorithms like the one I mentioned I am working on which sets k=n where n=the bit count of the key]. - -- Aryeh M. Friedman FloSoft Systems http://www.flosoft-systems.com Developer, not business, friendly -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHeClMzIOMjAek4JIRAlA+AKCVC0oOblPhF7QZARtkfUmdGX4hVACfcyPd qhtFfOt2lOaxcmCDt6/wXsE= =jztY -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 00:22:31 2007 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A36AA16A421; Mon, 31 Dec 2007 00:22:31 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 67BA113C46B; Mon, 31 Dec 2007 00:22:31 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.13.7) with ESMTP id lBV0BuWf086564; Sun, 30 Dec 2007 16:11:57 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id lBV0BuvT086563; Sun, 30 Dec 2007 16:11:56 -0800 (PST) Date: Sun, 30 Dec 2007 16:11:56 -0800 (PST) From: Matthew Dillon Message-Id: <200712310011.lBV0BuvT086563@apollo.backplane.com> To: Kris Kennaway References: <47759641.9090604@FreeBSD.org> Cc: alc@freebsd.org, hackers@freebsd.org Subject: Re: prefaulting MAP_ANONYMOUS pages X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 00:22:31 -0000 :1) Lots of page faults, which drop performance by a factor of 10 :compared to the case where everything is faulted in. : :... :Is there a way to achieve this that I am overlooking? If not can :someone give me some advice about what is needed? : :Kris The main problem here is that the backing pages do not exist anywhere. No backing store is assigned and the pages themselves aren't even allocated until the user touches the memory. You might be able to work magic with the default pager. The default pager handles all UNBACKED anonymous mappings. Default pager objects are quietly converted to swap pager objects when the kernel tries to page them out. In otherwords, the 'backing' store for the default pager is always zero-fill. default_pager_haspage() returns FALSE which basically causes the VM layer to fall through to zero-fill fault code. It optimizes the fault by allocating out of the zero'd page pool in order to avoid having to bzero the page. Because default_pager_haspage() returns FALSE, default_pager_getpages() is never called. You could theoretically have default_pager_haspage() return TRUE and also load up the before and after variables, and then manually zero-fill the pages in default_pager_getpages(). This is probably the simplest solution, but not necessarily optimal since you will not necessarily be provided with pre-zero'd pages. Another possibility would be to have the VM code which handles a FALSE return from *_pager_haspage() to check additional adjacent pages and zero-fill a larger block instead of just one page. This would be more work but would also be a more optimal solution. A third possibility would be to add an additional argument to *_pager_haspage() that allows it to tell the caller how many pages do not have backing store, so the kernel doesn't have to poll for each page and can zero-fill several at once. This is more complex but probably the most optimal solution. -- Note that these sorts of optimizations could create non-optimal operation for unrelated programs which rely on sparse allocations. I am not sure if FreeBSD has a heuristic for VM faults... I think it does. That heuristic could be used to determine how much fault-ahead to actually do. It may already exist, I haven't checked, and simply need to be applied to whatever code you cook up for the zero-fill fault. Also note that OBJT_DEFAULT and OBJT_SWAP (default and swap pagers) are special cased all over the code base. There might be unexpected side effects if you mess with the default_pager*() functions. -Matt Matthew Dillon From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 04:14:23 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BDCB16A419; Mon, 31 Dec 2007 04:14:23 +0000 (UTC) (envelope-from perryh@pluto.rain.com) Received: from agora.rdrop.com (agora.rdrop.com [199.26.172.34]) by mx1.freebsd.org (Postfix) with ESMTP id 4977813C4E5; Mon, 31 Dec 2007 04:14:23 +0000 (UTC) (envelope-from perryh@pluto.rain.com) Received: from agora.rdrop.com (66@localhost [127.0.0.1]) by agora.rdrop.com (8.13.1/8.12.7) with ESMTP id lBV4EI9g026519 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sun, 30 Dec 2007 20:14:18 -0800 (PST) (envelope-from perryh@pluto.rain.com) Received: (from uucp@localhost) by agora.rdrop.com (8.13.1/8.12.9/Submit) with UUCP id lBV4EIoR026518; Sun, 30 Dec 2007 20:14:18 -0800 (PST) Received: from fbsd61 by pluto.rain.com (4.1/SMI-4.1-pluto-M2060407) id AA25407; Sun, 30 Dec 07 20:01:09 PST Date: Sun, 30 Dec 2007 19:59:38 -0800 From: perryh@pluto.rain.com To: aryeh.friedman@gmail.com, des@des.no Message-Id: <4778692a.rVZUrFy36ANLKL7F%perryh@pluto.rain.com> References: <5950EE0C-383D-4D6B-9991-A0DEABD2ADE4@u.washington.edu> <20071228003716.GB48997@lor.one-eyed-alien.net> <4774EF27.90307@gmail.com> <86r6h5zpr3.fsf@ds4.des.no> <4778294C.3030905@gmail.com> In-Reply-To: <4778294C.3030905@gmail.com> User-Agent: nail 11.25 7/29/05 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, ivoras@freebsd.org Subject: Re: BSD license compatible hash algorithm? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 04:14:23 -0000 "Aryeh M. Friedman" wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Dag-Erling Sm??rgrav wrote: > > "Aryeh M. Friedman" writes: > >> All hashs have issues with pooling.... see > >> http://www.burtleburtle.net/bob/hash/index.html... btw it is > >> a old wives tale that the number of buckets should be prime > >> (mostly based on the very weak implementation Knuth offered) > > > > Not an "old wives' tale", but rather an easy way to implement a > > hash algorithm that is good enough for most simple uses: metric > > modulo table size, where metric is a number derived from the > > item in such a manner as to give a good spread. > > ... the above only applies if your using a very primitive hash > like Knuth's multiplication one.... every modern hash I know of > should have 2^k buckets actually for some k ... It very much depends on what is used for a rehash (collision step) value. The step value and the table size must have no common factor larger than 1, or there will be edge cases (bugs) in which some combinations of hash and step values will cause the table to appear full when in fact it is not. Making the table size prime is one simple way of preventing such problems, while still allowing the rehash value to depend on the key (thus reducing the liklihood of collision on the second probe). At the other extreme, the table can be any size at all if the step value is 1 (and the "modulo table size" operation will certainly be more efficient if the table size is 2^k). From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 04:18:17 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A73D16A417 for ; Mon, 31 Dec 2007 04:18:17 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: from mired.org (bhuda.mired.org [66.92.153.74]) by mx1.freebsd.org (Postfix) with SMTP id BCD8013C45A for ; Mon, 31 Dec 2007 04:18:16 +0000 (UTC) (envelope-from mwm-keyword-freebsdhackers2.e313df@mired.org) Received: (qmail 76114 invoked by uid 1001); 31 Dec 2007 04:18:10 -0000 Received: from bhuda.mired.org (localhost.localdomain [127.0.0.1]) by bhuda.mired.org (tmda-ofmipd) with ESMTP; Sun, 30 Dec 2007 23:18:09 -0500 Date: Sun, 30 Dec 2007 23:18:08 -0500 To: freebsd-hackers@freebsd.org Message-ID: <20071230231808.4eaa93d5@bhuda.mired.org> In-Reply-To: <20071230193718.GH31522@cicely12.cicely.de> References: <20071229.122221.-432830441.imp@bsdimp.com> <20071229193034.GA73845@freebie.xs4all.nl> <86prwp161k.fsf@ds4.des.no> <20071230093432.GF31522@cicely12.cicely.de> <20071230135506.47beac58@bhuda.mired.org> <20071230193718.GH31522@cicely12.cicely.de> Organization: Meyer Consulting X-Mailer: Claws Mail 2.9.1 (GTK+ 2.10.12; amd64-portbld-freebsd6.2) Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAG1BMVEXguIzRkGnhyaz069mXhW0WHRnbrnR9WCQ6LB0CchNMAAACSUlEQVQ4jV2TQW7jMAxFGaPQOgQEdZaGMsgBrAvUA03dCxj1Uu4U2gfwQD7AGNax51NK07RcxXz6/CSl0Ij450vkPG1jzpIZM1UwDCl/xB14TWnNX8A00Qj5a0mnVFVbVUz4MeErea2HikSRqZzY894zwg9p2+/AtO8LzxFED+tNAUFeU29iFOLRxlZAcdo9A8wi8ZBMV4BKPde82Oxrvs6BTkulQIClte0DLFzzsKk9j1MBex8iUaP00Bd78S/muyFScrTXz6zLkEUxJp+SabQfNOs4f4Jpx5qSZ/304PWwlEWP1cOn/mJQR7EOD+uKhjcBLziuL7xoY5Xm+VFAUSw/LwwwsHEHxihpwV4EJH0xXRkbw1PkRw+X4pEuSJwBggqk+HEYKkiL5/74/nQkogigzQsAFrakxZyfw3wMIEEZPv4AWMfxwqE5GNxGaERjmH+PG8AE0L4/w9g0lsp1raLYAN5azQa+AOoO9NwcpFkTrG2VKNMNEL5UKUUAw34tha0z7onUG0oBoNtczE04GwFE3wCHc0ChezAJ6A1WMV81AtY7wDAJSlXwV+4cwBvsOsrQMRawfQEBz0deEZ7WNpV2szckIKo5VpDHDSDvF1GItwqqAlG01Hh50BGtVhuUkjkasg/14bYFGCgWg1fSWHvmOoJck2xdp9ZvZBHzDVTzX23TkrOn7qe5U2COEw5D4Vx3qEQpFY2Z/3QFnJxzp7YCmSMG19nOUoe869zZfOQb5ywQuWu0yCn5+8gxZz+BE7vG3j4/wbf4D/sXN9Wug1s7AAAAAElFTkSuQmCC Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Delivery-Agent: TMDA/1.1.11 (Ladyburn) From: Mike Meyer Cc: ticso@cicely.de Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 04:18:17 -0000 On Sun, 30 Dec 2007 20:37:18 +0100 Bernd Walter = wrote: > On Sun, Dec 30, 2007 at 01:55:06PM -0500, Mike Meyer wrote: > > On Sun, 30 Dec 2007 10:34:33 +0100 Bernd Walter wrote: > > > On Sat, Dec 29, 2007 at 11:37:27PM +0100, Dag-Erling Sm=C3=B8rgrav wr= ote: > > > > Wilko Bulte writes: > > > > > In the past the alpha port had it too. > > > >=20 > > > > No, it was optional and defaulted to off. > > >=20 > > > It was never optional, since no alpha CPU can do missaligned access > > > Alphas can fix missaligned access by software trap handlers in the sa= me > > > way as most other strong alignment architectures can and we did this > > > for userland, but not in kernel. > >=20 > > > Nevertheless it is horribly slow to do this, but was required since > > > there was so many crappy software that days - fortunately this has > > > changed over time, although I still see aligment traps on new software > > > as well. > > > Sadly said we never implemented missaligment traps for x86 so > > [...] > >=20 > > Ok, I'm a bit confused. Since you're talking about moving code from > > the x86 to the alpha, I'm assuming you're talking about C code. Isn't > > it the *compilers* job to enforce alignment issues, unless the > > programmer specifically asks for byte-specific control of the layout > > of a set of variables? > Not in every case. > It is not unusual to read raw bytes from e.g. network and use > a struct pointer to it. > But in case of network data it may be missaligned and if the struct > contains anything else then bytes you may be doing missaligned accesses. Actually, network data is the most common case where the programmer has to ask for byte-specific control of the layout. As far as I know, the proper way to do this is to declare the struct properly - with a compiler-dependent request for packing - then read the data into that. If you could be reading more than one type of struct into it, then set up the appropriate unions for all of them. In either case, the compiler should DTRT. But it sounds like this people just read raw bytes, and then disassembled them by hand, which I would say qualifies as "crappy software." > > Or are these the issue, and the problem is that people do that and > > then don't use the appropriate APIs to pull data from them, thus > > causing you headaches? > No - they just don't think about alignment when parsing data from random > source. That seems to be common: if something doesn't matter on their platform, programmers tend not to worry about it. I saw plenty of evidence of that as Unix moved from 16 bit systems to 32 bit ones, and again as it went to 64 bits (though not quite so bad). Thanks, http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 08:18:17 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D865016A418 for ; Mon, 31 Dec 2007 08:18:17 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 7F70613C442 for ; Mon, 31 Dec 2007 08:18:17 +0000 (UTC) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id lBV8IFnq055756; Mon, 31 Dec 2007 09:18:15 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [10.1.1.14]) by cicely5.cicely.de (8.13.4/8.13.4) with ESMTP id lBV8I1iY096370 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Dec 2007 09:18:02 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.13.4/8.13.3) with ESMTP id lBV8I1fL049172; Mon, 31 Dec 2007 09:18:01 +0100 (CET) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.13.4/8.13.3/Submit) id lBV8I0nh049171; Mon, 31 Dec 2007 09:18:00 +0100 (CET) (envelope-from ticso) Date: Mon, 31 Dec 2007 09:18:00 +0100 From: Bernd Walter To: Mike Meyer Message-ID: <20071231081759.GN31522@cicely12.cicely.de> References: <20071229.122221.-432830441.imp@bsdimp.com> <20071229193034.GA73845@freebie.xs4all.nl> <86prwp161k.fsf@ds4.des.no> <20071230093432.GF31522@cicely12.cicely.de> <20071230135506.47beac58@bhuda.mired.org> <20071230193718.GH31522@cicely12.cicely.de> <20071230231808.4eaa93d5@bhuda.mired.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071230231808.4eaa93d5@bhuda.mired.org> X-Operating-System: FreeBSD cicely12.cicely.de 5.4-STABLE alpha User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, BAYES_00=-2.599 autolearn=ham version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on cicely12.cicely.de Cc: freebsd-hackers@freebsd.org, ticso@cicely.de Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 08:18:17 -0000 On Sun, Dec 30, 2007 at 11:18:08PM -0500, Mike Meyer wrote: > On Sun, 30 Dec 2007 20:37:18 +0100 Bernd Walter wrote: > > On Sun, Dec 30, 2007 at 01:55:06PM -0500, Mike Meyer wrote: > > > Ok, I'm a bit confused. Since you're talking about moving code from > > > the x86 to the alpha, I'm assuming you're talking about C code. Isn't > > > it the *compilers* job to enforce alignment issues, unless the > > > programmer specifically asks for byte-specific control of the layout > > > of a set of variables? > > Not in every case. > > It is not unusual to read raw bytes from e.g. network and use > > a struct pointer to it. > > But in case of network data it may be missaligned and if the struct > > contains anything else then bytes you may be doing missaligned accesses. > > Actually, network data is the most common case where the programmer > has to ask for byte-specific control of the layout. As far as I know, > the proper way to do this is to declare the struct properly - with a > compiler-dependent request for packing - then read the data into > that. If you could be reading more than one type of struct into it, > then set up the appropriate unions for all of them. In either case, > the compiler should DTRT. reading into a struct is one way. The other way is pointing into a read buffer. > But it sounds like this people just read raw bytes, and then > disassembled them by hand, which I would say qualifies as "crappy > software." It has it's reasons, but you have to be carefull if doing this and that's what people fail with - or better said other people fail with their software. In the same class of bugs we now have problems with arm from time to time, since arm pads single bytes to full 32bit aligment unless the struct is declared as being packed. > > > Or are these the issue, and the problem is that people do that and > > > then don't use the appropriate APIs to pull data from them, thus > > > causing you headaches? > > No - they just don't think about alignment when parsing data from random > > source. > > That seems to be common: if something doesn't matter on their > platform, programmers tend not to worry about it. I saw plenty of > evidence of that as Unix moved from 16 bit systems to 32 bit ones, and > again as it went to 64 bits (though not quite so bad). Yes - only for the last few years with all those amd64 systems out there those kind of problems are rare. But unlike missalignment bugs the compiler catches many of them as long as they are not reading random source data into structs... -- B.Walter http://www.bwct.de http://www.fizon.de bernd@bwct.de info@bwct.de support@fizon.de From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 10:19:27 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F098616A421 for ; Mon, 31 Dec 2007 10:19:27 +0000 (UTC) (envelope-from oceanare@pacific.net.sg) Received: from smtpgate2.pacific.net.sg (smtpgate2.pacific.net.sg [203.120.90.32]) by mx1.freebsd.org (Postfix) with SMTP id 2678913C46E for ; Mon, 31 Dec 2007 10:19:26 +0000 (UTC) (envelope-from oceanare@pacific.net.sg) Received: (qmail 23771 invoked from network); 31 Dec 2007 10:19:24 -0000 Received: from adsl54.dyn112.pacific.net.sg (HELO P2120.somewherefaraway.com) (oceanare@210.24.112.54) by smtpgate2.pacific.net.sg with ESMTPA; 31 Dec 2007 10:19:23 -0000 Message-ID: <4778B8A3.8040400@pacific.net.sg> Date: Mon, 31 Dec 2007 17:38:43 +0800 From: Erich Dollansky User-Agent: Thunderbird 2.0.0.6 (X11/20070826) MIME-Version: 1.0 To: Kostik Belousov References: <47760132.5040306@pacific.net.sg> <20071229111204.GX57756@deviant.kiev.zoral.com.ua> <20071230131056.GG57756@deviant.kiev.zoral.com.ua> In-Reply-To: <20071230131056.GG57756@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Kip Macy , Ivan Voras , freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 10:19:28 -0000 Hi, Kostik Belousov wrote: > On Sat, Dec 29, 2007 at 01:12:04PM +0200, Kostik Belousov wrote: >> On Sat, Dec 29, 2007 at 12:14:11AM -0800, Kip Macy wrote: > > I.e., it seems that gcc does not feel too guilty generating unaligned > half-word writes on i386. :( this should not be a problem inside a cache line. If the access goes accross two cache lines and the other cache line is not in the cache, it becomes real difficult. I can't tell you what the hardware actually does in this case. It should read the second affected cache line into the cache. But what happens if the second affected cache line is blocked by another CPU while the current CPU blocks the first cache line? Erich From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 11:33:32 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15A8E16A419; Mon, 31 Dec 2007 11:33:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id B217613C448; Mon, 31 Dec 2007 11:33:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=deviant.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1J9Itc-0001Yi-4N; Mon, 31 Dec 2007 13:33:30 +0200 Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id lBVBXRKa061172; Mon, 31 Dec 2007 13:33:27 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id lBVBXRgH061171; Mon, 31 Dec 2007 13:33:27 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 31 Dec 2007 13:33:27 +0200 From: Kostik Belousov To: Erich Dollansky Message-ID: <20071231113327.GN57756@deviant.kiev.zoral.com.ua> References: <47760132.5040306@pacific.net.sg> <20071229111204.GX57756@deviant.kiev.zoral.com.ua> <20071230131056.GG57756@deviant.kiev.zoral.com.ua> <4778B8A3.8040400@pacific.net.sg> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="rkfdwoJB1s8W8pdr" Content-Disposition: inline In-Reply-To: <4778B8A3.8040400@pacific.net.sg> User-Agent: Mutt/1.4.2.3i X-Scanner-Signature: 8d26fb09d298f8c1cb325fc86345eca8 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1976 [Dec 29 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: Kip Macy , Ivan Voras , freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 11:33:32 -0000 --rkfdwoJB1s8W8pdr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Dec 31, 2007 at 05:38:43PM +0800, Erich Dollansky wrote: > Hi, >=20 > Kostik Belousov wrote: > >On Sat, Dec 29, 2007 at 01:12:04PM +0200, Kostik Belousov wrote: > >>On Sat, Dec 29, 2007 at 12:14:11AM -0800, Kip Macy wrote: > > > >I.e., it seems that gcc does not feel too guilty generating unaligned > >half-word writes on i386. :( >=20 > this should not be a problem inside a cache line. >=20 > If the access goes accross two cache lines and the other cache line is=20 > not in the cache, it becomes real difficult. >=20 > I can't tell you what the hardware actually does in this case. >=20 > It should read the second affected cache line into the cache. But what=20 > happens if the second affected cache line is blocked by another CPU=20 > while the current CPU blocks the first cache line? =46rom the manual, 253668, 7.1.1: Accesses to cacheable memory that are split across bus widths, cache lines, and page boundaries are not guaranteed to be atomic by the Intel Core 2 Duo, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, P6 family, Pentium, and Intel486 processors. The Intel Core 2 Duo, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, and P6 family processors provide bus control signals that permit external memory subsystems to make split accesses atomic; however, nonaligned data accesses will seriously impact the performance of the processor and should be avoided. I think we might get any half of the operation as a result. --rkfdwoJB1s8W8pdr Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHeNOGC3+MBN1Mb4gRAufQAKClWTHINgpi/1zY2srN/SAMrdTnJQCgtnzh P9T/Ljo3zDfDVRzZ8wnTFoM= =jND5 -----END PGP SIGNATURE----- --rkfdwoJB1s8W8pdr-- From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 11:36:39 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 957A916A421; Mon, 31 Dec 2007 11:36:39 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4E76113C45A; Mon, 31 Dec 2007 11:36:39 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id C2A6A20B9; Mon, 31 Dec 2007 12:36:29 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.1/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id A716B2099; Mon, 31 Dec 2007 12:36:29 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 87E79844F0; Mon, 31 Dec 2007 12:36:29 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: "Aryeh M. Friedman" References: <5950EE0C-383D-4D6B-9991-A0DEABD2ADE4@u.washington.edu> <20071228003716.GB48997@lor.one-eyed-alien.net> <4774EF27.90307@gmail.com> <86r6h5zpr3.fsf@ds4.des.no> <4778294C.3030905@gmail.com> Date: Mon, 31 Dec 2007 12:36:29 +0100 In-Reply-To: <4778294C.3030905@gmail.com> (Aryeh M. Friedman's message of "Sun\, 30 Dec 2007 18\:27\:08 -0500") Message-ID: <86ejd3t7sy.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Ivan Voras Subject: Re: BSD license compatible hash algorithm? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 11:36:39 -0000 "Aryeh M. Friedman" writes: > Dag-Erling Sm=C3=B8rgrav writes: > > "Aryeh M. Friedman" writes: > > > All hashs have issues with pooling.... see > > > http://www.burtleburtle.net/bob/hash/index.html... btw it is a > > > old wives tale that the number of buckets should be prime (mostly > > > based on the very weak implementation Knuth offered) > > Not an "old wives' tale", but rather an easy way to implement a > > hash algorithm that is good enough for most simple uses: metric > > modulo table size, where metric is a number derived from the item > > in such a manner as to give a good spread. > Sorry for taking a while to reply.... but the above only applies if > your using a very primitive hash like Knuth's multiplication one.... You are overlooking a very common case: the use of a hash table to implement an in-memory dictionary (aka associative array) where the key is an integer with poor variability in the high-order bits. K % N where K is the key and N is the size of the table requires very little code, is reasonably efficient, and, provided that N is prime, gives a reasonably good spread (excpet in pathological cases where the values of K are clustered around multiples of N). > every modern hash I know of should have 2^k buckets actually for some > k<2^32 [in almost all cases <2^16 except for algorithms like the one I > mentioned I am working on which sets k=3Dn where n=3Dthe bit count of the > key]. I certainly hope not. 2^(2^32) is several of billion orders of magnitude more than the number of elementary particles in the known universe (currently estimated at 10^80). Even 2^(2^16) is too big by about sixty thousand orders of magnitude. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 12:30:45 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB82616A419 for ; Mon, 31 Dec 2007 12:30:45 +0000 (UTC) (envelope-from oceanare@pacific.net.sg) Received: from smtpgate1.pacific.net.sg (smtpgate1.pacific.net.sg [203.120.90.31]) by mx1.freebsd.org (Postfix) with SMTP id D995D13C4CC for ; Mon, 31 Dec 2007 12:30:44 +0000 (UTC) (envelope-from oceanare@pacific.net.sg) Received: (qmail 13851 invoked from network); 31 Dec 2007 12:30:42 -0000 Received: from adsl120.dyn229.pacific.net.sg (HELO P2120.somewherefaraway.com) (oceanare@210.24.229.120) by smtpgate1.pacific.net.sg with ESMTPA; 31 Dec 2007 12:30:41 -0000 Message-ID: <4778E0EB.8040709@pacific.net.sg> Date: Mon, 31 Dec 2007 20:30:35 +0800 From: Erich Dollansky User-Agent: Thunderbird 2.0.0.6 (X11/20070826) MIME-Version: 1.0 To: Kostik Belousov References: <47760132.5040306@pacific.net.sg> <20071229111204.GX57756@deviant.kiev.zoral.com.ua> <20071230131056.GG57756@deviant.kiev.zoral.com.ua> <4778B8A3.8040400@pacific.net.sg> <20071231113327.GN57756@deviant.kiev.zoral.com.ua> In-Reply-To: <20071231113327.GN57756@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Kip Macy , Ivan Voras , freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 12:30:45 -0000 Hi, Kostik Belousov wrote: > On Mon, Dec 31, 2007 at 05:38:43PM +0800, Erich Dollansky wrote: >> Kostik Belousov wrote: >>> On Sat, Dec 29, 2007 at 01:12:04PM +0200, Kostik Belousov wrote: >>>> On Sat, Dec 29, 2007 at 12:14:11AM -0800, Kip Macy wrote: >>> I.e., it seems that gcc does not feel too guilty generating unaligned >>> half-word writes on i386. :( >> this should not be a problem inside a cache line. >> >> If the access goes accross two cache lines and the other cache line is >> not in the cache, it becomes real difficult. >> >> I can't tell you what the hardware actually does in this case. >> >> It should read the second affected cache line into the cache. But what >> happens if the second affected cache line is blocked by another CPU >> while the current CPU blocks the first cache line? > > From the manual, 253668, 7.1.1: > > I think we might get any half of the operation as a result. so, both CPUs are blocked as none can access the other cache line. Is there really nothing in a normal PC to handle this? I do not know. The hardware I developed earlier was able to handle this by aborting both bus cycles. It was then task of the operating system to handle this. Erich From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 13:04:21 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4FD316A41A for ; Mon, 31 Dec 2007 13:04:21 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id 5D73313C505 for ; Mon, 31 Dec 2007 13:04:21 +0000 (UTC) (envelope-from erikt@midgard.homeip.net) Received: from c83-253-25-183.bredband.comhem.se ([83.253.25.183]:56618 helo=falcon.midgard.homeip.net) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1J9KJY-00069t-4V for freebsd-hackers@freebsd.org; Mon, 31 Dec 2007 14:04:20 +0100 Received: (qmail 20579 invoked by uid 1001); 31 Dec 2007 14:04:17 +0100 Date: Mon, 31 Dec 2007 14:04:17 +0100 From: Erik Trulsson To: Erich Dollansky Message-ID: <20071231130417.GA20470@falcon.midgard.homeip.net> Mail-Followup-To: Erich Dollansky , Kostik Belousov , Kip Macy , Ivan Voras , freebsd-hackers@freebsd.org References: <47760132.5040306@pacific.net.sg> <20071229111204.GX57756@deviant.kiev.zoral.com.ua> <20071230131056.GG57756@deviant.kiev.zoral.com.ua> <4778B8A3.8040400@pacific.net.sg> <20071231113327.GN57756@deviant.kiev.zoral.com.ua> <4778E0EB.8040709@pacific.net.sg> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4778E0EB.8040709@pacific.net.sg> User-Agent: Mutt/1.5.17 (2007-11-01) X-Originating-IP: 83.253.25.183 X-Scan-Result: No virus found in message 1J9KJY-00069t-4V. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1J9KJY-00069t-4V 13fba8c846417beaf444714e1e9f2633 Cc: Kostik Belousov , Kip Macy , Ivan Voras , freebsd-hackers@freebsd.org Subject: Re: Architectures with strict alignment? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 13:04:21 -0000 On Mon, Dec 31, 2007 at 08:30:35PM +0800, Erich Dollansky wrote: > Hi, > > Kostik Belousov wrote: >> On Mon, Dec 31, 2007 at 05:38:43PM +0800, Erich Dollansky wrote: >>> Kostik Belousov wrote: >>>> On Sat, Dec 29, 2007 at 01:12:04PM +0200, Kostik Belousov wrote: >>>>> On Sat, Dec 29, 2007 at 12:14:11AM -0800, Kip Macy wrote: >>>> I.e., it seems that gcc does not feel too guilty generating unaligned >>>> half-word writes on i386. :( >>> this should not be a problem inside a cache line. >>> >>> If the access goes accross two cache lines and the other cache line is >>> not in the cache, it becomes real difficult. >>> >>> I can't tell you what the hardware actually does in this case. >>> >>> It should read the second affected cache line into the cache. But what >>> happens if the second affected cache line is blocked by another CPU while >>> the current CPU blocks the first cache line? >> >> From the manual, 253668, 7.1.1: >> >> I think we might get any half of the operation as a result. > > so, both CPUs are blocked as none can access the other cache line. > > Is there really nothing in a normal PC to handle this? No, the CPUs are not blocked, and the hardware handles it in a typical modern multi-core system (so the programmers can't get it wrong.) If one CPU tries a write that crosses a cache-line boundary then what might happen (details vary among architectures) is that it will first get one cache-line, modify it, and write it back to memory. Then it will handle the other cache-line. This way you cannot get any deadlocks, since the CPU is waiting for at most one cache-line at a time, but you can get inconsistent results since the write is not atomic. (Another CPU can modify either cache-line in the time after the first CPU has handled the first cacheline, but before it has gotten to the second cacheline.) There are many other possible solutions as well, but hardware designers are usually quite careful to make sure deadlocks do not occur. For more information consult a good book on computer architecture. The book usually recommended is "Computer architecture: A quantitative approach" by Hennessy and Patterson. (A good book on databases or distributed systems might also be useful, since the problems with locking and concurrent accesses are essentially the same in all these areas.) > > I do not know. The hardware I developed earlier was able to handle this by > aborting both bus cycles. > > It was then task of the operating system to handle this. That is another possibility, but one which is not used all that much. -- Erik Trulsson ertr1013@student.uu.se From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 17:22:02 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8365716A417 for ; Mon, 31 Dec 2007 17:22:02 +0000 (UTC) (envelope-from markus.hoenicka@mhoenicka.de) Received: from rrzmta1.rz.uni-regensburg.de (rrzmta1.rz.uni-regensburg.de [194.94.155.51]) by mx1.freebsd.org (Postfix) with ESMTP id 1DFF713C447 for ; Mon, 31 Dec 2007 17:22:01 +0000 (UTC) (envelope-from markus.hoenicka@mhoenicka.de) Received: from rrzmta1.rz.uni-regensburg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 9774950A2A for ; Mon, 31 Dec 2007 18:02:07 +0100 (CET) Received: from yeti.mininet (rrzras1-10.rz.uni-regensburg.de [132.199.208.20]) by rrzmta1.rz.uni-regensburg.de (Postfix) with ESMTP id 3A2905091D for ; Mon, 31 Dec 2007 18:02:05 +0100 (CET) X-Mailer: emacs 21.3.1 (via feedmail 8 Q); VM 7.19 under Emacs 21.3.1 From: "Markus Hoenicka" MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18297.6718.750894.937199@yeti.mininet> Date: Mon, 31 Dec 2007 17:35:10 +0100 To: freebsd-hackers@freebsd.org X-Mailman-Approved-At: Mon, 31 Dec 2007 17:50:17 +0000 Subject: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 17:22:02 -0000 Hi, I've been redirected by Giorgos Keramidas to this list after reporting a problem on the freebsd-questions list. I'd greatly appreciate if you could have a look at the following problem. Apparently programs are doomed to segfault on FreeBSD if dlopen()ed modules install exit handlers via atexit(). Similar problem reports have cropped up before, see e.g. http://www.imagemagick.org/pipermail/magick-developers/2006-March/002523.html My system runs: FreeBSD yeti.mininet 6.1-RELEASE FreeBSD 6.1-RELEASE #1: Mon Aug 28 22:24:48 CEST 2006 markus@yeti.mininet:/usr/src/sys/i386/compile/YETI i386 I'm one of the developers of libdbi, a database abstraction layer for C, see http://libdbi.sourceforge.net. libdbi is a library for programs which are supposed to be able to access different database engines with a unified API. libdbi essentially maps generic API calls to the specific database client library calls of a particular database engine. To do this, libdbi loads available database drivers at runtime via dlopen() calls. Each of these drivers is linked against one database client library. E.g. the Firebird driver is linked against libfbclient.so. When libdbi is properly shut down, it unloads all loaded drivers by calling dlclose() on each of them. This design works well on all supported platforms and with all supported database engines, with one exception: the Firebird driver on FreeBSD invariably causes a segfault when the application linked against libdbi exits: #0 0x28514fe4 in ?? () #1 0x281507c3 in __cxa_finalize () from /lib/libc.so.6 #2 0x281503fe in exit () from /lib/libc.so.6 #3 0x0804a40f in main (argc=1, argv=0xbfbfe754) at test_dbi.c:419 The reason appears to be that the Firebird client libraries install exit handlers via atexit(). Remember that due to libdbi's design to load all available drivers whether or not they are used later, libdbi will cause a crash even if no Firebird database is accessed - it is sufficient that the driver has been loaded. As per Giorgos' suggestion it is simple to circumvent this segfault by avoiding the call to dlclose() before exiting, but I wonder whether there is a more robust solution for this problem. The attached minimal testcase is sufficient to illustrate the problem. atexitmod.c defines a module which is loaded by datest.c Make sure to fix the hardcoded path in datest.c before building the app. To build the test program and watch it crash, do the following: gcc -shared -o atexitmod.so atexitmod.c gcc -o datest datest.c ./datest Commenting out either the atexit() call in atexitmod.c or the dlclose() call in datest.c prevent the segfault. If you find some solution, please cc me as I'm not subscribed to freebsd-hackers. regards, Markus --8< datest.c --- #include #include #include int main (void) { void *dlhandle; void (*hellodriver) (void); dlhandle = dlopen("/home/markus/prog/datest/atexitmod.so", RTLD_LAZY); if (!dlhandle) { printf("cannot open module\n"); exit(1); } hellodriver = dlsym(dlhandle, "hellodriver"); if (!hellodriver) { printf("cannot locate function\n"); exit(1); } hellodriver(); dlclose(dlhandle); exit(0); } --8<------------- --8< atexitmod.c --- #include #include void exithandler(void); void hellodriver(void) { printf("hello driver\n"); fflush(stdout); atexit(exithandler); } void exithandler(void) { printf("now exiting\n"); fflush(stdout); } --8<------------- -- Markus Hoenicka markus.hoenicka@cats.de (Spam-protected email: replace the quadrupeds with "mhoenicka") http://www.mhoenicka.de From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 19:26:28 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B1F416A41A for ; Mon, 31 Dec 2007 19:26:28 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id B625713C4CE for ; Mon, 31 Dec 2007 19:26:27 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so990372anc.13 for ; Mon, 31 Dec 2007 11:26:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer:mime-version:content-type; bh=qO3B/461Q3YkUOUgOkBmjDpZ6muAyY7na1DggY3H/Xg=; b=X0Ha9HGxroJnR+I+R0mcvvLVlFSosceklou/81LWIgyDMgP5ZJqZLl5ZH0OE4QMreNvjdi9lmfu5ZgJCMaPcx3DEesJRjqAUeg3Lc8I8tSh1GYeNhPK/Is3X7mKMKFFvBJI8zVwZHsyReFfje2yGxUsQ+V1mZGWX7QOmEg6h06o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer:mime-version:content-type; b=p2na/sawkUZXk4puoOCHeDL3MZdTl1mwFRyCw9rFa/xriJlpV4nsOIoChmxEXYc4fjRr87MvJDZAg5KTgI3+Xywnbh1VQrv87UoVeI8OIuuboSxh1nhOZsu2MGn6uEw4wafqLULPyw8uoYrlS8qD08vQQttWUkjpNSDgGk31Cdk= Received: by 10.101.1.7 with SMTP id d7mr26456494ani.91.1199129186992; Mon, 31 Dec 2007 11:26:26 -0800 (PST) Received: from kan.dnsalias.net ( [24.218.183.247]) by mx.google.com with ESMTPS id o29sm20383089elf.14.2007.12.31.11.26.25 (version=SSLv3 cipher=OTHER); Mon, 31 Dec 2007 11:26:26 -0800 (PST) Date: Mon, 31 Dec 2007 14:26:20 -0500 From: Alexander Kabaev To: "Markus Hoenicka" Message-ID: <20071231142620.39f2fbd2@kan.dnsalias.net> In-Reply-To: <18297.6718.750894.937199@yeti.mininet> References: <18297.6718.750894.937199@yeti.mininet> X-Mailer: Claws Mail 3.0.2 (GTK+ 2.12.1; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/GwOo5uNn_S_7+PXNyx5=9Pm"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-hackers@freebsd.org Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 19:26:28 -0000 --Sig_/GwOo5uNn_S_7+PXNyx5=9Pm Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Mon, 31 Dec 2007 17:35:10 +0100 "Markus Hoenicka" wrote: > Hi, >=20 > I've been redirected by Giorgos Keramidas to this list after reporting > a problem on the freebsd-questions list. I'd greatly appreciate if you > could have a look at the following problem. Apparently programs are > doomed to segfault on FreeBSD if dlopen()ed modules install exit > handlers via atexit(). Similar problem reports have cropped up before, > see e.g. >=20 > http://www.imagemagick.org/pipermail/magick-developers/2006-March/002523.= html >=20 > My system runs: >=20 > FreeBSD yeti.mininet 6.1-RELEASE FreeBSD 6.1-RELEASE #1: Mon Aug 28 > 22:24:48 CEST 2006 > markus@yeti.mininet:/usr/src/sys/i386/compile/YETI i386 >=20 > I'm one of the developers of libdbi, a database abstraction layer for > C, see http://libdbi.sourceforge.net. >=20 > libdbi is a library for programs which are supposed to be able to > access different database engines with a unified API. libdbi > essentially maps generic API calls to the specific database client > library calls of a particular database engine. To do this, libdbi > loads available database drivers at runtime via dlopen() calls. Each > of these drivers is linked against one database client > library. E.g. the Firebird driver is linked against > libfbclient.so. When libdbi is properly shut down, it unloads all > loaded drivers by calling dlclose() on each of them. >=20 > This design works well on all supported platforms and with all > supported database engines, with one exception: the Firebird driver on > FreeBSD invariably causes a segfault when the application linked > against libdbi exits: >=20 > #0 0x28514fe4 in ?? () > #1 0x281507c3 in __cxa_finalize () from /lib/libc.so.6 > #2 0x281503fe in exit () from /lib/libc.so.6 > #3 0x0804a40f in main (argc=3D1, argv=3D0xbfbfe754) at test_dbi.c:419 >=20 > The reason appears to be that the Firebird client libraries install > exit handlers via atexit(). Remember that due to libdbi's design to > load all available drivers whether or not they are used later, libdbi > will cause a crash even if no Firebird database is accessed - it is > sufficient that the driver has been loaded. As per Giorgos' suggestion > it is simple to circumvent this segfault by avoiding the call to > dlclose() before exiting, but I wonder whether there is a more robust > solution for this problem. >=20 > The attached minimal testcase is sufficient to illustrate the > problem. atexitmod.c defines a module which is loaded by datest.c Make > sure to fix the hardcoded path in datest.c before building the app. To > build the test program and watch it crash, do the following: >=20 > gcc -shared -o atexitmod.so atexitmod.c > gcc -o datest datest.c > ./datest >=20 > Commenting out either the atexit() call in atexitmod.c or the > dlclose() call in datest.c prevent the segfault. >=20 > If you find some solution, please cc me as I'm not subscribed to > freebsd-hackers. >=20 > regards, > Markus As designed. atexit should not be used by shared objects that do not expect themselves to live until actual exit() happens. ELF provides proper _init/_fini sections to support shared object initialization/destruction. --=20 Alexander Kabaev --Sig_/GwOo5uNn_S_7+PXNyx5=9Pm Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHeUJcQ6z1jMm+XZYRAiCzAJ94kVQ5yrRdhSdtjxzrhHZKPK3awACcDrSU q6TUk0RsoiMf0oN/S73q0nE= =9at7 -----END PGP SIGNATURE----- --Sig_/GwOo5uNn_S_7+PXNyx5=9Pm-- From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 20:05:02 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36B3416A417 for ; Mon, 31 Dec 2007 20:05:02 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from canonware.com (canonware.com [64.183.146.166]) by mx1.freebsd.org (Postfix) with ESMTP id 1CF6013C45A for ; Mon, 31 Dec 2007 20:05:02 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from [192.168.168.201] (unknown [192.168.168.201]) by canonware.com (Postfix) with ESMTP id 838BF1298C1; Mon, 31 Dec 2007 11:32:32 -0800 (PST) Message-ID: <477943B3.5080605@freebsd.org> Date: Mon, 31 Dec 2007 11:32:03 -0800 From: Jason Evans User-Agent: Thunderbird 1.5.0.12 (X11/20071018) MIME-Version: 1.0 To: Markus Hoenicka References: <18297.6718.750894.937199@yeti.mininet> In-Reply-To: <18297.6718.750894.937199@yeti.mininet> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 20:05:02 -0000 Markus Hoenicka wrote: > I've been redirected by Giorgos Keramidas to this list after reporting > a problem on the freebsd-questions list. I'd greatly appreciate if you > could have a look at the following problem. Apparently programs are > doomed to segfault on FreeBSD if dlopen()ed modules install exit > handlers via atexit(). Similar problem reports have cropped up before, It seems to me that you should *expect* a crash under the circumstances you describe. You are dlopen()ing a module, saving a pointer to a function within that module, unloading the module, then trying to call a function that is no longer mapped. The only way this could possibly work is if dlclose() doesn't really unmap the module. It is up to the programmer to avoid dangling pointers to unmapped modules. There are all sorts of variations on this bug, such as storing a pointer to a const string. You have to be really careful to be able to safely dlclose() a module. Jason From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 20:42:13 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61FB216A41B for ; Mon, 31 Dec 2007 20:42:13 +0000 (UTC) (envelope-from markus.hoenicka@mhoenicka.de) Received: from rrzmta2.rz.uni-regensburg.de (rrzmta2.rz.uni-regensburg.de [194.94.155.53]) by mx1.freebsd.org (Postfix) with ESMTP id 252D213C44B for ; Mon, 31 Dec 2007 20:42:12 +0000 (UTC) (envelope-from markus.hoenicka@mhoenicka.de) Received: from rrzmta2.rz.uni-regensburg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 398AB55506 for ; Mon, 31 Dec 2007 21:42:18 +0100 (CET) Received: from yeti.mininet (rrzras1-21.rz.uni-regensburg.de [132.199.208.31]) by rrzmta2.rz.uni-regensburg.de (Postfix) with ESMTP id 5E74D55526 for ; Mon, 31 Dec 2007 21:42:06 +0100 (CET) X-Mailer: emacs 21.3.1 (via feedmail 8 Q); VM 7.19 under Emacs 21.3.1 From: "Markus Hoenicka" MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18297.20596.564077.568365@yeti.mininet> Date: Mon, 31 Dec 2007 21:26:28 +0100 To: freebsd-hackers@freebsd.org In-Reply-To: <20071231142620.39f2fbd2@kan.dnsalias.net> References: <18297.6718.750894.937199@yeti.mininet> <20071231142620.39f2fbd2@kan.dnsalias.net> X-Mailman-Approved-At: Mon, 31 Dec 2007 22:15:41 +0000 Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 20:42:13 -0000 Alexander Kabaev writes: > As designed. atexit should not be used by shared objects that do not > expect themselves to live until actual exit() happens. ELF provides > proper _init/_fini sections to support shared object > initialization/destruction. > That is, the only real solution to this problem is to convince the Firebird folks to remove their atexit() calls from the client libraries? I'll try, but I'm not very confident this is going to work. Also, I'm wondering how other OSes handle this. I don't see this code crash on Linux, contrary to its design as you say. Thanks anyway to you, and to Jason Evans, for your replies. regards, Markus -- Markus Hoenicka markus.hoenicka@cats.de (Spam-protected email: replace the quadrupeds with "mhoenicka") http://www.mhoenicka.de From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 22:20:41 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9033F16A418 for ; Mon, 31 Dec 2007 22:20:41 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outF.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id 781E513C468 for ; Mon, 31 Dec 2007 22:20:41 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Mon, 31 Dec 2007 14:20:40 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id D9255126DCF; Mon, 31 Dec 2007 14:20:39 -0800 (PST) Message-ID: <47796B43.9050704@elischer.org> Date: Mon, 31 Dec 2007 14:20:51 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Markus Hoenicka References: <18297.6718.750894.937199@yeti.mininet> <20071231142620.39f2fbd2@kan.dnsalias.net> <18297.20596.564077.568365@yeti.mininet> In-Reply-To: <18297.20596.564077.568365@yeti.mininet> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 22:20:41 -0000 Markus Hoenicka wrote: > Alexander Kabaev writes: > > As designed. atexit should not be used by shared objects that do not > > expect themselves to live until actual exit() happens. ELF provides > > proper _init/_fini sections to support shared object > > initialization/destruction. > > > > That is, the only real solution to this problem is to convince the > Firebird folks to remove their atexit() calls from the client > libraries? I'll try, but I'm not very confident this is going to > work. Also, I'm wondering how other OSes handle this. I don't see this > code crash on Linux, contrary to its design as you say. > > Thanks anyway to you, and to Jason Evans, for your replies. Not sure but I'd guess that each library calls its at_exit entrypoints as part of its unload handling. > > regards, > Markus > From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 22:35:39 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F84716A417 for ; Mon, 31 Dec 2007 22:35:39 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail6.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id BE89D13C457 for ; Mon, 31 Dec 2007 22:35:38 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8q) with ESMTP id 226848344-1834499 for multiple; Mon, 31 Dec 2007 17:35:35 -0500 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id lBVMZZel099982; Mon, 31 Dec 2007 17:35:35 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Mon, 31 Dec 2007 17:35:42 -0500 User-Agent: KMail/1.9.6 References: <18297.6718.750894.937199@yeti.mininet> <18297.20596.564077.568365@yeti.mininet> <47796B43.9050704@elischer.org> In-Reply-To: <47796B43.9050704@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200712311735.43320.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Mon, 31 Dec 2007 17:35:35 -0500 (EST) X-Virus-Scanned: ClamAV 0.91.2/5314/Mon Dec 31 16:46:09 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Julian Elischer , Markus Hoenicka Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 22:35:39 -0000 On Monday 31 December 2007 05:20:51 pm Julian Elischer wrote: > Markus Hoenicka wrote: > > Alexander Kabaev writes: > > > As designed. atexit should not be used by shared objects that do not > > > expect themselves to live until actual exit() happens. ELF provides > > > proper _init/_fini sections to support shared object > > > initialization/destruction. > > > > > > > That is, the only real solution to this problem is to convince the > > Firebird folks to remove their atexit() calls from the client > > libraries? I'll try, but I'm not very confident this is going to > > work. Also, I'm wondering how other OSes handle this. I don't see this > > code crash on Linux, contrary to its design as you say. > > > > Thanks anyway to you, and to Jason Evans, for your replies. > > Not sure but I'd guess that each library calls its at_exit entrypoints > as part of its unload handling. Older gcc used to use atexit() for destructors (2.95.x for example) but newer versions of gcc use fini routines. We have a hack in the run time linker that walks the atexit() list for 4.x binaries and manually pulls out any routines in the mapped region of a shared object during dlclose() to invoke the destructors and fixup the atexit list. However, newer binaries don't need this. If you used a regular old static C++ singleton on 6.x instead of trying to be cute and call atexit() directly you would be fine. I've no idea if Linux treats atexit() special. The documentation below for atexit() is very clear that handlers are only invoked for normal program termination, not for dlclose(), so you definitely have buggy code. http://www.opengroup.org/onlinepubs/009695399/functions/atexit.html -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 23:06:14 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA60016A479 for ; Mon, 31 Dec 2007 23:06:14 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by mx1.freebsd.org (Postfix) with ESMTP id 776D213C469 for ; Mon, 31 Dec 2007 23:06:14 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so1007454anc.13 for ; Mon, 31 Dec 2007 15:06:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer:mime-version:content-type; bh=JYgISVvQ9+GJ0BObwtMFi75WPDRXw671YCZIHSJl/Zc=; b=YBVe2w9lj3cyenPk3x6ucjo4WqKC3Rp4MMs8bO3yvOqLAW0QRCPtgSN60BkKZPpJXHYbwITH/4IGgmugqc7wPaFqjq9TM6OXB5/b+w+tKRT28Y+zRPi2Qv1gVuQyKhNpfZwJ+AsVcg+UYjlIBTdzLxlf6nXXTZOjvkSe6wVDh1Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer:mime-version:content-type; b=rFdycBk/2kFv7kljgGzlrSqJqcL5qceOZQ8os56rhiSFdlRAel6lc8HPd/xZV8X2DIN0xabd7qMDBLNk2M+bV7gw1QFBwplNBWnkvTTTDdThze9yFwCGpXGUd39dbBCCvRw8LU6WhId7sR8o+PPONpFwttBd/1OaW4AqXewo80U= Received: by 10.100.249.9 with SMTP id w9mr26822930anh.105.1199142373721; Mon, 31 Dec 2007 15:06:13 -0800 (PST) Received: from kan.dnsalias.net ( [24.218.183.247]) by mx.google.com with ESMTPS id a42sm16609753rne.11.2007.12.31.15.06.12 (version=SSLv3 cipher=OTHER); Mon, 31 Dec 2007 15:06:12 -0800 (PST) Date: Mon, 31 Dec 2007 18:06:05 -0500 From: Alexander Kabaev To: Julian Elischer Message-ID: <20071231180605.1fb939a9@kan.dnsalias.net> In-Reply-To: <47796B43.9050704@elischer.org> References: <18297.6718.750894.937199@yeti.mininet> <20071231142620.39f2fbd2@kan.dnsalias.net> <18297.20596.564077.568365@yeti.mininet> <47796B43.9050704@elischer.org> X-Mailer: Claws Mail 3.0.2 (GTK+ 2.12.1; i386-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/GxjZ=1Wx=Y9c_dyXrLIsawV"; protocol="application/pgp-signature"; micalg=PGP-SHA1 Cc: freebsd-hackers@freebsd.org, Markus Hoenicka Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 23:06:14 -0000 --Sig_/GxjZ=1Wx=Y9c_dyXrLIsawV Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Mon, 31 Dec 2007 14:20:51 -0800 Julian Elischer wrote: > Markus Hoenicka wrote: > > Alexander Kabaev writes: > > > As designed. atexit should not be used by shared objects that do > > > not expect themselves to live until actual exit() happens. ELF > > > provides proper _init/_fini sections to support shared object > > > initialization/destruction. > > >=20 > >=20 > > That is, the only real solution to this problem is to convince the > > Firebird folks to remove their atexit() calls from the client > > libraries? I'll try, but I'm not very confident this is going to > > work. Also, I'm wondering how other OSes handle this. I don't see > > this code crash on Linux, contrary to its design as you say. > >=20 > > Thanks anyway to you, and to Jason Evans, for your replies. >=20 > Not sure but I'd guess that each library calls its at_exit entrypoints > as part of its unload handling. >=20 This is not possible in general case. The object that calls atexit() is not necessarily object that contains clean-up routine, so when atexit() is to be called: when object that contains the routine itself is being unloaded or when the object that registered it goes away? Also, calling atexit hooks at that time is semantically wrong for atexit() as it is defined, dlclose(3) is NOT exit(2). C++ ABI defines a special __cxa_atexit function that does have proper semantics to work around atexit() deficiencies. We can hack rtld/libc to pull dangling atexit off the list entries when shared objects are being unloaded or to add extra references to libraries pointed to by atexit entries, but that will just paper over the general issue of Firebird (and like) folks using the API improperly. --=20 Alexander Kabaev --Sig_/GxjZ=1Wx=Y9c_dyXrLIsawV Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHeXXdQ6z1jMm+XZYRAseDAJ9V8uuZ7O6k+Fz9RpaGB0ZKMZ6nbQCcDy9X PluFdI27c2S0HY6dcmgk+mc= =qkx9 -----END PGP SIGNATURE----- --Sig_/GxjZ=1Wx=Y9c_dyXrLIsawV-- From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 31 23:18:34 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 083A816A41A for ; Mon, 31 Dec 2007 23:18:34 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outS.internet-mail-service.net (outS.internet-mail-service.net [216.240.47.242]) by mx1.freebsd.org (Postfix) with ESMTP id DDDE813C455 for ; Mon, 31 Dec 2007 23:18:33 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Mon, 31 Dec 2007 15:18:33 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id A75D4126DDD; Mon, 31 Dec 2007 15:18:32 -0800 (PST) Message-ID: <477978D5.3000200@elischer.org> Date: Mon, 31 Dec 2007 15:18:45 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Kabaev References: <18297.6718.750894.937199@yeti.mininet> <20071231142620.39f2fbd2@kan.dnsalias.net> <18297.20596.564077.568365@yeti.mininet> <47796B43.9050704@elischer.org> <20071231180605.1fb939a9@kan.dnsalias.net> In-Reply-To: <20071231180605.1fb939a9@kan.dnsalias.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Markus Hoenicka Subject: Re: dlopen(), atexit() crash on FreeBSD (testcase included) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2007 23:18:34 -0000 Alexander Kabaev wrote: > On Mon, 31 Dec 2007 14:20:51 -0800 > Julian Elischer wrote: > >> Markus Hoenicka wrote: >>> Alexander Kabaev writes: >>> > As designed. atexit should not be used by shared objects that do >>> > not expect themselves to live until actual exit() happens. ELF >>> > provides proper _init/_fini sections to support shared object >>> > initialization/destruction. >>> > >>> >>> That is, the only real solution to this problem is to convince the >>> Firebird folks to remove their atexit() calls from the client >>> libraries? I'll try, but I'm not very confident this is going to >>> work. Also, I'm wondering how other OSes handle this. I don't see >>> this code crash on Linux, contrary to its design as you say. >>> >>> Thanks anyway to you, and to Jason Evans, for your replies. >> Not sure but I'd guess that each library calls its at_exit entrypoints >> as part of its unload handling. >> > This is not possible in general case. The object that calls atexit() > is not necessarily object that contains clean-up routine, so when > atexit() is to be called: when object that contains the routine itself > is being unloaded or when the object that registered it goes away? Also, > calling atexit hooks at that time is semantically wrong for atexit() as > it is defined, dlclose(3) is NOT exit(2). C++ ABI defines a special > __cxa_atexit function that does have proper semantics to work around > atexit() deficiencies. > > We can hack rtld/libc to pull dangling atexit off the list entries when > shared objects are being unloaded or to add extra references to > libraries pointed to by atexit entries, but that will just paper over > the general issue of Firebird (and like) folks using the API improperly. > Yes.. I wasn't saying it was a good idea, just that it may be something that the linux folks might have done.