From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 05:14:42 2006 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9189416A408; Mon, 17 Apr 2006 05:14:42 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C57843D4C; Mon, 17 Apr 2006 05:14:41 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 2BB801A4E30; Sun, 16 Apr 2006 22:14:41 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 720795169B; Mon, 17 Apr 2006 01:14:40 -0400 (EDT) Date: Mon, 17 Apr 2006 01:14:40 -0400 From: Kris Kennaway To: current@FreeBSD.org, smp@FreeBSD.org Message-ID: <20060417051440.GB60956@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="s/l3CgOIzMHHjg/5" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Cc: Subject: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 05:14:42 -0000 --s/l3CgOIzMHHjg/5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I have been measuring performance with the supersmack benchmark on a 12-cpu sparc64 E4500. This is in the context of a patch I made to do exponential backoff during mutex spins, which has a large positive effect (~17%) in this case. However, even with CVS sources I found something very odd: when MUTEX_PROFILING is compiled in but not enabled (which usually still makes things slower), I see a clear 38% +/- 3% performance *increase* on this benchmark. The effects of MUTEX_PROFILING are (as far as I can see): - Increases size of struct mtx - Enables MUTEX_NOINLINE - Enables unconditional atomic_add_int at kern_mutex.c:484 (this may be unintentional; I think it should be conditional on mutex_prof_enable = 1) - Adds extra reads/writes of the struct mtx in the mutex lock/unlock ops - Adds a chunk of code to _mtx_unlock_flags But: - Removing the atomic_add_int from kern_mutex.c:484 seems to make it about 5% *slower*. I didn't measure this carefully yet though. When MUTEX_PROFILING is disabled: - MUTEX_NOINLINE doesn't help performance noticeably - Unconditionally changing size of struct mtx doesn't help - Compiling -O2 instead of -O doesn't help - Gutting the bulk of the mutex profiling code and leaving only the extra struct mtx accesses did not help - Here is the readelf output of the function table in kern_mutex.o in the non-mutex profiling case: Num: Value Size Type Bind Vis Ndx Name 9: 0000000000000900 404 FUNC LOCAL DEFAULT 1 db_show_mtx 18: 0000000000000000 68 FUNC GLOBAL DEFAULT 1 _mtx_lock_flags 19: 0000000000000220 420 FUNC GLOBAL DEFAULT 1 _mtx_lock_sleep 20: 0000000000000060 64 FUNC GLOBAL DEFAULT 1 _mtx_unlock_flags 21: 0000000000000500 508 FUNC GLOBAL DEFAULT 1 _mtx_unlock_sleep 22: 00000000000000a0 112 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin_flags 24: 00000000000003e0 236 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin 25: 0000000000000120 92 FUNC GLOBAL DEFAULT 1 _mtx_unlock_spin_flags 27: 0000000000000180 132 FUNC GLOBAL DEFAULT 1 _mtx_trylock 30: 00000000000004e0 8 FUNC GLOBAL DEFAULT 1 do_nothing 42: 0000000000000700 36 FUNC GLOBAL DEFAULT 1 mtx_sysinit 43: 0000000000000740 140 FUNC GLOBAL DEFAULT 1 mtx_init 45: 00000000000007e0 96 FUNC GLOBAL DEFAULT 1 mtx_destroy 48: 0000000000000840 188 FUNC GLOBAL DEFAULT 1 mutex_init and in the mutex profiling case: 9: 0000000000000ec0 404 FUNC LOCAL DEFAULT 1 db_show_mtx 44: 0000000000000000 40 FUNC LOCAL DEFAULT 1 nanoseconds 46: 0000000000000040 584 FUNC LOCAL DEFAULT 1 dump_mutex_prof_stats 51: 00000000000002a0 192 FUNC LOCAL DEFAULT 1 reset_mutex_prof_stats 68: 0000000000000ce0 36 FUNC GLOBAL DEFAULT 1 mtx_sysinit 69: 0000000000000dc0 96 FUNC GLOBAL DEFAULT 1 mtx_destroy 71: 0000000000000740 92 FUNC GLOBAL DEFAULT 1 _mtx_unlock_spin_flags 75: 00000000000006c0 112 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin_flags 81: 0000000000000360 132 FUNC GLOBAL DEFAULT 1 _mtx_lock_flags 82: 0000000000000840 484 FUNC GLOBAL DEFAULT 1 _mtx_lock_sleep 83: 0000000000000400 684 FUNC GLOBAL DEFAULT 1 _mtx_unlock_flags 86: 0000000000000b60 360 FUNC GLOBAL DEFAULT 1 _mtx_unlock_sleep 88: 0000000000000a40 236 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin 90: 00000000000007a0 132 FUNC GLOBAL DEFAULT 1 _mtx_trylock 93: 0000000000000b40 8 FUNC GLOBAL DEFAULT 1 do_nothing 105: 0000000000000d20 160 FUNC GLOBAL DEFAULT 1 mtx_init 109: 0000000000000e20 160 FUNC GLOBAL DEFAULT 1 mutex_init Most mutex operations grow a bit with MUTEX_PROFILING (except _mtx_unlock_sleep which shrinks because of the MUTEX_NOINLINE, see above), but the biggest growth is from the extra blob of unused (in this case) code in _mtx_unlock_flags(). This function is 0x100 aligned in the mutex profiling case. * I tried forcing 0x100 alignment on kern_mutex.o in the !mutex profiling case, which did not help (also _mtx_unlock_flags() should not be taking much CPU time; the bulk of it should be in the mutex spins since there are some very heavily contended mutexes) * Kernel profiling is apparently not supported on sparc, so I can't easily work out where it is spending time. * Our best guess is that mutex profiling is doing something that reduces contention on this very heavily contended mutex (unp), but I'd like to know what is happening precisely so I can maybe make use of it. Can anyone think of what may be happening that I've missed? Kris --s/l3CgOIzMHHjg/5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEQyQ/Wry0BWjoQKURAv24AKC37QSA9t8kT53QwZAUj9oeFjwl5gCgodOY oDwnnMAMxSnVY3Nb+dYOdGE= =l38l -----END PGP SIGNATURE----- --s/l3CgOIzMHHjg/5-- From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 05:16:12 2006 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BD5F716A400; Mon, 17 Apr 2006 05:16:12 +0000 (UTC) (envelope-from matusita@violin.ocn.ne.jp) Received: from mv-osn-hcb003.ocn.ad.jp (mv-osn-hcb003.ocn.ad.jp [60.37.51.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8584F43D46; Mon, 17 Apr 2006 05:16:11 +0000 (GMT) (envelope-from matusita@violin.ocn.ne.jp) Received: from vcviolin.ocn.ne.jp (mv-osn-hcb003 [60.37.51.8]) by mv-osn-hcb003.ocn.ad.jp (Postfix) with ESMTP id B1B42341AD; Mon, 17 Apr 2006 14:16:10 +0900 (JST) Received: from violin.ocn.ne.jp (p1178-ipad68osakakita.osaka.ocn.ne.jp [221.185.124.178]) by vcviolin.ocn.ne.jp (Postfix) with ESMTP; Mon, 17 Apr 2006 14:16:10 +0900 (JST) Received: from syvalion.matatabi.or.jp (3k0jfth5rci9069j@localhost [127.0.0.1]) by syvalion.matatabi.or.jp (8.13.4/8.13.4) with ESMTP id k3H5G9hO026061 for ; Mon, 17 Apr 2006 14:16:09 +0900 (JST) (envelope-from owner-freebsd-current@freebsd.org) Received: (from cyrus@localhost) by syvalion.matatabi.or.jp (8.13.4/8.13.4/Submit) id k3H5G9Bj026060 for makoto.matusita.home@gmail.com; Mon, 17 Apr 2006 14:16:09 +0900 (JST) (envelope-from owner-freebsd-current@freebsd.org) X-Authentication-Warning: galtvalion.matatabi.or.jp: cyrus set sender to owner-freebsd-current@freebsd.org using -f Received: from galtvalion.matatabi.or.jp ([unix socket]) by galtvalion.matatabi.or.jp (Cyrus v2.3.1) with LMTPA; Mon, 17 Apr 2006 14:16:09 +0900 X-Sieve: CMU Sieve 2.3 Received: from localhost [::1] by galtvalion.matatabi.or.jp with POP3 (fetchmail-6.3.2) for (single-drop); Mon, 17 Apr 2006 14:16:09 +0900 (JST) Received: from castle.jp.FreeBSD.org (castle.jp.FreeBSD.org [210.226.20.15]) by ns.matatabi.or.jp (8.12.11.20060308/3.7W/MATATABI-1.0v9-NS1.2) with ESMTP id k3H5FLsg068105 for ; Mon, 17 Apr 2006 14:15:22 +0900 (JST) (envelope-from owner-freebsd-current@freebsd.org) Received: from mx2.freebsd.org (mx2.freebsd.org [216.136.204.119]) by castle.jp.FreeBSD.org (8.11.6p2+3.4W/8.11.3) with ESMTP/inet id k3H5FKQ21819 for ; Mon, 17 Apr 2006 14:15:20 +0900 (JST) (envelope-from owner-freebsd-current@freebsd.org) Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18]) by mx2.freebsd.org (Postfix) with ESMTP id 8AC5C583CC for ; Mon, 17 Apr 2006 05:14:53 +0000 (GMT) (envelope-from owner-freebsd-current@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 98F5816A445; Mon, 17 Apr 2006 05:14:52 +0000 (UTC) (envelope-from owner-freebsd-current@freebsd.org) X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9189416A408; Mon, 17 Apr 2006 05:14:42 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C57843D4C; Mon, 17 Apr 2006 05:14:41 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 2BB801A4E30; Sun, 16 Apr 2006 22:14:41 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 720795169B; Mon, 17 Apr 2006 01:14:40 -0400 (EDT) Date: Mon, 17 Apr 2006 01:14:40 -0400 From: Kris Kennaway To: current@freebsd.org, smp@freebsd.org Message-ID: <20060417051440.GB60956@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="s/l3CgOIzMHHjg/5" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Sender: owner-freebsd-current@freebsd.org Errors-To: owner-freebsd-current@freebsd.org X-UIDL: [X@!!PWh"!-#p"!2/%"! X-Text-Classification: ok X-POPFile-Link: http://syvalion.matatabi.or.jp:11080/jump_to_message?view=475783 Cc: Subject: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 05:16:12 -0000 --s/l3CgOIzMHHjg/5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I have been measuring performance with the supersmack benchmark on a 12-cpu sparc64 E4500. This is in the context of a patch I made to do exponential backoff during mutex spins, which has a large positive effect (~17%) in this case. However, even with CVS sources I found something very odd: when MUTEX_PROFILING is compiled in but not enabled (which usually still makes things slower), I see a clear 38% +/- 3% performance *increase* on this benchmark. The effects of MUTEX_PROFILING are (as far as I can see): - Increases size of struct mtx - Enables MUTEX_NOINLINE - Enables unconditional atomic_add_int at kern_mutex.c:484 (this may be unintentional; I think it should be conditional on mutex_prof_enable = 1) - Adds extra reads/writes of the struct mtx in the mutex lock/unlock ops - Adds a chunk of code to _mtx_unlock_flags But: - Removing the atomic_add_int from kern_mutex.c:484 seems to make it about 5% *slower*. I didn't measure this carefully yet though. When MUTEX_PROFILING is disabled: - MUTEX_NOINLINE doesn't help performance noticeably - Unconditionally changing size of struct mtx doesn't help - Compiling -O2 instead of -O doesn't help - Gutting the bulk of the mutex profiling code and leaving only the extra struct mtx accesses did not help - Here is the readelf output of the function table in kern_mutex.o in the non-mutex profiling case: Num: Value Size Type Bind Vis Ndx Name 9: 0000000000000900 404 FUNC LOCAL DEFAULT 1 db_show_mtx 18: 0000000000000000 68 FUNC GLOBAL DEFAULT 1 _mtx_lock_flags 19: 0000000000000220 420 FUNC GLOBAL DEFAULT 1 _mtx_lock_sleep 20: 0000000000000060 64 FUNC GLOBAL DEFAULT 1 _mtx_unlock_flags 21: 0000000000000500 508 FUNC GLOBAL DEFAULT 1 _mtx_unlock_sleep 22: 00000000000000a0 112 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin_flags 24: 00000000000003e0 236 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin 25: 0000000000000120 92 FUNC GLOBAL DEFAULT 1 _mtx_unlock_spin_flags 27: 0000000000000180 132 FUNC GLOBAL DEFAULT 1 _mtx_trylock 30: 00000000000004e0 8 FUNC GLOBAL DEFAULT 1 do_nothing 42: 0000000000000700 36 FUNC GLOBAL DEFAULT 1 mtx_sysinit 43: 0000000000000740 140 FUNC GLOBAL DEFAULT 1 mtx_init 45: 00000000000007e0 96 FUNC GLOBAL DEFAULT 1 mtx_destroy 48: 0000000000000840 188 FUNC GLOBAL DEFAULT 1 mutex_init and in the mutex profiling case: 9: 0000000000000ec0 404 FUNC LOCAL DEFAULT 1 db_show_mtx 44: 0000000000000000 40 FUNC LOCAL DEFAULT 1 nanoseconds 46: 0000000000000040 584 FUNC LOCAL DEFAULT 1 dump_mutex_prof_stats 51: 00000000000002a0 192 FUNC LOCAL DEFAULT 1 reset_mutex_prof_stats 68: 0000000000000ce0 36 FUNC GLOBAL DEFAULT 1 mtx_sysinit 69: 0000000000000dc0 96 FUNC GLOBAL DEFAULT 1 mtx_destroy 71: 0000000000000740 92 FUNC GLOBAL DEFAULT 1 _mtx_unlock_spin_flags 75: 00000000000006c0 112 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin_flags 81: 0000000000000360 132 FUNC GLOBAL DEFAULT 1 _mtx_lock_flags 82: 0000000000000840 484 FUNC GLOBAL DEFAULT 1 _mtx_lock_sleep 83: 0000000000000400 684 FUNC GLOBAL DEFAULT 1 _mtx_unlock_flags 86: 0000000000000b60 360 FUNC GLOBAL DEFAULT 1 _mtx_unlock_sleep 88: 0000000000000a40 236 FUNC GLOBAL DEFAULT 1 _mtx_lock_spin 90: 00000000000007a0 132 FUNC GLOBAL DEFAULT 1 _mtx_trylock 93: 0000000000000b40 8 FUNC GLOBAL DEFAULT 1 do_nothing 105: 0000000000000d20 160 FUNC GLOBAL DEFAULT 1 mtx_init 109: 0000000000000e20 160 FUNC GLOBAL DEFAULT 1 mutex_init Most mutex operations grow a bit with MUTEX_PROFILING (except _mtx_unlock_sleep which shrinks because of the MUTEX_NOINLINE, see above), but the biggest growth is from the extra blob of unused (in this case) code in _mtx_unlock_flags(). This function is 0x100 aligned in the mutex profiling case. * I tried forcing 0x100 alignment on kern_mutex.o in the !mutex profiling case, which did not help (also _mtx_unlock_flags() should not be taking much CPU time; the bulk of it should be in the mutex spins since there are some very heavily contended mutexes) * Kernel profiling is apparently not supported on sparc, so I can't easily work out where it is spending time. * Our best guess is that mutex profiling is doing something that reduces contention on this very heavily contended mutex (unp), but I'd like to know what is happening precisely so I can maybe make use of it. Can anyone think of what may be happening that I've missed? Kris --s/l3CgOIzMHHjg/5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEQyQ/Wry0BWjoQKURAv24AKC37QSA9t8kT53QwZAUj9oeFjwl5gCgodOY oDwnnMAMxSnVY3Nb+dYOdGE= =l38l -----END PGP SIGNATURE----- --s/l3CgOIzMHHjg/5-- From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 07:27:52 2006 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0A8C216A401; Mon, 17 Apr 2006 07:27:52 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD98A43D45; Mon, 17 Apr 2006 07:27:51 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id A3B231A4E34; Mon, 17 Apr 2006 00:27:51 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 86E8051664; Mon, 17 Apr 2006 03:27:50 -0400 (EDT) Date: Mon, 17 Apr 2006 03:27:50 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060417072750.GA62515@xor.obsecurity.org> References: <20060417051440.GB60956@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DocE+STaALJfprDB" Content-Disposition: inline In-Reply-To: <20060417051440.GB60956@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: smp@FreeBSD.org, current@FreeBSD.org Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 07:27:52 -0000 --DocE+STaALJfprDB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 17, 2006 at 01:14:40AM -0400, Kris Kennaway wrote: > * Our best guess is that mutex profiling is doing something that > reduces contention on this very heavily contended mutex (unp), but I'd > like to know what is happening precisely so I can maybe make use of > it. >=20 > Can anyone think of what may be happening that I've missed? I think it is just doing effectively the same thing as my exponential spin backoff patch, namely introducing delays with the effect of reducing common memory accesses. When I turn the maximum spin backoff limit *way* up (from 1600 to 51200) I get performance that slightly exceeds what I see from mutex profiling alone (adding mutex profiling again on top of this gives a small further increase, but only a few % and so probably achievable by further increasing the backoff limit). A limit of 51200 is not an appropriate default since it penalizes the common case of light to moderate contention. The point is that here basically all 12 CPUs are spinning on a single lock (kern/uipc_usrreq.c:308), so it's massively over-contended and all we can do is mitigate the effects of this. On this system, the maximum supersmack performance (3700 queries/sec) comes when there are only 6 clients, so (as jasone eloquently put it) with 10 clients the difference between 2300 queries/sec (with absurdly high backoff limits or mutex profiling) and 1450/sec (with reasonable backoff limits) is the difference between "slow" and "ass slow". Fortunately rwatson is working on breaking up this lock, so that should help a lot here :-) Kris --DocE+STaALJfprDB Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEQ0N1Wry0BWjoQKURAt3WAJ9Q3kipclq/L5pCLHeQUeYX+Z1DVwCffq4X Z0quoa5vJNHbZcITo1rZiGU= =W46k -----END PGP SIGNATURE----- --DocE+STaALJfprDB-- From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 07:54:09 2006 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9EFA116A403 for ; Mon, 17 Apr 2006 07:54:09 +0000 (UTC) (envelope-from surerlistmail@gmail.com) Received: from nproxy.gmail.com (nproxy.gmail.com [64.233.182.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id B498C43D4C for ; Mon, 17 Apr 2006 07:54:08 +0000 (GMT) (envelope-from surerlistmail@gmail.com) Received: by nproxy.gmail.com with SMTP id h2so366832nfe for ; Mon, 17 Apr 2006 00:54:07 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=l2+B6J6+/sWaobY7CpT6pBC2gPA5a4rVNJ2zF7Tn8vduzqkxXNuOvqEa/LhDqHq5v1OWRncyDCpgZhXZy0+Od9lPSkKmK1SWhFzfnaPMvdjuhBmh9pA9YMNQ0EXv8tvvWhrib7+DrWayA+UYFnhvZeJR3twUZKNT2lC08z6xlKk= Received: by 10.49.9.9 with SMTP id m9mr2350669nfi; Mon, 17 Apr 2006 00:54:07 -0700 (PDT) Received: by 10.49.61.12 with HTTP; Mon, 17 Apr 2006 00:54:07 -0700 (PDT) Message-ID: Date: Mon, 17 Apr 2006 03:54:07 -0400 From: "Surer Dink" To: "Kris Kennaway" , smp@freebsd.org, current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Cc: Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 07:54:09 -0000 On Mon, 17 Apr 2006, Kris Kennaway wrote: > On Mon, Apr 17, 2006 at 01:14:40AM -0400, Kris Kennaway wrote: > >> * Our best guess is that mutex profiling is doing something that >> reduces contention on this very heavily contended mutex (unp), but I'd >> like to know what is happening precisely so I can maybe make use of >> it. >> >> Can anyone think of what may be happening that I've missed? > > I think it is just doing effectively the same thing as my exponential > spin backoff patch, namely introducing delays with the effect of > reducing common memory accesses. When I turn the maximum spin backoff > limit *way* up (from 1600 to 51200) I get performance that slightly > exceeds what I see from mutex profiling alone (adding mutex profiling > again on top of this gives a small further increase, but only a few % > and so probably achievable by further increasing the backoff limit). > > A limit of 51200 is not an appropriate default since it penalizes the > common case of light to moderate contention. The point is that here > basically all 12 CPUs are spinning on a single lock > (kern/uipc_usrreq.c:308), so it's massively over-contended and all we > can do is mitigate the effects of this. > > On this system, the maximum supersmack performance (3700 queries/sec) > comes when there are only 6 clients, so (as jasone eloquently put it) > with 10 clients the difference between 2300 queries/sec (with absurdly > high backoff limits or mutex profiling) and 1450/sec (with reasonable > backoff limits) is the difference between "slow" and "ass slow". Please excuse if this is a stupid question - but might using MCS or QOLB locks in this situation be useful? From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 08:16:09 2006 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9859D16A401; Mon, 17 Apr 2006 08:16:09 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28DD143D46; Mon, 17 Apr 2006 08:16:08 +0000 (GMT) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id E465C20A1; Mon, 17 Apr 2006 10:16:03 +0200 (CEST) X-Spam-Tests: AWL,BAYES_00,FORGED_RCVD_HELO X-Spam-Learn: ham X-Spam-Score: -2.4/3.0 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on tim.des.no Received: from dwp.des.no (des.no [80.203.243.180]) by tim.des.no (Postfix) with ESMTP id CBACC20A0; Mon, 17 Apr 2006 10:16:03 +0200 (CEST) Received: by dwp.des.no (Postfix, from userid 1001) id 75A7EB811; Mon, 17 Apr 2006 10:16:03 +0200 (CEST) From: des@des.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=) To: Kris Kennaway References: <20060417051440.GB60956@xor.obsecurity.org> Date: Mon, 17 Apr 2006 10:16:03 +0200 In-Reply-To: <20060417051440.GB60956@xor.obsecurity.org> (Kris Kennaway's message of "Mon, 17 Apr 2006 01:14:40 -0400") Message-ID: <86mzekbosc.fsf@dwp.des.no> User-Agent: Gnus/5.110003 (No Gnus v0.3) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: smp@FreeBSD.org, current@FreeBSD.org Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 08:16:09 -0000 Kris Kennaway writes: > However, even with CVS sources I found something very odd: when > MUTEX_PROFILING is compiled in but not enabled (which usually still > makes things slower), I see a clear 38% +/- 3% performance *increase* > on this benchmark. Heisenberg strikes! Heisenberg strikes! Heisenberg strikes! You are uncertain. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 16:22:18 2006 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C4AF216A40E; Mon, 17 Apr 2006 16:22:18 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8042A43D48; Mon, 17 Apr 2006 16:22:18 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 67B051A4E55; Mon, 17 Apr 2006 09:22:18 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 4EB6951659; Mon, 17 Apr 2006 12:22:17 -0400 (EDT) Date: Mon, 17 Apr 2006 12:22:17 -0400 From: Kris Kennaway To: Surer Dink Message-ID: <20060417162216.GA90886@xor.obsecurity.org> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BOKacYhQ+x31HxR3" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: smp@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 16:22:18 -0000 --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 17, 2006 at 03:54:07AM -0400, Surer Dink wrote: > On Mon, 17 Apr 2006, Kris Kennaway wrote: >=20 > > On Mon, Apr 17, 2006 at 01:14:40AM -0400, Kris Kennaway wrote: > > > >> * Our best guess is that mutex profiling is doing something that > >> reduces contention on this very heavily contended mutex (unp), but I'd > >> like to know what is happening precisely so I can maybe make use of > >> it. > >> > >> Can anyone think of what may be happening that I've missed? > > > > I think it is just doing effectively the same thing as my exponential > > spin backoff patch, namely introducing delays with the effect of > > reducing common memory accesses. When I turn the maximum spin backoff > > limit *way* up (from 1600 to 51200) I get performance that slightly > > exceeds what I see from mutex profiling alone (adding mutex profiling > > again on top of this gives a small further increase, but only a few % > > and so probably achievable by further increasing the backoff limit). > > > > A limit of 51200 is not an appropriate default since it penalizes the > > common case of light to moderate contention. The point is that here > > basically all 12 CPUs are spinning on a single lock > > (kern/uipc_usrreq.c:308), so it's massively over-contended and all we > > can do is mitigate the effects of this. > > > > On this system, the maximum supersmack performance (3700 queries/sec) > > comes when there are only 6 clients, so (as jasone eloquently put it) > > with 10 clients the difference between 2300 queries/sec (with absurdly > > high backoff limits or mutex profiling) and 1450/sec (with reasonable > > backoff limits) is the difference between "slow" and "ass slow". >=20 > Please excuse if this is a stupid question - but might using MCS or > QOLB locks in this situation be useful? What are they? Kris --BOKacYhQ+x31HxR3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEQ8C4Wry0BWjoQKURAvt5AKCMRE+4/1wrRTGDt0LTHcHXKmtldQCg3XLw ZiMsQve4FuiR6QoL+N9ClhI= =WCKf -----END PGP SIGNATURE----- --BOKacYhQ+x31HxR3-- From owner-freebsd-smp@FreeBSD.ORG Mon Apr 17 18:13:00 2006 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 049A316A402 for ; Mon, 17 Apr 2006 18:13:00 +0000 (UTC) (envelope-from surerlistmail@gmail.com) Received: from nproxy.gmail.com (nproxy.gmail.com [64.233.182.189]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5DF0D43D53 for ; Mon, 17 Apr 2006 18:12:59 +0000 (GMT) (envelope-from surerlistmail@gmail.com) Received: by nproxy.gmail.com with SMTP id m18so536584nfc for ; Mon, 17 Apr 2006 11:12:57 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=iiPbSqKcuWJqD2SKR8u5sOSQhKWCsz533sY6WiJejKlOmMKBFrR0rZeK+mokcIgGMUUHzTy1HsDB3JQmGjRanR5h3nUpVUcD6g6ksDmKtuMpFg6T/GISEfzz0hLPzHUXF9tjEoQfjSdanzIckKsg2f3j6WMjZgp64U2ri0kkfuo= Received: by 10.48.31.4 with SMTP id e4mr724892nfe; Mon, 17 Apr 2006 11:12:57 -0700 (PDT) Received: by 10.49.61.12 with HTTP; Mon, 17 Apr 2006 11:12:57 -0700 (PDT) Message-ID: Date: Mon, 17 Apr 2006 14:12:57 -0400 From: "Surer Dink" To: "Kris Kennaway" , smp@freebsd.org, current@freebsd.org In-Reply-To: <20060417162216.GA90886@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <20060417162216.GA90886@xor.obsecurity.org> Cc: Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2006 18:13:00 -0000 On 4/17/06, Kris Kennaway wrote: > On Mon, Apr 17, 2006 at 03:54:07AM -0400, Surer Dink wrote: > > Please excuse if this is a stupid question - but might using MCS or > > QOLB locks in this situation be useful? > > What are they? Mellor-Crummy Scott: http://www.cs.rochester.edu/u/scott/papers/1991_ASPLOS_sync.pdf An overview comparing various possible optimizations for a few lock types, including MCS and QOLB: ftp://ftp.cs.utexas.edu/pub/dburger/papers/ISCA97_qolb.pdf I believe the QOLB proposal only suggested hardware modificaition for performance improvement, but could be implemented entirely in software - the overheads are high, but offer substantial performance benefit in high contention situations. MCS is based on QOSB, however fully implemented in software. There is also a proposal for changing lock to MCS dynamically, however I have not read it: ftp://ftp.cag.lcs.mit.edu/pub/papers/pdf/reactive.pdf From owner-freebsd-smp@FreeBSD.ORG Tue Apr 18 00:03:56 2006 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2D45516A406; Tue, 18 Apr 2006 00:03:56 +0000 (UTC) (envelope-from tlambert2@mindspring.com) Received: from elasmtp-kukur.atl.sa.earthlink.net (elasmtp-kukur.atl.sa.earthlink.net [209.86.89.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 21EBD43D46; Tue, 18 Apr 2006 00:03:54 +0000 (GMT) (envelope-from tlambert2@mindspring.com) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dk20050327; d=mindspring.com; b=dinjTO8y91sED8bypkTHIS+aI9DFOWWqPHjQ2q7OfJhJJiHVuuOibxXBWQOu/83l; h=Received:Message-ID:Date:From:Reply-To:To:Subject:Mime-Version:Content-Type:Content-Transfer-Encoding:X-Mailer:X-ELNK-Trace:X-Originating-IP; Received: from [209.86.224.28] (helo=mswamui-blood.atl.sa.earthlink.net) by elasmtp-kukur.atl.sa.earthlink.net with asmtp (Exim 4.34) id 1FVdhA-0008AZ-SI; Mon, 17 Apr 2006 20:03:52 -0400 Message-ID: <18400116.1145318632626.JavaMail.root@mswamui-blood.atl.sa.earthlink.net> Date: Mon, 17 Apr 2006 17:03:52 -0700 (GMT-07:00) From: Terry Lambert To: Surer Dink , Kris Kennaway , smp@freebsd.org, current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: EarthLink Zoo Mail 1.0 X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a4b11c7e29a110beee3e546ab6f01399b1a2d4e88014a4647c350badd9bab72f9c350badd9bab72f9c X-Originating-IP: 209.86.224.28 Cc: Subject: Re: Anomalous performance increase from mutex profiling X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Terry Lambert List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Apr 2006 00:03:56 -0000 >From: Surer Dink >Sent: Apr 17, 2006 11:12 AM >To: Kris Kennaway , smp@freebsd.org, current@freebsd.org >Subject: Re: Anomalous performance increase from mutex profiling > >On 4/17/06, Kris Kennaway wrote: >> On Mon, Apr 17, 2006 at 03:54:07AM -0400, Surer Dink wrote: >> > Please excuse if this is a stupid question - but might using MCS or >> > QOLB locks in this situation be useful? >> >> What are they? > >Mellor-Crummy Scott: >http://www.cs.rochester.edu/u/scott/papers/1991_ASPLOS_sync.pdf > >An overview comparing various possible optimizations for a few lock >types, including MCS and QOLB: >ftp://ftp.cs.utexas.edu/pub/dburger/papers/ISCA97_qolb.pdf > >I believe the QOLB proposal only suggested hardware modificaition for >performance improvement, but could be implemented entirely in software >- the overheads are high, but offer substantial performance benefit in >high contention situations. MCS is based on QOSB, however fully >implemented in software. > >There is also a proposal for changing lock to MCS dynamically, however >I have not read it: >ftp://ftp.cag.lcs.mit.edu/pub/papers/pdf/reactive.pdf Implementation under the GPL, including MCAS and other lockless algorithms is available here (unfortunately, GPL'ed): http://www.cl.cam.ac.uk/Research/SRG/netos/lock-free/ Note that there is no general implementation that does not require use of asembly language; e.g. the PPC does not have a CAS instruction, and it has a weak memory model which means you will need to use an explicit barrier in your implementation (Mac OS X/Darwin provide implementations of the necessary atomic operations in the header file as of the Tiger release). -- Terry From owner-freebsd-smp@FreeBSD.ORG Wed Apr 19 08:01:32 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 706A616A400 for ; Wed, 19 Apr 2006 08:01:32 +0000 (UTC) (envelope-from dgoldstein@web.de) Received: from fmmailgate05.web.de (fmmailgate05.web.de [217.72.192.243]) by mx1.FreeBSD.org (Postfix) with ESMTP id 78CBB43D46 for ; Wed, 19 Apr 2006 08:01:31 +0000 (GMT) (envelope-from dgoldstein@web.de) Received: by fmmailgate05.web.de (8.12.10/8.12.10/webde Linux 0.7) with SMTP id k3J81Tw8009808 for ; Wed, 19 Apr 2006 10:01:29 +0200 Received: from [62.206.228.99] by freemailng0704.web.de with HTTP; Wed, 19 Apr 2006 10:01:27 +0200 Date: Wed, 19 Apr 2006 10:01:27 +0200 Message-Id: <257331718@web.de> MIME-Version: 1.0 From: Dmitri Goldstein To: freebsd-smp@freebsd.org Precedence: fm-user Organization: http://freemail.web.de/ Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Subject: FreeBSD crashes on dual core pentium. Help! X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2006 08:01:32 -0000 Can anybody help me? I have Pentium D920 on Asus P5ND2 SLI with NForce 4 SLI Intel edition. This processor is Dual Core! At last weekend I tried to install a new 6.1-RC1 but unsuccessfull. At first I tried 6.1-RC1 amd64 (for EM64T support) The notmal UP kernel start successfull. The SMP kernel crashes after "SMP: cpu #1 Lunched..." I get "kernel trap 12 with interrupts disabled." If I turn off the HT in BIOS both UP and SMP kernel start successfull. Yesterday I tried also 6.0-RELEASE - with the same result. 6.0-RELEASE i386 smp - same result. The SMP kernel with customisations, such as IPFIREWALL, NETGRAPH etc, starts successfull but during 1s-12h craches with the same message "kernel trap 12 with interrupts disabled." Please HELP me!!! --- Thanks in advance, dgo _______________________________________________________________ SMS schreiben mit WEB.DE FreeMail - einfach, schnell und kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192 From owner-freebsd-smp@FreeBSD.ORG Wed Apr 19 11:38:33 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C7A616A400 for ; Wed, 19 Apr 2006 11:38:33 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 924B743D45 for ; Wed, 19 Apr 2006 11:38:32 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (zion.baldwin.cx [192.168.0.7]) (authenticated bits=0) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k3JBcV7N002998; Wed, 19 Apr 2006 07:38:31 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-smp@freebsd.org Date: Wed, 19 Apr 2006 07:38:29 -0400 User-Agent: KMail/1.8.3 References: <257331718@web.de> In-Reply-To: <257331718@web.de> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200604190738.30463.jhb@freebsd.org> X-Virus-Scanned: ClamAV 0.87.1/1407/Tue Apr 18 17:01:55 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: Dmitri Goldstein Subject: Re: FreeBSD crashes on dual core pentium. Help! X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2006 11:38:33 -0000 On Wednesday 19 April 2006 04:01 am, Dmitri Goldstein wrote: > Can anybody help me? > > I have Pentium D920 on Asus P5ND2 SLI with NForce 4 SLI Intel edition. > This processor is Dual Core! > At last weekend I tried to install a new 6.1-RC1 but unsuccessfull. > At first I tried 6.1-RC1 amd64 (for EM64T support) > The notmal UP kernel start successfull. > The SMP kernel crashes after "SMP: cpu #1 Lunched..." I get "kernel trap = 12 > with interrupts disabled." If I turn off the HT in BIOS both UP and SMP > kernel start successfull. Yesterday I tried also 6.0-RELEASE - with the > same result. 6.0-RELEASE i386 smp - same result. The SMP kernel with > customisations, such as IPFIREWALL, NETGRAPH etc, starts successfull but > during 1s-12h craches with the same message "kernel trap 12 with interrup= ts > disabled." > > Please HELP me!!! Can you use a serial console to capture the dmesg along with the crash=20 messages? If you could compile DDB into the kernel and get a stack trace=20 that would also be very helpful. We can't do anything without more details= =20 however. =2D-=20 John Baldwin =A0<>< =A0http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" =A0=3D =A0http://www.FreeBSD.org From owner-freebsd-smp@FreeBSD.ORG Wed Apr 19 17:57:19 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF9F916A479 for ; Wed, 19 Apr 2006 17:57:19 +0000 (UTC) (envelope-from hvleest@signet.nl) Received: from bsd.local.jellevanleest.nl (cp417515-b.dbsch1.nb.home.nl [84.27.37.200]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3CF0A43D6A for ; Wed, 19 Apr 2006 17:57:17 +0000 (GMT) (envelope-from hvleest@signet.nl) Received: from [192.168.100.101] (unknown [192.168.100.101]) by bsd.local.jellevanleest.nl (Postfix) with ESMTP id CCDEB1143B for ; Wed, 19 Apr 2006 19:57:17 +0200 (CEST) Message-ID: <444679FB.7090602@signet.nl> Date: Wed, 19 Apr 2006 19:57:15 +0200 From: Hans van Leest User-Agent: Mozilla Thunderbird 1.0.7 (Windows/20050923) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-smp@freebsd.org References: <257331718@web.de> <200604190738.30463.jhb@freebsd.org> In-Reply-To: <200604190738.30463.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: FreeBSD crashes on dual core pentium. Help! X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2006 17:57:20 -0000 I've got simular problems, only on a dual xeon on a ASUS board I think it's an ACPI problem of the ASUS bios. There are more problems with ASUS bios'es. So I'm trying to disable ACPI in the kernel etc. Using 6.1-RC1 I'll post the results Greez Hans John Baldwin wrote: > On Wednesday 19 April 2006 04:01 am, Dmitri Goldstein wrote: > >>Can anybody help me? >> >>I have Pentium D920 on Asus P5ND2 SLI with NForce 4 SLI Intel edition. >>This processor is Dual Core! >>At last weekend I tried to install a new 6.1-RC1 but unsuccessfull. >>At first I tried 6.1-RC1 amd64 (for EM64T support) >>The notmal UP kernel start successfull. >>The SMP kernel crashes after "SMP: cpu #1 Lunched..." I get "kernel trap 12 >>with interrupts disabled." If I turn off the HT in BIOS both UP and SMP >>kernel start successfull. Yesterday I tried also 6.0-RELEASE - with the >>same result. 6.0-RELEASE i386 smp - same result. The SMP kernel with >>customisations, such as IPFIREWALL, NETGRAPH etc, starts successfull but >>during 1s-12h craches with the same message "kernel trap 12 with interrupts >>disabled." >> >>Please HELP me!!! > > > Can you use a serial console to capture the dmesg along with the crash > messages? If you could compile DDB into the kernel and get a stack trace > that would also be very helpful. We can't do anything without more details > however. > From owner-freebsd-smp@FreeBSD.ORG Wed Apr 19 18:15:07 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7D5BB16A418 for ; Wed, 19 Apr 2006 18:15:07 +0000 (UTC) (envelope-from goldshtain@gmail.com) Received: from uproxy.gmail.com (uproxy.gmail.com [66.249.92.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id 803D143D5E for ; Wed, 19 Apr 2006 18:14:53 +0000 (GMT) (envelope-from goldshtain@gmail.com) Received: by uproxy.gmail.com with SMTP id m3so926969ugc for ; Wed, 19 Apr 2006 11:14:52 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=tkTxOKQP4PoDTSyeoNME8CaEYVAhGFa7gOaMcrFGw4bQ0SYfua5WBFz/ToO3IP1M+r+EW8HrbiSk43AJrwUQ4itG3kklnOjWrNCEry/VBMzipR8jNiCckZaov0P0oCkBvHYov54ctViCy7SiBP++nLJLMTDYqgOeFtQqG3fD+cs= Received: by 10.78.32.16 with SMTP id f16mr49530huf; Wed, 19 Apr 2006 11:14:51 -0700 (PDT) Received: by 10.78.27.17 with HTTP; Wed, 19 Apr 2006 11:14:51 -0700 (PDT) Message-ID: <380ab9b80604191114s546d9b39y820bae417a5c7142@mail.gmail.com> Date: Wed, 19 Apr 2006 20:14:51 +0200 From: "Dmitry Goldshtain" To: "Hans van Leest" In-Reply-To: <444679FB.7090602@signet.nl> MIME-Version: 1.0 References: <257331718@web.de> <200604190738.30463.jhb@freebsd.org> <444679FB.7090602@signet.nl> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-smp@freebsd.org Subject: Re: FreeBSD crashes on dual core pentium. Help! X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Apr 2006 18:15:07 -0000 I think without ACPI OS can't recognise many (any modern?) devices, including dual core processor :( 2006/4/19, Hans van Leest : > > I've got simular problems, only on a dual xeon on a ASUS board > I think it's an ACPI problem of the ASUS bios. There are more problems > with ASUS bios'es. > > So I'm trying to disable ACPI in the kernel etc. Using 6.1-RC1 > > I'll post the results > > Greez Hans > > John Baldwin wrote: > > On Wednesday 19 April 2006 04:01 am, Dmitri Goldstein wrote: > > > >>Can anybody help me? > >> > >>I have Pentium D920 on Asus P5ND2 SLI with NForce 4 SLI Intel edition. > >>This processor is Dual Core! > >>At last weekend I tried to install a new 6.1-RC1 but unsuccessfull. > >>At first I tried 6.1-RC1 amd64 (for EM64T support) > >>The notmal UP kernel start successfull. > >>The SMP kernel crashes after "SMP: cpu #1 Lunched..." I get "kernel tra= p > 12 > >>with interrupts disabled." If I turn off the HT in BIOS both UP and SMP > >>kernel start successfull. Yesterday I tried also 6.0-RELEASE - with the > >>same result. 6.0-RELEASE i386 smp - same result. The SMP kernel with > >>customisations, such as IPFIREWALL, NETGRAPH etc, starts successfull bu= t > >>during 1s-12h craches with the same message "kernel trap 12 with > interrupts > >>disabled." > >> > >>Please HELP me!!! > > > > > > Can you use a serial console to capture the dmesg along with the crash > > messages? If you could compile DDB into the kernel and get a stack > trace > > that would also be very helpful. We can't do anything without more > details > > however. > > > > _______________________________________________ > freebsd-smp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-smp > To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" > -- Dipl.-Inf. Dimitrij Goldstein Software Architect Please note our new contacts. idesis GmbH Rellinghauser Stra=DFe 334F D-45136 Essen www.idesis.de T +49 (0700) 433747 01 F +49 (0700) 433747 99 M +49 (0163) 433747 6 E dimitrij.goldstein@idesis.de ______________________________________ From owner-freebsd-smp@FreeBSD.ORG Thu Apr 20 01:54:30 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 328C316A405 for ; Thu, 20 Apr 2006 01:54:30 +0000 (UTC) (envelope-from yckuo@yckuo.org) Received: from home.yckuo.org (220-134-63-174.HINET-IP.hinet.net [220.134.63.174]) by mx1.FreeBSD.org (Postfix) with ESMTP id A4F6543D49 for ; Thu, 20 Apr 2006 01:54:29 +0000 (GMT) (envelope-from yckuo@yckuo.org) Received: from PM14G (wi-ap.yckuo.org [192.168.1.253]) by home.yckuo.org (Postfix) with SMTP id B62619942E for ; Thu, 20 Apr 2006 09:54:27 +0800 (CST) From: "Ying-Chih Kuo" To: Date: Thu, 20 Apr 2006 09:54:24 +0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="big5" Content-Transfer-Encoding: base64 X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1807 Subject: SMP Kernel Panic in 6.1-RC1 during mount root filesystem X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: yckuo@yckuo.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Apr 2006 01:54:30 -0000 SSBoYXZlIGEgcXVhZCB4ZW9uIHBpaWktNTUwIGJveCAsaXQgd29ya3Mgb24gNC4xMSBTTVAgdmVy eSB3ZWxsLg0Kbm93IGkgImluc3RhbGwiIChub3QgdXBncmFkZSkgIDYuMS1SQzEgaW4gdGhpcyBi b3guDQppdCB3b3JrcyB2ZXJ5IHdlbGwgd2l0aG91dCBTTVAsIGJ1dCB3aXRoIFNNUCAod2l0aCBv ciB3aXRoIGFjcGkpICxhbHdheXMgUEFOSUMsUEFOSUMsUEFOSUMuLi4NCg0KaSBnb3QgRERCIHN0 YWNrIHRyYWNlLCBodHRwOi8vd3d3LmZyZWVic2Qub3JnL2NnaS9xdWVyeS1wci5jZ2k/cHI9OTYw NDkNCg0KaGVscCBtZSxwbGVhc2UuDQoNCnlja3Vv From owner-freebsd-smp@FreeBSD.ORG Thu Apr 20 02:05:01 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A793C16A400 for ; Thu, 20 Apr 2006 02:05:01 +0000 (UTC) (envelope-from yckuo@yckuo.org) Received: from home.yckuo.org (220-134-63-174.HINET-IP.hinet.net [220.134.63.174]) by mx1.FreeBSD.org (Postfix) with ESMTP id 39B0F43D49 for ; Thu, 20 Apr 2006 02:05:01 +0000 (GMT) (envelope-from yckuo@yckuo.org) Received: from PM14G (wi-ap.yckuo.org [192.168.1.253]) by home.yckuo.org (Postfix) with SMTP id 3EA2A9942E for ; Thu, 20 Apr 2006 10:05:00 +0800 (CST) From: "Ying-Chih Kuo" To: Date: Thu, 20 Apr 2006 10:04:58 +0800 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="big5" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1807 Subject: SMP Kernel Panic in 6.1-RC1 during mount root filesystem X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: yckuo@yckuo.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Apr 2006 02:05:01 -0000 sorry for last mail ,it encoded. I have a quad xeon piii-550 box ,it works on 4.11 SMP very well. now i "install" (not upgrade) 6.1-RC1 in this box. it works very well without SMP, but with SMP (with or with acpi) ,always PANIC,PANIC,PANIC... i got DDB stack trace, http://www.freebsd.org/cgi/query-pr.cgi?pr=96049 help me,please. yckuo From owner-freebsd-smp@FreeBSD.ORG Fri Apr 21 00:08:49 2006 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CA01816A401 for ; Fri, 21 Apr 2006 00:08:49 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9F05F43D45 for ; Fri, 21 Apr 2006 00:08:49 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [10.251.19.131]) ([10.251.19.131]) by a50.ironport.com with ESMTP; 20 Apr 2006 16:45:22 -0700 Message-ID: <44481D12.1000207@elischer.org> Date: Thu, 20 Apr 2006 16:45:22 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: en-us, en MIME-Version: 1.0 To: smp@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: SMP on 4.11++ with > 4 cpus X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Apr 2006 00:08:49 -0000 Does this work? We have a Dual CPU dualcore with HTT that enumerates to 8 cpus. It hangs badly under heavy networking load under 4.x. Does anyone have an 8-way 4.x machine that works? We do have a small patch installed that helps it enumerate the CPUS. (so that we see all the cpus). From owner-freebsd-smp@FreeBSD.ORG Fri Apr 21 02:19:51 2006 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F33F16A400 for ; Fri, 21 Apr 2006 02:19:51 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id DE0D643D45 for ; Fri, 21 Apr 2006 02:19:50 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (zion.baldwin.cx [192.168.0.7]) (authenticated bits=0) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k3L2Jic5019186; Thu, 20 Apr 2006 22:19:44 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-smp@freebsd.org Date: Thu, 20 Apr 2006 22:19:42 -0400 User-Agent: KMail/1.8.3 References: <44481D12.1000207@elischer.org> In-Reply-To: <44481D12.1000207@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200604202219.43742.jhb@freebsd.org> X-Virus-Scanned: ClamAV 0.87.1/1411/Thu Apr 20 18:23:28 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: Julian Elischer Subject: Re: SMP on 4.11++ with > 4 cpus X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Apr 2006 02:19:51 -0000 On Thursday 20 April 2006 07:45 pm, Julian Elischer wrote: > Does this work? > > We have a Dual CPU dualcore with HTT that enumerates to 8 cpus. > It hangs badly under heavy networking load under 4.x. Hang as in deadlock or hang as in livelock? (Does ping still work for example)=20 > Does anyone have an 8-way 4.x machine that works? > > We do have a small patch installed that helps it enumerate the CPUS. > (so that we see all the cpus). What did you have to patch? It's supposed to enumerate all the HTT cpus out of the box. (And multicore should look like HTT, but possibly with 4 HTTs). =2D-=20 John Baldwin =A0<>< =A0http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" =A0=3D =A0http://www.FreeBSD.org