From owner-freebsd-arch@FreeBSD.ORG Fri Jun 24 02:47:41 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (unknown [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 814BB106564A for ; Fri, 24 Jun 2011 02:47:41 +0000 (UTC) (envelope-from peter.jeremy@alcatel-lucent.com) Received: from ihemail3.lucent.com (ihemail3.lucent.com [135.245.0.37]) by mx1.freebsd.org (Postfix) with ESMTP id 46C418FC08 for ; Fri, 24 Jun 2011 02:47:40 +0000 (UTC) Received: from usnavsmail1.ndc.alcatel-lucent.com (usnavsmail1.ndc.alcatel-lucent.com [135.3.39.9]) by ihemail3.lucent.com (8.13.8/IER-o) with ESMTP id p5O2UrAK016585 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 23 Jun 2011 21:30:53 -0500 (CDT) Received: from unixmail.au.alcatel-lucent.com (unixmail.au.alcatel-lucent.com [139.188.42.130]) by usnavsmail1.ndc.alcatel-lucent.com (8.14.3/8.14.3/GMO) with ESMTP id p5O2UmDs032310 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Thu, 23 Jun 2011 21:30:52 -0500 Received: from insmb.au.alcatel-lucent.com (insmb.au.alcatel-lucent.com [139.188.42.184]) by unixmail.au.alcatel-lucent.com (8.13.8+Sun/8.13.3) with ESMTP id p5O2UmIl010661 for ; Fri, 24 Jun 2011 12:30:48 +1000 (EST) Received: from pjdesk.au.alcatel-lucent.com (pjdesk.au.alcatel-lucent.com [139.188.2.2]) by insmb.au.alcatel-lucent.com (8.13.8+Sun/8.13.8) with ESMTP id p5O2Fi0S010683; Fri, 24 Jun 2011 12:15:44 +1000 (EST) X-Bogosity: Ham, spamicity=0.000000 Received: from pjdesk.au.alcatel-lucent.com (localhost [127.0.0.1]) by pjdesk.au.alcatel-lucent.com (8.14.4/8.14.4) with ESMTP id p5O2FcFU003690; Fri, 24 Jun 2011 12:15:38 +1000 (EST) (envelope-from peter.jeremy@alcatel-lucent.com) Received: (from pjeremy@localhost) by pjdesk.au.alcatel-lucent.com (8.14.4/8.14.4/Submit) id p5O2Fc04003689; Fri, 24 Jun 2011 12:15:38 +1000 (EST) (envelope-from peter.jeremy@alcatel-lucent.com) Date: Fri, 24 Jun 2011 12:15:37 +1000 From: Peter Jeremy To: freebsd-arch@freebsd.org Message-ID: <20110624021537.GD10304@pjdesk.au.alcatel-lucent.com> References: <20110623053051.GL65891@pjdesk.au.alcatel-lucent.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="sgneBHv3152wZ8jf" Content-Disposition: inline In-Reply-To: <20110623053051.GL65891@pjdesk.au.alcatel-lucent.com> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.57 on 135.245.2.37 X-Scanned-By: MIMEDefang 2.64 on 135.3.39.9 Subject: Context Switching Oddities X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jun 2011 02:47:41 -0000 --sgneBHv3152wZ8jf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I originally asked a variant of this in -sparc64 since the oddities initially seemed to be related to sparc64 but it was suggested that this might be a more appropriate list. I've also added more results. I have a tool that measures the rate at which a single-byte token can be passed between two processes via a socketpair - basically counting while (1) { read(fd[0], buf, 1); write(fd[1], "a", 1); } I was initially running it multiple copies of it on an otherwise idle V890 (16-CPU, 64GB RAM running -current from about a week ago), capturing 'vmstat -s' output. In the process, I have found several oddities. That prompted me to additionally test it on a V440 (4-CPU, 8GB RAM, same world as the V890), a SB1500 (1 CPU, 4GB RAM, -current =66rom a few days ago) and an Athlon-64 (dual-core, 8GB RAM, -stable amd64 from a few months ago). The SPARC systems are all using 4BSD and the amd64 is using ULE. I don't have access to any large x86 boxes to cross-check. 1) The number of context switches doesn't match my expectations. See http://i.imgur.com/28OHu.jpg (1 CPU) http://i.imgur.com/6YRh8.jpg (2 core) http://i.imgur.com/r0v7M.jpg (4 CPU) http://i.imgur.com/hkCA2.jpg (16 CPU) http://i.imgur.com/9Tt9Q.gif (combined) Based on one process writing a token to a second process requiring one context switch, I would expect the number of context switches to roughly match the green (based on token passing rate) or blue (based on syscall rate) lines. Instead, it's generally far too low, though the 4- and 16-CPU graphs start out unexpectedly high. 2) The transfer rate doesn't gradually tail off See http://i.imgur.com/YoDQ5.jpg (1 CPU) http://i.imgur.com/0Wl1Y.jpg (2 core) http://i.imgur.com/699zr.jpg (4 CPU) http://i.imgur.com/0ujRN.jpg (16 CPU) http://i.imgur.com/omxG1.gif (combined & scaled) I would expect a fairly flat peak from 1 to about n-CPU pairs (since there are that many execution threads available) that then tailed off as scheduler overheads increased. Instead the 4- & 16-CPU tests show a dip initially then rising to a peak before tailing off. Each graph shows both the rate reported by the program and the rate estimated =66rom the syscall rate. Can anyone offer an explanation for this behaviour? --=20 Peter Jeremy --sgneBHv3152wZ8jf Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iEYEARECAAYFAk4D80kACgkQ/opHv/APuIc38wCfYhu4ewvyV4EETy00ObKPH7iV 2+EAnRi99cQ0oDhkBDE/rWrnJZCPwRgl =rZo+ -----END PGP SIGNATURE----- --sgneBHv3152wZ8jf--