From owner-freebsd-performance@FreeBSD.ORG Mon Nov 4 15:49:10 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EFEBB7C3 for ; Mon, 4 Nov 2013 15:49:09 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C16202E6D for ; Mon, 4 Nov 2013 15:49:09 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.7/8.14.7) with ESMTP id rA4Fn8LC007887 for ; Mon, 4 Nov 2013 07:49:08 -0800 (PST) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.7/8.14.7/Submit) id rA4Fn8iq007886 for freebsd-performance@freebsd.org; Mon, 4 Nov 2013 07:49:08 -0800 (PST) (envelope-from david) Date: Mon, 4 Nov 2013 07:49:08 -0800 From: David Wolfskill To: freebsd-performance@freebsd.org Subject: Reality check? compat.ia32.maxdsiz in jailed "32-bit" environment Message-ID: <20131104154908.GD1711@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pQhZXvAqiZgbeUkD" Content-Disposition: inline User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Nov 2013 15:49:10 -0000 --pQhZXvAqiZgbeUkD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I am supporting a performance-sensitive software development environment that: * Is presently contrained to operate in a FreeBSD/i386 (32-bit) environment. * Primarily uses svn, gcc (with various target architectures), and bmake. * Has more main memory than FreeBSD appears to be able to make constructive use of for this workload, but not enough to just put the entire set of files involved in a malloc-based memory disk. Our current implementation involves running the processes involved in a "full system image" jail on FreeBSD/amd64 8.x hosts.(We presently =20 have but one jail on each host; the developers are, in general, only =20 permitted to login to the jailed environment, not the host itself. Several develeopers are usually logged in for each jail, and concurrent builds are not uncommon.) My "test" environment (which I intend to provide for deployment ... soon) is similar, but switches the host to FreeBSD/amd64 9.2-S while leaving the jailed image unchanged; the performance is about 18% better (using elapsed time of the software build as the salient metric), and system CPU is significantly reduced as well (which is a bug issue when we have multiple such builds running concurrently). I have (also) started experimenting with increasing compat.ia32.maxdsiz beyond its default of 512MB: Initially, I kicked it to 2GB; more recently, I tried setting it to 3GB. While I have not noticed any negative issues from either of the above non-default settings, I have done but limited testing with concurrent software builds in my test environment. Our swap usage is neglible, and given that there is a more-than-adequate amount of RAM on the machines, I wouldn't expect that increasing the compat.ia32.maxdsiz value to be a constraint even with concurrent builds (as they are in separate processes, and thus, separate address spaces), but one of the reasons for this note is to ask for a reality check on that perception. Oddly enough, I have not noted any statistically significant performance change from setting compat.ia32.maxdsiz. On the other hand, I have noticed one significant behavioral change: some svn merge operations that had been failing (for alck of available memory) with the default (512MB) compat.ia32.maxdsiz no longer fail when it is set to 2GB. (I have not actually tested that for 3GB, though I expect that it would also work.) Does this make sense to anyone else? (I have a vague memory of gcc altering its behavior to take advantage of additional memory if it finds "enough" memory available -- but it's been a couple of decades since I poked around in the internals of gcc, and that's not actually an experience I'd prefer to repeat.) Thanks for any insights you're willing to share! Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --pQhZXvAqiZgbeUkD Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlJ3wfMACgkQmprOCmdXAD2kpgCfSA5MZi39wdrgDIZMKVYrMtFR x6AAniCLGuZufztsoZ5JE5k3ilHgFyVZ =1hjR -----END PGP SIGNATURE----- --pQhZXvAqiZgbeUkD-- From owner-freebsd-performance@FreeBSD.ORG Mon Nov 4 16:23:37 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CC6AA592 for ; Mon, 4 Nov 2013 16:23:37 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-lb0-x235.google.com (mail-lb0-x235.google.com [IPv6:2a00:1450:4010:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4CED42104 for ; Mon, 4 Nov 2013 16:23:37 +0000 (UTC) Received: by mail-lb0-f181.google.com with SMTP id x18so5516979lbi.40 for ; Mon, 04 Nov 2013 08:23:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=wSfyAAwEYmkxvQjoTvSBj8AcO9W1k+PpUniHUtZnrV8=; b=d7EBti9bAxgCfXdEmtPzZp9b9Gu9f0q1MKYbeemDJDW7sOBKSFD3JuHQJ9d6t7813b Uyni/36RUI26+Z7s2lBb6q72J+/0DUNCfKxAb+G//ciNJQhQDse+xzs9AasNPglD0IRN Xf5q97ZnoxjYnGJa9PMSZZJeezetjCRnsqj7NTcMPW3E/qQW7n3xfEemixHL2VNBeU6g q17j/pABqaOnnYklchLQt2hNaXhPSQR6CnyRh9TFWzlX151pStWkbAatWDwg8HyMWBD5 qJuQG4HQbmsZT+0xStmKfiHXnon0NlNUnNNYGzFon/phJSrHNPZQ+PctMqGGnyYDC2On ZJmg== MIME-Version: 1.0 X-Received: by 10.152.28.194 with SMTP id d2mr12491046lah.2.1383582215054; Mon, 04 Nov 2013 08:23:35 -0800 (PST) Received: by 10.112.5.138 with HTTP; Mon, 4 Nov 2013 08:23:34 -0800 (PST) In-Reply-To: <20131104154908.GD1711@albert.catwhisker.org> References: <20131104154908.GD1711@albert.catwhisker.org> Date: Mon, 4 Nov 2013 16:23:34 +0000 Message-ID: Subject: Re: Reality check? compat.ia32.maxdsiz in jailed "32-bit" environment From: Tom Evans To: David Wolfskill Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Nov 2013 16:23:37 -0000 On Mon, Nov 4, 2013 at 3:49 PM, David Wolfskill wrot= e: > I am supporting a performance-sensitive software development > environment that: > =E2=80=A6 > I have (also) started experimenting with increasing compat.ia32.maxdsiz > beyond its default of 512MB: Initially, I kicked it to 2GB; more > recently, I tried setting it to 3GB. > > While I have not noticed any negative issues from either of the above > non-default settings, I have done but limited testing with concurrent > software builds in my test environment. IIRC, when running on pure i386, setting kern.maxdsiz limits the amount of kernel address space (or rather, dsize+KVA =3D 4 GB), and so you had to be careful when increasing kern.maxdsiz. This probably has no effect on a i386 on amd64 jail though. Cheers Tom From owner-freebsd-performance@FreeBSD.ORG Mon Nov 4 16:34:13 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4DA3C955 for ; Mon, 4 Nov 2013 16:34:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D022221AB for ; Mon, 4 Nov 2013 16:34:12 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id rA4GXwAd001851; Mon, 4 Nov 2013 18:33:58 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua rA4GXwAd001851 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id rA4GXwSf001850; Mon, 4 Nov 2013 18:33:58 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 4 Nov 2013 18:33:58 +0200 From: Konstantin Belousov To: David Wolfskill Subject: Re: Reality check? compat.ia32.maxdsiz in jailed "32-bit" environment Message-ID: <20131104163358.GR59496@kib.kiev.ua> References: <20131104154908.GD1711@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jsBaDDfOofhewO/R" Content-Disposition: inline In-Reply-To: <20131104154908.GD1711@albert.catwhisker.org> User-Agent: Mutt/1.5.22 (2013-10-16) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Nov 2013 16:34:13 -0000 --jsBaDDfOofhewO/R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 04, 2013 at 07:49:08AM -0800, David Wolfskill wrote: > I am supporting a performance-sensitive software development > environment that: > * Is presently contrained to operate in a FreeBSD/i386 (32-bit) > environment. > * Primarily uses svn, gcc (with various target architectures), and bmake. > * Has more main memory than FreeBSD appears to be able to make > constructive use of for this workload, but not enough to just put the > entire set of files involved in a malloc-based memory disk. >=20 > Our current implementation involves running the processes involved > in a "full system image" jail on FreeBSD/amd64 8.x hosts.(We presently = =20 > have but one jail on each host; the developers are, in general, only = =20 > permitted to login to the jailed environment, not the host itself. > Several develeopers are usually logged in for each jail, and concurrent > builds are not uncommon.) >=20 > My "test" environment (which I intend to provide for deployment ... > soon) is similar, but switches the host to FreeBSD/amd64 9.2-S while > leaving the jailed image unchanged; the performance is about 18% > better (using elapsed time of the software build as the salient > metric), and system CPU is significantly reduced as well (which is > a bug issue when we have multiple such builds running concurrently). >=20 > I have (also) started experimenting with increasing compat.ia32.maxdsiz > beyond its default of 512MB: Initially, I kicked it to 2GB; more > recently, I tried setting it to 3GB. >=20 > While I have not noticed any negative issues from either of the above > non-default settings, I have done but limited testing with concurrent > software builds in my test environment. Our swap usage is neglible, and > given that there is a more-than-adequate amount of RAM on the machines, > I wouldn't expect that increasing the compat.ia32.maxdsiz value to > be a constraint even with concurrent builds (as they are in separate > processes, and thus, separate address spaces), but one of the reasons > for this note is to ask for a reality check on that perception. >=20 > Oddly enough, I have not noted any statistically significant > performance change from setting compat.ia32.maxdsiz. On the other > hand, I have noticed one significant behavioral change: some svn > merge operations that had been failing (for alck of available memory) > with the default (512MB) compat.ia32.maxdsiz no longer fail when > it is set to 2GB. (I have not actually tested that for 3GB, though > I expect that it would also work.) Does this make sense to anyone > else? (I have a vague memory of gcc altering its behavior to take > advantage of additional memory if it finds "enough" memory available > -- but it's been a couple of decades since I poked around in the > internals of gcc, and that's not actually an experience I'd prefer > to repeat.) >=20 > Thanks for any insights you're willing to share! For 32 bit kernel, the user VA layout is empty - text - data - bss ->|<- mmap area, including dso -->|<- stack ->| 0 maxdsiz 3G-68M 3G-4M For the 64bit kernel executing 32bit process, the layout is the same, but the top of the user VA is at 4G, and bottom of stack is at 4G-64M. By increasing maxdsiz to 3G, you allow for the larger heap in exchange for the limiting the space for mmap(2) to use, and shared libraries to load. In your load, where toolchain binaries are linked statically, the 'dso' is put out of the scope for memory-hungry processes. This would not work on the 32bit kernel, because 3G is the total address space available to the user process. --jsBaDDfOofhewO/R Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSd8x1AAoJEJDCuSvBvK1BQ9YP/0sqWjn5H/iG7LUor1FdiyUi qrr6HuFUqrQtZko7I4gow9cmbaqC9tV3lTI9eXSleBbItH1Y3JW/jV4/1UDS5xpD OwlbD21jypvA5lzHT6y9SLTedv611qf6nhHTI1nyzfFHEQDXxRbJwk3Wv5Gg9IJY NgEuXSDw8LuuCUCBICiprlAywgiNd9jOI6JvIz2ALyVodw3vW6tatW/Q06Az1my0 p2Vh6y2ifyxg/HWNxetnMwhO4sKhCdxl1jywcwd+l4CKqTQJsfLYlWaOTz8resT7 DDPnfmLtSc9xwCVp43BxnWzXtN4/irjS9QP5GOZ63QGf3eOmEXp9K6WVwEtcYzYK GxP2ybHOGhh04yopmiwl3gmfLNNhSZ+41K/1LuDFrl/6TYQHSZ5FoYPOwSz+c84l AWzss0HRjEkue3vQZFNRiSsA+x8KhC/02Mf/6rO0F5/dKCGcJcpx275sHZhH4Lzu 0gj9c9f7opEPRX4H/5IXOqjsxacQJB9MBMmO5pkMAU7EwbsPVCuLdJzMfEyPQzxd tO/6jqzQ7l/WUgAC1STL2UEHgBY1l9oSDtUqUoF89vTFeRbLuA9LdKfUuCC66nKY g8uXV7QtqyRLDJsvYG1LTKOxtCi3wALqbdUrL8XwNQHK6VqYNEEjLcefCJogEyvX WyC0mG/fFqybbP0bxQ29 =0kx0 -----END PGP SIGNATURE----- --jsBaDDfOofhewO/R--