From owner-freebsd-performance@FreeBSD.ORG Fri Apr 6 12:33:35 2012 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25065106566B; Fri, 6 Apr 2012 12:33:34 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 7D4878FC08; Fri, 6 Apr 2012 12:33:34 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1SG8M9-0008BQ-NB>; Fri, 06 Apr 2012 14:33:33 +0200 Received: from e178024241.adsl.alicedsl.de ([85.178.24.241] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1SG8M9-00072d-Ge>; Fri, 06 Apr 2012 14:33:33 +0200 Message-ID: <4F7EE297.2000402@zedat.fu-berlin.de> Date: Fri, 06 Apr 2012 14:33:27 +0200 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.3) Gecko/20120314 Thunderbird/10.0.3 MIME-Version: 1.0 To: Arnaud Lacombe References: In-Reply-To: X-Enigmail-Version: 1.4 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig0453718867B7A11D2983DCD6" X-Originating-IP: 85.178.24.241 Cc: freebsd-performance@freebsd.org, FreeBSD Current Subject: Re: Scheduler + IPC performance on FreeBSD 7.4, 8.2, 9.0 and -CURRENT X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2012 12:33:35 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig0453718867B7A11D2983DCD6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Am 04/05/12 20:03, schrieb Arnaud Lacombe: > Hi folks, >=20 > Over the past months, I ran on a couple of unused box the > `hackbench'[HACKBENCH] benchmark used by the Linux folks for tracking > down various kind of regression/improvement. `hackbench' is a > scheduler + IPC test (socket xor pipe). It creates producers/consumers > groups and let a variable quantity of small messages flow happily. > Producers and consumers are either processes xor threads. >=20 > Tested platforms were > - Atom D510, Intel, (incomplete) > - Core 2 Quad Q9560, Intel > - Soekris net5501, AMD (incomplete) > - Xeon E5645, Intel (incomplete) > - Xeon E5620 (dual package), Intel > - Xeon E5-1650 (pending completion) > - Vortex86, DMP >=20 > Tested kernel were: > - FreeBSD 7.4-RELEASE > - FreeBSD 8.2-RELEASE > - FreeBSD 9.0-RC3 and FreeBSD 9.0-RELEASE > - FreeBSD 10-CURRENT as of r231573 >=20 > on the following architecture: > - amd64 (if supported, incomplete) > - i386 >=20 > 1) DISCLAMER >=20 > Let me start by pointing out something important: >=20 > [I] "I am _not_ interested in testing released version FOO with > feature BAR enabled, if enabling BAR require a kernel rebuild." >=20 > All tests for release kernels were made for the kernel shipped > officially, it is the developers responsability to decide whether or > not enable a feature by default, not mine. If option BAR gives a 20% > performance improvement, enable it, don't complain the test to be 20% > slower. >=20 > [II] "I am _not_ interested in altering any hints, tunables or > sysctl, unless they prevent the execution of the test." >=20 > The exception added to the above rule is due to limitation introduced > by `kern.threads.max_threads_per_proc' and `kern.ipc.maxpipekva'. > Those were respectively set to 8192 and between 16M/64M. >=20 > note: rule [I] is alleviated for -CURRENT kernels, which were built > with the same alteration made to GENERIC during the CURRENT->RELEASE > transition (ie. WITNESS and a couple of other option disabled). >=20 >=20 > 2) Tests description >=20 > `hackbench' has the following tunable: > - IPC to use for messaging, either `pipe' or `socket'. > - Threading model, either `thread' or `process' > - Number of iteration to run > - Number of group to create >=20 > The tests covered all of these adjustments more or less heavily > depending on the platform capability. >=20 >=20 > 3) Scripts >=20 > Test scripts are available in the `master' branch of the git repository= at: >=20 > https://github.com/lacombar/hackbench >=20 > in the `hackbench/' directory. >=20 >=20 > 4) Results >=20 > Full results are available in the `runs/*' branches of the GitHub repos= itory. >=20 >=20 > 5) Quick results summary >=20 > * UP case >=20 > FreeBSD 9.0 behaves better than FreeBSD 8.2 in process mode, > especially with sockets. Results are comparable with thread. 9.0-RC3 > shows a 10% hits in thread/socket mode on the LX800, this will need > confirmation. >=20 > Linux is stable and scales linearly in all situation. It is only > beaten by FreeBSD 8.2-RELEASE with thread/socket. >=20 > * MP case >=20 > These is a pretty bad regression with FreeBSD 9.0 in thread/pipe mode, > which scale almost in O(N^2), ending up in way worse performance than > FreeBSD 7.4 or 8.2 on the Core 2 Quad. Beside that, it is really > difficult to draw a general trends, ie. whether FreeBSD 9.0 behaves > better than FreeBSD 8.2, or the other way around. Pretty much all > situation arises, FreeBSD 9.0 can beat FreeBSD 8.2 on some workload, > behave the same, or be beaten on others. None really scales regularly > either. Pretty much every runs shows thresholds where scheduling > decision change and/or became erratic. >=20 > 6) Anticipated question and remarks >=20 > Q1: "You should truly enable kick-ass feature BAZ in the kernel." > R1: "I'm lazy. Do your job as a developer to integrate the feature. If > it should be the default, make it the default." >=20 > Q2: "You should set `kern.vm.whatever' to 42, or enable feature BAZ in > the kernel, to get full performance from the Warp engine on > Constellation-class starship." > R2: "Would you ask Lt. Worf to re-aligh plasma injectors or would ask > Lt. Commander La Forge to plan an assault, seriously ?" >=20 > Q3: "You built the Linux kernel, why can't you rebuild FreeBSD's ?" > For a couple of reason: >=20 > - the Linux kernel does not provide binary release per-se. >=20 > - the Linux kernel was not the focus of the tests, but merely a > comparative of what others-can-do. >=20 > - I did not tweak the Linux kernel configuration. The kernels > configuration tested derived from the `defconfig', with very few > amendment[0], mostly about hardware support not enabled by default >=20 > Q3: "Could you post all the graph ?" > R3: I could, but there is really tons of them, so posting a subset of > them would be subjective, all the materials is available on the git > repository. >=20 > Q4: "So, how can I get all the graph ?" > R4: All you need is git, a posix shell, a couple of utility (find, > sort, ...), a recent gnuplot, and a ruby interpreter. >=20 > Comments and suggestions will be greatly appreciated. >=20 > - Arnaud >=20 > [HACKBENCH]: http://people.redhat.com/mingo/cfs-scheduler/tools/hackben= ch.c >=20 > [0]: the exact list is: >=20 > # CONFIG_KERNEL_GZIP is not set > CONFIG_KERNEL_XZ=3Dy > CONFIG_IKCONFIG=3Dy > CONFIG_IKCONFIG_PROC=3Dy > # CONFIG_MODULES is not set > CONFIG_X86_BIGSMP=3Dy > CONFIG_NR_CPUS=3D32 > CONFIG_PATA_IT8213=3Dy > CONFIG_PATA_IT821X=3Dy > CONFIG_IGB=3Dy > CONFIG_IGBVF=3Dy > CONFIG_IXGB=3Dy > CONFIG_IXGBE=3Dy > CONFIG_IXGBEVF=3Dy > # CONFIG_EXT3_FS is not set > CONFIG_EXT4_FS=3Dy Thank you very much for this work! --------------enig0453718867B7A11D2983DCD6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQEcBAEBAgAGBQJPfuKdAAoJEOgBcD7A/5N8qTEIAMduemjFoPHfNvfrEpF5pE0w sdKXOtKm59QEQ58NdJlR3s5AzY147tmhgm8ESSczc7LnFFSze530dlKxcMorSVgC zbQT3CbTtdi5ZlDsroX9qRcKT2gVYGgnFMSnN9HLF55YjveUHr1xgfyzEWLRB0LC oqVQKrnP5md2M+mhVAKI6WdQU8yvs+odNN/Wd5igG/G5IppNneeJDli4+00sm3Np Pyx/SFasxRhQPhHbnpanHU2KZb2g4VvHeMRFV1daJ1e5z/h9qu3SpZcaNMf0BhsS L1T9DGjDso88pE+MSkNdccT8he5Mlatj1ML66m6//D6CicVH1svvnfI0rnDPlig= =z50+ -----END PGP SIGNATURE----- --------------enig0453718867B7A11D2983DCD6--