From owner-freebsd-amd64@FreeBSD.ORG Wed Sep 7 02:50:09 2005 Return-Path: X-Original-To: freebsd-amd64@hub.freebsd.org Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 06A0016A41F for ; Wed, 7 Sep 2005 02:50:09 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBB9443D46 for ; Wed, 7 Sep 2005 02:50:07 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id j872o76i005972 for ; Wed, 7 Sep 2005 02:50:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id j872o7EF005971; Wed, 7 Sep 2005 02:50:07 GMT (envelope-from gnats) Resent-Date: Wed, 7 Sep 2005 02:50:07 GMT Resent-Message-Id: <200509070250.j872o7EF005971@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, NAKATA@FreeBSD.org, Maho Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4718116A41F for ; Wed, 7 Sep 2005 02:40:06 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id EAD1843D46 for ; Wed, 7 Sep 2005 02:40:05 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id j872e5CR009549 for ; Wed, 7 Sep 2005 02:40:05 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id j872e5fk009548; Wed, 7 Sep 2005 02:40:05 GMT (envelope-from nobody) Message-Id: <200509070240.j872e5fk009548@www.freebsd.org> Date: Wed, 7 Sep 2005 02:40:05 GMT From: NAKATA@FreeBSD.org, Maho To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Cc: Subject: amd64/85820: 1.5 times slower performance with SCHED_ULE than SCHED_4BSD X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Sep 2005 02:50:09 -0000 >Number: 85820 >Category: amd64 >Synopsis: 1.5 times slower performance with SCHED_ULE than SCHED_4BSD >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Sep 07 02:50:07 GMT 2005 >Closed-Date: >Last-Modified: >Originator: NAKATA, Maho >Release: FreeBSD/amd64 5.4-RELEASE >Organization: FreeBSD.org >Environment: FreeBSD ligeti.private.org 5.4-RELEASE-p5 FreeBSD 5.4-RELEASE-p5 #0: Thu Jul 28 09:34:57 JST 2005 maho@ligeti.private.org:/usr/src/sys/amd64/compile/MAHO amd64 >Description: I have noticed (last year) that SCHED_ULE is slower than SCHED_4BSD, and raised a PR. At that time it was not convincing because 5.3-RELEASE/amd64 was not stable enough with large amount of memory, etc. My recent 5.4-RELEASE/amd64 is extremely stable even with large amount of memory and high load, and I can say something definite. Someone who is interested in my e-mail, please test it. I also prepared statically linked binaries to reproduce my test easily, because this test uses ATLAS (math/atlas-devel) which is a really really pain to build. my conclusion in short: SCHED_ULE is slower than SCHED_4BSD by 1.5 times in FreeBSD 5.4-RELEASE/amd64. this means both SCHED_4BSD and SCHED_ULE are definitely SMP aware, but SCHED_ULE scheduling is not efficient for very large jobs. Whereas 4BSD is almost optimal. my opetron box: o Tyan S2885 Tiger K8W o Opteron 242x2 (1.6GHzx2) o Transcend 2Gx8 (total 16G) How to repeat: fetch http://people.freebsd.org/~maho/scheduler_amd64.tar.gz tar xvfz scheduler_amd64.tar.gz cd scheduler_amd64/ sysctl kern.sched.name ; /usr/bin/time ./xdinvtst_pt -N 7000 9000 200 My results: o 4BSD sysctl kern.sched.name ; /usr/bin/time ./xdinvtst_pt -N 7000 9000 200 kern.sched.name: 4BSD NREPS ORDER UPLO N LDA TIME MFLOP RESID ===== ===== ===== ===== ===== ======== ======== ============ 0 Col GE 7000 7000 157.028 4368.50 2.110725e-02 0 Col GE 7200 7200 168.527 4429.39 2.106386e-02 0 Col GE 7400 7400 185.014 4380.32 2.099199e-02 0 Col GE 7600 7600 198.622 4420.07 2.073756e-02 0 Col GE 7800 7800 214.284 4429.04 2.089531e-02 0 Col GE 8000 8000 232.126 4411.27 2.142018e-02 0 Col GE 8200 8200 255.809 4310.65 2.041516e-02 0 Col GE 8400 8400 265.088 4471.62 2.092699e-02 0 Col GE 8600 8600 285.403 4457.11 2.119786e-02 0 Col GE 8800 8800 306.969 4439.88 2.257722e-02 0 Col GE 9000 9000 324.456 4493.54 2.347010e-02 11 cases: 11 passed, 0 skipped, 0 failed 4707.78 real 9019.12 user 38.34 sys o ULE sysctl kern.sched.name ; /usr/bin/time ./xdinvtst_pt -N 7000 9000 200 kern.sched.name: ule NREPS ORDER UPLO N LDA TIME MFLOP RESID ===== ===== ===== ===== ===== ======== ======== ============ 0 Col GE 7000 7000 284.579 2410.49 2.110725e-02 0 Col GE 7200 7200 176.769 4222.87 2.106386e-02 0 Col GE 7400 7400 183.035 4427.67 2.099199e-02 0 Col GE 7600 7600 195.830 4483.10 2.073756e-02 0 Col GE 7800 7800 228.077 4161.20 2.089531e-02 0 Col GE 8000 8000 267.382 3829.61 2.142018e-02 0 Col GE 8200 8200 247.578 4453.95 2.041516e-02 0 Col GE 8400 8400 261.590 4531.42 2.092699e-02 0 Col GE 8600 8600 308.443 4124.18 2.119786e-02 0 Col GE 8800 8800 331.672 4109.20 2.257722e-02 0 Col GE 9000 9000 320.790 4544.91 2.347010e-02 11 cases: 11 passed, 0 skipped, 0 failed 6964.19 real 8720.26 user 34.31 sys o What are my test doing? what is xdinvtst_pt? this program calculates inversion of randomly genrated matrices (double precision). _pt means pthread, and this creates two threads at a time to calculate the inversion of the matrix. We performed calculation from 7000x7000 matrix to 9000x9000 gradually increasing row and column by 200. how to make xdivntst_pt? build math/atlas-devel with smp machine. this port knows # of processors installed. You will build atlas after a very very long time; 1.5 day or so after typing make many times (10-20 times!) since this port is fragile. Then go down the work directory and manually fix some makefiles to point Fortran BLAS/LAPACK (via math/lapack) and can build by yourself. so this is why i included in archive and prepared statically linked binaries. o Perfomance of Opteron and effect of SMP Theoretical peak of calculation in double precision using SSE2 for 1.6GHz opteron is 3.2GFlops, so 6.4GFlops for dual processors. Performance of the largest test (inversion of 9000x9000 matrix in double precision) is about 4.5Gflops. namely 70% of theoretical peak. this is very good. From my experience, best experimental perfomance in single processor is ~80% such achivement might be found at much more primitive calculations. o ULE vs 4BSD please see this row: 4BSD 0 Col GE 9000 9000 324.456 4493.54 2.347010e-02 ule 0 Col GE 9000 9000 320.790 4544.91 2.347010e-02 these line shows 4.5Flops performance by inverting matrix. 324 seconds have passed by 4BSD and 320 seconds have passed by ule. This doesn't mean what I say was wrong; ~320 seconds have passed by both processors. namely ~160 seconds by one processor and ~160 seconds by another processor, then atlas measure as ~320 seconds have passed as total and this is the best case. We definitely need at least 320 seconds to invert the matrix and how actual time has passed is not measured in this context. With ULE, for example ~240 have passed in one processor, and ~80s in another processor. so we *must* wait for 240 seconds, while with 4BSD, we only wait for 160 seconds. we can know from actual difference between ULE and 4BSD by /usr/bin/time 4BSD 4707.78 real 9019.12 user 38.34 sys ULE 6964.19 real 8720.26 user 34.31 sys and 6964.19/4707.78=1.479. ~9000 seconds have passed by 4BSD and 8700s by ULE. and real is ~4700 seconds for 4BSD and ~7000s by ULE. so time consumed by actual works are both same (~9000s and ~8700s). but scheduling is not efficinet for this calculation and so, ULE needs more time. o Scheduling threads / processes? scheduling threads and processes can be different. but other experiments show that if we run same process at a time, ULE is also ~1.5 times slower than 4BSD. o conclusion 4BSD is near the optimal for large calculations and ULE is ~1.5 times slower than 4BSD. Both scheduling algorithm smp aware. I don't think ULE as default is good choice. >How-To-Repeat: described in Full Description section >Fix: none. >Release-Note: >Audit-Trail: >Unformatted: