From owner-freebsd-performance@FreeBSD.ORG Sun May 7 00:29:15 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id A8BAB16A407; Sun, 7 May 2006 00:29:14 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Sun, 7 May 2006 08:29:10 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060506214838.Q46997@fledge.watson.org> <445D25BA.9020508@fsn.hu> In-Reply-To: <445D25BA.9020508@fsn.hu> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605070829.11195.davidxu@freebsd.org> Cc: Attila Nagy , Robert Watson , current@freebsd.org, performance@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 00:29:16 -0000 On Sunday 07 May 2006 06:39, Attila Nagy wrote: > On 2006. 05. 06. 22:50, Robert Watson wrote: > >> The machine is a quad core Xeon LV server, the client side is > >> sysbench, accessing mysql 4.1.8 on a socket. Heap table, simple test. > > > > Which threading library is that with, btw? If libpthread, could you run > > the same test with libthr, and vice versa? > > thr and with dynamically linked mysql, because when I link it with > -static, it dies with sig11 at startup. > Our static thread library build is broken, my standpoint is weak symbol overriding never works correctly, if you have any idea, we can discuss it on arch@, we of course will be at disadvantageous position if it won't be fixed. > You can find the updated picture at the previous link. David Xu From owner-freebsd-performance@FreeBSD.ORG Sun May 7 00:29:15 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id A8BAB16A407; Sun, 7 May 2006 00:29:14 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Sun, 7 May 2006 08:29:10 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060506214838.Q46997@fledge.watson.org> <445D25BA.9020508@fsn.hu> In-Reply-To: <445D25BA.9020508@fsn.hu> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605070829.11195.davidxu@freebsd.org> Cc: Attila Nagy , Robert Watson , current@freebsd.org, performance@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 00:29:16 -0000 On Sunday 07 May 2006 06:39, Attila Nagy wrote: > On 2006. 05. 06. 22:50, Robert Watson wrote: > >> The machine is a quad core Xeon LV server, the client side is > >> sysbench, accessing mysql 4.1.8 on a socket. Heap table, simple test. > > > > Which threading library is that with, btw? If libpthread, could you run > > the same test with libthr, and vice versa? > > thr and with dynamically linked mysql, because when I link it with > -static, it dies with sig11 at startup. > Our static thread library build is broken, my standpoint is weak symbol overriding never works correctly, if you have any idea, we can discuss it on arch@, we of course will be at disadvantageous position if it won't be fixed. > You can find the updated picture at the previous link. David Xu From owner-freebsd-performance@FreeBSD.ORG Sun May 7 08:07:46 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 5A36416A400; Sun, 7 May 2006 08:07:14 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Sun, 7 May 2006 16:07:10 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> In-Reply-To: <20060506150622.C17611@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605071607.10869.davidxu@freebsd.org> Cc: performance@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 08:07:47 -0000 On Saturday 06 May 2006 22:16, Robert Watson wrote: > Dear all, > > Attached, please find a patch implementing more fine-grained locking for > the POSIX local socket subsystem (UNIX domain socket subsystem). In the > current implementation, we use a single global subsystem lock to protect > all local IPC over the PF_LOCAL socket type. This has low overhead, but > can result in significant contention, especially for workloads like MySQL > which push a great deal of data through UNIX domain sockets, and involve > many parties. The hope is that by improving granularity, we can reduce > contention sufficiently to overcome the added cost of increased locking > overhead (a side-effect of greater granularity). At a high level, here are > the changes made: I have tested the patch on my dual PIII machine, the test is super-smack's select-key.smack, as you said, performance is improved. it seems uidinfo lock is one of the bottlenecks in my test, it might be caused by chgsbsize() because it seems each socket I/O has to execute the code though they don't have any relations. Note unlike kris, I don't have local changes. max total count avg cnt_hold cnt_lock name 2913 1110657 1200300 0 25690 34374 kern/kern_resource.c:1172 (sleep mtxpool) 5603 7259406 204321 35 20899 20075 kern/kern_descrip.c:378 (Giant) 6987 1817410 1369257 1 10739 7324 kern/kern_descrip.c:1990 (filedesc structure) 3713 3771857 120053 31 4553 4612 kern/uipc_usrreq.c:617 (unp_global_mtx) 7339 1903685 1574656 1 3389 3710 kern/kern_descrip.c:2145 (sleep mtxpool) 91334 1798700 1369257 1 3227 7916 kern/kern_descrip.c:2011 (filedesc structure) 4764 223419 204440 1 2549 1693 kern/kern_descrip.c:385 (filedesc structure) 410546 2002932 1369257 1 2238 4103 kern/kern_descrip.c:2010 (sleep mtxpool) 5064 248004 169944 1 1152 1777 kern/kern_sig.c:1002 (process lock) 39 149033 74420 2 760 866 kern/kern_synch.c:220 (process lock) 5567 209654 204321 1 691 1566 kern/kern_descrip.c:438 (filedesc structure) 70 386915 63807 6 527 412 kern/subr_sleepqueue.c:374 (process lock) 4358 486842 66802 7 463 291 kern/vfs_bio.c:2424 (vnode interlock) 6430 488186 214393 2 347 420 kern/kern_lock.c:163 (lockbuilder mtxpool) 3057 251010 68159 3 313 2290 kern/vfs_vnops.c:796 (vnode interlock) 13126 15731880 1080092 14 294 227 kern/uipc_usrreq.c:581 (so_snd) 3161 67316 66402 1 293 267 kern/vfs_bio.c:1464 (vnode interlock) 3395 205078 204321 1 270 447 kern/kern_descrip.c:433 (sleep mtxpool) 3011 97692 2323 42 213 239 kern/kern_synch.c:218 (Giant) 71 9933 3721 2 185 185 i386/i386/pmap.c:2235 (vm page queue mutex) 14 3454 3512 0 121 155 vm/vm_fault.c:909 (vm page queue mutex) 3700 2120096 120046 17 119 22 kern/uipc_usrreq.c:705 (so_rcv) 9 2379 2024 1 103 121 vm/vm_fault.c:346 (vm page queue mutex) 9389 131186 120070 1 94 1 kern/uipc_socket.c:1101 (so_snd) 4 2400 3016 0 85 4385 vm/vm_fault.c:851 (vm page queue mutex) 99 6403 596 10 84 50 i386/i386/pmap.c:2649 (vm page queue mutex) 5109 201972 204330 0 77 306 kern/sys_socket.c:176 (so_snd) 1892 47770 2030 23 66 15 vm/vm_fault.c:686 (vm object) 3380 360770 66716 5 61 117 kern/vfs_bio.c:1445 (buf queue lock) 7 1597 1380 1 53 90 vm/vm_fault.c:136 (vm page queue mutex) 347 89307 8267 10 50 330 kern/kern_timeout.c:240 (Giant) 14 10199 12448 0 38 100 kern/kern_sx.c:157 (lockbuilder mtxpool) 15 10421 3046 3 37 32 vm/vm_fault.c:1009 (vm page queue mutex) 3704 2757298 120046 22 31 0 kern/uipc_usrreq.c:696 (unp_mtx) 27 1519 510 2 27 21 vm/vm_object.c:637 (vm page queue mutex) 29 7651 5374 1 26 44 geom/geom_io.c:68 (bio queue) 4517 25540 6495 3 23 250 kern/kern_umtx.c:194 (umtxq_lock) 39 1910 400 4 20 9 i386/i386/pmap.c:2126 (vm page queue mutex) 3 669 648 1 19 24 vm/vm_object.c:1551 (vm page queue mutex) 13346 24581 2592 9 18 36 vm/vnode_pager.c:1164 (vm object) 293 24130 3559 6 17 37 vm/vm_fault.c:994 (vm object) 11121 73890 994 74 17 8 kern/kern_intr.c:661 (Giant) 396 6808 141 48 17 4 vm/vm_map.c:1427 (vm page queue mutex) 2100 64657 66580 0 16 46 kern/vfs_bio.c:357 (needsbuffer lock) 4082 304716 66778 4 12 92 sys/buf.h:296 (lockbuilder mtxpool) 491 10312 12453 0 12 21 kern/kern_sx.c:245 (lockbuilder mtxpool) 86 1726 210 8 11 7 i386/i386/pmap.c:1966 (vm page queue mutex) 809 60435 66602 0 11 0 kern/vfs_bio.c:315 (needsbuffer lock) From owner-freebsd-performance@FreeBSD.ORG Sun May 7 08:07:46 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 5A36416A400; Sun, 7 May 2006 08:07:14 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Sun, 7 May 2006 16:07:10 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> In-Reply-To: <20060506150622.C17611@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605071607.10869.davidxu@freebsd.org> Cc: performance@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 08:07:47 -0000 On Saturday 06 May 2006 22:16, Robert Watson wrote: > Dear all, > > Attached, please find a patch implementing more fine-grained locking for > the POSIX local socket subsystem (UNIX domain socket subsystem). In the > current implementation, we use a single global subsystem lock to protect > all local IPC over the PF_LOCAL socket type. This has low overhead, but > can result in significant contention, especially for workloads like MySQL > which push a great deal of data through UNIX domain sockets, and involve > many parties. The hope is that by improving granularity, we can reduce > contention sufficiently to overcome the added cost of increased locking > overhead (a side-effect of greater granularity). At a high level, here are > the changes made: I have tested the patch on my dual PIII machine, the test is super-smack's select-key.smack, as you said, performance is improved. it seems uidinfo lock is one of the bottlenecks in my test, it might be caused by chgsbsize() because it seems each socket I/O has to execute the code though they don't have any relations. Note unlike kris, I don't have local changes. max total count avg cnt_hold cnt_lock name 2913 1110657 1200300 0 25690 34374 kern/kern_resource.c:1172 (sleep mtxpool) 5603 7259406 204321 35 20899 20075 kern/kern_descrip.c:378 (Giant) 6987 1817410 1369257 1 10739 7324 kern/kern_descrip.c:1990 (filedesc structure) 3713 3771857 120053 31 4553 4612 kern/uipc_usrreq.c:617 (unp_global_mtx) 7339 1903685 1574656 1 3389 3710 kern/kern_descrip.c:2145 (sleep mtxpool) 91334 1798700 1369257 1 3227 7916 kern/kern_descrip.c:2011 (filedesc structure) 4764 223419 204440 1 2549 1693 kern/kern_descrip.c:385 (filedesc structure) 410546 2002932 1369257 1 2238 4103 kern/kern_descrip.c:2010 (sleep mtxpool) 5064 248004 169944 1 1152 1777 kern/kern_sig.c:1002 (process lock) 39 149033 74420 2 760 866 kern/kern_synch.c:220 (process lock) 5567 209654 204321 1 691 1566 kern/kern_descrip.c:438 (filedesc structure) 70 386915 63807 6 527 412 kern/subr_sleepqueue.c:374 (process lock) 4358 486842 66802 7 463 291 kern/vfs_bio.c:2424 (vnode interlock) 6430 488186 214393 2 347 420 kern/kern_lock.c:163 (lockbuilder mtxpool) 3057 251010 68159 3 313 2290 kern/vfs_vnops.c:796 (vnode interlock) 13126 15731880 1080092 14 294 227 kern/uipc_usrreq.c:581 (so_snd) 3161 67316 66402 1 293 267 kern/vfs_bio.c:1464 (vnode interlock) 3395 205078 204321 1 270 447 kern/kern_descrip.c:433 (sleep mtxpool) 3011 97692 2323 42 213 239 kern/kern_synch.c:218 (Giant) 71 9933 3721 2 185 185 i386/i386/pmap.c:2235 (vm page queue mutex) 14 3454 3512 0 121 155 vm/vm_fault.c:909 (vm page queue mutex) 3700 2120096 120046 17 119 22 kern/uipc_usrreq.c:705 (so_rcv) 9 2379 2024 1 103 121 vm/vm_fault.c:346 (vm page queue mutex) 9389 131186 120070 1 94 1 kern/uipc_socket.c:1101 (so_snd) 4 2400 3016 0 85 4385 vm/vm_fault.c:851 (vm page queue mutex) 99 6403 596 10 84 50 i386/i386/pmap.c:2649 (vm page queue mutex) 5109 201972 204330 0 77 306 kern/sys_socket.c:176 (so_snd) 1892 47770 2030 23 66 15 vm/vm_fault.c:686 (vm object) 3380 360770 66716 5 61 117 kern/vfs_bio.c:1445 (buf queue lock) 7 1597 1380 1 53 90 vm/vm_fault.c:136 (vm page queue mutex) 347 89307 8267 10 50 330 kern/kern_timeout.c:240 (Giant) 14 10199 12448 0 38 100 kern/kern_sx.c:157 (lockbuilder mtxpool) 15 10421 3046 3 37 32 vm/vm_fault.c:1009 (vm page queue mutex) 3704 2757298 120046 22 31 0 kern/uipc_usrreq.c:696 (unp_mtx) 27 1519 510 2 27 21 vm/vm_object.c:637 (vm page queue mutex) 29 7651 5374 1 26 44 geom/geom_io.c:68 (bio queue) 4517 25540 6495 3 23 250 kern/kern_umtx.c:194 (umtxq_lock) 39 1910 400 4 20 9 i386/i386/pmap.c:2126 (vm page queue mutex) 3 669 648 1 19 24 vm/vm_object.c:1551 (vm page queue mutex) 13346 24581 2592 9 18 36 vm/vnode_pager.c:1164 (vm object) 293 24130 3559 6 17 37 vm/vm_fault.c:994 (vm object) 11121 73890 994 74 17 8 kern/kern_intr.c:661 (Giant) 396 6808 141 48 17 4 vm/vm_map.c:1427 (vm page queue mutex) 2100 64657 66580 0 16 46 kern/vfs_bio.c:357 (needsbuffer lock) 4082 304716 66778 4 12 92 sys/buf.h:296 (lockbuilder mtxpool) 491 10312 12453 0 12 21 kern/kern_sx.c:245 (lockbuilder mtxpool) 86 1726 210 8 11 7 i386/i386/pmap.c:1966 (vm page queue mutex) 809 60435 66602 0 11 0 kern/vfs_bio.c:315 (needsbuffer lock) From owner-freebsd-performance@FreeBSD.ORG Sun May 7 16:50:08 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E640816A400; Sun, 7 May 2006 16:50:07 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx1.starman.ee (smtp-out1.starman.ee [85.253.0.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 179E143D49; Sun, 7 May 2006 16:50:06 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx1.starman.ee (Postfix) with ESMTP id 839C123C157; Sun, 7 May 2006 19:50:03 +0300 (EEST) From: Sven Petai To: freebsd-performance@freebsd.org Date: Sun, 7 May 2006 19:49:53 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> In-Reply-To: <20060506150622.C17611@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605071949.54978.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx1.starman.ee Cc: performance@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 16:50:08 -0000 hi I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in total). In general with 10 parallel smacker threads the performance seems to go up with your patch by ~44% and with 100 parallel threads it goes down ~25% Here's a graph of select smack performance with your patch: http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/ULE_+_pthread_vs._thr.png Server crashed before I managed to get enough data to plot graphs comparing pre-patch and patched performance. I'll try to generate that and another one, comparing the performance against linux on the same hardware, tomorrow. But below you can find some pre and after patch data that should give some idea of how the patch has impacted performance. Various test data, benchmarking scripts I used etc. can be found @ http://bsd.ee/~hadara/debug/mysql4/ I have included nice value of mysqld in the tests, because I have found that the impact of giving mysqld higher priority can have dramatic effects, especially under 6.X. This happens probably because super-smack with its 100 procs would otherwise get scheduling advantage over mysqld with 100 threads. Hardware: motherboard: Thunder K8QSD Pro hdd: scsi seagate cheetah 10K7 ram: 8 * 3200 CL3 kingston ECC 1G cpu: 4 * opteron 870 (2Ghz dualcore) Environment: mysql: ver: 4.1.18_2 from the ports table type: MyISAM config: my-huge.cnf with disabled bin-log and max-connections bumped to 400 database located on a slice mounted with softupdates super-smack: ver: from the ports smacks: default select and update All tests were run 2 times and mean value was used, unless stated otherwise. ==== FreeBSD current (06.05.2006) ==== scheduler: 4BSD thr_lib nice queries threads update select pthread 0 10000 100 3280 8253 pthread 0 100000 10 5037 19972 thr 0 10000 100 4470 20317 thr 0 100000 10 6341 20353 thr -10 10000 100 4457 20530 thr -10 100000 10 6318 21240 scheduler: ULE thr_lib nice queries threads update select pthread 0 10000 100 3640 8413 pthread 0 100000 10 5269 21248 thr 0 10000 100 4422 16299* thr 0 100000 10 6400 22068 thr -10 10000 100 4400 21154 thr -10 100000 10 6211 21877 * - mean over 5 runs since results were rather unstable: 14088.81, 17629.83, 21210.02, 14173.42, 14395.31 ==== FreeBSD current (06.05.2006) + uds locking patch ==== scheduler: 4BSD thr_lib nice queries threads update select pthread 0 10000 100 3410 8237 pthread 0 100000 10 5150 19786 thr 0 10000 100 4559 14992 thr 0 100000 10 6422 29371 thr -10 10000 100 4535 12589 thr -10 100000 10 6446 30300 scheduler: ULE thr_lib nice queries threads update select pthread 0 10000 100 3455 8062 ---- lost contact with machine ---- results with various threadcounts at 10000 queries is available though @ http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/smack_results_ULE.txt ==== FreeBSD 6.1 RC2 ==== scheduler: 4BSD thr_lib socket nice queries threads update select pthread unix 0 10000 100 2609 6683 pthread unix 0 100000 10 3752 15052 thr unix 0 10000 100 4802 11496 thr unix 0 100000 10 6398 20738 thr unix -10 10000 100 4607 21066 thr unix -10 100000 10 6379 21411 thr tcp -10 10000 100 4335 9952 scheduler: ULE thr_lib socket nice queries threads update select thr unix 0 10000 100 4779 13724 thr unix 0 100000 10 6473 25172 thr unix -10 10000 100 4969 20662 thr unix -10 100000 10 6418 25204 ==== Linux ==== suse - suse enterprise linux 9 with kernel 2.6.5-7.97-smp, mysql 4.0.18-32.1, reiserfs fedora - fedora core with kernel 2.6.9-22-ELsmp, mysql 4.1.12, ext3 distro queries threads update select suse 10000 100 10047* 76857 fedora 10000 100 8072* 67000 * - on linux the bin-log option was not disabled in the mysql config, so it would probably have gotten even higher update smack scores if I had disabled it like I did on freebsd From owner-freebsd-performance@FreeBSD.ORG Sun May 7 16:50:08 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E640816A400; Sun, 7 May 2006 16:50:07 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx1.starman.ee (smtp-out1.starman.ee [85.253.0.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id 179E143D49; Sun, 7 May 2006 16:50:06 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx1.starman.ee (Postfix) with ESMTP id 839C123C157; Sun, 7 May 2006 19:50:03 +0300 (EEST) From: Sven Petai To: freebsd-performance@freebsd.org Date: Sun, 7 May 2006 19:49:53 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> In-Reply-To: <20060506150622.C17611@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605071949.54978.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx1.starman.ee Cc: performance@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 16:50:08 -0000 hi I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in total). In general with 10 parallel smacker threads the performance seems to go up with your patch by ~44% and with 100 parallel threads it goes down ~25% Here's a graph of select smack performance with your patch: http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/ULE_+_pthread_vs._thr.png Server crashed before I managed to get enough data to plot graphs comparing pre-patch and patched performance. I'll try to generate that and another one, comparing the performance against linux on the same hardware, tomorrow. But below you can find some pre and after patch data that should give some idea of how the patch has impacted performance. Various test data, benchmarking scripts I used etc. can be found @ http://bsd.ee/~hadara/debug/mysql4/ I have included nice value of mysqld in the tests, because I have found that the impact of giving mysqld higher priority can have dramatic effects, especially under 6.X. This happens probably because super-smack with its 100 procs would otherwise get scheduling advantage over mysqld with 100 threads. Hardware: motherboard: Thunder K8QSD Pro hdd: scsi seagate cheetah 10K7 ram: 8 * 3200 CL3 kingston ECC 1G cpu: 4 * opteron 870 (2Ghz dualcore) Environment: mysql: ver: 4.1.18_2 from the ports table type: MyISAM config: my-huge.cnf with disabled bin-log and max-connections bumped to 400 database located on a slice mounted with softupdates super-smack: ver: from the ports smacks: default select and update All tests were run 2 times and mean value was used, unless stated otherwise. ==== FreeBSD current (06.05.2006) ==== scheduler: 4BSD thr_lib nice queries threads update select pthread 0 10000 100 3280 8253 pthread 0 100000 10 5037 19972 thr 0 10000 100 4470 20317 thr 0 100000 10 6341 20353 thr -10 10000 100 4457 20530 thr -10 100000 10 6318 21240 scheduler: ULE thr_lib nice queries threads update select pthread 0 10000 100 3640 8413 pthread 0 100000 10 5269 21248 thr 0 10000 100 4422 16299* thr 0 100000 10 6400 22068 thr -10 10000 100 4400 21154 thr -10 100000 10 6211 21877 * - mean over 5 runs since results were rather unstable: 14088.81, 17629.83, 21210.02, 14173.42, 14395.31 ==== FreeBSD current (06.05.2006) + uds locking patch ==== scheduler: 4BSD thr_lib nice queries threads update select pthread 0 10000 100 3410 8237 pthread 0 100000 10 5150 19786 thr 0 10000 100 4559 14992 thr 0 100000 10 6422 29371 thr -10 10000 100 4535 12589 thr -10 100000 10 6446 30300 scheduler: ULE thr_lib nice queries threads update select pthread 0 10000 100 3455 8062 ---- lost contact with machine ---- results with various threadcounts at 10000 queries is available though @ http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/smack_results_ULE.txt ==== FreeBSD 6.1 RC2 ==== scheduler: 4BSD thr_lib socket nice queries threads update select pthread unix 0 10000 100 2609 6683 pthread unix 0 100000 10 3752 15052 thr unix 0 10000 100 4802 11496 thr unix 0 100000 10 6398 20738 thr unix -10 10000 100 4607 21066 thr unix -10 100000 10 6379 21411 thr tcp -10 10000 100 4335 9952 scheduler: ULE thr_lib socket nice queries threads update select thr unix 0 10000 100 4779 13724 thr unix 0 100000 10 6473 25172 thr unix -10 10000 100 4969 20662 thr unix -10 100000 10 6418 25204 ==== Linux ==== suse - suse enterprise linux 9 with kernel 2.6.5-7.97-smp, mysql 4.0.18-32.1, reiserfs fedora - fedora core with kernel 2.6.9-22-ELsmp, mysql 4.1.12, ext3 distro queries threads update select suse 10000 100 10047* 76857 fedora 10000 100 8072* 67000 * - on linux the bin-log option was not disabled in the mysql config, so it would probably have gotten even higher update smack scores if I had disabled it like I did on freebsd From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:16:44 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 20F9716A43A; Sun, 7 May 2006 18:16:44 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5FEFA43D9B; Sun, 7 May 2006 18:16:35 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 8B9FB46D42; Sun, 7 May 2006 14:16:34 -0400 (EDT) Date: Sun, 7 May 2006 19:16:34 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Sven Petai In-Reply-To: <200605071949.54978.hadara@bsd.ee> Message-ID: <20060507190844.K46997@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:16:44 -0000 On Sun, 7 May 2006, Sven Petai wrote: > I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in > total). > > In general with 10 parallel smacker threads the performance seems to go up > with your patch by ~44% and with 100 parallel threads it goes down ~25% This is an interesting effect I need to explore. Kris reported much increased contention on locks within the process (between threads) when running with my patch. It would be interesting to know what the effect on average query time is -- perhaps it has gone down and we're looking at increased scheduler related contention. I noticed the results in the tests seem somewhat variable. I've noticed that MySQL bennchmarking is heavily affected by test run time and order. It's not atypical when running a series of identical tests to see a first result half the end rate, a second result *better* than the end rate, and then it balance out between the two. For example, I see the following on a dual Xeon: orangutan# foreach i (`jot 12`) super-smack /usr/share/smacks/select-key.smack 2 1000 | grep index | awk -F' ' '{print $5}' end foreach? super-smack /usr/share/smacks/select-key.smack 2 1000 | grep index | awk -F' ' '{print $5}' end foreach? end 4253.60 8685.55 8324.28 8680.88 8321.53 8391.27 8557.10 8034.79 7532.90 8196.65 7931.77 7981.55 This is quite usual, and is sometimes more pronounced. I'm not surprised the first run is slower while things get going, binaries are pulled off disk, code enters the cache, etc. What surprises me is how it bumps up higher initialially, then goes down to balance at a somewhat lower rate. Anyhow, what I'm getting at is this: make sure when you measure MySQL performance, you do a series of runs, discard the first few, and then take an average of the remainder, and watch out for outliers. Robert N M Watson > > Here's a graph of select smack performance with your patch: > http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/ULE_+_pthread_vs._thr.png > > Server crashed before I managed to get enough data to plot graphs > comparing pre-patch and patched performance. > I'll try to generate that and another one, comparing the performance against > linux on the same hardware, tomorrow. > > But below you can find some pre and after patch data that should > give some idea of how the patch has impacted performance. > > Various test data, benchmarking scripts I used etc. can be found @ > http://bsd.ee/~hadara/debug/mysql4/ > > I have included nice value of mysqld in the tests, because I have found that > the impact of giving mysqld higher priority can have dramatic effects, > especially under 6.X. > This happens probably because super-smack with its 100 procs would > otherwise get scheduling advantage over mysqld with 100 threads. > > Hardware: > motherboard: Thunder K8QSD Pro > hdd: scsi seagate cheetah 10K7 > ram: 8 * 3200 CL3 kingston ECC 1G > cpu: 4 * opteron 870 (2Ghz dualcore) > > Environment: > mysql: > ver: 4.1.18_2 from the ports > table type: MyISAM > config: my-huge.cnf with disabled bin-log and max-connections bumped to 400 > database located on a slice mounted with softupdates > super-smack: > ver: from the ports > smacks: default select and update > All tests were run 2 times and mean value was used, unless stated otherwise. > > ==== FreeBSD current (06.05.2006) ==== > scheduler: 4BSD > thr_lib nice queries threads update select > pthread 0 10000 100 3280 8253 > pthread 0 100000 10 5037 19972 > thr 0 10000 100 4470 20317 > thr 0 100000 10 6341 20353 > thr -10 10000 100 4457 20530 > thr -10 100000 10 6318 21240 > > scheduler: ULE > thr_lib nice queries threads update select > pthread 0 10000 100 3640 8413 > pthread 0 100000 10 5269 21248 > thr 0 10000 100 4422 16299* > thr 0 100000 10 6400 22068 > thr -10 10000 100 4400 21154 > thr -10 100000 10 6211 21877 > > * - mean over 5 runs since results were rather unstable: > 14088.81, 17629.83, 21210.02, 14173.42, 14395.31 > > ==== FreeBSD current (06.05.2006) + uds locking patch ==== > scheduler: 4BSD > thr_lib nice queries threads update select > pthread 0 10000 100 3410 8237 > pthread 0 100000 10 5150 19786 > thr 0 10000 100 4559 14992 > thr 0 100000 10 6422 29371 > thr -10 10000 100 4535 12589 > thr -10 100000 10 6446 30300 > > scheduler: ULE > thr_lib nice queries threads update select > pthread 0 10000 100 3455 8062 > ---- lost contact with machine ---- > results with various threadcounts at 10000 queries > is available though @ > http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/smack_results_ULE.txt > > > ==== FreeBSD 6.1 RC2 ==== > scheduler: 4BSD > thr_lib socket nice queries threads update select > pthread unix 0 10000 100 2609 6683 > pthread unix 0 100000 10 3752 15052 > thr unix 0 10000 100 4802 11496 > thr unix 0 100000 10 6398 20738 > thr unix -10 10000 100 4607 21066 > thr unix -10 100000 10 6379 21411 > thr tcp -10 10000 100 4335 9952 > > scheduler: ULE > thr_lib socket nice queries threads update select > thr unix 0 10000 100 4779 13724 > thr unix 0 100000 10 6473 25172 > thr unix -10 10000 100 4969 20662 > thr unix -10 100000 10 6418 25204 > > ==== Linux ==== > suse - suse enterprise linux 9 with kernel 2.6.5-7.97-smp, mysql > 4.0.18-32.1, reiserfs > fedora - fedora core with kernel 2.6.9-22-ELsmp, mysql 4.1.12, ext3 > > distro queries threads update select > suse 10000 100 10047* 76857 > fedora 10000 100 8072* 67000 > > * - on linux the bin-log option was not disabled in the mysql > config, so it would probably have gotten even higher update smack scores > if I had disabled it like I did on freebsd > > From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:16:44 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 20F9716A43A; Sun, 7 May 2006 18:16:44 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5FEFA43D9B; Sun, 7 May 2006 18:16:35 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 8B9FB46D42; Sun, 7 May 2006 14:16:34 -0400 (EDT) Date: Sun, 7 May 2006 19:16:34 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Sven Petai In-Reply-To: <200605071949.54978.hadara@bsd.ee> Message-ID: <20060507190844.K46997@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:16:44 -0000 On Sun, 7 May 2006, Sven Petai wrote: > I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in > total). > > In general with 10 parallel smacker threads the performance seems to go up > with your patch by ~44% and with 100 parallel threads it goes down ~25% This is an interesting effect I need to explore. Kris reported much increased contention on locks within the process (between threads) when running with my patch. It would be interesting to know what the effect on average query time is -- perhaps it has gone down and we're looking at increased scheduler related contention. I noticed the results in the tests seem somewhat variable. I've noticed that MySQL bennchmarking is heavily affected by test run time and order. It's not atypical when running a series of identical tests to see a first result half the end rate, a second result *better* than the end rate, and then it balance out between the two. For example, I see the following on a dual Xeon: orangutan# foreach i (`jot 12`) super-smack /usr/share/smacks/select-key.smack 2 1000 | grep index | awk -F' ' '{print $5}' end foreach? super-smack /usr/share/smacks/select-key.smack 2 1000 | grep index | awk -F' ' '{print $5}' end foreach? end 4253.60 8685.55 8324.28 8680.88 8321.53 8391.27 8557.10 8034.79 7532.90 8196.65 7931.77 7981.55 This is quite usual, and is sometimes more pronounced. I'm not surprised the first run is slower while things get going, binaries are pulled off disk, code enters the cache, etc. What surprises me is how it bumps up higher initialially, then goes down to balance at a somewhat lower rate. Anyhow, what I'm getting at is this: make sure when you measure MySQL performance, you do a series of runs, discard the first few, and then take an average of the remainder, and watch out for outliers. Robert N M Watson > > Here's a graph of select smack performance with your patch: > http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/ULE_+_pthread_vs._thr.png > > Server crashed before I managed to get enough data to plot graphs > comparing pre-patch and patched performance. > I'll try to generate that and another one, comparing the performance against > linux on the same hardware, tomorrow. > > But below you can find some pre and after patch data that should > give some idea of how the patch has impacted performance. > > Various test data, benchmarking scripts I used etc. can be found @ > http://bsd.ee/~hadara/debug/mysql4/ > > I have included nice value of mysqld in the tests, because I have found that > the impact of giving mysqld higher priority can have dramatic effects, > especially under 6.X. > This happens probably because super-smack with its 100 procs would > otherwise get scheduling advantage over mysqld with 100 threads. > > Hardware: > motherboard: Thunder K8QSD Pro > hdd: scsi seagate cheetah 10K7 > ram: 8 * 3200 CL3 kingston ECC 1G > cpu: 4 * opteron 870 (2Ghz dualcore) > > Environment: > mysql: > ver: 4.1.18_2 from the ports > table type: MyISAM > config: my-huge.cnf with disabled bin-log and max-connections bumped to 400 > database located on a slice mounted with softupdates > super-smack: > ver: from the ports > smacks: default select and update > All tests were run 2 times and mean value was used, unless stated otherwise. > > ==== FreeBSD current (06.05.2006) ==== > scheduler: 4BSD > thr_lib nice queries threads update select > pthread 0 10000 100 3280 8253 > pthread 0 100000 10 5037 19972 > thr 0 10000 100 4470 20317 > thr 0 100000 10 6341 20353 > thr -10 10000 100 4457 20530 > thr -10 100000 10 6318 21240 > > scheduler: ULE > thr_lib nice queries threads update select > pthread 0 10000 100 3640 8413 > pthread 0 100000 10 5269 21248 > thr 0 10000 100 4422 16299* > thr 0 100000 10 6400 22068 > thr -10 10000 100 4400 21154 > thr -10 100000 10 6211 21877 > > * - mean over 5 runs since results were rather unstable: > 14088.81, 17629.83, 21210.02, 14173.42, 14395.31 > > ==== FreeBSD current (06.05.2006) + uds locking patch ==== > scheduler: 4BSD > thr_lib nice queries threads update select > pthread 0 10000 100 3410 8237 > pthread 0 100000 10 5150 19786 > thr 0 10000 100 4559 14992 > thr 0 100000 10 6422 29371 > thr -10 10000 100 4535 12589 > thr -10 100000 10 6446 30300 > > scheduler: ULE > thr_lib nice queries threads update select > pthread 0 10000 100 3455 8062 > ---- lost contact with machine ---- > results with various threadcounts at 10000 queries > is available though @ > http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/smack_results_ULE.txt > > > ==== FreeBSD 6.1 RC2 ==== > scheduler: 4BSD > thr_lib socket nice queries threads update select > pthread unix 0 10000 100 2609 6683 > pthread unix 0 100000 10 3752 15052 > thr unix 0 10000 100 4802 11496 > thr unix 0 100000 10 6398 20738 > thr unix -10 10000 100 4607 21066 > thr unix -10 100000 10 6379 21411 > thr tcp -10 10000 100 4335 9952 > > scheduler: ULE > thr_lib socket nice queries threads update select > thr unix 0 10000 100 4779 13724 > thr unix 0 100000 10 6473 25172 > thr unix -10 10000 100 4969 20662 > thr unix -10 100000 10 6418 25204 > > ==== Linux ==== > suse - suse enterprise linux 9 with kernel 2.6.5-7.97-smp, mysql > 4.0.18-32.1, reiserfs > fedora - fedora core with kernel 2.6.9-22-ELsmp, mysql 4.1.12, ext3 > > distro queries threads update select > suse 10000 100 10047* 76857 > fedora 10000 100 8072* 67000 > > * - on linux the bin-log option was not disabled in the mysql > config, so it would probably have gotten even higher update smack scores > if I had disabled it like I did on freebsd > > From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:23:14 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 826CA16A404; Sun, 7 May 2006 18:23:14 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3CC6643D48; Sun, 7 May 2006 18:23:14 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 17B301A3C19; Sun, 7 May 2006 11:23:14 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 1DD2C51695; Sun, 7 May 2006 14:23:13 -0400 (EDT) Date: Sun, 7 May 2006 14:23:12 -0400 From: Kris Kennaway To: Robert Watson Message-ID: <20060507182312.GA185@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline In-Reply-To: <20060507190844.K46997@fledge.watson.org> User-Agent: Mutt/1.4.2.1i Cc: Sven Petai , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:23:15 -0000 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 07:16:34PM +0100, Robert Watson wrote: >=20 > On Sun, 7 May 2006, Sven Petai wrote: >=20 > >I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in= =20 > >total). > > > >In general with 10 parallel smacker threads the performance seems to go = up=20 > >with your patch by ~44% and with 100 parallel threads it goes down ~25% >=20 > This is an interesting effect I need to explore. Kris reported much=20 > increased contention on locks within the process (between threads) when= =20 > running with my patch. It would be interesting to know what the effect o= n=20 > average query time is -- perhaps it has gone down and we're looking at=20 > increased scheduler related contention. >=20 > I noticed the results in the tests seem somewhat variable. I've noticed= =20 > that MySQL bennchmarking is heavily affected by test run time and order. = =20 > It's not atypical when running a series of identical tests to see a first= =20 > result half the end rate, a second result *better* than the end rate, and= =20 > then it balance out between the two. For example, I see the following on= a=20 Also, I see a slow but statistically significant deterioration in performance over time. Maybe mysql's memory is getting fragmented or something. Kris --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXjsQWry0BWjoQKURAtNGAJ0fM1JzVK9Rbj5c+7bunFj6oVGL/ACdFY+B gnscH3roaT5uePFlo4ek4vo= =af63 -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:23:14 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 826CA16A404; Sun, 7 May 2006 18:23:14 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3CC6643D48; Sun, 7 May 2006 18:23:14 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 17B301A3C19; Sun, 7 May 2006 11:23:14 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 1DD2C51695; Sun, 7 May 2006 14:23:13 -0400 (EDT) Date: Sun, 7 May 2006 14:23:12 -0400 From: Kris Kennaway To: Robert Watson Message-ID: <20060507182312.GA185@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline In-Reply-To: <20060507190844.K46997@fledge.watson.org> User-Agent: Mutt/1.4.2.1i Cc: Sven Petai , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:23:15 -0000 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 07:16:34PM +0100, Robert Watson wrote: >=20 > On Sun, 7 May 2006, Sven Petai wrote: >=20 > >I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in= =20 > >total). > > > >In general with 10 parallel smacker threads the performance seems to go = up=20 > >with your patch by ~44% and with 100 parallel threads it goes down ~25% >=20 > This is an interesting effect I need to explore. Kris reported much=20 > increased contention on locks within the process (between threads) when= =20 > running with my patch. It would be interesting to know what the effect o= n=20 > average query time is -- perhaps it has gone down and we're looking at=20 > increased scheduler related contention. >=20 > I noticed the results in the tests seem somewhat variable. I've noticed= =20 > that MySQL bennchmarking is heavily affected by test run time and order. = =20 > It's not atypical when running a series of identical tests to see a first= =20 > result half the end rate, a second result *better* than the end rate, and= =20 > then it balance out between the two. For example, I see the following on= a=20 Also, I see a slow but statistically significant deterioration in performance over time. Maybe mysql's memory is getting fragmented or something. Kris --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXjsQWry0BWjoQKURAtNGAJ0fM1JzVK9Rbj5c+7bunFj6oVGL/ACdFY+B gnscH3roaT5uePFlo4ek4vo= =af63 -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:30:26 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB54916A43B for ; Sun, 7 May 2006 18:30:26 +0000 (UTC) (envelope-from mikej@rogers.com) Received: from smtp102.rog.mail.re2.yahoo.com (smtp102.rog.mail.re2.yahoo.com [206.190.36.80]) by mx1.FreeBSD.org (Postfix) with SMTP id B410743D7F for ; Sun, 7 May 2006 18:30:19 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 63229 invoked from network); 7 May 2006 18:30:18 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=FO2M/SLxHLfoKarnayVXI/UEsCpXOpX7SkgTo71YYePQWMHWwZyMv1IBN47HAI2CVP7xTqVLWgDfV9/wdjc5lULHObCc1mTcgBpHT+rBXNoNyIJ2FxG+wP+Fz/1IhK88fOphaW9d9UOnrnY/OdY2acgPaj4YnQWg6IUnrZinGSk= ; Received: from unknown (HELO ?70.31.50.218?) (mikej@rogers.com@70.31.50.218 with plain) by smtp102.rog.mail.re2.yahoo.com with SMTP; 7 May 2006 18:30:18 -0000 Message-ID: <445E3CBE.1040105@rogers.com> Date: Sun, 07 May 2006 14:30:22 -0400 From: Mike Jakubik User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: Sven Petai References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> In-Reply-To: <200605071949.54978.hadara@bsd.ee> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Robert Watson , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:30:27 -0000 Sven Petai wrote: > scheduler: ULE > thr_lib socket nice queries threads update select > thr unix 0 10000 100 4779 13724 > thr unix 0 100000 10 6473 25172 > thr unix -10 10000 100 4969 20662 > thr unix -10 100000 10 6418 25204 > > ==== Linux ==== > suse - suse enterprise linux 9 with kernel 2.6.5-7.97-smp, mysql > 4.0.18-32.1, reiserfs > fedora - fedora core with kernel 2.6.9-22-ELsmp, mysql 4.1.12, ext3 > > distro queries threads update select > suse 10000 100 10047* 76857 > fedora 10000 100 8072* 67000 > The difference in performance is just ridiculous. Is mysql written to be slow on freebsd or is there a problem with freebsd? From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:30:28 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6D05816A45F for ; Sun, 7 May 2006 18:30:28 +0000 (UTC) (envelope-from mikej@rogers.com) Received: from smtp102.rog.mail.re2.yahoo.com (smtp102.rog.mail.re2.yahoo.com [206.190.36.80]) by mx1.FreeBSD.org (Postfix) with SMTP id 85D8F43D83 for ; Sun, 7 May 2006 18:30:19 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 63229 invoked from network); 7 May 2006 18:30:18 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=FO2M/SLxHLfoKarnayVXI/UEsCpXOpX7SkgTo71YYePQWMHWwZyMv1IBN47HAI2CVP7xTqVLWgDfV9/wdjc5lULHObCc1mTcgBpHT+rBXNoNyIJ2FxG+wP+Fz/1IhK88fOphaW9d9UOnrnY/OdY2acgPaj4YnQWg6IUnrZinGSk= ; Received: from unknown (HELO ?70.31.50.218?) (mikej@rogers.com@70.31.50.218 with plain) by smtp102.rog.mail.re2.yahoo.com with SMTP; 7 May 2006 18:30:18 -0000 Message-ID: <445E3CBE.1040105@rogers.com> Date: Sun, 07 May 2006 14:30:22 -0400 From: Mike Jakubik User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: Sven Petai References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> In-Reply-To: <200605071949.54978.hadara@bsd.ee> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Robert Watson , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:30:29 -0000 Sven Petai wrote: > scheduler: ULE > thr_lib socket nice queries threads update select > thr unix 0 10000 100 4779 13724 > thr unix 0 100000 10 6473 25172 > thr unix -10 10000 100 4969 20662 > thr unix -10 100000 10 6418 25204 > > ==== Linux ==== > suse - suse enterprise linux 9 with kernel 2.6.5-7.97-smp, mysql > 4.0.18-32.1, reiserfs > fedora - fedora core with kernel 2.6.9-22-ELsmp, mysql 4.1.12, ext3 > > distro queries threads update select > suse 10000 100 10047* 76857 > fedora 10000 100 8072* 67000 > The difference in performance is just ridiculous. Is mysql written to be slow on freebsd or is there a problem with freebsd? From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:47:29 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9490E16A402; Sun, 7 May 2006 18:47:29 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 21EB243D4C; Sun, 7 May 2006 18:47:28 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 2396746CBA; Sun, 7 May 2006 14:47:27 -0400 (EDT) Date: Sun, 7 May 2006 19:47:27 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Mike Jakubik In-Reply-To: <445E3CBE.1040105@rogers.com> Message-ID: <20060507193651.O46997@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <445E3CBE.1040105@rogers.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Sven Petai , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:47:29 -0000 On Sun, 7 May 2006, Mike Jakubik wrote: > The difference in performance is just ridiculous. Is mysql written to be > slow on freebsd or is there a problem with freebsd? In past discussion, I think a reasonable conclusion has been some amount of both. We've identified a few particular areas where there are differences deserving attention: - MySQL has increasingly made use of time query calls, especially in more recent versions. On FreeBSD, we provide very precise time at somewhat higher cost than Linux. Presumably the design choice has been made in MySQL that since time is free [on Linux], it should be checked lots, which tends to hurt MySQL performance more on non-Linux platforms. Some experimental work has been done to offer reduced time quality via alternative APIs, but we haven't settled on a particular approach. Currently APIs don't allow you to say "I don't want to pay a lot for my muf... timestamp.". I have a wrapper libc library somewhere that causes applications to query lower quality time stamps using faster interfaces which has been shown to have a significant impact on MySQL performance. - MySQL seems to really like Linux threading and scheduling. For example, in the past it has been observed that you get significantly better MySQL performance on FreeBSD when you use the Linux threading implementation. libthr closes some of that gap, but last I checked wasn't there yet. I'm not really familiar with the issues here so can't speak to them in detail. - Locking granularity has been an issue on SMP for MySQL on FreeBSD. The patch I posted is part of working to address this. We have been seeing very high contention on IPC locks in particular. With my change, Kris reports much lower IPC lock contention -- instead we start hitting process-related lock bottlenecks. There's probably a lot more to this -- in the past there has also been speculation (although not, as far as I know, a lot of substantiation) that FreeBSD is running into some file system meta-data update bottlenecks, and that our sync interfaces for the file system are either more complete, less granular, or work too hard. Again, outside of my areas of expertise. Part of the problem is probably an application optimization problem to do with optimizing MySQL to use features that are more cheap on Linux than FreeBSD, rather than vice versa -- there are many other applications that perform measurably better on FreeBSD than Linux, especially when hitting the VM, which suggests that while we probably do have problems we need to fix, something disportionate may have happened for MySQL performance. We do have significant overheads in our communications code that we've been working on actively, and differences in threading that need to be explored more. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:47:29 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9490E16A402; Sun, 7 May 2006 18:47:29 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 21EB243D4C; Sun, 7 May 2006 18:47:28 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 2396746CBA; Sun, 7 May 2006 14:47:27 -0400 (EDT) Date: Sun, 7 May 2006 19:47:27 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Mike Jakubik In-Reply-To: <445E3CBE.1040105@rogers.com> Message-ID: <20060507193651.O46997@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <445E3CBE.1040105@rogers.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Sven Petai , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:47:29 -0000 On Sun, 7 May 2006, Mike Jakubik wrote: > The difference in performance is just ridiculous. Is mysql written to be > slow on freebsd or is there a problem with freebsd? In past discussion, I think a reasonable conclusion has been some amount of both. We've identified a few particular areas where there are differences deserving attention: - MySQL has increasingly made use of time query calls, especially in more recent versions. On FreeBSD, we provide very precise time at somewhat higher cost than Linux. Presumably the design choice has been made in MySQL that since time is free [on Linux], it should be checked lots, which tends to hurt MySQL performance more on non-Linux platforms. Some experimental work has been done to offer reduced time quality via alternative APIs, but we haven't settled on a particular approach. Currently APIs don't allow you to say "I don't want to pay a lot for my muf... timestamp.". I have a wrapper libc library somewhere that causes applications to query lower quality time stamps using faster interfaces which has been shown to have a significant impact on MySQL performance. - MySQL seems to really like Linux threading and scheduling. For example, in the past it has been observed that you get significantly better MySQL performance on FreeBSD when you use the Linux threading implementation. libthr closes some of that gap, but last I checked wasn't there yet. I'm not really familiar with the issues here so can't speak to them in detail. - Locking granularity has been an issue on SMP for MySQL on FreeBSD. The patch I posted is part of working to address this. We have been seeing very high contention on IPC locks in particular. With my change, Kris reports much lower IPC lock contention -- instead we start hitting process-related lock bottlenecks. There's probably a lot more to this -- in the past there has also been speculation (although not, as far as I know, a lot of substantiation) that FreeBSD is running into some file system meta-data update bottlenecks, and that our sync interfaces for the file system are either more complete, less granular, or work too hard. Again, outside of my areas of expertise. Part of the problem is probably an application optimization problem to do with optimizing MySQL to use features that are more cheap on Linux than FreeBSD, rather than vice versa -- there are many other applications that perform measurably better on FreeBSD than Linux, especially when hitting the VM, which suggests that while we probably do have problems we need to fix, something disportionate may have happened for MySQL performance. We do have significant overheads in our communications code that we've been working on actively, and differences in threading that need to be explored more. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Sun May 7 19:00:55 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B445416A42F; Sun, 7 May 2006 19:00:55 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx2.starman.ee (smtp-out4.starman.ee [85.253.0.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 25DAD43D58; Sun, 7 May 2006 19:00:52 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx2.starman.ee (Postfix) with ESMTP id 1311432C265; Sun, 7 May 2006 22:00:50 +0300 (EEST) From: Sven Petai To: Robert Watson Date: Sun, 7 May 2006 22:00:41 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> In-Reply-To: <20060507190844.K46997@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605072200.42529.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx2.starman.ee Cc: performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 19:00:56 -0000 > Anyhow, what I'm getting at is this: make sure when you measure MySQL > performance, you do a series of runs, discard the first few, and then take > an average of the remainder, and watch out for outliers. The results in my mail were mean values over 2 runs, only once did I see really huge (more than 10%) differences between several subsequent runs with same settings, this case was clearly mentioned in the results. graph was drawn from resultset available @ http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006_uds_patch/smack_results_ULE.txt as you can see each setting was tested twise and mean value was used, there doesnt seem to be any significant outliers in that dataset too. Tomorrow I'll try to get mutex profile while running with your patch, profiles for non-patched case are already available @ http://bsd.ee/~hadara/debug/mysql4/freebsd_current_06.02.2006/mutex_profile_thr_4bsd_select_smack_100-10000.txt most contended locks: max total count avg cnt_hold cnt_lock name 8143 926138412 36000820 25 111228313 34861067 /usr/src/sys/kern/uipc_usrreq.c:309 (unp) 9 1359 223 6 167 31657721 /usr/src/sys/kern/uipc_usrreq.c:709 (unp) 6107 123376889 4000519 30 12373564 3900209 /usr/src/sys/kern/uipc_usrreq.c:364 (unp) 34045 553265364 7972969 69 3951680 3101074 /usr/src/sys/kern/kern_descrip.c:378 (Giant) 1893100 355809996 44119592 8 602861 927204 /usr/src/sys/kern/kern_descrip.c:2011 (filedesc structure) 807479 389358045 44119592 8 1399635 853369 /usr/src/sys/kern/kern_descrip.c:1990 (filedesc structure) 146 1629 86 18 28 544914 /usr/src/sys/kern/kern_conf.c:311 (Giant) 991109 37883055 8003082 4 615328 509225 /usr/src/sys/kern/kern_descrip.c:385 (filedesc structure) 33986 33267790 7972969 4 244622 405542 /usr/src/sys/kern/kern_descrip.c:438 (filedesc structure) 13 16943 5357 3 475 381640 /usr/src/sys/kern/sys_generic.c:691 (filedesc structure) From owner-freebsd-performance@FreeBSD.ORG Sun May 7 18:27:49 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8383C16A43B; Sun, 7 May 2006 18:27:49 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from lh.synack.net (lh.synack.net [204.152.188.37]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2BE5C43D46; Sun, 7 May 2006 18:27:49 +0000 (GMT) (envelope-from jasone@FreeBSD.org) Received: by lh.synack.net (Postfix, from userid 100) id A97CF5E4927; Sun, 7 May 2006 11:27:48 -0700 (PDT) Received: from [192.168.168.201] (moscow-cuda-gen2-68-64-60-20.losaca.adelphia.net [68.64.60.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by lh.synack.net (Postfix) with ESMTP id 4407D5E4912; Sun, 7 May 2006 11:27:47 -0700 (PDT) Message-ID: <445E3C21.50206@FreeBSD.org> Date: Sun, 07 May 2006 11:27:45 -0700 From: Jason Evans User-Agent: Mozilla Thunderbird 1.0.8-1.4.1 (X11/20060420) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kris Kennaway References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> <20060507182312.GA185@xor.obsecurity.org> In-Reply-To: <20060507182312.GA185@xor.obsecurity.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on lh.synack.net X-Spam-Level: * X-Spam-Status: No, score=1.8 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 X-Mailman-Approved-At: Sun, 07 May 2006 19:05:49 +0000 Cc: performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 18:27:50 -0000 Kris Kennaway wrote: > > Also, I see a slow but statistically significant deterioration in > performance over time. Maybe mysql's memory is getting fragmented or > something. I have tools to qualitatively assess fragmentation. I can help with this, should you be interested in looking into it. Jason From owner-freebsd-performance@FreeBSD.ORG Sun May 7 19:16:55 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E16C316A4D7; Sun, 7 May 2006 19:16:54 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id C14F843D66; Sun, 7 May 2006 19:16:42 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 317AA1A3C1A; Sun, 7 May 2006 12:16:42 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 8FBC851695; Sun, 7 May 2006 15:16:41 -0400 (EDT) Date: Sun, 7 May 2006 15:16:41 -0400 From: Kris Kennaway To: Sven Petai Message-ID: <20060507191641.GA1851@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> <200605072200.42529.hadara@bsd.ee> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oyUTqETQ0mS9luUI" Content-Disposition: inline In-Reply-To: <200605072200.42529.hadara@bsd.ee> User-Agent: Mutt/1.4.2.1i Cc: performance@freebsd.org, Robert Watson , current@freebsd.org, csjp@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 19:16:55 -0000 --oyUTqETQ0mS9luUI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > The results in my mail were mean values over 2 runs, > only once did I see really huge (more than 10%) differences between sever= al=20 > subsequent runs with same settings, this case was clearly mentioned in th= e results. FYI, 2 is not really enough, you should do at least 10 repetitions of each test to reduce variance (which can be a lot, despite what you saw!) and so that differences between them can be accurately estimated. Ministat is really helpful for this. > most contended locks: > max total count avg cnt_hold cnt_lock name > 8143 926138412 36000820 25 111228313 34861067 /usr/src/= sys/kern/uipc_usrreq.c:309 (unp) > 9 1359 223 6 167 31657721 /usr/src/= sys/kern/uipc_usrreq.c:709 (unp) > 6107 123376889 4000519 30 12373564 3900209 /usr/src/= sys/kern/uipc_usrreq.c:364 (unp) > 34045 553265364 7972969 69 3951680 3101074 /usr/src/= sys/kern/kern_descrip.c:378 (Giant) > 1893100 355809996 44119592 8 602861 927204 /usr/src= /sys/kern/kern_descrip.c:2011 (filedesc structure) > 807479 389358045 44119592 8 1399635 853369 /usr/src/= sys/kern/kern_descrip.c:1990 (filedesc structure) > 146 1629 86 18 28 544914 /usr/src/= sys/kern/kern_conf.c:311 (Giant) > 991109 37883055 8003082 4 615328 509225 /usr/src/= sys/kern/kern_descrip.c:385 (filedesc structure) > 33986 33267790 7972969 4 244622 405542 /usr/src/= sys/kern/kern_descrip.c:438 (filedesc structure) > 13 16943 5357 3 475 381640 /usr/src/= sys/kern/sys_generic.c:691 (filedesc structure) Another patch that helps a lot (gets rid of the Giant in the above trace and gives me a modest performance boost) is here: http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff He hasn't committed it yet because there is some evidence it helps to make an unrelated devfs bug more common in certain test loads. I've come to the opinion that it should just be committed since the problem does not seem to be directly caused by this change. Kris --oyUTqETQ0mS9luUI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXkeYWry0BWjoQKURAikpAJ0WLlDSKCPs2qQDI0Srf93kFNKU7gCg1l8V k27vEp5TmcG43FnKJkn0w4s= =SyIO -----END PGP SIGNATURE----- --oyUTqETQ0mS9luUI-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 19:57:22 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7B58A16A402; Sun, 7 May 2006 19:57:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2937243D48; Sun, 7 May 2006 19:57:22 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 9E1ED46C09; Sun, 7 May 2006 15:57:21 -0400 (EDT) Date: Sun, 7 May 2006 20:57:21 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kris Kennaway In-Reply-To: <20060507191641.GA1851@xor.obsecurity.org> Message-ID: <20060507205627.B46997@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> <200605072200.42529.hadara@bsd.ee> <20060507191641.GA1851@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Sven Petai , performance@freebsd.org, current@freebsd.org, csjp@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 19:57:22 -0000 > On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > >> The results in my mail were mean values over 2 runs, >> only once did I see really huge (more than 10%) differences between several >> subsequent runs with same settings, this case was clearly mentioned in the results. > > FYI, 2 is not really enough, you should do at least 10 repetitions of each > test to reduce variance (which can be a lot, despite what you saw!) and so > that differences between them can be accurately estimated. Ministat is > really helpful for this. Typically, I do 12 runs of supersmack in each configuration, and discard the first 2 runs in which the cache and scheduler (etc) are still settling, as I'm interested in the steady state. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Sun May 7 19:59:44 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A4B3716A418; Sun, 7 May 2006 19:59:44 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 88D8A43D66; Sun, 7 May 2006 19:59:42 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 63A9D1A3C1A; Sun, 7 May 2006 12:59:42 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id BA46251DA6; Sun, 7 May 2006 15:59:41 -0400 (EDT) Date: Sun, 7 May 2006 15:59:41 -0400 From: Kris Kennaway To: Robert Watson Message-ID: <20060507195941.GA2978@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> <200605072200.42529.hadara@bsd.ee> <20060507191641.GA1851@xor.obsecurity.org> <20060507205627.B46997@fledge.watson.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zYM0uCDKw75PZbzx" Content-Disposition: inline In-Reply-To: <20060507205627.B46997@fledge.watson.org> User-Agent: Mutt/1.4.2.1i Cc: Sven Petai , csjp@FreeBSD.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 19:59:44 -0000 --zYM0uCDKw75PZbzx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 08:57:21PM +0100, Robert Watson wrote: >=20 > >On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > > >>The results in my mail were mean values over 2 runs, > >>only once did I see really huge (more than 10%) differences between=20 > >>several > >>subsequent runs with same settings, this case was clearly mentioned in= =20 > >>the results. > > > >FYI, 2 is not really enough, you should do at least 10 repetitions of ea= ch=20 > >test to reduce variance (which can be a lot, despite what you saw!) and = so=20 > >that differences between them can be accurately estimated. Ministat is= =20 > >really helpful for this. >=20 > Typically, I do 12 runs of supersmack in each configuration, and discard= =20 > the first 2 runs in which the cache and scheduler (etc) are still settlin= g,=20 > as I'm interested in the steady state. Yeah, forgot to mention that too. Also keeping in mind that mysql performance is not time-constant, e.g. you should restart it in between test configurations instead of comparing early and late times. Kris --zYM0uCDKw75PZbzx Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXlGtWry0BWjoQKURAh/BAKCPgPuGgxUzTeBWbstBc85S8zAEQQCgpDtT qP+XbWrbbL5xauyq3dOie18= =gMww -----END PGP SIGNATURE----- --zYM0uCDKw75PZbzx-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 20:09:01 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4B3316A403; Sun, 7 May 2006 20:09:01 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8126643D55; Sun, 7 May 2006 20:09:01 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 11F2146C8B; Sun, 7 May 2006 16:09:01 -0400 (EDT) Date: Sun, 7 May 2006 21:09:00 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kris Kennaway In-Reply-To: <20060507195941.GA2978@xor.obsecurity.org> Message-ID: <20060507210824.F17611@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <20060507190844.K46997@fledge.watson.org> <200605072200.42529.hadara@bsd.ee> <20060507191641.GA1851@xor.obsecurity.org> <20060507205627.B46997@fledge.watson.org> <20060507195941.GA2978@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Sven Petai , performance@freebsd.org, current@freebsd.org, csjp@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 20:09:01 -0000 On Sun, 7 May 2006, Kris Kennaway wrote: >> Typically, I do 12 runs of supersmack in each configuration, and discard >> the first 2 runs in which the cache and scheduler (etc) are still settling, >> as I'm interested in the steady state. > > Yeah, forgot to mention that too. Also keeping in mind that mysql > performance is not time-constant, e.g. you should restart it in between test > configurations instead of comparing early and late times. Yeah, very much so -- my normal test procedure is to reboot between runs, allowing the first to cycles in the test to warm things up. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Sun May 7 20:27:42 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 67ACD16A463; Sun, 7 May 2006 20:27:42 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx2.starman.ee (smtp-out4.starman.ee [85.253.0.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id D843643D6B; Sun, 7 May 2006 20:27:33 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx2.starman.ee (Postfix) with ESMTP id 178A932C255; Sun, 7 May 2006 23:27:31 +0300 (EEST) From: Sven Petai To: Kris Kennaway Date: Sun, 7 May 2006 23:27:22 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> <200605072200.42529.hadara@bsd.ee> <20060507191641.GA1851@xor.obsecurity.org> In-Reply-To: <20060507191641.GA1851@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605072327.23901.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx2.starman.ee Cc: rwatson@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 20:27:43 -0000 On Sunday 07 May 2006 22:16, you wrote: > On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > The results in my mail were mean values over 2 runs, > > only once did I see really huge (more than 10%) differences between > > several subsequent runs with same settings, this case was clearly > > mentioned in the results. > > FYI, 2 is not really enough, you should do at least 10 repetitions of > each test to reduce variance (which can be a lot, despite what you > saw!) and so that differences between them can be accurately > estimated. Ministat is really helpful for this. > I'm well aware that 2 is not enough for quality measurements and I certainly would have liked to do more repetitions, but I was running against a clock - this machine might be shipped out to client any time and I wanted to test several combinations of OS [fbsd 6, fbsd current, current + watsons patch, linux] with different threading library and scheduler combinations at different thread counts and nice values. This creates nice combinatorial explosion. Even a single run on one OS ver + one of the schedulers with 2 repetitions of each test takes about 2 hours, so I had to make some compromises. But i believe the trends are clear enough from these results and while I certainly can't say anything about <~5% performance changes what we see are consistent trends and 20%+ performance differences. From owner-freebsd-performance@FreeBSD.ORG Sun May 7 20:32:33 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ACC8D16A402; Sun, 7 May 2006 20:32:33 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6638943D45; Sun, 7 May 2006 20:32:33 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 179A01A3C1A; Sun, 7 May 2006 13:32:32 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 06C0851695; Sun, 7 May 2006 16:32:31 -0400 (EDT) Date: Sun, 7 May 2006 16:32:30 -0400 From: Kris Kennaway To: Sven Petai Message-ID: <20060507203230.GA3870@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605072200.42529.hadara@bsd.ee> <20060507191641.GA1851@xor.obsecurity.org> <200605072327.23901.hadara@bsd.ee> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="+HP7ph2BbKc20aGI" Content-Disposition: inline In-Reply-To: <200605072327.23901.hadara@bsd.ee> User-Agent: Mutt/1.4.2.1i Cc: rwatson@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 20:32:33 -0000 --+HP7ph2BbKc20aGI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 11:27:22PM +0300, Sven Petai wrote: > On Sunday 07 May 2006 22:16, you wrote: > > On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > > The results in my mail were mean values over 2 runs, > > > only once did I see really huge (more than 10%) differences between > > > several subsequent runs with same settings, this case was clearly > > > mentioned in the results. > > > > FYI, 2 is not really enough, you should do at least 10 repetitions of > > each test to reduce variance (which can be a lot, despite what you > > saw!) and so that differences between them can be accurately > > estimated. Ministat is really helpful for this. > > >=20 > I'm well aware that 2 is not enough for quality measurements and > I certainly would have liked to do more repetitions, but I was running ag= ainst=20 > a clock - this machine might be shipped out to client any > time and I wanted to test several combinations of OS [fbsd 6, fbsd curren= t,=20 > current + watsons patch, linux] with different threading library and=20 > scheduler combinations at different thread counts and nice values.=20 > This creates nice combinatorial explosion. FYI, ULE is probably not worth bothering about. It has too many performance problems and no owner. Kris --+HP7ph2BbKc20aGI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXlleWry0BWjoQKURAmN9AJ4+rdkseL9s+SS44k1FHLy7jUPobQCfaCQT mY4gf5mjZoQvumvcIn9OdPE= =UtMz -----END PGP SIGNATURE----- --+HP7ph2BbKc20aGI-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 21:04:28 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1E02016A400; Sun, 7 May 2006 21:04:28 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id B8B7A43D48; Sun, 7 May 2006 21:04:27 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 2BCD81A3C1A; Sun, 7 May 2006 14:04:27 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 969825161D; Sun, 7 May 2006 17:04:26 -0400 (EDT) Date: Sun, 7 May 2006 17:04:26 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060507210426.GA4422@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OXfL5xGRrasGEqWY" Content-Disposition: inline In-Reply-To: <20060506221908.GB51268@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 21:04:28 -0000 --OXfL5xGRrasGEqWY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 06, 2006 at 06:19:08PM -0400, Kris Kennaway wrote: > x norwatson-8 > + rwatson-8 > +------------------------------------------------------------+ > | + | > | + + + x x| > |+ + +++ + x xxxxx x x| > | |_________A___M______| |___A___| | > +------------------------------------------------------------+ > N Min Max Median Avg Stdd= ev > x 10 2601.46 2700.26 2650.52 2653.441 30.7580= 34 > + 10 2240.86 2516.87 2496.085 2468.468 81.8685= 76 > Difference at 95.0% confidence > -184.973 +/- 58.1052 > -6.97106% +/- 2.1898% > (Student's t, pooled s =3D 61.8406) >=20 > We see the drop in performance in both cases indicating that we are in > the "overloaded" regime. The fact that your patch seems to give worse > performance is puzzling at first sight. >=20 > Running mutex profiling (and only keeping the unp mutex entries and > the 10 most contended for clarity) shows the following: >=20 > norwatson, 8 clients: >=20 > debug.mutex.prof.stats: > max total count avg cnt_hold cnt_lock name > 5 40 9 4 0 3 kern/uipc= _usrreq.c:170 (unp) > 8 8 1 8 0 0 vm/uma_co= re.c:2101 (unpcb) > 13 283 52 5 0 0 vm/uma_co= re.c:890 (unpcb) > 14 1075 200 5 0 0 vm/uma_co= re.c:1885 (unpcb) > 4 52 18 2 4 6 kern/uipc= _usrreq.c:577 (unp) > 5 39 9 4 4 2 kern/uipc= _usrreq.c:534 (unp) > 5 35 11 3 6 6 kern/uipc= _usrreq.c:974 (unp) > 5 45 11 4 7 4 kern/uipc= _usrreq.c:210 (unp) > 171 1164 9 129 7 2 kern/uipc= _usrreq.c:917 (unp) > 14 78 20 3 11 2872481 kern/uipc= _usrreq.c:709 (unp) > 70 156 11 14 13 4 kern/uipc= _usrreq.c:895 (unp) > 43 581 20 29 24 6 kern/uipc= _usrreq.c:239 (unp) > 44 429 18 23 26 8 kern/uipc= _usrreq.c:518 (unp) > 55 491 12 40 30 10 kern/uipc= _usrreq.c:251 (unp) > ... > 449 20000519 320038 62 15158 0 kern/uipc= _usrreq.c:431 (so_rcv) > 459 86616085 2880079 30 15699 4944 kern/uipc= _usrreq.c:319 (so_snd) > 146 2273360 640315 3 27918 29789 kern/kern= _sig.c:1002 (process lock) > 387 3325481 640099 5 38143 47670 kern/kern= _descrip.c:420 (filedesc structure) > 150 1881990 640155 2 64111 49033 kern/kern= _descrip.c:368 (filedesc structure) > 496 13792853 3685885 3 101692 132480 kern/kern= _descrip.c:1988 (filedesc structure) > 207 4061793 551604 7 115427 118242 kern/kern= _synch.c:220 (process lock) > 391 10332282 3685885 2 194387 129547 kern/kern= _descrip.c:1967 (filedesc structure) > 465 25504709 320042 79 1632192 294498 kern/uipc= _usrreq.c:364 (unp) > 470 124263922 2880084 43 13222757 2685853 kern/uipc= _usrreq.c:309 (unp) >=20 > i.e. there is indeed heavy contention on the unp lock (column 5 counts > the number of times we tried to acquire it and failed because someone > else had the lock) - in fact about 5 times as many contentions as > successful acquisitions. >=20 > With your patch and the same load: >=20 > 3 20 9 2 0 0 kern/uipc= _usrreq.c:1028 (unp_mtx) > 3 22 9 2 0 0 kern/uipc= _usrreq.c:1161 (unp_mtx) > 5 29 9 3 0 2 kern/uipc= _usrreq.c:1065 (unp_global_mtx) > 5 53 18 2 0 76488 kern/uipc= _usrreq.c:287 (unp_global_mtx) > 6 33 9 3 0 0 kern/uipc= _usrreq.c:236 (unp_mtx) > 6 37 9 4 0 0 kern/uipc= _usrreq.c:819 (unp_mtx) > 7 7 1 7 0 0 vm/uma_co= re.c:2101 (unpcb) > 8 49 9 5 0 0 kern/uipc= _usrreq.c:1101 (unp_mtx) > 11 136 18 7 0 1 kern/uipc= _usrreq.c:458 (unp_global_mtx) > 32 143 9 15 0 1 kern/uipc= _usrreq.c:1160 (unp_global_mtx) > 44 472 18 26 0 0 kern/uipc= _usrreq.c:801 (unp_mtx) > 123 310 9 34 0 0 kern/uipc= _usrreq.c:1100 (unp_mtx) > 147 452 9 50 0 0 kern/uipc= _usrreq.c:1099 (unp_mtx) > 172 748 9 83 0 0 kern/uipc= _usrreq.c:473 (unp_mtx) > 337 1592 9 176 0 0 kern/uipc= _usrreq.c:1147 (unp_mtx) > 350 1790 9 198 0 0 kern/uipc= _usrreq.c:1146 (unp_mtx) > 780 39405928 320038 123 0 0 kern/uipc= _usrreq.c:618 (unp_mtx) > 18 140 9 15 1 0 kern/uipc= _usrreq.c:235 (unp_global_mtx) > 70 717 18 39 1 3 kern/uipc= _usrreq.c:800 (unp_global_mtx) > 528 2444 9 271 1 1 kern/uipc= _usrreq.c:1089 (unp_global_mtx) > 158 616 9 68 2 2 kern/uipc= _usrreq.c:476 (unp_mtx) > 794 175382857 2880084 60 2 7686 kern/uipc= _usrreq.c:574 (unp_mtx) > 4 25 9 2 3 2 kern/uipc= _usrreq.c:422 (unp_global_mtx) > 186 874 9 97 3 3 kern/uipc= _usrreq.c:472 (unp_global_mtx) > 768 33783759 320038 105 7442 0 kern/uipc= _usrreq.c:696 (unp_mtx) > ... > 465 913127 320045 2 43130 35046 kern/uipc= _socket.c:1101 (so_snd) > 483 2453927 628737 3 44768 46177 kern/kern= _sig.c:1002 (process lock) > 767 124298544 2880082 43 70037 59994 kern/uipc= _usrreq.c:581 (so_snd) > 794 45176699 320038 141 83252 72140 kern/uipc= _usrreq.c:617 (unp_global_mtx) > 549 9858281 3200210 3 579269 712643 kern/kern= _resource.c:1172 (sleep mtxpool) > 554 17122245 631715 27 641888 268243 kern/kern= _descrip.c:420 (filedesc structure) > 388 3009912 631753 4 653540 260590 kern/kern= _descrip.c:368 (filedesc structure) > 642 49626755 3681446 13 1642954 682669 kern/kern= _descrip.c:1988 (filedesc structure) > 530 13802687 3681446 3 1663244 616899 kern/kern= _descrip.c:1967 (filedesc structure) > 477 23472709 2810986 8 5671248 1900047 kern/kern= _synch.c:220 (process lock) >=20 > The top 10 heavily contended mutexes are very different (but note the > number of mutex acquisitions, column 3, is about the same). >=20 > There is not much contention on unp_global_mtx any longer, but there > is a lot more on some of the other mutexes, especially the process > lock via msleep(). Off-hand I don't know what is the cause of this > bottleneck (note: libthr is used as threading library and libpthread > is not ported to sparc64). >=20 > Also, a lot of the contention that used to be on the unp lock seems to > have fallen through onto contending *two* of the filedesc locks (all > about 1.6 million contentions). This may also help to explain the > performance drop. I profiled the wait channels in msleep and found: Before: 1 getblk 3 wait 1113 sigwait 2228 umtxq_busy 4505 fdesc 9405 umtx 14264 sbwait ----- 31515 total After: 1 getblk 1 wait 1413 sigwait 2794 umtxq_busy 4638 fdesc 10779 umtx 11663 sbwait ----- 31287 total i.e. the same number of msleep calls in both cases, but the patch shuffles them from sbwait onto everything else. Relative change: sbwait -19% umtx +14% fdesc +3% umtxq_busy +25% sigwait +27% This may help to understand why performance drops at high load when unp locking is moved out of the way. Kris --OXfL5xGRrasGEqWY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXmDZWry0BWjoQKURAu/0AKClaJIDOtbXoiyk38hh8XPOM2iixwCfdn5z 6Ls/SEAnr4MpzYr6hner4rU= =5hwo -----END PGP SIGNATURE----- --OXfL5xGRrasGEqWY-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 21:41:55 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A147E16A404; Sun, 7 May 2006 21:41:55 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 59BDA43D45; Sun, 7 May 2006 21:41:55 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 442C31A3C19; Sun, 7 May 2006 14:41:55 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 3814C515DC; Sun, 7 May 2006 17:41:54 -0400 (EDT) Date: Sun, 7 May 2006 17:41:53 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060507214153.GA5275@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="C7zPtVaVf+AK4Oqc" Content-Disposition: inline In-Reply-To: <20060507210426.GA4422@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 21:41:55 -0000 --C7zPtVaVf+AK4Oqc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 05:04:26PM -0400, Kris Kennaway wrote: > > 477 23472709 2810986 8 5671248 1900047 kern/ke= rn_synch.c:220 (process lock) > >=20 > > The top 10 heavily contended mutexes are very different (but note the > > number of mutex acquisitions, column 3, is about the same). > >=20 > > There is not much contention on unp_global_mtx any longer, but there > > is a lot more on some of the other mutexes, especially the process > > lock via msleep(). Off-hand I don't know what is the cause of this > > bottleneck (note: libthr is used as threading library and libpthread > > is not ported to sparc64). > i.e. the same number of msleep calls in both cases, but the patch > shuffles them from sbwait onto everything else. Oops, they're the same number because that was the size of my ktr buffer, duh :) Nevertheless it's a sample from the end of the same workload, so the distribution should be comparable. > sigwait +27% This appears to be the only relevant msleep wait channel that is using the process lock. static int kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi, struct timespec *timeout) { =2E.. td->td_sigmask =3D savedmask; SIGSETNAND(td->td_sigmask, waitset); signotify(td); error =3D msleep(&ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", hz); i.e. several threads in the process are all sleeping for 1 tick and then doing a thundering herd on the same proc lock when they wake up. To confirm this I added a KTR tracepoint before the msleep call. It's indeed being called from the various threads in mysqld. It's hard to say for sure how many CPUs are contending at once, but plotting a histogram of the time distribution of the calls indicates that several CPUs are frequently executing this code in close proximity. Kris --C7zPtVaVf+AK4Oqc Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXmmhWry0BWjoQKURAl4gAKCHsE1VGXgIJP7jm0HRUQ5p0VpS5ACcDzwX FRuZa8snPICD00VfJdFPym4= =0uuO -----END PGP SIGNATURE----- --C7zPtVaVf+AK4Oqc-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 23:04:35 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4CD8B16A42F; Sun, 7 May 2006 23:04:35 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA7C343D5F; Sun, 7 May 2006 23:04:31 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 9309E1A3C1E; Sun, 7 May 2006 16:04:31 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id F0F2B516B8; Sun, 7 May 2006 19:04:30 -0400 (EDT) Date: Sun, 7 May 2006 19:04:30 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060507230430.GA6872@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="EeQfGwPcQSOJBaQU" Content-Disposition: inline In-Reply-To: <20060507214153.GA5275@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 23:04:37 -0000 --EeQfGwPcQSOJBaQU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, May 07, 2006 at 05:41:53PM -0400, Kris Kennaway wrote: > static int > kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi, > struct timespec *timeout) > { > ... > td->td_sigmask =3D savedmask; > SIGSETNAND(td->td_sigmask, waitset); > signotify(td); > error =3D msleep(&ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", hz); >=20 > i.e. several threads in the process are all sleeping for 1 tick and > then doing a thundering herd on the same proc lock when they wake up. Oops, actually hz=3D0; it comes via sigwait(2). It's still a thundering herd situation though. I also profiled the sleepq_broadcast call to look at thundering herds from wakeup(), and I found that the umtxq_chain is waking the following distribution of CPUs at once: Before patch: Freq #CPUs 1111 1 366 2 136 3 54 4 17 5 2 6 After patch: Freq #CPUs 1139 1 332 2 139 3 44 4 12 5 2 6 i.e. apparently not a large difference, but still a large proportion of cases where multiple CPUs are woken at once on the same chain. Kris --EeQfGwPcQSOJBaQU Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXnz+Wry0BWjoQKURAt9bAJ98pfbOhmC9J0RECAq9WEB9IEi5mwCfQkp7 rN1OtiYv5coGIatwx6cntG4= =9lr+ -----END PGP SIGNATURE----- --EeQfGwPcQSOJBaQU-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 23:22:11 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id BAB7F16A403; Sun, 7 May 2006 23:22:10 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Mon, 8 May 2006 07:22:06 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> In-Reply-To: <20060507230430.GA6872@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605080722.06435.davidxu@freebsd.org> Cc: Robert Watson , performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 23:22:11 -0000 On Monday 08 May 2006 07:04, Kris Kennaway wrote: > i.e. apparently not a large difference, but still a large proportion > of cases where multiple CPUs are woken at once on the same chain. > > Kris This becauses there is no sleepable mutex available, so I had to use msleep and wakeup, this is suboptimal, I may put flag MTX_QUIET there to let WITNESS shut up. From owner-freebsd-performance@FreeBSD.ORG Sun May 7 23:22:11 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id BAB7F16A403; Sun, 7 May 2006 23:22:10 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Mon, 8 May 2006 07:22:06 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> In-Reply-To: <20060507230430.GA6872@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605080722.06435.davidxu@freebsd.org> Cc: Robert Watson , performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 23:22:11 -0000 On Monday 08 May 2006 07:04, Kris Kennaway wrote: > i.e. apparently not a large difference, but still a large proportion > of cases where multiple CPUs are woken at once on the same chain. > > Kris This becauses there is no sleepable mutex available, so I had to use msleep and wakeup, this is suboptimal, I may put flag MTX_QUIET there to let WITNESS shut up. From owner-freebsd-performance@FreeBSD.ORG Sun May 7 23:23:58 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D399516A454; Sun, 7 May 2006 23:23:58 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id ECCCC43DB2; Sun, 7 May 2006 23:23:33 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 4FDF51A3C1A; Sun, 7 May 2006 16:23:32 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id B8B1B516B8; Sun, 7 May 2006 19:23:31 -0400 (EDT) Date: Sun, 7 May 2006 19:23:31 -0400 From: Kris Kennaway To: David Xu Message-ID: <20060507232331.GB7670@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <200605080722.06435.davidxu@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="St7VIuEGZ6dlpu13" Content-Disposition: inline In-Reply-To: <200605080722.06435.davidxu@freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: Robert Watson , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 23:23:59 -0000 --St7VIuEGZ6dlpu13 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2006 at 07:22:06AM +0800, David Xu wrote: > On Monday 08 May 2006 07:04, Kris Kennaway wrote: >=20 > > i.e. apparently not a large difference, but still a large proportion > > of cases where multiple CPUs are woken at once on the same chain. > > > > Kris > This becauses there is no sleepable mutex available, so I had to use > msleep and wakeup, this is suboptimal, I may put flag MTX_QUIET there > to let WITNESS shut up. Can you use wakeup_one()? Kris --St7VIuEGZ6dlpu13 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXoFzWry0BWjoQKURAgVVAJ9oD9FK+zYc12YEJzOZfAN8MmzapwCgttR9 VeRXaBbj3eWlKdlBegnB0HE= =o3CD -----END PGP SIGNATURE----- --St7VIuEGZ6dlpu13-- From owner-freebsd-performance@FreeBSD.ORG Sun May 7 23:23:58 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D399516A454; Sun, 7 May 2006 23:23:58 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id ECCCC43DB2; Sun, 7 May 2006 23:23:33 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 4FDF51A3C1A; Sun, 7 May 2006 16:23:32 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id B8B1B516B8; Sun, 7 May 2006 19:23:31 -0400 (EDT) Date: Sun, 7 May 2006 19:23:31 -0400 From: Kris Kennaway To: David Xu Message-ID: <20060507232331.GB7670@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <200605080722.06435.davidxu@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="St7VIuEGZ6dlpu13" Content-Disposition: inline In-Reply-To: <200605080722.06435.davidxu@freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: Robert Watson , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 May 2006 23:23:59 -0000 --St7VIuEGZ6dlpu13 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2006 at 07:22:06AM +0800, David Xu wrote: > On Monday 08 May 2006 07:04, Kris Kennaway wrote: >=20 > > i.e. apparently not a large difference, but still a large proportion > > of cases where multiple CPUs are woken at once on the same chain. > > > > Kris > This becauses there is no sleepable mutex available, so I had to use > msleep and wakeup, this is suboptimal, I may put flag MTX_QUIET there > to let WITNESS shut up. Can you use wakeup_one()? Kris --St7VIuEGZ6dlpu13 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXoFzWry0BWjoQKURAgVVAJ9oD9FK+zYc12YEJzOZfAN8MmzapwCgttR9 VeRXaBbj3eWlKdlBegnB0HE= =o3CD -----END PGP SIGNATURE----- --St7VIuEGZ6dlpu13-- From owner-freebsd-performance@FreeBSD.ORG Mon May 8 01:34:22 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D22B716A401; Mon, 8 May 2006 01:34:22 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from mail.localelinks.com (web.localelinks.com [64.39.75.54]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0A29E43D48; Mon, 8 May 2006 01:34:21 +0000 (GMT) (envelope-from fullermd@over-yonder.net) Received: from draco.over-yonder.net (adsl-072-148-013-213.sip.jan.bellsouth.net [72.148.13.213]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.localelinks.com (Postfix) with ESMTP id 111323DB; Sun, 7 May 2006 20:34:21 -0500 (CDT) Received: by draco.over-yonder.net (Postfix, from userid 100) id 166B061C2B; Sun, 7 May 2006 20:34:20 -0500 (CDT) Date: Sun, 7 May 2006 20:34:20 -0500 From: "Matthew D. Fuller" To: Robert Watson Message-ID: <20060508013419.GD4202@over-yonder.net> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <445E3CBE.1040105@rogers.com> <20060507193651.O46997@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060507193651.O46997@fledge.watson.org> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.11-fullermd.3 Cc: Mike Jakubik , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 01:34:23 -0000 On Sun, May 07, 2006 at 07:47:27PM +0100 I heard the voice of Robert Watson, and lo! it spake thus: > > In past discussion, I think a reasonable conclusion has been some > amount of both. I've still got sitting around in my bookmarks from a previous iteration. Point (5) makes me twitch every time I read it ;) -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-performance@FreeBSD.ORG Mon May 8 01:34:22 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D22B716A401; Mon, 8 May 2006 01:34:22 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from mail.localelinks.com (web.localelinks.com [64.39.75.54]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0A29E43D48; Mon, 8 May 2006 01:34:21 +0000 (GMT) (envelope-from fullermd@over-yonder.net) Received: from draco.over-yonder.net (adsl-072-148-013-213.sip.jan.bellsouth.net [72.148.13.213]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.localelinks.com (Postfix) with ESMTP id 111323DB; Sun, 7 May 2006 20:34:21 -0500 (CDT) Received: by draco.over-yonder.net (Postfix, from userid 100) id 166B061C2B; Sun, 7 May 2006 20:34:20 -0500 (CDT) Date: Sun, 7 May 2006 20:34:20 -0500 From: "Matthew D. Fuller" To: Robert Watson Message-ID: <20060508013419.GD4202@over-yonder.net> References: <20060506150622.C17611@fledge.watson.org> <200605071949.54978.hadara@bsd.ee> <445E3CBE.1040105@rogers.com> <20060507193651.O46997@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060507193651.O46997@fledge.watson.org> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.11-fullermd.3 Cc: Mike Jakubik , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 01:34:23 -0000 On Sun, May 07, 2006 at 07:47:27PM +0100 I heard the voice of Robert Watson, and lo! it spake thus: > > In past discussion, I think a reasonable conclusion has been some > amount of both. I've still got sitting around in my bookmarks from a previous iteration. Point (5) makes me twitch every time I read it ;) -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-performance@FreeBSD.ORG Mon May 8 06:52:11 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2270116A401; Mon, 8 May 2006 06:52:11 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id BE69C43D48; Mon, 8 May 2006 06:52:10 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 4034C1A3C1A; Sun, 7 May 2006 23:52:10 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id CA2D85169A; Mon, 8 May 2006 02:52:08 -0400 (EDT) Date: Mon, 8 May 2006 02:52:08 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060508065207.GA20386@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="r5Pyd7+fXNt84Ff3" Content-Disposition: inline In-Reply-To: <20060507230430.GA6872@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 06:52:11 -0000 --r5Pyd7+fXNt84Ff3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline OK, David's patch fixes the umtx thundering herd (and seems to give a 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which was also waking up 2-7 CPUs at once about 30% of the time) by doing s/wakeup/wakeup_one/. This did not seem to give a performance impact on this test though. It seems to me that a more useful way to sort the mutex profiling list is by ration of contention to total acquisitions. Here is the list resorted by cnt_hold/count, keeping only the top 40 values of count and the mutexes with nonzero contention: Before: max total count avg cnt_hold cnt_lock ratio name 275 507115 166457 3 907 1348 .005 kern/vfs_bio.c:357 (needsbuffer lock) 310 487209 166460 2 1158 645 .006 kern/vfs_bio.c:315 (needsbuffer lock) 1084 3860336 166507 23 1241 1377 .007 kern/vfs_bio.c:1445 (buf queue lock) 1667 35018604 320038 109 3877 0 .012 kern/uipc_usrreq.c:696 (unp_mtx) 379 2143505 635740 3 10736 37083 .016 kern/sys_socket.c:176 (so_snd) 1503 4311935 502656 8 8664 9312 .017 kern/kern_lock.c:163 (lockbuilder mtxpool) 875 3495175 166487 20 3394 4272 .020 kern/vfs_bio.c:2424 (vnode interlock) 2084 121390320 2880081 42 67339 79525 .023 kern/uipc_usrreq.c:581 (so_snd) 909 1809346 165769 10 4454 9597 .026 kern/vfs_vnops.c:796 (vnode interlock) 277 518716 166442 3 5034 5172 .030 kern/vfs_bio.c:1464 (vnode interlock) 1565 10515648 282278 37 15760 10821 .055 kern/subr_sleepqueue.c:374 (process lock) 492 2500241 634835 3 54003 62520 .085 kern/kern_sig.c:1002 (process lock) 569 335913 30022 11 3262 2176 .108 kern/kern_sx.c:245 (lockbuilder mtxpool) 1378 27840143 320038 86 42183 1453 .131 kern/uipc_usrreq.c:705 (so_rcv) 300 1011100 320045 3 52423 30742 .163 kern/uipc_socket.c:1101 (so_snd) 437 10472850 3200213 3 576918 615361 .180 kern/kern_resource.c:1172 (sleep mtxpool) 2052 46242974 320039 144 80690 80729 .252 kern/uipc_usrreq.c:617 (unp_global_mtx) 546 48160602 3683470 13 1488801 696814 .404 kern/kern_descrip.c:1988 (filedesc structure) 395 13842967 3683470 3 1568927 685295 .425 kern/kern_descrip.c:1967 (filedesc structure) 644 16700212 635731 26 606615 278511 .954 kern/kern_descrip.c:420 (filedesc structure) 384 2863741 635774 4 654035 280340 1.028 kern/kern_descrip.c:368 (filedesc structure) 604 22164433 2721994 8 5564709 2225496 2.044 kern/kern_synch.c:220 (process lock) After: max total count avg cnt_hold cnt_lock ratio name 168 467413 166364 2 1025 2655 .006 kern/vfs_bio.c:357 (needsbuffer lock) 264 453972 166364 2 1688 44 .010 kern/vfs_bio.c:315 (needsbuffer lock) 240 2011519 640106 3 12032 48460 .018 kern/sys_socket.c:176 (so_snd) 425 5394174 514469 10 12838 15343 .024 kern/kern_lock.c:163 (lockbuilder mtxpool) 514 5127131 166383 30 4417 5666 .026 kern/vfs_bio.c:1445 (buf queue lock) 261 199860 38442 5 1405 475 .036 kern/kern_sx.c:245 (lockbuilder mtxpool) 707 174604101 2880083 60 119723 84566 .041 kern/uipc_usrreq.c:581 (so_snd) 126 520485 166351 3 7850 8574 .047 kern/vfs_bio.c:1464 (vnode interlock) 364 1850567 165607 11 8077 22156 .048 kern/vfs_vnops.c:796 (vnode interlock) 499 3233479 166432 19 9258 8468 .055 kern/vfs_bio.c:2424 (vnode interlock) 754 42181810 320038 131 21236 0 .066 kern/uipc_usrreq.c:696 (unp_mtx) 462 21081419 3685605 5 316514 243585 .085 kern/kern_descrip.c:1988 (filedesc structure) 577 12178436 321182 37 28585 21082 .088 kern/subr_sleepqueue.c:374 (process lock) 221 2410704 640387 3 75056 77553 .117 kern/kern_sig.c:1002 (process lock) 309 12026860 3685605 3 468707 331121 .127 kern/kern_descrip.c:1967 (filedesc structure) 299 973885 320046 3 60629 72506 .189 kern/uipc_socket.c:1101 (so_snd) 471 6132557 640097 9 125478 98778 .196 kern/kern_descrip.c:420 (filedesc structure) 737 33114067 320038 103 85243 1 .266 kern/uipc_usrreq.c:705 (so_rcv) 454 5866777 878113 6 240669 364921 .274 kern/kern_synch.c:220 (process lock) 365 2308060 640133 3 183152 142569 .286 kern/kern_descrip.c:368 (filedesc structure) 220 10297249 3200211 3 1117448 1175412 .349 kern/kern_resource.c:1172 (sleep mtxpool) 947 57806295 320040 180 132456 109179 .413 kern/uipc_usrreq.c:617 (unp_global_mtx) filedesc contention is down by a factor of 3-4, with corresponding reduction in the average hold time. The process lock contention coming from the signal delivery wakeup has also gone way down for some reason. unp contention has risen a bit. The other big gain is to sleep mtxpool contention, which roughly doubled: /* * Change the total socket buffer size a user has used. */ int chgsbsize(uip, hiwat, to, max) struct uidinfo *uip; u_int *hiwat; u_int to; rlim_t max; { rlim_t new; UIDINFO_LOCK(uip); So the next question is how can that be optimized? Kris --r5Pyd7+fXNt84Ff3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEXuqXWry0BWjoQKURAooFAKDolGQ8HXdVW06/t3LYDdllYL/TAgCfanPu b/FT/nnB0xb6Lon1bQJi2Cs= =Db4k -----END PGP SIGNATURE----- --r5Pyd7+fXNt84Ff3-- From owner-freebsd-performance@FreeBSD.ORG Mon May 8 07:35:13 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5168C16A404; Mon, 8 May 2006 07:35:13 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 02D9B43D53; Mon, 8 May 2006 07:35:11 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 621F146C13; Mon, 8 May 2006 03:35:11 -0400 (EDT) Date: Mon, 8 May 2006 08:35:11 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kris Kennaway In-Reply-To: <20060508065207.GA20386@xor.obsecurity.org> Message-ID: <20060508083359.I17611@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: performance@FreeBSD.org, current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 07:35:13 -0000 On Mon, 8 May 2006, Kris Kennaway wrote: > unp contention has risen a bit. The other big gain is to sleep mtxpool > contention, which roughly doubled: In the general case, you can increase the size of the mutex pool. However, since this is per-uid for the socket buffer resource limits, that won't help you as the same user will always contend with itself. We'll need to think about this some -- good activity for the plane today :-). Nice work so far, btw. Sounds like we're making real headway! Robert N M Watson > > /* > * Change the total socket buffer size a user has used. > */ > int > chgsbsize(uip, hiwat, to, max) > struct uidinfo *uip; > u_int *hiwat; > u_int to; > rlim_t max; > { > rlim_t new; > > UIDINFO_LOCK(uip); > > So the next question is how can that be optimized? > > Kris From owner-freebsd-performance@FreeBSD.ORG Mon May 8 10:43:37 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id D34D816A402; Mon, 8 May 2006 10:43:36 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Mon, 8 May 2006 18:43:31 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> In-Reply-To: <20060508065207.GA20386@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605081843.31825.davidxu@freebsd.org> Cc: Robert Watson , performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 10:43:37 -0000 On Monday 08 May 2006 14:52, Kris Kennaway wrote: > OK, David's patch fixes the umtx thundering herd (and seems to give a > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > was also waking up 2-7 CPUs at once about 30% of the time) by doing > s/wakeup/wakeup_one/. This did not seem to give a performance impact > on this test though. >.... > filedesc contention is down by a factor of 3-4, with corresponding > reduction in the average hold time. The process lock contention > coming from the signal delivery wakeup has also gone way down for some > reason. > I found that mysqld frequently calls alarm() in its file thr_alarm.c and thr_kill() to send SIGALRM to its timer thread to wake it up, the timer thread itself is being blocked in sigwait(), normally the alarm timer will be expired in a second, so the kernel will periodically call psignal to find a thread which can handle the signal, it means kernel has to periodically walk through thread list with process lock and scheduler held, this is very expensive. thr_kill will in most time wake up the timer thread earlier, in thr_kill syscall, kernel has to walk through thread list to find a thread whose thread is matching the given id, the function thread_find() uses a linear searching algorithm, it is slow, if there are lots of thread in the process, the process lock will be holden too long, I think that's the reason why you have seen so many process lock contention, if you define USE_ALARM_THREAD in mysql header file, the contention should be decreased ( I hope ), patch: --- my_pthread.h.old Mon May 8 18:16:56 2006 +++ my_pthread.h Mon May 8 18:17:07 2006 @@ -267,6 +267,8 @@ /* Test first for RTS or FSU threads */ +#define USE_ALARM_THREAD + #if defined(PTHREAD_SCOPE_GLOBAL) && !defined(PTHREAD_SCOPE_SYSTEM) #define HAVE_rts_threads extern int my_pthread_create_detached; > unp contention has risen a bit. The other big gain is to sleep > mtxpool contention, which roughly doubled: > > /* > * Change the total socket buffer size a user has used. > */ > int > chgsbsize(uip, hiwat, to, max) > struct uidinfo *uip; > u_int *hiwat; > u_int to; > rlim_t max; > { > rlim_t new; > > UIDINFO_LOCK(uip); > > So the next question is how can that be optimized? > may use atomic_cmpset_int in a loop to avoid context switch or use an adaptive mutex, but there is no adaptive mutex type you can specify. rlim_t is a 64bit integer, so atomic operation can not be used, but 64bit integer might not be necessary for socket buffer size. > Kris David Xu From owner-freebsd-performance@FreeBSD.ORG Mon May 8 10:43:37 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id D34D816A402; Mon, 8 May 2006 10:43:36 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Mon, 8 May 2006 18:43:31 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> In-Reply-To: <20060508065207.GA20386@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605081843.31825.davidxu@freebsd.org> Cc: Robert Watson , performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 10:43:37 -0000 On Monday 08 May 2006 14:52, Kris Kennaway wrote: > OK, David's patch fixes the umtx thundering herd (and seems to give a > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > was also waking up 2-7 CPUs at once about 30% of the time) by doing > s/wakeup/wakeup_one/. This did not seem to give a performance impact > on this test though. >.... > filedesc contention is down by a factor of 3-4, with corresponding > reduction in the average hold time. The process lock contention > coming from the signal delivery wakeup has also gone way down for some > reason. > I found that mysqld frequently calls alarm() in its file thr_alarm.c and thr_kill() to send SIGALRM to its timer thread to wake it up, the timer thread itself is being blocked in sigwait(), normally the alarm timer will be expired in a second, so the kernel will periodically call psignal to find a thread which can handle the signal, it means kernel has to periodically walk through thread list with process lock and scheduler held, this is very expensive. thr_kill will in most time wake up the timer thread earlier, in thr_kill syscall, kernel has to walk through thread list to find a thread whose thread is matching the given id, the function thread_find() uses a linear searching algorithm, it is slow, if there are lots of thread in the process, the process lock will be holden too long, I think that's the reason why you have seen so many process lock contention, if you define USE_ALARM_THREAD in mysql header file, the contention should be decreased ( I hope ), patch: --- my_pthread.h.old Mon May 8 18:16:56 2006 +++ my_pthread.h Mon May 8 18:17:07 2006 @@ -267,6 +267,8 @@ /* Test first for RTS or FSU threads */ +#define USE_ALARM_THREAD + #if defined(PTHREAD_SCOPE_GLOBAL) && !defined(PTHREAD_SCOPE_SYSTEM) #define HAVE_rts_threads extern int my_pthread_create_detached; > unp contention has risen a bit. The other big gain is to sleep > mtxpool contention, which roughly doubled: > > /* > * Change the total socket buffer size a user has used. > */ > int > chgsbsize(uip, hiwat, to, max) > struct uidinfo *uip; > u_int *hiwat; > u_int to; > rlim_t max; > { > rlim_t new; > > UIDINFO_LOCK(uip); > > So the next question is how can that be optimized? > may use atomic_cmpset_int in a loop to avoid context switch or use an adaptive mutex, but there is no adaptive mutex type you can specify. rlim_t is a 64bit integer, so atomic operation can not be used, but 64bit integer might not be necessary for socket buffer size. > Kris David Xu From owner-freebsd-performance@FreeBSD.ORG Mon May 8 18:43:32 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF80616A400; Mon, 8 May 2006 18:43:32 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A37243D4C; Mon, 8 May 2006 18:43:27 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 271391A3C1A; Mon, 8 May 2006 11:43:25 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 73733515DC; Mon, 8 May 2006 14:43:22 -0400 (EDT) Date: Mon, 8 May 2006 14:43:22 -0400 From: Kris Kennaway To: David Xu Message-ID: <20060508184322.GA49344@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <200605081843.31825.davidxu@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3V7upXqbjpZ4EhLz" Content-Disposition: inline In-Reply-To: <200605081843.31825.davidxu@freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: performance@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 18:43:32 -0000 --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2006 at 06:43:31PM +0800, David Xu wrote: > On Monday 08 May 2006 14:52, Kris Kennaway wrote: > > OK, David's patch fixes the umtx thundering herd (and seems to give a > > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > > was also waking up 2-7 CPUs at once about 30% of the time) by doing > > s/wakeup/wakeup_one/. This did not seem to give a performance impact > > on this test though. > >.... > > filedesc contention is down by a factor of 3-4, with corresponding > > reduction in the average hold time. The process lock contention > > coming from the signal delivery wakeup has also gone way down for some > > reason. > > >=20 > I found that mysqld frequently calls alarm() in its file thr_alarm.c and= =20 > thr_kill() to send SIGALRM to its timer thread to wake it up, the timer= =20 > thread itself is being blocked in sigwait(), normally the alarm timer will > be expired in a second, so the kernel will periodically call psignal to f= ind > a thread which can handle the signal, it means kernel has to periodically > walk through thread list with process lock and scheduler held, this is > very expensive. >=20 > thr_kill will in most time wake up the timer thread earlier, in thr_kill > syscall, kernel has to walk through thread list to find a thread whose > thread is matching the given id, the function thread_find() > uses a linear searching algorithm, it is slow, if there are lots of threa= d in > the process, the process lock will be holden too long, I think that's th= e=20 > reason why you have seen so many process lock contention, if you > define USE_ALARM_THREAD in mysql header file, the contention should > be decreased ( I hope ), patch: Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't yet had time to investigate. Is anyone else seeing this? Kris --3V7upXqbjpZ4EhLz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEX5FJWry0BWjoQKURAs44AJ9xvSlgI2Ged4dpfI5Au728e4UxJQCfQ+MG Re10oKdTr9I47XG0rxx3yvA= =xvgt -----END PGP SIGNATURE----- --3V7upXqbjpZ4EhLz-- From owner-freebsd-performance@FreeBSD.ORG Mon May 8 18:43:32 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF80616A400; Mon, 8 May 2006 18:43:32 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A37243D4C; Mon, 8 May 2006 18:43:27 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 271391A3C1A; Mon, 8 May 2006 11:43:25 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 73733515DC; Mon, 8 May 2006 14:43:22 -0400 (EDT) Date: Mon, 8 May 2006 14:43:22 -0400 From: Kris Kennaway To: David Xu Message-ID: <20060508184322.GA49344@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <200605081843.31825.davidxu@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3V7upXqbjpZ4EhLz" Content-Disposition: inline In-Reply-To: <200605081843.31825.davidxu@freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: performance@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 18:43:32 -0000 --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2006 at 06:43:31PM +0800, David Xu wrote: > On Monday 08 May 2006 14:52, Kris Kennaway wrote: > > OK, David's patch fixes the umtx thundering herd (and seems to give a > > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > > was also waking up 2-7 CPUs at once about 30% of the time) by doing > > s/wakeup/wakeup_one/. This did not seem to give a performance impact > > on this test though. > >.... > > filedesc contention is down by a factor of 3-4, with corresponding > > reduction in the average hold time. The process lock contention > > coming from the signal delivery wakeup has also gone way down for some > > reason. > > >=20 > I found that mysqld frequently calls alarm() in its file thr_alarm.c and= =20 > thr_kill() to send SIGALRM to its timer thread to wake it up, the timer= =20 > thread itself is being blocked in sigwait(), normally the alarm timer will > be expired in a second, so the kernel will periodically call psignal to f= ind > a thread which can handle the signal, it means kernel has to periodically > walk through thread list with process lock and scheduler held, this is > very expensive. >=20 > thr_kill will in most time wake up the timer thread earlier, in thr_kill > syscall, kernel has to walk through thread list to find a thread whose > thread is matching the given id, the function thread_find() > uses a linear searching algorithm, it is slow, if there are lots of threa= d in > the process, the process lock will be holden too long, I think that's th= e=20 > reason why you have seen so many process lock contention, if you > define USE_ALARM_THREAD in mysql header file, the contention should > be decreased ( I hope ), patch: Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't yet had time to investigate. Is anyone else seeing this? Kris --3V7upXqbjpZ4EhLz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEX5FJWry0BWjoQKURAs44AJ9xvSlgI2Ged4dpfI5Au728e4UxJQCfQ+MG Re10oKdTr9I47XG0rxx3yvA= =xvgt -----END PGP SIGNATURE----- --3V7upXqbjpZ4EhLz-- From owner-freebsd-performance@FreeBSD.ORG Mon May 8 22:43:36 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 2A29D16A404; Mon, 8 May 2006 22:43:36 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Kris Kennaway Date: Tue, 9 May 2006 06:43:31 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <200605081843.31825.davidxu@freebsd.org> <20060508184322.GA49344@xor.obsecurity.org> In-Reply-To: <20060508184322.GA49344@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605090643.31434.davidxu@freebsd.org> Cc: performance@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 22:43:36 -0000 On Tuesday 09 May 2006 02:43, Kris Kennaway wrote: > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > yet had time to investigate. Is anyone else seeing this? > > Kris I only have tested mysql 4.0, I will try 4.1 later. From owner-freebsd-performance@FreeBSD.ORG Mon May 8 22:43:36 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 2A29D16A404; Mon, 8 May 2006 22:43:36 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: Kris Kennaway Date: Tue, 9 May 2006 06:43:31 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <200605081843.31825.davidxu@freebsd.org> <20060508184322.GA49344@xor.obsecurity.org> In-Reply-To: <20060508184322.GA49344@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605090643.31434.davidxu@freebsd.org> Cc: performance@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 22:43:36 -0000 On Tuesday 09 May 2006 02:43, Kris Kennaway wrote: > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > yet had time to investigate. Is anyone else seeing this? > > Kris I only have tested mysql 4.0, I will try 4.1 later. From owner-freebsd-performance@FreeBSD.ORG Mon May 8 23:56:35 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 970A416A45C for ; Mon, 8 May 2006 23:56:35 +0000 (UTC) (envelope-from mikej@rogers.com) Received: from smtp105.rog.mail.re2.yahoo.com (smtp105.rog.mail.re2.yahoo.com [206.190.36.83]) by mx1.FreeBSD.org (Postfix) with SMTP id 5E6DD43D49 for ; Mon, 8 May 2006 23:56:33 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 44601 invoked from network); 8 May 2006 23:56:32 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=vUIuVYH8tAvRiGtlCeD+l6eo5uMI1uU29gErbbzKjLgLa41Tz5jw/+iTgf4VL6yZwb3THsh6rpUAYOHGEBpRfchSJ1LNruHjsEqw4iAqncCtwcvXheP8BzyG5Rrzi4rz/igGQlmEd6pFy0XwnAq74wuzKire6mu4tm5IBy0q0SE= ; Received: from unknown (HELO ?70.31.50.218?) (mikej@rogers.com@70.31.50.218 with plain) by smtp105.rog.mail.re2.yahoo.com with SMTP; 8 May 2006 23:56:32 -0000 Message-ID: <445FDAC1.6000300@rogers.com> Date: Mon, 08 May 2006 19:56:49 -0400 From: Mike Jakubik User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: David Xu References: <20060506150622.C17611@fledge.watson.org> <200605081843.31825.davidxu@freebsd.org> <20060508184322.GA49344@xor.obsecurity.org> <200605090643.31434.davidxu@freebsd.org> In-Reply-To: <200605090643.31434.davidxu@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Robert Watson , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 23:56:36 -0000 David Xu wrote: > On Tuesday 09 May 2006 02:43, Kris Kennaway wrote: > > >> Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't >> yet had time to investigate. Is anyone else seeing this? >> >> Kris >> > > I only have tested mysql 4.0, I will try 4.1 later. > What about 5.0? Have any of you done tests on this version? I'm curious to know the difference in performance between 4 and 5 on freebsd. From owner-freebsd-performance@FreeBSD.ORG Mon May 8 23:56:35 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9225C16A459 for ; Mon, 8 May 2006 23:56:35 +0000 (UTC) (envelope-from mikej@rogers.com) Received: from smtp105.rog.mail.re2.yahoo.com (smtp105.rog.mail.re2.yahoo.com [206.190.36.83]) by mx1.FreeBSD.org (Postfix) with SMTP id 5E26843D45 for ; Mon, 8 May 2006 23:56:33 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 44601 invoked from network); 8 May 2006 23:56:32 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=vUIuVYH8tAvRiGtlCeD+l6eo5uMI1uU29gErbbzKjLgLa41Tz5jw/+iTgf4VL6yZwb3THsh6rpUAYOHGEBpRfchSJ1LNruHjsEqw4iAqncCtwcvXheP8BzyG5Rrzi4rz/igGQlmEd6pFy0XwnAq74wuzKire6mu4tm5IBy0q0SE= ; Received: from unknown (HELO ?70.31.50.218?) (mikej@rogers.com@70.31.50.218 with plain) by smtp105.rog.mail.re2.yahoo.com with SMTP; 8 May 2006 23:56:32 -0000 Message-ID: <445FDAC1.6000300@rogers.com> Date: Mon, 08 May 2006 19:56:49 -0400 From: Mike Jakubik User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: David Xu References: <20060506150622.C17611@fledge.watson.org> <200605081843.31825.davidxu@freebsd.org> <20060508184322.GA49344@xor.obsecurity.org> <200605090643.31434.davidxu@freebsd.org> In-Reply-To: <200605090643.31434.davidxu@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Robert Watson , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2006 23:56:36 -0000 David Xu wrote: > On Tuesday 09 May 2006 02:43, Kris Kennaway wrote: > > >> Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't >> yet had time to investigate. Is anyone else seeing this? >> >> Kris >> > > I only have tested mysql 4.0, I will try 4.1 later. > What about 5.0? Have any of you done tests on this version? I'm curious to know the difference in performance between 4 and 5 on freebsd. From owner-freebsd-performance@FreeBSD.ORG Tue May 9 00:35:18 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8627016A44B; Tue, 9 May 2006 00:35:18 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx1.starman.ee (smtp-out1.starman.ee [85.253.0.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id B354543D8F; Tue, 9 May 2006 00:35:12 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx1.starman.ee (Postfix) with ESMTP id 8AB7123C157; Tue, 9 May 2006 03:35:08 +0300 (EEST) From: Sven Petai To: freebsd-performance@freebsd.org Date: Tue, 9 May 2006 03:34:59 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> <200605081843.31825.davidxu@freebsd.org> <20060508184322.GA49344@xor.obsecurity.org> In-Reply-To: <20060508184322.GA49344@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605090335.00451.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx1.starman.ee Cc: Kris Kennaway , David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 00:35:21 -0000 > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > yet had time to investigate. Is anyone else seeing this? > Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make much difference though. I ran the tests again on the 8 core machine with and without rwatsons patch and this time with 6 tests for each setting and generated graphs: http://bsd.ee/~hadara/debug/mysql4/stats.html thr + select smack dynamics with the patch really do look quite interesting PS I'm currently running testround with rwatsons patch + http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff + mysqld change davidxu suggested. Are there any other patches out there that I should try ? From owner-freebsd-performance@FreeBSD.ORG Tue May 9 00:42:07 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0FDA316A411; Tue, 9 May 2006 00:42:07 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id BA0C743D45; Tue, 9 May 2006 00:42:06 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 91FF91A3C1F; Mon, 8 May 2006 17:42:06 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id C176D515DC; Mon, 8 May 2006 20:42:03 -0400 (EDT) Date: Mon, 8 May 2006 20:42:03 -0400 From: Kris Kennaway To: Sven Petai Message-ID: <20060509004203.GA55852@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605081843.31825.davidxu@freebsd.org> <20060508184322.GA49344@xor.obsecurity.org> <200605090335.00451.hadara@bsd.ee> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BXVAT5kNtrzKuDFl" Content-Disposition: inline In-Reply-To: <200605090335.00451.hadara@bsd.ee> User-Agent: Mutt/1.4.2.1i Cc: freebsd-performance@freebsd.org, Kris Kennaway , David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 00:42:07 -0000 --BXVAT5kNtrzKuDFl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > > yet had time to investigate. Is anyone else seeing this? > > >=20 > Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make muc= h=20 > difference though. >=20 > I ran the tests again on the 8 core machine with and without rwatsons pat= ch=20 > and this time with 6 tests for each setting and generated graphs: > http://bsd.ee/~hadara/debug/mysql4/stats.html >=20 > thr + select smack dynamics with the patch really do look quite interesti= ng >=20 > PS > I'm currently running testround with rwatsons patch +=20 > http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff + mysqld= =20 > change davidxu suggested.=20 > Are there any other patches out there that I should try ? Sorry, I meant http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff instead of that patch. Also try turning down HZ to 100, and changing the wakeup()s to wakeup_one() in sys/filedesc.h. A mutex profiling trace from the 8 core machine will also be very interesting. Kris --BXVAT5kNtrzKuDFl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEX+VbWry0BWjoQKURAgm/AKD2YwlrftVlpb7kN7QeIQIJKl2bmACg78i2 abDlSzbEkADZRGjNfX8iNk0= =5TPT -----END PGP SIGNATURE----- --BXVAT5kNtrzKuDFl-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 00:43:39 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B30BF16A479; Tue, 9 May 2006 00:43:39 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id B313943D5C; Tue, 9 May 2006 00:43:29 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 9AAB21A3C1F; Mon, 8 May 2006 17:43:29 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 350BE51664; Mon, 8 May 2006 20:43:28 -0400 (EDT) Date: Mon, 8 May 2006 20:43:28 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060509004328.GB55852@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="WYTEVAkct0FjGQmd" Content-Disposition: inline In-Reply-To: <20060508065207.GA20386@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 00:43:40 -0000 --WYTEVAkct0FjGQmd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > OK, David's patch fixes the umtx thundering herd (and seems to give a > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > was also waking up 2-7 CPUs at once about 30% of the time) by doing > s/wakeup/wakeup_one/. This did not seem to give a performance impact > on this test though. Turning down kern.hz from 1000 to 100 also made a big difference on 12 CPUs (+6.1%). Note also that the system is no less than 40% idle during the runs (at any load), so the bottlenecks are serious. Kris --WYTEVAkct0FjGQmd Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEX+WvWry0BWjoQKURAt8YAKCPaRA18V5W8hP+Oafj2+7gPDJd2QCfeNaW vxHwPuOyQDZfJ9KVeJGH14o= =Ik2U -----END PGP SIGNATURE----- --WYTEVAkct0FjGQmd-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 01:11:15 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B30116A45A; Tue, 9 May 2006 01:11:15 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7B25043D6E; Tue, 9 May 2006 01:11:10 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 5AA501A3C1F; Mon, 8 May 2006 18:11:10 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id B566651742; Mon, 8 May 2006 21:11:09 -0400 (EDT) Date: Mon, 8 May 2006 21:11:09 -0400 From: Kris Kennaway To: Kris Kennaway Message-ID: <20060509011109.GA57060@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <20060509004328.GB55852@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="OXfL5xGRrasGEqWY" Content-Disposition: inline In-Reply-To: <20060509004328.GB55852@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 01:11:16 -0000 --OXfL5xGRrasGEqWY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote: > On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > > OK, David's patch fixes the umtx thundering herd (and seems to give a > > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > > was also waking up 2-7 CPUs at once about 30% of the time) by doing > > s/wakeup/wakeup_one/. This did not seem to give a performance impact > > on this test though. >=20 > Turning down kern.hz from 1000 to 100 also made a big difference on 12 > CPUs (+6.1%). >=20 > Note also that the system is no less than 40% idle during the runs (at > any load), so the bottlenecks are serious. top -H shows the threads mostly in umtx state. Kris --OXfL5xGRrasGEqWY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEX+wtWry0BWjoQKURApXBAKDG+HoSkP+edodHh+glPxAYokNvtgCg0KQ8 ACuFSrfTnqu3yauVgzLaCCM= =paPx -----END PGP SIGNATURE----- --OXfL5xGRrasGEqWY-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 01:21:07 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F4E116A40B; Tue, 9 May 2006 01:21:07 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id EB68943D46; Tue, 9 May 2006 01:21:06 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id BE2B71A3C1A; Mon, 8 May 2006 18:21:06 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 3AA4251664; Mon, 8 May 2006 21:21:06 -0400 (EDT) Date: Mon, 8 May 2006 21:21:06 -0400 From: Kris Kennaway To: Maxim Sobolev Message-ID: <20060509012106.GA57271@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <20060509004328.GB55852@xor.obsecurity.org> <445FEDDA.6010001@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="9jxsPFA5p3P2qPhR" Content-Disposition: inline In-Reply-To: <445FEDDA.6010001@FreeBSD.org> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 01:21:07 -0000 --9jxsPFA5p3P2qPhR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 08, 2006 at 06:18:18PM -0700, Maxim Sobolev wrote: > Kris Kennaway wrote: > >On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > >>OK, David's patch fixes the umtx thundering herd (and seems to give a > >>4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > >>was also waking up 2-7 CPUs at once about 30% of the time) by doing > >>s/wakeup/wakeup_one/. This did not seem to give a performance impact > >>on this test though. > > > >Turning down kern.hz from 1000 to 100 also made a big difference on 12 > >CPUs (+6.1%). > > > >Note also that the system is no less than 40% idle during the runs (at > >any load), so the bottlenecks are serious. >=20 > Maybe HDD just can't keep up with the pace? There is no disk I/O involved. Kris --9jxsPFA5p3P2qPhR Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEX+6BWry0BWjoQKURArFWAJ0XbBnww8FPfL/TD3fmr5/1MiBTEQCg0Xca wf4fD55J7ULkaas0a3GlpV4= =+Prk -----END PGP SIGNATURE----- --9jxsPFA5p3P2qPhR-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 01:18:32 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7B3D416A440; Tue, 9 May 2006 01:18:32 +0000 (UTC) (envelope-from sobomax@FreeBSD.org) Received: from sippysoft.com (gk.360sip.com [72.236.70.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6937543D49; Tue, 9 May 2006 01:18:31 +0000 (GMT) (envelope-from sobomax@FreeBSD.org) Received: from [192.168.1.73] ([204.244.149.125]) (authenticated bits=0) by sippysoft.com (8.13.6/8.13.6) with ESMTP id k491IRPL091048 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 8 May 2006 18:18:29 -0700 (PDT) (envelope-from sobomax@FreeBSD.org) Message-ID: <445FEDDA.6010001@FreeBSD.org> Date: Mon, 08 May 2006 18:18:18 -0700 From: Maxim Sobolev Organization: Sippy Software, Inc. User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: Kris Kennaway References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <20060509004328.GB55852@xor.obsecurity.org> In-Reply-To: <20060509004328.GB55852@xor.obsecurity.org> Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 09 May 2006 02:15:15 +0000 Cc: Robert Watson , performance@FreeBSD.org, current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 01:18:32 -0000 Kris Kennaway wrote: > On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: >> OK, David's patch fixes the umtx thundering herd (and seems to give a >> 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which >> was also waking up 2-7 CPUs at once about 30% of the time) by doing >> s/wakeup/wakeup_one/. This did not seem to give a performance impact >> on this test though. > > Turning down kern.hz from 1000 to 100 also made a big difference on 12 > CPUs (+6.1%). > > Note also that the system is no less than 40% idle during the runs (at > any load), so the bottlenecks are serious. Maybe HDD just can't keep up with the pace? -Maxim From owner-freebsd-performance@FreeBSD.ORG Tue May 9 01:24:24 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F1AE116A417; Tue, 9 May 2006 01:24:23 +0000 (UTC) (envelope-from sobomax@FreeBSD.org) Received: from sippysoft.com (gk.360sip.com [72.236.70.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7988943D45; Tue, 9 May 2006 01:24:23 +0000 (GMT) (envelope-from sobomax@FreeBSD.org) Received: from [192.168.1.73] ([204.244.149.125]) (authenticated bits=0) by sippysoft.com (8.13.6/8.13.6) with ESMTP id k491OKWs091125 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 8 May 2006 18:24:22 -0700 (PDT) (envelope-from sobomax@FreeBSD.org) Message-ID: <445FEF3B.3060509@FreeBSD.org> Date: Mon, 08 May 2006 18:24:11 -0700 From: Maxim Sobolev Organization: Sippy Software, Inc. User-Agent: Thunderbird 1.5.0.2 (Windows/20060308) MIME-Version: 1.0 To: Kris Kennaway References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <20060509004328.GB55852@xor.obsecurity.org> <445FEDDA.6010001@FreeBSD.org> <20060509012106.GA57271@xor.obsecurity.org> In-Reply-To: <20060509012106.GA57271@xor.obsecurity.org> Content-Type: text/plain; charset=KOI8-U; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 09 May 2006 02:15:46 +0000 Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 01:24:24 -0000 Kris Kennaway wrote: > On Mon, May 08, 2006 at 06:18:18PM -0700, Maxim Sobolev wrote: >> Kris Kennaway wrote: >>> On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: >>>> OK, David's patch fixes the umtx thundering herd (and seems to give a >>>> 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which >>>> was also waking up 2-7 CPUs at once about 30% of the time) by doing >>>> s/wakeup/wakeup_one/. This did not seem to give a performance impact >>>> on this test though. >>> Turning down kern.hz from 1000 to 100 also made a big difference on 12 >>> CPUs (+6.1%). >>> >>> Note also that the system is no less than 40% idle during the runs (at >>> any load), so the bottlenecks are serious. >> Maybe HDD just can't keep up with the pace? > > There is no disk I/O involved. Oh, sorry. -Maxim From owner-freebsd-performance@FreeBSD.ORG Tue May 9 15:14:48 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 45C1B16A405; Tue, 9 May 2006 15:14:48 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mail.neti.ee (mx1.elion.ee [194.126.101.123]) by mx1.FreeBSD.org (Postfix) with ESMTP id 378EF43D46; Tue, 9 May 2006 15:14:34 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from nat-155.nat (test.estpak.ee [194.126.115.47]) by Relayhost1.neti.ee (Postfix) with ESMTP id B9D061884F; Tue, 9 May 2006 18:14:27 +0300 (EEST) From: Sven Petai To: Kris Kennaway Date: Tue, 9 May 2006 18:18:11 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> In-Reply-To: <20060509004203.GA55852@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605091818.12676.hadara@bsd.ee> X-Virus-Scanned: by amavisd-new-2.2.1 (20041222) (Debian) at neti.ee Cc: Sven Petai , freebsd-performance@freebsd.org, David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 15:14:49 -0000 On Tuesday 09 May 2006 03:42, Kris Kennaway wrote: > On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > > > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > > > yet had time to investigate. Is anyone else seeing this? > > > > Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make > > much difference though. > > > > I ran the tests again on the 8 core machine with and without rwatsons > > patch and this time with 6 tests for each setting and generated graphs: > > http://bsd.ee/~hadara/debug/mysql4/stats.html > > > > thr + select smack dynamics with the patch really do look quite > > interesting > > > > PS > > I'm currently running testround with rwatsons patch + > > http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff + mysqld > > change davidxu suggested. > > Are there any other patches out there that I should try ? > > Sorry, I meant > http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff > instead of that patch. Also try turning down HZ to 100, and changing > the wakeup()s to wakeup_one() in sys/filedesc.h. I preformed additional tests with the settings you suggested, updated graphs are available @ http://bsd.ee/~hadara/debug/mysql4/stats.html * with rwatsons patch + mysqld change suggested by davidxu + http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff there doesnt seem to be much difference, but in general things seem to be couple of percent better mutex profiles for this configuration are available @ with 10 smack threads: http://bsd.ee/~hadara/debug/mysql4/freebsd_cur_ps2/mutex_freebsd_cur2_p2-select_10_10000.txt with 100 smack threads: http://bsd.ee/~hadara/debug/mysql4/freebsd_cur_ps2/mutex_freebsd_cur_p2-select_100_1000.txt * second run was with same patches, but HZ set to 100 this helps performance a lot but creates large fluxuations in select results with >20 threads + thr, sometimes difference between 2 runs with same settings was 10000+ q/s I'm currently still testing wakeup() -> wakeup_one() suggestion. In addition I did full testrun on linux to see how much this hardware is really capable of... graphs for that are available on the stats page too. From owner-freebsd-performance@FreeBSD.ORG Tue May 9 17:35:08 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D802016A638; Tue, 9 May 2006 17:35:07 +0000 (UTC) (envelope-from julian@elischer.org) Received: from a50.ironport.com (a50.ironport.com [63.251.108.112]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7A3D443D46; Tue, 9 May 2006 17:35:07 +0000 (GMT) (envelope-from julian@elischer.org) Received: from unknown (HELO [10.251.19.131]) ([10.251.19.131]) by a50.ironport.com with ESMTP; 09 May 2006 10:35:08 -0700 Message-ID: <4460D2CA.5090808@elischer.org> Date: Tue, 09 May 2006 10:35:06 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.13) Gecko/20060414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Sven Petai References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> In-Reply-To: <200605091818.12676.hadara@bsd.ee> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org, freebsd-performance@freebsd.org, David Xu , Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 17:35:08 -0000 Sven Petai wrote: are there any patches that take the gettimeofday() calls and replace them with something that is cheap such as only doing every 10th one and just returning the last value ++ 1 uSec for the other ones.. a ktrace of Mysql shows a LOT of gettimeofday() calls. >On Tuesday 09 May 2006 03:42, Kris Kennaway wrote: > > >>On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: >> >> >>>>Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't >>>>yet had time to investigate. Is anyone else seeing this? >>>> >>>> >>>Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make >>>much difference though. >>> >>>I ran the tests again on the 8 core machine with and without rwatsons >>>patch and this time with 6 tests for each setting and generated graphs: >>>http://bsd.ee/~hadara/debug/mysql4/stats.html >>> >>>thr + select smack dynamics with the patch really do look quite >>>interesting >>> >>>PS >>>I'm currently running testround with rwatsons patch + >>>http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff + mysqld >>>change davidxu suggested. >>>Are there any other patches out there that I should try ? >>> >>> >>Sorry, I meant >>http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff >>instead of that patch. Also try turning down HZ to 100, and changing >>the wakeup()s to wakeup_one() in sys/filedesc.h. >> >> > >I preformed additional tests with the settings you suggested, >updated graphs are available @ >http://bsd.ee/~hadara/debug/mysql4/stats.html > > * with rwatsons patch + mysqld change suggested by davidxu + > http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff > there doesnt seem to be much difference, but in general things seem to be >couple of percent better > mutex profiles for this configuration are available @ > with 10 smack threads: >http://bsd.ee/~hadara/debug/mysql4/freebsd_cur_ps2/mutex_freebsd_cur2_p2-select_10_10000.txt > with 100 smack threads: >http://bsd.ee/~hadara/debug/mysql4/freebsd_cur_ps2/mutex_freebsd_cur_p2-select_100_1000.txt > > * second run was with same patches, but HZ set to 100 > this helps performance a lot but creates large fluxuations in select results >with >20 threads + thr, sometimes difference between 2 runs with same >settings was 10000+ q/s > >I'm currently still testing wakeup() -> wakeup_one() suggestion. > >In addition I did full testrun on linux to see how much this hardware >is really capable of... graphs for that are available on the stats page too. >_______________________________________________ >freebsd-current@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-current >To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > From owner-freebsd-performance@FreeBSD.ORG Tue May 9 18:07:15 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9444C16A5C4; Tue, 9 May 2006 18:07:15 +0000 (UTC) (envelope-from maxim@macomnet.ru) Received: from mp2.macomnet.net (mp2.macomnet.net [195.128.64.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id E261943D6A; Tue, 9 May 2006 18:07:14 +0000 (GMT) (envelope-from maxim@macomnet.ru) Received: from localhost (localhost [127.0.0.1]) by mp2.macomnet.net (8.13.4/8.13.3) with ESMTP id k49I789T046129; Tue, 9 May 2006 22:07:08 +0400 (MSD) (envelope-from maxim@macomnet.ru) Date: Tue, 9 May 2006 22:07:08 +0400 (MSD) From: Maxim Konovalov To: Julian Elischer In-Reply-To: <4460D2CA.5090808@elischer.org> Message-ID: <20060509220230.W46073@mp2.macomnet.net> References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Sven Petai , freebsd-performance@freebsd.org, Kris Kennaway , current@freebsd.org, David Xu Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 18:07:17 -0000 On Tue, 9 May 2006, 10:35-0700, Julian Elischer wrote: > Sven Petai wrote: > > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap such as only doing every 10th one > and just returning the last value ++ 1 uSec for the other ones.. > > a ktrace of Mysql shows a LOT of gettimeofday() calls. [...] Robert posted them. I failed to found these messages in our archives (does mailman archive messages properly?) and put them there: http://maxim.int.ru/stuff/gtd/ We used to use a very similar approch on RELENG_4 and found it helped a lot. -- Maxim Konovalov From owner-freebsd-performance@FreeBSD.ORG Tue May 9 18:07:23 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F21616A877; Tue, 9 May 2006 18:07:23 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id 49F1343D70; Tue, 9 May 2006 18:07:20 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [85.236.96.23]) (MDaemon.PRO.v8.1.3.R) with ESMTP id md50002544741.msg; Tue, 09 May 2006 19:06:35 +0100 Message-ID: <004501c67393$4d4aeae0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Julian Elischer" , "Sven Petai" References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org><200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> Date: Tue, 9 May 2006 19:06:26 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 X-Spam-Processed: multiplay.co.uk, Tue, 09 May 2006 19:06:35 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Tue, 09 May 2006 19:06:37 +0100 Cc: freebsd-performance@freebsd.org, Kris Kennaway , current@freebsd.org, David Xu Subject: Re: Fine-grained locking for POSIX local sockets (UNIXdomain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 18:07:25 -0000 Julian Elischer wrote: > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap > such as only doing every 10th one and just returning the last value > ++ 1 > uSec for the other ones.. > > a ktrace of Mysql shows a LOT of gettimeofday() calls. Yes there are check out the following: http://wikitest.freebsd.org/moin.cgi/MySQL http://phk.freebsd.dk/patch/ Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Tue May 9 18:13:07 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8E78316A93C; Tue, 9 May 2006 18:13:07 +0000 (UTC) (envelope-from grog@lemis.com) Received: from ext-gw.lemis.com (ext-gw.lemis.com [150.101.14.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id C20C943D70; Tue, 9 May 2006 18:13:06 +0000 (GMT) (envelope-from grog@lemis.com) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.135]) by ext-gw.lemis.com (Postfix) with ESMTP id C9E56131DE0; Wed, 10 May 2006 03:43:05 +0930 (CST) Received: from eucla.lemis.com (wantadilla.lemis.com [192.109.197.135]) by wantadilla.lemis.com (Postfix) with ESMTP id 394B384632; Wed, 10 May 2006 03:43:05 +0930 (CST) Received: from eucla.lemis.com (localhost [127.0.0.1]) by eucla.lemis.com (Postfix) with ESMTP id CFEA2D5DA4; Tue, 9 May 2006 11:13:03 -0700 (PDT) Received: (from grog@localhost) by eucla.lemis.com (8.13.6/8.13.4/Submit) id k49ID3FL005063; Tue, 9 May 2006 11:13:03 -0700 (PDT) (envelope-from grog) Date: Tue, 9 May 2006 11:13:02 -0700 From: "Greg 'groggy' Lehey" To: Kris Kennaway Message-ID: <20060509181302.GD3636@eucla.lemis.com> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <20060509004328.GB55852@xor.obsecurity.org> <20060509011109.GA57060@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IMjqdzrDRly81ofr" Content-Disposition: inline In-Reply-To: <20060509011109.GA57060@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 VoIP: sip:0871270137@sip.internode.on.net WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Cc: Robert Watson , performance@FreeBSD.org, current@FreeBSD.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 18:13:11 -0000 --IMjqdzrDRly81ofr Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Monday, 8 May 2006 at 21:11:09 -0400, Kris Kennaway wrote: > On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote: >> On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: >>> OK, David's patch fixes the umtx thundering herd (and seems to give a >>> 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which >>> was also waking up 2-7 CPUs at once about 30% of the time) by doing >>> s/wakeup/wakeup_one/. This did not seem to give a performance impact >>> on this test though. >> >> Turning down kern.hz from 1000 to 100 also made a big difference on 12 >> CPUs (+6.1%). >> >> Note also that the system is no less than 40% idle during the runs (at >> any load), so the bottlenecks are serious. > > top -H shows the threads mostly in umtx state. This doesn't lend much support to the idea that the gettimeofday() calls are a bottleneck. I've just been through the code, but if there are any mutexes acquired there, I missed them. Greg -- See complete headers for address and phone numbers --IMjqdzrDRly81ofr Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQFEYNutIubykFB6QiMRAhriAJ9eot/N1vVl0VsXdZX0JocYC7/qxACfSM+V GyLMmRNfOAhK+sLjc7M4YsQ= =vjXK -----END PGP SIGNATURE----- --IMjqdzrDRly81ofr-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 18:14:01 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 743F316A7CE; Tue, 9 May 2006 18:14:01 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1820343D60; Tue, 9 May 2006 18:13:48 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id EDCDF1A3C26; Tue, 9 May 2006 11:13:47 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id DBEA85161D; Tue, 9 May 2006 14:13:46 -0400 (EDT) Date: Tue, 9 May 2006 14:13:46 -0400 From: Kris Kennaway To: Sven Petai Message-ID: <20060509181346.GA92714@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="rwEMma7ioTxnRzrJ" Content-Disposition: inline In-Reply-To: <200605091818.12676.hadara@bsd.ee> User-Agent: Mutt/1.4.2.1i Cc: current@freebsd.org, freebsd-performance@freebsd.org, David Xu , Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 18:14:05 -0000 --rwEMma7ioTxnRzrJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 09, 2006 at 06:18:11PM +0300, Sven Petai wrote: > On Tuesday 09 May 2006 03:42, Kris Kennaway wrote: > > On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > > > > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > > > > yet had time to investigate. Is anyone else seeing this? > > > > > > Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make > > > much difference though. > > > > > > I ran the tests again on the 8 core machine with and without rwatsons > > > patch and this time with 6 tests for each setting and generated graph= s: > > > http://bsd.ee/~hadara/debug/mysql4/stats.html > > > > > > thr + select smack dynamics with the patch really do look quite > > > interesting > > > > > > PS > > > I'm currently running testround with rwatsons patch + > > > http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff + mys= qld > > > change davidxu suggested. > > > Are there any other patches out there that I should try ? > > > > Sorry, I meant > > http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff > > instead of that patch. Also try turning down HZ to 100, and changing > > the wakeup()s to wakeup_one() in sys/filedesc.h. >=20 > I preformed additional tests with the settings you suggested, > updated graphs are available @ > http://bsd.ee/~hadara/debug/mysql4/stats.html Very useful, thanks! I imagine we'll be discussing this issue in detail at bsdcan over the next few days. Kris --rwEMma7ioTxnRzrJ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEYNvaWry0BWjoQKURAoPnAJwLYQ2hNGmzTk0gawpiAJH6DNlZ2QCfXUrH kUCY7+YBsQkUYDproOht5q4= =K+Wk -----END PGP SIGNATURE----- --rwEMma7ioTxnRzrJ-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 18:23:36 2006 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B1B8D16A9DE; Tue, 9 May 2006 18:23:36 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id B39DF43D64; Tue, 9 May 2006 18:23:31 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 9C9071A3C26; Tue, 9 May 2006 11:23:31 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 13BEA515BB; Tue, 9 May 2006 14:23:31 -0400 (EDT) Date: Tue, 9 May 2006 14:23:30 -0400 From: Kris Kennaway To: Greg 'groggy' Lehey Message-ID: <20060509182330.GB92714@xor.obsecurity.org> References: <20060506150622.C17611@fledge.watson.org> <20060506221908.GB51268@xor.obsecurity.org> <20060507210426.GA4422@xor.obsecurity.org> <20060507214153.GA5275@xor.obsecurity.org> <20060507230430.GA6872@xor.obsecurity.org> <20060508065207.GA20386@xor.obsecurity.org> <20060509004328.GB55852@xor.obsecurity.org> <20060509011109.GA57060@xor.obsecurity.org> <20060509181302.GD3636@eucla.lemis.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="hQiwHBbRI9kgIhsi" Content-Disposition: inline In-Reply-To: <20060509181302.GD3636@eucla.lemis.com> User-Agent: Mutt/1.4.2.1i Cc: performance@FreeBSD.org, Robert Watson , current@FreeBSD.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 18:23:37 -0000 --hQiwHBbRI9kgIhsi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 09, 2006 at 11:13:02AM -0700, Greg 'groggy' Lehey wrote: > On Monday, 8 May 2006 at 21:11:09 -0400, Kris Kennaway wrote: > > On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote: > >> On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > >>> OK, David's patch fixes the umtx thundering herd (and seems to give a > >>> 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > >>> was also waking up 2-7 CPUs at once about 30% of the time) by doing > >>> s/wakeup/wakeup_one/. This did not seem to give a performance impact > >>> on this test though. > >> > >> Turning down kern.hz from 1000 to 100 also made a big difference on 12 > >> CPUs (+6.1%). > >> > >> Note also that the system is no less than 40% idle during the runs (at > >> any load), so the bottlenecks are serious. > > > > top -H shows the threads mostly in umtx state. >=20 > This doesn't lend much support to the idea that the gettimeofday() > calls are a bottleneck. There are at least several issues here: * Factor of >two difference in performance across the board (all loads) relative to Linux. This may be general issues like gettimeofday() on Linux vs FreeBSD; clearly there is something *very big* to blame here. Mysql does do *lots* of such calls, so the cost of them is surely a component in performance, the only question is if it's the main one. * When you get some of the locking out of the way (per this thread) FreeBSD has 44% better peak performance on Sven's test on amd64, but tails off by about 33% at higher loads compared to unpatched. I see similar changes on 12-CPU E4500, but not as much performance gain (may be due to other reasons). i.e. optimizing the locking allows a new, bigger bottleneck to suck on center stage. This is the basis for my observation about libthr at high loads. It is not the same as the above issue. Kris --hQiwHBbRI9kgIhsi Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEYN4iWry0BWjoQKURAnNGAKDveYJUb9GfgI60pnxB+lxlgwWXiACdEy5h j5L+YV64tSOPEWKpN7+/iWE= =kYah -----END PGP SIGNATURE----- --hQiwHBbRI9kgIhsi-- From owner-freebsd-performance@FreeBSD.ORG Tue May 9 18:31:28 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 598E516A969; Tue, 9 May 2006 18:31:28 +0000 (UTC) (envelope-from jd@ugcs.caltech.edu) Received: from retch.ugcs.caltech.edu (retch.ugcs.caltech.edu [131.215.176.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9E2EF43D6A; Tue, 9 May 2006 18:31:27 +0000 (GMT) (envelope-from jd@ugcs.caltech.edu) Received: by retch.ugcs.caltech.edu (Postfix, from userid 3640) id 53FEFEBB67; Tue, 9 May 2006 11:31:27 -0700 (PDT) Date: Tue, 9 May 2006 11:31:27 -0700 From: Paul Allen To: Julian Elischer Message-ID: <20060509183127.GD17515@retch.ugcs.caltech.edu> References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4460D2CA.5090808@elischer.org> Sender: jd@ugcs.caltech.edu X-Mailman-Approved-At: Tue, 09 May 2006 21:24:55 +0000 Cc: Sven Petai , freebsd-performance@freebsd.org, Kris Kennaway , current@freebsd.org, David Xu Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 18:31:33 -0000 >From Julian Elischer , Tue, May 09, 2006 at 10:35:06AM -0700: > Sven Petai wrote: > > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap > such as only doing every 10th one and just returning the last value ++ 1 > uSec for the other ones.. Better yet, just realize that during any given scheduler quantum the process is running on the same CPU. Therefore, you should just read the TSC. For that matter, if libc would just remember an accurate synchronized timestamp and TSC pair on a per CPU basis, it should be trivial to get cheap, synchronized, and accurate TSC time on SMP systems. TSC drift isn't horrible--and best of all if the process drifts from CPU to CPU libc will have a decent chance at doing incremental calibrations. Simply giving libc easy access to a counter of scheduler ticks can be used to ensure this process delivers monotonic time. Let me formalize this a bit, you have a noisy, but cheap time source: the TSC always available provided you compute your deltas a per cpu basis. You have another low resolution, low noise, but cheap time source: the count of scheduler ticks. Rather than coding an ad hoc algorithm, this information should be fed into a kalman filter. There are some lingering details: you need to invalidate the TSC when the processor speed changes: but this is controlled by powerd no? Second, if you can manage to throttle the CPU it suggests that you can also manage to pay higher time query costs and force clock_gettime calls. It seems obvious that the mechanism should adapt to the workload. Paul > > a ktrace of Mysql shows a LOT of gettimeofday() calls. > > > >On Tuesday 09 May 2006 03:42, Kris Kennaway wrote: > > > > > >>On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > >> > >> > >>>>Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > >>>>yet had time to investigate. Is anyone else seeing this? > >>>> > >>>> > >>>Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make > >>>much difference though. > >>> > >>>I ran the tests again on the 8 core machine with and without rwatsons > >>>patch and this time with 6 tests for each setting and generated graphs: > >>>http://bsd.ee/~hadara/debug/mysql4/stats.html > >>> > >>>thr + select smack dynamics with the patch really do look quite > >>>interesting > >>> > >>>PS > >>>I'm currently running testround with rwatsons patch + > >>>http://people.freebsd.org/~csjp/kern_descrip.c.1145074052.diff + mysqld > >>>change davidxu suggested. > >>>Are there any other patches out there that I should try ? > >>> > >>> > >>Sorry, I meant > >>http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff > >>instead of that patch. Also try turning down HZ to 100, and changing > >>the wakeup()s to wakeup_one() in sys/filedesc.h. > >> > >> > > > >I preformed additional tests with the settings you suggested, > >updated graphs are available @ > >http://bsd.ee/~hadara/debug/mysql4/stats.html > > > >* with rwatsons patch + mysqld change suggested by davidxu + > > http://people.freebsd.org/~csjp/sys.mpsafe.fileops.1145896495.diff > > there doesnt seem to be much difference, but in general things seem to be > >couple of percent better > > mutex profiles for this configuration are available @ > > with 10 smack threads: > >http://bsd.ee/~hadara/debug/mysql4/freebsd_cur_ps2/mutex_freebsd_cur2_p2-select_10_10000.txt > > with 100 smack threads: > >http://bsd.ee/~hadara/debug/mysql4/freebsd_cur_ps2/mutex_freebsd_cur_p2-select_100_1000.txt > > > >* second run was with same patches, but HZ set to 100 > > this helps performance a lot but creates large fluxuations in select > > results with >20 threads + thr, sometimes difference between 2 runs with > >same settings was 10000+ q/s > > > >I'm currently still testing wakeup() -> wakeup_one() suggestion. > > > >In addition I did full testrun on linux to see how much this hardware > >is really capable of... graphs for that are available on the stats page > >too. > >_______________________________________________ > >freebsd-current@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-current > >To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > > > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Tue May 9 23:26:34 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id D668816A44A; Tue, 9 May 2006 23:26:33 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Wed, 10 May 2006 07:26:27 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> In-Reply-To: <20060509182330.GB92714@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605100726.28243.davidxu@freebsd.org> Cc: Greg 'groggy' Lehey , Robert Watson , performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 23:26:35 -0000 On Wednesday 10 May 2006 02:23, Kris Kennaway wrote: > There are at least several issues here: > > * Factor of >two difference in performance across the board (all > loads) relative to Linux. This may be general issues like > gettimeofday() on Linux vs FreeBSD; clearly there is something *very > big* to blame here. Mysql does do *lots* of such calls, so the cost > of them is surely a component in performance, the only question is if > it's the main one. > My last recall is that gettimeofday is not a syscall on Linux, they called it vgettimeofday, they also have a lower overhead syscall: vsyscall which uses sysenter/sysexit when CPU supports, they do calculation in userland, the page is exported by kernel which can be executed by userland. at least I saw the idea on one serious hacker's blog, but now I can not find the URL. > * When you get some of the locking out of the way (per this thread) > FreeBSD has 44% better peak performance on Sven's test on amd64, but > tails off by about 33% at higher loads compared to unpatched. I see > similar changes on 12-CPU E4500, but not as much performance gain (may > be due to other reasons). i.e. optimizing the locking allows a new, > bigger bottleneck to suck on center stage. This is the basis for my > observation about libthr at high loads. It is not the same as the > above issue. > > Kris Fixing one of big lock contentions is not enough, you have to fix them all, it is easy to see that a second contention becomes a top one. :-) From owner-freebsd-performance@FreeBSD.ORG Tue May 9 23:26:34 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id D668816A44A; Tue, 9 May 2006 23:26:33 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Wed, 10 May 2006 07:26:27 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> In-Reply-To: <20060509182330.GB92714@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605100726.28243.davidxu@freebsd.org> Cc: Greg 'groggy' Lehey , Robert Watson , performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 May 2006 23:26:35 -0000 On Wednesday 10 May 2006 02:23, Kris Kennaway wrote: > There are at least several issues here: > > * Factor of >two difference in performance across the board (all > loads) relative to Linux. This may be general issues like > gettimeofday() on Linux vs FreeBSD; clearly there is something *very > big* to blame here. Mysql does do *lots* of such calls, so the cost > of them is surely a component in performance, the only question is if > it's the main one. > My last recall is that gettimeofday is not a syscall on Linux, they called it vgettimeofday, they also have a lower overhead syscall: vsyscall which uses sysenter/sysexit when CPU supports, they do calculation in userland, the page is exported by kernel which can be executed by userland. at least I saw the idea on one serious hacker's blog, but now I can not find the URL. > * When you get some of the locking out of the way (per this thread) > FreeBSD has 44% better peak performance on Sven's test on amd64, but > tails off by about 33% at higher loads compared to unpatched. I see > similar changes on 12-CPU E4500, but not as much performance gain (may > be due to other reasons). i.e. optimizing the locking allows a new, > bigger bottleneck to suck on center stage. This is the basis for my > observation about libthr at high loads. It is not the same as the > above issue. > > Kris Fixing one of big lock contentions is not enough, you have to fix them all, it is easy to see that a second contention becomes a top one. :-) From owner-freebsd-performance@FreeBSD.ORG Wed May 10 01:19:07 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3380116A424; Wed, 10 May 2006 01:19:07 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx1.starman.ee (smtp-out1.starman.ee [85.253.0.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id BE8FD43D46; Wed, 10 May 2006 01:19:06 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx1.starman.ee (Postfix) with ESMTP id 8D23423C1A7; Wed, 10 May 2006 04:19:02 +0300 (EEST) From: Sven Petai To: freebsd-performance@freebsd.org Date: Wed, 10 May 2006 04:18:53 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> In-Reply-To: <4460D2CA.5090808@elischer.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605100418.54475.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx1.starman.ee Cc: Kris Kennaway , David Xu , Julian Elischer , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 01:19:07 -0000 On Tuesday 09 May 2006 20:35, Julian Elischer wrote: > Sven Petai wrote: > > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap > such as only doing every 10th one and just returning the last value ++ 1 > uSec for the other ones.. > > a ktrace of Mysql shows a LOT of gettimeofday() calls. > well I have actually done that although in a very hackish way that is suitable *only* for benchmarking, but my goal really was just to find out if gettimeofday really is the bottleneck. Basically I just preloaded my version of "buffering" time() over libc's. That function asked kernel for time only after every 5000 invocations. I could get away with that for benchmarking since about 99,99% of gettimeofday() calls from mysqld really come through time() which has anyway only 1 second resolution and mysqld seems to use the value for purposes that are not really all that critical, like stats and safeguard timers. It's also called far more than 5000 times in a second while benchmark is running so it should return correct values too. Anyway, the results really astonished me, what I got for reducing gettimeofday() calls by ~96% was performance *decrease* of about 4-5%. I thought something was wrong with my measurement techniques, so I wrote the buffering time() function that used TSC directly into mysqld, replaced all the time() calls with that, so that number of gettimeofday() syscalls ktrace reported was down to only the initial one, but still got the same 5% decrease. I just can't explain it, but can't find any mistakes in methods too. The library I used is available @ http://bsd.ee/~hadara/debug/mysql4/time_lib_hack.c From owner-freebsd-performance@FreeBSD.ORG Wed May 10 00:31:42 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CEE8D16A40A; Wed, 10 May 2006 00:31:42 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6409343D45; Wed, 10 May 2006 00:31:42 +0000 (GMT) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.221] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id k4A0VbOZ058953; Tue, 9 May 2006 17:31:38 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <44613469.2050000@freebsd.org> Date: Tue, 09 May 2006 17:31:37 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Xu References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> In-Reply-To: <200605100726.28243.davidxu@freebsd.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 10 May 2006 01:37:11 +0000 Cc: performance@freebsd.org, current@freebsd.org, Robert Watson , Kris Kennaway , Greg 'groggy' Lehey , freebsd-performance@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 00:31:44 -0000 David Xu wrote: > On Wednesday 10 May 2006 02:23, Kris Kennaway wrote: > >>There are at least several issues here: >> >>...This may be general issues like >>gettimeofday() on Linux vs FreeBSD; clearly there is something *very >>big* to blame here. Mysql does do *lots* of such calls ... > > My last recall is that gettimeofday is not a syscall on Linux... I recall Matt talking about implementing gettimeofday() without a syscall. The basic idea is to have the kernel record some constants in a page that's mapped across all processes, then libc can just read the time from a known location. It might be nice to combine this with some of the other ideas being tossed around here: * On each clock tick, store a base time in a known location (page mapped read-only, no-execute across all memory maps) * libc can just read the base time (accurate to the clock rate) from a constant. Very, very fast. * For higher resolution, the kernel could record TSC and CPU clock speed data (per-CPU? Hmmm...) and libc could use that to fine-tune the time? Still some details I need to think through... Tim From owner-freebsd-performance@FreeBSD.ORG Wed May 10 00:31:42 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CEE8D16A40A; Wed, 10 May 2006 00:31:42 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6409343D45; Wed, 10 May 2006 00:31:42 +0000 (GMT) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.221] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id k4A0VbOZ058953; Tue, 9 May 2006 17:31:38 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <44613469.2050000@freebsd.org> Date: Tue, 09 May 2006 17:31:37 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Xu References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> In-Reply-To: <200605100726.28243.davidxu@freebsd.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 10 May 2006 01:37:20 +0000 Cc: performance@freebsd.org, current@freebsd.org, Robert Watson , Kris Kennaway , Greg 'groggy' Lehey , freebsd-performance@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 00:31:44 -0000 David Xu wrote: > On Wednesday 10 May 2006 02:23, Kris Kennaway wrote: > >>There are at least several issues here: >> >>...This may be general issues like >>gettimeofday() on Linux vs FreeBSD; clearly there is something *very >>big* to blame here. Mysql does do *lots* of such calls ... > > My last recall is that gettimeofday is not a syscall on Linux... I recall Matt talking about implementing gettimeofday() without a syscall. The basic idea is to have the kernel record some constants in a page that's mapped across all processes, then libc can just read the time from a known location. It might be nice to combine this with some of the other ideas being tossed around here: * On each clock tick, store a base time in a known location (page mapped read-only, no-execute across all memory maps) * libc can just read the base time (accurate to the clock rate) from a constant. Very, very fast. * For higher resolution, the kernel could record TSC and CPU clock speed data (per-CPU? Hmmm...) and libc could use that to fine-tune the time? Still some details I need to think through... Tim From owner-freebsd-performance@FreeBSD.ORG Wed May 10 01:38:48 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F48716A406; Wed, 10 May 2006 01:38:48 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5989243D6A; Wed, 10 May 2006 01:38:44 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.14] (imini.samsco.home [192.168.254.14]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id k4A1cdOH096135; Tue, 9 May 2006 19:38:39 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <4461441E.60304@samsco.org> Date: Tue, 09 May 2006 19:38:38 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.7) Gecko/20050416 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Sven Petai References: <20060506150622.C17611@fledge.watson.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> <200605100418.54475.hadara@bsd.ee> In-Reply-To: <200605100418.54475.hadara@bsd.ee> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: Julian Elischer , freebsd-performance@freebsd.org, current@freebsd.org, David Xu , Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 01:38:48 -0000 Sven Petai wrote: > On Tuesday 09 May 2006 20:35, Julian Elischer wrote: > >>Sven Petai wrote: >> >>are there any patches that take the gettimeofday() calls and replace >>them with something that is cheap >>such as only doing every 10th one and just returning the last value ++ 1 >>uSec for the other ones.. >> >>a ktrace of Mysql shows a LOT of gettimeofday() calls. >> > > > well I have actually done that although in a very hackish way that > is suitable *only* for benchmarking, but my goal really was > just to find out if gettimeofday really is the bottleneck. > > Basically I just preloaded my version of "buffering" time() over libc's. > That function asked kernel for time only after every 5000 invocations. > I could get away with that for benchmarking since > about 99,99% of gettimeofday() calls from mysqld really come through > time() which has anyway only 1 second resolution and mysqld seems to use > the value for purposes that are not really all that critical, like > stats and safeguard timers. > It's also called far more than 5000 times in a second while benchmark is > running so it should return correct values too. > > Anyway, the results really astonished me, what I got for reducing > gettimeofday() calls by ~96% was performance *decrease* of about 4-5%. > I thought something was wrong with my measurement techniques, so I > wrote the buffering time() function that used TSC directly into mysqld, > replaced all the time() calls with that, so that number of gettimeofday() > syscalls ktrace reported was down to only the initial one, but still got the > same 5% decrease. > > I just can't explain it, but can't find any mistakes in methods too. > > The library I used is available @ > http://bsd.ee/~hadara/debug/mysql4/time_lib_hack.c Were you testing on SMP, and if so, was Hyperthreading enabled? Scott From owner-freebsd-performance@FreeBSD.ORG Wed May 10 02:38:38 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from [127.0.0.1] (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id D101E16A401; Wed, 10 May 2006 02:38:35 +0000 (UTC) (envelope-from davidxu@freebsd.org) Message-ID: <4461522D.9060405@freebsd.org> Date: Wed, 10 May 2006 10:38:37 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060302 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tim Kientzle References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> <44613469.2050000@freebsd.org> In-Reply-To: <44613469.2050000@freebsd.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: performance@freebsd.org, current@freebsd.org, Robert Watson , Kris Kennaway , Greg 'groggy' Lehey , freebsd-performance@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 02:38:38 -0000 Tim Kientzle wrote: > I recall Matt talking about implementing gettimeofday() > without a syscall. The basic idea is to have the kernel > record some constants in a page that's mapped across > all processes, then libc can just read the time from > a known location. > > It might be nice to combine this with some of the > other ideas being tossed around here: > * On each clock tick, store a base time in > a known location (page mapped read-only, no-execute > across all memory maps) > * libc can just read the base time (accurate > to the clock rate) from a constant. Very, very fast. > * For higher resolution, the kernel could record > TSC and CPU clock speed data (per-CPU? Hmmm...) > and libc could use that to fine-tune the time? > > Still some details I need to think through... > > Tim > > One of the problems to implement it is that atomic operations, if there are multiple integer needs to be updated by kernel, userland maybe gets an inconsistent result, the way to avoid the problem is using two generation numbers. http://gsu.linux.org.tr/~mpekmezci/kernelapi/unitedlinux/arch/x86_64/kernel/vsyscall.c.html check do_vgettimeofday(struct timeval * tv): Another problem is how you tell userland the address of the kernel page ? do you use fixed address or tell it via program headers like the PT_TLS set by kernel, check /usr/src/lib/libc/gen/tls.c. David Xu From owner-freebsd-performance@FreeBSD.ORG Wed May 10 02:38:38 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from [127.0.0.1] (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id D101E16A401; Wed, 10 May 2006 02:38:35 +0000 (UTC) (envelope-from davidxu@freebsd.org) Message-ID: <4461522D.9060405@freebsd.org> Date: Wed, 10 May 2006 10:38:37 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060302 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tim Kientzle References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> <44613469.2050000@freebsd.org> In-Reply-To: <44613469.2050000@freebsd.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: performance@freebsd.org, current@freebsd.org, Robert Watson , Kris Kennaway , Greg 'groggy' Lehey , freebsd-performance@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 02:38:38 -0000 Tim Kientzle wrote: > I recall Matt talking about implementing gettimeofday() > without a syscall. The basic idea is to have the kernel > record some constants in a page that's mapped across > all processes, then libc can just read the time from > a known location. > > It might be nice to combine this with some of the > other ideas being tossed around here: > * On each clock tick, store a base time in > a known location (page mapped read-only, no-execute > across all memory maps) > * libc can just read the base time (accurate > to the clock rate) from a constant. Very, very fast. > * For higher resolution, the kernel could record > TSC and CPU clock speed data (per-CPU? Hmmm...) > and libc could use that to fine-tune the time? > > Still some details I need to think through... > > Tim > > One of the problems to implement it is that atomic operations, if there are multiple integer needs to be updated by kernel, userland maybe gets an inconsistent result, the way to avoid the problem is using two generation numbers. http://gsu.linux.org.tr/~mpekmezci/kernelapi/unitedlinux/arch/x86_64/kernel/vsyscall.c.html check do_vgettimeofday(struct timeval * tv): Another problem is how you tell userland the address of the kernel page ? do you use fixed address or tell it via program headers like the PT_TLS set by kernel, check /usr/src/lib/libc/gen/tls.c. David Xu From owner-freebsd-performance@FreeBSD.ORG Wed May 10 07:51:06 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0A0C116A404; Wed, 10 May 2006 07:51:06 +0000 (UTC) (envelope-from hadara@bsd.ee) Received: from mx2.starman.ee (smtp-out5.starman.ee [85.253.0.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3547B43D48; Wed, 10 May 2006 07:51:05 +0000 (GMT) (envelope-from hadara@bsd.ee) Received: from depression.softematic.com (depression.softematic.com [62.65.205.81]) by mx2.starman.ee (Postfix) with ESMTP id BB44F32C263; Wed, 10 May 2006 10:50:57 +0300 (EEST) From: Sven Petai To: Scott Long Date: Wed, 10 May 2006 10:50:49 +0300 User-Agent: KMail/1.9.1 References: <20060506150622.C17611@fledge.watson.org> <200605100418.54475.hadara@bsd.ee> <4461441E.60304@samsco.org> In-Reply-To: <4461441E.60304@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605101050.49888.hadara@bsd.ee> X-Virus-Scanned: by Amavisd-New at mx2.starman.ee Cc: Julian Elischer , freebsd-performance@freebsd.org, current@freebsd.org, David Xu , Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets ( UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 07:51:06 -0000 On Wednesday 10 May 2006 04:38, Scott Long wrote: > > Were you testing on SMP, and if so, was Hyperthreading enabled? > I tested it on 3 amd64 machines, so no hyperthreading possible. The library preload hack was tested on: 1 * dualcore amd X2 4 * dualcore opteron Hacked mysqld that used TSC was tested on UP amd64 box, with powernow disabled, so that tsc would be usable. From owner-freebsd-performance@FreeBSD.ORG Wed May 10 08:43:03 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A677C16A418; Wed, 10 May 2006 08:43:03 +0000 (UTC) (envelope-from ssouhlal@FreeBSD.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3B6FA43D45; Wed, 10 May 2006 08:42:52 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [137.122.74.158] (host158.thm10.resonet.uottawa.ca [137.122.74.158]) by elvis.mu.org (Postfix) with ESMTP id 4C4431A4D7A; Wed, 10 May 2006 01:42:52 -0700 (PDT) Message-ID: <4461A77F.5080508@FreeBSD.org> Date: Wed, 10 May 2006 10:42:39 +0200 From: Suleiman Souhlal User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051204) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Paul Allen References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> <20060509183127.GD17515@retch.ugcs.caltech.edu> In-Reply-To: <20060509183127.GD17515@retch.ugcs.caltech.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 08:43:04 -0000 Paul Allen wrote: >>From Julian Elischer , Tue, May 09, 2006 at 10:35:06AM -0700: > >>Sven Petai wrote: >> >>are there any patches that take the gettimeofday() calls and replace >>them with something that is cheap >>such as only doing every 10th one and just returning the last value ++ 1 >>uSec for the other ones.. > > Better yet, just realize that during any given scheduler quantum the process > is running on the same CPU. Therefore, you should just read the TSC. > > For that matter, if libc would just remember an accurate synchronized > timestamp and TSC pair on a per CPU basis, it should be trivial to get > cheap, synchronized, and accurate TSC time on SMP systems. TSC drift > isn't horrible--and best of all if the process drifts from CPU to CPU > libc will have a decent chance at doing incremental calibrations. Simply > giving libc easy access to a counter of scheduler ticks can be used to > ensure this process delivers monotonic time. > > Let me formalize this a bit, you have a noisy, but cheap time source: > the TSC always available provided you compute your deltas a per cpu basis. > You have another low resolution, low noise, but cheap time source: the > count of scheduler ticks. Rather than coding an ad hoc algorithm, > this information should be fed into a kalman filter. > > There are some lingering details: you need to invalidate the > TSC when the processor speed changes: but this is controlled > by powerd no? Second, if you can manage to throttle the CPU > it suggests that you can also manage to pay higher time > query costs and force clock_gettime calls. That's not enough. On some CPUs (like the current Opterons), the TSC slows down when the CPU executes a HLT instruction, so if you want good accuracy, you'll need to take that into account too. -- Suleiman From owner-freebsd-performance@FreeBSD.ORG Wed May 10 10:31:52 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E0BED16A402; Wed, 10 May 2006 10:31:52 +0000 (UTC) (envelope-from ssouhlal@FreeBSD.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id A273043D46; Wed, 10 May 2006 10:31:52 +0000 (GMT) (envelope-from ssouhlal@FreeBSD.org) Received: from [137.122.74.158] (host158.thm10.resonet.uottawa.ca [137.122.74.158]) by elvis.mu.org (Postfix) with ESMTP id 452E41A4D89; Wed, 10 May 2006 03:31:52 -0700 (PDT) Message-ID: <4461C10B.7020703@FreeBSD.org> Date: Wed, 10 May 2006 12:31:39 +0200 From: Suleiman Souhlal User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051204) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Jeremy References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> <20060509183127.GD17515@retch.ugcs.caltech.edu> <4461A77F.5080508@FreeBSD.org> <20060510101119.GB700@turion.vk2pj.dyndns.org> In-Reply-To: <20060510101119.GB700@turion.vk2pj.dyndns.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 10:31:53 -0000 Peter Jeremy wrote: > On Wed, 2006-May-10 10:42:39 +0200, Suleiman Souhlal wrote: > >>That's not enough. On some CPUs (like the current Opterons), the TSC >>slows down when the CPU executes a HLT instruction, so if you want good >>accuracy, you'll need to take that into account too. > > > The CPU can't be executing any instructions whilst it's halted. All > you need to do is update the base time/TSC count between when you exit > halt and when you return to userland. > > Since most sane code doesn't call gettimeofday() multiple times per > tick, one option might be to unmap the page when the base data becomes > invalid and update/remap the page when it is first accessed. > Another way would be to catch the first rdtsc call after a HLT, and reset the TSC to a good value. You can do this by making rdtsc usable only in ring0, which will make any userland rdtsc generate a general protection fault. You have to be careful, though, because on some Intel CPUs, you can only change the lower 32 bits of the TSC (the upper 32 bits get reset to 0 every time you write to the TSC). We actually implemented this at work, on Linux, to make sure that the TSCs between every processor is synchronized, and it appears to work pretty well. -- Suleiman From owner-freebsd-performance@FreeBSD.ORG Wed May 10 10:11:23 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B5A8416A444; Wed, 10 May 2006 10:11:23 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail16.syd.optusnet.com.au (mail16.syd.optusnet.com.au [211.29.132.197]) by mx1.FreeBSD.org (Postfix) with ESMTP id C7D8743D46; Wed, 10 May 2006 10:11:22 +0000 (GMT) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-19-236.belrs4.nsw.optusnet.com.au [220.239.19.236]) by mail16.syd.optusnet.com.au (8.12.11/8.12.11) with ESMTP id k4AABKCE015137 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 10 May 2006 20:11:20 +1000 Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.6/8.13.6) with ESMTP id k4AABJhH001372; Wed, 10 May 2006 20:11:19 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.6/8.13.6/Submit) id k4AABJP8001371; Wed, 10 May 2006 20:11:19 +1000 (EST) (envelope-from peter) Date: Wed, 10 May 2006 20:11:19 +1000 From: Peter Jeremy To: Suleiman Souhlal Message-ID: <20060510101119.GB700@turion.vk2pj.dyndns.org> Mail-Followup-To: Peter Jeremy , Suleiman Souhlal , freebsd-performance@freebsd.org, current@freebsd.org References: <20060506150622.C17611@fledge.watson.org> <200605090335.00451.hadara@bsd.ee> <20060509004203.GA55852@xor.obsecurity.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> <20060509183127.GD17515@retch.ugcs.caltech.edu> <4461A77F.5080508@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4461A77F.5080508@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.11 X-Mailman-Approved-At: Wed, 10 May 2006 11:39:32 +0000 Cc: freebsd-performance@freebsd.org, current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 10:11:23 -0000 On Wed, 2006-May-10 10:42:39 +0200, Suleiman Souhlal wrote: >That's not enough. On some CPUs (like the current Opterons), the TSC >slows down when the CPU executes a HLT instruction, so if you want good >accuracy, you'll need to take that into account too. The CPU can't be executing any instructions whilst it's halted. All you need to do is update the base time/TSC count between when you exit halt and when you return to userland. Since most sane code doesn't call gettimeofday() multiple times per tick, one option might be to unmap the page when the base data becomes invalid and update/remap the page when it is first accessed. -- Peter Jeremy From owner-freebsd-performance@FreeBSD.ORG Wed May 10 14:25:07 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 57E2016A403; Wed, 10 May 2006 14:25:07 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF65843D46; Wed, 10 May 2006 14:25:03 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.14] (imini.samsco.home [192.168.254.14]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id k4AEOjwv000213; Wed, 10 May 2006 08:24:46 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <4461F7AD.3000705@samsco.org> Date: Wed, 10 May 2006 08:24:45 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.7) Gecko/20050416 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Sven Petai References: <20060506150622.C17611@fledge.watson.org> <200605100418.54475.hadara@bsd.ee> <4461441E.60304@samsco.org> <200605101050.49888.hadara@bsd.ee> In-Reply-To: <200605101050.49888.hadara@bsd.ee> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: Julian Elischer , freebsd-performance@freebsd.org, current@freebsd.org, David Xu , Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2006 14:25:07 -0000 Sven Petai wrote: > On Wednesday 10 May 2006 04:38, Scott Long wrote: > >>Were you testing on SMP, and if so, was Hyperthreading enabled? >> > > > I tested it on 3 amd64 machines, so no hyperthreading possible. > > The library preload hack was tested on: > 1 * dualcore amd X2 > 4 * dualcore opteron > > Hacked mysqld that used TSC was tested on UP amd64 box, > with powernow disabled, so that tsc would be usable. That is very puzzling. Scott From owner-freebsd-performance@FreeBSD.ORG Thu May 11 13:54:12 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 567A216A454; Thu, 11 May 2006 13:54:12 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id A2DBF43D9F; Thu, 11 May 2006 13:54:08 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 927E846CD3; Thu, 11 May 2006 09:54:06 -0400 (EDT) Date: Thu, 11 May 2006 14:54:06 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: David Xu In-Reply-To: <200605100726.28243.davidxu@freebsd.org> Message-ID: <20060511145049.I72925@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Greg 'groggy' Lehey , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 13:54:13 -0000 On Wed, 10 May 2006, David Xu wrote: > Fixing one of big lock contentions is not enough, you have to fix them all, > it is easy to see that a second contention becomes a top one. :-) So I guess the real question is: do we want to merge the UNIX domain socket locking work? The MySQL gains sound good, the performance drop under very high load seems problematic, and there are more general questions about performance with other workloads. Maintaining this patch for a month or so is no problem, but as the tree changes it will get harder. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Thu May 11 13:54:12 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 567A216A454; Thu, 11 May 2006 13:54:12 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id A2DBF43D9F; Thu, 11 May 2006 13:54:08 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 927E846CD3; Thu, 11 May 2006 09:54:06 -0400 (EDT) Date: Thu, 11 May 2006 14:54:06 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: David Xu In-Reply-To: <200605100726.28243.davidxu@freebsd.org> Message-ID: <20060511145049.I72925@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Greg 'groggy' Lehey , freebsd-performance@freebsd.org, performance@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 13:54:13 -0000 On Wed, 10 May 2006, David Xu wrote: > Fixing one of big lock contentions is not enough, you have to fix them all, > it is easy to see that a second contention becomes a top one. :-) So I guess the real question is: do we want to merge the UNIX domain socket locking work? The MySQL gains sound good, the performance drop under very high load seems problematic, and there are more general questions about performance with other workloads. Maintaining this patch for a month or so is no problem, but as the tree changes it will get harder. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Thu May 11 16:01:15 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC38516A6F6; Thu, 11 May 2006 16:01:15 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id BC21E43EB2; Thu, 11 May 2006 16:00:07 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id E4846119CCF; Thu, 11 May 2006 17:59:13 +0200 (CEST) X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id x6XAKjKKgNag; Thu, 11 May 2006 17:59:13 +0200 (CEST) Received: from oxy (dsl217-197-176-208.pool.tvnet.hu [217.197.176.208]) by green.field.hu (Postfix) with ESMTP id 92708119CA2; Thu, 11 May 2006 17:59:13 +0200 (CEST) Message-ID: <000801c67513$f814f7f0$0201a8c0@oxy> From: "OxY" To: Date: Thu, 11 May 2006 18:00:05 +0200 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 16:01:22 -0000 hi! i just bought an Intel Pentium D-930 (3ghz dual-core) and wonder why i just only have CPU0 and CPU1, when i compile with = SMP... sysctl says: machdep.cpu_idle_hlt: 0 is it right or i should see CPU 2 and 3 starting during the boot? thanks for your help! ps: i even compiled this option in kernel, can't see any change options MPTABLE_FORCE_HTT # Enable HTT CPUs with the MP = Table From owner-freebsd-performance@FreeBSD.ORG Thu May 11 16:04:48 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3FFF16A7FC; Thu, 11 May 2006 16:04:48 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1B9A943DA4; Thu, 11 May 2006 16:04:46 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.14] (imini.samsco.home [192.168.254.14]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id k4BG4emg008812; Thu, 11 May 2006 10:04:45 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <44636098.2010903@samsco.org> Date: Thu, 11 May 2006 10:04:40 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.7) Gecko/20050416 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> <20060511145049.I72925@fledge.watson.org> In-Reply-To: <20060511145049.I72925@fledge.watson.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: freebsd-performance@freebsd.org, Kris Kennaway , performance@freebsd.org, David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 16:04:50 -0000 Robert Watson wrote: > > On Wed, 10 May 2006, David Xu wrote: > >> Fixing one of big lock contentions is not enough, you have to fix them >> all, it is easy to see that a second contention becomes a top one. :-) > > > So I guess the real question is: do we want to merge the UNIX domain > socket locking work? The MySQL gains sound good, the performance drop > under very high load seems problematic, and there are more general > questions about performance with other workloads. > > Maintaining this patch for a month or so is no problem, but as the tree > changes it will get harder. > > Robert N M Watson The only thing I'm afraid of is that it'll get pushed onto the back-burner once it's in CVS, and we'll have a mad scramble to fix it when it's time for 7.0. That's not a show-stopper for it going in, as there are also numerous benefits. It's just something that needs to be tracked and worked on. Scott From owner-freebsd-performance@FreeBSD.ORG Thu May 11 16:04:48 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3FFF16A7FC; Thu, 11 May 2006 16:04:48 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1B9A943DA4; Thu, 11 May 2006 16:04:46 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.14] (imini.samsco.home [192.168.254.14]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id k4BG4emg008812; Thu, 11 May 2006 10:04:45 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <44636098.2010903@samsco.org> Date: Thu, 11 May 2006 10:04:40 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.7) Gecko/20050416 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> <20060511145049.I72925@fledge.watson.org> In-Reply-To: <20060511145049.I72925@fledge.watson.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on pooker.samsco.org Cc: freebsd-performance@freebsd.org, Kris Kennaway , performance@freebsd.org, David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 16:04:54 -0000 Robert Watson wrote: > > On Wed, 10 May 2006, David Xu wrote: > >> Fixing one of big lock contentions is not enough, you have to fix them >> all, it is easy to see that a second contention becomes a top one. :-) > > > So I guess the real question is: do we want to merge the UNIX domain > socket locking work? The MySQL gains sound good, the performance drop > under very high load seems problematic, and there are more general > questions about performance with other workloads. > > Maintaining this patch for a month or so is no problem, but as the tree > changes it will get harder. > > Robert N M Watson The only thing I'm afraid of is that it'll get pushed onto the back-burner once it's in CVS, and we'll have a mad scramble to fix it when it's time for 7.0. That's not a show-stopper for it going in, as there are also numerous benefits. It's just something that needs to be tracked and worked on. Scott From owner-freebsd-performance@FreeBSD.ORG Thu May 11 16:16:53 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A9B616A8A5; Thu, 11 May 2006 16:16:53 +0000 (UTC) (envelope-from luke@foolishgames.com) Received: from mail.foolishgames.com (mail.foolishgames.com [206.222.28.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id E37C843D96; Thu, 11 May 2006 16:16:44 +0000 (GMT) (envelope-from luke@foolishgames.com) Received: from [192.168.0.150] (24-176-58-245.dhcp.klmz.mi.charter.com [24.176.58.245]) (authenticated bits=0) by mail.foolishgames.com (8.13.6/8.13.3) with ESMTP id k4BGGaL8008150 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Thu, 11 May 2006 12:16:36 -0400 (EDT) (envelope-from luke@foolishgames.com) X-Habeas-Swe-9: mark in spam to . X-Habeas-Swe-6: email in exchange for a license for this Habeas Content-Transfer-Encoding: 7bit X-Habeas-Swe-3: like Habeas SWE (tm) Date: Thu, 11 May 2006 12:16:35 -0400 X-Priority: 3 X-Habeas-Swe-8: Message (HCM) and not spam. Please report use of this X-Habeas-Swe-5: Sender Warranted Email (SWE) (tm). The sender of this From: Lucas Holt X-Habeas-Swe-2: brightly anticipated In-Reply-To: <000801c67513$f814f7f0$0201a8c0@oxy> References: <000801c67513$f814f7f0$0201a8c0@oxy> To: OxY X-Habeas-Swe-7: warrant mark warrants that this is a Habeas Compliant Mime-Version: 1.0 (Apple Message framework v749.3) X-Habeas-Swe-4: Copyright 2002 Habeas (tm) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed X-Habeas-Swe-1: winter into spring Message-Id: <1DD6C5CE-047E-4489-AF66-902B27020CD0@foolishgames.com> X-Mailer: Apple Mail (2.749.3) X-Virus-Scanned: ClamAV 0.88.2/1456/Thu May 11 01:57:31 2006 on mail.foolishgames.com X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 16:17:00 -0000 I think intel chips with dual core don't have hyperthreading. The operating system sees the second core like it would with hyperthreading only. On May 11, 2006, at 12:00 PM, OxY wrote: > hi! > > i just bought an Intel Pentium D-930 (3ghz dual-core) > and wonder why i just only have CPU0 and CPU1, when i compile with > SMP... > > sysctl says: > machdep.cpu_idle_hlt: 0 > > is it right or i should see CPU 2 and 3 starting during the boot? > > thanks for your help! > > ps: i even compiled this option in kernel, can't see any change > > options MPTABLE_FORCE_HTT # Enable HTT CPUs with the > MP Table > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@freebsd.org" Lucas Holt Luke@FoolishGames.com ________________________________________________________ FoolishGames.com (Jewel Fan Site) JustJournal.com (Free blogging) FoolishGames.net (Enemy Territory site) From owner-freebsd-performance@FreeBSD.ORG Thu May 11 17:35:03 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BC3EF16A4EC for ; Thu, 11 May 2006 17:35:03 +0000 (UTC) (envelope-from Shane@007Marketing.com) Received: from smtp3.adl2.internode.on.net (smtp3.adl2.internode.on.net [203.16.214.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id D4F1343D5F for ; Thu, 11 May 2006 17:35:02 +0000 (GMT) (envelope-from Shane@007Marketing.com) Received: from [192.168.8.200] (ppp247-71.static.internode.on.net [203.122.247.71]) by smtp3.adl2.internode.on.net (8.13.6/8.13.5) with ESMTP id k4BHZ0LS021809 for ; Fri, 12 May 2006 03:05:01 +0930 (CST) (envelope-from Shane@007Marketing.com) User-Agent: Microsoft-Entourage/10.1.4.030702.0 Date: Fri, 12 May 2006 03:04:59 +0930 From: Shane Ambler To: FreeBSD Mailing Lists Message-ID: In-Reply-To: <000801c67513$f814f7f0$0201a8c0@oxy> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Subject: Re: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 17:35:04 -0000 On 12/5/2006 1:30, "OxY" wrote: > hi! > > i just bought an Intel Pentium D-930 (3ghz dual-core) > and wonder why i just only have CPU0 and CPU1, when i compile with SMP... > > sysctl says: > machdep.cpu_idle_hlt: 0 > > is it right or i should see CPU 2 and 3 starting during the boot? > > thanks for your help! > > ps: i even compiled this option in kernel, can't see any change > > options MPTABLE_FORCE_HTT # Enable HTT CPUs with the MP Table > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > Sounds right to me - the 930 is a dual core and will show up as two cpu's cpu0 and cpu1. The early dual core's don't support hyperthreading - the Pentium-D extreme edition does (Pentium D-955 at 3.46Ghz is one) and will show up as 4 cpu's. Currently at 3x the price. -- Shane Ambler Shane@007Marketing.com From owner-freebsd-performance@FreeBSD.ORG Thu May 11 17:58:51 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 396B416A476; Thu, 11 May 2006 17:58:51 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id D3CAA43D53; Thu, 11 May 2006 17:58:50 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id A11A946D18; Thu, 11 May 2006 13:58:50 -0400 (EDT) Date: Thu, 11 May 2006 18:58:50 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Scott Long In-Reply-To: <44636098.2010903@samsco.org> Message-ID: <20060511185632.V90111@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> <20060511145049.I72925@fledge.watson.org> <44636098.2010903@samsco.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, Kris Kennaway , performance@freebsd.org, David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 17:58:51 -0000 On Thu, 11 May 2006, Scott Long wrote: >> So I guess the real question is: do we want to merge the UNIX domain socket >> locking work? The MySQL gains sound good, the performance drop under very >> high load seems problematic, and there are more general questions about >> performance with other workloads. >> >> Maintaining this patch for a month or so is no problem, but as the tree >> changes it will get harder. > > The only thing I'm afraid of is that it'll get pushed onto the back-burner > once it's in CVS, and we'll have a mad scramble to fix it when it's time for > 7.0. That's not a show-stopper for it going in, as there are also numerous > benefits. It's just something that needs to be tracked and worked on. I should be able to support/improve UNIX domain sockets moving forward without a problem -- the maintenance issue is maintaining it in P4 indefinitely, not in the tree indefinitely, as the patch basically touches every line in the file, so any change in the vendor branch (FreeBSD CVS) will put the entire file into conflict. To be specific: I'll track and own this, but want to avoid having it in P4 indefinitely, because it will get stale :-). Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Thu May 11 17:58:51 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 396B416A476; Thu, 11 May 2006 17:58:51 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id D3CAA43D53; Thu, 11 May 2006 17:58:50 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id A11A946D18; Thu, 11 May 2006 13:58:50 -0400 (EDT) Date: Thu, 11 May 2006 18:58:50 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Scott Long In-Reply-To: <44636098.2010903@samsco.org> Message-ID: <20060511185632.V90111@fledge.watson.org> References: <20060506150622.C17611@fledge.watson.org> <20060509181302.GD3636@eucla.lemis.com> <20060509182330.GB92714@xor.obsecurity.org> <200605100726.28243.davidxu@freebsd.org> <20060511145049.I72925@fledge.watson.org> <44636098.2010903@samsco.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, Kris Kennaway , performance@freebsd.org, David Xu , current@freebsd.org Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 17:58:51 -0000 On Thu, 11 May 2006, Scott Long wrote: >> So I guess the real question is: do we want to merge the UNIX domain socket >> locking work? The MySQL gains sound good, the performance drop under very >> high load seems problematic, and there are more general questions about >> performance with other workloads. >> >> Maintaining this patch for a month or so is no problem, but as the tree >> changes it will get harder. > > The only thing I'm afraid of is that it'll get pushed onto the back-burner > once it's in CVS, and we'll have a mad scramble to fix it when it's time for > 7.0. That's not a show-stopper for it going in, as there are also numerous > benefits. It's just something that needs to be tracked and worked on. I should be able to support/improve UNIX domain sockets moving forward without a problem -- the maintenance issue is maintaining it in P4 indefinitely, not in the tree indefinitely, as the patch basically touches every line in the file, so any change in the vendor branch (FreeBSD CVS) will put the entire file into conflict. To be specific: I'll track and own this, but want to avoid having it in P4 indefinitely, because it will get stale :-). Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Thu May 11 18:37:40 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A1CB216A46D; Thu, 11 May 2006 18:37:40 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id E1FC6448B4; Thu, 11 May 2006 18:37:39 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id B61F7119CCF; Thu, 11 May 2006 20:36:45 +0200 (CEST) X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 4U+Xjr7ZFwOC; Thu, 11 May 2006 20:36:45 +0200 (CEST) Received: from oxy (dsl217-197-176-208.pool.tvnet.hu [217.197.176.208]) by green.field.hu (Postfix) with ESMTP id 35764119CA2; Thu, 11 May 2006 20:36:45 +0200 (CEST) Message-ID: <000801c67529$faaf9e00$0201a8c0@oxy> From: "OxY" To: "Achim Patzner" References: <000801c67513$f814f7f0$0201a8c0@oxy> <7BBD3DC5-8E81-459E-B3AA-C6B21B49A5FF@bnc.net> Date: Thu, 11 May 2006 20:37:38 +0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 18:37:40 -0000 930 is perfect, fast enough, no need to spend more money, thanks :) ----- Original Message ----- From: "Achim Patzner" To: "OxY" Cc: ; Sent: Thursday, May 11, 2006 8:16 PM Subject: Re: enabling hyper-threading on intel dual-core >> i just bought an Intel Pentium D-930 (3ghz dual-core) >> and wonder why i just only have CPU0 and CPU1, when i compile with >> SMP... > > You would need a Pentium Extreme (840, 955 or 965) for this. And > quite a bit of cash. > > > Achim > From owner-freebsd-performance@FreeBSD.ORG Thu May 11 18:44:39 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 14ED916A6B9; Thu, 11 May 2006 18:44:35 +0000 (UTC) (envelope-from ap@bnc.net) Received: from mailomat.net (mailomat.net [217.110.117.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id DC32F442EA; Thu, 11 May 2006 18:16:43 +0000 (GMT) (envelope-from ap@bnc.net) X-BNC-SpamCatcher-Score: 2 [X] Received: from [212.202.24.165] (account ap HELO [194.39.192.148]) by mailomat.net (CommuniGate Pro SMTP 4.3.6) with ESMTPSA id 8113952; Thu, 11 May 2006 20:16:38 +0200 In-Reply-To: <000801c67513$f814f7f0$0201a8c0@oxy> References: <000801c67513$f814f7f0$0201a8c0@oxy> Mime-Version: 1.0 (Apple Message framework v749.3) X-Priority: 3 Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <7BBD3DC5-8E81-459E-B3AA-C6B21B49A5FF@bnc.net> Content-Transfer-Encoding: 7bit From: Achim Patzner Date: Thu, 11 May 2006 20:16:16 +0200 To: "OxY" X-Mailer: Apple Mail (2.749.3) Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 18:44:49 -0000 > i just bought an Intel Pentium D-930 (3ghz dual-core) > and wonder why i just only have CPU0 and CPU1, when i compile with > SMP... You would need a Pentium Extreme (840, 955 or 965) for this. And quite a bit of cash. Achim From owner-freebsd-performance@FreeBSD.ORG Thu May 11 23:28:38 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 9646B16A58F; Thu, 11 May 2006 23:28:38 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Fri, 12 May 2006 07:28:31 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <44636098.2010903@samsco.org> <20060511185632.V90111@fledge.watson.org> In-Reply-To: <20060511185632.V90111@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605120728.31786.davidxu@freebsd.org> Cc: performance@freebsd.org, Scott Long , Robert Watson , current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets ( UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 23:28:43 -0000 On Friday 12 May 2006 01:58, Robert Watson wrote: > On Thu, 11 May 2006, Scott Long wrote: > >> So I guess the real question is: do we want to merge the UNIX domain > >> socket locking work? The MySQL gains sound good, the performance drop > >> under very high load seems problematic, and there are more general > >> questions about performance with other workloads. > >> > >> Maintaining this patch for a month or so is no problem, but as the tree > >> changes it will get harder. > > > > The only thing I'm afraid of is that it'll get pushed onto the > > back-burner once it's in CVS, and we'll have a mad scramble to fix it > > when it's time for 7.0. That's not a show-stopper for it going in, as > > there are also numerous benefits. It's just something that needs to be > > tracked and worked on. > > I should be able to support/improve UNIX domain sockets moving forward > without a problem -- the maintenance issue is maintaining it in P4 > indefinitely, not in the tree indefinitely, as the patch basically touches > every line in the file, so any change in the vendor branch (FreeBSD CVS) > will put the entire file into conflict. To be specific: I'll track and own > this, but want to avoid having it in P4 indefinitely, because it will get > stale :-). > > Robert N M Watson Your patch makes other bottlenecks more visible than before, for example, file descriptor locking, but it is not a problem of your patch, so I think it is fine to commit it. David Xu From owner-freebsd-performance@FreeBSD.ORG Thu May 11 23:28:38 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from localhost.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 9646B16A58F; Thu, 11 May 2006 23:28:38 +0000 (UTC) (envelope-from davidxu@freebsd.org) From: David Xu To: freebsd-performance@freebsd.org Date: Fri, 12 May 2006 07:28:31 +0800 User-Agent: KMail/1.8.2 References: <20060506150622.C17611@fledge.watson.org> <44636098.2010903@samsco.org> <20060511185632.V90111@fledge.watson.org> In-Reply-To: <20060511185632.V90111@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200605120728.31786.davidxu@freebsd.org> Cc: performance@freebsd.org, Scott Long , Robert Watson , current@freebsd.org, Kris Kennaway Subject: Re: Fine-grained locking for POSIX local sockets ( UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2006 23:28:43 -0000 On Friday 12 May 2006 01:58, Robert Watson wrote: > On Thu, 11 May 2006, Scott Long wrote: > >> So I guess the real question is: do we want to merge the UNIX domain > >> socket locking work? The MySQL gains sound good, the performance drop > >> under very high load seems problematic, and there are more general > >> questions about performance with other workloads. > >> > >> Maintaining this patch for a month or so is no problem, but as the tree > >> changes it will get harder. > > > > The only thing I'm afraid of is that it'll get pushed onto the > > back-burner once it's in CVS, and we'll have a mad scramble to fix it > > when it's time for 7.0. That's not a show-stopper for it going in, as > > there are also numerous benefits. It's just something that needs to be > > tracked and worked on. > > I should be able to support/improve UNIX domain sockets moving forward > without a problem -- the maintenance issue is maintaining it in P4 > indefinitely, not in the tree indefinitely, as the patch basically touches > every line in the file, so any change in the vendor branch (FreeBSD CVS) > will put the entire file into conflict. To be specific: I'll track and own > this, but want to avoid having it in P4 indefinitely, because it will get > stale :-). > > Robert N M Watson Your patch makes other bottlenecks more visible than before, for example, file descriptor locking, but it is not a problem of your patch, so I think it is fine to commit it. David Xu From owner-freebsd-performance@FreeBSD.ORG Fri May 12 03:27:16 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 70CFF16A400 for ; Fri, 12 May 2006 03:27:16 +0000 (UTC) (envelope-from gtrbsd@gmail.com) Received: from wx-out-0102.google.com (wx-out-0102.google.com [66.249.82.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3B01043E2A for ; Fri, 12 May 2006 03:09:26 +0000 (GMT) (envelope-from gtrbsd@gmail.com) Received: by wx-out-0102.google.com with SMTP id t11so276868wxc for ; Thu, 11 May 2006 20:09:25 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=WRjN2xxWGanZJ1IeIyXum5sXZfuER+sIbGYnoMCrOViM85JhQSukbE0irweb+TD4tOaANvityyHx9uC962cKuFuHgGLgYfJf9g4X0ekgGTPsb783krNfXS3Sa3m51iQlh2WpcPiQ5dFds2H+Z9/jL0xMAb3lPugT/JGBReOAQu0= Received: by 10.70.84.17 with SMTP id h17mr2538929wxb; Thu, 11 May 2006 20:09:25 -0700 (PDT) Received: by 10.70.130.16 with HTTP; Thu, 11 May 2006 20:09:25 -0700 (PDT) Message-ID: Date: Fri, 12 May 2006 15:09:25 +1200 From: YSC To: freebsd-performance@freebsd.org In-Reply-To: <1DD6C5CE-047E-4489-AF66-902B27020CD0@foolishgames.com> MIME-Version: 1.0 References: <000801c67513$f814f7f0$0201a8c0@oxy> <1DD6C5CE-047E-4489-AF66-902B27020CD0@foolishgames.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 May 2006 03:27:22 -0000 if you want dual core + ht you gotta buy intel extream cpu ( dual core + ht =3D 4cpu) On 5/12/06, Lucas Holt wrote: > > I think intel chips with dual core don't have hyperthreading. The > operating system sees the second core like it would with > hyperthreading only. > > > On May 11, 2006, at 12:00 PM, OxY wrote: > > > hi! > > > > i just bought an Intel Pentium D-930 (3ghz dual-core) > > and wonder why i just only have CPU0 and CPU1, when i compile with > > SMP... > > > > sysctl says: > > machdep.cpu_idle_hlt: 0 > > > > is it right or i should see CPU 2 and 3 starting during the boot? > > > > thanks for your help! > > > > ps: i even compiled this option in kernel, can't see any change > > > > options MPTABLE_FORCE_HTT # Enable HTT CPUs with the > > MP Table > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance- > > unsubscribe@freebsd.org" > > > Lucas Holt > Luke@FoolishGames.com > ________________________________________________________ > FoolishGames.com (Jewel Fan Site) > JustJournal.com (Free blogging) > FoolishGames.net (Enemy Territory site) > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Fri May 12 11:18:12 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B7A8D16A401 for ; Fri, 12 May 2006 11:18:12 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2973543D46 for ; Fri, 12 May 2006 11:18:11 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id 0D5CB119CCF; Fri, 12 May 2006 13:17:12 +0200 (CEST) X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id ADmgzdU21n8j; Fri, 12 May 2006 13:17:11 +0200 (CEST) Received: from oxy (dsl217-197-176-208.pool.tvnet.hu [217.197.176.208]) by green.field.hu (Postfix) with ESMTP id 8A026119CA2; Fri, 12 May 2006 13:17:11 +0200 (CEST) Message-ID: <002401c675b5$c033abe0$0201a8c0@oxy> From: "OxY" To: "Derrick T. Woolworth" References: <000801c67513$f814f7f0$0201a8c0@oxy> <10fd06c60605111203y2f7a32degc225c3cc7b17af2@mail.gmail.com> <000d01c6752d$e81f95c0$0201a8c0@oxy> <10fd06c60605111209l64ca8a7dm8701786edb4da263@mail.gmail.com> Date: Fri, 12 May 2006 13:18:09 +0200 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: enabling hyper-threading on intel dual-core X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 May 2006 11:18:13 -0000 I made some tests on i386 and amd64 system with my new hardware, and got = interesting results.. benchmarked with sysbench on i386 the test was done in 47seconds, then checked on amd64 and got = incredible results!! !!!23sec!!! awesome, it worth to reinstall the whole system :) double performance! ----- Original Message -----=20 From: Derrick T. Woolworth=20 To: OxY=20 Sent: Thursday, May 11, 2006 9:09 PM Subject: Re: enabling hyper-threading on intel dual-core Huge performance advantage and difference between i386 and amd64 = kernel. i386 is 32-bit and amd64 is obviously 64-bit, but from my = tests, the difference in performance is big - like night and day. In fact, this is why I'm not so versed in the Intel world - I = typically only use AMD processors and love them.=20 I have had some issues with amd64 and IPSec - this was with versions = 5.4 and 6.0, so I'm unclear if all of those issues have been resolved. = If you're not using IPSec though, you won't have any other problems. AMD64 is pretty gosh darn fast... D On 5/11/06, OxY wrote: thanks, that was impressive :) i have only one question left: can i have any performance advantage from amd64 kernel compared to = i386? does it worth to start again from scratch? thanks! --=20 Derrick T. Woolworth, President ServeTheWeb, LLC. http://www.ServeTheWeb.com From owner-freebsd-performance@FreeBSD.ORG Fri May 12 18:19:51 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C4A2016AD38 for ; Fri, 12 May 2006 18:19:51 +0000 (UTC) (envelope-from jinmtb@sbcglobal.net) Received: from smtp108.sbc.mail.mud.yahoo.com (smtp108.sbc.mail.mud.yahoo.com [68.142.198.207]) by mx1.FreeBSD.org (Postfix) with SMTP id 9A39F43D83 for ; Fri, 12 May 2006 18:19:50 +0000 (GMT) (envelope-from jinmtb@sbcglobal.net) Received: (qmail 45413 invoked from network); 12 May 2006 18:19:49 -0000 Received: from unknown (HELO ?192.168.2.8?) (jinmtb@sbcglobal.net@68.127.184.186 with plain) by smtp108.sbc.mail.mud.yahoo.com with SMTP; 12 May 2006 18:19:49 -0000 Message-ID: <4464D242.4050908@sbcglobal.net> Date: Fri, 12 May 2006 11:21:54 -0700 From: Jin Guojun User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060317 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: Scott Long References: <20060506150622.C17611@fledge.watson.org> <200605091818.12676.hadara@bsd.ee> <4460D2CA.5090808@elischer.org> <200605100418.54475.hadara@bsd.ee> <4461441E.60304@samsco.org> In-Reply-To: <4461441E.60304@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org, David Xu , Kris Kennaway , Sven Petai , freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 May 2006 18:19:55 -0000 Scott Long wrote: > Sven Petai wrote: > >> On Tuesday 09 May 2006 20:35, Julian Elischer wrote: >> >>> Sven Petai wrote: >>> >>> are there any patches that take the gettimeofday() calls and replace >>> them with something that is cheap >>> such as only doing every 10th one and just returning the last value >>> ++ 1 >>> uSec for the other ones.. >>> >>> a ktrace of Mysql shows a LOT of gettimeofday() calls. >>> >> >> >> well I have actually done that although in a very hackish way that >> is suitable *only* for benchmarking, but my goal really was >> just to find out if gettimeofday really is the bottleneck. >> >> Basically I just preloaded my version of "buffering" time() over libc's. >> That function asked kernel for time only after every 5000 invocations. >> I could get away with that for benchmarking since >> about 99,99% of gettimeofday() calls from mysqld really come through >> time() which has anyway only 1 second resolution and mysqld seems to use >> the value for purposes that are not really all that critical, like >> stats and safeguard timers. >> It's also called far more than 5000 times in a second while benchmark is >> running so it should return correct values too. >> >> Anyway, the results really astonished me, what I got for reducing >> gettimeofday() calls by ~96% was performance *decrease* of about 4-5%. >> I thought something was wrong with my measurement techniques, so I >> wrote the buffering time() function that used TSC directly into mysqld, >> replaced all the time() calls with that, so that number of >> gettimeofday() >> syscalls ktrace reported was down to only the initial one, but still >> got the same 5% decrease. >> >> I just can't explain it, but can't find any mistakes in methods too. > >> The library I used is available @ >> http://bsd.ee/~hadara/debug/mysql4/time_lib_hack.c > > > Were you testing on SMP, and if so, was Hyperthreading enabled? > > Scott I think that depends on the type of testing, not hardware. I simply replaced gettimeofday() syscall with a super simple user space code, for testing with no time accuracy, in libc (libc.so.6.test, which is mapped for mysqld and mysql via libmap -- see details at the end), and found that code has no effect on "select-only" super-smack test, and has some impact (7-8%) for super-smack "update-select" test. The code itself is about 80~90 times faster than gettimeofday syscall. So, the slow is not mainly caused by timestamp, but I/O and other syscalls. 129 /tmp: ldd /usr/local/libexec/mysqld /usr/local/libexec/mysqld: ... skipped libpthread.so.2 => /usr/lib/libthr.so.2 (0x884fc000) libc.so.6 => /lib/libc.so.6 (0x8850e000) 130 /tmp: super-smack -d mysql /usr/local/share/super-smack/update-select.smack 10 7000 Query Barrel Report for client smacker connect: max=4ms min=4ms avg= 4ms from 10 clients Query_type num_queries max_time min_time q_per_s select_index 70000 1 0 3125.63 update_index 70000 1 0 3125.63 2.742u 4.058s 0:22.52 30.1% 119+519k 0+0io 0pf+0w ----- edit /etc/libmap.conf and restart mysqld ---------- 131 /tmp: ldd /usr/local/libexec/mysqld /usr/local/libexec/mysqld: ... skiipped libpthread.so.2 => /usr/lib/libthr.so.2 (0x884fc000) libc.so.6 => /lib/libc.so.6.test (0x8850e000) 132 /tmp: super-smack -d mysql /usr/local/share/super-smack/update-select.smack 10 7000 Query Barrel Report for client smacker connect: max=5ms min=2ms avg= 3ms from 10 clients Query_type num_queries max_time min_time q_per_s select_index 70000 1 0 3356.99 update_index 70000 1 0 3356.99 2.634u 4.198s 0:20.97 32.5% 116+506k 0+0io 0pf+0w 133 /tmp: super-smack -d mysql /usr/local/share/super-smack/update-select.smack 10 7000 Query Barrel Report for client smacker connect: max=248ms min=1ms avg= 132ms from 10 clients Query_type num_queries max_time min_time q_per_s select_index 70000 1 0 3334.93 update_index 70000 1 0 3334.93 2.604u 4.247s 0:21.11 32.4% 115+503k 0+0io 0pf+0w ------------- original gettimeofday code ---------- < #include "SYS.h" < RSYSCALL(gettimeofday) --------- replacement code --- #include static struct timeval tmv_base={$COMPILING_TIME}; #ifndef FAKE_INC #define FAKE_INC 700 /* 100-times longer as per gettimeofday cost */ #endif #define ONE_MEGA 1000000 int gettimeofday(struct timeval *tp, struct timezone *nouse) { tmv_base.tv_usec += FAKE_INC; if (tmv_base.tv_usec >= ONE_MEGA) tmv_base.tv_usec -= ONE_MEGA, ++tmv_base.tv_sec; if (tp) { *tp = tmv_base; return 0; } return -1; }