From owner-freebsd-performance@FreeBSD.ORG Sun Jun 19 15:56:52 2005 Return-Path: X-Original-To: performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7BDDE16A41C; Sun, 19 Jun 2005 15:56:52 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2B7AB43D1F; Sun, 19 Jun 2005 15:56:52 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 915D246B20; Sun, 19 Jun 2005 11:56:51 -0400 (EDT) Date: Sun, 19 Jun 2005 16:59:30 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: David Sze In-Reply-To: <6.2.1.2.2.20050617091736.05949298@mail.distrust.net> Message-ID: <20050619165349.U6413@fledge.watson.org> References: <746fd037f6ca8131a8fb8938f1e346e9@lonres.com> <20050610170537.GA67849@bibipentium.lonres.com> <20050611085604.J75625@fledge.watson.org> <20050616161506.GB28794@bibipentium.lonres.com> <6.2.1.2.2.20050617091736.05949298@mail.distrust.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Mon, 20 Jun 2005 12:01:12 +0000 Cc: Steve Roome , freebsd-stable@freebsd.org, steve@pepcross.com, Jon Dama , Guy Helmer , Kris Kennaway , Greg 'groggy' Lehey , performance@FreeBSD.org, Daniel Eischen , Xin LI , Thomas Hurst Subject: Re: FreeBSD MySQL still WAY slower than Linux X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jun 2005 15:56:52 -0000 On Fri, 17 Jun 2005, David Sze wrote: > FreeBSD/amd64 5.4-RELEASE (libpthread, system and process scope) > FreeBSD/amd64 6.0-CURRENT (libpthread, libthr, system and process > scope) > CentOS/amd64 4.0 (i.e. RHEL4.0) > > I couldn't get libthr to work on FreeBSD/amd64 5.4-RELEASE, mysqld would > immediately coredump and the stacktrace looked like it was corrupted > (i.e. hundreds of stack frames, all of which were ???). The libthr in 5.x is basically an early prototype compared to the more mature work on 6.x, so this result is not all that surprising. David Xu has put a very large amount of work into the libthr on 6.x with some very good results. > It turns out that the problem was the same thing everyone usually points > the finger at, but no one actually mentioned this time: Linux mounts > its partitions async by default. I don't have the exact numbers in > front of me right now, but these were the ballpark figures (I'm not > going to separate out results for all of the different threading > libraries for FreeBSD because the deltas weren't huge): This is actually an interesting observation -- on 6.x using P4 Xeon hardware, I've seen a substantial performance improvement from running with libthr with MySQL. Sometimes as much as 30% - 50%. However, amd64 is quite a different beast. BTW, could you run 'file' on the Linux and FreeBSD MySQL binaries and confirm that they're both either 32-bit i386 binaries, or 64-bit amd64 binaries? > That last CentOS number is not a typo, it was an order of magnitude > slower. I didn't try other file systems on CentOS, just the default > ext3. It's possible that reiserfs or xfs might not be as affected by > switching from async to sync. > > So my production server is now happily running mysql 4.1 on 6.0-CURRENT > :). I was interested to observe, in some benchmarking a few months ago, that 4.0 MySQL performance on FreeBSD 6.x is a *lot* faster than 4.1 MySQL performance. I don't track MySQL feature development, but I couldn't help but wonder if either there had been substantial reliability changes that impacted file system semantics (which might be indicated by sync/async issues), or whether there had been a lot of performance optimization work for Linux that had swung its use of IPC/etc primitives towards ones that work better on Linux than FreeBSD? BTW, could you confirm that on 6.x, you're setting /etc/malloc.conf so that it's not scrubbing memory on free()? In particular: ln -s jz /etc/malloc.conf as root. Most 6.x debugging features are set as part of the kernel configuration, and your performance results suggest that you've caught them all already, I think, but this one is user space. Also, make sure you are compiling without INVARIANT_SUPPORT -- for packet send tests, I find that even without INVARIANTS, I see a 4% hit for having INVARIANT_SUPPORT in the kernel. Robert N M Watson