From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 05:34:11 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8964016A4CE; Thu, 17 Jun 2004 05:34:11 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9BD443D45; Thu, 17 Jun 2004 05:34:10 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5H5VwJK015223; Thu, 17 Jun 2004 01:31:58 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5H5Vw3q015220; Thu, 17 Jun 2004 01:31:58 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 01:31:57 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 05:34:11 -0000 On Wed, 16 Jun 2004, Julian Elischer wrote: > Robert, can you try see if HZ=5000 (up from 100) changes the > performance? I haven't explored the HZ=BIGGER case very thoroughly, but ran a couple of test cases with HZ at 1000 instead of 100, and found that generally it resulted in a slight (1.5%) performance drop in the smattering of cases I looked at. The cases I looked at were NETPERF-SMP-MPSAFE-ADMTX-HZ=1000, NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT-HZ=1000. In general, I found that the biggest variables improving performance relative to out-of-box SMP configuration were: - Removing Giant from UNIX domain sockets - Running with SCHED_4BSD instead of SCHED_ULE - Disabling HTT - Using ADAPTIVE_MUTEXES By doing these things, I went from an out-of-box queries/sec for the simple select "smack" with 11 clients from 2667 q/s to 6955 q/s, or a bit over doubling the transaction rate. That as compared to 4939 q/s, which was the best UP result I got (Giant-free UNIX domain sockets, SCHED_4BSD): 20040615-UP 4898.71 q/s (22.39 stddev) 20040615-SMP 2666.93 q/s (32.01 stddev) NETPERF-UP-MPSAFE-4BSD 4939.89 q/s (74.06 stddev) NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT 6955.18 q/s (156.91 stddev) Some of my 4BSD performance numbers were also slightly pessimized with what appears to be a property of the 4BSD scheduling period being longer than ULE. With ULE, the results settled on one loop of the benchmark. With 4BSD, it sometimes took 2-3 loops of the benchmark to settle. As a result, 4BSD stddev's are also generally higher. I haven't tried regenerating the results dropping those outliers. Using out-of-box -CURRENT without the netperf patches necessary to run UNIX domain sockets safely, I got best results using: 20040615-UP-4BSD 4886.84 q/s (46.03 stddev) 20040615-SMP-ADMTX-4BSD-!HTT 5838.76 q/s (45.11 stddev) In my environment, I couldn't easily test ULE without HTT, since the HTT disabling sysctl/tunable with ULE results in a hang. (Obviously, in all of the above, WITNESS and INVARIANTS are disabled; I didn't disable the userspace malloc debugging flags). Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > On Wed, 16 Jun 2004, Robert Watson wrote: > > > > > On Tue, 15 Jun 2004, Julian Elischer wrote: > > > > > On Mon, 14 Jun 2004, mike wrote: > > > > > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > > > as well. it sounds like scheduler/threading is to blame but we were not > > > > able to give sufficient or proper motivation to the folks who could > > > > examine this deeper - we even offered $500 cash to whomever stepped up to > > > > help resolve this. > > > > > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > > > and we get nearly the same results running in single CPU mode vs. dual CPU > > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > > > combination. > > > > > > You complained about this some time ago and you have still not responded > > > with the information I suggest.. > > > > I sent this to Jeremy privately, since it was just some preliminary > > measurements, but figured I'd send it publically since the results were > > interesting (if tentative, I need to do a lot more work to make them > > useful. There are a number of variables I need to look at including: > > > > - Disabling HTT. A chat with Scott Long this evening suggests that HTT > > may be substantially hurting the test cases given increased IPIs, etc. > > Unfortunately, it looks like I can't easily twiddle HTT without being > > local to the machine, and I'm at home right now (it being 1:30am and > > all). Removing HTT may help substantially with the dip in performance > > in the SMP configuration. > > > > - I'd like to compare against RELENG_4 and a recent Linux kernel. > > Unfortunately, the box is configured for neither right now. > > > > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd > > like to try SCHED_4BSD. > > > > - This was without adaptive mutexes, which seem to be helpful for others, > > so I should give them a try. > > > > I don't have any amd64 hardware, so I don't know what if any role it will > > play in the results. The performance drop observed in the report appears > > to be on amd64 (I may have misread). > > > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > > > > ---- > > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) > > From: Robert Watson > > To: JG > > Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > > > > > On Tue, 15 Jun 2004, JG wrote: > > > > > Fwiw, it has to be something that was committed between May 18th and > > > yesterday. ~May 18th was the last time I built -CURRENT during my last > > > round of testing and I did not have any of these problems. Then someone > > > emailed me recently and said there were some commits that might effect > > > the outcome of the mysql benchmarks. > > > > Ok, so these results are on a dual-processor XEON + hyperthreads, so four > > logical processors. I used two dates off CVS, 20040515 and 20040615. I > > also benchmarked my netperf branch. I don't have RELENG_4 on the box, but > > might be able to load RELENG_4 on it later this week. In each case, I > > took ten samples, dropped the first value as getting into the cache, and > > took the mean of the rest. For this test, I used the select test; I'll > > try the other smack query set tomorrow. In each case, I ran with "10 > > 1000" as the arguments to the test. I used the default threading > > configuration in -CURRENT, which is libpthread (libkse). > > > > Mean Stdev > > 20040515-UP 4752.27 14.63 > > 20040515-SMP 2550.35 19.23 > > > > 20040615-UP 4898.71 22.39 > > 20040615-SMP 2666.93 32.01 > > > > Netperf-UP-giant 4902.41 14.3 > > Netperf-SMP-giant 2566.18 16.83 > > Netperf-UP-mpsafe 4799.35 22.04 > > Netperf-SMP-mpsafe 3022.51 18.06 > > > > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages > > the SMP numbers a fair amount due to additional IPIs without benefit. > > However, the numbers basically suggest that on my hardware, the UP > > configuration is marginally faster than it was last month, and that if you > > throw in the netperf branch, the SMP case is a moderate amount faster. > > This suggests that either I'm just lucky, or that the performance loss > > might be specific to the amd64 version of FreeBSD. I'm going to run some > > more numbers tomorrow and try to post something more rigorous to the > > -threads list. > > > > I don't have RELENG_4 on the box or Linux on the box, but I may get a > > chance to later this week. > > > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > > > > > >