Date: Wed, 16 Jun 2004 18:39:55 -0700 (PDT) From: Julian Elischer <julian@elischer.org> To: Robert Watson <rwatson@freebsd.org> Cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? Message-ID: <Pine.BSF.4.21.0406161839130.65222-100000@InterJet.elischer.org> In-Reply-To: <Pine.NEB.3.96L.1040616011931.75630J-100000@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Robert, can you try see if HZ=5000 (up from 100) changes the performance? On Wed, 16 Jun 2004, Robert Watson wrote: > > On Tue, 15 Jun 2004, Julian Elischer wrote: > > > On Mon, 14 Jun 2004, mike wrote: > > > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > > as well. it sounds like scheduler/threading is to blame but we were not > > > able to give sufficient or proper motivation to the folks who could > > > examine this deeper - we even offered $500 cash to whomever stepped up to > > > help resolve this. > > > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > > and we get nearly the same results running in single CPU mode vs. dual CPU > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > > combination. > > > > You complained about this some time ago and you have still not responded > > with the information I suggest.. > > I sent this to Jeremy privately, since it was just some preliminary > measurements, but figured I'd send it publically since the results were > interesting (if tentative, I need to do a lot more work to make them > useful. There are a number of variables I need to look at including: > > - Disabling HTT. A chat with Scott Long this evening suggests that HTT > may be substantially hurting the test cases given increased IPIs, etc. > Unfortunately, it looks like I can't easily twiddle HTT without being > local to the machine, and I'm at home right now (it being 1:30am and > all). Removing HTT may help substantially with the dip in performance > in the SMP configuration. > > - I'd like to compare against RELENG_4 and a recent Linux kernel. > Unfortunately, the box is configured for neither right now. > > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd > like to try SCHED_4BSD. > > - This was without adaptive mutexes, which seem to be helpful for others, > so I should give them a try. > > I don't have any amd64 hardware, so I don't know what if any role it will > play in the results. The performance drop observed in the report appears > to be on amd64 (I may have misread). > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > ---- > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) > From: Robert Watson <rwatson@FreeBSD.org> > To: JG <amd64list@jpgsworld.com> > Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > > On Tue, 15 Jun 2004, JG wrote: > > > Fwiw, it has to be something that was committed between May 18th and > > yesterday. ~May 18th was the last time I built -CURRENT during my last > > round of testing and I did not have any of these problems. Then someone > > emailed me recently and said there were some commits that might effect > > the outcome of the mysql benchmarks. > > Ok, so these results are on a dual-processor XEON + hyperthreads, so four > logical processors. I used two dates off CVS, 20040515 and 20040615. I > also benchmarked my netperf branch. I don't have RELENG_4 on the box, but > might be able to load RELENG_4 on it later this week. In each case, I > took ten samples, dropped the first value as getting into the cache, and > took the mean of the rest. For this test, I used the select test; I'll > try the other smack query set tomorrow. In each case, I ran with "10 > 1000" as the arguments to the test. I used the default threading > configuration in -CURRENT, which is libpthread (libkse). > > Mean Stdev > 20040515-UP 4752.27 14.63 > 20040515-SMP 2550.35 19.23 > > 20040615-UP 4898.71 22.39 > 20040615-SMP 2666.93 32.01 > > Netperf-UP-giant 4902.41 14.3 > Netperf-SMP-giant 2566.18 16.83 > Netperf-UP-mpsafe 4799.35 22.04 > Netperf-SMP-mpsafe 3022.51 18.06 > > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages > the SMP numbers a fair amount due to additional IPIs without benefit. > However, the numbers basically suggest that on my hardware, the UP > configuration is marginally faster than it was last month, and that if you > throw in the netperf branch, the SMP case is a moderate amount faster. > This suggests that either I'm just lucky, or that the performance loss > might be specific to the amd64 version of FreeBSD. I'm going to run some > more numbers tomorrow and try to post something more rigorous to the > -threads list. > > I don't have RELENG_4 on the box or Linux on the box, but I may get a > chance to later this week. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0406161839130.65222-100000>