Date: Tue, 17 Feb 2004 17:21:49 -0800 (PST) From: Julian Elischer <julian@elischer.org> To: Kris Gale <kris-fbsd@asn.net> Cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 Message-ID: <Pine.BSF.4.21.0402171711020.81059-100000@InterJet.elischer.org> In-Reply-To: <49313.68.106.19.246.1077066168.squirrel@mail.asn.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 17 Feb 2004, Kris Gale wrote: > >> What I seem to be seeing is a bogging down of MySQL > >> when new threads are being created in bursts. This > >> causes MySQL to temporarily become unresponsive, > >> and will sometimes crash the whole system. > > > > My first question is: > > "In what state are these threads waiting for work? > > are they in the kernel, or are they in userland? > > is there a single thread that listens on a socket and then hands the > > work to worker threads using some userland synchronisation, or do the > > threads enter the kernel and wait on the sockets themselves? > > (i.e. what does ps -H show?) " > > I'm not really sure how to answer this. During the startup > of my test script, when I'm seeing all of the connect errors, > "ps auxwH" shows a growing number of mysqld threads. how many? What is most important is, "How many (in ps) when it has 1800 threads and no work.." (how do you know it has 1800 threads?) what does teh ps show? can you send it to me? > All but one have "SL" listed in the STAT column. The other > one has "RL" listed. > > > what happens if you have 1 process with X*Y threads? > > (just curious). > > No problems. 1800 threads run fine. The problem > seems to be with the child processes trying to initialize > connections simultaneously. I can also "solve" the > problem by putting a sleep in so the child processes > aren't forked at the same time. However, this is > not a workable real-world solution, since the application > I'm trying to simulate is a theoretical burst of connections > from a web cluster. > > > is the server threaded process you are debugging, or is the > > test program what is being debugged? > > The test program is just meant to simulate the problems I > saw in production. I'm trying to test MySQL. > > > use vmstat -z to check the values for: > > Here's a snapshot from the system when mysql first starts: > > UPCALL: 44, 0, 7, 2151, 11723 > KSE: 84, 0, 384, 2158, 12096 > KSEGRP: 128, 0, 381, 2099, 12081 > THREAD: 328, 0, 389, 1915, 17090 > PROC: 440, 0, 182, 196, 1611 > yes but you've run it before.. teh system is still caching over 2000 threads :-) > Here's a snapshot during startup of my test script, while > I'm seeing lots of connect errors: > > UPCALL: 44, 0, 346, 1812, 11725 > KSE: 84, 0, 723, 1819, 12227 > KSEGRP: 128, 0, 720, 1760, 12253 > THREAD: 328, 0, 727, 1577, 17323 > PROC: 440, 0, 273, 105, 1642 > > I wanted to include a snapshot of the system after > everything stabalized, but I got the trap 12 panic > just after running the second vmstat. (p.s. put DDB in your kernel please try again. it is the most important one... Looks like you have lots of spare threads available (though they may not all be fully set up yet) so it's possibly not the kernel. > > > Also, are they system scope threds or process scope? > > What can I do to figure this out? part of it is the quesrtion of how many are in 'ps' but it looks to me like they are system scope threads or just badly implemented process scope threads.. > > > what happens with libthr? > > I haven't done very much testing with libthr, but the > little I've done shows that I still see the problems > handling that many connections, although it "catches > up" faster if I let it run. I did also see the trap 12 > crash with libthr. Not unlikely.. they share a lot of code. > > The perl script I've written to do all of this is > totally self-contained, creates the table it needs > for testing, and should work on any system. > > It needs p5-DBI, p5-DBD-mysql and p5-Digest-MD5 > from ports. > > http://hutta.com/test-threads.pl I wouldn't know perl if it came and kicked me in the shins.. how about (as dan said) a recipe to get to a test system from a newly installed freebsd system, with no packages loaded. > > I'd be interested to see if people have the same > problems I'm having on different hardware, with > different threading configurations, etc. try Dan's hack to the thread library to FORCE process scope threads. > > > of course it's fixable.. we just needd more info :-) > > I appreciate everyone's help on this, and I'd be glad > to provide any info I can. Just let me know what's > needed. > > Kris Gale > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.BSF.4.21.0402171711020.81059-100000>