Date: Tue, 17 Feb 2004 18:02:48 -0700 (MST) From: "Kris Gale" <kris-fbsd@asn.net> To: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 Message-ID: <49313.68.106.19.246.1077066168.squirrel@mail.asn.net> In-Reply-To: <Pine.BSF.4.21.0402171544310.81059-100000@InterJet.elischer.org> References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <Pine.BSF.4.21.0402171544310.81059-100000@InterJet.elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
>> What I seem to be seeing is a bogging down of MySQL >> when new threads are being created in bursts. This >> causes MySQL to temporarily become unresponsive, >> and will sometimes crash the whole system. > > My first question is: > "In what state are these threads waiting for work? > are they in the kernel, or are they in userland? > is there a single thread that listens on a socket and then hands the > work to worker threads using some userland synchronisation, or do the > threads enter the kernel and wait on the sockets themselves? > (i.e. what does ps -H show?) " I'm not really sure how to answer this. During the startup of my test script, when I'm seeing all of the connect errors, "ps auxwH" shows a growing number of mysqld threads. All but one have "SL" listed in the STAT column. The other one has "RL" listed. > what happens if you have 1 process with X*Y threads? > (just curious). No problems. 1800 threads run fine. The problem seems to be with the child processes trying to initialize connections simultaneously. I can also "solve" the problem by putting a sleep in so the child processes aren't forked at the same time. However, this is not a workable real-world solution, since the application I'm trying to simulate is a theoretical burst of connections from a web cluster. > is the server threaded process you are debugging, or is the > test program what is being debugged? The test program is just meant to simulate the problems I saw in production. I'm trying to test MySQL. > use vmstat -z to check the values for: Here's a snapshot from the system when mysql first starts: UPCALL: 44, 0, 7, 2151, 11723 KSE: 84, 0, 384, 2158, 12096 KSEGRP: 128, 0, 381, 2099, 12081 THREAD: 328, 0, 389, 1915, 17090 PROC: 440, 0, 182, 196, 1611 Here's a snapshot during startup of my test script, while I'm seeing lots of connect errors: UPCALL: 44, 0, 346, 1812, 11725 KSE: 84, 0, 723, 1819, 12227 KSEGRP: 128, 0, 720, 1760, 12253 THREAD: 328, 0, 727, 1577, 17323 PROC: 440, 0, 273, 105, 1642 I wanted to include a snapshot of the system after everything stabalized, but I got the trap 12 panic just after running the second vmstat. > Also, are they system scope threds or process scope? What can I do to figure this out? > what happens with libthr? I haven't done very much testing with libthr, but the little I've done shows that I still see the problems handling that many connections, although it "catches up" faster if I let it run. I did also see the trap 12 crash with libthr. The perl script I've written to do all of this is totally self-contained, creates the table it needs for testing, and should work on any system. It needs p5-DBI, p5-DBD-mysql and p5-Digest-MD5 from ports. http://hutta.com/test-threads.pl I'd be interested to see if people have the same problems I'm having on different hardware, with different threading configurations, etc. > of course it's fixable.. we just needd more info :-) I appreciate everyone's help on this, and I'd be glad to provide any info I can. Just let me know what's needed. Kris Gale
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49313.68.106.19.246.1077066168.squirrel>