Date: Tue, 17 Feb 2004 19:37:09 -0500 (EST) From: Daniel Eischen <eischen@vigrid.com> To: Kris Gale <kris-fbsd@asn.net> Cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 Message-ID: <Pine.GSO.4.10.10402171926550.4193-100000@pcnet5.pcnet.com> In-Reply-To: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 17 Feb 2004, Kris Gale wrote: > Hey Everyone, > > I've been trying to create a simple program to simulate > the load my production environment puts on MySQL. > > I'm not sure if I'm creating exactly the same problems > as I was seeing in production (and that I've described > on this list), but I have found some pretty interesting > things. > > What I seem to be seeing is a bogging down of MySQL > when new threads are being created in bursts. This > causes MySQL to temporarily become unresponsive, > and will sometimes crash the whole system. > > Here's what my test program is doing: > > - Fork X number of child processes, each opening > Y number of connections to the database. > > - Each child process loops through the Y connections it > has open, executing one select statement for each, then > starting over from the first. > > - If a particular database connection drops, it will enter > a loop attempting to reconnect, forever. > > When using 45 child processes and 20 connections for > each, everything is fine. (900 threads) You expect 900 threads on the server (perhaps a few more for overhead tasks), right? > If I bump it up to 90 children and 20 connections, I > start to see problems. The database is unable to > serve the incoming connections fast enough, and > existing connections become slow or entirely > unresponsive. However, if I leave it alone, eventually > things "catch up."* That is, as the database server > slowly manages to create new threads, all of the > incoming connect requests eventually succeed > (remember, they're looping). Once everything is > reconnected, I see 1800 threads in MySQL, and the > same query/second rate that I saw with 900 threads. What are the settings for kern.thread.*? > * Okay, not always. About half of the time, once > MySQL falls behind the incoming connections, and > connect attempts start to fail, the system will crash > with a "fatal trap 12: page fault while in kernel mode" > > In the X=90, Y=20 scenario (1800 threads), if the > test is allowed to continue until everything catches > up (about 5-10 minutes with KSE), I can stop and > start the test, triggering the burst of connection > attempts, but I see only a handful of connect errors. > However, if I stop and start mysql, I'll see the 10 > minutes of connect errors again. > > This seems to imply that somehow these threads > are being cached, or something is happening that > allows us to skip whatever bottleneck was causing > things to bog down. Userland does cache threads, but at most 100 threads and 50 (userland) KSEs and KSEGs. This seems to imply the problem is related to kernel resources. Hmm, userland also caches thread stacks (unlimited number), but thread stacks of default size are mmap'd onto the user stack. > Does this look like a fixable problem with KSE > to anyone on this list? I would like to see how MySQL works when using process scope threads. Do you know enough to hack it to do that? > Let me know if you'd like a copy of the perl script > I've written to try out all of these things. A step-by-step process of how to set up MySQL and run the tests would be nice along with the perl script. -- Dan Eischen
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.GSO.4.10.10402171926550.4193-100000>