From owner-freebsd-threads@FreeBSD.ORG Mon Feb 16 09:33:27 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F31BD16A4CE; Mon, 16 Feb 2004 09:33:26 -0800 (PST) Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id EA4C043D1D; Mon, 16 Feb 2004 09:33:26 -0800 (PST) (envelope-from rodrigc@crodrigues.org) Received: from h00609772adf0.ne.client2.attbi.com ([66.31.45.197]) by comcast.net (rwcrmhc13) with ESMTP id <2004021617332601500gs1f0e>; Mon, 16 Feb 2004 17:33:26 +0000 Received: from h00609772adf0.ne.client2.attbi.com (localhost.crodrigues.org [127.0.0.1])i1GHXghd060138; Mon, 16 Feb 2004 12:33:42 -0500 (EST) (envelope-from rodrigc@h00609772adf0.ne.client2.attbi.com) Received: (from rodrigc@localhost)i1GHXfaF060137; Mon, 16 Feb 2004 12:33:41 -0500 (EST) (envelope-from rodrigc) Date: Mon, 16 Feb 2004 12:33:41 -0500 From: Craig Rodrigues To: freebsd-gnats-submit@freebsd.org Message-ID: <20040216173341.GA60096@crodrigues.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i cc: hook@aktrad.ru cc: freebsd-threads@freebsd.org Subject: Re: misc/18824: gethostbyname is not thread safe X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Feb 2004 17:33:27 -0000 > The gethostbyname() is not thread safe. It causes lock ups if >compiled with threads. It would be helpful if this is mentioned in >the docs or if a thread-safe version is provided (gethostbyname_r >?) The gethostbyname() man page mentions in the BUGS section: "These functions use static data storage; if the data is needed for future use, it should be copied before any subsequent calls overwrite it." You should use the getaddrinfo() function which is reentrant, since it doesn't use static data storage. I think this PR should be closed. -- Craig Rodrigues rodrigc@crodrigues.org From owner-freebsd-threads@FreeBSD.ORG Mon Feb 16 09:40:13 2004 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B705E16A4CE for ; Mon, 16 Feb 2004 09:40:13 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 244B943D41 for ; Mon, 16 Feb 2004 09:40:07 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i1GHe7bv021113 for ; Mon, 16 Feb 2004 09:40:07 -0800 (PST) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i1GHe7vJ021112; Mon, 16 Feb 2004 09:40:07 -0800 (PST) (envelope-from gnats) Date: Mon, 16 Feb 2004 09:40:07 -0800 (PST) Message-Id: <200402161740.i1GHe7vJ021112@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Craig Rodrigues Subject: Re: misc/18824: gethostbyname is not thread safe X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Craig Rodrigues List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Feb 2004 17:40:13 -0000 The following reply was made to PR misc/18824; it has been noted by GNATS. From: Craig Rodrigues To: freebsd-gnats-submit@freebsd.org Cc: hook@aktrad.ru, freebsd-threads@freebsd.org Subject: Re: misc/18824: gethostbyname is not thread safe Date: Mon, 16 Feb 2004 12:33:41 -0500 > The gethostbyname() is not thread safe. It causes lock ups if >compiled with threads. It would be helpful if this is mentioned in >the docs or if a thread-safe version is provided (gethostbyname_r >?) The gethostbyname() man page mentions in the BUGS section: "These functions use static data storage; if the data is needed for future use, it should be copied before any subsequent calls overwrite it." You should use the getaddrinfo() function which is reentrant, since it doesn't use static data storage. I think this PR should be closed. -- Craig Rodrigues rodrigc@crodrigues.org From owner-freebsd-threads@FreeBSD.ORG Mon Feb 16 11:01:47 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6C07D16A4CE for ; Mon, 16 Feb 2004 11:01:47 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 508FB43D1D for ; Mon, 16 Feb 2004 11:01:47 -0800 (PST) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (peter@localhost [127.0.0.1]) i1GJ1lbv034284 for ; Mon, 16 Feb 2004 11:01:47 -0800 (PST) (envelope-from owner-bugmaster@freebsd.org) Received: (from peter@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i1GJ1kCA034278 for freebsd-threads@freebsd.org; Mon, 16 Feb 2004 11:01:46 -0800 (PST) (envelope-from owner-bugmaster@freebsd.org) Date: Mon, 16 Feb 2004 11:01:46 -0800 (PST) Message-Id: <200402161901.i1GJ1kCA034278@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: peter set sender to owner-bugmaster@freebsd.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Feb 2004 19:01:47 -0000 Current FreeBSD problem reports Critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/06/13] kern/19247 threads uthread_sigaction.c does not do anything 1 problem total. Serious problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/07/18] kern/20016 threads pthreads: Cannot set scheduling timer/Can o [2000/08/26] misc/20861 threads libc_r does not honor socket timeouts o [2001/01/19] bin/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVT o [2001/01/25] bin/24632 threads libc_r delicate deviation from libc in ha o [2001/01/25] misc/24641 threads pthread_rwlock_rdlock can deadlock o [2001/04/02] bin/26307 threads libc_r aborts when using the KDE media pl o [2001/11/26] bin/32295 threads pthread dont dequeue signals o [2002/02/01] i386/34536 threads accept() blocks other threads o [2002/05/25] kern/38549 threads the procces compiled whith pthread stoppe o [2002/06/27] bin/39922 threads [PATCH?] Threaded applications executed w o [2002/08/04] misc/41331 threads Pthread library open sets O_NONBLOCK flag o [2003/03/02] bin/48856 threads Setting SIGCHLD to SIG_IGN still leaves z o [2003/03/10] bin/49087 threads Signals lost in programs linked with libc a [2003/04/08] bin/50733 threads buildworld won't build, because of linkin o [2003/05/07] bin/51949 threads thread in accept cannot be cancelled 15 problems total. Non-critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/05/25] misc/18824 threads gethostbyname is not thread safe o [2000/10/21] misc/22190 threads A threaded read(2) from a socketpair(2) f o [2001/09/09] bin/30464 threads pthread mutex attributes -- pshared o [2002/05/02] bin/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwri o [2002/07/16] misc/40671 threads pthread_cancel doesn't remove thread from 5 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Feb 16 16:10:18 2004 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C065016A4D7 for ; Mon, 16 Feb 2004 16:10:18 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id BB29143D1F for ; Mon, 16 Feb 2004 16:10:18 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i1H0AIbv084553 for ; Mon, 16 Feb 2004 16:10:18 -0800 (PST) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i1H0AI09084537; Mon, 16 Feb 2004 16:10:18 -0800 (PST) (envelope-from gnats) Date: Mon, 16 Feb 2004 16:10:18 -0800 (PST) Message-Id: <200402170010.i1H0AI09084537@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Craig Rodrigues Subject: Re: misc/40671: pthread_cancel doesn't remove thread from condition queue X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Craig Rodrigues List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Feb 2004 00:10:18 -0000 The following reply was made to PR misc/40671; it has been noted by GNATS. From: Craig Rodrigues To: freebsd-gnats-submit@freebsd.org Cc: richard@cogsci.ed.ac.uk Subject: Re: misc/40671: pthread_cancel doesn't remove thread from condition queue Date: Mon, 16 Feb 2004 19:06:32 -0500 Hi, On FreeBSD-CURRENT, if I link your testcase to libc_r, I get this error: Fatal error '_waitq_remove: Not in queue' at line 350 in file /usr/src/lib/libc_r/uthread/uthread_priority_queue.c (errno = 0) Abort trap (core dumped) If I link your testcase to libpthread, I get no error. It looks like this is not a problem in the new libpthread library. -- Craig Rodrigues rodrigc@crodrigues.org From owner-freebsd-threads@FreeBSD.ORG Mon Feb 16 20:43:53 2004 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4D95616A4CE; Mon, 16 Feb 2004 20:43:53 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D59143D1F; Mon, 16 Feb 2004 20:43:53 -0800 (PST) (envelope-from deischen@FreeBSD.org) Received: from freefall.freebsd.org (deischen@localhost [127.0.0.1]) i1H4hrbv023708; Mon, 16 Feb 2004 20:43:53 -0800 (PST) (envelope-from deischen@freefall.freebsd.org) Received: (from deischen@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i1H4hqRm023704; Mon, 16 Feb 2004 20:43:52 -0800 (PST) (envelope-from deischen) Date: Mon, 16 Feb 2004 20:43:52 -0800 (PST) From: Daniel Eischen Message-Id: <200402170443.i1H4hqRm023704@freefall.freebsd.org> To: richard@cogsci.ed.ac.uk, deischen@FreeBSD.org, freebsd-threads@FreeBSD.org Subject: Re: misc/40671: pthread_cancel doesn't remove thread from condition queue X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Feb 2004 04:43:53 -0000 Synopsis: pthread_cancel doesn't remove thread from condition queue State-Changed-From-To: open->suspended State-Changed-By: deischen State-Changed-When: Mon Feb 16 20:39:48 PST 2004 State-Changed-Why: This seems to only be a problem in libc_r and works OK in libpthread. Mark it as suspended for now, perhaps a project for someone wanting to keep libc_r on life-support for a little while longer. In discussion with: Craig Rodrigues http://www.freebsd.org/cgi/query-pr.cgi?pr=40671 From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 14:52:29 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 191A216A4CE for ; Tue, 17 Feb 2004 14:52:29 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id F3E5A43D1F for ; Tue, 17 Feb 2004 14:52:28 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 42504 invoked by uid 80); 17 Feb 2004 22:52:28 -0000 Received: from 68.106.19.246 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Tue, 17 Feb 2004 15:52:28 -0700 (MST) Message-ID: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> Date: Tue, 17 Feb 2004 15:52:28 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Feb 2004 22:52:29 -0000 Hey Everyone, I've been trying to create a simple program to simulate the load my production environment puts on MySQL. I'm not sure if I'm creating exactly the same problems as I was seeing in production (and that I've described on this list), but I have found some pretty interesting things. What I seem to be seeing is a bogging down of MySQL when new threads are being created in bursts. This causes MySQL to temporarily become unresponsive, and will sometimes crash the whole system. Here's what my test program is doing: - Fork X number of child processes, each opening Y number of connections to the database. - Each child process loops through the Y connections it has open, executing one select statement for each, then starting over from the first. - If a particular database connection drops, it will enter a loop attempting to reconnect, forever. When using 45 child processes and 20 connections for each, everything is fine. (900 threads) If I bump it up to 90 children and 20 connections, I start to see problems. The database is unable to serve the incoming connections fast enough, and existing connections become slow or entirely unresponsive. However, if I leave it alone, eventually things "catch up."* That is, as the database server slowly manages to create new threads, all of the incoming connect requests eventually succeed (remember, they're looping). Once everything is reconnected, I see 1800 threads in MySQL, and the same query/second rate that I saw with 900 threads. * Okay, not always. About half of the time, once MySQL falls behind the incoming connections, and connect attempts start to fail, the system will crash with a "fatal trap 12: page fault while in kernel mode" In the X=90, Y=20 scenario (1800 threads), if the test is allowed to continue until everything catches up (about 5-10 minutes with KSE), I can stop and start the test, triggering the burst of connection attempts, but I see only a handful of connect errors. However, if I stop and start mysql, I'll see the 10 minutes of connect errors again. This seems to imply that somehow these threads are being cached, or something is happening that allows us to skip whatever bottleneck was causing things to bog down. Does this look like a fixable problem with KSE to anyone on this list? Let me know if you'd like a copy of the perl script I've written to try out all of these things. Kris Gale From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 15:31:37 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DE7BE16A4CE for ; Tue, 17 Feb 2004 15:31:37 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id AFE2043D2D for ; Tue, 17 Feb 2004 15:31:37 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 44330 invoked by uid 80); 17 Feb 2004 23:31:37 -0000 Received: from 68.106.19.246 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Tue, 17 Feb 2004 16:31:37 -0700 (MST) Message-ID: <49166.68.106.19.246.1077060697.squirrel@mail.asn.net> In-Reply-To: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> Date: Tue, 17 Feb 2004 16:31:37 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Feb 2004 23:31:38 -0000 > * Okay, not always. About half of the time, once > MySQL falls behind the incoming connections, and > connect attempts start to fail, the system will crash > with a "fatal trap 12: page fault while in kernel mode" By the way, the panic message says that the error is in line 819 of trap.c Kris Gale From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 16:05:20 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC9CA16A4CE for ; Tue, 17 Feb 2004 16:05:20 -0800 (PST) Received: from sccrmhc13.comcast.net (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9661B43D1F for ; Tue, 17 Feb 2004 16:05:20 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc13) with ESMTP id <20040218000514016003hfise>; Wed, 18 Feb 2004 00:05:19 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA83035; Tue, 17 Feb 2004 16:05:10 -0800 (PST) Date: Tue, 17 Feb 2004 16:05:06 -0800 (PST) From: Julian Elischer To: Kris Gale In-Reply-To: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 00:05:20 -0000 On Tue, 17 Feb 2004, Kris Gale wrote: > Hey Everyone, > > I've been trying to create a simple program to simulate > the load my production environment puts on MySQL. great! > > What I seem to be seeing is a bogging down of MySQL > when new threads are being created in bursts. This > causes MySQL to temporarily become unresponsive, > and will sometimes crash the whole system. My first question is: "In what state are these threads waiting for work? are they in the kernel, or are they in userland? is there a single thread that listens on a socket and then hands the work to worker threads using some userland synchronisation, or do the threads enter the kernel and wait on the sockets themselves? (i.e. what does ps -H show?) " > > Here's what my test program is doing: > > - Fork X number of child processes, each opening > Y number of connections to the database. WHere are the threads coming into this? Is it a htreaded server or a threaded test program openning tcp connections to the server? what happens if you have 1 process with X*Y threads? (just curious). > > - Each child process loops through the Y connections it > has open, executing one select statement for each, then > starting over from the first. > > - If a particular database connection drops, it will enter > a loop attempting to reconnect, forever. > > When using 45 child processes and 20 connections for > each, everything is fine. (900 threads) All talking to teh same database server? is the server threaded process you are debugging, or is the test program what is being debugged? > > If I bump it up to 90 children and 20 connections, I > start to see problems. The database is unable to > serve the incoming connections fast enough, and > existing connections become slow or entirely > unresponsive. However, if I leave it alone, eventually > things "catch up."* That is, as the database server > slowly manages to create new threads, all of the > incoming connect requests eventually succeed > (remember, they're looping). So it looks like you are trying to debug threads in the server.. what state are threads in that are not doing work? thread creation has several parts depending on whether the threads are to run in the kernel or not.. Actually creating the threads in userland takes memory allocations and structure munging.. making them come to life may require the creation of a new thread in the kernel. Threads are held in 'UMA' and are cached.. use vmstat -z to check the values for: UPCALL: 40, 0, 0, 0, 0 KSE: 64, 0, 200, 110, 200 KSEGRP: 112, 0, 200, 52, 200 THREAD: 236, 0, 200, 24, 200 PROC: 508, 0, 159, 41, 672667 To help you account for the m, you should know that each process comes with a single preallocated KSEG , KSE and thread. so 159+41 processes are created, (159 in use) meaning that there are 200 KSE, KSEGroup and thread structures associated with them, (even teh ones not yet in use). > Once everything is > reconnected, I see 1800 threads in MySQL, and the > same query/second rate that I saw with 900 threads. > > * Okay, not always. About half of the time, once > MySQL falls behind the incoming connections, and > connect attempts start to fail, the system will crash > with a "fatal trap 12: page fault while in kernel mode" "where" in the kernel? > > In the X=90, Y=20 scenario (1800 threads), if the > test is allowed to continue until everything catches > up (about 5-10 minutes with KSE), I can stop and > start the test, triggering the burst of connection > attempts, but I see only a handful of connect errors. > However, if I stop and start mysql, I'll see the 10 > minutes of connect errors again. This suggests that the slowness is in malloc'ing 1800 stacks etc. but much depends on what stat those threads are in when not working.. Also, are they system scope threds or process scope? what happens with libthr? > > This seems to imply that somehow these threads > are being cached, or something is happening that > allows us to skip whatever bottleneck was causing > things to bog down. yes threads are cached both in userland and in the kernel > > Does this look like a fixable problem with KSE > to anyone on this list? of course it's fixable.. we just needd more info :-) > > Let me know if you'd like a copy of the perl script > I've written to try out all of these things. > > Kris Gale > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 16:08:44 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4932116A4CE for ; Tue, 17 Feb 2004 16:08:44 -0800 (PST) Received: from sccrmhc12.comcast.net (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id 169FD43D2D for ; Tue, 17 Feb 2004 16:08:44 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc12) with ESMTP id <2004021800084201200jecjme>; Wed, 18 Feb 2004 00:08:43 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA83046; Tue, 17 Feb 2004 16:08:42 -0800 (PST) Date: Tue, 17 Feb 2004 16:08:42 -0800 (PST) From: Julian Elischer To: Kris Gale In-Reply-To: <49166.68.106.19.246.1077060697.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 00:08:44 -0000 umm what version of trap.c? On Tue, 17 Feb 2004, Kris Gale wrote: > > * Okay, not always. About half of the time, once > > MySQL falls behind the incoming connections, and > > connect attempts start to fail, the system will crash > > with a "fatal trap 12: page fault while in kernel mode" > > By the way, the panic message says that the error is > in line 819 of trap.c > > Kris Gale > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 16:37:14 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 94DBB16A4D0 for ; Tue, 17 Feb 2004 16:37:14 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3BFF743D1F for ; Tue, 17 Feb 2004 16:37:14 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1I0b9SQ007939; Tue, 17 Feb 2004 19:37:09 -0500 (EST) Date: Tue, 17 Feb 2004 19:37:09 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Gale In-Reply-To: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 00:37:14 -0000 On Tue, 17 Feb 2004, Kris Gale wrote: > Hey Everyone, > > I've been trying to create a simple program to simulate > the load my production environment puts on MySQL. > > I'm not sure if I'm creating exactly the same problems > as I was seeing in production (and that I've described > on this list), but I have found some pretty interesting > things. > > What I seem to be seeing is a bogging down of MySQL > when new threads are being created in bursts. This > causes MySQL to temporarily become unresponsive, > and will sometimes crash the whole system. > > Here's what my test program is doing: > > - Fork X number of child processes, each opening > Y number of connections to the database. > > - Each child process loops through the Y connections it > has open, executing one select statement for each, then > starting over from the first. > > - If a particular database connection drops, it will enter > a loop attempting to reconnect, forever. > > When using 45 child processes and 20 connections for > each, everything is fine. (900 threads) You expect 900 threads on the server (perhaps a few more for overhead tasks), right? > If I bump it up to 90 children and 20 connections, I > start to see problems. The database is unable to > serve the incoming connections fast enough, and > existing connections become slow or entirely > unresponsive. However, if I leave it alone, eventually > things "catch up."* That is, as the database server > slowly manages to create new threads, all of the > incoming connect requests eventually succeed > (remember, they're looping). Once everything is > reconnected, I see 1800 threads in MySQL, and the > same query/second rate that I saw with 900 threads. What are the settings for kern.thread.*? > * Okay, not always. About half of the time, once > MySQL falls behind the incoming connections, and > connect attempts start to fail, the system will crash > with a "fatal trap 12: page fault while in kernel mode" > > In the X=90, Y=20 scenario (1800 threads), if the > test is allowed to continue until everything catches > up (about 5-10 minutes with KSE), I can stop and > start the test, triggering the burst of connection > attempts, but I see only a handful of connect errors. > However, if I stop and start mysql, I'll see the 10 > minutes of connect errors again. > > This seems to imply that somehow these threads > are being cached, or something is happening that > allows us to skip whatever bottleneck was causing > things to bog down. Userland does cache threads, but at most 100 threads and 50 (userland) KSEs and KSEGs. This seems to imply the problem is related to kernel resources. Hmm, userland also caches thread stacks (unlimited number), but thread stacks of default size are mmap'd onto the user stack. > Does this look like a fixable problem with KSE > to anyone on this list? I would like to see how MySQL works when using process scope threads. Do you know enough to hack it to do that? > Let me know if you'd like a copy of the perl script > I've written to try out all of these things. A step-by-step process of how to set up MySQL and run the tests would be nice along with the perl script. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 16:49:54 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2EFD816A4CE for ; Tue, 17 Feb 2004 16:49:54 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id E358443D1F for ; Tue, 17 Feb 2004 16:49:53 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1I0noSQ011531; Tue, 17 Feb 2004 19:49:50 -0500 (EST) Date: Tue, 17 Feb 2004 19:49:50 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Gale In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 00:49:54 -0000 On Tue, 17 Feb 2004, Daniel Eischen wrote: > On Tue, 17 Feb 2004, Kris Gale wrote: > > Does this look like a fixable problem with KSE > > to anyone on this list? > > I would like to see how MySQL works when using process > scope threads. Do you know enough to hack it to do that? Alternatively, this patch to libpthread should force all threads to be process scope. -- Dan Eischen Index: thread/thr_create.c =================================================================== RCS file: /opt/FreeBSD/cvs/src/lib/libpthread/thread/thr_create.c,v retrieving revision 1.53 diff -u -r1.53 thr_create.c --- thread/thr_create.c 8 Jan 2004 15:37:09 -0000 1.53 +++ thread/thr_create.c 18 Feb 2004 00:45:48 -0000 @@ -128,6 +128,7 @@ new_thread->attr = _pthread_attr_default; else { new_thread->attr = *(*attr); + new_thread->attr &= ~PTHREAD_SCOPE_SYSTEM; if ((*attr)->sched_inherit == PTHREAD_INHERIT_SCHED) { /* inherit scheduling contention scop */ if (curthread->attr.flags &= PTHREAD_SCOPE_SYSTEM) From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 17:02:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C557616A4CE for ; Tue, 17 Feb 2004 17:02:48 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id 87B3343D1D for ; Tue, 17 Feb 2004 17:02:48 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 47092 invoked by uid 80); 18 Feb 2004 01:02:48 -0000 Received: from 68.106.19.246 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Tue, 17 Feb 2004 18:02:48 -0700 (MST) Message-ID: <49313.68.106.19.246.1077066168.squirrel@mail.asn.net> In-Reply-To: References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> Date: Tue, 17 Feb 2004 18:02:48 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 01:02:49 -0000 >> What I seem to be seeing is a bogging down of MySQL >> when new threads are being created in bursts. This >> causes MySQL to temporarily become unresponsive, >> and will sometimes crash the whole system. > > My first question is: > "In what state are these threads waiting for work? > are they in the kernel, or are they in userland? > is there a single thread that listens on a socket and then hands the > work to worker threads using some userland synchronisation, or do the > threads enter the kernel and wait on the sockets themselves? > (i.e. what does ps -H show?) " I'm not really sure how to answer this. During the startup of my test script, when I'm seeing all of the connect errors, "ps auxwH" shows a growing number of mysqld threads. All but one have "SL" listed in the STAT column. The other one has "RL" listed. > what happens if you have 1 process with X*Y threads? > (just curious). No problems. 1800 threads run fine. The problem seems to be with the child processes trying to initialize connections simultaneously. I can also "solve" the problem by putting a sleep in so the child processes aren't forked at the same time. However, this is not a workable real-world solution, since the application I'm trying to simulate is a theoretical burst of connections from a web cluster. > is the server threaded process you are debugging, or is the > test program what is being debugged? The test program is just meant to simulate the problems I saw in production. I'm trying to test MySQL. > use vmstat -z to check the values for: Here's a snapshot from the system when mysql first starts: UPCALL: 44, 0, 7, 2151, 11723 KSE: 84, 0, 384, 2158, 12096 KSEGRP: 128, 0, 381, 2099, 12081 THREAD: 328, 0, 389, 1915, 17090 PROC: 440, 0, 182, 196, 1611 Here's a snapshot during startup of my test script, while I'm seeing lots of connect errors: UPCALL: 44, 0, 346, 1812, 11725 KSE: 84, 0, 723, 1819, 12227 KSEGRP: 128, 0, 720, 1760, 12253 THREAD: 328, 0, 727, 1577, 17323 PROC: 440, 0, 273, 105, 1642 I wanted to include a snapshot of the system after everything stabalized, but I got the trap 12 panic just after running the second vmstat. > Also, are they system scope threds or process scope? What can I do to figure this out? > what happens with libthr? I haven't done very much testing with libthr, but the little I've done shows that I still see the problems handling that many connections, although it "catches up" faster if I let it run. I did also see the trap 12 crash with libthr. The perl script I've written to do all of this is totally self-contained, creates the table it needs for testing, and should work on any system. It needs p5-DBI, p5-DBD-mysql and p5-Digest-MD5 from ports. http://hutta.com/test-threads.pl I'd be interested to see if people have the same problems I'm having on different hardware, with different threading configurations, etc. > of course it's fixable.. we just needd more info :-) I appreciate everyone's help on this, and I'd be glad to provide any info I can. Just let me know what's needed. Kris Gale From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 17:12:49 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3618116A4CE for ; Tue, 17 Feb 2004 17:12:49 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id 1E14F43D1D for ; Tue, 17 Feb 2004 17:12:49 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 47447 invoked by uid 80); 18 Feb 2004 01:12:48 -0000 Received: from 68.106.19.246 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Tue, 17 Feb 2004 18:12:48 -0700 (MST) Message-ID: <49327.68.106.19.246.1077066768.squirrel@mail.asn.net> In-Reply-To: References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> Date: Tue, 17 Feb 2004 18:12:48 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 01:12:49 -0000 > You expect 900 threads on the server (perhaps a few more > for overhead tasks), right? Yep, and that's what I see. MySQL reports 900 threads in this instance. > What are the settings for kern.thread.*? kern.threads.debug: 0 kern.threads.max_threads_per_proc: 2500 kern.threads.max_groups_per_proc: 2500 kern.threads.max_threads_hits: 0 kern.threads.virtual_cpu: 4 > I would like to see how MySQL works when using process > scope threads. Do you know enough to hack it to do that? Wish I did. =) > A step-by-step process of how to set up MySQL and run > the tests would be nice along with the perl script. The script is at http://hutta.com/test-threads.pl A current build will automatically use KSE for MySQL, but a build from a few weeks ago will need the proper entries in libmap.conf. Set OVERWRITE_DB=yes when installing the port (databases/mysql40-server) to get default databases set up, and my perl script will create everything it needs for testing. I haven't seen any difference in using BUILD_OPTIMIZED or BUILD_STATIC, so use those as you so feel. I'm using the following config file for mysql: ---[ BEGIN /etc/my.cnf ]--- [client] port = 3306 socket = /tmp/mysql.sock [mysqld] set-variable = max_connections=6000 port = 3306 socket = /tmp/mysql.sock skip-locking set-variable = key_buffer=1024M set-variable = max_allowed_packet=1M set-variable = table_cache=512 set-variable = sort_buffer=1M set-variable = record_buffer=1M set-variable = myisam_sort_buffer_size=128M set-variable = thread_cache=8 set-variable = thread_concurrency=4 server-id = 1 skip-innodb tmpdir = /usr/tmp/ [mysql] no-auto-rehash ---[ END /etc/my.cnf ]---- Kris Gale From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 17:21:52 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 691EC16A4D0 for ; Tue, 17 Feb 2004 17:21:52 -0800 (PST) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60D7343D2D for ; Tue, 17 Feb 2004 17:21:52 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <20040218012151013007mut7e>; Wed, 18 Feb 2004 01:21:51 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id RAA83891; Tue, 17 Feb 2004 17:21:51 -0800 (PST) Date: Tue, 17 Feb 2004 17:21:49 -0800 (PST) From: Julian Elischer To: Kris Gale In-Reply-To: <49313.68.106.19.246.1077066168.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 01:21:52 -0000 On Tue, 17 Feb 2004, Kris Gale wrote: > >> What I seem to be seeing is a bogging down of MySQL > >> when new threads are being created in bursts. This > >> causes MySQL to temporarily become unresponsive, > >> and will sometimes crash the whole system. > > > > My first question is: > > "In what state are these threads waiting for work? > > are they in the kernel, or are they in userland? > > is there a single thread that listens on a socket and then hands the > > work to worker threads using some userland synchronisation, or do the > > threads enter the kernel and wait on the sockets themselves? > > (i.e. what does ps -H show?) " > > I'm not really sure how to answer this. During the startup > of my test script, when I'm seeing all of the connect errors, > "ps auxwH" shows a growing number of mysqld threads. how many? What is most important is, "How many (in ps) when it has 1800 threads and no work.." (how do you know it has 1800 threads?) what does teh ps show? can you send it to me? > All but one have "SL" listed in the STAT column. The other > one has "RL" listed. > > > what happens if you have 1 process with X*Y threads? > > (just curious). > > No problems. 1800 threads run fine. The problem > seems to be with the child processes trying to initialize > connections simultaneously. I can also "solve" the > problem by putting a sleep in so the child processes > aren't forked at the same time. However, this is > not a workable real-world solution, since the application > I'm trying to simulate is a theoretical burst of connections > from a web cluster. > > > is the server threaded process you are debugging, or is the > > test program what is being debugged? > > The test program is just meant to simulate the problems I > saw in production. I'm trying to test MySQL. > > > use vmstat -z to check the values for: > > Here's a snapshot from the system when mysql first starts: > > UPCALL: 44, 0, 7, 2151, 11723 > KSE: 84, 0, 384, 2158, 12096 > KSEGRP: 128, 0, 381, 2099, 12081 > THREAD: 328, 0, 389, 1915, 17090 > PROC: 440, 0, 182, 196, 1611 > yes but you've run it before.. teh system is still caching over 2000 threads :-) > Here's a snapshot during startup of my test script, while > I'm seeing lots of connect errors: > > UPCALL: 44, 0, 346, 1812, 11725 > KSE: 84, 0, 723, 1819, 12227 > KSEGRP: 128, 0, 720, 1760, 12253 > THREAD: 328, 0, 727, 1577, 17323 > PROC: 440, 0, 273, 105, 1642 > > I wanted to include a snapshot of the system after > everything stabalized, but I got the trap 12 panic > just after running the second vmstat. (p.s. put DDB in your kernel please try again. it is the most important one... Looks like you have lots of spare threads available (though they may not all be fully set up yet) so it's possibly not the kernel. > > > Also, are they system scope threds or process scope? > > What can I do to figure this out? part of it is the quesrtion of how many are in 'ps' but it looks to me like they are system scope threads or just badly implemented process scope threads.. > > > what happens with libthr? > > I haven't done very much testing with libthr, but the > little I've done shows that I still see the problems > handling that many connections, although it "catches > up" faster if I let it run. I did also see the trap 12 > crash with libthr. Not unlikely.. they share a lot of code. > > The perl script I've written to do all of this is > totally self-contained, creates the table it needs > for testing, and should work on any system. > > It needs p5-DBI, p5-DBD-mysql and p5-Digest-MD5 > from ports. > > http://hutta.com/test-threads.pl I wouldn't know perl if it came and kicked me in the shins.. how about (as dan said) a recipe to get to a test system from a newly installed freebsd system, with no packages loaded. > > I'd be interested to see if people have the same > problems I'm having on different hardware, with > different threading configurations, etc. try Dan's hack to the thread library to FORCE process scope threads. > > > of course it's fixable.. we just needd more info :-) > > I appreciate everyone's help on this, and I'd be glad > to provide any info I can. Just let me know what's > needed. > > Kris Gale > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 17:23:37 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ACB6C16A4CE for ; Tue, 17 Feb 2004 17:23:37 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id 7CB7F43D1F for ; Tue, 17 Feb 2004 17:23:37 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 47829 invoked by uid 80); 18 Feb 2004 01:23:37 -0000 Received: from 68.106.19.246 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Tue, 17 Feb 2004 18:23:37 -0700 (MST) Message-ID: <49363.68.106.19.246.1077067417.squirrel@mail.asn.net> In-Reply-To: <20040218000052.GA70706@crodrigues.org> References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <20040218000052.GA70706@crodrigues.org> Date: Tue, 17 Feb 2004 18:23:37 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 01:23:37 -0000 > (1) steps you used to build MySQL, ie. cd /usr/ports/blah, make install This is assuming you have a current build (with KSE as the default threading system.) # cd /usr/ports/databases/mysql40-server # make OVERWRITE_DB=yes install Then, to install the dependencies for my script, and to fetch it: # cd ../p5-DBI # make install # cd ../p5-DBD-mysql40 # make install # cd ~ # fetch http://hutta.com/test-threads.pl To test: # /usr/local/etc/rc.d/mysql-server.sh start # ./test-threads.pl > (2) output of the following: > > sysctl hw.model hw.physmem hw.usermem kern.threads.max_threads_per_proc \ > kern.threads.max_groups_per_proc kern.threads.max_threads_hits \ > kern.threads.virtual_cpu hw.model: Intel(R) Xeon(TM) CPU 2.66GHz hw.physmem: 4151197696 hw.usermem: 4111593472 kern.threads.max_threads_per_proc: 2500 kern.threads.max_groups_per_proc: 2500 kern.threads.max_threads_hits: 0 kern.threads.virtual_cpu: 4 Kris Gale From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 19:45:18 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 78FBD16A4CE for ; Tue, 17 Feb 2004 19:45:18 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 37E5543D1D for ; Tue, 17 Feb 2004 19:45:18 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1I3jESQ001628; Tue, 17 Feb 2004 22:45:14 -0500 (EST) Date: Tue, 17 Feb 2004 22:45:14 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Gale In-Reply-To: <49313.68.106.19.246.1077066168.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 03:45:18 -0000 On Tue, 17 Feb 2004, Kris Gale wrote: > >> What I seem to be seeing is a bogging down of MySQL > >> when new threads are being created in bursts. This > >> causes MySQL to temporarily become unresponsive, > >> and will sometimes crash the whole system. > > > > My first question is: > > "In what state are these threads waiting for work? > > are they in the kernel, or are they in userland? > > is there a single thread that listens on a socket and then hands the > > work to worker threads using some userland synchronisation, or do the > > threads enter the kernel and wait on the sockets themselves? > > (i.e. what does ps -H show?) " > > I'm not really sure how to answer this. During the startup > of my test script, when I'm seeing all of the connect errors, > "ps auxwH" shows a growing number of mysqld threads. > All but one have "SL" listed in the STAT column. The other > one has "RL" listed. > > > what happens if you have 1 process with X*Y threads? > > (just curious). > > No problems. 1800 threads run fine. The problem > seems to be with the child processes trying to initialize > connections simultaneously. I can also "solve" the > problem by putting a sleep in so the child processes > aren't forked at the same time. However, this is > not a workable real-world solution, since the application > I'm trying to simulate is a theoretical burst of connections > from a web cluster. I'm not sure what all the mumbo-jumbo is in your /etc/my.cnf, but commenting out this line: #set-variable = key_buffer=1024M let your perl script run with 90 children w/20 connections per. Everything seems to be running here, but I'm not sure what I'm looking for. Hmm, I also set: set-variable = thread_cache=1 set-variable = thread_concurrency=1 -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 22:46:50 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ECD3516A4CE for ; Tue, 17 Feb 2004 22:46:50 -0800 (PST) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4929F43D2D for ; Tue, 17 Feb 2004 22:46:50 -0800 (PST) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h81.vuokselantie10.fi [193.64.42.129]) by rms04.rommon.net (8.12.9p1/8.12.9) with ESMTP id i1I6kicM095162; Wed, 18 Feb 2004 08:46:45 +0200 (EET) (envelope-from pete@he.iki.fi) Message-ID: <40330A4C.7030508@he.iki.fi> Date: Wed, 18 Feb 2004 08:46:36 +0200 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Daniel Eischen References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 06:46:51 -0000 Daniel Eischen wrote: >A step-by-step process of how to set up MySQL and run >the tests would be nice along with the perl script. > > > As a related note, since upgrading to 5.2.1-RC (Feb 6) mysql now shutdowns properly for me. No unresponsiveness which used to plague kse since day one any more. (so far my test data is limited but IŽll report if anything changes for the worse) Pete From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 22:50:14 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9540016A4CE for ; Tue, 17 Feb 2004 22:50:14 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3C59C43D1D for ; Tue, 17 Feb 2004 22:50:14 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1I6oASQ021176; Wed, 18 Feb 2004 01:50:10 -0500 (EST) Date: Wed, 18 Feb 2004 01:50:10 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Petri Helenius In-Reply-To: <40330A4C.7030508@he.iki.fi> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 06:50:14 -0000 On Wed, 18 Feb 2004, Petri Helenius wrote: > Daniel Eischen wrote: >=20 > >A step-by-step process of how to set up MySQL and run > >the tests would be nice along with the perl script. > > > > =20 > > > As a related note, since upgrading to 5.2.1-RC (Feb 6) mysql now=20 > shutdowns properly for me. No unresponsiveness which used to plague kse= =20 > since day one any more. (so far my test data is limited but I=B4ll report= =20 > if anything changes for the worse) Oh, I assumed you guys were tracking -current, not sticking with the last release... --=20 Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Feb 17 23:02:46 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F3BEE16A4CE for ; Tue, 17 Feb 2004 23:02:45 -0800 (PST) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4F84743D1F for ; Tue, 17 Feb 2004 23:02:45 -0800 (PST) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h81.vuokselantie10.fi [193.64.42.129]) by rms04.rommon.net (8.12.9p1/8.12.9) with ESMTP id i1I72fcM095212; Wed, 18 Feb 2004 09:02:41 +0200 (EET) (envelope-from pete@he.iki.fi) Message-ID: <40330E09.9090900@he.iki.fi> Date: Wed, 18 Feb 2004 09:02:33 +0200 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Daniel Eischen References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 07:02:46 -0000 Daniel Eischen wrote: >Oh, I assumed you guys were tracking -current, not sticking with >the last release... > > > I could to that on the test boxes but IŽm also trying to figure out if I could switch from linuxthreads to kse in production stuff. After IŽm convinced that can be done, I have more load to put on the -CURRENT boxes. It would be kind of scary to run current "for real" :). How large the differences are between RELENG_5_2 and HEAD ? (I could look in cvs but it probably does not give the right picture) Pete From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 01:19:23 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DA1C416A4CE for ; Wed, 18 Feb 2004 01:19:23 -0800 (PST) Received: from exchhz01.viatech.com.cn (ip-40-162-97-218.anlai.com [218.97.162.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1457743D1D for ; Wed, 18 Feb 2004 01:19:16 -0800 (PST) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (DAVIDWNT [10.4.1.99]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 1WN49C80; Wed, 18 Feb 2004 17:19:04 +0800 Message-ID: <40332E18.6090209@freebsd.org> Date: Wed, 18 Feb 2004 17:19:20 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kris Gale References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <49327.68.106.19.246.1077066768.squirrel@mail.asn.net> In-Reply-To: <49327.68.106.19.246.1077066768.squirrel@mail.asn.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 09:19:24 -0000 Can you tell us what scheduler are you using ? if it is ULE, can you try 4BSD too ? I think this is a heavy load for scheduler too, not just for pthread library. Kris Gale wrote: >>You expect 900 threads on the server (perhaps a few more >>for overhead tasks), right? >> >> > >Yep, and that's what I see. MySQL reports 900 threads in >this instance. > > > >>What are the settings for kern.thread.*? >> >> > >kern.threads.debug: 0 >kern.threads.max_threads_per_proc: 2500 >kern.threads.max_groups_per_proc: 2500 >kern.threads.max_threads_hits: 0 >kern.threads.virtual_cpu: 4 > > From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 06:59:51 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D29716A4CF for ; Wed, 18 Feb 2004 06:59:51 -0800 (PST) Received: from smtp.des.no (flood.des.no [217.116.83.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6BEA343D2D for ; Wed, 18 Feb 2004 06:59:51 -0800 (PST) (envelope-from des@des.no) Received: by smtp.des.no (Pony Express, from userid 666) id 9A6305309; Wed, 18 Feb 2004 15:59:49 +0100 (CET) Received: from dwp.des.no (des.no [80.203.228.37]) by smtp.des.no (Pony Express) with ESMTP id 3A1875308; Wed, 18 Feb 2004 15:59:42 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 2602) id D2A7C33C6F; Wed, 18 Feb 2004 15:59:41 +0100 (CET) To: "Kris Gale" References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <49166.68.106.19.246.1077060697.squirrel@mail.asn.net> From: des@des.no (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Wed, 18 Feb 2004 15:59:41 +0100 In-Reply-To: <49166.68.106.19.246.1077060697.squirrel@mail.asn.net> (Kris Gale's message of "Tue, 17 Feb 2004 16:31:37 -0700 (MST)") Message-ID: User-Agent: Gnus/5.090024 (Oort Gnus v0.24) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on flood.des.no X-Spam-Level: X-Spam-Status: No, hits=0.0 required=5.0 tests=AWL autolearn=no version=2.63 cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 14:59:51 -0000 "Kris Gale" writes: > By the way, the panic message says that the error is > in line 819 of trap.c that's completely useless, "fatal trap 12" panics will always come from line 819 of trap.c. the real error is somewhere else, and we need a backtrace to figure out where. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 07:18:49 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 151AD16A4D1 for ; Wed, 18 Feb 2004 07:18:49 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id C5FF143D3F for ; Wed, 18 Feb 2004 07:18:48 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1IFIfSQ006883; Wed, 18 Feb 2004 10:18:44 -0500 (EST) Date: Wed, 18 Feb 2004 10:18:41 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Petri Helenius In-Reply-To: <40330E09.9090900@he.iki.fi> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 15:18:49 -0000 On Wed, 18 Feb 2004, Petri Helenius wrote: > Daniel Eischen wrote: >=20 > >Oh, I assumed you guys were tracking -current, not sticking with > >the last release... > > > > =20 > > > I could to that on the test boxes but I=B4m also trying to figure out if = I=20 > could switch from linuxthreads to kse in production stuff. After I=B4m=20 > convinced that can be done, I have more load to put on the -CURRENT=20 > boxes. It would be kind of scary to run current "for real" :). >=20 > How large the differences are between RELENG_5_2 and HEAD ? (I could=20 > look in cvs but it probably does not give the right picture) http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libpthread/thread/?sortby= =3Ddate#dirlist Mostly some cancellation cleaups, adding cancellation points for accept() and connect(), deadlock prevention for recursive rwlocks, cancellation points for sem_[try|timed]wait(). --=20 Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 08:33:19 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 88A4516A4CE for ; Wed, 18 Feb 2004 08:33:19 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id 5805943D1F for ; Wed, 18 Feb 2004 08:33:19 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 97139 invoked by uid 80); 18 Feb 2004 16:33:18 -0000 Received: from 68.3.131.72 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Wed, 18 Feb 2004 09:33:18 -0700 (MST) Message-ID: <50155.68.3.131.72.1077121998.squirrel@mail.asn.net> In-Reply-To: <40332E18.6090209@freebsd.org> References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <49327.68.106.19.246.1077066768.squirrel@mail.asn.net> <40332E18.6090209@freebsd.org> Date: Wed, 18 Feb 2004 09:33:18 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 16:33:19 -0000 > Can you tell us what scheduler are you using ? if it is ULE, can you > try 4BSD too ? > I think this is a heavy load for scheduler too, not just for pthread > library. Originally, I was using 4BSD, but I switched to ULE yesterday. Same problems either way. Kris Gale From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 08:36:06 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9109716A4CE for ; Wed, 18 Feb 2004 08:36:06 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id 7DD0E43D1D for ; Wed, 18 Feb 2004 08:36:06 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 97234 invoked by uid 80); 18 Feb 2004 16:36:06 -0000 Received: from 68.3.131.72 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Wed, 18 Feb 2004 09:36:06 -0700 (MST) Message-ID: <50172.68.3.131.72.1077122166.squirrel@mail.asn.net> In-Reply-To: References: <40330A4C.7030508@he.iki.fi> Date: Wed, 18 Feb 2004 09:36:06 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 16:36:06 -0000 > Oh, I assumed you guys were tracking -current, not sticking with > the last release... I'm tracking -current. But then again, I never saw the problem with MySQL refusing to quit. Kris Gale From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 08:50:41 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E391E16A4CE for ; Wed, 18 Feb 2004 08:50:41 -0800 (PST) Received: from mail.asn.net (mail.asn.net [66.235.231.4]) by mx1.FreeBSD.org (Postfix) with SMTP id CAAAC43D2F for ; Wed, 18 Feb 2004 08:50:41 -0800 (PST) (envelope-from kris-fbsd@asn.net) Received: (qmail 97588 invoked by uid 80); 18 Feb 2004 16:50:41 -0000 Received: from 68.3.131.72 (SquirrelMail authenticated user kgale) by mail.asn.net with HTTP; Wed, 18 Feb 2004 09:50:41 -0700 (MST) Message-ID: <50189.68.3.131.72.1077123041.squirrel@mail.asn.net> In-Reply-To: References: <49313.68.106.19.246.1077066168.squirrel@mail.asn.net> Date: Wed, 18 Feb 2004 09:50:41 -0700 (MST) From: "Kris Gale" To: freebsd-threads@freebsd.org User-Agent: SquirrelMail/1.4.2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 Importance: Normal Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 16:50:42 -0000 > I'm not sure what all the mumbo-jumbo is in your /etc/my.cnf, > but commenting out this line: > > #set-variable = key_buffer=1024M This key buffer is shared amongst the threads. This is the value I have set on an identical machine running -STABLE. I'll do some testing today with this value set different ways, and try out KSE vs. libthr to see what happens. Ideally, I need to get things running with this value quite large. Can anyone speculate as to why a big chunk of shared memory would slow down thread creation? I don't know much about how threading works. > let your perl script run with 90 children w/20 connections per. > Everything seems to be running here, but I'm not sure what I'm > looking for. Using my perl script, the problems I was describing would look like a huge scrolling mess of connect errors. If it gets past the messages that say "Starting child..." and then starts to output the information on threads every few seconds, everything is running fine. A good experiment is to stop and restart MySQL while my perl script continues to run. I will frequently see the kernel panics in this situation. Assuming there is no panic, you'll see lots of connect errors for about 10 minutes (much less time with libthr), then eventually things will be fine. > set-variable = thread_concurrency=1 The MySQL docs look like they're saying this only applies to Solaris. >From http://www.mysql.com/doc/en/SHOW_VARIABLES.html : thread_concurrency On Solaris, mysqld will call thr_setconcurrency() with this value. thr_setconcurrency() permits the application to give the threads system a hint for the desired number of threads that should be run at the same time. It recommends setting it to number of CPUs * 2, which is why I had it at 4. Kris Gale From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 11:22:46 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9C76216A4CE for ; Wed, 18 Feb 2004 11:22:46 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4454E43D1F for ; Wed, 18 Feb 2004 11:22:46 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1IJMfSQ016283; Wed, 18 Feb 2004 14:22:42 -0500 (EST) Date: Wed, 18 Feb 2004 14:22:41 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Gale In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 19:22:46 -0000 On Tue, 17 Feb 2004, Daniel Eischen wrote: > On Tue, 17 Feb 2004, Daniel Eischen wrote: > > On Tue, 17 Feb 2004, Kris Gale wrote: > > > Does this look like a fixable problem with KSE > > > to anyone on this list? > > > > I would like to see how MySQL works when using process > > scope threads. Do you know enough to hack it to do that? > > Alternatively, this patch to libpthread should force all > threads to be process scope. That patch doesn't work. It's not as simple to force the library to use scope process threads, since we make use of a special thread to handle signals and it has to be scope system. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 16:53:41 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6003816A4CE for ; Wed, 18 Feb 2004 16:53:41 -0800 (PST) Received: from exchhz01.viatech.com.cn (ip-40-162-97-218.anlai.com [218.97.162.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id A625243D2D for ; Wed, 18 Feb 2004 16:53:38 -0800 (PST) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (DAVIDWNT [10.4.1.99]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 1WN49F3X; Thu, 19 Feb 2004 08:53:41 +0800 Message-ID: <40340920.6010002@freebsd.org> Date: Thu, 19 Feb 2004 08:53:52 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kris Gale References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <49327.68.106.19.246.1077066768.squirrel@mail.asn.net> <40332E18.6090209@freebsd.org> <50155.68.3.131.72.1077121998.squirrel@mail.asn.net> In-Reply-To: <50155.68.3.131.72.1077121998.squirrel@mail.asn.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 00:53:41 -0000 Kris Gale wrote: >>Can you tell us what scheduler are you using ? if it is ULE, can you >>try 4BSD too ? >>I think this is a heavy load for scheduler too, not just for pthread >>library. >> >> > >Originally, I was using 4BSD, but I switched to ULE yesterday. Same >problems either way. > > With ULE, your test program failed on my machine, but with 4BSD, I got: Threads (connected/running/created/cached): 1801/91/3607/0 Threads (connected/running/created/cached): 1801/90/3607/0 Threads (connected/running/created/cached): 1801/91/3607/0 Threads (connected/running/created/cached): 1801/91/3607/0 Threads (connected/running/created/cached): 1801/91/3607/0 ... It is working fine 10 minutes without any problem. >Kris Gale >_______________________________________________ >freebsd-threads@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-threads >To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > > > > From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 17:16:59 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7038316A4CE for ; Wed, 18 Feb 2004 17:16:59 -0800 (PST) Received: from bento.FreeBSD.org (bento.freebsd.org [216.136.204.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id 685CB43D1F; Wed, 18 Feb 2004 17:16:59 -0800 (PST) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (localhost [127.0.0.1]) by bento.FreeBSD.org (8.12.10/8.12.10) with ESMTP id i1J1GuQt031780; Wed, 18 Feb 2004 17:16:57 -0800 (PST) (envelope-from davidxu@freebsd.org) Message-ID: <40340E9E.3090302@freebsd.org> Date: Thu, 19 Feb 2004 09:17:18 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Daniel Eischen References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 01:16:59 -0000 Daniel Eischen wrote: >On Tue, 17 Feb 2004, Daniel Eischen wrote: > > > >>On Tue, 17 Feb 2004, Daniel Eischen wrote: >> >> >>>On Tue, 17 Feb 2004, Kris Gale wrote: >>> >>> >>>>Does this look like a fixable problem with KSE >>>>to anyone on this list? >>>> >>>> >>>I would like to see how MySQL works when using process >>>scope threads. Do you know enough to hack it to do that? >>> >>> >>Alternatively, this patch to libpthread should force all >>threads to be process scope. >> >> > >That patch doesn't work. It's not as simple to force the >library to use scope process threads, since we make use >of a special thread to handle signals and it has to be >scope system. > I am running with following patch and 4BSD scheduler, it is working without deadlock. ;-) Index: thr_create.c =================================================================== RCS file: /home/ncvs/src/lib/libpthread/thread/thr_create.c,v retrieving revision 1.53 diff -u -r1.53 thr_create.c --- thr_create.c 8 Jan 2004 15:37:09 -0000 1.53 +++ thr_create.c 19 Feb 2004 01:13:11 -0000 @@ -142,6 +142,9 @@ } #ifdef SYSTEM_SCOPE_ONLY new_thread->attr.flags |= PTHREAD_SCOPE_SYSTEM; +#else + if (_thr_sig_daemon) + new_thread->attr.flags &= ~PTHREAD_SCOPE_SYSTEM; #endif if (create_stack(&new_thread->attr) != 0) { /* Insufficient memory to create a stack: */ From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 20:22:56 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A58BF16A4CE for ; Wed, 18 Feb 2004 20:22:56 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4CEC643D1D for ; Wed, 18 Feb 2004 20:22:56 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1J4MpSQ014126; Wed, 18 Feb 2004 23:22:52 -0500 (EST) Date: Wed, 18 Feb 2004 23:22:51 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Gale In-Reply-To: <50189.68.3.131.72.1077123041.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 04:22:56 -0000 On Wed, 18 Feb 2004, Kris Gale wrote: > > I'm not sure what all the mumbo-jumbo is in your /etc/my.cnf, > > but commenting out this line: > > > > #set-variable = key_buffer=1024M > > This key buffer is shared amongst the threads. This is the value > I have set on an identical machine running -STABLE. > > I'll do some testing today with this value set different ways, and > try out KSE vs. libthr to see what happens. > > Ideally, I need to get things running with this value quite large. > Can anyone speculate as to why a big chunk of shared memory > would slow down thread creation? I don't know much about > how threading works. The library doesn't do anything with shared memory so it should have no effect on it. > > let your perl script run with 90 children w/20 connections per. > > Everything seems to be running here, but I'm not sure what I'm > > looking for. > > Using my perl script, the problems I was describing would look > like a huge scrolling mess of connect errors. If it gets past the > messages that say "Starting child..." and then starts to output > the information on threads every few seconds, everything is > running fine. Yes, I got that when key_buffer was 1024M. Are you sure you want 1GB for this value? What is the default value? > A good experiment is to stop and restart MySQL while my perl > script continues to run. I will frequently see the kernel panics > in this situation. Assuming there is no panic, you'll see lots of > connect errors for about 10 minutes (much less time with libthr), > then eventually things will be fine. I haven't tried starting and stopping mysql. > > set-variable = thread_concurrency=1 > > The MySQL docs look like they're saying this only applies to > Solaris. > > >From http://www.mysql.com/doc/en/SHOW_VARIABLES.html : > thread_concurrency On Solaris, mysqld will call > thr_setconcurrency() with this value. thr_setconcurrency() > permits the application to give the threads system a hint for > the desired number of threads that should be run at the > same time. > > It recommends setting it to number of CPUs * 2, which is why > I had it at 4. We do have a pthread_setconcurrency(), but it really only does something if you try and lower the concurrency below kern.thread.virtual_cpu. By default, the number of KSEs within the main KSEG is set to kern.thread.virtual_cpu. All process scope threads are scheduled within the main KSEG -- having more than one virtual_cpu (KSE) is like having a pool of system scope thingies that can run all your process scope threads. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Feb 18 22:30:13 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 23F7016A4CE for ; Wed, 18 Feb 2004 22:30:13 -0800 (PST) Received: from telecom.net.et (sparrow.telecom.net.et [213.55.64.38]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3FAB743D1F for ; Wed, 18 Feb 2004 22:30:02 -0800 (PST) (envelope-from mtm@identd.net) Received: from [213.55.65.88] (HELO pool-151-200-10-97.res.east.verizon.net) by telecom.net.et (CommuniGate Pro SMTP 3.4.8) with ESMTP-TLS id 37035279 for freebsd-threads@freebsd.org; Thu, 19 Feb 2004 09:23:55 +0300 Received: from mobile.acs-et.com (localhost [127.0.0.1]) ESMTP id i1J6T10s008621 for ; Thu, 19 Feb 2004 09:29:09 +0300 (EAT) (envelope-from mtm@mobile.acs-et.com) Received: (from mtm@localhost) by mobile.acs-et.com (8.12.10/8.12.10/Submit) id i1J6SxAq008620 for freebsd-threads@freebsd.org; Thu, 19 Feb 2004 09:28:59 +0300 (EAT) (envelope-from mtm) Date: Thu, 19 Feb 2004 09:28:50 +0300 From: Mike Makonnen To: freebsd-threads@freebsd.org Message-ID: <20040219062850.GC1074@mobile.acs-et.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="kXdP64Ggrk/fb43R" Content-Disposition: inline User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD/5.2-CURRENT (i386) Subject: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 06:30:13 -0000 --kXdP64Ggrk/fb43R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello folks, The following patch stops libthr from using signals for syncronizng threads. I've tested it localy and have seen *really large* performance improvements in heavy thread/mutex contention situations. I'm interested in hearing your experiences. P.S. - The patch to the kernel applies two new syscalls so you must regenerate some files *after* applying the patch. Do the following: cd /usr/src/sys/kern sh makesyscalls.sh syscalls.master cd /usr/src/sys/compat/freebsd32 sh ../../kern/makesyscalls.sh syscalls.master syscalls.conf and then build your kernel in the usual way. Cheers. -- Mike Makonnen | GPG-KEY: http://www.identd.net/~mtm/mtm.asc mtm@identd.net | Fingerprint: AC7B 5672 2D11 F4D0 EBF8 5279 5359 2B82 7CD4 1F55 mtm@FreeBSD.Org| FreeBSD - Unleash the Daemon ! --kXdP64Ggrk/fb43R Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="diff.sys" Index: sys/compat/freebsd32/syscalls.master =================================================================== RCS file: /home/ncvs/src/sys/compat/freebsd32/syscalls.master,v retrieving revision 1.34 diff -u -r1.34 syscalls.master --- sys/compat/freebsd32/syscalls.master 3 Feb 2004 05:18:48 -0000 1.34 +++ sys/compat/freebsd32/syscalls.master 12 Feb 2004 14:03:47 -0000 @@ -604,3 +604,5 @@ 439 UNIMPL extattr_list_link 440 UNIMPL kse_switchin 441 UNIMPL ksem_timedwait +442 MNOPROTO { int thr_suspend(const struct timespec *timeout); } +443 MNOPROTO { int thr_wake(thr_id_t id); } Index: sys/kern/kern_thr.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_thr.c,v retrieving revision 1.14 diff -u -r1.14 kern_thr.c --- sys/kern/kern_thr.c 14 Aug 2003 03:56:24 -0000 1.14 +++ sys/kern/kern_thr.c 15 Feb 2004 06:46:08 -0000 @@ -254,3 +254,47 @@ PROC_UNLOCK(p); return (error); } + +int +thr_suspend(struct thread *td, struct thr_suspend_args *uap) + /* const struct timespec *timeout */ +{ + struct timespec ts; + struct timeval tv; + int error; + int hz; + + hz = 0; + error = 0; + if (uap->timeout != NULL) { + error = copyin((const void *)uap->timeout, (void *)&ts, + sizeof(struct timespec)); + if (error != 0) + return (error); + if (ts.tv_nsec < 0 || ts.tv_nsec > 1000000000) + return (EINVAL); + if (ts.tv_sec == 0 && ts.tv_nsec == 0) + return (ETIMEDOUT); + TIMESPEC_TO_TIMEVAL(&tv, &ts); + hz = tvtohz(&tv); + } + PROC_LOCK(td->td_proc); + if ((td->td_lthrflags & LTF_THRWAKEUP) == 0) { + error = msleep((void *)td, &td->td_proc->p_mtx, + td->td_priority | PCATCH, "lthr", hz); + td->td_lthrflags &= ~LTF_THRWAKEUP; + } + PROC_UNLOCK(td->td_proc); + return (error == EWOULDBLOCK ? ETIMEDOUT : error); +} + +int +thr_wake(struct thread *td, struct thr_wake_args *uap) + /* thr_id_t id */ +{ + PROC_LOCK(((struct thread *)uap->id)->td_proc); + ((struct thread *)uap->id)->td_lthrflags |= LTF_THRWAKEUP; + PROC_UNLOCK(((struct thread *)uap->id)->td_proc); + wakeup_one((void *)uap->id); + return (0); +} Index: sys/kern/syscalls.master =================================================================== RCS file: /home/ncvs/src/sys/kern/syscalls.master,v retrieving revision 1.166 diff -u -r1.166 syscalls.master --- sys/kern/syscalls.master 3 Feb 2004 05:08:32 -0000 1.166 +++ sys/kern/syscalls.master 12 Feb 2004 18:48:12 -0000 @@ -629,5 +629,7 @@ 440 MSTD { int kse_switchin(const struct __mcontext *mcp, \ long val, long *loc); } 441 MNOSTD { int ksem_timedwait(semid_t id, struct timespec *abstime); } +442 MSTD { int thr_suspend(const struct timespec *timeout); } +443 MSTD { int thr_wake(thr_id_t id); } ; Please copy any additions and changes to the following compatability tables: ; sys/compat/freebsd32/syscalls.master Index: sys/sys/proc.h =================================================================== RCS file: /home/ncvs/src/sys/sys/proc.h,v retrieving revision 1.367 diff -u -r1.367 proc.h --- sys/sys/proc.h 4 Feb 2004 21:52:56 -0000 1.367 +++ sys/sys/proc.h 12 Feb 2004 13:13:22 -0000 @@ -301,6 +301,7 @@ sigset_t td_siglist; /* (c) Sigs arrived, not delivered. */ sigset_t *td_waitset; /* (c) Wait set for sigwait. */ TAILQ_ENTRY(thread) td_umtx; /* (c?) Link for when we're blocked. */ + int td_lthrflags; /* (c) Libthr thread flags. */ volatile u_int td_generation; /* (k) Enable detection of preemption */ stack_t td_sigstk; /* (k) Stack ptr and on-stack flag. */ Index: sys/sys/thr.h =================================================================== RCS file: /home/ncvs/src/sys/sys/thr.h,v retrieving revision 1.1 diff -u -r1.1 thr.h --- sys/sys/thr.h 1 Apr 2003 00:30:30 -0000 1.1 +++ sys/sys/thr.h 17 Feb 2004 06:15:29 -0000 @@ -37,6 +37,10 @@ #define THR_SUSPENDED 0x0001 /* Create the thread in the suspended state. */ +/* flags stored in td_lthrflags member of the thread structure */ +#define LTF_THRWAKEUP 0x000001 /* Thread must not suspend itself */ +#define LTF_UMTXWAKEUP 0x000002 /* Thread must not suspend on a umtx */ + /* * See pthread_* */ @@ -46,6 +50,8 @@ int thr_self(thr_id_t *id); void thr_exit(void); int thr_kill(thr_id_t id, int sig); +int thr_suspend(const struct timespec *timeout); +int thr_wake(thr_id_t id); #endif /* !_KERNEL */ --kXdP64Ggrk/fb43R Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="diff.lib" Index: lib/libthr/thread/thr_cond.c =================================================================== RCS file: /home/ncvs/src/lib/libthr/thread/thr_cond.c,v retrieving revision 1.10 diff -u -r1.10 thr_cond.c --- lib/libthr/thread/thr_cond.c 9 Dec 2003 11:12:11 -0000 1.10 +++ lib/libthr/thread/thr_cond.c 15 Feb 2004 08:15:20 -0000 @@ -285,7 +285,7 @@ _thread_critical_exit(curthread); COND_UNLOCK(*cond); rval = _thread_suspend(curthread, (struct timespec *)abstime); - if (rval != 0 && rval != EAGAIN && rval != EINTR) { + if (rval != 0 && rval != ETIMEDOUT && rval != EINTR) { printf("foo"); fflush(stdout); abort(); Index: lib/libthr/thread/thr_kern.c =================================================================== RCS file: /home/ncvs/src/lib/libthr/thread/thr_kern.c,v retrieving revision 1.17 diff -u -r1.17 thr_kern.c --- lib/libthr/thread/thr_kern.c 29 Jan 2004 09:44:36 -0000 1.17 +++ lib/libthr/thread/thr_kern.c 15 Feb 2004 08:26:04 -0000 @@ -74,10 +74,6 @@ * Block all signals. */ SIGFILLSET(set); - SIGADDSET(set, SIGTHR); -#ifdef _PTHREADS_INVARIANTS - SIGDELSET(set, SIGABRT); -#endif SIGDELSET(set, SIGTRAP); /* If we have already blocked signals, just up the refcount */ @@ -121,7 +117,6 @@ { struct timespec remaining; struct timespec *ts; - siginfo_t info; int error; /* @@ -139,19 +134,12 @@ ts = &remaining; /* - * If the absolute timeout has already passed set the - * relative timeout to 0 sec. so that sigtimedwait() - * returns immediately. * NOTE: timespecsub() makes sure the tv_nsec member >= 0. */ - if (ts->tv_sec < 0) { - ts->tv_sec = 0; - ts->tv_nsec = 0; - } + if (ts->tv_sec < 0) + return (ETIMEDOUT); } else ts = NULL; - - error = sigtimedwait(&_thread_suspend_sigset, &info, ts); - error = (error == -1) ? errno : 0; - return (error); + error = thr_suspend(ts); + return (error == -1 ? errno : error); } Index: lib/libthr/thread/thr_mutex.c =================================================================== RCS file: /home/ncvs/src/lib/libthr/thread/thr_mutex.c,v retrieving revision 1.21 diff -u -r1.21 thr_mutex.c --- lib/libthr/thread/thr_mutex.c 19 Jan 2004 15:00:57 -0000 1.21 +++ lib/libthr/thread/thr_mutex.c 15 Feb 2004 08:16:53 -0000 @@ -1402,7 +1402,7 @@ _thread_critical_exit(curthread); _SPINUNLOCK(&mutexp->lock); error = _thread_suspend(curthread, abstime); - if (error != 0 && error != EAGAIN && error != EINTR) + if (error != 0 && error != ETIMEDOUT && error != EINTR) PANIC("Cannot suspend on mutex."); _SPINLOCK(&mutexp->lock); _thread_critical_enter(curthread); Index: lib/libthr/thread/thr_private.h =================================================================== RCS file: /home/ncvs/src/lib/libthr/thread/thr_private.h,v retrieving revision 1.30 diff -u -r1.30 thr_private.h --- lib/libthr/thread/thr_private.h 22 Jan 2004 15:31:56 -0000 1.30 +++ lib/libthr/thread/thr_private.h 12 Feb 2004 10:17:55 -0000 @@ -139,10 +139,8 @@ } while (0) #define PTHREAD_NEW_STATE(thrd, newstate) do { \ - if (newstate == PS_RUNNING) { \ - if (thr_kill(thrd->thr_id, SIGTHR)) \ - abort(); \ - } \ + if (newstate == PS_RUNNING) \ + thr_wake(thrd->thr_id); \ PTHREAD_SET_STATE(thrd, newstate); \ } while (0) Index: lib/libthr/thread/thr_resume_np.c =================================================================== RCS file: /home/ncvs/src/lib/libthr/thread/thr_resume_np.c,v retrieving revision 1.5 diff -u -r1.5 thr_resume_np.c --- lib/libthr/thread/thr_resume_np.c 12 Dec 2003 09:51:39 -0000 1.5 +++ lib/libthr/thread/thr_resume_np.c 12 Feb 2004 13:41:31 -0000 @@ -84,6 +84,5 @@ resume_common(struct pthread *thread) { thread->flags &= ~PTHREAD_FLAGS_SUSPENDED; - if (thr_kill(thread->thr_id, SIGTHR)) - abort(); + thr_wake(thread->thr_id); } --kXdP64Ggrk/fb43R-- From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 00:56:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CFD3C16A4CE for ; Thu, 19 Feb 2004 00:56:48 -0800 (PST) Received: from exchhz01.viatech.com.cn (ip-40-162-97-218.anlai.com [218.97.162.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9753643D1F for ; Thu, 19 Feb 2004 00:55:40 -0800 (PST) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (DAVIDWNT [10.4.1.99]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id FHN0J507; Thu, 19 Feb 2004 16:52:20 +0800 Message-ID: <4034793E.8040105@freebsd.org> Date: Thu, 19 Feb 2004 16:52:14 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Makonnen References: <20040219062850.GC1074@mobile.acs-et.com> In-Reply-To: <20040219062850.GC1074@mobile.acs-et.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 08:56:48 -0000 Yes, I think sigwait is too slow for synchronizing, it is better to use dedicated function, this looks fine for me. David Xu Mike Makonnen wrote: >Hello folks, > >The following patch stops libthr from using signals for syncronizng threads. >I've tested it localy and have seen *really large* performance >improvements in heavy thread/mutex contention situations. I'm interested >in hearing your experiences. > >P.S. - The patch to the kernel applies two new syscalls so you must > regenerate some files *after* applying the patch. Do the following: > cd /usr/src/sys/kern > sh makesyscalls.sh syscalls.master > cd /usr/src/sys/compat/freebsd32 > sh ../../kern/makesyscalls.sh syscalls.master syscalls.conf > > and then build your kernel in the usual way. > >Cheers. > > From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 05:05:59 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C1BE216A4CE for ; Thu, 19 Feb 2004 05:05:59 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 68CE743D1F for ; Thu, 19 Feb 2004 05:05:59 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1JD5vSQ024471; Thu, 19 Feb 2004 08:05:57 -0500 (EST) Date: Thu, 19 Feb 2004 08:05:57 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Mike Makonnen In-Reply-To: <20040219062850.GC1074@mobile.acs-et.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 13:05:59 -0000 On Thu, 19 Feb 2004, Mike Makonnen wrote: > Hello folks, > > The following patch stops libthr from using signals for syncronizng threads. > I've tested it localy and have seen *really large* performance > improvements in heavy thread/mutex contention situations. I'm interested > in hearing your experiences. If you used KSEs, you could use kse_release() and kse_wakeup() :-) :-) -- Dan From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 05:48:15 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4B5F316A4D0 for ; Thu, 19 Feb 2004 05:48:15 -0800 (PST) Received: from smtp02.syd.iprimus.net.au (smtp02.syd.iprimus.net.au [210.50.76.52]) by mx1.FreeBSD.org (Postfix) with ESMTP id 20FFD43D1D for ; Thu, 19 Feb 2004 05:48:15 -0800 (PST) (envelope-from tim@robbins.dropbear.id.au) Received: from robbins.dropbear.id.au (210.50.217.1) by smtp02.syd.iprimus.net.au (7.0.024) id 402CF87000199257; Fri, 20 Feb 2004 00:48:13 +1100 Received: by robbins.dropbear.id.au (Postfix, from userid 1000) id 6D96341C0; Fri, 20 Feb 2004 00:47:33 +1100 (EST) Date: Fri, 20 Feb 2004 00:47:33 +1100 From: Tim Robbins To: Mike Makonnen Message-ID: <20040219134733.GA38023@cat.robbins.dropbear.id.au> References: <20040219062850.GC1074@mobile.acs-et.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040219062850.GC1074@mobile.acs-et.com> User-Agent: Mutt/1.4.1i cc: freebsd-threads@freebsd.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 13:48:15 -0000 On Thu, Feb 19, 2004 at 09:28:50AM +0300, Mike Makonnen wrote: > Index: sys/kern/kern_thr.c [...] > +int > +thr_wake(struct thread *td, struct thr_wake_args *uap) > + /* thr_id_t id */ > +{ > + PROC_LOCK(((struct thread *)uap->id)->td_proc); > + ((struct thread *)uap->id)->td_lthrflags |= LTF_THRWAKEUP; > + PROC_UNLOCK(((struct thread *)uap->id)->td_proc); > + wakeup_one((void *)uap->id); > + return (0); > +} [...] Make sure you fix thr_wake() to check that uap->id is valid before you commit this patch. Something like this would be safer but slower: struct thread *td1; PROC_LOCK(td->td_proc); FOREACH_THREAD_IN_PROC(td->td_proc, td1) if (td1 == (struct thread *)uap->id) break; if (td1 == NULL) { PROC_UNLOCK(td->td_proc); return (ESRCH); } td1->td_lthrflags |= LTF_THRWAKEUP; wakeup_one(td1); PROC_UNLOCK(td->td_proc); return (0); (I'm not sure that it's safe to call wakeup_one() on a thread pointer that isn't curthread without holding the proc lock, so I changed that too.) Tim From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 10:50:55 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 374D416A4CE for ; Thu, 19 Feb 2004 10:50:55 -0800 (PST) Received: from odot.okladot.state.ok.us (odot.okladot.state.ok.us [192.149.244.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id DA6B343D1F for ; Thu, 19 Feb 2004 10:50:54 -0800 (PST) (envelope-from root@techpc04.okladot.state.ok.us) Received: from notes9c.okladot.state.ok.us (notes9a.okladot.state.ok.us [10.36.36.31])MAA72290; Thu, 19 Feb 2004 12:50:52 -0600 Received: from techpc04.okladot.state.ok.us ([199.27.9.37]) by notes9c.okladot.state.ok.us (Lotus Domino Release 5.0.12) with ESMTP id 2004021912504699:38489 ; Thu, 19 Feb 2004 12:50:46 -0600 Received: by techpc04.okladot.state.ok.us (Postfix, from userid 0) id 7CD365C22; Thu, 19 Feb 2004 12:50:55 -0600 (CST) To: Mike Makonnen From: "Paul Seniura" In-Reply-To: <20040219062850.GC1074@mobile.acs-et.com> References: <20040219062850.GC1074@mobile.acs-et.com> Message-Id: <20040219185055.7CD365C22@techpc04.okladot.state.ok.us> Date: Thu, 19 Feb 2004 12:50:55 -0600 (CST) X-MIMETrack: Itemize by SMTP Server on Notes9c/ODOT(Release 5.0.12 |February 13, 2003) at 02/19/2004 12:50:47 PM,2003) at 02/19/2004 12:50:48 PM, Serialize complete at 02/19/2004 12:50:48 PM cc: Paul Seniura cc: freebsd-threads@freebsd.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Paul Seniura List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 18:50:55 -0000 Hi, Looks like there were some commits done since the date on your patches. I had to change the following to make it match as of last night's CTM deltas: +++ lib/libthr/thread/thr_mutex.c 15 Feb 2004 08:16:53 -0000 -@@ -1402,7 +1402,7 @@ +@@ -892,7 +892,7 @@ There was a recent commit for lib/libthr/thread/thr_private.h also, but the line numbers still match. I'm whipping up a new kernel with these patches, too, and will try Tim Robbins's changes also. I'm *very* anxious to squeeze more from this Puny Pentium2 system "they" gave me for this project. ;-) Thank you, -- Paul Seniura System Specialist State of Okla. D.O.T. From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 12:00:05 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 997D016A4CE for ; Thu, 19 Feb 2004 12:00:05 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4373443D2F for ; Thu, 19 Feb 2004 12:00:05 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1JJxwSQ024603; Thu, 19 Feb 2004 14:59:58 -0500 (EST) Date: Thu, 19 Feb 2004 14:59:58 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Kris Gale In-Reply-To: <50189.68.3.131.72.1077123041.squirrel@mail.asn.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 20:00:05 -0000 On Wed, 18 Feb 2004, Kris Gale wrote: > > I'm not sure what all the mumbo-jumbo is in your /etc/my.cnf, > > but commenting out this line: > > > > #set-variable = key_buffer=1024M > > This key buffer is shared amongst the threads. This is the value > I have set on an identical machine running -STABLE. I can't even get mysqld to start with a value this large. I patched mysql to use process scope threads. Here is the patch to databases/mysql40-server: http://people.freebsd.org/~deischen/mysql40-server.diffs The port still uses -lc_r even though it detects -lpthread. I haven't fixed that, so be sure to keep using libmap.conf. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 12:05:12 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 54AF216A4CE for ; Thu, 19 Feb 2004 12:05:12 -0800 (PST) Received: from sccrmhc12.comcast.net (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1E88D43D1D for ; Thu, 19 Feb 2004 12:05:12 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc12) with ESMTP id <2004021920050901200fqm0fe>; Thu, 19 Feb 2004 20:05:10 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id MAA09911; Thu, 19 Feb 2004 12:05:08 -0800 (PST) Date: Thu, 19 Feb 2004 12:05:07 -0800 (PST) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 20:05:12 -0000 This brings up something that I think we should discuss with Mike.. Mike, the libthr kernel code has lots of code (e.g. in the scheduler) to make libthr implement 1:1 syste, scope threads. this is "silly" necause we designed the KSE based kernel framework to support 1:1 system scoep threads.. one just needs to allocate a KSEGROUP with each thread and then everything would be right automatically. Do you know teh reason that this was not done? do you know how muvch work it would be to do this? if it was it would allow both threading pacakges to follow the same cade paths (KSE can be compiled to run in 1:1 mode and does it this way) and you coukld as Dan suggested, use the infrastructure we already have for this sort of thing... Julian On Thu, 19 Feb 2004, Daniel Eischen wrote: > On Thu, 19 Feb 2004, Mike Makonnen wrote: > > > Hello folks, > > > > The following patch stops libthr from using signals for syncronizng threads. > > I've tested it localy and have seen *really large* performance > > improvements in heavy thread/mutex contention situations. I'm interested > > in hearing your experiences. > > If you used KSEs, you could use kse_release() and kse_wakeup() :-) :-) > > -- > Dan > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 12:22:28 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9A04B16A4CE for ; Thu, 19 Feb 2004 12:22:28 -0800 (PST) Received: from rwcrmhc12.comcast.net (rwcrmhc12.comcast.net [216.148.227.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 93BAE43D1F for ; Thu, 19 Feb 2004 12:22:28 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc12) with ESMTP id <2004021920222401400lkfuae>; Thu, 19 Feb 2004 20:22:28 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id MAA10117; Thu, 19 Feb 2004 12:22:24 -0800 (PST) Date: Thu, 19 Feb 2004 12:22:24 -0800 (PST) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 20:22:28 -0000 On Thu, 19 Feb 2004, Daniel Eischen wrote: > On Wed, 18 Feb 2004, Kris Gale wrote: > > > > I'm not sure what all the mumbo-jumbo is in your /etc/my.cnf, > > > but commenting out this line: > > > > > > #set-variable = key_buffer=1024M > > > > This key buffer is shared amongst the threads. This is the value > > I have set on an identical machine running -STABLE. > > I can't even get mysqld to start with a value this large. you probably don't have the 4GB of RAM he does :-).. > > I patched mysql to use process scope threads. Here is the > patch to databases/mysql40-server: > > http://people.freebsd.org/~deischen/mysql40-server.diffs > > The port still uses -lc_r even though it detects -lpthread. > I haven't fixed that, so be sure to keep using libmap.conf. > > -- > Dan Eischen > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 12:32:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 037D816A4CE for ; Thu, 19 Feb 2004 12:32:48 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id B710143D1D for ; Thu, 19 Feb 2004 12:32:47 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1JKWdSQ003716; Thu, 19 Feb 2004 15:32:39 -0500 (EST) Date: Thu, 19 Feb 2004 15:32:39 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Feb 2004 20:32:48 -0000 On Thu, 19 Feb 2004, Julian Elischer wrote: > > On Thu, 19 Feb 2004, Daniel Eischen wrote: > > > On Wed, 18 Feb 2004, Kris Gale wrote: > > > > > > I'm not sure what all the mumbo-jumbo is in your /etc/my.cnf, > > > > but commenting out this line: > > > > > > > > #set-variable = key_buffer=1024M > > > > > > This key buffer is shared amongst the threads. This is the value > > > I have set on an identical machine running -STABLE. > > > > I can't even get mysqld to start with a value this large. > > you probably don't have the 4GB of RAM he does :-).. Not quite. CPU: Intel Pentium III (999.52-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x68a Stepping = 10 Features=0x383fbff real memory = 1073676288 (1023 MB) avail memory = 1041158144 (992 MB) I didn't now what 'key_buffer' really meant; I assumed it would be backed by swap and that it wouldn't result in being wired. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 19:00:33 2004 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B16B216A4CE for ; Thu, 19 Feb 2004 19:00:33 -0800 (PST) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 92E6543D2F for ; Thu, 19 Feb 2004 19:00:33 -0800 (PST) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i1K30Vbv012199 for ; Thu, 19 Feb 2004 19:00:31 -0800 (PST) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i1K30V37012198; Thu, 19 Feb 2004 19:00:31 -0800 (PST) (envelope-from gnats) Date: Thu, 19 Feb 2004 19:00:31 -0800 (PST) Message-Id: <200402200300.i1K30V37012198@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Craig Rodrigues Subject: Re: bin/24632: libc_r delicate deviation from libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Craig Rodrigues List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 03:00:33 -0000 The following reply was made to PR bin/24632; it has been noted by GNATS. From: Craig Rodrigues To: freebsd-gnats-submit@FreeBSD.org, barak@cwnt.com Cc: rodrigc@crodrigues.org Subject: Re: bin/24632: libc_r delicate deviation from libc Date: Thu, 19 Feb 2004 21:52:12 -0500 (EST) Hi, I tried the testcase in bin/24632 under -CURRENT. I linked against -lc_r, -lthr, and -lpthread. I could not reproduce the problem in this testcase. The patch submitted in this PR is for a very old version of libc_r. For example, SA_ONSTACK is not specified in the signal code in libc_r. From owner-freebsd-threads@FreeBSD.ORG Thu Feb 19 23:38:31 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 093B016A4CE; Thu, 19 Feb 2004 23:38:31 -0800 (PST) Received: from telecom.net.et (ns2.telecom.net.et [213.55.64.38]) by mx1.FreeBSD.org (Postfix) with ESMTP id E4AF643D2D; Thu, 19 Feb 2004 23:38:26 -0800 (PST) (envelope-from mtm@identd.net) Received: from [213.55.67.100] (HELO pool-151-200-10-97.res.east.verizon.net) by telecom.net.et (CommuniGate Pro SMTP 3.4.8) with ESMTP-TLS id 37116927; Fri, 20 Feb 2004 10:32:43 +0300 Received: from mobile.acs-et.com (localhost [127.0.0.1]) ESMTP id i1K7cYCN001477; Fri, 20 Feb 2004 10:38:36 +0300 (EAT) (envelope-from mtm@mobile.acs-et.com) Received: (from mtm@localhost) by mobile.acs-et.com (8.12.10/8.12.10/Submit) id i1K7cX5H001476; Fri, 20 Feb 2004 10:38:33 +0300 (EAT) (envelope-from mtm) Date: Fri, 20 Feb 2004 10:38:30 +0300 From: Mike Makonnen To: Tim Robbins Message-ID: <20040220073830.GB1089@mobile.acs-et.com> References: <20040219062850.GC1074@mobile.acs-et.com> <20040219134733.GA38023@cat.robbins.dropbear.id.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040219134733.GA38023@cat.robbins.dropbear.id.au> User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD/5.2-CURRENT (i386) cc: freebsd-threads@FreeBSD.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 07:38:31 -0000 On Fri, Feb 20, 2004 at 12:47:33AM +1100, Tim Robbins wrote: > On Thu, Feb 19, 2004 at 09:28:50AM +0300, Mike Makonnen wrote: > > Make sure you fix thr_wake() to check that uap->id is valid before you > commit this patch. Something like this would be safer but slower: Hmm, you're right. I should've thought of that. Thanks for catching it. > > struct thread *td1; > > PROC_LOCK(td->td_proc); > FOREACH_THREAD_IN_PROC(td->td_proc, td1) > if (td1 == (struct thread *)uap->id) > break; > if (td1 == NULL) { > PROC_UNLOCK(td->td_proc); > return (ESRCH); > } > td1->td_lthrflags |= LTF_THRWAKEUP; > wakeup_one(td1); > PROC_UNLOCK(td->td_proc); > return (0); > > (I'm not sure that it's safe to call wakeup_one() on a thread pointer > that isn't curthread without holding the proc lock, so I changed that too.) Sorry, I don't follow. How can a thread call wakeup_one() on itself (if it's supposed to be asleep)? A cursory look through the call path doesn't show anything that needs a proc lock. There is access to p->p_state but either sched_lock or proc lock is sufficient for that (and the accessing function does hold sched_lock). Cheers. -- Mike Makonnen | GPG-KEY: http://www.identd.net/~mtm/mtm.asc mtm@identd.net | Fingerprint: AC7B 5672 2D11 F4D0 EBF8 5279 5359 2B82 7CD4 1F55 mtm@FreeBSD.Org| FreeBSD - Unleash the Daemon ! From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 00:25:25 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2984216A4CE for ; Fri, 20 Feb 2004 00:25:25 -0800 (PST) Received: from smtp02.syd.iprimus.net.au (smtp02.syd.iprimus.net.au [210.50.76.52]) by mx1.FreeBSD.org (Postfix) with ESMTP id EECB743D1D for ; Fri, 20 Feb 2004 00:25:24 -0800 (PST) (envelope-from tim@robbins.dropbear.id.au) Received: from robbins.dropbear.id.au (210.50.203.147) by smtp02.syd.iprimus.net.au (7.0.024) id 402CF870001D4185; Fri, 20 Feb 2004 19:25:21 +1100 Received: by robbins.dropbear.id.au (Postfix, from userid 1000) id 8469A41BF; Fri, 20 Feb 2004 19:25:17 +1100 (EST) Date: Fri, 20 Feb 2004 19:25:17 +1100 From: Tim Robbins To: Mike Makonnen Message-ID: <20040220082517.GA42604@cat.robbins.dropbear.id.au> References: <20040219062850.GC1074@mobile.acs-et.com> <20040219134733.GA38023@cat.robbins.dropbear.id.au> <20040220073830.GB1089@mobile.acs-et.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040220073830.GB1089@mobile.acs-et.com> User-Agent: Mutt/1.4.1i cc: freebsd-threads@FreeBSD.org Subject: Re: libthr patch X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 08:25:25 -0000 On Fri, Feb 20, 2004 at 10:38:30AM +0300, Mike Makonnen wrote: > On Fri, Feb 20, 2004 at 12:47:33AM +1100, Tim Robbins wrote: > > On Thu, Feb 19, 2004 at 09:28:50AM +0300, Mike Makonnen wrote: > > > > Make sure you fix thr_wake() to check that uap->id is valid before you > > commit this patch. Something like this would be safer but slower: > > Hmm, you're right. I should've thought of that. Thanks for catching it. > > > > > struct thread *td1; > > > > PROC_LOCK(td->td_proc); > > FOREACH_THREAD_IN_PROC(td->td_proc, td1) > > if (td1 == (struct thread *)uap->id) > > break; > > if (td1 == NULL) { > > PROC_UNLOCK(td->td_proc); > > return (ESRCH); > > } > > td1->td_lthrflags |= LTF_THRWAKEUP; > > wakeup_one(td1); > > PROC_UNLOCK(td->td_proc); > > return (0); > > > > (I'm not sure that it's safe to call wakeup_one() on a thread pointer > > that isn't curthread without holding the proc lock, so I changed that too.) > > Sorry, I don't follow. How can a thread call wakeup_one() on itself (if it's > supposed to be asleep)? > > A cursory look through the call path doesn't show anything that needs a > proc lock. There is access to p->p_state but either sched_lock or proc lock > is sufficient for that (and the accessing function does hold sched_lock). What I mean is: I don't know whether you can be sure that the thread hasn't already been freed between the time unlock the proc and when you call wakeup_one(). I assumed the list of threads associated with a process was protected by the proc lock. Tim From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 05:08:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D919F16A4CE for ; Fri, 20 Feb 2004 05:08:48 -0800 (PST) Received: from smtp.des.no (flood.des.no [217.116.83.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id A874C43D1D for ; Fri, 20 Feb 2004 05:08:48 -0800 (PST) (envelope-from des@des.no) Received: by smtp.des.no (Pony Express, from userid 666) id 3F7345309; Fri, 20 Feb 2004 14:08:47 +0100 (CET) Received: from dwp.des.no (des.no [80.203.228.37]) by smtp.des.no (Pony Express) with ESMTP id 2E9EE5308 for ; Fri, 20 Feb 2004 14:08:41 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 2602) id 12E8533C6F; Fri, 20 Feb 2004 14:08:41 +0100 (CET) To: threads@freebsd.org From: des@des.no (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Fri, 20 Feb 2004 14:08:41 +0100 Message-ID: User-Agent: Gnus/5.090024 (Oort Gnus v0.24) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on flood.des.no X-Spam-Level: X-Spam-Status: No, hits=0.0 required=5.0 tests=AWL autolearn=no version=2.63 Subject: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 13:08:49 -0000 - What happens when a thread locks a mutex and then starts another thread? Does the new thread in any way inherit the lock, or will it have to acquire its own? - What about spin locks? Are there any semantic differences between mutexes and spin locks in POSIX, or are spin locks simply mutexes that spin instead of sleeping? DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 07:02:03 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C3F1E16A4CE for ; Fri, 20 Feb 2004 07:02:03 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 826AF43D2F for ; Fri, 20 Feb 2004 07:02:03 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1KF21SQ002727; Fri, 20 Feb 2004 10:02:01 -0500 (EST) Date: Fri, 20 Feb 2004 10:02:01 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 15:02:03 -0000 On Fri, 20 Feb 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote: > - What happens when a thread locks a mutex and then starts another > thread? Does the new thread in any way inherit the lock, or will it > have to acquire its own? No, it doesn't inherit any locks. > - What about spin locks? Are there any semantic differences between > mutexes and spin locks in POSIX, or are spin locks simply mutexes > that spin instead of sleeping? POSIX differentiates between spinlocks and mutexes, and that is why there is a pthread_spinlock_t instead of using pthread_mutex_t for both lock operations. I believe the implementation is allowed to spin indefinitely or schedule another thread if the lock is busy. I can't find a rationale for spinlocks; I _thought_ there was one in some draft of the spec that I had read. I wouldn't really advocate using spinlocks due to possible priority inversion problems and wasting cycles. --=20 Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 08:02:56 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E096316A4CE for ; Fri, 20 Feb 2004 08:02:56 -0800 (PST) Received: from smtp.des.no (flood.des.no [217.116.83.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF7D943D39 for ; Fri, 20 Feb 2004 08:02:56 -0800 (PST) (envelope-from des@des.no) Received: by smtp.des.no (Pony Express, from userid 666) id 50E3C5309; Fri, 20 Feb 2004 17:02:55 +0100 (CET) Received: from dwp.des.no (des.no [80.203.228.37]) by smtp.des.no (Pony Express) with ESMTP id 690195308; Fri, 20 Feb 2004 17:02:48 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 2602) id 53F7D33C6F; Fri, 20 Feb 2004 17:02:48 +0100 (CET) To: Daniel Eischen References: From: des@des.no (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Fri, 20 Feb 2004 17:02:48 +0100 In-Reply-To: (Daniel Eischen's message of "Fri, 20 Feb 2004 10:02:01 -0500 (EST)") Message-ID: User-Agent: Gnus/5.090024 (Oort Gnus v0.24) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on flood.des.no X-Spam-Level: X-Spam-Status: No, hits=0.0 required=5.0 tests=AWL autolearn=no version=2.63 cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 16:02:57 -0000 Daniel Eischen writes: > POSIX differentiates between spinlocks and mutexes, and that is > why there is a pthread_spinlock_t instead of using pthread_mutex_t > for both lock operations. I believe the implementation is allowed > to spin indefinitely or schedule another thread if the lock is > busy. In other words, the semantics for the two are identical? BTW, I looked at the libpthread sources, and it seems to me that a spinlock is just a wrapper around a mutex, though the comments contradict this. Going off on a tangent, is there a value one can assign to a pthread_t to make it unambiguously invalid? Obviously, NULL or 0 will work in FreeBSD since our pthread_t is a pointer to a struct pthread, but I'm looking for something more portable. DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 11:45:46 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 24E1916A4CE for ; Fri, 20 Feb 2004 11:45:46 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id D3EDB43D1D for ; Fri, 20 Feb 2004 11:45:45 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1KJjhSQ022361; Fri, 20 Feb 2004 14:45:44 -0500 (EST) Date: Fri, 20 Feb 2004 14:45:43 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 19:45:46 -0000 On Fri, 20 Feb 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote: > Daniel Eischen writes: > > POSIX differentiates between spinlocks and mutexes, and that is > > why there is a pthread_spinlock_t instead of using pthread_mutex_t > > for both lock operations. I believe the implementation is allowed > > to spin indefinitely or schedule another thread if the lock is > > busy. >=20 > In other words, the semantics for the two are identical? Yes, you use them in the same way. > BTW, I looked at the libpthread sources, and it seems to me that a > spinlock is just a wrapper around a mutex, though the comments > contradict this. I think you are looking at spinlocks, not pthread_spinlocks. Look at thr_pspinlock.c, not thr_spinlock.c. The latter is used for internal spinlock usage by libc. Libc spinlocks are unfortunately named and should just be libc_locks or something. > Going off on a tangent, is there a value one can assign to a pthread_t > to make it unambiguously invalid? Obviously, NULL or 0 will work in > FreeBSD since our pthread_t is a pointer to a struct pthread, but I'm > looking for something more portable. Not that I know of. --=20 Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 11:48:27 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C06DF16A4CE for ; Fri, 20 Feb 2004 11:48:27 -0800 (PST) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 87B9243D1F for ; Fri, 20 Feb 2004 11:48:27 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc11) with ESMTP id <200402201948260110092cuke>; Fri, 20 Feb 2004 19:48:26 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id LAA24272; Fri, 20 Feb 2004 11:48:25 -0800 (PST) Date: Fri, 20 Feb 2004 11:48:24 -0800 (PST) From: Julian Elischer To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 19:48:27 -0000 On Fri, 20 Feb 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote: > - What happens when a thread locks a mutex and then starts another > thread? Does the new thread in any way inherit the lock, or will it > have to acquire its own? do you mean userland mutexes or kernel mutexes? I'll only cover kernel mutexes.. In KSE the new thread in teh kernel does not start out in the same context as the parent thread.. in this way it is not like fork(). the new thread starts out about to return to user land pretty much as if it has just completed a special system call. this return-without-having-done-the-call-in-the-first-place is called an upcall. The upcall is into the userland thread scheduler and a new userland thread is jumped into in userland.. the original thread is still blocked or running in the kernel and still owns the mutex. >=20 > - What about spin locks? Are there any semantic differences between > mutexes and spin locks in POSIX, or are spin locks simply mutexes > that spin instead of sleeping? >=20 > DES > --=20 > Dag-Erling Sm=F8rgrav - des@des.no > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org= " >=20 From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 12:28:12 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 85E8E16A4CE for ; Fri, 20 Feb 2004 12:28:12 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2AD2643D1F for ; Fri, 20 Feb 2004 12:28:12 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1KKSASQ004828; Fri, 20 Feb 2004 15:28:10 -0500 (EST) Date: Fri, 20 Feb 2004 15:28:10 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 20:28:12 -0000 On Fri, 20 Feb 2004, Daniel Eischen wrote: > I can't find a rationale for spinlocks; I _thought_ there was > one in some draft of the spec that I had read. I wouldn't > really advocate using spinlocks due to possible priority > inversion problems and wasting cycles. Found it. It's in the Rationale -> System Interfaces -> General Info. Here's a link that might get you there: http://www.opengroup.org/onlinepubs/007904975/xrat/xsh_chap02.html#tag_03_02_09 -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 14:08:35 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EFAFE16A520 for ; Fri, 20 Feb 2004 14:08:35 -0800 (PST) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id AC02743D2D for ; Fri, 20 Feb 2004 14:08:35 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i1KM8XSQ002761; Fri, 20 Feb 2004 17:08:33 -0500 (EST) Date: Fri, 20 Feb 2004 17:08:33 -0500 (EST) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN Content-Transfer-Encoding: QUOTED-PRINTABLE cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 22:08:36 -0000 On Fri, 20 Feb 2004, Dag-Erling [iso-8859-1] Sm=F8rgrav wrote: > Daniel Eischen writes: > > Found it. It's in the Rationale -> System Interfaces -> General Info. > > Here's a link that might get you there: > > > > http://www.opengroup.org/onlinepubs/007904975/xrat/xsh_chap02.html#ta= g_03_02_09 >=20 > OK. It seems to me then that our spin locks are only superficially > conformant. They are far heavier than the rationale suggests: they > depend on malloc(), and they call pthread_self() which requires a full > libpthread initialization. That makes them unsuitable in many cases. > For instance, a malloc() implementation can't use our spin locks to > protect its management structures. This happens to be a very real > issue for me in my day job; I ended up writing my own spin lock > implementation based solely on (anyone know of a > good way to thoroughly test it, BTW?) Put it in a shared memory region and have another process poke at it? Our synchronization primitives (mutexes, CVs, etc) need to change from a * to , but that is an ABI change, causes a libc version bump, etc. The plan for 6.0 is to change this, but since we have multiple thread libraries it also requires some coordination. We really need to make this change to support the pshared attribute (sharing the locks across processes in shared memory regions). --=20 Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Feb 20 14:24:37 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4082A16A4CE for ; Fri, 20 Feb 2004 14:24:37 -0800 (PST) Received: from smtp.des.no (flood.des.no [217.116.83.31]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0DDBC43D31 for ; Fri, 20 Feb 2004 14:24:37 -0800 (PST) (envelope-from des@des.no) Received: by smtp.des.no (Pony Express, from userid 666) id B3DCE5309; Fri, 20 Feb 2004 22:54:36 +0100 (CET) Received: from dwp.des.no (des.no [80.203.228.37]) by smtp.des.no (Pony Express) with ESMTP id 93B675308; Fri, 20 Feb 2004 22:54:29 +0100 (CET) Received: by dwp.des.no (Postfix, from userid 2602) id 1EF1A33C6F; Fri, 20 Feb 2004 22:54:29 +0100 (CET) To: Daniel Eischen References: From: des@des.no (Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?=) Date: Fri, 20 Feb 2004 22:54:29 +0100 In-Reply-To: (Daniel Eischen's message of "Fri, 20 Feb 2004 15:28:10 -0500 (EST)") Message-ID: User-Agent: Gnus/5.090024 (Oort Gnus v0.24) Emacs/21.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on flood.des.no X-Spam-Level: X-Spam-Status: No, hits=0.0 required=5.0 tests=AWL autolearn=no version=2.63 cc: threads@freebsd.org Subject: Re: cross-thread locking X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Feb 2004 22:24:37 -0000 Daniel Eischen writes: > Found it. It's in the Rationale -> System Interfaces -> General Info. > Here's a link that might get you there: > > http://www.opengroup.org/onlinepubs/007904975/xrat/xsh_chap02.html#tag_= 03_02_09 OK. It seems to me then that our spin locks are only superficially conformant. They are far heavier than the rationale suggests: they depend on malloc(), and they call pthread_self() which requires a full libpthread initialization. That makes them unsuitable in many cases. For instance, a malloc() implementation can't use our spin locks to protect its management structures. This happens to be a very real issue for me in my day job; I ended up writing my own spin lock implementation based solely on (anyone know of a good way to thoroughly test it, BTW?) DES --=20 Dag-Erling Sm=F8rgrav - des@des.no From owner-freebsd-threads@FreeBSD.ORG Sat Feb 21 17:37:46 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39A8D16A4CE; Sat, 21 Feb 2004 17:37:46 -0800 (PST) Received: from mail.chesapeake.net (chesapeake.net [208.142.252.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id E334543D1F; Sat, 21 Feb 2004 17:37:45 -0800 (PST) (envelope-from jroberson@chesapeake.net) Received: from mail.chesapeake.net (localhost [127.0.0.1]) by mail.chesapeake.net (8.12.10/8.12.10) with ESMTP id i1M1bhXJ038194; Sat, 21 Feb 2004 20:37:43 -0500 (EST) (envelope-from jroberson@chesapeake.net) Received: from localhost (jroberson@localhost)i1M1bhYw038188; Sat, 21 Feb 2004 20:37:43 -0500 (EST) (envelope-from jroberson@chesapeake.net) X-Authentication-Warning: mail.chesapeake.net: jroberson owned process doing -bs Date: Sat, 21 Feb 2004 20:37:43 -0500 (EST) From: Jeff Roberson To: David Xu In-Reply-To: <40340920.6010002@freebsd.org> Message-ID: <20040221203641.E35589@mail.chesapeake.net> References: <56666.68.106.19.246.1077058348.squirrel@mail.asn.net> <40332E18.6090209@freebsd.org><40340920.6010002@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: More on MySQL -- Fatal trap 12 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Feb 2004 01:37:46 -0000 On Thu, 19 Feb 2004, David Xu wrote: > Kris Gale wrote: > > >>Can you tell us what scheduler are you using ? if it is ULE, can you > >>try 4BSD too ? > >>I think this is a heavy load for scheduler too, not just for pthread > >>library. > >> > >> > > > >Originally, I was using 4BSD, but I switched to ULE yesterday. Same > >problems either way. > > > > > With ULE, your test program failed on my machine, but with 4BSD, I got: > > Threads (connected/running/created/cached): 1801/91/3607/0 > Threads (connected/running/created/cached): 1801/90/3607/0 > Threads (connected/running/created/cached): 1801/91/3607/0 > Threads (connected/running/created/cached): 1801/91/3607/0 > Threads (connected/running/created/cached): 1801/91/3607/0 > ... > > It is working fine 10 minutes without any problem. If KSE is failing with ULE in some way then you are not properly using the sched api. The scheduler has to be told when threads, kses, ksegrps, and procs are created and destroyed. ULE is more sensitive to this than 4BSD because it tracks more state on each object. Cheers, Jeff > > >Kris Gale > >_______________________________________________ > >freebsd-threads@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-threads > >To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > > > > > > > > > > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" >