From owner-freebsd-threads@FreeBSD.ORG Sun Apr 18 09:10:59 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 790A916A4CE for ; Sun, 18 Apr 2004 09:10:59 -0700 (PDT) Received: from geri.cc.fer.hr (geri.cc.fer.hr [161.53.72.107]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD58D43D1D for ; Sun, 18 Apr 2004 09:10:58 -0700 (PDT) (envelope-from ivoras@geri.cc.fer.hr) Received: from geri.cc.fer.hr (geri.cc.fer.hr [161.53.72.107]) by geri.cc.fer.hr (8.12.9p2/8.12.8) with ESMTP id i3IGAa1m094976 for ; Sun, 18 Apr 2004 18:10:36 +0200 (CEST) (envelope-from ivoras@geri.cc.fer.hr) Message-ID: <4082A88E.5060605@geri.cc.fer.hr> Date: Sun, 18 Apr 2004 18:10:54 +0200 From: Ivan Voras User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-threads@freebsd.org X-Enigmail-Version: 0.83.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Siege segfaulting X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2004 16:10:59 -0000 Hi! I was asked to send details about a problem I found with siege on 5.2-current. Here is my original post from current@freebsd.org: There are problems (segfault) running siege (web benchmark program) on a recent FreeBSD 5.2-current (a few days ago). When I switch it to libc_r via libmap.conf, everything works ok. Since the software is ported to a large number of platforms and works ok, it's likely a libpthread bug Siege uses threading to make simultaneous HTTP requests. I tried with a recent release (2.59) and beta (2.60) version if siege, with same behaviour. This does not happend with small number of threads (2-3), but if I specify a larger number (usually 8+), it crashes in the middle of work. In the faulting instances, siege was started as: siege -c20 -v -f list -t5m -d1 Thats: 20 threads, verbose output (writes status of each request to the console), "list" is the file that contains URLs, 5m is the duration of the test, d1 stands for: insert a random delay between 0 and 1 seconds before each request. The web application tested is made in PHP and requests take from about 0.02sec to 1.5sec to serve. Changing the duration (-t), random delay (-d) did not have any effect. Kernel is "mostly" GENERIC, with support for older processors, ipv6 and unneeded drivers removed (as well as witness and other debugging options), and with ULE scheduler. -- C isn't that hard: void (*(*f[])())() defines f as an array of unspecified size, of pointers to functions that return pointers to functions that return void. From owner-freebsd-threads@FreeBSD.ORG Sun Apr 18 12:34:43 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BD1D316A4CE; Sun, 18 Apr 2004 12:34:43 -0700 (PDT) Received: from geri.cc.fer.hr (geri.cc.fer.hr [161.53.72.107]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1E42F43D46; Sun, 18 Apr 2004 12:34:43 -0700 (PDT) (envelope-from ivoras@geri.cc.fer.hr) Received: from geri.cc.fer.hr (geri.cc.fer.hr [161.53.72.107]) by geri.cc.fer.hr (8.12.9p2/8.12.8) with ESMTP id i3IJYK1m000764; Sun, 18 Apr 2004 21:34:21 +0200 (CEST) (envelope-from ivoras@geri.cc.fer.hr) Message-ID: <4082D84C.2010001@geri.cc.fer.hr> Date: Sun, 18 Apr 2004 21:34:36 +0200 From: Ivan Voras User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: In-Reply-To: X-Enigmail-Version: 0.83.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: current@FreeBSD.org cc: freebsd-threads@FreeBSD.org Subject: Re: siege and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2004 19:34:44 -0000 Robert Watson wrote: > (1) libthr.so substituition via libmap.conf. > (2) linuxthreads substitution -- not sure if that can be done with > libmap.conf. I don't think linuxthreads can be done with libmap, and even if it does, I can't do it for another couple of weeks... With libthr it's the same. During the test I noticed in 'top' somewhat strange differences in time spent in the 'system' mode. Running under /usr/bin/time confirms that: ============= libthr: alfredo:~> /usr/bin/time -hlp siege -c20 -t1m -d1 -f web_list ** Siege 2.59 ** Preparing 20 concurrent users for battle. The server is now under siege...time: command terminated abnormally real 19.88 user 0.76 sys 1.65 3700 maximum resident set size 55 average shared memory size 997 average unshared data size 137 average unshared stack size 700 page reclaims 0 page faults 0 swaps 1 block input operations 4 block output operations 501 messages sent 7463 messages received 0 signals received 3261 voluntary context switches 13445 involuntary context switches Segmentation fault ============= libpthread: alfredo:~> /usr/bin/time -hlp siege -c20 -t1m -d1 -f web_list ** Siege 2.59 ** Preparing 20 concurrent users for battle. The server is now under siege...time: command terminated abnormally real 32.79 user 0.95 sys 0.66 4340 maximum resident set size 59 average shared memory size 1406 average unshared data size 145 average unshared stack size 902 page reclaims 0 page faults 0 swaps 33 block input operations 34 block output operations 995 messages sent 25512 messages received 0 signals received 25287 voluntary context switches 5625 involuntary context switches Segmentation fault ============= libc_r: alfredo:~> /usr/bin/time -hlp siege -c20 -t1m -d1 -f web_list ** Siege 2.59 ** Preparing 20 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 1804 hits Availability: 100.00 % Elapsed time: 59.74 secs Data transferred: 68221721 bytes Response time: 0.14 secs Transaction rate: 30.20 trans/sec Throughput: 1141935.44 bytes/sec Concurrency: 4.11 Successful transactions: 1804 Failed transactions: 0 real 59.78 user 1.40 sys 1.94 3884 maximum resident set size 50 average shared memory size 1031 average unshared data size 123 average unshared stack size 693 page reclaims 0 page faults 0 swaps 0 block input operations 0 block output operations 1808 messages sent 47267 messages received 311 signals received 44678 voluntary context switches 9919 involuntary context switches But then again, those probably doesn't mean anything, because the measurements are taken for different time periods. -- C isn't that hard: void (*(*f[])())() defines f as an array of unspecified size, of pointers to functions that return pointers to functions that return void. From owner-freebsd-threads@FreeBSD.ORG Sun Apr 18 14:00:22 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F2CD816A535 for ; Sun, 18 Apr 2004 14:00:21 -0700 (PDT) Received: from mta4.rcsntx.swbell.net (mta4.rcsntx.swbell.net [151.164.30.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id C2C0043D3F for ; Sun, 18 Apr 2004 14:00:21 -0700 (PDT) (envelope-from mbsd@pacbell.net) Received: from atlas (adsl-64-160-45-15.dsl.snfc21.pacbell.net [64.160.45.15]) i3IL0Jh8024718; Sun, 18 Apr 2004 16:00:20 -0500 (CDT) Date: Sun, 18 Apr 2004 14:00:19 -0700 (PDT) From: =?ISO-8859-1?Q?Mikko_Ty=F6l=E4j=E4rvi?= X-X-Sender: mikko@atlas.home To: Ivan Voras In-Reply-To: <4082A88E.5060605@geri.cc.fer.hr> Message-ID: <20040418133853.A31913@atlas.home> References: <4082A88E.5060605@geri.cc.fer.hr> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: Siege segfaulting X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2004 21:00:22 -0000 On Sun, 18 Apr 2004, Ivan Voras wrote: > I was asked to send details about a problem I found with siege on > 5.2-current. Here is my original post from current@freebsd.org: > > There are problems (segfault) running siege (web benchmark program) on a > recent FreeBSD 5.2-current (a few days ago). When I switch it to libc_r > via libmap.conf, everything works ok. Since the software is ported to a > large number of platforms and works ok, it's likely a libpthread bug Or not. Last time I tried to use siege, I ran into so many bugs that I gave up on it. > Siege uses threading to make simultaneous HTTP requests. I tried with a > recent release (2.59) and beta (2.60) version if siege, with same behaviour. > > This does not happend with small number of threads (2-3), but if I > specify a larger number (usually 8+), it crashes in the middle of work. Looks like a porting error; siege is calling gethostbyname(), which is non-reentrant. Try this patch (put it into siege/files/patch-src::sock.c) and rebuild the port: % cd /usr/ports/benchmarks/siege && cat files/patch-src::sock.c --- src/sock.c.orig Sun Apr 18 13:34:08 2004 +++ src/sock.c Sun Apr 18 13:34:43 2004 @@ -132,7 +132,7 @@ if((gethostbyname_r( hn, &hent, hbf, sizeof(hbf), &hp, &herrno ) < 0)){ hp = NULL; } -#elif defined(sun) +#elif defined(sun) || defined(__FreeBSD__) # ifdef HAVE_GETIPNODEBYNAME hp = getipnodebyname( hn, AF_INET, 0, &herrno ); # else /* default use gethostbyname_r*/ @@ -155,7 +155,7 @@ if( hp == NULL ){ return -1; } memset((void*) &cli, 0, sizeof( cli )); memcpy( &cli.sin_addr, hp->h_addr, hp->h_length ); -#if defined(sun) +#if defined(sun) || defined(__FreeBSD__) # ifdef HAVE_FREEHOSTENT freehostent(hp); # endif/*HAVE_FREEHOSTENT*/ @@ -257,6 +257,7 @@ return( C->sock ); } +#if 0 /* unused */ /** * local function * returns hostent based on OS specifics @@ -313,6 +314,7 @@ xfree( buf ); return( hp ); } +#endif /** * makes the socket non-blocking, I ran some quick tests on a less than a month old -CURRENT, and with this patch siege runs to completion. Without the patch, siege crashes within half a minute. $.02, /Mikko From owner-freebsd-threads@FreeBSD.ORG Mon Apr 19 11:01:39 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D230016A51B for ; Mon, 19 Apr 2004 11:01:39 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id B629643D2D for ; Mon, 19 Apr 2004 11:01:39 -0700 (PDT) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (peter@localhost [127.0.0.1]) i3JI1dbv042913 for ; Mon, 19 Apr 2004 11:01:39 -0700 (PDT) (envelope-from owner-bugmaster@freebsd.org) Received: (from peter@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i3JI1dWG042907 for freebsd-threads@freebsd.org; Mon, 19 Apr 2004 11:01:39 -0700 (PDT) (envelope-from owner-bugmaster@freebsd.org) Date: Mon, 19 Apr 2004 11:01:39 -0700 (PDT) Message-Id: <200404191801.i3JI1dWG042907@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: peter set sender to owner-bugmaster@freebsd.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2004 18:01:40 -0000 Current FreeBSD problem reports Critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/06/13] kern/19247 threads uthread_sigaction.c does not do anything s [2004/03/15] kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/un 2 problems total. Serious problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/07/18] kern/20016 threads pthreads: Cannot set scheduling timer/Can o [2000/08/26] misc/20861 threads libc_r does not honor socket timeouts o [2001/01/19] bin/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVT o [2001/01/25] bin/24632 threads libc_r delicate deviation from libc in ha o [2001/01/25] misc/24641 threads pthread_rwlock_rdlock can deadlock o [2001/11/26] bin/32295 threads pthread dont dequeue signals o [2002/02/01] i386/34536 threads accept() blocks other threads o [2002/05/25] kern/38549 threads the procces compiled whith pthread stoppe o [2002/06/27] bin/39922 threads [PATCH?] Threaded applications executed w o [2002/08/04] misc/41331 threads Pthread library open sets O_NONBLOCK flag o [2003/03/02] bin/48856 threads Setting SIGCHLD to SIG_IGN still leaves z o [2003/03/10] bin/49087 threads Signals lost in programs linked with libc a [2003/04/08] bin/50733 threads buildworld won't build, because of linkin o [2003/05/07] bin/51949 threads thread in accept cannot be cancelled 14 problems total. Non-critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/05/25] misc/18824 threads gethostbyname is not thread safe o [2000/10/21] misc/22190 threads A threaded read(2) from a socketpair(2) f o [2001/09/09] bin/30464 threads pthread mutex attributes -- pshared o [2002/05/02] bin/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwri s [2002/07/16] misc/40671 threads pthread_cancel doesn't remove thread from 5 problems total. From owner-freebsd-threads@FreeBSD.ORG Tue Apr 20 02:27:33 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4655516A4CF for ; Tue, 20 Apr 2004 02:27:33 -0700 (PDT) Received: from relay.pair.com (relay.pair.com [209.68.1.20]) by mx1.FreeBSD.org (Postfix) with SMTP id AAD8243D64 for ; Tue, 20 Apr 2004 02:27:32 -0700 (PDT) (envelope-from silby@silby.com) Received: (qmail 51240 invoked from network); 20 Apr 2004 09:27:31 -0000 Received: from niwun.pair.com (HELO localhost) (209.68.2.70) by relay.pair.com with SMTP; 20 Apr 2004 09:27:31 -0000 X-pair-Authenticated: 209.68.2.70 Date: Tue, 20 Apr 2004 06:23:34 -0500 (CDT) From: Mike Silbersack To: threads@freebsd.org Message-ID: <20040420062033.S20848@odysseus.silby.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: MySQL and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2004 09:27:33 -0000 I haven't had time to fully track it down yet, but FYI I've found that MySQL and libpthread don't seem to get along well for me; with sources from april 12th and mysql 4.1 from ports, mysqld just seems to hang for long periods of time, and is unkillable. When I libmap it to libthr or libc_r, all is well. The test I was running was simply mysql's included benchmark suite, using local socket connections to mysql. If one of the libpthread developers can not reproduce this behavior, I'll try more tests later in the week to see if I can pin it down more. Mike "Silby" Silbersack From owner-freebsd-threads@FreeBSD.ORG Tue Apr 20 02:53:25 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EEF9E16A4CE for ; Tue, 20 Apr 2004 02:53:25 -0700 (PDT) Received: from publicd.ub.mng.net (publicd.ub.mng.net [202.179.0.88]) by mx1.FreeBSD.org (Postfix) with ESMTP id 423AF43D31 for ; Tue, 20 Apr 2004 02:53:24 -0700 (PDT) (envelope-from ganbold@micom.mng.net) Received: from [202.179.0.164] (helo=ganbold.micom.mng.net) by publicd.ub.mng.net with asmtp (Exim 4.30; FreeBSD) id 1BFrwI-000C8Z-L7; Tue, 20 Apr 2004 17:53:14 +0800 Message-Id: <6.0.3.0.2.20040420185111.02ad3968@202.179.0.80> X-Sender: ganbold@micom.mng.net@202.179.0.80 X-Mailer: QUALCOMM Windows Eudora Version 6.0.3.0 Date: Tue, 20 Apr 2004 18:58:17 +0900 To: Mike Silbersack From: Ganbold In-Reply-To: <20040420062033.S20848@odysseus.silby.com> References: <20040420062033.S20848@odysseus.silby.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed cc: threads@freebsd.org Subject: Re: MySQL and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2004 09:53:26 -0000 Hi, I patched mysql with Daniel Eischen's patch that makes it use scope process threads. Also I'm using SCHED_4BSD scheduler and now mysql behaves much better. Mysql with libpthread runs without any crash since April 16th. I have cvs sources from April 16th and mysql-4.0.18 from ports. Ganbold At 08:23 PM 20.04.2004, you wrote: >I haven't had time to fully track it down yet, but FYI I've found that >MySQL and libpthread don't seem to get along well for me; with sources >from april 12th and mysql 4.1 from ports, mysqld just seems to hang for >long periods of time, and is unkillable. When I libmap it to libthr or >libc_r, all is well. > >The test I was running was simply mysql's included benchmark suite, using >local socket connections to mysql. If one of the libpthread developers >can not reproduce this behavior, I'll try more tests later in the week to >see if I can pin it down more. > >Mike "Silby" Silbersack >_______________________________________________ >freebsd-threads@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-threads >To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" From owner-freebsd-threads@FreeBSD.ORG Tue Apr 20 04:39:24 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BFF8E16A4CE for ; Tue, 20 Apr 2004 04:39:24 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 396C743D5A for ; Tue, 20 Apr 2004 04:39:24 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i3KBdNtf026104; Tue, 20 Apr 2004 07:39:23 -0400 (EDT) Date: Tue, 20 Apr 2004 07:39:23 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Mike Silbersack In-Reply-To: <20040420062033.S20848@odysseus.silby.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: MySQL and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2004 11:39:24 -0000 On Tue, 20 Apr 2004, Mike Silbersack wrote: > > I haven't had time to fully track it down yet, but FYI I've found that > MySQL and libpthread don't seem to get along well for me; with sources > from april 12th and mysql 4.1 from ports, mysqld just seems to hang for > long periods of time, and is unkillable. When I libmap it to libthr or > libc_r, all is well. There are a lot of posts regarding mysql in the threads archives. Two things -- mysql with libwrap support doesn't work with default CFLAGS (-march), and mysql uses scope system threads which chew up more kernel resources and can hang when you hit kern.threads.* limits. To make mysql40 use scope process threads: http://people.freebsd.org/~deischen/mysql40-server.diffs > The test I was running was simply mysql's included benchmark suite, using > local socket connections to mysql. If one of the libpthread developers > can not reproduce this behavior, I'll try more tests later in the week to > see if I can pin it down more. I've got mysql40-server running here with the above patches and a test script that causes creation of 1800 mysql threads. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Apr 20 12:13:22 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A95916A4CE for ; Tue, 20 Apr 2004 12:13:22 -0700 (PDT) Received: from miramanee.icarz.com (miramanee.icarz.com [207.99.22.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id CFA7343D1D for ; Tue, 20 Apr 2004 12:13:21 -0700 (PDT) (envelope-from kenfreebsd@icarz.com) Received: from deanna.icarz.com (deanna.icarz.com [207.99.22.19]) by miramanee.icarz.com (8.12.10/8.12.10) with ESMTP id i3KJD9BQ026867 for ; Tue, 20 Apr 2004 15:13:09 -0400 (EDT) (envelope-from kenfreebsd@icarz.com) Received: from kenxp (netb-138.icarz.com [209.123.219.138]) by deanna.icarz.com (8.12.9p2/8.12.9) with SMTP id i3KJD6Rd075592 for ; Tue, 20 Apr 2004 15:13:06 -0400 (EDT) (envelope-from kenfreebsd@icarz.com) Message-ID: <05b101c4270b$82e696d0$8adb7bd1@icarz.com> From: "Ken Menzel" To: References: Date: Tue, 20 Apr 2004 15:13:06 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 X-Spam-Score: -100.001 () BAYES_40,USER_IN_WHITELIST X-Scanned-By: MIMEDefang 2.39 Subject: Re: MySQL and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Ken Menzel List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2004 19:13:22 -0000 Hi Dan, Looks like a simple enough patch, is this something that should be suggested to the mysql folks to include in future versions for FreeBSD? Looks like this patch does not break 4.9 with or without Linux threads. (I checked). Is there any downside to this for FreeBSD? Would other platforms perhaps benefit from this change? Ken ----- Original Message ----- From: "Daniel Eischen" To: "Mike Silbersack" Cc: Sent: Tuesday, April 20, 2004 7:39 AM Subject: Re: MySQL and libpthread > On Tue, 20 Apr 2004, Mike Silbersack wrote: > > > > > I haven't had time to fully track it down yet, but FYI I've found that > > MySQL and libpthread don't seem to get along well for me; with sources > > from april 12th and mysql 4.1 from ports, mysqld just seems to hang for > > long periods of time, and is unkillable. When I libmap it to libthr or > > libc_r, all is well. > > There are a lot of posts regarding mysql in the threads > archives. Two things -- mysql with libwrap support doesn't > work with default CFLAGS (-march), and mysql uses scope > system threads which chew up more kernel resources and > can hang when you hit kern.threads.* limits. > > To make mysql40 use scope process threads: > > http://people.freebsd.org/~deischen/mysql40-server.diffs > > > The test I was running was simply mysql's included benchmark suite, using > > local socket connections to mysql. If one of the libpthread developers > > can not reproduce this behavior, I'll try more tests later in the week to > > see if I can pin it down more. > > I've got mysql40-server running here with the above patches > and a test script that causes creation of 1800 mysql threads. > > -- > Dan Eischen > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Apr 20 16:11:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EC32816A4CE for ; Tue, 20 Apr 2004 16:11:48 -0700 (PDT) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9DE4A43D48 for ; Tue, 20 Apr 2004 16:11:48 -0700 (PDT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc11) with ESMTP id <2004042023114701100k6qr8e>; Tue, 20 Apr 2004 23:11:47 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA21104; Tue, 20 Apr 2004 16:11:46 -0700 (PDT) Date: Tue, 20 Apr 2004 16:11:45 -0700 (PDT) From: Julian Elischer To: Mike Silbersack In-Reply-To: <20040420062033.S20848@odysseus.silby.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: MySQL and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Apr 2004 23:11:49 -0000 what does ps auxH show for that process when it's "hung"? or is it alxH? I guess the latter. On Tue, 20 Apr 2004, Mike Silbersack wrote: > > I haven't had time to fully track it down yet, but FYI I've found that > MySQL and libpthread don't seem to get along well for me; with sources > from april 12th and mysql 4.1 from ports, mysqld just seems to hang for > long periods of time, and is unkillable. When I libmap it to libthr or > libc_r, all is well. > > The test I was running was simply mysql's included benchmark suite, using > local socket connections to mysql. If one of the libpthread developers > can not reproduce this behavior, I'll try more tests later in the week to > see if I can pin it down more. > > Mike "Silby" Silbersack > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Apr 21 23:52:55 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EDD2116A4CE for ; Wed, 21 Apr 2004 23:52:55 -0700 (PDT) Received: from mail.violasystem.net (YahooBB219195104055.bbtec.net [219.195.104.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id D777543D53 for ; Wed, 21 Apr 2004 23:52:54 -0700 (PDT) (envelope-from kaakun@highway.ne.jp) Received: from face.violasystem.net ([192.168.11.5])i3M6RftJ000926 for ; Thu, 22 Apr 2004 15:27:41 +0900 (JST) (envelope-from kaakun@highway.ne.jp) Date: Thu, 22 Apr 2004 15:27:05 +0900 From: Kazuaki Oda To: threads@freebsd.org Message-Id: <20040422152705.01f574e1.kaakun@highway.ne.jp> X-Mailer: Sylpheed version 0.9.8a-gtk2-20040109 (GTK+ 2.2.4; i386-portbld-freebsd5.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: kse_release and kse_wakeup problem X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Apr 2004 06:52:56 -0000 Hi, I think, after switching to use new sleep queue interface, there is a problem when using scope system threads on MP machine. This problem occurs when one CPU is executing kse_release and another is kse_wakeup. There is a scenario (thread A and B is on the same process): CPU0: thread A aquires PROC_LOCK in kse_release (kern_thread.c 621) CPU0: thread A releases PROC_LOCK in msleep (kern_synch.c 209) CPU1: thread B aquires PROC_LOCK in kse_wakeup (kern_thread.c 667) CPU1: thread B looks up kse_upcall (kern_thread.c 669-687) CPU1: thread B gets kse_upcall owner and this is thread A (kern_thread.c 689) CPU1: thread B sets KUF_DOUPCALL flag (kern_thread.c 697) because thread A is not on sleep queue yet, sleepq_abort (kern_thread.c 695) is not executed. CPU1: thread B releases PROC_LOCK (kern_thread.c 700) CPU0: thread A puts himself on sleep queue (kern_synch.c 221) CPU0: thread A sets TDF_SINTR flag (subr_sleepqueue.c 310) CPU0: thread A sleeps and context switch occurs... I think, thread B should call sleepq_abort and thread A should do upcall as soon as possible. The following patch is for thread A to release PROC_LOCK after putting himself on sleep queue and setting TDF_SINTR flag. I don't think this patch is so good (obviously setting TDF_SINTR here is not good), but enough for test. And, after patching, MySQL does not hang up on heavy load on my machine (P4 2.40GHz, HTT enabled). --- kern_synch.c.orig Thu Apr 22 14:00:40 2004 +++ kern_synch.c Thu Apr 22 12:20:06 2004 @@ -203,11 +203,6 @@ td, p->p_pid, p->p_comm, wmesg, ident); DROP_GIANT(); - if (mtx != NULL) { - mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); - WITNESS_SAVE(&mtx->mtx_object, mtx); - mtx_unlock(mtx); - } /* * We put ourselves on the sleep queue and start our timeout @@ -219,6 +214,13 @@ * return from cursig(). */ sleepq_add(sq, ident, mtx, wmesg, 0); + if (catch) + td->td_flags |= TDF_SINTR; + if (mtx != NULL) { + mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); + WITNESS_SAVE(&mtx->mtx_object, mtx); + mtx_unlock(mtx); + } if (timo) sleepq_set_timeout(ident, timo); if (catch) { -- Kazuaki Oda From owner-freebsd-threads@FreeBSD.ORG Thu Apr 22 05:50:21 2004 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BA23A16A4CE for ; Thu, 22 Apr 2004 05:50:21 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id A6B1343D53 for ; Thu, 22 Apr 2004 05:50:21 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i3MCoLbv062518 for ; Thu, 22 Apr 2004 05:50:21 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i3MCoLuK062517; Thu, 22 Apr 2004 05:50:21 -0700 (PDT) (envelope-from gnats) Resent-Date: Thu, 22 Apr 2004 05:50:21 -0700 (PDT) Resent-Message-Id: <200404221250.i3MCoLuK062517@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-threads@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Max Fomitchev Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1D25016A4CE for ; Thu, 22 Apr 2004 05:47:32 -0700 (PDT) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id 01A6043D46 for ; Thu, 22 Apr 2004 05:47:32 -0700 (PDT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.12.10/8.12.10) with ESMTP id i3MClV72013762 for ; Thu, 22 Apr 2004 05:47:31 -0700 (PDT) (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.12.10/8.12.10/Submit) id i3MClVLW013761; Thu, 22 Apr 2004 05:47:31 -0700 (PDT) (envelope-from nobody) Message-Id: <200404221247.i3MClVLW013761@www.freebsd.org> Date: Thu, 22 Apr 2004 05:47:31 -0700 (PDT) From: Max Fomitchev To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Subject: threads/65883: libkse's sigwait does not work after fork X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Apr 2004 12:50:21 -0000 >Number: 65883 >Category: threads >Synopsis: libkse's sigwait does not work after fork >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-threads >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Apr 22 05:50:21 PDT 2004 >Closed-Date: >Last-Modified: >Originator: Max Fomitchev >Release: 5.1-RELEASE, 5.2-RELEASE, 5-CURRENT >Organization: Kaspersky Labs >Environment: FreeBSD freebsd-smtpgw.testlab2k.avp.ru 5.1-RELEASE FreeBSD 5.1-RELEASE #0: Thu Jun 5 02:55:42 GMT 2003 root@wv1u.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC i386 >Description: sigwait() call cannot receive a signal after fork() if the following code (simplified model of my current application) linked with libkse. This code works well with a various thread libraries (libc_r for example). Another interesting feature, if you do not start threads explicitly at all, sigwait() will receive SIGINT from the scheduler(???) and the program exits. //kse_test.c begin #include #include #include #include static pthread_mutex_t m_lock; static int ended; static void *thr (void *arg) { for (;;) { pthread_mutex_lock(&m_lock); if (ended == 1) { pthread_mutex_unlock(&m_lock); break; } pthread_mutex_unlock(&m_lock); sleep(1); } return NULL; } static int dmn() { switch (fork()) { case 0: break; case -1: return -1; default: _exit(0); } if (setsid() < 0) return -1; switch (fork()) { case 0: break; case -1: return -1; default: _exit(0); } return 0; } static int close_ios() { int fd_l = 0; int fd = 0; errno = 0; chdir("/"); fd_l = sysconf(_SC_OPEN_MAX); if ((fd_l < 0) || (errno != 0)) return -1; for (fd = 0; fd < fd_l; fd++) close(fd); return 0; } static void signal_manager() { int sign = 0; sigset_t sigs; sigemptyset(&sigs); sigaddset(&sigs, SIGINT); //exit sigprocmask(SIG_BLOCK, &sigs, NULL); for (;;) { sigwait(&sigs, &sign); switch (sign) { case SIGINT: pthread_mutex_lock(&m_lock); ended = 1; pthread_mutex_unlock(&m_lock); return; default: continue; } } } int main(int argc, const char *argv[]) { pthread_mutexattr_t mattr; pthread_attr_t pattr; pthread_t tid1, tid2; void *exit_status; ended = 0; if (dmn() != 0) return 10; if (close_ios() != 0) return 20; if ((pthread_attr_init(&pattr) != 0) || (pthread_attr_setdetachstate(&pattr, PTHREAD_CREATE_JOINABLE) != 0)) return 30; if ((pthread_mutexattr_init(&mattr) != 0) || (pthread_mutex_init(&m_lock, &mattr) != 0)) return 40; if (pthread_create(&tid1, &pattr, thr, NULL) != 0) return 51; if (pthread_create(&tid2, &pattr, thr, NULL) != 0) return 52; signal_manager(); pthread_join(tid1, &exit_status); pthread_join(tid2, &exit_status); return 0; } //kse_test.c end >How-To-Repeat: bash-2.05b# gcc -v Using built-in specs. Configured with: FreeBSD/i386 system compiler Thread model: posix gcc version 3.2.2 [FreeBSD] 20030205 (release) bash-2.05b# gcc kse_test.c -o kse_test -lc_r bash-2.05b# ldd kse_test kse_test: libc_r.so.5 => /usr/lib/libc_r.so.5 (0x2806b000) libc.so.5 => /usr/lib/libc.so.5 (0x2808e000) bash-2.05b# ./kse_test; echo $? 0 bash-2.05b# ps -ax | grep kse_test 62277 ?? S 0:00.00 ./kse_test bash-2.05b# kill -INT 62277 bash-2.05b# ps -ax | grep kse_test bash-2.05b# gcc kse_test.c -o kse_test -lkse bash-2.05b# ldd kse_test kse_test: libkse.so.1 => /usr/lib/libkse.so.1 (0x2806b000) libc.so.5 => /usr/lib/libc.so.5 (0x2808b000) bash-2.05b# ./kse_test; echo $? 0 bash-2.05b# ps -ax | grep kse_test 62288 ?? S 0:00.00 ./kse_test bash-2.05b# kill -INT 62288 bash-2.05b# ps -ax | grep kse_test 62288 ?? R 0:00.00 ./kse_test bash-2.05b# kill -9 62288 bash-2.05b# ps -ax | grep kse_test >Fix: >Release-Note: >Audit-Trail: >Unformatted: From owner-freebsd-threads@FreeBSD.ORG Thu Apr 22 06:15:02 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B0C6916A4D0; Thu, 22 Apr 2004 06:15:02 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 604D543D41; Thu, 22 Apr 2004 06:15:02 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i3MDF1tf019400; Thu, 22 Apr 2004 09:15:01 -0400 (EDT) Date: Thu, 22 Apr 2004 09:15:01 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Max Fomitchev In-Reply-To: <200404221247.i3MClVLW013761@www.freebsd.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-gnats-submit@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: threads/65883: libkse's sigwait does not work after fork X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Apr 2004 13:15:02 -0000 On Thu, 22 Apr 2004, Max Fomitchev wrote: > > >How-To-Repeat: > bash-2.05b# gcc kse_test.c -o kse_test -lkse > bash-2.05b# ldd kse_test > kse_test: > libkse.so.1 => /usr/lib/libkse.so.1 (0x2806b000) > libc.so.5 => /usr/lib/libc.so.5 (0x2808b000) > bash-2.05b# ./kse_test; echo $? > 0 > bash-2.05b# ps -ax | grep kse_test > 62288 ?? S 0:00.00 ./kse_test > bash-2.05b# kill -INT 62288 > bash-2.05b# ps -ax | grep kse_test > 62288 ?? R 0:00.00 ./kse_test > bash-2.05b# kill -9 62288 > bash-2.05b# ps -ax | grep kse_test I can't repeat this with libpthread (it's no longer libkse). -bash-2.05b$ gcc -Wall -o kse_test kse_test.c -lpthread -bash-2.05b$ ldd kse_test kse_test: libpthread.so.1 => /usr/lib/libpthread.so.1 (0x2807a000) libc.so.5 => /lib/libc.so.5 (0x2809d000) -bash-2.05b$ kse_test -bash-2.05b$ echo $? 0 -bash-2.05b$ ps -ax | grep kse_test 712 ?? S 0:00.02 kse_test 698 p1 I 0:00.85 nedit kse_test.c 727 p1 D+ 0:00.02 grep kse_test -bash-2.05b$ kill -INT 712 -bash-2.05b$ ps -ax | grep kse_test 698 p1 I 0:00.85 nedit kse_test.c 729 p1 S+ 0:00.03 grep kse_test -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Apr 22 06:20:24 2004 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8988516A4CE for ; Thu, 22 Apr 2004 06:20:24 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 810D243D45 for ; Thu, 22 Apr 2004 06:20:24 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) i3MDKObv036297 for ; Thu, 22 Apr 2004 06:20:24 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.10/8.12.10/Submit) id i3MDKOSC036296; Thu, 22 Apr 2004 06:20:24 -0700 (PDT) (envelope-from gnats) Date: Thu, 22 Apr 2004 06:20:24 -0700 (PDT) Message-Id: <200404221320.i3MDKOSC036296@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Daniel Eischen Subject: Re: threads/65883: libkse's sigwait does not work after fork X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Apr 2004 13:20:24 -0000 The following reply was made to PR threads/65883; it has been noted by GNATS. From: Daniel Eischen To: Max Fomitchev Cc: freebsd-gnats-submit@freebsd.org, freebsd-threads@freebsd.org Subject: Re: threads/65883: libkse's sigwait does not work after fork Date: Thu, 22 Apr 2004 09:15:01 -0400 (EDT) On Thu, 22 Apr 2004, Max Fomitchev wrote: > > >How-To-Repeat: > bash-2.05b# gcc kse_test.c -o kse_test -lkse > bash-2.05b# ldd kse_test > kse_test: > libkse.so.1 => /usr/lib/libkse.so.1 (0x2806b000) > libc.so.5 => /usr/lib/libc.so.5 (0x2808b000) > bash-2.05b# ./kse_test; echo $? > 0 > bash-2.05b# ps -ax | grep kse_test > 62288 ?? S 0:00.00 ./kse_test > bash-2.05b# kill -INT 62288 > bash-2.05b# ps -ax | grep kse_test > 62288 ?? R 0:00.00 ./kse_test > bash-2.05b# kill -9 62288 > bash-2.05b# ps -ax | grep kse_test I can't repeat this with libpthread (it's no longer libkse). -bash-2.05b$ gcc -Wall -o kse_test kse_test.c -lpthread -bash-2.05b$ ldd kse_test kse_test: libpthread.so.1 => /usr/lib/libpthread.so.1 (0x2807a000) libc.so.5 => /lib/libc.so.5 (0x2809d000) -bash-2.05b$ kse_test -bash-2.05b$ echo $? 0 -bash-2.05b$ ps -ax | grep kse_test 712 ?? S 0:00.02 kse_test 698 p1 I 0:00.85 nedit kse_test.c 727 p1 D+ 0:00.02 grep kse_test -bash-2.05b$ kill -INT 712 -bash-2.05b$ ps -ax | grep kse_test 698 p1 I 0:00.85 nedit kse_test.c 729 p1 S+ 0:00.03 grep kse_test -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Apr 22 12:28:11 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A9BA016A4CE for ; Thu, 22 Apr 2004 12:28:11 -0700 (PDT) Received: from relay.pair.com (relay.pair.com [209.68.1.20]) by mx1.FreeBSD.org (Postfix) with SMTP id 2CE7543D49 for ; Thu, 22 Apr 2004 12:28:11 -0700 (PDT) (envelope-from silby@silby.com) Received: (qmail 71941 invoked from network); 22 Apr 2004 19:28:10 -0000 Received: from niwun.pair.com (HELO localhost) (209.68.2.70) by relay.pair.com with SMTP; 22 Apr 2004 19:28:10 -0000 X-pair-Authenticated: 209.68.2.70 Date: Thu, 22 Apr 2004 14:57:27 -0500 (CDT) From: Mike Silbersack To: Julian Elischer In-Reply-To: Message-ID: <20040422145637.O21358@odysseus.silby.com> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: MySQL and libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Apr 2004 19:28:11 -0000 On Tue, 20 Apr 2004, Julian Elischer wrote: > what does ps auxH show for that process when it's "hung"? > or is it alxH? I guess the latter. I'll take a stab at this once we're done with the rest of the testing for our project. I just decided to go with libthr for now, we're only running one or two concurrent clients, so threading isn't a huge deal. Mike "Silby" Silbersack From owner-freebsd-threads@FreeBSD.ORG Thu Apr 22 15:02:02 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 452F016A4CE; Thu, 22 Apr 2004 15:02:02 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id C7C5243D49; Thu, 22 Apr 2004 15:02:01 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i3MM20tf016670; Thu, 22 Apr 2004 18:02:00 -0400 (EDT) Date: Thu, 22 Apr 2004 18:02:00 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: John Baldwin In-Reply-To: <200404220954.00469.jhb@FreeBSD.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org cc: davidxu@FreeBSD.org Subject: Re: kse_release and kse_wakeup problem (fwd) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Apr 2004 22:02:02 -0000 Sorry, I should have included threads@. On Thu, 22 Apr 2004, John Baldwin wrote: > On Thursday 22 April 2004 09:01 am, Daniel Eischen wrote: > > What do you guys think of this patch? > > I think that the thread code should check for the upcall case the same way The thread code where? Everywhere msleep() is called? > that we check for signals by calliing cursig() in sleepq_catch_signals(), > that is: > > /* Mark thread as being in an interruptible sleep. */ > mtx_lock_spin(&sched_lock); > MPASS(TD_ON_SLEEPQ(td)); > td->td_flags |= TDF_SINTR; > mtx_unlock_spin(&sched_lock); > sleepq_release(wchan); > > /* See if there are any pending signals for this thread. */ > PROC_LOCK(p); > mtx_lock(&p->p_sigacts->ps_mtx); > sig = cursig(td); > mtx_unlock(&p->p_sigacts->ps_mtx); > if (sig == 0 && thread_suspend_check(1)) > sig = SIGSTOP; > PROC_UNLOCK(p); > > /* > * If there were pending signals and this thread is still on > * the sleep queue, remove it from the sleep queue. > */ > > The thread_suspend_check() code should also be checking for the UPCALL case > and as well. Maybe something like: > > if (sig == 0 && thread_suspend_check(1)) > sig == SIGSTOP; > if (sig == 0 && thread_upcall_check()) > doing_upcall = 1; > > and then dequeue if we are doing an upcall just like we do if there is already > a signal. This mirrors the way we handle signals since UPCALLs are a kind of > special cased signal. The patch below has incorrect locking (td_flags could > get trashed) by the way. > > > ---------- Forwarded message ---------- > > Date: Thu, 22 Apr 2004 15:27:05 +0900 > > From: Kazuaki Oda > > To: threads@freebsd.org > > Subject: kse_release and kse_wakeup problem > > > > Hi, > > > > I think, after switching to use new sleep queue interface, there is a > > problem when using scope system threads on MP machine. > > This problem occurs when one CPU is executing kse_release and another > > is kse_wakeup. > > > > > > There is a scenario (thread A and B is on the same process): > > > > CPU0: thread A aquires PROC_LOCK in kse_release (kern_thread.c 621) > > CPU0: thread A releases PROC_LOCK in msleep (kern_synch.c 209) > > CPU1: thread B aquires PROC_LOCK in kse_wakeup (kern_thread.c 667) > > CPU1: thread B looks up kse_upcall (kern_thread.c 669-687) > > CPU1: thread B gets kse_upcall owner and this is thread A (kern_thread.c > > 689) CPU1: thread B sets KUF_DOUPCALL flag (kern_thread.c 697) > > because thread A is not on sleep queue yet, > > sleepq_abort (kern_thread.c 695) is not executed. > > CPU1: thread B releases PROC_LOCK (kern_thread.c 700) > > CPU0: thread A puts himself on sleep queue (kern_synch.c 221) > > CPU0: thread A sets TDF_SINTR flag (subr_sleepqueue.c 310) > > CPU0: thread A sleeps and context switch occurs... > > > > > > I think, thread B should call sleepq_abort and thread A should do > > upcall as soon as possible. > > The following patch is for thread A to release PROC_LOCK after putting > > himself on sleep queue and setting TDF_SINTR flag. > > I don't think this patch is so good (obviously setting TDF_SINTR here is > > not good), but enough for test. > > > > And, after patching, MySQL does not hang up on heavy load on my machine > > (P4 2.40GHz, HTT enabled). > > > > > > --- kern_synch.c.orig Thu Apr 22 14:00:40 2004 > > +++ kern_synch.c Thu Apr 22 12:20:06 2004 > > @@ -203,11 +203,6 @@ > > td, p->p_pid, p->p_comm, wmesg, ident); > > > > DROP_GIANT(); > > - if (mtx != NULL) { > > - mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); > > - WITNESS_SAVE(&mtx->mtx_object, mtx); > > - mtx_unlock(mtx); > > - } > > > > /* > > * We put ourselves on the sleep queue and start our timeout > > @@ -219,6 +214,13 @@ > > * return from cursig(). > > */ > > sleepq_add(sq, ident, mtx, wmesg, 0); > > + if (catch) > > + td->td_flags |= TDF_SINTR; > > + if (mtx != NULL) { > > + mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); > > + WITNESS_SAVE(&mtx->mtx_object, mtx); > > + mtx_unlock(mtx); > > + } > > if (timo) > > sleepq_set_timeout(ident, timo); > > if (catch) { > > -- > John Baldwin <>< http://www.FreeBSD.org/~jhb/ > "Power Users Use the Power to Serve" = http://www.FreeBSD.org > -- "Some folks are into open source, but me, I'm into open bar." -- Spencer F. Katt From owner-freebsd-threads@FreeBSD.ORG Thu Apr 22 22:32:51 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 20BA716A4CE; Thu, 22 Apr 2004 22:32:51 -0700 (PDT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id B173143D2F; Thu, 22 Apr 2004 22:32:50 -0700 (PDT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i3N5Wntf025368; Fri, 23 Apr 2004 01:32:49 -0400 (EDT) Date: Fri, 23 Apr 2004 01:32:49 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: John Baldwin In-Reply-To: <200404220954.00469.jhb@FreeBSD.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@FreeBSD.org cc: davidxu@FreeBSD.org Subject: Re: kse_release and kse_wakeup problem (fwd) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2004 05:32:51 -0000 On Thu, 22 Apr 2004, John Baldwin wrote: > On Thursday 22 April 2004 09:01 am, Daniel Eischen wrote: > > What do you guys think of this patch? > > I think that the thread code should check for the upcall case the same way > that we check for signals by calliing cursig() in sleepq_catch_signals(), > that is: > > /* Mark thread as being in an interruptible sleep. */ > mtx_lock_spin(&sched_lock); > MPASS(TD_ON_SLEEPQ(td)); > td->td_flags |= TDF_SINTR; > mtx_unlock_spin(&sched_lock); > sleepq_release(wchan); > > /* See if there are any pending signals for this thread. */ > PROC_LOCK(p); > mtx_lock(&p->p_sigacts->ps_mtx); > sig = cursig(td); > mtx_unlock(&p->p_sigacts->ps_mtx); > if (sig == 0 && thread_suspend_check(1)) > sig = SIGSTOP; > PROC_UNLOCK(p); > > /* > * If there were pending signals and this thread is still on > * the sleep queue, remove it from the sleep queue. > */ > > The thread_suspend_check() code should also be checking for the UPCALL case > and as well. Maybe something like: > > if (sig == 0 && thread_suspend_check(1)) > sig == SIGSTOP; > if (sig == 0 && thread_upcall_check()) > doing_upcall = 1; > > and then dequeue if we are doing an upcall just like we do if there is already > a signal. This mirrors the way we handle signals since UPCALLs are a kind of > special cased signal. The patch below has incorrect locking (td_flags could > get trashed) by the way. I think td_flags can get (conditionally) set just before the sched_lock is released (before GIANT is dropped). Other than that, the patch makes sense to me. It just makes sure that the thread is put on the sleep queue before the mutex is released. As long as the mutex is held while fiddling with KUF_DOUPCALL, it seems like that should work. I'm not sure what you are trying to say above. Should the thread code not be using msleep()? I don't like that it has to know a little something about sleepq. Usually with mutexes, CV's, and the like, you don't wake up particular threads which is what we need to do in this case. Index: kern_synch.c =================================================================== RCS file: /opt/FreeBSD/cvs/src/sys/kern/kern_synch.c,v retrieving revision 1.246 diff -u -r1.246 kern_synch.c --- kern_synch.c 12 Mar 2004 19:06:18 -0000 1.246 +++ kern_synch.c 23 Apr 2004 05:19:37 -0000 @@ -202,16 +202,13 @@ } } } + if (catch) + td->td_flags |= TDF_SINTR; mtx_unlock_spin(&sched_lock); CTR5(KTR_PROC, "msleep: thread %p (pid %d, %s) on %s (%p)", td, p->p_pid, p->p_comm, wmesg, ident); DROP_GIANT(); - if (mtx != NULL) { - mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); - WITNESS_SAVE(&mtx->mtx_object, mtx); - mtx_unlock(mtx); - } /* * We put ourselves on the sleep queue and start our timeout @@ -223,6 +220,11 @@ * return from cursig(). */ sleepq_add(sq, ident, mtx, wmesg, 0); + if (mtx != NULL) { + mtx_assert(mtx, MA_OWNED | MA_NOTRECURSED); + WITNESS_SAVE(&mtx->mtx_object, mtx); + mtx_unlock(mtx); + } if (timo) sleepq_set_timeout(ident, timo); if (catch) { -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Apr 23 11:08:14 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D0B0A16A4E4 for ; Fri, 23 Apr 2004 11:08:13 -0700 (PDT) Received: from mail2.speakeasy.net (mail2.speakeasy.net [216.254.0.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id 40A0B43D46 for ; Fri, 23 Apr 2004 11:08:13 -0700 (PDT) (envelope-from jhb@FreeBSD.org) Received: (qmail 30356 invoked from network); 23 Apr 2004 18:08:12 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 23 Apr 2004 18:08:12 -0000 Received: from 10.50.40.205 (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.10/8.12.10) with ESMTP id i3NI7v6E001779; Fri, 23 Apr 2004 14:08:09 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: Daniel Eischen Date: Fri, 23 Apr 2004 13:57:23 -0400 User-Agent: KMail/1.6 References: In-Reply-To: MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200404231357.23096.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: threads@FreeBSD.org cc: davidxu@FreeBSD.org Subject: Re: kse_release and kse_wakeup problem (fwd) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Apr 2004 18:08:14 -0000 On Thursday 22 April 2004 06:02 pm, Daniel Eischen wrote: > Sorry, I should have included threads@. > > On Thu, 22 Apr 2004, John Baldwin wrote: > > On Thursday 22 April 2004 09:01 am, Daniel Eischen wrote: > > > What do you guys think of this patch? > > > > I think that the thread code should check for the upcall case the same > > way > > The thread code where? Everywhere msleep() is called? No, in sleepq_catch_signals() just as I quoted below: > > that we check for signals by calliing cursig() in sleepq_catch_signals(), > > that is: > > > > /* Mark thread as being in an interruptible sleep. */ > > mtx_lock_spin(&sched_lock); > > MPASS(TD_ON_SLEEPQ(td)); > > td->td_flags |= TDF_SINTR; > > mtx_unlock_spin(&sched_lock); > > sleepq_release(wchan); > > > > /* See if there are any pending signals for this thread. */ > > PROC_LOCK(p); > > mtx_lock(&p->p_sigacts->ps_mtx); > > sig = cursig(td); > > mtx_unlock(&p->p_sigacts->ps_mtx); > > if (sig == 0 && thread_suspend_check(1)) > > sig = SIGSTOP; > > PROC_UNLOCK(p); > > > > /* > > * If there were pending signals and this thread is still on > > * the sleep queue, remove it from the sleep queue. > > */ > > > > The thread_suspend_check() code should also be checking for the UPCALL > > case and as well. Maybe something like: > > > > if (sig == 0 && thread_suspend_check(1)) > > sig == SIGSTOP; > > if (sig == 0 && thread_upcall_check()) > > doing_upcall = 1; > > > > and then dequeue if we are doing an upcall just like we do if there is > > already a signal. This mirrors the way we handle signals since UPCALLs > > are a kind of special cased signal. The patch below has incorrect > > locking (td_flags could get trashed) by the way. I.e. do the upcall check in sleepq_catch_signals() right where you already do thread_suspend_check(1). The only reason you have to do this, btw, is because the kse_release() code is trying to mess with thread state internals using sleepq_abort(), etc. The other in-kernel code that does that (signals) already does the check in sleepq_catch_signals() and has done the same type of check in msleep()/tsleep() for quite a while. If the kse_release() stuff was just using sleep/wakeup() rather than trying to manually abort sleeps it wouldn't have to be so intimate with the sleep interface. Note that thr's thr_wakeup() and thr_sleep() manage to simulate synchronization w/o having to abort sleeps, but it is probably also easier to do that than for the M:N case. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org