From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 11:02:10 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 15B7116A4CE for ; Mon, 14 Jun 2004 11:02:10 +0000 (GMT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id EC4DF43D2D for ; Mon, 14 Jun 2004 11:02:09 +0000 (GMT) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (peter@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.11/8.12.11) with ESMTP id i5EB1hgB072664 for ; Mon, 14 Jun 2004 11:01:43 GMT (envelope-from owner-bugmaster@freebsd.org) Received: (from peter@localhost) by freefall.freebsd.org (8.12.11/8.12.11/Submit) id i5EB1hVa072658 for freebsd-threads@freebsd.org; Mon, 14 Jun 2004 11:01:43 GMT (envelope-from owner-bugmaster@freebsd.org) Date: Mon, 14 Jun 2004 11:01:43 GMT Message-Id: <200406141101.i5EB1hVa072658@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: peter set sender to owner-bugmaster@freebsd.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 11:02:10 -0000 Current FreeBSD problem reports Critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/06/13] kern/19247 threads uthread_sigaction.c does not do anything s [2004/03/15] kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/un o [2004/04/22] threads/65883threads libkse's sigwait does not work after fork 3 problems total. Serious problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/07/18] kern/20016 threads pthreads: Cannot set scheduling timer/Can o [2000/08/26] misc/20861 threads libc_r does not honor socket timeouts o [2001/01/20] bin/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVT o [2001/01/25] bin/24632 threads libc_r delicate deviation from libc in ha o [2001/01/25] misc/24641 threads pthread_rwlock_rdlock can deadlock o [2001/11/26] bin/32295 threads pthread dont dequeue signals o [2002/02/01] i386/34536 threads accept() blocks other threads o [2002/05/25] kern/38549 threads the procces compiled whith pthread stoppe o [2002/06/27] bin/39922 threads [PATCH?] Threaded applications executed w o [2002/08/04] misc/41331 threads Pthread library open sets O_NONBLOCK flag o [2003/03/02] bin/48856 threads Setting SIGCHLD to SIG_IGN still leaves z o [2003/03/10] bin/49087 threads Signals lost in programs linked with libc o [2003/05/08] bin/51949 threads thread in accept cannot be cancelled 13 problems total. Non-critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/05/26] misc/18824 threads gethostbyname is not thread safe o [2000/10/21] misc/22190 threads A threaded read(2) from a socketpair(2) f o [2001/09/09] bin/30464 threads pthread mutex attributes -- pshared o [2002/05/02] bin/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwri s [2002/07/16] misc/40671 threads pthread_cancel doesn't remove thread from 5 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 18:01:26 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 89D3816A4CE; Mon, 14 Jun 2004 18:01:26 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id DAA8F43D5D; Mon, 14 Jun 2004 18:01:23 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 20:01:21 +0200 Message-ID: <01c301c45239$940781f0$6401a8c0@animal> From: "Lasse Laursen" To: freebsd-threads@freebsd.org, freebsd-current@freebsd.org Date: Mon, 14 Jun 2004 20:00:41 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 Subject: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 18:01:26 -0000 Hi, I posted this on the 2freebsq-questions" list some time ago but didn't get any response. Maybe someone on this list can help: We are running MySQL 4.0.20 on FreeBSD 5.2-CURRENT. The machine is a dual Xeon with 4GByte of memory with HTT enabled. Some info about the system: FreeBSD dbnode3 5.2-CURRENT FreeBSD 5.2-CURRENT #0: Tue Jun 8 14:41:02 CEST 2004 laursen@dbnode3:/usr/obj/usr/src/sys/DBNODE3 i386 the /etc/libmap.conf contains: libc_r.so.5 libpthread.so.1 libc_r.so libpthread.so /etc/make.conf contains: # -- use.perl generated deltas -- # # Created: Wed Jun 9 10:18:52 2004 # Setting to use base perl from ports: PERL_VER=5.8.4 PERL_VERSION=5.8.4 PERL_ARCH=mach NOPERL=yo NO_PERL=yo NO_PERL_WRAPPER=yo MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be broken) and WITH_PROC_SCOPE_PTH set to yes. WE use the SCHED_ULE, and: # Memory options options MAXDSIZ="(1536*1024*1024)" options MAXSSIZ="(1024*1024*1024)" options DFLDSIZ="(1536*1024*1024)" The server runs fine until a single thread/query suddenly locks up the entire MySQL daemon. After that all queries are just queued and a restart of the daemon is needed to unlock the system. The system itself is stable enough as far as I can see. 'top' reports a lot of locks (*Giant) so I assume that it's some weird problem with the threading? We used to use FreeBSD on non-SMP machines without any problems. I have tried with linux threads as well but the same problem occurs. Have anyone experienced any similar problems and found a solution to this rather weird problem? Feel free to request any further information about the system/setup and I will do my best to provide it to the list. Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 18:15:39 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D7B3B16A4D0; Mon, 14 Jun 2004 18:15:39 +0000 (GMT) Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id B446643D2D; Mon, 14 Jun 2004 18:15:39 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc13) with ESMTP id <2004061418153801500fdmi3e>; Mon, 14 Jun 2004 18:15:39 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id LAA31533; Mon, 14 Jun 2004 11:15:37 -0700 (PDT) Date: Mon, 14 Jun 2004 11:15:34 -0700 (PDT) From: Julian Elischer To: Lasse Laursen In-Reply-To: <01c301c45239$940781f0$6401a8c0@animal> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 18:15:40 -0000 On Mon, 14 Jun 2004, Lasse Laursen wrote: > > MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be > broken) and WITH_PROC_SCOPE_PTH set to yes. > > WE use the SCHED_ULE, and: > > # Memory options > options MAXDSIZ="(1536*1024*1024)" > options MAXSSIZ="(1024*1024*1024)" > options DFLDSIZ="(1536*1024*1024)" > > The server runs fine until a single thread/query suddenly locks up the > entire MySQL daemon. After that all queries are just queued and a restart of > the daemon is needed to unlock the system. The system itself is stable > enough as far as I can see. 'top' reports a lot of locks (*Giant) so I > assume that it's some weird problem with the threading? We used to use > FreeBSD on non-SMP machines without any problems. > > I have tried with linux threads as well but the same problem occurs. This does suggest a bug in MySQL, but let's investigate anyhow.. > > Have anyone experienced any similar problems and found a solution to this > rather weird problem? > > Feel free to request any further information about the system/setup and I > will do my best to provide it to the list. the output of "ps -alxH might be useful. also: do you have teh kernel debugger compiled into the kernel? (option DDB) if so.. note the PID of the process sysctl debug.enter_debugger=ddb [you are now in the debugger] ps [note the Thread address of the threads that seem to be blocking things] (I may ask for specific threads after seeing the 'ps' output) do show thread {address} to get a stack backtrace of those threads. if possible use a serial console to do all this.. That way you can log it all. to make a serial console, connect 2 machines com1 by a null-modem cable. on the other machine do: script tip com1 on the SQL machine add console="comconsole" into /boot/loader.conf and reboot all console out put from teh boot-loader-on will now go to teh serial port and appear in teh 'tip' output. (which is being logged by the 'script' command). julian From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 18:35:33 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3706916A4CE; Mon, 14 Jun 2004 18:35:33 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id BB54D43D31; Mon, 14 Jun 2004 18:35:32 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 20:33:54 +0200 Message-ID: <028001c4523e$202da9d0$6401a8c0@animal> From: "Lasse Laursen" To: "Julian Elischer" References: Date: Mon, 14 Jun 2004 20:33:14 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 18:35:33 -0000 Hi, Please find answers below: Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Julian Elischer" To: "Lasse Laursen" Cc: ; Sent: Monday, June 14, 2004 8:15 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > On Mon, 14 Jun 2004, Lasse Laursen wrote: > > > > > MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be > > broken) and WITH_PROC_SCOPE_PTH set to yes. > > > > WE use the SCHED_ULE, and: > > > > # Memory options > > options MAXDSIZ="(1536*1024*1024)" > > options MAXSSIZ="(1024*1024*1024)" > > options DFLDSIZ="(1536*1024*1024)" > > > > The server runs fine until a single thread/query suddenly locks up the > > entire MySQL daemon. After that all queries are just queued and a restart of > > the daemon is needed to unlock the system. The system itself is stable > > enough as far as I can see. 'top' reports a lot of locks (*Giant) so I > > assume that it's some weird problem with the threading? We used to use > > FreeBSD on non-SMP machines without any problems. > > > > I have tried with linux threads as well but the same problem occurs. > > This does suggest a bug in MySQL, but let's investigate anyhow.. We have used FreeBSD on non-smp machines for a long time (even the 5.x branch) without any problems. However on our new servers that are SMP the problems occur. It's the exact same version of MySQL that runs on the servers. > the output of "ps -alxH might be useful. I will post that the next time the machine locks up. :) > also: > do you have teh kernel debugger compiled into the kernel? > (option DDB) > > if so.. > note the PID of the process > sysctl debug.enter_debugger=ddb > [you are now in the debugger] > ps > [note the Thread address of the threads that seem to be blocking things] > (I may ask for specific threads after seeing the 'ps' output) > do > > show thread {address} > > to get a stack backtrace of those threads. > > if possible use a serial console to do all this.. That way you can log > it all. > > to make a serial console, connect 2 machines com1 by a null-modem cable. > > on the other machine do: > script > tip com1 > > on the SQL machine add > console="comconsole" > into /boot/loader.conf and reboot > > all console out put from teh boot-loader-on will now go to teh serial > port and appear in teh 'tip' output. (which is being logged by the > 'script' command). > > julian I will look into that as well - thanks for the promte reply. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 18:58:20 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75AC516A4CE; Mon, 14 Jun 2004 18:58:20 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2510543D54; Mon, 14 Jun 2004 18:58:20 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5EIwDqb009277; Mon, 14 Jun 2004 14:58:13 -0400 (EDT) Date: Mon, 14 Jun 2004 14:58:13 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Lasse Laursen In-Reply-To: <01c301c45239$940781f0$6401a8c0@animal> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 18:58:20 -0000 On Mon, 14 Jun 2004, Lasse Laursen wrote: > Hi, > > I posted this on the 2freebsq-questions" list some time ago but didn't get > any response. Maybe someone on this list can help: > > We are running MySQL 4.0.20 on FreeBSD 5.2-CURRENT. The machine is a dual > Xeon with 4GByte of memory with HTT enabled. > > Some info about the system: > > FreeBSD dbnode3 5.2-CURRENT FreeBSD 5.2-CURRENT #0: Tue Jun 8 14:41:02 CEST > 2004 laursen@dbnode3:/usr/obj/usr/src/sys/DBNODE3 i386 > > the /etc/libmap.conf contains: > > libc_r.so.5 libpthread.so.1 > libc_r.so libpthread.so > > /etc/make.conf contains: > > # -- use.perl generated deltas -- # > # Created: Wed Jun 9 10:18:52 2004 > # Setting to use base perl from ports: > PERL_VER=5.8.4 > PERL_VERSION=5.8.4 > PERL_ARCH=mach > NOPERL=yo > NO_PERL=yo > NO_PERL_WRAPPER=yo > > MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be > broken) and WITH_PROC_SCOPE_PTH set to yes. > > WE use the SCHED_ULE, and: > > # Memory options > options MAXDSIZ="(1536*1024*1024)" > options MAXSSIZ="(1024*1024*1024)" > options DFLDSIZ="(1536*1024*1024)" Try using SCHED_4BSD, then try removing all memory options. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 19:03:02 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 07E0A16A4D0; Mon, 14 Jun 2004 19:03:02 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id 69E7D43D53; Mon, 14 Jun 2004 19:03:01 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 21:02:38 +0200 Message-ID: <02c601c45242$240df470$6401a8c0@animal> From: "Lasse Laursen" To: "Daniel Eischen" References: Date: Mon, 14 Jun 2004 21:01:58 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 19:03:02 -0000 Hi, Please find answers below: Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Daniel Eischen" To: "Lasse Laursen" Cc: ; Sent: Monday, June 14, 2004 8:58 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > WE use the SCHED_ULE, and: > > > > # Memory options > > options MAXDSIZ="(1536*1024*1024)" > > options MAXSSIZ="(1024*1024*1024)" > > options DFLDSIZ="(1536*1024*1024)" > > Try using SCHED_4BSD, then try removing all memory options. Wouldn't this limit us to 512 MB of memory per process running on the server? we tried with each scheduler with same results. I will try to recompile without the memory options and use the old scheduler. Any hints why this might work? From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 19:10:02 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4ECF16A4CE; Mon, 14 Jun 2004 19:10:02 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7446843D1D; Mon, 14 Jun 2004 19:10:02 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5EJA1qb011698; Mon, 14 Jun 2004 15:10:01 -0400 (EDT) Date: Mon, 14 Jun 2004 15:10:01 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Lasse Laursen In-Reply-To: <02c601c45242$240df470$6401a8c0@animal> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 19:10:03 -0000 On Mon, 14 Jun 2004, Lasse Laursen wrote: > > > WE use the SCHED_ULE, and: > > > > > > # Memory options > > > options MAXDSIZ="(1536*1024*1024)" > > > options MAXSSIZ="(1024*1024*1024)" > > > options DFLDSIZ="(1536*1024*1024)" > > > > Try using SCHED_4BSD, then try removing all memory options. > > Wouldn't this limit us to 512 MB of memory per process running on the > server? we tried with each scheduler with same results. I will try to > recompile without the memory options and use the old scheduler. Any hints > why this might work? No, just trying to see what happens with a more stock kernel. Also, try removing /etc/my.conf to use default MySQL settings. -- Dan From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 19:14:53 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8A0DC16A4CE; Mon, 14 Jun 2004 19:14:53 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4AC6D43D62; Mon, 14 Jun 2004 19:14:53 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 21:14:50 +0200 Message-ID: <02da01c45243$d86439b0$6401a8c0@animal> From: "Lasse Laursen" To: "Daniel Eischen" References: Date: Mon, 14 Jun 2004 21:14:11 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 19:14:53 -0000 Hi, I will try that. Would it be better to use 4.10 and MySQL w. LinuxThreads instead? It's production databases that we run on the machine so stability is a must (yes I know - 5.2 isn't stable yet...) The reason why we decided to use 5.2-CURRENT was that the improved SMP support - which seems to still have some bugs? Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Daniel Eischen" To: "Lasse Laursen" Cc: ; Sent: Monday, June 14, 2004 9:10 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > On Mon, 14 Jun 2004, Lasse Laursen wrote: > > > > > WE use the SCHED_ULE, and: > > > > > > > > # Memory options > > > > options MAXDSIZ="(1536*1024*1024)" > > > > options MAXSSIZ="(1024*1024*1024)" > > > > options DFLDSIZ="(1536*1024*1024)" > > > > > > Try using SCHED_4BSD, then try removing all memory options. > > > > Wouldn't this limit us to 512 MB of memory per process running on the > > server? we tried with each scheduler with same results. I will try to > > recompile without the memory options and use the old scheduler. Any hints > > why this might work? > > No, just trying to see what happens with a more stock > kernel. Also, try removing /etc/my.conf to use default > MySQL settings. > > -- > Dan > > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 19:44:17 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4B1C016A4CE; Mon, 14 Jun 2004 19:44:17 +0000 (GMT) Received: from mail.mikehost.net (lvs-1.voxel.net [207.99.115.72]) by mx1.FreeBSD.org (Postfix) with ESMTP id CA97043D5C; Mon, 14 Jun 2004 19:44:14 +0000 (GMT) (envelope-from mike@mike2k.com) Received: by mail.mikehost.net (Postfix, from userid 502) id 104EC1C139; Mon, 14 Jun 2004 11:20:40 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.mikehost.net (Postfix) with ESMTP id 0A0041C0F0; Mon, 14 Jun 2004 11:20:40 -0700 (PDT) Date: Mon, 14 Jun 2004 11:20:39 -0700 (PDT) From: mike X-X-Sender: mike@sql01.internal.mikehost.net To: Lasse Laursen In-Reply-To: <01c301c45239$940781f0$6401a8c0@animal> Message-ID: References: <01c301c45239$940781f0$6401a8c0@animal> mike: yes MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=Windows-1252 Content-Transfer-Encoding: QUOTED-PRINTABLE cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 19:44:17 -0000 welcome to our hell. we've been experiencing mysql problems on freebsd 5.x as well. it sounds like scheduler/threading is to blame but we were not able to give sufficient or proper motivation to the folks who could examine this deeper - we even offered $500 cash to whomever stepped up to help resolve this. linux runs almost 2x as fast on the same hardware with no configuring - and we get nearly the same results running in single CPU mode vs. dual CPU mode on fbsd... something is definately fubar with the mysql+fbsd5.x combination. On Mon, 14 Jun 2004, Lasse Laursen wrote: > Hi, > > I posted this on the 2freebsq-questions" list some time ago but didn't ge= t > any response. Maybe someone on this list can help: > > We are running MySQL 4.0.20 on FreeBSD 5.2-CURRENT. The machine is a dual > Xeon with 4GByte of memory with HTT enabled. > > Some info about the system: > > FreeBSD dbnode3 5.2-CURRENT FreeBSD 5.2-CURRENT #0: Tue Jun 8 14:41:02 C= EST > 2004 laursen@dbnode3:/usr/obj/usr/src/sys/DBNODE3 i386 > > the /etc/libmap.conf contains: > > libc_r.so.5 libpthread.so.1 > libc_r.so libpthread.so > > /etc/make.conf contains: > > # -- use.perl generated deltas -- # > # Created: Wed Jun 9 10:18:52 2004 > # Setting to use base perl from ports: > PERL_VER=3D5.8.4 > PERL_VERSION=3D5.8.4 > PERL_ARCH=3Dmach > NOPERL=3Dyo > NO_PERL=3Dyo > NO_PERL_WRAPPER=3Dyo > > MySQL 4.0.20 was compiled with --without-libwrap (libwrapper seems to be > broken) and WITH_PROC_SCOPE_PTH set to yes. > > WE use the SCHED_ULE, and: > > # Memory options > options MAXDSIZ=3D"(1536*1024*1024)" > options MAXSSIZ=3D"(1024*1024*1024)" > options DFLDSIZ=3D"(1536*1024*1024)" > > The server runs fine until a single thread/query suddenly locks up the > entire MySQL daemon. After that all queries are just queued and a restart= of > the daemon is needed to unlock the system. The system itself is stable > enough as far as I can see. 'top' reports a lot of locks (*Giant) so I > assume that it's some weird problem with the threading? We used to use > FreeBSD on non-SMP machines without any problems. > > I have tried with linux threads as well but the same problem occurs. > > Have anyone experienced any similar problems and found a solution to this > rather weird problem? > > Feel free to request any further information about the system/setup and I > will do my best to provide it to the list. > > Regards > -- > Lasse Laursen =B7 VP, Hosting Technology =B7 NetGroup Processing Aps > St. Kongensgade 40H =B7 DK-1264 Copenhagen K, Denmark > Phone: +45 3370 1526 =B7 Fax: +45 3313 0066 > > - Don't be fooled by cheap finnish imitations - BSD is the One True Code > > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org= " > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 20:12:02 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E213F16A4CE; Mon, 14 Jun 2004 20:12:02 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6DDE943D2D; Mon, 14 Jun 2004 20:12:02 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5EKC1qb024559; Mon, 14 Jun 2004 16:12:01 -0400 (EDT) Date: Mon, 14 Jun 2004 16:12:01 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: mike In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 20:12:03 -0000 On Mon, 14 Jun 2004, mike wrote: > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > as well. it sounds like scheduler/threading is to blame but we were not > able to give sufficient or proper motivation to the folks who could > examine this deeper - we even offered $500 cash to whomever stepped up to > help resolve this. > > linux runs almost 2x as fast on the same hardware with no configuring - > and we get nearly the same results running in single CPU mode vs. dual CPU > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > combination. The problem is not the same. Lasse is getting lockups even with Linuxthreads. You were able to run with all threading libraries, but native linux was faster. -- DE From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 20:38:18 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1DBC116A4CE; Mon, 14 Jun 2004 20:38:18 +0000 (GMT) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id BDAA543D2D; Mon, 14 Jun 2004 20:38:16 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h86.vuokselantie10.fi [193.64.42.134]) by rms04.rommon.net (8.12.10/8.12.9) with ESMTP id i5EKbm3v032279; Mon, 14 Jun 2004 23:37:48 +0300 (EEST) (envelope-from pete@he.iki.fi) Message-ID: <40CE0C9B.1080305@he.iki.fi> Date: Mon, 14 Jun 2004 23:37:47 +0300 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: mike References: <01c301c45239$940781f0$6401a8c0@animal> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 20:38:18 -0000 mike wrote: >linux runs almost 2x as fast on the same hardware with no configuring - >and we get nearly the same results running in single CPU mode vs. dual CPU >mode on fbsd... something is definately fubar with the mysql+fbsd5.x >combination. > > > mysql runs fine on freebsd 5.x both with linuxthreads and pthreads (on -CURRENT). 5.2.1 release was broken though. This is on i386. Pete From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 20:43:27 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EEF8816A4CE; Mon, 14 Jun 2004 20:43:27 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id A75C543D2D; Mon, 14 Jun 2004 20:43:27 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 22:41:34 +0200 Message-ID: <03db01c4524f$f60d5760$6401a8c0@animal> From: "Lasse Laursen" To: "Petri Helenius" , "mike" References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> Date: Mon, 14 Jun 2004 22:40:50 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 20:43:28 -0000 Hi, Also on a SMP machine? I think that the problem is somehow related to SMP machines since the problems started when we moved to a SMP box :( Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Petri Helenius" To: "mike" Cc: "Lasse Laursen" ; ; Sent: Monday, June 14, 2004 10:37 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > mike wrote: > > >linux runs almost 2x as fast on the same hardware with no configuring - > >and we get nearly the same results running in single CPU mode vs. dual CPU > >mode on fbsd... something is definately fubar with the mysql+fbsd5.x > >combination. > > > > > > > mysql runs fine on freebsd 5.x both with linuxthreads and pthreads (on > -CURRENT). 5.2.1 release was broken though. > This is on i386. > > Pete > > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 20:47:57 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 15D0D16A4CE; Mon, 14 Jun 2004 20:47:57 +0000 (GMT) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4DBC343D53; Mon, 14 Jun 2004 20:47:56 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h86.vuokselantie10.fi [193.64.42.134]) by rms04.rommon.net (8.12.10/8.12.9) with ESMTP id i5EKkn3v032330; Mon, 14 Jun 2004 23:46:49 +0300 (EEST) (envelope-from pete@he.iki.fi) Message-ID: <40CE0EB8.4000303@he.iki.fi> Date: Mon, 14 Jun 2004 23:46:48 +0300 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Lasse Laursen References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> In-Reply-To: <03db01c4524f$f60d5760$6401a8c0@animal> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 20:47:57 -0000 Lasse Laursen wrote: >Hi, > >Also on a SMP machine? I think that the problem is somehow related to SMP >machines since the problems started when we moved to a SMP box :( > > > With both "classic" dual CPU, dual Xeons with HT enabled or disabled. But Iīve seen motherboards where SMP is not an option although they have two CPU sockets :-) BTW, do you have PAE enabled? We donīt. Pete From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 20:52:12 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0289816A4CE; Mon, 14 Jun 2004 20:52:12 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id B68C143D53; Mon, 14 Jun 2004 20:52:11 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 22:51:21 +0200 Message-ID: <03fb01c45251$541fdf20$6401a8c0@animal> From: "Lasse Laursen" To: "Petri Helenius" References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> <40CE0EB8.4000303@he.iki.fi> Date: Mon, 14 Jun 2004 22:50:41 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 20:52:12 -0000 Hi, No. PAE isn't enabled. I have copied the config file for the kernel to a webserver: http://solidcore.dk/DBNODE3 Feel free to comment the settings. The machine is an IBM-x-something, dual Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid 10. Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Petri Helenius" To: "Lasse Laursen" Cc: "mike" ; ; Sent: Monday, June 14, 2004 10:46 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > >Also on a SMP machine? I think that the problem is somehow related to SMP > >machines since the problems started when we moved to a SMP box :( > > > > > > > With both "classic" dual CPU, dual Xeons with HT enabled or disabled. > > But Iīve seen motherboards where SMP is not an option although they have > two CPU sockets :-) > > BTW, do you have PAE enabled? We donīt. From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 21:04:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C0DC16A600; Mon, 14 Jun 2004 21:04:48 +0000 (GMT) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6CA0F43D5F; Mon, 14 Jun 2004 21:04:47 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h86.vuokselantie10.fi [193.64.42.134]) by rms04.rommon.net (8.12.10/8.12.9) with ESMTP id i5EL4Y3v032412; Tue, 15 Jun 2004 00:04:34 +0300 (EEST) (envelope-from pete@he.iki.fi) Message-ID: <40CE12E1.4020108@he.iki.fi> Date: Tue, 15 Jun 2004 00:04:33 +0300 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Lasse Laursen References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> <40CE0EB8.4000303@he.iki.fi> <03fb01c45251$541fdf20$6401a8c0@animal> In-Reply-To: <03fb01c45251$541fdf20$6401a8c0@animal> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 21:04:48 -0000 Lasse Laursen wrote: >Hi, > >No. PAE isn't enabled. I have copied the config file for the kernel to a >webserver: > >http://solidcore.dk/DBNODE3 > >Feel free to comment the settings. The machine is an IBM-x-something, dual >Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid >10. > > > My suspect (without proof) would be SCHED_ULE. Not sure also how 4G works out. We have 1-3G on the boxes. Pete From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 21:08:22 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6953C16A4CE; Mon, 14 Jun 2004 21:08:22 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1CBE243D4C; Mon, 14 Jun 2004 21:08:22 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 23:08:19 +0200 Message-ID: <042d01c45253$b2dc4600$6401a8c0@animal> From: "Lasse Laursen" To: "Petri Helenius" References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> <40CE0EB8.4000303@he.iki.fi> <03fb01c45251$541fdf20$6401a8c0@animal> <40CE12E1.4020108@he.iki.fi> Date: Mon, 14 Jun 2004 23:07:39 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 21:08:22 -0000 Hi, I tried with both schedulers with the same result :( I think that I have tried every possible combination in order to track down the problem ... Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Petri Helenius" To: "Lasse Laursen" Cc: "mike" ; ; Sent: Monday, June 14, 2004 11:04 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > Lasse Laursen wrote: > > >Hi, > > > >No. PAE isn't enabled. I have copied the config file for the kernel to a > >webserver: > > > >http://solidcore.dk/DBNODE3 > > > >Feel free to comment the settings. The machine is an IBM-x-something, dual > >Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid > >10. > > > > > > > My suspect (without proof) would be SCHED_ULE. > > Not sure also how 4G works out. We have 1-3G on the boxes. > > Pete > > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 21:12:08 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E774D16A4CF; Mon, 14 Jun 2004 21:12:08 +0000 (GMT) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id E7E7743D2F; Mon, 14 Jun 2004 21:12:07 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h86.vuokselantie10.fi [193.64.42.134]) by rms04.rommon.net (8.12.10/8.12.9) with ESMTP id i5ELBl3v032445; Tue, 15 Jun 2004 00:11:47 +0300 (EEST) (envelope-from pete@he.iki.fi) Message-ID: <40CE1492.1050704@he.iki.fi> Date: Tue, 15 Jun 2004 00:11:46 +0300 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Lasse Laursen References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> <40CE0EB8.4000303@he.iki.fi> <03fb01c45251$541fdf20$6401a8c0@animal> <40CE12E1.4020108@he.iki.fi> <042d01c45253$b2dc4600$6401a8c0@animal> In-Reply-To: <042d01c45253$b2dc4600$6401a8c0@animal> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 21:12:09 -0000 Lasse Laursen wrote: >Hi, > >I tried with both schedulers with the same result :( I think that I have >tried every possible combination in order to track down the problem ... > > > I would take out one DIMM to reduce the memory and if that would not help, swap the hardware. Obviously the earlier questions about where it does hang are valid. Pete >- Don't be fooled by cheap finnish imitations - BSD is the One True Code > > P.S. funny sig :) From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 21:16:20 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6F33E16A4CE; Mon, 14 Jun 2004 21:16:20 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A04B43D5A; Mon, 14 Jun 2004 21:16:20 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Mon, 14 Jun 2004 23:15:08 +0200 Message-ID: <043d01c45254$a6720fc0$6401a8c0@animal> From: "Lasse Laursen" To: "Petri Helenius" References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> <40CE0EB8.4000303@he.iki.fi> <03fb01c45251$541fdf20$6401a8c0@animal> <40CE12E1.4020108@he.iki.fi> <042d01c45253$b2dc4600$6401a8c0@animal> <40CE1492.1050704@he.iki.fi> Date: Mon, 14 Jun 2004 23:14:28 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 21:16:20 -0000 I find it hard to believe that it's the RAM amount that can cause it - also the current RAM setup is some exotic IBM configuration with some nice features (according to their product page) - also the disk controller is too expensive to replace :( The sig: We had some really nasty problems with Linux in the early days (2.0.14) ... :( Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Petri Helenius" To: "Lasse Laursen" Cc: "mike" ; ; Sent: Monday, June 14, 2004 11:11 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > Lasse Laursen wrote: > > >Hi, > > > >I tried with both schedulers with the same result :( I think that I have > >tried every possible combination in order to track down the problem ... > > > > > > > I would take out one DIMM to reduce the memory and if that would not > help, swap the hardware. > > Obviously the earlier questions about where it does hang are valid. > > Pete > > >- Don't be fooled by cheap finnish imitations - BSD is the One True Code > > > > > P.S. funny sig :) > > From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 21:44:23 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C05E16A4CE; Mon, 14 Jun 2004 21:44:23 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9341C43D49; Mon, 14 Jun 2004 21:44:22 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5ELh5Lk043235; Mon, 14 Jun 2004 17:43:05 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5ELh5nZ043231; Mon, 14 Jun 2004 17:43:05 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Mon, 14 Jun 2004 17:43:05 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Lasse Laursen In-Reply-To: <03db01c4524f$f60d5760$6401a8c0@animal> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 21:44:23 -0000 On Mon, 14 Jun 2004, Lasse Laursen wrote: > Also on a SMP machine? I think that the problem is somehow related to > SMP machines since the problems started when we moved to a SMP box :( If you haven't already, it would certainly be worth removing SMP from the kernel on that box and just running a UP kernel to confirm that it's a problem using SMP. I don't doubt it's the case, but it's worth checking anyway to be sure. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research >=20 > Regards > -- > Lasse Laursen =B7 VP, Hosting Technology =B7 NetGroup Processing Aps > St. Kongensgade 40H =B7 DK-1264 Copenhagen K, Denmark > Phone: +45 3370 1526 =B7 Fax: +45 3313 0066 >=20 > - Don't be fooled by cheap finnish imitations - BSD is the One True Code > ----- Original Message -----=20 > From: "Petri Helenius" > To: "mike" > Cc: "Lasse Laursen" ; ; > > Sent: Monday, June 14, 2004 10:37 PM > Subject: Re: Possible Threading problem with -CURRENT / MySQL? >=20 >=20 > > mike wrote: > > > > >linux runs almost 2x as fast on the same hardware with no configuring = - > > >and we get nearly the same results running in single CPU mode vs. dual > CPU > > >mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > >combination. > > > > > > > > > > > mysql runs fine on freebsd 5.x both with linuxthreads and pthreads (on > > -CURRENT). 5.2.1 release was broken though. > > This is on i386. > > > > Pete > > > > >=20 >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " >=20 From owner-freebsd-threads@FreeBSD.ORG Mon Jun 14 22:30:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0EE4716A4CE; Mon, 14 Jun 2004 22:30:48 +0000 (GMT) Received: from mail.mikehost.net (lvs-1.voxel.net [207.99.115.72]) by mx1.FreeBSD.org (Postfix) with ESMTP id D7BA643D2D; Mon, 14 Jun 2004 22:30:47 +0000 (GMT) (envelope-from mike@mike2k.com) Received: by mail.mikehost.net (Postfix, from userid 502) id 4C2FB1C221; Mon, 14 Jun 2004 15:13:18 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.mikehost.net (Postfix) with ESMTP id 4A9F31C11C; Mon, 14 Jun 2004 15:13:18 -0700 (PDT) Date: Mon, 14 Jun 2004 15:13:18 -0700 (PDT) From: mike X-X-Sender: mike@sql01.internal.mikehost.net To: Petri Helenius In-Reply-To: <40CE0C9B.1080305@he.iki.fi> Message-ID: References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> mike: yes MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jun 2004 22:30:48 -0000 there's a difference between running "fine" and actually performing to the hardware's specifications. if you tell me 5.2.1 was broken, fine, but i believe JG tried some other versions and still came out with the same results. this was on AMD64 but it sounds like all platforms are having the problems - it's just that there is no linuxthreads port for AMD64 so we could not test with that (however he did install i386 freebsd and tried linuxthreads and there were no significant performance gains) On Mon, 14 Jun 2004, Petri Helenius wrote: > mike wrote: > > >linux runs almost 2x as fast on the same hardware with no configuring - > >and we get nearly the same results running in single CPU mode vs. dual CPU > >mode on fbsd... something is definately fubar with the mysql+fbsd5.x > >combination. > > > > > > > mysql runs fine on freebsd 5.x both with linuxthreads and pthreads (on > -CURRENT). 5.2.1 release was broken though. > This is on i386. > > Pete > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 01:37:08 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F023316A4CE for ; Tue, 15 Jun 2004 01:37:07 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id E31D443D48 for ; Tue, 15 Jun 2004 01:37:07 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 658 invoked by uid 89); 15 Jun 2004 01:36:31 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 15 Jun 2004 01:36:31 -0000 Message-Id: <5.2.0.9.2.20040614183523.015cdaa0@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Mon, 14 Jun 2004 18:36:45 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: References: <40CE0C9B.1080305@he.iki.fi> <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 01:37:08 -0000 I just posted this to -AMD64 and someone brought this discussion to my attention... anyway I'll repost it here since it seems to be a more relevant discussion: I got a couple emails from people telling me I should retest because some changes have been committed recently that might effect my MySQL benchmarks. Well I just did a buildworld and kernel to -CURRENT & it did effect the results alright... just not the way we wanted them to: amd64f# super-smack update-select.smack 30 10000 Query Barrel Report for client smacker connect: max=46ms min=2ms avg= 26ms from 30 clients Query_type num_queries max_time min_time q_per_s select_index 300000 2 0 958.05 update_index 300000 2 0 958.05 In all my previous tests I think that number was at least 2000/qps (and that was bad) and as much as 4000/qps. Whatever was changed recently really killed performance in this area. This is a SMP kerenel using SCHED_ULE. .. debugging is turned off in kernel of course. Whatever was changed is producing very erratic benchmark results from bad to worse.... here is an example: (keep in mind this box is for testing only, nothing else is going on on this box) I ran some short tests below (notice 100 below vs 10000 above) amd64f# super-smack update-select.smack 30 100 Query Barrel Report for client smacker connect: max=24ms min=2ms avg= 10ms from 30 clients Query_type num_queries max_time min_time q_per_s select_index 3000 1119 0 60.64 update_index 3000 2185 0 60.64 amd64f# super-smack update-select.smack 30 100 Query Barrel Report for client smacker connect: max=292ms min=1ms avg= 54ms from 30 clients Query_type num_queries max_time min_time q_per_s select_index 3000 4 0 3167.06 update_index 3000 1 0 3167.06 amd64f# super-smack update-select.smack 30 100 Query Barrel Report for client smacker connect: max=26ms min=3ms avg= 10ms from 30 clients Query_type num_queries max_time min_time q_per_s select_index 3000 1891 0 32.73 update_index 3000 1891 0 32.73 amd64f# amd64f# super-smack update-select.smack 30 100 Query Barrel Report for client smacker connect: max=37ms min=3ms avg= 10ms from 30 clients Query_type num_queries max_time min_time q_per_s select_index 3000 10 0 757.05 update_index 3000 11 0 757.05 amd64f# super-smack update-select.smack 30 100 Query Barrel Report for client smacker connect: max=39ms min=5ms avg= 15ms from 30 clients Query_type num_queries max_time min_time q_per_s select_index 3000 11 0 2988.89 update_index 3000 2 0 2988.89 Those were ran one after the other with nothing changed in between. Just thought I would give you guys a heads up. From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 05:28:24 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F142616A4CE; Tue, 15 Jun 2004 05:28:24 +0000 (GMT) Received: from mx1.imp.ch (mx1.imp.ch [157.161.9.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id D6F8B43D5A; Tue, 15 Jun 2004 05:28:23 +0000 (GMT) (envelope-from mb@imp.ch) Received: from mx2.imp.ch (mx2 [157.161.9.17]) by mx1.imp.ch (8.12.11/8.12.11) with ESMTP id i5F5SFw7095672; Tue, 15 Jun 2004 07:28:16 +0200 (CEST) (envelope-from mb@imp.ch) Received: from mx2.imp.ch (localhost [127.0.0.1]) by mx2.imp.ch (8.12.11/8.12.11/Submit) with ESMTP id i5F5SDN1014105; Tue, 15 Jun 2004 07:28:14 +0200 (CEST) (envelope-from mb@imp.ch) Received: (from clamav@localhost) by mx2.imp.ch (8.12.11/8.12.11/Submit) id i5F5SD5s014101; Tue, 15 Jun 2004 07:28:13 +0200 (CEST) (envelope-from mb@imp.ch) Received: from cvs.imp.ch (cvs.imp.ch [157.161.4.9]) by ns1.imp.ch (MIMEDefang) with ESMTP id i5F5S8CH080912; Tue, 15 Jun 2004 07:28:13 +0200 (CEST) Date: Tue, 15 Jun 2004 07:28:08 +0200 (CEST) From: Martin Blapp To: Lasse Laursen In-Reply-To: <03fb01c45251$541fdf20$6401a8c0@animal> Message-ID: <20040615072441.F47987@cvs.imp.ch> References: <01c301c45239$940781f0$6401a8c0@animal> <40CE0C9B.1080305@he.iki.fi> <03db01c4524f$f60d5760$6401a8c0@animal> <40CE0EB8.4000303@he.iki.fi> <03fb01c45251$541fdf20$6401a8c0@animal> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Spam-Checksum: 75132a9075685370b313d620a64e6cbf X-Virus-Status: No X-Spam-Status: No, hits=-4.9 required=4 tests=BAYES_00 X-Scanned-By: MIMEDefang 2.42 cc: mike cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 05:28:25 -0000 Hi, > Feel free to comment the settings. The machine is an IBM-x-something, dual > Xeon with HTT enabled. 4 GByte of memory and a ServeRAID controller w. raid > 10. We have the same config as you have. IBM X-series 345, FreeBSD Current and have no locking problems with mysql on a very busy SMP server. With HTT enabled, we encountered the same problems you had, and some severe crashes additionally from time to time. Please turn off HTT and try again.. It has been told to us that the chipset (serverworks) has buggy HTT support. Martin From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 11:07:08 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCF0616A549; Tue, 15 Jun 2004 11:07:08 +0000 (GMT) Received: from angel.grumly.eu.org (angel.grumly.eu.org [193.138.90.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60E3F43D64; Tue, 15 Jun 2004 11:07:08 +0000 (GMT) (envelope-from ced@grumly.eu.org) Received: by angel.grumly.eu.org (Postfix, from userid 1001) id 7B9B9FDAA; Tue, 15 Jun 2004 13:06:13 +0200 (CEST) Date: Tue, 15 Jun 2004 13:06:13 +0200 From: Cedric Tabary To: freebsd-current@freebsd.org Message-ID: <20040615110613.GB81104@efrei.fr> References: <03db01c4524f$f60d5760$6401a8c0@animal> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.1i X-URL: http://tabary.eu.org/ cc: mike cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 11:07:08 -0000 On 14/06/2004 17:43, Robert Watson wrote: > On Mon, 14 Jun 2004, Lasse Laursen wrote: > > > Also on a SMP machine? I think that the problem is somehow related to > > SMP machines since the problems started when we moved to a SMP box :( > > If you haven't already, it would certainly be worth removing SMP from the > kernel on that box and just running a UP kernel to confirm that it's a > problem using SMP. I don't doubt it's the case, but it's worth checking > anyway to be sure. I have a similar smp config (dell poweredge 2650, dual Xeon 2.8, RAID5) I have best performance with linux 2.4.21-smp (not tried 2.6), but very poor (5x less) with mysqld using libpthreads on CURRENT. mysqld process is always in state 'kserel' or sometimes '*Giant' http://grumly.eu.org/~ced/dmesg.txt http://grumly.eu.org/~ced/CED-SMP.txt (no invariants, no witness) mysqld with linuxthreads seems to work better ... and even better with libc_r (but using only 1 cpu) Everything with HTT disabled in bios. Enabling HTT gives even worst performance :/ -- Cédric Tabary From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 11:23:28 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 29DC216A4CE; Tue, 15 Jun 2004 11:23:28 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id B365443D46; Tue, 15 Jun 2004 11:23:27 +0000 (GMT) (envelope-from laursen@netgroup.dk) Message-ID: <045c01c452cb$16f4b0b0$ce01000a@animal> From: "Lasse Laursen" To: "Cedric Tabary" , freebsd-current@freebsd.org References: <03db01c4524f$f60d5760$6401a8c0@animal> <20040615110613.GB81104@efrei.fr> Date: Tue, 15 Jun 2004 13:12:18 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: mike cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 11:23:28 -0000 Hi, Have you had any lockups in MySQL with your setup? We have tried to disable HTT and hopefully that will solve the problem. Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Cedric Tabary" To: Cc: "Lasse Laursen" ; ; "mike" ; "Petri Helenius" Sent: Tuesday, June 15, 2004 1:06 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > On 14/06/2004 17:43, Robert Watson wrote: > > On Mon, 14 Jun 2004, Lasse Laursen wrote: > > > > > Also on a SMP machine? I think that the problem is somehow related to > > > SMP machines since the problems started when we moved to a SMP box :( > > > > If you haven't already, it would certainly be worth removing SMP from the > > kernel on that box and just running a UP kernel to confirm that it's a > > problem using SMP. I don't doubt it's the case, but it's worth checking > > anyway to be sure. > > I have a similar smp config (dell poweredge 2650, dual Xeon 2.8, RAID5) > > I have best performance with linux 2.4.21-smp (not tried 2.6), > but very poor (5x less) with mysqld using libpthreads on CURRENT. > mysqld process is always in state 'kserel' or sometimes '*Giant' > > http://grumly.eu.org/~ced/dmesg.txt > http://grumly.eu.org/~ced/CED-SMP.txt (no invariants, no witness) > > mysqld with linuxthreads seems to work better ... > and even better with libc_r (but using only 1 cpu) > > Everything with HTT disabled in bios. Enabling HTT gives even worst > performance :/ From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 13:08:24 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7457F16A4CE; Tue, 15 Jun 2004 13:08:24 +0000 (GMT) Received: from angel.grumly.eu.org (angel.grumly.eu.org [193.138.90.69]) by mx1.FreeBSD.org (Postfix) with ESMTP id 088DC43D53; Tue, 15 Jun 2004 13:08:24 +0000 (GMT) (envelope-from ced@grumly.eu.org) Received: by angel.grumly.eu.org (Postfix, from userid 1001) id C472EFDAA; Tue, 15 Jun 2004 15:07:34 +0200 (CEST) Date: Tue, 15 Jun 2004 15:07:34 +0200 From: Cedric Tabary To: Lasse Laursen Message-ID: <20040615130734.GC81104@efrei.fr> References: <03db01c4524f$f60d5760$6401a8c0@animal> <20040615110613.GB81104@efrei.fr> <045c01c452cb$16f4b0b0$ce01000a@animal> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <045c01c452cb$16f4b0b0$ce01000a@animal> User-Agent: Mutt/1.4.2.1i X-URL: http://tabary.eu.org/ cc: freebsd-current@freebsd.org cc: mike cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 13:08:24 -0000 On 15/06/2004 13:12, Lasse Laursen wrote: > Hi, > > Have you had any lockups in MySQL with your setup? We have tried to disable > HTT and hopefully that will solve the problem. No lockups for now, but it is not heavily loaded. -- Cédric Tabary From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 13:33:50 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C38516A4CE; Tue, 15 Jun 2004 13:33:50 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id D727843D48; Tue, 15 Jun 2004 13:33:49 +0000 (GMT) (envelope-from laursen@netgroup.dk) Message-ID: <05c501c452dd$561037d0$ce01000a@animal> From: "Lasse Laursen" To: "Cedric Tabary" References: <03db01c4524f$f60d5760$6401a8c0@animal> <20040615110613.GB81104@efrei.fr> <045c01c452cb$16f4b0b0$ce01000a@animal> <20040615130734.GC81104@efrei.fr> Date: Tue, 15 Jun 2004 15:33:24 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-current@freebsd.org cc: mike cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 13:33:50 -0000 Hi, We just had our first lockup for today - and the last one. Next step is to downgrade to Linux :( *crud* Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Cedric Tabary" To: "Lasse Laursen" Cc: ; "mike" ; "Petri Helenius" ; Sent: Tuesday, June 15, 2004 3:07 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > On 15/06/2004 13:12, Lasse Laursen wrote: > > Hi, > > > > Have you had any lockups in MySQL with your setup? We have tried to disable > > HTT and hopefully that will solve the problem. > > No lockups for now, but it is not heavily loaded. > > -- > Cédric Tabary > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 13:39:14 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB69616A4CE; Tue, 15 Jun 2004 13:39:14 +0000 (GMT) Received: from mailout11.sul.t-online.com (mailout11.sul.t-online.com [194.25.134.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 88A2B43D5D; Tue, 15 Jun 2004 13:39:14 +0000 (GMT) (envelope-from Alexander@Leidinger.net) Received: from fwd06.aul.t-online.de by mailout11.sul.t-online.com with smtp id 1BaE9Y-0005dF-05; Tue, 15 Jun 2004 15:39:04 +0200 Received: from Andro-Beta.Leidinger.net (rC-XzUZaZelH0HbfL75bQmDDWl+tDxb9c7u5JVlx1LARz8BsIwvW8B@[84.128.201.82]) by fmrl06.sul.t-online.com with esmtp id 1BaE9B-22XnQO0; Tue, 15 Jun 2004 15:38:41 +0200 Received: from Magellan.Leidinger.net (Magellan.Leidinger.net [192.168.1.1]) i5FDcqdn073548; Tue, 15 Jun 2004 15:38:52 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Date: Tue, 15 Jun 2004 15:40:30 +0200 From: Alexander Leidinger To: "Lasse Laursen" Message-Id: <20040615154030.31669246@Magellan.Leidinger.net> In-Reply-To: <045c01c452cb$16f4b0b0$ce01000a@animal> References: <03db01c4524f$f60d5760$6401a8c0@animal> <20040615110613.GB81104@efrei.fr> <045c01c452cb$16f4b0b0$ce01000a@animal> X-Mailer: Sylpheed version 0.9.11claws (GTK+ 1.2.10; i386-portbld-freebsd5.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Seen: false X-ID: rC-XzUZaZelH0HbfL75bQmDDWl+tDxb9c7u5JVlx1LARz8BsIwvW8B@t-dialin.net cc: freebsd-threads@freebsd.org cc: Cedric Tabary cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 13:39:15 -0000 On Tue, 15 Jun 2004 13:12:18 +0200 "Lasse Laursen" wrote: > Hi, > > Have you had any lockups in MySQL with your setup? We have tried to disable > HTT and hopefully that will solve the problem. I'm not sure I've seen all replies, but I think I haven't seen a mention of the malloc flags so far. If you haven't changed the default malloc flags please do so and try again: ---snip--- % ll /etc/malloc.conf lrwx------ 1 root wheel 2B 18 Aug 2001 /etc/malloc.conf@ -> aj ---snip--- The 'a' isn't important here, but the 'j' affects the performance. Bye, Alexander. -- I'm available to get hired (preferred in .lu). http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 14:03:06 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 89C1516A4CE; Tue, 15 Jun 2004 14:03:06 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1892843D2D; Tue, 15 Jun 2004 14:03:06 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5FE2rqb014227; Tue, 15 Jun 2004 10:02:53 -0400 (EDT) Date: Tue, 15 Jun 2004 10:02:53 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Lasse Laursen In-Reply-To: <05c501c452dd$561037d0$ce01000a@animal> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: Cedric Tabary cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 14:03:06 -0000 On Tue, 15 Jun 2004, Lasse Laursen wrote: > Hi, > > We just had our first lockup for today - and the last one. Next step is to > downgrade to Linux :( *crud* We still haven't heard whether removing memory options from the kernel, disabling HTT, and removing /etc/my.conf have had any effect. -- DE From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 14:15:54 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D3E7816A4CE for ; Tue, 15 Jun 2004 14:15:54 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id B195043D2D for ; Tue, 15 Jun 2004 14:15:54 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 26587 invoked by uid 89); 15 Jun 2004 14:15:04 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 15 Jun 2004 14:15:04 -0000 Message-Id: <5.2.0.9.2.20040615071038.03bfdc18@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 07:15:14 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: References: <05c501c452dd$561037d0$ce01000a@animal> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 14:15:54 -0000 > >We still haven't heard whether removing memory options from >the kernel, disabling HTT, and removing /etc/my.conf have >had any effect. > >-- >DE Yes you have (though you may not have known)... look at my benchmark results posted yesterday. My results were without HTT (It was FreeBSD-AMD64) I didn't have any memory options or a mysql config file. _This problem is directly related to a commit that was made between ~May 18 and yesterday._ From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 15:10:37 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3681E16A4CF; Tue, 15 Jun 2004 15:10:37 +0000 (GMT) Received: from mx1.imp.ch (mx1.imp.ch [157.161.9.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3BE3843D64; Tue, 15 Jun 2004 15:10:35 +0000 (GMT) (envelope-from mb@imp.ch) Received: from mx2.imp.ch (mx2 [157.161.9.17]) by mx1.imp.ch (8.12.11/8.12.11) with ESMTP id i5FF9xDR011613; Tue, 15 Jun 2004 17:10:00 +0200 (CEST) (envelope-from mb@imp.ch) Received: from mx2.imp.ch (localhost [127.0.0.1]) by mx2.imp.ch (8.12.11/8.12.11/Submit) with ESMTP id i5FF9vpL053815; Tue, 15 Jun 2004 17:09:57 +0200 (CEST) (envelope-from mb@imp.ch) Received: (from clamav@localhost) by mx2.imp.ch (8.12.11/8.12.11/Submit) id i5FF9uvO053805; Tue, 15 Jun 2004 17:09:56 +0200 (CEST) (envelope-from mb@imp.ch) Received: from cvs.imp.ch (cvs.imp.ch [157.161.4.9]) by ns1.imp.ch (MIMEDefang) with ESMTP id i5FF9pCH031041; Tue, 15 Jun 2004 17:09:56 +0200 (CEST) Date: Tue, 15 Jun 2004 17:09:51 +0200 (CEST) From: Martin Blapp To: mike In-Reply-To: Message-ID: <20040615170316.T84297@cvs.imp.ch> References: <01c301c45239$940781f0$6401a8c0@animal> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Spam-Checksum: 3e629ff7caceec649e71d44aaed56da4 X-Virus-Status: No X-Spam-Status: No, hits=-4.899 required=4 tests=BAYES_00, UPPERCASE_25_50 X-Scanned-By: MIMEDefang 2.42 cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 15:10:37 -0000 Hi all, Just to list our working setup: (I don't know if mysql is very fast, but since I rewrote all mysql queries to use LEFT/RIGHT JOIN and UNION and redesigned the databases, the mysql load is not significant anymore) Has someone a test database and e benchmark tool ? - We use SCHED_4BSD. - HTT is disabled in the BIOS - WITNESS and INVARIANTS are turned off (important) ! uptime 5:07PM up 35 days, 3:52, 9 users, load averages: 2.27, 1.70, 1.54 No hangs and crashes or lockups. 38073 idms 20 -15 55160K 17200K kserel 0 26:45 0.00% 0.00% mysqld my.cnf config: [mysqld] skip-innodb set-variable = max_connections=1000 set-variable = key_buffer_size=16M set-variable = table_cache=256 set-variable = record_buffer=1M set-variable = sort_buffer=16M CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3059.98-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 Features=0xbfebfbff real memory = 2147397632 (2047 MB) avail memory = 2095964160 (1998 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 6 Hope this helps anybody: Martin From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 15:27:05 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1780C16A4CE for ; Tue, 15 Jun 2004 15:27:05 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8977E43D2F for ; Tue, 15 Jun 2004 15:27:04 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5FFQsqb029875; Tue, 15 Jun 2004 11:26:54 -0400 (EDT) Date: Tue, 15 Jun 2004 11:26:54 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: JG In-Reply-To: <5.2.0.9.2.20040615071038.03bfdc18@mail.ojoink.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 15:27:05 -0000 On Tue, 15 Jun 2004, JG wrote: > > > > >We still haven't heard whether removing memory options from > >the kernel, disabling HTT, and removing /etc/my.conf have > >had any effect. > > > >-- > >DE > > > Yes you have (though you may not have known)... look at my benchmark > results posted yesterday. This isn't to you. This is to the original poster who was having lockups. > My results were without HTT (It was FreeBSD-AMD64) I didn't have any memory > options > or a mysql config file. > > _This problem is directly related to a commit that was made between ~May 18 > and yesterday._ There was a subsystem lock(s) put in for networking a few days ago. There were no changes to the threads library that would exhibit the behavior you are seeing. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 16:27:01 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A56B716A4CE for ; Tue, 15 Jun 2004 16:27:01 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 82E2643D55 for ; Tue, 15 Jun 2004 16:27:01 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 30624 invoked by uid 89); 15 Jun 2004 16:25:57 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 15 Jun 2004 16:25:57 -0000 Message-Id: <5.2.0.9.2.20040615091911.01536010@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 09:26:06 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: References: <5.2.0.9.2.20040615071038.03bfdc18@mail.ojoink.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 16:27:01 -0000 > > > Yes you have (though you may not have known)... look at my benchmark > > results posted yesterday. > >This isn't to you. This is to the original poster who was having >lockups. Regardless, it suggests that the problem doesn't have to do his with HTT, my.cnf, or memory settings & has more to do with recent commits to the base since ~May 15. I am having these lockups as well (as reflected by the benchmarks). Robert Watson said he was going to try to look into this problem within the next few days. From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 16:52:26 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82F4216A4CE for ; Tue, 15 Jun 2004 16:52:26 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2C66F43D39 for ; Tue, 15 Jun 2004 16:52:26 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5FGpXqb016243; Tue, 15 Jun 2004 12:51:33 -0400 (EDT) Date: Tue, 15 Jun 2004 12:51:33 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: JG In-Reply-To: <5.2.0.9.2.20040615091911.01536010@mail.ojoink.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 16:52:26 -0000 On Tue, 15 Jun 2004, JG wrote: > > > > > > Yes you have (though you may not have known)... look at my benchmark > > > results posted yesterday. > > > >This isn't to you. This is to the original poster who was having > >lockups. > > Regardless, it suggests that the problem doesn't have to do his with HTT, > my.cnf, > or memory settings & has more to do with recent commits to the base since > ~May 15. > > I am having these lockups as well (as reflected by the benchmarks). His lockups are complete; yours are not. Also, others have mentioned that HTT causes problems. Please let him try various things instead of discouraging him. Any information he can give us is helpful. > Robert Watson said he was going to try to look into this problem within the > next few days. DE From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:00:53 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E93BA16A4CE for ; Tue, 15 Jun 2004 17:00:53 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id B1A0843D48 for ; Tue, 15 Jun 2004 17:00:53 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Tue, 15 Jun 2004 18:58:47 +0200 Message-ID: <07df01c452fa$0059c000$ce01000a@animal> From: "Lasse Laursen" To: "Daniel Eischen" , "JG" References: Date: Tue, 15 Jun 2004 18:57:58 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:00:54 -0000 Hi all, I'm still determined to track down the problem and I have tried most of the things suggested to me in your posts. I have disabled HTT and removed the memory options in the kernel config. It's "only" the mysqld that locks up - all new queries are simply queued and everything grinds to a hold. A simple restart of the daemon usually solves the problem (start - stop of the daemon works fine.) I will keep you all updated on my the progress of the system. :) Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Daniel Eischen" To: "JG" Cc: Sent: Tuesday, June 15, 2004 6:51 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > On Tue, 15 Jun 2004, JG wrote: > > > > > > > > > > Yes you have (though you may not have known)... look at my benchmark > > > > results posted yesterday. > > > > > >This isn't to you. This is to the original poster who was having > > >lockups. > > > > Regardless, it suggests that the problem doesn't have to do his with HTT, > > my.cnf, > > or memory settings & has more to do with recent commits to the base since > > ~May 15. > > > > I am having these lockups as well (as reflected by the benchmarks). > > His lockups are complete; yours are not. Also, others have mentioned > that HTT causes problems. Please let him try various things instead > of discouraging him. Any information he can give us is helpful. > > > Robert Watson said he was going to try to look into this problem within the > > next few days. > > DE > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:12:46 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8216116A4CF for ; Tue, 15 Jun 2004 17:12:46 +0000 (GMT) Received: from web14107.mail.yahoo.com (web14107.mail.yahoo.com [216.136.172.137]) by mx1.FreeBSD.org (Postfix) with SMTP id 6337043D68 for ; Tue, 15 Jun 2004 17:12:46 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Message-ID: <20040615171225.711.qmail@web14107.mail.yahoo.com> Received: from [194.248.174.50] by web14107.mail.yahoo.com via HTTP; Tue, 15 Jun 2004 19:12:25 CEST Date: Tue, 15 Jun 2004 19:12:25 +0200 (CEST) From: =?iso-8859-1?q?Claus=20Guttesen?= To: Daniel Eischen , Lasse Laursen In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit cc: freebsd-current@freebsd.org cc: Cedric Tabary cc: mike cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:12:46 -0000 > > We just had our first lockup for today - and the > last one. Next step is to > > downgrade to Linux :( *crud* > Maby not the greatest comfort, but I've had a Dell 2650 with the Perc3-controller working for weeks. I installed rt3 (troubleticket-system) which needed Mysql (or postgresql). I installed Mysql 4.0 and haven't had problems with it. I increased the amount of RAM than Mysql can utilize, but that's about it. All installed from /usr/ports using portinstall. FreeBSD 5.2.1, 2 GB RAM, dual Xeon @ 2.4 Ghz. Other than enabling -O2 and CPUTYPE=i686 in /etc/make.conf and make a customized kernel with amr in my kernel, everything else is pretty much default. I never touched HTT in BIOS nor using sysctl, that's working out of the box. /etc/make.conf: CPUTYPE=i686 CFLAGS= -O2 -pipe -funroll-loops -fno-strict-aliasing COPTFLAGS= -O2 -pipe -funroll-loops ... etc. etc. etc. /usr/src/sys/i386/conf/DELL2650: machine i386 cpu I686_CPU ident DELL options SCHED_4BSD #4BSD scheduler options SMP # Symmetric MultiProcessor Kernel device apic device amr # AMI MegaRAID device bge # Broadcom BCM570xx Gigabit Ethernet regards Claus Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:21:39 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 14D8B16A4CE for ; Tue, 15 Jun 2004 17:21:39 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id E625343D46 for ; Tue, 15 Jun 2004 17:21:38 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 32612 invoked by uid 89); 15 Jun 2004 17:20:57 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 15 Jun 2004 17:20:57 -0000 Message-Id: <5.2.0.9.2.20040615101805.015f6028@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 10:21:05 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: <20040615171225.711.qmail@web14107.mail.yahoo.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:21:39 -0000 > >FreeBSD 5.2.1, 2 GB RAM, dual Xeon @ 2.4 Ghz. Other >than enabling -O2 and CPUTYPE=i686 in /etc/make.conf >and make a customized kernel with amr in my kernel, >everything else is pretty much default. Claus, I think this problem applies to a recent (< 1 month) change in -CURRENT and that is most likely why you have not had any problems. - Jeremy From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:23:13 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F318616A4CF; Tue, 15 Jun 2004 17:23:12 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id BA9AA43D1D; Tue, 15 Jun 2004 17:23:12 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Tue, 15 Jun 2004 19:23:07 +0200 Message-ID: <082101c452fd$66bd1dd0$ce01000a@animal> From: "Lasse Laursen" To: "Claus Guttesen" , "Daniel Eischen" References: <20040615171225.711.qmail@web14107.mail.yahoo.com> Date: Tue, 15 Jun 2004 19:22:22 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org cc: mike cc: Cedric Tabary Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:23:13 -0000 Hi Claus, How many queries per second does the machine handle? We are at approx 100 queries per second in prime time. The load on the server is low and the performance is OK (haven't tried anything else but FreeBSD so I can't compare with Linux). Everything is working fine until a single thread suddenly locks up the entire system. :( There are no special compile options for neither FreeBSD or for MySQL (from ports). Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Claus Guttesen" To: "Daniel Eischen" ; "Lasse Laursen" Cc: ; "Cedric Tabary" ; "mike" ; Sent: Tuesday, June 15, 2004 7:12 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > > We just had our first lockup for today - and the > > last one. Next step is to > > > downgrade to Linux :( *crud* > > > > Maby not the greatest comfort, but I've had a Dell > 2650 with the Perc3-controller working for weeks. I > installed rt3 (troubleticket-system) which needed > Mysql (or postgresql). I installed Mysql 4.0 and > haven't had problems with it. I increased the amount > of RAM than Mysql can utilize, but that's about it. > > All installed from /usr/ports using portinstall. > > FreeBSD 5.2.1, 2 GB RAM, dual Xeon @ 2.4 Ghz. Other > than enabling -O2 and CPUTYPE=i686 in /etc/make.conf > and make a customized kernel with amr in my kernel, > everything else is pretty much default. > > I never touched HTT in BIOS nor using sysctl, that's > working out of the box. > > /etc/make.conf: > CPUTYPE=i686 > CFLAGS= -O2 -pipe -funroll-loops -fno-strict-aliasing > COPTFLAGS= -O2 -pipe -funroll-loops > ... > etc. etc. etc. > > /usr/src/sys/i386/conf/DELL2650: > machine i386 > cpu I686_CPU > ident DELL > options SCHED_4BSD #4BSD scheduler > options SMP # Symmetric > MultiProcessor Kernel > device apic > device amr # AMI MegaRAID > device bge # Broadcom BCM570xx > Gigabit Ethernet > > regards > Claus > > > Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:33:23 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A8F0416A4CE for ; Tue, 15 Jun 2004 17:33:23 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 887F743D55 for ; Tue, 15 Jun 2004 17:33:23 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 33038 invoked by uid 89); 15 Jun 2004 17:32:39 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 15 Jun 2004 17:32:39 -0000 Message-Id: <5.2.0.9.2.20040615102224.015312c0@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 10:32:47 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: <5.2.0.9.2.20040615101805.015f6028@mail.ojoink.com> References: <20040615171225.711.qmail@web14107.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:33:23 -0000 >Claus, > >I think this problem applies to a recent (< 1 month) change in -CURRENT >and that is most likely why you have not had any problems. > >- Jeremy It should be said that this may not be a discussion for freebsd-threads and instead might belong more on freebsd-kernel or freebsd-smp - since according to Daniel Eischen, (who has commit privileges) "there were no changes to the threads library that would exhibit the behavior..." I guess my early benchmarks suggested it was more of a kernel / SMP problem as well... figuring we would get damn near the same results on a UP kernel on an otherwise identical configuration. I suppose that on it's own warrants a post to freebsd-smp no? - Jeremy From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:37:06 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6553816A4CE; Tue, 15 Jun 2004 17:37:06 +0000 (GMT) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.208.78.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4890043D48; Tue, 15 Jun 2004 17:37:06 +0000 (GMT) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) i5FHahFo020776; Tue, 15 Jun 2004 10:36:43 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost)i5FHah5K020774; Tue, 15 Jun 2004 10:36:43 -0700 (PDT) (envelope-from sgk) Date: Tue, 15 Jun 2004 10:36:42 -0700 From: Steve Kargl To: Claus Guttesen Message-ID: <20040615173642.GA20669@troutmask.apl.washington.edu> References: <20040615171225.711.qmail@web14107.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040615171225.711.qmail@web14107.mail.yahoo.com> User-Agent: Mutt/1.4.1i cc: Cedric Tabary cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org cc: mike Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:37:06 -0000 On Tue, Jun 15, 2004 at 07:12:25PM +0200, Claus Guttesen wrote: >>> We just had our first lockup for today - and the >>> last one. Next step is to >>> downgrade to Linux :( *crud* >> > > /etc/make.conf: > CPUTYPE=i686 > CFLAGS= -O2 -pipe -funroll-loops -fno-strict-aliasing > COPTFLAGS= -O2 -pipe -funroll-loops man make.conf Now read the descriptions for CFLAGS and COPTFLAGS. -- Steve From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 17:47:58 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BDB0416A4CE for ; Tue, 15 Jun 2004 17:47:58 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9BA1343D41 for ; Tue, 15 Jun 2004 17:47:58 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 33622 invoked by uid 89); 15 Jun 2004 17:47:08 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 15 Jun 2004 17:47:08 -0000 Message-Id: <5.2.0.9.2.20040615104046.015286d8@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 10:47:16 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: <20040615173642.GA20669@troutmask.apl.washington.edu> References: <20040615171225.711.qmail@web14107.mail.yahoo.com> <20040615171225.711.qmail@web14107.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 17:47:58 -0000 At 10:36 AM 6/15/2004 -0700, you wrote: >On Tue, Jun 15, 2004 at 07:12:25PM +0200, Claus Guttesen wrote: > >>> We just had our first lockup for today - and the > >>> last one. Next step is to > >>> downgrade to Linux :( *crud* > >> > > > > /etc/make.conf: > > CPUTYPE=i686 > > CFLAGS= -O2 -pipe -funroll-loops -fno-strict-aliasing > > COPTFLAGS= -O2 -pipe -funroll-loops > >man make.conf > >Now read the descriptions for CFLAGS and COPTFLAGS. > >-- >Steve >_______________________________________________ >freebsd-threads@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-threads >To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" Err.... nice editing there. It should be noted that it was NOT Claus that posted "..Next step is to downgrade to linux" it was Lasse - the person who started this topic. Claus came along saying that his 5.2.1 (not -CURRENT) was not having the problems described in this thread. (Most likely because these problems are due to a recent commit.) The make.conf file was from Claus's "stable" system, not Lasse's. It doesn't make his make.conf any more or less right or wrong, but the way you chose to edit and reply makes it look like person having this lockup problem has a make.conf that should be re-evaluated. Just trying to keep it real & honest ;) - Jeremy From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 18:50:45 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 73ED816A507; Tue, 15 Jun 2004 18:50:44 +0000 (GMT) Received: from mail.ngdc.net (mail.ngdc.net [195.190.153.169]) by mx1.FreeBSD.org (Postfix) with ESMTP id E2A9743D4C; Tue, 15 Jun 2004 18:50:43 +0000 (GMT) (envelope-from laursen@netgroup.dk) Received: from animal (port390.ds1-noe.adsl.cybercity.dk [217.157.177.19]) (AUTH: LOGIN laursen@solidcore.dk) by bunsen.solidcore.dk with esmtp; Tue, 15 Jun 2004 20:50:02 +0200 Message-ID: <08d001c45309$8ac93630$ce01000a@animal> From: "Lasse Laursen" To: "Daniel Eischen" , freebsd-smp@freebsd.org References: Date: Tue, 15 Jun 2004 20:49:20 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1409 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 18:50:46 -0000 Hi again, The system still "locks up" on some queries after HTT has been disabled and all memory options removed from the kernel. I have posted this on the freebsd-smp@freebsd.org as well since it's probably a more suitable place for this bug/problem. Thank you all for your suggestions so far :) Regards -- Lasse Laursen · VP, Hosting Technology · NetGroup Processing Aps St. Kongensgade 40H · DK-1264 Copenhagen K, Denmark Phone: +45 3370 1526 · Fax: +45 3313 0066 - Don't be fooled by cheap finnish imitations - BSD is the One True Code ----- Original Message ----- From: "Daniel Eischen" To: "JG" Cc: Sent: Tuesday, June 15, 2004 6:51 PM Subject: Re: Possible Threading problem with -CURRENT / MySQL? > On Tue, 15 Jun 2004, JG wrote: > His lockups are complete; yours are not. Also, others have mentioned > that HTT causes problems. Please let him try various things instead > of discouraging him. Any information he can give us is helpful. > > > Robert Watson said he was going to try to look into this problem within the > > next few days. From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 21:18:13 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76BA416A4CE for ; Tue, 15 Jun 2004 21:18:13 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 27A7E43D58 for ; Tue, 15 Jun 2004 21:18:13 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5FLGrCZ073445; Tue, 15 Jun 2004 17:16:53 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5FLGrcC073442; Tue, 15 Jun 2004 17:16:53 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 15 Jun 2004 17:16:53 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 21:18:13 -0000 On Tue, 15 Jun 2004, Daniel Eischen wrote: > > My results were without HTT (It was FreeBSD-AMD64) I didn't have any memory > > options > > or a mysql config file. > > > > _This problem is directly related to a commit that was made between ~May 18 > > and yesterday._ > > There was a subsystem lock(s) put in for networking a few days ago. > There were no changes to the threads library that would exhibit the > behavior you are seeing. The candidate time range includes a number of big and relevant changes: - mbuma becomes the mbuf allocator, which would affect heavy UNIX domain socket I/O since mbufs are used for that I/O on local mysql configurations. Since general benchmarks I've seen suggest mbuma helps or has the same cost as the previous allocator, presumably it would be some special case regression. - Coarse-grained locking was introduced in UNIX domain sockets, but Giant has not yet been removed. This means we're paying an additional (and redundant) locking overhead without the benefits of the finer-grained locking model in the base tree. I've set up a test box to look at the performance impact of that today to see if it resulted in a substantial regression. I'd expect a few percent, but maybe it's worse and it would be good to know. - There have been a number of amd64-related VM changes, optimizations, etc. Maybe one didn't work out so well. I don't have an amd64-box so I can't really reason about that too much just now. If we can track down the date a little closer, that would help a lot. Even just adding a new data point for June 1 or June 7 would be helpful. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 23:19:20 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4B3A216A4D0; Tue, 15 Jun 2004 23:19:20 +0000 (GMT) Received: from sccrmhc12.comcast.net (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id EB0F843D54; Tue, 15 Jun 2004 23:19:19 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc12) with ESMTP id <20040615231910012002hokie>; Tue, 15 Jun 2004 23:19:11 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50694; Tue, 15 Jun 2004 16:19:08 -0700 (PDT) Date: Tue, 15 Jun 2004 16:19:07 -0700 (PDT) From: Julian Elischer To: Lasse Laursen In-Reply-To: <01c301c45239$940781f0$6401a8c0@animal> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 23:19:20 -0000 On Mon, 14 Jun 2004, Lasse Laursen wrote: > The server runs fine until a single thread/query suddenly locks up the > entire MySQL daemon. After that all queries are just queued and a restart of > the daemon is needed to unlock the system. The system itself is stable > enough as far as I can see. 'top' reports a lot of locks (*Giant) so I > assume that it's some weird problem with the threading? We used to use > FreeBSD on non-SMP machines without any problems. > > I have tried with linux threads as well but the same problem occurs. > > Have anyone experienced any similar problems and found a solution to this > rather weird problem? When this happens you need to examine the state of each thread. I have not seen teh result of a 'ps alxH' when this happens.. Also you should try 'ktrace' the process for a while when stopped as it might be retrying something , which may show up. julian From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 23:21:08 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7634316A4CF; Tue, 15 Jun 2004 23:21:08 +0000 (GMT) Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6865143D5C; Tue, 15 Jun 2004 23:21:08 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc13) with ESMTP id <2004061523204001500ff6e3e>; Tue, 15 Jun 2004 23:20:51 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50750; Tue, 15 Jun 2004 16:20:39 -0700 (PDT) Date: Tue, 15 Jun 2004 16:20:38 -0700 (PDT) From: Julian Elischer To: mike In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 23:21:08 -0000 On Mon, 14 Jun 2004, mike wrote: > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > as well. it sounds like scheduler/threading is to blame but we were not > able to give sufficient or proper motivation to the folks who could > examine this deeper - we even offered $500 cash to whomever stepped up to > help resolve this. > > linux runs almost 2x as fast on the same hardware with no configuring - > and we get nearly the same results running in single CPU mode vs. dual CPU > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > combination. > > You complained about this some time ago and you have still not responded with the information I suggest.. ---- quote from my previous mail--------------- I'm not saying I can help particularly but the problem is that the people who can help are generally not interested in databases.. what needs to happen is for some analysis to be made as to where the time is being spent.. I vaguely remember you saying htat teh cpu was not piegged when you had this problem... You didn't indicate whether the disks were at 100%. to check this you should try running systat -vmstat while noticing the slowdown and try see what is at 100%. if it turns out that you are NOT seeing disk or CPU limitted then the next step would be to take a snapshot of operations using ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off again. then looking at the output may give an idea of what operations are taking too long, and what is waiting on what.. ---------------------------------------------- Julian From owner-freebsd-threads@FreeBSD.ORG Tue Jun 15 23:22:45 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2DB5616A4CE for ; Tue, 15 Jun 2004 23:22:45 +0000 (GMT) Received: from sccrmhc13.comcast.net (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id E80DD43D31 for ; Tue, 15 Jun 2004 23:22:44 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc13) with ESMTP id <200406152322410160083sv8e>; Tue, 15 Jun 2004 23:22:42 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id QAA50793; Tue, 15 Jun 2004 16:22:40 -0700 (PDT) Date: Tue, 15 Jun 2004 16:22:39 -0700 (PDT) From: Julian Elischer To: JG In-Reply-To: <5.2.0.9.2.20040614183523.015cdaa0@mail.ojoink.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 23:22:45 -0000 On Mon, 14 Jun 2004, JG wrote: > > I just posted this to -AMD64 and someone brought this discussion to my > attention... > anyway I'll repost it here since it seems to be a more relevant discussion: > > > I got a couple emails from people telling me I should retest > because some changes have been committed recently that > might effect my MySQL benchmarks. > > Well I just did a buildworld and kernel to -CURRENT & it did > effect the results alright... just not the way we wanted them to: erk! :-) As I suggested elsewhere: ----- quoted---- To check this you should try running systat -vmstat while noticing the slowdown and try see what is at 100%. if it turns out that you are NOT seeing disk or CPU limitted then the next step would be to take a snapshot of operations using ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off again. then looking at the output may give an idea of what operations are taking too long, and what is waiting on what.. Julian From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 04:09:45 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6507416A4CE for ; Wed, 16 Jun 2004 04:09:45 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3C5E743D54 for ; Wed, 16 Jun 2004 04:09:45 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 57551 invoked by uid 89); 16 Jun 2004 04:08:48 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 16 Jun 2004 04:08:48 -0000 Message-Id: <5.2.0.9.2.20040615210256.016c55e0@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 21:08:51 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: References: <5.2.0.9.2.20040614183523.015cdaa0@mail.ojoink.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 04:09:45 -0000 > >erk! :-) > >As I suggested elsewhere: >----- quoted---- > >To check this you should try running systat -vmstat >while noticing the slowdown and try see what is at 100%. >if it turns out that you are NOT seeing disk or CPU limitted >then the next step would be to take a snapshot of operations using >ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off >again. > >then looking at the output may give an idea of what operations are >taking too long, and what is waiting on what.. > >Julian Julian, I think I'll wait for someone more qualified to evaluate these problems. I can't code, don't have commit access, and don't even know how to see what was added or removed to -CURRENT when or where etc. The problem has been laid on the table of the FreeBSD gods by many people now. FreeBSD would be dead in the water if it relied on the likes of me to do assessments and debugging. Once I identified what was taking too long and what was waiting on what - I would have no clue what to do or try from there, but I'm sure it would require some coding experience. - Jeremy From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 04:10:59 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D0CA16A4CE for ; Wed, 16 Jun 2004 04:10:59 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1C40043D39 for ; Wed, 16 Jun 2004 04:10:59 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 57643 invoked by uid 89); 16 Jun 2004 04:10:07 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 16 Jun 2004 04:10:07 -0000 Message-Id: <5.2.0.9.2.20040615210919.01574ec8@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Tue, 15 Jun 2004 21:10:12 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: References: <5.2.0.9.2.20040614183523.015cdaa0@mail.ojoink.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 04:10:59 -0000 > >As I suggested elsewhere: >----- quoted---- > >To check this you should try running systat -vmstat >while noticing the slowdown and try see what is at 100%. >if it turns out that you are NOT seeing disk or CPU limitted >then the next step would be to take a snapshot of operations using >ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off >again. But actually that doesn't sound too rough.... I'll try it tomorrow evening and give a report. - Jeremy From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 04:13:00 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 97F0D16A4CE for ; Wed, 16 Jun 2004 04:13:00 +0000 (GMT) Received: from exchhz01.viatech.com.cn (ip-40-162-97-218.anlai.com [218.97.162.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id 11CEB43D1D for ; Wed, 16 Jun 2004 04:12:58 +0000 (GMT) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (DAVIDWNT [10.4.1.99]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id NAMYV88X; Wed, 16 Jun 2004 12:12:46 +0800 Message-ID: <40CFC966.3080107@freebsd.org> Date: Wed, 16 Jun 2004 12:15:34 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: JG References: <5.2.0.9.2.20040614183523.015cdaa0@mail.ojoink.com> <5.2.0.9.2.20040615210256.016c55e0@mail.ojoink.com> In-Reply-To: <5.2.0.9.2.20040615210256.016c55e0@mail.ojoink.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 04:13:00 -0000 I think we need a page to summarize the problem. David Xu JG wrote: > >> >> erk! :-) >> >> As I suggested elsewhere: >> ----- quoted---- >> From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 04:49:24 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C696816A4CE for ; Wed, 16 Jun 2004 04:49:24 +0000 (GMT) Received: from sccrmhc13.comcast.net (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8330843D2F for ; Wed, 16 Jun 2004 04:49:24 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc13) with ESMTP id <200406160449230160083bjte>; Wed, 16 Jun 2004 04:49:23 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id VAA54442; Tue, 15 Jun 2004 21:49:21 -0700 (PDT) Date: Tue, 15 Jun 2004 21:49:20 -0700 (PDT) From: Julian Elischer To: JG In-Reply-To: <5.2.0.9.2.20040615210256.016c55e0@mail.ojoink.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 04:49:24 -0000 but you can run systat -vmstat when it is as fast as it can go... and you can do teh ktrace (man ktrace) and make the file available... :-) On Tue, 15 Jun 2004, JG wrote: > > > > >erk! :-) > > > >As I suggested elsewhere: > >----- quoted---- > > > >To check this you should try running systat -vmstat > >while noticing the slowdown and try see what is at 100%. > >if it turns out that you are NOT seeing disk or CPU limitted > >then the next step would be to take a snapshot of operations using > >ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off > >again. > > > >then looking at the output may give an idea of what operations are > >taking too long, and what is waiting on what.. > > > >Julian > > > Julian, > > I think I'll wait for someone more qualified to evaluate these problems. > > I can't code, don't have commit access, and don't even know how to see > what was added or removed to -CURRENT when or where etc. > > The problem has been laid on the table of the FreeBSD gods by many > people now. FreeBSD would be dead in the water if it relied on the likes > of me to do assessments and debugging. > > Once I identified what was taking too long and what was waiting on what > - I would have no clue what to do or try from there, but I'm sure it would > require some coding experience. > > - Jeremy > > > > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 05:12:51 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F2F816A4CE for ; Wed, 16 Jun 2004 05:12:51 +0000 (GMT) Received: from web53708.mail.yahoo.com (web53708.mail.yahoo.com [206.190.37.29]) by mx1.FreeBSD.org (Postfix) with SMTP id 067F043D41 for ; Wed, 16 Jun 2004 05:12:51 +0000 (GMT) (envelope-from chancedj@yahoo.com) Message-ID: <20040616051052.98903.qmail@web53708.mail.yahoo.com> Received: from [24.170.135.170] by web53708.mail.yahoo.com via HTTP; Tue, 15 Jun 2004 22:10:52 PDT Date: Tue, 15 Jun 2004 22:10:52 -0700 (PDT) From: Daryl Chance To: freebsd-current@freebsd.org, freebsd-threads@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: mike@mike2k.com cc: julian@elischer.org Subject: RE: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: chancedj@yahoo.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 05:12:51 -0000 Sorry for the top post. We've had this same problem (with mysqld locking up). I was able to get a trace, but since it's a live sql box, I didn't have time to find your email (If I knew I had kept a copy in my inbox, I would have done this correctly, instead I thought I had to look up the email in the archives) but I was able to get a ktrace -p pid output of the process. I let it run about 10 seconds and then killall -11 mysqld'd the process (it allows mysqld to shutdown gracefully, thus not corrupting our tables). Here is the link to the .out file http://sql.tribalwar.com/ktrace.out here is the link to the .txt file of the kdump output http://sql.tribalwar.com/kdump.txt That was all I could remember to do in the short time I had to get it back up and running :). It's a stock kernel (w/ some minor things commented out). It's FBSD 5.2.1 p6 http://sql.tribalwar.com/SQL Hopefully this can help out a little. Daryl -----Original Message----- From: owner-freebsd-current@freebsd.org [mailto:owner-freebsd-current@freebsd.org] On Behalf Of Julian Elischer Sent: Tuesday, June 15, 2004 7:21 PM To: mike Cc: Lasse Laursen; freebsd-current@freebsd.org; freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? On Mon, 14 Jun 2004, mike wrote: > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > as well. it sounds like scheduler/threading is to blame but we were not > able to give sufficient or proper motivation to the folks who could > examine this deeper - we even offered $500 cash to whomever stepped up to > help resolve this. > > linux runs almost 2x as fast on the same hardware with no configuring - > and we get nearly the same results running in single CPU mode vs. dual CPU > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > combination. > > You complained about this some time ago and you have still not responded with the information I suggest.. ---- quote from my previous mail--------------- I'm not saying I can help particularly but the problem is that the people who can help are generally not interested in databases.. what needs to happen is for some analysis to be made as to where the time is being spent.. I vaguely remember you saying htat teh cpu was not piegged when you had this problem... You didn't indicate whether the disks were at 100%. to check this you should try running systat -vmstat while noticing the slowdown and try see what is at 100%. if it turns out that you are NOT seeing disk or CPU limitted then the next step would be to take a snapshot of operations using ktrace -d -p (pid) for 20 second followe by krtace -C to turn it off again. then looking at the output may give an idea of what operations are taking too long, and what is waiting on what.. ---------------------------------------------- Julian _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 05:25:52 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3562516A4CE; Wed, 16 Jun 2004 05:25:52 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9093543D41; Wed, 16 Jun 2004 05:25:51 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5G5NtMq083977; Wed, 16 Jun 2004 01:23:55 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5G5NteE083974; Wed, 16 Jun 2004 01:23:55 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Wed, 16 Jun 2004 01:23:55 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 05:25:52 -0000 On Tue, 15 Jun 2004, Julian Elischer wrote: > On Mon, 14 Jun 2004, mike wrote: > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > as well. it sounds like scheduler/threading is to blame but we were not > > able to give sufficient or proper motivation to the folks who could > > examine this deeper - we even offered $500 cash to whomever stepped up to > > help resolve this. > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > and we get nearly the same results running in single CPU mode vs. dual CPU > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > combination. > > You complained about this some time ago and you have still not responded > with the information I suggest.. I sent this to Jeremy privately, since it was just some preliminary measurements, but figured I'd send it publically since the results were interesting (if tentative, I need to do a lot more work to make them useful. There are a number of variables I need to look at including: - Disabling HTT. A chat with Scott Long this evening suggests that HTT may be substantially hurting the test cases given increased IPIs, etc. Unfortunately, it looks like I can't easily twiddle HTT without being local to the machine, and I'm at home right now (it being 1:30am and all). Removing HTT may help substantially with the dip in performance in the SMP configuration. - I'd like to compare against RELENG_4 and a recent Linux kernel. Unfortunately, the box is configured for neither right now. - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd like to try SCHED_4BSD. - This was without adaptive mutexes, which seem to be helpful for others, so I should give them a try. I don't have any amd64 hardware, so I don't know what if any role it will play in the results. The performance drop observed in the report appears to be on amd64 (I may have misread). Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research ---- Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) From: Robert Watson To: JG Subject: Re: Possible Threading problem with -CURRENT / MySQL? On Tue, 15 Jun 2004, JG wrote: > Fwiw, it has to be something that was committed between May 18th and > yesterday. ~May 18th was the last time I built -CURRENT during my last > round of testing and I did not have any of these problems. Then someone > emailed me recently and said there were some commits that might effect > the outcome of the mysql benchmarks. Ok, so these results are on a dual-processor XEON + hyperthreads, so four logical processors. I used two dates off CVS, 20040515 and 20040615. I also benchmarked my netperf branch. I don't have RELENG_4 on the box, but might be able to load RELENG_4 on it later this week. In each case, I took ten samples, dropped the first value as getting into the cache, and took the mean of the rest. For this test, I used the select test; I'll try the other smack query set tomorrow. In each case, I ran with "10 1000" as the arguments to the test. I used the default threading configuration in -CURRENT, which is libpthread (libkse). Mean Stdev 20040515-UP 4752.27 14.63 20040515-SMP 2550.35 19.23 20040615-UP 4898.71 22.39 20040615-SMP 2666.93 32.01 Netperf-UP-giant 4902.41 14.3 Netperf-SMP-giant 2566.18 16.83 Netperf-UP-mpsafe 4799.35 22.04 Netperf-SMP-mpsafe 3022.51 18.06 Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages the SMP numbers a fair amount due to additional IPIs without benefit. However, the numbers basically suggest that on my hardware, the UP configuration is marginally faster than it was last month, and that if you throw in the netperf branch, the SMP case is a moderate amount faster. This suggests that either I'm just lucky, or that the performance loss might be specific to the amd64 version of FreeBSD. I'm going to run some more numbers tomorrow and try to post something more rigorous to the -threads list. I don't have RELENG_4 on the box or Linux on the box, but I may get a chance to later this week. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 05:45:30 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0045C16A4CE; Wed, 16 Jun 2004 05:45:30 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9BC7C43D48; Wed, 16 Jun 2004 05:45:29 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5G5jNqb011596; Wed, 16 Jun 2004 01:45:23 -0400 (EDT) Date: Wed, 16 Jun 2004 01:45:23 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Daryl Chance In-Reply-To: <20040616051052.98903.qmail@web53708.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: julian@elischer.org cc: mike@mike2k.com cc: freebsd-threads@freebsd.org Subject: RE: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 05:45:30 -0000 On Tue, 15 Jun 2004, Daryl Chance wrote: > Sorry for the top post. > > We've had this same problem (with mysqld locking up). > I was able to get a trace, but since it's a live sql > box, I didn't have time to find your email (If I knew > I had kept a copy in my inbox, I would have done this > correctly, instead I thought I had to look up the > email in the archives) but I was able to get a ktrace > -p pid output of the process. I let it run about 10 > seconds and then killall -11 mysqld'd the process (it > allows mysqld to shutdown gracefully, thus not > corrupting our tables). > > Here is the link to the .out file > http://sql.tribalwar.com/ktrace.out > > here is the link to the .txt file of the kdump output > http://sql.tribalwar.com/kdump.txt > > That was all I could remember to do in the short time > I had to get it back up and running :). > > It's a stock kernel (w/ some minor things commented > out). It's FBSD 5.2.1 p6 > http://sql.tribalwar.com/SQL I don't think 5.2.1 is going to tell us much. We know there was a problem in 5.2-release. We really need -current. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 05:49:57 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D52D16A4CE for ; Wed, 16 Jun 2004 05:49:57 +0000 (GMT) Received: from web53702.mail.yahoo.com (web53702.mail.yahoo.com [206.190.37.23]) by mx1.FreeBSD.org (Postfix) with SMTP id E4FB343D1F for ; Wed, 16 Jun 2004 05:49:52 +0000 (GMT) (envelope-from chancedj@yahoo.com) Message-ID: <20040616054951.37343.qmail@web53702.mail.yahoo.com> Received: from [24.170.135.170] by web53702.mail.yahoo.com via HTTP; Tue, 15 Jun 2004 22:49:51 PDT Date: Tue, 15 Jun 2004 22:49:51 -0700 (PDT) From: Daryl Chance To: freebsd-current@freebsd.org, freebsd-threads@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: RE: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: chancedj@yahoo.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 05:49:57 -0000 We're actually upgrading tomorrow night to -CURRENT, if it continues, I will let you guys know. :). -----Original Message----- From: owner-freebsd-current@freebsd.org [mailto:owner-freebsd-current@freebsd.org] On Behalf Of Daniel Eischen Sent: Wednesday, June 16, 2004 1:45 AM To: Daryl Chance Cc: freebsd-current@freebsd.org; julian@elischer.org; mike@mike2k.com; freebsd-threads@freebsd.org Subject: RE: Possible Threading problem with -CURRENT / MySQL? On Tue, 15 Jun 2004, Daryl Chance wrote: > Sorry for the top post. > > We've had this same problem (with mysqld locking up). > I was able to get a trace, but since it's a live sql > box, I didn't have time to find your email (If I knew > I had kept a copy in my inbox, I would have done this > correctly, instead I thought I had to look up the > email in the archives) but I was able to get a ktrace > -p pid output of the process. I let it run about 10 > seconds and then killall -11 mysqld'd the process (it > allows mysqld to shutdown gracefully, thus not > corrupting our tables). > > Here is the link to the .out file > http://sql.tribalwar.com/ktrace.out > > here is the link to the .txt file of the kdump output > http://sql.tribalwar.com/kdump.txt > > That was all I could remember to do in the short time > I had to get it back up and running :). > > It's a stock kernel (w/ some minor things commented > out). It's FBSD 5.2.1 p6 > http://sql.tribalwar.com/SQL I don't think 5.2.1 is going to tell us much. We know there was a problem in 5.2-release. We really need -current. -- Dan Eischen _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 05:53:17 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8CB7E16A4CE; Wed, 16 Jun 2004 05:53:17 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1E26743D1D; Wed, 16 Jun 2004 05:53:17 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5G5phEG084287; Wed, 16 Jun 2004 01:51:43 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5G5phga084284; Wed, 16 Jun 2004 01:51:43 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Wed, 16 Jun 2004 01:51:42 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: mike cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 05:53:17 -0000 On Wed, 16 Jun 2004, Robert Watson wrote: > Netperf-UP-giant 4902.41 14.3 > Netperf-SMP-giant 2566.18 16.83 > Netperf-UP-mpsafe 4799.35 22.04 > Netperf-SMP-mpsafe 3022.51 18.06 FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes to 4000 q/s on the netperf+smp+mpsafenet number. I'm off to bed now, but it seems like (at least in the HTT configuration), it makes a big difference. I'll run the remainder of that set with ADAPTIVE_MUTEXES tomorrow as well. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 06:43:14 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A7D8116A4CE; Wed, 16 Jun 2004 06:43:14 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 52D3A43D39; Wed, 16 Jun 2004 06:43:14 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5G6fmJx085048; Wed, 16 Jun 2004 02:41:49 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5G6fmud085045; Wed, 16 Jun 2004 02:41:48 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Wed, 16 Jun 2004 02:41:48 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 06:43:14 -0000 On Wed, 16 Jun 2004, Robert Watson wrote: > On Wed, 16 Jun 2004, Robert Watson wrote: > > > Netperf-UP-giant 4902.41 14.3 > > Netperf-SMP-giant 2566.18 16.83 > > Netperf-UP-mpsafe 4799.35 22.04 > > Netperf-SMP-mpsafe 3022.51 18.06 > > FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes > to 4000 q/s on the netperf+smp+mpsafenet number. > > I'm off to bed now, but it seems like (at least in the HTT > configuration), it makes a big difference. I'll run the remainder of > that set with ADAPTIVE_MUTEXES tomorrow as well. And switching to 4BSD from ULE takes the mean to 6426 q/s when combined with netperf, smp, debug.mpsafenet=1, and adaptive mutexes. (Keeping in mind that this mysql benchmark is basically an inter-thread/process kernel IPC benchmark using UNIX domain sockets, since the workload is fairly minimal in userspace, this makes reasonable sense). Now I'm really going to bed. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 07:49:07 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EFDB016A4CE; Wed, 16 Jun 2004 07:49:07 +0000 (GMT) Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id D7E1C43D5E; Wed, 16 Jun 2004 07:49:07 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc13) with ESMTP id <2004061607485101500fbl01e>; Wed, 16 Jun 2004 07:48:52 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id AAA56755; Wed, 16 Jun 2004 00:48:50 -0700 (PDT) Date: Wed, 16 Jun 2004 00:48:49 -0700 (PDT) From: Julian Elischer To: Robert Watson In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 07:49:08 -0000 On Wed, 16 Jun 2004, Robert Watson wrote: > > On Wed, 16 Jun 2004, Robert Watson wrote: > > > On Wed, 16 Jun 2004, Robert Watson wrote: > > > > > Netperf-UP-giant 4902.41 14.3 > > > Netperf-SMP-giant 2566.18 16.83 > > > Netperf-UP-mpsafe 4799.35 22.04 > > > Netperf-SMP-mpsafe 3022.51 18.06 > > > > FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes > > to 4000 q/s on the netperf+smp+mpsafenet number. > > > > I'm off to bed now, but it seems like (at least in the HTT > > configuration), it makes a big difference. I'll run the remainder of > > that set with ADAPTIVE_MUTEXES tomorrow as well. > > And switching to 4BSD from ULE takes the mean to 6426 q/s when combined > with netperf, smp, debug.mpsafenet=1, and adaptive mutexes. > > (Keeping in mind that this mysql benchmark is basically an > inter-thread/process kernel IPC benchmark using UNIX domain sockets, since > the workload is fairly minimal in userspace, this makes reasonable sense). > > Now I'm really going to bed. One thing that was happenning at one time, was that when a process is made runnable on one processor, If another processor is idle, it remains idle until the next scheduling tick. I proposed patches about a year ago that fixed this but "as far as I know" it is still true. The solution is to generate an IPI when you make a thread runnable and you see an idle processor.. To see if this is the problem a quick test would be to move HZ to 5000 or something and see what happens. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 07:57:40 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F90E16A4CF; Wed, 16 Jun 2004 07:57:40 +0000 (GMT) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8981543D1D; Wed, 16 Jun 2004 07:57:40 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <200406160741030130097uq9e>; Wed, 16 Jun 2004 07:41:12 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id AAA56709; Wed, 16 Jun 2004 00:41:01 -0700 (PDT) Date: Wed, 16 Jun 2004 00:41:00 -0700 (PDT) From: Julian Elischer To: Robert Watson In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 07:57:40 -0000 I'd like to see what happens with HZ=1000 (or even 5000) On Wed, 16 Jun 2004, Robert Watson wrote: > > On Wed, 16 Jun 2004, Robert Watson wrote: > > > Netperf-UP-giant 4902.41 14.3 > > Netperf-SMP-giant 2566.18 16.83 > > Netperf-UP-mpsafe 4799.35 22.04 > > Netperf-SMP-mpsafe 3022.51 18.06 > > FYI, when I add ADAPTIVE_MUTEXES on this box, the mean of 3000 q/s goes to > 4000 q/s on the netperf+smp+mpsafenet number. > > I'm off to bed now, but it seems like (at least in the HTT configuration), > it makes a big difference. I'll run the remainder of that set with > ADAPTIVE_MUTEXES tomorrow as well. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 15:03:49 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8EF5716A4F1 for ; Wed, 16 Jun 2004 15:03:49 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6269B43D45 for ; Wed, 16 Jun 2004 15:03:49 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 75654 invoked by uid 89); 16 Jun 2004 15:03:13 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 16 Jun 2004 15:03:13 -0000 Message-Id: <5.2.0.9.2.20040616080116.014819f8@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Wed, 16 Jun 2004 08:03:14 -0700 To: Robert Watson From: JG In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 15:03:49 -0000 > >I don't have any amd64 hardware, so I don't know what if any role it will >play in the results. The performance drop observed in the report appears >to be on amd64 (I may have misread). If it's within the next few days or so, I can make my AMD-64 SMP box available to you Robert. - Jeremy From owner-freebsd-threads@FreeBSD.ORG Wed Jun 16 18:34:05 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8DE1C16A4CE for ; Wed, 16 Jun 2004 18:34:05 +0000 (GMT) Received: from creme-brulee.marcuscom.com (rrcs-midsouth-24-172-16-118.biz.rr.com [24.172.16.118]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2479943D39 for ; Wed, 16 Jun 2004 18:34:05 +0000 (GMT) (envelope-from marcus@FreeBSD.org) Received: from [10.2.1.4] (vpn-client-4.marcuscom.com [10.2.1.4]) i5GIXKNo018951 for ; Wed, 16 Jun 2004 14:33:20 -0400 (EDT) (envelope-from marcus@FreeBSD.org) From: Joe Marcus Clarke To: threads@FreeBSD.org Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-3tm8KMWdX3U5FITJdU/S" Organization: FreeBSD, Inc. Message-Id: <1087410860.888.22.camel@gyros> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Wed, 16 Jun 2004 14:34:20 -0400 X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on creme-brulee.marcuscom.com Subject: Illegal instruction in firefox/thunderbird with libpthread X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 18:34:05 -0000 --=-3tm8KMWdX3U5FITJdU/S Content-Type: text/plain Content-Transfer-Encoding: quoted-printable I'm running -CURRENT: FreeBSD gyros.marcuscom.com 5.2-CURRENT FreeBSD 5.2-CURRENT #33: Fri Jun 11 17:44:01 EDT 2004 =20 marcus@gyros.marcuscom.com:/usr/obj/usr/src/sys/GYROS i386 And trying to port the new thunderbird. When I run it on a fresh account, I immediately get an illegal instruction and a core dump. If I libmap libpthread to libc_r or libthr, the problem goes away, and thunderbird starts. If I then libmap libpthread back to libpthread, I no longer get the problem. Therefore, it looks like one code path in thunderbird (and one user has reported the same thing with firefox-0.9) causes a crash with KSE. Here is the backtrace I have so far: Illegal instruction (core dumped) #0 0x28872477 in kse_create () at {standard input}:15 No locals. #1 0x2886a539 in _kse_setthreaded (threaded=3D-1077946880) at /usr/src/lib/libpthread/thread/thr_kern.c:432 sigset =3D {__bits =3D {135174400, 3217020440, 671471161, 0}} #2 0x2885fe99 in _pthread_create (thread=3D0x0, attr=3D0xbfbfd608, start_routine=3D0x2884b31c , arg=3D0x80e9900) at /usr/src/lib/libpthread/thread/thr_private.h:971 curthread =3D (struct pthread *) 0x807e2b0 new_thread =3D (struct pthread *) 0x807be00 kse =3D (struct kse *) 0x0 kseg =3D (struct kse_group *) 0x0 crit =3D 0x1 ret =3D 1 #3 0x2884b759 in PR_Select () from /usr/X11R6/lib/thunderbird/lib/thunderbird-0.7/libnspr4.so No symbol table info available. This is highly reproduceable with thunderbird, and I can make the port available if someone wants it. Thanks. Joe --=20 Joe Marcus Clarke FreeBSD GNOME Team :: marcus@FreeBSD.org gnome@FreeBSD.org FreeNode / #freebsd-gnome http://www.FreeBSD.org/gnome --=-3tm8KMWdX3U5FITJdU/S Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQBA0JKsb2iPiv4Uz4cRAhl7AJ9DzzTcna9htH5KIKK6rbmcpDpdWACfXLwY clfRnRAUlTZ/hDuR/UBsonE= =Kv2h -----END PGP SIGNATURE----- --=-3tm8KMWdX3U5FITJdU/S-- From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 01:40:50 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5778F16A4CE; Thu, 17 Jun 2004 01:40:50 +0000 (GMT) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 38A0343D58; Thu, 17 Jun 2004 01:40:50 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <2004061701395801300gfbgme>; Thu, 17 Jun 2004 01:39:58 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id SAA68093; Wed, 16 Jun 2004 18:39:57 -0700 (PDT) Date: Wed, 16 Jun 2004 18:39:55 -0700 (PDT) From: Julian Elischer To: Robert Watson In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 01:40:50 -0000 Robert, can you try see if HZ=5000 (up from 100) changes the performance? On Wed, 16 Jun 2004, Robert Watson wrote: > > On Tue, 15 Jun 2004, Julian Elischer wrote: > > > On Mon, 14 Jun 2004, mike wrote: > > > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > > as well. it sounds like scheduler/threading is to blame but we were not > > > able to give sufficient or proper motivation to the folks who could > > > examine this deeper - we even offered $500 cash to whomever stepped up to > > > help resolve this. > > > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > > and we get nearly the same results running in single CPU mode vs. dual CPU > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > > combination. > > > > You complained about this some time ago and you have still not responded > > with the information I suggest.. > > I sent this to Jeremy privately, since it was just some preliminary > measurements, but figured I'd send it publically since the results were > interesting (if tentative, I need to do a lot more work to make them > useful. There are a number of variables I need to look at including: > > - Disabling HTT. A chat with Scott Long this evening suggests that HTT > may be substantially hurting the test cases given increased IPIs, etc. > Unfortunately, it looks like I can't easily twiddle HTT without being > local to the machine, and I'm at home right now (it being 1:30am and > all). Removing HTT may help substantially with the dip in performance > in the SMP configuration. > > - I'd like to compare against RELENG_4 and a recent Linux kernel. > Unfortunately, the box is configured for neither right now. > > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd > like to try SCHED_4BSD. > > - This was without adaptive mutexes, which seem to be helpful for others, > so I should give them a try. > > I don't have any amd64 hardware, so I don't know what if any role it will > play in the results. The performance drop observed in the report appears > to be on amd64 (I may have misread). > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > ---- > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) > From: Robert Watson > To: JG > Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > > On Tue, 15 Jun 2004, JG wrote: > > > Fwiw, it has to be something that was committed between May 18th and > > yesterday. ~May 18th was the last time I built -CURRENT during my last > > round of testing and I did not have any of these problems. Then someone > > emailed me recently and said there were some commits that might effect > > the outcome of the mysql benchmarks. > > Ok, so these results are on a dual-processor XEON + hyperthreads, so four > logical processors. I used two dates off CVS, 20040515 and 20040615. I > also benchmarked my netperf branch. I don't have RELENG_4 on the box, but > might be able to load RELENG_4 on it later this week. In each case, I > took ten samples, dropped the first value as getting into the cache, and > took the mean of the rest. For this test, I used the select test; I'll > try the other smack query set tomorrow. In each case, I ran with "10 > 1000" as the arguments to the test. I used the default threading > configuration in -CURRENT, which is libpthread (libkse). > > Mean Stdev > 20040515-UP 4752.27 14.63 > 20040515-SMP 2550.35 19.23 > > 20040615-UP 4898.71 22.39 > 20040615-SMP 2666.93 32.01 > > Netperf-UP-giant 4902.41 14.3 > Netperf-SMP-giant 2566.18 16.83 > Netperf-UP-mpsafe 4799.35 22.04 > Netperf-SMP-mpsafe 3022.51 18.06 > > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages > the SMP numbers a fair amount due to additional IPIs without benefit. > However, the numbers basically suggest that on my hardware, the UP > configuration is marginally faster than it was last month, and that if you > throw in the netperf branch, the SMP case is a moderate amount faster. > This suggests that either I'm just lucky, or that the performance loss > might be specific to the amd64 version of FreeBSD. I'm going to run some > more numbers tomorrow and try to post something more rigorous to the > -threads list. > > I don't have RELENG_4 on the box or Linux on the box, but I may get a > chance to later this week. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 05:34:11 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8964016A4CE; Thu, 17 Jun 2004 05:34:11 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id E9BD443D45; Thu, 17 Jun 2004 05:34:10 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5H5VwJK015223; Thu, 17 Jun 2004 01:31:58 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5H5Vw3q015220; Thu, 17 Jun 2004 01:31:58 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 01:31:57 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 05:34:11 -0000 On Wed, 16 Jun 2004, Julian Elischer wrote: > Robert, can you try see if HZ=5000 (up from 100) changes the > performance? I haven't explored the HZ=BIGGER case very thoroughly, but ran a couple of test cases with HZ at 1000 instead of 100, and found that generally it resulted in a slight (1.5%) performance drop in the smattering of cases I looked at. The cases I looked at were NETPERF-SMP-MPSAFE-ADMTX-HZ=1000, NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT-HZ=1000. In general, I found that the biggest variables improving performance relative to out-of-box SMP configuration were: - Removing Giant from UNIX domain sockets - Running with SCHED_4BSD instead of SCHED_ULE - Disabling HTT - Using ADAPTIVE_MUTEXES By doing these things, I went from an out-of-box queries/sec for the simple select "smack" with 11 clients from 2667 q/s to 6955 q/s, or a bit over doubling the transaction rate. That as compared to 4939 q/s, which was the best UP result I got (Giant-free UNIX domain sockets, SCHED_4BSD): 20040615-UP 4898.71 q/s (22.39 stddev) 20040615-SMP 2666.93 q/s (32.01 stddev) NETPERF-UP-MPSAFE-4BSD 4939.89 q/s (74.06 stddev) NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT 6955.18 q/s (156.91 stddev) Some of my 4BSD performance numbers were also slightly pessimized with what appears to be a property of the 4BSD scheduling period being longer than ULE. With ULE, the results settled on one loop of the benchmark. With 4BSD, it sometimes took 2-3 loops of the benchmark to settle. As a result, 4BSD stddev's are also generally higher. I haven't tried regenerating the results dropping those outliers. Using out-of-box -CURRENT without the netperf patches necessary to run UNIX domain sockets safely, I got best results using: 20040615-UP-4BSD 4886.84 q/s (46.03 stddev) 20040615-SMP-ADMTX-4BSD-!HTT 5838.76 q/s (45.11 stddev) In my environment, I couldn't easily test ULE without HTT, since the HTT disabling sysctl/tunable with ULE results in a hang. (Obviously, in all of the above, WITNESS and INVARIANTS are disabled; I didn't disable the userspace malloc debugging flags). Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > On Wed, 16 Jun 2004, Robert Watson wrote: > > > > > On Tue, 15 Jun 2004, Julian Elischer wrote: > > > > > On Mon, 14 Jun 2004, mike wrote: > > > > > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > > > as well. it sounds like scheduler/threading is to blame but we were not > > > > able to give sufficient or proper motivation to the folks who could > > > > examine this deeper - we even offered $500 cash to whomever stepped up to > > > > help resolve this. > > > > > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > > > and we get nearly the same results running in single CPU mode vs. dual CPU > > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > > > combination. > > > > > > You complained about this some time ago and you have still not responded > > > with the information I suggest.. > > > > I sent this to Jeremy privately, since it was just some preliminary > > measurements, but figured I'd send it publically since the results were > > interesting (if tentative, I need to do a lot more work to make them > > useful. There are a number of variables I need to look at including: > > > > - Disabling HTT. A chat with Scott Long this evening suggests that HTT > > may be substantially hurting the test cases given increased IPIs, etc. > > Unfortunately, it looks like I can't easily twiddle HTT without being > > local to the machine, and I'm at home right now (it being 1:30am and > > all). Removing HTT may help substantially with the dip in performance > > in the SMP configuration. > > > > - I'd like to compare against RELENG_4 and a recent Linux kernel. > > Unfortunately, the box is configured for neither right now. > > > > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd > > like to try SCHED_4BSD. > > > > - This was without adaptive mutexes, which seem to be helpful for others, > > so I should give them a try. > > > > I don't have any amd64 hardware, so I don't know what if any role it will > > play in the results. The performance drop observed in the report appears > > to be on amd64 (I may have misread). > > > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > > > > ---- > > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) > > From: Robert Watson > > To: JG > > Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > > > > > On Tue, 15 Jun 2004, JG wrote: > > > > > Fwiw, it has to be something that was committed between May 18th and > > > yesterday. ~May 18th was the last time I built -CURRENT during my last > > > round of testing and I did not have any of these problems. Then someone > > > emailed me recently and said there were some commits that might effect > > > the outcome of the mysql benchmarks. > > > > Ok, so these results are on a dual-processor XEON + hyperthreads, so four > > logical processors. I used two dates off CVS, 20040515 and 20040615. I > > also benchmarked my netperf branch. I don't have RELENG_4 on the box, but > > might be able to load RELENG_4 on it later this week. In each case, I > > took ten samples, dropped the first value as getting into the cache, and > > took the mean of the rest. For this test, I used the select test; I'll > > try the other smack query set tomorrow. In each case, I ran with "10 > > 1000" as the arguments to the test. I used the default threading > > configuration in -CURRENT, which is libpthread (libkse). > > > > Mean Stdev > > 20040515-UP 4752.27 14.63 > > 20040515-SMP 2550.35 19.23 > > > > 20040615-UP 4898.71 22.39 > > 20040615-SMP 2666.93 32.01 > > > > Netperf-UP-giant 4902.41 14.3 > > Netperf-SMP-giant 2566.18 16.83 > > Netperf-UP-mpsafe 4799.35 22.04 > > Netperf-SMP-mpsafe 3022.51 18.06 > > > > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages > > the SMP numbers a fair amount due to additional IPIs without benefit. > > However, the numbers basically suggest that on my hardware, the UP > > configuration is marginally faster than it was last month, and that if you > > throw in the netperf branch, the SMP case is a moderate amount faster. > > This suggests that either I'm just lucky, or that the performance loss > > might be specific to the amd64 version of FreeBSD. I'm going to run some > > more numbers tomorrow and try to post something more rigorous to the > > -threads list. > > > > I don't have RELENG_4 on the box or Linux on the box, but I may get a > > chance to later this week. > > > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > > > > > > From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 06:08:28 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 401B116A4CE; Thu, 17 Jun 2004 06:08:28 +0000 (GMT) Received: from vhost109.his.com (vhost109.his.com [216.194.225.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6CF6843D46; Thu, 17 Jun 2004 06:08:27 +0000 (GMT) (envelope-from brad.knowles@skynet.be) Received: from [10.0.1.3] (localhost.his.com [127.0.0.1]) by vhost109.his.com (8.12.11/8.12.3) with ESMTP id i5H67uBj016947; Thu, 17 Jun 2004 02:07:59 -0400 (EDT) (envelope-from brad.knowles@skynet.be) Mime-Version: 1.0 X-Sender: bs663385@pop.skynet.be Message-Id: In-Reply-To: References: Date: Thu, 17 Jun 2004 08:07:50 +0200 To: Robert Watson From: Brad Knowles Content-Type: text/plain; charset="us-ascii" ; format="flowed" cc: mike cc: Julian Elischer cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 06:08:28 -0000 At 1:31 AM -0400 2004-06-17, Robert Watson wrote: > - Removing Giant from UNIX domain sockets Out of curiosity, is this something that could be relatively safely done in general? Any ideas on what the plan is for doing this as the default on -CURRENT? > - Disabling HTT > - Using ADAPTIVE_MUTEXES These both sound like typical improvements, based on what I've seen on this list. Any ideas on when they might become the default? > - Running with SCHED_4BSD instead of SCHED_ULE This is the only one that really concerns me. This shows that we clearly need more work on ULE. Is there one particular thing that we seem to be tripping up on, or is it a multitude of things? -- Brad Knowles, "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin, Historical Review of Pennsylvania. SAGE member since 1995. See for more info. From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 06:22:40 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E4D7716A4CF; Thu, 17 Jun 2004 06:22:40 +0000 (GMT) Received: from rms04.rommon.net (rms04.rommon.net [212.54.2.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9351043D58; Thu, 17 Jun 2004 06:22:39 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from he.iki.fi (h86.vuokselantie10.fi [193.64.42.134]) by rms04.rommon.net (8.12.10/8.12.9) with ESMTP id i5H6LZ3v044467; Thu, 17 Jun 2004 09:21:35 +0300 (EEST) (envelope-from pete@he.iki.fi) Message-ID: <40D1386F.1000703@he.iki.fi> Date: Thu, 17 Jun 2004 09:21:35 +0300 From: Petri Helenius User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Brad Knowles References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: freebsd-threads@freebsd.org cc: mike cc: Robert Watson cc: Julian Elischer cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 06:22:41 -0000 Brad Knowles wrote: > >> - Disabling HTT >> - Using ADAPTIVE_MUTEXES > > > These both sound like typical improvements, based on what I've > seen on this list. Any ideas on when they might become the default? > HTT defaults to the BIOS setting, which I think is the logical course of action. I donīt think the OS should make assumptions the way or another over what the user has set in the BIOS. (this used not to be the case earlier, and I asked it to default to disable HTT back then) Pete From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 06:37:40 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 725F316A4CE for ; Thu, 17 Jun 2004 06:37:40 +0000 (GMT) Received: from cmsrelay03.mx.net (cmsrelay03.mx.net [165.212.11.112]) by mx1.FreeBSD.org (Postfix) with SMTP id E6F1743D2F for ; Thu, 17 Jun 2004 06:37:39 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from uadvg130.cms.usa.net (165.212.11.130) by cmsoutbound.mx.net with SMTP; 17 Jun 2004 06:35:44 -0000 Received: from optimator.noacks.org [70.240.243.152] by uadvg130.cms.usa.net (ASMTP/noackjr@usa.net) via mtad (C8.MAIN.3.13N) with ESMTP id 825iFqgjQ0291M30; Thu, 17 Jun 2004 06:35:41 GMT X-USANET-Auth: 70.240.243.152 AUTH noackjr@usa.net optimator.noacks.org Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id 0B0216134; Thu, 17 Jun 2004 01:35:41 -0500 (CDT) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 08574-05; Thu, 17 Jun 2004 01:35:39 -0500 (CDT) Received: from compgeek.noacks.org (compgeek [192.168.1.10]) by optimator.noacks.org (Postfix) with ESMTP id CFBFF617E; Thu, 17 Jun 2004 01:35:39 -0500 (CDT) Received: from [127.0.0.1] (localhost.noacks.org [127.0.0.1]) by compgeek.noacks.org (8.12.11/8.12.11) with ESMTP id i5H6ZZBg067028; Thu, 17 Jun 2004 01:35:35 -0500 (CDT) (envelope-from noackjr@alumni.rice.edu) Message-ID: <40D13BB6.3020709@alumni.rice.edu> Date: Thu, 17 Jun 2004 01:35:34 -0500 From: Jon Noack User-Agent: Mozilla Thunderbird 0.6 (X11/20040531) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Brad Knowles References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at noacks.org cc: freebsd-threads@freebsd.org cc: mike cc: Robert Watson cc: Julian Elischer cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 06:37:40 -0000 On 06/17/04 01:07, Brad Knowles wrote: > At 1:31 AM -0400 2004-06-17, Robert Watson wrote: >> - Removing Giant from UNIX domain sockets > > Out of curiosity, is this something that could be relatively safely > done in general? Any ideas on what the plan is for doing this as the > default on -CURRENT? Wasn't this already done? See this commit: http://lists.freebsd.org/pipermail/cvs-src/2004-June/025082.html >> - Disabling HTT - Using ADAPTIVE_MUTEXES > > These both sound like typical improvements, based on what I've seen > on this list. Any ideas on when they might become the default? ADAPTIVE_MUTEXES was already enabled by default on amd64. See this commit (read thread for discussion): http://lists.freebsd.org/pipermail/cvs-src/2004-June/025234.html It appears no comprehensive testing has been done to check whether it really does improve performance. Many signs do point that way, though. >> - Running with SCHED_4BSD instead of SCHED_ULE > > This is the only one that really concerns me. This shows that we > clearly need more work on ULE. Is there one particular thing that we > seem to be tripping up on, or is it a multitude of things? I think I'll switch to 4BSD until I see more work being done on ULE (I read cvs-src as a hobby already so I'll know when to switch back). I noticed a performance drop from 5.2.1-p8 (with 4BSD) to -CURRENT (with ULE) when I upgraded, but I attributed it to other things. However, the system still feels slower than before despite having since disabled INVARIANTS, WITNESS, and userspace malloc debugging flags. Jon Noack From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 07:47:25 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5DA4F16A4CE; Thu, 17 Jun 2004 07:47:25 +0000 (GMT) Received: from rwcrmhc13.comcast.net (rwcrmhc13.comcast.net [204.127.198.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id EFE0B43D39; Thu, 17 Jun 2004 07:47:24 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc13) with ESMTP id <2004061707464701500ss87qe>; Thu, 17 Jun 2004 07:46:47 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id AAA71430; Thu, 17 Jun 2004 00:46:46 -0700 (PDT) Date: Thu, 17 Jun 2004 00:46:45 -0700 (PDT) From: Julian Elischer To: Robert Watson In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 07:47:25 -0000 Robert, when it is going as fast as yu can make it, what is the bottleneck? CPU? disk? net? or just a procedural bottleneck somewhere.? BTW what about libthr? From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 08:36:28 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 33E8F16A4CE; Thu, 17 Jun 2004 08:36:28 +0000 (GMT) Received: from expert.ukrtel.net (expert.ukrtel.net [195.5.6.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id C47EC43D1D; Thu, 17 Jun 2004 08:36:26 +0000 (GMT) (envelope-from astesin@ukrtelecom.net) Received: from hoexc010.ho.ukrtelecom.net (doctor.ukrtel.net [195.5.37.134]) by expert.ukrtel.net (Netscape Messaging Server 3.5) with ESMTP id AAA31EC; Thu, 17 Jun 2004 11:34:38 +0300 Received: by hoexc010.ukrtelecom.net with Internet Mail Service (5.5.2653.19) id ; Thu, 17 Jun 2004 11:36:13 +0300 Message-ID: <1152675CA9EDD71187130002B3CE5ADA08A6F91B@hoexc010.ukrtelecom.net> From: astesin@ukrtelecom.net To: freebsd-threads@freebsd.org, freebsd-current@freebsd.org Date: Thu, 17 Jun 2004 11:36:12 +0300 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="koi8-r" Subject: HA: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 08:36:28 -0000 > the HTT disabling sysctl/tunable with ULE results in a hang. It's not to be news maybe, anyway I confirm this. From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 12:52:02 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1CB1216A4CE; Thu, 17 Jun 2004 12:52:02 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81DED43D41; Thu, 17 Jun 2004 12:52:01 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5HCo9sI029568; Thu, 17 Jun 2004 08:50:09 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5HCo952029565; Thu, 17 Jun 2004 08:50:09 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 08:50:08 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Brad Knowles In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: mike cc: Julian Elischer cc: freebsd-current@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 12:52:02 -0000 On Thu, 17 Jun 2004, Brad Knowles wrote: > At 1:31 AM -0400 2004-06-17, Robert Watson wrote: > > > - Removing Giant from UNIX domain sockets > > Out of curiosity, is this something that could be relatively > safely done in general? Any ideas on what the plan is for doing this as > the default on -CURRENT? I'm in the process of merging the necessary support to CVS, but took a break for a day or two to run some benchmarks and wait for some reviews of specific pieces of it. I'll restart the merge process this evening. You can find more more information at: http://www.watson.org/~robert/freebsd/netperf/ > > - Disabling HTT > > - Using ADAPTIVE_MUTEXES > > These both sound like typical improvements, based on what I've > seen on this list. Any ideas on when they might become the default? There's an interaction with HTT that prevents the OS from disabling HTT when running SCHED_ULE. The benefits and costs of HTT vary based on load; with this particular benchmark, HTT hurts quite a bit. Regarding ADAPTIVE_MUTEXES -- I think we still need more information. I'm convinced that on high-contention local IPC workloads, ADAPTIVE_MUTEXES is a clear win across all other variables -- regardless of scheduler, use of HTT, fine-grained locking, etc. However, I haven't benchmarked more userspace-intensive workloads, and that's where you'd expect it might hurt. ADAPTIVE_MUTEXES should (may?) hurt "less" for high user loads when contention spots are brief, such as in a more fine-grained locking scenario rather than with a Giant lock. After I finish the netperf merge, I'll be doing a set of detailed performance measurements and optimizations and we'll see if that notion plays out or not. It could also well be that ADAPTIVE_MUTEXES are simply a win. There are also two proposed modifications to ADAPTIVE_MUTEXES, one by Bosko Milekic and the other by Alfred Perlstein, and we'll want to look at them as well. > > - Running with SCHED_4BSD instead of SCHED_ULE > > This is the only one that really concerns me. This shows that > we clearly need more work on ULE. Is there one particular thing that we > seem to be tripping up on, or is it a multitude of things? It's not immediately clear; this benchmark is fairly thread-centric, so it may just be that ULE is doing a bad job of sharing a quantum among multiple threads in the same process. Or, we could be looking at a load-balancing issue or other issue. For this benchmark, which involved lots of smaller IPCs across processors, ULE hurt quite a bit compared to 4BSD. On UP, the results were identical. Since this benchmark is a very specific and narrow workload (yet important), I think we don't yet have enough information to make a decision as to whether to keep refining ULE for 5.3, or switch back to 4BSD for 5.3. Presumably the largest single concern is actually whether Jeff or others will have time to work on ULE.y Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 12:54:09 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A7D9C16A4CE; Thu, 17 Jun 2004 12:54:09 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5405743D1D; Thu, 17 Jun 2004 12:54:09 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5HCqClQ029619; Thu, 17 Jun 2004 08:52:12 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5HCqC42029616; Thu, 17 Jun 2004 08:52:12 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 08:52:12 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Petri Helenius In-Reply-To: <40D1386F.1000703@he.iki.fi> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE cc: Brad Knowles cc: mike cc: Julian Elischer cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 12:54:09 -0000 On Thu, 17 Jun 2004, Petri Helenius wrote: > Brad Knowles wrote: >=20 > >> - Disabling HTT > >> - Using ADAPTIVE_MUTEXES > > > > These both sound like typical improvements, based on what I've=20 > > seen on this list. Any ideas on when they might become the default? > > > HTT defaults to the BIOS setting, which I think is the logical course of= =20 > action. I don=B4t think the OS should make assumptions the way or another= =20 > over what the user has set in the BIOS. > (this used not to be the case earlier, and I asked it to default to=20 > disable HTT back then) I agree in principle; however, as a variable in this benchmark, it seems like regardless of scheduler it's a net loss. A question we haven't yet answered is whether (for this workload, or other interesting workloads) it should be a net loss, and therefore whether it's simply our support to HTT is not yet mature enough to offer the theoretical advantage, or whether HTT is just a bad idea. :-) Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 13:02:18 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9489516A4CE; Thu, 17 Jun 2004 13:02:18 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id B139443D41; Thu, 17 Jun 2004 13:02:12 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5HD08t4029767; Thu, 17 Jun 2004 09:00:08 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5HD08oR029764; Thu, 17 Jun 2004 09:00:08 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 09:00:08 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Jon Noack In-Reply-To: <40D13BB6.3020709@alumni.rice.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Brad Knowles cc: mike cc: Julian Elischer cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 13:02:18 -0000 On Thu, 17 Jun 2004, Jon Noack wrote: > On 06/17/04 01:07, Brad Knowles wrote: > > At 1:31 AM -0400 2004-06-17, Robert Watson wrote: > >> - Removing Giant from UNIX domain sockets > > > > Out of curiosity, is this something that could be relatively safely > > done in general? Any ideas on what the plan is for doing this as the > > default on -CURRENT? > > Wasn't this already done? See this commit: > http://lists.freebsd.org/pipermail/cvs-src/2004-June/025082.html The locks have been introduced to protect UNIX domain sockets, but the merge of locking for the cross-protocol socket infrastructure isn't complete yet. This means that if you disable Giant, the UNIX domain bits work fine, but the socket buffers, socket state, and socket event handling will probably corrupt themselves rapidly on SMP. Per an earlier e-mail, I'm working to complete that merge, but took a break to run some benchmarks. Looks for more information here in about a week and a half. > >> - Disabling HTT - Using ADAPTIVE_MUTEXES > > > > These both sound like typical improvements, based on what I've seen > > on this list. Any ideas on when they might become the default? > > ADAPTIVE_MUTEXES was already enabled by default on amd64. See this > commit (read thread for discussion): > http://lists.freebsd.org/pipermail/cvs-src/2004-June/025234.html > > It appears no comprehensive testing has been done to check whether it > really does improve performance. Many signs do point that way, though. Agreed. I think we need to run a broad range of benchmarks across a number of hardware configurations in order to make this decision. Results on a couple of interesting benchmarks, though, are very suggestive, so we should probably do this sooner rather than later. > >> - Running with SCHED_4BSD instead of SCHED_ULE > > > > This is the only one that really concerns me. This shows that we > > clearly need more work on ULE. Is there one particular thing that we > > seem to be tripping up on, or is it a multitude of things? > > I think I'll switch to 4BSD until I see more work being done on ULE (I > read cvs-src as a hobby already so I'll know when to switch back). I > noticed a performance drop from 5.2.1-p8 (with 4BSD) to -CURRENT (with > ULE) when I upgraded, but I attributed it to other things. However, the > system still feels slower than before despite having since disabled > INVARIANTS, WITNESS, and userspace malloc debugging flags. ULE seems to do a very good job of scheduling interactive tasks over other workloads, resulting in a very "snappy" feel on my boxes, despite heavy CPU load from background builds, etc. The workload I looked at had no real "interactive" component, although it was a latency-centric RPC test, so timely hand-off as well as high throughput would be important. I know that Jeff's measurement work on ULE had a substantial focus on deadlines -- whether or not ULE was timely in scheduling tasks, etc, and that he demonstrated that it was much stronger than most other available schedulers in this area. One of the next obvious steps in optimizing either ULE or 4BSD is going to be to spend a lot of time sitting with KTR(4) and looking at context switch traces for "dumb things", such as bouncing between CPUs, rapid switches back and forth, undo multiple wakeups, etc. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 13:09:20 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7DF7916A4D3; Thu, 17 Jun 2004 13:09:20 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 27B5B43D41; Thu, 17 Jun 2004 13:09:20 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5HD7RIJ029951; Thu, 17 Jun 2004 09:07:27 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5HD7RRm029948; Thu, 17 Jun 2004 09:07:27 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 09:07:27 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: astesin@ukrtelecom.net In-Reply-To: <1152675CA9EDD71187130002B3CE5ADA08A6F91B@hoexc010.ukrtelecom.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: jhb@freebsd.org cc: freebsd-threads@freebsd.org Subject: Re: HA: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 13:09:20 -0000 On Thu, 17 Jun 2004 astesin@ukrtelecom.net wrote: > > the HTT disabling sysctl/tunable with ULE results in a hang. > > It's not to be news maybe, anyway I confirm this. Ditto. Ran into this yesterday while doing benchmarking. Jeff mentioned to me last night that this was something he thought John Baldwin was going to fix, but I'm not sure if John knows that. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 14:45:50 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4EC2116A4CE for ; Thu, 17 Jun 2004 14:45:50 +0000 (GMT) Received: from ojoink.com (center.ojoink.com [216.65.123.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1D38543D48 for ; Thu, 17 Jun 2004 14:45:50 +0000 (GMT) (envelope-from amd64list@jpgsworld.com) Received: (qmail 27760 invoked by uid 89); 17 Jun 2004 14:45:26 -0000 Received: from unknown (HELO MAINBX.jpgsworld.com) (amd64list@jpgsworld.com@24.10.96.33) by center.ojoink.com with SMTP; 17 Jun 2004 14:45:26 -0000 Message-Id: <5.2.0.9.2.20040617071734.03251a28@mail.ojoink.com> X-Sender: amd64list@jpgsworld.com@mail.ojoink.com X-Mailer: QUALCOMM Windows Eudora Version 5.2.0.9 Date: Thu, 17 Jun 2004 07:45:20 -0700 To: freebsd-threads@freebsd.org From: JG In-Reply-To: <40D13BB6.3020709@alumni.rice.edu> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 14:45:50 -0000 On FreeBSD-AMD64.... I've just confirmed that a kernel compiled with ULE will cause performance locks during the super-smack mysql benchmarks. ("locks" being - resulting in highly erratic q/ps results for the same test... from 50qps to 3000qps and anywhere in between) From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 17:24:46 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6407116A4CE for ; Thu, 17 Jun 2004 17:24:46 +0000 (GMT) Received: from cmsrelay02.mx.net (cmsrelay02.mx.net [165.212.11.111]) by mx1.FreeBSD.org (Postfix) with SMTP id D36D343D4C for ; Thu, 17 Jun 2004 17:24:45 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from uadvg130.cms.usa.net (165.212.11.130) by cmsoutbound.mx.net with SMTP; 17 Jun 2004 17:24:38 -0000 Received: from optimator.noacks.org [70.240.231.39] by uadvg130.cms.usa.net (ASMTP/noackjr@usa.net) via mtad (C8.MAIN.3.13N) with ESMTP id 826iFqRyj0380M30; Thu, 17 Jun 2004 17:24:35 GMT X-USANET-Auth: 70.240.231.39 AUTH noackjr@usa.net optimator.noacks.org Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id 178286153; Thu, 17 Jun 2004 12:24:35 -0500 (CDT) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 51460-08; Thu, 17 Jun 2004 12:24:33 -0500 (CDT) Received: from compgeek.noacks.org (compgeek [192.168.1.10]) by optimator.noacks.org (Postfix) with ESMTP id D0D5C6140; Thu, 17 Jun 2004 12:24:33 -0500 (CDT) Received: from [127.0.0.1] (localhost.noacks.org [127.0.0.1]) by compgeek.noacks.org (8.12.11/8.12.11) with ESMTP id i5HHOXsx061453; Thu, 17 Jun 2004 12:24:33 -0500 (CDT) (envelope-from noackjr@alumni.rice.edu) Message-ID: <40D1D3D1.9010209@alumni.rice.edu> Date: Thu, 17 Jun 2004 12:24:33 -0500 From: Jon Noack User-Agent: Mozilla Thunderbird 0.6 (X11/20040531) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at noacks.org cc: Brad Knowles cc: mike cc: Julian Elischer cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 17:24:46 -0000 On 06/17/04 08:00, Robert Watson wrote: > On Thu, 17 Jun 2004, Jon Noack wrote: >> On 06/17/04 01:07, Brad Knowles wrote: >>> At 1:31 AM -0400 2004-06-17, Robert Watson wrote: >>>> - Running with SCHED_4BSD instead of SCHED_ULE >>> >>> This is the only one that really concerns me. This shows that we >>> clearly need more work on ULE. Is there one particular thing >>> that we seem to be tripping up on, or is it a multitude of >>> things? >> >> I think I'll switch to 4BSD until I see more work being done on ULE >> (I read cvs-src as a hobby already so I'll know when to switch >> back). I noticed a performance drop from 5.2.1-p8 (with 4BSD) to >> -CURRENT (with ULE) when I upgraded, but I attributed it to other >> things. However, the system still feels slower than before despite >> having since disabled INVARIANTS, WITNESS, and userspace malloc >> debugging flags. > > ULE seems to do a very good job of scheduling interactive tasks over > other workloads, resulting in a very "snappy" feel on my boxes, > despite heavy CPU load from background builds, etc. The workload I > looked at had no real "interactive" component, although it was a > latency-centric RPC test, so timely hand-off as well as high > throughput would be important. I know that Jeff's measurement work > on ULE had a substantial focus on deadlines -- whether or not ULE was > timely in scheduling tasks, etc, and that he demonstrated that it was > much stronger than most other available schedulers in this area. Well, I switched to 4BSD and noticed this right away. I occasionally get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit ridiculous. There's a PR open about it (can't remember which off the top of my head), but sound will skip and slow down (play at half speed or slower). I know the sound locking is not all it should be, so this may not be a representative test. Regardless, my workstations are back on ULE... Jon [1] Often when using tar or bsdtar. To reproduce: cd /usr/ports/www/firefox; make extract From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 19:04:50 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C6CDA16A4CE for ; Thu, 17 Jun 2004 19:04:50 +0000 (GMT) Received: from mail2.speakeasy.net (mail2.speakeasy.net [216.254.0.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA59E43D55 for ; Thu, 17 Jun 2004 19:04:50 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 6116 invoked from network); 17 Jun 2004 19:02:20 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 17 Jun 2004 19:02:20 -0000 Received: from 10.50.41.233 (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i5HJ2Gc5081771; Thu, 17 Jun 2004 15:02:16 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: Robert Watson Date: Thu, 17 Jun 2004 15:03:03 -0400 User-Agent: KMail/1.6 References: In-Reply-To: MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200406171503.03031.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: freebsd-current@FreeBSD.org cc: astesin@ukrtelecom.net cc: freebsd-threads@FreeBSD.org Subject: Re: HA: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 19:04:50 -0000 On Thursday 17 June 2004 09:07 am, Robert Watson wrote: > On Thu, 17 Jun 2004 astesin@ukrtelecom.net wrote: > > > the HTT disabling sysctl/tunable with ULE results in a hang. > > > > It's not to be news maybe, anyway I confirm this. > > Ditto. Ran into this yesterday while doing benchmarking. Jeff mentioned > to me last night that this was something he thought John Baldwin was going > to fix, but I'm not sure if John knows that. Oh, hrmm. I can put it on my todo list. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 19:35:38 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DE69716A4CE; Thu, 17 Jun 2004 19:35:38 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5DE1543D4C; Thu, 17 Jun 2004 19:35:38 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5HJJ8SZ046360; Thu, 17 Jun 2004 15:19:08 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5HJJ72I046357; Thu, 17 Jun 2004 15:19:07 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 15:19:07 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Jon Noack In-Reply-To: <40D1D3D1.9010209@alumni.rice.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Brad Knowles cc: mike cc: Julian Elischer cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 19:35:39 -0000 On Thu, 17 Jun 2004, Jon Noack wrote: > > ULE seems to do a very good job of scheduling interactive tasks over > > other workloads, resulting in a very "snappy" feel on my boxes, > > despite heavy CPU load from background builds, etc. The workload I > > looked at had no real "interactive" component, although it was a > > latency-centric RPC test, so timely hand-off as well as high > > throughput would be important. I know that Jeff's measurement work > > on ULE had a substantial focus on deadlines -- whether or not ULE was > > timely in scheduling tasks, etc, and that he demonstrated that it was > > much stronger than most other available schedulers in this area. > > Well, I switched to 4BSD and noticed this right away. I occasionally > get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit > ridiculous. There's a PR open about it (can't remember which off the > top of my head), but sound will skip and slow down (play at half speed > or slower). I know the sound locking is not all it should be, so this > may not be a representative test. Regardless, my workstations are back > on ULE... Hrm. So, I'm not really a scheduler guy, but I have some ideas about how to investigate what's going on. If you feel like playing with kernel tracing facilities, there are some really neat things you can do with ktr(4). It allows you took hook context switch and interrupt delivery events and dump a trace to userspace. One possible cause of the symptoms you're seeing is poor interrupt handling response time -- possibly a failure to schedule the ithread in a timely manner. It would be quite interesting to compare the latency in scheduling (and running) the ithread across ULE and 4BSD and see how well each is managing to do in getting it running quickly. I'm not sure there's a tutorial on how to get KTR(4) running -- I found the man page helpful, but a little confusing. What you'll want to do, assuming you're willing to get into this, is turn on KTR tracing with KTR_INTR and KTR_PROC. You'll need to set the flags in the kernel compile, and then set them using a sysctl. You can use ktrdump to dump a record trace -- I'd use a number of flags present to add timestamps, cpu numbers if SMP, etc. With some post-processing, it should be possible to generate a distribution of "interrupt time" to "running interrupt handler". It would be useful to demonstrate to oneself that ithreads are running quickly, preempting things they need to, etc. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research From owner-freebsd-threads@FreeBSD.ORG Thu Jun 17 22:35:32 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4454716A4CE for ; Thu, 17 Jun 2004 22:35:32 +0000 (GMT) Received: from wm1.todt.org (wm2.todt.org [207.218.27.21]) by mx1.FreeBSD.org (Postfix) with SMTP id 0430043D31 for ; Thu, 17 Jun 2004 22:35:31 +0000 (GMT) (envelope-from jachmann@unitix.org) Received: (qmail 4712 invoked from network); 17 Jun 2004 22:31:36 -0000 Received: from unknown (HELO habbib.dhis.org) (82.82.135.98) by 0 with SMTP; 17 Jun 2004 22:31:36 -0000 Received: (qmail 1844 invoked by uid 1001); 17 Jun 2004 22:34:51 -0000 Date: Fri, 18 Jun 2004 00:34:51 +0200 From: Christian Jachmann To: Martin Blapp Message-ID: <20040617223451.GA1800@habbib.thc> References: <01c301c45239$940781f0$6401a8c0@animal> <20040615170316.T84297@cvs.imp.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20040615170316.T84297@cvs.imp.ch> User-Agent: Mutt/1.5.6i cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 22:35:32 -0000 On Tue, Jun 15, 2004 at 05:09:51PM +0200, Martin Blapp wrote: > > set-variable = key_buffer_size=16M > set-variable = table_cache=256 > set-variable = record_buffer=1M > set-variable = sort_buffer=16M Why only so uninspired values? > avail memory = 2095964160 (1998 MB) there seems to be lot of memory? -- Christian Jachmann From owner-freebsd-threads@FreeBSD.ORG Fri Jun 18 07:35:29 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DCFD216A4D8; Fri, 18 Jun 2004 07:35:26 +0000 (GMT) Received: from mailout1.pacific.net.au (mailout1.pacific.net.au [61.8.0.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5620343D45; Fri, 18 Jun 2004 07:35:26 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au [61.8.0.86])i5I7YC4u031254; Fri, 18 Jun 2004 17:34:12 +1000 Received: from gamplex.bde.org (katana.zip.com.au [61.8.7.246]) i5I7Y6hp019462; Fri, 18 Jun 2004 17:34:07 +1000 Date: Fri, 18 Jun 2004 17:34:05 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Robert Watson In-Reply-To: Message-ID: <20040618164402.P3080@gamplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org cc: mike cc: Julian Elischer cc: Jon Noack cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2004 07:35:29 -0000 On Thu, 17 Jun 2004, Robert Watson wrote: > On Thu, 17 Jun 2004, Jon Noack wrote: > > > > ... I know that Jeff's measurement work > > > on ULE had a substantial focus on deadlines -- whether or not ULE was > > > timely in scheduling tasks, etc, and that he demonstrated that it was > > > much stronger than most other available schedulers in this area. > > > > Well, I switched to 4BSD and noticed this right away. I occasionally > > get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit > > ridiculous. There's a PR open about it (can't remember which off the > > top of my head), but sound will skip and slow down (play at half speed > > or slower). I know the sound locking is not all it should be, so this > > may not be a representative test. Regardless, my workstations are back > > on ULE... > > Hrm. So, I'm not really a scheduler guy, but I have some ideas about how > to investigate what's going on. If you feel like playing with kernel > tracing facilities, there are some really neat things you can do with > ktr(4). It allows you took hook context switch and interrupt delivery > events and dump a trace to userspace. Interrupt delivery if well known to be broken (interrupts that can't switch to their ithread immediately, perhaps because they are scheduled while in a critical region, just set a flag, and the flag is not checked until return to user mode which may happen a long time later, so the current current thread keeps running until it either gives up control or another interrupt that can be switched to immediately is recieved (then the flag becomes irrelevant and the highest priority ithread is switched too; it may even be the one for the first interrupt). The effects of this can be seen in ping latency. For "ping -fq -c1000000 besplex" over 100Mb/S ethernet using an old version of ping that I keep for this benchmark: Under -current on an unloaded Celeron 366: --- besplex.bde.org ping statistics --- 100002 packets transmitted, 100000 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.172/0.393/20.336/0.191 ms 13.60 real 0.74 user 7.66 sys The latency is sometimes 20ms (2 clock ticks). Under my version of -current (which is missing the bug) on the same machine: --- besplex.bde.org ping statistics --- 100001 packets transmitted, 100000 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.156/0.204/2.449/0.099 ms 10.92 real 0.80 user 5.91 sys The worst-case latency is still bad, but is much smaller, and the real and system times are much smaller too (the latter may not be significant -- flood pings sometimes give strange behaviour (capture effect?) where synchronization gives a larger latency but a smaller real time. But here the latency is smaller too. > One possible cause of the symptoms you're seeing is poor interrupt > handling response time -- possibly a failure to schedule the ithread in a > timely manner. It would be quite interesting to compare the latency in > scheduling (and running) the ithread across ULE and 4BSD and see how well > each is managing to do in getting it running quickly. I'm not sure The scheduler really shouldn't affect scheduling of ithreads. The 4BSD scheduler just picks the highest priority ithread (round robin among equal priorities) and runs it, and doing much more than that would be wrong. ULE may be reducing the effect of bugs by scheduling more often. Or perhaps it is better for sound because it handles competition between user threads and ithreads better. Ithreads are supposed to always have higher priority than user threads; however, the 4BSD scheduler still has the bugfeature of not enforcing this, so that when a user thread in kernel mode wakes up with a high kernel priority, it essentially keeps the kernel priority when it returns to user mode (until it gives up control or is interrupted). This bugfeature seems to be more responsible for the 4BSD scheduler's good interactive behaviour than anything it does intentionally. I believe the ULE scheduler does similar things intentionally. > there's a tutorial on how to get KTR(4) running -- I found the man page > helpful, but a little confusing. What you'll want to do, assuming you're > willing to get into this, is turn on KTR tracing with KTR_INTR and > KTR_PROC. You'll need to set the flags in the kernel compile, and then > set them using a sysctl. You can use ktrdump to dump a record trace -- > I'd use a number of flags present to add timestamps, cpu numbers if SMP, > etc. With some post-processing, it should be possible to generate a > distribution of "interrupt time" to "running interrupt handler". > > It would be useful to demonstrate to oneself that ithreads are running > quickly, preempting things they need to, etc. I think listening to the sound is the actually easiest way to see if interrupt latency is not too large, at least if 100uS (corresponding to 10kHz) is not too large :-). Too bad I can't hear it well enough. When I fixed interrupt problems for 386BSD-0.0, I set up a visual indicator using about 10 words of VGA memory at the top of the screen. It was updated on every change to the interrupt masks. Latency bugs in the 100uS range are far too short to see, but I could easily see some larger bugs. Bruce From owner-freebsd-threads@FreeBSD.ORG Fri Jun 18 08:54:52 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7AAEE16A4CE; Fri, 18 Jun 2004 08:54:52 +0000 (GMT) Received: from exchhz01.viatech.com.cn (ip-40-162-97-218.anlai.com [218.97.162.40]) by mx1.FreeBSD.org (Postfix) with ESMTP id C0DEE43D2F; Fri, 18 Jun 2004 08:54:45 +0000 (GMT) (envelope-from davidxu@freebsd.org) Received: from freebsd.org (DAVIDWNT [10.4.1.99]) by exchhz01.viatech.com.cn with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id NAMYWJJZ; Fri, 18 Jun 2004 16:54:32 +0800 Message-ID: <40D2AE58.3040303@freebsd.org> Date: Fri, 18 Jun 2004 16:56:56 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-current@freebsd.org cc: mike cc: Julian Elischer cc: Jon Noack cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2004 08:54:52 -0000 When this pthread program runs under ULE, I can not interrupt it pressing ctrl+c has no effect, I have this problem for a long time. http://people.freebsd.org/~davidxu/kse/locktest.c David Xu Robert Watson wrote: > ULE seems to do a very good job of scheduling interactive tasks over > other > workloads, resulting in a very "snappy" feel on my boxes, despite heavy > CPU load from background builds, etc. The workload I looked at had no > real "interactive" component, although it was a latency-centric RPC > test, > so timely hand-off as well as high throughput would be important. I > know > that Jeff's measurement work on ULE had a substantial focus on deadlines > -- whether or not ULE was timely in scheduling tasks, etc, and that he > demonstrated that it was much stronger than most other available > schedulers in this area. > > One of the next obvious steps in optimizing either ULE or 4BSD is going > to > be to spend a lot of time sitting with KTR(4) and looking at context > switch traces for "dumb things", such as bouncing between CPUs, rapid > switches back and forth, undo multiple wakeups, etc. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to > "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Fri Jun 18 09:21:11 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 666C016A4CF for ; Fri, 18 Jun 2004 09:21:11 +0000 (GMT) Received: from web14107.mail.yahoo.com (web14107.mail.yahoo.com [216.136.172.137]) by mx1.FreeBSD.org (Postfix) with SMTP id 46D6E43D45 for ; Fri, 18 Jun 2004 09:21:11 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Message-ID: <20040618092009.67564.qmail@web14107.mail.yahoo.com> Received: from [194.248.174.58] by web14107.mail.yahoo.com via HTTP; Fri, 18 Jun 2004 11:20:09 CEST Date: Fri, 18 Jun 2004 11:20:09 +0200 (CEST) From: =?iso-8859-1?q?Claus=20Guttesen?= To: David Xu , Robert Watson In-Reply-To: <40D2AE58.3040303@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit cc: freebsd-current@freebsd.org cc: Julian Elischer cc: mike cc: freebsd-threads@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2004 09:21:11 -0000 > When this pthread program runs under ULE, I can not > interrupt it > pressing ctrl+c has no effect, I have this problem > for a long time. > http://people.freebsd.org/~davidxu/kse/locktest.c > David Xu frodo~/temp%>cc -O2 -Wall -o pthread_lock pthread_lock.c -lpthread frodo~/temp%>./pthread_lock ^C frodo~/temp%>echo $? 130 Pressing ctrl-c worked here. ULE, current from June 15'th. regards Claus Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan From owner-freebsd-threads@FreeBSD.ORG Fri Jun 18 14:32:16 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 75BFB16A4E9 for ; Fri, 18 Jun 2004 14:32:13 +0000 (GMT) Received: from web14101.mail.yahoo.com (web14101.mail.yahoo.com [216.136.172.131]) by mx1.FreeBSD.org (Postfix) with SMTP id 514F943D55 for ; Fri, 18 Jun 2004 14:32:13 +0000 (GMT) (envelope-from cguttesen@yahoo.dk) Message-ID: <20040618143134.4512.qmail@web14101.mail.yahoo.com> Received: from [194.248.174.58] by web14101.mail.yahoo.com via HTTP; Fri, 18 Jun 2004 16:31:34 CEST Date: Fri, 18 Jun 2004 16:31:34 +0200 (CEST) From: =?iso-8859-1?q?Claus=20Guttesen?= To: Daniel Eischen In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org cc: Robert Watson cc: David Xu cc: mike cc: Julian Elischer Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2004 14:32:16 -0000 > > frodo~/temp%>cc -O2 -Wall -o pthread_lock > > pthread_lock.c -lpthread > > > > frodo~/temp%>./pthread_lock > > ^C > > frodo~/temp%>echo $? > > 130 > > > > Pressing ctrl-c worked here. ULE, current from > June > > 15'th. > > SMP or UP? SMP, ACPI APIC Table: , dual PIII, 1 GB of RAM. regards Claus Yahoo! Mail (http://dk.mail.yahoo.com) - Gratis: 6 MB lagerplads, spamfilter og virusscan From owner-freebsd-threads@FreeBSD.ORG Fri Jun 18 14:54:03 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 87FCA16A4CE; Fri, 18 Jun 2004 14:54:03 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9F71243D55; Fri, 18 Jun 2004 14:54:02 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i5IEAAqb025151; Fri, 18 Jun 2004 10:10:10 -0400 (EDT) Date: Fri, 18 Jun 2004 10:10:10 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: =?iso-8859-1?q?Claus=20Guttesen?= In-Reply-To: <20040618092009.67564.qmail@web14107.mail.yahoo.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org cc: Robert Watson cc: David Xu cc: mike cc: Julian Elischer Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jun 2004 14:54:03 -0000 On Fri, 18 Jun 2004, [iso-8859-1] Claus Guttesen wrote: > > When this pthread program runs under ULE, I can not > > interrupt it > > pressing ctrl+c has no effect, I have this problem > > for a long time. > > http://people.freebsd.org/~davidxu/kse/locktest.c > > David Xu > > frodo~/temp%>cc -O2 -Wall -o pthread_lock > pthread_lock.c -lpthread > > frodo~/temp%>./pthread_lock > ^C > frodo~/temp%>echo $? > 130 > > Pressing ctrl-c worked here. ULE, current from June > 15'th. SMP or UP? -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Sat Jun 19 14:04:47 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3783116A4CE; Sat, 19 Jun 2004 14:04:47 +0000 (GMT) Received: from meitner.wh.uni-dortmund.de (meitner.wh.Uni-Dortmund.DE [129.217.129.133]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8C4F643D2D; Sat, 19 Jun 2004 14:04:46 +0000 (GMT) (envelope-from michaelnottebrock@gmx.net) Received: from lofi.dyndns.org (pc2-105.intern.meitner [10.3.12.105]) by meitner.wh.uni-dortmund.de (Postfix) with ESMTP id 2D4F9167522; Sat, 19 Jun 2004 16:04:31 +0200 (CEST) Received: from [192.168.8.4] (lofi@kiste.my.domain [192.168.8.4]) (authenticated bits=0) by lofi.dyndns.org (8.12.10/8.12.10) with ESMTP id i5JE4Qxm016639 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Sat, 19 Jun 2004 16:04:27 +0200 (CEST) (envelope-from michaelnottebrock@gmx.net) From: Michael Nottebrock To: freebsd-threads@freebsd.org Date: Sat, 19 Jun 2004 16:04:22 +0200 User-Agent: KMail/1.6.2 MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Boundary-03=_qfE1Am/nZCnkLgS"; charset="us-ascii" Content-Transfer-Encoding: 7bit Message-Id: <200406191604.26550.michaelnottebrock@gmx.net> X-Virus-Scanned: by amavisd-new cc: kde@freebsd.org cc: eti@erata.net Subject: Fwd: [kde-freebsd] Problem with qt on freebsd X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jun 2004 14:04:47 -0000 --Boundary-03=_qfE1Am/nZCnkLgS Content-Type: multipart/mixed; boundary="Boundary-01=_mfE1AoEOjaB5jhH" Content-Transfer-Encoding: 7bit Content-Disposition: inline --Boundary-01=_mfE1AoEOjaB5jhH Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline I and others have been able to reproduce the problem on -STABLE and 5.2.1-R= ,=20 but not on -CURRENT (my guess is it's probably reproducable on -CURRENT as= =20 well if qt is linked against libc_r instead of libpthread). Backtraces yield nothing despite full debugging enabled in QT and KDE (you = can=20 enable debugging in the QT port by compiling with -DWANT_QT_DEBUG): (gdb) run Starting program: /usr/home/lofi/./qt-test Program received signal SIGSEGV, Segmentation fault. 0x28de44f8 in __tcf_0 () at kstyle.cpp:295 Could this be some libc_r bug? It's a little heisenbug-ish, not all QT styl= es=20 seem to trigger this reliably, perhaps it's even necessary to have some KDE= =20 styles (i.e. kdelibs) installed, I couldn't check yet. =2D-=20 ,_, | Michael Nottebrock | lofi@freebsd.org (/^ ^\) | FreeBSD - The Power to Serve | http://www.freebsd.org \u/ | K Desktop Environment on FreeBSD | http://freebsd.kde.org --Boundary-01=_mfE1AoEOjaB5jhH Content-Type: message/rfc822; name="forwarded message" Content-Transfer-Encoding: 7bit Content-Description: etila rootila : [kde-freebsd] Problem with qt on freebsd Content-Disposition: inline Return-Path: Received: from pop.gmx.net (lofi@localhost [127.0.0.1]) by lofi.dyndns.org (8.12.10/8.12.10) with ESMTP id i5J9F5xo013083 for ; Sat, 19 Jun 2004 11:15:10 +0200 (CEST) (envelope-from kde-freebsd-bounces@freebsd.kde.org) X-Flags: 0000 Delivered-To: GMX delivery to michaelnottebrock@gmx.net Received: (qmail 3670 invoked by uid 65534); 19 Jun 2004 09:11:56 -0000 Received: from h2.liquidneon.com (EHLO h2.liquidneon.com) (216.38.206.182) by mx0.gmx.net (mx016) with SMTP; 19 Jun 2004 11:11:56 +0200 Received: (qmail 78363 invoked from network); 19 Jun 2004 09:11:53 -0000 Received: from h2.liquidneon.com (216.38.206.182) by h2.liquidneon.com with SMTP; 19 Jun 2004 09:11:53 -0000 Received: (qmail 78353 invoked by uid 98); 19 Jun 2004 09:11:49 -0000 Received: from etishor@yahoo.com by h2.liquidneon.com by uid 82 with qmail-scanner-1.20 (clamuko: 0.65. Clear:RC:0(216.136.204.119):. Processed in 0.071322 secs); 19 Jun 2004 09:11:49 -0000 X-Qmail-Scanner-Mail-From: etishor@yahoo.com via h2.liquidneon.com X-Qmail-Scanner: 1.20 (Clear:RC:0(216.136.204.119):. Processed in 0.071322 secs) Received: from mx2.freebsd.org (216.136.204.119) by h2.liquidneon.com with SMTP; 19 Jun 2004 09:11:49 -0000 Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18]) by mx2.freebsd.org (Postfix) with ESMTP id 7F7E255457 for ; Sat, 19 Jun 2004 09:11:48 +0000 (GMT) (envelope-from etishor@yahoo.com) Received: by hub.freebsd.org (Postfix) id 7BFA516A4CF; Sat, 19 Jun 2004 09:11:48 +0000 (GMT) Delivered-To: kde@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7A24616A4CE for ; Sat, 19 Jun 2004 09:11:48 +0000 (GMT) Received: from web60608.mail.yahoo.com (web60608.mail.yahoo.com [216.109.119.82]) by mx1.FreeBSD.org (Postfix) with SMTP id 2855843D39 for ; Sat, 19 Jun 2004 09:11:48 +0000 (GMT) (envelope-from etishor@yahoo.com) Message-ID: <20040619091142.72297.qmail@web60608.mail.yahoo.com> Received: from [65.75.178.20] by web60608.mail.yahoo.com via HTTP; Sat, 19 Jun 2004 02:11:42 PDT Date: Sat, 19 Jun 2004 02:11:42 -0700 (PDT) From: etila rootila To: kde@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: [kde-freebsd] Problem with qt on freebsd X-BeenThere: kde-freebsd@freebsd.kde.org X-Mailman-Version: 2.1.3 Precedence: list List-Id: KDE/FreeBSD QA/Development Team List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: kde-freebsd-bounces@freebsd.kde.org Errors-To: kde-freebsd-bounces@freebsd.kde.org X-GMX-Antivirus: -1 (not scanned, may not use virus scanner) X-GMX-Antispam: -2 (not scanned, spam filter disabled) X-Virus-Scanned: by amavisd-new X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on lofi.dyndns.org X-Spam-Status: No, hits=-4.9 required=3.0 tests=BAYES_00 autolearn=ham version=2.63 X-Spam-Level: X-UID: 6995 hi, There seems to be a problem with QListView widgtet when compiling & runing on freebsd. I found a report of a similar problema on the mailing lists from trolltech: http://lists.trolltech.com/qt-interest/ 2004-04/thread00962-0.html The problema is that the application containing a QListView witch has been populated with items segfaults on exit. example: #include #include int main( int argc, char ** argv ) { QApplication a( argc, argv ); QListView w; w.addColumn("column"); QListViewItem *item; item=new QListViewItem(&w); item->setText(0,"a"); item=new QListViewItem(item); item->setText(0,"b"); a.setMainWidget(&w); w.show(); return a.exec(); } if you compile and run the application, and if you close the main window without opening the first item nothing happens. The problem is if you close the application after opening the first item then it segfaults. I'm using freebsd 4.10-stable with: qt-3.3.2_2 bash-2.05b$ gcc -v Using builtin specs. gcc version 2.95.4 20020320 [FreeBSD] ai've tryed compiling with gcc33 but the results are the same. bash-2.05b$ gcc33 -v Reading specs from /usr/local/lib/gcc-lib/i386-portbld-freebsd4.9/3.3.4/specs Configured with: ./..//gcc-3.3-20040505/configure --disable-nls --with-system-zlib --program-suffix=33 --with-gxx-include-dir=/usr/local/lib/gcc-lib/i386-portbld-freebsd4.9/3.3.4/include/c++/ --disable-shared --prefix=/usr/local i386-portbld-freebsd4.9 Thread model: posix gcc version 3.3.4 20040505 (prerelease) [FreeBSD] The problem does not apear on linux or windows ... so i guess it's freebsd specific. (i'm not 100% sure but i think the problem exists on 5.X too) Also i'm not sure if it's a qt problem or a libc problem. If you answer please cc to eti@erata.net (i can't email from there cos the smtp servre does not have a reverse and freebsd's MX does not accept mail from there. thanks. Regards, Iulian margarintescu __________________________________ Do you Yahoo!? Read only the mail you want - Yahoo! Mail SpamGuard. http://promotions.yahoo.com/new_mail _______________________________________________ kde-freebsd mailing list kde-freebsd@freebsd.kde.org http://freebsd.kde.org/mailman/listinfo/kde-freebsd --Boundary-01=_mfE1AoEOjaB5jhH-- --Boundary-03=_qfE1Am/nZCnkLgS Content-Type: application/pgp-signature Content-Description: signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQBA1EfqXhc68WspdLARAuo9AJ4i6clyd0xsZxYEVL56fYUYZthOPgCfa2p+ 2VNrIDlPzAHhRniT31qgMf0= =zsBG -----END PGP SIGNATURE----- --Boundary-03=_qfE1Am/nZCnkLgS-- From owner-freebsd-threads@FreeBSD.ORG Sat Jun 19 16:13:32 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F170316A4CE; Sat, 19 Jun 2004 16:13:32 +0000 (GMT) Received: from meitner.wh.uni-dortmund.de (meitner.wh.Uni-Dortmund.DE [129.217.129.133]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9294F43D2F; Sat, 19 Jun 2004 16:13:32 +0000 (GMT) (envelope-from michaelnottebrock@gmx.net) Received: from lofi.dyndns.org (pc2-105.intern.meitner [10.3.12.105]) by meitner.wh.uni-dortmund.de (Postfix) with ESMTP id D7E691675A8; Sat, 19 Jun 2004 18:12:49 +0200 (CEST) Received: from kiste.my.domain (kiste.my.domain [192.168.8.4]) (authenticated bits=0) by lofi.dyndns.org (8.12.10/8.12.10) with ESMTP id i5JGCgxm018138 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Sat, 19 Jun 2004 18:12:44 +0200 (CEST) (envelope-from michaelnottebrock@gmx.net) From: Michael Nottebrock To: kde@freebsd.org Date: Sat, 19 Jun 2004 18:12:38 +0200 User-Agent: KMail/1.6.2 References: <200406191604.26550.michaelnottebrock@gmx.net> <200406191611.26624.raabe@kde.org> <200406191635.24187.raabe@kde.org> In-Reply-To: <200406191635.24187.raabe@kde.org> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Boundary-02=_5XG1AC9lPlok/3h"; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <200406191812.41970.michaelnottebrock@gmx.net> X-Virus-Scanned: by amavisd-new cc: freebsd-threads@freebsd.org Subject: Re: Fwd: [kde-freebsd] Problem with qt on freebsd X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jun 2004 16:13:33 -0000 --Boundary-02=_5XG1AC9lPlok/3h Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Saturday 19 June 2004 16:35, Frerich Raabe wrote: > I attached a simple testcase which demonstrates the problem using nothing > but standard C and C++ library calls, so you don't have to install KDE to > reproduce this crash. Note that making "s" (in plugin.cc) a normal (auto) > variable and not static does not trigger the crash. I'm going to toss this onto freebsd-current@ ... it still works for me on=20 recent -CURRENT, but not on 5.2.1-R, no clue what's causing it. =2D-=20 ,_, | Michael Nottebrock | lofi@freebsd.org (/^ ^\) | FreeBSD - The Power to Serve | http://www.freebsd.org \u/ | K Desktop Environment on FreeBSD | http://freebsd.kde.org --Boundary-02=_5XG1AC9lPlok/3h Content-Type: application/pgp-signature Content-Description: signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQBA1GX5Xhc68WspdLARAi7nAJ0Uil6tM5D9TJKqtfCSpMMDkuBpJQCdEwDJ eQX38WXCtaFeA785BfXEaEw= =S9JK -----END PGP SIGNATURE----- --Boundary-02=_5XG1AC9lPlok/3h--