From owner-freebsd-threads@FreeBSD.ORG Sun Oct 5 03:44:56 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A4F21065688; Sun, 5 Oct 2008 03:44:56 +0000 (UTC) (envelope-from dixit@netapp.com) Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by mx1.freebsd.org (Postfix) with ESMTP id 130A98FC0A; Sun, 5 Oct 2008 03:44:55 +0000 (UTC) (envelope-from dixit@netapp.com) X-IronPort-AV: E=Sophos;i="4.33,362,1220252400"; d="scan'208";a="67197082" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 04 Oct 2008 20:44:55 -0700 Received: from dixit-lxp.nane.netapp.com ([10.30.32.59]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id m953isgZ001777; Sat, 4 Oct 2008 20:44:55 -0700 (PDT) Message-ID: <48E83836.6090305@netapp.com> Date: Sat, 04 Oct 2008 23:44:54 -0400 From: Amol Dixit User-Agent: Thunderbird 2.0.0.17 (X11/20080914) MIME-Version: 1.0 To: David Xu References: <48DAABAF.9030709@netapp.com> <48DB0950.3060405@freebsd.org> In-Reply-To: <48DB0950.3060405@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: SIGTRAP during thr_new syscall X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Oct 2008 03:44:56 -0000 David, I tried your patch, but it doesn't seem to fix the problem because even though _pthread_create() calls thr_new(), this system call is never made for the first thread created. Breakpoint set in kernel code of thr_new is hit only for the initial_thread (_libpthread_init())...skips the first thread...and then is hit again for 2nd thread onwards. This patch may be necessary, but it doesn't fix my issue of SIGTRAP on thread_start(). Any clue why the first thread created using pthread_create() doesn't enter thr_new() syscall inside the kernel at all? I get the same SIGTRAP if I single-step through this thr_new() line in _pthread_create() using _local_ gdb. I am running freebsd6.0 configured for SYSTEM_SCOPE threading (1:1). Thanks, Amol David Xu wrote: > Amol Dixit wrote: >> Hi, >> I am seeing an unexpected SIGTRAP being reported to gdbserver when >> the debugged process creates a new thread via the _pthread_create() >> call of libthr library. [libthr/thread/thr_create.c,v 1.22.4.1, >> Freebsd 6.0] >> Gdbserver has internally set a breakpoint on address of >> _thread_bp_create() so that it gets notified on thread creation and >> is expecting a SIGTRAP at address (stop pc) of _thread_bp_create(). >> But instead SIGTRAP happens as a side-effect of thr_new() system call >> and the stop pc at that point is that of routine thread_start() which >> is the starting function of new thread. So gdbserver cannot match >> expected breakpoint (ie. _thread_bp_create) and is confused. >> For testing purpose, if I call _thread_bp_create() before thr_new() >> in _pthread_create(), I get the _expected_ SIGTRAP with address of >> _thread_bp_create. But that is not the fix. >> Does anyone have any idea about this SIGTRAP being reported to >> tracing process gdbserver as part of thr_new? Where is it originating >> from and why? >> Thanks, >> Amol >> > > I found kernel clears trap flag for new process but not for new thread > in cpu_fork(), you may try following patch: > > Index: i386/i386/vm_machdep.c > =================================================================== > --- i386/i386/vm_machdep.c (revision 183337) > +++ i386/i386/vm_machdep.c (working copy) > @@ -413,6 +413,15 @@ > bcopy(td0->td_frame, td->td_frame, sizeof(struct trapframe)); > > /* > + * If the current thread has the trap bit set (i.e. a debugger had > + * single stepped the process to the system call), we need to clear > + * the trap flag from the new frame. Otherwise, the new thread will > + * receive a (likely unexpected) SIGTRAP when it executes the first > + * instruction after returning to userland. > + */ > + td->td_frame->tf_eflags &= ~PSL_T; > + > + /* > * Set registers for trampoline to user mode. Leave space for the > * return address on stack. These are the kernel mode register > values. > */ From owner-freebsd-threads@FreeBSD.ORG Mon Oct 6 11:07:03 2008 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75C19106568B for ; Mon, 6 Oct 2008 11:07:03 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 645F08FC1C for ; Mon, 6 Oct 2008 11:07:03 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m96B73lM035643 for ; Mon, 6 Oct 2008 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m96B72cU035639 for freebsd-threads@FreeBSD.org; Mon, 6 Oct 2008 11:07:02 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 6 Oct 2008 11:07:02 GMT Message-Id: <200810061107.m96B72cU035639@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 11:07:03 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/126950 threads [patch] rtld(1): rtld malloc is thread-unsafe o kern/126128 threads [patch] pthread_condattr_getpshared is broken o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s kern/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 38 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Oct 6 21:29:10 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F36991065699 for ; Mon, 6 Oct 2008 21:29:09 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from silver.he.iki.fi (mx.helenius.fi [IPv6:2001:1bc8:1018::42]) by mx1.freebsd.org (Postfix) with ESMTP id 83C9B8FC22 for ; Mon, 6 Oct 2008 21:29:09 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from localhost (localhost [127.0.0.1]) by silver.he.iki.fi (Postfix) with ESMTP id 8CA6ABBFE for ; Tue, 7 Oct 2008 00:29:06 +0300 (EEST) Received: from silver.he.iki.fi ([127.0.0.1]) by localhost (silver.he.iki.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MP4am6FurKmU for ; Tue, 7 Oct 2008 00:29:03 +0300 (EEST) Received: from [IPv6:2001:1bc8:1018:0:5585:2de2:2674:216b] (unknown [IPv6:2001:1bc8:1018:0:5585:2de2:2674:216b]) by silver.he.iki.fi (Postfix) with ESMTP for ; Tue, 7 Oct 2008 00:29:02 +0300 (EEST) Message-ID: <48EA8320.5050500@helenius.fi> Date: Tue, 07 Oct 2008 00:29:04 +0300 From: Petri Helenius User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: freebsd-threads@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: getrusage with pthreads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Oct 2008 21:29:10 -0000 Is there a way to get thread-specific CPU utilization like the getrusage returns for the whole process? Looking at the kernel sources the utilization is accumulated on thread basis and then summed up but it would be useful to be able to determine utilization on a single thread between two checkpoints. Would pmc or ITIMER_PROF help? Pete From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 03:30:46 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC9B8106568A for ; Tue, 7 Oct 2008 03:30:46 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 928DB8FC15; Tue, 7 Oct 2008 03:30:46 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from apple.my.domain (root@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m973Uh9Y016872; Tue, 7 Oct 2008 03:30:45 GMT (envelope-from davidxu@freebsd.org) Message-ID: <48EAD851.7000404@freebsd.org> Date: Tue, 07 Oct 2008 11:32:33 +0800 From: David Xu User-Agent: Thunderbird 2.0.0.9 (X11/20080612) MIME-Version: 1.0 To: Petri Helenius References: <48EA8320.5050500@helenius.fi> In-Reply-To: <48EA8320.5050500@helenius.fi> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: getrusage with pthreads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 03:30:46 -0000 Petri Helenius wrote: > > Is there a way to get thread-specific CPU utilization like the getrusage > returns for the whole process? Looking at the kernel sources the > utilization is accumulated on thread basis and then summed up but it > would be useful to be able to determine utilization on a single thread > between two checkpoints. Would pmc or ITIMER_PROF help? > > Pete > You can use clock_gettime(CLOCK_THREAD_CPUTIME_ID), but I admit getrusage like function is better. Regards, David Xu From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 05:02:52 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AB7A1065688; Tue, 7 Oct 2008 05:02:52 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from silver.he.iki.fi (mx.helenius.fi [IPv6:2001:1bc8:1018::42]) by mx1.freebsd.org (Postfix) with ESMTP id 9F38F8FC1B; Tue, 7 Oct 2008 05:02:51 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from localhost (localhost [127.0.0.1]) by silver.he.iki.fi (Postfix) with ESMTP id EEE78BBFE; Tue, 7 Oct 2008 08:02:49 +0300 (EEST) Received: from silver.he.iki.fi ([127.0.0.1]) by localhost (silver.he.iki.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bZMs1CN06b3v; Tue, 7 Oct 2008 08:02:46 +0300 (EEST) Received: from [IPv6:2001:1bc8:1018:0:2869:106e:310c:4282] (unknown [IPv6:2001:1bc8:1018:0:2869:106e:310c:4282]) by silver.he.iki.fi (Postfix) with ESMTP; Tue, 7 Oct 2008 08:02:46 +0300 (EEST) Message-ID: <48EAED75.9010900@helenius.fi> Date: Tue, 07 Oct 2008 08:02:45 +0300 From: Petri Helenius User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: David Xu References: <48EA8320.5050500@helenius.fi> <48EAD851.7000404@freebsd.org> In-Reply-To: <48EAD851.7000404@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: getrusage with pthreads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 05:02:52 -0000 David Xu wrote: > > You can use clock_gettime(CLOCK_THREAD_CPUTIME_ID), but I admit > getrusage like function is better. > Is this implemented in FreeBSD starting from some specific version? I cannot seem to find the header. Pete > > From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 05:09:47 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB27F1065690 for ; Tue, 7 Oct 2008 05:09:47 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EC1228FC13; Tue, 7 Oct 2008 05:09:46 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from apple.my.domain (root@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m9759ju0028697; Tue, 7 Oct 2008 05:09:46 GMT (envelope-from davidxu@freebsd.org) Message-ID: <48EAEF87.8050803@freebsd.org> Date: Tue, 07 Oct 2008 13:11:35 +0800 From: David Xu User-Agent: Thunderbird 2.0.0.9 (X11/20080612) MIME-Version: 1.0 To: Petri Helenius References: <48EA8320.5050500@helenius.fi> <48EAD851.7000404@freebsd.org> <48EAED75.9010900@helenius.fi> In-Reply-To: <48EAED75.9010900@helenius.fi> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org Subject: Re: getrusage with pthreads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 05:09:47 -0000 Petri Helenius wrote: > David Xu wrote: >> >> You can use clock_gettime(CLOCK_THREAD_CPUTIME_ID), but I admit >> getrusage like function is better. >> > Is this implemented in FreeBSD starting from some specific version? I > cannot seem to find the header. > > Pete > It only exists in -CURRENT. From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 11:54:11 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67C501065693; Tue, 7 Oct 2008 11:54:11 +0000 (UTC) (envelope-from qj@huawei.com) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [119.145.14.65]) by mx1.freebsd.org (Postfix) with ESMTP id 095B98FC17; Tue, 7 Oct 2008 11:54:10 +0000 (UTC) (envelope-from qj@huawei.com) Received: from huawei.com (szxga02-in [172.24.2.6]) by szxga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0K8D00BEE9XGRL@szxga02-in.huawei.com>; Tue, 07 Oct 2008 19:44:05 +0800 (CST) Received: from huawei.com ([172.24.1.18]) by szxga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0K8D00H4R9XGTN@szxga02-in.huawei.com>; Tue, 07 Oct 2008 19:44:04 +0800 (CST) Received: from q00130354 ([10.111.9.38]) by szxml03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPA id <0K8D00GJZ9XDFL@szxml03-in.huawei.com>; Tue, 07 Oct 2008 19:44:04 +0800 (CST) Date: Tue, 07 Oct 2008 19:44:00 +0800 From: =?gb2312?B?x/G9ow==?= To: freebsd-questions@freebsd.org Message-id: <004001c92871$fdec0a10$01000001@china.huawei.com> MIME-version: 1.0 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.3350 X-Mailer: Microsoft Office Outlook 11 Thread-index: Ackocf2blBPcE5DVRuCv/5eRYj2bEg== Content-Type: text/plain; charset=gb2312 Content-Transfer-Encoding: 7BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@FreeBSD.org, freebsd-threads@freebsd.org Subject: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 11:54:11 -0000 Hi, folks, I did kernel profiling when a single thread client sends UDP packets to a single thread server on the same machine. In the output kernel profile, the first few kernel functions that consumes the most CPU time are listed below: granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds % cumulative self self total time seconds seconds calls ms/call ms/call name 42.4 10.88 10.88 0 100.00% __mcount [1] 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] 0.7 22.80 0.19 3145852 0.00 0.00 free [47] 0.6 22.96 0.15 6292172 0.00 0.00 uma_zfree_arg [52] 0.6 23.10 0.14 5243413 0.00 0.00 generic_bzero [53] 0.5 23.23 0.14 1048581 0.00 0.00 ip_output [23] 0.5 23.36 0.13 4221855 0.00 0.00 generic_bcopy [57] 0.4 23.47 0.11 36865859 0.00 0.00 critical_enter [61] 0.4 23.57 0.10 36865859 0.00 0.00 critical_exit [62] 0.4 23.67 0.09 17937541 0.00 0.00 spinlock_enter [63] 0.4 23.76 0.09 1048582 0.00 0.00 udp_input [21] 0.3 23.85 0.09 2108904 0.00 0.00 syscall [5] 0.3 23.93 0.08 1048587 0.00 0.00 ip_input [20] 0.3 24.00 0.07 2097156 0.00 0.00 getsock [65] 0.3 24.07 0.07 1048576 0.00 0.00 udp_send [22] It is very strange that spinlock_exit consumes over 36% CPU time while it seems a very simple function. For clarity, I paste the code of spinlock_exit here: void spinlock_exit(void) { struct thread *td; td = curthread; critical_exit(); td->td_md.md_spinlock_count--; if (td->td_md.md_spinlock_count == 0) intr_restore(td->td_md.md_saved_flags); } Since critical_exit consumes only 0.4% CPU time, does this mean the rest of spinlock_exit consume ~36% CPU time? Am I missing something? Could anybody help me understand this? Many thanks. BTW, the kernel is compiled with SMP and PREEMPTION disabled. The scheduler is ULE. Best regards, Qiu Jian From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 12:13:51 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BBF710656A6 for ; Tue, 7 Oct 2008 12:13:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA03.emeryville.ca.mail.comcast.net (qmta03.emeryville.ca.mail.comcast.net [76.96.30.32]) by mx1.freebsd.org (Postfix) with ESMTP id 4EFFC8FC47 for ; Tue, 7 Oct 2008 12:13:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA09.emeryville.ca.mail.comcast.net ([76.96.30.20]) by QMTA03.emeryville.ca.mail.comcast.net with comcast id PmXe1a0070S2fkCA3nxrGz; Tue, 07 Oct 2008 11:57:51 +0000 Received: from koitsu.dyndns.org ([69.181.141.110]) by OMTA09.emeryville.ca.mail.comcast.net with comcast id Pnxo1a00R2P6wsM8VnxpSF; Tue, 07 Oct 2008 11:57:49 +0000 X-Authority-Analysis: v=1.0 c=1 a=j1-vOjvzWS4A:10 a=qbkBlHUnetAA:10 a=QycZ5dHgAAAA:8 a=830efeqVd1-BpMj3v6YA:9 a=OaTEfa5PdL42xPuouuoA:7 a=_rQjeZhtyJV0AH9nAlltBONLT1IA:4 a=EoioJ0NPDVgA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id 7D66EC9419; Tue, 7 Oct 2008 04:57:48 -0700 (PDT) Date: Tue, 7 Oct 2008 04:57:48 -0700 From: Jeremy Chadwick To: ???? Message-ID: <20081007115748.GA48154@icarus.home.lan> References: <004001c92871$fdec0a10$01000001@china.huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <004001c92871$fdec0a10$01000001@china.huawei.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@FreeBSD.org, freebsd-questions@freebsd.org, freebsd-threads@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 12:13:51 -0000 On Tue, Oct 07, 2008 at 07:44:00PM +0800, ???? wrote: > Hi, folks, > > I did kernel profiling when a single thread client sends UDP packets to a > single thread server on the same machine. > > In the output kernel profile, the first few kernel functions that consumes > the most CPU time are listed below: > > granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds > > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 42.4 10.88 10.88 0 100.00% __mcount [1] > 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] > 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] > 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] > 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] > 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] > 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] > 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] > 0.7 22.80 0.19 3145852 0.00 0.00 free [47] > 0.6 22.96 0.15 6292172 0.00 0.00 uma_zfree_arg [52] > 0.6 23.10 0.14 5243413 0.00 0.00 generic_bzero [53] > 0.5 23.23 0.14 1048581 0.00 0.00 ip_output [23] > 0.5 23.36 0.13 4221855 0.00 0.00 generic_bcopy [57] > 0.4 23.47 0.11 36865859 0.00 0.00 critical_enter [61] > 0.4 23.57 0.10 36865859 0.00 0.00 critical_exit [62] > 0.4 23.67 0.09 17937541 0.00 0.00 spinlock_enter [63] > 0.4 23.76 0.09 1048582 0.00 0.00 udp_input [21] > 0.3 23.85 0.09 2108904 0.00 0.00 syscall [5] > 0.3 23.93 0.08 1048587 0.00 0.00 ip_input [20] > 0.3 24.00 0.07 2097156 0.00 0.00 getsock [65] > 0.3 24.07 0.07 1048576 0.00 0.00 udp_send [22] > > It is very strange that spinlock_exit consumes over 36% CPU time while it > seems a very simple function. > > For clarity, I paste the code of spinlock_exit here: > > void > spinlock_exit(void) > { > struct thread *td; > > td = curthread; > critical_exit(); > td->td_md.md_spinlock_count--; > if (td->td_md.md_spinlock_count == 0) > intr_restore(td->td_md.md_saved_flags); > } > > Since critical_exit consumes only 0.4% CPU time, does this mean the rest of > spinlock_exit consume ~36% CPU time? > > Am I missing something? Could anybody help me understand this? Many thanks. > > BTW, the kernel is compiled with SMP and PREEMPTION disabled. The scheduler > is ULE. What FreeBSD version, and what build date of the kernel? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 13:28:07 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D3B1106568B for ; Tue, 7 Oct 2008 13:28:07 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outP.internet-mail-service.net (outp.internet-mail-service.net [216.240.47.239]) by mx1.freebsd.org (Postfix) with ESMTP id 765938FC22 for ; Tue, 7 Oct 2008 13:28:07 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id E883022C0; Tue, 7 Oct 2008 06:28:07 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id CE2C52D601A; Tue, 7 Oct 2008 06:28:03 -0700 (PDT) Message-ID: <48EB63E3.60604@elischer.org> Date: Tue, 07 Oct 2008 21:28:03 +0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.17 (Macintosh/20080914) MIME-Version: 1.0 To: =?GB2312?B?x/G9ow==?= References: <004001c92871$fdec0a10$01000001@china.huawei.com> In-Reply-To: <004001c92871$fdec0a10$01000001@china.huawei.com> Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: 8bit Cc: freebsd-net@FreeBSD.org, freebsd-questions@freebsd.org, freebsd-threads@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 13:28:07 -0000 Ηρ½£ wrote: > Hi, folks, [...] spinlocks disable interrupts so the profiling interrupt is held off from the moment that the spinlock is entered to the moment it is exited, and all of that time is attributed to spinlock_exit(). so that this tells you that 3% of your time is spent under spinlocks which is a lot. as others have asked, "what version"? you should look up lock profiling to see WHICH lock is teh ine in question. From owner-freebsd-threads@FreeBSD.ORG Tue Oct 7 14:04:17 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 841F51065695; Tue, 7 Oct 2008 14:04:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 060658FC08; Tue, 7 Oct 2008 14:04:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m97E4AvC042408; Tue, 7 Oct 2008 10:04:10 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-threads@freebsd.org Date: Tue, 7 Oct 2008 09:38:04 -0400 User-Agent: KMail/1.9.7 References: <004001c92871$fdec0a10$01000001@china.huawei.com> In-Reply-To: <004001c92871$fdec0a10$01000001@china.huawei.com> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200810070938.04673.jhb@freebsd.org> Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Tue, 07 Oct 2008 10:04:10 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/8384/Tue Oct 7 06:37:18 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-net@freebsd.org, freebsd-questions@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2008 14:04:17 -0000 On Tuesday 07 October 2008 07:44:00 am =C7=F1=BD=A3 wrote: > Hi, folks, > =20 > I did kernel profiling when a single thread client sends UDP packets to a > single thread server on the same machine. > =20 > In the output kernel profile, the first few kernel functions that consumes > the most CPU time are listed below: > =20 > granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds >=20 > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 42.4 10.88 10.88 0 100.00% __mcount [1] > 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] > 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] > 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] > 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] > 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] > 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] > 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [= 9] > > It is very strange that spinlock_exit consumes over 36% CPU time while it > seems a very simple function. It's because the intr_restore() re-enables interrupts and the resulting tim= e=20 spent executing the handlers for any pending interrupts are attributed to=20 spinlock_exit(). =2D-=20 John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 04:45:24 2008 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5EC8106564A for ; Wed, 8 Oct 2008 04:45:24 +0000 (UTC) (envelope-from jhein@timing.com) Received: from Daffy.timing.com (smtp.timing.com [206.168.13.218]) by mx1.freebsd.org (Postfix) with ESMTP id 941B88FC16 for ; Wed, 8 Oct 2008 04:45:24 +0000 (UTC) (envelope-from jhein@timing.com) Received: from gromit.timing.com (gromit.timing.com [206.168.13.209]) by Daffy.timing.com (8.13.1/8.13.1) with ESMTP id m984KVUe020498 for ; Tue, 7 Oct 2008 22:20:32 -0600 (MDT) (envelope-from jhein@timing.com) Received: from gromit.timing.com (localhost [127.0.0.1]) by gromit.timing.com (8.14.2/8.14.2) with ESMTP id m983RvTS063513; Tue, 7 Oct 2008 21:27:57 -0600 (MDT) (envelope-from jhein@gromit.timing.com) Received: (from jhein@localhost) by gromit.timing.com (8.14.3/8.14.3/Submit) id m983Rv93063510; Tue, 7 Oct 2008 21:27:57 -0600 (MDT) (envelope-from jhein) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18668.10429.689076.595798@gromit.timing.com> Date: Tue, 7 Oct 2008 21:27:57 -0600 From: John Hein To: threads@freebsd.org X-Mailer: VM 7.19 under Emacs 22.1.1 X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on Daffy.timing.com X-Virus-Status: Clean Cc: Subject: (no subject) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 04:45:24 -0000 In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros that look like so... +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ + { \ + struct _pthread_cleanup_info __cleanup_info__; \ + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ + &__cleanup_info__); \ + { + +#define pthread_cleanup_pop(execute) \ + } \ + __pthread_cleanup_pop_imp(execute); \ + } + This breaks code where the pop is done in an inner block level. simplified example: pthread_cleanup_push(docleanup, &foo); try { dostuff(); } catch (...) { pthread_cleanup_pop(1); throw; } pthread_cleanup_pop(1); Does anyone have a recommended fix? From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 04:45:25 2008 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DEF51065686 for ; Wed, 8 Oct 2008 04:45:25 +0000 (UTC) (envelope-from jhein@timing.com) Received: from Daffy.timing.com (smtp.timing.com [206.168.13.218]) by mx1.freebsd.org (Postfix) with ESMTP id 0AE5B8FC19 for ; Wed, 8 Oct 2008 04:45:24 +0000 (UTC) (envelope-from jhein@timing.com) Received: from gromit.timing.com (gromit.timing.com [206.168.13.209]) by Daffy.timing.com (8.13.1/8.13.1) with ESMTP id m984KVUc020498 for ; Tue, 7 Oct 2008 22:20:32 -0600 (MDT) (envelope-from jhein@timing.com) Received: from gromit.timing.com (localhost [127.0.0.1]) by gromit.timing.com (8.14.2/8.14.2) with ESMTP id m983SXao063616 for ; Tue, 7 Oct 2008 21:28:33 -0600 (MDT) (envelope-from jhein@gromit.timing.com) Received: (from jhein@localhost) by gromit.timing.com (8.14.3/8.14.3/Submit) id m983SXLv063613; Tue, 7 Oct 2008 21:28:33 -0600 (MDT) (envelope-from jhein) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18668.10465.699531.162573@gromit.timing.com> Date: Tue, 7 Oct 2008 21:28:33 -0600 X-Mailer: VM 7.19 under Emacs 22.1.1 From: John Hein To: threads@freebsd.org X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on Daffy.timing.com X-Virus-Status: Clean Cc: Subject: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 04:45:25 -0000 In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros that look like so... +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ + { \ + struct _pthread_cleanup_info __cleanup_info__; \ + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ + &__cleanup_info__); \ + { + +#define pthread_cleanup_pop(execute) \ + } \ + __pthread_cleanup_pop_imp(execute); \ + } + This breaks code where the pop is done in an inner block level. simplified example: pthread_cleanup_push(docleanup, &foo); try { dostuff(); } catch (...) { pthread_cleanup_pop(1); throw; } pthread_cleanup_pop(1); Does anyone have a recommended fix? From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 05:12:16 2008 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9F27106568F for ; Wed, 8 Oct 2008 05:12:16 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 99F7B8FC12 for ; Wed, 8 Oct 2008 05:12:16 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 43AE91A3C36; Tue, 7 Oct 2008 21:54:47 -0700 (PDT) Date: Tue, 7 Oct 2008 21:54:47 -0700 From: Alfred Perlstein To: John Hein Message-ID: <20081008045447.GY36572@elvis.mu.org> References: <18668.10465.699531.162573@gromit.timing.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18668.10465.699531.162573@gromit.timing.com> User-Agent: Mutt/1.4.2.3i Cc: threads@freebsd.org Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 05:12:16 -0000 * John Hein [081007 21:45] wrote: > In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros > that look like so... Hey John, I found the same problem when working on QNX a while back, however that is really how it's supposed to be set up. I would suggest the following construct to fix the problem, make your own per-thread stack of destructors that are callable as functions and not macros. It's not too hard to do. Just use a pthread_key and pthread_once thingy to write a library to do it, shouldn't take more than a hundred lines of code. FWIW, OS X and QNX have the same set of macros, not sure about other OSes. -Alfred > > +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ > + { \ > + struct _pthread_cleanup_info __cleanup_info__; \ > + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ > + &__cleanup_info__); \ > + { > + > +#define pthread_cleanup_pop(execute) \ > + } \ > + __pthread_cleanup_pop_imp(execute); \ > + } > + > > > This breaks code where the pop is done in an inner block level. > > simplified example: > > pthread_cleanup_push(docleanup, &foo); > try > { > dostuff(); > } > catch (...) > { > pthread_cleanup_pop(1); > throw; > } > pthread_cleanup_pop(1); > > Does anyone have a recommended fix? > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" -- - Alfred Perlstein From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 06:31:04 2008 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E9941065687 for ; Wed, 8 Oct 2008 06:31:04 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id 2CA168FC23 for ; Wed, 8 Oct 2008 06:31:04 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.3/8.14.3/NETPLEX) with ESMTP id m986Ia7p015190; Wed, 8 Oct 2008 02:18:36 -0400 (EDT) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.0 (mail.netplex.net [204.213.176.10]); Wed, 08 Oct 2008 02:18:36 -0400 (EDT) Date: Wed, 8 Oct 2008 02:18:36 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Alfred Perlstein In-Reply-To: <20081008045447.GY36572@elvis.mu.org> Message-ID: References: <18668.10465.699531.162573@gromit.timing.com> <20081008045447.GY36572@elvis.mu.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: threads@freebsd.org, John Hein Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 06:31:04 -0000 On Tue, 7 Oct 2008, Alfred Perlstein wrote: > * John Hein [081007 21:45] wrote: >> In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros >> that look like so... > > Hey John, I found the same problem when working on QNX a while back, > however that is really how it's supposed to be set up. > > I would suggest the following construct to fix the problem, > make your own per-thread stack of destructors that are callable > as functions and not macros. > > It's not too hard to do. > > Just use a pthread_key and pthread_once thingy to write a library > to do it, shouldn't take more than a hundred lines of code. > > FWIW, OS X and QNX have the same set of macros, not sure about > other OSes. Solaris as well. Just conditionally undef them before you use them. #ifdef pthread_cleanup_push #undef pthread_cleanup_push #endif #ifdef pthread_cleanup_pop #undef pthread_cleanup_pop #endif The library versions are still there (they have to be in order to be callable from non-C/C++ languages). -- DE From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 07:13:10 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23D7110656D9; Wed, 8 Oct 2008 07:13:10 +0000 (UTC) (envelope-from qj@huawei.com) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [119.145.14.65]) by mx1.freebsd.org (Postfix) with ESMTP id CD7FD8FC2F; Wed, 8 Oct 2008 07:13:09 +0000 (UTC) (envelope-from qj@huawei.com) Received: from huawei.com (szxga02-in [172.24.2.6]) by szxga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0K8E00L7NRZK4Y@szxga02-in.huawei.com>; Wed, 08 Oct 2008 15:11:44 +0800 (CST) Received: from huawei.com ([172.24.1.12]) by szxga02-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0K8E00IHGRZKD4@szxga02-in.huawei.com>; Wed, 08 Oct 2008 15:11:44 +0800 (CST) Received: from q00130354 ([10.111.9.38]) by szxml05-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTPA id <0K8E00A8ZRZFIX@szxml05-in.huawei.com>; Wed, 08 Oct 2008 15:11:44 +0800 (CST) Date: Wed, 08 Oct 2008 15:11:40 +0800 From: =?gb2312?B?x/G9ow==?= In-reply-to: <20081007115748.GA48154@icarus.home.lan> To: 'Jeremy Chadwick' Message-id: <000301c92915$1cd0ae60$01000001@china.huawei.com> MIME-version: 1.0 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.3350 X-Mailer: Microsoft Office Outlook 11 Content-type: text/plain; charset=gb2312 Content-transfer-encoding: 7BIT Thread-index: AckodL2oSBjwz6W/R9ecg3XW/kxmlwAoABWQ Cc: freebsd-net@FreeBSD.org, freebsd-questions@freebsd.org, freebsd-threads@freebsd.org Subject: RE: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 07:13:10 -0000 Forgot to meantion that the test is based on FreeBSD kernel 7.0 2000807 snapshot. The kernel was compiled with a modified version of GENERIC configuration. With SMP and PREEMPTION disabled and kernel profiling enabled. -----Original Message----- From: Jeremy Chadwick [mailto:koitsu@FreeBSD.org] Sent: Tuesday, October 07, 2008 7:58 PM To: ???? Cc: freebsd-questions@freebsd.org; freebsd-net@FreeBSD.org; freebsd-threads@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. On Tue, Oct 07, 2008 at 07:44:00PM +0800, ???? wrote: > Hi, folks, > > I did kernel profiling when a single thread client sends UDP packets > to a single thread server on the same machine. > > In the output kernel profile, the first few kernel functions that > consumes the most CPU time are listed below: > > granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 > seconds > > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 42.4 10.88 10.88 0 100.00% __mcount [1] > 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] > 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] > 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] > 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] > 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] > 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] > 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] > 0.7 22.80 0.19 3145852 0.00 0.00 free [47] > 0.6 22.96 0.15 6292172 0.00 0.00 uma_zfree_arg [52] > 0.6 23.10 0.14 5243413 0.00 0.00 generic_bzero [53] > 0.5 23.23 0.14 1048581 0.00 0.00 ip_output [23] > 0.5 23.36 0.13 4221855 0.00 0.00 generic_bcopy [57] > 0.4 23.47 0.11 36865859 0.00 0.00 critical_enter [61] > 0.4 23.57 0.10 36865859 0.00 0.00 critical_exit [62] > 0.4 23.67 0.09 17937541 0.00 0.00 spinlock_enter [63] > 0.4 23.76 0.09 1048582 0.00 0.00 udp_input [21] > 0.3 23.85 0.09 2108904 0.00 0.00 syscall [5] > 0.3 23.93 0.08 1048587 0.00 0.00 ip_input [20] > 0.3 24.00 0.07 2097156 0.00 0.00 getsock [65] > 0.3 24.07 0.07 1048576 0.00 0.00 udp_send [22] > > It is very strange that spinlock_exit consumes over 36% CPU time while > it seems a very simple function. > > For clarity, I paste the code of spinlock_exit here: > > void > spinlock_exit(void) > { > struct thread *td; > > td = curthread; > critical_exit(); > td->td_md.md_spinlock_count--; > if (td->td_md.md_spinlock_count == 0) > intr_restore(td->td_md.md_saved_flags); > } > > Since critical_exit consumes only 0.4% CPU time, does this mean the > rest of spinlock_exit consume ~36% CPU time? > > Am I missing something? Could anybody help me understand this? Many thanks. > > BTW, the kernel is compiled with SMP and PREEMPTION disabled. The > scheduler is ULE. What FreeBSD version, and what build date of the kernel? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 19:11:02 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 125D61065765; Wed, 8 Oct 2008 19:11:01 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id EC2A98FC20; Wed, 8 Oct 2008 19:10:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m98JAd6H057756; Wed, 8 Oct 2008 15:10:51 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: =?gb2312?q?=C7=F1=BD=A3?= Date: Wed, 8 Oct 2008 11:16:10 -0400 User-Agent: KMail/1.9.7 References: <000a01c9291a$b81fa560$01000001@china.huawei.com> In-Reply-To: <000a01c9291a$b81fa560$01000001@china.huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200810081116.10298.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Wed, 08 Oct 2008 15:10:51 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/8395/Wed Oct 8 12:44:51 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.1 required=4.2 tests=BAYES_00, DATE_IN_PAST_03_06, NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-net@freebsd.org, freebsd-questions@freebsd.org, freebsd-threads@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:11:02 -0000 On Wednesday 08 October 2008 03:51:48 am =C7=F1=BD=A3 wrote: > Many thanks for the information. >=20 > Could we say that interrupt handlers consumed ~36% execution time? >=20 > Is this number too high? Is it possible that we abuse the use of critical > sections in kernel? I think whether or not it is high depends on the workload. =2D-=20 John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 19:11:17 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 539F2106569B for ; Wed, 8 Oct 2008 19:11:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id CC0788FC1E for ; Wed, 8 Oct 2008 19:11:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m98JB4tV057790; Wed, 8 Oct 2008 15:11:10 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-threads@freebsd.org Date: Wed, 8 Oct 2008 14:09:31 -0400 User-Agent: KMail/1.9.7 References: <18668.10465.699531.162573@gromit.timing.com> In-Reply-To: <18668.10465.699531.162573@gromit.timing.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200810081409.31822.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Wed, 08 Oct 2008 15:11:10 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/8395/Wed Oct 8 12:44:51 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.4 required=4.2 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: John Hein Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:11:17 -0000 On Tuesday 07 October 2008 11:28:33 pm John Hein wrote: > In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros > that look like so... > > +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ > + { \ > + struct _pthread_cleanup_info __cleanup_info__; \ > + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ > + &__cleanup_info__); \ > + { > + > +#define pthread_cleanup_pop(execute) \ > + } \ > + __pthread_cleanup_pop_imp(execute); \ > + } > + > > > This breaks code where the pop is done in an inner block level. > > simplified example: > > pthread_cleanup_push(docleanup, &foo); > try > { > dostuff(); > } > catch (...) > { > pthread_cleanup_pop(1); > throw; > } > pthread_cleanup_pop(1); > > Does anyone have a recommended fix? Don't Do That (tm). The spec explicitly states that they can be defined as macros and that code using it needs to put them in the same block. -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 19:32:40 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7321B106569B; Wed, 8 Oct 2008 19:32:40 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id 044E18FC13; Wed, 8 Oct 2008 19:32:39 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail05.syd.optusnet.com.au (mail05.syd.optusnet.com.au [211.29.132.186]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m98C9SS1004217; Wed, 8 Oct 2008 23:09:28 +1100 Received: from c211-30-84-14.carlnfd3.nsw.optusnet.com.au (c211-30-84-14.carlnfd3.nsw.optusnet.com.au [211.30.84.14]) by mail05.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m98C9OVj014049 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 8 Oct 2008 23:09:26 +1100 Date: Wed, 8 Oct 2008 22:09:24 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: John Baldwin In-Reply-To: <200810070938.04673.jhb@freebsd.org> Message-ID: <20081008210104.S20625@delplex.bde.org> References: <004001c92871$fdec0a10$01000001@china.huawei.com> <200810070938.04673.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-242184313-1223467764=:20625" Cc: freebsd-net@freebsd.org, freebsd-questions@freebsd.org, freebsd-threads@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:32:40 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-242184313-1223467764=:20625 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Tue, 7 Oct 2008, John Baldwin wrote: > On Tuesday 07 October 2008 07:44:00 am =C7=F1=BD=A3 wrote: >> Hi, folks, >> >> I did kernel profiling when a single thread client sends UDP packets to = a >> single thread server on the same machine. >> >> In the output kernel profile, the first few kernel functions that consum= es >> the most CPU time are listed below: >> >> granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 second= s >> >> % cumulative self self total >> time seconds seconds calls ms/call ms/call name >> 42.4 10.88 10.88 0 100.00% __mcount [1] >> 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] >> 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] >> 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43= ] >> 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [4= 8] >> 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [= 3] >> 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46= ] >> 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic = [9] >> >> It is very strange that spinlock_exit consumes over 36% CPU time while i= t >> seems a very simple function. > > It's because the intr_restore() re-enables interrupts and the resulting t= ime > spent executing the handlers for any pending interrupts are attributed to > spinlock_exit(). This is one of many defects that are not present in high resolution kernel profiling (kgmon -B instead of kgmon -b; availaible on amd64 and i386). However, high resolution kernel profiling doesn't work right with SMP, and was completely broken by gcc-4. Ordinary profiling was less completely broken by gcc-4, and you can recover the old behaviour by turning off new optimizations (mainly -funit-at-a-time and/or -finline-functions-called-once and or all of -O2). Bruce --0-242184313-1223467764=:20625-- From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 19:53:25 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8FF011065688 for ; Wed, 8 Oct 2008 19:53:25 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 778D48FC1E for ; Wed, 8 Oct 2008 19:53:25 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id F16891A3C36; Wed, 8 Oct 2008 12:53:24 -0700 (PDT) Date: Wed, 8 Oct 2008 12:53:24 -0700 From: Alfred Perlstein To: John Hein Message-ID: <20081008195324.GM36572@elvis.mu.org> References: <18668.10465.699531.162573@gromit.timing.com> <200810081409.31822.jhb@freebsd.org> <18669.3729.459915.769388@gromit.timing.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18669.3729.459915.769388@gromit.timing.com> User-Agent: Mutt/1.4.2.3i Cc: Daniel Eischen , freebsd-threads@freebsd.org Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 19:53:25 -0000 * John Hein [081008 12:48] wrote: > John Baldwin wrote at 14:09 -0400 on Oct 8, 2008: > > Don't Do That (tm). The spec explicitly states that they can be defined as > > macros and that code using it needs to put them in the same block. > > Thanks for the responses. I like deischen's response best to work > around the problem short term, but I suspect that's not very portable. > And as jhb describes using it outside the same block violates the > spirit of intended use. > > Note that the example I gave (pop off the cleanup stack in a catch > block) is not our only "violation". We also push in one function and > either never [explicitly] pop or possibly pop in another function. > > So that leaves me wondering about possible techniques for cleanup on > thread cancellation or exit. Alfred touched on one method, and I see > that pthread_key_create takes an optional destructor, so that sounds > like a possible avenue to explore. I'm positive that you'll have a solution given my starting point within a few hours. :) All the cpp_cleanup_push() needs to do is to: pthread_once() -> set up the pthread_key and destructor as you said. then pthread_getspecific to get the stack if no stack, make one and set it. then push your cleanup function. good luck, -- - Alfred Perlstein From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 20:40:31 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74F22106564A; Wed, 8 Oct 2008 20:40:31 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id 2DB898FC14; Wed, 8 Oct 2008 20:40:31 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.3/8.14.3/NETPLEX) with ESMTP id m98KQmwP002461; Wed, 8 Oct 2008 16:26:48 -0400 (EDT) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.0 (mail.netplex.net [204.213.176.10]); Wed, 08 Oct 2008 16:26:49 -0400 (EDT) Date: Wed, 8 Oct 2008 16:26:48 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Alfred Perlstein In-Reply-To: <20081008195324.GM36572@elvis.mu.org> Message-ID: References: <18668.10465.699531.162573@gromit.timing.com> <200810081409.31822.jhb@freebsd.org> <18669.3729.459915.769388@gromit.timing.com> <20081008195324.GM36572@elvis.mu.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: John Hein , freebsd-threads@freebsd.org Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:40:31 -0000 On Wed, 8 Oct 2008, Alfred Perlstein wrote: > * John Hein [081008 12:48] wrote: >> John Baldwin wrote at 14:09 -0400 on Oct 8, 2008: >> > Don't Do That (tm). The spec explicitly states that they can be defined as >> > macros and that code using it needs to put them in the same block. >> >> Thanks for the responses. I like deischen's response best to work >> around the problem short term, but I suspect that's not very portable. It's probably more portable than you think. The library calls really need to be there so they can be linked to from non-C languages. >> And as jhb describes using it outside the same block violates the >> spirit of intended use. >> >> Note that the example I gave (pop off the cleanup stack in a catch >> block) is not our only "violation". We also push in one function and >> either never [explicitly] pop or possibly pop in another function. >> >> So that leaves me wondering about possible techniques for cleanup on >> thread cancellation or exit. Alfred touched on one method, and I see >> that pthread_key_create takes an optional destructor, so that sounds >> like a possible avenue to explore. > > I'm positive that you'll have a solution given my starting point > within a few hours. :) > > All the cpp_cleanup_push() needs to do is to: > > pthread_once() -> set up the pthread_key and destructor as you said. > then pthread_getspecific to get the stack > if no stack, make one and set it. > then push your cleanup function. > > good luck, -- DE From owner-freebsd-threads@FreeBSD.ORG Wed Oct 8 20:45:25 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55BEF106568A; Wed, 8 Oct 2008 20:45:25 +0000 (UTC) (envelope-from jhein@timing.com) Received: from Daffy.timing.com (w.timing.com [206.168.13.218]) by mx1.freebsd.org (Postfix) with ESMTP id 1CAC38FC32; Wed, 8 Oct 2008 20:45:24 +0000 (UTC) (envelope-from jhein@timing.com) Received: from gromit.timing.com (gromit.timing.com [206.168.13.209]) by Daffy.timing.com (8.13.1/8.13.1) with ESMTP id m98KOcEV060930; Wed, 8 Oct 2008 14:24:39 -0600 (MDT) (envelope-from jhein@timing.com) Received: from gromit.timing.com (localhost [127.0.0.1]) by gromit.timing.com (8.14.2/8.14.2) with ESMTP id m98JmXEP070814; Wed, 8 Oct 2008 13:48:33 -0600 (MDT) (envelope-from jhein@gromit.timing.com) Received: (from jhein@localhost) by gromit.timing.com (8.14.3/8.14.3/Submit) id m98JmXuM070811; Wed, 8 Oct 2008 13:48:33 -0600 (MDT) (envelope-from jhein) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18669.3729.459915.769388@gromit.timing.com> Date: Wed, 8 Oct 2008 13:48:33 -0600 From: John Hein To: John Baldwin In-Reply-To: <200810081409.31822.jhb@freebsd.org> References: <18668.10465.699531.162573@gromit.timing.com> <200810081409.31822.jhb@freebsd.org> X-Mailer: VM 7.19 under Emacs 22.2.1 X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on Daffy.timing.com X-Virus-Status: Clean Cc: Daniel Eischen , freebsd-threads@freebsd.org Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Oct 2008 20:45:25 -0000 John Baldwin wrote at 14:09 -0400 on Oct 8, 2008: > Don't Do That (tm). The spec explicitly states that they can be defined as > macros and that code using it needs to put them in the same block. Thanks for the responses. I like deischen's response best to work around the problem short term, but I suspect that's not very portable. And as jhb describes using it outside the same block violates the spirit of intended use. Note that the example I gave (pop off the cleanup stack in a catch block) is not our only "violation". We also push in one function and either never [explicitly] pop or possibly pop in another function. So that leaves me wondering about possible techniques for cleanup on thread cancellation or exit. Alfred touched on one method, and I see that pthread_key_create takes an optional destructor, so that sounds like a possible avenue to explore. From owner-freebsd-threads@FreeBSD.ORG Thu Oct 9 01:24:09 2008 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE8B2106568C for ; Thu, 9 Oct 2008 01:24:09 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BD4E28FC24; Thu, 9 Oct 2008 01:24:09 +0000 (UTC) (envelope-from davidxu@freebsd.org) Received: from apple.my.domain (root@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m991O8XT009288; Thu, 9 Oct 2008 01:24:09 GMT (envelope-from davidxu@freebsd.org) Message-ID: <48ED5DA4.9060803@freebsd.org> Date: Thu, 09 Oct 2008 09:25:56 +0800 From: David Xu User-Agent: Thunderbird 2.0.0.9 (X11/20080612) MIME-Version: 1.0 To: John Hein References: <18668.10465.699531.162573@gromit.timing.com> In-Reply-To: <18668.10465.699531.162573@gromit.timing.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: threads@freebsd.org Subject: Re: pthread_cleanup_push & pthread_cleanup_pop usage X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Oct 2008 01:24:09 -0000 John Hein wrote: > In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros > that look like so... > > +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ > + { \ > + struct _pthread_cleanup_info __cleanup_info__; \ > + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ > + &__cleanup_info__); \ > + { > + > +#define pthread_cleanup_pop(execute) \ > + } \ > + __pthread_cleanup_pop_imp(execute); \ > + } > + > > > This breaks code where the pop is done in an inner block level. > > simplified example: > > pthread_cleanup_push(docleanup, &foo); > try > { > dostuff(); > } > catch (...) > { > pthread_cleanup_pop(1); > throw; > } > pthread_cleanup_pop(1); > > Does anyone have a recommended fix? Your problem can be fixed as deischen@ said. but I think it can be resolved by stack unwinding code I am working in progress, when a C++ exception is thrown, onstack C++ objects along with pthread cleanup handlers will be invoked automatically. Regards, David Xu