From owner-freebsd-threads@FreeBSD.ORG Mon Dec 5 11:02:30 2005 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 23DE116A420 for ; Mon, 5 Dec 2005 11:02:30 +0000 (GMT) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id F310243D64 for ; Mon, 5 Dec 2005 11:02:13 +0000 (GMT) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (peter@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB5B2CQk024079 for ; Mon, 5 Dec 2005 11:02:12 GMT (envelope-from owner-bugmaster@freebsd.org) Received: (from peter@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jB5B2CLJ024073 for freebsd-threads@freebsd.org; Mon, 5 Dec 2005 11:02:12 GMT (envelope-from owner-bugmaster@freebsd.org) Date: Mon, 5 Dec 2005 11:02:12 GMT Message-Id: <200512051102.jB5B2CLJ024073@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: peter set sender to owner-bugmaster@freebsd.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Dec 2005 11:02:30 -0000 Current FreeBSD problem reports Critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2005/01/26] threads/76690threads fork hang in child for (-lc_r & -lthr) 1 problem total. Serious problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/07/18] kern/20016 threads pthreads: Cannot set scheduling timer/Can o [2000/08/26] kern/20861 threads libc_r does not honor socket timeouts o [2001/01/20] threads/24472threads libc_r does not honor SO_SNDTIMEO/SO_RCVT o [2001/01/25] threads/24632threads libc_r delicate deviation from libc in ha o [2001/01/25] kern/24641 threads pthread_rwlock_rdlock can deadlock o [2001/11/26] bin/32295 threads pthread dont dequeue signals o [2002/02/01] threads/34536threads accept() blocks other threads o [2002/05/25] kern/38549 threads the procces compiled whith pthread stoppe o [2002/06/27] threads/39922threads [threads] [patch] Threaded applications e o [2002/08/04] kern/41331 threads Pthread library open sets O_NONBLOCK flag o [2003/03/02] threads/48856threads Setting SIGCHLD to SIG_IGN still leaves z o [2003/03/10] threads/49087threads Signals lost in programs linked with libc s [2004/03/15] kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/un o [2004/08/26] threads/70975threads unexpected and unreliable behaviour when o [2004/10/05] threads/72353threads Assertion fails in /usr/src/lib/libpthrea o [2004/10/07] threads/72429threads threads blocked in stdio (fgets, etc) are o [2004/10/21] threads/72953threads fork() unblocks blocked signals w/o PTHRE o [2004/12/19] threads/75273threads FBSD 5.3 libpthread (KSE) bug o [2004/12/21] threads/75374threads pthread_kill() ignores SA_SIGINFO flag o [2005/01/26] threads/76694threads fork cause hang in dup()/close() function o [2005/03/10] threads/78660threads Java hangs unkillably in STOP state after o [2005/04/08] threads/79683threads svctcp_create() fails if multiple threads o [2005/04/28] threads/80435threads panic on high loads o [2005/05/19] threads/81258threads Thread specific data is sometimes assigne o [2005/08/02] threads/84483threads problems with devel/nspr and -lc_r on 4.x s [2005/08/10] threads/84778threads libpthread busy loop/hang with Java when o [2005/08/20] threads/85160threads [libthr] [patch] libobjc + libpthread/lib o [2005/11/19] threads/89262threads [kernel] [patch] multi-threaded process h 28 problems total. Non-critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/06/13] kern/19247 threads uthread_sigaction.c does not do anything o [2000/10/21] kern/22190 threads A threaded read(2) from a socketpair(2) f o [2001/09/09] threads/30464threads pthread mutex attributes -- pshared o [2002/05/02] threads/37676threads libc_r: msgsnd(), msgrcv(), pread(), pwri s [2002/07/16] threads/40671threads pthread_cancel doesn't remove thread from o [2004/07/13] threads/69020threads pthreads library leaks _gc_mutex o [2004/09/21] threads/71966threads Mlnet Core Dumped : Fatal error '_pq_inse o [2004/11/21] threads/74180threads KSE problem. Applications those riched ma o [2005/04/13] threads/79887threads [patch] freopen() isn't thread-safe o [2005/05/13] threads/80992threads abort() sometimes not caught by gdb depen o [2005/05/26] threads/81534threads [libc_r] [patch] libc_r close() will fail 11 problems total. From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 10:00:04 2005 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4500816A41F; Tue, 6 Dec 2005 10:00:04 +0000 (GMT) (envelope-from mv@roq.com) Received: from vault.mel.jumbuck.com (ppp166-27.static.internode.on.net [150.101.166.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8140E43D60; Tue, 6 Dec 2005 10:00:03 +0000 (GMT) (envelope-from mv@roq.com) Received: from vault.mel.jumbuck.com (localhost [127.0.0.1]) by vault.mel.jumbuck.com (Postfix) with ESMTP id 6C0198A035; Tue, 6 Dec 2005 20:59:57 +1100 (EST) Received: from [192.168.46.52] (unknown [192.168.46.250]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by vault.mel.jumbuck.com (Postfix) with ESMTP id 4FF1D8A014; Tue, 6 Dec 2005 20:59:57 +1100 (EST) Message-ID: <43956122.5020106@roq.com> Date: Tue, 06 Dec 2005 21:00:02 +1100 From: Michael Vince User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.7.12) Gecko/20051202 X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-java@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Cc: performance@freebsd.org, freebsd-threads@FreeBSD.org Subject: More threads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 10:00:04 -0000 Hi All, I been benchmarking a Java servlet I have created and I want to be able to handle a massive amount of simultaneous connections. So far using benchmarking tools I have been able to get around 2,565 threads on the Tomcat 5.5 Java process (with native 1.4.2 Java) according to ps -auxwH | grep -c 'java' But I just can't seem to get past that mark, I have a lot of memory currently the Tomcat is allocated 2gigs of memory. When I max it out with my benchmarks I get this in catalina Tomcat 5.5 but there is still plenty of free memory to the Tomcat process. SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.SocketAcceptor@f36e59, terminatingthread I have been testing with my libmap.conf with these different types of implementations. [/usr/local/jdk1.4.2/] #libpthread.so libc_r.so #libpthread.so.2 libc_r.so.6 libpthread.so.2 libthr.so.2 libpthread.so libthr.so libc_r.so.6 libthr.so.2 libc_r.so libthr.so I have been getting the best performance with libthr and have been using the above libthr and with these settings below. I have set my max_threads in /etc/sysctl.conf to a massive amount. kern.threads.max_threads_per_proc=40000 kern.threads.max_groups_per_proc=40000 ps -auxwH | grep -c 'java' 2565 I am using Apache2.2 with the new built in AJP module which has been a great addition to Apache 2.2. I have been able to get the setup performing most stable (no 503 status errors) but with less performance / threads with the libmap.conf below. [/usr/local/jdk1.4.2/] libpthread.so libc_r.so libpthread.so.2 libc_r.so.6 I am on 6.0 Release i386, dual Intel P4, generic SMP kernel. With Tomcat on Windows XP I have been able to get it running better. Does any one know of some other sysctls that increase the amount of threads in libthr which I am assuming utilizes the above sysctls. Does any one know how many threads can be created in Java on FreeBSD? Cheers, Mike From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 10:12:09 2005 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5F36D16A41F; Tue, 6 Dec 2005 10:12:09 +0000 (GMT) (envelope-from achill@matrix.gatewaynet.com) Received: from matrix.gatewaynet.com (host2.dynacom.ondsl.gr [62.103.35.210]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8E20243D68; Tue, 6 Dec 2005 10:12:08 +0000 (GMT) (envelope-from achill@matrix.gatewaynet.com) Received: from matrix.gatewaynet.com (localhost.localdomain [127.0.0.1]) by matrix.gatewaynet.com (8.12.8/8.12.8) with ESMTP id jB6BAUKc017209; Tue, 6 Dec 2005 13:10:30 +0200 Received: from localhost (achill@localhost) by matrix.gatewaynet.com (8.12.8/8.12.8/Submit) with ESMTP id jB6BAUgg017205; Tue, 6 Dec 2005 13:10:30 +0200 Date: Tue, 6 Dec 2005 13:10:30 +0200 (EET) From: Achilleus Mantzios To: Michael Vince In-Reply-To: <43956122.5020106@roq.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-threads@freebsd.org, performance@freebsd.org, freebsd-java@freebsd.org Subject: Re: More threads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 10:12:09 -0000 Maybe take a look at http://mail-archives.apache.org/mod_mbox/jakarta-tomcat-user/200312.mbox/<3FD2066D.2040008%40joedog.org> -- -Achilleus From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 10:15:22 2005 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 726F816A420 for ; Tue, 6 Dec 2005 10:15:22 +0000 (GMT) (envelope-from chad.brubaker@gmail.com) Received: from smtp101.sbc.mail.mud.yahoo.com (smtp101.sbc.mail.mud.yahoo.com [68.142.198.200]) by mx1.FreeBSD.org (Postfix) with SMTP id 6AD5143D55 for ; Tue, 6 Dec 2005 10:15:21 +0000 (GMT) (envelope-from chad.brubaker@gmail.com) Received: (qmail 67453 invoked from network); 6 Dec 2005 10:15:20 -0000 Received: from unknown (HELO ?192.168.1.106?) (emochila@sbcglobal.net@71.141.245.145 with plain) by smtp101.sbc.mail.mud.yahoo.com with SMTP; 6 Dec 2005 10:15:20 -0000 In-Reply-To: <43956122.5020106@roq.com> References: <43956122.5020106@roq.com> Mime-Version: 1.0 (Apple Message framework v746.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Chad Brubaker Date: Tue, 6 Dec 2005 02:15:19 -0800 To: Michael Vince X-Mailer: Apple Mail (2.746.2) Cc: freebsd-threads@FreeBSD.org, performance@freebsd.org, freebsd-java@FreeBSD.org Subject: Re: More threads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 10:15:22 -0000 You can try setting your heap size. JAVA_OPTS="-Xms512M - Xmx1024M" - actually, in your case you could set the min and max to be much higher, but this is the command i use on my machine. I'm pretty sure this is definitely your problem. I upgraded to a machine with more memory thinking i had solved my OutOfMemeoryError, - but low and behold, it was my heap size allocation.... hope this helps. On Dec 6, 2005, at 2:00 AM, Michael Vince wrote: > Hi All, > > I been benchmarking a Java servlet I have created and I want to be > able to handle a massive amount of simultaneous connections. > So far using benchmarking tools I have been able to get around > 2,565 threads on the Tomcat 5.5 Java process (with native 1.4.2 > Java) according to ps -auxwH | grep -c 'java' > > But I just can't seem to get past that mark, I have a lot of memory > currently the Tomcat is allocated 2gigs of memory. > > When I max it out with my benchmarks I get this in catalina Tomcat > 5.5 but there is still plenty of free memory to the Tomcat process. > SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to > create new native thread) executing > org.apache.jk.common.SocketAcceptor@f36e59, terminatingthread > > I have been testing with my libmap.conf with these different types > of implementations. > > [/usr/local/jdk1.4.2/] > #libpthread.so libc_r.so > #libpthread.so.2 libc_r.so.6 > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > libc_r.so.6 libthr.so.2 > libc_r.so libthr.so > > I have been getting the best performance with libthr and have been > using the above libthr and with these settings below. > > I have set my max_threads in /etc/sysctl.conf to a massive amount. > > kern.threads.max_threads_per_proc=40000 > kern.threads.max_groups_per_proc=40000 > > ps -auxwH | grep -c 'java' > 2565 > > I am using Apache2.2 with the new built in AJP module which has > been a great addition to Apache 2.2. > I have been able to get the setup performing most stable (no 503 > status errors) but with less performance / threads with the > libmap.conf below. > [/usr/local/jdk1.4.2/] > libpthread.so libc_r.so > libpthread.so.2 libc_r.so.6 > > I am on 6.0 Release i386, dual Intel P4, generic SMP kernel. > > With Tomcat on Windows XP I have been able to get it running better. > Does any one know of some other sysctls that increase the amount of > threads in libthr which I am assuming utilizes the above sysctls. > Does any one know how many threads can be created in Java on FreeBSD? > > Cheers, > Mike > > > > > > _______________________________________________ > freebsd-java@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-java > To unsubscribe, send any mail to "freebsd-java- > unsubscribe@freebsd.org" From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 10:33:30 2005 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C17016A420; Tue, 6 Dec 2005 10:33:30 +0000 (GMT) (envelope-from davidxu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8F12543D5E; Tue, 6 Dec 2005 10:33:29 +0000 (GMT) (envelope-from davidxu@freebsd.org) Received: from [127.0.0.1] (davidxu@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB6AXQMD040880; Tue, 6 Dec 2005 10:33:27 GMT (envelope-from davidxu@freebsd.org) Message-ID: <439568FB.1000804@freebsd.org> Date: Tue, 06 Dec 2005 18:33:31 +0800 From: David Xu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20050928 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Michael Vince References: <43956122.5020106@roq.com> In-Reply-To: <43956122.5020106@roq.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-threads@freebsd.org, performance@freebsd.org, freebsd-java@freebsd.org Subject: Re: More threads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 10:33:30 -0000 Michael Vince wrote: > Hi All, > > I been benchmarking a Java servlet I have created and I want to be able > to handle a massive amount of simultaneous connections. > So far using benchmarking tools I have been able to get around 2,565 > threads on the Tomcat 5.5 Java process (with native 1.4.2 Java) > according to ps -auxwH | grep -c 'java' > > But I just can't seem to get past that mark, I have a lot of memory > currently the Tomcat is allocated 2gigs of memory. > > When I max it out with my benchmarks I get this in catalina Tomcat 5.5 > but there is still plenty of free memory to the Tomcat process. > SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create > new native thread) executing org.apache.jk.common.SocketAcceptor@f36e59, > terminatingthread > > I have been testing with my libmap.conf with these different types of > implementations. > > [/usr/local/jdk1.4.2/] > #libpthread.so libc_r.so > #libpthread.so.2 libc_r.so.6 > libpthread.so.2 libthr.so.2 > libpthread.so libthr.so > libc_r.so.6 libthr.so.2 > libc_r.so libthr.so > > I have been getting the best performance with libthr and have been using > the above libthr and with these settings below. > > I have set my max_threads in /etc/sysctl.conf to a massive amount. > > kern.threads.max_threads_per_proc=40000 > kern.threads.max_groups_per_proc=40000 > > ps -auxwH | grep -c 'java' > 2565 > > I am using Apache2.2 with the new built in AJP module which has been a > great addition to Apache 2.2. > I have been able to get the setup performing most stable (no 503 status > errors) but with less performance / threads with the libmap.conf below. > [/usr/local/jdk1.4.2/] > libpthread.so libc_r.so > libpthread.so.2 libc_r.so.6 > > I am on 6.0 Release i386, dual Intel P4, generic SMP kernel. > > With Tomcat on Windows XP I have been able to get it running better. > Does any one know of some other sysctls that increase the amount of > threads in libthr which I am assuming utilizes the above sysctls. Does > any one know how many threads can be created in Java on FreeBSD? > > Cheers, > Mike > Number of threads you can create if you use libthr is limited by following factors: 1) sysctl: kern.threads.max_threads_per_proc kern.threads.max_groups_per_proc 2) stack per-thread userland stack, default number on 64 bits platform is 2M, on 32 bits platform, it is 1M. I don't know whether java supports adjusting default per-thread stack size. if can not, we may add an environment variable to thread libraries, for example: LIBPTHREAD_THREAD_STACKSIZE allows user to set default stack size. Thread also needs a kernel mode stack when it is in kernel, if I am right, it is 16K bytes per-thread. if you create too many threads, make sure both side won't exhaust address space, and because kernel stack can not be swapped out when process is running, you'd make sure physical memory can not be exhausted. 3) check memory limits type limits command: Resource limits (current): cputime infinity secs filesize infinity kB datasize 524288 kB stacksize 65536 kB coredumpsize infinity kB memoryuse infinity kB memorylocked infinity kB maxprocesses 5547 openfiles 11095 sbsize infinity bytes vmemoryuse infinity kB if address space is not large enough, you have to reconfigure kernel to allow larger space. Fix me if I am wrong. David Xu From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 22:40:10 2005 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 174F916A420 for ; Tue, 6 Dec 2005 22:40:10 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2FE0443D5C for ; Tue, 6 Dec 2005 22:40:06 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB6Me6Ek069813 for ; Tue, 6 Dec 2005 22:40:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jB6Me5hA069812; Tue, 6 Dec 2005 22:40:06 GMT (envelope-from gnats) Date: Tue, 6 Dec 2005 22:40:06 GMT Message-Id: <200512062240.jB6Me5hA069812@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: freebsd@spatula.net Cc: Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@spatula.net List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 22:40:10 -0000 The following reply was made to PR threads/84778; it has been noted by GNATS. From: freebsd@spatula.net To: bug-followup@FreeBSD.org Cc: freebsd-java@freebsd.org Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec Date: Tue, 6 Dec 2005 14:36:20 -0800 (PST) This is still happening on FreeBSD 6.0-STABLE as of 12/6/2005. World and kernel were compiled without any CPUTYPE or CFLAGS options, as was Java. This problem is trivial to reproduce. One merely has to start httpd.sh in Resin 3.0 with no arguments, wait for the JVM to start running, and hit ^Z or ^C. The process then goes into an infinite loop in kse_release. truss -p shows the following: kse_release(0x8056f44) = 0 (0x0) kse_release(0x8056f44) = 0 (0x0) kse_release(0x8056f4c) = 0 (0x0) kse_release(0x8056f44) = 0 (0x0) kse_release(0x8056f44) = 0 (0x0) kse_release(0x8056f4c) = 0 (0x0) ktrace shows a bit more detail of the infinite loop: 21357 java CALL gettimeofday(0xbf275e70,0) 21357 java RET gettimeofday 0 21357 java CALL gettimeofday(0xbf275ec0,0) 21357 java RET gettimeofday 0 21357 java CALL kse_release(0x8056f4c) 21357 java RET kse_release 0 21357 java CALL gettimeofday(0xbf275e70,0) 21357 java RET gettimeofday 0 21357 java CALL gettimeofday(0xbf275ec0,0) 21357 java RET gettimeofday 0 21357 java CALL kse_release(0x8056f4c) 21357 java RET kse_release 0 gcore -s 21357; gdb /usr/local/jdk14/bin/java core.21357 shows: Cannot get thread info: generic error #0 0x4809f1af in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x480a01dd in __error () from /usr/lib/libpthread.so.2 #2 0x4808adeb in pthread_getschedparam () from /usr/lib/libpthread.so.2 #3 0x4808feef in pthread_create () from /usr/lib/libpthread.so.2 #4 0x48465f10 in os::create_thread () from /usr/local/jdk1.4.2/jre/lib/i386/client/libjvm.so #5 0x484afb28 in JavaThread::JavaThread () from /usr/local/jdk1.4.2/jre/lib/i386/client/libjvm.so #6 0x483f927c in JVM_StartThread () from /usr/local/jdk1.4.2/jre/lib/i386/client/libjvm.so followed by a smashed-looking stack for about 60 frames, and then: #65 0x485735c0 in __JCR_LIST__ () from /usr/local/jdk1.4.2/jre/lib/i386/client/libjvm.so #66 0x4a3c20d7 in ?? () #67 0xbf073d00 in ?? () #68 0xbf073d48 in ?? () #69 0x4839032b in JavaCalls::call_helper () from /usr/local/jdk1.4.2/jre/lib/i386/client/libjvm.so Previous frame inner to this frame (corrupt stack?) Note that the valid-looking portion of the stack trace still isn't valid higher than pthread_create(), though this could have still been a result of the process still making calls while gcore was running, since the process does not respond to the STOP signal. I'm inclined to suspect that this is a Java bug, considering the smashed stack. Nick -- "The aptly-named morons.org is an obscenity-laced screed..." -- Robert P. Lockwood, Catholic League director of research Nick Johnson, version 2.1 http://web.morons.org/ From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 23:18:29 2005 Return-Path: X-Original-To: freebsd-threads@freebsd.org Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3132516A41F; Tue, 6 Dec 2005 23:18:29 +0000 (GMT) (envelope-from glewis@eyesbeyond.com) Received: from misty.eyesbeyond.com (glewis.dsl.xmission.com [166.70.56.15]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3C1AF43D4C; Tue, 6 Dec 2005 23:18:27 +0000 (GMT) (envelope-from glewis@eyesbeyond.com) Received: from misty.eyesbeyond.com (localhost.eyesbeyond.com [127.0.0.1]) by misty.eyesbeyond.com (8.13.3/8.13.3) with ESMTP id jB6NIQ9q045157; Tue, 6 Dec 2005 16:18:26 -0700 (MST) (envelope-from glewis@eyesbeyond.com) Received: (from glewis@localhost) by misty.eyesbeyond.com (8.13.3/8.13.3/Submit) id jB6NIPEq045156; Tue, 6 Dec 2005 16:18:25 -0700 (MST) (envelope-from glewis@eyesbeyond.com) X-Authentication-Warning: misty.eyesbeyond.com: glewis set sender to glewis@eyesbeyond.com using -f Date: Tue, 6 Dec 2005 16:18:25 -0700 From: Greg Lewis To: David Xu Message-ID: <20051206231825.GA44966@misty.eyesbeyond.com> References: <43956122.5020106@roq.com> <439568FB.1000804@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <439568FB.1000804@freebsd.org> User-Agent: Mutt/1.4.2.1i Cc: freebsd-java@freebsd.org, performance@freebsd.org, freebsd-threads@freebsd.org Subject: Re: More threads X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 23:18:29 -0000 On Tue, Dec 06, 2005 at 06:33:31PM +0800, David Xu wrote: > Michael Vince wrote: [...] > >But I just can't seem to get past that mark, I have a lot of memory > >currently the Tomcat is allocated 2gigs of memory. [...] > Number of threads you can create if you use libthr is limited by > following factors: > 1) sysctl: > kern.threads.max_threads_per_proc > kern.threads.max_groups_per_proc > > 2) stack > per-thread userland stack, default number on 64 bits platform is 2M, > on 32 bits platform, it is 1M. I don't know whether java supports > adjusting default per-thread stack size. if can not, we may add > an environment variable to thread libraries, for example: > LIBPTHREAD_THREAD_STACKSIZE allows user to set default stack size. > > Thread also needs a kernel mode stack when it is in kernel, if > I am right, it is 16K bytes per-thread. if you create too many > threads, make sure both side won't exhaust address space, and > because kernel stack can not be swapped out when process is > running, you'd make sure physical memory can not be exhausted. > > 3) check memory limits > type limits command: > > Resource limits (current): > cputime infinity secs > filesize infinity kB > datasize 524288 kB > stacksize 65536 kB > coredumpsize infinity kB > memoryuse infinity kB > memorylocked infinity kB > maxprocesses 5547 > openfiles 11095 > sbsize infinity bytes > vmemoryuse infinity kB > > if address space is not large enough, you have to reconfigure kernel > to allow larger space. Fix me if I am wrong. I think that Michael's real problem is the amount of memory he has allocated. If he has 2GB allocated already then he is out of luck if he needs to allocate more (e.g. to create another thread). See mmap(2), particularly the BUGS section. -- Greg Lewis Email : glewis@eyesbeyond.com Eyes Beyond Web : http://www.eyesbeyond.com Information Technology FreeBSD : glewis@FreeBSD.org From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 23:20:19 2005 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF67716A41F for ; Tue, 6 Dec 2005 23:20:19 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5674543D77 for ; Tue, 6 Dec 2005 23:20:15 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB6NK9x0075946 for ; Tue, 6 Dec 2005 23:20:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jB6NK9RS075945; Tue, 6 Dec 2005 23:20:09 GMT (envelope-from gnats) Date: Tue, 6 Dec 2005 23:20:09 GMT Message-Id: <200512062320.jB6NK9RS075945@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: freebsd@spatula.net Cc: Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@spatula.net List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 23:20:19 -0000 The following reply was made to PR threads/84778; it has been noted by GNATS. From: freebsd@spatula.net To: bug-followup@FreeBSD.org Cc: freebsd-java@freebsd.org Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec Date: Tue, 6 Dec 2005 15:16:16 -0800 (PST) Another bit of information of interest: sending the process a kill -stop (-17) after trying to stop it with ^Z *will* cause the process to stop. Sending it a kill -stop without hitting ^Z will also cause the process to stop normally. In either case, however, resuming from the shell with fg will cause the same infinite-loop behaviour. Resuming with a kill -19 will also cause the infinite-loop behaviour. Sending a regular kill signal also causes the infinite-loop behaviour. The process has to be killed with a kill -9 in all cases once this behaiour starts. -- "The aptly-named morons.org is an obscenity-laced screed..." -- Robert P. Lockwood, Catholic League director of research Nick Johnson, version 2.1 http://web.morons.org/ From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 23:40:21 2005 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A3E4F16A41F for ; Tue, 6 Dec 2005 23:40:21 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id B060E43D88 for ; Tue, 6 Dec 2005 23:40:06 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB6Ne6GO077801 for ; Tue, 6 Dec 2005 23:40:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jB6Ne604077800; Tue, 6 Dec 2005 23:40:06 GMT (envelope-from gnats) Date: Tue, 6 Dec 2005 23:40:06 GMT Message-Id: <200512062340.jB6Ne604077800@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: freebsd@spatula.net Cc: Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@spatula.net List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 23:40:21 -0000 The following reply was made to PR threads/84778; it has been noted by GNATS. From: freebsd@spatula.net To: bug-followup@FreeBSD.org Cc: freebsd-java@FreeBSD.org Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec Date: Tue, 6 Dec 2005 15:34:32 -0800 (PST) As I continue to muck about with this I'm finding a few new things out. I sent the process a ^Z, which caused the process to begin running away. Then I sent it a kill -stop and waited for the process to stop completely. Then I ran gcore, and although the first few stack frames are probably caused by the stop signal, the last stack frame may be telling: Cannot get thread info: generic error (gdb) bt #0 0x480a0090 in __error () from /usr/lib/libpthread.so.2 #1 0x4808d074 in sigaction () from /usr/lib/libpthread.so.2 #2 0x4808d624 in sigaction () from /usr/lib/libpthread.so.2 #3 0x48097bd0 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 #4 0x00000000 in ?? () Looks like sending a ^Z may be causing Java to dereference a null. I'll continue investigating to see what else can be discovered. -- "The aptly-named morons.org is an obscenity-laced screed..." -- Robert P. Lockwood, Catholic League director of research Nick Johnson, version 2.1 http://web.morons.org/ From owner-freebsd-threads@FreeBSD.ORG Tue Dec 6 23:50:05 2005 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 35B0016A41F for ; Tue, 6 Dec 2005 23:50:05 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id CD0E443D45 for ; Tue, 6 Dec 2005 23:50:04 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB6No4o0078041 for ; Tue, 6 Dec 2005 23:50:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jB6No4d9078040; Tue, 6 Dec 2005 23:50:04 GMT (envelope-from gnats) Date: Tue, 6 Dec 2005 23:50:04 GMT Message-Id: <200512062350.jB6No4d9078040@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: freebsd@spatula.net Cc: Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@spatula.net List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2005 23:50:05 -0000 The following reply was made to PR threads/84778; it has been noted by GNATS. From: freebsd@spatula.net To: bug-followup@FreeBSD.org Cc: freebsd-java@FreeBSD.org Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec Date: Tue, 6 Dec 2005 15:48:31 -0800 (PST) I have found the problem, at long last. Resin uses a custom JNI library for acceleration of some features. In their compilation process, they specify -lc_r for the threading library used in their JNI library. This in turn seems to be the cause of unexpected and erratic behaviour when running Java with libpthread, and probably explains why the problems completely disappear when mapping Java to use libc_r. Modifying the makefile to link the JNI library to libpthread completely alleviates the signal handling problems. The process can be suspended and resumed from the keyboard ad-nauseam. Removing the JNI library from the picture entirely has the same effect (Resin will use the native Java code if the JNI library cannot be loaded.) I will follow up with the Resin folks and submit a patch to detect the thread library that is in use and use that one when compiling the JNI library. Please close this bug. Nick -- "The aptly-named morons.org is an obscenity-laced screed..." -- Robert P. Lockwood, Catholic League director of research Nick Johnson, version 2.1 http://web.morons.org/ From owner-freebsd-threads@FreeBSD.ORG Wed Dec 7 00:22:33 2005 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C98E16A420; Wed, 7 Dec 2005 00:22:33 +0000 (GMT) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id E2A8643D45; Wed, 7 Dec 2005 00:22:32 +0000 (GMT) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id jB70MWBA084996; Wed, 7 Dec 2005 00:22:32 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id jB70MWZf084992; Wed, 7 Dec 2005 00:22:32 GMT (envelope-from linimon) Date: Wed, 7 Dec 2005 00:22:32 GMT From: Mark Linimon Message-Id: <200512070022.jB70MWZf084992@freefall.freebsd.org> To: freebsd@spatula.net, linimon@FreeBSD.org, freebsd-threads@FreeBSD.org Cc: Subject: Re: threads/84778: libpthread busy loop/hang with Java when handling signals and Runtime.exec X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2005 00:22:33 -0000 Synopsis: libpthread busy loop/hang with Java when handling signals and Runtime.exec State-Changed-From-To: suspended->closed State-Changed-By: linimon State-Changed-When: Wed Dec 7 00:22:06 GMT 2005 State-Changed-Why: Submitted reports that this is an application error which results in linking against two thread libraries. http://www.freebsd.org/cgi/query-pr.cgi?pr=84778