From owner-freebsd-java@FreeBSD.ORG Mon Mar 31 21:01:17 2008 Return-Path: Delivered-To: freebsd-java@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E13781065671 for ; Mon, 31 Mar 2008 21:01:16 +0000 (UTC) (envelope-from lists@intricatesoftware.com) Received: from mail1.intricatesoftware.com (cl-18.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:11::2]) by mx1.freebsd.org (Postfix) with ESMTP id A0F518FC2A for ; Mon, 31 Mar 2008 21:01:15 +0000 (UTC) (envelope-from lists@intricatesoftware.com) Received: from seraph.intricatesoftware.com (relay@localhost.intricatesoftware.com [IPv6:::1]) by mail1.intricatesoftware.com (8.14.1/8.13.4) with ESMTP id m2VL0tY8025616; Mon, 31 Mar 2008 17:00:56 -0400 (EDT) Received: from seraph.intricatesoftware.com (truk@localhost.intricatesoftware.com [127.0.0.1]) by seraph.intricatesoftware.com (8.14.1/8.14.1) with ESMTP id m2VL0t9n007162; Mon, 31 Mar 2008 17:00:55 -0400 (EDT) Received: (from truk@localhost) by seraph.intricatesoftware.com (8.14.1/8.14.1/Submit) id m2VL0slu013326; Mon, 31 Mar 2008 17:00:54 -0400 (EDT) X-Authentication-Warning: seraph.intricatesoftware.com: truk set sender to lists@intricatesoftware.com using -f From: Kurt Miller To: freebsd-java@freebsd.org Date: Mon, 31 Mar 2008 17:00:54 -0400 User-Agent: KMail/1.9.7 References: <20080323191540.9FCAC8FC16@mx1.freebsd.org> In-Reply-To: <20080323191540.9FCAC8FC16@mx1.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803311700.54567.lists@intricatesoftware.com> X-SMTP-Vilter-Version: 1.3.6 X-SMTP-Vilter-Virus-Backend: clamd X-SMTP-Vilter-Status: clean X-SMTP-Vilter-clamd-Virus-Status: clean X-Spamd-Symbols: ALL_TRUSTED,BAYES_00 X-SMTP-Vilter-Spam-Backend: spamd X-Spam-Score: -4.4 X-Spam-Threshold: 5.0 X-Spam-Probability: -0.9 X-SMTP-Vilter-Unwanted-Backend: attachment X-SMTP-Vilter-attachment-Unwanted-Status: clean Cc: Dominic Bishop Subject: Re: File descriptor leaks under java 1.6 X-BeenThere: freebsd-java@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kurt@intricatesoftware.com List-Id: Porting Java to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Mar 2008 21:01:17 -0000 On Sunday 23 March 2008 3:13:33 pm Dominic Bishop wrote: > I have an application that makes use of 2 way socket communcations, so uses > Socket/ServerSocket, this is all standard IO, no java.nio is involved. > > When running this under diablo 1.5 everything works fine, watching allocated > file descriptors with lsof shows nothing abnormal whatsoever. > > If I run this under a port build JDK 1.6 then it starts leaking file > descriptors originating from TCP, until eventually the available descriptors > are exhausted and the application crashes with a 'too many open files' > error. > > I've tried both patchset 3 and 4 of the port, under amd64 and i386 > architecture FreeBSD 6.3 and the same problem occurs under all > circumstances. > > The problem file descriptors in the lsof output appear as: > > java 47065 djb 46u IPv4 0xffffff01354a4000 0t0 TCP *:* > (CLOSED) > java 47065 djb 47u IPv4 0xffffff006b2875f0 0t0 TCP *:* > (CLOSED) > java 47065 djb 48u IPv4 0xffffff0001bdcbe0 0t0 TCP *:* > (CLOSED) > java 47065 djb 49u IPv4 0xffffff015924c000 0t0 TCP *:* > (CLOSED) > java 47065 djb 50u IPv4 0xffffff01375078e8 0t0 TCP *:* > (CLOSED) > > I ran the application with -Xrunhprof enabled under jdk 1.6 and then used > jhat to look at the resulting dump. > > I managed to determine from this that the FD number (ie 46u) corresponds to > the 'fd' field in a java.io.FileDescriptor instance. > > Using the Object Query Language in jhat I went looking for these file > descriptor instances using: > > 'select f from java.io.FileDescriptor f where f.fd == ##' > > For numbers which were genuine file descriptors (ie connected sockets etc) > this would always find the file descriptor instance and walking through the > referenced objects/fields it was clear it had located the correct file > descriptor. By that I mean the connected address, or for actual files, the > pointed to filename agreed with the lsof output. > > If however I tried this query looking for file descriptor numbers which lsof > was showing in the CLOSED state then it would fail to find them at all, so > no FileDescriptor instance in the java heap referenced these file descriptor > numbers. > > To make sure I did a: > 'select f.fd from java.io.FileDescriptor f' > > This listed the 'fd' field value for all instances of java.io.FileDescriptor > in the heap, I had numbers for the ones expected (ie actual bound sockets, > open files/pipes etc that lsof showed) as well as quite a few set to -1 and > a lot set to 0. Presumably these last 2 are either newly created instances > or ones awaiting garbage collection. > > I've basically completely run out of ideas as to where these file > descriptors are leaking, and given the working behaviour under diablo 1.5 > and the lack of references in the heap dump I'm no longer sure that this is > actually a problem in my code. > > Anyone more familiar with the JVM internals got any suggestions as to where > to go next? Hi Dominic, My next step would be to create a minimal test case and post it here to see if others can reproduce it. A minimal test case is also useful to narrow the scope of what JVM code needs to be reviewed to find the cause. -Kurt