From owner-freebsd-performance@FreeBSD.ORG Thu Nov 24 18:05:52 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E33B16A41F for ; Thu, 24 Nov 2005 18:05:52 +0000 (GMT) (envelope-from m@obmail.net) Received: from stewie.obfuscated.net (stewie.obfuscated.net [66.118.188.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 53B2B43D46 for ; Thu, 24 Nov 2005 18:05:44 +0000 (GMT) (envelope-from m@obmail.net) Received: from [192.168.1.101] (163-2.26-24.tampabay.res.rr.com [24.26.2.163]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by stewie.obfuscated.net (Postfix) with ESMTP id D303361F5 for ; Thu, 24 Nov 2005 13:05:38 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v746.2) In-Reply-To: <20051123221952.D32130@fledge.watson.org> References: <6C48A943-1AB3-4DF4-B8CC-CD75B9F36E98@obmail.net> <20051123221952.D32130@fledge.watson.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <3EF0A80D-0577-46E2-9143-B849CFF48197@obmail.net> Content-Transfer-Encoding: 7bit From: Michael Conlen Date: Thu, 24 Nov 2005 13:05:39 -0500 To: freebsd-performance@freebsd.org X-Mailer: Apple Mail (2.746.2) Subject: Re: mmap() X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2005 18:05:52 -0000 On Nov 23, 2005, at 5:21 PM, Robert Watson wrote: > > On Wed, 23 Nov 2005, Michael Conlen wrote: > >> Sorry if this is the wrong place for this, but I haven't been >> getting answers elsewhere. >> >> I'm trying to tune the system to allow very large mmap()'s in a >> single process space, something on the order of 1.5 GB so I can >> pass very large values for -Xms and -Xmx to java. I know I had >> been able to do this on FreeBSD in the past but recent versions of >> either Java or FreeBSD aren't playing nicely. currently.. > > BTW, you may find it useful to use procfs to inspect the address > space layout of your process. You can d: > > mkdir /proc > mount -t procfs proc /proc > cd /proc/pid > dd if=map of=/dev/stdout bs=20k > > This can help you look for fragmentation of process address space, > among other things. Thanks. I forgot when you mmap you have to have contiguous space... ... 0x8048000 0x8049000 0 0 0xc3c79630 r-x 1 0 0x0 COW NC vnode /tmp/foo 0x8049000 0x804a000 1 0 0xc5ad718c rw- 2 0 0x2180 NCOW NNC default - 0x804a000 0x804c000 2 0 0xc5ad718c rwx 2 0 0x2180 NCOW NNC default - 0x87f49000 0x87f69000 32 0 0xc10406b4 r-x 80 29 0x4 COW NC vnode / libexec/ld-elf That looks like about 2 GB of space between the last two lines. The rest is as follows 0x87f69000 0x87f6a000 1 0 0xc5ae1d68 rw- 1 0 0x2180 COW NNC vnode / libexec/ld-elf.so.1 0x87f6a000 0x87f6f000 5 0 0xc3b4db58 rw- 2 0 0x2180 NCOW NNC default - 0x87f6f000 0x87f77000 7 0 0xc3b4db58 rwx 2 0 0x2180 NCOW NNC default - 0x87f77000 0x88041000 108 0 0xc3b3bd68 r-x 1 0 0x2180 COW NNC vnode / lib/libc.so.5 0x88041000 0x88042000 1 0 0xc5ad6840 r-x 1 0 0x2180 COW NNC vnode / lib/libc.so.5 0x88042000 0x88047000 5 0 0xc5ad418c rwx 1 0 0x2180 COW NNC vnode / lib/libc.so.5 0x88047000 0x8805c000 6 0 0xc5ad75ac rwx 1 0 0x2180 NCOW NNC default - 0xbfbe0000 0xbfc00000 3 0 0xc3c9718c rwx 1 0 0x2180 NCOW NNC default - If I change from using mmap to malloc() I get the following first four lines show malloc() on the heap 0x8048000 0x8049000 1 0 0xc3b2c318 r-x 1 0 0x0 COW NC vnode /tmp/foo 0x8049000 0x804a000 1 0 0xc5ae3108 rw- 2 0 0x2180 NCOW NNC default - 0x804a000 0x3fc4c000 2 0 0xc5ae3108 rwx 2 0 0x2180 NCOW NNC default - 0x87f49000 0x87f69000 32 0 0xc10406b4 r-x 83 31 0x4 COW NC vnode / libexec/ld-elf and it looks like a smaller mmap() shows up at the end 0x88041000 0x88042000 1 0 0xc5adcce4 r-x 1 0 0x2180 COW NNC vnode / lib/libc.so.5 0x88042000 0x88047000 5 0 0xc5ad6420 rwx 1 0 0x2180 COW NNC vnode / lib/libc.so.5 0x88047000 0x8805b000 5 0 0xc5ae1ce4 rwx 3 0 0x2180 NCOW NNC default - 0x8805b000 0xbfb5b000 0 0 0xc5ae1ce4 --- 3 0 0x2180 NCOW NNC default - 0xbfb5b000 0xbfb5c000 1 0 0xc5ae1ce4 rwx 3 0 0x2180 NCOW NNC default - I presume the stack is coming up from the bottom. The end looks about 5 MB short of 3 GB. In all it looks like about 1 GB off the top (kernel I presume), the heap, followed by the libraries which IIRC are mmap()'ed coming down after the end of the heap and I presume the stack coming up from the bottom. If I reduce the limit for the datasize locally using limit it doesn't seem to actually free up space for the stack. If I change loader.conf to reduce the datasize then the space is freed up to do the mmap(). If I leave the max datasize and reduce the default to 1 GB I don't get any change in the memory map at all. So here's the problem, I've got a DB server that needs a large datasize and a tomcat server which occasionally needs to use a lot of memory, which java allocates from a memory mapped space. Any ideas how to get the system to allow processes to have either/or? -- Michael Conlen