From owner-freebsd-questions@FreeBSD.ORG Sat May 31 02:07:07 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 500AA106567A for ; Sat, 31 May 2008 02:07:07 +0000 (UTC) (envelope-from penoff@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.226]) by mx1.freebsd.org (Postfix) with ESMTP id 37FDE8FC23 for ; Sat, 31 May 2008 02:07:06 +0000 (UTC) (envelope-from penoff@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so129245rvf.43 for ; Fri, 30 May 2008 19:07:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=5g8QNVUWk/jot7y7V3L1izd5QRFsOlYA0hNYH+vsphI=; b=U/IqwVfsdBsoTHKDL1SY3AZeqvxWm0hyU5jUlwX+9ISQwCI2/i8wXf7Lsb5zn3tLL7sr/FO/2YKouNdoIMxt+BqJhZIqRVhf2jfxTbyElS3BX/XSQL9KrXHMH7ZYdZEqVCqQAHlPtqgz3iovV/rmwZgVrom+iUkzj1w6965HPPI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=enU34hqJwUzpxX6/R4fTrJM6g2h98+0FEg784p4XXStAfotE4mlcy2s1zSaubpTz2sTHGitFj0B1V57UPdAcXEXc4kOZzW6mVg4zRUOwL3MZYbPT1C+O2onHbR1Ve8hG5K3WmOM9FwdlhPNNZd8a3qXODJqKVSjfyjpTv6EDUOQ= Received: by 10.140.166.21 with SMTP id o21mr3482665rve.167.1212199626614; Fri, 30 May 2008 19:07:06 -0700 (PDT) Received: by 10.141.113.14 with HTTP; Fri, 30 May 2008 19:07:06 -0700 (PDT) Message-ID: Date: Fri, 30 May 2008 19:07:06 -0700 From: "Brad Penoff" Sender: penoff@gmail.com To: "Joshua Isom" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080520165437.36413c46.wmoran@potentialtech.com> <20080521090919.e3e777ec.wmoran@potentialtech.com> <20080521133645.e327ffdf.wmoran@potentialtech.com> X-Google-Sender-Auth: 24f1aa9394e57e06 Cc: Bill Moran , freebsd-questions@freebsd.org Subject: Re: resident memory limit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 May 2008 02:07:07 -0000 On Wed, May 21, 2008 at 11:46 AM, Joshua Isom wrote: > > On May 21, 2008, at 12:36 PM, Bill Moran wrote: > >> In response to "Brad Penoff" : >>> >>> On Wed, May 21, 2008 at 6:09 AM, Bill Moran >>> wrote: >>>> >>>> In response to "Brad Penoff" : >>>> >>>>> On Tue, May 20, 2008 at 2:39 PM, Brad Penoff wrote: >>>>>> >>>>>> On Tue, May 20, 2008 at 1:54 PM, Bill Moran >>>>>> wrote: >>>>>>> >>>>>>> In response to "Brad Penoff" : >>>>>>>> >>>>>>>> I have an application that runs on Linux or Mac OS X but seems to >>>>>>>> have >>>>>>>> a problem when I run on FreeBSD (6.3 or 7). The issue is the memory >>>>>>>> footprint for the application (osubw_sctpclien below) is quite >>>>>>>> large; >>>>>>>> on Linux it can be as much as 950 MB in resident memory, according >>>>>>>> to >>>>>>>> top. However, on FreeBSD I start to get ENOMEM always around the >>>>>>>> time >>>>>>>> my resident memory size is about 200 MB. >>>>>>>> >>>>>>>> I read a few posts and have seen people fixing their problems by >>>>>>>> adjusting kern.maxdsiz in /boot/loader.conf and/or by adding a swap >>>>>>>> file. I've tried both and for my application, it still seems to be >>>>>>>> limited to 200 MB resident memory regardless of maxdsize and swap >>>>>>>> file >>>>>>>> setting. I wrote a toy application (malloctest below) that calls >>>>>>>> malloc in a while(1) and breaks once it gets ENOMEM (doing another >>>>>>>> while(1) so it doesn't exit); this application's memory size in top >>>>>>>> always matches the kern.maxdsiz setting, however it has a very low >>>>>>>> resident memory number, according to top. >>>>>>> >>>>>>> Have a look at /etc/login.conf and the associated man pages. >>>>>>> >>>>> >>>>> BTW, we've seen the exact behavior on FreeBSD 7 as well (6.3 was >>>>> reported here). We've tried on different hardware as well, and keep >>>>> getting haunted by this resident memory limit that we don't know how >>>>> to set. >>>>> >>>>> Any idea why, in the data I originally reported, I can allocate >>>>> kern.maxdsiz + swap (see SIZE from top output) for malloc(1 MB) in a >>>>> while loop, yet the top value for RES is always really low? >>>>> >>>>> How come, in contrast, my application starts to report ENOMEM when >>>>> SIZE is 203 MB and RES is 201 MB? This is why I titled the thread >>>>> asking about an unknown (to me ;-) limit for resident memory... >>>> >>>> It's called memory overcommit. If the OS thinks it _might_ be able >>>> to get you the memory, it will allow it. You only actually use the >>>> memory when you start putting data in it (hence the difference between >>>> SIZE and RES) Add a statement to fill up the malloc()ed memory with >>>> some sort of data in your loop, and you'll see different behaviour. >>>> >>>> As to what's limiting your application, I'm not sure. What does the >>>> output of 'ulimit -a' say? >>>> >>> >>> Thanks again for your time. >>> >>> >>> With the default loader.conf, my "limit -a" output is: >>> >>> Resource limits (current): >>> cputime infinity secs >>> filesize infinity kB >>> datasize 524288 kB >>> stacksize 65536 kB >>> coredumpsize infinity kB >>> memoryuse infinity kB >>> memorylocked infinity kB >>> maxprocesses 5547 >>> openfiles 11095 >>> sbsize infinity bytes >>> vmemoryuse infinity kB >>> >>> My application starts getting ENOMEM when I have 201 MB of resident >>> memory. >>> >>> >>> >>> When I change my loader.conf to match the 2 GB of physical memory that I >>> have: >>> kern.maxdsiz="2147483648" >>> kern.maxssiz="2147483648" >>> kern.dfldsiz="2147483648" >>> >>> ...and reboot, then my "limit -a" output is: >>> >>> Resource limits (current): >>> cputime infinity secs >>> filesize infinity kB >>> datasize 2097152 kB >>> stacksize 2097152 kB >>> coredumpsize infinity kB >>> memoryuse infinity kB >>> memorylocked infinity kB >>> maxprocesses 5547 >>> openfiles 11095 >>> sbsize infinity bytes >>> vmemoryuse infinity kB >>> >>> >>> However, the application still seems to max out at 201 MB of resident >>> memory. >>> >>> >>> People suggest to fix my login.conf but the memory related fields are >>> set to unlimited... Any ideas where this 201 MB limit of resident >>> memory comes from? >> >> That's pretty strange. If I had to guess, I would guess that there is no >> 201M limit, but that you're hitting some other limit that just happens to >> predictably occur at 201M with that program. >> >> I'm kind of grasping at straws here, so hopefully I won't lead you on a >> wild goose chase, but I would look next at putting some debugging in >> /etc/malloc.conf and seeing if you get any useful information from it >> (see the malloc man page). From there, possibly a ktrace of the process. >> Hopefully you have source code for the program and can compile it with >> debugging and run it under gdb. >> >> -- >> Bill Moran >> http://www.potentialtech.com >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to >> "freebsd-questions-unsubscribe@freebsd.org" >> >> I'm still playing around with malloc.conf and ktrace, searching for answers (I'll report if I find any) but in the meantime, I'll try to answer your questions... > > Here's a question I haven't seen asked yet. How much memory is it trying to allocate? If it can't get everything it's asking for it can fail. This same application runs on both Mac OS X and Linux. On Linux, the maximum memory demands shown in "top" were VIRT=955MB RES=929MB. This was actually on a machine with only 1 GB of RAM. This same program starts getting ENOMEM on FreeBSD on a machine with 2 GB of RAM when RES is 201 MB; the same occurs on FreeBSD on a machine with 1 GB of RAM (same HW as Linux). On FreeBSD, I can run a dummy program that calls malloc in a loop essential until the value of SIZE on "top" (same as VIRT in Linux) is kern.maxdsiz (which I set to 2GB in loader.conf). > Also, how is the application being started? There could be some setting in the shell startup that's putting a limit. I run it as root which uses /bin/csh . I did a chsh as root to bash and retried the same app but still reached the same magic 201 MB limit. > Is it a native FreeBSD program or is it a linux executable running under simulation? It's a native FreeBSD program. brad