From owner-freebsd-hackers@freebsd.org Wed May 24 09:07:19 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38942D7B9CB for ; Wed, 24 May 2017 09:07:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4AD61707 for ; Wed, 24 May 2017 09:07:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v4O97DZa030630 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 24 May 2017 12:07:13 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v4O97DZa030630 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v4O97DIY030629; Wed, 24 May 2017 12:07:13 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 24 May 2017 12:07:13 +0300 From: Konstantin Belousov To: Shrikanth Kamath Cc: freebsd-hackers@freebsd.org Subject: Re: Difference between p_vmspace quota between stable/11 and stable/10 Message-ID: <20170524090713.GG1622@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.8.2 (2017-04-18) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 May 2017 09:07:19 -0000 On Wed, May 24, 2017 at 01:00:51AM -0700, Shrikanth Kamath wrote: > I have a certain application(32 bit user space running in compat mode > in 64 bit system X86) which does a fork, and the child process does a > series of mmap calls as part of a scaling test. I am currently > debugging an issue with this application which hits a ENOMEM when it > is run on a stable/11 based setup and fails subsequent mmap and/or any > malloc calls v/s stable/10 where the issue is not seen.. I probed the > vm_map_find function using DTrace when "execname" was my application > in question, and got these readings > > fbt:kernel:vm_map_find:entry > /self->trace == 1/ /*enabled only during sys_mmap call of this application */ > { > @bytes[args[4]] = sum(args[4]); > printf("request length [%x]", args[4]); > } > > For stable_10 --> Total of 124 requests (length requested was > 0x500000) with the test successful > 124 * 0x500000 (5MB) ~ 620MB > > For stable_11 --> Total of 109 mmap requests > (0x500000/0x200000/0x3ff000 are the different vm_size_t length > arguments in vm_map_find). The test fails after 386MB has been > approved. > 24 * 0x500000 (5MB) ~ 120MB > 82 * 0x200000 (2MB) ~ 164MB > 3 * 0x3ff000 (4MB) ~ 12MB > > > The process parent rlimits are > > # cat /proc/5058/rlimit > > cpu -1 -1 > fsize -1 -1 > data 3221225472 3221225472 > stack 16777216 16777216 > core -1 -1 > rss 67108864 33265819648 > memlock 67108864 33265819648 > nproc 512 512 > nofile 1024 1024 > sbsize -1 -1 > vmem -1 -1 > npts -1 -1 > swap -1 -1 > kqueues -1 -1 > umtx -1 -1 > > The requests started failing in stable/11 with just 386 MB approved > v/s stable/10 which was successful in approving ~620MB. > > My stable/11 is from early days and is at GRN 302410 (probably 10 months old) > Any pointers or tips on what to probe further will be very helpful. Is > there any limits breach that I should probe further? The limits set > when a process is forked? > Should I probe the p->vmspace initiazliation? I doubt that limits are relevant for your issue. Look at the process address map at the moment when the request failed, I suspect that it is fragmented. Use procstat -v to examine the address space. You may spawn the tool from your program when mmap(2) fails.