From owner-freebsd-hackers@freebsd.org Thu May 25 08:13:41 2017 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06890D81EBD for ; Thu, 25 May 2017 08:13:41 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-qk0-x232.google.com (mail-qk0-x232.google.com [IPv6:2607:f8b0:400d:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B711910AD for ; Thu, 25 May 2017 08:13:40 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: by mail-qk0-x232.google.com with SMTP id a72so171705320qkj.2 for ; Thu, 25 May 2017 01:13:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=vemppGDg1iRY9CtTyPoL3F82RV+s1zptpvX0OhHrqdc=; b=NvrBxRni2rJvoDwDpeYs+l356i0JGFsZLDPE4guS2YgcLDZ+4juvrCEjtPw/Qpqjoa v4G4yWCFAPH4oFw6gXenXT/Y4Vl5LpwZHxMDAE8zL0xF0AwktJHCoQKeUVFZvMDBYvfz Zi5UVki7QgGCVbc2s5xig7gjAru0T22Rk8QwnjqB26n9Iq2ja0VKaNvGzmdtiBuU0cAL t5NnKmF1uPiziK1iZsh9WaWCzarj5MbEd8l54Kl2CwFl77yY58BVvfw/jP+Iylqoooi3 SZBxMMneqlEpzXWBm5m8rFX31xUiwdiP7SamDeoKv5+w2X+xC2ZJueX+c1s5fEpndU9F 6lgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=vemppGDg1iRY9CtTyPoL3F82RV+s1zptpvX0OhHrqdc=; b=n46T9PevNI7hkKxqFVTeLgxQ155GrIT5u+I4ve8ocNAv8dSXx9QQJmGSIKgSmx9H96 eY08JK7xzSOk5fYjInjiLk97EP1GoLGYirpf09UB7pPDHr4QXBUk5TCwgFGUYZzW8/H3 jzufxKAQJ8fxbbWUe/OZMRHz4M5WK756xyu8GrnrZIliCSo0AWCPRcxKhi7HzAp0DAl+ hjk6qd80VMY6JzJgcGw2E40Qd53eo5zT35B+71wnWptujIk7+nflgZ8V04ALQeeOZdjl iVocilhpANMGyG4PC6mwPCkLP0+y+DqMaspYjAvjCPgFSxuZehcVb8AOUyHjYWVaPDcS E8Pw== X-Gm-Message-State: AODbwcBR8WivsGv8SKAUurvEZJ1z3pk+tkxSDP5EQr2a74wBRK6AWztE 7Kg1sxJpdn0SqhaMgfOpqw2d88lbCQ== X-Received: by 10.55.73.133 with SMTP id w127mr37212503qka.4.1495700019503; Thu, 25 May 2017 01:13:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.38.241 with HTTP; Thu, 25 May 2017 01:13:39 -0700 (PDT) From: Shrikanth Kamath Date: Thu, 25 May 2017 01:13:39 -0700 Message-ID: Subject: Re: Difference between p_vmspace quota between stable/11 and stable/10 To: Konstantin Belousov , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 08:13:41 -0000 Thanks for the reply Konstantin, I captured the procstat -v snapshots for the forked process (got the readings for both stable/10 vs stable/11). I am still trying to figure how to interpret these mappings, In stable/11 the first few lines of procstat -v show PID START END PRT RES PRES REF SHD FLAG TP PATH 19933 0x8048000 0x877e000 r-x 933 1003 2 1 CN-- vn /packages/mnt/junos-platform/sbin/dcd 19933 0x877e000 0x87f2000 rw- 70 0 1 0 C--- vn /packages/mnt/junos-platform/sbin/dcd 19933 0x87f2000 0x8a73000 rw- 59 59 1 0 C--- df 19933 0xc8797000 0xc87a1000 rw- 10 10 1 0 CN-- df The same for stable/10 show PID START END PRT RES PRES REF SHD FL TP PATH 43678 0x8048000 0x8779000 r-x 943 1014 2 1 CN-- vn /packages/mnt/junos-platform/sbin/dcd 43678 0x8779000 0x87ed000 rw- 70 0 1 0 C--- vn /packages/mnt/junos-platform/sbin/dcd 43678 0x87ed000 0x2cc00000 rw- 145872 145872 1 0 C-S- df 43678 0xc8792000 0xc879c000 rw- 10 10 1 0 C--- df The third entry in two cases show a stark difference, does this indicated the space that was setup was much lower compared to stable/10? -- Shrikanth R K From: Konstantin Belousov To: Shrikanth Kamath Cc: freebsd-hackers@freebsd.org Subject: Re: Difference between p_vmspace quota between stable/11 and stable/10 Message-ID: <20170524090713.GG1622@kib.kiev.ua> Content-Type: text/plain; charset=us-ascii On Wed, May 24, 2017 at 01:00:51AM -0700, Shrikanth Kamath wrote: > I have a certain application(32 bit user space running in compat mode > in 64 bit system X86) which does a fork, and the child process does a > series of mmap calls as part of a scaling test. I am currently > debugging an issue with this application which hits a ENOMEM when it > is run on a stable/11 based setup and fails subsequent mmap and/or any > malloc calls v/s stable/10 where the issue is not seen.. I probed the > vm_map_find function using DTrace when "execname" was my application > in question, and got these readings > > fbt:kernel:vm_map_find:entry > /self->trace == 1/ /*enabled only during sys_mmap call of this application */ > { > @bytes[args[4]] = sum(args[4]); > printf("request length [%x]", args[4]); > } > > For stable_10 --> Total of 124 requests (length requested was > 0x500000) with the test successful > 124 * 0x500000 (5MB) ~ 620MB > > For stable_11 --> Total of 109 mmap requests > (0x500000/0x200000/0x3ff000 are the different vm_size_t length > arguments in vm_map_find). The test fails after 386MB has been > approved. > 24 * 0x500000 (5MB) ~ 120MB > 82 * 0x200000 (2MB) ~ 164MB > 3 * 0x3ff000 (4MB) ~ 12MB > > > The process parent rlimits are > > # cat /proc/5058/rlimit > > cpu -1 -1 > fsize -1 -1 > data 3221225472 3221225472 > stack 16777216 16777216 > core -1 -1 > rss 67108864 33265819648 > memlock 67108864 33265819648 > nproc 512 512 > nofile 1024 1024 > sbsize -1 -1 > vmem -1 -1 > npts -1 -1 > swap -1 -1 > kqueues -1 -1 > umtx -1 -1 > > The requests started failing in stable/11 with just 386 MB approved > v/s stable/10 which was successful in approving ~620MB. > > My stable/11 is from early days and is at GRN 302410 (probably 10 months old) > Any pointers or tips on what to probe further will be very helpful. Is > there any limits breach that I should probe further? The limits set > when a process is forked? > Should I probe the p->vmspace initiazliation? I doubt that limits are relevant for your issue. Look at the process address map at the moment when the request failed, I suspect that it is fragmented. Use procstat -v to examine the address space. You may spawn the tool from your program when mmap(2) fails. -- Shrikanth R K