From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 13 21:45:42 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DCC11065670; Fri, 13 Apr 2012 21:45:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id E37948FC0C; Fri, 13 Apr 2012 21:45:41 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3DLjaw7093212; Sat, 14 Apr 2012 00:45:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3DLjapt081824; Sat, 14 Apr 2012 00:45:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3DLja5d081823; Sat, 14 Apr 2012 00:45:36 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 14 Apr 2012 00:45:36 +0300 From: Konstantin Belousov To: Sushanth Rai Message-ID: <20120413214536.GL2358@deviant.kiev.zoral.com.ua> References: <1334342264.8672.YahooMailClassic@web180006.mail.gq1.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="p9yXrYA3ANr0wrJ4" Content-Disposition: inline In-Reply-To: <1334342264.8672.YahooMailClassic@web180006.mail.gq1.yahoo.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: alc@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Apr 2012 21:45:42 -0000 --p9yXrYA3ANr0wrJ4 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Apr 13, 2012 at 11:37:44AM -0700, Sushanth Rai wrote: > I've attached the simple program that creates 5 threads. Following is the= o/p of /proc//map when this program is running. Note that I modified= =20 > sys/fs/procfs/procfs_map.c to print whether a region is wired. As you can= see from this o/p, none of stack areas get wired.=20 >=20 > 0x400000 0x401000 1 0 0xffffff002d943bd0 r-x 1 0 0x1000 COW NC wired vnod= e /var/tmp/thread1 > 0x500000 0x501000 1 0 0xffffff002dd13e58 rw- 2 0 0x3100 NCOW NNC wired de= fault - > 0x501000 0x600000 255 0 0xffffff002dd13e58 rwx 2 0 0x3100 NCOW NNC wired = default - > 0x800500000 0x800526000 38 0 0xffffff0025574000 r-x 192 46 0x1004 COW NC = wired vnode /libexec/ld-elf.so.1 > 0x800526000 0x800537000 17 0 0xffffff002d9f81b0 rw- 1 0 0x3100 NCOW NNC w= ired default - > 0x800626000 0x80062d000 7 0 0xffffff002dd13bd0 rw- 1 0 0x3100 COW NNC wir= ed vnode /libexec/ld-elf.so.1 > 0x80062d000 0x800633000 6 0 0xffffff002dd145e8 rw- 1 0 0x3100 NCOW NNC wi= red default - > 0x800633000 0x800645000 18 0 0xffffff00256d71b0 r-x 63 42 0x4 COW NC wire= d vnode /lib/libthr.so.3 > 0x800645000 0x800646000 1 0 0xffffff002d975510 r-x 1 0 0x3100 COW NNC wir= ed vnode /lib/libthr.so.3 > 0x800646000 0x800746000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC no= t-wired default - > 0x800746000 0x80074a000 4 0 0xffffff002572a288 rw- 1 0 0x3100 COW NNC wir= ed vnode /lib/libthr.so.3 > 0x80074a000 0x80074c000 2 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wi= red default - > 0x80074c000 0x80083e000 242 0 0xffffff001cd226c0 r-x 238 92 0x1004 COW NC= wired vnode /lib/libc.so.7 > 0x80083e000 0x80083f000 1 0 0xffffff002dd12000 r-x 1 0 0x3100 COW NNC wir= ed vnode /lib/libc.so.7 > 0x80083f000 0x80093e000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC no= t-wired default - > 0x80093e000 0x80095d000 31 0 0xffffff002dddc360 rw- 1 0 0x3100 COW NNC wi= red vnode /lib/libc.so.7 > 0x80095d000 0x800974000 23 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC w= ired default - > 0x800a00000 0x800b00000 256 0 0xffffff002dbd1798 rw- 1 0 0x3100 NCOW NNC = wired default - > 0x800b00000 0x800c00000 256 0 0xffffff002dd14948 rw- 1 0 0x3100 NCOW NNC = wired default - > 0x7fffff3db000 0x7fffff3fb000 1 0 0xffffff002dbb4360 rw- 1 0 0x3100 NCOW = NNC not-wired default - > 0x7fffff5dc000 0x7fffff5fc000 1 0 0xffffff002dc66af8 rw- 1 0 0x3100 NCOW = NNC not-wired default - > 0x7fffff7dd000 0x7fffff7fd000 1 0 0xffffff002dbea438 rw- 1 0 0x3100 NCOW = NNC not-wired default - > 0x7fffff9de000 0x7fffff9fe000 1 0 0xffffff002dd7fd80 rw- 1 0 0x3100 NCOW = NNC not-wired default - > 0x7fffffbdf000 0x7fffffbff000 1 0 0xffffff002dbe9438 rw- 1 0 0x3100 NCOW = NNC not-wired default - > 0x7fffffbff000 0x7fffffc00000 0 0 0 --- 0 0 0x0 NCOW NNC not-wired none - > 0x7ffffffe0000 0x800000000000 32 0 0xffffff002dd125e8 rwx 1 0 0x3100 NCOW= NNC wired default - >=20 > --- On Fri, 4/13/12, Konstantin Belousov wrote: >=20 > > From: Konstantin Belousov > > Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN > > To: "Sushanth Rai" > > Cc: freebsd-hackers@freebsd.org > > Date: Friday, April 13, 2012, 1:11 AM > > On Thu, Apr 12, 2012 at 08:10:26PM > > -0700, Sushanth Rai wrote: > > > >=20 > > > > Then it should be fixed in r190885. > > > >=20 > > >=20 > > > Thanks. That works like a charm.=20 > > >=20 > > > mlockall() mostly works now. There is still a, issue in > > wiring the stacks of multithreaded program when the program > > uses default stack allocation scheme. Thread library > > allocates stack for each thread by calling mmap() and > > sending address and size to be mapped. The kernel adjusts > > the start address to sgrowsz in=9A vm_map_stack() and > > maps at the adjusted address. But the subsequent wiring is > > done using the original address, which fails.=20 Oh, I see. The problem is the VM_MAP_WIRE_NOHOLES flag. Since we map only the initial stack fragment even for the MCL_WIREFUTURE maps, there is a hole in the stack region. In fact, for MCL_WIREFUTURE, we probably should map the whole stack at once, prefaulting all pages. Below are two patches. The change for vm_mmap.c would fix your immediate problem by allowing holes in wired region. The change for vm_map.c prefaults the whole stack instead of the initial fragment. The single-threaded programs still get a fault on stack growth. diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c index 6198629..2fd18d1 100644 --- a/sys/vm/vm_map.c +++ b/sys/vm/vm_map.c @@ -3259,7 +3259,10 @@ vm_map_stack(vm_map_t map, vm_offset_t addrbos, vm_s= ize_t max_ssize, addrbos + max_ssize < addrbos) return (KERN_NO_SPACE); =20 - init_ssize =3D (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz; + if (map->flags & MAP_WIREFUTURE) + init_ssize =3D max_ssize; + else + init_ssize =3D (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz; =20 PROC_LOCK(curthread->td_proc); vmemlim =3D lim_cur(curthread->td_proc, RLIMIT_VMEM); diff --git a/sys/vm/vm_mmap.c b/sys/vm/vm_mmap.c index 2588c85..3fccd9e 100644 --- a/sys/vm/vm_mmap.c +++ b/sys/vm/vm_mmap.c @@ -1561,9 +1561,11 @@ vm_mmap(vm_map_t map, vm_offset_t *addr, vm_size_t s= ize, vm_prot_t prot, * If the process has requested that all future mappings * be wired, then heed this. */ - if (map->flags & MAP_WIREFUTURE) + if (map->flags & MAP_WIREFUTURE) { vm_map_wire(map, *addr, *addr + size, - VM_MAP_WIRE_USER | VM_MAP_WIRE_NOHOLES); + VM_MAP_WIRE_USER | ((flags & MAP_STACK) ? + VM_MAP_WIRE_HOLESOK : VM_MAP_WIRE_NOHOLES)); + } } else { /* * If this mapping was accounted for in the vnode's --p9yXrYA3ANr0wrJ4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+Inn8ACgkQC3+MBN1Mb4iolQCcCiKLilWM5Ok8fuf5rSOAyQlu h/IAoOL9B0Pn/HYRABYAN0uKsOXUy+bP =QwC2 -----END PGP SIGNATURE----- --p9yXrYA3ANr0wrJ4--