Date: Sat, 21 Apr 2012 13:22:31 -0500 From: Alan Cox <alc@rice.edu> To: Konstantin Belousov <kostikbel@gmail.com> Cc: alc@freebsd.org, freebsd-hackers@freebsd.org, Sushanth Rai <sushanth_rai@yahoo.com> Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN Message-ID: <4F92FAE7.3090702@rice.edu> In-Reply-To: <20120413214536.GL2358@deviant.kiev.zoral.com.ua> References: <1334342264.8672.YahooMailClassic@web180006.mail.gq1.yahoo.com> <20120413214536.GL2358@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On 04/13/2012 16:45, Konstantin Belousov wrote:
> On Fri, Apr 13, 2012 at 11:37:44AM -0700, Sushanth Rai wrote:
>> I've attached the simple program that creates 5 threads. Following is the o/p of /proc/<pid>/map when this program is running. Note that I modified
>> sys/fs/procfs/procfs_map.c to print whether a region is wired. As you can see from this o/p, none of stack areas get wired.
>>
>> 0x400000 0x401000 1 0 0xffffff002d943bd0 r-x 1 0 0x1000 COW NC wired vnode /var/tmp/thread1
>> 0x500000 0x501000 1 0 0xffffff002dd13e58 rw- 2 0 0x3100 NCOW NNC wired default -
>> 0x501000 0x600000 255 0 0xffffff002dd13e58 rwx 2 0 0x3100 NCOW NNC wired default -
>> 0x800500000 0x800526000 38 0 0xffffff0025574000 r-x 192 46 0x1004 COW NC wired vnode /libexec/ld-elf.so.1
>> 0x800526000 0x800537000 17 0 0xffffff002d9f81b0 rw- 1 0 0x3100 NCOW NNC wired default -
>> 0x800626000 0x80062d000 7 0 0xffffff002dd13bd0 rw- 1 0 0x3100 COW NNC wired vnode /libexec/ld-elf.so.1
>> 0x80062d000 0x800633000 6 0 0xffffff002dd145e8 rw- 1 0 0x3100 NCOW NNC wired default -
>> 0x800633000 0x800645000 18 0 0xffffff00256d71b0 r-x 63 42 0x4 COW NC wired vnode /lib/libthr.so.3
>> 0x800645000 0x800646000 1 0 0xffffff002d975510 r-x 1 0 0x3100 COW NNC wired vnode /lib/libthr.so.3
>> 0x800646000 0x800746000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC not-wired default -
>> 0x800746000 0x80074a000 4 0 0xffffff002572a288 rw- 1 0 0x3100 COW NNC wired vnode /lib/libthr.so.3
>> 0x80074a000 0x80074c000 2 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wired default -
>> 0x80074c000 0x80083e000 242 0 0xffffff001cd226c0 r-x 238 92 0x1004 COW NC wired vnode /lib/libc.so.7
>> 0x80083e000 0x80083f000 1 0 0xffffff002dd12000 r-x 1 0 0x3100 COW NNC wired vnode /lib/libc.so.7
>> 0x80083f000 0x80093e000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC not-wired default -
>> 0x80093e000 0x80095d000 31 0 0xffffff002dddc360 rw- 1 0 0x3100 COW NNC wired vnode /lib/libc.so.7
>> 0x80095d000 0x800974000 23 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wired default -
>> 0x800a00000 0x800b00000 256 0 0xffffff002dbd1798 rw- 1 0 0x3100 NCOW NNC wired default -
>> 0x800b00000 0x800c00000 256 0 0xffffff002dd14948 rw- 1 0 0x3100 NCOW NNC wired default -
>> 0x7fffff3db000 0x7fffff3fb000 1 0 0xffffff002dbb4360 rw- 1 0 0x3100 NCOW NNC not-wired default -
>> 0x7fffff5dc000 0x7fffff5fc000 1 0 0xffffff002dc66af8 rw- 1 0 0x3100 NCOW NNC not-wired default -
>> 0x7fffff7dd000 0x7fffff7fd000 1 0 0xffffff002dbea438 rw- 1 0 0x3100 NCOW NNC not-wired default -
>> 0x7fffff9de000 0x7fffff9fe000 1 0 0xffffff002dd7fd80 rw- 1 0 0x3100 NCOW NNC not-wired default -
>> 0x7fffffbdf000 0x7fffffbff000 1 0 0xffffff002dbe9438 rw- 1 0 0x3100 NCOW NNC not-wired default -
>> 0x7fffffbff000 0x7fffffc00000 0 0 0 --- 0 0 0x0 NCOW NNC not-wired none -
>> 0x7ffffffe0000 0x800000000000 32 0 0xffffff002dd125e8 rwx 1 0 0x3100 NCOW NNC wired default -
>>
>> --- On Fri, 4/13/12, Konstantin Belousov<kostikbel@gmail.com> wrote:
>>
>>> From: Konstantin Belousov<kostikbel@gmail.com>
>>> Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN
>>> To: "Sushanth Rai"<sushanth_rai@yahoo.com>
>>> Cc: freebsd-hackers@freebsd.org
>>> Date: Friday, April 13, 2012, 1:11 AM
>>> On Thu, Apr 12, 2012 at 08:10:26PM
>>> -0700, Sushanth Rai wrote:
>>>>> Then it should be fixed in r190885.
>>>>>
>>>> Thanks. That works like a charm.
>>>>
>>>> mlockall() mostly works now. There is still a, issue in
>>> wiring the stacks of multithreaded program when the program
>>> uses default stack allocation scheme. Thread library
>>> allocates stack for each thread by calling mmap() and
>>> sending address and size to be mapped. The kernel adjusts
>>> the start address to sgrowsz in vm_map_stack() and
>>> maps at the adjusted address. But the subsequent wiring is
>>> done using the original address, which fails.
> Oh, I see. The problem is the VM_MAP_WIRE_NOHOLES flag. Since we
> map only the initial stack fragment even for the MCL_WIREFUTURE maps,
> there is a hole in the stack region.
>
> In fact, for MCL_WIREFUTURE, we probably should map the whole
> stack at once, prefaulting all pages.
>
> Below are two patches. The change for vm_mmap.c would fix your immediate
> problem by allowing holes in wired region.
>
> The change for vm_map.c prefaults the whole stack instead of the
> initial fragment. The single-threaded programs still get a fault
> on stack growth.
The vm_mmap.c change looks ok to me. Please commit it. I haven't yet
had a chance to think about the other change.
Alan
> diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
> index 6198629..2fd18d1 100644
> --- a/sys/vm/vm_map.c
> +++ b/sys/vm/vm_map.c
> @@ -3259,7 +3259,10 @@ vm_map_stack(vm_map_t map, vm_offset_t addrbos, vm_size_t max_ssize,
> addrbos + max_ssize< addrbos)
> return (KERN_NO_SPACE);
>
> - init_ssize = (max_ssize< sgrowsiz) ? max_ssize : sgrowsiz;
> + if (map->flags& MAP_WIREFUTURE)
> + init_ssize = max_ssize;
> + else
> + init_ssize = (max_ssize< sgrowsiz) ? max_ssize : sgrowsiz;
>
> PROC_LOCK(curthread->td_proc);
> vmemlim = lim_cur(curthread->td_proc, RLIMIT_VMEM);
> diff --git a/sys/vm/vm_mmap.c b/sys/vm/vm_mmap.c
> index 2588c85..3fccd9e 100644
> --- a/sys/vm/vm_mmap.c
> +++ b/sys/vm/vm_mmap.c
> @@ -1561,9 +1561,11 @@ vm_mmap(vm_map_t map, vm_offset_t *addr, vm_size_t size, vm_prot_t prot,
> * If the process has requested that all future mappings
> * be wired, then heed this.
> */
> - if (map->flags& MAP_WIREFUTURE)
> + if (map->flags& MAP_WIREFUTURE) {
> vm_map_wire(map, *addr, *addr + size,
> - VM_MAP_WIRE_USER | VM_MAP_WIRE_NOHOLES);
> + VM_MAP_WIRE_USER | ((flags& MAP_STACK) ?
> + VM_MAP_WIRE_HOLESOK : VM_MAP_WIRE_NOHOLES));
> + }
> } else {
> /*
> * If this mapping was accounted for in the vnode's
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F92FAE7.3090702>