From owner-freebsd-hackers@freebsd.org Sat Jan 27 16:55:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1959EECB5E3 for ; Sat, 27 Jan 2018 16:55:32 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A847B703F9 for ; Sat, 27 Jan 2018 16:55:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.ysv.freebsd.org (Postfix) id 64EF0ECB5E2; Sat, 27 Jan 2018 16:55:31 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27DDCECB5E1 for ; Sat, 27 Jan 2018 16:55:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ADE27703F8 for ; Sat, 27 Jan 2018 16:55:30 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 59704260115; Sat, 27 Jan 2018 17:55:20 +0100 (CET) Subject: Re: allocating large piece of contiguous kernel memory To: Konstantin Belousov , Hartmut Brandt Cc: hackers@freebsd.org References: <20180125172116.GS55707@kib.kiev.ua> <20180127135013.GA55707@kib.kiev.ua> From: Hans Petter Selasky Message-ID: Date: Sat, 27 Jan 2018 17:52:28 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <20180127135013.GA55707@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 16:55:32 -0000 On 01/27/18 14:50, Konstantin Belousov wrote: > On Fri, Jan 26, 2018 at 10:05:14AM +0100, Hartmut Brandt wrote: >> Hi, >> >> On Thu, 25 Jan 2018, Konstantin Belousov wrote: >> >> KB>On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: >> KB>> Hi all, >> KB>> >> KB>> for a device driver communicating with an FPGA I would like to allocate >> KB>> an 8GByte piece or two 4GByte pieces of contiguous memory in the device >> KB>> driver. The machine is amd64 and has 256GByte. >> KB>> >> KB>> The maximum I can allocate is 512MByte. If I try more (in one piece or in >> KB>> two pieces) contigfree() crashes in pmap_resident_count_dec() with an >> KB>> "pmap resident count underflow". >> KB>> >> KB>> Is this something which is not supposed to work? Or is there something >> KB>> tunable or is this a bug? >> KB> >> KB>I suspect a bug. Print out the value of >> KB>kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also >> KB>interesting are the values of the resident count and decrement at the >> KB>panic time (but they are already printed by the panic, you did not shown >> KB>them). >> >> This is the panic: >> >> knot_attach: attaching >> kernel_pmap->pm_stats.resident_count=57322 >> cmem=0xfffffe0101000000 >> kernel_pmap->pm_stats.resident_count=57712 >> kernel_pmap->pm_stats.resident_count=57712 >> panic: pmap 0xffffffff80e40188 resident count underflow 368 512 >> cpuid = 3 >> time = 1516956458 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0100d0d1c0 >> vpanic() at vpanic+0x19c/frame 0xfffffe0100d0d240 >> kassert_panic() at kassert_panic+0x126/frame 0xfffffe0100d0d2b0 >> pmap_remove_pde() at pmap_remove_pde+0x645/frame 0xfffffe0100d0d340 >> pmap_remove() at pmap_remove+0x484/frame 0xfffffe0100d0d3c0 >> kmem_unback() at kmem_unback+0x3a/frame 0xfffffe0100d0d400 >> kmem_free() at kmem_free+0x3d/frame 0xfffffe0100d0d430 >> contigfree() at contigfree+0x27/frame 0xfffffe0100d0d460 >> knot_attach() at knot_attach+0xec/frame 0xfffffe0100d0d4c0 >> device_attach() at device_attach+0x3f7/frame 0xfffffe0100d0d510 >> pci_driver_added() at pci_driver_added+0xe9/frame 0xfffffe0100d0d550 >> devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe0100d0d590 >> devclass_add_driver() at devclass_add_driver+0x144/frame 0xfffffe0100d0d5d0 >> module_register_init() at module_register_init+0xc0/frame 0xfffffe0100d0d600 >> linker_load_module() at linker_load_module+0xb78/frame 0xfffffe0100d0d910 >> kern_kldload() at kern_kldload+0xa9/frame 0xfffffe0100d0d950 >> sys_kldload() at sys_kldload+0x5b/frame 0xfffffe0100d0d980 >> amd64_syscall() at amd64_syscall+0x271/frame 0xfffffe0100d0dab0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0100d0dab0 >> --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x80087228a, rsp = >> 0x7fffffffd508, rbp = 0x7fffffffda80 --- >> KDB: enter: panic >> [ thread pid 1150 tid 100146 ] >> Stopped at kdb_enter+0x3b: movq $0,kdb_why >> db> >> >> The first printf() is just before contigmalloc(), then it prints the >> returned pointer. The next to prints() are just before the contigfree(). >> >> Doesn't the change of the count by 390 look strange when allocating >> 8GByte? > I cannot reproduce your issue. I suspect this is something local to your > system. With the patch below, if I do > # sysctl debug.a=1 > I see > # sysctl debug.a=1 > debug.a: before 31310 > 0after alloc 2128462 > after free 31310 > -> 0 > > The numbers are exactly as what I expect. > > diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c > index 12c5b695e23..3c707444169 100644 > --- a/sys/amd64/amd64/pmap.c > +++ b/sys/amd64/amd64/pmap.c > @@ -8040,3 +8040,26 @@ DB_SHOW_COMMAND(phys2dmap, pmap_phys2dmap) > } > } > #endif > + > +static int > +pmap_contigalloc(SYSCTL_HANDLER_ARGS) > +{ > + void *ptr; > + size_t sz; > + int error, i; > + > + i = 0; > + error = sysctl_handle_int(oidp, &i, 0, req); > + if (error != 0 || req->newptr == NULL) > + return (error); > + sz = 8UL * 1024 * 1024 * 1024; > + printf("before %ld\n", kernel_pmap->pm_stats.resident_count); > + ptr = contigmalloc(sz, M_TEMP, M_WAITOK, 0, ~0UL, 0, 0); > + printf("after alloc %ld\n", kernel_pmap->pm_stats.resident_count); > + contigfree(ptr, sz, M_TEMP); > + printf("after free %ld\n", kernel_pmap->pm_stats.resident_count); > + return (0); > +} > +SYSCTL_PROC(_debug, OID_AUTO, a, CTLTYPE_INT | CTLFLAG_RW | > + CTLFLAG_MPSAFE, NULL, 0, pmap_contigalloc, "I", > + ""); Hi, What is the earliest point in the SYSINIT sequence which contigmalloc() can be called? Is contigmalloc() being called too early for this big allocations? --HPS