Date: Mon, 11 Apr 2022 17:37:27 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= <roger.pau@citrix.com> To: Ze Dupsys <zedupsys@gmail.com> Cc: <freebsd-xen@freebsd.org>, <buhrow@nfbcal.org> Subject: Re: ZFS + FreeBSD XEN dom0 panic Message-ID: <YlRLN7aAxYzgb7kr@Air-de-Roger> In-Reply-To: <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com> References: <Yj7YrW9CG2aXT%2BiC@Air-de-Roger> <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com> <Yj8lZWqeHbD%2BkfOQ@Air-de-Roger> <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com> <YkAqxjiMM1M1QdgR@Air-de-Roger> <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com> <Ykxev3fangqRGQcn@Air-de-Roger> <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com> <YlBOgFYNokZ0rTgD@Air-de-Roger> <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Apr 11, 2022 at 11:47:50AM +0300, Ze Dupsys wrote: > On 2022.04.08. 18:02, Roger Pau Monné wrote: > > On Fri, Apr 08, 2022 at 10:45:12AM +0300, Ze Dupsys wrote: > > > On 2022.04.05. 18:22, Roger Pau Monné wrote: > > > > .. Thanks, sorry for the late reply, somehow the message slip. > > > > > > > > I've been able to get the file:line for those, and the trace is kind > > > > of weird, I'm not sure I know what's going on TBH. It seems to me the > > > > backend instance got freed while being in the process of connecting. > > > > > > > > I've made some changes, that might mitigate this, but having not a > > > > clear understanding of what's going on makes this harder. > > > > > > > > I've pushed the changes to: > > > > > > > > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/for-leak > > > > > > > > (This is on top of main branch). > > > > > > > > I'm also attaching the two patches on this email. > > > > > > > > Let me know if those make a difference to stabilize the system. > > > > > > Hi, > > > > > > Yes, it stabilizes the system, but there is still a memleak somewhere, i > > > think. > > > > > > System could run tests for approximately 41 hour, did not panic, but started > > > to OOM kill everything. > > > > > > I did not know how to git clone given commit, thus i just applied patches to > > > 13.0-RELEASE sources. > > > > > > Serial logs have nothing unusual, just that at some point OOM kill starts. > > > > Well, I think that's good^W better than before. Thanks again for all > > the testing. > > > > It might be helpful now to start dumping `vmstat -m` periodically > > while running the stress tests. As there are (hopefully) no more > > panics now vmstat might report us what subsystem is hogging the > > memory. It's possible it's blkback (again). > > > > Thanks, Roger. > > > > Yes, it certainly is better. Applied patch on my pre-production server, have > not had any panic since then, still testing though. > > On my stressed lab server, it's a bit different story. On occasion i see a > panic with this trace on serial (can not reliably repeat, but sometimes upon > starting dom id 1 and 2, sometimes mid-stress-test, dom id > 95). > panic: pmap_growkernel: no memory to grow kernel > cpuid = 2 > time = 1649485133 > KDB: stack backtrace: > #0 0xffffffff80c57385 at kdb_backtrace+0x65 > #1 0xffffffff80c09d61 at vpanic+0x181 > #2 0xffffffff80c09bd3 at panic+0x43 > #3 0xffffffff81073eed at pmap_growkernel+0x27d > #4 0xffffffff80f2d918 at vm_map_insert+0x248 > #5 0xffffffff80f30079 at vm_map_find+0x549 > #6 0xffffffff80f2bda6 at kmem_init+0x226 > #7 0xffffffff80c731a1 at vmem_xalloc+0xcb1 > #8 0xffffffff80c72a9b at vmem_xalloc+0x5ab > #9 0xffffffff80c724a6 at vmem_alloc+0x46 > #10 0xffffffff80f2ac6b at kva_alloc+0x2b > #11 0xffffffff8107f0eb at pmap_mapdev_attr+0x27b > #12 0xffffffff810588ca at nexus_add_irq+0x65a > #13 0xffffffff81058710 at nexus_add_irq+0x4a0 > #14 0xffffffff810585b9 at nexus_add_irq+0x349 > #15 0xffffffff80c495c1 at bus_alloc_resource+0xa1 > #16 0xffffffff8105e940 at xenmem_free+0x1a0 > #17 0xffffffff80a7e0dd at xbd_instance_create+0x943d > > | sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e > /usr/lib/debug/boot/kernel/kernel.debug > /usr/src/sys/kern/subr_kdb.c:443 > /usr/src/sys/kern/kern_shutdown.c:0 > /usr/src/sys/kern/kern_shutdown.c:843 > /usr/src/sys/amd64/amd64/pmap.c:0 > /usr/src/sys/vm/vm_map.c:0 > /usr/src/sys/vm/vm_map.c:0 > /usr/src/sys/vm/vm_kern.c:712 > /usr/src/sys/kern/subr_vmem.c:928 > /usr/src/sys/kern/subr_vmem.c:0 > /usr/src/sys/kern/subr_vmem.c:1350 > /usr/src/sys/vm/vm_kern.c:150 > /usr/src/sys/amd64/amd64/pmap.c:0 > /usr/src/sys/x86/x86/nexus.c:0 > /usr/src/sys/x86/x86/nexus.c:449 > /usr/src/sys/x86/x86/nexus.c:412 > /usr/src/sys/kern/subr_bus.c:4620 > /usr/src/sys/x86/xen/xenpv.c:123 > /usr/src/sys/dev/xen/blkback/blkback.c:3010 > > With gdb backtrace i think i can get a better trace though: > #0 __curthread at /usr/src/sys/amd64/include/pcpu_aux.h:55 > #1 doadump at /usr/src/sys/kern/kern_shutdown.c:399 > #2 kern_reboot at /usr/src/sys/kern/kern_shutdown.c:486 > #3 vpanic at /usr/src/sys/kern/kern_shutdown.c:919 > #4 panic at /usr/src/sys/kern/kern_shutdown.c:843 > #5 pmap_growkernel at /usr/src/sys/amd64/amd64/pmap.c:208 > #6 vm_map_insert at /usr/src/sys/vm/vm_map.c:1752 > #7 vm_map_find at /usr/src/sys/vm/vm_map.c:2259 > #8 kva_import at /usr/src/sys/vm/vm_kern.c:712 > #9 vmem_import at /usr/src/sys/kern/subr_vmem.c:928 > #10 vmem_try_fetch at /usr/src/sys/kern/subr_vmem.c:1049 > #11 vmem_xalloc at /usr/src/sys/kern/subr_vmem.c:1449 > #12 vmem_alloc at /usr/src/sys/kern/subr_vmem.c:1350 > #13 kva_alloc at /usr/src/sys/vm/vm_kern.c:150 > #14 pmap_mapdev_internal at /usr/src/sys/amd64/amd64/pmap.c:8974 > #15 pmap_mapdev_attr at /usr/src/sys/amd64/amd64/pmap.c:8990 > #16 nexus_map_resource at /usr/src/sys/x86/x86/nexus.c:523 > #17 nexus_activate_resource at /usr/src/sys/x86/x86/nexus.c:448 > #18 nexus_alloc_resource at /usr/src/sys/x86/x86/nexus.c:412 > #19 BUS_ALLOC_RESOURCE at ./bus_if.h:321 > #20 bus_alloc_resource at /usr/src/sys/kern/subr_bus.c:4617 > #21 xenpv_alloc_physmem at /usr/src/sys/x86/xen/xenpv.c:121 > #22 xbb_alloc_communication_mem at > /usr/src/sys/dev/xen/blkback/blkback.c:3010 > #23 xbb_connect at /usr/src/sys/dev/xen/blkback/blkback.c:3336 > #24 xenbusb_back_otherend_changed at > /usr/src/sys/xen/xenbus/xenbusb_back.c:228 > #25 xenwatch_thread at /usr/src/sys/dev/xen/xenstore/xenstore.c:1003 > #26 in fork_exit at /usr/src/sys/kern/kern_fork.c:1069 > #27 <signal handler called> > > > There is some sort of mismatch in info, because panic message printed > "panic: pmap_growkernel: no memory to grow kernel", but gdb backtrace in > #5 0xffffffff81073eed in pmap_growkernel at > /usr/src/sys/amd64/amd64/pmap.c:208 > leads to lines: > switch (pmap->pm_type) { > .. > panic("pmap_valid_bit: invalid pm_type %d", pmap->pm_type) > > So either trace is off the mark or message in serial logs. If this was only > memleak related, then it should not happen when dom id 1 is started, i > suppose. That's weird, I would rather trust the printed panic message rather than the symbol resolution. Seems to be a kind of memory exhaustion, as the kernel is failing to allocate a page for use in the kernel page table. I will try to see what can be done here. Thanks, Roger.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YlRLN7aAxYzgb7kr>