Date: Tue, 29 Oct 2013 02:26:51 +0400 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: d@delphij.net Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org, Jordan Hubbard <jkh@ixsystems.com> Subject: Re: ZFS txg implementation flaw Message-ID: <20131028222651.GZ63359@zxy.spb.ru> In-Reply-To: <526EDD81.8000109@delphij.net> References: <20131028092844.GA24997@zxy.spb.ru> <9A00B135-7D28-47EB-ADB3-E87C38BAC6B6@ixsystems.com> <20131028213204.GX63359@zxy.spb.ru> <526ED956.10202@delphij.net> <20131028214552.GY63359@zxy.spb.ru> <526EDD81.8000109@delphij.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 28, 2013 at 02:56:17PM -0700, Xin Li wrote:
> >>> Semi-indirect. dtrace -n 'fbt:kernel:vm_object_terminate:entry
> >>> { @traces[stack()] = count(); }'
> >>>
> >>> After some (2-3) seconds
> >>>
> >>> kernel`vnode_destroy_vobject+0xb9
> >>> zfs.ko`zfs_freebsd_reclaim+0x2e kernel`VOP_RECLAIM_APV+0x78
> >>> kernel`vgonel+0x134 kernel`vnlru_free+0x362
> >>> kernel`vnlru_proc+0x61e kernel`fork_exit+0x11f
> >>> kernel`0xffffffff80cdbfde 2490
> >
> > 0xffffffff80cdbfd0 <fork_trampoline>: mov %r12,%rdi
> > 0xffffffff80cdbfd3 <fork_trampoline+3>: mov %rbx,%rsi
> > 0xffffffff80cdbfd6 <fork_trampoline+6>: mov %rsp,%rdx
> > 0xffffffff80cdbfd9 <fork_trampoline+9>: callq 0xffffffff808db560
> > <fork_exit> 0xffffffff80cdbfde <fork_trampoline+14>: jmpq
> > 0xffffffff80cdca80 <doreti> 0xffffffff80cdbfe3
> > <fork_trampoline+19>: nopw 0x0(%rax,%rax,1)
> > 0xffffffff80cdbfe9 <fork_trampoline+25>: nopl 0x0(%rax)
> >
> >
> >>> I don't have user process created threads nor do fork/exit.
> >>
> >> This has nothing to do with fork/exit but does suggest that you
> >> are running of vnodes. What does sysctl -a | grep vnode say?
> >
> > kern.maxvnodes: 1095872 kern.minvnodes: 273968
> > vm.stats.vm.v_vnodepgsout: 0 vm.stats.vm.v_vnodepgsin: 62399
> > vm.stats.vm.v_vnodeout: 0 vm.stats.vm.v_vnodein: 10680
> > vfs.freevnodes: 275107 vfs.wantfreevnodes: 273968 vfs.numvnodes:
> > 316321 debug.sizeof.vnode: 504
>
> Try setting vfs.wantfreevnodes to 547936 (double it).
Now fork_trampoline was gone, but I still see prcfr (and zfod/totfr
too). Currently half of peeak traffic and I can't check impact to IRQ
handling.
kern.maxvnodes: 1095872
kern.minvnodes: 547936
vm.stats.vm.v_vnodepgsout: 0
vm.stats.vm.v_vnodepgsin: 63134
vm.stats.vm.v_vnodeout: 0
vm.stats.vm.v_vnodein: 10836
vfs.freevnodes: 481873
vfs.wantfreevnodes: 547936
vfs.numvnodes: 517331
debug.sizeof.vnode: 504
Now dtrace -n 'fbt:kernel:vm_object_terminate:entry { @traces[stack()] = count(); }'
kernel`vm_object_deallocate+0x520
kernel`vm_map_entry_deallocate+0x4c
kernel`vm_map_process_deferred+0x3d
kernel`sys_munmap+0x16c
kernel`amd64_syscall+0x5ea
kernel`0xffffffff80cdbd97
56
I think this is nginx memory management (allocation|dealocation). Can
I tune malloc to disable free pages?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131028222651.GZ63359>