Date: Fri, 21 Sep 2018 11:17:43 -0400 From: Josh Gitlin <jgitlin@goboomtown.com> To: freebsd-fs@freebsd.org Subject: Re: Troubleshooting kernel panic with zfs Message-ID: <373102CE-B53A-4547-8665-DF5EC74C23DD@goboomtown.com> In-Reply-To: <c4d7e51f-09c9-c50c-075d-8eff65e64abe@sentex.net> References: <D54225AD-CC96-45E7-A203-D2C52E984963@goboomtown.com> <c4d7e51f-09c9-c50c-075d-8eff65e64abe@sentex.net>
index | next in thread | previous in thread | raw e-mail
Thanks for the help Mike! In reviewing our settings I also found that we're setting vm.kmem_size to 1.5x physical RAM installed, based on a tuning guide someone found. I personally believe that might be misguided and may be related to the panic we saw, as this causes zfs-stats -a to show the kernel memory map being larger than system RAM: Kernel Memory Map: 24.00 GiB Size: 21.72% 5.21 GiB Free: 78.28% 18.79 GiB I'll try tuning these settings down and see if the problem happens again. Appreciate the support! -- <http://www.goboomtown.com/>; Josh Gitlin Senior DevOps Engineer (415) 690-1610 x155 Stay up to date and join the conversation in Relay <http://relay.goboomtown.com/>. > On Sep 21, 2018, at 9:04 AM, Mike Tancsa <mike@sentex.net> wrote: > > On 9/20/2018 8:27 PM, Josh Gitlin wrote: >> >> My hunch is that, given this was inside kmem_malloc, we were unable to allocate memory for a zio_write_compress call (the pool does have ZFS compression on) and hence this is a tuning issue and not a bug... but I am looking for confirmation and/or suggested changes/troubleshooting steps. The ZFS tuning configuration has been stable for years, to it may be a change in behavior or traffic... If this looks like it might be a bug, I will be able to get more information from a minidump if it reoccurs and can follow up on this thread. > > Given all the issues with memory pressure perhaps ARC is eating too > much, I would try artificially limiting it so it does not eat up too > much RAM. We had a similar purposed server but with 32G RAM and I found > one day the system was randomly shooting processes to reclaim RAM > > eg > pid xxx (yyy), uid zzzz, was killed: out of swap space > > > Perhaps > > sysctl -w vfs.zfs.arc_max=6000000000 > > eg. > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229764 > > https://lists.freebsd.org/pipermail/freebsd-questions/2018-July/282261.html > > ---Mike > > > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 x203 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canadahome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?373102CE-B53A-4547-8665-DF5EC74C23DD>