Date: Fri, 21 Sep 2018 11:17:43 -0400 From: Josh Gitlin <jgitlin@goboomtown.com> To: freebsd-fs@freebsd.org Subject: Re: Troubleshooting kernel panic with zfs Message-ID: <373102CE-B53A-4547-8665-DF5EC74C23DD@goboomtown.com> In-Reply-To: <c4d7e51f-09c9-c50c-075d-8eff65e64abe@sentex.net> References: <D54225AD-CC96-45E7-A203-D2C52E984963@goboomtown.com> <c4d7e51f-09c9-c50c-075d-8eff65e64abe@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks for the help Mike! In reviewing our settings I also found that = we're setting vm.kmem_size to 1.5x physical RAM installed, based on a = tuning guide someone found. I personally believe that might be misguided = and may be related to the panic we saw, as this causes zfs-stats -a to = show the kernel memory map being larger than system RAM: Kernel Memory Map: 24.00 GiB Size: 21.72% 5.21 GiB Free: 78.28% 18.79 GiB I'll try tuning these settings down and see if the problem happens = again. Appreciate the support! -- <http://www.goboomtown.com/>=09 Josh Gitlin Senior DevOps Engineer (415) 690-1610 x155 Stay up to date and join the conversation in Relay = <http://relay.goboomtown.com/>. > On Sep 21, 2018, at 9:04 AM, Mike Tancsa <mike@sentex.net> wrote: >=20 > On 9/20/2018 8:27 PM, Josh Gitlin wrote: >>=20 >> My hunch is that, given this was inside kmem_malloc, we were unable = to allocate memory for a zio_write_compress call (the pool does have ZFS = compression on) and hence this is a tuning issue and not a bug... but I = am looking for confirmation and/or suggested changes/troubleshooting = steps. The ZFS tuning configuration has been stable for years, to it may = be a change in behavior or traffic... If this looks like it might be a = bug, I will be able to get more information from a minidump if it = reoccurs and can follow up on this thread. >=20 > Given all the issues with memory pressure perhaps ARC is eating too > much, I would try artificially limiting it so it does not eat up too > much RAM. We had a similar purposed server but with 32G RAM and I = found > one day the system was randomly shooting processes to reclaim RAM >=20 > eg > pid xxx (yyy), uid zzzz, was killed: out of swap space >=20 >=20 > Perhaps >=20 > sysctl -w vfs.zfs.arc_max=3D6000000000 >=20 > eg. > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229764 >=20 > = https://lists.freebsd.org/pipermail/freebsd-questions/2018-July/282261.htm= l >=20 > ---Mike >=20 >=20 >=20 >=20 > --=20 > ------------------- > Mike Tancsa, tel +1 519 651 3400 x203 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?373102CE-B53A-4547-8665-DF5EC74C23DD>