Date: Mon, 5 Oct 2009 10:29:04 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: freebsd-stable@freebsd.org Subject: Still possible to panic RELENG_7 with ZFS (kmem exhaustion)? Message-ID: <20091005172904.GA56362@icarus.home.lan>
next in thread | raw e-mail | index | archive | help
(Note: please keep me CC'd, as I am not subscribed to freebsd-stable) Is it still possible with ZFS to panic a RELENG_7 amd64 box (kernel/world from recent[1] source) with "kmem map too small" or similar conditions? Why I ask: Our production SQL/backup box kernel panic'd a couple days ago. Sadly, the box also acts as a serial console server, so I don't have the exact message spit back from the kernel prior to being dumped to DDB, but the backtrace looks very much like the historic problem of the ZFS ARC exhausting all kernel memory, so I'm betting the message prior to being dumped to DDB was "kmem map too small": db> bt Tracing pid 40738 tid 100168 td 0xffffff001f078720 kdb_enter_why() at kdb_enter_why+0x3d panic() at panic+0x176 kmem_malloc() at kmem_malloc+0x548 uma_large_malloc() at uma_large_malloc+0x3c malloc() at malloc+0xc1 arc_get_data_buf() at arc_get_data_buf+0x1bb arc_buf_alloc() at arc_buf_alloc+0xa1 arc_read_nolock() at arc_read_nolock+0xd1 arc_read() at arc_read+0x71 dbuf_prefetch() at dbuf_prefetch+0x135 dmu_zfetch_dofetch() at dmu_zfetch_dofetch+0xe3 dmu_zfetch() at dmu_zfetch+0xa58 dbuf_read() at dbuf_read+0x433 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x119 dmu_buf_hold_array() at dmu_buf_hold_array+0x57 dmu_read_uio() at dmu_read_uio+0x3f zfs_freebsd_read() at zfs_freebsd_read+0x55a vn_read() at vn_read+0x1ef dofileread() at dofileread+0x88 kern_readv() at kern_readv+0x43 read() at read+0x4d syscall() at syscall+0x247 Xfast_syscall() at Xfast_syscall+0xab The machine in question has absolutely no loader.conf tuning applied, and kernel/world was built from RELENG_7 dated 2009/06/11. The ZFS pool consisted of a single (entire) disk; nothing special. I do not have sysctl counters from the box before it panic'd, but these are what are active presently: hw.physmem: 4286558208 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 1381478400 With regards to the above counters: ZFS is not in use. I had to switch back to UFS2 (zpool destroy + newfs -O2 -U...) because of stability concerns relating to the question at hand. If someone familiar with the FreeBSD ZFS internals, and/or the VM, please make a statement I think it would beneficial to those who are considering using/migrating to ZFS on FreeBSD. The only "semi-official" statements I've read as of late are here: http://lists.freebsd.org/pipermail/freebsd-stable/2009-September/051810.html http://lists.freebsd.org/pipermail/freebsd-stable/2009-September/051830.html And what's in src/UPDATING: 20090207: ZFS users on amd64 machines with 4GB or more of RAM should reevaluate their need for setting vm.kmem_size_max and vm.kmem_size manually. In fact, after recent changes to the kernel, the default value of vm.kmem_size is larger than the suggested manual setting in most ZFS/FreeBSD tuning guides. Thanks! [1]: "Recent" means post-February 2009, specifically after Alan Cox's commits listed here: http://svn.freebsd.org/changeset/base/188291 http://svn.freebsd.org/changeset/base/187523 http://svn.freebsd.org/changeset/base/187522 http://svn.freebsd.org/changeset/base/187520 http://svn.freebsd.org/changeset/base/187485 http://svn.freebsd.org/changeset/base/187466 http://svn.freebsd.org/changeset/base/187465 http://svn.freebsd.org/changeset/base/187464 http://svn.freebsd.org/changeset/base/187458 http://svn.freebsd.org/changeset/base/187428 http://svn.freebsd.org/changeset/base/187425 http://svn.freebsd.org/changeset/base/187420 http://svn.freebsd.org/changeset/base/187419 http://svn.freebsd.org/changeset/base/187416 http://svn.freebsd.org/changeset/base/187414 http://svn.freebsd.org/changeset/base/187408 http://svn.freebsd.org/changeset/base/187407 http://svn.freebsd.org/changeset/base/187404 http://svn.freebsd.org/changeset/base/187400 -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20091005172904.GA56362>