From owner-freebsd-fs@FreeBSD.ORG Thu Oct 11 01:03:23 2007 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACA5A16A419; Thu, 11 Oct 2007 01:03:23 +0000 (UTC) (envelope-from avatar@mmlab.cse.yzu.edu.tw) Received: from www.mmlab.cse.yzu.edu.tw (www.mmlab.cse.yzu.edu.tw [140.138.150.166]) by mx1.freebsd.org (Postfix) with ESMTP id 6FFBC13C43E; Thu, 11 Oct 2007 01:03:23 +0000 (UTC) (envelope-from avatar@mmlab.cse.yzu.edu.tw) Received: by www.mmlab.cse.yzu.edu.tw (qmail, from userid 1000) id 313078C9B01; Thu, 11 Oct 2007 08:43:42 +0800 (CST) Received: from localhost (localhost [127.0.0.1]) by www.mmlab.cse.yzu.edu.tw (qmail) with ESMTP id D811B8C9B00; Thu, 11 Oct 2007 08:43:42 +0800 (CST) Date: Thu, 11 Oct 2007 08:43:42 +0800 (CST) From: Tai-hwa Liang To: Pawel Jakub Dawidek In-Reply-To: <20071008121523.GM2327@garage.freebsd.pl> Message-ID: <0710110840301.59863@www.mmlab.cse.yzu.edu.tw> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Oct 2007 01:03:23 -0000 On Mon, 8 Oct 2007, Pawel Jakub Dawidek wrote: > Here are some updates: > > I was able to reproduce the panic by rsyncing big files and trying > bonnie++ test suggested in this thread. > > Can you guys retry with this patch: > > http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch > > It's a hack, yes, but allows to mitigate the problem quite well. I'm > looking for a solution that can be used for 7.0 before we find a better > fix. > > BTW. To use ZFS you _must_ increase vm.kmem_size/vm.kmem_size_max. > If you have the problem discussed here and you're using standard values, > please retry with vm.kmem_size/vm.kmem_size_max set to at least 600MB in > /boot/loader.conf. > > I'm not sure if it's not too late to ask re@ about increasing the > default kmem size at least on amd64. ~300MB we have there is silly > small. The latest patch does keep the system surviving longer than before; however, it eventually panicked with different message. - Testing case: while true; do bonnie++ -s 2048 -c 50 -x 20 done - ZFS related settings: vfs.root.mountfrom="zfs:universe" vm.kmem_size_max="629145600" vm.kmem_size="629145600" - zpool status: pool: universe state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM universe ONLINE 0 0 0 ad0s3d ONLINE 0 0 0 errors: No known data errors - panic messages: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x3916545d fault code = supervisor write, page not present instruction pointer = 0x20:0xc04d33c0 stack pointer = 0x28:0xf7b0e930 frame pointer = 0x28:0xf7b0e944 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3477 (bonnie++) [thread pid 3477 tid 100159 ] Stopped at malloc_type_zone_allocated+0x70: addb %cl,0x758bf45d(%ebx) db> wh Tracing pid 3477 tid 100159 td 0xc4258a50 malloc_type_zone_allocated(c1076d20,0,2,1,0,...) at malloc_type_zone_allocated+0x70 malloc(40,c0824060,2,f7b0e9fc,c080df70,...) at malloc+0x69 zfs_kmem_alloc(40,2,c0607d3e,c3d66228,f7b0e998,...) at zfs_kmem_alloc+0x20 zfs_range_lock(c90da0ec,22f2da,0,1,0,...) at zfs_range_lock+0x20 zfs_freebsd_write(f7b0ebc4,0,0,0,c06be120,...) at zfs_freebsd_write+0x24b VOP_WRITE_APV(c08257c0,f7b0ebc4,c4258a50,c068d7f8,242,...) at VOP_WRITE_APV+0xb6 vn_write(c53bd0d8,f7b0ec60,c7835e00,0,c4258a50,...) at vn_write+0x247 dofilewrite(f7b0ec60,ffffffff,ffffffff,0,c53bd0d8,...) at dofilewrite+0x97 kern_writev(c4258a50,3,f7b0ec60,bfbfdf8b,1,...) at kern_writev+0x58 write(c4258a50,f7b0ecfc,c,f7b0eca4,c065d061,...) at write+0x4f syscall(f7b0ed38) at syscall+0x319 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x282779a3, esp = 0xbfbfdf3c, ebp = 0xbfbfdf58 --- db> -- Cheers, Tai-hwa Liang