From owner-freebsd-bugs@FreeBSD.ORG Fri Dec 7 21:30:04 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8308B16A421 for ; Fri, 7 Dec 2007 21:30:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6168013C45D for ; Fri, 7 Dec 2007 21:30:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lB7LU4OW031433 for ; Fri, 7 Dec 2007 21:30:04 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lB7LU4Xe031430; Fri, 7 Dec 2007 21:30:04 GMT (envelope-from gnats) Resent-Date: Fri, 7 Dec 2007 21:30:04 GMT Resent-Message-Id: <200712072130.lB7LU4Xe031430@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Michael Haro Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 351DF16A41B; Fri, 7 Dec 2007 21:25:17 +0000 (UTC) (envelope-from mharo@bitsurf.net) Received: from bitsurf.net (cowabunga.bitsurf.net [66.92.2.81]) by mx1.freebsd.org (Postfix) with ESMTP id C5C0613C448; Fri, 7 Dec 2007 21:25:16 +0000 (UTC) (envelope-from mharo@bitsurf.net) Received: from zfsserver.mtv.bitsurf.net (localhost [127.0.0.1]) by zfsserver.mtv.bitsurf.net (8.14.2/8.14.2) with ESMTP id lB7KxGGG001299; Fri, 7 Dec 2007 12:59:16 -0800 (PST) (envelope-from mharo@zfsserver.mtv.bitsurf.net) Received: (from root@localhost) by zfsserver.mtv.bitsurf.net (8.14.2/8.14.2/Submit) id lB7KxGoP001298; Fri, 7 Dec 2007 12:59:16 -0800 (PST) (envelope-from mharo) Message-Id: <200712072059.lB7KxGoP001298@zfsserver.mtv.bitsurf.net> Date: Fri, 7 Dec 2007 12:59:16 -0800 (PST) From: Michael Haro To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: mharo@FreeBSD.org Subject: kern/118487: zfs related kernel panic on RELENG_7 (7.0BETA4): kmem_malloc(XXX): kmem_map too small X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Michael Haro List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Dec 2007 21:30:04 -0000 >Number: 118487 >Category: kern >Synopsis: zfs related kernel panic on RELENG_7 (7.0BETA4): kmem_malloc(XXX): kmem_map too small >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Dec 07 21:30:04 UTC 2007 >Closed-Date: >Last-Modified: >Originator: Michael Haro >Release: FreeBSD 7.0-BETA4 i386 >Organization: >Environment: System: FreeBSD zfsserver.mtv.bitsurf.net 7.0-BETA4 FreeBSD 7.0-BETA4 #2: Thu Dec 6 23:02:30 PST 2007 root@zfsserver.mtv.bitsurf.net:/usr/obj/usr/src/sys/KERNEL i386 I have 3 kernel dumps. I suspect they're related... Unread portion of the kernel message buffer: panic: kmem_malloc(131072): kmem_map too small: 418766848 total allocated cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c09337fe,f3155af0,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0951d54,0,c094a215,f3155afc,0,...) at kdb_backtrace+0x29 panic(c094a215,20000,18f5e000,c094a20f,7d0,...) at panic+0x10f kmem_malloc(c147108c,20000,2,f3155b7c,c08541b0,...) at kmem_malloc+0x284 page_alloc(0,20000,f3155b6f,2,24539a8,...) at page_alloc+0x27 uma_large_malloc(20000,2,c683ccf0,c6488ac8,28000,...) at uma_large_malloc+0x50 malloc(20000,c0b9f1c0,2,f3155bc0,c0b7a719,...) at malloc+0x8d zfs_kmem_alloc(20000,2,f3155c48,c0b699bc,20000,...) at zfs_kmem_alloc+0x20 zio_buf_alloc(20000,c6488ac8,1,3b7,33860000,...) at zio_buf_alloc+0x19 vdev_queue_io_to_issue(f3155c64,c6bcf228,c0b9c2d0,133,c43ab2e4,...) at vdev_queue_io_to_issue+0x1bc vdev_queue_io_done(c6bcf228,f3155ce0,f3155ce0,f3155cf4,c0b23ba0,...) at vdev_queue_io_done+0xa8 vdev_geom_io_done(c6bcf228,0,c0b976f3,33d,2f652a7e,...) at vdev_geom_io_done+0x12 taskq_thread(c3f9a7f8,f3155d38,4,0,85,...) at taskq_thread+0x140 fork_exit(c0b23a60,c3f9a7f8,f3155d38) at fork_exit+0x97 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf3155d70, ebp = 0 --- Uptime: 9m48s Physical memory: 1011 MB Dumping 473 MB: 458 442 426 410 394 378 362 346 330 314 298 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10 Unread portion of the kernel message buffer: ad16: FAILURE - device detached subdisk16: detached ad16: detached Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x2c fault code = supervisor write, page not present instruction pointer = 0x20:0xc0631085 stack pointer = 0x28:0xef33bc5c frame pointer = 0x28:0xef33bc70 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3 (g_up) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c09337fe,ef33baf8,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0951d54,0,c0913383,ef33bb04,0,...) at kdb_backtrace+0x29 panic(c0913383,c0952ffb,c3f544d0,1,1,...) at panic+0x10f trap_fatal(c0a12600,0,2,8,76c5ec12,...) at trap_fatal+0x333 trap_pfault(c0a120a8,ef33bb90,c066ac9d,ef33bbb4,c,...) at trap_pfault+0x250 trap(ef33bc1c) at trap+0x3c6 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0631085, esp = 0xef33bc5c, ebp = 0xef33bc70 --- _mtx_lock_flags(1c,0,c0b9de0d,1d8,c0b932b0,...) at _mtx_lock_flags+0x15 vdev_geom_io_intr(c431c948,c09c6304,0,0,0) at vdev_geom_io_intr+0x44 biodone(c431c948,c0a12008,24c,c092bdb9,64,...) at biodone+0xad g_io_schedule_up(c3f0dc60,4c,c092ca83,5b,0,...) at g_io_schedule_up+0x7f g_up_procbody(0,ef33bd38,102640,ffffefff,ffffffff,...) at g_up_procbody+0x6c fork_exit(c05ed200,0,ef33bd38) at fork_exit+0x97 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef33bd70, ebp = 0 --- Uptime: 11m42s Physical memory: 1011 MB Dumping 340 MB: 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5 Unread portion of the kernel message buffer: panic: kmem_malloc(32768): kmem_map too small: 533876736 total allocated cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c09337fe,f320c034,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0951d54,0,c094a215,f320c040,0,...) at kdb_backtrace+0x29 panic(c094a215,8000,1fd25000,c094a20f,7d0,...) at panic+0x10f kmem_malloc(c147108c,8000,2,f320c0c0,c08541b0,...) at kmem_malloc+0x284 page_alloc(0,8000,f320c0b3,2,2eed800,...) at page_alloc+0x27 uma_large_malloc(8000,2,400,c44d8000,0,...) at uma_large_malloc+0x50 malloc(8000,c0b9f1c0,2,f320c104,c0b7a719,...) at malloc+0x8d zfs_kmem_alloc(8000,2,f320c1f8,c0b6b570,8000,...) at zfs_kmem_alloc+0x20 zio_buf_alloc(8000,0,400,0,f320c13c,...) at zio_buf_alloc+0x19 vdev_raidz_io_start(c67ab8a0,c859ee00,c43a9400,efeb8000,95,...) at vdev_raidz_io_start+0x390 vdev_mirror_io_start(c67abac8,20000,0,c859ee00,1,...) at vdev_mirror_io_start+0x1fc zio_dva_allocate(c67abac8,24a24,0,c859ee00,c61679c4,...) at zio_dva_allocate+0x115 zio_write_compress(c67abac8,c0b9cbac,329,326,1,...) at zio_write_compress+0x178 zio_wait_for_children(c67abac8,c6cb1900,7dd,760,c44dc9b4,...) at zio_wait_for_children+0x63 dbuf_sync_list(c8515c38,c6cb1900,c0b97eb7,74a,c4509cb0,...) at dbuf_sync_list+0x327 dbuf_sync_list(c67c0c38,c6cb1900,c0b97eb7,74a,c4509cb0,...) at dbuf_sync_list+0x117 dbuf_sync_list(c4509d50,c6cb1900,257,250,0,...) at dbuf_sync_list+0x117 dnode_sync(c4509cb0,c6cb1900,0,a0,c4fd2800,...) at dnode_sync+0x1e2 dmu_objset_sync_dnodes(c510bae0,c6cb1900,7,3,2,...) at dmu_objset_sync_dnodes+0x37 dmu_objset_sync(c4fd2800,c8e5f678,c6cb1900,0,c4e8d898,...) at dmu_objset_sync+0x15a dsl_pool_sync(c4e8d800,24a24,0,0,0,...) at dsl_pool_sync+0x98 spa_sync(c41bc000,24a24,0,13a,c4e8d940,...) at spa_sync+0x3f7 txg_sync_thread(c4e8d800,f320cd38,fe36743f,21a9e16c,ac45c4ff,...) at txg_sync_thread+0x1ea fork_exit(c0b61b90,c4e8d800,f320cd38) at fork_exit+0x97 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf320cd70, ebp = 0 --- Uptime: 17m48s Physical memory: 1011 MB Dumping 559 MB: 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 dmesg: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-BETA4 #2: Thu Dec 6 23:02:30 PST 2007 root@zfsserver.mtv.bitsurf.net:/usr/obj/usr/src/sys/KERNEL Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) XP 2000+ (1659.61-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x680 Stepping = 0 Features=0x383f9ff AMD Features=0xc0400800 real memory = 1073676288 (1023 MB) avail memory = 1036881920 (988 MB) kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 agp0: on hostb0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: mem 0xce000000-0xceffffff,0xb0000000-0xb7ffffff,0xbdb80000-0xbdbfffff irq 11 at device 0.0 on pci1 isab0: at device 2.0 on pci0 isa0: on isab0 ohci0: mem 0xcfffd000-0xcfffdfff irq 10 at device 2.2 on pci0 ohci0: [GIANT-LOCKED] ohci0: [ITHREAD] usb0: OHCI version 1.0, legacy support usb0: on ohci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 3 ports with 3 removable, self powered ohci1: mem 0xcfffe000-0xcfffefff irq 12 at device 2.3 on pci0 ohci1: [GIANT-LOCKED] ohci1: [ITHREAD] usb1: OHCI version 1.0, legacy support usb1: on ohci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 3 ports with 3 removable, self powered atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 2.5 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 2.7 (no driver attached) sis0: port 0xc800-0xc8ff mem 0xcfffc000-0xcfffcfff irq 10 at device 3.0 on pci0 miibus0: on sis0 rlphy0: PHY 1 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto sis0: Ethernet address: 00:0a:e6:8a:3b:e3 sis0: [ITHREAD] pci0: at device 11.0 (no driver attached) pci0: at device 13.0 (no driver attached) atapci1: port 0xdc00-0xdc3f,0xd800-0xd80f,0xd400-0xd47f mem 0xcffff000-0xcfffffff,0xcffc0000-0xcffdffff irq 10 at device 15.0 on pci0 atapci1: [ITHREAD] atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] atapci2: port 0xc400-0xc47f,0xc000-0xc0ff mem 0xcfffb000-0xcfffbfff,0xcff80000-0xcff9ffff irq 11 at device 17.0 on pci0 atapci2: [ITHREAD] atapci2: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] ata6: on atapci2 ata6: [ITHREAD] ata7: on atapci2 ata7: [ITHREAD] ata8: on atapci2 ata8: [ITHREAD] uhci0: port 0xb800-0xb81f irq 11 at device 19.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb2: on uhci0 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci1: port 0xbc00-0xbc1f irq 11 at device 19.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb3: on uhci1 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xcfffaf00-0xcfffafff irq 5 at device 19.2 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 0.95 usb4: companion controllers, 2 ports each: usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 4 ports with 4 removable, self powered acpi_button0: on acpi0 fdc0: port 0x3f2-0x3f3,0x3f4-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] pmtimer0 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 uhub5: on uhub2 uhub5: 3 ports with 2 removable, bus powered ukbd0: on uhub5 kbd2 at ukbd0 uhid0: on uhub5 WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounter "TSC" frequency 1659613591 Hz quality 800 Timecounters tick every 1.000 msec ad0: 114473MB at ata0-master UDMA100 ZFS filesystem version 6 ZFS storage pool version 6 acd0: DVDROM at ata1-slave UDMA33 ad8: 239372MB at ata4-master UDMA133 ad10: 238475MB at ata5-master SATA150 ad12: 238475MB at ata6-master SATA150 ad14: 238475MB at ata7-master SATA150 ad16: 238475MB at ata8-master SATA150 Trying to mount root from zfs:tank Using the GENERIC kernel with DDB and KDB additions. /boot/loader.conf: autoboot_delay="2" zfs_load="YES" vfs.root.mountfrom="zfs:tank" #vfs.zfs.prefetch_disable="1" vm.kmem_size="512M" vm.kmem_size_max="512M" /usr/src was synced to RELENG_7 yesterday (and used to build the kernel that paniced). zfsserver# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT data 1.14T 812G 352G 69% ONLINE - tank 110G 71.5G 38.5G 64% ONLINE - zfsserver# zfs list NAME USED AVAIL REFER MOUNTPOINT data 649G 267G 511G /data data/backups 30.4K 267G 30.4K /data/backups data/mythtv 138G 267G 138G /data/mythtv tank 72.5G 35.8G 462M legacy tank/backups 20.0G 35.8G 20.0G /tank/backups tank/home 126K 35.8G 26.5K /home tank/home/mharo 100K 35.8G 100K /home/mharo tank/random 47.0G 35.8G 47.0G /tank/random tank/swap 155K 36.8G 155K - tank/usr 2.64G 35.8G 2.11G /usr tank/usr/ports 392M 35.8G 139M /usr/ports tank/usr/ports/distfiles 254M 35.8G 254M /usr/ports/distfiles tank/usr/src 152M 35.8G 152M /usr/src tank/var 1.43G 35.8G 1.43G /var tank/var/log 414K 35.8G 414K /var/log >Description: After a few minutes of moving data from one zfs partition to another in the same zpool the machine panics (see above). I suspect this is zfs related, but don't know for sure. >How-To-Repeat: cd /data mv -v mythtv.orig/recordings/*.nuv mythtv/recordings/ wait a few minutes I think it happens when copying files > 1G in size, but I haven't traced it down to which file was being moved when the system paniced. I've tried setting vfs.zfs.prefetch_disable="1" and vfs.zfs.prefetch_disable="0" but it doesn't seemt to matter. I also tried applying http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch I *think* that made my system more stable, but I'm still crashing in under 30 minutes. >Fix: Unknown. Willing to try other things. >Release-Note: >Audit-Trail: >Unformatted: