Date: Fri, 7 Dec 2007 12:59:16 -0800 (PST) From: Michael Haro <mharo@FreeBSD.org> To: FreeBSD-gnats-submit@FreeBSD.org Cc: mharo@FreeBSD.org Subject: kern/118487: zfs related kernel panic on RELENG_7 (7.0BETA4): kmem_malloc(XXX): kmem_map too small Message-ID: <200712072059.lB7KxGoP001298@zfsserver.mtv.bitsurf.net> Resent-Message-ID: <200712072130.lB7LU4Xe031430@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 118487 >Category: kern >Synopsis: zfs related kernel panic on RELENG_7 (7.0BETA4): kmem_malloc(XXX): kmem_map too small >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Dec 07 21:30:04 UTC 2007 >Closed-Date: >Last-Modified: >Originator: Michael Haro >Release: FreeBSD 7.0-BETA4 i386 >Organization: >Environment: System: FreeBSD zfsserver.mtv.bitsurf.net 7.0-BETA4 FreeBSD 7.0-BETA4 #2: Thu Dec 6 23:02:30 PST 2007 root@zfsserver.mtv.bitsurf.net:/usr/obj/usr/src/sys/KERNEL i386 I have 3 kernel dumps. I suspect they're related... Unread portion of the kernel message buffer: panic: kmem_malloc(131072): kmem_map too small: 418766848 total allocated cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c09337fe,f3155af0,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0951d54,0,c094a215,f3155afc,0,...) at kdb_backtrace+0x29 panic(c094a215,20000,18f5e000,c094a20f,7d0,...) at panic+0x10f kmem_malloc(c147108c,20000,2,f3155b7c,c08541b0,...) at kmem_malloc+0x284 page_alloc(0,20000,f3155b6f,2,24539a8,...) at page_alloc+0x27 uma_large_malloc(20000,2,c683ccf0,c6488ac8,28000,...) at uma_large_malloc+0x50 malloc(20000,c0b9f1c0,2,f3155bc0,c0b7a719,...) at malloc+0x8d zfs_kmem_alloc(20000,2,f3155c48,c0b699bc,20000,...) at zfs_kmem_alloc+0x20 zio_buf_alloc(20000,c6488ac8,1,3b7,33860000,...) at zio_buf_alloc+0x19 vdev_queue_io_to_issue(f3155c64,c6bcf228,c0b9c2d0,133,c43ab2e4,...) at vdev_queue_io_to_issue+0x1bc vdev_queue_io_done(c6bcf228,f3155ce0,f3155ce0,f3155cf4,c0b23ba0,...) at vdev_queue_io_done+0xa8 vdev_geom_io_done(c6bcf228,0,c0b976f3,33d,2f652a7e,...) at vdev_geom_io_done+0x12 taskq_thread(c3f9a7f8,f3155d38,4,0,85,...) at taskq_thread+0x140 fork_exit(c0b23a60,c3f9a7f8,f3155d38) at fork_exit+0x97 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf3155d70, ebp = 0 --- Uptime: 9m48s Physical memory: 1011 MB Dumping 473 MB: 458 442 426 410 394 378 362 346 330 314 298 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10 Unread portion of the kernel message buffer: ad16: FAILURE - device detached subdisk16: detached ad16: detached Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x2c fault code = supervisor write, page not present instruction pointer = 0x20:0xc0631085 stack pointer = 0x28:0xef33bc5c frame pointer = 0x28:0xef33bc70 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3 (g_up) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c09337fe,ef33baf8,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0951d54,0,c0913383,ef33bb04,0,...) at kdb_backtrace+0x29 panic(c0913383,c0952ffb,c3f544d0,1,1,...) at panic+0x10f trap_fatal(c0a12600,0,2,8,76c5ec12,...) at trap_fatal+0x333 trap_pfault(c0a120a8,ef33bb90,c066ac9d,ef33bbb4,c,...) at trap_pfault+0x250 trap(ef33bc1c) at trap+0x3c6 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0631085, esp = 0xef33bc5c, ebp = 0xef33bc70 --- _mtx_lock_flags(1c,0,c0b9de0d,1d8,c0b932b0,...) at _mtx_lock_flags+0x15 vdev_geom_io_intr(c431c948,c09c6304,0,0,0) at vdev_geom_io_intr+0x44 biodone(c431c948,c0a12008,24c,c092bdb9,64,...) at biodone+0xad g_io_schedule_up(c3f0dc60,4c,c092ca83,5b,0,...) at g_io_schedule_up+0x7f g_up_procbody(0,ef33bd38,102640,ffffefff,ffffffff,...) at g_up_procbody+0x6c fork_exit(c05ed200,0,ef33bd38) at fork_exit+0x97 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xef33bd70, ebp = 0 --- Uptime: 11m42s Physical memory: 1011 MB Dumping 340 MB: 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5 Unread portion of the kernel message buffer: panic: kmem_malloc(32768): kmem_map too small: 533876736 total allocated cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper(c09337fe,f320c034,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0951d54,0,c094a215,f320c040,0,...) at kdb_backtrace+0x29 panic(c094a215,8000,1fd25000,c094a20f,7d0,...) at panic+0x10f kmem_malloc(c147108c,8000,2,f320c0c0,c08541b0,...) at kmem_malloc+0x284 page_alloc(0,8000,f320c0b3,2,2eed800,...) at page_alloc+0x27 uma_large_malloc(8000,2,400,c44d8000,0,...) at uma_large_malloc+0x50 malloc(8000,c0b9f1c0,2,f320c104,c0b7a719,...) at malloc+0x8d zfs_kmem_alloc(8000,2,f320c1f8,c0b6b570,8000,...) at zfs_kmem_alloc+0x20 zio_buf_alloc(8000,0,400,0,f320c13c,...) at zio_buf_alloc+0x19 vdev_raidz_io_start(c67ab8a0,c859ee00,c43a9400,efeb8000,95,...) at vdev_raidz_io_start+0x390 vdev_mirror_io_start(c67abac8,20000,0,c859ee00,1,...) at vdev_mirror_io_start+0x1fc zio_dva_allocate(c67abac8,24a24,0,c859ee00,c61679c4,...) at zio_dva_allocate+0x115 zio_write_compress(c67abac8,c0b9cbac,329,326,1,...) at zio_write_compress+0x178 zio_wait_for_children(c67abac8,c6cb1900,7dd,760,c44dc9b4,...) at zio_wait_for_children+0x63 dbuf_sync_list(c8515c38,c6cb1900,c0b97eb7,74a,c4509cb0,...) at dbuf_sync_list+0x327 dbuf_sync_list(c67c0c38,c6cb1900,c0b97eb7,74a,c4509cb0,...) at dbuf_sync_list+0x117 dbuf_sync_list(c4509d50,c6cb1900,257,250,0,...) at dbuf_sync_list+0x117 dnode_sync(c4509cb0,c6cb1900,0,a0,c4fd2800,...) at dnode_sync+0x1e2 dmu_objset_sync_dnodes(c510bae0,c6cb1900,7,3,2,...) at dmu_objset_sync_dnodes+0x37 dmu_objset_sync(c4fd2800,c8e5f678,c6cb1900,0,c4e8d898,...) at dmu_objset_sync+0x15a dsl_pool_sync(c4e8d800,24a24,0,0,0,...) at dsl_pool_sync+0x98 spa_sync(c41bc000,24a24,0,13a,c4e8d940,...) at spa_sync+0x3f7 txg_sync_thread(c4e8d800,f320cd38,fe36743f,21a9e16c,ac45c4ff,...) at txg_sync_thread+0x1ea fork_exit(c0b61b90,c4e8d800,f320cd38) at fork_exit+0x97 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xf320cd70, ebp = 0 --- Uptime: 17m48s Physical memory: 1011 MB Dumping 559 MB: 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 dmesg: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-BETA4 #2: Thu Dec 6 23:02:30 PST 2007 root@zfsserver.mtv.bitsurf.net:/usr/obj/usr/src/sys/KERNEL Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) XP 2000+ (1659.61-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x680 Stepping = 0 Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE> AMD Features=0xc0400800<SYSCALL,MMX+,3DNow!+,3DNow!> real memory = 1073676288 (1023 MB) avail memory = 1036881920 (988 MB) kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: <AMIINT SiS735XX> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 agp0: <SiS 735 host to AGP bridge> on hostb0 pcib1: <PCI-PCI bridge> at device 1.0 on pci0 pci1: <PCI bus> on pcib1 vgapci0: <VGA-compatible display> mem 0xce000000-0xceffffff,0xb0000000-0xb7ffffff,0xbdb80000-0xbdbfffff irq 11 at device 0.0 on pci1 isab0: <PCI-ISA bridge> at device 2.0 on pci0 isa0: <ISA bus> on isab0 ohci0: <SiS 5571 USB controller> mem 0xcfffd000-0xcfffdfff irq 10 at device 2.2 on pci0 ohci0: [GIANT-LOCKED] ohci0: [ITHREAD] usb0: OHCI version 1.0, legacy support usb0: <SiS 5571 USB controller> on ohci0 usb0: USB revision 1.0 uhub0: <SiS OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 3 ports with 3 removable, self powered ohci1: <SiS 5571 USB controller> mem 0xcfffe000-0xcfffefff irq 12 at device 2.3 on pci0 ohci1: [GIANT-LOCKED] ohci1: [ITHREAD] usb1: OHCI version 1.0, legacy support usb1: <SiS 5571 USB controller> on ohci1 usb1: USB revision 1.0 uhub1: <SiS OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 3 ports with 3 removable, self powered atapci0: <SiS 735 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 2.5 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] pci0: <multimedia, audio> at device 2.7 (no driver attached) sis0: <SiS 900 10/100BaseTX> port 0xc800-0xc8ff mem 0xcfffc000-0xcfffcfff irq 10 at device 3.0 on pci0 miibus0: <MII bus> on sis0 rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto sis0: Ethernet address: 00:0a:e6:8a:3b:e3 sis0: [ITHREAD] pci0: <multimedia, video> at device 11.0 (no driver attached) pci0: <multimedia, video> at device 13.0 (no driver attached) atapci1: <Promise PDC20375 SATA150 controller> port 0xdc00-0xdc3f,0xd800-0xd80f,0xd400-0xd47f mem 0xcffff000-0xcfffffff,0xcffc0000-0xcffdffff irq 10 at device 15.0 on pci0 atapci1: [ITHREAD] atapci1: [ITHREAD] ata2: <ATA channel 0> on atapci1 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci1 ata3: [ITHREAD] ata4: <ATA channel 2> on atapci1 ata4: [ITHREAD] atapci2: <Promise PDC40518 SATA150 controller> port 0xc400-0xc47f,0xc000-0xc0ff mem 0xcfffb000-0xcfffbfff,0xcff80000-0xcff9ffff irq 11 at device 17.0 on pci0 atapci2: [ITHREAD] atapci2: [ITHREAD] ata5: <ATA channel 0> on atapci2 ata5: [ITHREAD] ata6: <ATA channel 1> on atapci2 ata6: [ITHREAD] ata7: <ATA channel 2> on atapci2 ata7: [ITHREAD] ata8: <ATA channel 3> on atapci2 ata8: [ITHREAD] uhci0: <VIA 83C572 USB controller> port 0xb800-0xb81f irq 11 at device 19.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb2: <VIA 83C572 USB controller> on uhci0 usb2: USB revision 1.0 uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 uhub2: 2 ports with 2 removable, self powered uhci1: <VIA 83C572 USB controller> port 0xbc00-0xbc1f irq 11 at device 19.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb3: <VIA 83C572 USB controller> on uhci1 usb3: USB revision 1.0 uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: <VIA VT6202 USB 2.0 controller> mem 0xcfffaf00-0xcfffafff irq 5 at device 19.2 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 0.95 usb4: companion controllers, 2 ports each: usb2 usb3 usb4: <VIA VT6202 USB 2.0 controller> on ehci0 usb4: USB revision 2.0 uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4 uhub4: 4 ports with 4 removable, self powered acpi_button0: <Sleep Button> on acpi0 fdc0: <floppy drive controller> port 0x3f2-0x3f3,0x3f4-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] pmtimer0 on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 ppc0: [GIANT-LOCKED] ppc0: [ITHREAD] vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 uhub5: <Dell Dell USB Keyboard Hub, class 9/0, rev 1.10/2.00, addr 2> on uhub2 uhub5: 3 ports with 2 removable, bus powered ukbd0: <Dell Dell USB Keyboard Hub, class 0/0, rev 1.10/2.00, addr 3> on uhub5 kbd2 at ukbd0 uhid0: <Dell Dell USB Keyboard Hub, class 0/0, rev 1.10/2.00, addr 3> on uhub5 WARNING: ZFS is considered to be an experimental feature in FreeBSD. Timecounter "TSC" frequency 1659613591 Hz quality 800 Timecounters tick every 1.000 msec ad0: 114473MB <WDC WD1200BB-00CJA1 17.07W17> at ata0-master UDMA100 ZFS filesystem version 6 ZFS storage pool version 6 acd0: DVDROM <HL-DT-STDVD-ROM GDR8163B/0L23> at ata1-slave UDMA33 ad8: 239372MB <Maxtor 6Y250P0 YAR41BW0> at ata4-master UDMA133 ad10: 238475MB <HDS722525VLSA80 V36OA6MA> at ata5-master SATA150 ad12: 238475MB <HDS722525VLSA80 V36OA6MA> at ata6-master SATA150 ad14: 238475MB <HDS722525VLSA80 V36OA6MA> at ata7-master SATA150 ad16: 238475MB <HDS722525VLSA80 V36OA6MA> at ata8-master SATA150 Trying to mount root from zfs:tank Using the GENERIC kernel with DDB and KDB additions. /boot/loader.conf: autoboot_delay="2" zfs_load="YES" vfs.root.mountfrom="zfs:tank" #vfs.zfs.prefetch_disable="1" vm.kmem_size="512M" vm.kmem_size_max="512M" /usr/src was synced to RELENG_7 yesterday (and used to build the kernel that paniced). zfsserver# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT data 1.14T 812G 352G 69% ONLINE - tank 110G 71.5G 38.5G 64% ONLINE - zfsserver# zfs list NAME USED AVAIL REFER MOUNTPOINT data 649G 267G 511G /data data/backups 30.4K 267G 30.4K /data/backups data/mythtv 138G 267G 138G /data/mythtv tank 72.5G 35.8G 462M legacy tank/backups 20.0G 35.8G 20.0G /tank/backups tank/home 126K 35.8G 26.5K /home tank/home/mharo 100K 35.8G 100K /home/mharo tank/random 47.0G 35.8G 47.0G /tank/random tank/swap 155K 36.8G 155K - tank/usr 2.64G 35.8G 2.11G /usr tank/usr/ports 392M 35.8G 139M /usr/ports tank/usr/ports/distfiles 254M 35.8G 254M /usr/ports/distfiles tank/usr/src 152M 35.8G 152M /usr/src tank/var 1.43G 35.8G 1.43G /var tank/var/log 414K 35.8G 414K /var/log >Description: After a few minutes of moving data from one zfs partition to another in the same zpool the machine panics (see above). I suspect this is zfs related, but don't know for sure. >How-To-Repeat: cd /data mv -v mythtv.orig/recordings/*.nuv mythtv/recordings/ wait a few minutes I think it happens when copying files > 1G in size, but I haven't traced it down to which file was being moved when the system paniced. I've tried setting vfs.zfs.prefetch_disable="1" and vfs.zfs.prefetch_disable="0" but it doesn't seemt to matter. I also tried applying http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch I *think* that made my system more stable, but I'm still crashing in under 30 minutes. >Fix: Unknown. Willing to try other things. >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200712072059.lB7KxGoP001298>