Date: Tue, 6 Oct 2015 21:27:57 +0200 From: Freek Dijkstra <public@macfreek.nl> To: freebsd-questions@freebsd.org Subject: ZFS fatal double fault upon reboot after upgrade to 10.2 Message-ID: <561420BD.6090700@macfreek.nl>
next in thread | raw e-mail | index | archive | help
Hi all, After an upgrade from FreeBSD 10.0 to 10.2, my system no longer boots with an error: Trying to mount root from zfs:zroot []... Fatal double fault: When booting from an USB stick, I'm able to mount my zpool just fine, so I suspect this is a software or configuration error, rather than a hardware error. I appreciate any advice how to proceed. Details of my case are below. Context: I have a system running FreeBSD as my router, since about 3 year. Hardware: Soekris net6501-50 Intel Atom E640 CPU (32 bit), just 1 GByte memory. Disks: two SSD (mini-SATA) Intel SSD 525 Series 30GByte. ZFS config: pool: zroot state: ONLINE scan: resilvered 13.0M in 0h0m with 0 errors on Sat Oct 3 15:37:46 2015 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/system0 ONLINE 0 0 0 gpt/system1 ONLINE 0 0 0 errors: No known data errors System is running FreeBSD 10.0, so the upgrade to 10.2 is actually long overdue. `freebsd-update -r 10.2-RELEASE upgrade` worked fine, just had to restart because of lost connection (broken pipe) and I forgot to use `screen`. /usr/sbin/freebsd-update install Installing updates...^@^@^@^@^@^@^@^@^@^@^@^@^@ Kernel updates have been installed. Please reboot and run "/usr/sbin/freebsd-update install" again to finish installing updates. [root@wolfje] ~# /sbin/shutdown -r now Shutdown NOW! shutdown: [pid 70238] Then after reboot, system did not come up. Memory check (by Soekris BIOS) is fine. Excerpt from the tty output: /boot/kernel/kernel text=0xf79f95 data=0xdb7e8+0x160f88 syms=[0x4+0xd7fa0+0x4+0x15f0e8] /boot/kernel/zfs.ko text=0x19267c data=0x9d24+0x546cc syms=[0x4+0x113f0+0x4+0x181ed] loading required module 'opensolaris' /boot/kernel/opensolaris.ko text=0x1758 data=0x18c+0xc24 syms=[0x4+0x850+0x4+0x882] /boot/kernel/if_tap.ko text=0x3cb8 data=0x2c4+0x38 syms=[0x4+0xac0+0x4+0xa6c] /boot/kernel/ip_mroute.ko text=0x904c data=0x2cc+0x360c syms=[0x4+0xe70+0x4+0x1014] Booting... ACPI BIOS Error (bug): A valid RSDP was not found (20150515/tbxfroot-258) Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 19:31:38 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC i386 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Genuine Intel(R) CPU @ 1.00GHz (1000.02-MHz 686-class CPU) Origin="GenuineIntel" Id=0x20661 Family=0x6 Model=0x26 Stepping=1 Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x40e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE> AMD Features=0x20100000<NX,LM> AMD Features2=0x1<LAHF> VT-x: (disabled in BIOS) HLT,PAUSE TSC: P-state invariant, performance statistics real memory = 1073610752 (1023 MB) avail memory = 1025433600 (977 MB) MPTable: <Soekris net6501 > [...] ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) [...] ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 uhub5: 1 port with 1 removable, self powered ada0: <INTEL SSDMCEAC030B3 LLLi> ACS-2 ATA SATA 3.x device ada0: Serial Number CVLI3212005D030H ada0: 300.000MB/s transfersuhub6: 1 port with 1 removable, self powered (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 28626MB (58626288 512 byte sectors: 16H 63S/T 16383C) SMP: AP CPU #1 Launched! Timecounter "TSC" frequency 1000022290 Hz quality 1000 Root mount waiting for: usbus7 usbus3 uhub3: 3 ports with 3 removable, self powered uhub7: 3 ports with 3 removable, self powered Trying to mount root from zfs:zroot []... Fatal double fault: eip = 0xc181f6d4 esp = 0xda377ff0 ebp = 0xda378024 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: stack backtrace: #0 0xc0b720f2 at kdb_backtrace+0x52 #1 0xc0b332cb at vpanic+0x11b #2 0xc0b331ab at panic+0x1b #3 0xc10554eb at dblfault_handler+0xab Uptime: 4s Automatic reboot in 15 seconds - press a key on the console to abort After a reboot, the same error occurs. The eip, esp and ebp number are exactly the same for each reboot, so I suspect a software issue (as opposed to a hardware issue). The warning about kernel memory also occured with FreeBSD 10.0, but I had ignored that since everything seemed to work in practice. I wanted to check if the zpool is fine, so I booted the machine from an USB stick with FreeBSD-10.2-RELEASE-i386-mini-memstick.img. This worked fine. I could even mount my zpool: # zpool list ZFS NOTICE: Prefetch is disabled by default on i386 -- to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version: 5 ZFS storage pool version: features support (5000) no pools available # zpool import pool: zroot id: 13393886111405456989 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: http://illumos.org/msg/ZFS-8000-EY config: zroot ONLINE mirror-0 ONLINE gpt/system0 ONLINE gpt/system1 ONLINE # ls /mnt # zpool import -f -o readonly=on -R /mnt zroot # ls /mnt .cshrc boot lib proc tmp .profile dev libexec rescue usr .rnd entropy man8 root var COPYRIGHT etc media sbin bin home mnt sys How to proceed? A search on the Internet for "ZFS fatal double fault" gave the following result: http://lists.freebsd.org/pipermail/freebsd-stable/2014-December /081196.html > Either switch to amd64 or increase the stack size of your kernel. Would that help here as well? How do I increase the stack size of the kernel? Regards, Freek Dijkstra
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?561420BD.6090700>