From owner-freebsd-emulation@FreeBSD.ORG Sun Jan 31 16:56:43 2010 Return-Path: Delivered-To: freebsd-emulation@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05B17106568B for ; Sun, 31 Jan 2010 16:56:43 +0000 (UTC) (envelope-from jamesbrandongooch@gmail.com) Received: from mail-pz0-f194.google.com (mail-pz0-f194.google.com [209.85.222.194]) by mx1.freebsd.org (Postfix) with ESMTP id D06168FC08 for ; Sun, 31 Jan 2010 16:56:42 +0000 (UTC) Received: by pzk32 with SMTP id 32so933951pzk.27 for ; Sun, 31 Jan 2010 08:56:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=Vy1C2NTzKqkSrv7B7pxOFZYqY2zfKKRRxWT4S4oGoa4=; b=xicb/6SyNtGQ/m+YGOt6e5dkIdlfhE37TD/eJmRibJqxwyalLG0gmSBx/d4z4MlEtR CHOGQZ6Wt+loWpBY+yID48C8b+Q/BS9Tez3gmgkQO5DzSaRJARCb5dGVkwYZcT9br0n0 NbS9vVz4Tr9/kefgK6rymru5pgSUFxFUSHqvY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=uaPsJ4FLw5xXzLggoLtDyNIqq+p6nofzhQOV17mmRscnIMEZPAEP1xSyXeol3ERDur gpFUiOdcDD1MyU62vCDcgg+xjkwvkLLDhY8V110acyNYEvrhoqkrq9C1OmcbAARXa910 qbYGaMgENuRC5OrFPaXgMuTM4j/QMl1eUL5c8= MIME-Version: 1.0 Received: by 10.142.248.4 with SMTP id v4mr2323750wfh.258.1264957002190; Sun, 31 Jan 2010 08:56:42 -0800 (PST) In-Reply-To: <179b97fb1001270941m2d8e9c8au20abc798c16b9c11@mail.gmail.com> References: <179b97fb1001270941m2d8e9c8au20abc798c16b9c11@mail.gmail.com> Date: Sun, 31 Jan 2010 16:56:42 +0000 Message-ID: <179b97fb1001310856j6ab9474bve2a0ee40bfccc1ab@mail.gmail.com> From: Brandon Gooch To: freebsd-emulation@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: ZFS and sh(1) panic: spin lock [lock addr] (smp rendezvous) held by [sh(1) proc tid] too long X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jan 2010 16:56:43 -0000 I've a few more details now (the original subject seems a little off now). I added INVARIANTS and WITNESS to my kernel config. I feel the first textdump wasn't as revealing as this most recent one. >From the most recent textdump: spin lock 0xffffffff80852940 (smp rendezvous) held by 0xffffff0005eeaae0 (tid 10 0170) too long panic: spin lock held too long cpuid = 2 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 _mtx_lock_spin_failed() at _mtx_lock_spin_failed+0x61 _mtx_lock_spin() at _mtx_lock_spin+0xb2 _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x149 smp_rendezvous_cpus() at smp_rendezvous_cpus+0xd3 RTMpPokeCpu() at RTMpPokeCpu+0x35 g_aAdapters() at g_aAdapters+0x54b2 g_aAdapters() at 0xffffffff81e64795 g_aAdapters() at 0xffffffff81e64a7b supdrvIOCtl() at supdrvIOCtl+0x17bd VBoxDrvFreeBSDIOCtl() at VBoxDrvFreeBSDIOCtl+0x1f6 devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x1e4 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800c8c9fc, rsp = 0x7fffff46cbe8, rbp = 0x7fffff46cbf0 --- KDB: enter: panic exclusive sx IPRT Fast Mutex Semaphore (IPRT Fast Mutex Semaphore) r = 0 (0xffff ff0007123a98) locked @ /usr/ports/emulators/virtualbox-ose-kmod/work/VirtualBox- 3.1.2_OSE/out/freebsd.amd64/release/bin/src/vboxdrv/r0drv/freebsd/semfastmutex-r 0drv-freebsd.c:103 exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0007d8d648) locked @ /usr/src/ sys/kern/uipc_sockbuf.c:148 exclusive sleep mutex vm object (standard object) r = 0 (0xffffff0007ba3948) loc ked @ /usr/src/sys/vm/vm_fault.c:297 shared sx user map (user map) r = 0 (0xffffff0007268200) locked @ /usr/src/sys/vm/vm_map.c:3532 exclusive sleep mutex pmap (pmap) r = 0 (0xffffffff80a4c3c0) locked @ /usr/src/sys/amd64/amd64/pmap.c:2463 exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff80a02940) locked @ /usr/src/sys/amd64/amd64/pmap.c:2462 exclusive sleep mutex system map (system map) r = 0 (0xffffff0002000180) locked @ /usr/src/sys/vm/vm_map.c:2772 exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff8082fec0) locked @ /usr/src/sys/kern/kern_sysctl.c:1417 shared sx sysctl lock (sysctl lock) r = 0 (0xffffffff80830860) locked @ /usr/src/sys/kern/kern_sysctl.c:1521 I can make the entire textdump available on request. Also, I can provide any other detail necessary to help solve this, including access to the box. Thanks to whomever may be able to take a look at this! -Brandon