From owner-freebsd-current@FreeBSD.ORG Wed Jun 20 15:53:50 2007 Return-Path: X-Original-To: Current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6229B16A469 for ; Wed, 20 Jun 2007 15:53:50 +0000 (UTC) (envelope-from marcus@FreeBSD.org) Received: from creme-brulee.marcuscom.com (creme-brulee.marcuscom.com [24.172.16.118]) by mx1.freebsd.org (Postfix) with ESMTP id F19C313C45A for ; Wed, 20 Jun 2007 15:53:49 +0000 (UTC) (envelope-from marcus@FreeBSD.org) Received: from [IPv6:2001:470:1f00:2464::4] (shumai.marcuscom.com [IPv6:2001:470:1f00:2464::4]) by creme-brulee.marcuscom.com (8.14.1/8.14.1) with ESMTP id l5KFseUF011470 for ; Wed, 20 Jun 2007 11:54:40 -0400 (EDT) (envelope-from marcus@FreeBSD.org) From: Joe Marcus Clarke To: Current@FreeBSD.org Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-gme0goPbq6jcP1AhuPl7" Organization: FreeBSD, Inc. Date: Wed, 20 Jun 2007 11:53:43 -0400 Message-Id: <1182354823.6504.23.camel@shumai.marcuscom.com> Mime-Version: 1.0 X-Mailer: Evolution 2.10.2 FreeBSD GNOME Team Port X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,NO_RELAYS autolearn=no version=3.2.0 X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on creme-brulee.marcuscom.com Cc: Subject: ZFS and deadlock with {nullfs,NFS} X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jun 2007 15:53:50 -0000 --=-gme0goPbq6jcP1AhuPl7 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable I've resurrected by amd64 Tinderbox with a ZFS base, and I've been seeing a 100% reproducible deadlock when I use it with either localhost NFS or nullfs. When this occurs, the CPU is 100% idle, but I can no longer connect via SSH, and the box will only reboot from the debugger. I know there are some tuning bits I can tweak, but all I've run across is for memory consumption. Any pointers would be helpful. I'm also at the debugger, so if there is anything I can do to help troubleshoot why this is happening, please let me know. =20 This box is -CURRENT as of June 19, 2007. It has a GENERIC kernel minus devices I do not have (i.e. SMP kernel). I am currently using nullfs for the Tinderbox. The process that most regularly locks up is mtree. Here is the trace: FreeBSD cobbler.marcuscom.com 7.0-CURRENT FreeBSD 7.0-CURRENT #32: Tue Jun 19 12:53:28 EDT 2007 marcus@cobbler.marcuscom.com:/usr/obj/usr/src/sys/COBBLER amd64 Tracing command mtree pid 34844 tid 100169 td 0xffffff0060350680 sched_switch() at sched_switch+0x176 mi_switch() at mi_switch+0x231 sleepq_switch() at sleepq_switch+0xc7 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x14e zio_wait() at zio_wait+0x5f dbuf_read() at dbuf_read+0x2d5 dbuf_findbp() at dbuf_findbp+0xf7 dbuf_prefetch() at dbuf_prefetch+0xa9 dmu_zfetch_dofetch() at dmu_zfetch_dofetch+0xd4 dmu_zfetch() at dmu_zfetch+0x603 dbuf_read() at dbuf_read+0x535 dnode_hold_impl() at dnode_hold_impl+0xe7 dmu_bonus_hold() at dmu_bonus_hold+0x32 zfs_zget() at zfs_zget+0x6b zfs_dirent_lock() at zfs_dirent_lock+0x2fc zfs_dirlook() at zfs_dirlook+0x58 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x13d vfs_cache_lookup() at vfs_cache_lookup+0xf8 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x6d lookup() at lookup+0x363 namei() at namei+0x2ca kern_lstat() at kern_lstat+0x5e lstat() at lstat+0x2a syscall() at syscall+0x1ce Xfast_syscall() at Xfast_syscall+0xab --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800807e5c, rsp =3D 0x7fffffffe5b8, rbp =3D 0x556170 --- A full process list from the debugger can be found at http://www.marcuscom.com/downloads/cobbler_proc.txt . Joe --=20 Joe Marcus Clarke FreeBSD GNOME Team :: gnome@FreeBSD.org FreeNode / #freebsd-gnome http://www.FreeBSD.org/gnome --=-gme0goPbq6jcP1AhuPl7 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQBGeU2Gb2iPiv4Uz4cRAm1vAJsG+eaMf5x5amP2+/dT0FHtykomWQCfc96V lR7KxcTu6q7/htqyzQBosjw= =a1GC -----END PGP SIGNATURE----- --=-gme0goPbq6jcP1AhuPl7--