From owner-freebsd-stable@freebsd.org Wed Mar 2 09:12:34 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97DA6AC1054 for ; Wed, 2 Mar 2016 09:12:34 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 79CC810AD for ; Wed, 2 Mar 2016 09:12:34 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id 79856AC1052; Wed, 2 Mar 2016 09:12:34 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F5DAAC1051 for ; Wed, 2 Mar 2016 09:12:34 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F9AA10AA for ; Wed, 2 Mar 2016 09:12:33 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-wm0-x230.google.com with SMTP id p65so70420184wmp.1 for ; Wed, 02 Mar 2016 01:12:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:date:message-id:subject:from:to:cc; bh=M3eErwsYcbIGbul4VGdGyj18qwbKY9IslEmwfvimpgU=; b=zWojK1QQP9vzc0VQtM828FEDjKA4FpTQaSkt661Ct6uoeh5WIEYtea0Q8gRMjucS75 CECBtol3FWOd4TeD/V7qQGUXg3eiENhq4162ZTLfedTZoD9iJd6JKyR9vRimW2fzzx16 wCZGG50NjV0gp0Xq7sK4N9SI3OUNWkOnHoOEMKiNloIfeHE2ZjKNwl6VP48eAyeXTp2E kLh1Yufo2m9bsPc3KLHHZKpa5xe35kZ4/M3CyINk/W0CFko4jIcb6ZmN85jO++FcQZy8 ubWR1N1oPc51mQTyFjjsW+Vvz2BRpCZiywZOc5NLypth1SE41UKQ64WmnXefZopQJ99o FB0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:cc; bh=M3eErwsYcbIGbul4VGdGyj18qwbKY9IslEmwfvimpgU=; b=lF6G/+To0iSJaXAJsLD2a2BL57akyqYXk95IuKJ1neYDsST1MDFptPBC1vKGZkGVbD i+WWdg15dM1GPv6ePh9qlMoxd77gV02K86Bdl0+5FAaiivLtWRM8vrxAKZDsajEIFVXv nCfWEVqzHnEfpdjqVLbzyiwv8ke2+0vcQGL6iCgijgSlaFe6KDlj/iH10/pG2CRXBudq kfwS7VsgnkAUE40Wxv+w+0TxuSqm9WeTgZ82P9jzxCwVfXDn4Iij63ABLiR1PNIG8ZOl He2xNEoJF/NI3dy81Ioar/DQ38g06QRaEoektoplOY0b70qF2PYA5dard7qRDGzRWiLt syMA== X-Gm-Message-State: AD7BkJI34kZ2Zrzm3cJAW2e2ZNj6HWFKuaPiLsZ8pXNzwWZXhFxr7FNtZf4zNix6ejZcFloNXzfO+TzhQM8ZwZAp MIME-Version: 1.0 X-Received: by 10.28.148.16 with SMTP id w16mr1148701wmd.90.1456909951915; Wed, 02 Mar 2016 01:12:31 -0800 (PST) Sender: sobomax@sippysoft.com Received: by 10.27.218.12 with HTTP; Wed, 2 Mar 2016 01:12:31 -0800 (PST) Date: Wed, 2 Mar 2016 01:12:31 -0800 X-Google-Sender-Auth: 2EwUDXaJj1zU5SYvfTyflX5yKdc Message-ID: Subject: Process stuck in "vnread" From: Maxim Sobolev To: stable@freebsd.org, freebsd-fs@freebsd.org Cc: Kirk McKusick , kib@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 09:12:34 -0000 Hi, I've encountered cp(1) process stuck in the vnread state on one of my build machines that got recently upgraded to 10.3. 0 79596 1 0 20 0 17092 1396 wait I 1 0:00.00 /bin/sh /usr/local/bin/autoreconf -f -i 0 79602 79596 0 52 0 41488 9036 wait I 1 0:00.07 /usr/local/bin/perl -w /usr/local/bin/autoreconf-2.69 -f -i 0 79639 79602 0 72 0 0 0 - Z 1 0:00.27 0 79762 79602 0 20 0 17092 1396 wait I 1 0:00.00 /bin/sh /usr/local/bin/automake --add-missing --copy --force-missing 0 79768 79762 0 52 0 49736 13936 wait I 1 0:00.11 /usr/local/bin/perl -w /usr/local/bin/automake-1.15 --add-missing --copy --force-missing 0 79962 79768 0 20 0 12368 1024 vnread DL 1 0:00.00 cp /usr/local/share/automake-1.15/compile ./compile I am not sure if it's related to that OS version upgrade, but I have not seen any such issues on the same machine in 2-3 years running essentially the same build process with version 9.x, 10.0, 10.1 and 10.2. $ uname -a FreeBSD van01.sippysoft.com 10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #1 80de3e2(master)-dirty: Tue Feb 2 12:19:57 PST 2016 sobomax@abc.sippysoft.com:/usr/obj/usr/home/sobomax/projects/freebsd103/sys/ABC amd64 The kernel stack trace is: (kgdb) thread 360 [Switching to thread 360 (Thread 100515)]#0 0xffffffff8095244e in sched_switch () (kgdb) bt #0 0xffffffff8095244e in sched_switch () #1 0xffffffff809313b1 in mi_switch () #2 0xffffffff8097089a in sleepq_wait () #3 0xffffffff80930dd7 in _sleep () #4 0xffffffff809b230e in bwait () #5 0xffffffff80b511f3 in vnode_pager_generic_getpages () #6 0xffffffff80dd1607 in VOP_GETPAGES_APV () #7 0xffffffff80b4f59a in vnode_pager_getpages () #8 0xffffffff80b30031 in vm_fault_hold () #9 0xffffffff80b2f797 in vm_fault () #10 0xffffffff80cb5a75 in trap_pfault () #11 0xffffffff80cb51dd in trap () #12 0xffffffff80c9b122 in calltrap () #13 0xffffffff80cb36f1 in copyin () #14 0xffffffff80977ddf in uiomove_faultflag () The FS stack configuration is somewhat unique, so I am not sure if I am hitting some rare race condition or lock ordering issues specific to that. It's basically ZFS (ZRAID) on top of pair or SATA SSDs with big file on that FS attached via md(4) and UFS2 on that md(4). The build itself runs in chroot with that UFS2 fs as its primary root. Just maybe additional bit of info, attempting to list the directory with that UFS image also got my bash process stuck in "zfs" state, backtrace from that is: (kgdb) thread 353 [Switching to thread 353 (Thread 100508)]#0 0xffffffff8095244e in sched_switch () (kgdb) bt #0 0xffffffff8095244e in sched_switch () #1 0xffffffff809313b1 in mi_switch () #2 0xffffffff8097089a in sleepq_wait () #3 0xffffffff809069ad in sleeplk () #4 0xffffffff809060e0 in __lockmgr_args () #5 0xffffffff809b8b7c in vop_stdlock () #6 0xffffffff80dd0a3b in VOP_LOCK1_APV () #7 0xffffffff809d6d23 in _vn_lock () #8 0xffffffff81a8c9cd in ?? () #9 0x0000000000000000 in ?? ()