From owner-freebsd-fs@freebsd.org Wed Mar 2 17:39:57 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C845CAC0F98 for ; Wed, 2 Mar 2016 17:39:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B11F319C7 for ; Wed, 2 Mar 2016 17:39:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B05EEAC0F97; Wed, 2 Mar 2016 17:39:57 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFF41AC0F96 for ; Wed, 2 Mar 2016 17:39:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3013A19C6; Wed, 2 Mar 2016 17:39:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u22Hdkvv029701 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 2 Mar 2016 19:39:47 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u22Hdkvv029701 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u22Hdk9S029700; Wed, 2 Mar 2016 19:39:46 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 2 Mar 2016 19:39:46 +0200 From: Konstantin Belousov To: Maxim Sobolev Cc: Kirk McKusick , peter@holm.cc, fs@freebsd.org Subject: Re: Process stuck in "vnread" Message-ID: <20160302173946.GL67250@kib.kiev.ua> References: <20160302095339.GB67250@kib.kiev.ua> <20160302115707.GF67250@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 17:39:58 -0000 On Wed, Mar 02, 2016 at 09:06:37AM -0800, Maxim Sobolev wrote: > Konstantin, this is nullfs mounted over UFS and nullfs is pointing over to > the part of the ZFS tree. I am not sure if it's what you are talking about > or not. > > > storage/builder on /builder (zfs, local, nfsv4acls) > > md0 vnode 3200M /builder/tmp/sspicd_tmp.ufs > > /dev/md0 on /builder/mnt (ufs, asynchronous, local, noatime) > /builder/usr/ports-bitbucket on /builder/mnt/usr/ports (nullfs, local) > > So, stuck process refers to file effectively being copied over from > /builder/mnt/usr/local/share/automake-1.15/compile to > /builder/usr/ports-bitbucket/SOMETHING/./compile by the process chrooted > into /builder/mnt, and it could be either in the read path or in the write > path. However looking at the full kernel side of stack trace of that cp(1), > I'd say it's probably the latter, as this would have to traverse through > top level vfs/ufs first, to nullfs layer and then via zfs, none of the last > two is compiled in so that there is no proper traceback. The nullfs mount > is used to allow it accessing ZFS tree on the upper level, i.e. > /builder/usr. > > Unfortunately I cannot find a way to figure out specific system call that > cp got stuck in. Attempting to attach gdb causes gdb to hang in turn. So > unless somebody got any other ideas on how to get some useful post-mortem > debug out of this situation I'll have to restart the box soon to recover it. > > I will put your patch in and see if it helps. I'd also compile nullfs > statically, so at least if it hits again we have some post-mortem evidence > to work with. No, my patch would not help you. Your problem is hung ZFS.