From owner-freebsd-stable@freebsd.org Mon Mar 28 16:23:24 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E8A8AE0CF5 for ; Mon, 28 Mar 2016 16:23:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 870121D7E for ; Mon, 28 Mar 2016 16:23:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 86598AE0CF3; Mon, 28 Mar 2016 16:23:24 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85DE2AE0CF2; Mon, 28 Mar 2016 16:23:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0436A1D7D; Mon, 28 Mar 2016 16:23:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2SGNAoD056823 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 28 Mar 2016 19:23:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2SGNAoD056823 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2SGNAuY056822; Mon, 28 Mar 2016 19:23:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 28 Mar 2016 19:23:10 +0300 From: Konstantin Belousov To: Maxim Sobolev Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Pawel Jakub Dawidek , Kirk McKusick , kib@freebsd.org Subject: Re: Process stuck in "vnread" Message-ID: <20160328162310.GJ1741@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Mar 2016 16:23:24 -0000 On Mon, Mar 28, 2016 at 08:52:03AM -0700, Maxim Sobolev wrote: > Done some head scratching, it looks like it's got page fault in the > copyin() (cp(1) AFAIK mmaps source file). There might be some interlock > issue between competing write to the same ZFS, the md0 device is locked > forever waiting for the write operation to complete at the very same time. > I am curious as to whether we are allowed to sleep in the dmu_write_uio_dbuf(), > AFAIK dmu is ZFS's transaction layer, so maybe copyin() should be done > earlier to avoid possible page fault in there? No idea about ZFS, but if the issue is due to copyin(9) recursing into VM and then VFS while owning file system locks, it is well-known and long-standing issue. I sometimes call it 'ups deadlock', for some reasons, see tools/test/upsdl/ for the distilled test case. It is handled for UFS and NFS, read the long comment starting with 'The vn_io_fault() is a wrapper' in sys/kern/vfs_vnops.c, which describes the deadlock in details and explains the mechanism which is used to prevent it. Filesystems must opt-in into it by specifiying MNTK_NO_IOPF flag, and then being ready to get an array of pages for io instead of the buffer KVA.