From owner-freebsd-stable@freebsd.org Fri Aug 5 15:03:41 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6037BAF9E5 for ; Fri, 5 Aug 2016 15:03:41 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ABC3211F0; Fri, 5 Aug 2016 15:03:41 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi0-x22e.google.com with SMTP id j185so366589497oih.0; Fri, 05 Aug 2016 08:03:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=WlTq8zSzVs6tOz/UKzk/8I4NrWQZAJo8YzVr933W/JM=; b=efhnuvt4Sz5Y4ZcnnznBreezOAHvP+FelDX6Vfio2ilBw1K+/fwQ0YidwAv//iXthH Azqxt8hT1joO4bHZX2dStwIM60KPQTNEgIGmKjjvLeQ8hPqMETplW53r2YQ3q/HFBEa7 biOWA6iz5Z6aSInx/xxY2q7YhiIAMP8oghPQT0JJpFmQ0JZNtJyHtG4NfqhiNg1zBIwm WcB1DUSM5PrNA1br4+Ys3JK6dzuoUZeRL4yztu7FUk1eACCM1otnERvbFaxMqDPGxuxp 4Z2SkkUBwvY97p5fSRaDMfapB2ICtbylqLg7Y2YyTwMs0YlTVQGrw7kBbbMOWwoek/7B dDww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=WlTq8zSzVs6tOz/UKzk/8I4NrWQZAJo8YzVr933W/JM=; b=VRpmqrlYdK7OwQfxK+fij0sLbo0Egf6vtntdiMHAzBHn8srjJHHVi4ACHLkXkLGCgU WSC12TOuAfaGhC5iB2hXXOzAxX7Uwi8VZxmI7J7eiv8r+cdWvU+TcNI/FIt15De6aOie RTn0eb+o9MxNS0+u/GY8XGHzHrZDVPsuwwYU4DyBGtkIq1hds4SkpZnOl3gLkEEkAL4n tSOUo6e6tsvH2lWZ2E3gnanbAUo3EoJozCFGBxy9cOh1z8zoxzCY5hTqGfdKvcShJKna gtTU4thlvQo7Jf3Ut3Aexsb27iFA6+RVBZt2fnnTHBg5mRGtIlxK4RFMyo8R/2xLQC5d TXnw== X-Gm-Message-State: AEkoouv6umMNNSzfGnUVtzBQFB6G0Zv699gnvOHULonhlcv+3BEeOcbWTQfSM7DH8TkBCm/fOjHYF41Uq2y3Hg== X-Received: by 10.202.65.213 with SMTP id o204mr447792oia.105.1470409420870; Fri, 05 Aug 2016 08:03:40 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.202.196.149 with HTTP; Fri, 5 Aug 2016 08:03:40 -0700 (PDT) In-Reply-To: <1eb419ed-4180-11c2-3bf3-5d3013a07197@gmx.de> References: <1df33129-0c3e-dfc3-6867-46ab0473ae57@gmx.de> <1eb419ed-4180-11c2-3bf3-5d3013a07197@gmx.de> From: Alan Somers Date: Fri, 5 Aug 2016 09:03:40 -0600 X-Google-Sender-Auth: FkHyF02VNQ1Jmri8TYKYu1-9RWw Message-ID: Subject: Re: zfs recv causes nfs server to throw NFSERR_IO i/o errors To: Daniel Genis Cc: Alan Somers , FreeBSD Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Aug 2016 15:03:42 -0000 Glad to help. BTW, that setting is the default in FreeBSD 11. On Fri, Aug 5, 2016 at 8:54 AM, Daniel Genis wrote: > Thank you, that probably is what the doctor ordered! > My quick testing shows that it's very likely fixed. Kudos! :-) > > On 08/05/2016 04:11 PM, Alan Somers wrote: >> On Fri, Aug 5, 2016 at 7:22 AM, Daniel Genis wrote: >>> Hi everyone, >>> >>> we've been tracing an issue where snapshot replication is causing >>> interruptions for the NFS serivce. >>> >>> The problem is as follows: >>> >>> Every time a zfs recv finishes, there is a chance for the NFS server to >>> return an NFSERR_IO for a GETATTR call. This shows up as input/output >>> errors on the nfs clients. >>> >>> Here the tcpdump showing the NFS conversation: >>> https://nopaste.me/view/95d1a79d >>> >>> NFS 202 V3 GETATTR Call (Reply In 6043), FH: 0x8c711a60 >>> NFS 98 V3 GETATTR Reply (Call In 6042) Error: NFS3ERR_IO >>> NFS 222 V3 LOOKUP Call (Reply In 6046), DH: 0x6694634f/example.file.txt >>> NFS 102 V3 LOOKUP Reply (Call In 6045) Error: NFS3ERR_ACCES >>> >>> We've been able to verify that there is a _direct_ correlation between >>> the zfs recv command and these NFS errors. For every input/output error >>> we can find a log entry of a replication just finishing (zfs recv exiting). >>> >>> The receiving server is running 10.3-RELEASE >>> >>> I've read about a VFS/ZFS deadlock issue which is to be included/fixed >>> in Freebsd 11.0-BETA4. >>> >>> Could our issue be related? >>> Otherwise does anyone have any suggestions how to tackle this issue? >>> >>> >>> For the record, say we have two volumes: >>> tank/volumeA and tank/volumeB >>> >>> If there is a zfs recv busy for tank/volumeA then tank/volumeB can get >>> these NFS "io" errors, it does not have to be the same volume. >>> >>> >>> Has anyone else seen/experience this as well? >>> >>> Any insights are appreciated! >>> >>> With kind regards, >>> >>> Daniel >> Try adding mountd_flags="-S" to /etc/rc.conf. > >