From owner-freebsd-fs@freebsd.org Sun Jul 5 21:01:29 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80E9998E26E for ; Sun, 5 Jul 2015 21:01:29 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 289D81DEB; Sun, 5 Jul 2015 21:01:29 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: by wgjx7 with SMTP id x7so124532510wgj.2; Sun, 05 Jul 2015 14:01:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=Xd8TEDa9/mDxNeuMEdQ53NbhkWd1H2+Nvn5qZZnSahM=; b=Qnnuc9zNdsL4ZjjgftntiazxwSWdCGYAl2mLn6kiHj2t6CS+GvTeEJuKZyNbkHAiTN v2O13IPweCauo6RL+TsnNuIEvin76Gmv83NdKPVUafho5r4mogEvi1Tib+2AIBUIp5n0 ERT0Yf+7dnLqqxeS26kMqcfBnWlgsQobinHo+j5KSzLXWpI854aCJAj7dwUQc5J+KiXZ p94tTnwqy+Gz52w1lO6INSn3B53kTI/HNEmeveSejWnS3BcOK+bsKsGCZNUnrr+Wffn/ 0a46PkXjZEQb/l7p29zxuMSBVjDAStRvGXSLSxAiTrgFkTe3PnYsPbua+olePnuGCvXh /G6A== X-Received: by 10.180.79.133 with SMTP id j5mr85788441wix.38.1436130086095; Sun, 05 Jul 2015 14:01:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.6.143 with HTTP; Sun, 5 Jul 2015 14:01:06 -0700 (PDT) In-Reply-To: <2010996878.3611963.1435884702063.JavaMail.zimbra@uoguelph.ca> References: <1491630362.2785531.1435799383802.JavaMail.zimbra@uoguelph.ca> <5594B008.10202@freebsd.org> <1022558302.2863702.1435838360534.JavaMail.zimbra@uoguelph.ca> <791936587.3443190.1435873993955.JavaMail.zimbra@uoguelph.ca> <2010996878.3611963.1435884702063.JavaMail.zimbra@uoguelph.ca> From: Ahmed Kamal Date: Sun, 5 Jul 2015 23:01:06 +0200 Message-ID: Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) To: Rick Macklem Cc: Julian Elischer , freebsd-fs@freebsd.org, Xin LI Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jul 2015 21:01:29 -0000 Hi folks, Just a quick update. I did not test Xin's patches yet .. What I did so far is to increase the tcp highwater tunable and increase nfsd threads to 60. Today (a working day) I noticed I only got one bad sequence error message! Check this: # grep 'bad sequence' messages* | awk '{print $1 $2}' | uniq -c 1 messages:Jul5 39 messages.1:Jun28 15 messages.1:Jun29 4 messages.1:Jun30 9 messages.1:Jul1 23 messages.1:Jul2 1 messages.1:Jul4 1 messages.2:Jun28 So there seems to be an improvement! Not sure if the Linux nfs4 client is able to somehow recover from those bad-sequence situations or not .. I did get some user complaints that running "ls -l" is sometimes slow and takes a couple of seconds to finish. One final question .. Do you folks think nfs4.1 is more reliable in general than nfs4 .. I've always only used nfs3 (I guess it can't work here with /home/* being separate zfs filesystems) .. So should I go through the pain of upgrading a few servers to RHEL-6 to try out nfs4.1 ? Basically do you expect the protocol to be more solid ? I know it's a fluffy question, just give me your thoughts. Thanks a lot! On Fri, Jul 3, 2015 at 2:51 AM, Rick Macklem wrote: > Ahmed Kamal wrote: > > PS: Today (after adjusting tcp.highwater) I didn't get any screaming > > reports from users about hung vnc sessions. So maybe just maybe, linux > > clients are able to somehow recover from this bad sequence messages. I > > could still see the bad sequence error message in logs though > > > > Why isn't the highwater tunable set to something better by default ? I > mean > > this server is certainly not under a high or unusual load (it's only 40 > PCs > > mounting from it) > > > > On Fri, Jul 3, 2015 at 1:15 AM, Ahmed Kamal < > email.ahmedkamal@googlemail.com > > > wrote: > > > > > Thanks all .. I understand now we're doing the "right thing" .. > Although > > > if mounting keeps wedging, I will have to solve it somehow! Either > using > > > Xin's patch .. or Upgrading RHEL to 6.x and using NFS4.1. > > > > > > Regarding Xin's patch, is it possible to build the patched nfsd code, > as a > > > kernel module ? I'm looking to minimize my delta to upstream. > > > > Yes, you can build the nfsd as a module. If your kernel config does not > include > "options NFSD" the module will get loaded/used. It is also possible to > replace > the module without rebooting, but you need to kill of the nfsd daemon then > kldunload nfsd.ko and replace nfsd.ko with the new one. (In > /boot/.) > > > > Also would adopting Xin's patch and hiding it behind a > > > kern.nfs.allow_linux_broken_client be an option (I'm probably not the > last > > > person on earth to hit this) ? > > > > If it fixes your problem, I think this is reasonable. > I'm also hoping that someone that works on the Linux client reports > if/when this > was changed. > > rick > > > > Thanks a lot for all the help! > > > > > > On Thu, Jul 2, 2015 at 11:53 PM, Rick Macklem > > > wrote: > > > > > >> Ahmed Kamal wrote: > > >> > Appreciating the fruitful discussion! Can someone please explain to > me, > > >> > what would happen in the current situation (linux client doing this > > >> > skip-by-1 thing, and freebsd not doing it) ? What is the effect of > that? > > >> Well, as you've seen, the Linux client doesn't function correctly > against > > >> the FreeBSD server (and probably others that don't support this > > >> "skip-by-1" > > >> case). > > >> > > >> > What do users see? Any chances of data loss? > > >> Hmm. Mostly it will cause Opens to fail, but I can't guess what the > Linux > > >> client behaviour is after receiving NFS4ERR_BAD_SEQID. You're the guy > > >> observing > > >> it. > > >> > > >> > > > >> > Also, I find it strange that netapp have acknowledged this is a bug > on > > >> > their side, which has been fixed since then! > > >> Yea, I think Netapp screwed up. For some reason their server allowed > this, > > >> then was fixed to not allow it and then someone decided that was > broken > > >> and > > >> reversed it. > > >> > > >> > I also find it strange that I'm the first to hit this :) Is no one > > >> running > > >> > nfs4 yet! > > >> > > > >> Well, it seems to be slowly catching on. I suspect that the Linux > client > > >> mounting a Netapp is the most common use of it. Since it appears that > they > > >> flip flopped w.r.t. who's bug this is, it has probably persisted. > > >> > > >> It may turn out that the Linux client has been fixed or it may turn > out > > >> that most servers allowed this "skip-by-1" even though David Noveck > (one > > >> of the main authors of the protocol) seems to agree with me that it > should > > >> not be allowed. > > >> > > >> It is possible that others have bumped into this, but it wasn't > isolated > > >> (I wouldn't have guessed it, so it was good you pointed to the RedHat > > >> discussion) > > >> and they worked around it by reverting to NFSv3 or similar. > > >> The protocol is rather complex in this area and changed completely for > > >> NFSv4.1, > > >> so many have also probably moved onto NFSv4.1 where this won't be an > > >> issue. > > >> (NFSv4.1 uses sessions to provide exactly once RPC semantics and > doesn't > > >> use > > >> these seqid fields.) > > >> > > >> This is all just mho, rick > > >> > > >> > On Thu, Jul 2, 2015 at 1:59 PM, Rick Macklem > > >> wrote: > > >> > > > >> > > Julian Elischer wrote: > > >> > > > On 7/2/15 9:09 AM, Rick Macklem wrote: > > >> > > > > I am going to post to nfsv4@ietf.org to see what they say. > Please > > >> > > > > let me know if Xin Li's patch resolves your problem, even > though I > > >> > > > > don't believe it is correct except for the UINT32_MAX case. > Good > > >> > > > > luck with it, rick > > >> > > > and please keep us all in the loop as to what they say! > > >> > > > > > >> > > > the general N+2 bit sounds like bullshit to me.. its always N+1 > in a > > >> > > > number field that has a > > >> > > > bit of slack at wrap time (probably due to some ambiguity in the > > >> > > > original spec). > > >> > > > > > >> > > Actually, since N is the lock op already done, N + 1 is the next > lock > > >> > > operation in order. Since lock ops need to be strictly ordered, > > >> allowing > > >> > > N + 2 (which means N + 2 would be done before N + 1) makes no > sense. > > >> > > > > >> > > I think the author of the RFC meant that N + 2 or greater fails, > but > > >> it > > >> > > was poorly worded. > > >> > > > > >> > > I will pass along whatever I get from nfsv4@ietf.org. (There is > an > > >> archive > > >> > > of it somewhere, but I can't remember where.;-) > > >> > > > > >> > > rick > > >> > > _______________________________________________ > > >> > > freebsd-fs@freebsd.org mailing list > > >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > >> > > To unsubscribe, send any mail to " > freebsd-fs-unsubscribe@freebsd.org" > > >> > > > > >> > > > >> > > > > > > > > >