From owner-freebsd-fs@freebsd.org Thu Jul 2 01:09:46 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE25C9923C1 for ; Thu, 2 Jul 2015 01:09:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id A01651239 for ; Thu, 2 Jul 2015 01:09:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2B3AwA/jpRV/61jaINbg2ZfBoMZuhoJgWQKhS5KAoICFAEBAQEBAQGBCoQiAQEBAgEBAQEBIAQnIAsFCwIBCBgCAg0ZAgInAQkmAhMEARkBAgSIBggNtgeWXgEBAQEBBQEBAQEBAQEbgSGKKYQ0AQEBBBcBMweCaIFDBYcIhQ6BJIZWhF2ENoQIRINQgw2MDYNbAiaCCx2BbiIxAQaBBTqBAgEBAQ X-IronPort-AV: E=Sophos;i="5.15,389,1432612800"; d="scan'208";a="221510258" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 01 Jul 2015 21:09:45 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0C63A15F533; Wed, 1 Jul 2015 21:09:45 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 9mQrzFqhObDP; Wed, 1 Jul 2015 21:09:43 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id E5CC615F54D; Wed, 1 Jul 2015 21:09:43 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id URnLzD9XECl1; Wed, 1 Jul 2015 21:09:43 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id C891915F533; Wed, 1 Jul 2015 21:09:43 -0400 (EDT) Date: Wed, 1 Jul 2015 21:09:43 -0400 (EDT) From: Rick Macklem To: d@delphij.net Cc: Ahmed Kamal , freebsd-fs@freebsd.org Message-ID: <1491630362.2785531.1435799383802.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <55947C6E.5060409@delphij.net> References: <684628776.2772174.1435793776748.JavaMail.zimbra@uoguelph.ca> <55947C6E.5060409@delphij.net> Subject: Re: Linux NFSv4 clients are getting (bad sequence-id error!) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: Linux NFSv4 clients are getting (bad sequence-id error!) Thread-Index: 551uJ4KbtV+co3xU0l+6BLywRhu40w== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jul 2015 01:09:47 -0000 Xin Li wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 07/01/15 16:44, Ahmed Kamal via freebsd-fs wrote: > > The not so great news is, after updating sysctl and rebooting the > > nas box, I still saw a few (NFS: v4 server nas returned a bad > > sequence-id error!) lines in logs. Users have already left, so I > > don't know how bad is it .. > > > > Could you share more info on what this error means? RedHat seems to > > think the client can skip-by-1 and choose larger IDs and that would > > be totally fine ? Also how serious is this error, would it cause > > NFS session stall like that ? > I just looked at RFC-7530 (which replaced RFC-3530 for NFSv4.0) and here is the snippet w.r.t. this: When a request is received, its sequence number (r) is compared to that of the last one received (L). Only if it has the correct next sequence, normally L + 1, is the request processed beyond the point of seqid checking. Given a properly functioning client, the response to (r) must have been received before the last request (L) was sent. If a duplicate of last request (r == L) is received, the stored response is returned. If the sequence value received is any other value, it is rejected with the return of error NFS4ERR_BAD_SEQID. Sequence history is reinitialized whenever the SETCLIENTID/ SETCLIENTID_CONFIRM sequence changes the client verifier. It seems clear that "L + 2" is not supposed to be allowed. (The wording in RFC-3530 meant that L + 2 was the case of past the correct one, but it was poorly worded I'd say.) However, the updated RFC has changed one case. For RFC-3530, the correct value that follows UINT32_MAX is 0. However, for RFC-7530, the code is supposed to skip over 0 and use 1. (I think that means that the server must accept both 0 and 1 as the correct value following UINT32_MAX.) It seems that is why the above states "normally L + 1". I am going to post to nfsv4@ietf.org to see what they say. Please let me know if Xin Li's patch resolves your problem, even though I don't believe it is correct except for the UINT32_MAX case. Good luck with it, rick > I wonder if this would help, which loosen the check: > > Index: sys/fs/nfsserver/nfs_nfsdstate.c > =================================================================== > - --- sys/fs/nfsserver/nfs_nfsdstate.c (revision 285016) > +++ sys/fs/nfsserver/nfs_nfsdstate.c (working copy) > @@ -3805,7 +3805,8 @@ nfsrv_checkseqid(struct nfsrv_descript *nd, u_int3 > printf("refcnt=%d\n", stp->ls_op->rc_refcnt); > panic("nfsrvstate op refcnt"); > } > - - if ((stp->ls_seq + 1) == seqid) { > + if ((stp->ls_seq + 1) == seqid || > + (stp->ls_seq + 2) == seqid) { > if (stp->ls_op) > nfsrvd_derefcache(stp->ls_op); > stp->ls_op = op; > > > Personally I don't quite buy the skip-by-1 is Okay argument but it > seems that the RFC text can be interpreted that way. > > Cheers, > > > > On Thu, Jul 2, 2015 at 1:36 AM, Rick Macklem > > wrote: > > > >> Ahmed Kamal wrote: > >>> Hi all, > >>> > >>> I'm a refugee from linux land. I just set up my first freebsd > >>> 10.1 zfs > >> box, > >>> sharing /home over nfs. Since every home directory is its own > >>> zfs > >> dataset, > >>> I chose to use nfsv4 to enable recursively sharing/mounting any > >>> directory under /home (I understand nfs4 is a must in this > >>> scenario!) > >>> > >>> I'm able to mount form linux (rhel5 latest kernel) > >>> successfully. Users > >> are > >>> working fine. However every now and then a user screams that > >>> his session > >> is > >>> frozen. Usually the processes are stuck in nfs_wait or rpc_* > >>> state. I > >> tried > >>> using a much newer linux kernel (3.2 however it still faced the > >>> same problem). The errors in Linux log files are mostly: Jul 1 > >>> 17:41:47 mammoth kernel: NFS: v4 server nas returned a *bad > >>> sequence-id error*! Jul 1 17:52:32 mammoth kernel: > >>> nfs4_reclaim_locks: unhandled error -11. Zeroing state Jul 1 > >>> 17:52:32 mammoth kernel: nfs4_reclaim_open_state: Lock reclaim > >>> failed! > >>> > >> Btw, a client should only do "reclaim" operations after the > >> server has replied with NFS4ERR_STALE_CLIENTID or > >> NFS4ERR_STALE_STATEID. I am pretty certain that the FreeBSD NFSv4 > >> server only generates these replies after it has rebooted, so > >> assuming the server didn't reboot, I have no idea why the client > >> would attempt these and am not surprised they failed. > >> > >> I'm guessing that the DRC constipation somehow caused the Linux > >> client to go into recovery mode? > >> > >> rick > >> > >>> My search led me to > >>> (https://access.redhat.com/solutions/1328073) a detailed > >>> analysis of the issue, which you can read over here > >>> https://dl.dropboxusercontent.com/u/51939288/nfs4-bad-seq.pdf > >>> .. NetApp confirmed this was a bug for them (I'm wondering if > >>> this is still in FreeBSD?!) > >>> > >>> PS: Right before sending this, I saw dmesg on the freebsd box > >>> advising increasing vfs.nfsd.tcphighwater .. So I up'ed that to > >>> 64000. I also > >> up'ed > >>> the number of nfs server threads (-t) from 10 to 60 (we're > >>> roughly 40 > >> linux > >>> machines) > >>> > >>> Any advice is most appreciated! > >>> > >>> Thanks _______________________________________________ > >>> freebsd-fs@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs To > >>> unsubscribe, send any mail to > >>> "freebsd-fs-unsubscribe@freebsd.org" > >>> > >> > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs To > > unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > > > - -- > Xin LI https://www.delphij.net/ > FreeBSD - The Power to Serve! Live free or die > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.1.5 (FreeBSD) > > iQIcBAEBCgAGBQJVlHxuAAoJEJW2GBstM+nskvsP/ire8QyTfL6mF1njMNZwI/k5 > AQ+BwWs5r8LzcRN/4v7/gelbS+lXnYVbVHMl8q6j+HzUzQ3yId4ZGlJWpJtHDNnj > +gV8kmFt/og1QTrQRbN81i4GEr914SlKWmo7LsxrWmEhAiKsN0sYsjELD/mH5BZX > 1wRe3vTvyrMwm+6u1krqT8ZrxRANBFBmNqiFb8sag7B3oJQZsGhAyUSsJvUhb00o > ozwC2NT5y8Jv0QcZdC/wGeYc8FmRNQTAjE22WkzbsUey/e7FxL7vflCGgngYCIxE > zbZNW65xThZO8fti5MxiepJ27VPa5ocX0CQihBFYp5haG6fzWBGalV/ggAOwYL44 > nz1caLhdKIj9JSd8QwLdTArq8+6H8Sx4jp4iGzQnppNo8PtG/AlHlw9uDKaUF4iw > H+tMb6qMu2FQJ9X+phtplzvjZxCbBbwY205GeTm5eElOkYzIyYvqIvZasos02ze0 > v3SQXtpIHjrnndXMVNRJOkhYquGxVFxUm5IJ7o+0wrgVJp1V3cBKd4vs0o84Mgu5 > EPGKCyt8x/B6ujCxkunODpNOb+sFyq6aqsDLAO6JSih5HfQntpxoZTjm8p4KjsG6 > nPqXQXmi2NoOd6WPOunp7w/y+fKA4YdLAhPC7rbXQwpLL81UqNH141BrtscN0ovi > pyRlJ4r3Zs75qUwVSkzL > =3/OG > -----END PGP SIGNATURE----- >