From owner-freebsd-fs@FreeBSD.ORG Fri Jun 20 14:58:42 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E6AD2FFE for ; Fri, 20 Jun 2014 14:58:42 +0000 (UTC) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BA652986 for ; Fri, 20 Jun 2014 14:58:42 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id c11so2465025lbj.13 for ; Fri, 20 Jun 2014 07:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=3geeks.org; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/Ptdy09GgBtc72Uo7HgptmHO//Mmn+A5fsulKvtF3WE=; b=O0pEoYZlffKuUX0D7T2Y4v+LbDhCj/XuSbkDXXc7x1syPAzfx9mKrRW7nKERtBUyeD 74LtqUI9RFoB+4I9hb07eGNjfw0Z2wbP9+qy6LAosi/Gt5Jc5fnHNCoOtvUEHVaD74W8 4ISwNGbXoNG2fIv63/ItLxdsCSCV5atctFams= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=/Ptdy09GgBtc72Uo7HgptmHO//Mmn+A5fsulKvtF3WE=; b=JIdUVt0tvKhYdyl+zwVxaE/QH/X94gaHqIEGi23zhEshieLvGc1iCGV0D5QJ/WZT2e Dvvlp7HXH61h3MPeS4gFuU/3vwOKmeDhNt1AYZihbG/Jo6e5nq2SJw/br0PWM5EgqcKM kPHzPBjunmjlVy5oqcJLA2bbSLJB3264cuHBOcV6TRm30QxHTWmg0r/rh54+CSwtWlUF W1j3idPTU3Z1+xKSDoMzsr249nWvC+HtDj+Do3kGNIPhHYrGIP6YyWDYwmUQfGzKgSNr 36VRCkR+AYdDnA4osbs9+dZ232ZdEgX7JX1sSG0ctf5SgA3oL9eUj4ByjGzBM3bIESD7 NNSg== X-Gm-Message-State: ALoCoQniDiGsSM3Mw6f/yV4ElyQdclhz9yOznIfu9u/g+Bymj7u+DyAVIFE4TVj55rkc5xV32qvX MIME-Version: 1.0 X-Received: by 10.112.136.2 with SMTP id pw2mr2669950lbb.13.1403276319900; Fri, 20 Jun 2014 07:58:39 -0700 (PDT) Received: by 10.152.2.9 with HTTP; Fri, 20 Jun 2014 07:58:39 -0700 (PDT) X-Originating-IP: [64.236.208.26] In-Reply-To: <538359689.1860054.1403270164794.JavaMail.root@uoguelph.ca> References: <0016EC7C-7DCC-47B4-AD12-798525045F89@3geeks.org> <538359689.1860054.1403270164794.JavaMail.root@uoguelph.ca> Date: Fri, 20 Jun 2014 10:58:39 -0400 Message-ID: Subject: Re: Debugging newnfs From: Daniel Mayfield To: Rick Macklem Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2014 14:58:43 -0000 The server side is a set of vlans on a lagg of 4 igbs. The Xen side is the same setup, with the VMs in question attached to two different vlans. Many different mounts, but the mount options all look like this: nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=4048762,timeout=120,retrans=2 The permissions do not change, but repeat operations succeed and fail randomly. There aren't any clients concurrently accessing the same mount. On Fri, Jun 20, 2014 at 9:16 AM, Rick Macklem wrote: > Daniel Mayfield wrote: > > I have a very strange problem between an NFS server running FreeBSD > > 10 w/ ZFS and a number of FreeBSD 10 VMs running on a XenServer 6.2 > > SP1 host. The problem manifests as seemingly random permissions > > issues and/or IO errors on the clients when the ZFS pool is busy. > > There are no entries in dmesg on either side, and no errors logged > > in nfsstat either. If I keep the traffic down, the errors subside, > > but not completely. Other than tcpdump, how can I go about > > debugging this? > > > Well, you didn't mention what mount options you are using or what > network interfaces that you are using, but here's a few things that > might be worth looking at... > > The TSO max transmit segments issue: > - Without going into all the details (there have been some recent > commits like r264630 to try and alleviate this), if a net device > driver cannot handle 35 mbufs in a transmit TSO segment, things > will get broken. > - Xen/netfront is a weird exception, which I think is ok so long > as lagg or a vlan isn't layered on top of it. > --> If can disable TSO on both server and clients or reduce rsize,wsize > to 32K on all client mounts and see if the problem persists, that > is probably the best way to check this. (Since Xen/netfront is > such a weird case, I am not 100% sure if doing the above will fix > this problem, if it is being used) > > I also don't know if it is possible to have corrupted packets due to > a hardware problem (bad memory or...) where the Xen/netfront world > doesn't catch it. > > If you use the "soft" mount option, you could easily get this when > the server is slow to respond. I'd strongly recommend using "tcp" > and not "soft" for your mounts. ("nfsstat -m" on the client will > show you what the actual mount options is use are. This can be > somewhat different than what is specified on the command line, since > servers limit rsize/wsize, as an example.) > > When you get a "permissions failure" case, check on the server to > see if the permissions for the file appear correct on ZFS. If they > are (or the problem disappears when you retry a command without > changing permissions), you could have a caching issue. Other than > capturing the packets and looking at them in wireshark (which knows > NFS, unlike tcpdump) all you can do is try fiddling with the mount > options related to caching and see if that helps. (Note that NFS > does not have a cache coherency protocol, so if files are concurrently > shared among multiple clients, all bets are off w.r.t. what the > behaviour is. jhb@ is much better at this than I, since he seems > to find lots of these weird cases at his workplace.) > > Good luck with it, rick > > > Dan > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > >