From owner-freebsd-questions@freebsd.org Tue Jul 28 15:32:10 2015 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DECB9AC8EC for ; Tue, 28 Jul 2015 15:32:10 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5379EA5 for ; Tue, 28 Jul 2015 15:32:09 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by wibxm9 with SMTP id xm9so166390362wib.1 for ; Tue, 28 Jul 2015 08:32:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type; bh=UtMlR2DgLeidugjSmrooj1mYvf+UOmRMFgzQXzf/ZtE=; b=U0lA6KSGWSPquo2ViQXT73ZGldvM7PuqIrFScy87oOWBepqBSph8C1/LdciwYv01RX vt/Za7wj2Eip1ZO68Wdc+vXJ8VeXogXLfJVYb2K1o9Wu2AOzn5sEr6dZVV6kg88+RESy 2y8Psa3iZRINV5C8v8twyPzlohmVyUnMRQBitiCntrVfkmCCnSLVdsFJaD6SbdWh7MeZ cmxON7vAOg3wd9HNLBluvMCz2OGCYAe5IZ/9kttmRxUsW9YCggE1IvzrO90+lU1zXqKp bHiO0cZymvjFfciTYre0Wga0dnZhkhXBZdpAjtIF6finbw1Ekelk5GTZiNzsmz7jcpCV 4B4Q== X-Received: by 10.194.93.3 with SMTP id cq3mr66474358wjb.20.1438097528089; Tue, 28 Jul 2015 08:32:08 -0700 (PDT) Received: from x220.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.googlemail.com with ESMTPSA id gj6sm19560246wib.22.2015.07.28.08.32.06 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Jul 2015 08:32:07 -0700 (PDT) Subject: Re: NFS Server keeps timing out on a client? To: freebsd-questions References: <55B29A4F.4040506@gmail.com> From: Kaya Saman Message-ID: <55B7A076.7040107@gmail.com> Date: Tue, 28 Jul 2015 16:32:06 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55B29A4F.4040506@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jul 2015 15:32:10 -0000 It looks like this is restricted to one particular client for some reason? The server is fine as have tested with other machines and all fine, so either the client doesn't have enough power to handle the I/O on a GbE connection - as another lesser powered client is working fine on 100Mb/s, or there is something wrong with the client itself? Anyway, continuing to troubleshoot!! :-) --K On 07/24/2015 09:04 PM, Kaya Saman wrote: > Hi, > > this is a really difficult one to diagnose :-( > > The situation is this: > > I have a server which I'm trying to serve up a home directory for. So > when a user logs in they will be taken to /mnt/home/ NFS mount. > > The home directory is a zfs filesystem stored on a ZPOOL. > > The server runs FreeBSD 10.1 64bit and is powered by a quad core Intel > Xeon with 8GB RAM and is connected to the network using a 4GbE lagg trunk. > > > The client side is Arch Linux, 64 bit connected via gigabit link. > > > They are on different vlans, and the network uses Jumbo MTU's however, > I have also tested with regular mtu size of 1500 and same thing seems > to be happening. > > > As the firewall/router also runs *BSD I tried to see what was > happening using: tcpdump -eni (vlan) host > > > During the timeout periods I keep seeing this message: > > 2042: truncated-ip - 4 bytes missing!192.168.20.30.981 > 192.168.1.80.2049: xid 0x3b14188c (NFSv3) 1976 write [|nfs] (DF)2042: truncated-ip - 4 bytes missing!192.168.20.30.981 > 192.168.1.80.2049: xid 0x3b14188c (NFSv3) 1976 write [|nfs] (DF) > > 2042: truncated-ip - 4 bytes missing!.981 > > .2049: xid 0x3b14188c (NFSv3) 1976 write [|nfs] (DF) > > > Unfortunately setting the "debug" flag in mountd on FreeBSD doesn't > show any thing other then "mount ok". > > > On the client side I have put NFS into debug mode and am seeing this > through dmesg: > > [24012.265800] NFS: nfs_update_inode(0:41/1653343 fh_crc=0x9257f59a > ct=1 info=0x27e7f) > [24012.266207] NFS: nfs_update_inode(0:41/1653343 fh_crc=0x9257f59a > ct=1 info=0x27e7f) > [24012.266215] NFS: permission(0:41/1653343), mask=0x26, res=0 > [24012.284651] NFS: nfs_update_inode(0:41/1653339 fh_crc=0x998992f7 > ct=1 info=0x7feff) > [24012.285181] NFS: nfs_update_inode(0:41/1653339 fh_crc=0x998992f7 > ct=1 info=0x7feff) > [24012.285251] NFS: > dentry_delete(entries/ADDBB8F380DD96E0D9B49F9AF600706F6A1431BA, 4808cc) > [24012.294851] NFS: nfs_update_inode(0:41/1653196 fh_crc=0x6513c0a6 > ct=1 info=0x7feff) > [24012.295399] NFS: nfs_update_inode(0:41/1653196 fh_crc=0x6513c0a6 > ct=1 info=0x7feff) > [24012.295510] NFS: > dentry_delete(entries/9AEB965A3D7D6F96E61BD8427D8296215AE6CE13, 4808cc) > [24013.731223] NFS: permission(0:41/4), mask=0x1, res=0 > [24013.731237] NFS: permission(0:41/1649925), mask=0x1, res=0 > [24013.731244] NFS: permission(0:41/1649944), mask=0x1, res=0 > [24013.731251] NFS: dentry_delete(.config/menus, 808cc) > > > Has anyone experienced this type of behavior before, and possibly help > guide me on how to debug it to get to the root of the problem? > > > Many thanks! > > > Regards, > > > Kaya > >