From owner-freebsd-fs@freebsd.org Wed Apr 4 20:29:03 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DC02F73EFF for ; Wed, 4 Apr 2018 20:29:03 +0000 (UTC) (envelope-from rcarter@pinyon.org) Received: from h2.pinyon.org (h2.pinyon.org [65.101.20.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 831D9720ED for ; Wed, 4 Apr 2018 20:29:02 +0000 (UTC) (envelope-from rcarter@pinyon.org) Received: by h2.pinyon.org (Postfix, from userid 58) id 179CF3275B; Wed, 4 Apr 2018 13:20:36 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=pinyon.org; s=DKIM; t=1522873236; bh=brbQdYKz0oioCUFsw9oUmxj08Ef2WgOfwe2N/Tvmczw=; h=Subject:To:References:From:Cc:Date:In-Reply-To; b=mbHal0O34kzVVeGniyJkBbcJoM1J7V6myWV8VizPMz+6TSIku4obGjCQsZMvA9sSQ 6A6Otay0R+modQKsCSTJESpuzX8/HeTQGD4NzkobUSqFGxd1m7dxDsL4a7vXl2Cu/i G4Ac6qZYdTewzRsiBwDTnCKuhdSoJ/qQMJ+gfEBc= X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on h2.n1.pinyon.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from [10.0.10.15] (h1.pinyon.org [65.101.20.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by h2.pinyon.org (Postfix) with ESMTPSA id 0BF753273E; Wed, 4 Apr 2018 13:20:34 -0700 (MST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=pinyon.org; s=DKIM; t=1522873234; bh=brbQdYKz0oioCUFsw9oUmxj08Ef2WgOfwe2N/Tvmczw=; h=Subject:To:References:From:Cc:Date:In-Reply-To; b=JF1DH6G5/PyKuTzw+oNw43J4cJp9J8mW1h5EAgiqrKTjsIWN/p1tzprBWq4j8PTek mkexcjWrZ85nVIcYmWGydtjri1b18TUgwoRpVqAy2qBncNtqKGW9cKYdT86sHEUKbQ Hgxm9Q8Sy7WjSlO8y0J2htFQH80s7sE2joumLWRY= Subject: Re: Linux NFS client and FreeBSD server strangeness To: Mike Tancsa References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> From: "Russell L. Carter" Cc: "freebsd-fs@freebsd.org" Message-ID: <6ca61caa-f4a9-68e4-32d9-6142e175e09b@pinyon.org> Date: Wed, 4 Apr 2018 13:20:33 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2018 20:29:03 -0000 On 04/04/18 11:27, Mike Tancsa wrote: > Not sure where the tweaking needs to happen, but I am getting strange > behaviour between a Linux nfs client and FreeBSD RELENG_11 NFS server. I initially had terrible performance with a Linux client writing to a FreeBSD server several years back, and after a lot of false starts this snippet in /etc/sysctl.conf solved it: # NFS client sux0rs w/o: net.inet.tcp.tso=0 A caveat is I use NFS V4.1, though. Good luck, Russell > The FreeBSD server starts with > > > nfs_client_enable="YES" > nfs_server_enable="YES" > > > rpcbind_enable="YES" > rpc_lockd_enable="YES" > rpc_statd_enable="YES" > nfs_server_flags="-u -t -n 16" > > and on the Linux client I have been trying various options to no avail. > The mount works, but on a straight up write to the FreeBSD server, > everything is very bursty. I noticed this (I think) a few months ago > where Linux dumps across an nfs mount seemed to take a lot longer and > were getting very bursty. > > It seems if there are a mixture of reads and writes, everything is > pretty fast. But if a client is just writing to the server, something, > somewhere is blocking. Doing something simple like > ls -l /nfsmount > from the client "wakes" up the server/client so that write stream can > keep going. Otherwise, it will do a big blast of writes and then several > seconds of pausing on the dump. > > Linux Dump is a simple > > /sbin/dump u -0 -f - / | /bin/bzip2 >/backup/dump-root-0.bz2 > > Mount is > > mount.nfs -o tcp,intr,noatime,vers=3 192.168.yy.xx:/path > > If I run ifstat on the FreeBSD nfs server, the traffic pattern looks like > > # ifstat -b -i cxl0 > cxl0 > Kbps in Kbps out > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 8.12e+06 45127.03 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 0.00 0.00 > 6.04e+06 33525.76 > 901122.1 4983.72 > 0.00 0.00 > > if I do a bunch of ls -l /nfsmount on the client > > eg > > while true > do > ls -l /backup/ > /dev/null > done > > traffic pattern is > > > cxl0 > Kbps in Kbps out > 0.00 0.00 > 3.31e+06 18520.03 > 5.89e+06 32571.52 > 4.84e+06 28325.71 > 2.12e+06 19466.56 > 614727.0 12246.10 > 874927.6 13557.18 > 1.06e+06 14386.78 > 917865.4 13696.87 > 1.09e+06 14608.64 > 1.06e+06 14376.12 > 164077.3 5286.64 > > > Leading up to the stall, pcap snippet attached. > > Note, doing something like > > dd if=/dev/zero of=/backup/test.bin bs=4096 count=5000000 > > I can saturate the 10G link and max out the disk on the server > > # dd if=/dev/zero of=/backup/test.bin bs=4096 count=5000000 > 5000000+0 records in > 5000000+0 records out > 20480000000 bytes (20 GB, 19 GiB) copied, 36.6238 s, 559 MB/s > > and its a pretty steady stream unlike the dump. Any ideas whats going > on and how I might be able to work around this ? > > > 192.168.xx.yy:/zbackup1/virtbox4b/backup on /backup type nfs > (rw,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.242.254,mountvers=3,mountport=774,mountproto=tcp,local_lock=none,addr=192.168.yy.xx) > > > > ---Mike > > > ------------------- > Mike Tancsa, tel +1 519 651 3400 x203 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >