From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 21 23:05:30 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 707C1106566C for ; Mon, 21 Dec 2009 23:05:30 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 2C8888FC0A for ; Mon, 21 Dec 2009 23:05:29 +0000 (UTC) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.2/8.14.1) with ESMTP id nBLN5T7u022862; Mon, 21 Dec 2009 15:05:29 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.14.2/8.13.4/Submit) id nBLN5S9I022861; Mon, 21 Dec 2009 15:05:28 -0800 (PST) Date: Mon, 21 Dec 2009 15:05:28 -0800 (PST) From: Matthew Dillon Message-Id: <200912212305.nBLN5S9I022861@apollo.backplane.com> To: Zaphod Beeblebrox References: <5f67a8c40912182147t1adc158ew9fd3d94c4c4c955f@mail.gmail.com> <200912212042.nBLKga0G021550@apollo.backplane.com> <5f67a8c40912211306k657b9a2aj7c037d3f08b8ab21@mail.gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: scp more perfectly fills the pipe than NFS/TCP X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Dec 2009 23:05:30 -0000 Oh, one more thing... I'm assuming you haven't used tcpdump with NFS much. tcpdump has issues parsing the NFS RPC's out of a TCP stream. For the purposes of testing you may want to temporarily use a UDP NFS mount. tcpdump can parse the NFS RPCs out of the UDP stream far more easily. If you use a UDP mount use the dumbtimer option and set it to something big, like 10 seconds, so you don't get caught up in NFS/UDP's retry code (which will confuse your parsing of the output). A typical tcpdump line would be something like this: tcpdump -n -i nfe0 -l -s 4096 -vvv not port 2049 Where the port is whatever port the NFS RPC's are running over while you are running the test. You'd want to display it on a max-width xterm, or record a bunch of it to a file and then review it via less. The purpose of running the tcpdump is to validate all your assumptions as well as determine whether basic features such as read-ahead are actually running. You can also determine if packet loss is occuring, if requests are being sent or responded to out of order (the RPC tcpdump parses includes the request id's and the file offsets so it should be easy to figure that out). You can also determine the actual latency by looking at the timestamps for the request vs the reply. Once you've figured out as much as you can from that you can try tcpdumping the TCP stream. In this case you may not be able to pick out RPCs but you should be able to determine whether the requests are being pipelined and whether any packet loss is occurring or not. You can also determine whether the TCP link is working properly... i.e. that the TCP packets are properly flagging the 'P'ushes and not delaying the responses, and that the link isn't blowing out its socket buffer or TCP window (those are two separate things). The kernel should be scaling things properly but you never know. -Matt