From owner-freebsd-net@FreeBSD.ORG Thu Jan 23 05:04:11 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D994578 for ; Thu, 23 Jan 2014 05:04:11 +0000 (UTC) Received: from mail-ig0-x22f.google.com (mail-ig0-x22f.google.com [IPv6:2607:f8b0:4001:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0B29F1DC8 for ; Thu, 23 Jan 2014 05:04:11 +0000 (UTC) Received: by mail-ig0-f175.google.com with SMTP id uq10so15962629igb.2 for ; Wed, 22 Jan 2014 21:04:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=JqSSVCva9wq8BBEN7o1M9gjsLMRHpg4AgfW1p9Si4Zk=; b=f80P/oLHtI1fWOxIHpq8yP/XAeuwiqIEeHu7IHM86XRkcAVhPY7dnvvCOPoDezhsc7 VKYA2PwZCN65zUdJGmRizvTddepmFITbCPt3iL0Qkxb9HwRiEzwWu54pEuzXZjTTH4n/ Tyl0FHR2Czrd2QqHepytxV69xv5u7KUKizGabq3E6OM/KhGrOQy8y+qSZ6a4R4u9iPx0 uj1oiPibHePI+kNr+lxEkWdNOOlab/9FB/Ig8qNAxVS7SyqPkgF5DoU0R9t/AuyiPq0Y VEy8Avd4a5KSawY8e1inTSmiFDqb0Jb1R5c29GN4PNY9adnllFfAdEuDBz22CnKjfurL I4zQ== MIME-Version: 1.0 X-Received: by 10.43.49.1 with SMTP id uy1mr4350440icb.48.1390453450441; Wed, 22 Jan 2014 21:04:10 -0800 (PST) Sender: jdavidlists@gmail.com Received: by 10.42.170.8 with HTTP; Wed, 22 Jan 2014 21:04:10 -0800 (PST) In-Reply-To: <1891524918.14888294.1390450374695.JavaMail.root@uoguelph.ca> References: <1891524918.14888294.1390450374695.JavaMail.root@uoguelph.ca> Date: Thu, 23 Jan 2014 00:04:10 -0500 X-Google-Sender-Auth: aUsw749YtA2pZlL8NYaStWz9fCI Message-ID: Subject: Re: Terrible NFS performance under 9.2-RELEASE? From: J David To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jan 2014 05:04:11 -0000 On Wed, Jan 22, 2014 at 11:12 PM, Rick Macklem wrote: > So, do you consider the 32K results as reasonable or terrible performance? > (They are obviously much better than 64K, except for the reread case.) It's the 64k numbers that prompted the "terrible" thread title. A 196K/sec write speed on a 10+ Gbit network is pretty disastrous. The 32k numbers are, as you say, better. Possibly reasonable, but I'm not sure if they're optimal. It's hard to tell the latency of the virtual network, which would be needed to make that determination. It would be best if FreeBSD out of the box blows the doors of Debian out of the box and FreeBSD tuned to the gills blows the doors off of Debian tuned to the gills. Right now, Debian seems to be the one with the edge and with FreeBSD's illustrious history as the NFS performance king for so many years, that just won't do. :) > Btw, I don't think you've mentioned what network device driver gets used > for this virtual environment. That might be useful, in case the maintainer > of that driver is aware of some issue/patch. KVM uses virtio. >> 00:38:07.932732 IP (tos 0x0, ttl 64, id 38912, offset 0, flags [DF], >> proto TCP (6), length 53628) >> > I don't know why this would be so large. A 32K write should be under > 33Kbytes in size, not 53Kbytes. I suspect tcpdump is confused? Since TCP is stream oriented, is there a reason to expect 1:1 correlation between NFS writes and TCP packets? > Well, it seems Debian is doing 4096 byte writes, which won't have anywhere > near the effect on the network driver/virtual hardware that a 64K (about > 45 IP datagrams) in one NFS RPC will. Debian's kernel says it is doing 64k reads/writes on that mount. So again, possibly an expectation of 1:1 correlation between NFS writes and TCP packets is not being satisfied. However, iozone is doing 4k reads/writes for these tests, so it's also possible that Debian is not coalescing them at all (which FreeBSD apparently is) and the 4k writes are hitting the virtual wire as-is. Also, both sides have TSO and LRO, so it would be surprising (and incorrect?) behavior if a 64k packet were actually fragmented into 45 IP datagrams. Although if something is happening to temporarily provoke exactly that behavior, it might explain the 1500 byte packets, so that's definitely a lead. Maybe it would be possible for me to peek at the stream from various different points and establish who is doing the fragmenting. It could be that if Debian is basically disregarding the 64k setting and using only 4k packets, it's simply not hitting whatever large-packet bad behavior that is harming FreeBSD. However it also performs better in the server role, with the client requesting the larger packets. So that's not definitive. > Yea, looking at this case in wireshark might make what is going on > apparent. Possibly, but that would likely have to be done by someone with more NFS protocol familiarity than I. Also, the incorrect checksums on outbound packets are normal because the interface supports checksum offloading. The checksum simply hasn't been calculated yet when tcpdump sees it. Thanks!