From owner-freebsd-net@FreeBSD.ORG Fri Jan 31 17:58:16 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C69861A1; Fri, 31 Jan 2014 17:58:16 +0000 (UTC) Received: from mail-ie0-x232.google.com (mail-ie0-x232.google.com [IPv6:2607:f8b0:4001:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 895181945; Fri, 31 Jan 2014 17:58:16 +0000 (UTC) Received: by mail-ie0-f178.google.com with SMTP id x13so4720460ief.9 for ; Fri, 31 Jan 2014 09:58:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=OuiCkJqdf7ZDhRD3EbTuIQDZsZXdHn1CR3hukVW+0tI=; b=ZFlmxfNWy3UAL1DvI9ja4/X7Fc1SRLbmhUVRi0GVQqcnoR/buj/CbfEGN7I2VbTTqm 6QHg8ni0bHYDadbkM5mftBxLqMEfcHg4ntNlGaaTTxv0Ap3fqaKfrGwnVAWBezgIGCSS wRmHKYvt44rQ6BYjNlxf5S/9x9EqpMwVu3FXUbF+EVVQENQ2otSjV2ARqCucaHHq9CTG 86QfqKd7JvTQOADpTmBRfXEH7MGRtV93I5JUoA1v/x5kQb/R8d+6keWOEZ9KLebb0akP Cb6RDbQC98e37hyk8Xsi31eTaEEtMjc4O1bB94wdBqjJjfYWPZlHDZwowCf07g8efrDG UdBQ== MIME-Version: 1.0 X-Received: by 10.43.82.69 with SMTP id ab5mr992946icc.95.1391191096052; Fri, 31 Jan 2014 09:58:16 -0800 (PST) Sender: jdavidlists@gmail.com Received: by 10.42.170.8 with HTTP; Fri, 31 Jan 2014 09:58:15 -0800 (PST) In-Reply-To: <201401310618.s0V6IVJv027167@hergotha.csail.mit.edu> References: <87942875.478893.1391121843834.JavaMail.root@uoguelph.ca> <201401310618.s0V6IVJv027167@hergotha.csail.mit.edu> Date: Fri, 31 Jan 2014 12:58:15 -0500 X-Google-Sender-Auth: RGZ1-iXDuUwPxh2EdH2xGmc6rCo Message-ID: Subject: Re: Terrible NFS performance under 9.2-RELEASE? From: J David To: Garrett Wollman Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jan 2014 17:58:16 -0000 On Fri, Jan 31, 2014 at 1:18 AM, wrote: > This is almost entirely wrong in its description of the non-offload > case. Yes, you're quite right; I confused myself. GSO works a little differently, but FreeBSD doesn't use that. > The whole mess is then passed on to the hardware for > offload, if it fits. That's the point, NFS is creating a situation where it never fits. It can't shove 65k into 64k, so it ends up looping back through the whole output routine again for a tiny tail of data, and then the same for the input routine on the other side. Arguably that makes rsize/wsize 65536 negligibly different than rsize/wsize 32768 in the long run because the average data output per pass is about the same (64k + 1k vs 33k + 33k). Except, of course, in the case where almost all files are between 32k and 60k. Please don't get me wrong, I'm not suggesting there's anything more than a small CPU reduction to be obtained by changing this. Which is not nothing if the client is CPU-limited due to the other work it's doing, but it's not much. To get real speedups from NFS would require a change to the punishing read-before-write behavior, which is pretty clearly not going to happen. > RPC responses will only get smushed together if > tcp_output() wasn't able to schedule the transmit immediately, and if > the network is working properly, that will only happen if there's more > than one client-side-receive-window's-worth of data to be transmitted. This is something I have seen live in tcpdump, but then I have had so many problems with NFS and congestion control that the "network is working properly" condition probably isn't satisfied. Hopefully the jumbo cluster changes will resolve that once and for all. Thanks!