From owner-freebsd-net@FreeBSD.ORG Fri Jan 24 23:37:38 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D38D6F3A; Fri, 24 Jan 2014 23:37:38 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9321914BF; Fri, 24 Jan 2014 23:37:38 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id tq11so3666447ieb.12 for ; Fri, 24 Jan 2014 15:37:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=HdLDnHYjQTX6MyMfL1n+GBW3CiJXkp1xO/xZ3SxcrUI=; b=lNL9K8aHKqX72jqW9561Xos7SabwdbKackXEpdiB5TSXwhrn072ct/W/+aCeMdm7lR 3DZbRIfqbVPCGtRxa28PQLcKK2IOLltUPpysUPeqzGO1iiArz0nQ4Ze4fd6R/4R+7ZsK F0O3osK1wx6BSNuOT31fH0SNji9gbmv0UzclZin2wefYEqqT+fJH+dQFGuw8WawMNVfy M0KlfeYsRAV6AjMdS+OWlXrfPw0F2b6de96canfq0z7ffPwmjpoIImcYaYBkDMBQN1F2 trWBYNkzoXoPki+5CFry/kwTZZmf2ztME8UhtsspcmQsVvczOxKPGaIhkxpWu3vFHb4r lbCw== MIME-Version: 1.0 X-Received: by 10.43.161.2 with SMTP id me2mr13175103icc.20.1390606658118; Fri, 24 Jan 2014 15:37:38 -0800 (PST) Sender: jdavidlists@gmail.com Received: by 10.42.170.8 with HTTP; Fri, 24 Jan 2014 15:37:38 -0800 (PST) In-Reply-To: <659117348.16015750.1390604069888.JavaMail.root@uoguelph.ca> References: <659117348.16015750.1390604069888.JavaMail.root@uoguelph.ca> Date: Fri, 24 Jan 2014 18:37:38 -0500 X-Google-Sender-Auth: 9muO36VsdQ0aOJ6d6VrC7cXUcv8 Message-ID: Subject: Re: Terrible NFS performance under 9.2-RELEASE? From: J David To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jan 2014 23:37:39 -0000 On Fri, Jan 24, 2014 at 5:54 PM, Rick Macklem wrote: > But disabling it will identify if that is causing the problem. And it > is a workaround that often helps people get things to work. (With real > hardware, there may be no way to "fix" such things, depending on the > chipset, etc.) There are two problems that are crippling NFS performance with large block sizes. One is the extraneous NFS read-on-write issue I documented earlier today that has nothing to do with network topology or packet size. You might have more interest in that one. This other thing is a five-way negative interaction between 64k NFS, TSO, LRO, delayed ack, and congestion control. Disabling *any* one of them is sufficient to see significant improvement, but does not serve to identify that it is causing the problem since it is not a unique characterstic. (Even if it was, that would not determine whether a problem was with component X or with component Y's ability to interact with component X.) Figuring out what's really happening has proven very difficult for me, largely due to my limited knowledge of these areas. And the learning curve on the TCP code is pretty steep. The "simple" explanation appears to be that NFS generates two packets, one just under 64k and one containing "the rest" and the alternating sizes prevent the delayed ack code from ever seeing two full-size segments in a row, so traffic gets pinned down to one packet per net.inet.tcp.delacktime (100ms default), for 10pps, as observed earlier. But unfortunately, like a lot of simple explanations, this one appears to have the disadvantage of being more or less completely wrong. > ps: If you had looked at the link I had in the email, you would have > seen that he gets very good performance once he disables TSO. As > they say, your mileage may vary. Pretty much every word written on this subject has come across my screens at this point. "Very good performance" is relative. Yes, you can get about 10-20x better performance by disabling TSO, at the expense of using vastly more CPU. Which is definitely a big improvement, and may be sufficient for many applications. But in absolute terms, the overall performance and particularly the efficiency remains unsatisfactory. Thanks!