From owner-freebsd-net@freebsd.org Tue Aug 2 23:14:05 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 563C3BAC7C1 for ; Tue, 2 Aug 2016 23:14:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0056.outbound.protection.outlook.com [157.56.111.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0E821D85; Tue, 2 Aug 2016 23:14:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM (10.165.220.17) by YTXPR01MB0496.CANPRD01.PROD.OUTLOOK.COM (10.165.220.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.549.15; Tue, 2 Aug 2016 22:41:17 +0000 Received: from YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM ([10.165.220.17]) by YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM ([10.165.220.17]) with mapi id 15.01.0549.022; Tue, 2 Aug 2016 22:41:17 +0000 From: Rick Macklem To: Alan Somers , =?iso-8859-1?Q?Gerrit_K=FChn?= CC: FreeBSD Net Subject: Re: NFS on 10G interfaces still painfully slow Thread-Topic: NFS on 10G interfaces still painfully slow Thread-Index: AQHR7JuWK5VNejCg2ESCcdbpFf4ey6A1v7WAgACDZOg= Date: Tue, 2 Aug 2016 22:41:17 +0000 Message-ID: References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-originating-ip: [24.57.164.61] x-ms-office365-filtering-correlation-id: f852f848-80bd-40a4-324a-08d3bb261dc7 x-microsoft-exchange-diagnostics: 1; YTXPR01MB0496; 6:xMgOdnC6poMGYHE91aEdnPjenrfeJyPRYMckD2u9+Jaxo1D5CvFY5ZnSHvv86acHYgU+F4u/lpLpT+jsclsVC//knqyxDaYmKB69G9vVJWOZZCVt/yleNBnqq+DhpyZX82pkgwP+Vv/utUk5/DicJ2Yhlo2AkA2wcvWt9PBg1QAWoolgmQ/HocXwkNXQsAGRWn1ccUqhGSKnhaU4qGfMka7J/ioAEAXN3R202vAvvYdUlgwLfm+rPR+omKRvSHSCyZ8pUxcXopyXhT/46nuFYuGs6/rn6EERylMV83bzRjrLwk6JF0wmJ9k/RKxTGQXk; 5:QMsa++/wg6ozQbPJLGUDkiC4KE7z9D9RY2hT29YxLc2Dajb89d0Ct8KKYDtlixhyLYI9xTlOoy7DEVjgBvW6AE17PrDx2icyuao4IAkKHbhU2Ioe2duxUEuEBnEuB2ySK09l8Ne9tMx4QhL78zvO5Q==; 24:VUgve7MUfi1spdyOP0ldKQBXFaEe2J1+YmVRQSh4eh1cnzU7BaYFuRoaTR7tkpXuVDxQEIXfJ5o6IPCsOmlE9qzyrBBwooLvZwHUxI1kKYo=; 7:iOF1a7alAzKveK2LardD8lieHrtFk/9u+vDlW8M/MjPpIUepTJHy1E4uePEpP4/30po9FJT3hfLzzVxqgncnA2zRGjZPX67xwdAQ0IykuUL3iLTBIMrkFVrHFXAOVJ09Kcl0bTOgvyPF0lPcRsr7HBcQ66thXlQLvQEKyytFyADBABmhg/95QTZ9Ik3hKz3hp4x7UZuo6AGbtXbKmXHTJBOJ9yYiq10rbkqXTxOe+cR6OEvFD0uhgvYLHdxmDM3R x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:YTXPR01MB0496; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040130)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041072)(6043046); SRVR:YTXPR01MB0496; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0496; x-forefront-prvs: 0022134A87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916002)(377454003)(199003)(24454002)(53754006)(189002)(16236675004)(19580395003)(19580405001)(81156014)(81166006)(2900100001)(106356001)(9686002)(4326007)(106116001)(5002640100001)(76176999)(8936002)(92566002)(189998001)(8676002)(3846002)(6116002)(50986999)(74482002)(2950100001)(102836003)(101416001)(3660700001)(33656002)(3280700002)(66066001)(2906002)(19625215002)(68736007)(97736004)(7906003)(5001770100001)(586003)(19617315012)(15975445007)(77096005)(105586002)(54356999)(7846002)(11100500001)(86362001)(7696003)(7736002)(74316002)(87936001)(122556002)(10400500002)(19627405001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0496; H:YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Aug 2016 22:41:17.0385 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0496 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 23:14:05 -0000 Alan Somers wrote: >On Tue, Aug 2, 2016 at 2:49 AM, Gerrit K=FChn wr= ote: >> Hi all, >> >> I already reported this issue here a year ago and unfortunately was not >> able to fix it back then. Now I had another run at it, using two recent >> 10.3-machines with a direct 10G link. I still see nfs is painfully >> slow (around 20-80MB/s). I tried both nfsv3 and nfsv4, with almost the s= ame >> results. Everything I tried so far (mtu size, wcommitsize, readahead...) >> only makes things worse or at least not much better. >> Moving data in different ways (scp, ggate) is much faster, so plain >> network speed should not be an issue. >> >> Is there anyone around here who can confirm that nfs can go faster over >> 10G links? >> Any hints for further tuning/debugging are greatly appreciated. >> I can't help much, but a couple of things you can try: - Disable TSO - Turn off/reduce interrupt moderation on the net interface. (NFS perf. depends on response time and anything that delays interrupt servicing will slow it down.) Good luck with it, rick >> >> cu >> Gerrit > > >I can get 1GB/s over NFS on a 10G link, so it's not always slow. >There's probably something about your setup that's slowing it down. "async" mount option instead of "sync". What benchmark are you using you post your /etc/exports and the client's /etc/fstab, that might >reveal something. >_______________________________________________ >freebsd-net@freebsd.org mailing list >https://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"