From owner-freebsd-infiniband@freebsd.org Mon Oct 29 05:16:15 2018 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D2CE10F086A; Mon, 29 Oct 2018 05:16:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670073.outbound.protection.outlook.com [40.107.67.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 107867E8D8; Mon, 29 Oct 2018 05:16:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1899.CANPRD01.PROD.OUTLOOK.COM (52.132.49.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.25; Mon, 29 Oct 2018 05:16:13 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::a086:adbc:2b38:1018%2]) with mapi id 15.20.1273.027; Mon, 29 Oct 2018 05:16:13 +0000 From: Rick Macklem To: Andrew Vylegzhanin , "freebsd-infiniband@freebsd.org" , "freebsd-fs@freebsd.org" Subject: Re: NFS + Infiniband problem Thread-Topic: NFS + Infiniband problem Thread-Index: AQHUbzV7uxRrUqx4EESM4oETGDUinKU1rGlW Date: Mon, 29 Oct 2018 05:16:13 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1899; 6:jUEUM4mhLPavH9axN7rp7y95BvXxalJNNngx8oNOa02CADyfeFvW8R8TpPHB58sntXIv633B6tzgUcbwBTnnATcieDgi+jwt9r19/EZ8aMwM/YCHgTMpwciVg3Ne83vGoCQRPbFV5PDcBrjMzcvGBkOkNyH7B1aSFUMPtBxttxXxsRiqHr8x9V/J8fu4E+naZ2GnlvFByUR+BfvMfsWjyDEwlzcu9xEURm0OcfulGB5SlJe0mb0eLDU44DbWHMqbZul1v5WlNcVkuG3WmrPWPk4zWkv2pEJHP51/QXgoP584RNxK30bX6mtW2Vyd5Y785+WImKY6Y3Wzak1UpGfgbENYmwcSbwg01nhwynSHAI6433+GGxJ2X/0XdAJ9QwTAdiiitkHrKPUZ04LFN0mg8sLLTRZwNiQLL15I/XofVeDThJmRCq2MKk9JYIWAB47R0naulhLCYi5OdDON5gOAcw==; 5:lbQ8OfRBk9eSLoyx5p/6hOFLxO23Klzh3PfI8vthOAlRKaFaQd0gGTkbmHJ4a7mdF5HDO0PXoOD01jE//tcNPvBOTeQnYLEaUwpbK6Fi468A3/2O566zy+V8AHxhnStzxcq37pujXZx8bqbpJ2iLNCCQ6yqEMKxFgDxqzLeutUI=; 7:ofrX/rnslXHFmTANd/N16XFNGgv2bJLonTqs5Y4oWTOVloJPBCaC9c9lMy9Ecn2SnF6IOKcaqio+ZTjo4xez9s9It8740A47YtRSpar9M8gZkW1NsYHWFIuSahLD3IIqLDyKEBAMwdV0XdStpOpXHQ== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 6bb000ad-79ed-4330-ddce-08d63d5da556 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1899; x-ms-traffictypediagnostic: YTOPR0101MB1899: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231382)(944501410)(52105095)(148016)(149066)(150057)(6041310)(20161123562045)(20161123564045)(20161123560045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1899; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1899; x-forefront-prvs: 084080FC15 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(346002)(376002)(136003)(39860400002)(366004)(189003)(199004)(39060400002)(74482002)(186003)(316002)(105586002)(33656002)(256004)(5250100002)(2501003)(102836004)(55016002)(25786009)(786003)(7696005)(76176011)(14454004)(110136005)(106356001)(6506007)(81166006)(6436002)(68736007)(2906002)(81156014)(2900100001)(53936002)(9686003)(99286004)(486006)(6246003)(97736004)(71190400001)(74316002)(86362001)(46003)(2201001)(8676002)(476003)(229853002)(5660300001)(11346002)(478600001)(446003)(8936002)(305945005)(71200400001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1899; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: 8ip4rNKt1HbRXthqs8r9tA9MoSfoZoNnIfZ100IiAD/7WNeG+f3nLAXB+DSD5ypoGkqPWt8LqMu3pESV3eqKZTQG44PsGTKwQcAPlrzbRIis0ajWNGjqvlQy1l2riHWC3bCx5srQ9sp7H21xfBZr8PukXmljRTJLcjbi54CMkfPR6kHrbKjN3c6m3LgVvsuETxMwy9A6jUifeAwuLzABvcOhjnnKi2OfpABi+gfEq0NEvAj0KR0QIprg8Y4ufk2M1E++WRq84zCaxz/rpwsstUm2oBrrSfgBAvHDVjlJDHKdLcYqiiWoVO/NSZFPTsEa8dj0oa6wsUFgm7Ni3i86+skpCtx+YDuk9wCA1wAico4= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 6bb000ad-79ed-4330-ddce-08d63d5da556 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2018 05:16:13.4065 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1899 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 05:16:15 -0000 [stuff related to slow performance snipped] Try disabling TSO, that is the most common cause of NFS RPC transport issues. # sysctl net.inet.tcp.tso=3D0 will do it, if you can't do it for the interface. (Also, try disabling LSO, LRO if the interface will let you do so.) You can also try mounting with "rsize=3D8192,wsize=3D8192" and if that work= s fairly well, then just keep doubling it until it no longer works well. I know nothing about InfiniBand, so if the above doesn't help, hopefully someone familiar with InfiniBand can help. Good luck with it, rick