From owner-freebsd-infiniband@freebsd.org Thu Nov 1 00:27:19 2018 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3636910D373A; Thu, 1 Nov 2018 00:27:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660062.outbound.protection.outlook.com [40.107.66.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C1641735BC; Thu, 1 Nov 2018 00:27:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM (52.132.50.155) by YTOPR0101MB1546.CANPRD01.PROD.OUTLOOK.COM (52.132.49.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1273.26; Thu, 1 Nov 2018 00:27:17 +0000 Received: from YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b]) by YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM ([fe80::9c71:6eb6:1bff:727b%3]) with mapi id 15.20.1294.021; Thu, 1 Nov 2018 00:27:17 +0000 From: Rick Macklem To: Andrew Vylegzhanin CC: "Rodney W. Grimes" , "freebsd-fs@freebsd.org" , "freebsd-infiniband@freebsd.org" Subject: Re: NFS + Infiniband problem Thread-Topic: NFS + Infiniband problem Thread-Index: AQHUbzV7uxRrUqx4EESM4oETGDUinKU2U4sAgAACL+GAAOXvgIABagv/gAElKYCAAEaZ4w== Date: Thu, 1 Nov 2018 00:27:16 +0000 Message-ID: References: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB1546; 6:pegFwrocxH1W7ICIxfAFgOryQEvmi8NKbCWsQYdXUI+zKbmcULytGT25PLf7T4Ne2tdfzCHW5PTxkdReHDZGQQ6I0pPXiXO27vjvYKpSfZHZOvU1pT+i4UW0EpdGZhkr1bDkNqfidBzecZfP1KCLB4nvm63wOniKKXmza5IaCoVbup2gA78XiGBqXb/3e8w0Lyp3mQ8T9WMXYIMJX8fI4con4mtsuVGWxrcHphpeUk/1ONIssCw03vdg7NrUHDkcPCBzFmIss6PdBmEPsMOH5blt5wTr8+wyabeXRr9mV4aGPnE83w/QVZuyC63JfBywObmpkRnOfmge0gRRHtnLDVzL7i30KujV7RIQnJj3HRLbTOIhKdfhFkfCzP1PlTxwC4u5A8cXlo+lbI4qwI4/LAN4SNvTWyQxc8TMoxi7TBXsJh7agNR7mE0tsgkloXP8B0iKkoMMNvVfWt5QplCd/Q==; 5:6kzEtACuDzgk+evXMg0j0QwPiDYKONy3JCx6vA7W3Kl1+GOF3ukIN/YOxUpgHnUj1qRkCLLH2DhuU423QIjSHZzEsFfWwUiAkBJS3peVBTGopUeLMfdSocHBHxMHTvM8sDeN1t/99tSVuo+2exrVe6sY7/hcpKRZXcPJYQ2klyw=; 7:x5XABEz59c51KCXN/3J+F8PGaZtDkueGF6l+VotXWM0zP8Q5x5hyLcvr3MmxmBF85lWIeaPaL1g0KS13ZOC38VhkgTLR6BBNO6Ds/9qFb7efv91jffOdnhzRqV+1sCfIwRkGSq9yAt258wUTREwuug== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 863b1fdc-90eb-446b-38fe-08d63f90c740 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB1546; x-ms-traffictypediagnostic: YTOPR0101MB1546: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3231382)(944501410)(52105095)(93006095)(93001095)(3002001)(10201501046)(148016)(149066)(150057)(6041310)(20161123560045)(20161123564045)(20161123558120)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:YTOPR0101MB1546; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB1546; x-forefront-prvs: 0843C17679 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(346002)(396003)(39860400002)(376002)(366004)(189003)(199004)(71190400001)(6436002)(105586002)(33656002)(6506007)(5660300001)(76176011)(106356001)(74316002)(99286004)(68736007)(478600001)(6916009)(7696005)(2906002)(1411001)(71200400001)(305945005)(97736004)(229853002)(476003)(446003)(46003)(93886005)(6246003)(4326008)(8676002)(54906003)(81166006)(11346002)(14454004)(81156014)(39060400002)(9686003)(25786009)(102836004)(5250100002)(53936002)(256004)(55016002)(2900100001)(8936002)(786003)(316002)(86362001)(186003)(486006)(74482002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB1546; H:YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: QJNnb3pcZUvn+43cV1LCDqwHT4QZWQ5o9abUCvhEosfA8SbhxWqgfY9yRJAh5u7L21c7eGC+ALBI4QMU5GAVPFyi71yjCI9kCMLIkzgsFsEpG2np0NJRcM1jHNygIExtJgBC4MCdfHH0VOhLFLfrY9mG/Hv89a8i4wA4i4BZsglwmKEP4qKx+jG2MUBN2oieClG6kdCYFGVbr5jtnyTQ7S6Z10mk9OqKARI0Zg+2VKWYsnKG0ycdoCY8pJc+HcQqppaUvZxK0WtzglMWmtWbgmBoLG58AYNkjoq3CmLHmMGmel2jVttbjXENSr4IvQvqjFGobfVPfXytRS/CBN3AfrGd+9wc1Lv2Nz4o2EiifaA= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 863b1fdc-90eb-446b-38fe-08d63f90c740 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2018 00:27:16.9281 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1546 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2018 00:27:19 -0000 Andrew Vylegzhanin wrote: [stuff snipped] >With this TCP settings same server serve NFS requests via 40G Ethernet on = multiply >clients with speed via 1G Eth ~ 105/110 MB/sec write/read. >Of course I'll try to change congestion algorithm, but I don't think that = will help. Yes, I doubt changing the congestion algorithm will make much difference. >Also need to test setup with infniband set from connected mode to datagram= mode. Are you using IPv6 by any chance? Why I ask is that there was a problem with IPv6 fragmentation re-assembly. = If InfiniBand is using a larger MTU than the ethernet, the switch would probab= ly fragment the IP datagram. If any fragment is lost (or fragmentation re-asse= mbly is broken), it isn't going to work well. "netstat -s | fgrep "fragments dropped after" will show you the count of fragments dropped after timeout. If this value i= s increasing, then fragmentation re-assembly is an issue. (Check on the recei= ving end. The server for writes.) You might want to post on freebsd-net@, since someone there might know more about InfiniBand (and someone over there will definitely know about the IPv= 6 fragmentation problem. rick=