From owner-freebsd-net@freebsd.org Fri Apr 20 22:02:41 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A35D1FA6504 for ; Fri, 20 Apr 2018 22:02:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670084.outbound.protection.outlook.com [40.107.67.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3523B699D1 for ; Fri, 20 Apr 2018 22:02:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB2002.CANPRD01.PROD.OUTLOOK.COM (52.132.71.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.12; Fri, 20 Apr 2018 22:02:39 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a%13]) with mapi id 15.20.0696.013; Fri, 20 Apr 2018 22:02:39 +0000 From: Rick Macklem To: hiren panchasara , Garrett Wollman CC: "freebsd-net@freebsd.org" Subject: Re: Diagnosing terrible ixl performance Thread-Topic: Diagnosing terrible ixl performance Thread-Index: AQHT2O7cpk1RY9D0akiazz05k2oUaKQKNHrx Date: Fri, 20 Apr 2018 22:02:39 +0000 Message-ID: References: <23257.26265.720293.659892@hergotha.csail.mit.edu>, <20180420212045.GB3653@strugglingcoder.info> In-Reply-To: <20180420212045.GB3653@strugglingcoder.info> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: strugglingcoder.info; dkim=none (message not signed) header.d=none;strugglingcoder.info; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB2002; 7:rLMLH9UHkf4ZgQTKJSsYdm6KyM5SAlvJb6KX/MD/Xm0RsY0xExk7Bx4KceqL/YgyyvZmDVaVcDCfIF6JszLbZlRM7L42EoFmAHZdipwyBUQArmHGCH9E8q80TZ2fcy3qE8vkfybu7NECKv3DXYMpA+17Ikq8WRe2ox4RX1mGqEgLwDZcZoIWmlzTtpIGsAFLSRwt142jryDocG1VMFdzYphk7rAa/FuCU72ak5mli193gir94kpyaue3NkoCUVh9 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB2002; x-ms-traffictypediagnostic: YQBPR0101MB2002: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231232)(944501327)(52105095)(3002001)(10201501046)(6041310)(20161123564045)(20161123558120)(20161123562045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(6072148)(201708071742011); SRVR:YQBPR0101MB2002; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB2002; x-forefront-prvs: 0648FCFFA8 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(39380400002)(396003)(39860400002)(366004)(346002)(9686003)(478600001)(6436002)(110136005)(6306002)(2906002)(74482002)(6246003)(8936002)(186003)(55016002)(74316002)(25786009)(4326008)(5250100002)(5660300001)(3480700004)(86362001)(966005)(53936002)(446003)(305945005)(476003)(3280700002)(11346002)(2900100001)(102836004)(3660700001)(229853002)(786003)(33656002)(76176011)(7696005)(26005)(81166006)(53546011)(6506007)(8676002)(316002)(6606295002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB2002; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; MLV:sfv; x-microsoft-antispam-message-info: KaCyNUC3BMIzBrIBTnpgTaKwdWN/f6rJHq5M9BzhQTtK2CaR5f/lRDBnHK3BK2u4yO7pXKIASon6/vmqZzZT1jXUfNn41qG7O2tTwt3F6TkydTTEOex8BjFKfarAG6HVDoulr8Y22wHJtnxqiGVRYtZkf9XhWjIcox58Dm6H23vPmQoqmKdjzJpI/qxccwdS spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: eeb5feb0-ee78-4bf8-eeb3-08d5a70a6eef X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: eeb5feb0-ee78-4bf8-eeb3-08d5a70a6eef X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Apr 2018 22:02:39.4741 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB2002 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 22:02:42 -0000 I don't know if this post is helpful, but just in case: http://docs.FreeBSD.org/cgi/mid.cgi?04125f40-6388-f074-d935-ce6c16d220fa Hope you don't mind a top post, rick ________________________________________ From: owner-freebsd-net@freebsd.org on beha= lf of hiren panchasara Sent: Friday, April 20, 2018 5:20:45 PM To: Garrett Wollman Cc: freebsd-net@freebsd.org Subject: Re: Diagnosing terrible ixl performance On 04/20/18 at 12:03P, Garrett Wollman wrote: > I'm commissioning a new NFS server with an Intel dual-40G XL710 > interface, running 11.1. I have a few other servers with this > adapter, although not running 40G, and they work fine so long as you > disable TSO. This one ... not so much. On the receive side, it gets > about 600 Mbit/s with lots of retransmits. On the *sending* side, > though, it's not even able to sustain 10 Mbit/s -- but there's no > evidence of retransmissions, it's just sending really really slowly. > (Other machines with XL710 adapters are able to sustain full 10G.) > There is no evidence of any errors on either the adapter or the switch > it's connected to. > > So far, I've tried: > > - Using the latest Intel driver (no change) > - Using the latest Intel firmware (breaks the adapter) > - Disabling performance tweaks in loader.conf and sysctl.conf > - Changing congestion-control algorithms > > Anyone have suggestions while I still have time to test this? (My > plan B is to fall back to an X520 card that I have in my spares kit, > because I *know* those work great with no faffing about.) Any > relevant MIBs to inspect? > > The test I'm doing here is simple iperf over TCP, with MTU 9120. It > takes about 10 seconds for the sending side to complete, but buffers > are severely constipated for 20 seconds after that (delaying all > traffic, including ssh connections). > > I'm at the point of trying different switch ports just to eliminate > that as a possibility. You are already trying to check if the switch in-between isn't causing the problem. A few other (probably obvious) things to try: - sysctl -a | grep hw.ixl or dev.ixl to see if you find anything useful (actual name might not be ixl, but you get the point) - Try with lower mtu to see if that's causing anything interesting - If you can reproduce easily, a single stream pcap might be useful from both send and recv side to understand the slowness. Cheers, Hiren