From owner-freebsd-net@freebsd.org Tue Jun 16 19:56:24 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CEA58331771 for ; Tue, 16 Jun 2020 19:56:24 +0000 (UTC) (envelope-from gfoster@panasas.com) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on20601.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5a::601]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49mf8l3ctxz4c0Z; Tue, 16 Jun 2020 19:56:23 +0000 (UTC) (envelope-from gfoster@panasas.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DQB7jO4DUZGknsEzSUEcUCyGTcJSPuQtETeK4/x+OWsGYYZVPfPxuJf8g+c99MswyrWCdfuu4sX8A3DObxefwT9u2MgOqsQQCcVpqO9ojC/Jsav/zMXyRxFZO8QOYU0DsGYg8NpqtbjW3PeDgsfr+jtcuzwAaJuzNuSpu8+P3v2Bf7PYs+1hBYqjP0uLC33R5mieUGWYgwFr4cpy1i8Fj0Qty3WhWuDsyHCQbi36ph8F2CDFBtSCZ2QnxrDp0o4/sbSSQmwVnSn0O091FVRIM90VU3E3lkhJsm6+GWBeqY+qTJopcP16D34xrf72uKPKGoKtSxbJo+v0Jw3CgvAijw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wp5+zkVk90PcmKg1nPmVPNRWIPXeKpFifZ1KzP2KPv8=; b=Aeo4eIsO91yC0bKTIdNHoxS18pngW1aGhnYN9JvBDqL/cZgPISLNs3I8bpA7uQjttNlX0n6tVCf1mC87z8oAEg2ZAiwO7tLVuA7Lkdu6GlwNVkUJgHevycvfYo4bzzpoBpnJhnP6RW40tgc8NapBof0Vv5qF88n+rPyC2zBKAi4HuWJ4+GmwExmDhZ9KTqcFeceD7a3+OM4IN2QfXOrUdfuPK49Wmf/MLlce4dcs3zClxN9T1NvFbtvKENF0P8gzU74TDMeDldfyl8x2V//z+smmqDiOn0LdDGDV4tUgvTf9fGprZELuHAWC4ryUtMbI/wGjiGLaw9/hCaDpIauxPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=panasas.com; dmarc=pass action=none header.from=panasas.com; dkim=pass header.d=panasas.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=panasas.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wp5+zkVk90PcmKg1nPmVPNRWIPXeKpFifZ1KzP2KPv8=; b=D8tK1RCLfxQMx0JX/2kKhRtvXzhiOIYOJ3le79dkPZl/z8nn6FSNAtnzljUg0/J9s5kpiVLKrzAKNidMoZd84Ljw+Du/32JVzBN38IypwIbaMTVvrR6GgoRH9XK7YEwW9lQVX9TLdyL33TA+/o2nptZrH4XIjgJ0n1qQ15Z9Jps= Received: from BYAPR08MB6038.namprd08.prod.outlook.com (2603:10b6:a03:c6::31) by BYAPR08MB5559.namprd08.prod.outlook.com (2603:10b6:a03:ce::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.21; Tue, 16 Jun 2020 19:56:20 +0000 Received: from BYAPR08MB6038.namprd08.prod.outlook.com ([fe80::9077:5c2f:269c:d196]) by BYAPR08MB6038.namprd08.prod.outlook.com ([fe80::9077:5c2f:269c:d196%7]) with mapi id 15.20.3088.028; Tue, 16 Jun 2020 19:56:20 +0000 From: "Foster, Greg" To: Navdeep Parhar CC: "freebsd-net@freebsd.org" Subject: RE: How to Increase TX Queue Priority for LACP Packets Thread-Topic: How to Increase TX Queue Priority for LACP Packets Thread-Index: AdZBE8R05PE/72svTY6segyLIx/vzAAGao0AALpZlVA= Date: Tue, 16 Jun 2020 19:56:19 +0000 Message-ID: References: <20200613025031.GA34653@x270> In-Reply-To: <20200613025031.GA34653@x270> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [75.141.175.213] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1d2805c6-52ce-484c-1f6d-08d8122f56b6 x-ms-traffictypediagnostic: BYAPR08MB5559: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 04362AC73B x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 1LK3yaUFs0KuJrPjklsavdSBr2G0/pz4vrToUYHkl6dexzAfvj9syPUvCvf9FMJ+XvFKpuTaT681oLb2BNrh3of2G3A3IEF47aGS/gMO1goYXwRegAyZsqE/b3rggfVk4tgwkhP8J4IlH9bektZabV9zAaXJLM3VAhPXT1zVKsBidwUidQJrb/cAdiuTUt31mg6qxcBX5GYtIBQxF9U+rYJdVzWj0EVUuPYvFBJMpVoQj896Vp3biW9R73rk8l583HeF8dTm23RLbyyWvbt9vqSvZIio/5Ht+Azhq9A2uxoVKx1Am9tX4v7jF8Y7SecODAr4E3D16ZXmw+lWZwyPLw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR08MB6038.namprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(39840400004)(366004)(136003)(396003)(346002)(376002)(86362001)(30864003)(6506007)(2906002)(8676002)(66446008)(66946007)(52536014)(55016002)(66476007)(66556008)(8936002)(316002)(6916009)(64756008)(76116006)(53546011)(9686003)(966005)(186003)(7696005)(26005)(33656002)(5660300002)(450100002)(478600001)(71200400001)(4326008)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: KOmORCxJu7lZUyffdOGaBZq8mzU2KN3rVtfR0aAAnqmi0ZO79Vo/E6dSVw+ezjwOiiR4sP4Z77tEwgaWq/PMv+WwN24kC9lS7UsPBajZDha/VPdeL33u43uWI1z95lQproaLJnTjeSsCEEa6UgCbRcnrN5fos4zkP3LV/d/hF7rVYOmgCAaVQy0LcN1oml2jl780Izbx3dH7wZaDMPhiXhWHj3IVQ7P5/+luW1aQh0+b14KxO13IzIzcI88yuFobpHcmPgKYV4F+tcu88Hj8/evjtMIYzbuSnQluyvA5cLI0aPOAdFB+D3ejnrTtPyZzRJzYiIBuxkBGz8lb4KtZGjrESFnqU5r5j/M22XAddQHioagR2+kZ1w5+i2EzvXk+8cP/nUKFWPsuD2Z3rF8uVgR3j+ZpAjFpzfIj+KoWrHu3SyaXkl0bl4+I/8wOUrJALfW+ciKik7hEqpRlxEn7xHBlxaEJOGnGfaglYlkzMMgoR9yv6a3HJYgLxhVcmKde x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1d2805c6-52ce-484c-1f6d-08d8122f56b6 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jun 2020 19:56:19.8785 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: UkWST2isxl5GjJJEFuBaIsEMYaXfKkqh54Vb0CIl0LqIWZnSX4oen9pcBGEm4D3E8d2cmWvzEQ/Oiq7nYYM/Dw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR08MB5559 X-Rspamd-Queue-Id: 49mf8l3ctxz4c0Z X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=panasas.com header.s=selector1 header.b=D8tK1RCL; dmarc=none; spf=pass (mx1.freebsd.org: domain of gfoster@panasas.com designates 2a01:111:f400:fe5a::601 as permitted sender) smtp.mailfrom=gfoster@panasas.com X-Spamd-Result: default: False [-3.78 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.979]; R_DKIM_ALLOW(-0.20)[panasas.com:s=selector1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[panasas.com]; NEURAL_HAM_LONG(-1.01)[-1.010]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[panasas.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.29)[-0.290]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jun 2020 19:56:24 -0000 HI Navdeep, Thanks for the information! I've integrated the changes and will be testin= g more today. We have seen the LACP port flapping under different scenarios, most we beli= eve are traffic/load based. I did see the flapping unexpectedly when I just enabled LACP debug (e.g., sysctl net.link.lagg.lacp.debug=3D1). Is this a known problem? Thanks Greg -----Original Message----- From: Navdeep Parhar On Behalf Of Navdeep Parhar Sent: Friday, June 12, 2020 7:51 PM To: Foster, Greg Cc: freebsd-net@freebsd.org Subject: Re: How to Increase TX Queue Priority for LACP Packets On Fri, Jun 12, 2020 at 11:47:41PM +0000, Foster, Greg wrote: > FreeBSD Networkers, > > We are seeing LACP port flapping on our FreeBSD 10.4/12.1 systems > under different conditions. > > Can someone explain or point me to the information on how to queue the > LACP packets to a higher priority queue ? > > We are using the Chelsio T580-LP-CR adapter/cxgbe driver. The Cheslio > NICs have 8 TX/RX queues each, but I don't know how to explicitly put > the LACP packets in the higher priority TX queue. > > I've read about PF/ALTQ and think this may be overkill our needs, and > was wondering if there was a simpler method. This is cxgbe specific but that's what you're using so it'll do. Add "hw.cxgbe.rsrv_noflowq=3D1" to your /boot/loader.conf. That reserves o= ne tx queue for non-RSS traffic (like ARP, LACP). You might also want to i= ncrease the number of tx queues to compensate for the one that's now reserv= ed. Use "hw.cxgbe.ntxq=3D9" for that. The ntxq knob might be different on= 10.4 but the man page matching the driver should have its exact name. Regards, Navdeep