From owner-freebsd-transport@freebsd.org Fri Sep 11 17:15:46 2020 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38AB53E0DAF for ; Fri, 11 Sep 2020 17:15:46 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4Bp2TF6nxyz4bw9 for ; Fri, 11 Sep 2020 17:15:45 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: by mailman.nyi.freebsd.org (Postfix) id E5E343E0D32; Fri, 11 Sep 2020 17:15:45 +0000 (UTC) Delivered-To: transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E54163E0FC3; Fri, 11 Sep 2020 17:15:45 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2043.outbound.protection.outlook.com [40.107.92.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bp2TD3sQ4z4blL; Fri, 11 Sep 2020 17:15:43 +0000 (UTC) (envelope-from Richard.Scheffenegger@netapp.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e6wEmEaAQOMK6EVaoPeQFqsv2uixdtaXSFi5hVjHMUZ+J0L4iaq3/do35cfgfOafuvKscNzFBiCxOmwasSRkkzqTthV1HX6iVtQp26WJOXVfHPPawg41SJ5B5oqFLfebC7Ud130UqKMQdPmqLCc1N0hEigynEBtorpIHrG5PfeLBJlX5PZm3FZZAW4mk+N32GZLSEWPu4KCgUqb5/h9QjOQmaJzM6qZxrFzVXw4WyftzBUzq9+UGvRt6fc5hl14eV/b2zha41nNCfcspSYrHxaaY66f54LKc+zX6u6g+YNxZv5CLIi0InF+sOukrlVZSWIpoIU0L9IIikARivSQJtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wIIlLi7zTFskA/vKFfjBPcbmSAIYjYFA+9WpC9gM9mQ=; b=MW3r/4x5T5p8Z8wYAzGnyuRxBzNDc3u2RT77t+lwPvsEUgOTzD2sHel3jvP3nlW9+7AoyrMepAjy3wyA2xsRh/iyCvtHIhHmH+GGCbLHLgraYBGFX8X4fctbkl+GZgyPIX7VYvkqyAQV+Ne03pPttc/XOQCsJqPCzvewXVhvyKGex451NmJfL6obzLLNIxt9BgMWFdA6TmFhYHtS/XQVD0rJRdjQMD65O2DEib/HJF8uqv0Or5SmqBDDNt/ZRVtmJwP+rCJtBgNcJ9RKxpftKVmntgfeMs7wnQq2w9Q+rp4tuh8p80aqWLYjtZ+u2i1ejHscBeIwl+Hz12sM4QFMEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.onmicrosoft.com; s=selector1-netapp-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wIIlLi7zTFskA/vKFfjBPcbmSAIYjYFA+9WpC9gM9mQ=; b=v/9pvVXtoDDo4cpV17XQ0KV/atW3kZHOiwsppauu/yrqB3d5T2De3IBjRgc3A26Twz9Cd8W+c0IkRMZtt9nBJB6E/KopXUUAt7+OgRGKP4IL51CaGy9X+mC6z0QA7WLloIcpS8x+KjAPmNXXG0s+W5ZFVXD3JF+j0tUHCL6xdSU= Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SA0PR06MB7004.namprd06.prod.outlook.com (2603:10b6:806:bd::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3370.16; Fri, 11 Sep 2020 17:15:43 +0000 Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::84a9:fc74:7eed:ca3a]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::84a9:fc74:7eed:ca3a%7]) with mapi id 15.20.3370.017; Fri, 11 Sep 2020 17:15:43 +0000 From: "Scheffenegger, Richard" To: "sthaug@nethelp.no" CC: "net@FreeBSD.org" , "transport@freebsd.org" Subject: RE: Socket option to configure Ethernet PCP / CoS per-flow Thread-Topic: Socket option to configure Ethernet PCP / CoS per-flow Thread-Index: AdaIV6Ig8St2MdM0SrW3cDX5AzmWpwABJZYAAABYzLA= Date: Fri, 11 Sep 2020 17:15:42 +0000 Message-ID: References: <20200911.185432.122001633.sthaug@nethelp.no> In-Reply-To: <20200911.185432.122001633.sthaug@nethelp.no> Accept-Language: de-AT, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [217.70.211.11] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9a5edaf1-261f-475a-223b-08d85676508c x-ms-traffictypediagnostic: SA0PR06MB7004: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ducvOnf1QDLk/bM5xUZg6aAfYz5hQkbk5a9t5BExV1Z/VXOflkD2ss1VZ9y6g91Vq5eJ10Dzq+EUx2xDwRVAq1azTpjJh1/nMI8s/c8fWe9SgrznKLPjE8zMzIOW682XxN8tyOTR1LDqwbxRoCSEkd5BQQ2hSPZf/I4pD/d+YTEUXufCJ5LR8CthxX2sApmlzBgksmEjqn/iCT1Wp6GO7oBlmxjHbb9I8T9EECP4gO2yrM3yXE1elJvUlI8p40rbhHJ2Vg9E3OoKuwlQxo/wT0HUCQRsGDncRntDLtsXLO624UtTDGWsNUhoJT1dBdq1 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(346002)(136003)(376002)(396003)(39860400002)(54906003)(76116006)(66446008)(478600001)(71200400001)(86362001)(2906002)(186003)(316002)(5660300002)(33656002)(6916009)(26005)(66946007)(66556008)(64756008)(7696005)(53546011)(6506007)(66476007)(8936002)(52536014)(4326008)(83380400001)(55016002)(8676002)(9686003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: UXp6yQNWSxIERgWRgxCjlwIo4VWwiR2000Fs+My2JC63DSHD7VCpLrzVvAMGlHOr5OmKCwKYqFjwRqw4TURAqWMYA58lcjdYDWxxLXWj65+MrDSu5AAz8uCfT8uNp+g1CGX+h6eGp9Ttb3vWrpV/LRfRHLUJAPSvPEyYlv9L18RpgDLN1i1RJ93HwoeUBZzr6aKy2H3H/twH/CZL7I5qmmcdh7iDiRgP3uDqN3z/jv4aty4kIcvdFGFqHROQV2NQtTvwbhngTtCpN3dntBqzMwmBqnQpU16azXxvVOlun6aUwW3qc64OLPQkW+Ta+RRZ6KMe3hh7IxNTDdQQ4AAUR3ZrCFP/IvHABVV4vObzVZQNAIlrj5mZggXeernKA7m7Prj4QPq86RZNTrxrKQ04C2B4HaxXCcn15mJR89DPAFbvIDo8TD/nw8YnTEs6TQPpgu96b1IFC+5YsW1+zqW2g5zqjejFXDJqm8fwMsHYuL/Mp8QRo8j4gpbWfozj5PRU4EuW9OLUpiq/cFicuZl4TNXlbGyjsZ3Sy0g0c/0/Ch7T1p4WyG8EODhyZbEwebJ+s+iXaSzypFPqIaU38jJRMWbh3Vp6x4UUFliWvLfWcOGvt4hemJ6Zt/gNQ/xtn9uyiixPRNidsFXnXfdS9haTUA== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: netapp.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9a5edaf1-261f-475a-223b-08d85676508c X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Sep 2020 17:15:42.8766 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: G615wrjUKL/ucoZC6ALnZzG/HtSW0MAaUMfGkkxPngyhOdIdVDzGdbCNcnUk/uai2AaknqwpBURi6MJske8pmQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR06MB7004 X-Rspamd-Queue-Id: 4Bp2TD3sQ4z4blL X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=netapp.onmicrosoft.com header.s=selector1-netapp-onmicrosoft-com header.b=v/9pvVXt; dmarc=none; spf=pass (mx1.freebsd.org: domain of Richard.Scheffenegger@netapp.com designates 40.107.92.43 as permitted sender) smtp.mailfrom=Richard.Scheffenegger@netapp.com X-Spamd-Result: default: False [-4.02 / 15.00]; NEURAL_HAM_MEDIUM(-0.98)[-0.977]; R_DKIM_ALLOW(-0.20)[netapp.onmicrosoft.com:s=selector1-netapp-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.004]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[netapp.com]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[netapp.onmicrosoft.com:+]; NEURAL_HAM_SHORT(-0.54)[-0.537]; RCVD_IN_DNSWL_NONE(0.00)[40.107.92.43:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[net,transport]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.92.43:from] X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 17:15:46 -0000 Thank you for the quick feedback. On a related note - it just occurred to me, that the PCP functionality coul= d be extended to make more effective use of PFC (priority flow control) wit= hout explicitly managing it on an application level directly. Right now, PFC typically degenerates to good-old Flow control, as all traff= ic is handled just in the default class (0, or whatever is set up using the= IOCTL interface API). Typically, the different Ethernet classes come with a notion of prioritizat= ion between them - traffic in a "higher" class may be forwarded prior to tr= affic in a lower class. But that is not a strong requirement - using WRR wi= th 1/8th bandwidth "reserved" for each class in a switch, assigning flows t= o a random PCP value, PFC could work in a more scalable fashion - only bloc= king a fraction of traffic, that is actually queue building (has to go over= a lower bandwidth link, or a NIC excessively pausing its ingress), thus re= ducing the chance of the formation of congrestion trees... E.g. PCP runs from 0 (default) to 7;=20 Adding a socket option to explicitly assign traffic to one of these flows w= ould allow testing and configuring applications to make use of "real" prior= itization capabilities of modern switches. And what I was just pondering was a special interface level setting (e.g. 8= ), which results in a socket to pick a "random" value when created, to dist= ribute packets across all the queues available in hardware, allowing PFC to= no longer collapse in effect to old FC style "on"/"off" for all traffic...= =20 Perhaps someone here has experience with congestion tree formation in multi= -hop switching environments, and can comment if the above approach would be= feasible to address that FC issue? Richard Scheffenegger -----Original Message----- From: sthaug@nethelp.no =20 Sent: Freitag, 11. September 2020 18:55 To: Scheffenegger, Richard Cc: net@FreeBSD.org; transport@freebsd.org Subject: Re: Socket option to configure Ethernet PCP / CoS per-flow NetApp Security WARNING: This is an external email. Do not click links or o= pen attachments unless you recognize the sender and know the content is saf= e. > However, while this allows all traffic sent via a specific interface to b= e marked with a PCP (priority code point), it defeats the purpose of PFC (p= riority flow control) which works by individually pausing different queues = of an interface, provided there is an actual differentiation of traffic int= o those various classes. > > Internally, we have added a socket option (SO_VLAN_PCP) to change the PCP= specifically for traffic associated with that socket, to be marked differe= ntly from whatever the interface default is (unmarked, or the default PCP). > > Does the community see value in having such a socket option widely availa= ble? (Linux currently doesn't seem to have a per-socket option either, only= a per-interface IOCTL API). I've been doing quite a bit of network testing using iperf3 and similar too= ls, and have wanted this type of functionality since the interface option b= ecame available. Having this on a socket level would make it possible to te= ach iperf3, ping and other tools to set PCP and facilitate/simplify testing= of L2 networks. So the answer is a definite yes! This would be valuable. Steinar Haug, Nethelp consulting, sthaug@nethelp.no