From nobody Thu Jun 29 09:51:11 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsDH73wxSz4k5dr for ; Thu, 29 Jun 2023 09:51:15 +0000 (UTC) (envelope-from muralik1@vmware.com) Received: from MW2PR02CU002.outbound.protection.outlook.com (mail-westus2azon11013006.outbound.protection.outlook.com [52.101.49.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsDH650pCz4W7q for ; Thu, 29 Jun 2023 09:51:14 +0000 (UTC) (envelope-from muralik1@vmware.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=vmware.com header.s=selector2 header.b=Xsyv0HE4; spf=pass (mx1.freebsd.org: domain of muralik1@vmware.com designates 52.101.49.6 as permitted sender) smtp.mailfrom=muralik1@vmware.com; dmarc=pass (policy=quarantine) header.from=vmware.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gQ/6rHlJ3P8UBdvuaREGWvtU8NtdcOqzDG1srJ/VUoXYF6QKXz3wewQ2abha2MLBhkZB6fZDLjLS35uue8sBZqZDE2nI+zTfitdYQClCcyUBkaPFkDDGt6A5oxFT9hWQK2+jAtyq8lPsfU9CxWVs010xIUmNXjgLv4YSPYkZjSJyFlxgFGgQIn+GvoNffDzxKJg44eUVKhIHfe5uJ/QRuZZqqE+kSuJFRJI2D1wZuCjPUebYjU6l+m85oR1MvVgVUu4jHnuiBzhVlIy6V+6FBzp4KzuA+82rEbTuqdeI/yTLOHFyh0VsuG/pIEoMF2TfXnNYoBpGqk0WSBX6ENWMzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X8BA7l1ZrNDl7sNhvj+In4v2gc1v/FzKdMp+FfH+Xcg=; b=bH+q+iLMSQ7Qawb9y8TCCwhPrao+LHY9ORCJwA/ugJl9W3cJJOpZOt0LOYcCLDRhfFKEgJtudmPv+eCBfOWRh2/ia1pA24HGBck2w7+6ItT+6z2zhVqPIcJTIM1/BMmiddPpxGKxLIAifBBqs07lI3mLPY+c1qU9F0Te3HDDUS6fgFDCX3C9sjddT+rSPjjYPBurWyP7Gw3kN49gNOoJ27ML7sywyDWeZE3MZMZVVQm3w1Ze8+zAK5TgeQS1Srvac8jJYplDHmW0nO9TUvSgHg1FSHkPGT/vp2600YKJDIaegdZRsnCZmfhacYYKlbg9w/MdEJqMs2CwcsBydLyNaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X8BA7l1ZrNDl7sNhvj+In4v2gc1v/FzKdMp+FfH+Xcg=; b=Xsyv0HE4WYGpFDDQRik2j20KuO44AuTG31WPk9jKZoBBUp6uEMQSr6qjoh+yqzIWEzESjkIeQ00hokktOp7o1aaF6IQSXZRGHzhUq6sk5IiaXmw7MuQzs0hJVGixi/4fniOdDdWxEs1cdcRdoT9vVq8RWs7AtHU4cSiV1H6QRR4= Received: from PH0PR05MB10064.namprd05.prod.outlook.com (2603:10b6:510:29d::8) by CO6PR05MB7540.namprd05.prod.outlook.com (2603:10b6:5:340::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.26; Thu, 29 Jun 2023 09:51:11 +0000 Received: from PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593]) by PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593%4]) with mapi id 15.20.6500.026; Thu, 29 Jun 2023 09:51:11 +0000 From: Murali Krishnamurthy To: "freebsd-transport@FreeBSD.org" Subject: FreeBSD TCP (with iperf3) comparison with Linux Thread-Topic: FreeBSD TCP (with iperf3) comparison with Linux Thread-Index: AQHZjxV6iK08kPjzWEy1GrMDiqKRKg== Date: Thu, 29 Jun 2023 09:51:11 +0000 Message-ID: Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR05MB10064:EE_|CO6PR05MB7540:EE_ x-ms-office365-filtering-correlation-id: 40d6ec62-a152-4fcb-9073-08db78865eb8 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: De3bp0LWh3FLNUXw2s7XQ4/C+oxtOjAV0ismj8g0JrxGESQB5krp0U8cmgYIXIRthHqqmxJkQsyce9eK8XnFQUNSTIkexnJhpagpjuDG2H0sTfs01RcPYmp8ZXzXW+FobIr06Rh0QyKDTFF7shOyznm5+jyPn14AJWDUtJ8DEpS6M8Ba7vwCQCl6rc0CGHxnEXunCBDNqYBTJ4TkTrpkxpvtM/jDKrEezGBG/tYFEU4AyCgMjbBgqk7+2kxavKeV8pD1J3xARdrh/HH3z1HLVA0qgBbhflZL8ZDXQGr1Mnq8XNXJHau4jEIhWyskrfrVtFR3ml6apy/CvSCcNgvMxdKBnjLr1V2ksXqtdVRTHYal8gz15OBAQUEh1rqK0+ovNOl0758nG+gFoY+iiwLXDsmzpvSXzq3OAloomZbFeldjmxji3C6yFcMZ9cTdj8NsTpggOJpVFzXD4oyPipz8drL+/vMqBJnRXVcovuisGz2aBFVoDF3mXk4qv9IedQZ8iGGImepZ/5Erc/wxJwoBXK2j0Vb/ytPIVlTN56xBR9vUrsw2UVtUIDmuxuAB7XX5ansTHCq/DURmWY0G/am0n3YJGdNkNBNfUXLq+VZP+2sewOr/AOzaR0byxq22FMo/8P1SUWuyZemWwRc2wZB9yur3xERXkaFi/jx26gkjxOY= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR05MB10064.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(39860400002)(346002)(136003)(396003)(376002)(366004)(451199021)(38070700005)(122000001)(38100700002)(83380400001)(33656002)(86362001)(9686003)(71200400001)(91956017)(41300700001)(66946007)(316002)(66556008)(55016003)(8936002)(8676002)(66446008)(76116006)(26005)(6506007)(6916009)(7696005)(186003)(66476007)(64756008)(2906002)(66899021)(52536014)(478600001)(5660300002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?KJsodru4GrFomic3MPdfUlQnEqN1ec1b9jIJi073WEhxb2FwOZzp8zFSM7DF?= =?us-ascii?Q?eVuNAvZeWwhv2QynjUfqf2tDOMhY8yKjf1fhrjpT4+mUwUOyJUJvFIEsfEMv?= =?us-ascii?Q?2pAJfD0drdkoy48Qv/8UMmE7oHbNdC1nBCY4AzWInavtYJ1Hh6VUeMgIvruB?= =?us-ascii?Q?nW9xBT1EikeE+dpTiWqL6G7tmBm6D21ftMJhahWwSvf73NkcATCBdWpf7Dnr?= =?us-ascii?Q?ZWREcf3ZVDmlc14oKMp8Uu7/8+cjbwG1kas+6n2oOs1afwfns66jAKHK5xrn?= =?us-ascii?Q?Go75YUeIFHBEWMd3w6GzHX/btc8iao09++74WjQSLgeLTxcGZ/4fvl3WQkVd?= =?us-ascii?Q?rhhpiH4fqIBU4uadRLRah57G5KC6h+CKHJfEz9vCzFVgelpSFMNfru+Eml0O?= =?us-ascii?Q?1B9NctzCYqvK7kTvLkgCYz9jFhYsrZj0SskmnJbBR291Gnt+tQRqS7xBd7U/?= =?us-ascii?Q?3eBqNXuQyvKcRhJNMZpSOp4RV4PXU8fcDNlqoUZSdwcp3j6VNfqwjW8/b1Bd?= =?us-ascii?Q?3wokQs0RArkciLot1N+wupd9mIgFr8aJ+CIfsiJGepgaE+pQFrBQXvtZVTjH?= =?us-ascii?Q?cgmBmlwhGHRiDlAhqufwQqr8423aawbNK+DogI/HlZlHQaN+p/oN+u1fuppB?= =?us-ascii?Q?o/U1wUUj0khdyKcNuwtgwzXWWMTwHCFyeGVtV+EprFo60bqZQ2CQ7agjAEzK?= =?us-ascii?Q?Ff7GGSR+KvPBTLzXd41DFcxD5yPOyO9CxkgET8JltBk+qocZaecGOqwxER2w?= =?us-ascii?Q?IJd3esPPAtgH4hhkAVrlE8FFkLY9ef34vtPwZWu74VG3ULlhcv0UIoCWlokX?= =?us-ascii?Q?RPlrxetX3xaQES6lt2SWH0GNS6oCcuTjkRm3I1iiyLhFXLUHmaxAfenXxs/E?= =?us-ascii?Q?OTJhc7J2GTsZAREdCd2jyIAkuMrtjHfze5Z2hlDq2c6r/EeBEmwfHNsdPQ5R?= =?us-ascii?Q?KoDi628OKv/CpwzZKiiWCjZfhjQyXhN6q50fEpO8zjpnXn0HCZI1tXtcquX/?= =?us-ascii?Q?xukxPEFqhQe7JNpNns+4Z9/1gYGbHZxHBcNcr9MkEmZy/lktPOWhQIfqpmib?= =?us-ascii?Q?M4RyOF7bK0Pi316TMVBaYkJo5H7eQqWiFxgXcrXMzHppQaaBGcySs5BRJ1Na?= =?us-ascii?Q?R0W7jli5Qbb/Q9HuZG9sRp9+8MTaZX5u2Uhk7pus3TDmm68qUclGtYSwk3MZ?= =?us-ascii?Q?92m8+yUyEGrdHcos+ZVeyyN/mVz5+rom04LA5/+PnOYAS2IL+AUkbEVYPwYq?= =?us-ascii?Q?LpTsufFEhMZhb6uNbUJjGhVzw0UJ8ZFJ8m+FAJqLcqokA0sr/p9bHgbXbLOz?= =?us-ascii?Q?RTf9MQur9aEosr/V6XpZfsm0JA37tud4pmJy0zqXwXxj9kIdl8GaIH+q/aDG?= =?us-ascii?Q?FxXYGjHVeDduE57yUJXzM6u2g1gGEAnBOIuwWkM/kGGjLTchJKQKIvsPGwGi?= =?us-ascii?Q?NKiwfJY0SgBzCyoz6uRJGiVXybiWzk3PJAUNxbh/MoDjFmkKuOBXyyaOqBSH?= =?us-ascii?Q?e11xMDcVCuXiPvHzpgwZ5Q3dx7whlLDsGqDts3oXpsqCC4nbvb7OQ1NvHQNk?= =?us-ascii?Q?FqqtzQlLFhofmLfbAQhjM5wkmSKsikewra3hwK3klG2RAptxzh7YUUoAd0pN?= =?us-ascii?Q?aA=3D=3D?= Content-Type: multipart/alternative; boundary="_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_" List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR05MB10064.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 40d6ec62-a152-4fcb-9073-08db78865eb8 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jun 2023 09:51:11.2762 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: MuemmuWp888ds0lYDI4bAShD7FNipkFf+F0iqHjwXdl5Q9drNaAZA8AhJJE7hQO98ITsmXC2ZlyFujiJUsDQsA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR05MB7540 X-Spamd-Result: default: False [-11.50 / 15.00]; DWL_DNSWL_HI(-3.50)[vmware.com:dkim]; WHITELIST_SPF_DKIM(-3.00)[vmware.com:d:+,vmware.com:s:+]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[vmware.com,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:52.100.0.0/14]; R_DKIM_ALLOW(-0.20)[vmware.com:s=selector2]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-transport@FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:52.96.0.0/12, country:US]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[vmware.com:+]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[52.101.49.6:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RWL_MAILSPIKE_POSSIBLE(0.00)[52.101.49.6:from] X-Rspamd-Queue-Id: 4QsDH650pCz4W7q X-Spamd-Bar: ----------- X-ThisMailContainsUnwantedMimeParts: N --_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello FreeBSD Transport experts, We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in long RT= T setup and happened to compare the performance with Linux VM with same hyp= ervisor. We see a substantially better performance with Linux getting close to the B= DP limit, whereas BSD 13 not filling up the pipe enough. We are trying to figure out what could lead to such a huge difference and f= eel we could be missing something here. Could you please help us to know if there is a way to make it perform bette= r? Setup details: We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu 23.04/= Linux kernel 6.2) were launched on each hypervisor. Then we ran iperf between, 1. BSD 13 <-> BSD 13 2. Ubuntu <-> Ubuntu Even though the network environment were same in both cases, we see Ubuntu = performing much better. Below are connection parameters: Socket buffer: 16MB TCP CC Algo: Cubic. We used this as this is suitable for Long Fat Networks. Ping RTT: 100 ms between the two end points. We kept all other parameters to default on both Linux and BSD. BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 102= 4 =3D 1.25 Gbps Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP limi= t. FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So it s= eems to be doing half as good as Linux. For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same and = able to meet BDP of 300 Mbps consistently. Larger socket buffer seems to have an issue. Please let us know if there are ways to fine tune the system parameters to = make BSD perform better. Or any other suggestions/queries welcome. Regards Murali --_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello FreeBSD Transport experts= ,

 

We are evaluating performance o= f FreeBSD 13 VM on ESX hypervisor in long RTT setup and happened to compare= the performance with Linux VM with same hypervisor.

We see a substantially better p= erformance with Linux getting close to the BDP limit, whereas BSD 13 not fi= lling up the pipe enough.

We are trying to figure out wha= t could lead to such a huge difference and feel we could be missing somethi= ng here.

 

Could you please help us to kno= w if there is a way to make it perform better?

 

Setup details:

 

We have 2 ESX hypervisors where= 2 VMs (one FreeBSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched= on each hypervisor.

Then we ran iperf between,=

  1. BSD 13 <-> BSD 13
    2. Ubuntu <-> Ubuntu

     

    Even though the network environ= ment were same in both cases, we see Ubuntu performing much better.

     

    Below are connection parameters= :

    Socket buffer: 16MB<= /span>

    TCP CC Algo: Cubic. We used thi= s as this is suitable for Long Fat Networks.

    Ping RTT:  100 ms between = the two end points.

    We kept all other parameters to= default on both Linux and BSD.

     

    BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 1024 =3D 1.25 Gbps

     

    Ubuntu consistently hits around= 1 Gbps Bitrate almost reaching the BDP limit.

    FreeBSD 13 shows a Bit rate bet= ween the range of 300-600 Mbps = only. So it seems to be doing half as good as Linux.

    For lower socket buffer of 4MB,= both FreeBSD and Linux seem to do same and able to meet BDP of 300 Mbps co= nsistently.

    Larger socket buffer seems to h= ave an issue.

     

    Please let us know if there are= ways to fine tune the system parameters to make BSD perform better.

    Or any other suggestions/querie= s welcome.

     

    Regards

    Murali

--_000_PH0PR05MB10064288C78F8F3A0116D023DFB469PH0PR05MB10064na_-- From nobody Thu Jun 29 11:46:00 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsGqc5pqdz4kXHc for ; Thu, 29 Jun 2023 11:46:04 +0000 (UTC) (envelope-from rrs@netflix.com) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsGqc4XN7z3qGk for ; Thu, 29 Jun 2023 11:46:04 +0000 (UTC) (envelope-from rrs@netflix.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-55b1238cab4so312278a12.2 for ; Thu, 29 Jun 2023 04:46:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netflix.com; s=google; t=1688039163; x=1690631163; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=wghCVvSr25nh1VMnd282hqjxtVfrg9k40NQkh/Fce+I=; b=pdXo36hx59Q+v9B0Tm2rnwSljdVTfnZc92xAk/D3H+uCneOUiCV7kkITixBnxmxpNk i9Lv7kOeze7HewiWwKGSvI7TsAFSAHhuokx+ZfLU/PDjwM4zmc50fihjs40mOLzKZ9ga rZ5EuTA/ZtDXvMfzK2PjIyFvXzhCZmAtJgpTQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688039163; x=1690631163; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wghCVvSr25nh1VMnd282hqjxtVfrg9k40NQkh/Fce+I=; b=GB3aSnQPAmSzLMzaaWpJvhK1tzsDBkNmkbVSwvubT1WqXWQlrcA3zFnF6KYOsio7WT c7RbhgJzJUnAXGACDJcDoAIACdhy1XDSpLDgkzOnEjUdMdStFLWPtRzJfDFTcIvj/Bgi R+pXWHXSkdGqyURuSTDBl8kXZvt9bZFcqwkXxhYm3JKds7A+BzSGMp6Tlm5xnepkgXaU Ei1PnpolxCVHoqCiNjgwDRcgRmcDYZFiL082iYqJtQH+vOjiNQf5ngfp4aVcLVCFJmRg jN+VO6muamioWo3bYxUQLebtBpdciphdxa0BU0IiH+fqLwRSBZ/thWNWHKP0YifWYoeM Wrgw== X-Gm-Message-State: AC+VfDx/d+qJk/iswH/LbWeUcnmxb3gbWkEkDjcJZwasMeqws4ZTGUKR xgVu4NHIxahFyqtbdnICSflz2/SNr2dkzVii8p4= X-Google-Smtp-Source: ACHHUZ4uIoaqyP7JbyAs31aVpwq/XDDrNzobDpaIqzY98mQNzJ4lMW3altS22KubJ8ZlSfufQCR2iw== X-Received: by 2002:a05:6a20:7f8a:b0:118:2e5a:efab with SMTP id d10-20020a056a207f8a00b001182e5aefabmr37205977pzj.4.1688039163152; Thu, 29 Jun 2023 04:46:03 -0700 (PDT) Received: from smtpclient.apple (072-239-139-254.res.spectrum.com. [72.239.139.254]) by smtp.gmail.com with ESMTPSA id x18-20020a170902ec9200b001b66a71a4a0sm4171361plg.32.2023.06.29.04.46.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Jun 2023 04:46:02 -0700 (PDT) From: Randall Stewart Message-Id: <089C6BBB-C978-46DE-9479-1285B112E90D@netflix.com> Content-Type: multipart/signed; boundary="Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036"; protocol="application/pkcs7-signature"; micalg=sha-256 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.3\)) Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Date: Thu, 29 Jun 2023 07:46:00 -0400 In-Reply-To: Cc: "freebsd-transport@FreeBSD.org" To: Murali Krishnamurthy References: X-Mailer: Apple Mail (2.3696.120.41.1.3) X-Rspamd-Queue-Id: 4QsGqc4XN7z3qGk X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036 Content-Type: multipart/alternative; boundary="Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707" --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Greetings Murali: So I am unclear from your =E2=80=9CSocket buffer=E2=80=9D as to what you = did.. Did you set both the send and receive windows to 16Mbps with the SO_SNDBUF and = SO_RCVBUF options? Or were you just using auto-scaling to have the socket buffers advance? Thanks R > On Jun 29, 2023, at 5:51 AM, Murali Krishnamurthy = wrote: >=20 > Hello FreeBSD Transport experts, > =20 > We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in = long RTT setup and happened to compare the performance with Linux VM = with same hypervisor. > We see a substantially better performance with Linux getting close to = the BDP limit, whereas BSD 13 not filling up the pipe enough. > We are trying to figure out what could lead to such a huge difference = and feel we could be missing something here. > =20 > Could you please help us to know if there is a way to make it perform = better? > =20 > Setup details: > =20 > We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu = 23.04/Linux kernel 6.2) were launched on each hypervisor. > Then we ran iperf between, > BSD 13 <-> BSD 13 > Ubuntu <-> Ubuntu > =20 > Even though the network environment were same in both cases, we see = Ubuntu performing much better. > =20 > Below are connection parameters: > Socket buffer: 16MB > TCP CC Algo: Cubic. We used this as this is suitable for Long Fat = Networks. > Ping RTT: 100 ms between the two end points. > We kept all other parameters to default on both Linux and BSD. > =20 > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 = bits/ 1024 =3D 1.25 Gbps > =20 > Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP = limit. > FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So = it seems to be doing half as good as Linux. > For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same = and able to meet BDP of 300 Mbps consistently. > Larger socket buffer seems to have an issue. > =20 > Please let us know if there are ways to fine tune the system = parameters to make BSD perform better. > Or any other suggestions/queries welcome. > =20 > Regards > Murali ------ Randall Stewart rrs@netflix.com --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Greetings Murali:

So I am unclear from your =E2=80=9CSocket buffer=E2=80=9D as = to what you did.. Did you
set both the send and = receive windows to 16Mbps with the SO_SNDBUF and SO_RCVBUF = options?
Or were you just using auto-scaling to = have the socket buffers advance?

Thanks

R

On Jun = 29, 2023, at 5:51 AM, Murali Krishnamurthy <muralik1@vmware.com>= wrote:

Hello FreeBSD = Transport experts,
 
We are = evaluating performance of FreeBSD 13 VM on ESX hypervisor in long RTT = setup and happened to compare the performance with Linux VM with same = hypervisor.
We see a substantially better performance with = Linux getting close to the BDP limit, whereas BSD 13 not filling up the = pipe enough.
We are trying to figure out what could lead to = such a huge difference and feel we could be missing something here.
 
Could you please help us to know if there is a = way to make it perform better?
 
Setup details:
 
We have 2 ESX hypervisors where 2 VMs (one = FreeBSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched on each = hypervisor.
Then we ran iperf between,
  1. BSD = 13 <-> BSD 13
  2. Ubuntu = <-> Ubuntu
 
Even though the network environment were same = in both cases, we see Ubuntu performing much better.
 
Below are connection parameters:
Socket buffer: 16MB
TCP CC Algo: = Cubic. We used this as this is suitable for Long Fat Networks.
Ping RTT:  100 ms between the two end points.
We kept all other parameters to default on both Linux and = BSD.
 
BDP for = 16MB Socket = buffer: 16 MB * (1000 ms * 100ms = latency) * 8 bits/ 1024 =3D 1.25 = Gbps
 
Ubuntu consistently hits around 1 Gbps Bitrate = almost reaching the BDP limit.
FreeBSD 13 shows = a Bit rate between the range of 300-600 = Mbps only. So it seems to be doing half = as good as Linux.
For lower socket buffer of 4MB, both FreeBSD = and Linux seem to do same and able to meet BDP of 300 Mbps = consistently.
Larger socket buffer seems to have an = issue.
 
Please let us = know if there are ways to fine tune the system parameters to make BSD = perform better.
Or any other suggestions/queries welcome.
 
Regards
Murali

------
Randall = Stewart



= --Apple-Mail=_D4C7D065-0C42-45C8-9FF3-05D3B02AB707-- --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCAzYw ggMyMIICGqADAgECAgqxywKqrHPB2ybTMA0GCSqGSIb3DQEBCwUAMEcxGDAWBgNVBAMTD1JhbmRh bGwgU3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzAe Fw0yMTAxMjQxMjIwMTRaFw0yNjAxMjQxMjIwMTRaMEcxGDAWBgNVBAMTD1JhbmRhbGwgU3Rld2Fy dDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUzCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAMUAht2nr/NFlK+tmmN9PdO3DBPfeYh9fLcbVihR+/dipO41 AsFy9y+2uDVaFhTEvp406P0o9PQQTuYXqrCr76eWQIj3V787e1WKjTup1mIyQeWHGf1gvb/7vmI2 zHg6QZEIC4W8xeO8SLKyHiwlFHZn8Rn1HxtB7Ge+NulygkgUgJYhXD5E29jVGXAc6Qn9Vr9AexPf KaOhHCaNB/Twcinayz6D8CO/Ym1LOs3+ceSOa4cB07fepmbqDSXDkOeA3U7KLaluHrRTlj6DO+JU nqKXX7jJ68KTYSZ0qH4fZsk8cxFkwYI/3HDJi+oF+FDkf7SRo1Q2w+e3M/5MReLIQ7sCAwEAAaMg MB4wDwYJKoZIhvcvAQEKBAIFADALBgNVHQ8EBAMCB4AwDQYJKoZIhvcNAQELBQADggEBAHJfum1j 1WIVFjOJT/hqMIN751aXkablmwesW94lNJKjslPULbbcP5nZGg2lGpHcZ+0I5F/1TTiEsT2H2rhA uAnSsUxTpxRA+aoe+xtJOa5vle3CprhFkHAvB7EIoLiNaPd0DNK6kKYsbvr5Z5Eq7TF3SIO77Qh9 /8VgUfOb0ARDgix59Q6MM0NmIabEwh0cDWQYlGgDDtN9DNk5PGM4pjs48suwEdLmFTMOmGTkCp7I Vq6iHDNinBiB6+BB4VYMAO1o9qS+0pnfdmPJybt0zVGrhm/c1Fmm3Jec7NEuiKeXmhPIwdwMkKyp AsX0sHuFCYwioBTHHZpvnir+H2rRakgxggHrMIIB5wIBATBVMEcxGDAWBgNVBAMTD1JhbmRhbGwg U3Rld2FydDEeMBwGCSqGSIb3DQEJARYPcnJzQG5ldGZsaXguY29tMQswCQYDVQQGEwJVUwIKscsC qqxzwdsm0zANBglghkgBZQMEAgEFAKBpMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI hvcNAQkFMQ8XDTIzMDYyOTExNDYwMFowLwYJKoZIhvcNAQkEMSIEILaabTtURq4atG0MExNLW/wO 8Ouv2lkNCnBERFs6wJwSMA0GCSqGSIb3DQEBCwUABIIBAIdtF+DXNOHZ2pJB1YkkLSaNc3uTedsy w45aDrz//YcaClhr/KWJnZ2vStcGv1O6tqXQEmfXPYACQFB/iOwsIahe8NjY71ysxl1f+37JEmnY 1GlOEIkjSoHr7JPmPDvRGtbaVQWsjeLDXJ5fZ1cUOmKB2h1g+sBdD4fTITRfN9z2JTDjWkVlSdw0 BvI1H+jke17uLoGLqVtPPNyW3NpWoKaTWz+1Ug7Bk/Qkz14Yhcq8elF0w+IZ/ySZQc32j9Hhjwl/ xOFH/xpyxgNx8yFEOJFNpwurBpGvfO14uEAzAxJ8DZva9sn6C5VfoQKa+ApkYlvo2xCib/tlT2on tra52DwAAAAAAAA= --Apple-Mail=_F3B8C638-6298-4CA3-B37E-159AD1A4F036-- From nobody Thu Jun 29 12:52:44 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsJJl1TRlz4k0rt for ; Thu, 29 Jun 2023 12:52:55 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsJJk6g2lz4DtB for ; Thu, 29 Jun 2023 12:52:54 +0000 (UTC) (envelope-from tuexen@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (unknown [IPv6:2a02:8109:1140:c3d:35f7:d883:641f:3e25]) (Authenticated sender: micmac) by mail-n.franken.de (Postfix) with ESMTPSA id 1F0B38008A033; Thu, 29 Jun 2023 14:52:45 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux From: tuexen@freebsd.org In-Reply-To: Date: Thu, 29 Jun 2023 14:52:44 +0200 Cc: "freebsd-transport@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: <19F174D3-F0DB-4CD0-B43C-7A3BD8F47AFF@freebsd.org> References: To: Murali Krishnamurthy X-Mailer: Apple Mail (2.3731.600.7) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: 4QsJJk6g2lz4DtB X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On 29. Jun 2023, at 11:51, Murali Krishnamurthy = wrote: >=20 > Hello FreeBSD Transport experts, > We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in = long RTT setup and happened to compare the performance with Linux VM = with same hypervisor. > We see a substantially better performance with Linux getting close to = the BDP limit, whereas BSD 13 not filling up the pipe enough. > We are trying to figure out what could lead to such a huge difference = and feel we could be missing something here. > Could you please help us to know if there is a way to make it perform = better? > Setup details: > We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu = 23.04/Linux kernel 6.2) were launched on each hypervisor. > Then we ran iperf between, > =E2=80=A2 BSD 13 <-> BSD 13 > =E2=80=A2 Ubuntu <-> Ubuntu > Even though the network environment were same in both cases, we see = Ubuntu performing much better. > Below are connection parameters: > Socket buffer: 16MB > TCP CC Algo: Cubic. We used this as this is suitable for Long Fat = Networks. > Ping RTT: 100 ms between the two end points. > We kept all other parameters to default on both Linux and BSD. > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 = bits/ 1024 =3D 1.25 Gbps > Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP = limit. > FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So = it seems to be doing half as good as Linux. > For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same = and able to meet BDP of 300 Mbps consistently. > Larger socket buffer seems to have an issue. > Please let us know if there are ways to fine tune the system = parameters to make BSD perform better. > Or any other suggestions/queries welcome. > Regards > Murali Hi Murali, I'll bring this up on the biweekly transport call. Best regards Michael From nobody Thu Jun 29 14:03:32 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsKtK0NVJz4kmf9 for ; Thu, 29 Jun 2023 14:03:37 +0000 (UTC) (envelope-from muralik1@vmware.com) Received: from DM4PR02CU001.outbound.protection.outlook.com (mail-centralusazlp170120005.outbound.protection.outlook.com [IPv6:2a01:111:f403:c111::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsKtJ2tjDz3MCZ for ; Thu, 29 Jun 2023 14:03:36 +0000 (UTC) (envelope-from muralik1@vmware.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NGykbFw/8My/XIy1nrr8h48zKNS3fu7TecHMQ7PHHD9zaLl/OIPxgYO5wvzZVjp3FISugCtTnSnk6fjsgcL+64tjOKHTe93LMuKNH3/X2yqHDsYvIYLkBfuAGRKhuFGPi7h39pUSEE/Sf8/tQflerEkycohq4nDsflx90fnG134i+qzVaU337PXtVCx7oqk9ENQZLu4ErN5PbjWyCbYokmjUHXTa3NBu9aihdpkHGqils4fRdYRYoNqzMMabjEHJem8P1FTk7vlG3V5+8lXksfC0EPaRmcEV/3IiAAENhZTiER49q2IgjiitYhAynnWmm8v3aiigTy9zBCnNxoTrog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SgDfiAyA7xD+Dyka0Wmdk63Zs2BHgP+dWBZMRemyPBw=; b=MB481WcqKzbHKairhplUsYbuuUtW13TbmgsamY3pMdjlgw+6xTMm7+BqAXgdarS05O2mUKEKP86EbTkI+JUqwabXIr0Zcr2qkSwoY6avRIvZ971COm3HdTBwofG4PnfSuw6YM1va/glfQ1XpB2a6ILYZxAJVJFFabbpr2krgTXScq+OJ4NX2METo7oDf1gLAl2UnYA/pit1g3b0BVrbUz3wHgpNtR05HeWkuRfHmqfgp12irIlKTzyGd2jMQNBLrITBZQ/NO85FWcmW7O4KwMtaEDWFRW7hFjnL5MUf/m6psX3941La7QBGkHRdOMY8C65+dUYD6g+u9LJOqI+OqpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SgDfiAyA7xD+Dyka0Wmdk63Zs2BHgP+dWBZMRemyPBw=; b=Tq6E43y2hUDN+1FMjvZwRcuvhEAc5xOZ1J7Gxq799j1eSGzHjf8K2iExiVpF+DNdREYIOekQTPl2ptvY2kew7mjNxTdaZI2zynPAFdG8afDKCEvNiKfI0Z2L9xOdwORJHFGdHbsukn2bw4VekWzVe6hfhH57ZXeQR+o4BsIGnDo= Received: from PH0PR05MB10064.namprd05.prod.outlook.com (2603:10b6:510:29d::8) by BY3PR05MB7907.namprd05.prod.outlook.com (2603:10b6:a03:363::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.15; Thu, 29 Jun 2023 14:03:32 +0000 Received: from PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593]) by PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593%4]) with mapi id 15.20.6500.026; Thu, 29 Jun 2023 14:03:32 +0000 From: Murali Krishnamurthy To: Randall Stewart CC: "freebsd-transport@FreeBSD.org" Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Thread-Topic: FreeBSD TCP (with iperf3) comparison with Linux Thread-Index: AQHZjxV6iK08kPjzWEy1GrMDiqKRKq+h4GEAgAAksV8= Date: Thu, 29 Jun 2023 14:03:32 +0000 Message-ID: References: <089C6BBB-C978-46DE-9479-1285B112E90D@netflix.com> In-Reply-To: <089C6BBB-C978-46DE-9479-1285B112E90D@netflix.com> Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR05MB10064:EE_|BY3PR05MB7907:EE_ x-ms-office365-filtering-correlation-id: d234f4a3-707c-48b7-f84e-08db78a99f64 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Qam+VrOU49XQYYyv6HvE5P0zeb9kZW++mUFCmeTqnfyNnJNjDHpQbfq4+QBR4h+eZ41XfpXsiIzwKihTERKIK5bStrvZJS7dLLRV4+SCgIFUQZBwxWlpsfsY6CEfbTBUMti98PlGusTbqfDPnnjM7zmBX+xH5qWPB4U47NT4i0rQ2iIsOcbNuHzKSohyDelrGGDfVXf1HmBpnZnvnmTNSyJuvlrAPb6VDSmoW33rZRIeUgRSsA2Xil446bEID+sqVj37xQFrfVW/tNWDnQqQRTIJMcKwdNa4tY8Zp8oP22zgRGSOGwOmiJxzayXG9Fp0K8usC3gMouUoSWj6OhbS3GZ7jlrHEVtU+3up3WanR7JGwVJdeAYBv5gW0n+4E/oWCj+iLP7Yoerd4RD8ppwS8ieZhNAk45VMEgIYJUyRMtXPvCD2Jb37g64ZD68U+I3/YJZh5P0a61Cr15RXsefdmFbXQ9hRKsms3Ehh7MmPierykx4LNoEBngK7tNmw9O68rUT7GfNI2hhNAvPNTSZ86VS2OqsqD98+AO6r2EIqzG0Bq3z5MgYxy2Vf/26qywjRG9m/IiQbcretKbtSaDk8BzxRjwzyCb547KRhqV2fe+lW2q1lts5eipMvWlHbAqxLidke2p1k2WlsLRnNi0Rs1oeRBOoCZWzyCLoTgC/WX78= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR05MB10064.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(376002)(346002)(396003)(39860400002)(136003)(366004)(451199021)(55016003)(4326008)(2906002)(186003)(83380400001)(316002)(64756008)(6916009)(66946007)(91956017)(66476007)(66556008)(66446008)(71200400001)(7696005)(6506007)(5660300002)(478600001)(9686003)(26005)(8936002)(53546011)(41300700001)(8676002)(52536014)(38100700002)(122000001)(86362001)(33656002)(38070700005)(66899021)(76116006);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?NXY3SUJ0UXRWbGg0VWpacFpzT1dDcVRJWC9DQzhCL3VNcHdLZm5wQ0RyYno3?= =?utf-8?B?cWFZYXhGbjJFTGJFQktCMzNYbnhiWmNlVzllcHZWUnZ6WGxmcHFiaFZXc2ht?= =?utf-8?B?SDlTamhsenBFaGVvaS9FL3psTm5mOHZHSmJZZCtGK1UyRnZIRFEya1d1QlBI?= =?utf-8?B?dE5HY01DMk1jWVJsUjd2dmMyNmRZZEJ6MEtKUlNIMi9aNy8wN1I3cTFMQWUy?= =?utf-8?B?aVJVaWFhTVZqRnVENndEZ2I1aGEzZnhKNUxWOTBieXE4OUtKZHRmYVdVMUNH?= =?utf-8?B?QnBDNk9pSmNPUzNmRmR4NWY4VjZFdUMya2EzZTN6b0V2YXVDeE1GRWFaQnlZ?= =?utf-8?B?R2xyLy8xMEhzM2RzQjc2L21IZmhnV3RoNDhtTHB6c3lTMk9WbWlYUE9Pc0hO?= =?utf-8?B?ZFMvVEx1RDc1Mi9EY0NNVlh6cG5DeEtrcHBnSWdLR1JvZ3lZRDY1alBtTjZa?= =?utf-8?B?RGg0OGtFR0ZWR0R4UURUL0w4T0hzd204cVRKY1JCc3VsUjBTZU5GanZzTlQw?= =?utf-8?B?eldidGJYTEExUDRPbWY0c2N3WFlxTDZ0UlNNT0JNRCtPbnV2TitteXMvUUFG?= =?utf-8?B?ZWdQeENSTDMzVjZOelpMcDhQL0tTa295cU5rb2I1L200RGpkenlhKzdtUHpP?= =?utf-8?B?RTlvK3JoZU9aRWgrUmtCQ1p3SmVjajRJbHo3S3YvL0Y2dysvNHVFNE1QSEpX?= =?utf-8?B?TEE0NHhMRm4xM0JTck41aU1WS1VJK2tsMWZ5WmRyY1BscC9HSGhnOXF0S3NL?= =?utf-8?B?NWk1VjYyN2c3a0lhMGwyQ2lyc01QbU5Kc00xS29EcE0xay9UQno3STdSMDhD?= =?utf-8?B?VzJwYkRpNUxGUVh1eEdFRU9JQXJJNURZTzU0bU9UcWxNazZRMUt5WHFoTTdU?= =?utf-8?B?NUMrQjVGQjkwcWY4RXZVZ2FzeUIxd294Q1FtbjhORGRUUWsrc3ZEZkZ1Nzdy?= =?utf-8?B?Y2kvQlFSVm9WbEZKWXBGS2ZoK1kzbGdzdXVJOVR0MjVybGhqWjRHQXN2UGcv?= =?utf-8?B?L1RXeFQxZWVVNllqU1ZCNXd6c1dKQUJOLytMTEZKVU5hcENIWVdpVlpwY3Bx?= =?utf-8?B?ajZPS2xGbnBKQkJBT2FIL0pDT1doc3JYK05McWtjVXNMdUhsM0xqMXFyeGdO?= =?utf-8?B?Mm5QRWZTcXA4K0VEcmdHQUIzWkZPODlpeEhYekt1Si9aQ2JYeXpjTkY3MlRm?= =?utf-8?B?Ym5mL0pDSUFsVzZJUWNSZFFoVXFFVTdmMGpnNjAzM215UWFSYVhLZjJVM3l2?= =?utf-8?B?RUpieVV3NnVqQkcrcHhtVEUxcW5HaDB1SDRlVWhwemFqUTNqazdRRkhJMURU?= =?utf-8?B?eGUzc0ZhTVE5a050RU5ydXNDckt1eDh0UWN5QkRPc1ZJdVgyU1BlUWFtV2xm?= =?utf-8?B?UVlZN2lBUkM0MkJTdlNPRVRVRFVDWjZibWZNQVB2YnBTeENMMHZFRHVHQmlw?= =?utf-8?B?bEZPcEhRdEpkV2FISG42ZlRvMENOKytGL2k2cTlIVzhENVpFcnNLVE9VTXJE?= =?utf-8?B?UU5HZHRtMzZsU1RDUEswdmNSTzV0bGt3NVpjSm94aGlpT0Nla2UwNUwrdzd3?= =?utf-8?B?bTYxbEQ5Z2VDQzRDVHNxMWdIcTFyRW9ySUxJSjAyU3BnN1plQndWQWkreThP?= =?utf-8?B?ZGYyamUrVU5WaDloYjRkcU5DaURYdHlZa3NYeWVGYnAxdnFHdlZWdld6OHFH?= =?utf-8?B?MEVTb3cvOFlseHI4Z3VTV0FFRktCT29UY2pzYUo1VGZrWURUUGVjM0s2elZq?= =?utf-8?B?UkxUTWRBVDVyeFgwS05ha1NKNlVBU2VXWmxpdm40NC8wRFJOQmVIMXJBUmtZ?= =?utf-8?B?UXBZMFFlbWV6Y0xaR0Zicjc1UkprQmd6ZVhMc25FMG5GbHAraXNyMzlYSkxm?= =?utf-8?B?UmIycFo4WUlrdUV4RlJuL2FOMFVOVDJrT3NNRy9qam1Dc29ESEtmSmY5aW1n?= =?utf-8?B?UytOZXBEcTBPVVVRTnNPL0hZQU5sZ1EvZjhYNW5taTA4TnkxWFJNN0hraXVN?= =?utf-8?B?REd2OWRMcjFOMUZNN0dlVjdNcnZ3ZmdXdlhmMlFWYWpqYWVubFQ5eFJvUFVm?= =?utf-8?B?d3U5VVJDR2xEUDZ6Z2F1SVlDaWUraHN4QXBUcGh3d0VLZmZQYitaajFOWng3?= =?utf-8?B?MHhzNzgwaFM0OVNzazgvVjl1R0tiMEhWQU9oZC9LN2RVQjBLMjFlS1Y3c0pX?= =?utf-8?B?bnc9PQ==?= Content-Type: multipart/alternative; boundary="_000_PH0PR05MB10064C83AAD620C3AB838FC6FFB25APH0PR05MB10064na_" List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR05MB10064.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: d234f4a3-707c-48b7-f84e-08db78a99f64 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jun 2023 14:03:32.2017 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: u6PMLiwMzwFXu5ST3MxnTDXudlFGqSkWo6PzjjL5qYGEelZZkqPECwvD7rHeMXXRlZn2lecaKAaQm1Yuw3Cj1w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR05MB7907 X-Rspamd-Queue-Id: 4QsKtJ2tjDz3MCZ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --_000_PH0PR05MB10064C83AAD620C3AB838FC6FFB25APH0PR05MB10064na_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 UmFuZGFsbCwNCg0KVGhhbmsgeW91IGZvciB5b3VyIHJlc3BvbnNlLg0KDQpJIHNldCBiZWxvdyBz eXNjdGxzDQoNCnN5c2N0bCBrZXJuLmlwYy5tYXhzb2NrYnVmPTE4ODc0MzY4ICAgICAgICAg4oCi IDE4TUINCg0Kc3lzY3RsIG5ldC5pbmV0LnRjcC5zZW5kYnVmX21heD0xODg3NDM2OA0Kc3lzY3Rs IG5ldC5pbmV0LnRjcC5yZWN2YnVmX21heD0xODg3NDM2OA0KDQoNCkxvYWRlZCBDdWJpYyBDQyBh bGdvcml0aG0sDQoNCjEuICAgICBMb2FkIHRoZSBtb2R1bGUgdXNpbmcNCg0Ka2xkbG9hZCBjY19j dWJpYw0KDQoNCg0KMi4gICAgIFNldCB0aGUgY2MgYWxnb3JpdGhtIGJ5DQoNCnN5c2N0bCBuZXQu aW5ldC50Y3AuY2MuYWxnb3JpdGhtPWN1YmljDQoNCg0KDQpCZWxvdyBpcyB0aGUgY29tbWFuZCBJ IHVzZWQgdG8gcnVuIGlwZXJmMw0KDQpzZXJ2ZXI6DQppcGVyZjMgLXMgLUIgMTkyLjE2OC4xLjgN CmNsaWVudDoNCmlwZXJmMyDigJNjIDE5Mi4xNjguMS44IC1CIDE5Mi4xNjguMi44IC10IDE4MCAt dyAxNm0gLUMgY3ViaWMNCg0KVGhpcyBzaG91bGQgc2V0IHRoZSBzb2NrZXQgYnVmZmVycyBmb3Ig dGhlIGNvbm5lY3Rpb25zIHRvIDE2TUIgYW5kIHJ1biB0aGUgdGVzdHMgZm9yIDE4MCBzZWNzLg0K DQoNCkhvcGUgdGhpcyBpcyBjbGVhci4NCg0KDQpSZWdhcmRzDQpNdXJhbGkNCg0KDQpGcm9tOiBS YW5kYWxsIFN0ZXdhcnQgPHJyc0BuZXRmbGl4LmNvbT4NCkRhdGU6IFRodXJzZGF5LCAyOSBKdW5l IDIwMjMgYXQgNToxNiBQTQ0KVG86IE11cmFsaSBLcmlzaG5hbXVydGh5IDxtdXJhbGlrMUB2bXdh cmUuY29tPg0KQ2M6IGZyZWVic2QtdHJhbnNwb3J0QEZyZWVCU0Qub3JnIDxmcmVlYnNkLXRyYW5z cG9ydEBGcmVlQlNELm9yZz4NClN1YmplY3Q6IFJlOiBGcmVlQlNEIFRDUCAod2l0aCBpcGVyZjMp IGNvbXBhcmlzb24gd2l0aCBMaW51eA0KR3JlZXRpbmdzIE11cmFsaToNCg0KU28gSSBhbSB1bmNs ZWFyIGZyb20geW91ciDigJxTb2NrZXQgYnVmZmVy4oCdIGFzIHRvIHdoYXQgeW91IGRpZC4uIERp ZCB5b3UNCnNldCBib3RoIHRoZSBzZW5kIGFuZCByZWNlaXZlIHdpbmRvd3MgdG8gMTZNYnBzIHdp dGggdGhlIFNPX1NOREJVRiBhbmQgU09fUkNWQlVGIG9wdGlvbnM/DQpPciB3ZXJlIHlvdSBqdXN0 IHVzaW5nIGF1dG8tc2NhbGluZyB0byBoYXZlIHRoZSBzb2NrZXQgYnVmZmVycyBhZHZhbmNlPw0K DQpUaGFua3MNCg0KUg0KDQoNCk9uIEp1biAyOSwgMjAyMywgYXQgNTo1MSBBTSwgTXVyYWxpIEty aXNobmFtdXJ0aHkgPG11cmFsaWsxQHZtd2FyZS5jb208bWFpbHRvOm11cmFsaWsxQHZtd2FyZS5j b20+PiB3cm90ZToNCg0KSGVsbG8gRnJlZUJTRCBUcmFuc3BvcnQgZXhwZXJ0cywNCg0KV2UgYXJl IGV2YWx1YXRpbmcgcGVyZm9ybWFuY2Ugb2YgRnJlZUJTRCAxMyBWTSBvbiBFU1ggaHlwZXJ2aXNv ciBpbiBsb25nIFJUVCBzZXR1cCBhbmQgaGFwcGVuZWQgdG8gY29tcGFyZSB0aGUgcGVyZm9ybWFu Y2Ugd2l0aCBMaW51eCBWTSB3aXRoIHNhbWUgaHlwZXJ2aXNvci4NCldlIHNlZSBhIHN1YnN0YW50 aWFsbHkgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggTGludXggZ2V0dGluZyBjbG9zZSB0byB0aGUg QkRQIGxpbWl0LCB3aGVyZWFzIEJTRCAxMyBub3QgZmlsbGluZyB1cCB0aGUgcGlwZSBlbm91Z2gu DQpXZSBhcmUgdHJ5aW5nIHRvIGZpZ3VyZSBvdXQgd2hhdCBjb3VsZCBsZWFkIHRvIHN1Y2ggYSBo dWdlIGRpZmZlcmVuY2UgYW5kIGZlZWwgd2UgY291bGQgYmUgbWlzc2luZyBzb21ldGhpbmcgaGVy ZS4NCg0KQ291bGQgeW91IHBsZWFzZSBoZWxwIHVzIHRvIGtub3cgaWYgdGhlcmUgaXMgYSB3YXkg dG8gbWFrZSBpdCBwZXJmb3JtIGJldHRlcj8NCg0KU2V0dXAgZGV0YWlsczoNCg0KV2UgaGF2ZSAy IEVTWCBoeXBlcnZpc29ycyB3aGVyZSAyIFZNcyAob25lIEZyZWVCU0QgMTMgYW5kIG9uZSBVYnVu dHUgMjMuMDQvTGludXgga2VybmVsIDYuMikgd2VyZSBsYXVuY2hlZCBvbiBlYWNoIGh5cGVydmlz b3IuDQpUaGVuIHdlIHJhbiBpcGVyZiBiZXR3ZWVuLA0KDQogIDEuICBCU0QgMTMgPC0+IEJTRCAx Mw0KICAyLiAgVWJ1bnR1IDwtPiBVYnVudHUNCg0KRXZlbiB0aG91Z2ggdGhlIG5ldHdvcmsgZW52 aXJvbm1lbnQgd2VyZSBzYW1lIGluIGJvdGggY2FzZXMsIHdlIHNlZSBVYnVudHUgcGVyZm9ybWlu ZyBtdWNoIGJldHRlci4NCg0KQmVsb3cgYXJlIGNvbm5lY3Rpb24gcGFyYW1ldGVyczoNClNvY2tl dCBidWZmZXI6IDE2TUINClRDUCBDQyBBbGdvOiBDdWJpYy4gV2UgdXNlZCB0aGlzIGFzIHRoaXMg aXMgc3VpdGFibGUgZm9yIExvbmcgRmF0IE5ldHdvcmtzLg0KUGluZyBSVFQ6ICAxMDAgbXMgYmV0 d2VlbiB0aGUgdHdvIGVuZCBwb2ludHMuDQpXZSBrZXB0IGFsbCBvdGhlciBwYXJhbWV0ZXJzIHRv IGRlZmF1bHQgb24gYm90aCBMaW51eCBhbmQgQlNELg0KDQpCRFAgZm9yIDE2TUIgU29ja2V0IGJ1 ZmZlcjogMTYgTUIgKiAoMTAwMCBtcyAqIDEwMG1zIGxhdGVuY3kpICogOCBiaXRzLyAxMDI0ID0g MS4yNSBHYnBzDQoNClVidW50dSBjb25zaXN0ZW50bHkgaGl0cyBhcm91bmQgMSBHYnBzIEJpdHJh dGUgYWxtb3N0IHJlYWNoaW5nIHRoZSBCRFAgbGltaXQuDQpGcmVlQlNEIDEzIHNob3dzIGEgQml0 IHJhdGUgYmV0d2VlbiB0aGUgcmFuZ2Ugb2YgMzAwLTYwMCBNYnBzIG9ubHkuIFNvIGl0IHNlZW1z IHRvIGJlIGRvaW5nIGhhbGYgYXMgZ29vZCBhcyBMaW51eC4NCkZvciBsb3dlciBzb2NrZXQgYnVm ZmVyIG9mIDRNQiwgYm90aCBGcmVlQlNEIGFuZCBMaW51eCBzZWVtIHRvIGRvIHNhbWUgYW5kIGFi bGUgdG8gbWVldCBCRFAgb2YgMzAwIE1icHMgY29uc2lzdGVudGx5Lg0KTGFyZ2VyIHNvY2tldCBi dWZmZXIgc2VlbXMgdG8gaGF2ZSBhbiBpc3N1ZS4NCg0KUGxlYXNlIGxldCB1cyBrbm93IGlmIHRo ZXJlIGFyZSB3YXlzIHRvIGZpbmUgdHVuZSB0aGUgc3lzdGVtIHBhcmFtZXRlcnMgdG8gbWFrZSBC U0QgcGVyZm9ybSBiZXR0ZXIuDQpPciBhbnkgb3RoZXIgc3VnZ2VzdGlvbnMvcXVlcmllcyB3ZWxj b21lLg0KDQpSZWdhcmRzDQpNdXJhbGkNCg0KLS0tLS0tDQpSYW5kYWxsIFN0ZXdhcnQNCnJyc0Bu ZXRmbGl4LmNvbTxtYWlsdG86cnJzQG5ldGZsaXguY29tPg0KDQoNCg0K --_000_PH0PR05MB10064C83AAD620C3AB838FC6FFB25APH0PR05MB10064na_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iR2VuZXJhdG9yIiBjb250ZW50PSJNaWNyb3NvZnQgV29yZCAxNSAoZmlsdGVyZWQg bWVkaXVtKSI+DQo8c3R5bGU+PCEtLQ0KLyogRm9udCBEZWZpbml0aW9ucyAqLw0KQGZvbnQtZmFj ZQ0KCXtmb250LWZhbWlseTpIZWx2ZXRpY2E7DQoJcGFub3NlLTE6MCAwIDAgMCAwIDAgMCAwIDAg MDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OldpbmdkaW5nczsNCglwYW5vc2UtMTo1IDAg MCAwIDAgMCAwIDAgMCAwO30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0 aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQt ZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7fQ0KQGZvbnQt ZmFjZQ0KCXtmb250LWZhbWlseToiU2Vnb2UgVUkiOw0KCXBhbm9zZS0xOjIgMTEgNSAyIDQgMiA0 IDIgMiAzO30NCi8qIFN0eWxlIERlZmluaXRpb25zICovDQpwLk1zb05vcm1hbCwgbGkuTXNvTm9y bWFsLCBkaXYuTXNvTm9ybWFsDQoJe21hcmdpbjowY207DQoJZm9udC1zaXplOjEwLjBwdDsNCglm b250LWZhbWlseToiQ2FsaWJyaSIsc2Fucy1zZXJpZjt9DQpoNA0KCXttc28tc3R5bGUtcHJpb3Jp dHk6OTsNCgltc28tc3R5bGUtbGluazoiSGVhZGluZyA0IENoYXIiOw0KCW1hcmdpbi10b3A6Mi4w cHQ7DQoJbWFyZ2luLXJpZ2h0OjBjbTsNCgltYXJnaW4tYm90dG9tOjBjbTsNCgltYXJnaW4tbGVm dDowY207DQoJcGFnZS1icmVhay1hZnRlcjphdm9pZDsNCglmb250LXNpemU6MTAuMHB0Ow0KCWZv bnQtZmFtaWx5OiJDYWxpYnJpIExpZ2h0IixzYW5zLXNlcmlmOw0KCWNvbG9yOiMyRjU0OTY7DQoJ Zm9udC13ZWlnaHQ6bm9ybWFsOw0KCWZvbnQtc3R5bGU6aXRhbGljO30NCmE6bGluaywgc3Bhbi5N c29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4 dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpwcmUNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0K CW1zby1zdHlsZS1saW5rOiJIVE1MIFByZWZvcm1hdHRlZCBDaGFyIjsNCgltYXJnaW46MGNtOw0K CWZvbnQtc2l6ZToxMC4wcHQ7DQoJZm9udC1mYW1pbHk6IkNvdXJpZXIgTmV3Ijt9DQpwLk1zb0xp c3RQYXJhZ3JhcGgsIGxpLk1zb0xpc3RQYXJhZ3JhcGgsIGRpdi5Nc29MaXN0UGFyYWdyYXBoDQoJ e21zby1zdHlsZS1wcmlvcml0eTozNDsNCgltc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzsNCgltYXJn aW4tcmlnaHQ6MGNtOw0KCW1zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvOw0KCW1hcmdpbi1sZWZ0 OjBjbTsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNl cmlmO30NCnNwYW4uYXBwbGUtY29udmVydGVkLXNwYWNlDQoJe21zby1zdHlsZS1uYW1lOmFwcGxl LWNvbnZlcnRlZC1zcGFjZTt9DQpzcGFuLkVtYWlsU3R5bGUyMg0KCXttc28tc3R5bGUtdHlwZTpw ZXJzb25hbC1yZXBseTsNCglmb250LWZhbWlseToiQ2FsaWJyaSIsc2Fucy1zZXJpZjsNCgljb2xv cjp3aW5kb3d0ZXh0O30NCnNwYW4uSFRNTFByZWZvcm1hdHRlZENoYXINCgl7bXNvLXN0eWxlLW5h bWU6IkhUTUwgUHJlZm9ybWF0dGVkIENoYXIiOw0KCW1zby1zdHlsZS1wcmlvcml0eTo5OTsNCglt c28tc3R5bGUtbGluazoiSFRNTCBQcmVmb3JtYXR0ZWQiOw0KCWZvbnQtZmFtaWx5OiJDb3VyaWVy IE5ldyI7fQ0Kc3Bhbi5IZWFkaW5nNENoYXINCgl7bXNvLXN0eWxlLW5hbWU6IkhlYWRpbmcgNCBD aGFyIjsNCgltc28tc3R5bGUtcHJpb3JpdHk6OTsNCgltc28tc3R5bGUtbGluazoiSGVhZGluZyA0 IjsNCglmb250LWZhbWlseToiQ2FsaWJyaSBMaWdodCIsc2Fucy1zZXJpZjsNCgljb2xvcjojMkY1 NDk2Ow0KCWZvbnQtc3R5bGU6aXRhbGljO30NCi5Nc29DaHBEZWZhdWx0DQoJe21zby1zdHlsZS10 eXBlOmV4cG9ydC1vbmx5Ow0KCWZvbnQtc2l6ZToxMC4wcHQ7DQoJbXNvLWxpZ2F0dXJlczpub25l O30NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXplOjYxMi4wcHQgNzkyLjBwdDsNCgltYXJnaW46 NzIuMHB0IDcyLjBwdCA3Mi4wcHQgNzIuMHB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpX b3JkU2VjdGlvbjE7fQ0KLyogTGlzdCBEZWZpbml0aW9ucyAqLw0KQGxpc3QgbDANCgl7bXNvLWxp c3QtaWQ6NTgwNDgyNTEwOw0KCW1zby1saXN0LXRlbXBsYXRlLWlkczoxOTM5Nzk5NzE2O30NCkBs aXN0IGwxDQoJe21zby1saXN0LWlkOjEyNjQ4MDAzMTQ7DQoJbXNvLWxpc3QtdGVtcGxhdGUtaWRz Oi0xNjEwMzI3OTY2O30NCkBsaXN0IGwxOmxldmVsMQ0KCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1h dDphbHBoYS1sb3dlcjsNCgltc28tbGV2ZWwtdGFiLXN0b3A6MzYuMHB0Ow0KCW1zby1sZXZlbC1u dW1iZXItcG9zaXRpb246bGVmdDsNCgl0ZXh0LWluZGVudDotMTguMHB0O30NCkBsaXN0IGwxOmxl dmVsMg0KCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDphbHBoYS1sb3dlcjsNCgltc28tbGV2ZWwt dGFiLXN0b3A6NzIuMHB0Ow0KCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsNCgl0ZXh0 LWluZGVudDotMTguMHB0O30NCkBsaXN0IGwxOmxldmVsMw0KCXttc28tbGV2ZWwtbnVtYmVyLWZv cm1hdDphbHBoYS1sb3dlcjsNCgltc28tbGV2ZWwtdGFiLXN0b3A6MTA4LjBwdDsNCgltc28tbGV2 ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7DQoJdGV4dC1pbmRlbnQ6LTE4LjBwdDt9DQpAbGlzdCBs MTpsZXZlbDQNCgl7bXNvLWxldmVsLW51bWJlci1mb3JtYXQ6YWxwaGEtbG93ZXI7DQoJbXNvLWxl dmVsLXRhYi1zdG9wOjE0NC4wcHQ7DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0Ow0K CXRleHQtaW5kZW50Oi0xOC4wcHQ7fQ0KQGxpc3QgbDE6bGV2ZWw1DQoJe21zby1sZXZlbC1udW1i ZXItZm9ybWF0OmFscGhhLWxvd2VyOw0KCW1zby1sZXZlbC10YWItc3RvcDoxODAuMHB0Ow0KCW1z by1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsNCgl0ZXh0LWluZGVudDotMTguMHB0O30NCkBs aXN0IGwxOmxldmVsNg0KCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDphbHBoYS1sb3dlcjsNCglt c28tbGV2ZWwtdGFiLXN0b3A6MjE2LjBwdDsNCgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxl ZnQ7DQoJdGV4dC1pbmRlbnQ6LTE4LjBwdDt9DQpAbGlzdCBsMTpsZXZlbDcNCgl7bXNvLWxldmVs LW51bWJlci1mb3JtYXQ6YWxwaGEtbG93ZXI7DQoJbXNvLWxldmVsLXRhYi1zdG9wOjI1Mi4wcHQ7 DQoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0Ow0KCXRleHQtaW5kZW50Oi0xOC4wcHQ7 fQ0KQGxpc3QgbDE6bGV2ZWw4DQoJe21zby1sZXZlbC1udW1iZXItZm9ybWF0OmFscGhhLWxvd2Vy Ow0KCW1zby1sZXZlbC10YWItc3RvcDoyODguMHB0Ow0KCW1zby1sZXZlbC1udW1iZXItcG9zaXRp b246bGVmdDsNCgl0ZXh0LWluZGVudDotMTguMHB0O30NCkBsaXN0IGwxOmxldmVsOQ0KCXttc28t bGV2ZWwtbnVtYmVyLWZvcm1hdDphbHBoYS1sb3dlcjsNCgltc28tbGV2ZWwtdGFiLXN0b3A6MzI0 LjBwdDsNCgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7DQoJdGV4dC1pbmRlbnQ6LTE4 LjBwdDt9DQpvbA0KCXttYXJnaW4tYm90dG9tOjBjbTt9DQp1bA0KCXttYXJnaW4tYm90dG9tOjBj bTt9DQotLT48L3N0eWxlPg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iRU4tSU4iIGxpbms9ImJsdWUi IHZsaW5rPSJwdXJwbGUiIHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZDstd2Via2l0LW5ic3At bW9kZTpzcGFjZTtsaW5lLWJyZWFrOmFmdGVyLXdoaXRlLXNwYWNlIj4NCjxkaXYgY2xhc3M9Ildv cmRTZWN0aW9uMSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjExLjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+UmFuZGFsbCw8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEx LjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7 bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPlRoYW5rIHlvdSBmb3IgeW91ciByZXNwb25zZS48 bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjExLjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8 L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMS4wcHQ7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkkgc2V0IGJlbG93IHN5c2N0 bHM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExLjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJz cDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi10 b3A6Ny41cHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFtaWx5OiZxdW90 O0NvdXJpZXIgTmV3JnF1b3Q7O2NvbG9yOiMxNzJCNEQiPnN5c2N0bCBrZXJuLmlwYy5tYXhzb2Nr YnVmPTE4ODc0MzY4Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7DQo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6V2lu Z2RpbmdzO2NvbG9yOiMxNzJCNEQiPu+Dnzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEw LjVwdDtmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90Oztjb2xvcjojMTcyQjREIj4g MThNQjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwcmUgc3R5bGU9Im1hcmdpbi10b3A6Ny41cHQi PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2NvbG9yOiMxNzJCNEQiPnN5c2N0bCBuZXQu aW5ldC50Y3Auc2VuZGJ1Zl9tYXg9MTg4NzQzNjgmbmJzcDsgPGJyPnN5c2N0bCBuZXQuaW5ldC50 Y3AucmVjdmJ1Zl9tYXg9MTg4NzQzNjg8bzpwPjwvbzpwPjwvc3Bhbj48L3ByZT4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O21zby1mYXJlYXN0LWxh bmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDttc28tZmFyZWFzdC1sYW5ndWFn ZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t VVMiPkxvYWRlZCBDdWJpYyBDQyBhbGdvcml0aG0sPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7bXNvLWZhcmVh c3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxoNCBzdHls ZT0ibXNvLW1hcmdpbi10b3AtYWx0OjBjbTttYXJnaW4tcmlnaHQ6MGNtO21hcmdpbi1ib3R0b206 MGNtO21hcmdpbi1sZWZ0OjM2LjBwdDttYXJnaW4tYm90dG9tOi4wMDAxcHQ7dGV4dC1pbmRlbnQ6 LTE4LjBwdDttc28tbGlzdDpsMCBsZXZlbDEgbGZvMiI+DQo8IVtpZiAhc3VwcG9ydExpc3RzXT48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtTZWdvZSBVSSZx dW90OyxzYW5zLXNlcmlmO2NvbG9yOiMxNzJCNEQ7bGV0dGVyLXNwYWNpbmc6LS4wNXB0Ij48c3Bh biBzdHlsZT0ibXNvLWxpc3Q6SWdub3JlIj4xLjxzcGFuIHN0eWxlPSJmb250OjcuMHB0ICZxdW90 O1RpbWVzIE5ldyBSb21hbiZxdW90OyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7DQo8L3NwYW4+ PC9zcGFuPjwvc3Bhbj48IVtlbmRpZl0+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9u dC1mYW1pbHk6JnF1b3Q7U2Vnb2UgVUkmcXVvdDssc2Fucy1zZXJpZjtjb2xvcjojMTcyQjREO2xl dHRlci1zcGFjaW5nOi0uMDVwdCI+TG9hZCB0aGUgbW9kdWxlIHVzaW5nJm5ic3A7PG86cD48L286 cD48L3NwYW4+PC9oND4NCjxwcmUgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDo3LjVwdDttYXJn aW4tcmlnaHQ6MGNtO21hcmdpbi1ib3R0b206MGNtO21hcmdpbi1sZWZ0OjM2LjBwdDttYXJnaW4t Ym90dG9tOi4wMDAxcHQiPjxzcGFuIHN0eWxlPSJjb2xvcjojMTcyQjREIj5rbGRsb2FkIGNjX2N1 YmljPG86cD48L286cD48L3NwYW4+PC9wcmU+DQo8cHJlIHN0eWxlPSJtYXJnaW4tdG9wOjcuNXB0 Ij48c3BhbiBzdHlsZT0iY29sb3I6IzE3MkI0RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9w cmU+DQo8aDQgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDowY207bWFyZ2luLXJpZ2h0OjBjbTtt YXJnaW4tYm90dG9tOjBjbTttYXJnaW4tbGVmdDozNi4wcHQ7bWFyZ2luLWJvdHRvbTouMDAwMXB0 O3RleHQtaW5kZW50Oi0xOC4wcHQ7bXNvLWxpc3Q6bDAgbGV2ZWwxIGxmbzIiIGlkPSJGcmVlQlNE MTNWTXRvVk1JUEVSRlBlcmZvcm1hbmNlVGVzdGluZy1TZXR0aGVjY2FsZ29yaXRobWJ5Ij4NCjwh W2lmICFzdXBwb3J0TGlzdHNdPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFt aWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3MkI0RDtsZXR0ZXIt c3BhY2luZzotLjA1cHQiPjxzcGFuIHN0eWxlPSJtc28tbGlzdDpJZ25vcmUiPjIuPHNwYW4gc3R5 bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7Ij4mbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsNCjwvc3Bhbj48L3NwYW4+PC9zcGFuPjwhW2VuZGlmXT48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtTZWdvZSBVSSZxdW90OyxzYW5zLXNl cmlmO2NvbG9yOiMxNzJCNEQ7bGV0dGVyLXNwYWNpbmc6LS4wNXB0Ij5TZXQgdGhlIGNjIGFsZ29y aXRobSBieTxvOnA+PC9vOnA+PC9zcGFuPjwvaDQ+DQo8cCBzdHlsZT0ibXNvLW1hcmdpbi10b3At YWx0OjcuNXB0O21hcmdpbi1yaWdodDowY207bWFyZ2luLWJvdHRvbTowY207bWFyZ2luLWxlZnQ6 MzYuMHB0Ij4NCjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFtaWx5OiZxdW90 O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3MkI0RCI+c3lzY3RsIG5ldC5pbmV0 LnRjcC5jYy5hbGdvcml0aG09Y3ViaWM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDttc28tZmFyZWFzdC1sYW5n dWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6 RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExLjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+QmVs b3cgaXMgdGhlIGNvbW1hbmQgSSB1c2VkIHRvIHJ1biBpcGVyZjM8bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtt c28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0K PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi10b3A6Ny41cHQiPjxiPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNh bnMtc2VyaWY7Y29sb3I6IzE3MkI0RDtsZXR0ZXItc3BhY2luZzotLjA1cHQiPnNlcnZlcjo8bzpw PjwvbzpwPjwvc3Bhbj48L2I+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdp bi10b3A6Ny41cHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFtaWx5OiZx dW90O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3MkI0RCI+aXBlcmYzIC1zIC1C IDE5Mi4xNjguMS44Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1hcmdpbi10b3A6MjIuNXB0Ij48Yj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtTZWdvZSBVSSZxdW90OyxzYW5zLXNlcmlmO2NvbG9y OiMxNzJCNEQ7bGV0dGVyLXNwYWNpbmc6LS4wNXB0Ij5jbGllbnQ6PG86cD48L286cD48L3NwYW4+ PC9iPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tdG9wOjcuNXB0Ij48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtTZWdvZSBVSSZx dW90OyxzYW5zLXNlcmlmO2NvbG9yOiMxNzJCNEQiPmlwZXJmMyDigJNjIDE5Mi4xNjguMS44IC1C IDE5Mi4xNjguMi44IC10IDE4MCAtdyAxNm0gLUMgY3ViaWM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDttc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7bXNvLWZhcmVh c3QtbGFuZ3VhZ2U6RU4tVVMiPlRoaXMgc2hvdWxkIHNldCB0aGUgc29ja2V0IGJ1ZmZlcnMgZm9y IHRoZSBjb25uZWN0aW9ucyB0byAxNk1CIGFuZCBydW4gdGhlIHRlc3RzIGZvciAxODAgc2Vjcy48 bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjExLjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8 L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMS4wcHQ7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTEuMHB0O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5Ib3BlIHRoaXMgaXMgY2xlYXIuPG86 cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMS4wcHQ7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9v OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTEuMHB0O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEx LjBwdDttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+UmVnYXJkczxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0 O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5NdXJhbGk8bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDttc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7bXNvLWZhcmVh c3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXYgaWQ9 Im1haWwtZWRpdG9yLXJlZmVyZW5jZS1tZXNzYWdlLWNvbnRhaW5lciI+DQo8ZGl2Pg0KPGRpdiBz dHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjQjVDNERGIDEuMHB0O3BhZGRpbmc6 My4wcHQgMGNtIDBjbSAwY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1i b3R0b206MTIuMHB0Ij48Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtjb2xvcjpibGFj ayI+RnJvbToNCjwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Y29sb3I6 YmxhY2siPlJhbmRhbGwgU3Rld2FydCAmbHQ7cnJzQG5ldGZsaXguY29tJmd0Ozxicj4NCjxiPkRh dGU6IDwvYj5UaHVyc2RheSwgMjkgSnVuZSAyMDIzIGF0IDU6MTYgUE08YnI+DQo8Yj5UbzogPC9i Pk11cmFsaSBLcmlzaG5hbXVydGh5ICZsdDttdXJhbGlrMUB2bXdhcmUuY29tJmd0Ozxicj4NCjxi PkNjOiA8L2I+ZnJlZWJzZC10cmFuc3BvcnRARnJlZUJTRC5vcmcgJmx0O2ZyZWVic2QtdHJhbnNw b3J0QEZyZWVCU0Qub3JnJmd0Ozxicj4NCjxiPlN1YmplY3Q6IDwvYj5SZTogRnJlZUJTRCBUQ1Ag KHdpdGggaXBlcmYzKSBjb21wYXJpc29uIHdpdGggTGludXg8bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEu MHB0Ij5HcmVldGluZ3MgTXVyYWxpOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXY+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD4mbmJz cDs8L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPlNvIEkgYW0gdW5jbGVhciBmcm9tIHlvdXIg 4oCcU29ja2V0IGJ1ZmZlcuKAnSBhcyB0byB3aGF0IHlvdSBkaWQuLiBEaWQgeW91PG86cD48L286 cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPnNldCBib3RoIHRoZSBzZW5kIGFuZCByZWNlaXZlIHdp bmRvd3MgdG8gMTZNYnBzIHdpdGggdGhlIFNPX1NOREJVRiBhbmQgU09fUkNWQlVGIG9wdGlvbnM/ PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPk9yIHdlcmUgeW91IGp1c3QgdXNpbmcg YXV0by1zY2FsaW5nIHRvIGhhdmUgdGhlIHNvY2tldCBidWZmZXJzIGFkdmFuY2U/PG86cD48L286 cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwv ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTEuMHB0Ij5UaGFua3M8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD4mbmJz cDs8L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPlI8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 L2Rpdj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMS4wcHQiPjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxibG9j a3F1b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4wcHQiPg0KPGRp dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij5P biBKdW4gMjksIDIwMjMsIGF0IDU6NTEgQU0sIE11cmFsaSBLcmlzaG5hbXVydGh5ICZsdDs8YSBo cmVmPSJtYWlsdG86bXVyYWxpazFAdm13YXJlLmNvbSI+bXVyYWxpazFAdm13YXJlLmNvbTwvYT4m Z3Q7IHdyb3RlOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+Jm5ic3A7PC9vOnA+PC9z cGFuPjwvcD4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0i RU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij5IZWxsbyBGcmVlQlNEIFRyYW5zcG9ydCBl eHBlcnRzLDwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD48L286cD48 L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFu Zz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRp dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1z aXplOjExLjBwdCI+V2UgYXJlIGV2YWx1YXRpbmcgcGVyZm9ybWFuY2Ugb2YgRnJlZUJTRCAxMyBW TSBvbiBFU1ggaHlwZXJ2aXNvciBpbiBsb25nIFJUVCBzZXR1cCBhbmQgaGFwcGVuZWQgdG8gY29t cGFyZSB0aGUgcGVyZm9ybWFuY2Ugd2l0aCBMaW51eCBWTSB3aXRoIHNhbWUgaHlwZXJ2aXNvci48 L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+V2Ugc2VlIGEgc3Vic3RhbnRpYWxseSBiZXR0ZXIg cGVyZm9ybWFuY2Ugd2l0aCBMaW51eCBnZXR0aW5nIGNsb3NlIHRvIHRoZSBCRFAgbGltaXQsIHdo ZXJlYXMgQlNEIDEzIG5vdCBmaWxsaW5nIHVwIHRoZSBwaXBlIGVub3VnaC48L3NwYW4+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0K PGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9u dC1zaXplOjExLjBwdCI+V2UgYXJlIHRyeWluZyB0byBmaWd1cmUgb3V0IHdoYXQgY291bGQgbGVh ZCB0byBzdWNoIGEgaHVnZSBkaWZmZXJlbmNlIGFuZCBmZWVsIHdlIGNvdWxkIGJlIG1pc3Npbmcg c29tZXRoaW5nIGhlcmUuPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpw PjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPiZuYnNwOzwvc3Bhbj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9k aXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxl PSJmb250LXNpemU6MTEuMHB0Ij5Db3VsZCB5b3UgcGxlYXNlIGhlbHAgdXMgdG8ga25vdyBpZiB0 aGVyZSBpcyBhIHdheSB0byBtYWtlIGl0IHBlcmZvcm0gYmV0dGVyPzwvc3Bhbj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2 Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNp emU6MTEuMHB0Ij4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+U2V0dXAgZGV0YWls czo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO LVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+Jm5ic3A7PC9zcGFuPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTEuMHB0Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTox MS4wcHQiPldlIGhhdmUgMiBFU1ggaHlwZXJ2aXNvcnMgd2hlcmUgMiBWTXMgKG9uZSBGcmVlQlNE IDEzIGFuZCBvbmUgVWJ1bnR1IDIzLjA0L0xpbnV4IGtlcm5lbCA2LjIpIHdlcmUgbGF1bmNoZWQg b24gZWFjaCBoeXBlcnZpc29yLjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+ PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij5UaGVuIHdlIHJh biBpcGVyZiBiZXR3ZWVuLDwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86 cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8b2wgc3R5bGU9Im1hcmdpbi10b3A6MGNtIiBz dGFydD0iMSIgdHlwZT0iYSI+DQo8bGkgY2xhc3M9Ik1zb0xpc3RQYXJhZ3JhcGgiIHN0eWxlPSJt YXJnaW4tdG9wOjBjbTttYXJnaW4tYm90dG9tOjBjbTttc28tbGlzdDpsMSBsZXZlbDEgbGZvMSI+ DQo8c3BhbiBsYW5nPSJFTi1VUyI+QlNEIDEzICZsdDstJmd0OyBCU0QgMTM8L3NwYW4+PG86cD48 L286cD48L2xpPjxsaSBjbGFzcz0iTXNvTGlzdFBhcmFncmFwaCIgc3R5bGU9Im1hcmdpbi10b3A6 MGNtO21hcmdpbi1ib3R0b206MGNtO21zby1saXN0OmwxIGxldmVsMSBsZm8xIj4NCjxzcGFuIGxh bmc9IkVOLVVTIj5VYnVudHUgJmx0Oy0mZ3Q7IFVidW50dTwvc3Bhbj48bzpwPjwvbzpwPjwvbGk+ PC9vbD4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQiPiZuYnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjExLjBwdCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij5F dmVuIHRob3VnaCB0aGUgbmV0d29yayBlbnZpcm9ubWVudCB3ZXJlIHNhbWUgaW4gYm90aCBjYXNl cywgd2Ugc2VlIFVidW50dSBwZXJmb3JtaW5nIG11Y2ggYmV0dGVyLjwvc3Bhbj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2 Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNp emU6MTEuMHB0Ij4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+QmVsb3cgYXJlIGNv bm5lY3Rpb24gcGFyYW1ldGVyczo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQi PjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+U29ja2V0IGJ1 ZmZlcjogMTZNQjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD48L286 cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g bGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij5UQ1AgQ0MgQWxnbzogQ3ViaWMu IFdlIHVzZWQgdGhpcyBhcyB0aGlzIGlzIHN1aXRhYmxlIGZvciBMb25nIEZhdCBOZXR3b3Jrcy48 L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+UGluZyBSVFQ6Jm5ic3A7IDEwMCBtcyBiZXR3ZWVu IHRoZSB0d28gZW5kIHBvaW50cy48L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQi PjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+V2Uga2VwdCBh bGwgb3RoZXIgcGFyYW1ldGVycyB0byBkZWZhdWx0IG9uIGJvdGggTGludXggYW5kIEJTRC48L3Nw YW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBz dHlsZT0iZm9udC1zaXplOjExLjBwdCI+Jm5ic3A7PC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTEuMHB0Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3Ryb25nPjxpPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2Zv bnQtZmFtaWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3MkI0RCI+ QkRQIGZvciAxNk1CPC9zcGFuPjwvaT48L3N0cm9uZz48c3BhbiBjbGFzcz0iYXBwbGUtY29udmVy dGVkLXNwYWNlIj48aT48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTom cXVvdDtTZWdvZSBVSSZxdW90OyxzYW5zLXNlcmlmO2NvbG9yOiMxNzJCNEQiPiZuYnNwOzwvc3Bh bj48L2k+PC9zcGFuPjxlbT48Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZh bWlseTomcXVvdDtTZWdvZSBVSSZxdW90OyxzYW5zLXNlcmlmO2NvbG9yOiMxNzJCNEQiPlNvY2tl dA0KIGJ1ZmZlcjo8L3NwYW4+PC9iPjwvZW0+PHNwYW4gY2xhc3M9ImFwcGxlLWNvbnZlcnRlZC1z cGFjZSI+PGk+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7 U2Vnb2UgVUkmcXVvdDssc2Fucy1zZXJpZjtjb2xvcjojMTcyQjREIj4mbmJzcDs8L3NwYW4+PC9p Pjwvc3Bhbj48ZW0+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6JnF1 b3Q7U2Vnb2UgVUkmcXVvdDssc2Fucy1zZXJpZjtjb2xvcjojMTcyQjREIj4xNiBNQiAqICgxMDAw DQogbXMgKiAxMDBtcyBsYXRlbmN5KSAqIDggYml0cy8gMTAyNDwvc3Bhbj48L2VtPjxzcGFuIGNs YXNzPSJhcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0 O2ZvbnQtZmFtaWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3MkI0 RCI+Jm5ic3A7PC9zcGFuPjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250 LWZhbWlseTomcXVvdDtTZWdvZSBVSSZxdW90OyxzYW5zLXNlcmlmO2NvbG9yOiMxNzJCNEQiPj08 c3BhbiBjbGFzcz0iYXBwbGUtY29udmVydGVkLXNwYWNlIj48aT4mbmJzcDs8L2k+PC9zcGFuPjxz dHJvbmc+PGk+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNh bnMtc2VyaWYiPjEuMjUNCiBHYnBzPC9zcGFuPjwvaT48L3N0cm9uZz48L3NwYW4+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRp dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1z aXplOjExLjBwdCI+Jm5ic3A7PC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48 bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPlVidW50dSBjb25z aXN0ZW50bHkgaGl0cyBhcm91bmQgMSBHYnBzIEJpdHJhdGUgYWxtb3N0IHJlYWNoaW5nIHRoZSBC RFAgbGltaXQuPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs YW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPkZyZWVCU0QgMTMgc2hvd3MgYSBC aXQgcmF0ZSBiZXR3ZWVuIHRoZSByYW5nZSBvZjxzcGFuIGNsYXNzPSJhcHBsZS1jb252ZXJ0ZWQt c3BhY2UiPiZuYnNwOzwvc3Bhbj48L3NwYW4+PHN0cm9uZz48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtTZWdvZSBVSSZxdW90OyxzYW5zLXNlcmlmO2NvbG9y OiMxNzJCNEQiPjMwMC02MDANCiBNYnBzPC9zcGFuPjwvc3Ryb25nPjxzcGFuIGNsYXNzPSJhcHBs ZS1jb252ZXJ0ZWQtc3BhY2UiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFt aWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3MkI0RDtiYWNrZ3Jv dW5kOndoaXRlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAu NXB0O2ZvbnQtZmFtaWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LHNhbnMtc2VyaWY7Y29sb3I6IzE3 MkI0RDtiYWNrZ3JvdW5kOndoaXRlIj5vbmx5Lg0KIFNvIGl0IHNlZW1zIHRvIGJlIGRvaW5nIGhh bGYgYXMgZ29vZCBhcyBMaW51eC48L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQi PjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+Rm9yIGxvd2Vy IHNvY2tldCBidWZmZXIgb2YgNE1CLCBib3RoIEZyZWVCU0QgYW5kIExpbnV4IHNlZW0gdG8gZG8g c2FtZSBhbmQgYWJsZSB0byBtZWV0IEJEUCBvZiAzMDAgTWJwcyBjb25zaXN0ZW50bHkuPC9zcGFu PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQiPkxhcmdlciBzb2NrZXQgYnVmZmVyIHNlZW1zIHRvIGhhdmUg YW4gaXNzdWUuPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs YW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPiZuYnNwOzwvc3Bhbj48c3BhbiBz dHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250 LXNpemU6MTEuMHB0Ij5QbGVhc2UgbGV0IHVzIGtub3cgaWYgdGhlcmUgYXJlIHdheXMgdG8gZmlu ZSB0dW5lIHRoZSBzeXN0ZW0gcGFyYW1ldGVycyB0byBtYWtlIEJTRCBwZXJmb3JtIGJldHRlci48 L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+T3IgYW55IG90aGVyIHN1Z2dlc3Rpb25zL3F1ZXJp ZXMgd2VsY29tZS48L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+Jm5ic3A7PC9zcGFuPjxzcGFu IHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZv bnQtc2l6ZToxMS4wcHQiPlJlZ2FyZHM8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4w cHQiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+TXVyYWxp PC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0Ij48bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdCI+PG86cD4mbmJzcDs8L286cD48 L3NwYW4+PC9wPg0KPGRpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjkuMHB0O2ZvbnQtZmFtaWx5OkhlbHZldGljYTtjb2xvcjpibGFjayI+LS0t LS0tPG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTo5LjBwdDtmb250LWZhbWlseTpIZWx2ZXRpY2E7 Y29sb3I6YmxhY2siPlJhbmRhbGwgU3Rld2FydDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2 Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6OS4w cHQ7Zm9udC1mYW1pbHk6SGVsdmV0aWNhO2NvbG9yOmJsYWNrIj48YSBocmVmPSJtYWlsdG86cnJz QG5ldGZsaXguY29tIj5ycnNAbmV0ZmxpeC5jb208L2E+PG86cD48L286cD48L3NwYW4+PC9wPg0K PC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6 ZTo5LjBwdDtmb250LWZhbWlseTpIZWx2ZXRpY2E7Y29sb3I6YmxhY2siPjxvOnA+Jm5ic3A7PC9v OnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQiPjxvOnA+ Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0K PC9ib2R5Pg0KPC9odG1sPg0K --_000_PH0PR05MB10064C83AAD620C3AB838FC6FFB25APH0PR05MB10064na_-- From nobody Thu Jun 29 16:14:21 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QsNnF67hvz4kv1R for ; Thu, 29 Jun 2023 16:14:25 +0000 (UTC) (envelope-from rscheff@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QsNnF5jjpz4K7T; Thu, 29 Jun 2023 16:14:25 +0000 (UTC) (envelope-from rscheff@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688055265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=AdKGi4aJFN0sZTq1wfOPJnH8pCSpwvbiyGN7u1g5xJI=; b=EExjVlvdsXmhAM3eCxIQ/SZVo8JzWx/X2hwwWv+rQlj5PiXjLbMruF2xgjF938HvenCH2u U0otRnbw5Znu1x8snu3d80m7r6t1C8O0i7DTULsMXScSPaUVVNzvGMxK9Em0fpBOJCVTLx ZIscKbJ500TBJju77dVp+mypvj7i5goP8pdKeAnQ//fT4ph25Vf54T7wbdkLTZV7PwNdj6 8wM5My8ksQz5GrX7V1zLA8SRC84uQWoQs1homelXhrwaAI706CMfgld05RWrVGDmQeKyGf qgDuNuA+0oTPa6+ljoP1GUMYEaY/ACVeN5hvMrfIvO0g5r22so1Ib1yPakudSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688055265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=AdKGi4aJFN0sZTq1wfOPJnH8pCSpwvbiyGN7u1g5xJI=; b=KeTwmde6zHG0op14c2z9J+VcaC95jPmduZJWY4uFC/o7R1SCztqpPE7J03tCW3fD2uw0rx wIOM3Mq+OKz3iHsiB3Z0d8IuHUM7bbJ5mRdb95bjBBTS4zWkwShiOqttVLA5ouE6hlFtqy 8q/7gZrPPfl6C998QFgRaZ77IHGNgT1bxNrcm4jG9MgP0hf5OpkqcIUiUBC15W/LOWmWYF b1BMTg7487pPrZghD01tErPrLosP4Eb4yUQUtiwSS8ygXwLunYWL6WyWFHZaZtrGVp13T1 3cmiMRg5gnFPAr6T118V2GRHk96Acvx1pi5CpI9F8p6scKZgjhNSt1BKNFasyA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688055265; a=rsa-sha256; cv=none; b=Xdym3QTzlSEfalasJmXScJB3Bfc6sGK+++lrPxfpDon0OD8li+dQSA5EBP3nuwzOTBbgxN 0dEONR/K/CHKxi6gAPsxGg3SNYgqTfl2QHzCz7kGiiRMjO8Mtqola5NxaePNuHkWxVq1uP +PGOAnA5vrYJar//JA0vPwiVLkI25lH7DPvWU05ZEf85RhOdkRrnJ0sD+ylouyJb+BJ5GU F6TVYevKYeMw+oC4o7nFX9zX8NezyaHSTie2JaCFVvnB2cEeC7IzxTQ23qxuS+kNXDzSN5 poEoCrICZ4kzWpZb27mH9yXz45EYxyOfRPweeRWhynJDMxmFpTxtfczi8RMG5g== Received: from [192.168.233.104] (unknown [185.236.167.136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: rscheff/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QsNnF2Gb6z18Gd; Thu, 29 Jun 2023 16:14:25 +0000 (UTC) (envelope-from rscheff@freebsd.org) Message-ID: <0826320c-936a-e3a4-78e8-809b8bab5f71@freebsd.org> Date: Thu, 29 Jun 2023 18:14:21 +0200 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: FreeBSD Transport , Murali Krishnamurthy From: "Scheffenegger, Richard" Autocrypt: addr=rscheff@freebsd.org; keydata= xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iziPF0FzrN K1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9yZz7CmgQTFgoAQhYh BDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGABQsJCAcCAyICAQYVCgkICwIE FgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/nvht8kExJ31M+3qpjOqdVypMp+/Ojvh5Z lsk96QEA5HCBkteJcrohwRA7llZvLH3m25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdV AQUBAQdA1Dim8ZWpXRS5i9hb3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2 S7eZrINEWrPNtHEXvliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bA EPqOH+JCIND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8= Subject: FreeBSD TCP (with iperf3) comparison with Linux Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------eBjGS22THVXJLa2iBXpgJshH" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------eBjGS22THVXJLa2iBXpgJshH Content-Type: multipart/mixed; boundary="------------P7pmeTlc5O9Awvgom5UwApur"; protected-headers="v1" From: "Scheffenegger, Richard" To: FreeBSD Transport , Murali Krishnamurthy Message-ID: <0826320c-936a-e3a4-78e8-809b8bab5f71@freebsd.org> Subject: FreeBSD TCP (with iperf3) comparison with Linux --------------P7pmeTlc5O9Awvgom5UwApur Content-Type: multipart/mixed; boundary="------------m1v32PQnBe670iYdt8FHMhWm" --------------m1v32PQnBe670iYdt8FHMhWm Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SGkgTXVyYWxpLA0KDQogIA0KPiBIZWxsbyBGcmVlQlNEIFRyYW5zcG9ydCBleHBlcnRzLA0K PiANCj4gV2UgYXJlIGV2YWx1YXRpbmcgcGVyZm9ybWFuY2Ugb2YgRnJlZUJTRCAxMyBWTSBv biBFU1ggaHlwZXJ2aXNvciBpbiBsb25nIFJUVCBzZXR1cCBhbmQgaGFwcGVuZWQgdG8gY29t cGFyZSB0aGUgcGVyZm9ybWFuY2Ugd2l0aCBMaW51eCBWTSB3aXRoIHNhbWUgaHlwZXJ2aXNv ci4NCj4gV2Ugc2VlIGEgc3Vic3RhbnRpYWxseSBiZXR0ZXIgcGVyZm9ybWFuY2Ugd2l0aCBM aW51eCBnZXR0aW5nIGNsb3NlIHRvIHRoZSBCRFAgbGltaXQsIHdoZXJlYXMgQlNEIDEzIG5v dCBmaWxsaW5nIHVwIHRoZSBwaXBlIGVub3VnaC4gDQo+IFdlIGFyZSB0cnlpbmcgdG8gZmln dXJlIG91dCB3aGF0IGNvdWxkIGxlYWQgdG8gc3VjaCBhIGh1Z2UgZGlmZmVyZW5jZSBhbmQg ZmVlbCB3ZSBjb3VsZCBiZSBtaXNzaW5nIHNvbWV0aGluZyBoZXJlLg0KPiANCj4gQ291bGQg eW91IHBsZWFzZSBoZWxwIHVzIHRvIGtub3cgaWYgdGhlcmUgaXMgYSB3YXkgdG8gbWFrZSBp dCBwZXJmb3JtIGJldHRlcj8NCj4gDQo+IFNldHVwIGRldGFpbHM6DQo+IA0KPiBXZSBoYXZl IDIgRVNYIGh5cGVydmlzb3JzIHdoZXJlIDIgVk1zIChvbmUgRnJlZUJTRCAxMyBhbmQgb25l IFVidW50dSAyMy4wNC9MaW51eCBrZXJuZWwgNi4yKSB3ZXJlIGxhdW5jaGVkIG9uIGVhY2gg aHlwZXJ2aXNvci4NCj4gVGhlbiB3ZSByYW4gaXBlcmYgYmV0d2VlbiwNCj4gYSkJQlNEIDEz IDwtPiBCU0QgMTMNCj4gYikJVWJ1bnR1IDwtPiBVYnVudHUNCg0KU2luY2UgeW91IG1lbnRp b24gdHdvIGh5cGVydmlzb3JzIC0gd2hhdCBpcyB0aGUgcGh5aXNjYWwgbmV0d29yayB0b3Bv bG9neSBpbiBiZXR3ZWVuIHRoZXNlIHR3byBzZXJ2ZXJzPyBXaGF0IHRoZW9yZXRpY2FsIGxp bmsgcmF0ZXMgd291bGQgYmUgYXR0YWluYWJsZT8NCg0KT3IgZGlkIHlvdSB3YW50IHRvIHNh eSB0aGF0IGVhY2ggdGVzdCB3YXMgcnVuIHdpdGhpbiBvbmUgYW5kIHRoZSBzYW1lIGh5cGVy dmlzb3IsIHByZXN1bWFibHkgd2l0aCB0aGUgc2FtZSBoYXJkd2FyZSBhbmQgdmlydHVhbCBz d2l0Y2gvaHlwZXJ2aXNvciBwcm92aWRlZCBvbiBib3RoIHRlc3Qgc2NlbmFyaW9zPw0KDQpT byB0aGF0IHRoZSB0aGVvcmV0aWNhbCBiYW5kd2lkdGggd291bGQgYmUgd2hhdGV2ZXIgcGFj a2V0cyBjYW4gYmUgc2h1ZmZsZWQgaW4gbWVtb3J5IGJldHdlZW4gVk0xLCBWc3dpdGNoICh3 aXRoIGRlbGF5IHNpbXVsYXRvciksIFZNMj8NCg0KDQpEaWQgeW91IHJ1biBpcGVyZjM/IERp ZCB0aGUgdHJhbnNtaXR0aW5nIGVuZHBvaW50IHJlcG9ydCBhbnkgcmV0cmFuc21pc3Npb25z IGJldHdlZW4gTGludXggb3IgRkJTRCBob3N0cz8NCg0KSW4gdGhlb3J5LCBpbiBhIHRydWUg YmFjay10by1iYWNrLCBsb3NzbGVzcyBlbnZpcm9ubWVudCwgaXQgc2hvdWxkIGJlIGZhaXJs eSBpcnJlbGV2YW50IHdoaWNoIENDIHlvdSBhcmUgcnVubmluZzsgaWYgdGhlcmUgYXJlIHBh Y2tldCBkcm9wcyAocmV0cmFuc21pc3Npb25zKSBpbiB0aGUgaXBlcmYzIG91dHB1dCwgcGxl YXNlIGNvbXBhcmUgdGhlaXIgZnJlcXVlbmN5IGJldHdlZW4gdGhlIExpbnV4IGFuZCBGQlNE IHNpZGUgLSBhIHNsaWdodGx5IGhpZ2hlciBpbmNpZGVuY2UgcmF0ZSBvZiBwYWNrZXQgZHJv cHMgKGUuZy4gcmluZy1idWZmZXIgb3ZlcmZsb3dzIHVzZWQgdG8gYmUgYSBwcm9taW5lbnQg aXNzdWUgYXQgaGlnaCBzcGVlZCBsaW5rcykgY291bGQgcG9zc2libHkgZXhwbGFpbiB3aGF0 IGFwcGVhcnMgdG8gYmUgc3VjaCBhIGh1Z2UgZGlmZmVyZW50aWFsICh0aGUgc3F1YXJlIG9m IHRoZSBsb3NzIHByb2JhYmlsaXR5IGlzIHJlbGV2YW50IGZvciBjbGFzc2ljIFRDUHMgLSBz byBhIG1pbm9yIGNoYW5nZSBjYW4gaGF2ZSBhIGh1Z2UgaW1wYWN0KS4NCg0KRGlkIHlvdSBj b250cmFzdCB5b3VyIGNjX2N1YmljIGZpbmRpbmdzIHdpdGggY2NfbmV3cmVubz8NCg0KPiBF dmVuIHRob3VnaCB0aGUgbmV0d29yayBlbnZpcm9ubWVudCB3ZXJlIHNhbWUgaW4gYm90aCBj YXNlcywgd2Ugc2VlIFVidW50dSBwZXJmb3JtaW5nIG11Y2ggYmV0dGVyLg0KPiANCj4gQmVs b3cgYXJlIGNvbm5lY3Rpb24gcGFyYW1ldGVyczoNCj4gU29ja2V0IGJ1ZmZlcjogMTZNQg0K PiBUQ1AgQ0MgQWxnbzogQ3ViaWMuIFdlIHVzZWQgdGhpcyBhcyB0aGlzIGlzIHN1aXRhYmxl IGZvciBMb25nIEZhdCBOZXR3b3Jrcy4NCj4gUGluZyBSVFQ6ICAxMDAgbXMgYmV0d2VlbiB0 aGUgdHdvIGVuZCBwb2ludHMuDQo+IFdlIGtlcHQgYWxsIG90aGVyIHBhcmFtZXRlcnMgdG8g ZGVmYXVsdCBvbiBib3RoIExpbnV4IGFuZCBCU0QuIA0KPiAgDQo+IEJEUCBmb3IgMTZNQiBT b2NrZXQgYnVmZmVyOiAxNiBNQiAqICgxMDAwIG1zICogMTAwbXMgbGF0ZW5jeSkgKiA4IGJp dHMvIDEwMjQgPSAxLjI1IEdicHMNCj4gDQo+IFVidW50dSBjb25zaXN0ZW50bHkgaGl0cyBh cm91bmQgMSBHYnBzIEJpdHJhdGUgYWxtb3N0IHJlYWNoaW5nIHRoZSBCRFAgbGltaXQuIA0K PiBGcmVlQlNEIDEzIHNob3dzIGEgQml0IHJhdGUgYmV0d2VlbiB0aGUgcmFuZ2Ugb2YgMzAw LTYwMCBNYnBzIG9ubHkuIFNvIGl0IHNlZW1zIHRvIGJlIGRvaW5nIGhhbGYgYXMgZ29vZCBh cyBMaW51eC4NCj4gRm9yIGxvd2VyIHNvY2tldCBidWZmZXIgb2YgNE1CLCBib3RoIEZyZWVC U0QgYW5kIExpbnV4IHNlZW0gdG8gZG8gc2FtZSBhbmQgYWJsZSB0byBtZWV0IEJEUCBvZiAz MDAgTWJwcyBjb25zaXN0ZW50bHkuDQo+IExhcmdlciBzb2NrZXQgYnVmZmVyIHNlZW1zIHRv IGhhdmUgYW4gaXNzdWUuDQo+IA0KPiBQbGVhc2UgbGV0IHVzIGtub3cgaWYgdGhlcmUgYXJl IHdheXMgdG8gZmluZSB0dW5lIHRoZSBzeXN0ZW0gcGFyYW1ldGVycyB0byBtYWtlIEJTRCBw ZXJmb3JtIGJldHRlci4NCj4gT3IgYW55IG90aGVyIHN1Z2dlc3Rpb25zL3F1ZXJpZXMgd2Vs Y29tZS4NCj4gDQo+IFJlZ2FyZHMNCj4gTXVyYWxpDQo= --------------m1v32PQnBe670iYdt8FHMhWm Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc" Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc" Content-Description: OpenPGP public key Content-Transfer-Encoding: quoted-printable -----BEGIN PGP PUBLIC KEY BLOCK----- xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m 25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb 3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8=3D =3DBwxS -----END PGP PUBLIC KEY BLOCK----- --------------m1v32PQnBe670iYdt8FHMhWm-- --------------P7pmeTlc5O9Awvgom5UwApur-- --------------eBjGS22THVXJLa2iBXpgJshH Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZJ2t3QUDAAAAAAAKCRAXvliZ4LFDmyv5 AQDw1GUFi2Rqjuk8zpFYeUorf1nbNe+RTxKt9b5A0doWwwEArxcYvz6qiOl9HKLTUZzKtN+JM9Co dV7hoKXpZ6eEgwY= =WxnE -----END PGP SIGNATURE----- --------------eBjGS22THVXJLa2iBXpgJshH-- From nobody Fri Jun 30 15:03:44 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qsz9K5B5cz4kh8s for ; Fri, 30 Jun 2023 15:03:49 +0000 (UTC) (envelope-from muralik1@vmware.com) Received: from CY4PR02CU007.outbound.protection.outlook.com (mail-westcentralusazon11011012.outbound.protection.outlook.com [40.93.199.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qsz9K1zZjz4DDR; Fri, 30 Jun 2023 15:03:49 +0000 (UTC) (envelope-from muralik1@vmware.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BY15HtQ01Ekz+hOoyBDuu8XWT0r9pYZBueudOnVYbdNaJxrnpfcJwPZ47Dn0uXysM7rR0WrMueVbdJiAFaQXDrc4Dd93SCBCaFM7km2yaGhDs26asgSMFYGdlFVsCvGwbkz2H+NmjGhYWkqtDF5Gmd8hTmPvLopuCW1eb+fM0xkGlynKcFffjT2o/YpLB3adiwneqHZH8r4pUw3QRai+pbs6uWx9jhT5DiQk/rtzNnwqhjQ8rVfRNsIHaK/iTRgwvohy/U8tmU79V7Tb1spLmtK9/z0fFMZMkmrHLNH34DsRuiYYmXfaChMkOCMwEvgD4uJUhPTaSBG2PKK2WZTAPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kpkbddn/YYtpDp2Oj4RyUEE7zhEMCQLYR4Q/PvLUO7Y=; b=SGxVozVAiP8ZpzEYV3Tj8oSs6iCjqKLooLtOZh/A8+iTjeAPojUsekrjCDmtwmOxgfvXOJMRn94cnzv227knOMm8SMXMc8S93NuKbHjF0td1s2cfNb/9KDnB6ITs5ghUZr3Qfu7GWVl/dHiYA9f/1XzgOCAwt6ietkoyMEn4boSYX/gKzySqw5rzm7kWnrozwCOH5WpUQuhdMSSxYIKMJ4f1ScOAcYvFe6EnngY+zra7Jow9c2Uc3jpMltYKdxjeEoIs3qfo8QA/S51yBsUcHl0W3rBa19ZxVCZM6OCLwtk1g3BQJIruIWIK4GipzAzS23jc+5c4Lrh4wfGu2qaaTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kpkbddn/YYtpDp2Oj4RyUEE7zhEMCQLYR4Q/PvLUO7Y=; b=k2dw87D/9Rn9eLoyATohr6aMHJsFaacM1WPYHA+Ov6gdUJS/EygQokza4949WIccDqnvN7ff9NFEaQTT2OhcVsxsapxo8cp0iBZH0HWJQWuePw5UqMzRIMuVEDV8U4yhhQl9o7UJ1SimQVWfh766ToDi75Ah270qn7QJhFzvpvc= Received: from PH0PR05MB10064.namprd05.prod.outlook.com (2603:10b6:510:29d::8) by PH7PR05MB9083.namprd05.prod.outlook.com (2603:10b6:510:1fd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.24; Fri, 30 Jun 2023 15:03:45 +0000 Received: from PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593]) by PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593%4]) with mapi id 15.20.6544.019; Fri, 30 Jun 2023 15:03:45 +0000 From: Murali Krishnamurthy To: "Scheffenegger, Richard" , FreeBSD Transport Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Thread-Topic: FreeBSD TCP (with iperf3) comparison with Linux Thread-Index: AQHZqqTJeOhY5ZSu6EKxBx6E7aUjbq+ja1xF Date: Fri, 30 Jun 2023 15:03:44 +0000 Message-ID: References: <0826320c-936a-e3a4-78e8-809b8bab5f71@freebsd.org> In-Reply-To: <0826320c-936a-e3a4-78e8-809b8bab5f71@freebsd.org> Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR05MB10064:EE_|PH7PR05MB9083:EE_ x-ms-office365-filtering-correlation-id: 58398e54-fe75-4463-c6d3-08db797b333c x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: J0Iggwh78rRvIqHO4+UU8lO1WNNIv2pg+1Eoh2Vh1rdxoYcYIDvwqIRXEYl2YNYjsFveODMItVFpAlv/8zZ3y4yvHOmxFE/GMgdfFJusXd64vWrwG1TDozjNGM6VSY6s1/xv8LxxKVZqwpFgLdzPZGGA8J9k7ovu98kcudmwc2b8pdeg5idSDzifXPM3wajeCcgPDMxa4WTBocN++BaMkqsyZLJF3R+19QzOdxYJSlcfPMGLTdVFOzX6F1axq5dSn/J8Uxg2ST9INEs2zqSa0bdNzSuDu0Xd1SMGieJk3LfytDny2ZJv6qWLYZ+Hh8rq5B1Tz1+v0/Y4HYD1r+eUjX5zfPIYcBgRSOVSXhDFUumFDzSIwmFOogdVPY9wO7tQ16vXv9gv+tgqQ9kryg/GH6lXouyJmB6ezhAsHsLf1ZXbSScwKBO92Irt08jZ7Ko7j8wcWXogbrvZd80mp0g+CJmurAv2NMCMWI0nYXAJMScMGUKYKfx2o3bXTOFnu3MX/albltTssRaBLezFxIMMa5x2dpmnR+bES25c+kz2BwNryqTr4Zs56ta5TSMjUHBz1jEyKcWHedIWskU8oZ865hPDGDlOuH/1TkMwCjMXr7/kk0nTR01N5fsgu6RMbGSaDT9XPyK4Aka8vQ/iCIJhW6EUQH6PhvDokS2DBJstn5g= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR05MB10064.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(39860400002)(396003)(136003)(376002)(366004)(346002)(451199021)(52536014)(5660300002)(91956017)(66946007)(76116006)(66556008)(66446008)(33656002)(64756008)(66476007)(450100002)(478600001)(316002)(8936002)(66899021)(8676002)(2906002)(26005)(110136005)(38070700005)(41300700001)(71200400001)(55016003)(7696005)(186003)(53546011)(6506007)(9686003)(122000001)(86362001)(38100700002)(83380400001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?7viqfgv44Ce72zmy6+oxLRGtRhucC2b1SGLBcIkbOWZ7dBvjPm8UA9ub?= =?Windows-1252?Q?+YOS5U8KxG/I7SHfeDCPPeTL4ADP0WZNuyQYS3XBEWnI1FmYV3v3PUI9?= =?Windows-1252?Q?dyszvg/EXWwiSELQlD2uAV8N5rNPYKg5L+BsEJeCmyJ+Q0VpGVDL3SaT?= =?Windows-1252?Q?KVsAmuOEOEv2EQsuHE/GgsMyJ9DtJjAPry8ZA1yMziIRnkYDW6B2otBs?= =?Windows-1252?Q?NGY6EnDGpdkl+uJStbd3e9qfbcZ04F+pYs3B/4THRsnqn9bHaHqfS0L3?= =?Windows-1252?Q?dKGXR66iaIkWfgiPhRgptL3Ig67julf40aG8SjdKGJ9eTBhAd8nxzJVx?= =?Windows-1252?Q?wroCwovD+5bmtLG7BB0qMUMKW5DDeTTY9HSaLJGRbvTm4dEAQy3Pkd89?= =?Windows-1252?Q?1f8VRnMxp8rEXOtCimu7lcgiNAdrb3vvK+q6RKAZB/WUe2yOFz8homRc?= =?Windows-1252?Q?AsQryRfC+bDzw2ZzaNioc24IW6u/iVqOeplg8oBi7SgVxoptgD+URbm6?= =?Windows-1252?Q?2QLU+4uIYLo9W0ThZmunpvYSkgvdtjUGlG/Xzh6dYgDFqoNI7yYrjvOc?= =?Windows-1252?Q?ZrI1bgZziSr9fKIFE81trMHtKx3aAoQlVLW8yi+JHohosD9fPA8V4rp8?= =?Windows-1252?Q?4ZOWLeCc/e6eRy7kmiB1fB7mBmKpnk1BLxCEkZ59CnRTAsO5z2H1NpAT?= =?Windows-1252?Q?y9mFdvZvo0cDdT3b1eAJoE+QY4Tnz43KZX3Ir1iVCJFsWWrIeSPUxvQA?= =?Windows-1252?Q?6uvtR+QrTBhqlk7sLvS7TSwyx1L4rlWs/LnbYaiUozRFBtkK4oMt73A8?= =?Windows-1252?Q?f2ASnfw62ZSTi1aRUQ4nZrr972Z+2DvmKK+B4YLaC/QoLa7yMz4fjEvX?= =?Windows-1252?Q?fEoQNOvISmPIdfashAVhThdh6kinua1XYP8xcegtCQuROpdNBQCrrHBE?= =?Windows-1252?Q?R9CyoehKy+Q9SpTXvXw+aKqIirIKL08PaW33GKqxXuj6Mj9QGHMZRpE6?= =?Windows-1252?Q?XUD3G9cu5X6DQhjDplSYOE01A3CCCAx2wuO9KQuRvSEOptQmIbN7j/Ru?= =?Windows-1252?Q?WPl3FnkB1ZZhUHuwGAaY6VAL00HWKzb2Bwbsl3Nnis4sSMdljt8lOkwn?= =?Windows-1252?Q?63XpI6cfaks8xN2pI/tu+xXrEPGt05kiLMw3RsONUG3irzflrQZz8f+9?= =?Windows-1252?Q?GqhsntaErYs1Y0TltaDuliZZwg/l3bdTBt9eRGVdReNaLyO5AaPfftHA?= =?Windows-1252?Q?FIAkRwUk4N7QUASepZE97l89oaf19mcVgGTfALhlSYV1Jtg3X1Nf3vVL?= =?Windows-1252?Q?myFtRLWn5LGbZWMPmBuzm3/BpxC4V3UHXvT84UdhtiR2kGW+QXz1eYwg?= =?Windows-1252?Q?9ccgLeXNjac/HeWFyIxRiCJgj2dJyuFhtJj+v1WAfmbQexlXihujQ1z3?= =?Windows-1252?Q?igWEWNOy2qCay0aE6Mo4cZ9aPaO76v3dn7dP7ZwH++BNkguikqfOPADI?= =?Windows-1252?Q?p07zqy6rfAnJwI7AcgN26x0rhGeqL0WMMdzv9t/P25j1u5izcjPTjvx8?= =?Windows-1252?Q?Fmh75vV1hG66wiJgHFyk0Oh7Y4lDuaBMRBLaz/nduZtJIOFuGPwSrdUe?= =?Windows-1252?Q?UteQYH0eQSk9nn29YecPE9AW5G2yqMC/aO7ZbpYwUNWqFvlrj6AoPCy9?= =?Windows-1252?Q?hLFUNLqTC4iTpNNo4ZHwhljTbY2vkVZ6sLZtCYdzZYlsdaPKMwnFiA?= =?Windows-1252?Q?=3D=3D?= Content-Type: multipart/alternative; boundary="_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_" List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR05MB10064.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58398e54-fe75-4463-c6d3-08db797b333c X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Jun 2023 15:03:45.0038 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: pFBCkaPHkkM5hv3mvMgpTTYGkfzbJnfZTHb6VCE6bR2ibq/riXkEf4Iu5ZutlMwODWMdn0BQAug0OPe06ZnxxQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR05MB9083 X-Rspamd-Queue-Id: 4Qsz9K1zZjz4DDR X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:40.80.0.0/12, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Richard, Please see my answers for your queries, Q. Since you mention two hypervisors - what is the phyiscal network topolog= y in between these two servers? What theoretical link rates would be attain= able? Here is the topology Iperf end points are on 2 different hypervisors. =97=97=97=97=97=97 =97=97=97=97=97=97-=97 = =97=97=97=97=97=97 = =97=97=97=97=97=97-=97 | Linux VM1 | | BSD 13 VM 1 | = | Linux VM2 | |= BSD 13 VM 2 | |___________| |_ ____ ____ ___ | = |___________ | |= _ ____ ____ ___ | | | | = | = | | | = | = | =97=97=97=97=97=97=97=97=97=97=97=97=97=97=97 = =97=97=97=97=97=97=97= =97=97=97=97=97=97=97=97 | ESX Hypervisor 1 | 10G link connected via L2= Switch | ESX Hypervisor 2 | | |=97=97=97=97=97=97=97=97= =97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97 | = | |=97=97=97=97=97=97=97=97=97=97=97=97=97=97 | = |=97=97=97=97=97=97=97= =97=97=97=97=97=97=97| Nic is of 10G capacity on both ESX server and it has below config. Name PCI Driver Link Speed = MAC Address MTU Descrip= tion vmnic4 0000:81:00.0 ixgben Up 10000Mbps Full Du= plex a0:36:9f:61:ca:d4 1500 Intel(R) Ethernet 10G 2P X520= Adapter iperf3 was run between, 1. BSD 13 VM 1 <-> BSD 13 VM 2 2. Linux VM 1 <-> Linux VM 2 BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 102= 4 =3D 1.25 Gbps So theoretically we should see close to 1.25Gbps of Bitrate and we see Linu= x reaching close to this number. But BSD is not able to do that. Q. Did you run iperf3? Did the transmitting endpoint report any retransmiss= ions between Linux or FBSD hosts? Yes, we used iper3. I see Linux doing less number retransmissions compared = to BSD. On BSD, the best performance was around 600 Mbps bitrate and the number of = retransmissions for this number seen is around 32K On Linux, the best performance was around 1.15 Gbps bitrate and the number = of retransmissions for this number seen is only 2K. So as you pointed the number of retransmissions in BSD could be the real is= sue here. Is there a way to reduce this packet loss by fine tuning some parameters w.= r.t ring buffer or any other areas? Q. Did you contrast your cc_cubic findings with cc_newreno? New reno does not perform well with long RTT networks. I could only see 200= Mbps bitrate even with no packet loss. So we did not want to pursue with New Reno with long RTT connections. Hope the answers are clear. Please let me know if you have any other sugges= tions to reduce the packet loss. Regards Murali On 29/06/23, 9:44 PM, "Scheffenegger, Richard" wrote: Hi Murali, > Hello FreeBSD Transport experts, > > We are evaluating performance of FreeBSD 13 VM on ESX hypervisor in long = RTT setup and happened to compare the performance with Linux VM with same h= ypervisor. > We see a substantially better performance with Linux getting close to the= BDP limit, whereas BSD 13 not filling up the pipe enough. > We are trying to figure out what could lead to such a huge difference and= feel we could be missing something here. > > Could you please help us to know if there is a way to make it perform bet= ter? > > Setup details: > > We have 2 ESX hypervisors where 2 VMs (one FreeBSD 13 and one Ubuntu 23.0= 4/Linux kernel 6.2) were launched on each hypervisor. > Then we ran iperf between, > a) BSD 13 <-> BSD 13 > b) Ubuntu <-> Ubuntu Since you mention two hypervisors - what is the phyiscal network topology i= n between these two servers? What theoretical link rates would be attainabl= e? Or did you want to say that each test was run within one and the same hyper= visor, presumably with the same hardware and virtual switch/hypervisor prov= ided on both test scenarios? So that the theoretical bandwidth would be whatever packets can be shuffled= in memory between VM1, Vswitch (with delay simulator), VM2? Did you run iperf3? Did the transmitting endpoint report any retransmission= s between Linux or FBSD hosts? In theory, in a true back-to-back, lossless environment, it should be fairl= y irrelevant which CC you are running; if there are packet drops (retransmi= ssions) in the iperf3 output, please compare their frequency between the Li= nux and FBSD side - a slightly higher incidence rate of packet drops (e.g. = ring-buffer overflows used to be a prominent issue at high speed links) cou= ld possibly explain what appears to be such a huge differential (the square= of the loss probability is relevant for classic TCPs - so a minor change c= an have a huge impact). Did you contrast your cc_cubic findings with cc_newreno? > Even though the network environment were same in both cases, we see Ubunt= u performing much better. > > Below are connection parameters: > Socket buffer: 16MB > TCP CC Algo: Cubic. We used this as this is suitable for Long Fat Network= s. > Ping RTT: 100 ms between the two end points. > We kept all other parameters to default on both Linux and BSD. > > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 1= 024 =3D 1.25 Gbps > > Ubuntu consistently hits around 1 Gbps Bitrate almost reaching the BDP li= mit. > FreeBSD 13 shows a Bit rate between the range of 300-600 Mbps only. So it= seems to be doing half as good as Linux. > For lower socket buffer of 4MB, both FreeBSD and Linux seem to do same an= d able to meet BDP of 300 Mbps consistently. > Larger socket buffer seems to have an issue. > > Please let us know if there are ways to fine tune the system parameters t= o make BSD perform better. > Or any other suggestions/queries welcome. > > Regards > Murali --_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Richard,<= o:p>

&nbs= p;

Please se= e my answers for your queries,

&nbs= p;

&nbs= p;

Q. Since you mention two hypervisors - what is the phyiscal network topology = in between these two servers? What theoretical link rates would be attainab= le?

&nbs= p;

&nbs= p;

Here is t= he topology

&nbs= p;

Iperf end= points are on 2 different hypervisors.

 

=97=97=97=97=97=97 &n= bsp;    =97=97=97=97=97=97-=97           &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;        =97=97=97=97=97=97   &nbs= p;            =97= =97=97=97=97=97-=97    &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;        

|  Linux = VM1 |      |  BSD 13 VM 1  |        =             &nb= sp;            =             &nb= sp;            =             &nb= sp;          |  Linux VM2  |      &nb= sp;         |  BSD 13 VM 2  |

|___________|  &= nbsp;   |_ ____ ____ ___ |&= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;      |___________ |    &n= bsp;           |_ = ____ ____ ___ |

|     =     |   &nb= sp;             = ;        |      =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            =               &= nbsp;           |      =             &nb= sp;             = ;   |

          |        = ;            &n= bsp;     |       &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;              = ;            &n= bsp;          |      =             &nb= sp;             = ;   |

=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97         &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;         =97=97=97=97=97=97=97=97=97=97=97=97=97=97=97

|    &nb= sp;      ESX Hypervisor 1          |      =      10G link connected via L2 Switch          =             |      =      ESX Hypervisor  2      =       |

|    &nb= sp;            =             &nb= sp;            =      |=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97= =97=97   |    =             &nb= sp;             = ;             &= nbsp;    |

|=97=97=97=97=97=97=97=97=97=97=97=97=97=97 |         &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;          |=97=97=97=97=97=97=97=97=97=97=97=97=97=97|

 

 

Nic is of 10G capacity on both ESX server and it has below = config.

 

Name    PCI        &n= bsp;       &= nbsp;           &nbs= p;  Driver    &= nbsp; Link    Speed            &= nbsp;            &nb= sp;             MAC Address                = ;   MTU    Descri= ption        &nbs= p;          

vmnic4  0= 000:81:00.0              &n= bsp; ixgben     = ; Up   10000Mbps      Full Duplex=         a0:36:9f:61:ca:d4          1500   Intel(R) Ethernet 10G 2P X520 Adapter

 

&nbs= p;

iperf3 wa= s run between,

  1. BSD 13 VM 1 <-> BSD 13 VM 2
  2. Linux VM 1 <-> Linux VM 2

&nbs= p;

&nbs= p;

&nbs= p;

BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms= latency) * 8 bits/ 1024 =3D 1.25 Gbps

&nbs= p;

So theore= tically we should see close to 1.25Gbps of Bitrate and we see Linux reachin= g close to this number.

But BSD i= s not able to do that.

&nbs= p;

&nbs= p;

Q. Did you run iperf3? Did the transmitting endpoint= report any retransmissions between Linux or FBSD hosts?

&nbs= p;

Yes, we u= sed iper3. I see Linux doing less number retransmissions compared to BSD.

On BSD, t= he best performance was around 600 Mbps bitrate and the number of retransmi= ssions for this number seen is around 32K

On Linux,= the best performance was around 1.15 Gbps bitrate and the number of retran= smissions for this number seen is only 2K.

So as you= pointed the number of retransmissions in BSD could be the real issue here.=

Is there = a way to reduce this packet loss by fine tuning some parameters w.r.t ring = buffer or any other areas?

&nbs= p;

Q. Did you contrast your cc_cubic findings with cc_newreno?

New reno = does not perform well with long RTT networks. I could only see 200 Mbps bit= rate even with no packet loss.

So we did= not want to pursue with New Reno with long RTT connections.

&nbs= p;

&nbs= p;

Hope the = answers are clear. Please let me know if you have any other suggestions to = reduce the packet loss.

&nbs= p;

Regards

Murali

&nbs= p;

&nbs= p;

&nbs= p;

On 29/06/23, 9:44 PM,= "Scheffenegger, Richard" <rscheff@freebsd.org> wrote:=

Hi Murali,

 

 

> Hello FreeBSD Transport experts,

>

> We are evaluating performance of FreeBSD 13 VM = on ESX hypervisor in long RTT setup and happened to compare the performance= with Linux VM with same hypervisor.

> We see a substantially better performance with = Linux getting close to the BDP limit, whereas BSD 13 not filling up the pip= e enough.

> We are trying to figure out what could lead to = such a huge difference and feel we could be missing something here.

>

> Could you please help us to know if there is a = way to make it perform better?

>

> Setup details:

>

> We have 2 ESX hypervisors where 2 VMs (one Free= BSD 13 and one Ubuntu 23.04/Linux kernel 6.2) were launched on each hypervi= sor.

> Then we ran iperf between,

> a)  &n= bsp;      BSD 13 <-> BSD 13

> b)  &n= bsp;     Ubuntu <-> Ubuntu

 

Since you mention two hypervisors - what is the phyi= scal network topology in between these two servers? What theoretical link r= ates would be attainable?

 

Or did you want to say that each test was run within= one and the same hypervisor, presumably with the same hardware and virtual= switch/hypervisor provided on both test scenarios?

 

So that the theoretical bandwidth would be whatever = packets can be shuffled in memory between VM1, Vswitch (with delay simulato= r), VM2?

 

 

Did you run iperf3? Did the transmitting endpoint re= port any retransmissions between Linux or FBSD hosts?

 

In theory, in a true back-to-back, lossless environm= ent, it should be fairly irrelevant which CC you are running; if there are = packet drops (retransmissions) in the iperf3 output, please compare their f= requency between the Linux and FBSD side - a slightly higher incidence rate of packet drops (e.g. ring-buffer = overflows used to be a prominent issue at high speed links) could possibly = explain what appears to be such a huge differential (the square of the loss= probability is relevant for classic TCPs - so a minor change can have a huge impact).

 

Did you contrast your cc_cubic findings with cc_newr= eno?

 

> Even though the network environment were same i= n both cases, we see Ubuntu performing much better.

>

> Below are connection parameters:

> Socket buffer: 16MB

> TCP CC Algo: Cubic. We used this as this is sui= table for Long Fat Networks.

> Ping RTT:  100 ms between the two end= points.

> We kept all other parameters to default on both= Linux and BSD.

>  

> BDP for 16MB Socket buffer: 16 MB * (1000 ms * = 100ms latency) * 8 bits/ 1024 =3D 1.25 Gbps

>

> Ubuntu consistently hits around 1 Gbps Bitrate = almost reaching the BDP limit.

> FreeBSD 13 shows a Bit rate between the range o= f 300-600 Mbps only. So it seems to be doing half as good as Linux.

> For lower socket buffer of 4MB, both FreeBSD an= d Linux seem to do same and able to meet BDP of 300 Mbps consistently.=

> Larger socket buffer seems to have an issue.

>

> Please let us know if there are ways to fine tu= ne the system parameters to make BSD perform better.

> Or any other suggestions/queries welcome.<= /o:p>

>

> Regards

> Murali

 

--_000_PH0PR05MB10064ECAB726B2636AF410338FB2AAPH0PR05MB10064na_-- From nobody Fri Jun 30 16:05:36 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qt0Xj2gyxz4lHnY for ; Fri, 30 Jun 2023 16:05:41 +0000 (UTC) (envelope-from rscheff@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qt0Xj18h3z3G22; Fri, 30 Jun 2023 16:05:41 +0000 (UTC) (envelope-from rscheff@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688141141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=MJmvqfsiJaJWb3FjN5MD6HYlRS4e2SEY9bQEaSPDow8=; b=CUKv4RLWCdJN8KyYxetqezVEPeBmlwYxUK5Ejdq1XP7yKL/LhskNZtVs5XptFCJUFbLJHG tUTRurGmdT1rOsYEqh2orl3Djf/rOg0b8k5hrXAStbzWn2UALHVjXDJ01xed6o0LtRjytV qH8pQ1wpMQ7jQ6pqhnoSfD6bBr41ro5cW+P+pwAUWhk130taNAhoe/HXuC59kV3O7/q6mW SZaHQ0n/Ma2BXnnZRTYAZeFUXnVGukKuk7Ict67qMfoQE/eOy17K5AbEVabVVxYrBJuaaM BVP/aE8sS5ZPcvQGTrwG0OmqZu2Q5oGSONP3mEQq3qbMcykTOI91UGJmveHtrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688141141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=MJmvqfsiJaJWb3FjN5MD6HYlRS4e2SEY9bQEaSPDow8=; b=o6jLJFYLfy/cJoXpAZ8nXMquEFSZnJaMPJsjfzyco1zdgaaqVWL6jbJrkFCzLJP1kq7jA4 XElc6LM2zLoM21DrbIT2fKHHvDMVsgTXF2c+eZ3hQeWlmXreHD4xZhZw8IpYE5ysR27ua0 KWJa2Bz+n9d1F3izdfZo+/BkEe6CkGRaUSwy3E6U1+eu9YcDAZpxnMY9ZsT/GXWwLVvk2p ubML1T625okAGvPtHYtXm7I36EoXZI0y9Wd+iWM6RUyVXHxpzuTlM929BPf/tOvdzGOIv6 /X3G+x+WQu4wdPHltPiZWidM4NOf6oaH/h8HpPjD0bN2C5Xh2wzBwA2VEswOfw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688141141; a=rsa-sha256; cv=none; b=C+3p57+EAHDI6+SC2JR+5XaaAMrDw0jyhymL19CS06em+uZ6J1it9+yIxS2Bn6jkhHG2+2 olg5Ua/eMlbUuDmup9vQ7U5Uzl3GlppHTHliuYheSkJvp00pessM4fAWNqn1dFKfe7jN9K 1CWDv3mnmDEvzAX05EfACf1pOCKomkF0S2mn2MQujpfsbCWqI9dxRTGy8n3EYr3NrsqrFh BD+aERNskAQ9xqf/+jHN9oraR00mQ81YYEzT8xcj5qD91P3kms9aFYYY6WEWioYp+tOMSW 8Xm11+1uTtTmjeM0nVfPu5rHv8mHUnTa4NVEXj7RXKZ4jSsOZk+5DB25qo05Jw== Received: from [192.168.233.109] (unknown [185.236.167.136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: rscheff/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Qt0Xh4wvVzdDQ; Fri, 30 Jun 2023 16:05:40 +0000 (UTC) (envelope-from rscheff@freebsd.org) Message-ID: <53aff274-b1a8-0730-6971-2755c7e7b688@freebsd.org> Date: Fri, 30 Jun 2023 18:05:36 +0200 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 To: Murali Krishnamurthy , FreeBSD Transport From: "Scheffenegger, Richard" Autocrypt: addr=rscheff@freebsd.org; keydata= xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iziPF0FzrN K1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9yZz7CmgQTFgoAQhYh BDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGABQsJCAcCAyICAQYVCgkICwIE FgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/nvht8kExJ31M+3qpjOqdVypMp+/Ojvh5Z lsk96QEA5HCBkteJcrohwRA7llZvLH3m25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdV AQUBAQdA1Dim8ZWpXRS5i9hb3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2 S7eZrINEWrPNtHEXvliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bA EPqOH+JCIND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8= Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------iU0c3mrMuS0JPe0SkCuH7MCV" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------iU0c3mrMuS0JPe0SkCuH7MCV Content-Type: multipart/mixed; boundary="------------z1bWLEBxMfrUj0cOUFNuxvsc"; protected-headers="v1" From: "Scheffenegger, Richard" To: Murali Krishnamurthy , FreeBSD Transport Message-ID: <53aff274-b1a8-0730-6971-2755c7e7b688@freebsd.org> Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux --------------z1bWLEBxMfrUj0cOUFNuxvsc Content-Type: multipart/mixed; boundary="------------e7087ap1ikMqylv8U1eiUcwi" --------------e7087ap1ikMqylv8U1eiUcwi Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 DQpIaSBNdXJhbGksDQoNCj4gUS4gU2luY2UgeW91IG1lbnRpb24gdHdvIGh5cGVydmlzb3Jz IC0gd2hhdCBpcyB0aGUgcGh5aXNjYWwgbmV0d29yayB0b3BvbG9neSBpbiBiZXR3ZWVuIHRo ZXNlIHR3byBzZXJ2ZXJzPyBXaGF0IHRoZW9yZXRpY2FsIGxpbmsgcmF0ZXMgd291bGQgYmUg YXR0YWluYWJsZT8NCj4gIA0KPiBIZXJlIGlzIHRoZSB0b3BvbG9neQ0KPiANCj4gSXBlcmYg ZW5kIHBvaW50cyBhcmUgb24gMiBkaWZmZXJlbnQgaHlwZXJ2aXNvcnMuIA0KPiANCj4gIOKA lOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlCAgICAgICAg4oCU4oCU4oCU4oCU4oCU 4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCU4oCUICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAg4oCU4oCU4oCU4oCU4oCU4oCUICAgICAgICAgICAgICAgIOKAlOKAlOKAlOKAlOKA lOKAlC3igJQgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICANCj4gfCBMaW51eCBWTTEgfCAgICAgIHwgIEJT RCAxMyBWTSAxICB8ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgTGludXggVk0yICB8 ICAgICAgICAgICAgICAgIHwgIEJTRCAxMyBWTSAyICB8DQo+IHxfX19fX19fX19fX3wgICAg ICB8XyBfX19fIF9fX18gX19fIHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfF9fX19f X19fX19fIHwgICAgICAgICAgICAgICAgfF8gX19fXyBfX19fIF9fXyB8DQo+IHwgICAgICAg ICAgfCAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICB8DQo+ICAgICAgICAgICB8ICAgICAgICAgICAgICAgICAgICAgICAgICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8DQo+IOKAlOKAlOKAlOKAlOKA lOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlCAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICDigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJQNCj4g fCAgICAgICAgICAgRVNYIEh5cGVydmlzb3IgMSAgICAgICAgICB8ICAgICAgICAgICAxMEcg bGluayBjb25uZWN0ZWQgdmlhIEwyIFN3aXRjaCAgICAgICAgICAgICAgICAgICAgICB8ICAg ICAgICAgICBFU1ggSHlwZXJ2aXNvciAgMiAgICAgICAgICAgIHwNCj4gfCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfOKAlOKAlOKAlOKAlOKAlOKA lOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKA lCAgIHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8 DQo+IHzigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJTigJQgfCAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgfOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKAlOKA lOKAlOKAlOKAlOKAlHwNCj4gDQo+IA0KPiBOaWMgaXMgb2YgMTBHIGNhcGFjaXR5IG9uIGJv dGggRVNYIHNlcnZlciBhbmQgaXQgaGFzIGJlbG93IGNvbmZpZy4NCg0KDQpTbywgd2hlbiBi b3RoIFZNcyBydW4gb24gdGhlIHNhbWUgSHlwZXJ2aXNvciwgbWF5YmUgd2l0aCBhbm90aGVy IFZNIHRvIHNpbXVsYXRlIHRoZSAxMDBtcyBkZWxheSwgY2FuIHlvdSBhdHRhaW4gYSBsb3Nz bGVzcyBiYXNlbGluZSBzY2VuYXJpbz8NCg0KDQo+IEJEUCBmb3IgMTZNQiBTb2NrZXQgYnVm ZmVyOiAxNiBNQiAqICgxMDAwIG1zICogMTAwbXMgbGF0ZW5jeSkgKiA4IGJpdHMvIDEwMjQg PSAxLjI1IEdicHMNCj4gDQo+IFNvIHRoZW9yZXRpY2FsbHkgd2Ugc2hvdWxkIHNlZSBjbG9z ZSB0byAxLjI1R2JwcyBvZiBCaXRyYXRlIGFuZCB3ZSBzZWUgTGludXggcmVhY2hpbmcgY2xv c2UgdG8gdGhpcyBudW1iZXIuDQoNClVuZGVyIG5vIGxvc3MsIHllcy4NCg0KDQo+IEJ1dCBC U0QgaXMgbm90IGFibGUgdG8gZG8gdGhhdC4NCj4gDQo+IA0KPiBRLiBEaWQgeW91IHJ1biBp cGVyZjM/IERpZCB0aGUgdHJhbnNtaXR0aW5nIGVuZHBvaW50IHJlcG9ydCBhbnkgcmV0cmFu c21pc3Npb25zIGJldHdlZW4gTGludXggb3IgRkJTRCBob3N0cz8NCj4gDQo+IFllcywgd2Ug dXNlZCBpcGVyMy4gSSBzZWUgTGludXggZG9pbmcgbGVzcyBudW1iZXIgcmV0cmFuc21pc3Np b25zIGNvbXBhcmVkIHRvIEJTRC4gDQo+IE9uIEJTRCwgdGhlIGJlc3QgcGVyZm9ybWFuY2Ug d2FzIGFyb3VuZCA2MDAgTWJwcyBiaXRyYXRlIGFuZCB0aGUgbnVtYmVyIG9mIHJldHJhbnNt aXNzaW9ucyBmb3IgdGhpcyBudW1iZXIgc2VlbiBpcyBhcm91bmQgMzJLDQo+IE9uIExpbnV4 LCB0aGUgYmVzdCBwZXJmb3JtYW5jZSB3YXMgYXJvdW5kIDEuMTUgR2JwcyBiaXRyYXRlIGFu ZCB0aGUgbnVtYmVyIG9mIHJldHJhbnNtaXNzaW9ucyBmb3IgdGhpcyBudW1iZXIgc2VlbiBp cyBvbmx5IDJLLiANCj4gU28gYXMgeW91IHBvaW50ZWQgdGhlIG51bWJlciBvZiByZXRyYW5z bWlzc2lvbnMgaW4gQlNEIGNvdWxkIGJlIHRoZSByZWFsIGlzc3VlIGhlcmUuDQoNClRoZXJl IGFyZSBvdGhlciBjYyBtb2R1bGVzIGF2YWlsYWJsZTsgYnV0IEkgYmVsaWV2ZSBvbmUgbWFq b3IgZGV2aWF0aW9uIGlzIHRoYXQgTGludXggY2FuIHBlcmZvcm0gbWVjaGFuaXNtcyBsaWtl IGh5c3RhcnQ7IEFDS2luZyBldmVyeSBwYWNrZXQgd2hlbiB0aGUgY2xpZW50IGRldGVjdHMg c2xvdyBzdGFydDsgcGVyZm9ybSBwYWNpbmcgdG8gYWNoaWV2ZSBtb3JlIHVuaWZvcm0gcGFj a2V0IHRyYW5zbWlzc2lvbnMuDQoNCkkgdGhpbmsgdGhlIG5leHQgc3RlcCB3b3VsZCBiZSB0 byBmaW5kIG91dCwgYXQgd2hpY2ggcXVldWUgdGhvc2UgcGFja2V0IGRpc2NhcmRzIGFyZSBj b21pbmcgZnJvbSAoZXh0ZXJuYWwgc3dpdGNoPyBkZWxheSBnZW5lcmF0b3I/IFZzd2l0Y2g/ IEV0aCBzdGFjayBpbnNpZGUgdGhlIFZNPykNCg0KT3IgYWx0ZXJuYXRpdmVseSwgcHJvdmlk ZSB5b3VyIEVTWCBoeXBlcnZpc29ycyB3aXRoIHZhc3RseSBtb3JlIGxpbmsgc3BlZWQsIHRv IHJ1bGUgb3V0IGFueSBMMiBpbmR1Y2VkIHBhY2tldCBkcm9wcyAtIHByb3ZpZGVkIHlvdXIg ZGVsYXkgZ2VuZXJhdG9yIGlzIG5vdCB0aGUgc291cmNlIHdoZW4gbW9tZW50YXJpbHkgb3Zl cmxvYWRlZC4NCg0KPiBJcyB0aGVyZSBhIHdheSB0byByZWR1Y2UgdGhpcyBwYWNrZXQgbG9z cyBieSBmaW5lIHR1bmluZyBzb21lIHBhcmFtZXRlcnMgdy5yLnQgcmluZyBidWZmZXIgb3Ig YW55IG90aGVyIGFyZWFzPyANCg0KRmluZGluZyB3aGVyZSB0aGVzZSBhcmlzZSAobG9va2lu ZyBhdCBxdWV1ZSBhbmQgcG9ydCBjb3VudGVycykgd291bGQgYmUgdGhlIG5leHQgc3RlcC4g QnV0IHRoaXMgaXMgbm90IHJlYWxseSBteSBzcGVjaWZpYyBhcmVhIG9mIGV4cGVydGlzZSBi ZXlvbmQgdGhlIGhpZ2ggbGV2ZWwsIHZlbmRvciBpbmRlcGVuZGVudCBvYnNlcnZhdGlvbnMu DQoNClN3aXRjaGluZyB0byBvdGhlciBjYyBtb2R1bGVzIG1heSBnaXZlIHNvbWUgbW9yZSBp bnNpZ2h0cy4gQnV0IGFnYWluLCBJIHN1c3BlY3QgdGhhdCBtb21lbnRhcnkgKG1pY3Jvc2Vj b25kKSBidXJzdGluZXNzIG9mIEJTRCBtYXkgYmUgY2F1c2luZyB0aGlzIHNpZ25pZmljYW50 bHkgaGlnaGVyIGxvc3MgcmF0ZS4NCg0KVENQIFJBQ0sgd291bGQgYmUgYW5vdGhlciBvcHRp b24uIFRoYXQgc3RhY2sgaGFzIHBhY2luZywgbW9yZSBmaW5lLWdyYWluZWQgdGltaW5nLCB0 aGUgUkFDSyBsb3NzIHJlY292ZXJ5IG1lY2hhbmlzbXMgZXRjLiBNYXliZSB0aGF0IGhlbHBz IHJlZHVjZSB0aGUgb2JzZXJ2ZWQgcGFja2V0IGRyb3BzIGJ5IGlwZXJmLCBhbmQgY29uc2Vx dWVudGx5LCB5aWVsZCBhIGhpZ2hlciBvdmVyYWxsIHRocm91aGdwdXQuDQoNCg0KDQo= --------------e7087ap1ikMqylv8U1eiUcwi Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc" Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc" Content-Description: OpenPGP public key Content-Transfer-Encoding: quoted-printable -----BEGIN PGP PUBLIC KEY BLOCK----- xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m 25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb 3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8=3D =3DBwxS -----END PGP PUBLIC KEY BLOCK----- --------------e7087ap1ikMqylv8U1eiUcwi-- --------------z1bWLEBxMfrUj0cOUFNuxvsc-- --------------iU0c3mrMuS0JPe0SkCuH7MCV Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZJ79UAUDAAAAAAAKCRAXvliZ4LFDm00x AQDYClXQqLe3lL/UQ8T7qbDypURrHz7h2yxvDJk90WlklQEAiwDYMpNa+pygWjidVUaP+MY5kKMK tAYxY8AiEeFw2QE= =r/2y -----END PGP SIGNATURE----- --------------iU0c3mrMuS0JPe0SkCuH7MCV-- From nobody Fri Jun 30 16:26:39 2023 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qt1102HmXz4k27M for ; Fri, 30 Jun 2023 16:26:44 +0000 (UTC) (envelope-from muralik1@vmware.com) Received: from CO1PR02CU001.outbound.protection.outlook.com (mail-westus2azlp170110001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c007::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qt10z6lkBz3Llb; Fri, 30 Jun 2023 16:26:43 +0000 (UTC) (envelope-from muralik1@vmware.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F+0PRhK1apKVXl8vlr1CiP+hzz1H3evslWwKvVNbkCmkthDkJYQtC2UcMvIShk0jy09rhGYieJvRZ60ES8pkhx4qu1s0P7YvyeGj3U2gbliBL6BHg2jQIw+svzGBYXzU7VRW1oliDcaTEOHpTOqHZvvywRoJ2bJTISdM7UazOV4MEfQoP59hYtCc/CoNRGzvoce6kLYzWyX3yJORZuuYgcqNyPKtnG2YJx+6Ku1upz4JrMs/H+nwkPFS+DFmgLzVnchTFdcgVnYxBrfi+MO23qRttlMiAr+82ZeUxXus/wbrc6gri1DOk5cZzngCTHz1j5ovhVfiiVN9GPS70FQQOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G34u697bFNTJPyps0xe2sJ0zHCKtFMvlZZfAKSlLkts=; b=ka92ait8/7kHdDHFJM2bGk1Tyiaw2DAHm2Mr3Oj7S4r9oqFrBaVtZDQT1Jl5ZltHmBRNv2BHO1ou6uygSfDwlAJBvmVlQpgdT43mfvaQTu9pohbBT1hBdq1ieb2VzgFDOF9XOe3fi5YbB5taxkSKTONcDmbfClMnJRh3do3buwj3gtdXh82X7tFQ8rc+RXXhkcQkw5domEhjwroN99/O6aTvGpzsRqekPy3Yri+UhNa1l5Z5apfWKPymlLG+iptXB7h0YhTvwW4EKQQ6rufmA4Eia5r06KV8LQxhbIFhobjAaM3obXQtW0f8LWY3632MyBanKx+pNyklBFL6dEUsqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vmware.com; dmarc=pass action=none header.from=vmware.com; dkim=pass header.d=vmware.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G34u697bFNTJPyps0xe2sJ0zHCKtFMvlZZfAKSlLkts=; b=sAxl6ZejDo7DsEQc5H6qBhYmZFqZ9ff6T3IuyiHpSFoW5+FuS/eMZO9w/jbzqNmXpQYVHeTtU+jq/xDxMtKyQ32hBj0BiX+z8f2vbsY+P+diDlasHIvpxaCmkRp3qgE2cgsrR7F3FYv6EsGnPPXtHNA4LJ1LTkXKxfz6NSusrWo= Received: from PH0PR05MB10064.namprd05.prod.outlook.com (2603:10b6:510:29d::8) by DS0PR05MB9620.namprd05.prod.outlook.com (2603:10b6:8:146::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.24; Fri, 30 Jun 2023 16:26:39 +0000 Received: from PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593]) by PH0PR05MB10064.namprd05.prod.outlook.com ([fe80::c63e:75e8:e8bf:a593%4]) with mapi id 15.20.6544.019; Fri, 30 Jun 2023 16:26:39 +0000 From: Murali Krishnamurthy To: "Scheffenegger, Richard" , FreeBSD Transport Subject: Re: FreeBSD TCP (with iperf3) comparison with Linux Thread-Topic: FreeBSD TCP (with iperf3) comparison with Linux Thread-Index: AQHZq2y0eOhY5ZSu6EKxBx6E7aUjbq+jhpqC Date: Fri, 30 Jun 2023 16:26:39 +0000 Message-ID: References: <53aff274-b1a8-0730-6971-2755c7e7b688@freebsd.org> In-Reply-To: <53aff274-b1a8-0730-6971-2755c7e7b688@freebsd.org> Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR05MB10064:EE_|DS0PR05MB9620:EE_ x-ms-office365-filtering-correlation-id: 0bf77515-5d4e-4a84-40fb-08db7986c82d x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: fExt5t536WvsSKNxrBO2eYcnJSJ52DMtlVWJQdOO1vL66AQqqQBYa8ZV5WYfb5jy02FB+MgoMUap/9xketZIY+ylc89SSquu6GykJu4Sy21f+peAafiXopUW6VZs4vFBnqDK96G4cjv7Y3OTqxAt8ZOpRG1rvUUUq8l8bD3I0cpCopxIzR58ObgWtfBRQ6EXENPh1xWWHfp6g7goJPZTf+5w2PxVhtCflDWRQRgfOLMdrtFZPf/6de2YD2I7Rs53MIUbefbeJkOAA3lMteo0qMoLhpky+D7N3hkC4pIsY9pKy1uDFiw6wdhGtlYwao2hUx4Gg+TVVJEJRtGR0NAJEFdWNESkRbUbOmFsQbuBZ6hafR4OxBV4OoDLYMzY26Btd9DPycuhpDjNkb3nnP5pccPmECTLre+YSVtuzfg+LMKM/ROYFS+0+MGocHtjfuPhG6AzAtw8oTwql5OFIVyA/kIJ4UptuppUmcK7e7uz9LjZxlHCViP99CsG7UpaMten2AH1aFQbWvjsDvTSCPNFi8NtutITmnR9Yn6zwkEux+Hat3vg7K8G5JTpmMF5mgQ8m+2zSO7UnY5Z1zG06yviN6wz4FIaYhg5uNBpMZH+oDBMy4vmtuGhx1zhvh3N3HkzN9lUDPtkNHQrRGMOniQEoML3w7jepBrRLqnF5alnwt4= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR05MB10064.namprd05.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(346002)(136003)(376002)(39860400002)(366004)(396003)(451199021)(52536014)(5660300002)(64756008)(66556008)(8936002)(66446008)(76116006)(66946007)(316002)(450100002)(33656002)(66476007)(91956017)(8676002)(38070700005)(478600001)(2906002)(6506007)(110136005)(55016003)(86362001)(41300700001)(26005)(7696005)(53546011)(71200400001)(186003)(9686003)(122000001)(38100700002)(83380400001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?we+zrSevzezuYLUztWAsZiyB/N5xzqG5gC22ZBwbj/nVx6dUJME68fna?= =?Windows-1252?Q?WHSjpba4bh1vS4QoqQ6w7yzagWNqXTx6HIzo2rlbmO7+rkTXmGZt4meu?= =?Windows-1252?Q?yn1AOBfRsr+SWTn9Z5Up+YLMO9vq5XLAl3/Yt6UJ9uEHAcu4ZmMMr8OP?= =?Windows-1252?Q?3XJ/R/+CfYp0Vd9UUALjDKjtk4u/PtD4tN1YbqX7qtXwP6hwvPmzeQkA?= =?Windows-1252?Q?P7GebhLNunVgoJEHRtoKDxWC8KHAAnAs+KH1QRD+f3YZ8hd7N6TGQkU3?= =?Windows-1252?Q?gSM8I0lkuKBpo+uDrjBwmJFIbPRV0g9GfhmabxnJTzVHlbDZphqCn/It?= =?Windows-1252?Q?7/syLPWv16v3Un5leAMbHbfUCnd9FhQ6sOl1eoET1f95JVVEiXZdzSWY?= =?Windows-1252?Q?lhG+6xDV6Fij4KNS2ijROH/hbeWz2qxOBinUmZGZj3KRBkyPLLqZ14dV?= =?Windows-1252?Q?xWp5SPSUIc39ty5OK4F8m2foqWY6CFZgnetdEFc6n6QCJfoRmnH3dRH1?= =?Windows-1252?Q?G0sKTELPlaSRkKDwR2R1qZvQktHlV5x2zfo6oc9fXxcU+YuBDQBJXep5?= =?Windows-1252?Q?55D4JuypYNzkAW/pmnn6e4eKxm9WVBSnlXvQvR+kaRyUu2YsnyMUxLTm?= =?Windows-1252?Q?rxLEWO/0XR0lLyY71KXYV4y9guDtWtEGmYqsYboGfyl4jPU2rNJsejHa?= =?Windows-1252?Q?FP0PI1rkrAazdstTi/AodKhlk4SfvSrL16bRK8S8AOgrXHYBbb10HPhR?= =?Windows-1252?Q?ExPO66ChqoAOo3aYiEmZ6zY1zkt6MF55hgHG4arCCue9yRZB5I2N0FSK?= =?Windows-1252?Q?1z+nrQGypVQrGW94DJvphuzJpzvEXYMQEeHRCdEDxkalfcYcLU/AUe0Q?= =?Windows-1252?Q?TVwJhkCql2oUZLA+5bsT6cLTlnijCl7jFg3wfiSTCmGC/16lHOBlvGU9?= =?Windows-1252?Q?KzU8uboxnVBxkllZlNADgzqYn/0yZkaB36mmuq9Vwd9d3jT+SvGwSZBc?= =?Windows-1252?Q?h6fvQ+PJAHkD6iZC4f3QwZd7lhVo5vgFHCfSyZkLDGaJDgBE58OdmdPD?= =?Windows-1252?Q?PVYrirAjwCmf7gSIvTLR7qfafK9qag46wq9lIq263Po5FbxfabiOm1DZ?= =?Windows-1252?Q?Kc0XG6JmfgfbxjrmXKojO8KHCC5HC7SWN5F6joThg+Ncr1poDdSgeOhz?= =?Windows-1252?Q?tRFnGHFOd65JTax77O7nLBenaqQlxs8sPu/ZcNS6ojf3y8t4artdHgR4?= =?Windows-1252?Q?SyYGvM5ZIYY68RzvaGLJMrTdps495r2thivrKQX6/DKzKz3AFq6DChN7?= =?Windows-1252?Q?THVpuj+Kx5LTUNenAUu7srbXEAnNPuImA2S7kFHA73eTBSQDBFZ8UgAn?= =?Windows-1252?Q?SHF8Uz6L3v0a7WejWnmGp5xqe3W3/gYFSFVpQPLmUM1zEWPkAm2qhpB0?= =?Windows-1252?Q?IZSidf39NLgA6CL4SdAFwkjV3dqtoYhcp1S9XBgQLaVk64kpWi+l+Hu6?= =?Windows-1252?Q?tujyehR8KalRct2JD132wqjkEU+L922lLW5GINqo86n6GuHVwyC1MP8T?= =?Windows-1252?Q?4cuefNsEIVf1+i32W4Fn6hnR9+I9tc5/JpGtPKFJHTtQrJIUdiE7uOEW?= =?Windows-1252?Q?S4LRNrUT8PbuApOQseFN56+CVcaUJzlur8qzxK75+wcv2qJgfeDn42A0?= =?Windows-1252?Q?efq3UWZ7MjE6a8Gde0ixYZVf9nhQEi4/ZtNOdlaSOAAa2/KWM9hCcg?= =?Windows-1252?Q?=3D=3D?= Content-Type: multipart/alternative; boundary="_000_PH0PR05MB100642BD041192E6B7EBDBFE1FB2AAPH0PR05MB10064na_" List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR05MB10064.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0bf77515-5d4e-4a84-40fb-08db7986c82d X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Jun 2023 16:26:39.3457 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ps3qXcNl35pq3Sp7lYVyF59NKCdyDPWd1AiwPOSl0h4g1GrU5jSMCcwaZnrL06j1nkqDXY0D0SLGl1QdkK12Qw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR05MB9620 X-Rspamd-Queue-Id: 4Qt10z6lkBz3Llb X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --_000_PH0PR05MB100642BD041192E6B7EBDBFE1FB2AAPH0PR05MB10064na_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Richard, Appreciate the useful inputs you have shared so far. Will try to figure out= regarding packet drops. Regarding HyStart, I see even BSD code base has support for this. May I kno= w by when can we see that in an release, if not already available ? Regarding this point : =93Switching to other cc modules may give some more = insights. But again, I suspect that momentary (microsecond) burstiness of B= SD may be causing this significantly higher loss rate.=94 Is there some info somewhere where I can understand more on this in detail? Regards Murali On 30/06/23, 9:35 PM, "owner-freebsd-transport@freebsd.org" wrote: Hi Murali, > Q. Since you mention two hypervisors - what is the phyiscal network topol= ogy in between these two servers? What theoretical link rates would be atta= inable? > > Here is the topology > > Iperf end points are on 2 different hypervisors. > > =97=97=97=97=97=97=97=97=97=97=97 =97=97=97=97=97=97=97=97=97=97= =97=97=97=97=97=97 = =97=97=97=97=97=97 =97=97=97=97=97= =97-=97 > | Linux VM1 | | BSD 13 VM 1 | = | Linux VM2 | = | BSD 13 VM 2 | > |___________| |_ ____ ____ ___ | = |___________ | = |_ ____ ____ ___ | > | | | = | = | > | | = | = | > =97=97=97=97=97=97=97=97=97=97=97=97=97=97=97 = =97=97=97=97=97=97=97= =97=97=97=97=97=97=97=97 > | ESX Hypervisor 1 | 10G link connected via = L2 Switch | ESX Hypervisor 2 | > | |=97=97=97=97=97=97=97=97= =97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97 | = | > |=97=97=97=97=97=97=97=97=97=97=97=97=97=97 | = |=97=97=97=97=97=97= =97=97=97=97=97=97=97=97| > > > Nic is of 10G capacity on both ESX server and it has below config. So, when both VMs run on the same Hypervisor, maybe with another VM to simu= late the 100ms delay, can you attain a lossless baseline scenario? > BDP for 16MB Socket buffer: 16 MB * (1000 ms * 100ms latency) * 8 bits/ 1= 024 =3D 1.25 Gbps > > So theoretically we should see close to 1.25Gbps of Bitrate and we see Li= nux reaching close to this number. Under no loss, yes. > But BSD is not able to do that. > > > Q. Did you run iperf3? Did the transmitting endpoint report any retransmi= ssions between Linux or FBSD hosts? > > Yes, we used iper3. I see Linux doing less number retransmissions compare= d to BSD. > On BSD, the best performance was around 600 Mbps bitrate and the number o= f retransmissions for this number seen is around 32K > On Linux, the best performance was around 1.15 Gbps bitrate and the numbe= r of retransmissions for this number seen is only 2K. > So as you pointed the number of retransmissions in BSD could be the real = issue here. There are other cc modules available; but I believe one major deviation is = that Linux can perform mechanisms like hystart; ACKing every packet when th= e client detects slow start; perform pacing to achieve more uniform packet = transmissions. I think the next step would be to find out, at which queue those packet dis= cards are coming from (external switch? delay generator? Vswitch? Eth stack= inside the VM?) Or alternatively, provide your ESX hypervisors with vastly more link speed,= to rule out any L2 induced packet drops - provided your delay generator is= not the source when momentarily overloaded. > Is there a way to reduce this packet loss by fine tuning some parameters = w.r.t ring buffer or any other areas? Finding where these arise (looking at queue and port counters) would be the= next step. But this is not really my specific area of expertise beyond the= high level, vendor independent observations. Switching to other cc modules may give some more insights. But again, I sus= pect that momentary (microsecond) burstiness of BSD may be causing this sig= nificantly higher loss rate. TCP RACK would be another option. That stack has pacing, more fine-grained = timing, the RACK loss recovery mechanisms etc. Maybe that helps reduce the = observed packet drops by iperf, and consequently, yield a higher overall th= rouhgput. --_000_PH0PR05MB100642BD041192E6B7EBDBFE1FB2AAPH0PR05MB10064na_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Richard,<= o:p>

&nbs= p;

Appreciat= e the useful inputs you have shared so far. Will try to figure out regardin= g packet drops.

&nbs= p;

Regarding= HyStart, I see even BSD code base has support for this. May I know by when= can we see that in an release, if not already available ?

&nbs= p;

Regarding= this point : =93Switching to other cc modules may give some more insigh= ts. But again, I suspect that momentary (microsecond) burstiness of BSD may= be causing this significantly higher loss rate.=94

Is there = some info somewhere where I can understand more on this in detail?

&nbs= p;

Regards

Murali

&nbs= p;

&nbs= p;

On 30/06/23, 9:35 PM,= "owner-freebsd-transport@freebsd.org" <owner-freebsd-transpor= t@freebsd.org> wrote:

 

Hi Murali,

 

> Q. Since you mention two hypervisors - what is = the phyiscal network topology in between these two servers? What theoretica= l link rates would be attainable?

>  

> Here is the topology

>

> Iperf end points are on 2 different hypervisors= .

>

>  =97=97=97=97=97=97=97=97=97=97=97&nb= sp;       =97=97=97=97=97=97=97=97=97=97= =97=97=97=97=97=97         &nb= sp;            =             &nb= sp;            =             &nb= sp;            =          =97=97=97=97=97=97 &n= bsp;            = ;  =97=97=97=97=97=97-=97      &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;      

> | Linux VM1 |     &nbs= p;|  BSD 13 VM 1  |      =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            |&nbs= p; Linux VM2  |       &nb= sp;        |  BSD 13 VM 2=   |

> |___________|     &nbs= p;|_ ____ ____ ___ |         &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;          |___________ = |            &n= bsp;   |_ ____ ____ ___ |

> |       &nbs= p;  |          =             &nb= sp;  |          &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp;        |    &n= bsp;            = ;            &n= bsp;     |

>        =    |          &= nbsp;           &nbs= p;   |         =             &nb= sp;            =             &nb= sp;            =             &nb= sp;            =             &nb= sp;          |  &nbs= p;            &= nbsp;           &nbs= p;       |

> =97=97=97=97=97=97=97=97=97=97=97=97=97=97=97&n= bsp;            = ;            &n= bsp;            = ;            &n= bsp;            = ;            &n= bsp;      =97=97=97=97=97=97=97=97=97=97=97= =97=97=97=97

> |       &nbs= p;   ESX Hypervisor 1       &n= bsp;  |         &nbs= p; 10G link connected via L2 Switch      &nbs= p;            &= nbsp;  |         &nb= sp; ESX Hypervisor  2       &n= bsp;    |

> |       &nbs= p;            &= nbsp;           &nbs= p;            &= nbsp; |=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97=97= =97=97   |         &= nbsp;           &nbs= p;            &= nbsp;           &nbs= p; |

> |=97=97=97=97=97=97=97=97=97=97=97=97=97=97 |&n= bsp;            = ;            &n= bsp;            = ;            &n= bsp;            = ;            &n= bsp;       |=97=97=97=97=97=97=97=97=97=97=97= =97=97=97|

>

>

> Nic is of 10G capacity on both ESX server and i= t has below config.

 

 

So, when both VMs run on the same Hypervisor, maybe = with another VM to simulate the 100ms delay, can you attain a lossless base= line scenario?

 

 

> BDP for 16MB Socket buffer: 16 MB * (1000 ms * = 100ms latency) * 8 bits/ 1024 =3D 1.25 Gbps

>

> So theoretically we should see close to 1.25Gbp= s of Bitrate and we see Linux reaching close to this number.

 

Under no loss, yes.

 

 

> But BSD is not able to do that.

>

>

> Q. Did you run iperf3? Did the transmitting end= point report any retransmissions between Linux or FBSD hosts?

>

> Yes, we used iper3. I see Linux doing less numb= er retransmissions compared to BSD.

> On BSD, the best performance was around 600 Mbp= s bitrate and the number of retransmissions for this number seen is around = 32K

> On Linux, the best performance was around 1.15 = Gbps bitrate and the number of retransmissions for this number seen is only= 2K.

> So as you pointed the number of retransmissions= in BSD could be the real issue here.

 

There are other cc modules available; but I believe = one major deviation is that Linux can perform mechanisms like hystart; ACKi= ng every packet when the client detects slow start; perform pacing to achie= ve more uniform packet transmissions.

 

I think the next step would be to find out, at which= queue those packet discards are coming from (external switch? delay genera= tor? Vswitch? Eth stack inside the VM?)

 

Or alternatively, provide your ESX hypervisors with = vastly more link speed, to rule out any L2 induced packet drops - provided = your delay generator is not the source when momentarily overloaded.

 

> Is there a way to reduce this packet loss by fi= ne tuning some parameters w.r.t ring buffer or any other areas?

 

Finding where these arise (looking at queue and port= counters) would be the next step. But this is not really my specific area = of expertise beyond the high level, vendor independent observations.

 

Switching to other cc modules may give some more ins= ights. But again, I suspect that momentary (microsecond) burstiness of BSD = may be causing this significantly higher loss rate.

 

TCP RACK would be another option. That stack has pac= ing, more fine-grained timing, the RACK loss recovery mechanisms etc. Maybe= that helps reduce the observed packet drops by iperf, and consequently, yi= eld a higher overall throuhgput.

 

 

 

 

--_000_PH0PR05MB100642BD041192E6B7EBDBFE1FB2AAPH0PR05MB10064na_--