From owner-freebsd-net@freebsd.org Sat Nov 21 22:22:07 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 960D4471E67 for ; Sat, 21 Nov 2020 22:22:07 +0000 (UTC) (envelope-from prvs=9594a43ac5=mark.saad@lucera.com) Received: from mx0b-0017d602.pphosted.com (mx0b-0017d602.pphosted.com [148.163.153.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.pphosted.com", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cdnvz2qwrz3jVj; Sat, 21 Nov 2020 22:22:07 +0000 (UTC) (envelope-from prvs=9594a43ac5=mark.saad@lucera.com) Received: from pps.filterd (m0101310.ppops.net [127.0.0.1]) by mx0b-0017d602.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 0ALMG7fJ017318; Sat, 21 Nov 2020 17:22:06 -0500 Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2101.outbound.protection.outlook.com [104.47.55.101]) by mx0b-0017d602.pphosted.com with ESMTP id 34yad6r3e5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 21 Nov 2020 17:22:05 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UU55d95yuRlBKYVSNn2Au5GPNQQsfQ30qgxtCgYnamqBBHuI9yxNZ/eVNb+eJ8EKMYnuxwEc34FVRn/O+qqoblrxUnm7tgOAGIejDmNaCSxyHWHpWhtcFmM/POOBDbx13yv7efDrTbmQjJnh/iUp0TX4Spr4deTnQ8e3P4RffZvVRjUwmD/8J/uqRWiiHLEpJxTMY0VjXv+6wxYt8gfmt4rAnPWw5jYZLhR5nslvCFpVIpIeE4Az03h6/wSbFWd0s1k8mPF8jqBz0Md3BeWgFuqOl9EIXVUO0FhVJOpsNPJeSJe4SnEv702EZ1+lwUl/p7gPDSgKSp3IlPIGeqlb+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SvigKgEz0s7tMOB/s05FIUeTuuX+8wiTADRfRIL6cZ4=; b=NeKoIzftMKS7XZUNevqeq4OMYOzSnL+s55EUyI/AKrkBuLMd1gFFCLsitzcLgd83mC8b4T8PDKblMS3MrsxvF1l9ZE5AP/EMdRzC14S2oz/QIvkPxJ9j1gc9wrO+4UG31ElLxD+EFs7PSxRi5aKnOIPi+sYKcXluVGMZSZ8zcATKTmKHJrYoZMSrEizmjaht1GKe6ym2UYqkwloorYihhn6PZA4fkQ8qq4J15TIYOWYm0ezYs4o+J+WBhXSjpe6xL6gDXHGnOEJx3sk1X2/PjwH/tZqaZT/P3QF3CTDYFz003NQ/BFwKgbO0dGltJwEk3W2nyHK1nmDep2vH5jdtbA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=lucera.com; dmarc=pass action=none header.from=lucera.com; dkim=pass header.d=lucera.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bgccs.onmicrosoft.com; s=selector1-bgccs-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SvigKgEz0s7tMOB/s05FIUeTuuX+8wiTADRfRIL6cZ4=; b=mjv8fe88pj7zFR9AiZNG79f9mTtO87aEZsb0ZXAgP1i5mLksXMVGMEkHOPmh5C2McBqqoNi/TGoPLQvl5TsBUpl8mETBUE+p6UDc4C51L4GqjZ/SCU0eHW3pCgBf4f6Msn5BNYCOBIlym4rAu5tEb67cVdAS3n15Sgka1hB7hws= Received: from BL0PR12MB4756.namprd12.prod.outlook.com (2603:10b6:208:8d::29) by BL0PR12MB4691.namprd12.prod.outlook.com (2603:10b6:208:84::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.22; Sat, 21 Nov 2020 22:22:00 +0000 Received: from BL0PR12MB4756.namprd12.prod.outlook.com ([fe80::ecf2:6e89:1581:11a]) by BL0PR12MB4756.namprd12.prod.outlook.com ([fe80::ecf2:6e89:1581:11a%7]) with mapi id 15.20.3564.035; Sat, 21 Nov 2020 22:22:00 +0000 From: "Saad, Mark" To: Michael Sierchio , "Sean C. Farley" CC: "freebsd-net@freebsd.org" Subject: Re: Determining cause of transfer limit Thread-Topic: Determining cause of transfer limit Thread-Index: AQHWwDOXAENA8npM3Ui/vCDCatVqsKnTIbUAgAAHA/k= Date: Sat, 21 Nov 2020 22:22:00 +0000 Message-ID: References: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [96.232.87.29] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c6be99bd-db83-440f-e3c6-08d88e6bdd90 x-ms-traffictypediagnostic: BL0PR12MB4691: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2512; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Ms9zhZCgVObWUeMNSZDM9J/iRG3KCnAFTEx/+xKv6o91+v3/6S+FICXRVQdO3gd0b9bna+gGtNm/0iqclZKVh6yFUX/HkoWCi2nKzHcbuT/hsgiTpzrhn0unNOUgbd46Hjoq8wDgI+KoU0JBH5kAcvOMN43yzKDPe3zBp/MvLDT5wM7T5Hic66+43Y1ZAxBb85Y/0lwirjv+oQ0xSu6Bz9hDK1UvcSv8gczWGcfGy9XJG69mST6xgTInGUW858ykc2toGeZpEa+7pq+YtT3ZLs+o8rROT9DJnfT+mjqHRRwRaaJAGnh512phDSrZa2Qma8AGOsGoSnHRZknK70+Q1kEdz0A6MYb4LOrkXcbCXMVvBJe/NvPwZ85p5z2CcZ9chlIsQpYvN6bue0Wg3fyWzg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4756.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(39860400002)(376002)(136003)(396003)(7696005)(71200400001)(9686003)(110136005)(8676002)(6506007)(478600001)(52536014)(316002)(53546011)(66446008)(8936002)(33656002)(76116006)(4326008)(186003)(66556008)(26005)(64756008)(66476007)(66946007)(966005)(5660300002)(83380400001)(55016002)(66574015)(86362001)(2906002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: hD1HmpgHKTyzCyidMCcbtEeo+Eemkv7pne7jTEfgwkLv6ljxOtWhH2mdsu8lXZ9zpKOagA9qpHW2AaExLpND8y66kydVUFaBf4oGkaUeKYxWET3ywaon1v62GL+3i9ZsloP5vpCF6I6jwbsCJ/p9/BNP4DT13mC8dw9BwDj4OJqqpow7Km+jwPl1uVCO7kj7RyCNEQtEVNWbMaBjGkYKWyD5bXBHoH+7DS/35t4TI+KHrgwlHDxa5FIAEPsvRgaB5CYEOmh9NdP21rIKDcU/RMUtlZPigGAupxwhm+1J+ABRW3WnGyOim1qFFt7PTc7E6uMpev6ppPci7X+pjgRsQWri9le7r8pYV8hCXCi4ins1mzvKk0gqYPR2T/Mn4EAQvoHRPUOETvL9SYXuc8bDobdLtsm+C2/RNrwEjZMco8dR/yDrMYazdMenN132zZzSn8c3D6uzm7qz9XhZJowVngzrhnVANv4uhbMY64jX9QpxStv1d51/WSXWOoXxUs9bxijAC/6cPfe+bJNLRIWZZ3AUxVkqkkCt3k4VFNut8ws8Pb64/kKaLSLMF1YwP9nMukQt03ZmWqtttJKhZ9eb8nQcQktHp1sAqWV3SZipC42g3GdWuU3h0aAHWKZTqrzEkxl4Ti1+vP2Vy45+IlRqaw== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-4" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: lucera.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4756.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c6be99bd-db83-440f-e3c6-08d88e6bdd90 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Nov 2020 22:22:00.0701 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: cfeb5f5e-839a-44b8-ab46-47157d8f1241 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: m9ZJQdWq6d6Mbdj+Ywdcfklp1C3tLcGNhVOdPI3obTl5lAqCSrYJN5PphmXS1iFCb0kKM9tbyTGYLuo7dCy4Xw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4691 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-21_04:2020-11-20, 2020-11-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 mlxscore=0 impostorscore=0 bulkscore=0 phishscore=0 adultscore=0 mlxlogscore=999 clxscore=1011 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011210156 X-Rspamd-Queue-Id: 4Cdnvz2qwrz3jVj X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 22:22:07 -0000 scf can I use polling on sfxge ? Also if I want to use polling on say ix or s= fxge does this break things like pf, bird or nsd ? --- Mark Saad Lucera Financial Infrastructures, LLC msaad@lucera.com ________________________________________ From: owner-freebsd-net@freebsd.org on beha= lf of Michael Sierchio Sent: Saturday, November 21, 2020 4:55 PM To: Sean C. Farley Cc: freebsd-net@freebsd.org Subject: Re: Determining cause of transfer limit Sorry for the top post. Have you tried device polling? From /usr/src/sys/amd64/conf/NOTES: ##################################################################### # NETWORKING OPTIONS # # DEVICE_POLLING adds support for mixed interrupt-polling handling # of network device drivers, which has significant benefits in terms # of robustness to overloads and responsivity, as well as permitting # accurate scheduling of the CPU time between kernel network processing # and other activities. The drawback is a moderate (up to 1/HZ seconds) # potential increase in response times. # It is strongly recommended to use HZ=3D1000 or 2000 with DEVICE_POLLING # to achieve smoother behaviour. # Additionally, you can enable/disable polling at runtime with help of # the ifconfig(8) utility, and select the CPU fraction reserved to # userland with the sysctl variable kern.polling.user_frac # (default 50, range 0..100). # # Not all device drivers support this mode of operation at the time of # this writing. See polling(4) for more details. options DEVICE_POLLING On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley wrote: > I have recently upped my Internet service and have now noticed a limit > being reached, but I am not certain which limit and best option to > resolve it. > > I am using a circa 2007 system as a multi-purpose router running FreeBSD > 12-STABLE (r367740). The issue is that it maxes out around 400Mb/s when > running a speed test through it between my workstation and various test > sites (i.e., DSL Reports and Speedtest). There are two NIC's (both are > Intel 82541PI) in use with one to the ISP and one to my workstation. > > At first, I saw one of them apparently hitting an interrupt rate of just > over 8000, so I bumped their rate limits higher with little to no > improvement. > > What makes me believe I can theoretically get faster speeds is that I > can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's > and nearly double the speed. The difference is that it is on the PCI-E > bus and has MSI support. > > irq16: em0:irq0+ > irq17: em1:irq0 > irq20: hpet0 > irq258: mskc0 > > I have many network settings, but changing them did nothing. Here are > the settings I am trying now that seem to squeak a little extra > performance. The commented-out lines are ones I tried without seeing > any change. I have also tested without these settings. > > /boot/loader.conf > hw.em.rx_process_limit=3D"-1" > # dev.em.0.iflib.override_nrxds=3D"2048" > # dev.em.1.iflib.override_nrxds=3D"2048" > # dev.em.2.iflib.override_nrxds=3D"2048" > # dev.em.0.iflib.override_ntxds=3D"2048" > # net.link.ifqmaxlen=3D"2048" > hw.em.max_interrupt_rate=3D"32000" > # net.isr.maxthreads=3D"-1" > # net.isr.bindthreads=3D"1" > > /etc/sysctl.conf > kern.random.harvest.mask=3D351 > dev.em.0.fc=3D0 > dev.em.1.fc=3D0 > dev.em.0.itr=3D122 # Allow past 8000 interrupts/second. > dev.em.1.itr=3D122 > net.inet.ip.redirect=3D0 > net.inet6.ip6.redirect=3D0 > > Increasing these from 66 to 250 did not help: > hw.em.rx_abs_int_delay: 66 > hw.em.tx_abs_int_delay: 66 > hw.em.tx_int_delay: 66 > > I am utilizing pf, but I doubt it is the issue since using the same > rules with the msk driver would have held the speed down to 400Mb/s. > > Am I hitting the limit of the PCI bus (memory or interrupt) or something > else? I can buy a new PCI-E NIC for the internal network, but I rather > fully utilize the Intel NIC's I have, if possible. > > Sean > -- > scf@FreeBSD.org > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- "Well," Brahm=E0 said, "even after ten thousand explanations, a fool is no wiser, but an intelligent person requires only two thousand five hundred." - The Mah=E0bh=E0rata _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"