From owner-freebsd-net@freebsd.org Sat Dec 12 03:07:02 2020 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 02FD647D700 for <freebsd-net@mailman.nyi.freebsd.org>; Sat, 12 Dec 2020 03:07:02 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from mail.farley.org (farley.org [104.129.130.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CtCHT4Z9Wz3qLG for <freebsd-net@FreeBSD.org>; Sat, 12 Dec 2020 03:07:01 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from thor.farley.org (thor.farley.org [192.168.1.5]) by mail.farley.org (8.16.1/8.16.1) with ESMTP id 0BC36qln015779 for <freebsd-net@FreeBSD.org>; Fri, 11 Dec 2020 22:06:52 -0500 (EST) (envelope-from scf@FreeBSD.org) Date: Fri, 11 Dec 2020 22:06:52 -0500 (EST) From: "Sean C. Farley" <scf@FreeBSD.org> To: freebsd-net@FreeBSD.org Subject: Re: Determining cause of transfer limit In-Reply-To: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> Message-ID: <814fa4a1-9771-53bc-16fa-ebfdb5fbbf7e@FreeBSD.org> References: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spam-Status: No, score=-1.0 required=4.0 tests=ALL_TRUSTED,SHORTCIRCUIT shortcircuit=ham autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mail.farley.org X-Rspamd-Queue-Id: 4CtCHT4Z9Wz3qLG X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; ASN(0.00)[asn:396949, ipnet:104.129.130.0/24, country:US] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sat, 12 Dec 2020 03:07:02 -0000 On Sat, 21 Nov 2020, Sean C. Farley wrote: > I have recently upped my Internet service and have now noticed a limit being > reached, but I am not certain which limit and best option to resolve it. > > I am using a circa 2007 system as a multi-purpose router running FreeBSD > 12-STABLE (r367740). The issue is that it maxes out around 400Mb/s when > running a speed test through it between my workstation and various test sites > (i.e., DSL Reports and Speedtest). There are two NIC's (both are Intel > 82541PI) in use with one to the ISP and one to my workstation. > > At first, I saw one of them apparently hitting an interrupt rate of just over > 8000, so I bumped their rate limits higher with little to no improvement. > > What makes me believe I can theoretically get faster speeds is that I can use > the onboard NIC (Marvell 88E8056) to replace one of the NIC's and nearly > double the speed. The difference is that it is on the PCI-E bus and has MSI > support. > > irq16: em0:irq0+ > irq17: em1:irq0 > irq20: hpet0 > irq258: mskc0 > > I have many network settings, but changing them did nothing. Here are the > settings I am trying now that seem to squeak a little extra performance. The > commented-out lines are ones I tried without seeing any change. I have also > tested without these settings. > > /boot/loader.conf > hw.em.rx_process_limit="-1" > # dev.em.0.iflib.override_nrxds="2048" > # dev.em.1.iflib.override_nrxds="2048" > # dev.em.2.iflib.override_nrxds="2048" > # dev.em.0.iflib.override_ntxds="2048" > # net.link.ifqmaxlen="2048" > hw.em.max_interrupt_rate="32000" > # net.isr.maxthreads="-1" > # net.isr.bindthreads="1" > > /etc/sysctl.conf > kern.random.harvest.mask=351 > dev.em.0.fc=0 > dev.em.1.fc=0 > dev.em.0.itr=122 # Allow past 8000 interrupts/second. > dev.em.1.itr=122 > net.inet.ip.redirect=0 > net.inet6.ip6.redirect=0 > > Increasing these from 66 to 250 did not help: > hw.em.rx_abs_int_delay: 66 > hw.em.tx_abs_int_delay: 66 > hw.em.tx_int_delay: 66 > > I am utilizing pf, but I doubt it is the issue since using the same rules > with the msk driver would have held the speed down to 400Mb/s. > > Am I hitting the limit of the PCI bus (memory or interrupt) or something > else? I can buy a new PCI-E NIC for the internal network, but I rather fully > utilize the Intel NIC's I have, if possible. Well, I surrendered with using those NIC's. If anyone runs across this thread in the future, I want to let you know that I fixed my issue by buying a used HP multi-port NIC using the Intel 82580 chipset. The only setting I made was to turn off flowcontrol. I may experiment with tweaks (such as bumping dev.igb.N.iflib.override_nrxds) in the future, but it looks like I am in a much better situation already. Sean -- scf@FreeBSD.org