From owner-freebsd-net@freebsd.org Sat Nov 21 18:24:35 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A43746D566 for ; Sat, 21 Nov 2020 18:24:35 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from mail.farley.org (farley.org [104.129.130.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cdhdt4XVFz3GSZ for ; Sat, 21 Nov 2020 18:24:31 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from thor.farley.org (thor.farley.org [192.168.1.5]) by mail.farley.org (8.16.1/8.16.1) with ESMTP id 0ALIONGC077813 for ; Sat, 21 Nov 2020 13:24:23 -0500 (EST) (envelope-from scf@FreeBSD.org) Date: Sat, 21 Nov 2020 13:24:23 -0500 (EST) From: "Sean C. Farley" To: freebsd-net@FreeBSD.org Subject: Determining cause of transfer limit Message-ID: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Spam-Status: No, score=-1.0 required=4.0 tests=ALL_TRUSTED,SHORTCIRCUIT shortcircuit=ham autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mail.farley.org X-Rspamd-Queue-Id: 4Cdhdt4XVFz3GSZ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:396949, ipnet:104.129.130.0/24, country:US]; local_wl_from(0.00)[FreeBSD.org] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 18:24:35 -0000 I have recently upped my Internet service and have now noticed a limit being reached, but I am not certain which limit and best option to resolve it. I am using a circa 2007 system as a multi-purpose router running FreeBSD 12-STABLE (r367740). The issue is that it maxes out around 400Mb/s when running a speed test through it between my workstation and various test sites (i.e., DSL Reports and Speedtest). There are two NIC's (both are Intel 82541PI) in use with one to the ISP and one to my workstation. At first, I saw one of them apparently hitting an interrupt rate of just over 8000, so I bumped their rate limits higher with little to no improvement. What makes me believe I can theoretically get faster speeds is that I can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's and nearly double the speed. The difference is that it is on the PCI-E bus and has MSI support. irq16: em0:irq0+ irq17: em1:irq0 irq20: hpet0 irq258: mskc0 I have many network settings, but changing them did nothing. Here are the settings I am trying now that seem to squeak a little extra performance. The commented-out lines are ones I tried without seeing any change. I have also tested without these settings. /boot/loader.conf hw.em.rx_process_limit="-1" # dev.em.0.iflib.override_nrxds="2048" # dev.em.1.iflib.override_nrxds="2048" # dev.em.2.iflib.override_nrxds="2048" # dev.em.0.iflib.override_ntxds="2048" # net.link.ifqmaxlen="2048" hw.em.max_interrupt_rate="32000" # net.isr.maxthreads="-1" # net.isr.bindthreads="1" /etc/sysctl.conf kern.random.harvest.mask=351 dev.em.0.fc=0 dev.em.1.fc=0 dev.em.0.itr=122 # Allow past 8000 interrupts/second. dev.em.1.itr=122 net.inet.ip.redirect=0 net.inet6.ip6.redirect=0 Increasing these from 66 to 250 did not help: hw.em.rx_abs_int_delay: 66 hw.em.tx_abs_int_delay: 66 hw.em.tx_int_delay: 66 I am utilizing pf, but I doubt it is the issue since using the same rules with the msk driver would have held the speed down to 400Mb/s. Am I hitting the limit of the PCI bus (memory or interrupt) or something else? I can buy a new PCI-E NIC for the internal network, but I rather fully utilize the Intel NIC's I have, if possible. Sean -- scf@FreeBSD.org