From owner-freebsd-net@freebsd.org Fri Nov 27 20:55:22 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 10B0246899A for ; Fri, 27 Nov 2020 20:55:22 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from mail.farley.org (farley.org [104.129.130.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CjRj53vZGz4SRR for ; Fri, 27 Nov 2020 20:55:21 +0000 (UTC) (envelope-from scf@FreeBSD.org) Received: from thor.farley.org (thor.farley.org [192.168.1.5] (may be forged)) by mail.farley.org (8.16.1/8.16.1) with ESMTP id 0ARKi2wc010844; Fri, 27 Nov 2020 15:44:02 -0500 (EST) (envelope-from scf@FreeBSD.org) Date: Fri, 27 Nov 2020 15:44:02 -0500 (EST) From: "Sean C. Farley" To: Michael Sierchio cc: "freebsd-net@freebsd.org" Subject: Re: Determining cause of transfer limit In-Reply-To: Message-ID: References: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Spam-Status: No, score=-1.0 required=4.0 tests=ALL_TRUSTED,SHORTCIRCUIT shortcircuit=ham autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on mail.farley.org X-Rspamd-Queue-Id: 4CjRj53vZGz4SRR X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; ASN(0.00)[asn:396949, ipnet:104.129.130.0/24, country:US] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2020 20:55:22 -0000 On Sat, 21 Nov 2020, Michael Sierchio wrote: > Sorry for the top post. Have you tried device polling? From > /usr/src/sys/amd64/conf/NOTES: ... > options DEVICE_POLLING *snip* I had not tried it and had actually forgotten about that capability. Anyway, I just tried it (kernel built with polling and enabled via ifconfig on the devices) and saw no observable change. I am not certain if the code paths can really make use of it with the switch to iflib. However, I was able to squeak a bit more from the NIC's by the equivalent of polling(4) by bumping the delay for interrupts with the key one being hw.em.rx_int_delay. It improved transfers going through the router, in both directions, by about 20Mb/s. The number of interrupts dropped a lot, so I doubt I am hitting an interrupt limit. /boot/loader.conf: hw.em.rx_int_delay="264" hw.em.rx_abs_int_delay="264" hw.em.tx_int_delay="264" hw.em.tx_abs_int_delay="264" One other thing to note, is that when I am testing going through the router, there are many drops and stalls/restarts when downloading but not uploading. Before test: dev.em.1.iflib.txq0.r_abdications: 0 dev.em.1.iflib.txq0.r_restarts: 0 dev.em.1.iflib.txq0.r_stalls: 0 dev.em.1.iflib.txq0.r_starts: 9239 dev.em.1.iflib.txq0.r_drops: 0 dev.em.1.iflib.txq0.r_enqueues: 9240 dev.em.1.iflib.txq0.txq_cleaned: 9640 dev.em.1.iflib.txq0.txq_processed: 9680 After test: dev.em.1.iflib.txq0.r_abdications: 14 dev.em.1.iflib.txq0.r_restarts: 7419 dev.em.1.iflib.txq0.r_stalls: 7419 dev.em.1.iflib.txq0.r_starts: 411677 dev.em.1.iflib.txq0.r_drops: 198 dev.em.1.iflib.txq0.r_enqueues: 878486 dev.em.1.iflib.txq0.txq_cleaned: 875005 dev.em.1.iflib.txq0.txq_processed: 875045 After 2nd test with Marvell as outgoing NIC starting from zero: dev.em.1.iflib.txq0.r_abdications: 1 dev.em.1.iflib.txq0.r_restarts: 7647 dev.em.1.iflib.txq0.r_stalls: 7647 dev.em.1.iflib.txq0.r_starts: 1092018 dev.em.1.iflib.txq0.r_drops: 811 dev.em.1.iflib.txq0.r_enqueues: 1570425 dev.em.1.iflib.txq0.txq_cleaned: 1567629 dev.em.1.iflib.txq0.txq_processed: 1567671 Unless someone knows anything else, I think I am going to have to buy a PCI-E NIC for the router to replace one of the Intel NIC's. I will either use it along with the Marvell or the other Intel NIC. When testing between my workstation and any one Intel NIC on the router using iperf3, I can achieve just over 900Mb/s. Sean -- scf@FreeBSD.org > On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley wrote: > >> I have recently upped my Internet service and have now noticed a limit >> being reached, but I am not certain which limit and best option to >> resolve it. >> >> I am using a circa 2007 system as a multi-purpose router running FreeBSD >> 12-STABLE (r367740). The issue is that it maxes out around 400Mb/s when >> running a speed test through it between my workstation and various test >> sites (i.e., DSL Reports and Speedtest). There are two NIC's (both are >> Intel 82541PI) in use with one to the ISP and one to my workstation. >> >> At first, I saw one of them apparently hitting an interrupt rate of just >> over 8000, so I bumped their rate limits higher with little to no >> improvement. >> >> What makes me believe I can theoretically get faster speeds is that I >> can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's >> and nearly double the speed. The difference is that it is on the PCI-E >> bus and has MSI support. >> >> irq16: em0:irq0+ >> irq17: em1:irq0 >> irq20: hpet0 >> irq258: mskc0 >> >> I have many network settings, but changing them did nothing. Here are >> the settings I am trying now that seem to squeak a little extra >> performance. The commented-out lines are ones I tried without seeing >> any change. I have also tested without these settings. >> >> /boot/loader.conf >> hw.em.rx_process_limit="-1" >> # dev.em.0.iflib.override_nrxds="2048" >> # dev.em.1.iflib.override_nrxds="2048" >> # dev.em.2.iflib.override_nrxds="2048" >> # dev.em.0.iflib.override_ntxds="2048" >> # net.link.ifqmaxlen="2048" >> hw.em.max_interrupt_rate="32000" >> # net.isr.maxthreads="-1" >> # net.isr.bindthreads="1" >> >> /etc/sysctl.conf >> kern.random.harvest.mask=351 >> dev.em.0.fc=0 >> dev.em.1.fc=0 >> dev.em.0.itr=122 # Allow past 8000 interrupts/second. >> dev.em.1.itr=122 >> net.inet.ip.redirect=0 >> net.inet6.ip6.redirect=0 >> >> Increasing these from 66 to 250 did not help: >> hw.em.rx_abs_int_delay: 66 >> hw.em.tx_abs_int_delay: 66 >> hw.em.tx_int_delay: 66 >> >> I am utilizing pf, but I doubt it is the issue since using the same >> rules with the msk driver would have held the speed down to 400Mb/s. >> >> Am I hitting the limit of the PCI bus (memory or interrupt) or something >> else? I can buy a new PCI-E NIC for the internal network, but I rather >> fully utilize the Intel NIC's I have, if possible. >> >> Sean