Date: Fri, 27 Nov 2020 15:44:02 -0500 (EST) From: "Sean C. Farley" <scf@FreeBSD.org> To: Michael Sierchio <kudzu@tenebras.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Determining cause of transfer limit Message-ID: <e4295db8-bf19-eb41-f6d7-f2c5d93657e7@FreeBSD.org> In-Reply-To: <CAHu1Y73dxUeAA1cMboU4TBsHWYDsa-C9F7P2iACbrWQN2f%2BygA@mail.gmail.com> References: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> <CAHu1Y73dxUeAA1cMboU4TBsHWYDsa-C9F7P2iACbrWQN2f%2BygA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 21 Nov 2020, Michael Sierchio wrote: > Sorry for the top post. Have you tried device polling? From > /usr/src/sys/amd64/conf/NOTES: ... > options DEVICE_POLLING *snip* I had not tried it and had actually forgotten about that capability. Anyway, I just tried it (kernel built with polling and enabled via ifconfig on the devices) and saw no observable change. I am not certain if the code paths can really make use of it with the switch to iflib. However, I was able to squeak a bit more from the NIC's by the equivalent of polling(4) by bumping the delay for interrupts with the key one being hw.em.rx_int_delay. It improved transfers going through the router, in both directions, by about 20Mb/s. The number of interrupts dropped a lot, so I doubt I am hitting an interrupt limit. /boot/loader.conf: hw.em.rx_int_delay="264" hw.em.rx_abs_int_delay="264" hw.em.tx_int_delay="264" hw.em.tx_abs_int_delay="264" One other thing to note, is that when I am testing going through the router, there are many drops and stalls/restarts when downloading but not uploading. Before test: dev.em.1.iflib.txq0.r_abdications: 0 dev.em.1.iflib.txq0.r_restarts: 0 dev.em.1.iflib.txq0.r_stalls: 0 dev.em.1.iflib.txq0.r_starts: 9239 dev.em.1.iflib.txq0.r_drops: 0 dev.em.1.iflib.txq0.r_enqueues: 9240 dev.em.1.iflib.txq0.txq_cleaned: 9640 dev.em.1.iflib.txq0.txq_processed: 9680 After test: dev.em.1.iflib.txq0.r_abdications: 14 dev.em.1.iflib.txq0.r_restarts: 7419 dev.em.1.iflib.txq0.r_stalls: 7419 dev.em.1.iflib.txq0.r_starts: 411677 dev.em.1.iflib.txq0.r_drops: 198 dev.em.1.iflib.txq0.r_enqueues: 878486 dev.em.1.iflib.txq0.txq_cleaned: 875005 dev.em.1.iflib.txq0.txq_processed: 875045 After 2nd test with Marvell as outgoing NIC starting from zero: dev.em.1.iflib.txq0.r_abdications: 1 dev.em.1.iflib.txq0.r_restarts: 7647 dev.em.1.iflib.txq0.r_stalls: 7647 dev.em.1.iflib.txq0.r_starts: 1092018 dev.em.1.iflib.txq0.r_drops: 811 dev.em.1.iflib.txq0.r_enqueues: 1570425 dev.em.1.iflib.txq0.txq_cleaned: 1567629 dev.em.1.iflib.txq0.txq_processed: 1567671 Unless someone knows anything else, I think I am going to have to buy a PCI-E NIC for the router to replace one of the Intel NIC's. I will either use it along with the Marvell or the other Intel NIC. When testing between my workstation and any one Intel NIC on the router using iperf3, I can achieve just over 900Mb/s. Sean -- scf@FreeBSD.org > On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley <scf@freebsd.org> wrote: > >> I have recently upped my Internet service and have now noticed a limit >> being reached, but I am not certain which limit and best option to >> resolve it. >> >> I am using a circa 2007 system as a multi-purpose router running FreeBSD >> 12-STABLE (r367740). The issue is that it maxes out around 400Mb/s when >> running a speed test through it between my workstation and various test >> sites (i.e., DSL Reports and Speedtest). There are two NIC's (both are >> Intel 82541PI) in use with one to the ISP and one to my workstation. >> >> At first, I saw one of them apparently hitting an interrupt rate of just >> over 8000, so I bumped their rate limits higher with little to no >> improvement. >> >> What makes me believe I can theoretically get faster speeds is that I >> can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's >> and nearly double the speed. The difference is that it is on the PCI-E >> bus and has MSI support. >> >> irq16: em0:irq0+ >> irq17: em1:irq0 >> irq20: hpet0 >> irq258: mskc0 >> >> I have many network settings, but changing them did nothing. Here are >> the settings I am trying now that seem to squeak a little extra >> performance. The commented-out lines are ones I tried without seeing >> any change. I have also tested without these settings. >> >> /boot/loader.conf >> hw.em.rx_process_limit="-1" >> # dev.em.0.iflib.override_nrxds="2048" >> # dev.em.1.iflib.override_nrxds="2048" >> # dev.em.2.iflib.override_nrxds="2048" >> # dev.em.0.iflib.override_ntxds="2048" >> # net.link.ifqmaxlen="2048" >> hw.em.max_interrupt_rate="32000" >> # net.isr.maxthreads="-1" >> # net.isr.bindthreads="1" >> >> /etc/sysctl.conf >> kern.random.harvest.mask=351 >> dev.em.0.fc=0 >> dev.em.1.fc=0 >> dev.em.0.itr=122 # Allow past 8000 interrupts/second. >> dev.em.1.itr=122 >> net.inet.ip.redirect=0 >> net.inet6.ip6.redirect=0 >> >> Increasing these from 66 to 250 did not help: >> hw.em.rx_abs_int_delay: 66 >> hw.em.tx_abs_int_delay: 66 >> hw.em.tx_int_delay: 66 >> >> I am utilizing pf, but I doubt it is the issue since using the same >> rules with the msk driver would have held the speed down to 400Mb/s. >> >> Am I hitting the limit of the PCI bus (memory or interrupt) or something >> else? I can buy a new PCI-E NIC for the internal network, but I rather >> fully utilize the Intel NIC's I have, if possible. >> >> Sean
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e4295db8-bf19-eb41-f6d7-f2c5d93657e7>