From owner-freebsd-net@freebsd.org Sat Nov 21 21:56:21 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 645E0471E84 for ; Sat, 21 Nov 2020 21:56:21 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CdnLD3tBLz3hZ3 for ; Sat, 21 Nov 2020 21:56:20 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: by mail-lj1-x22a.google.com with SMTP id 142so13845371ljj.10 for ; Sat, 21 Nov 2020 13:56:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenebras-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qQIwDCdsFYDhMy3nng8wEZaYM+RFdQsOvEUQzxWby+I=; b=mCTTfxrETC2U+78AryjwI3x5F2cmX6BtSrsYlGMyVr3P4DQbn2QcvE+27kfw3hkwef Oe1TA6TaSGdb6LUdeK4oVkunT+1zobkLDSXZOjFIuNOGN2z7+kgqWEgY4F6E8RdmLO8U VHCVa5AaNJkSUOv3fKi5MqhF3tfd46Glm6OGjudFv2qGa/ruWs0LbOq37vhkwt5eUHda B1utQXmEUVlbke7sB/GMLvOK8hjMSdc9uVJhB5Y6yVFNfV15rO0IESMKwb8WpDOhWiYv QH1Sq8dHojWpgHZ5kBp6M80tzLcgov9pkpLIcWeZ5TfwGvhICVQUIuDo9yw5yahSnkq+ 3HLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qQIwDCdsFYDhMy3nng8wEZaYM+RFdQsOvEUQzxWby+I=; b=sgQwRhHSNIvcpTkeDsESfUx+RMETxDoE8c9HioBfzluRSHJCJRXnl5h4nQREQUvoqn uKC+HjUjOCYAPSZMsOKnz4TvGpTiYVXoiwrfxW51hP1NFIajcEy8x6w0MpefVqtJ2nKa g3cLYQD7kJ8EShfSxRF7xJE8ntO/Bswdz5o1/TAledt2Oo0gaKlseMRMh/lCJNEn4/GP Zt1t4zYOn+uSmLNqbkBkWAlh8nntgRvFVB4HP4Gw7hGIp0H6UAv+SyouFyJx38H3T+Q0 jD+qPSaMayJqJ5EObyJ7NsZT8FvEv7BlGipzZzBG/jBzOa/Y/omqzzdSBjqiY5YCt6Rc 7O0w== X-Gm-Message-State: AOAM533GxTwXszdpfYcNIrvHoUGo6hfbpTBFbS+/nHPsAbYet/Mj/SWb SDvod5i8URvXYKKpoT4kQNjm/GbNDrKP0DFFal38hg== X-Google-Smtp-Source: ABdhPJyWLLg2SovfchSctLJEe+59gSRxOd0BYysVHdRhlf3a0BUHi7Hmh178yzM6QuooyyjEHe09E7KTeujr1cz6ps0= X-Received: by 2002:a2e:8891:: with SMTP id k17mr9888409lji.326.1605995778308; Sat, 21 Nov 2020 13:56:18 -0800 (PST) MIME-Version: 1.0 References: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> In-Reply-To: <9d7b39fb-7c1-fe7b-fa9a-ab1aa89cb96a@FreeBSD.org> From: Michael Sierchio Date: Sat, 21 Nov 2020 13:55:42 -0800 Message-ID: Subject: Re: Determining cause of transfer limit To: "Sean C. Farley" Cc: "freebsd-net@freebsd.org" X-Rspamd-Queue-Id: 4CdnLD3tBLz3hZ3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tenebras-com.20150623.gappssmtp.com header.s=20150623 header.b=mCTTfxrE; dmarc=none; spf=none (mx1.freebsd.org: domain of kudzu@tenebras.com has no SPF policy when checking 2a00:1450:4864:20::22a) smtp.mailfrom=kudzu@tenebras.com X-Spamd-Result: default: False [-3.30 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tenebras-com.20150623.gappssmtp.com:s=20150623]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[tenebras.com]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::22a:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[tenebras-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22a:from]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::22a:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 21:56:21 -0000 Sorry for the top post. Have you tried device polling? From /usr/src/sys/amd64/conf/NOTES: ##################################################################### # NETWORKING OPTIONS # # DEVICE_POLLING adds support for mixed interrupt-polling handling # of network device drivers, which has significant benefits in terms # of robustness to overloads and responsivity, as well as permitting # accurate scheduling of the CPU time between kernel network processing # and other activities. The drawback is a moderate (up to 1/HZ seconds) # potential increase in response times. # It is strongly recommended to use HZ=3D1000 or 2000 with DEVICE_POLLING # to achieve smoother behaviour. # Additionally, you can enable/disable polling at runtime with help of # the ifconfig(8) utility, and select the CPU fraction reserved to # userland with the sysctl variable kern.polling.user_frac # (default 50, range 0..100). # # Not all device drivers support this mode of operation at the time of # this writing. See polling(4) for more details. options DEVICE_POLLING On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley wrote: > I have recently upped my Internet service and have now noticed a limit > being reached, but I am not certain which limit and best option to > resolve it. > > I am using a circa 2007 system as a multi-purpose router running FreeBSD > 12-STABLE (r367740). The issue is that it maxes out around 400Mb/s when > running a speed test through it between my workstation and various test > sites (i.e., DSL Reports and Speedtest). There are two NIC's (both are > Intel 82541PI) in use with one to the ISP and one to my workstation. > > At first, I saw one of them apparently hitting an interrupt rate of just > over 8000, so I bumped their rate limits higher with little to no > improvement. > > What makes me believe I can theoretically get faster speeds is that I > can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's > and nearly double the speed. The difference is that it is on the PCI-E > bus and has MSI support. > > irq16: em0:irq0+ > irq17: em1:irq0 > irq20: hpet0 > irq258: mskc0 > > I have many network settings, but changing them did nothing. Here are > the settings I am trying now that seem to squeak a little extra > performance. The commented-out lines are ones I tried without seeing > any change. I have also tested without these settings. > > /boot/loader.conf > hw.em.rx_process_limit=3D"-1" > # dev.em.0.iflib.override_nrxds=3D"2048" > # dev.em.1.iflib.override_nrxds=3D"2048" > # dev.em.2.iflib.override_nrxds=3D"2048" > # dev.em.0.iflib.override_ntxds=3D"2048" > # net.link.ifqmaxlen=3D"2048" > hw.em.max_interrupt_rate=3D"32000" > # net.isr.maxthreads=3D"-1" > # net.isr.bindthreads=3D"1" > > /etc/sysctl.conf > kern.random.harvest.mask=3D351 > dev.em.0.fc=3D0 > dev.em.1.fc=3D0 > dev.em.0.itr=3D122 # Allow past 8000 interrupts/second. > dev.em.1.itr=3D122 > net.inet.ip.redirect=3D0 > net.inet6.ip6.redirect=3D0 > > Increasing these from 66 to 250 did not help: > hw.em.rx_abs_int_delay: 66 > hw.em.tx_abs_int_delay: 66 > hw.em.tx_int_delay: 66 > > I am utilizing pf, but I doubt it is the issue since using the same > rules with the msk driver would have held the speed down to 400Mb/s. > > Am I hitting the limit of the PCI bus (memory or interrupt) or something > else? I can buy a new PCI-E NIC for the internal network, but I rather > fully utilize the Intel NIC's I have, if possible. > > Sean > -- > scf@FreeBSD.org > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > --=20 "Well," Brahm=C4=81 said, "even after ten thousand explanations, a fool is = no wiser, but an intelligent person requires only two thousand five hundred." - The Mah=C4=81bh=C4=81rata