Date: Mon, 29 Jan 2024 12:54:49 -0500 From: Paul Procacci <pprocacci@gmail.com> To: Jim Long <freebsd-questions@umpquanet.com> Cc: freebsd-questions@freebsd.org Subject: Re: VirtIO/ipfw/natd throughput problem in hosted VM Message-ID: <CAFbbPui_RX%2Bk%2BtFd18yN2MHMfSAQSqqEjPLo3GY12AchnN0eCg@mail.gmail.com> In-Reply-To: <ZbfkhQXCobk0jKBg@sfo.umpquanet.com> References: <ZbfkhQXCobk0jKBg@sfo.umpquanet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000021cf506101957f3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Mon, Jan 29, 2024 at 12:47=E2=80=AFPM Jim Long <freebsd-questions@umpqua=
net.com>
wrote:
> I'm running FreeBSD 14.0-RELEASE in a quad-core, 12G VM commercially
> hosted under KVM (I'm told). It was installed from the main disc1.iso
> image, not any of the VM-centric ISOs.
>
> # grep -i network /var/run/dmesg.boot
> virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0xc000-0xc03f mem
> 0xfebd1000-0xfebd1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0
> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
> # ifconfig public
> public: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
> metric 0 mtu 1500
>
> options=3D4c079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,T=
SO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
> ether fa:16:3e:ca:b5:9c
> inet 10.1.170.27 netmask 0xffffff00 broadcast 10.1.170.255
> media: Ethernet autoselect (10Gbase-T <full-duplex>)
> status: active
> nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> (10.1.170.27 is my obfuscated routable public IP.)
>
> Using ipfw *without* any "divert" rule, I get good network speed.
> Transfering two larger files, one time apiece:
>
> # ipfw show
> 65000 2966704 2831806570 allow ip from any to any
> 65535 135 35585 deny ip from any to any
>
> # 128MB @ > 94MB/s:
> # rm -f random-data-test-128M
> # time rsync -Ppv example.com:random-data-test-128M .
> random-data-test-128M
> 134,217,728 100% 94.26MB/s 0:00:01 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 134,250,588 bytes 53,700,252.40 bytes/sec
> total size is 134,217,728 speedup is 1.00
>
> real 0m1.645s
> user 0m0.826s
> sys 0m0.788s
>
> # 1024MB @ > 105MB/s:
> # rm -f random-data-test-1G
> # time rsync -Ppv example.com:random-data-test-1G .
> random-data-test-1G
> 1,073,741,824 100% 105.98MB/s 0:00:09 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 1,074,004,060 bytes 102,286,105.05 bytes/sec
> total size is 1,073,741,824 speedup is 1.00
>
> real 0m9.943s
> user 0m4.701s
> sys 0m5.769s
>
>
>
> But with an "ipfw divert" rule in place (and natd running as 'natd -n
> public'), across 5 transfers of a 2M file of /dev/random, I get very
> poor transfer speeds:
>
> # ipfw add 65000 divert natd all from any to any via public
> # ipfw show
> 60000 3 292 divert 8668 ip from any to any via public
> 65000 2950208 2817524670 allow ip from any to any
> 65535 135 35585 deny ip from any to any
>
> Test 1 of 5, < 180kB/s:
>
> # rm -f random-data-test-2M
> # time rsync -Ppv example.com:random-data-test-2M .
> random-data-test-2M
> 2,097,152 100% 179.08kB/s 0:00:11 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 2,097,752 bytes 167,823.60 bytes/sec
> total size is 2,097,152 speedup is 1.00
>
> real 0m12.199s
> user 0m0.085s
> sys 0m0.027s
>
> Test 2 of 5, < 115kB/s:
>
> # rm -f random-data-test-2M
> # rsync -Ppv example.com:random-data-test-2M .
> random-data-test-2M
> 2,097,152 100% 114.40kB/s 0:00:17 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 2,097,752 bytes 107,579.23 bytes/sec
> total size is 2,097,152 speedup is 1.00
>
> real 0m19.300s
> user 0m0.072s
> sys 0m0.051s
>
> Test 3 of 5, < 37kB/s (almost 57s elapsed time):
>
> # rm -f random-data-test-2M
> # time rsync -Ppv example.com:random-data-test-2M .
> random-data-test-2M
> 2,097,152 100% 36.49kB/s 0:00:56 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 2,097,752 bytes 36,483.39 bytes/sec
> total size is 2,097,152 speedup is 1.00
>
> real 0m56.868s
> user 0m0.080s
> sys 0m0.023s
>
> Test 4 of 5, < 112kB/s:
>
> # rm -f random-data-test-2M
> # time rsync -Ppv example.com:random-data-test-2M .
> random-data-test-2M
> 2,097,152 100% 111.89kB/s 0:00:18 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 2,097,752 bytes 102,331.46 bytes/sec
> total size is 2,097,152 speedup is 1.00
>
> real 0m19.544s
> user 0m0.095s
> sys 0m0.015s
>
> Test 5 of 5, 130kB/s:
>
> # rm -f random-data-test-2M
> # time rsync -Ppv example.com:random-data-test-2M .
> random-data-test-2M
> 2,097,152 100% 130.21kB/s 0:00:15 (xfr#1, to-chk=3D0/1)
>
> sent 43 bytes received 2,097,752 bytes 127,139.09 bytes/sec
> total size is 2,097,152 speedup is 1.00
>
> real 0m16.583s
> user 0m0.072s
> sys 0m0.035s
>
>
> How can I tweak my network stack to get reasonable throughput from natd?
> I'm happy to respond to requests for additional details.
>
>
> Thank you!
>
>
>
>
The most glaringly obvious thing to me is to use in-kernel nat instead of
natd.
Packets won't have to leave the kernel at that point.
It's detailed in ipfw(8).
~Paul
--=20
__________________
:(){ :|:& };:
--000000000000021cf506101957f3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div dir=3D"ltr"><br></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Jan 29, 2024 at 12:=
47=E2=80=AFPM Jim Long <<a href=3D"mailto:freebsd-questions@umpquanet.co=
m">freebsd-questions@umpquanet.com</a>> wrote:<br></div><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r=
gb(204,204,204);padding-left:1ex">I'm running FreeBSD 14.0-RELEASE in a=
quad-core, 12G VM commercially<br>
hosted under KVM (I'm told).=C2=A0 It was installed from the main disc1=
.iso<br>
image, not any of the VM-centric ISOs.<br>
<br>
# grep -i network /var/run/dmesg.boot<br>
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0xc000-0xc03f=
mem 0xfebd1000-0xfebd1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pc=
i0<br>
vtnet0: <VirtIO Networking Adapter> on virtio_pci0<br>
# ifconfig public<br>
public: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&=
gt; metric 0 mtu 1500<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 options=3D4c079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN=
_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6><b=
r>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ether fa:16:3e:ca:b5:9c<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 inet 10.1.170.27 netmask 0xffffff00 broadcast 1=
0.1.170.255<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 media: Ethernet autoselect (10Gbase-T <full-=
duplex>)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 status: active<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_=
LINKLOCAL><br>
<br>
(10.1.170.27 is my obfuscated routable public IP.)<br>
<br>
Using ipfw *without* any "divert" rule, I get good network speed.=
<br>
Transfering two larger files, one time apiece:<br>
<br>
# ipfw show<br>
65000 2966704 2831806570 allow ip from any to any<br>
65535=C2=A0 =C2=A0 =C2=A0135=C2=A0 =C2=A0 =C2=A0 35585 deny ip from any to =
any<br>
<br>
# 128MB @ > 94MB/s:<br>
# rm -f random-data-test-128M<br>
# time rsync -Ppv example.com:random-data-test-128M .<br>
random-data-test-128M<br>
=C2=A0 =C2=A0 134,217,728 100%=C2=A0 =C2=A094.26MB/s=C2=A0 =C2=A0 0:00:01 (=
xfr#1, to-chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 134,250,588 bytes=C2=A0 53,700,252.40 bytes/se=
c<br>
total size is 134,217,728=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m1.645s<br>
user=C2=A0 =C2=A0 0m0.826s<br>
sys=C2=A0 =C2=A0 =C2=A00m0.788s<br>
<br>
# 1024MB @ > 105MB/s:<br>
# rm -f random-data-test-1G<br>
# time rsync -Ppv example.com:random-data-test-1G .<br>
random-data-test-1G<br>
=C2=A0 1,073,741,824 100%=C2=A0 105.98MB/s=C2=A0 =C2=A0 0:00:09 (xfr#1, to-=
chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 1,074,004,060 bytes=C2=A0 102,286,105.05 bytes=
/sec<br>
total size is 1,073,741,824=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m9.943s<br>
user=C2=A0 =C2=A0 0m4.701s<br>
sys=C2=A0 =C2=A0 =C2=A00m5.769s<br>
<br>
<br>
<br>
But with an "ipfw divert" rule in place (and natd running as '=
;natd -n<br>
public'), across 5 transfers of a 2M file of /dev/random, I get very<br=
>
poor transfer speeds:<br>
<br>
# ipfw add 65000 divert natd all from any to any via public<br>
# ipfw show<br>
60000=C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 =C2=A0 292 divert 866=
8 ip from any to any via public<br>
65000 2950208 2817524670 allow ip from any to any<br>
65535=C2=A0 =C2=A0 =C2=A0135=C2=A0 =C2=A0 =C2=A0 35585 deny ip from any to =
any<br>
<br>
Test 1 of 5, < 180kB/s:<br>
<br>
# rm -f random-data-test-2M<br>
# time rsync -Ppv example.com:random-data-test-2M .<br>
random-data-test-2M<br>
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 179.08kB/s=C2=A0 =C2=A0 0:00:11 (=
xfr#1, to-chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 167,823.60 bytes/sec<br>
total size is 2,097,152=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m12.199s<br>
user=C2=A0 =C2=A0 0m0.085s<br>
sys=C2=A0 =C2=A0 =C2=A00m0.027s<br>
<br>
Test 2 of 5, < 115kB/s:<br>
<br>
# rm -f random-data-test-2M<br>
# rsync -Ppv example.com:random-data-test-2M .<br>
random-data-test-2M<br>
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 114.40kB/s=C2=A0 =C2=A0 0:00:17 (=
xfr#1, to-chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 107,579.23 bytes/sec<br>
total size is 2,097,152=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m19.300s<br>
user=C2=A0 =C2=A0 0m0.072s<br>
sys=C2=A0 =C2=A0 =C2=A00m0.051s<br>
<br>
Test 3 of 5, < 37kB/s (almost 57s elapsed time):<br>
<br>
# rm -f random-data-test-2M<br>
# time rsync -Ppv example.com:random-data-test-2M .<br>
random-data-test-2M<br>
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 =C2=A036.49kB/s=C2=A0 =C2=A0 0:00=
:56 (xfr#1, to-chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 36,483.39 bytes/sec<br>
total size is 2,097,152=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m56.868s<br>
user=C2=A0 =C2=A0 0m0.080s<br>
sys=C2=A0 =C2=A0 =C2=A00m0.023s<br>
<br>
Test 4 of 5, < 112kB/s:<br>
<br>
# rm -f random-data-test-2M<br>
# time rsync -Ppv example.com:random-data-test-2M .<br>
random-data-test-2M<br>
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 111.89kB/s=C2=A0 =C2=A0 0:00:18 (=
xfr#1, to-chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 102,331.46 bytes/sec<br>
total size is 2,097,152=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m19.544s<br>
user=C2=A0 =C2=A0 0m0.095s<br>
sys=C2=A0 =C2=A0 =C2=A00m0.015s<br>
<br>
Test 5 of 5, 130kB/s:<br>
<br>
# rm -f random-data-test-2M<br>
# time rsync -Ppv example.com:random-data-test-2M .<br>
random-data-test-2M<br>
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 130.21kB/s=C2=A0 =C2=A0 0:00:15 (=
xfr#1, to-chk=3D0/1)<br>
<br>
sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 127,139.09 bytes/sec<br>
total size is 2,097,152=C2=A0 speedup is 1.00<br>
<br>
real=C2=A0 =C2=A0 0m16.583s<br>
user=C2=A0 =C2=A0 0m0.072s<br>
sys=C2=A0 =C2=A0 =C2=A00m0.035s<br>
<br>
<br>
How can I tweak my network stack to get reasonable throughput from natd?<br=
>
I'm happy to respond to requests for additional details.<br>
<br>
<br>
Thank you!<br>
<br>
<br>
<br>
</blockquote></div><br clear=3D"all"></div>The most glaringly obvious thing=
to me is to use in-kernel nat instead of natd.</div><div>Packets won't=
have to leave the kernel at that point.</div><div>It's detailed in ipf=
w(8).</div><div><br></div><div>~Paul</div><div><br></div><div><div><span cl=
ass=3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmai=
l_signature">__________________<br><br>:(){ :|:& };:</div></div></div><=
/div>
--000000000000021cf506101957f3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFbbPui_RX%2Bk%2BtFd18yN2MHMfSAQSqqEjPLo3GY12AchnN0eCg>