From nobody Mon Jan 29 17:54:49 2024 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TNwtC3yHNz596qL for ; Mon, 29 Jan 2024 17:54:43 +0000 (UTC) (envelope-from pprocacci@gmail.com) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TNwtC1lldz51M4 for ; Mon, 29 Jan 2024 17:54:43 +0000 (UTC) (envelope-from pprocacci@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a35385da5bbso328232266b.3 for ; Mon, 29 Jan 2024 09:54:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706550880; x=1707155680; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=s7xsdtczBVanOprgyaePLOyI0NdFCknlxiWerOKShlE=; b=eS2hzJdZcUkMVw/GBwCgt5Sg1G1ZNmTlmmvm6TZBpBk0q7c94srTToHlAzaU2fodko 4fvit8UVNTGxB1Layqo0Oi9UDIWCvh79bIbotV/swtp7GuPipjmrDZtL2lh63Qr/4eAi Ym9u0fxz8hnymhj8MaTr8cDBqKWiwD2cH6UtU+SqrU39hRDRHxW9LWLdcG49oflB9m6G Jp4bmdebA4GpF9LxcN0Mdcm4gYkPnW3qElcAxfxJ61nj/9Eq+tCe48NyXELpoU7WQ5/+ 4JrwAUWJ7rOw2FWSlSLbAwAJinTrDVbJkF0Mc39ArM9MTpPwT6KE8CcjMT42EUTnQps4 AI7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706550880; x=1707155680; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=s7xsdtczBVanOprgyaePLOyI0NdFCknlxiWerOKShlE=; b=DEx2LIjvR+6XE3MZ3IZxjVIu6Mv0nicx8AhVuRdXAall+SPCKwRGIGwuXX0I4tjj/e /ZCQlIH5jPwzmWOtEEDTTeWVxeJGe4ToROhCOfVd01aeI5cmwKJxiz6RO0XntQtEWZOo N4CNLIOk4U7RyqkMG92n1jAFvcf7uKjVPRpJ6MM48mvEO/xt3bwrcJBrhYucOdpvobfP wVg/sOBrLXw1y/dR9LIjECFZbSrzLoqAnTIUUCrB1FFBZMakyWGom3Wtf/9uMXbzox58 Lyhr77/6F+Vxajn+54nzO8y/Lzh/9Z1fM/Eq8UowkxW5LXLEkGynEhAkXQjdYEUxqErC 4scA== X-Gm-Message-State: AOJu0YyIpLOEmMcYScfydEJNYyO4DoQmg7Blq00rgoPW3g6etfQ+86cn YvCQ9ICJUilbTNOdlEjU8nsrFakpeHJhRoFtn+jV58jNcFVtDTkNxsYvVYXu/Jp8AcOVrOxegsd zCTu7chNTxH55IS7mCn14iVdhb8mmqO4= X-Google-Smtp-Source: AGHT+IHerCF5IGV7blof7k2iJV5ud6wEyuwk5nkRTziw3B1i4VqbteE09YXFB80KDA8KvIiJsCve8e7m3F31C1464wM= X-Received: by 2002:a17:906:140a:b0:a30:474a:916e with SMTP id p10-20020a170906140a00b00a30474a916emr5019321ejc.7.1706550880245; Mon, 29 Jan 2024 09:54:40 -0800 (PST) List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Paul Procacci Date: Mon, 29 Jan 2024 12:54:49 -0500 Message-ID: Subject: Re: VirtIO/ipfw/natd throughput problem in hosted VM To: Jim Long Cc: freebsd-questions@freebsd.org Content-Type: multipart/alternative; boundary="000000000000021cf506101957f3" X-Rspamd-Queue-Id: 4TNwtC1lldz51M4 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --000000000000021cf506101957f3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jan 29, 2024 at 12:47=E2=80=AFPM Jim Long wrote: > I'm running FreeBSD 14.0-RELEASE in a quad-core, 12G VM commercially > hosted under KVM (I'm told). It was installed from the main disc1.iso > image, not any of the VM-centric ISOs. > > # grep -i network /var/run/dmesg.boot > virtio_pci0: port 0xc000-0xc03f mem > 0xfebd1000-0xfebd1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0 > vtnet0: on virtio_pci0 > # ifconfig public > public: flags=3D1008843 > metric 0 mtu 1500 > > options=3D4c079b > ether fa:16:3e:ca:b5:9c > inet 10.1.170.27 netmask 0xffffff00 broadcast 10.1.170.255 > media: Ethernet autoselect (10Gbase-T ) > status: active > nd6 options=3D29 > > (10.1.170.27 is my obfuscated routable public IP.) > > Using ipfw *without* any "divert" rule, I get good network speed. > Transfering two larger files, one time apiece: > > # ipfw show > 65000 2966704 2831806570 allow ip from any to any > 65535 135 35585 deny ip from any to any > > # 128MB @ > 94MB/s: > # rm -f random-data-test-128M > # time rsync -Ppv example.com:random-data-test-128M . > random-data-test-128M > 134,217,728 100% 94.26MB/s 0:00:01 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 134,250,588 bytes 53,700,252.40 bytes/sec > total size is 134,217,728 speedup is 1.00 > > real 0m1.645s > user 0m0.826s > sys 0m0.788s > > # 1024MB @ > 105MB/s: > # rm -f random-data-test-1G > # time rsync -Ppv example.com:random-data-test-1G . > random-data-test-1G > 1,073,741,824 100% 105.98MB/s 0:00:09 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 1,074,004,060 bytes 102,286,105.05 bytes/sec > total size is 1,073,741,824 speedup is 1.00 > > real 0m9.943s > user 0m4.701s > sys 0m5.769s > > > > But with an "ipfw divert" rule in place (and natd running as 'natd -n > public'), across 5 transfers of a 2M file of /dev/random, I get very > poor transfer speeds: > > # ipfw add 65000 divert natd all from any to any via public > # ipfw show > 60000 3 292 divert 8668 ip from any to any via public > 65000 2950208 2817524670 allow ip from any to any > 65535 135 35585 deny ip from any to any > > Test 1 of 5, < 180kB/s: > > # rm -f random-data-test-2M > # time rsync -Ppv example.com:random-data-test-2M . > random-data-test-2M > 2,097,152 100% 179.08kB/s 0:00:11 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 2,097,752 bytes 167,823.60 bytes/sec > total size is 2,097,152 speedup is 1.00 > > real 0m12.199s > user 0m0.085s > sys 0m0.027s > > Test 2 of 5, < 115kB/s: > > # rm -f random-data-test-2M > # rsync -Ppv example.com:random-data-test-2M . > random-data-test-2M > 2,097,152 100% 114.40kB/s 0:00:17 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 2,097,752 bytes 107,579.23 bytes/sec > total size is 2,097,152 speedup is 1.00 > > real 0m19.300s > user 0m0.072s > sys 0m0.051s > > Test 3 of 5, < 37kB/s (almost 57s elapsed time): > > # rm -f random-data-test-2M > # time rsync -Ppv example.com:random-data-test-2M . > random-data-test-2M > 2,097,152 100% 36.49kB/s 0:00:56 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 2,097,752 bytes 36,483.39 bytes/sec > total size is 2,097,152 speedup is 1.00 > > real 0m56.868s > user 0m0.080s > sys 0m0.023s > > Test 4 of 5, < 112kB/s: > > # rm -f random-data-test-2M > # time rsync -Ppv example.com:random-data-test-2M . > random-data-test-2M > 2,097,152 100% 111.89kB/s 0:00:18 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 2,097,752 bytes 102,331.46 bytes/sec > total size is 2,097,152 speedup is 1.00 > > real 0m19.544s > user 0m0.095s > sys 0m0.015s > > Test 5 of 5, 130kB/s: > > # rm -f random-data-test-2M > # time rsync -Ppv example.com:random-data-test-2M . > random-data-test-2M > 2,097,152 100% 130.21kB/s 0:00:15 (xfr#1, to-chk=3D0/1) > > sent 43 bytes received 2,097,752 bytes 127,139.09 bytes/sec > total size is 2,097,152 speedup is 1.00 > > real 0m16.583s > user 0m0.072s > sys 0m0.035s > > > How can I tweak my network stack to get reasonable throughput from natd? > I'm happy to respond to requests for additional details. > > > Thank you! > > > > The most glaringly obvious thing to me is to use in-kernel nat instead of natd. Packets won't have to leave the kernel at that point. It's detailed in ipfw(8). ~Paul --=20 __________________ :(){ :|:& };: --000000000000021cf506101957f3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Mon, Jan 29, 2024 at 12:= 47=E2=80=AFPM Jim Long <freebsd-questions@umpquanet.com> wrote:
I'm running FreeBSD 14.0-RELEASE in a= quad-core, 12G VM commercially
hosted under KVM (I'm told).=C2=A0 It was installed from the main disc1= .iso
image, not any of the VM-centric ISOs.

# grep -i network /var/run/dmesg.boot
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0xc000-0xc03f= mem 0xfebd1000-0xfebd1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pc= i0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
# ifconfig public
public: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&= gt; metric 0 mtu 1500
=C2=A0 =C2=A0 =C2=A0 =C2=A0 options=3D4c079b<RXCSUM,TXCSUM,VLAN_MTU,VLAN= _HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6> =C2=A0 =C2=A0 =C2=A0 =C2=A0 ether fa:16:3e:ca:b5:9c
=C2=A0 =C2=A0 =C2=A0 =C2=A0 inet 10.1.170.27 netmask 0xffffff00 broadcast 1= 0.1.170.255
=C2=A0 =C2=A0 =C2=A0 =C2=A0 media: Ethernet autoselect (10Gbase-T <full-= duplex>)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 status: active
=C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_= LINKLOCAL>

(10.1.170.27 is my obfuscated routable public IP.)

Using ipfw *without* any "divert" rule, I get good network speed.=
Transfering two larger files, one time apiece:

# ipfw show
65000 2966704 2831806570 allow ip from any to any
65535=C2=A0 =C2=A0 =C2=A0135=C2=A0 =C2=A0 =C2=A0 35585 deny ip from any to = any

# 128MB @ > 94MB/s:
# rm -f random-data-test-128M
# time rsync -Ppv example.com:random-data-test-128M .
random-data-test-128M
=C2=A0 =C2=A0 134,217,728 100%=C2=A0 =C2=A094.26MB/s=C2=A0 =C2=A0 0:00:01 (= xfr#1, to-chk=3D0/1)

sent 43 bytes=C2=A0 received 134,250,588 bytes=C2=A0 53,700,252.40 bytes/se= c
total size is 134,217,728=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m1.645s
user=C2=A0 =C2=A0 0m0.826s
sys=C2=A0 =C2=A0 =C2=A00m0.788s

# 1024MB @ > 105MB/s:
# rm -f random-data-test-1G
# time rsync -Ppv example.com:random-data-test-1G .
random-data-test-1G
=C2=A0 1,073,741,824 100%=C2=A0 105.98MB/s=C2=A0 =C2=A0 0:00:09 (xfr#1, to-= chk=3D0/1)

sent 43 bytes=C2=A0 received 1,074,004,060 bytes=C2=A0 102,286,105.05 bytes= /sec
total size is 1,073,741,824=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m9.943s
user=C2=A0 =C2=A0 0m4.701s
sys=C2=A0 =C2=A0 =C2=A00m5.769s



But with an "ipfw divert" rule in place (and natd running as '= ;natd -n
public'), across 5 transfers of a 2M file of /dev/random, I get very poor transfer speeds:

# ipfw add 65000 divert natd all from any to any via public
# ipfw show
60000=C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 =C2=A0 292 divert 866= 8 ip from any to any via public
65000 2950208 2817524670 allow ip from any to any
65535=C2=A0 =C2=A0 =C2=A0135=C2=A0 =C2=A0 =C2=A0 35585 deny ip from any to = any

Test 1 of 5, < 180kB/s:

# rm -f random-data-test-2M
# time rsync -Ppv example.com:random-data-test-2M .
random-data-test-2M
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 179.08kB/s=C2=A0 =C2=A0 0:00:11 (= xfr#1, to-chk=3D0/1)

sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 167,823.60 bytes/sec
total size is 2,097,152=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m12.199s
user=C2=A0 =C2=A0 0m0.085s
sys=C2=A0 =C2=A0 =C2=A00m0.027s

Test 2 of 5, < 115kB/s:

# rm -f random-data-test-2M
# rsync -Ppv example.com:random-data-test-2M .
random-data-test-2M
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 114.40kB/s=C2=A0 =C2=A0 0:00:17 (= xfr#1, to-chk=3D0/1)

sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 107,579.23 bytes/sec
total size is 2,097,152=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m19.300s
user=C2=A0 =C2=A0 0m0.072s
sys=C2=A0 =C2=A0 =C2=A00m0.051s

Test 3 of 5, < 37kB/s (almost 57s elapsed time):

# rm -f random-data-test-2M
# time rsync -Ppv example.com:random-data-test-2M .
random-data-test-2M
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 =C2=A036.49kB/s=C2=A0 =C2=A0 0:00= :56 (xfr#1, to-chk=3D0/1)

sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 36,483.39 bytes/sec
total size is 2,097,152=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m56.868s
user=C2=A0 =C2=A0 0m0.080s
sys=C2=A0 =C2=A0 =C2=A00m0.023s

Test 4 of 5, < 112kB/s:

# rm -f random-data-test-2M
# time rsync -Ppv example.com:random-data-test-2M .
random-data-test-2M
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 111.89kB/s=C2=A0 =C2=A0 0:00:18 (= xfr#1, to-chk=3D0/1)

sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 102,331.46 bytes/sec
total size is 2,097,152=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m19.544s
user=C2=A0 =C2=A0 0m0.095s
sys=C2=A0 =C2=A0 =C2=A00m0.015s

Test 5 of 5, 130kB/s:

# rm -f random-data-test-2M
# time rsync -Ppv example.com:random-data-test-2M .
random-data-test-2M
=C2=A0 =C2=A0 =C2=A0 2,097,152 100%=C2=A0 130.21kB/s=C2=A0 =C2=A0 0:00:15 (= xfr#1, to-chk=3D0/1)

sent 43 bytes=C2=A0 received 2,097,752 bytes=C2=A0 127,139.09 bytes/sec
total size is 2,097,152=C2=A0 speedup is 1.00

real=C2=A0 =C2=A0 0m16.583s
user=C2=A0 =C2=A0 0m0.072s
sys=C2=A0 =C2=A0 =C2=A00m0.035s


How can I tweak my network stack to get reasonable throughput from natd? I'm happy to respond to requests for additional details.


Thank you!




The most glaringly obvious thing= to me is to use in-kernel nat instead of natd.
Packets won't= have to leave the kernel at that point.
It's detailed in ipf= w(8).

~Paul

--
__________________

:(){ :|:& };:
<= /div> --000000000000021cf506101957f3--