From nobody Fri Oct 25 12:13:24 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XZhXY6HrSz5Zfg0 for ; Fri, 25 Oct 2024 12:14:05 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XZhXX64XSz4F46 for ; Fri, 25 Oct 2024 12:14:04 +0000 (UTC) (envelope-from ccfreebsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of ccfreebsd@gmail.com designates 209.85.167.47 as permitted sender) smtp.mailfrom=ccfreebsd@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none) Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-539f1d96668so234506e87.3 for ; Fri, 25 Oct 2024 05:14:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729858442; x=1730463242; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Q8aABl+vUKHvu+wBy1vsVHyA7MOda6zok5EfiT0VPMg=; b=gw0pzmVVenM4dH3Kk12p9Vx92HM8GN9vNUD/Dx9qKCNt/Mf0qW5oX3kil2Ei4WVcz3 15tYjX0VrMkaYjri/P5NisNi+642SBxmygdSF1tK3mRXY8dpAy7RekV3TJzCFlp/5OJe u14XTaAZEAUx1/OZ/E5HfthZJvCEw/FXAn2MnYCrWKn6I13HxXwTuxb0/0M3t3ntunT/ T76HCCo2FxyHOTgXEKkjpNH93olllzZlqQwW5vJE1RjdBufH7wnqqpknDJrrYqk+YhvD Kd/qWzHFyQbuLPzmLZhjyLNvgsKOLMinyK1ZJLC+wIdqPuUuZ1GpmVL64nIm8kZyd/mC UGwQ== X-Gm-Message-State: AOJu0YzF/lrt6SiOEl8gk169G7elL2nWX1+6IknZZeBqttqZR+4WXYDE /omKyTgsWiEXYKGP3Zh8qOpJ45+CKWz78XkdjB7zQp3gtLQe6lW40Gtzsa7v X-Google-Smtp-Source: AGHT+IHC9YV7iNxuHkD0IgL+hk8ds+tQ+6GOHa0VU1qOYdbf/9uM3NaMmIuegt/pr7u1Ze6mdElsfA== X-Received: by 2002:a05:6512:6c3:b0:539:eaa9:738c with SMTP id 2adb3069b0e04-53b2f66c720mr241947e87.7.1729858441905; Fri, 25 Oct 2024 05:14:01 -0700 (PDT) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com. [209.85.208.173]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-53b2e12452fsm161056e87.73.2024.10.25.05.14.01 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 25 Oct 2024 05:14:01 -0700 (PDT) Received: by mail-lj1-f173.google.com with SMTP id 38308e7fff4ca-2f75d529b49so1227591fa.1 for ; Fri, 25 Oct 2024 05:14:01 -0700 (PDT) X-Received: by 2002:a05:651c:b26:b0:2fb:55e0:feca with SMTP id 38308e7fff4ca-2fcb6331828mr2171231fa.1.1729858441322; Fri, 25 Oct 2024 05:14:01 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Cheng Cui Date: Fri, 25 Oct 2024 08:13:24 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Performance test for CUBIC in stable/14 To: void Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="000000000000e8353806254c0df1" X-Spamd-Result: default: False [-2.89 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; FORGED_SENDER(0.30)[cc@freebsd.org,ccfreebsd@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_TO(0.00)[f-m.fm]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[cc@freebsd.org,ccfreebsd@gmail.com]; MISSING_XM_UA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.47:from]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.47:from,209.85.208.173:received] X-Rspamd-Queue-Id: 4XZhXX64XSz4F46 X-Spamd-Bar: -- --000000000000e8353806254c0df1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Here is my example. I am using two 6-core/12-threads desktops for my bhyve servers. CPU: AMD Ryzen 5 5560U with Radeon Graphics (2295.75-MHz K8-class CPU) You can find test results on VMs from my wiki: https://wiki.freebsd.org/chengcui/testD46046 All the CPU utilization results are low, especially for these throughput over 900 Mb/s. cc On Wed, Oct 23, 2024 at 5:43=E2=80=AFPM void wrote: > On Wed, Oct 23, 2024 at 03:14:08PM -0400, Cheng Cui wrote: > >I see. The result of `newreno` vs. `cubic` shows non-constant/infrequent > >packet > >retransmission. So TCP congestion control has little impact on improving > the > >performance. > > > >The performance bottleneck may come from somewhere else. For example, th= e > >sender CPU shows 97.7% utilization. Would there be any way to reduce CPU > >usage? > > There are 11 VMs running on the bhyve server. None of them are very busy > but the > server shows > % uptime > 9:54p.m. up 8 days, 6:08, 22 users, load averages: 0.82, 1.25, 1.74 > > The test vm vm4-fbsd14s: > % uptime > 9:55PM up 2 days, 3:12, 5 users, load averages: 0.35, 0.31, 0.21 > > It has > % sysctl hw.ncpu > hw.ncpu: 8 > > and > avail memory =3D 66843062272 (63746 MB) > > so it's not short of resources. > > A test just now gave these results: > - - - - - - - - - - - - - - - - - - - - - - - - - > Test Complete. Summary Results: > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-20.04 sec 1.31 GBytes 563 Mbits/sec 0 > sender > [ 5] 0.00-20.06 sec 1.31 GBytes 563 Mbits/sec > receiver > CPU Utilization: local/sender 94.1% (0.1%u/94.1%s), remote/receiver 15.5% > (1.5%u/13.9%s) > snd_tcp_congestion cubic > rcv_tcp_congestion cubic > > iperf Done. > > so I'm not sure how the utilization figure was synthesised, unless it's > derived > from something like 'top' where 1.00 is 100%. Load when running the test > got to > 0.83 as observed in 'top' in another terminal. Five mins after the test, > load in the vm is: 0.32, 0.31, 0.26 > on the bhyve host: 0.39, 0.61, 1.11 > > Before we began testing, I was looking at the speed issue as being caused > by > something to do with interrupts and/or polling, and/or HZ, somehting that > linux > handles differently and gives better results on the same bhyve host. > Maybe rebuilding the kernel with a different scheduler on both the host > and the > freebsd vms will give a better result for freebsd if tweaking sysctls > doesn't > make much of a difference. > > In terms of real-world bandwidth, I found that the combination of your > modified > cc_cubic + rack gave the best results in terms of overall throughput in a > speedtest context, although it's slower to get to its max throughput than > cubic > alone. I'm still testing with a webdav/rsync context (cubic against > cubic+rack) > > The next lot of testing after changing the scheduler will be on a KVM > host, > with various *BSDs as guests. > > There may be a tradeoff of stability against speed I guess. > -- > > --=20 Best Regards, Cheng Cui --000000000000e8353806254c0df1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Here is my example. I am using two 6-core/12-threads = desktops for my bhyve servers.
CPU: AMD Ryzen 5 5560U = with Radeon Graphics=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 (2295.75-MHz K8-class CPU)

<= /div>
You can find test results on VMs from my wiki:=C2=A0https://wiki.freebsd.org/chengc= ui/testD46046

All the CPU utilization results = are low, especially for these throughput over 900 Mb/s.

cc

On Wed, Oct 23, 2024 at 5:43=E2=80=AFPM void <void@f-m.fm> wrote:
On Wed, O= ct 23, 2024 at 03:14:08PM -0400, Cheng Cui wrote:
>I see. The result of `newreno` vs. `cubic` shows non-constant/infrequen= t
>packet
>retransmission. So TCP congestion control has little impact on improvin= g the
>performance.
>
>The performance bottleneck may come from somewhere else. For example, t= he
>sender CPU shows 97.7% utilization. Would there be any way to reduce CP= U
>usage?

There are 11 VMs running on the bhyve server. None of them are very busy bu= t the
server shows
% uptime
=C2=A0 9:54p.m.=C2=A0 up 8 days,=C2=A0 6:08, 22 users, load averages: 0.82,= 1.25, 1.74

The test vm vm4-fbsd14s:
% uptime
=C2=A0 9:55PM=C2=A0 up 2 days,=C2=A0 3:12, 5 users, load averages: 0.35, 0.= 31, 0.21

It has
% sysctl hw.ncpu
hw.ncpu: 8

and
avail memory =3D 66843062272 (63746 MB)

so it's not short of resources.

A test just now gave these results:
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Transfer=C2=A0 =C2= =A0 =C2=A0Bitrate=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Retr
[=C2=A0 5]=C2=A0 =C2=A00.00-20.04=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 =C2=A05= 63 Mbits/sec=C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= sender
[=C2=A0 5]=C2=A0 =C2=A00.00-20.06=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 =C2=A05= 63 Mbits/sec=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = receiver
CPU Utilization: local/sender 94.1% (0.1%u/94.1%s), remote/receiver 15.5% (1.5%u/13.9%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

so I'm not sure how the utilization figure was synthesised, unless it&#= 39;s derived
from something like 'top' where 1.00 is 100%. Load when running the= test got to
0.83 as observed in 'top' in another terminal. Five mins after the = test,
load in the vm is: 0.32, 0.31, 0.26
on the bhyve host: 0.39, 0.61, 1.11

Before we began testing, I was looking at the speed issue as being caused b= y
something to do with interrupts and/or polling, and/or HZ, somehting that l= inux
handles differently and gives better results on the same bhyve host.
Maybe rebuilding the kernel with a different scheduler on both the host and= the
freebsd vms will give a better result for freebsd if tweaking sysctls doesn= 't
make much of a difference.

In terms of real-world bandwidth, I found that the combination of your modi= fied
cc_cubic + rack gave the best results in terms of overall throughput in a speedtest context, although it's slower to get to its max throughput th= an cubic
alone. I'm still testing with a webdav/rsync context (cubic against cub= ic+rack)

The next lot of testing after changing the scheduler will be on a KVM host,=
with various *BSDs as guests.

There may be a tradeoff of stability against speed I guess.
--



--
Best Regards,
Cheng Cui
--000000000000e8353806254c0df1--