From nobody Mon Aug 28 15:43:03 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZFFl6gGsz4rgxf for ; Mon, 28 Aug 2023 15:43:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-54.consmr.mail.gq1.yahoo.com (sonic315-54.consmr.mail.gq1.yahoo.com [98.137.65.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZFFk4mQzz4fd0 for ; Mon, 28 Aug 2023 15:43:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Sgt+2E6+; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693237400; bh=pjs9FuzOOW2K3pdqNIJ4qcYFn+CsjGZZNcX+z2WJ2+U=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=Sgt+2E6+KMRVSdAnx4zJl9KRJUultmFrLJ1tQWxia8ikv3LXwUctKPJsJrHCFrF+Eg/HUr9yuL+yQxWSI93xIhyo+6WHRxJtUUV5ECtQb3fDZEH2jW+ARG/+bWP+QQDKOY33M8xvYlY971/W7xPkGsnbZTcHFwIQOPnkboENpn3k2ScnbzrSfeWzN6PnRkQ4RN+1gE6HWey3LQ+w0OA+XvB+m4JelXXGuwGijc1KlBWJbLAJ3das1XU6uv9LlN7te/ik8wIaJD+xgCydVFK3wjkN6kU80tELX4868KXWdMrobEZFRc9ISXM2r58ST/El8/9cT9jQjqZTqbWJHd9K/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693237400; bh=Hm10xH0HnaZw2zzbA5Y3WsxWDLz0O591OPpcUo2Mv9l=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=qkAh5R+62jIld3YbLIFUPWAift1qTiukdwHwo6RFsqBzsltKXNUx44HSonDNWY9ZDoaQqWYArGnfAqc4xRjclEG6l9BKtJNwEa0G3As1+s+r6QrGirhIhjhG65SBR1zmrY/Ji19Tz8YYqlt0dAX2yk0UKAZjLgwdYfx+1JWn4KnJHwOqEvEtZdPnfrjefRq/+mzT1PZ3rg0A8vKdldTqamj1W5j9J6CKBwgS8SGN8tJK4Eo6JG+JKhVdnb9/Zqml6WxL1eVACYMlZANpaBz09Yk9pyxhbPpp2V+8w8rg8ruvGRrkkvYNqpvTs53GtvDoxGPTgW5eqp1SqYwOTqyUJg== X-YMail-OSG: ruXq62wVM1lBUQ6aiK5jB31YPJA_0q.ZAQkbfa7nE9UJYdHKLBqF.vAlS63bVgM ThrBUuxzY5kVDaenb9SgCsotHp14lwfGbgwOVOgytjh6PRYoAoeMj7jJoAV9o40otJmpuO4dhvui 1xVIWQitS6S6SFlxJFSNGyUsYg6K0.40afgghDgf0sGyTLOTeWckSaINztfU4l58toB8HOiWG3q3 T7PKpjJljFTHmf0uaOZkxUGrWPTT8pD15deDz2kXdYHbD87i5ApMwaqFQmvzOTkLBRXiBqZg20jh A8UOxGEfYso1tzyXPT9OhTvytSnb6PWz2Fe99HiR_hKNqknwMlahCrgI80zqaQBnlLO5t3KE7BZ4 _IoK5d9PJUuB8aJOX6zv.Mfen.BXU3zRyv468aolbhb_1grXzfv0BSSPpcKBFznrQdKWEkSWCl57 FbzNUhzFTKkIC4Qnz9w5e4KD_ziZSWecggGVzn1fs3DE0d_DpCc_Ov2U44QPOR8jLm1jX_tKIF0A h3B9VC._B0Iac2ZNVYFC_JYUYejawrviX8XAsNY_0PvyCenJyYpot9NnqQ4vlZB7cymAeXc0vtHK 9htzALZdtz_wFCKRmKNihWM.ZCGuMxoYnfVJlTNwRr20d5oTLgP9mZYwtVzS84nLCOvZ5CXPa3vS U4dA8sBE_zEoJj5wMV3Pyt2u6du0p3paf.QXRf.dGrht7K8RkUG3J1fBIUDeNE.XCMKTcmIWoRWj 2MfzZmQfNPWAx037zn7YCL3AC85Kq6NFWqgZO_qQ83f_by5vioOo.gsHY_QQby6V1Nk4VtPLMlw6 PUVgRnFQCjIrNugAZOldSsa4B1uMOss7gLQDyTUKT1nKyuwQagcqR02aNM7K5H4IOa0KGpF5GQS7 J8AtEB39Tjt90RoZ0MCZPksdozoB1skm2wcQix5aUmOLVNOD.ffRd2sm8mzMo9ARzYqizTHPHuNd PPMNsK4Uhn7b0mpuvCZFzjOT8sJwi5JaZR0eqkfvKR_YppPI89_N3XQroY00NlKzXbGVUCLMtOkz XYCcOoXCMCkdJFUpKc8DbX2M1MvHR4mPKlBlq9bac_zQbAjVhD8qZj0VrxVxpSu25xOez7.vKKH8 93aEbP2R77O7uNBXJfl8DZ_OQqdb6gYgsrzXFr7os8.c4TMFgAnj.QrH5BtsBqhqf7eXgEcG7GGW 4IYZ5sWYDxh3zrdy4A__FPdhes1gzuXTwVq3MZVAQMetDt3trTcFhNs9rI95rK.CF6RyLA0ukB7t kzERrpvi8LjMr3zx8_xoSzk3Yp.Fp2_SNldpX24Sm1OpAQaxybuAkdgw8ZB2Bzq7w1h13_1HgmXH 1ofST7e0SzviXPl5KohojEvHyUm8yfVYHyZqKbaLgk1Ewf.vDScAYi_0vtN3sMrvMvwZqG1rDUP4 P3cuG87RwpixDLOhbsfLmy23USn3u.U.r3CA3kEHzEvJfBNjOkOkeG2_J70k9eHYj9cvYrVOkRG6 exm3qvoWuK5bTn.P21Yfos6PXlkQyCCUXVnxmvEK0MqZTvu8GcbOaGsEHvUKWQu.KVCrgIjU3gpx 0fuDI537hbATRS23e1ghZY3gXPFuCZbTs7laUzi6xzt32h7DU2D_Nt245VwyMNmIk5bBbJ9P_a4C bw3ym4eAFgmXH_kOiG5y4C7fD6ZGiNJ42XYCbptjYvB6sipoidAI2GsFKlT67mZS_2uJNC9Xq7HO nEiooygvncgRoU6IxcYJ8Fi.6eIEd9Mo30TZN82XtvPDQy6XNc3dKylmfPoa9Wj4q7wbHKjqWABb AVtnVv5D97WQ_UVfVs2fM90_mMvJTTtr4ioUzPYaMj86q66ekzeBWuHZ2mztkmH2P75Y82mQ.Frh XtXbvJghiYnRtSvdD9GEC.eIzbJIKE7nGhWPEn.WQFvQ3FxM9nLWLEphxC.6YFb7eHEFwwxYbQE2 huj76vQOp2i3k0y2dlvWbQly.YkP6bxkJUoYRFnolt7U3_Om9j7EWDP6gps3hv5ke_BE29ush9AI r8YNnlRh7l4uDKvsPDSbQ5FngvOvCHr8q.JVZucWN5dUoYWalP2t05yA4FLye_Xg5txfajLie3Mh NlqEFtA2epS6qzxgaoMEPXUbS1Ed_3MgdQampwPZr48Um5Cu9I1SpFRzn4Mo188WdhhwI4tk_LOw uFOBWpcHhHq8TUKAQO03blmVXPQqxOQ5yw4EyG_6nzXiLIdaBALkyncn6rVkG7N74Vrdhqx0vRa3 ZXcyXPlX6.LBUy7dOP2AgrSGMm5BAHu9qenn1drYgfDCM.74MfHhj2haqjYo3nCQoy.q_aq2p2Y0 Prw-- X-Sonic-MF: X-Sonic-ID: 0e65ab72-84eb-43d8-99b1-f5e55360bcac Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Aug 2023 15:43:20 +0000 Received: by hermes--production-ne1-7b767b77cc-msspp (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID df76bb844275db3a0b32046750ce2ae3; Mon, 28 Aug 2023 15:43:14 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Message-Id: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> Date: Mon, 28 Aug 2023 08:43:03 -0700 To: weh@microsoft.com, FreeBSD Hackers X-Mailer: Apple Mail (2.3731.700.6) References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.30:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.30:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZFFk4mQzz4fd0 Wei Hu wrote on Date: Mon, 28 Aug 2023 07:32:35 UTC : > When I was testing a new NIC, I found the single stream scp = performance was almost 8 time slower than Linux on the RX side. = Initially I thought it might be something with the NIC. But when I = switched to sending the file on localhost, the numbers stay the same.=20 >=20 > Here I was sending a 2GB file from sender to receiver using scp. = FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux = kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. = The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu. >=20 > Sender Receiver throughput > Linux FreeBSD 70 MB/s > Linux Linux 550 MB/s > FreeBSD FreeBSD 70 MB/s > FreeBSD Linux 350 MB/s > FreeBSD localhost 70 MB/s > Linux localhost 550 MB/s >=20 > =46rom theses test, it seems I can rule out the issue on NIC and its = driver. Looks the FreeBSD kernel network stack is much slower than Linux = on single stream TCP, or there are some problem with scp? >=20 > I also tried turning on following kernel parameters on FreeBSD kernel. = But it makes no difference, neither do the other tcp cc algorithms such = as htcp and newreno. >=20 > net.inet.tcp.soreceive_stream=3D"1" > net.isr.maxthreads=3D"-1" > net.isr.bindthreads=3D"1" >=20 > net.inet.ip.intr_queue_maxlen=3D2048 > net.inet.tcp.recvbuf_max=3D16777216 > net.inet.tcp.recvspace=3D419430 > net.inet.tcp.sendbuf_max=3D16777216 > net.inet.tcp.sendspace=3D209715 > kern.ipc.maxsockbuf=3D16777216 >=20 > Any ideas? You do not give explicit commands to try. Nor do you specify your hardware context that is involved, just that HyperV is involved. So, on a HoneyComb (16 cortex-A72's) with Optane boot media in its PCIe slot I, no HyperV or VM involved, tried: # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 120.2MB/s 00:42 It is not a high performance system. 64 GiBytes of RAM. So instead trying a ThreadRipper 1950X that also has Optane in a CPIe slot for its boot media, no HyperV or VM involved, # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 299.7MB/s 00:17 (These systems do not run with any tmpfs areas, not even /tmp . So I'm not providing that kind of example, at least for now.) 128 GiBytes of RAM. Both systems are ZFS based but with a simple single partition. (Used for bectl BE not for other types of reasons to use ZFS. I could boot UFS variants of the boot media and test that kind of context.) So both show between your FreeBSD figure and the Linux figure. I've no means of checking how reasonable the figures are relative to your test context. I just know the results are better than you report for localhost use. =3D=3D=3D Mark Millard marklmi at yahoo.com