From nobody Wed Feb 22 22:17:01 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMVrV41L1z3sHNG for ; Wed, 22 Feb 2023 22:17:14 +0000 (UTC) (envelope-from ggm@algebras.org) Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMVrT3jRrz3lRj for ; Wed, 22 Feb 2023 22:17:13 +0000 (UTC) (envelope-from ggm@algebras.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=algebras-org.20210112.gappssmtp.com header.s=20210112 header.b=YVVBYRFG; spf=pass (mx1.freebsd.org: domain of ggm@algebras.org designates 2001:4860:4864:20::2c as permitted sender) smtp.mailfrom=ggm@algebras.org; dmarc=none Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-1720887dfcdso11834150fac.6 for ; Wed, 22 Feb 2023 14:17:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=algebras-org.20210112.gappssmtp.com; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=6NntP/peRAQdzEFbZED+RnGAAGl0yabY9pmJ5SX+oLw=; b=YVVBYRFGZWrEty/x9U596U5ZyWqzhmVQoalfXEnviVDek0FYcJ0PkDjjV810RBub91 vqvSmZWWFJUaBsC3JcCU8Fm2g0Ly7IBpj6n2QY96wJzGMlV4fChL2PaAf3qontVPADJE ArIIAV/3eJE+Xo+fQS/aC9+JR06w7gnH+2dggemXHcFTiY+NrsnOhxn3oXwO0T/GecFP UTix5sF8UihnidvfpUnEzKxv1/VDyI4c3stf05MWZVWdRVeq6hjCAN+Sf08Mk9g/sU+N Ym6Lomnq27+6Zko3Kp4EctqmD7vNovohzwoWW1K6HJZWh8izQCEabSouK5Rx4DIlnfZz fGDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6NntP/peRAQdzEFbZED+RnGAAGl0yabY9pmJ5SX+oLw=; b=upeODiZEW0a9hhC6rSJn7Kz38z9q98VAITuacq8QkhAHAp5dtKrEvtiiXbbJd9Ybc0 yIlm4rcRMF/cHvp2vR88s81eNREfQJsUgchSN5y9uTXREGXp7Io32kF3urPMVUegjW/R YrFZtOfyM3v4PjDTN3ljdp4giBRn+AwE/Di7wPRRM8lNuLVLoqKtt21CZjJ0jpYRFKBq /RLoDpM0CQxbruuMCfrheYlrsG3Z7RHaEl7t3LnWq1HwYbtwBwXb6/oLIuAJh+6B+UDW 8qVJxjXF9IzowbhVjtZnpzMcijWF7AijIfkKynJvjU6mSgI8wENIwaRZtwLVQao/cHd/ Q7nQ== X-Gm-Message-State: AO0yUKXW6GCdMmR7z3Bino6X8bkYDa7wQFGj2nEK9Gm78fP9qmlPGceX ZL0Ns0cV2DeNIK7kRuOb6vJJ11kd8UBTkOf6E1gErmuy+/Rx1Q== X-Google-Smtp-Source: AK7set/A6pK7hbnLEoJveP1akSh4lapXwH4xxQLZ4Un/08wKyWhIiwigovjnMITZtDJysHpyybwWIpEZ0t02H/wuH7A= X-Received: by 2002:a05:6871:10a:b0:172:193e:a9a8 with SMTP id y10-20020a056871010a00b00172193ea9a8mr1000586oab.285.1677104232489; Wed, 22 Feb 2023 14:17:12 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> In-Reply-To: From: George Michaelson Date: Thu, 23 Feb 2023 08:17:01 +1000 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[algebras-org.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::2c:from]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[algebras-org.20210112.gappssmtp.com:+]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[algebras.org]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PMVrT3jRrz3lRj X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N I use mbuffer for this purpose. I'm only on protected "inside" links even if cross-router (ie not just a switching fabric) which may make my use-case a poor fit for your need. mbuffer has several strategies around how much buffer to use, what rate limits to apply, I believe can do tweaks to use scatter-gather models (the necessary ioctls to turn things on &c but these are almost certainly tuned to linux) you also need to look at your ethernet card offload. Normally beneficial. it can interact badly with in-kernel models of end-to-end flow, because it's performing its own work "on your behalf" I sort of miss having a "null" cipher in SSH. I didn't entirely understand why it got stripped out: the 'who are you" initialisation authorisation is beneficial, even if the datastream is (deliberately) unprotected. the RC2 fallback didn't seem to impose much cost, but thats gone too now. -G On Thu, Feb 23, 2023 at 7:43 AM Freddie Cash wrote: > > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get the "none" cipher working with ssh on FreeBSD. > > We use ssh to connect to the remote server and enable a netcat listener on port X, then pipe the send through netcat to the remote system on port X. That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. > > > > Cheers, > Freddie > > Typos due to smartphone keyboard. > > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrote: >> >> On 22/02/2023 22:08, mike tancsa wrote: >> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> >> Interresting numbers. I think I am the only one who get best speed >> >> with chacha20-poly1305@openssh.com >> >> >> >> >> >> It seems the speed of SSH is limited by single core performance which >> >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). >> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >> > >> > The CPU I have has >> > aesni0: on motherboard >> > >> > which probably helps. >> >> That explains it >> aesni0: No AES or SHA support. >> >> >> I know there were some HPN patches to ssh, beside that is there any >> >> option I can try to use less CPU? >> >> >> >> I will play with cpuset to pin ssh on one core and everything else on >> >> the other core. >> > >> > It looks like you are running into a CPU bottleneck TBH >> >> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but >> without some tweaks on ssh I will not gain more speed :( >> >> Thank you for your help! >> >> Miroslav Lachman >> >>