From nobody Sat Feb 18 15:48:15 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PJtPW60hZz3sf6P for ; Sat, 18 Feb 2023 15:48:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PJtPW51Fsz3wvL for ; Sat, 18 Feb 2023 15:48:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676735295; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p09BByFudTv690vFOqgWkDjYlMJAmUQjHFdRBf0T6jM=; b=hdCdu3PsGnRVAl/YpskiCzrI/v/DH7x70ZNKbcjZaRcP7HMuq/Cx7DuH07vjoiBhzt1JlL B1r9zpQoUZggFR3D3zRvvRROsqlLXYObiJBv637T2I7iSTblViIovYxXRganxCVT4BT+90 ReTafP7SKsXyWtoV33mUzXJoMDgR/2GrIm46lAZG4ED8DlXTnLVfV0r608DOdV0RAx1cOY 3RxN7BEbdYxjSfNE3t2AaGG8dxV2cx5v+4QRR7HSJvcaHf/I1tGSpxP5M83RcbRrMsmK5/ 936moX8ql77RauNMWew95ueQHnaZzydXzFTtBBJLE1KbKSu/tYLI74215NTiFg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676735295; a=rsa-sha256; cv=none; b=qVvj4u5w9BlldgunjO0GRnHKkoi+KshySnB/JlSCYEMKyOEhLo3EG/793YOaLK/NBk4knV mIUWLDRxZmWyFjWRiAi4QvDeEbaPFB/RqXy0MHo6G06MdTgrQkw8hA0uLyh1rudSkhoMEd jDYiARR5UAHYaqgb5kQ3Mk+HovivBD52IUi109MG7LBmOVfOZfubGjwlwJ6SDjHl+W+7a2 lGo8AfZUUSXS2SUzFj2tHISrx4cznOpcU9PXPLExzV+dq6/Enqjx88cP7ME5j8q9fSdlw8 t54Wuj38fj3VXgNzDCAPTTgk9zHnVmSBmgG+HZAWABJofwGJSXteVpJ4z+BwPA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PJtPW41V6zSxb for ; Sat, 18 Feb 2023 15:48:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31IFmFpK065374 for ; Sat, 18 Feb 2023 15:48:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31IFmF2j065373 for fs@FreeBSD.org; Sat, 18 Feb 2023 15:48:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 265951] ext2fs: kernel panic when trivial rsync operation from UFS2 (system disk) to ext3 partitions (data disk) Date: Sat, 18 Feb 2023 15:48:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D265951 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open See Also| |https://reviews.freebsd.org | |/D38503 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 19 05:48:19 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PKF2q6blHz3smNt for ; Sun, 19 Feb 2023 05:48:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PKF2q4t02z3MxP for ; Sun, 19 Feb 2023 05:48:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676785699; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MgVkOorjAY2TJkbCp7qcZi93u1g5EITIRF3gogbLELY=; b=hVrB8C5NP66zYw/6kqiJsobwPn8lx7fBF0EX8SzTUW1Z8szV4NxgI8LF3BEQ9Ntn98bVIg 50jIN1q8J9d5XdiFXmnVcz5jRDgAP67opB32XYqYsvTcJ7FpUyWvRG+aOtBCsngaVqKCT9 5WHP6bH48re2KDKl09/2Sd2SXzPB/qpVFZHTlO8xVodq12O4fdq9BYkWwrjgZFex2NAOxw 1Ii/dsqM4PFTJnTyI8chR5YlRSZkgRjPYcZp/xbKUEbBgJT99dPcSMIlxR5FX3As99V7wn O5q3019ubqrgXj6r3lo7sivZ/NblEXla66/Ve2Qb6BKaAZ4XY5LpQOEVZxIoIw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676785699; a=rsa-sha256; cv=none; b=jyfI7qWGK5SdWfLM9fWxigU6YCARr2CFkOs1a/FfSpkXZeUzlTrkAkv7OXlEyfDI4YwJgg lug/UwAQb9d6xlBKMjAzCT//Q0YMgiBjWg/x2S+gGCIPqgyK3lrljn8TPCEvvC2bsGC7+G k+4CRL8bF9R97KB+kBZyO7wJAw1ftYI7eYM1XGeIpY0baoKTqiE6U65HyFkpCduLFwx/j9 lyyJrY4AKg/KQs+1BD/l1+X26gyy5t0IZqO5Ip/hqb33CdRaXMuGBlwhpfZN4ErAjVtWEm dbmc1vP7vDH2KYoxWLTw6/Nud9EKKQda3tzd5jytNGrUTducJVVfG1Oaf79VsA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PKF2q3wl4zrTg for ; Sun, 19 Feb 2023 05:48:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31J5mJW6042085 for ; Sun, 19 Feb 2023 05:48:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31J5mJB7042084 for fs@FreeBSD.org; Sun, 19 Feb 2023 05:48:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 269341] [msdosfs] deadlock with sendfile Date: Sun, 19 Feb 2023 05:48:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: commit-hook@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269341 --- Comment #5 from commit-hook@FreeBSD.org --- A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3Dc2ee668306bbe3edf4a05246ed3a88f52= dfc94ae commit c2ee668306bbe3edf4a05246ed3a88f52dfc94ae Author: Konstantin Belousov AuthorDate: 2023-02-11 18:09:30 +0000 Commit: Konstantin Belousov CommitDate: 2023-02-19 05:16:25 +0000 msdosfs deextend: validate pages of the partial buffer PR: 269341 (cherry picked from commit 0152d453a08fa2bad694dc04a8184fce2b7faa10) sys/fs/msdosfs/msdosfs_denode.c | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 19 16:38:17 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PKWSn405Qz3rPjt for ; Sun, 19 Feb 2023 16:38:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PKWSn1VjSz3N3y for ; Sun, 19 Feb 2023 16:38:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676824697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lgAFRSuMSvHZRWkMwDcL6hu66xIIbDTetGrQH6dp6OM=; b=bamswlKKYl6I1ZxT/fdSniwVavwXlse6X1xObGLR7EqdyFe6dRI5RMexqaSvjo3bJqLlee 3fOgDY4O31rcGg74T5+fCYQpt1QWLmvGhlKBpUSifah69Ypq5wP3lPawAXZ+OKJYw/bR9m uWaq+2rwx9dOoAvzuw9G2SxcpAYnE9RVIyTpI0BAKzX6qRFYbckSwSvrLfo3GD6mkyiMtv u9VWyXt4SOkfgCynC06vr/aeIDMTycfgqb9K4+YL6t6a9H26WdhEEZoqDvkNYGLr49i0jr zHQpaiQzRup9TVMeZk3RWxp3fXFKTcJCnrhJXR6oyEDI0lCKp7OwA85sydGWeQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676824697; a=rsa-sha256; cv=none; b=hxyEGAaYQkJLA774gXI6cu2eoukpuudYUNsEO84TDlrgG5reyoYfMz1b9RVLikliYVvnNE rQbRRIP3u736vOMYL70rfAdBT+XAc5BAh9LEVxKwpBW3n8T7Wpovfi0Q2es7sI8a2rv03F k020T8OUM9wbVkga+xLf109pGcWt3xQ/EKdGcMqtmVTLKRAUIjug7nF1qcsGZ6uKr4phJJ Z+hFSA0uPnVbnFb1APwIMSCpcYoNRCyTTxmEw3TKqcuEXD0NWSbUaTVXhYU91uJ53XjwuC PdXtDI1ZWs5XAngP/evwJx85hB51LtKLQyL5itFlkuXkr83995k9+zQZobGfIg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PKWSn0TRVz17Z7 for ; Sun, 19 Feb 2023 16:38:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31JGcHFm064436 for ; Sun, 19 Feb 2023 16:38:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31JGcH7p064435 for fs@FreeBSD.org; Sun, 19 Feb 2023 16:38:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 269513] Crash when running poudriere with USE_TMPFS=all with recently upgraded CURRENT Date: Sun, 19 Feb 2023 16:38:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc bug_status keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269513 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |fs@FreeBSD.org CC| |grahamperrin@freebsd.org Status|New |Open Keywords| |crash --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 19 16:40:44 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PKWWd1rNCz3rPq9 for ; Sun, 19 Feb 2023 16:40:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PKWWd0sGbz3NRp for ; Sun, 19 Feb 2023 16:40:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676824845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WGj1pkmbypAq8hUt75mcfhKH1n7PH9U7mmuikpSZV/s=; b=wjEUCVHJqbq0C1aWUh86VPEJO64e/oRXtsOCYD3+LvB/kMtU5lExnb3mvy2+JGS0Cs9AGs Aq1nuIdzUyvKH/WVe+4i0sUr0OfFv/9M4ERTIHkS76wcbab/KBrWGGaglpARb5vKz6oFg3 D5jqfzozp7WCpkBrntEoVui1I/UGHBYHP8hMhPl+Snv4EslaR2+pCJSWqqU2ZB1Jfyu2GK gkqw1vNMex5dXW1ShK91B2BxEBHosLsAPtUx5jDmQgxeWQSK3U9QUMl68hb4W8E2ztG2x5 IQI3N9CKqDcJvHngWltKBl2T+e3UhzHPK54u0fC2qKpqvBf769+SGB4aWXVHRw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676824845; a=rsa-sha256; cv=none; b=gl72G29V7fLQvhETqLUHGbosh2C4GPAWWGuNAyqhJUysT4yvgg1wFk6Bw/igiAnegRqnb4 toIVr5nkz/djCH0Wz/FNeXREHkDFHoBZOaT7x1oNqwNpDalYjiNSTRePq2TDYemr34Gft7 pMIKd1QTNgO70SNnA/jJysY3hWp25BBBKqLYH884YKqe9vlHkF7qXJd6HsZzU5yRrRPczv 3Du0YqXvOqwtCU1x2Zjyljm9gmnO8/AUsWP9c+6hYk6mBAfHLwu6qoyLWwAwhmV1laXjN/ 7WHb32HUTbZCBxnTK8cScu7CGpjXOvcCoxRe3/BvSlL6Ft3QiQZ6W7VU1OAutw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PKWWc71sjz17pG for ; Sun, 19 Feb 2023 16:40:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31JGeiP4066089 for ; Sun, 19 Feb 2023 16:40:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31JGeieH066088 for fs@FreeBSD.org; Sun, 19 Feb 2023 16:40:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 269513] Crash when running poudriere with USE_TMPFS=all with recently upgraded CURRENT Date: Sun, 19 Feb 2023 16:40:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mjg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269513 Mateusz Guzik changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mjg@FreeBSD.org --- Comment #3 from Mateusz Guzik --- this should be fixed as of 3a3450eda6d4616df51a30f84a872d9d43669d78 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Feb 19 21:01:04 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PKdJ11WnHz3s1wP for ; Sun, 19 Feb 2023 21:01:05 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PKdJ03tQJz4D5s for ; Sun, 19 Feb 2023 21:01:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676840464; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=M5KVYFzli93T0jE5puEhNxQ1xiY5h/H/MqgYP1lAnio=; b=mB88wYyX/p09amSCU88w5NNbaJvERHoqKvQ2e/zFMG6tbHbn1UkBimOQyUf4n5V06/33Yz sOMQKFc3OcBSCwXqEP0p9gi3oOjeQZLFVkZPlikMl10xWBG3VtyoyufAwy+7dIuSPIGvRR +eLuSZW0n39yBHW6ieE341NTwZIsIVTDRFm1BtcA1ZIoIYDKfH+/PQ7gV0nhEqUu0w5/Ha ujXfKF90ciUXuqf5k7BNN/D2Tqo0mNhYVt4XQFi+2O5R9SKblBhDoVcZU5cm7rygVSu17/ KntrxeECXiOMRbdx5cKWh/YoPlHsN0dwyujgbuw1br0QWNoMbfwHP/msBDogDw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676840464; a=rsa-sha256; cv=none; b=TfbcW3rZiK3osVmTOwndWWMPjVLhrpqwHQVuTZ08O/OM/7m0cv6OAV7rPJ3t6ZMhZtMp9a OgQfi77kijQVVbjTpmHmGGi4WDBifFrwyCMqCgTa3iJpLrqWn48t+EMqR0omlZHov9EUEl w+o692wcmsONNoTf0ByfQ+hP2FyYdgdQfF6mU0oasZ6zib+MAlz2gBeE1lh0u9X/MW7zVx +PJYXWyXPCLgtKWLyxijAl3eyFaNSOZgUnuIusaV43F6QRLGiSR3k/vDAzQzPzdj/8ACpW mkBdsby6DWjakj125NKvPdQ6sY8zsrOxrztfgt5jQfk8U6ebkLqsXgD8GVrOrQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PKdJ030k6zGjD for ; Sun, 19 Feb 2023 21:01:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31JL14HK049821 for ; Sun, 19 Feb 2023 21:01:04 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31JL14uY049820 for fs@FreeBSD.org; Sun, 19 Feb 2023 21:01:04 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202302192101.31JL14uY049820@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 19 Feb 2023 21:01:04 +0000 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16768404642.7Bc84.44161" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N --16768404642.7Bc84.44161 Date: Sun, 19 Feb 2023 21:01:04 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w Open | 240831 | zfs: Panic during snapshot on 12.1-STABLE r352648 Open | 243973 | [zfs] rollback segmentation fault Open | 244656 | zfs: resilver doesn't provide enough information Open | 244692 | gjournal: Does not support TRIM Open | 251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W Open | 269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit 8 problems total for which you should take action. --16768404642.7Bc84.44161 Date: Sun, 19 Feb 2023 21:01:04 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    211491 | System hangs after "Uptime" on reboot with ZFS
Open        |    237067 | ZFS: Crash in vdev_dtl_reassess when using GELI w
Open        |    240831 | zfs: Panic during snapshot on 12.1-STABLE r352648
Open        |    243973 | [zfs] rollback segmentation fault
Open        |    244656 | zfs: resilver doesn't provide enough information 
Open        |    244692 | gjournal: Does not support TRIM
Open        |    251035 | ZFS: Allow 64 bit ZFS to support 32 bit ioctls (W
Open        |    269503 | docs.freebsd.org: default vfs.zfs.arc.meta_limit

8 problems total for which you should take action.
--16768404642.7Bc84.44161-- From nobody Wed Feb 22 16:20:30 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMLwv30dNz3t8XF for ; Wed, 22 Feb 2023 16:20:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMLwv2112z40xd for ; Wed, 22 Feb 2023 16:20:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677082831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F+o2QNd/s5mZd+5Jq9pihLl3E/jPRniaRgD3VrPopU8=; b=COyqK7DX5wUgrFYw6qK6FKnKhSKYvH+6zFnVaO1k52+mFbyGnzr+Ip0EIIuySWRyd04bL5 d1aZN0BJAFpfnI3PtJ8v2L0wVpH1vASjdFIHGg1e8Wl1NvGy9VS44fEmD6m6UE79Y1C4U7 ljv/EPcDrdqf7iJGUTZCj8oEDa8DTTWlpx0cnZ6LQApsrM5/4eLB3aNRmdA6oZEsWpJ/8S EinXuFVTG945G8PW4hhPL39en5yC1lOTfkX4p2AvogthunqUM+ivSutkV19RNs7PPv8Mfq j8hdWAUIjxffg1byE68dI2SjU3Ud9PnXg2/4N/St5LJeDK1udIGhlCDl5Dml5Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677082831; a=rsa-sha256; cv=none; b=n2gK3Ha5WT3jQ4vENHcDtsmSeTGXF4Pi7B5jgqx/iDTFhM7FmhktcZwkFAZv1fj26f2J+w TT6zJ6INkImA0Zrl/eT9xkvwwgXHguY2W+WJ1zGXrBlGdkaDZX+DkIsqfTSVbi1tBEIoTM 5s3PuzFlTy/AvZayMKjFyX8WsPumFAIORsxTyLXS+Q7naMaJaOAfipEx8eoXcwxHTnDiZY qKuW3F04Keb80rjZYbOsPCsWkY5EKcmuzTCiIbsXp+4mRckCgHZYBBmw061USDEw+mmnIb S2iGtmt/QFJFeoBNf9gja3c9f7WJsK/yiWCAOLYP8q75SuAkMgGadddp1C7OmA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PMLwv13nJz176B for ; Wed, 22 Feb 2023 16:20:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31MGKV79020983 for ; Wed, 22 Feb 2023 16:20:31 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31MGKVbk020982 for fs@FreeBSD.org; Wed, 22 Feb 2023 16:20:31 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 252700] page fault in zfsctl_snapdir_lookup Date: Wed, 22 Feb 2023 16:20:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: rew@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252700 Robert Wing changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|In Progress |Closed --- Comment #10 from Robert Wing --- fixed upstream in https://github.com/openzfs/zfs/commit/28251d81d723292a6813f93495f2c6c132938= 945 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Feb 22 16:36:26 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMMHG460nz3t9rk for ; Wed, 22 Feb 2023 16:36:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMMHF6zQzz432r for ; Wed, 22 Feb 2023 16:36:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677083786; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hTJOGqHpueuMNvCpRvaFgjfQ6V736vIrPMicYuEICS0=; b=d3+9BJTqq0dR1ByPBzk6CBalt8toojBrRhWh36k75ljjQecaQUfmVbc/6UiZGmWLg+AFf7 Dbg7x1DDpsCqc9gnBznr+eEikTSvxDPAueuwakvmlvE33p7oiT15B9bHVBKC9ZNqPthDpx bq8pEieCtNht99Dkl3AcjnJYx2h2DDD1+a43Dqu8jwYF7fecLu0AaYKXm1/Tek/8QLkwN9 x6IhULsv/WW2sGnpwtBQuD21I+Wak1LG+VidKHU3gB744a04gMjr5yP6o/nsDdNIXdSVpy IB256xWk/j+0A6BIUUBd/abzC3/KW32oEUEvQrIxJqw0JkCC7qrNXRQuyqjt1Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677083786; a=rsa-sha256; cv=none; b=J3SkTu1MZyrBa7wvY8MvBmmIkSuIbYR0SQTPq5PUbdkxBBewZ4/8YddjKN1jlBHrThhKBN Q4gNgDtBmuHXCy1b3RJ3ld3lsTTw/7YkP4eQT25STgfAkNu21p8EfAfe6DDkIVCR9sWwhH rM35zCzUxHa9UnZqKiQELyaUBgxD3paARJe1VlXdUlig9FSpGRKJp1+dcy+2jRMhFyf0Gd KE995tOtW/6njLH6vmYjtuaTmT+4jtM8Lwth8IxfBbCw4/MLTe0H7O5PlI/9sWH2kGT32V 7Jgn1JzATdGJPVU5xW2Ikbj6dD4tOVVCZJJCHFn4pnyO6xDEdQAGBaUz/WsNsw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PMMHF653Bz16hQ for ; Wed, 22 Feb 2023 16:36:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31MGaP9v045826 for ; Wed, 22 Feb 2023 16:36:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31MGaPk2045825 for fs@FreeBSD.org; Wed, 22 Feb 2023 16:36:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 252700] page fault in zfsctl_snapdir_lookup Date: Wed, 22 Feb 2023 16:36:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252700 --- Comment #11 from Andriy Gapon --- (In reply to Robert Wing from comment #10) Thank you for the fix! But I think that we move bugs to closed/fixed after a fix is actually in FreeBSD. Even more, when the fix is in all active branc= hes where it is needed / wanted. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Feb 22 18:06:06 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMPGp2zYKz3tFqs for ; Wed, 22 Feb 2023 18:06:10 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMPGn2W36z4HD5; Wed, 22 Feb 2023 18:06:09 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of "SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz"; dmarc=none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 898B6D788C; Wed, 22 Feb 2023 19:06:07 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 5A2FDD7896; Wed, 22 Feb 2023 19:06:06 +0100 (CET) Message-ID: Date: Wed, 22 Feb 2023 19:06:06 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) Content-Language: cs-Cestina, en-US To: Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-1.79 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[no SPF record]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[quip.cz]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4PMPGn2W36z4HD5 X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 17/05/2021 17:57, mike tancsa wrote: [...] > On the mail spool just via mbuffer (no ssh involved at all) > > zfs send > summary:  514 GiByte in 1h 09min 35.9sec - average of  126 MiB/s > zfs send -c > summary:  418 GiByte in 1h 05min 58.5sec - average of  108 MiB/s > > and the same dataset, sending just through OpenSSH took 1h:06m (zfs > send) and 1h:01m (zfs send -c) > > > On the large dataset (large VMDK files), similar pattern. I did find one > interesting thing, when I was testing with a smaller dataset of just > 12G.  As the server has 65G of ram, 29 allocated to ARC, sending a zfs > stream with -c made a giant difference. I guess there is some efficiency > with sending something thats already compressed in arc ? Or maybe its > just all cache effect. > > Testing with one with about 1TB of referenced data using mbuffer with > and without ssh  and just ssh > > zfs send with mbuffer and ssh > summary:  772 GiByte in 51min 06.2sec - average of  258 MiB/s > zfs send -c > summary:  772 GiByte in 1h 22min 09.3sec - average of  160 MiB/s > > And the same dataset just with ssh -- zfs send 53min and zfs send -c 55min > > and just mbuffer (no ssh) > > summary:  772 GiByte in 56min 45.7sec - average of  232 MiB/s (zfs send -c) > summary: 1224 GiByte in 53min 20.4sec - average of  392 MiB/s (zfs send) I am facing similar problem with low performance of zfs send over network. I have 2 machines in two different datacenters, both have 1Gbps NIC and I would like to saturate the network but it seems impossible even if "everything" seem to have enough unused resources. The sending side is very old HP ML110 G5 with bge0 NIC, receiving side is VM with enough CPU, RAM, 22TB storage and vtnet0 NIC. Sender has about 25% CPU idle during sending, disks are not saturated according to iostat -w -x but I still cannot see more than 52MiB/s. Everything about zfs snapshot, send, receive etc. is handled by syncoid from sanoid package (ssh + mbuffer + pv, no lzop) I thought it is slow because of ssh ciphers so I tried to change with --sshciper but it was 10MiB/s slower when I changed from default chacha20-poly1305@openssh.com to aes128-ctr The interesting thing was testing with scp with different ciphers (tested all available) and sending 1GB file full of random bytes from dd if=/dev/urandom. With each scp cipher it was able to achieve 65MiB/s. Even more interesting is that "zfs send -c" is slower than "zfs send" (without compressed stream). I would expect the opposite. Why is zfs send -c slower? Some datasets are transferred with much lower speed about 40MiB/s. What are the limiting factors for zfs send if there is 25% idle CPU, RAIDZ of 4x SATA disks should give more than 50MB/s (iostat show disks busy oscillating between 50% - 80% and receiving side is under 50% usage (CPU and disk) Sender stats looks like this: # iostat -x -w 30 ada{0,1,2,3} extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada0 227 18 10961.1 260.5 2 2 56 2 0 17 ada1 276 18 12996.9 260.8 2 2 62 2 0 18 ada2 225 17 10567.9 252.1 3 2 68 3 0 19 ada3 275 17 12217.1 252.4 2 2 77 2 0 19 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada0 185 11 6623.3 126.5 5 4 110 5 5 24 ada1 204 11 7238.8 124.0 3 4 109 4 3 24 ada2 187 11 6474.7 119.5 5 10 106 5 5 26 ada3 205 10 6825.5 115.7 4 8 108 4 2 26 extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b ada0 239 3 7698.6 59.3 5 1 46 4 0 29 ada1 241 3 7507.2 59.3 4 1 61 4 1 29 ada2 240 3 7476.6 59.3 5 1 40 5 0 30 ada3 235 3 7156.9 59.3 4 2 75 4 0 30 # ifstat -i bge0 60 bge0 KB/s in KB/s out 1245.04 49964.32 1321.39 53035.46 1323.03 53108.41 1319.83 52947.85 1248.55 50089.84 1240.00 49706.52 1278.22 51309.49 1274.60 51159.64 1279.98 51366.20 1157.62 46435.98 Network latency between machines round-trip min/avg/max/stddev = 0.326/0.492/0.897/0.156 ms As I need to transfer about 15TB of data I really would like to speed it up to the maximum. In the current state it would take a week to transfer it. Kind regards Miroslav Lachman From nobody Wed Feb 22 19:01:46 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMQVz1pWDz3tJ1W for ; Wed, 22 Feb 2023 19:01:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMQVz0RvJz4N3N for ; Wed, 22 Feb 2023 19:01:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677092507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bS4WL57okXCAqYlgk35s5c59GoLvVQ7GpoRzeU8wnoE=; b=bM0LoxQn5nsv4Sap0VdMPwcfsRUpJH0ERTVEPuppU/0/Wt6nzAoQroJE1h9eAKZsNno20n e6YObzyRqzWppSWP7bz3UDBUShEFuehlyC4f6oXvfmStS3y9QGtF/EW5dx4y52aOa7Dxhf bL9GUwxixuCTOz8JLigJdN/jCoe8ci3WAEhhZ7xBxeBQyujG9EQ/1eRCHS1CtdYXqwdS/S NxJUcBT144Gc/6FTIgjE7TpCNN5vorARVWclAQGXuipimYH+hfzO/ZJEVtB8dH5IGtG7TC Y0WG01ebONNQNqGGLFwM8I47AYxfLZ0Uo0TALnWsIjeXBzOMm1riblSzo8JCwA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677092507; a=rsa-sha256; cv=none; b=gTFsZwIVfbwqHHQDNoRqYkt1FmWDUatwv9iVA/tvjvA5IsEoreaRJljULC/DE0Mx/nLs8f s0+QTAqh77mTUI4L67bqJtX+n74ExMvGfa3zl2GBNPUHq434492QKLSbf3rnixR9zwY5ev oNuUmC7H9CYTqzlXn0zs+iH1GS0s2zTLcdtcKIqcOu2v4NMM+hdVS0JvSPrvx8e5ZaXXkr klwKb3S0RK4j6l8j++JUlhSmPqAqvl9+yg63fklmH0waaaqqEwWrsVgeEVxkGRTDzKuw4t nfnsTa7Bu8yECAG4NDwSzvn6l2d3CZPIzitcv64c+OuUyny++8DN8bzPu7Pwjw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PMQVy69hRz1C1L for ; Wed, 22 Feb 2023 19:01:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31MJ1klH062957 for ; Wed, 22 Feb 2023 19:01:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31MJ1k5s062956 for fs@FreeBSD.org; Wed, 22 Feb 2023 19:01:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 252700] page fault in zfsctl_snapdir_lookup Date: Wed, 22 Feb 2023 19:01:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: rew@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252700 Robert Wing changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Closed |Open Resolution|FIXED |--- --- Comment #12 from Robert Wing --- (In reply to Andriy Gapon from comment #11) Alright, I'll open this back up until the fix is merged into FreeBSD just in case someone else stumbles across this problem in the meantime. also - thanks for reviewing this change. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Feb 22 19:58:36 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMRmh1g59z3s7TY for ; Wed, 22 Feb 2023 19:58:44 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMRmg4lwmz3FKL for ; Wed, 22 Feb 2023 19:58:43 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Authentication-Results: mx1.freebsd.org; none Received: from scrappy.simplesystems.org (scrappy.simplesystems.org [65.66.246.73]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id 31MJwaJs027265; Wed, 22 Feb 2023 13:58:36 -0600 (CST) Date: Wed, 22 Feb 2023 13:58:36 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@scrappy.simplesystems.org To: Miroslav Lachman <000.fbsd@quip.cz> cc: freebsd-fs Subject: Re: speeding up zfs send | recv (update) In-Reply-To: Message-ID: References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> User-Agent: Alpine 2.20 (GSO 67 2015-01-07) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Wed, 22 Feb 2023 13:58:36 -0600 (CST) X-Rspamd-Queue-Id: 4PMRmg4lwmz3FKL X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7018, ipnet:65.64.0.0/13, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Wed, 22 Feb 2023, Miroslav Lachman wrote: > > I am facing similar problem with low performance of zfs send over network. I > have 2 machines in two different datacenters, both have 1Gbps NIC and I would > like to saturate the network but it seems impossible even if "everything" > seem to have enough unused resources. Have you tried doing your zfs send locally and send the output to /dev/null? Is the speed much faster than when sending over the network? Does 'zpool scrub' perform well for your pool? Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt From nobody Wed Feb 22 20:13:55 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMS6L3fRQz3s8Pb for ; Wed, 22 Feb 2023 20:14:02 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMS6K5rVgz3GSF; Wed, 22 Feb 2023 20:14:01 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 31MKDsKR093337 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Wed, 22 Feb 2023 15:13:54 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4::29] ([IPv6:2607:f3e0:0:4:0:0:0:29]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 31MKDrlH089627 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 22 Feb 2023 15:13:54 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: Date: Wed, 22 Feb 2023 15:13:55 -0500 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: speeding up zfs send | recv (update) Content-Language: en-US To: Miroslav Lachman <000.fbsd@quip.cz>, Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: mike tancsa In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 X-Rspamd-Queue-Id: 4PMS6K5rVgz3GSF X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 2/22/2023 1:06 PM, Miroslav Lachman wrote: > > I am facing similar problem with low performance of zfs send over > network. I have 2 machines in two different datacenters, both have > 1Gbps NIC and I would like to saturate the network but it seems > impossible even if "everything" seem to have enough unused resources. > The sending side is very old HP ML110 G5 with bge0 NIC, receiving side > is VM with enough CPU, RAM, 22TB storage and vtnet0 NIC. > Sender has about 25% CPU idle during sending, disks are not saturated > according to iostat -w -x but I still cannot see more than 52MiB/s. > Everything about zfs snapshot, send, receive etc. is handled by > syncoid from sanoid package (ssh + mbuffer + pv, no lzop) > > I thought it is slow because of ssh ciphers so I tried to change with > --sshciper but it was 10MiB/s slower when I changed from default > chacha20-poly1305@openssh.com to aes128-ctr aes128-gcm@openssh.com is what I settled on for the cipher.   If you just blast dd if=/dev/zero | ssh are you able to achieve close to wirespeed ?  As (I think) I mentioned in this thread, different zfs datasets transmit at different speeds.  Ones with tens of thousands of small files are much slower than those with a few multi gigabit files.  The disk seems to be the limiting factor for me as graphing "time spent in IO" via telegraf, shows the disks pretty well 100% busy. e.g. dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null"   20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s 20000+0 records in 19982+36 records out 20971520000 bytes transferred in 18.259553 secs (1148523199 bytes/sec) vs  dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null"   20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s 20000+0 records in 19961+78 records out 20971520000 bytes transferred in 43.947239 secs (477197671 bytes/sec) dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr mdtancsa@coldstore "cat - > /dev/null"   20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s 20000+0 records in 19973+54 records out 20971520000 bytes transferred in 29.263111 secs (716653818 bytes/sec)     ---Mike From nobody Wed Feb 22 21:03:44 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMTCt5DtWz3sCJf for ; Wed, 22 Feb 2023 21:03:54 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMTCt36Ylz3LPh; Wed, 22 Feb 2023 21:03:54 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id D8635D788C; Wed, 22 Feb 2023 22:03:45 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 20ABED7886; Wed, 22 Feb 2023 22:03:45 +0100 (CET) Message-ID: <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> Date: Wed, 22 Feb 2023 22:03:44 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) Content-Language: cs-Cestina To: mike tancsa , Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4PMTCt36Ylz3LPh X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 22/02/2023 21:13, mike tancsa wrote: > aes128-gcm@openssh.com is what I settled on for the cipher.   If you > just blast dd if=/dev/zero | ssh are you able to achieve close to > wirespeed ?  As (I think) I mentioned in this thread, different zfs > datasets transmit at different speeds.  Ones with tens of thousands of > small files are much slower than those with a few multi gigabit files. > The disk seems to be the limiting factor for me as graphing "time spent > in IO" via telegraf, shows the disks pretty well 100% busy. Yes, I read about it. Which is also interesting as I thought when zfs sends the whole filesystem it will not walk file by file as will rsync do. > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c > aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null" >   20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s >  dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c > chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null" >   20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr > mdtancsa@coldstore "cat - > /dev/null" >   20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s > Interresting numbers. I think I am the only one who get best speed with chacha20-poly1305@openssh.com # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-gcm@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null" 2074083328 bytes (2074 MB, 1978 MiB) transferred 50.003s, 41 MB/s # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c chacha20-poly1305@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null" 2069889024 bytes (2070 MB, 1974 MiB) transferred 33.006s, 63 MB/s # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-ctr quip@aa.bb.cc.dd "cat - > /dev/null" 2091909120 bytes (2092 MB, 1995 MiB) transferred 45.007s, 46 MB/s It seems the speed of SSH is limited by single core performance which is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). Even if CPU has 50% idle, ssh runs on 99.8% of single core. I know there were some HPN patches to ssh, beside that is there any option I can try to use less CPU? I will play with cpuset to pin ssh on one core and everything else on the other core. Kind regards Miroslav Lachman From nobody Wed Feb 22 21:08:49 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMTKY0bQqz3sCjc for ; Wed, 22 Feb 2023 21:08:49 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMTKY0B74z3LmW; Wed, 22 Feb 2023 21:08:49 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 31ML8mYp011994 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Wed, 22 Feb 2023 16:08:48 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4::29] ([IPv6:2607:f3e0:0:4:0:0:0:29]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 31ML8lKm005496 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 22 Feb 2023 16:08:48 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> Date: Wed, 22 Feb 2023 16:08:49 -0500 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: speeding up zfs send | recv (update) Content-Language: en-US To: Miroslav Lachman <000.fbsd@quip.cz>, Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> From: mike tancsa In-Reply-To: <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 X-Rspamd-Queue-Id: 4PMTKY0B74z3LmW X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > Interresting numbers. I think I am the only one who get best speed > with chacha20-poly1305@openssh.com > > > It seems the speed of SSH is limited by single core performance which > is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E2160). > Even if CPU has 50% idle, ssh runs on 99.8% of single core. The CPU I have has aesni0: on motherboard which probably helps. > > I know there were some HPN patches to ssh, beside that is there any > option I can try to use less CPU? > > I will play with cpuset to pin ssh on one core and everything else on > the other core. It looks like you are running into a CPU bottleneck TBH     ---Mike > > > Kind regards > Miroslav Lachman > From nobody Wed Feb 22 21:31:00 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMTqC0D9zz3sDgJ for ; Wed, 22 Feb 2023 21:31:03 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMTqB6SjHz3Q4p; Wed, 22 Feb 2023 21:31:02 +0000 (UTC) (envelope-from SRS0=ZcGF=6S=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 47AD6D7891; Wed, 22 Feb 2023 22:31:01 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 9295FD788C; Wed, 22 Feb 2023 22:31:00 +0100 (CET) Message-ID: <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> Date: Wed, 22 Feb 2023 22:31:00 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) Content-Language: cs-Cestina To: mike tancsa , Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4PMTqB6SjHz3Q4p X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which >> is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E2160). >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > The CPU I have has > aesni0: on motherboard > > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on >> the other core. > > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman From nobody Wed Feb 22 21:43:24 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMV5j3kkKz3sFS1 for ; Wed, 22 Feb 2023 21:43:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMV5j3J17z3hPm; Wed, 22 Feb 2023 21:43:37 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-536e10ae021so102351797b3.7; Wed, 22 Feb 2023 13:43:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=2SjvyeMs4Im0k1KmHMh4roJKEFQh8AzRLRta3IB7RWQ=; b=KeSktspGOnCCfL5hUCmz3kdaPcLw0J9D1Ft0KvCpbpVSU0KVYvzWCgETFY3spvRSjQ ptuFOd+gPlerK8/0uRQ/jQVeE0rof/z21oXH54aXdHdzcVpZR/E2SA9x42DCPytXcs5F mfGMfPRcLlKG2LtbO50I9IUgiWbBjFBDY4bMOrtPKoe7OWRDyT5Be4kXGbNGnLWKwAci +LzfmloncGs3oM9pXIgNxMLmtlwY0GwwxdqCjaiVL97CvvqXliFVyqJ2xtUQi+7eMrp9 vrJEYhNKkKJ8l50br9HPwGdkggr616mPugM1w4Tj3qwTHZCd3TslAYT5roq/alRzbYEM wu6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=2SjvyeMs4Im0k1KmHMh4roJKEFQh8AzRLRta3IB7RWQ=; b=GjmIs1HwV0kAdqvxenvBr6voJ3jL1fukfMc7PgAsFde6S1zFOWtPAV15X0p8Ii3IMZ gy0WUo2tGOzgpiLWCVh110RRkHG8Wl92DR8FZ+pecrQWIxQK8O08VAL54ok04Rr2f2CT ivEI6/+FLWJwqEbC0zU9j11+JHRvq3sy06pCTLPKIzP2pG+UaT55D9fh3D3PalDu8I7I 5pcXlReCyt81JnG2BH+U4m2eo66Z3y6PDETZC3eoUJyavoT30MnYc8sDoBUerf0TdBdA Pgs9pLuzVvbODyQRQXBx3b+vbBLxrXli8Rco5kpLBmZlHr7EFJGj9jlXfOd+R6FWiuph UMiQ== X-Gm-Message-State: AO0yUKXbXSttf4zTM3JgeQnXwNM+L+lA2M1sthiMxkg9hOTRQWHvMaJ+ 0mrqCNRvtabDT/MvD1JVXWZxqCrv3ESJc7pggKW6fyeK X-Google-Smtp-Source: AK7set+DIigyQx0zQua+kqfZli45uBWX/TRXG7VJglcYaNMyAkmaCW2makCKJhBfqDKK/KQTtPiwUbzTolQzBDoho0Y= X-Received: by 2002:a05:690c:a97:b0:52e:b74b:1b93 with SMTP id ci23-20020a05690c0a9700b0052eb74b1b93mr1772271ywb.0.1677102216687; Wed, 22 Feb 2023 13:43:36 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> In-Reply-To: <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> From: Freddie Cash Date: Wed, 22 Feb 2023 13:43:24 -0800 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: Miroslav Lachman <000.fbsd@quip.cz> Cc: mike tancsa , Alan Somers , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000e0b62c05f550c911" X-Rspamd-Queue-Id: 4PMV5j3J17z3hPm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000e0b62c05f550c911 Content-Type: text/plain; charset="UTF-8" [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the send stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get the "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on port X, then pipe the send through netcat to the remote system on port X. That way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrote: > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed > >> with chacha20-poly1305@openssh.com > >> > >> > >> It seems the speed of SSH is limited by single core performance which > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > > > The CPU I have has > > aesni0: on motherboard > > > > which probably helps. > > That explains it > aesni0: No AES or SHA support. > > >> I know there were some HPN patches to ssh, beside that is there any > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else on > >> the other core. > > > > It looks like you are running into a CPU bottleneck TBH > > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but > without some tweaks on ssh I will not gain more speed :( > > Thank you for your help! > > Miroslav Lachman > > > --000000000000e0b62c05f550c911 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
[Sorry for top part, GMail sucks for replies.]

If this is a LAN or private WAN where yo= u trust the network, piping the send stream through netcat will remove ssh = from the equation.

That's = what we switched to using once it became almost impossible to get the "= ;none" cipher working with ssh on FreeBSD.

=
We use ssh to connect to the remote server and enab= le a netcat listener on port X, then pipe the send through netcat to the re= mote system on port X. That way it's logged and uses ssh for authentica= tion.

We easily saturate= gigabit links between our ZFS systems using netcat.



Cheers,
Freddie

Typos due to smartphone keyboa= rd.

On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, &l= t;000.fbsd@quip.cz> wrote:
On 22/02/2023 22:08, mike tancsa wrote:<= br> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman


--000000000000e0b62c05f550c911-- From nobody Wed Feb 22 22:17:01 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMVrV41L1z3sHNG for ; Wed, 22 Feb 2023 22:17:14 +0000 (UTC) (envelope-from ggm@algebras.org) Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMVrT3jRrz3lRj for ; Wed, 22 Feb 2023 22:17:13 +0000 (UTC) (envelope-from ggm@algebras.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=algebras-org.20210112.gappssmtp.com header.s=20210112 header.b=YVVBYRFG; spf=pass (mx1.freebsd.org: domain of ggm@algebras.org designates 2001:4860:4864:20::2c as permitted sender) smtp.mailfrom=ggm@algebras.org; dmarc=none Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-1720887dfcdso11834150fac.6 for ; Wed, 22 Feb 2023 14:17:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=algebras-org.20210112.gappssmtp.com; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=6NntP/peRAQdzEFbZED+RnGAAGl0yabY9pmJ5SX+oLw=; b=YVVBYRFGZWrEty/x9U596U5ZyWqzhmVQoalfXEnviVDek0FYcJ0PkDjjV810RBub91 vqvSmZWWFJUaBsC3JcCU8Fm2g0Ly7IBpj6n2QY96wJzGMlV4fChL2PaAf3qontVPADJE ArIIAV/3eJE+Xo+fQS/aC9+JR06w7gnH+2dggemXHcFTiY+NrsnOhxn3oXwO0T/GecFP UTix5sF8UihnidvfpUnEzKxv1/VDyI4c3stf05MWZVWdRVeq6hjCAN+Sf08Mk9g/sU+N Ym6Lomnq27+6Zko3Kp4EctqmD7vNovohzwoWW1K6HJZWh8izQCEabSouK5Rx4DIlnfZz fGDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6NntP/peRAQdzEFbZED+RnGAAGl0yabY9pmJ5SX+oLw=; b=upeODiZEW0a9hhC6rSJn7Kz38z9q98VAITuacq8QkhAHAp5dtKrEvtiiXbbJd9Ybc0 yIlm4rcRMF/cHvp2vR88s81eNREfQJsUgchSN5y9uTXREGXp7Io32kF3urPMVUegjW/R YrFZtOfyM3v4PjDTN3ljdp4giBRn+AwE/Di7wPRRM8lNuLVLoqKtt21CZjJ0jpYRFKBq /RLoDpM0CQxbruuMCfrheYlrsG3Z7RHaEl7t3LnWq1HwYbtwBwXb6/oLIuAJh+6B+UDW 8qVJxjXF9IzowbhVjtZnpzMcijWF7AijIfkKynJvjU6mSgI8wENIwaRZtwLVQao/cHd/ Q7nQ== X-Gm-Message-State: AO0yUKXW6GCdMmR7z3Bino6X8bkYDa7wQFGj2nEK9Gm78fP9qmlPGceX ZL0Ns0cV2DeNIK7kRuOb6vJJ11kd8UBTkOf6E1gErmuy+/Rx1Q== X-Google-Smtp-Source: AK7set/A6pK7hbnLEoJveP1akSh4lapXwH4xxQLZ4Un/08wKyWhIiwigovjnMITZtDJysHpyybwWIpEZ0t02H/wuH7A= X-Received: by 2002:a05:6871:10a:b0:172:193e:a9a8 with SMTP id y10-20020a056871010a00b00172193ea9a8mr1000586oab.285.1677104232489; Wed, 22 Feb 2023 14:17:12 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> In-Reply-To: From: George Michaelson Date: Thu, 23 Feb 2023 08:17:01 +1000 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[algebras-org.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::2c:from]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[algebras-org.20210112.gappssmtp.com:+]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[algebras.org]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PMVrT3jRrz3lRj X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N I use mbuffer for this purpose. I'm only on protected "inside" links even if cross-router (ie not just a switching fabric) which may make my use-case a poor fit for your need. mbuffer has several strategies around how much buffer to use, what rate limits to apply, I believe can do tweaks to use scatter-gather models (the necessary ioctls to turn things on &c but these are almost certainly tuned to linux) you also need to look at your ethernet card offload. Normally beneficial. it can interact badly with in-kernel models of end-to-end flow, because it's performing its own work "on your behalf" I sort of miss having a "null" cipher in SSH. I didn't entirely understand why it got stripped out: the 'who are you" initialisation authorisation is beneficial, even if the datastream is (deliberately) unprotected. the RC2 fallback didn't seem to impose much cost, but thats gone too now. -G On Thu, Feb 23, 2023 at 7:43 AM Freddie Cash wrote: > > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get the "none" cipher working with ssh on FreeBSD. > > We use ssh to connect to the remote server and enable a netcat listener on port X, then pipe the send through netcat to the remote system on port X. That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. > > > > Cheers, > Freddie > > Typos due to smartphone keyboard. > > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrote: >> >> On 22/02/2023 22:08, mike tancsa wrote: >> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> >> Interresting numbers. I think I am the only one who get best speed >> >> with chacha20-poly1305@openssh.com >> >> >> >> >> >> It seems the speed of SSH is limited by single core performance which >> >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). >> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >> > >> > The CPU I have has >> > aesni0: on motherboard >> > >> > which probably helps. >> >> That explains it >> aesni0: No AES or SHA support. >> >> >> I know there were some HPN patches to ssh, beside that is there any >> >> option I can try to use less CPU? >> >> >> >> I will play with cpuset to pin ssh on one core and everything else on >> >> the other core. >> > >> > It looks like you are running into a CPU bottleneck TBH >> >> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but >> without some tweaks on ssh I will not gain more speed :( >> >> Thank you for your help! >> >> Miroslav Lachman >> >> From nobody Wed Feb 22 22:22:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMVyf6BjBz3sHZd for ; Wed, 22 Feb 2023 22:22:34 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp1.bway.net (smtp1.bway.net [216.220.96.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMVyf39Nxz3mGP; Wed, 22 Feb 2023 22:22:34 +0000 (UTC) (envelope-from spork@bway.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (pool-173-70-201-95.nwrknj.fios.verizon.net [173.70.201.95]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp1.bway.net (Postfix) with ESMTPSA id D3D7111816; Wed, 22 Feb 2023 17:22:32 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bway.net; s=mail; t=1677104553; bh=yq8Xcd5iGF7txSLvXp21z8+gl+CbX3s7gHc4eCWzITE=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=Oaz+qf98543xUMSa3bFh3ZPGyIQg/dO0MCpd6CChbSbf+vBjLZFkgitA/laB9BXGN ngTUyOcGgYPcKAaenIcT4qnhMuLldAT0UcvlEE2lG8X404u6rM/qayZbciTwbNFMsu GYdNc5Uwlmhtrc9Li1QFOxXBZNaeTVzmqm8Tn/54= From: Charles Sprickman Message-Id: <0171E506-3899-42B2-B7DC-4145BAA595D7@bway.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: speeding up zfs send | recv (update) Date: Wed, 22 Feb 2023 17:22:32 -0500 In-Reply-To: Cc: Miroslav Lachman <000.fbsd@quip.cz>, mike tancsa , Alan Somers , freebsd-fs To: Freddie Cash References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> X-Mailer: Apple Mail (2.3696.120.41.1.2) X-Rspamd-Queue-Id: 4PMVyf39Nxz3mGP X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8059, ipnet:216.220.96.0/19, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Feb 22, 2023, at 4:43 PM, Freddie Cash wrote: >=20 > [Sorry for top part, GMail sucks for replies.] >=20 > If this is a LAN or private WAN where you trust the network, piping = the send stream through netcat will remove ssh from the equation. >=20 > That's what we switched to using once it became almost impossible to = get the "none" cipher working with ssh on FreeBSD. >=20 > We use ssh to connect to the remote server and enable a netcat = listener on port X, then pipe the send through netcat to the remote = system on port X. That way it's logged and uses ssh for authentication. >=20 > We easily saturate gigabit links between our ZFS systems using netcat. This is kind of tangential, but is there any ssh client/server that is = able to make use of multiple CPU cores or is that just not easily = possible? The first set of hosts I worked with that had 10Gb/s internal network = ports kind of showed me how much of a bottleneck trying to encrypt with = a single core is. If using netcat or similar to avoid the ssh overhead, can IPSEC or a VPN = option (wireguard?) be a bit of a workaround? Do any VPN implementations = on FreeBSD put multiple cores to use? Thanks, Charles >=20 >=20 >=20 > Cheers, > Freddie >=20 > Typos due to smartphone keyboard. >=20 > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz = > wrote: > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed=20= > >> with chacha20-poly1305@openssh.com = > >> > >> > >> It seems the speed of SSH is limited by single core performance = which=20 > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160).=20= > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > >=20 > > The CPU I have has > > aesni0: on motherboard > >=20 > > which probably helps. >=20 > That explains it > aesni0: No AES or SHA support. >=20 > >> I know there were some HPN patches to ssh, beside that is there any=20= > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else = on=20 > >> the other core. > >=20 > > It looks like you are running into a CPU bottleneck TBH >=20 > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 > without some tweaks on ssh I will not gain more speed :( >=20 > Thank you for your help! >=20 > Miroslav Lachman >=20 >=20 --Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
On = Feb 22, 2023, at 4:43 PM, Freddie Cash <fjwcash@gmail.com> = wrote:

[Sorry for top part, GMail sucks for = replies.]

If this is a LAN or private WAN where you trust = the network, piping the send stream through netcat will remove ssh from = the equation.

That's what we switched to using once it became = almost impossible to get the "none" cipher working with ssh on = FreeBSD.

We use ssh to connect to the remote server and = enable a netcat listener on port X, then pipe the send through netcat to = the remote system on port X. That way it's logged and uses ssh for = authentication.

We easily saturate gigabit = links between our ZFS systems using netcat.

This is kind of tangential, but is there any ssh = client/server that is able to make use of multiple CPU cores or is that = just not easily possible?

The first = set of hosts I worked with that had 10Gb/s internal network ports kind = of showed me how much of a bottleneck trying to encrypt with a single = core is.

If using netcat or similar = to avoid the ssh overhead, can IPSEC or a VPN option (wireguard?) be a = bit of a workaround? Do any VPN implementations on FreeBSD put multiple = cores to use?

Thanks,

Charles




Cheers,
Freddie

Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, = <000.fbsd@quip.cz> wrote:
On 22/02/2023 22:08, mike tancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best = speed
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance = which
>> is very poor on this machine (Intel(R) Pentium(R) Dual  = CPU E2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on = motherboard
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there = any
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything = else on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



= --Apple-Mail=_9217D20E-B136-4AEA-9001-DD4231C0CFCD-- From nobody Thu Feb 23 03:28:55 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMdmD0Mmkz3ssNc for ; Thu, 23 Feb 2023 03:29:00 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from wilbur.contactoffice.com (wilbur.contactoffice.com [212.3.242.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMdmC52d6z4LS5 for ; Thu, 23 Feb 2023 03:28:59 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; none Received: from ichabod.co-bxl (ichabod.co-bxl [10.2.0.36]) by wilbur.contactoffice.com (Postfix) with ESMTP id A08852994; Thu, 23 Feb 2023 04:28:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1677122937; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Cc:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=7317; bh=Thvl5cftda4IBZ9l07SrlzL9+ZFlmWFRRy6MxH1inbE=; b=Qnkla9+UNpdCGokf9sWPXm9JQfNuVvMrElkWVPLdfRZMP9PyH5YNRMrWzESaHCIb wVJCCoSeMy1SOxYWz3orwT7PfsHkFEowCF2X1vKM2xOodtITnz40zGj/bpMrGhENjTl JuF4JjzexLee0grcuYGpsLAdyWiyU6b1+dY4wNKw+a2ezXCfhFu2tN81+55qUzOaVeh y6vTCMkKfiYRMrdrWB8m12n6HmlS7Ci42kC03AgTotLkiCtkYOfRXFMjM1vt8Vc6Xja tILSeQ3FBYtNLn6T3dZN4rO2FZbhyYc4vHwAvYi9d+u7Howas01Vcmr0Kfw7B30JwO1 w+GD77peKA== Date: Thu, 23 Feb 2023 04:28:55 +0100 (CET) From: Sysadmin Lists To: freebsd-fs Cc: Miroslav Lachman <000.fbsd@quip.cz>, Freddie Cash Message-ID: <741387429.91447.1677122934622@ichabod.co-bxl> In-Reply-To: References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> Subject: Re: speeding up zfs send | recv (update) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_91444_1729784993.1677122934622" X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Rspamd-Queue-Id: 4PMdmC52d6z4LS5 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:10753, ipnet:212.3.242.64/26, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N ------=_Part_91444_1729784993.1677122934622 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the sen= d stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get th= e "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on = port X, then pipe the send through netcat to the remote system on port X. T= hat way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrot= e: On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed=20 >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which=20 >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E2160).= =20 >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >=20 > The CPU I have has > aesni0: on motherboard >=20 > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any=20 >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on=20 >> the other core. >=20 > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman You could pipe the stream through an encrypting program before piping to netcat, then decrypt on the recieving end. $ zfs send | crypt | netcat ipaddr 2222 $ netcat -vl 2222 | crypt | zfs recv I don't know if zfs can handle that, but worth a try. $ man crypt =C2=A0 =C2=A0 The enigma utility, also known as crypt is a very simple encr= yption =C2=A0 =C2=A0 =C2=A0program, working on a =E2=80=9Csecret-key=E2=80=9D basi= s.=C2=A0 It operates as a filter, i.e., =C2=A0 =C2=A0 =C2=A0it encrypts or decrypts a stream of data from standard = input, and writes =C2=A0 =C2=A0 =C2=A0the result to standard output.=C2=A0 Since its operatio= n is fully symmetrical, =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it. -- Sent with https://mailfence.com Secure and private email ------=_Part_91444_1729784993.1677122934622 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=
On Feb 22, 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com>= wrote:
">
[Sorry for top part, = GMail sucks for replies.]

If t= his is a LAN or private WAN where you trust the network, piping the send st= ream through netcat will remove ssh from the equation.
That's what we switched to using once it became al= most impossible to get the "none" cipher working with ssh on FreeBSD.
=

We use ssh to connect to the = remote server and enable a netcat listener on port X, then pipe the send th= rough netcat to the remote system on port X. That way it's logged and uses = ssh for authentication.

= We easily saturate gigabit links between our ZFS systems using netcat.
<= /div>



Cheers,
Freddie

Typos due to= smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <= ;000.fbsd@quip.cz> wrote:
On 22/02/2023 22:08, mike tancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could pipe= the stream through an encrypting program before piping to
netcat= , then decrypt on the recieving end.

$ zfs send | = crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but worth a tr= y.

$ man crypt
    = The enigma utility, also known as crypt is a very simple encryption
     program, working on a =E2=80=9Csecret-key=E2=80=9D b= asis.  It operates as a filter, i.e.,
     it= encrypts or decrypts a stream of data from standard input, and writes
     the result to standard output.  Since its op= eration is fully symmetrical,
     feeding the enc= rypted data stream again through the engine (using the
  &nb= sp;  same secret key) will decrypt it.

=
--=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_91444_1729784993.1677122934622-- From nobody Thu Feb 23 03:51:03 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMfFn2ZSYz3stlZ for ; Thu, 23 Feb 2023 03:51:09 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from mailout-l3b-97.contactoffice.com (mailout-l3b-97.contactoffice.com [212.3.242.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMfFm3wtpz4P3v for ; Thu, 23 Feb 2023 03:51:08 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mailfence.com header.s=20210208-e7xh header.b=k8cyeO7w; spf=pass (mx1.freebsd.org: domain of sysadmin.lists@mailfence.com designates 212.3.242.97 as permitted sender) smtp.mailfrom=sysadmin.lists@mailfence.com; dmarc=pass (policy=quarantine) header.from=mailfence.com Received: from ichabod.co-bxl (ichabod.co-bxl [10.2.0.36]) by mailout-l3b-97.contactoffice.com (Postfix) with ESMTP id D209BB00 for ; Thu, 23 Feb 2023 04:51:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1677124265; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=9370; bh=wGBH19F+W/usqtgXfEv4vA9VwRMK9bNHrOASya4lsSs=; b=k8cyeO7w/EiIwXWvXQuCqS3lgTeFWFidEDEC6gTcRnzdq7IDK4ioULPbWqP0nIBy LnDUy+4QFq53/sLF5VrMFhFC1OY5opbownKUbUUV/6H7EiWSWbcTRPIAm446i1YGsG/ +pKxAG0otNBZhtmnbJ9Ni4e9eDuyS1vvSfziL7dm/Qam/98GEoCGvK3pOq4tTNdOo4h nBFV08aZ1jR/qIPwZWrLm+M0lR8llIrNGkUnynq0QY4EcteCqIASiSoLZEitAfMsou0 YxWLVJfSX8IdoQcNNUGLs52CWxI5vKxMgqy0eLJ45/6m9Upy5t68F5BYm3s0jhGJ9sG yzOUYNK8Bg== Date: Thu, 23 Feb 2023 04:51:03 +0100 (CET) From: Sysadmin Lists To: freebsd-fs Message-ID: <409401259.92260.1677124263649@ichabod.co-bxl> In-Reply-To: <741387429.91447.1677122934622@ichabod.co-bxl> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> Subject: Re: speeding up zfs send | recv (update) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_92257_810470091.1677124263648" X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Spamd-Result: default: False [-3.93 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.84)[-0.838]; DMARC_POLICY_ALLOW(-0.50)[mailfence.com,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:212.3.242.64/26]; R_DKIM_ALLOW(-0.20)[mailfence.com:s=20210208-e7xh]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[212.3.242.97:from]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:10753, ipnet:212.3.242.64/26, country:US]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[mailfence.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4PMfFm3wtpz4P3v X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N ------=_Part_92257_810470091.1677124263648 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Feb 22, 2023 at 7:28 PM, Sysadmin Lists w= rote: On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the sen= d stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get th= e "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on = port X, then pipe the send through netcat to the remote system on port X. T= hat way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrot= e: On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed=20 >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which=20 >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E2160).= =20 >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >=20 > The CPU I have has > aesni0: on motherboard >=20 > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any=20 >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on=20 >> the other core. >=20 > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman You could pipe the stream through an encrypting program before piping to netcat, then decrypt on the recieving end. $ zfs send | crypt | netcat ipaddr 2222 $ netcat -vl 2222 | crypt | zfs recv I don't know if zfs can handle that, but worth a try. $ man crypt =C2=A0 =C2=A0 The enigma utility, also known as crypt is a very simple encr= yption =C2=A0 =C2=A0 =C2=A0program, working on a =E2=80=9Csecret-key=E2=80=9D basi= s.=C2=A0 It operates as a filter, i.e., =C2=A0 =C2=A0 =C2=A0it encrypts or decrypts a stream of data from standard = input, and writes =C2=A0 =C2=A0 =C2=A0the result to standard output.=C2=A0 Since its operatio= n is fully symmetrical, =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it. -- Sent with https://mailfence.com Secure and private email Seems to work: # zfs create zroot/test # mount -t zfs zroot/test /mnt/test # date > /mnt/test/testfile # zfsnap snapshot -p testsend- zroot/test # zfs list -t snap zroot/test NAME ... zroot/test@testsend-2023-02-22_19.46.15--1m ... # nc -l 2222 | crypt | zfs recv zroot/newtest Enter key: # zfs send zroot/test@testsend-2023-02-22_19.46.15--1m | crypt | nc -w 5 lo= calhost 2222 Enter key: # zfs list zroot/newtest NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 USED=C2=A0 AVAIL=C2=A0 =C2=A0= =C2=A0REFER=C2=A0 MOUNTPOINT zroot/newtest=C2=A0 =C2=A0 96K=C2=A0 70.7G=C2=A0 =C2=A0 =C2=A0 =C2=A096K=C2= =A0 none -- Sent with https://mailfence.com Secure and private email ------=_Part_92257_810470091.1677124263648 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=

On Feb 22, 2023 at 7:28 PM, Sysadmin Lists <sysadmin.lists@= mailfence.com> wrote:
">

On Feb 22, 2023 at 1:43 PM, Freddie= Cash <fjwcash@gmail.com> wrote:
">
[Sorry for top part, GMail sucks for replies.]
<= br>
If this is a LAN or private WAN where you trust = the network, piping the send stream through netcat will remove ssh from the= equation.

That's what we swit= ched to using once it became almost impossible to get the "none" cipher wor= king with ssh on FreeBSD.

We use ssh to connect to the remote server and enable a netcat listener o= n port X, then pipe the send through netcat to the remote system on port X.= That way it's logged and uses ssh for authentication.

We easily saturate gigabit links between our= ZFS systems using netcat.



Cheers,=
Freddie

Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:= 31 p.m. Miroslav Lachman, <000.fbsd@= quip.cz> wrote:
On 22/02/2023 22:08, mi= ke tancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could pipe= the stream through an encrypting program before piping to
netcat= , then decrypt on the recieving end.

$ zfs send | = crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but worth a tr= y.

$ man crypt
    = The enigma utility, also known as crypt is a very simple encryption
     program, working on a =E2=80=9Csecret-key=E2=80=9D b= asis.  It operates as a filter, i.e.,
     it= encrypts or decrypts a stream of data from standard input, and writes
     the result to standard output.  Since its op= eration is fully symmetrical,
     feeding the enc= rypted data stream again through the engine (using the
  &nb= sp;  same secret key) will decrypt it.

=
--=20 Sent with https://mailfence.com =20 Secure and private email

Seems to work:

# zfs create zroot/test
# mount -t zfs= zroot/test /mnt/test
# date > /mnt/test/testfile
# = zfsnap snapshot -p testsend- zroot/test
# zfs list -t snap zroot/= test
NAME ...
zroot/test@testsend-2023-02-22_19.46.15--= 1m ...

# nc -l 2222 | crypt | zfs recv zroot/= newtest
Enter key:

# zfs send zroo= t/test@testsend-2023-02-22_19.46.15--1m | crypt | nc -w 5 localhost 2222
Enter key:

# zfs list zroot/newtest
<= div>NAME            USED  AVAIL  &n= bsp;  REFER  MOUNTPOINT
zroot/newtest    96K&= nbsp; 70.7G       96K  none
=
--=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_92257_810470091.1677124263648-- From nobody Thu Feb 23 17:09:07 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PMzyb1w2mz3sDx5 for ; Thu, 23 Feb 2023 17:09:11 +0000 (UTC) (envelope-from SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PMzyZ6jSwz3lJp for ; Thu, 23 Feb 2023 17:09:10 +0000 (UTC) (envelope-from SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 98539D7890; Thu, 23 Feb 2023 18:09:08 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 51B4BD78C0; Thu, 23 Feb 2023 18:09:07 +0100 (CET) Message-ID: Date: Thu, 23 Feb 2023 18:09:07 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) To: Freddie Cash Cc: mike tancsa , freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> Content-Language: cs-Cestina From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4PMzyZ6jSwz3lJp X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 22/02/2023 22:43, Freddie Cash wrote: > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the > send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get > the "none" cipher working with ssh on FreeBSD. It would be good if I can run it without ssh overhead but I don't found any example on the internet how to use Sanoid / Syncoid without SSH. It looks like ssh is requirement and mbuffer and pv is optional. > We use ssh to connect to the remote server and enable a netcat listener > on port X, then pipe the send through netcat to the remote system on > port X. That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. Nice trick. Next time I will need to move fet TB of zfs data I will try different solution instead of Syncoid to avoid slow ssh. Netcat or direct mbuffer listening on TCP port seems better. Kind regards Miroslav Lachman From nobody Thu Feb 23 17:11:06 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PN00r6qN3z3sFYL for ; Thu, 23 Feb 2023 17:11:08 +0000 (UTC) (envelope-from SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PN00r23b9z3ljQ for ; Thu, 23 Feb 2023 17:11:08 +0000 (UTC) (envelope-from SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of "SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz"; dmarc=none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 68EDED78C0 for ; Thu, 23 Feb 2023 18:11:07 +0100 (CET) Received: from [192.168.145.50] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id BB076D7890 for ; Thu, 23 Feb 2023 18:11:06 +0100 (CET) Message-ID: Date: Thu, 23 Feb 2023 18:11:06 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Subject: Re: speeding up zfs send | recv (update) Content-Language: cs-Cestina To: freebsd-fs@freebsd.org References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-1.19 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.977]; NEURAL_HAM_LONG(-0.42)[-0.415]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; R_SPF_NA(0.00)[no SPF record]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[quip.cz]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=5bAR=6T=quip.cz=000.fbsd@elsa.codelab.cz]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4PN00r23b9z3ljQ X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 22/02/2023 23:17, George Michaelson wrote: > I use mbuffer for this purpose. I'm only on protected "inside" links > even if cross-router (ie not just a switching fabric) which may make > my use-case a poor fit for your need. > > mbuffer has several strategies around how much buffer to use, what > rate limits to apply, I believe can do tweaks to use scatter-gather > models (the necessary ioctls to turn things on &c but these are almost > certainly tuned to linux) > > you also need to look at your ethernet card offload. Normally > beneficial. it can interact badly with in-kernel models of end-to-end > flow, because it's performing its own work "on your behalf" I tried disabling rxcsum, txcum, tso4 but then it became 10 MiB/s slower. > I sort of miss having a "null" cipher in SSH. I didn't entirely > understand why it got stripped out: the 'who are you" initialisation > authorisation is beneficial, even if the datastream is (deliberately) > unprotected. the RC2 fallback didn't seem to impose much cost, but > thats gone too now. It would be really useful for my usecase too. Kind regards Miroslav Lachman From nobody Thu Feb 23 19:15:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PN2mc5h9dz3sdCv for ; Thu, 23 Feb 2023 19:15:44 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PN2mc3lw2z43Kc for ; Thu, 23 Feb 2023 19:15:44 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102d.google.com with SMTP id cp7-20020a17090afb8700b0023756229427so360227pjb.1 for ; Thu, 23 Feb 2023 11:15:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=eSnL7H2JqspZngpdyxwyjiXUlI+P21ydRQwHKrXneO4=; b=UOMF8KoXyycLfzXJ3OSSy3nD8qrFhRj14R5MMPQBmdMmEr3iHecrKsb8j6a2JD6nQ5 cdW9uPP5EpuWCF+WUfuEc8JCm6gqZrzYgY+BEGXf3W9uMOkRQ4VBNNfzfqgcLe6W/Llc b9CcRnZiQQ5y3o+ex9lpujbkwGCCRiM/qbY8Be1Se9irdt6SpsNVquUHn9MDzDNA6BrI DEz3KlWqbS8mwbRemk3SwV6zr61bLSC5WGn/h56NPS8O8hsaKILlSpuDbI2kQGioPNd4 NFdcyBjg8cncZi9I1s7iL1KVW61OweFrut5Kuicm7g55KlzUd+9NjVMZRcUL2ia3PpKY g8FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eSnL7H2JqspZngpdyxwyjiXUlI+P21ydRQwHKrXneO4=; b=Oea4SE5tRERhJXL+jccC6jTSQrV0bMMrBxpz5PiBbVJhxNDLur8zngjvbbtYCxeOM8 Fw1Hj2lqHQnplIlC+dKgz+jSG2PLXyTTHkDeZbgkNaF0vt4YXX54Cp+FWUCcJYbWTmKF ab8VNRfZuyQ4uoaGMChyZvrn+h/4P0MiuA83pyOlDwDwowPr5V8EwxXbmofT5eSrDRJ+ 9BGjjso2LP7iygAYvcXM4OPLGz1uw5PUJS5oDXDDLy1CTo7CzGMGj6njRKN5lKv5TXLK jzeYRyYGY3lHtI2bZEQ4+htuTYOUM59CN+0DjFLBord80mmEQx4xAhnDVBpg0SKIhaq+ aA9Q== X-Gm-Message-State: AO0yUKVeKAz6zEsrmTPJeiVEKPfwkBxx/4HYKjIAZqYsQregJsAyFCG1 q/NRirYGP72VxpdawHOkUGH3H8rED2gYPhGg+HFcSQFd X-Google-Smtp-Source: AK7set/bETShe/DgE0Q+6rbl9v8G8/11PnymBWF0ByoTCnuHWniOPAM6WsC6FqgJnF9ritoEP2imYUAqrRekCD+ixoY= X-Received: by 2002:a17:903:3291:b0:19a:9f86:adab with SMTP id jh17-20020a170903329100b0019a9f86adabmr2031687plb.7.1677179743156; Thu, 23 Feb 2023 11:15:43 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> In-Reply-To: <741387429.91447.1677122934622@ichabod.co-bxl> From: Chris Watson Date: Thu, 23 Feb 2023 13:15:32 -0600 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: Sysadmin Lists Cc: Freddie Cash , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000d0c06905f562d633" X-Rspamd-Queue-Id: 4PN2mc3lw2z43Kc X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000d0c06905f562d633 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable [Sorry miroslav, I hit send without checking the To: this was meant to be public] I=E2=80=99m a bit late, but I mentioned this to someone on this thread priv= ately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentio= ned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted/massively overly complex/hard to secure just to simply have a secure pipe for doing ZFS send|recv. Simply configuring an spiped PtP pipe between A and B seems the simplest, most secure, performant option here. At least considering all the other options tossed out in this thread. No one=E2=80=99s using spiped? O.o Thoughts? Has anyone compared ssh to spiped regarding overhead and throughput in this scenario? Chris On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists wrote: > > On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: > > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the > send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get > the "none" cipher working with ssh on FreeBSD. > > We use ssh to connect to the remote server and enable a netcat listener o= n > port X, then pipe the send through netcat to the remote system on port X. > That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. > > > > Cheers, > Freddie > > Typos due to smartphone keyboard. > > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> > wrote: > > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed > >> with chacha20-poly1305@openssh.com > >> > >> > >> It seems the speed of SSH is limited by single core performance which > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > > > The CPU I have has > > aesni0: on motherboard > > > > which probably helps. > > That explains it > aesni0: No AES or SHA support. > > >> I know there were some HPN patches to ssh, beside that is there any > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else on > >> the other core. > > > > It looks like you are running into a CPU bottleneck TBH > > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but > without some tweaks on ssh I will not gain more speed :( > > Thank you for your help! > > Miroslav Lachman > > > > You could pipe the stream through an encrypting program before piping to > netcat, then decrypt on the recieving end. > > $ zfs send | crypt | netcat ipaddr 2222 > $ netcat -vl 2222 | crypt | zfs recv > > I don't know if zfs can handle that, but worth a try. > > $ man crypt > The enigma utility, also known as crypt is a very simple encryption > program, working on a =E2=80=9Csecret-key=E2=80=9D basis. It operat= es as a filter, > i.e., > it encrypts or decrypts a stream of data from standard input, and > writes > the result to standard output. Since its operation is fully > symmetrical, > feeding the encrypted data stream again through the engine (using th= e > same secret key) will decrypt it. > > > -- Sent with https://mailfence.com Secure and private email --000000000000d0c06905f562d633 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
[Sorry miroslav, I hit send without ch= ecking the To: this was meant to be public]=C2=A0

I=E2=80=99m a bit lat= e, but I mentioned this to someone on this thread privately, I=E2=80=99m cu= rious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentioned in this th= read. I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be,= imo, grossly unwarranted/massively overly complex/hard to secure just to s= imply have a secure pipe for doing ZFS send|recv.=C2=A0

Simply configuring an spiped P= tP pipe between A and B seems the simplest, most secure, performant option = here. At least considering all the other options tossed out in this thread.= =C2=A0

No o= ne=E2=80=99s using spiped? O.o

Thoughts?=C2=A0

Has anyone compared ssh to spiped regarding overhe= ad and throughput in this scenario?
Chris

On Wed, Feb 22, 202= 3 at 9:29 PM Sysadmin Lists <sysadmin.lists@mailfence.com> wrote:
=

= On Feb 22, 2023 at 1:43 PM, Freddie Cash <= fjwcash@gmail.com> wrote:
[Sorry for top part, GMail sucks for replies.]
=
If this= is a LAN or private WAN where you trust the network, piping the send strea= m through netcat will remove ssh from the equation.

That's what we switched= to using once it became almost impossible to get the "none" ciph= er working with ssh on FreeBSD.

We use ssh to connect to the remote server= and enable a netcat listener on port X, then pipe the send through netcat = to the remote system on port X. That way it's logged and uses ssh for a= uthentication.

We easily saturate gigabit links between our ZFS systems us= ing netcat.



Cheers,
Freddie
=
Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miro= slav Lachman, <000.fbsd@quip.cz> wro= te:
On 22/02/2023 22:08, mike t= ancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with = chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could = pipe the stream through an encrypting program before piping to
netcat, then decrypt on the= recieving end.

$ zfs sen= d | crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but wo= rth a try.
$ man cr= ypt
=C2=A0 =C2=A0 The enigma utili= ty, also known as crypt is a very simple encryption
=C2=A0 =C2=A0 =C2=A0program, working o= n a =E2=80=9Csecret-key=E2=80=9D basis.=C2=A0 It operates as a filter, i.e.= ,
=C2=A0 =C2=A0= =C2=A0it encrypts or decrypts a stream of data from standard input, and wr= ites
=C2=A0 =C2= =A0 =C2=A0the result to standard output.=C2=A0 Since its operation is fully= symmetrical,
= =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the
= =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it.


--=20 Sent with https://mailf= ence.com =20 Secure and private email
--000000000000d0c06905f562d633-- From nobody Thu Feb 23 20:28:08 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PN4NB2ZgSz3sjjm for ; Thu, 23 Feb 2023 20:28:10 +0000 (UTC) (envelope-from spork@bway.net) Received: from smtp1.bway.net (smtp1.bway.net [216.220.96.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PN4NB0lYtz4Ps2 for ; Thu, 23 Feb 2023 20:28:10 +0000 (UTC) (envelope-from spork@bway.net) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (pool-173-70-201-95.nwrknj.fios.verizon.net [173.70.201.95]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: spork@bway.net) by smtp1.bway.net (Postfix) with ESMTPSA id F0FF2118A9; Thu, 23 Feb 2023 15:28:08 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bway.net; s=mail; t=1677184089; bh=PDw5Anwb6EQzR+umcXyYq1N3hXcJA/WxubqSvOPbF38=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=bLVKE313g8aCP6JKkYUHJMb3Ay63HK/ScTGkBWfm/PHJmZwTyDFT4UkSr19qoxzRM SkS9P6FSyEEJ4eJMr7uS81Vlk6tMNy5kMOliADdUOLScsxxomBlOAEc4/4O7D+0DiQ A+zB1oKDCI/Q0KK/v3B43KzBo/evLo4pGJyq/6/c= From: Charles Sprickman Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: speeding up zfs send | recv (update) Date: Thu, 23 Feb 2023 15:28:08 -0500 In-Reply-To: Cc: Sysadmin Lists , Freddie Cash , freebsd-fs To: Chris Watson References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> X-Mailer: Apple Mail (2.3696.120.41.1.2) X-Rspamd-Queue-Id: 4PN4NB0lYtz4Ps2 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8059, ipnet:216.220.96.0/19, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Feb 23, 2023, at 2:15 PM, Chris Watson wrote: >=20 > [Sorry miroslav, I hit send without checking the To: this was meant to = be public]=20 >=20 > I=E2=80=99m a bit late, but I mentioned this to someone on this thread = privately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t= been mentioned in this thread. I=E2=80=99ve seen everything from = VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted/massively = overly complex/hard to secure just to simply have a secure pipe for = doing ZFS send|recv.=20 >=20 > Simply configuring an spiped PtP pipe between A and B seems the = simplest, most secure, performant option here. At least considering all = the other options tossed out in this thread.=20 Does it scale across multiple cores? Charles >=20 > No one=E2=80=99s using spiped? O.o >=20 > Thoughts?=20 >=20 > Has anyone compared ssh to spiped regarding overhead and throughput in = this scenario? >=20 > Chris >=20 > On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists = > = wrote: >=20 > On Feb 22, 2023 at 1:43 PM, Freddie Cash > wrote: >>=20 >> [Sorry for top part, GMail sucks for replies.] >>=20 >> If this is a LAN or private WAN where you trust the network, piping = the send stream through netcat will remove ssh from the equation. >>=20 >> That's what we switched to using once it became almost impossible to = get the "none" cipher working with ssh on FreeBSD. >>=20 >> We use ssh to connect to the remote server and enable a netcat = listener on port X, then pipe the send through netcat to the remote = system on port X. That way it's logged and uses ssh for authentication. >>=20 >> We easily saturate gigabit links between our ZFS systems using = netcat. >>=20 >>=20 >>=20 >> Cheers, >> Freddie >>=20 >> Typos due to smartphone keyboard. >>=20 >> On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz = > wrote: >> On 22/02/2023 22:08, mike tancsa wrote: >> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> >> Interresting numbers. I think I am the only one who get best speed=20= >> >> with chacha20-poly1305@openssh.com = >> >> >> >> >> >> It seems the speed of SSH is limited by single core performance = which=20 >> >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU = E2160).=20 >> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >> >=20 >> > The CPU I have has >> > aesni0: on motherboard >> >=20 >> > which probably helps. >>=20 >> That explains it >> aesni0: No AES or SHA support. >>=20 >> >> I know there were some HPN patches to ssh, beside that is there = any=20 >> >> option I can try to use less CPU? >> >> >> >> I will play with cpuset to pin ssh on one core and everything else = on=20 >> >> the other core. >> >=20 >> > It looks like you are running into a CPU bottleneck TBH >>=20 >> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 >> without some tweaks on ssh I will not gain more speed :( >>=20 >> Thank you for your help! >>=20 >> Miroslav Lachman >>=20 >>=20 >=20 > You could pipe the stream through an encrypting program before piping = to > netcat, then decrypt on the recieving end. >=20 > $ zfs send | crypt | netcat ipaddr 2222 > $ netcat -vl 2222 | crypt | zfs recv >=20 > I don't know if zfs can handle that, but worth a try. >=20 > $ man crypt > The enigma utility, also known as crypt is a very simple = encryption > program, working on a =E2=80=9Csecret-key=E2=80=9D basis. It = operates as a filter, i.e., > it encrypts or decrypts a stream of data from standard input, and = writes > the result to standard output. Since its operation is fully = symmetrical, > feeding the encrypted data stream again through the engine (using = the > same secret key) will decrypt it. >=20 >=20 > -- Sent with https://mailfence.com Secure and = private email --Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Feb 23, 2023, at 2:15 PM, Chris Watson <bsdunix44@gmail.com>= wrote:

[Sorry miroslav, I hit send without checking = the To: this was meant to be public] 

I=E2=80=99m a bit late, but I mentioned this = to someone on this thread privately, I=E2=80=99m curious why = =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentioned in this thread. = I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be, = imo, grossly unwarranted/massively overly complex/hard to secure just to = simply have a secure pipe for doing ZFS send|recv. 

Simply configuring an spiped PtP pipe between = A and B seems the simplest, most secure, performant option here. At = least considering all the other options tossed out in this = thread. 

Does it scale across multiple cores?

Charles


No one=E2=80=99s using spiped? O.o

Thoughts? 

Has anyone compared ssh to spiped regarding = overhead and throughput in this scenario?

Chris

On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists = <sysadmin.lists@mailfence.com> wrote:

On Feb 22, = 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com> wrote:
[Sorry for top part, GMail sucks for replies.]

If this is a LAN or private WAN where you trust = the network, piping the send stream through netcat will remove ssh from = the equation.

That's what = we switched to using once it became almost impossible to get the "none" = cipher working with ssh on FreeBSD.

We use ssh to connect to the remote server and = enable a netcat listener on port X, then pipe the send through netcat to = the remote system on port X. That way it's logged and uses ssh for = authentication.

We easily = saturate gigabit links between our ZFS systems using netcat.



Cheers,
Freddie

Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav = Lachman, <000.fbsd@quip.cz> wrote:
On 22/02/2023 22:08, mike tancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best = speed
>> with chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance = which
>> is very poor on this machine (Intel(R) Pentium(R) Dual  = CPU E2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on = motherboard
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there = any
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything = else on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could = pipe the stream through an encrypting program before piping to
netcat, = then decrypt on the recieving end.

$ zfs send | crypt | netcat ipaddr 2222
$ netcat = -vl 2222 | crypt | zfs recv

I don't = know if zfs can handle that, but worth a try.

$ man crypt
    The enigma utility, also known as = crypt is a very simple encryption
  =    program, working on a =E2=80=9Csecret-key=E2=80=9D = basis.  It operates as a filter, i.e.,
  =    it encrypts or decrypts a stream of data from standard = input, and writes
     the result to standard = output.  Since its operation is fully symmetrical,
  =    feeding the encrypted data stream again through the engine = (using the
     same secret key) will decrypt = it.


--=20 Sent with https://mailfence.com =20 Secure and private email

= --Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615-- From nobody Thu Feb 23 21:02:04 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PN57Y4TtPz3sljt for ; Thu, 23 Feb 2023 21:02:17 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-yw1-x1131.google.com (mail-yw1-x1131.google.com [IPv6:2607:f8b0:4864:20::1131]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PN57Y44JHz3FyY for ; Thu, 23 Feb 2023 21:02:17 +0000 (UTC) (envelope-from fjwcash@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1131.google.com with SMTP id 00721157ae682-536c02eea4dso217087537b3.4 for ; Thu, 23 Feb 2023 13:02:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=sohsxX2y8Z7PSAgmLhhEfuPmQPSXhtFYlLhRm57KNlA=; b=LbpjNyPHyaVAqQcpQEb4Ql/VEKN77ooNkKfNHzg2wdbNghnkTSalkgntGCCvAr+iAE 5yWKXXezHoThdLsQ8ZH3kVSv3u2/AEm0hDfEcfOEV+YoHdD3m8hh913qqxV6cjMohVCQ +wqmqi1lcjjWKlIS9YU9m/UYo6fGgf+iiJy4az18yRXyOpg74AqBjRwQULx/WVGln1Kd InNa8ffblQDfOz9roPQbCuiPSbTEvWqeOZ+RKCXnyrTG3/Btjb8nUmo8Zq/on3/Hag4C 3CqOX7njmaI4dx6oCowzSUvbQ5LgJ1aDP/lv+7CCfiS5fhYiMv3uFgwfoF9aSOZXoJIZ NTew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sohsxX2y8Z7PSAgmLhhEfuPmQPSXhtFYlLhRm57KNlA=; b=BcaI5+2bLhZdT2lxOXbp0YUUo4kgsasLwaIHcf+WrXQc2Uav1+C03LJATSTb80oFl4 8SgJn0E9PKnvlwG/HrgPN8goGcoUhyyAU9XT2XORq6xyMKyEH1id0DuwUnnSCAcQKfep R4nr1C6BIGt2AO62hKJFJw8YYzxT7K0mEyedz0als7CUVFly030VytVoq0ej0gaS1Jyl eQ+AWqagYaRfateYtVkm8bEsy8bfl8mm9YU6fdZ/7ApkYBNmFK9VVVrGbfndlYqoDCKZ lmNTTv3FDLv5TSOJFyoikSKbUo8Scl4Itfh3YxJlkZs4D9el3rAlx7BVJOOyqDEsweZO OaFA== X-Gm-Message-State: AO0yUKXYzUvc/2uaMZwLsm+HFTBI1fn5583/8bWYgPnu5UKDdKJ+pxY5 8KxuwGYdGb4v/JiySjYrMACeiJl/7OuPSxcqW0o= X-Google-Smtp-Source: AK7set8PfdYKPOZRdIaU85+Ce9CX90cELcsgPr2CBuiXYbAoiMHo31uhax98YG5VHxKTxZEQahyZkIYglcKyV7KovXM= X-Received: by 2002:a25:9392:0:b0:997:bdfe:78c5 with SMTP id a18-20020a259392000000b00997bdfe78c5mr2756849ybm.6.1677186135972; Thu, 23 Feb 2023 13:02:15 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> In-Reply-To: From: Freddie Cash Date: Thu, 23 Feb 2023 13:02:04 -0800 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: Chris Watson Cc: Sysadmin Lists , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000db5f5d05f5645322" X-Rspamd-Queue-Id: 4PN57Y44JHz3FyY X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000db5f5d05f5645322 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Feb 23, 2023 at 11:15 AM Chris Watson wrote: > [Sorry miroslav, I hit send without checking the To: this was meant to be > public] > > I=E2=80=99m a bit late, but I mentioned this to someone on this thread pr= ivately, > I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been ment= ioned in this thread. I=E2=80=99ve seen > everything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly > unwarranted/massively overly complex/hard to secure just to simply have a > secure pipe for doing ZFS send|recv. > > Simply configuring an spiped PtP pipe between A and B seems the simplest, > most secure, performant option here. At least considering all the other > options tossed out in this thread. > > No one=E2=80=99s using spiped? O.o > > Thoughts? > > Has anyone compared ssh to spiped regarding overhead and throughput in > this scenario? > Interesting, never heard of spiped before this message. Looks neat. Not really useful for our specific use case (sending ZFS snapshots across private LAN and WAN links without encryption). But could be useful in place of SSH connections when sending across public links, depending on cipher suites used. --=20 Freddie Cash fjwcash@gmail.com --000000000000db5f5d05f5645322 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, Feb 23, 2023 at 11:15 AM Chris Wa= tson <bsdunix44@gmail.com>= wrote:
[Sorry miroslav, I = hit send without checking the To: this was meant to be public]=C2=A0
<= div dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(= 49,49,49);color:rgb(49,49,49)">
I= =E2=80=99m a bit late, but I mentioned this to someone on this thread priva= tely, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been = mentioned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s t= o nc. VPNs would be, imo, grossly unwarranted/massively overly complex/hard= to secure just to simply have a secure pipe for doing ZFS send|recv.=C2=A0=

Simply con= figuring an spiped PtP pipe between A and B seems the simplest, most secure= , performant option here. At least considering all the other options tossed= out in this thread.=C2=A0

No one=E2=80=99s using spiped? O.o
Thoughts?=C2=A0

Has anyone compared ssh to spi= ped regarding overhead and throughput in this scenario?

Interesting, never heard of spiped before this mes= sage.=C2=A0 Looks neat.

Not really useful for our = specific use case (sending ZFS snapshots across private LAN and WAN links w= ithout encryption).=C2=A0 But could be useful in place of SSH connections w= hen sending across public links, depending on cipher suites used.

--
Freddie = Cash
fjwcash@gmai= l.com
--000000000000db5f5d05f5645322-- From nobody Fri Feb 24 14:00:57 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PNWl22fDVz3tHGN for ; Fri, 24 Feb 2023 14:01:02 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "GEANT OV RSA CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PNWl04sbtz452H for ; Fri, 24 Feb 2023 14:01:00 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tu-dortmund.de header.s=unimail header.b=Rigzg7lT; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de; dmarc=none Received: from [192.168.111.100] (i5387852E.versanet.de [83.135.133.46]) (authenticated bits=0) by unimail.uni-dortmund.de (8.17.1.9/8.17.1) with ESMTPSA id 31OE0w8b000471 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Fri, 24 Feb 2023 15:00:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1677247258; bh=eOXNDOkFjuYoGex2xWSkCC/fQjFG9MEHYjW9zVcikqw=; h=Date:From:To:Cc:Subject; b=Rigzg7lToztPBpGx/gV11l/MP0TnZGx3ujrJ1Kd7/elPaNJgJsyLzrMRlR2bXAjeE EsFy6G1d6hqfsYgK+32sm+dZRZmZDuNP8SEvWUClpQ4FWvj4x7C+sOzTgNd5kj3YuV V9Gc9Q9RjgxOHG7wDKwof11ipRI+EKAQAX5roQDU= Message-ID: <45d84dae-0ca9-95ed-f6fd-8243797453ff@tu-dortmund.de> Date: Fri, 24 Feb 2023 15:00:57 +0100 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.2 Content-Language: de-DE-1901, en-US From: Alexander Lochmann To: Konstantin Belousov Cc: freebsd-fs@freebsd.org Subject: Understanding locking for buf Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------6qEDoPgBZ0QxwsEXCqCF6A4E" X-Spamd-Result: default: False [-7.30 / 15.00]; SIGNED_PGP(-2.00)[]; DWL_DNSWL_LOW(-1.00)[tu-dortmund.de:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; R_DKIM_ALLOW(-0.20)[tu-dortmund.de:s=unimail]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24]; RWL_MAILSPIKE_GOOD(-0.10)[129.217.128.51:from]; MIME_BASE64_TEXT(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[tu-dortmund.de]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; RCPT_COUNT_TWO(0.00)[2]; HAS_ATTACHMENT(0.00)[]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[tu-dortmund.de:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4PNWl04sbtz452H X-Spamd-Bar: ------- X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------6qEDoPgBZ0QxwsEXCqCF6A4E Content-Type: multipart/mixed; boundary="------------x0bPb0tgMNaaCONevmCsVlMN"; protected-headers="v1" From: Alexander Lochmann To: Konstantin Belousov Cc: freebsd-fs@freebsd.org Message-ID: <45d84dae-0ca9-95ed-f6fd-8243797453ff@tu-dortmund.de> Subject: Understanding locking for buf --------------x0bPb0tgMNaaCONevmCsVlMN Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 SGkgS29uc3RhbnRpbiENCg0KSSdtIHNvcnJ5LiBJJ20gc3RpbGwgc3RydWdnbGluZyB0byB1 bmRlcnN0YW5kIGxvY2tpbmcgb2Ygc3RydWN0IGJ1Zi4NCkFzIGZhciBhcyBJIGtub3csIGEg c3RydWN0IGJ1ZiBpcyAobW9zdGx5KSBwcm90ZWN0ZWQgYnkgYl9sb2NrLg0KWW91IGFscmVh ZHkgZXhwbGFpbmVkIHRoZSBjb25jZXB0IG9mIExLX0tFUk5QUk9DLg0KDQpJLCBob3dldmVy LCBoYXZlIGFuIGluc3RhbmNlIG9mIHN0cnVjdCBidWYuIFRoYXQgcGFydGljdWxhciBvbmUg aXMgDQptb3N0bHkgYWNjZXNzZWQgZnJvbSBqdXN0IG9uZSBjb250ZXh0IChha2EgdGhyZWFk KSAtLSBhc3N1bWluZyANCnN5bmNocm9ub3VzIElPLg0KVGhlIG90aGVyLCB1bnByb3RlY3Rl ZCBhY2Nlc3NlcyBjb21lIGZyb20ganVzdCBvbmUgb3RoZXIgY29udGV4dC4gVGhvc2UgDQph Y2Nlc3NlcyBvcmlnaW5hdGUgZnJvbSAnZ192ZnNfZG9uZScgWzFdLg0KV2h5IGRvbid0IHlv dSByZWxlYXNlIGJfbG9jayB3aGVuIGJ1ZiBnb2VzIGZ1cnRoZXIgZG93biB0aGUgaW8gc3Rh Y2ssIA0KYW5kIGFjcXVpcmUgaXQgYWdhaW4gaW4gZ192ZnNfZG9uZT8NClRoaXMgd2F5IHRo ZSBjb250ZXh0IG9mIGdfdmZzX2RvbmUgd291bGQgb3duIHRoZSBiX2xvY2suDQoNClZpZXdp bmcgaXQgZnJvbSBhIGRpZmZlcmVudCBhbmdsZTogQXJlIGFjY2Vzc2VzIGluIGdfdmZzX2Rv bmUgc2FmZSANCmJlY2F1c2UgdGhlIGJ1ZiBpbnN0YW5jZSBpcyBhbHJlYWR5IGxvY2tlZCBm cm9tIGEgZ2xvYmFsIHBlcnNwZWN0aXZlPw0KSGVuY2UsIG90aGVyIGNvZGUgcGF0aHMgd291 bGQgYmxvY2sgb24gQlVGX0xPQ0soKS4NCg0KUmVnYXJkcywNCkFsZXgNCg0KWzFdIA0KaHR0 cHM6Ly9pcmlzLmNzLnR1LWRvcnRtdW5kLmRlL2ZyZWVic2QtbG9ja2RvYy9sYXRlc3Qvc291 cmNlL3N5cy9nZW9tL2dlb21fdmZzLmMjTDk2DQotLSANClRlY2huaXNjaGUgVW5pdmVyc2l0 w6R0IERvcnRtdW5kDQpDb21wdXRlciBTY2llbmNlIFhJSSAtIFN5c3RlbSBTb2Z0d2FyZSBH cm91cA0KQWxleGFuZGVyIExvY2htYW5uICAgICAgICAgICAgICAgIFBHUCBrZXk6IDB4QkMz RUY2RkQNCk90dG8tSGFobi1TdHIuIDE2ICAgICAgICAgICAgICAgICBwaG9uZTogICs0OS4y MzEuNzU1NjE0MQ0KRC00NDIyNyBEb3J0bXVuZCAgICAgICAgICAgICAgICAgIGZheDogICAg KzQ5LjIzMS43NTU2MTE2DQpodHRwczovL3N5cy5jcy50dS1kb3J0bXVuZC5kZS9hbA0K --------------x0bPb0tgMNaaCONevmCsVlMN-- --------------6qEDoPgBZ0QxwsEXCqCF6A4E Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEElhZsUHzVP0dbkjCRWT7tBbw+9v0FAmP4wxkFAwAAAAAACgkQWT7tBbw+9v1F jA/9ERO8YXoRVtmGYeA+RVxTZSVN4sXU5mj32FF6DbtOVOQqXMQCSwM12cUsMBJOY31VMaduCkCS xJ1WmD6c3ZcHjnsfewjJOnnnH1pQdawvA3cTPBV99zz9PyXaBn78zWwfcBRXPRZpcUdHTLNYD3Ji IFsXUvR2VsF5+0XWwjdcRRCc44OgbmcgnV0HuPpKLw4q/urAoQfFyUW9ut38qFC5vnUNVQW3BuwW nt8cMjpTP4rQ1+oaxxkHBmcP6HE9v2tiiiuBAeWxxFFYfvRA/Oy1iP7Ms+hazD2offGMv/3u12WR 3EDlWwwxvfMQkCnZhGiolgu9wIcKH1qp+9Gj61vyEbnOonQR3EadFQqBmeIWg5IFOQWF4k8p1Ymo 2k62AABE3bvKiIietPEvXa5rhqS6rjm1m2PWEUr/33O/USdBIVs57mCVhkM5KKPaccjASuP6pV/u Irgocy9RD7IcIEvJI86NjQAjmNDoQkigBGIflzJ/l1qc0KmBPmgIriSDGFmYLKLTjM7G0EBKXyrt 8hETMNq2XnQErpBDMwCb1bzV3a8qguTujryWRGxxLbkpe0VgTRdK555H9IDjcjg3Ls0m660y8338 vD7P1P1JHkixjt2OSgBpGgUXNo+JFFkxzPjLsC7hmdWoOMh0XA5dvPkx5w29u1+6nviCju8Oy6tg WTI= =4WJy -----END PGP SIGNATURE----- --------------6qEDoPgBZ0QxwsEXCqCF6A4E-- From nobody Fri Feb 24 20:41:48 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PNhdl62qbz3tvST for ; Fri, 24 Feb 2023 20:42:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PNhdl2SX5z3r9k for ; Fri, 24 Feb 2023 20:42:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 31OKfmmW099820 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 24 Feb 2023 22:41:51 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 31OKfmmW099820 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 31OKfmxE099819; Fri, 24 Feb 2023 22:41:48 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 24 Feb 2023 22:41:48 +0200 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs@freebsd.org Subject: Re: Understanding locking for buf Message-ID: References: <45d84dae-0ca9-95ed-f6fd-8243797453ff@tu-dortmund.de> List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <45d84dae-0ca9-95ed-f6fd-8243797453ff@tu-dortmund.de> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4PNhdl2SX5z3r9k X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Fri, Feb 24, 2023 at 03:00:57PM +0100, Alexander Lochmann wrote: > Hi Konstantin! > > I'm sorry. I'm still struggling to understand locking of struct buf. > As far as I know, a struct buf is (mostly) protected by b_lock. > You already explained the concept of LK_KERNPROC. > > I, however, have an instance of struct buf. That particular one is mostly > accessed from just one context (aka thread) -- assuming synchronous IO. > The other, unprotected accesses come from just one other context. Those > accesses originate from 'g_vfs_done' [1]. > Why don't you release b_lock when buf goes further down the io stack, and > acquire it again in g_vfs_done? > This way the context of g_vfs_done would own the b_lock. Unlocking the buffer means that other thread might take a lock. This gives the new owner a permission to modify the buffer state and data, causing the io operation in flight to corrupt data. Or the buffer might be reclaimed with brelse(), corrupting the kernel state. > > Viewing it from a different angle: Are accesses in g_vfs_done safe because > the buf instance is already locked from a global perspective? > Hence, other code paths would block on BUF_LOCK(). geom completion code is the only code that allowed to touch the buffer after the ownership was relinguished. I believe I already tell that to you: consider the buffer lock after LK_KERNPROC as a semaphore and not lock. From nobody Fri Feb 24 22:45:08 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PNlMm6Y46z3v2Y7 for ; Fri, 24 Feb 2023 22:45:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PNlMm5XcMz47L5 for ; Fri, 24 Feb 2023 22:45:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677278708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=it2+yAGvRVFA/IFPw+VZ5OW1vlsz9Dm7yDPepR6itJs=; b=bHH8VWK0HUYcQQN7MoMgi69nBJy/59Mfj38fiqZRLPmpxGZpy/9QhrECjtgknpK0ngeS3r LSD2fx84evpeqZL43bvMe1RaudFwQzzB2SxkAllHcn0RchiVwY8qtjImlNHgoFNIj1mGCG l9XU5uUEmts5WzhpfAx1dZraD0NIhvOnpy/WjP8KgDIlScbcARkYtqhX4th/eg4GIm8gWd 6W2NgLu3Gi13LcemVTxmBpPvdzJ1RhCSw6E2BgwVv0S1QUHrXnEUYhfhxUgjs6mSCwdUGy OgDdfA1w9B3NTd4P+pvolZBbr6lTb2Gl349Sv92q6xF7ajHUfBY4xLzrAEPbQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677278708; a=rsa-sha256; cv=none; b=kPChiDk+3LqtfI2DwGb2gI730w4Qx/M2lw00UlMlytOLhSXlmEyFKXs72XHoI3nb5V2ZGZ +1jWM++gvtL/1J2sdFksSiMuJ0BP83rWmeq4xGKab62KLzh1DEiQsDi5sIQ7BTnJa75vsZ zijtLWJgTt0W+n92xENBzWsBNy8IeqHaMlWxDXPSQrj6u58SHQi39wTwidn2sy5aL3r2Tv 4bpFQ48Mx1IzCLRoHDK+T/ozjTmxidTqJl4tLPpYW2UHTf9ovUVoj5dXeQJJx2G0hZTXLf g/RBZTdSCGkGZRC55deGB9XwXMXBzZF41yK6xqlp9PXtTOTxc/Wlp0lyDOrnGw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PNlMm4XR2zhBq for ; Fri, 24 Feb 2023 22:45:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31OMj8Jg092532 for ; Fri, 24 Feb 2023 22:45:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31OMj8Yb092531 for fs@FreeBSD.org; Fri, 24 Feb 2023 22:45:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 267028] kernel panics when booting with ZFS and amdgpu.ko (both are required) Date: Fri, 24 Feb 2023 22:45:08 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: george@m5p.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? maintainer-feedback? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D267028 --- Comment #78 from George Mitchell --- Although I have not yet managed to test this without ZFS, I have established that with zfs_load=3D"YES" but without "vboxnet_enable=3D"YES"" in /etc/rc.= conf (zfs.ko and vboxnetflt.ko seeming to be the two modules with which amdgpu.ko has, um, personality conflicts), I can now boot up without crashing (so far= ).=20 Does anyone have any idea what zfs.ko and vboxnetflt.ko do that other modul= es don't do? --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From nobody Fri Feb 24 22:45:15 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PNlN15GWnz3v2WG for ; Fri, 24 Feb 2023 22:45:21 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from wilbur.contactoffice.com (wilbur.contactoffice.com [212.3.242.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PNlN043mwz47vX for ; Fri, 24 Feb 2023 22:45:20 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mailfence.com header.s=20210208-e7xh header.b=WboHlsBo; spf=pass (mx1.freebsd.org: domain of sysadmin.lists@mailfence.com designates 212.3.242.68 as permitted sender) smtp.mailfrom=sysadmin.lists@mailfence.com; dmarc=pass (policy=quarantine) header.from=mailfence.com Received: from fidget.co-bxl (fidget.co-bxl [10.2.0.33]) by wilbur.contactoffice.com (Postfix) with ESMTP id 2230C1A1B; Fri, 24 Feb 2023 23:45:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1677278718; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Cc:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=15015; bh=EC0i/VAxrIYv/ZhW5nn2AYVOIrEMShMKTWPgmQ9oWR8=; b=WboHlsBo5vwT9pqhVhQ1dzjRguPNk3cSes5mSI0wMqudIJXTN8ybwyP8hk7dOgMV if96HidVa3MwONroY7P7v07bTUpF1u02vSkBo/QSKHzWXkX5XqOjsU7CPAAu1NK+SK9 2JWRyBYOFqYrP9+G2yQ7qMCMkpuMI/OoHlQ7nc+ok7vDCyjSShUhXKt9tyzoW16gyXr T7MO6Ug77eXvmecFaMWNGTzcWlYN/Wm3QkdkjGuNtzkUP9J/Mf4y2iKmLJEeuXcrNgE Vunqj4KMSOXpMqhynvX9aZvIjXE5EfwsBGCn0CaAEyWBcbHn+SLiG/4FViKmVO49X+M Sq9vAO1cPQ== Date: Fri, 24 Feb 2023 23:45:15 +0100 (CET) From: Sysadmin Lists To: freebsd-fs Cc: Chris Watson Message-ID: <1290947438.348129.1677278715319@fidget.co-bxl> In-Reply-To: References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> Subject: Re: speeding up zfs send | recv (update) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_348126_1685684535.1677278715318" X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Spamd-Result: default: False [-4.08 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[mailfence.com,quarantine]; R_DKIM_ALLOW(-0.20)[mailfence.com:s=20210208-e7xh]; R_SPF_ALLOW(-0.20)[+ip4:212.3.242.64/26]; RCVD_IN_DNSWL_LOW(-0.10)[212.3.242.68:from]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[mailfence.com:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:10753, ipnet:212.3.242.64/26, country:US]; FREEMAIL_CC(0.00)[gmail.com] X-Rspamd-Queue-Id: 4PNlN043mwz47vX X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N ------=_Part_348126_1685684535.1677278715318 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Feb 23, 2023 at 11:15 AM, Chris Watson wrote: [Sorry miroslav, I hit send without checking the To: this was meant to be p= ublic]=C2=A0 I=E2=80=99m a bit late, but I mentioned this to someone on this thread priv= ately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been= mentioned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s = to nc. VPNs would be, imo, grossly unwarranted/massively overly complex/har= d to secure just to simply have a secure pipe for doing ZFS send|recv.=C2= =A0 Simply configuring an spiped PtP pipe between A and B seems the simplest, m= ost secure, performant option here. At least considering all the other opti= ons tossed out in this thread.=C2=A0 No one=E2=80=99s using spiped? O.o Thoughts?=C2=A0 Has anyone compared ssh to spiped regarding overhead and throughput in this= scenario? Chris On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists wrote: On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the sen= d stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get th= e "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on = port X, then pipe the send through netcat to the remote system on port X. T= hat way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrot= e: On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed=20 >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which=20 >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E2160).= =20 >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >=20 > The CPU I have has > aesni0: on motherboard >=20 > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any=20 >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on=20 >> the other core. >=20 > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman You could pipe the stream through an encrypting program before piping to netcat, then decrypt on the recieving end. $ zfs send | crypt | netcat ipaddr 2222 $ netcat -vl 2222 | crypt | zfs recv I don't know if zfs can handle that, but worth a try. $ man crypt =C2=A0 =C2=A0 The enigma utility, also known as crypt is a very simple encr= yption =C2=A0 =C2=A0 =C2=A0program, working on a =E2=80=9Csecret-key=E2=80=9D basi= s.=C2=A0 It operates as a filter, i.e., =C2=A0 =C2=A0 =C2=A0it encrypts or decrypts a stream of data from standard = input, and writes =C2=A0 =C2=A0 =C2=A0the result to standard output.=C2=A0 Since its operatio= n is fully symmetrical, =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it. -- Sent with https://mailfence.com Secure and private email I've used it before, but forgot about it. But it's not part of base, and th= ere are tools in base which together perform a similar task, so that probably explains why many people haven't heard about it or forgot they had. Most everyone has at some point needed to transfer a couple files to a loca= l machine with a LAN connection but borked authentication services. In steps nc and optionally crypt or openssl to encrypt the data. Simple. -- Sent with https://mailfence.com Secure and private email ------=_Part_348126_1685684535.1677278715318 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=
On Feb 23, 2023 at 11:15 AM, Chris Watson <bsdunix44@gmail.com> = wrote:
">
[Sorry miroslav, I hit send without checking the To: this was mea= nt to be public] 

I=E2=80=99m a bit late, but I mentioned this to = someone on this thread privately, I=E2=80=99m curious why =E2=80=98spiped= =E2=80=99 hasn=E2=80=99t been mentioned in this thread. I=E2=80=99ve seen e= verything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted= /massively overly complex/hard to secure just to simply have a secure pipe = for doing ZFS send|recv. 

Simply configuring an spiped PtP pipe between A and B s= eems the simplest, most secure, performant option here. At least considerin= g all the other options tossed out in this thread. 

No one=E2=80=99s using spiped= ? O.o

Thoug= hts? 

= Has anyone compared ssh to spiped regarding overhead and throughput in this= scenario?

Chris

On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists <sysadmin.lists@mailfence.com> wrot= e:

On Feb 22, 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com> wrote:
[Sorry for top part, GMail suc= ks for replies.]

If this is a LAN or private WAN where you trust the network, pi= ping the send stream through netcat will remove ssh from the equation.

That's wh= at we switched to using once it became almost impossible to get the "none" = cipher working with ssh on FreeBSD.

We use ssh to connect to the remote se= rver and enable a netcat listener on port X, then pipe the send through net= cat to the remote system on port X. That way it's logged and uses ssh for a= uthentication.

We easily saturate gigabit links between our ZFS systems us= ing netcat.



Cheers,
Freddie
=
Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miro= slav Lachman, <000.fbsd@quip.cz> wro= te:
On 22/02/2023 22:08, mike t= ancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with = chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could = pipe the stream through an encrypting program before piping to
netcat, then decrypt on the= recieving end.

$ zfs sen= d | crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but worth = a try.
$ man crypt<= /div>
    The enigma utility, a= lso known as crypt is a very simple encryption
     program, working on a = =E2=80=9Csecret-key=E2=80=9D basis.  It operates as a filter, i.e.,
    &nb= sp;it encrypts or decrypts a stream of data from standard input, and writes=
    =  the result to standard output.  Since its operation is fully sym= metrical,
 = ;    feeding the encrypted data stream again through the engine (= using the
 = ;    same secret key) will decrypt it.


--=20 Sent with https://mailf= ence.com =20 Secure and private email

I've used it before, but forgot about it. But it's not part of base,= and there
are tools in base which together perform a similar tas= k, so that probably
explains why many people haven't heard about = it or forgot they had.

Most everyone has at some p= oint needed to transfer a couple files to a local
machine with a = LAN connection but borked authentication services. In steps
nc an= d optionally crypt or openssl to encrypt the data. Simple.


--=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_348126_1685684535.1677278715318-- From nobody Fri Feb 24 22:46:20 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PNlP90Pv1z3v2YD for ; Fri, 24 Feb 2023 22:46:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PNlP86Wp2z49rj for ; Fri, 24 Feb 2023 22:46:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677278780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0tsk5nZ6xntT+EarwzMRUrCJVEZtK5I81lbPxJMy1Zs=; b=htkTIZyKGY6JEuFw2ALMmD95JcBn6arBFHn+KhYIwNT1qRD27JuVe69JlGDMLFeIdEGfwi MssfprZ2yV6AbdLq0d5cuBM2ee7OkoITtj/VRzBCL++JA8WbIjSqwSw+Ho2q8K2ndpTtzU AKv/uLIDQdKTIbSsoNT+3o4Dmp8idk2f5Xs//xPSa0F2mzISIqKpGieI2cOuuiGxBY+rGe uXEIq0B8x93ZmPmaVhUntgYG2ZLeA5cMho9Go6aeqQc35R3NTnNkSeOUNX5T0oiY2smoXD 2x2uUv3Prkwnu1cRwSYsGuVsXDulgip1ccbCAkwUYWSQnLR24LJMBhw7L+SSjg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677278780; a=rsa-sha256; cv=none; b=ZWjoA9xyeneNHbW661AXRR+GpXGL2Ikla/xjPKKU8lMTlD/70hqvPnE4MV54vZ1WRmrSP7 sP3OyqxGo0GrhixXE3lctgb1p/87XN6odMMRpi7pkwUkfUgZWGfLDzlcC2lo0/CXByecUn ROYUXwuaRrKNrJkgqaDXT6k7rUQAGzvMjZ9m8kTvUyJb5nXKHuJLwHZCGGt9ukBp3toVrV NXyDCG2Jmp7BqRkVHJtQDSMKLkgF4zBcNM7G4nYpRlxoTAot22oETxv8qGI6VfOS/4igh5 IhKQ9rTAiNMYmJ9kQvGqgte/CpsNU2TzgeRGDsLb27YbP0JlJ0noA2iTW1NZ3A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PNlP85cn8zgb7 for ; Fri, 24 Feb 2023 22:46:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31OMkKKt092982 for ; Fri, 24 Feb 2023 22:46:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31OMkK9g092981 for fs@FreeBSD.org; Fri, 24 Feb 2023 22:46:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 267028] kernel panics when booting with ZFS and amdgpu.ko (both are required) Date: Fri, 24 Feb 2023 22:46:20 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: george@m5p.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? maintainer-feedback? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D267028 --- Comment #79 from George Mitchell --- I omitted an important phrase. It should have said, "with with zfs_load=3D= "YES" in /boot/loader.conf ..." --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From nobody Sat Feb 25 14:50:07 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PP8nD1J7Sz3tG9j for ; Sat, 25 Feb 2023 14:50:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PP8nD0GqWz3Cwp for ; Sat, 25 Feb 2023 14:50:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677336608; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n3Upjnjc1gViXs7u036Tn4RgsA/OaC+Rx+gkw/GA+ho=; b=fiySXfBUAc/SvSvNqOt4W3S9F+Olk0DzXDgwiTqmlvZF6gw9KQR0SnPycjXnmXbgcgbwhu KNY+/wLJSaqHpzSaXw4OQway0ZG6SXfcObbuljkjfr3Xcf97cb5u3dg+1XWUnY/x89gdpQ itVtFNoxjuAvEKIIXg+wqeG75KvcgicMmCLvNn564rcxfRcLsEohBsiBkfP5JX6sx5VeZj 7Mt26v7X70rMvjTuHBnw9A8PjcCECkxuJ8UhfwcmfNTLckqE24Us0M36wSLQzN+YFqQeXc 6e3JzbfdbgJut9GKUK0stKsH+6sKhBeXFbBErL13dpAAl6hJQK7GRrmI8sDMGA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677336608; a=rsa-sha256; cv=none; b=AuL6OghYDP7vWFwHrVWtx4cxl8jkQd6Fl5OesiNvabW80eP0EcF2rk6CrIaCY3cbDFSPGp TZjkJDeh3GgU0efnMvutYRJFUNxqYON4E9e/s4UHyJ7F5UgSMM/5TGkdF2j9bMPOyhdGQ5 T5kpph0V0mJha4C6sgD8+tKxwkUmFaVo6NB/mMyQoc4OXGEyHpphJQN8d1yu2UFJKg+B6d mhLrZT49ttWUG8XHz7n1Lc6DR5dUU0HtptNsKybklZal2A7sZ7jrAw7XD9DkkqmiuQkx1P C20/xarlLnZi1z8mFovUB7yoWRLS6z93ip9peb8wqU+gfQcopFBWWDd7Sv5wqg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PP8nC6MJlz17Pl for ; Sat, 25 Feb 2023 14:50:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 31PEo7Ua071036 for ; Sat, 25 Feb 2023 14:50:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 31PEo7kL071035 for fs@FreeBSD.org; Sat, 25 Feb 2023 14:50:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 267028] kernel panics when booting with ZFS and amdgpu.ko (both are required) Date: Sat, 25 Feb 2023 14:50:07 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? maintainer-feedback? X-Bugzilla-Changed-Fields: flagtypes.name cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D267028 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|maintainer-feedback? |maintainer-feedback?(vbox@F | |reeBSD.org) CC| |vbox@FreeBSD.org --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.=