From owner-dev-commits-src-main@freebsd.org Thu Jan 28 21:22:56 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5192A4EDC92; Thu, 28 Jan 2021 21:22:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DRYNJ1v35z4jG3; Thu, 28 Jan 2021 21:22:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 334CE16E96; Thu, 28 Jan 2021 21:22:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 10SLMugm057330; Thu, 28 Jan 2021 21:22:56 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 10SLMuob057329; Thu, 28 Jan 2021 21:22:56 GMT (envelope-from git) Date: Thu, 28 Jan 2021 21:22:56 GMT Message-Id: <202101282122.10SLMuob057329@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alexander Motin Subject: git: b75168ed24ca - main - Make software iSCSI more configurable. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b75168ed24ca74f65929e5c57d4fed5f0ab08f2a Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 21:22:56 -0000 The branch main has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=b75168ed24ca74f65929e5c57d4fed5f0ab08f2a commit b75168ed24ca74f65929e5c57d4fed5f0ab08f2a Author: Alexander Motin AuthorDate: 2021-01-28 20:53:49 +0000 Commit: Alexander Motin CommitDate: 2021-01-28 21:22:45 +0000 Make software iSCSI more configurable. Move software iSCSI tunables/sysctls into kern.icl.soft subtree. Replace several hardcoded length constants there with variables. While there, stretch the limits to better match Linux' open-iscsi and our own initiator with new MAXPHYS of 1MB. Our CTL target is also optimized for up to 1MB I/Os, so there is also a match now. For Windows 10 and VMware 6.7 initiators at default settings it should make no change, since previous limits were sufficient there. Tests of QD1 1MB writes from FreeBSD over 10GigE link show throughput increase by 29% on idle connection and 132% with concurrent QD8 reads. MFC after: 3 days Sponsored by: iXsystems, Inc. --- sys/dev/iscsi/icl.h | 2 -- sys/dev/iscsi/icl_soft.c | 45 +++++++++++++++++++++++++-------------------- 2 files changed, 25 insertions(+), 22 deletions(-) diff --git a/sys/dev/iscsi/icl.h b/sys/dev/iscsi/icl.h index 06108802766d..0b897a50302a 100644 --- a/sys/dev/iscsi/icl.h +++ b/sys/dev/iscsi/icl.h @@ -89,8 +89,6 @@ struct icl_pdu { #define ICL_CONN_STATE_DATA 4 #define ICL_CONN_STATE_DATA_DIGEST 5 -#define ICL_MAX_DATA_SEGMENT_LENGTH (128 * 1024) - #define ICL_NOCOPY (1 << 30) struct icl_conn { diff --git a/sys/dev/iscsi/icl_soft.c b/sys/dev/iscsi/icl_soft.c index 41d20b4abf69..a8986b3d4253 100644 --- a/sys/dev/iscsi/icl_soft.c +++ b/sys/dev/iscsi/icl_soft.c @@ -72,18 +72,29 @@ struct icl_soft_pdu { int error; }; +SYSCTL_NODE(_kern_icl, OID_AUTO, soft, CTLFLAG_RD | CTLFLAG_MPSAFE, 0, + "Software iSCSI"); static int coalesce = 1; -SYSCTL_INT(_kern_icl, OID_AUTO, coalesce, CTLFLAG_RWTUN, +SYSCTL_INT(_kern_icl_soft, OID_AUTO, coalesce, CTLFLAG_RWTUN, &coalesce, 0, "Try to coalesce PDUs before sending"); -static int partial_receive_len = 128 * 1024; -SYSCTL_INT(_kern_icl, OID_AUTO, partial_receive_len, CTLFLAG_RWTUN, +static int partial_receive_len = 256 * 1024; +SYSCTL_INT(_kern_icl_soft, OID_AUTO, partial_receive_len, CTLFLAG_RWTUN, &partial_receive_len, 0, "Minimum read size for partially received " "data segment"); -static int sendspace = 1048576; -SYSCTL_INT(_kern_icl, OID_AUTO, sendspace, CTLFLAG_RWTUN, +static int max_data_segment_length = 256 * 1024; +SYSCTL_INT(_kern_icl_soft, OID_AUTO, max_data_segment_length, CTLFLAG_RWTUN, + &max_data_segment_length, 0, "Maximum data segment length"); +static int first_burst_length = 1024 * 1024; +SYSCTL_INT(_kern_icl_soft, OID_AUTO, first_burst_length, CTLFLAG_RWTUN, + &first_burst_length, 0, "First burst length"); +static int max_burst_length = 1024 * 1024; +SYSCTL_INT(_kern_icl_soft, OID_AUTO, max_burst_length, CTLFLAG_RWTUN, + &max_burst_length, 0, "Maximum burst length"); +static int sendspace = 1536 * 1024; +SYSCTL_INT(_kern_icl_soft, OID_AUTO, sendspace, CTLFLAG_RWTUN, &sendspace, 0, "Default send socket buffer size"); -static int recvspace = 1048576; -SYSCTL_INT(_kern_icl, OID_AUTO, recvspace, CTLFLAG_RWTUN, +static int recvspace = 1536 * 1024; +SYSCTL_INT(_kern_icl_soft, OID_AUTO, recvspace, CTLFLAG_RWTUN, &recvspace, 0, "Default receive socket buffer size"); static MALLOC_DEFINE(M_ICL_SOFT, "icl_soft", "iSCSI software backend"); @@ -663,10 +674,8 @@ icl_conn_receive_pdu(struct icl_conn *ic, size_t *availablep) len = icl_pdu_data_segment_length(request); if (len > ic->ic_max_data_segment_length) { ICL_WARN("received data segment " - "length %zd is larger than negotiated " - "MaxDataSegmentLength %zd; " - "dropping connection", - len, ic->ic_max_data_segment_length); + "length %zd is larger than negotiated; " + "dropping connection", len); error = EINVAL; break; } @@ -1230,7 +1239,7 @@ icl_soft_new_conn(const char *name, struct mtx *lock) #ifdef DIAGNOSTIC refcount_init(&ic->ic_outstanding_pdus, 0); #endif - ic->ic_max_data_segment_length = ICL_MAX_DATA_SEGMENT_LENGTH; + ic->ic_max_data_segment_length = max_data_segment_length; ic->ic_name = name; ic->ic_offload = "None"; ic->ic_unmapped = false; @@ -1280,10 +1289,6 @@ icl_conn_start(struct icl_conn *ic) * For sendspace, this is required because the current code cannot * send a PDU in pieces; thus, the minimum buffer size is equal * to the maximum PDU size. "+4" is to account for possible padding. - * - * What we should actually do here is to use autoscaling, but set - * some minimal buffer size to "minspace". I don't know a way to do - * that, though. */ minspace = sizeof(struct iscsi_bhs) + ic->ic_max_data_segment_length + ISCSI_HEADER_DIGEST_SIZE + ISCSI_DATA_DIGEST_SIZE + 4; @@ -1520,10 +1525,10 @@ static int icl_soft_limits(struct icl_drv_limits *idl) { - idl->idl_max_recv_data_segment_length = 128 * 1024; - idl->idl_max_send_data_segment_length = 128 * 1024; - idl->idl_max_burst_length = 262144; - idl->idl_first_burst_length = idl->idl_max_burst_length; + idl->idl_max_recv_data_segment_length = max_data_segment_length; + idl->idl_max_send_data_segment_length = max_data_segment_length; + idl->idl_max_burst_length = max_burst_length; + idl->idl_first_burst_length = first_burst_length; return (0); }