From nobody Thu Feb 1 09:50:30 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQZ064jKNz58HPW for ; Thu, 1 Feb 2024 09:50:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQZ063JMVz44wf for ; Thu, 1 Feb 2024 09:50:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706781030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZOt9kBonEsLdWZ7JjFl5KimjGRiHf9E491zbhU6+MnA=; b=SQb77elc+5aDnldB40O968/dABBcB3ipc5+ohwp01Dh5K7Mr2yWZSlDp2NDqtu+KgBRwVu eOD4OfDTjp6X58IOn5qp/Pqqdh3mbWNbtV+42uwOMzfj5Zh//pLMSmreHCUIq0Dh7GP/rk AosTooZYgZoV7LlO2iCU8ZKRLvkN4bx00PjPzCm5g1XtFvekI5NZd8RMqCZ0ZkgQXQ6TXs 6a3fo35h9mVREZ997r2vdudb6HPDlptB8f3IPO1A1MDqQgmsAaV1BeZgNuM5gnCdoUsM1z nP0cudFeqjtZM+ZJUEL1ef63Yx5AcTHSJBBj4BGDJn5FilSsHUeTDRQKk2omQw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706781030; a=rsa-sha256; cv=none; b=yE17PXDUqP6UhARUABH+4dOmT0T7maPG40ebJbnVeyoF1tJVAEbrdfrL3C69S0mPcT9ZEM 42Qptpzxqs7jBI83gB/0ZDmSEEZj0K1uWpP7RIaOfHSNQuJNf84egCI0hm+Gtqg+Xwxic4 b+DYUQ/Mdtifji53vgf1Ad+omp+mBfonYISkUhc5TbSlXlPNjoeW20pDocWmMU2dfEABhh /um8Zlki7xEAqmqVlrYybPTg8GNAantP0Dn9vh/lzoYwfX4++Fo3iDnQnRaCHcYXswFD67 pjBYrNT1UOMeJePHUb1qdngnHUWkenvNeEwSbvhEKYiNJRxMgd/6zKB+j8DkFg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TQZ062M91zxfK for ; Thu, 1 Feb 2024 09:50:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 4119oULu008821 for ; Thu, 1 Feb 2024 09:50:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 4119oUha008818 for transport@FreeBSD.org; Thu, 1 Feb 2024 09:50:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Thu, 01 Feb 2024 09:50:30 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |transport@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 1 10:22:22 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQZht591bz58L64 for ; Thu, 1 Feb 2024 10:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQZht487Xz4705 for ; Thu, 1 Feb 2024 10:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706782942; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QppuuALZq8akK+phiwWIOG7SyVunhroSmU0H2uymONY=; b=Sp7lxEezCsI2rnWI2RK3nDUp+ONZ8if2XJn31GsXEp191FsRbd3u140CqWlsJ1NHnhF38I HrucYSMihEWUpmDBEurQYoEtzLoEGhgAT4sspIpKBlxPWBkOS3CtReMcFggh68oyF4AdoB prgvyvnUYsg1Bl8ZFtNcYB1zmQvzfOXpAuFfKtFJw7BbwjAqEw2qrISgxBH8ik3d0tu5g8 zb3rbk5NIMPvW+2bhF1anxf47AXzSydmg0Un/epuod6L+reOrD2qSXYxaLLEGcAh0pF1LX OvdvLf7uz7obIJJ1avBif57v//8xYHioYcaLXlnK+yGNzySdlTvigZdmCqNCCg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706782942; a=rsa-sha256; cv=none; b=BVjdRyZU9XNJ5kGJ+TeXRZ5rwaY+Y0KJjvIXFVE4db+jSbn/FpETtr9eD/VSVh51VNj5hX EmBhLXAnWaIUtFl7yZRAFATfC31z3ZgH8Aao8fZPcLSuEfO9NqVsqLCYIghCouJW2kIiwY brLn9HhsQyWcbfE7L4U8yX9oBkUUJuXm4uagFZZRv5lL5mthuow6znHgn1qD16YtbHrjOS 1HPHqQioiuIcOcqJJLOPvnCcHqGohVn18uW8fIhwm7RRNasCLhmrBsExarPPEGqLIyxdWT Lu3uuK95MBPLbQIMIWztxWS6wTOPzNu2yGh0mCKvGatqYyQdeT0F8MfBmMBwWw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TQZht3DCXzyPP for ; Thu, 1 Feb 2024 10:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 411AMMIf087790 for ; Thu, 1 Feb 2024 10:22:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 411AMMbx087789 for transport@FreeBSD.org; Thu, 1 Feb 2024 10:22:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Thu, 01 Feb 2024 10:22:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: tuexen@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 --- Comment #1 from Michael Tuexen --- How can we access the core? How do you reproduce it? --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 1 10:30:40 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQZtS6BKyz58LfW for ; Thu, 1 Feb 2024 10:30:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQZtS56G6z47S7 for ; Thu, 1 Feb 2024 10:30:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706783440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+D6Ff++naTj2QIpyE59OXSXBIZvBGKLHAtuAHQO7e8E=; b=Oe+uwamgDI8HLuFLjnE0/V48QtXmfZmayq5ADegkHFqUOHnr/PmkcvGzO8ckg25Va/Nhbp 624P9fNBRgVEO0OSAdybsWt0NN1MIg0vK7/tG7BvJmQxXDgeBXiCd5qypBpjLpscvy1nyI pk5qMPLSayf21fKmyBDmNb2k6Lgc6zSZCprk26zn6Irj2MX2uECe+ZUf0N4DJUihTJdKLy p4pwp+YymCyRwjsgxS3OODaT595vmaomrKzOT32VcSCyh7ujWhe8Y9qPtOB/8S9vR0Ibgq wizmbTDj73RFfuclzs4Az6cQ/hqGJCH+vainSzr0wlwwelEXYIRfZZcbD9kIig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706783440; a=rsa-sha256; cv=none; b=MRFfd/lQ61RwueGndrHCxck2w9/T1B2OpC9KLKwhwdNsEVMHdNGLvwYbjF5d291KmrxWcO 2ZuQs+kHFrU3dl+ItP3P2IVyIcZafgrlgDRJjUeIvjlXbGV0oocHSRrr5tpoU+3mBnr6f9 p0A69zcLjf6I3vR23RxeACVhSXcCsEtlSHO8I3loHCqEo2QbT5VBhcLSzJtFbsn9AW1wb1 /mfP6jhZZQuuV5kejUsJLGXcWasZzo4DZlrJzoR1pljW1c18qDNjI9/rLV57gpYSUQzDxa x31peZ0Tx7vbtSKxmciMyYKq+ctX5HqPY5cWaebLtKKTI4DOmU07xWgK9EI3kw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TQZtS458xzyTX for ; Thu, 1 Feb 2024 10:30:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 411AUe40017321 for ; Thu, 1 Feb 2024 10:30:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 411AUeqS017320 for transport@FreeBSD.org; Thu, 1 Feb 2024 10:30:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Thu, 01 Feb 2024 10:30:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rscheff@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 Richard Scheffenegger changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |glebius@FreeBSD.org --- Comment #2 from Richard Scheffenegger --- Daniel, If you can deterministically reproduce this, that this is probably what Gleb was looking for. It is likely that this is an instance where the retransmit timer is active = on a closed (or closing) session, where the socket buffers were already freed... The BT certainly looks like this - the open question is, why not all outstanding timers get cancelled when a session is closing - one codepath s= eems to be missing the relevant cleanups. (This appears to have been exposed by my recent change of not discarding the SACK scoreboard on an RTO - SACK retransmissions do re-arm the RTO timer.) If you are looking for a quick workaround, we have one, but would really appreciate your help into understanding the actual root cause (codepath whe= re session closure is not properly cleaning up). --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 1 10:33:37 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQZxs59mwz58M12 for ; Thu, 1 Feb 2024 10:33:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQZxs46bRz47wq for ; Thu, 1 Feb 2024 10:33:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706783617; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pSaSKRmQx5Cb6vjnUzJSxvBGbqW4dC0HnHEw3zXm7B0=; b=LbgPoSZUrhC6K7ks9XEB66KsKqju9s27ujTjz5/2SYWTEaWviWbijZHerJDP4pXjcnO6cU bTgPRNp36qV4FF0Brh/HtXJ/GqjDlbJFJUKL6p3pHJGynG00pNMyrme1rxREz/sV+/S032 p4TZlA824XwdbDczr1VkAwTGN0WW+2Hv7jryak0xRatxz3TZUYZpCsPHJDS4/xzFLQp8uo G8IqeMvRPm32FVRy9f98oLohVU+OlLtTtBXAS1IUeLlGtidVBnK4a/KqQ+9+3kiHJFylrs iqY5I0dWMj6D/vE/wsmAOlGyeB8JCex7phBtHsc6+1IVSvMNLO1pqDEWWbBZJw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706783617; a=rsa-sha256; cv=none; b=FQPlCpvNhlHpPDkilOwJFUwUXee2bfAXYEHHrYtXFyQb1darhw8KDA1t4xLNLU9zEzxWX8 9nlD9c95f6i/3YRjcSkTdM2/6C+Pfv3gLqN4SDj+l38rVnuWwkmlom1UzgORwphsVJuXy5 ne7FmmtqqUtE5+WICXbBMSfxYDbxDCL6WVEaUiTZbPDMlY53vlAklnm5qUFd6NDfAFw6bY trgH0VsN/K68EQMyaHYjOX99moiqBk1qmr0R2RCTrcQB0A2RkT5hob/4+7oPqm1gV+ONiz zHeTZlZOPewgtx+98QB14wpcBW03IswwbjRbqKg7IG8V/rS5NqF7Qu0U8kcbhQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TQZxs3BCrzykw for ; Thu, 1 Feb 2024 10:33:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 411AXbY7043092 for ; Thu, 1 Feb 2024 10:33:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 411AXbVq043091 for transport@FreeBSD.org; Thu, 1 Feb 2024 10:33:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Thu, 01 Feb 2024 10:33:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ddaniel@nvidia.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 --- Comment #3 from Daniel --- Its happening during IPsec testing by sending TCP traffic using iperf tool Please download the core from here and let me know when I can remove it from my drive https://drive.google.com/file/d/1LxoaCE9QoZuwGYKGFqKYeVt7ozVS4aXm/view?usp= =3Dsharing --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 1 10:33:43 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQZy003lBz58MVc for ; Thu, 1 Feb 2024 10:33:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQZxz3RGxz47lr for ; Thu, 1 Feb 2024 10:33:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706783623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RAfXq/FfZJQlZel/+YRzPZj7/Awzeog0bCov83Hktoo=; b=kafOvOHOZjLqaDwEX1MeEPwMbUl9waSOxQqqB70F2zGEhAqEKZJrV8khA/2OUnDCgKSP4O AZfEtQ1hezH1xNQTshx7GBSaLbJ45/Jcj66U7372+E1ZynKAqTgYKBn18+98IVHbhq3NcW 0cMimo0/zU/x3txi830oO9yZy2R7VGoKtfuLtgfziWiUdgJSbuhYXkjTB2X6pRWJfsvUhx YrlGSOIQMXXfkx+GlAFR5qo1JCAhD7GmcLvmT8XX64o9qoq7jnuZFd31oTtqWUpqWeYh4m nstehqH4l1/2LU7D0Kn30v8dFu5uhJnvZdMYmQ8L5arZ8FqzxnwOob/rjqIRiA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706783623; a=rsa-sha256; cv=none; b=Fg+BxnDwGCfDQ/C//ZTg230br+WPJ4c7j85cPBVhX2eUw73Ft31KcpA0JR4/UKxWlbULvE l4hPxmabPr3h3o2VcpU8E7WT3p/VnZy0j1OBsJX33kdS63hWOJO8HjIQbonTa0I0cpRRDh d8JJW4QczqgqL5vzGRAuaO4p37Q1otnXnN4N5qNi3wlrfKZbTieimAq2DEDxp/2abHeKDc DayaA0adHj2NJTvXXLgApC3Sahu8BkIT0LkMl0RLqVTK22dG/TTORqFAL45yLEfVOBtLMc Yr2gTsooaCOIkeJ2+N94p+NzgikydPGHNviuzl26TteXW+51+Hv49jeajEMYCw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TQZxz2W1bzyvT for ; Thu, 1 Feb 2024 10:33:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 411AXh77043394 for ; Thu, 1 Feb 2024 10:33:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 411AXh7O043389 for transport@FreeBSD.org; Thu, 1 Feb 2024 10:33:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Thu, 01 Feb 2024 10:33:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ddaniel@nvidia.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 --- Comment #4 from Daniel --- Its happening during IPsec testing by sending TCP traffic using iperf tool Please download the core from here and let me know when I can remove it from my drive https://drive.google.com/file/d/1LxoaCE9QoZuwGYKGFqKYeVt7ozVS4aXm/view?usp= =3Dsharing --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu Feb 1 18:11:50 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TQn6b03v6z58MBY for ; Thu, 1 Feb 2024 18:11:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TQn6Z67r0z4BH9 for ; Thu, 1 Feb 2024 18:11:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706811110; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gc6GdqbPbLA0W0In2EhBOB/5upeoxJ3RHNsiOs7NPTQ=; b=i5NUcdW9v39R6e6Wazfgc2LH9VMcvjmVy//Y5NVa7xIYUHI6m3rRZjK2yLpzY+d7p1pt9h BUgccgJm/HGOUtc7YxW+m4fSsk+tr2xnQ262EJ198Ntmk7olkOSqjVOnMHdeo9Da5x0pjL fO2p9ATnXZCNsDoleMnRe0TMvsGj9WIv79q4IRSmWRK0oFTI0vY07Ve6sRA9IbTWsikAn8 Im92CQU4AXGj66ZI0UFBl1Oc5S39X+8kj1HHXpiXY+5IUpWWhiXUntaP6ciBzBQ15OFf9O mZ11mgahXKT2gTLbVxbSArv1tSLzLfVf+71TUDJlUzWffvH1rgjts0fmeD3ayQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706811110; a=rsa-sha256; cv=none; b=rpl/JBvGcnWI6A1cckHellN/zoHSIq2LhZbKLoMtLZb1EKc6aKQogSmFmCVC72oIUfJ2KV jFXPPN0alhz81ImAtYRkgqlCeA8lo31WJ8oWjP+Wr2y5EdqqLo1RQ7jTLMUNoCBwnC9PRk lN03+/lFO2goGJtz47E9VQy+jqcBivNPSoLGfzROSvBZve2/KRlahZZDOApNYwReFhnnmD KqTiU5VO1vYYJmZ9+8+H547V3RPVPueIRiWTW9d5LAppZ4xKBWYywJybK8Sr9qbIptNLJ0 ejXbbmPw6oPey6xRWq98DmmWeBRs21n4sYc1flE3d7ktJ6WttUh05zOeSUey8A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TQn6Z4Hz8z1BZf for ; Thu, 1 Feb 2024 18:11:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 411IBoJ8057638 for ; Thu, 1 Feb 2024 18:11:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 411IBovs057637 for transport@FreeBSD.org; Thu, 1 Feb 2024 18:11:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Thu, 01 Feb 2024 18:11:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: glebius@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 --- Comment #5 from Gleb Smirnoff --- Daniel, yes the core looks the same as several I already have. As Richard said we have workarounds against this panic and we know what commit opened it up. The commit itself isn't a mistake, though. Instead of committing any of our workarounds we are looking to find the root cause of the panic, a path that would flush socket send buffer but leave sack scoreboard. If you have any clue how reproduce the bug, please share. You mentioned certa= in iperf action/configuration. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 2 09:21:46 2024 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TR9Jb2Mjqz58MNV; Fri, 2 Feb 2024 09:21:51 +0000 (UTC) (envelope-from rscheff@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TR9Jb1sCTz4mGT; Fri, 2 Feb 2024 09:21:51 +0000 (UTC) (envelope-from rscheff@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706865711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=mOvaPzYeToomQJOU+lrH7efIQCYNOF7zA2xnSoEvWaI=; b=VR8mq/XW0roKUy0nIca2JSO4Yohp1vvMeCoiulRNKOZhA3fZfj+sadun88VzvZFSVy+Vpj dd3XgtXuYc1WkuVOk8cTerIKdqJArj0JSVLZ/nUuX8ErlyBqM6dOrjoXd3HVLwGVK8AE0k pe0zRjn/9d0QoXjcFKuLR7tbYnUiH5W1F2jvGSAnrAWj4iEqdSWUcwR1+Z19ZGUk476ZvD 9/gu79DlGt+XBMjLDf8mSAP7y+xw6zwKHLnvr8uVWdjcvN01erAM62ZPR14sQGHldNJvtf NlWAqXINJo/kLr4rx/GuAFcDMQe51sXcyaVoRy4w/w6BSzmSEYX4G/PrVHQ6dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706865711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type:autocrypt:autocrypt; bh=mOvaPzYeToomQJOU+lrH7efIQCYNOF7zA2xnSoEvWaI=; b=mO7fFfkTcB8PldRS1vsIdumZGSwBY7jtLc7HLl5labbxBGlJ4Fa85liE9v1kkXFfvO51Rq DGW+fzhYf3fFEHEH32Fz1Ujp91NVaP8zxEhpMkwzObDkWnWXVW4CnR35HhssGhskcZf4v6 kjA6rzqn0/9MFM6alYnBmNuoT/Yr6TLlAYHiH+K2IJ89ReyK5XbO5s1dliJUpZ9IdRdveV NWynk4dRIESOe44f0Z6jLF5SNVo/QDSQqgKanyKo6Qeji6aiir4Q1J5chdEnPI4jZcWf2b /woOdLfp9TVwS3XETwxdkgH/fMtPsHbOFB96g9InNfyfxT2ffF4kWXixQ954dQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706865711; a=rsa-sha256; cv=none; b=UG0VuFqZNJnUt6I6C83ddZAjetfbtHyHwL7dw5jXSs491KPkaQeZ6IWTyVS7g2FFKKTLwp 9MaKpybrcquCDlR0VC0LbwasCzlgnKE8xYD83VscgAqK+r94lVOiZMYA097ZRPZX5eqpr3 9RunNUJR4B5oiP0Lmvuk87+dE12Gx+t2No5pHIo4jTJys13ylMl0hrYILopSZRN/pgbzrS nzfDbwfA3EFBw3cwFOQIvulBRVcIRSBcY0mm5LS8erU/jNy2mwFv6fS7IHwtB9Y+D6+Jv6 u9V4aV1gNLS7YGeVru6thQzKLfPyoR1VT0Avmmv84wQg5BvCldmM/9wi4fnUsA== Received: from [192.168.233.122] (unknown [185.236.167.136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: rscheff/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TR9JZ3h3RzXQM; Fri, 2 Feb 2024 09:21:50 +0000 (UTC) (envelope-from rscheff@freebsd.org) Message-ID: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Date: Fri, 2 Feb 2024 10:21:46 +0100 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: "freebsd-net@FreeBSD.org" , FreeBSD Transport , rmacklem@freebsd.org, gallatin@FreeBSD.org, kp@FreeBSD.org From: "Scheffenegger, Richard" Subject: Increasing TCP TSO size support Autocrypt: addr=rscheff@freebsd.org; keydata= xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iziPF0FzrN K1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9yZz7CmgQTFgoAQhYh BDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGABQsJCAcCAyICAQYVCgkICwIE FgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/nvht8kExJ31M+3qpjOqdVypMp+/Ojvh5Z lsk96QEA5HCBkteJcrohwRA7llZvLH3m25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdV AQUBAQdA1Dim8ZWpXRS5i9hb3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2 S7eZrINEWrPNtHEXvliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bA EPqOH+JCIND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8= Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------cFlXxpdNPSvFDnJ2Q7jJvaOb" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------cFlXxpdNPSvFDnJ2Q7jJvaOb Content-Type: multipart/mixed; boundary="------------imiNeDBqBOWYIblw01WKAtFz"; protected-headers="v1" From: "Scheffenegger, Richard" To: "freebsd-net@FreeBSD.org" , FreeBSD Transport , rmacklem@freebsd.org, gallatin@FreeBSD.org, kp@FreeBSD.org Message-ID: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Subject: Increasing TCP TSO size support --------------imiNeDBqBOWYIblw01WKAtFz Content-Type: multipart/mixed; boundary="------------f8bM96TDLFPthBSFcPuWJDK5" --------------f8bM96TDLFPthBSFcPuWJDK5 Content-Type: multipart/alternative; boundary="------------g30wiijxOrV2m8hMPa7cSCrN" --------------g30wiijxOrV2m8hMPa7cSCrN Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 DQpIaSwNCg0KV2UgaGF2ZSBydW4gYSB0ZXN0IGZvciBhIFJQQyB3b3JrbG9hZCB3aXRoIDFN QiBJTyBzaXplcywgYW5kIGNvbGxlY3RlZCANCnRoZSB0Y3BfZGVmYXVsdF9vdXRwdXQoKSBs ZW4oZ3RoKSBkdXJpbmcgdGhlIGZpcnN0IHBhc3MgaW4gdGhlIG91dHB1dCBsb29wLg0KDQpJ biBzdWNoIGEgc2NlbmFyaW8sIHdoZXJlIHRoZSBhcHBsaWNhdGlvbiBmcmVxdWVudGx5IGlu dHJvZHVjZXMgc21hbGwgDQpwYXVzZXMgKHNpbmNlIHRoZSBuZXh0IGxhcmdlIElPIGlzIG9u bHkgc2VudCBhZnRlciB0aGUgY29ycmVzcG9uZGluZyANCnJlcXVlc3QgZnJvbSB0aGUgY2xp ZW50IGhhcyBiZWVuIHJlY2VpdmVkIGFuZCBwcm9jZXNzZWQpIGJldHdlZW4gc2VuZGluZyAN CmFkZGl0aW9uYWwgZGF0YSwgdGhlIGN1cnJlbnQgVFNPIGxpbWl0IG9mIDY0a0IgVFNPIG1h eGltdW0gKDQ1KjE0NDggaW4gDQplZmZlY3QpIHJlcXVpcmVzIG11bHRpcGxlIHBhc3NlcyBp biB0aGUgb3V0cHV0IHJvdXRpbmUgdG8gc2VuZCBhbGwgdGhlIA0KYWxsb3dhYmxlIChjd25k IGxpbWl0ZWQpIGRhdGEuDQoNCkknbGwgdHJ5IHRvIGdldCBhIGRhdGEgY29sbGVjdGlvbiB3 aXRoIGJldHRlciBncmFudWxhcml5IGFib3ZlIDkwIDAwMCANCmJ5dGVzIC0gYnV0IGV2ZW4g aGVyZSB0aGUgYXZlcmFnZSBzdHJvbmdseSBpbmRpY2F0ZXMgdGhhdCBhIG1ham9yaXR5IG9m IA0KdHJhbnNtaXNzaW9uIG9wcG9ydHVuaXRpZXMgYXJlIGluIHRoZSA1MTIga0IgYXJlYSAt IHByb2JhYmx5IGFsc28gaGF2aW5nIA0KdG8gZG8gd2l0aCBMUk8gYW5kIEFDSyB0aGlubmlu ZyBlZmZlY3RzIGJ5IHRoZSBjbGllbnQuDQoNCldpdGggb3RoZXIgd29yZHMsIHRoZSB0Y3Ag b3V0cHV0IGhhcyB0byBydW4gYWJvdXQgOSB0aW1lcyB3aXRoIFRTTywgdG8gDQp0cmFuc21p dCBhbGwgZWxlZ2libGUgZGF0YSAtIGluY3JlYXNpbmcgdGhlIEZyZWVCU0Qgc3VwcG9ydGVk IG1heGltdW0gDQpUU08gc2l6ZSB0byB3aGF0IGN1cnJlbnQgaGFyZHdhcmUgY291bGQgaGFu ZGxlICgyNTZrQi4uMU1CKSB3b3VsZCByZWR1Y2UgDQp0aGUgQ1BVIGJ1cmRlbiBoZXJlLg0K DQoNCklzIGluY3JlYXNpbmcgdGhlIHNvZndhcmUgc3VwcG9ydGVkIFRTTyBzaXplIHRvIGFs bG93IGZvciB3aGF0IHRoZSBOSUNzIA0KY291bGQgbm93YWRheXMgZG8gc29tZXRoaW5nIGFu eW9uZSBhcGFydCBmcm9tIHVzIHdvdWxkIGJlIGludGVyZXN0ZWQgaW4gDQooaW4gcGFydGlj dWxhciwgdGhvc2Ugd2hvIHdvcmsgd2l0aCB0aGUgZHJpdmVycyk/DQoNCg0KQmVzdCByZWdh cmRzLA0KDQogwqAgUmljaGFyZA0KDQoNCg0KDQp0c28gc2l6ZSAodHJhbnNtaXNzaW9ucyA8 IDE0NDggd291bGQgbm90IGJlIGFjY291bnRlZCBoZXJlIGF0IGFsbCkNCg0KIMKgwqDCoCDC oMKgwqAgwqDCoMKgIMKgwqDCoCDCoMKgwqAgIyBjb3VudA0KDQo8MTAwMCAJMA0KPDIwMDAg CTIzDQo8MzAwMCAJMTExDQo8NDAwMCAJNDANCjw1MDAwIAkzMA0KPDcwMDAgCTE0DQo8ODAw MCAJMTM0DQo8OTAwMCAJNDQyDQo8MTAwMDAgCTkzOTYNCjwyMDAwMCAJNDYyMjcNCjwzMDAw MCAJMjU2NDYNCjw0MDAwMCAJMzMwNjANCjw2MDAwMCAJMjMxNjINCjw3MDAwMCAJMjQzNjgN Cjw4MDAwMCAJMTk3NzINCjw5MDAwMCAJNDAxMDENCiA+PTkwMDAwIAk3NTM4NDE2OQ0KQXZl cmFnZTogCTU3ODg0NC40NA0KDQo= --------------g30wiijxOrV2m8hMPa7cSCrN Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


Hi,

We have run a test for a RPC workload with 1MB IO sizes, and collected the tcp_default_output() len(gth) during the first pass in the output loop.

In such a scenario, where the application frequently introduces small pauses (since the next large IO is only sent after the corresponding request from the client has been received and processed) between sending additional data, the current TSO limit of 64kB TSO maximum (45*1448 in effect) requires multiple passes in the output routine to send all the allowable (cwnd limited) data.

I'll try to get a data collection with better granulariy above 90 000 bytes - but even here the average strongly indicates that a majority of transmission opportunities are in the 512 kB area - probably also having to do with LRO and ACK thinning effects by the client.

With other words, the tcp output has to run about 9 times with TSO, to transmit all elegible data - increasing the FreeBSD supported maximum TSO size to what current hardware could handle (256kB..1MB) would reduce the CPU burden here.


Is increasing the sofware supported TSO size to allow for what the NICs could nowadays do something anyone apart from us would be interested in (in particular, those who work with the drivers)?

=


Best regards,

=C2=A0 Richard




tso size (transmissions < 1448 would not be accounted here at all)

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 # count

<1000 0
<2000 23
<3000 111
<4000 40
<5000 30
<7000 14
<8000 134
<9000 442
<10000 9396
<20000 46227
<30000 25646
<40000 33060
<60000 23162
<70000 24368
<80000 19772
<90000 40101
>=3D90000 75384169
Average: 578844.44
--------------g30wiijxOrV2m8hMPa7cSCrN-- --------------f8bM96TDLFPthBSFcPuWJDK5 Content-Type: application/pgp-keys; name="OpenPGP_0x17BE5899E0B1439B.asc" Content-Disposition: attachment; filename="OpenPGP_0x17BE5899E0B1439B.asc" Content-Description: OpenPGP public key Content-Transfer-Encoding: quoted-printable -----BEGIN PGP PUBLIC KEY BLOCK----- xjMEY/i74RYJKwYBBAHaRw8BAQdAwtnvjlFVnnzNXO9hjHtB6MPGSY19L/BHh/iz iPF0FzrNK1JpY2hhcmQgU2NoZWZmZW5lZ2dlciA8cnNjaGVmZkBmcmVlYnNkLm9y Zz7CmgQTFgoAQhYhBDZLt5msg0Ras820cRe+WJngsUObBQJj+LvhAhsDBQkJZgGA BQsJCAcCAyICAQYVCgkICwIEFgIDAQIeBwIXgAAKCRAXvliZ4LFDm4ylAQCSw2/n vht8kExJ31M+3qpjOqdVypMp+/Ojvh5Zlsk96QEA5HCBkteJcrohwRA7llZvLH3m 25hcJdzmDh39mc0cSgPOOARj+LvhEgorBgEEAZdVAQUBAQdA1Dim8ZWpXRS5i9hb 3O4RNHub8XvqTTkYyiZ2lSkXDwYDAQgHwn4EGBYKACYWIQQ2S7eZrINEWrPNtHEX vliZ4LFDmwUCY/i74QIbDAUJCWYBgAAKCRAXvliZ4LFDm2TGAQDcg+bAEPqOH+JC IND8wZ62MwnjFyXFv73qevXkUHHNSgEApUgpHW9f6UaIAQpc3R185xjz6tk8XXBx eYpxKgIAeQ8=3D =3DBwxS -----END PGP PUBLIC KEY BLOCK----- --------------f8bM96TDLFPthBSFcPuWJDK5-- --------------imiNeDBqBOWYIblw01WKAtFz-- --------------cFlXxpdNPSvFDnJ2Q7jJvaOb Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQQ2S7eZrINEWrPNtHEXvliZ4LFDmwUCZby0KgUDAAAAAAAKCRAXvliZ4LFDm73n AQC00+GZ7IDDzaPlORXmsNmMrrKgmFRT/rH6+MIpnbvA8AEA+130h9m/ksyrLjAOFWgXqG68ByH7 +gaZN4mhV5Q13Q0= =QnaZ -----END PGP SIGNATURE----- --------------cFlXxpdNPSvFDnJ2Q7jJvaOb-- From nobody Fri Feb 2 15:17:04 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TRKBT0qQVz58wcZ for ; Fri, 2 Feb 2024 15:17:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TRKBS6fBbz4LLv for ; Fri, 2 Feb 2024 15:17:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706887024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GYRBU6mwhgA4dxOGuhPBi5iyKXh6lGSpFFlsJ4C27I4=; b=gv6zi49J2UANjUbMcNUv7uz93NkkCtlk7H8EGODP5xwRwqw5SSFUONJEllUaGJYt/IqPL2 VdpNzlmshET3qxlRYb9O2e2c/vGYlFzUYn4rtfpyDGHGHWI2/x7Qefy142L6IsOf6i2AuC cxrpV4z9Pm9AcfWzbgBs2OasF0jSbHygnC4yzejd5BlMDTSpAGm3tWaY2ayadH9jHeAOXo ExL2OQmdH5yASHttfSZwY5mT7ay+LQIfxlNzXEMtNQqlRpwyCIegnyvXy3W5OP8BJZndGU yzmd9BnsEUTcQPOYwjvskSo3rBs3w5zYAMCPK0lf2XA7JNqrY5/fbi4H/Qr5tA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706887024; a=rsa-sha256; cv=none; b=Um84BJiklVII8PCwX14bVEARXBGcJQUEWwLyHLBOjZjTHczwa+vdDuGw+iQNe6RofleEUo w2pts0V1KLLEBE0Oi7VVcnVdC6pqeFrKCCUM+ivoRcgMVmTJtaZMBNJ3oqlYGu/rpKmGcZ 1OoP1R8NZ7r5OehdfyiAT4jocIXd38UmvN7Poo+gpk8lzjcheCsp3FqirX8Ee4Bibq1m2v dAqbbO/ePVV32NPrRwK0yNEh0Q5j2D2pUhFF4Y2dhgV0Bo987A2JNaZQ5Hbxkv2dpxB8qc TJT09zpZbUutBS4BAgGqGrQwBstPybk2M0304kw1ty6RsDntB4vxs9HuzUfdqQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TRKBS5XvPzbXk for ; Fri, 2 Feb 2024 15:17:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 412FH453049727 for ; Fri, 2 Feb 2024 15:17:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 412FH4qG049726 for transport@FreeBSD.org; Fri, 2 Feb 2024 15:17:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Fri, 02 Feb 2024 15:17:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ddaniel@nvidia.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 --- Comment #6 from Daniel --- Hi=20 I was able to reproduce the bug by running the following test in a loop ~20 times test flow (ran it ~20 times till reproduced): run in loop 8 esp\IKE alg (twice): 1. set IPsec configuration 2. start ipsec - ipsec start 3. send IPerf stress traffic ~10 sec iperf params: -P 100 -t 20 --format g -p 17123 4. stop iperf traffic stop iperf traffic after ~10 sec - killall iperf 5. stop ipsec - ipsec stop 6. change IKE and esp alg 7. return to stage 1. IKE and ESP algorithms I am running in a loop 1. aes256-sha2_256-modp1024! 2. aes256-sha256-modp4096 3. aes256gcm16-prfsha384-ecp384 4. aes256gcm16-sha256-modp1024 5. aes128-sha2_256-modp1024! 6. aes128gcm16-prfsha256-ecp256 7. aes128-sha256-ecp256 8. aes128gcm16-sha256-modp1024 IPSEC version FreeBSD strongSwan U5.9.10/K15.0-CURRENT iperf version: 2.1.9 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 2 15:26:35 2024 X-Original-To: transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TRKPR6xsHz58xgF for ; Fri, 2 Feb 2024 15:26:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TRKPR5mr1z4M57 for ; Fri, 2 Feb 2024 15:26:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706887595; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IIe3lfDVneK4Z+yJ0p558KbaN4jE2gA33K+PZUpahgM=; b=F2QmU8LWUdXziBXFZbvUwTFu0e+ZyJjReyqmS3SQqUTZq6DUs6Jyy4cUTzbMSZDISrOUr3 65Kw1eRCfrnIwXCgr5jXaurMX7Tu9LTz5y/V9WJ//MAn3f5K/1uUZnK5cmSeMSCbZwYZ3S 1E3FhdPUJIFPqcGH78HGHOLPnERNYZ4nnEXbs5dqJRrLzdHmQBSOrFtszkMJHG5ZFzqT9/ 4vx1X/8tEVzjq8Q3p05aGpp7fNNKq83uInj91VKYy+YKkKW5+OWAivfBHZEWIlrDkVwg23 rVg2PbUmHz9DfA8MIuKWKWztooLOHpP+0IrZOVcamZNXF59DLlleRTbasMJBTw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706887595; a=rsa-sha256; cv=none; b=D9ChYW1ApyIB8toi8hfstXSulYdUK2mOdDevcte1lZBjDIgSpwvmKkENfHvvcMcR+GXA36 fA5VzYdMcw3uFn2SKZQBK/Mb3fHBeEDSXZcnMaxLMjomR9LTfmYlozfQ7VZqbqwb5HUjHk XhCWZBv1SzpWluhgOWS+CJ+l+XPnWFXReZNwMMSsbphH9ZLQ2rT/7pXE02BGPvyV58SLjF Xhb/kfTNloPcITWyJxzPLEvlwvJIKAhJ0JTob3tWB+xv/VkZbWFiY0aZu923SYgcyP1v3S WPCiCdqndUzmOpbq18LdeCA+SReoIwfzSzONBeqO77Tr/IebT0bSE3Fu3B1czQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4TRKPR4mtWzbQD for ; Fri, 2 Feb 2024 15:26:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 412FQZ31099028 for ; Fri, 2 Feb 2024 15:26:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 412FQZjx099027 for transport@FreeBSD.org; Fri, 2 Feb 2024 15:26:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: transport@FreeBSD.org Subject: [Bug 276761] panic: sbsndptr_noadv: sb_mb is NULL Date: Fri, 02 Feb 2024 15:26:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rscheff@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: transport@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D276761 --- Comment #7 from Richard Scheffenegger --- Last TCP / IPSec bug I looked at was an oversight in error handling (on the= TCP side), when ip_output() would return ENOBUFS (which has a higher likelyhood apparently when IPSec is in play). Maybe something along these lines plays a role here as well (just speculati= ng, didn't really check all the codepaths for closing a session, and what/if any potential error is always properly handled...) --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Feb 2 22:14:15 2024 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TRVSF2spcz58Nrp; Fri, 2 Feb 2024 22:14:37 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TRVSF25c5z40RM; Fri, 2 Feb 2024 22:14:37 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706912077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DjtbGOYAruVAO+vh5MF8LOAoRii6LS0uXXBjX+ehR1U=; b=QzX/5UC0ozDjLhqFyBtpK2PejmbCytLfc+jP15nI2VcItz4Rb8qldzOS95535nl56A7H37 R2hxR9DHdloUNnzXe0KpCR1CHJuCOgeLn9pFkIrwui4ZcSqmREQl1As1iZNvlRKzBV2A2m yAzpCefwfOo0eT9ousAVC/MrVAle7mHB56g3hnzOa1sovUdObE0R9zDlZ+CqjwNdTm2mTt +wOBOnDtTic4OloKFGKBjYZZV2rmjhrfwjdSh07no5sbnZHI5Ynb/LSFqBrXcEdoRnmkpb b2JB6UNf1pxNCyHRzH88u4TbfpnU6Z4D7vykclLsHCKQsFznl6AtFq9FjTEOXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706912077; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DjtbGOYAruVAO+vh5MF8LOAoRii6LS0uXXBjX+ehR1U=; b=QfQkV0Y0i/C9rOBGyHM417YMKxTWJN78+rDZS2sCsUOkH5CyIdvrAyjXLWKkWx3gjszWdJ 90mUAdNtq0HzulpjjVwe526JqwKHqdUwH+vlo4z9LvDF3oFHym3kfT/smLEkrJhzlKNkx7 txGs1tgs21MV+efLxZOjjDtVIU5nmpoyMoGDjikD+1s6N6BOhwayRbt3NN7LUHv7+RNids jOMmXGOQOgA/ANYchsutY3ml386XN35fWCjqU3MuiRdWG04mnmdIrmkKCP4v3mfMyn+38Y Y4xvMPkEenxV1fsuXgh1wtQKAx1e4Ryd4JomkTEdbKoDL9FE1cXAzj0qxmWXuA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706912077; a=rsa-sha256; cv=none; b=ak3YWdgJ1XQ9UQtVnnDDBqsFKN3uDwEL6W/y0tNlmqz9l7BH5ip+fiHqP3cjzWGKdvegep 4/jgAOCOG1xPuI5GDqlKbnLqTCrdHBBtN4fQrSWHwuS+msHESDIXfT7spc29OQTmfqj/Zj eyembYZQD5SHsjzAy9RwkxqdYTFiEnmx1ODllRLP+3qgRlQqc3rS4bfMk+SasrF6rJZHjd WhJlHUDjwxJs/CoAPGyG9cxx3JBrCVfd+055U8yEmA4qdeh4qdp74b6JZNYShG9XdGtH20 lPJmiojrfMA4LG/Uo8f0LS+ufsYOB+SSQRRRrEtAaAZ3++IQT6IWMEuAbo1SyA== Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com [66.111.4.227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TRVSF0qxMz11tq; Fri, 2 Feb 2024 22:14:37 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id 2010D27C005B; Fri, 2 Feb 2024 17:14:36 -0500 (EST) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Fri, 02 Feb 2024 17:14:36 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedugedgudehjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefofgggkfgjfhffhffvufgtsegrtderreerredtnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpeehjeevleeigeefffdtgedtudegheetteeigfeileejuedv gefhvdekjedvhefhveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehgrghllhgrthhinhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihht hidqudeffeehledvvdduiedqvdelhedtgedukeegqdhgrghllhgrthhinheppehfrhgvvg gsshgurdhorhhgsehfrghsthhmrghilhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id D7AF6364006B; Fri, 2 Feb 2024 17:14:35 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 Message-Id: <95e76a2c-44c8-4fbb-ab45-8bcffe80d4a3@app.fastmail.com> In-Reply-To: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> References: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Date: Fri, 02 Feb 2024 17:14:15 -0500 From: "Drew Gallatin" To: "Richard Scheffenegger" , "freebsd-net@FreeBSD.org" , "FreeBSD Transport" , rmacklem@freebsd.org, kp@FreeBSD.org Subject: Re: Increasing TCP TSO size support Content-Type: multipart/alternative; boundary=16179d38f0f74410b648a6fe3220c987 --16179d38f0f74410b648a6fe3220c987 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable What is the link speed that you're working with? A long time ago, when I worked for a now-defunct 10GbE NIC vendor, I exp= erimented with the benefits of TSO as we varied the max TSO size. I ca= nnot recall the platform (it could have been OSX, Solaris, FreeBSD or Li= nux). At the time (~2006?) the CPU saving benefits of increasing the ma= x TSO size from 8k to 64k was fairly minimal. In fact, I seem to reca= ll that there was almost no benefit to TSO sizes larger than 16K. I was wondering if you see any difference in your benchmark if you cap m= ax TSO size to 8k, 16k,32k, and the default of 64k. Any change in CPU = use, or in your benchmark's performance would be interesting to hear abo= ut. Naively, I'd expect the benchmark performance to remain unchanged until = you'd reduced the TSO size so much as to make the host slower than the w= ire, thereby inserting gaps between TSOs. That would be reflected in th= e CPU use as well.. Drew On Fri, Feb 2, 2024, at 4:21 AM, Scheffenegger, Richard wrote: >=20 >=20 > Hi, >=20 > We have run a test for a RPC workload with 1MB IO sizes, and collected= the tcp_default_output() len(gth) during the first pass in the output l= oop. >=20 > In such a scenario, where the application frequently introduces small = pauses (since the next large IO is only sent after the corresponding req= uest from the client has been received and processed) between sending ad= ditional data, the current TSO limit of 64kB TSO maximum (45*1448 in eff= ect) requires multiple passes in the output routine to send all the allo= wable (cwnd limited) data. >=20 > I'll try to get a data collection with better granulariy above 90 000 = bytes - but even here the average strongly indicates that a majority of = transmission opportunities are in the 512 kB area - probably also having= to do with LRO and ACK thinning effects by the client. >=20 > With other words, the tcp output has to run about 9 times with TSO, to= transmit all elegible data - increasing the FreeBSD supported maximum T= SO size to what current hardware could handle (256kB..1MB) would reduce = the CPU burden here. >=20 >=20 >=20 > Is increasing the sofware supported TSO size to allow for what the NIC= s could nowadays do something anyone apart from us would be interested i= n (in particular, those who work with the drivers)? >=20 >=20 >=20 > Best regards, >=20 > Richard >=20 >=20 >=20 >=20 >=20 >=20 >=20 > tso size (transmissions < 1448 would not be accounted here at all) >=20 > # count >=20 >=20 >=20 > <1000 > 0 > <2000 > 23 > <3000 > 111 > <4000 > 40 > <5000 > 30 > <7000 > 14 > <8000 > 134 > <9000 > 442 > <10000 > 9396 > <20000 > 46227 > <30000 > 25646 > <40000 > 33060 > <60000 > 23162 > <70000 > 24368 > <80000 > 19772 > <90000 > 40101 > >=3D90000 > 75384169 > Average: > 578844.44 >=20 > *Attachments:* > =E2=80=A2 OpenPGP_0x17BE5899E0B1439B.asc > =E2=80=A2 OpenPGP_signature.asc --16179d38f0f74410b648a6fe3220c987 Content-Type: text/html Content-Transfer-Encoding: quoted-printable
What is the lin= k speed that you're working with?

A long ti= me ago, when I worked for a now-defunct 10GbE NIC vendor, I experimented= with the benefits of  TSO as we varied the max TSO size.  I c= annot recall the platform (it could have been OSX, Solaris, FreeBSD or L= inux).  At the time (~2006?) the CPU saving benefits of increasing = the max TSO size from 8k to 64k was fairly minimal.    In= fact, I seem to recall that there was almost no benefit to TSO sizes la= rger than 16K.

I was wondering if you see a= ny difference in your benchmark if you cap max TSO size to 8k,  16k= ,32k, and the default of 64k.  Any change in CPU use, or in your be= nchmark's performance would be interesting to hear about.
=
Naively, I'd expect the benchmark performance to remain u= nchanged until you'd reduced the TSO size so much as to make the host sl= ower than the wire, thereby inserting gaps between TSOs.  That woul= d be reflected in the CPU use as well..

Dre= w

On Fri, Feb 2, 2024, at 4:21 AM, Scheffen= egger, Richard wrote:


Hi,

We have run a test for a RPC workload = with 1MB IO sizes, and collected the tcp_default_output() len(gth) during the first pass in the output loop.

In such a scenario, where the applic= ation frequently introduces small pauses (since the next large IO is only sent after the corresponding request from the client has been received and processed) between sending additional data, the current TSO limit of 64kB TSO maximum (45*1448 in effect) requires multiple passes in the output routine to send all the allowable (cwnd limited) data.

I'll try to get a data collection with better gran= ulariy above 90 000 bytes - but even here the average strongly indicates that a majority of transmission opportunities are in the 512 kB area - probably also having to do with LRO and ACK thinning effects by the client.

With other words, the tcp output has to run = about 9 times with TSO, to transmit all elegible data - increasing the FreeBSD supported maximum TSO size to what current hardware could handle (256kB..1MB) would reduce the CPU burden here.


<= p>Is increasing the sofware supported TSO size to allow for what the NICs could nowadays do something anyone apart from us would be interested in (in particular, those who work with the drivers)?


Best regards,

  Richard


=



tso size (transmissions < 1448 would not= be accounted here at all)

          &= nbsp;         # count


<1000
0
&l= t;2000
23
<300= 0
111
<400040
<5000
30
<7000
14
<8000
134
<= td style=3D"height:14.4pt;" height=3D"19"><9000
442
<10000
9396
<20000
46227
<30000
25646
<40000
= 33060
<60000
231= 62
<70000
2436= 8
<80000
19772=
<90000
40101<= br>>=3D90000
7538= 4169
Average:
578844.44

Attachments:
    =
  • OpenPGP_0x17BE5899E0B1439B.asc
  • OpenPGP_signature.asc
    =

--16179d38f0f74410b648a6fe3220c987-- From nobody Sat Feb 3 00:47:40 2024 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TRYsF0Zlxz58f7J; Sat, 3 Feb 2024 00:48:01 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TRYsF00qYz4KQ6; Sat, 3 Feb 2024 00:48:01 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706921281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=RJhZ2pmFa8Mh9nqZ8Q6UxZA2Ks25X7DGKKB3ByQW1tU=; b=tLBNgZUZSmIZEcnVsMrdD7YyUvXmym8JRAE9Wq9DhTeNq52i+RBIdvjlJgX10oeixCtjyC UA6vw+i+8oAHIgnepD4q34xU3x97Gr+9M2/nOjXtZLyN1yu49gcKCzApdPGL4xgyYrmSrS 0iP9Ljwgr7vRcPmdcOTOZOVHHLJhOJhO/fsXyq+NQP+A+ka2P/oqBAO6LZfJ2ZXYJupiH/ ECOiaaOsb0qqbzP87lfxj98Y5jESU2CcT0AFAAo/4dVOUzACIkejW8UjjGp619kJn4/ib1 NjOlLxXoaYz5uTWdcmg3sEdOrq0g+mrat4uwJ/X/FFwzE9M9Q3//mnSKCaxDqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706921281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=RJhZ2pmFa8Mh9nqZ8Q6UxZA2Ks25X7DGKKB3ByQW1tU=; b=MiPYRNOKuSgyPWhacpL48sAhBf1uopCHaUvUjIaloa7D9ffju2GwMHL/1idwFy4Y8eykQF RWLsOs9rj94HcSqdx0etEHYiSipeWorOLIJheH/yT0S6Fkq8k1rhCVIvN2mXN7oR4clAe1 kh0+tQfA7UD9KfPBVQuhny1WdX7oR5wJ5mnVcv3nRd1OW/P+8W0hO4c48+j+lxK0J55lfw x0R5IyigII7HY/wUJyqnWrvsmbV7oJm5EiRwqD7ykhWUEjARHNQMM23PPPhg5JcXvTskVf 2krL8wZ8tQ6PJ69VV7M2Wj4YuELqgv+a2AalPH0Vqyf3bO557pAFCTS/V06Qig== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706921281; a=rsa-sha256; cv=none; b=cepMKxf3i6raG/McEVMYvW7FMn6AE5aVW6ogWtKXjQ1aTPKZKA11aG9/PuzITyOvNNYng3 gYyVFDBGUNN2QKKravf/vepUclSlRiY54n38tba423Ld7knnZ1upZ+g435sSx8TKT4SkRP QlkBxWR9hYS/MeZwP00jS4TvPO8ge7GTCM01sMIYDHOW9URxx3N14qfXA0tR6VC5lfoKeL vJeOAnulYm25jGj4ZgGcmW4zWPTdYBunJmhpxP1wbrl0vjWSn9hRLWxqBdC9F5cjrkIS+b wNX1mlctNUadLHVt+eCvY5s/QUCZlZtiwG64Uo+70QHM55cZVO1LJPCG7QV5nA== Received: from auth2-smtp.messagingengine.com (auth2-smtp.messagingengine.com [66.111.4.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TRYsD5nLyz14jZ; Sat, 3 Feb 2024 00:48:00 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id A5FDA27C005B; Fri, 2 Feb 2024 19:48:00 -0500 (EST) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Fri, 02 Feb 2024 19:48:00 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfeduhedgvdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerredtnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpeeggfeugeevuedtuedvleefffduteegtdffudeihefhgfeg feekffeiueevkeeuudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehgrghllhgrthhinhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihht hidqudeffeehledvvdduiedqvdelhedtgedukeegqdhgrghllhgrthhinheppehfrhgvvg gsshgurdhorhhgsehfrghsthhmrghilhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 68C46364006B; Fri, 2 Feb 2024 19:48:00 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: References: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Date: Fri, 02 Feb 2024 19:47:40 -0500 From: "Drew Gallatin" To: "Rick Macklem" , "Richard Scheffenegger" Cc: "freebsd-net@FreeBSD.org" , "FreeBSD Transport" , rmacklem@freebsd.org, kp@freebsd.org Subject: Re: Increasing TCP TSO size support Content-Type: multipart/alternative; boundary=d72aaec284da4bab8e1160d4085e3fc4 --d72aaec284da4bab8e1160d4085e3fc4 Content-Type: text/plain On Fri, Feb 2, 2024, at 6:13 PM, Rick Macklem wrote: > A factor here is the if_hw_tsomaxsegcount limit. For example, a 1Mbyte NFS write request > or read reply will result in a 514 element mbuf chain. Each of these (mostly 2K mbuf clusters) > are non-contiguous data segments. (I suspect most NICs do not handle this many segments well, > if at all.) Excellent point > > The NFS code does know how to use M_EXTPG mbufs (for NFS over TLS, for the ktls), but I do not > know what it would take to make these work for non-KTLS TSO? Sendfile already uses M_EXTPG mbufs... When I was initially doing M_EXTPG stuff for kTLS, I added support for using M_EXTPG mbufs in sendfile regardless of whether or not kTLS was in use. That reduced CPU use marginally on 64-bit platforms (due to reducing socket buffer lengths, and hence reducing pointer chasing), and quite a bit more on 32-bit platforms (due to also not needing to map memory into the kernel map, and by reducing pointer chasing even more, as more pages fit into an M_EXTPG mbuf when a paddr_t is 32-bits. > I do not know how the TSO loop in tcp_output handles M_EXTPG mbufs. > Does it assume each M_EXTPG mbuf is one contiguous data segment? No, its fully aware of how to handle M_EXTPG mbufs. Look at tcp_m_copy(). We added code in the segment counting part of that function to count the hdr/trailer parts of an M_EXTPG mbuf, and to deal with the start/end page being misaligned. > I do see that ip_output() will call mb_unmapped_to_ext() when the NIC does not have IFCAP_MEXTPG set. > (If IFCAP_MEXTPG is set, do the pages need to be contiguous so that it can become > a single contiguous data segment for TSO or ???) No, it just means that a NIC driver has been verified to call not mtod() an M_EXTPGS mbuf and deref the resulting data pointer. (which would make it go "boom"). But the page size is only 4K on most platforms. So while an M_EXTPGS mbuf can hold 5 pages (..from memory, too lazy to do the math right now) and reduces socket buffer mbuf chain lengths by a factor of 10 or so (2k vs 20k per mbuf), the S/G list that a NIC will need to consume would likely decrease only by a factor of 2. And even then only if the busdma code to map mbufs for DMA is not coalescing adjacent mbufs. I know busdma does some coalescing, but I can't recall if it coalesces physcally adjacent mbufs. > If TSO and the code beneath it (NIC and maybe mb_unmapped_to_ext() being called) were to > all work ok for M_EXTPG mbufs, it would be easy to enable that for NFS (non-TLS case). It does. You should enable it for at least TCP. Drew --d72aaec284da4bab8e1160d4085e3fc4 Content-Type: text/html Content-Transfer-Encoding: quoted-printable

=
On Fri, Feb 2, 2024, at 6:13 PM, Rick Macklem wrote:
<= /div>
 A factor here is the if_hw_tsomaxse= gcount limit. For example, a 1Mbyte NFS write request
or read r= eply will result in a 514 element mbuf chain. Each of these (mostly 2K m= buf clusters)
are non-contiguous data segments. (I suspect most NICs d= o not handle this many segments well,
if at all.)

Excellent point
=

The NFS code does know how to= use M_EXTPG mbufs (for NFS over TLS, for the ktls), but I do not
know= what it would take to make these work for non-KTLS TSO?
=


Sendfile alr= eady uses M_EXTPG mbufs... When I was initially doing M_EXTPG stuff for = kTLS, I added support for using M_EXTPG mbufs in sendfile regardless of = whether or not kTLS was in use.  That reduced CPU use marginally on= 64-bit platforms (due to reducing socket buffer lengths, and hence redu= cing pointer chasing), and quite a bit more on 32-bit platforms (due to = also not needing to map memory into the kernel map, and by reducing poin= ter chasing even more, as more pages fit into an M_EXTPG mbuf when a pad= dr_t is 32-bits.


I do not know how the TSO loop in tcp_output handles M_E= XTPG mbufs.
Does it assume each M_EXTPG mbuf is one contiguous data se= gment?

No, i= ts fully aware of how to handle M_EXTPG mbufs.  Look at tcp_m_copy(= ).  We added code in the segment counting part of that function to = count the hdr/trailer parts of an M_EXTPG mbuf, and to deal with the sta= rt/end page being misaligned.

I do see that ip_output() will call mb_unmapped_to_ext() wh= en the NIC does not have IFCAP_MEXTPG set.
(If IFCAP_MEXTPG is set, do= the pages need to be contiguous so that it can become
a single contig= uous data segment for TSO or ???)

No, it just means that a NIC driver has been verif= ied to call not mtod() an M_EXTPGS mbuf and deref the resulting data poi= nter. (which would make it go "boom").

But = the page size is only 4K on most platforms.  So while an M_EXTPGS m= buf can hold 5 pages (..from memory, too lazy to do the math right now) = and reduces socket buffer mbuf chain lengths by a factor of 10 or so (2k= vs 20k per mbuf), the S/G list that a NIC will need to consume would li= kely decrease only by a factor of 2.  And even then only if the bus= dma code to map mbufs for DMA is not coalescing adjacent mbufs.  I = know busdma does some coalescing, but I can't recall if it coalesces phy= scally adjacent mbufs. 

If TSO and the code beneath it (NIC and maybe mb_unmapped_t= o_ext() being called) were to
all work ok for M_EXTPG mbufs, it would = be easy to enable that for NFS (non-TLS case).


It does.  You sho= uld enable it for at least TCP.

Drew
--d72aaec284da4bab8e1160d4085e3fc4-- From nobody Sat Feb 3 02:19:41 2024 X-Original-To: freebsd-transport@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TRbvQ2BgNz58nyd; Sat, 3 Feb 2024 02:20:02 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TRbvQ1WWTz4d94; Sat, 3 Feb 2024 02:20:02 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706926802; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=etFOi/HDfAtXaVCgcNdajbpYwcPoJvXj7bW98kLT/po=; b=NfQUwXtYrjwXJuRycqG9AMtFvc9AD3FqMzTyKIdWPYbuGRjXhgzEG9OoxM3DpVdlzPn6Zg ISsCksi54gNvCe5cOgFV2GfICF06RpLWXklRdEKnJV3bnBWq8lz336O1wlrD8f802TLaKF +Dpj8zhRmprcIulqs9uHgVwA8OCdz2hSI+rYGiYs8ryEzVShZqbZdZvvyvYvy9FQHX2M/9 5aOgNvftHBp25rTDV8gc7kZqayj3TGQY0ihNuiWW6duF862ZVbEb2NNRQKhHgnG+NBOTnz SHcrcwm69tQtcRBxZetUV1Xh4WSYaZgjCvPheF0L8ZTJ6kPi3Wrhv3VmtteoAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1706926802; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=etFOi/HDfAtXaVCgcNdajbpYwcPoJvXj7bW98kLT/po=; b=GJn5vXcU8PXqN6BCV6aeNpocNHK9GZgCMm0yk+By33va4SLvcNwTQL+GTYRGqqutLny29W qWyjwfHchMCEskTGpBdeFcaptIxJnvIwLevhIs2ID2AZ6gCO0cgkm45PaBXaVuQXWlo/81 ctwxqdWsqGFJgQN5kncf7r+flzvepv/DnkpjCn7O/Tbto8BVA6lRUmeSgLdETjf+fKQ5CZ O5uFCbbWoUuXKyqcvLpVxujD05d1TS7TosS3u1hRcTUuZE9/gPbd8TClQT1PA+V8X9QLKF Vwd/NWu1WT3uGI9jkyX0YV3ts+hvQd9UJhU9JT7MBZdC/MfOvG5DmTrV0jV2Zg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1706926802; a=rsa-sha256; cv=none; b=iY0bGE0hlhgJUq9hmfDKaq0Z8kYL3t/R454OIOpDr39dp/AkYneAnpz4QhD6wMmhU+OugS vkn2fnQMXfyJM9/i8id/BUFvClJfmtIaqbgBjtHXDMqzbLzA5HTSwlLD07jkzSrFqoh0ME 8rvZ7avWslvbiyPhpjTxaQWAe35jm/o0+bpq4wPpwZwISQFiActxOAiT0oK52AoH1W4DRd SNcXiV0i3f4dZuyj+jVCtjsu6beZjrGTKPsJgO4V/eIJsG/5JS2pbrWZtfCeJiEy6uBsDL 3j2KdsOheYYIoDczKTVWuLtIsK8RQiZaj7x9MU3JYfwKB0sgfWqELAhIz4heeA== Received: from auth1-smtp.messagingengine.com (auth1-smtp.messagingengine.com [66.111.4.227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TRbvQ0Dp2z15Vv; Sat, 3 Feb 2024 02:20:02 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailauth.nyi.internal (Postfix) with ESMTP id E525E27C0061; Fri, 2 Feb 2024 21:20:01 -0500 (EST) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Fri, 02 Feb 2024 21:20:01 -0500 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfeduhedggeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerredtnecuhfhrohhmpedfffhr vgifucfirghllhgrthhinhdfuceoghgrlhhlrghtihhnsehfrhgvvggsshgurdhorhhgqe enucggtffrrghtthgvrhhnpeeggfeugeevuedtuedvleefffduteegtdffudeihefhgfeg feekffeiueevkeeuudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehgrghllhgrthhinhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihht hidqudeffeehledvvdduiedqvdelhedtgedukeegqdhgrghllhgrthhinheppehfrhgvvg gsshgurdhorhhgsehfrghsthhmrghilhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id A1D73364006B; Fri, 2 Feb 2024 21:20:01 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-transport List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-transport@freebsd.org X-BeenThere: freebsd-transport@freebsd.org MIME-Version: 1.0 Message-Id: <2fac0ac3-ba3a-4bca-b0d4-fafb0c5b75fd@app.fastmail.com> In-Reply-To: References: <2c31ac44-b34b-469c-a6de-fdd927ec2f9e@freebsd.org> Date: Fri, 02 Feb 2024 21:19:41 -0500 From: "Drew Gallatin" To: "Rick Macklem" Cc: "Richard Scheffenegger" , "freebsd-net@FreeBSD.org" , "FreeBSD Transport" , rmacklem@freebsd.org, kp@freebsd.org Subject: Re: Increasing TCP TSO size support Content-Type: multipart/alternative; boundary=40239dadffc4465dbc528566ad3b21da --40239dadffc4465dbc528566ad3b21da Content-Type: text/plain On Fri, Feb 2, 2024, at 9:05 PM, Rick Macklem wrote: > > But the page size is only 4K on most platforms. So while an M_EXTPGS mbuf can hold 5 pages (..from memory, too lazy to do the math right now) and reduces socket buffer mbuf chain lengths by a factor of 10 or so (2k vs 20k per mbuf), the S/G list that a NIC will need to consume would likely decrease only by a factor of 2. And even then only if the busdma code to map mbufs for DMA is not coalescing adjacent mbufs. I know busdma does some coalescing, but I can't recall if it coalesces physcally adjacent mbufs. > > I'm guessing the factor of 2 comes from the fact that each page is a > contiguous segment? Actually, no, I'm being dumb. I was thinking that pages would be split up, but that's wrong. Without M_EXTPGS, each mbuf generated by sendfile (or nfs) would be an M_EXT with a wrapper around a single 4K page. So the scatter/gather list would be exactly the same. The win would be if the pages themselves were contiguous (which they often are), and if the bus_dma mbuf mapping code coalesced those segments, and if the device could handle DMA across a 4K boundary. That's what would get you shorter s/g lists. I think tcp_m_copy() can handle this now, as if_hw_tsomaxsegsize is set by the driver to express how long the max contiguous segment they can handle is. BTW, I really hate the mixing of bus dma restrictions with the hw_tsomax stuff. It always makes my head explode.. Drew --40239dadffc4465dbc528566ad3b21da Content-Type: text/html Content-Transfer-Encoding: quoted-printable

=
On Fri, Feb 2, 2024, at 9:05 PM, Rick Macklem wrote:
<= /div>
> But the pa= ge size is only 4K on most platforms.  So while an M_EXTPGS mbuf ca= n hold 5 pages (..from memory, too lazy to do the math right now) and re= duces socket buffer mbuf chain lengths by a factor of 10 or so (2k vs 20= k per mbuf), the S/G list that a NIC will need to consume would likely d= ecrease only by a factor of 2.  And even then only if the busdma co= de to map mbufs for DMA is not coalescing adjacent mbufs.  I know b= usdma does some coalescing, but I can't recall if it coalesces physcally= adjacent mbufs.

I'm guessing the factor of= 2 comes from the fact that each page is a
contiguous segm= ent?

Actually, no, I'm being d= umb.  I was thinking that pages would be split up, but that's wrong= .  Without M_EXTPGS, each mbuf generated by sendfile (or nfs) would= be an M_EXT with a wrapper around a single 4K page.  So the scatte= r/gather list would be exactly the same.

Th= e win would be if the pages themselves were contiguous (which they often= are), and if the bus_dma mbuf mapping code coalesced those segments, an= d if the device could handle DMA across a 4K boundary.  That's what= would get you shorter s/g lists.

I think tcp_m_copy()= can handle this now, as if_hw_tsomaxsegsize is set by the driver to exp= ress how long the max contiguous segment they can handle is.

BTW, I really hate the mixing of bus dma restrictions = with the hw_tsomax stuff.  It always makes my head explode..

Drew

--40239dadffc4465dbc528566ad3b21da--