From owner-dev-commits-src-all@freebsd.org Thu Aug 12 16:06:15 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E72F466E9C3; Thu, 12 Aug 2021 16:06:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gls4R65Lrz3DDr; Thu, 12 Aug 2021 16:06:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B8B7025A74; Thu, 12 Aug 2021 16:06:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 17CG6FK8038817; Thu, 12 Aug 2021 16:06:15 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 17CG6FWj038816; Thu, 12 Aug 2021 16:06:15 GMT (envelope-from git) Date: Thu, 12 Aug 2021 16:06:15 GMT Message-Id: <202108121606.17CG6FWj038816@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 2eb0e53a6b5e - main - cxgbei: Wait for the final CPL to be received in icl_cxgbei_conn_close. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 2eb0e53a6b5ec1a72be70e966d4e562e1a8d4e88 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Aug 2021 16:06:16 -0000 The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=2eb0e53a6b5ec1a72be70e966d4e562e1a8d4e88 commit 2eb0e53a6b5ec1a72be70e966d4e562e1a8d4e88 Author: John Baldwin AuthorDate: 2021-08-12 15:48:14 +0000 Commit: John Baldwin CommitDate: 2021-08-12 15:48:35 +0000 cxgbei: Wait for the final CPL to be received in icl_cxgbei_conn_close. A socket in the FIN_WAIT_1 state is marked disconnected by do_close_con_rpl() even though there might still receive data pending. This is because the socket at that point has set SBS_CANTRCVMORE which causes the protocol layer to discard any data received before the FIN. However, icl_cxgbei_conn_close needs to wait until all the data has been discarded. Replace the wait for SS_ISDISCONNECTED with instead waiting for final_cpl_received() to be called. Reported by: Jithesh Arakkan @ Chelsio Sponsored by: Chelsio Communications --- sys/dev/cxgbe/cxgbei/icl_cxgbei.c | 28 +++++++++++++++++++++------- sys/dev/cxgbe/tom/t4_tom.c | 12 +++++++++++- sys/dev/cxgbe/tom/t4_tom.h | 1 + 3 files changed, 33 insertions(+), 8 deletions(-) diff --git a/sys/dev/cxgbe/cxgbei/icl_cxgbei.c b/sys/dev/cxgbe/cxgbei/icl_cxgbei.c index a57d26ae21b8..5526388915f7 100644 --- a/sys/dev/cxgbe/cxgbei/icl_cxgbei.c +++ b/sys/dev/cxgbe/cxgbei/icl_cxgbei.c @@ -947,6 +947,18 @@ icl_cxgbei_conn_close(struct icl_conn *ic) icl_cxgbei_pdu_done(ip, ENOTCONN); } SOCKBUF_UNLOCK(sb); + + /* + * Grab a reference to use when waiting for the final + * CPL to be received. If toep->inp is NULL, then + * final_cpl_received() has already been called (e.g. + * due to the peer sending a RST). + */ + if (toep->inp != NULL) { + toep = hold_toepcb(toep); + toep->flags |= TPF_WAITING_FOR_FINAL; + } else + toep = NULL; } INP_WUNLOCK(inp); @@ -959,7 +971,6 @@ icl_cxgbei_conn_close(struct icl_conn *ic) * queues were purged instead of delivered reliably but soabort isn't * really general purpose and wouldn't do the right thing here. */ - soref(so); soclose(so); /* @@ -969,12 +980,15 @@ icl_cxgbei_conn_close(struct icl_conn *ic) * Callers assume that it is safe to free buffers for tasks * and transfers after this function returns. */ - SOCK_LOCK(so); - while ((so->so_state & SS_ISDISCONNECTED) == 0) - mtx_sleep(&so->so_timeo, SOCK_MTX(so), PSOCK, "conclo2", 0); - CURVNET_SET(so->so_vnet); - sorele(so); - CURVNET_RESTORE(); + if (toep != NULL) { + struct mtx *lock = mtx_pool_find(mtxpool_sleep, toep); + + mtx_lock(lock); + while ((toep->flags & TPF_WAITING_FOR_FINAL) != 0) + mtx_sleep(toep, lock, PSOCK, "conclo2", 0); + mtx_unlock(lock); + free_toepcb(toep); + } } static void diff --git a/sys/dev/cxgbe/tom/t4_tom.c b/sys/dev/cxgbe/tom/t4_tom.c index 1d065ade77c4..a444f0c9d690 100644 --- a/sys/dev/cxgbe/tom/t4_tom.c +++ b/sys/dev/cxgbe/tom/t4_tom.c @@ -1008,6 +1008,7 @@ void final_cpl_received(struct toepcb *toep) { struct inpcb *inp = toep->inp; + bool need_wakeup; KASSERT(inp != NULL, ("%s: inp is NULL", __func__)); INP_WLOCK_ASSERT(inp); @@ -1022,7 +1023,8 @@ final_cpl_received(struct toepcb *toep) else if (ulp_mode(toep) == ULP_MODE_TLS) tls_detach(toep); toep->inp = NULL; - toep->flags &= ~TPF_CPL_PENDING; + need_wakeup = (toep->flags & TPF_WAITING_FOR_FINAL) != 0; + toep->flags &= ~(TPF_CPL_PENDING | TPF_WAITING_FOR_FINAL); mbufq_drain(&toep->ulp_pduq); mbufq_drain(&toep->ulp_pdu_reclaimq); @@ -1031,6 +1033,14 @@ final_cpl_received(struct toepcb *toep) if (!in_pcbrele_wlocked(inp)) INP_WUNLOCK(inp); + + if (need_wakeup) { + struct mtx *lock = mtx_pool_find(mtxpool_sleep, toep); + + mtx_lock(lock); + wakeup(toep); + mtx_unlock(lock); + } } void diff --git a/sys/dev/cxgbe/tom/t4_tom.h b/sys/dev/cxgbe/tom/t4_tom.h index 10d9cfe87960..dc462d4b4a66 100644 --- a/sys/dev/cxgbe/tom/t4_tom.h +++ b/sys/dev/cxgbe/tom/t4_tom.h @@ -76,6 +76,7 @@ enum { TPF_INITIALIZED = (1 << 12), /* init_toepcb has been called */ TPF_TLS_RECEIVE = (1 << 13), /* should receive TLS records */ TPF_TLS_ESTABLISHED = (1 << 14), /* TLS handshake timer initialized */ + TPF_WAITING_FOR_FINAL = (1<< 15), /* waiting for wakeup on final CPL */ }; enum {