From owner-svn-src-stable@freebsd.org Mon Apr 16 17:24:34 2018 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30EBDF95CE6; Mon, 16 Apr 2018 17:24:34 +0000 (UTC) (envelope-from trasz@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D20017FECA; Mon, 16 Apr 2018 17:24:33 +0000 (UTC) (envelope-from trasz@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C8D9A205AB; Mon, 16 Apr 2018 17:24:33 +0000 (UTC) (envelope-from trasz@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w3GHOXfp031503; Mon, 16 Apr 2018 17:24:33 GMT (envelope-from trasz@FreeBSD.org) Received: (from trasz@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w3GHOXcE031501; Mon, 16 Apr 2018 17:24:33 GMT (envelope-from trasz@FreeBSD.org) Message-Id: <201804161724.w3GHOXcE031501@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: trasz set sender to trasz@FreeBSD.org using -f From: Edward Tomasz Napierala Date: Mon, 16 Apr 2018 17:24:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r332622 - stable/11/sys/cam/ctl X-SVN-Group: stable-11 X-SVN-Commit-Author: trasz X-SVN-Commit-Paths: stable/11/sys/cam/ctl X-SVN-Commit-Revision: 332622 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Apr 2018 17:24:34 -0000 Author: trasz Date: Mon Apr 16 17:24:33 2018 New Revision: 332622 URL: https://svnweb.freebsd.org/changeset/base/332622 Log: MFC r331013: Fix iSCSI target crash on session reinstation. The crash scenario goes like this: there's a thread waiting on "reinstate"; because it doesn't update the timeout counter it gets terminated by the callout; at this point the maintenance thread starts the termination routine. The first thread finishes waiting, proceeds to icl_conn_handoff(), and drops the refcount, which allows the maintenance thread to free its resources. At this point another thread receives a PDU. Boom. PR: 222898, 219866 Sponsored by: playkey.net Modified: stable/11/sys/cam/ctl/ctl_frontend_iscsi.c stable/11/sys/cam/ctl/ctl_frontend_iscsi.h Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/cam/ctl/ctl_frontend_iscsi.c ============================================================================== --- stable/11/sys/cam/ctl/ctl_frontend_iscsi.c Mon Apr 16 17:22:51 2018 (r332621) +++ stable/11/sys/cam/ctl/ctl_frontend_iscsi.c Mon Apr 16 17:24:33 2018 (r332622) @@ -1162,11 +1162,11 @@ cfiscsi_maintenance_thread(void *arg) for (;;) { CFISCSI_SESSION_LOCK(cs); - if (cs->cs_terminating == false) + if (cs->cs_terminating == false || cs->cs_handoff_in_progress) cv_wait(&cs->cs_maintenance_cv, &cs->cs_lock); CFISCSI_SESSION_UNLOCK(cs); - if (cs->cs_terminating) { + if (cs->cs_terminating && cs->cs_handoff_in_progress == false) { /* * We used to wait up to 30 seconds to deliver queued @@ -1194,8 +1194,6 @@ static void cfiscsi_session_terminate(struct cfiscsi_session *cs) { - if (cs->cs_terminating) - return; cs->cs_terminating = true; cv_signal(&cs->cs_maintenance_cv); #ifdef ICL_KERNEL_PROXY @@ -1266,6 +1264,13 @@ cfiscsi_session_new(struct cfiscsi_softc *softc, const cv_init(&cs->cs_login_cv, "cfiscsi_login"); #endif + /* + * The purpose of this is to avoid racing with session shutdown. + * Otherwise we could have the maintenance thread call icl_conn_close() + * before we call icl_conn_handoff(). + */ + cs->cs_handoff_in_progress = true; + cs->cs_conn = icl_new_conn(offload, false, "cfiscsi", &cs->cs_lock); if (cs->cs_conn == NULL) { free(cs, M_CFISCSI); @@ -1376,8 +1381,18 @@ cfiscsi_accept(struct socket *so, struct sockaddr *sa, icl_conn_handoff_sock(cs->cs_conn, so); cs->cs_initiator_sa = sa; cs->cs_portal_id = portal_id; + cs->cs_handoff_in_progress = false; cs->cs_waiting_for_ctld = true; cv_signal(&cfiscsi_softc.accept_cv); + + CFISCSI_SESSION_LOCK(cs); + /* + * Wake up the maintenance thread if we got scheduled for termination + * somewhere between cfiscsi_session_new() and icl_conn_handoff_sock(). + */ + if (cs->cs_terminating) + cfiscsi_session_terminate(cs); + CFISCSI_SESSION_UNLOCK(cs); } #endif @@ -1556,6 +1571,7 @@ cfiscsi_ioctl_handoff(struct ctl_iscsi *ci) mtx_lock(&softc->lock); if (ct->ct_online == 0) { mtx_unlock(&softc->lock); + cs->cs_handoff_in_progress = false; cfiscsi_session_terminate(cs); cfiscsi_target_release(ct); ci->status = CTL_ISCSI_ERROR; @@ -1566,7 +1582,6 @@ cfiscsi_ioctl_handoff(struct ctl_iscsi *ci) cs->cs_target = ct; mtx_unlock(&softc->lock); - refcount_acquire(&cs->cs_outstanding_ctl_pdus); restart: if (!cs->cs_terminating) { mtx_lock(&softc->lock); @@ -1603,8 +1618,8 @@ restart: #endif error = icl_conn_handoff(cs->cs_conn, cihp->socket); if (error != 0) { + cs->cs_handoff_in_progress = false; cfiscsi_session_terminate(cs); - refcount_release(&cs->cs_outstanding_ctl_pdus); ci->status = CTL_ISCSI_ERROR; snprintf(ci->error_str, sizeof(ci->error_str), "%s: icl_conn_handoff failed with error %d", @@ -1629,7 +1644,16 @@ restart: } #endif - refcount_release(&cs->cs_outstanding_ctl_pdus); + CFISCSI_SESSION_LOCK(cs); + cs->cs_handoff_in_progress = false; + + /* + * Wake up the maintenance thread if we got scheduled for termination. + */ + if (cs->cs_terminating) + cfiscsi_session_terminate(cs); + CFISCSI_SESSION_UNLOCK(cs); + ci->status = CTL_ISCSI_OK; } Modified: stable/11/sys/cam/ctl/ctl_frontend_iscsi.h ============================================================================== --- stable/11/sys/cam/ctl/ctl_frontend_iscsi.h Mon Apr 16 17:22:51 2018 (r332621) +++ stable/11/sys/cam/ctl/ctl_frontend_iscsi.h Mon Apr 16 17:24:33 2018 (r332622) @@ -83,6 +83,7 @@ struct cfiscsi_session { int cs_timeout; struct cv cs_maintenance_cv; bool cs_terminating; + bool cs_handoff_in_progress; bool cs_tasks_aborted; size_t cs_max_data_segment_length; size_t cs_max_burst_length;