From owner-svn-src-stable@freebsd.org Thu Jun 14 16:19:06 2018 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6A8A1010FB9; Thu, 14 Jun 2018 16:19:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C3B77248F; Thu, 14 Jun 2018 16:19:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3EBFC2B82; Thu, 14 Jun 2018 16:19:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w5EGJ6fC062467; Thu, 14 Jun 2018 16:19:06 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w5EGJ6SK062466; Thu, 14 Jun 2018 16:19:06 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201806141619.w5EGJ6SK062466@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 14 Jun 2018 16:19:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r335149 - stable/11/sys/dev/nvme X-SVN-Group: stable-11 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: stable/11/sys/dev/nvme X-SVN-Commit-Revision: 335149 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jun 2018 16:19:06 -0000 Author: mav Date: Thu Jun 14 16:19:05 2018 New Revision: 335149 URL: https://svnweb.freebsd.org/changeset/base/335149 Log: MFC r330954, r330955 (by imp): When tearing down a queue pair, also delete the queue entries. The NVME standard has required in section 7.2.6, since at least 1.1, that a clean shutdown is signalled by deleting the subission and the completion queues before setting the shutdown bit in CC. The 1.0 standard, apparently, did not and many of the early Intel cards didn't care. Some newer cards care, at least one whose beta firmware can scramble the card on an unclean shutdown. Linux has done this for some time. To make it possible to move forward with an evaluation of this pre-release card with wonky firmware, delete the queues on the card when we delete the qpair structures. Modified: stable/11/sys/dev/nvme/nvme_ctrlr.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/dev/nvme/nvme_ctrlr.c ============================================================================== --- stable/11/sys/dev/nvme/nvme_ctrlr.c Thu Jun 14 16:11:50 2018 (r335148) +++ stable/11/sys/dev/nvme/nvme_ctrlr.c Thu Jun 14 16:19:05 2018 (r335149) @@ -493,6 +493,34 @@ nvme_ctrlr_create_qpairs(struct nvme_controller *ctrlr } static int +nvme_ctrlr_destroy_qpair(struct nvme_controller *ctrlr, struct nvme_qpair *qpair) +{ + struct nvme_completion_poll_status status; + + status.done = 0; + nvme_ctrlr_cmd_delete_io_sq(ctrlr, qpair, + nvme_completion_poll_cb, &status); + while (!atomic_load_acq_int(&status.done)) + pause("nvme", 1); + if (nvme_completion_is_error(&status.cpl)) { + nvme_printf(ctrlr, "nvme_destroy_io_sq failed!\n"); + return (ENXIO); + } + + status.done = 0; + nvme_ctrlr_cmd_delete_io_cq(ctrlr, qpair, + nvme_completion_poll_cb, &status); + while (!atomic_load_acq_int(&status.done)) + pause("nvme", 1); + if (nvme_completion_is_error(&status.cpl)) { + nvme_printf(ctrlr, "nvme_destroy_io_cq failed!\n"); + return (ENXIO); + } + + return (0); +} + +static int nvme_ctrlr_construct_namespaces(struct nvme_controller *ctrlr) { struct nvme_namespace *ns; @@ -1216,6 +1244,7 @@ nvme_ctrlr_destruct(struct nvme_controller *ctrlr, dev destroy_dev(ctrlr->cdev); for (i = 0; i < ctrlr->num_io_queues; i++) { + nvme_ctrlr_destroy_qpair(ctrlr, &ctrlr->ioq[i]); nvme_io_qpair_destroy(&ctrlr->ioq[i]); }