From nobody Wed Jul 5 22:05:04 2023 X-Original-To: dev-commits-doc-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QxDH45qt3z4lpb3 for ; Wed, 5 Jul 2023 22:05:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QxDH45KW4z40nQ; Wed, 5 Jul 2023 22:05:04 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688594704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BKzTUY8ncy9s5BoyE2ShI9Vi14Z7Suo0BhZsF6n2GWk=; b=w5ogMlAmxA1CPSP7kdSFzNzii9pwk2z8o5l26OX+qgbtPFusd+1MxfvXaN6aQdHU6tZUqM qZ8avCrs6jSkn7ouHD+SqVtEgj3OsJ00Y5A78x1vNJlJBYQmZbUNfegWlP2q3Lw8tyxKEV tdWc7KI5kjhbae0F9V8kzHZKuxj0qOipB6uZWyChPV+WvAhkdc7r4Pr+iNt3swilc7GFF9 wpf7LfMv3nhGRDyo9xjeIJ5GQcCDNUEHudWUmM9yZsUK4RAJInmipdL0t/NzeM7S5RF8qd 30W53rL3krA7vYW0pKC6xi91cetnz9qwpOxJHf2SV0ba/3SBZQYeU8k9Xh8Yhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688594704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=BKzTUY8ncy9s5BoyE2ShI9Vi14Z7Suo0BhZsF6n2GWk=; b=pZ1Vb4+ElD292J0uWdd7mGJFhDPia011YE/GFQf0oM4odyFalYLWcb+gzohoA1iwRXEs4a bTpRxyV7L/xLA5bqzdUQgZ63f2pkO0SzW8DamvOPV4VnqdHV6sdIizTPjqACAsgGMIWBuM v/WIfrk7SylYcTQZVioQ05Pyk6x9eJocmbesgRlp5DIVUzhBGy0ABMmtm7rKHdamyamBce EKNHutIT48cqghQ+F/fwzgWbr1wCyFX8x2m9lLy5orvKnjrYy/E4BvkY6gEYZsze7SSAAu uWkGbyiV9KvKNbfE/LwQavU7+6Nwcvn4RJEXKcd2V+EviqH9E+u4oJ56l25ATw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688594704; a=rsa-sha256; cv=none; b=VwMTtiIQSs4L7tH3/UmAX7lxNT66y2Z9X4xss8nWy95+LQrhkVoPt2RJqYMlDMechC/szh PD5GNdM+TkYGV0d89xEIR4BhvlRz4iDteV/UCZHW8bsZtJQzYLo5n911727Kohur2CR9x9 O25+SiFK3CPdCtsHU1iwPYYrD2N6TWtCi9E4WR5NLNb3PRwLMge5OWoYqkF5vd1KmpP/Yw q8CknGkW21C9NT5dBgDnu1TKE0LWnf6f1yBrjLdvq1ET7CBxobADUcmatIuCq3azESPvrn +HCqrOn5QuOLbY+jokXYrvCPUhOlw1Its01SnlqQL4adjbiya0DZ8cr6ADV2QQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QxDH44NJRzXBW; Wed, 5 Jul 2023 22:05:04 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 365M546P062350; Wed, 5 Jul 2023 22:05:04 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 365M5422062349; Wed, 5 Jul 2023 22:05:04 GMT (envelope-from git) Date: Wed, 5 Jul 2023 22:05:04 GMT Message-Id: <202307052205.365M5422062349@gitrepo.freebsd.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org From: John Baldwin Subject: git: 3ba5fc40d0 - main - 2023Q2 status report for NVMe over Fabrics List-Id: Commit messages for all branches of the doc repository List-Archive: https://lists.freebsd.org/archives/dev-commits-doc-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-doc-all@freebsd.org X-BeenThere: dev-commits-doc-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: doc X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 3ba5fc40d0f5a946ffb89268c319c8967365d5fb Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/doc/commit/?id=3ba5fc40d0f5a946ffb89268c319c8967365d5fb commit 3ba5fc40d0f5a946ffb89268c319c8967365d5fb Author: John Baldwin AuthorDate: 2023-07-05 22:04:31 +0000 Commit: John Baldwin CommitDate: 2023-07-05 22:04:31 +0000 2023Q2 status report for NVMe over Fabrics Reviewed by: Pau Amma , salvadore Differential Revision: https://reviews.freebsd.org/D40792 --- .../en/status/report-2023-04-2023-06/nvmf.adoc | 71 ++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/website/content/en/status/report-2023-04-2023-06/nvmf.adoc b/website/content/en/status/report-2023-04-2023-06/nvmf.adoc new file mode 100644 index 0000000000..445119c7f9 --- /dev/null +++ b/website/content/en/status/report-2023-04-2023-06/nvmf.adoc @@ -0,0 +1,71 @@ +=== NVMe over Fabrics + +Links: + +link:https://github.com/bsdjhb/freebsd/tree/nvmf2[nvmf2 branch] URL: link:https://github.com/bsdjhb/freebsd/tree/nvmf2[] + +Contact: John Baldwin + +NVMe over Fabrics enables communication with a storage device using +the NVMe protocol over a network fabric. +This is similar to using iSCSI to export a storage device over a +network using SCSI commands. + +NVMe over Fabrics currently defines network transports for +Fibre Channel, RDMA, and TCP. + +The work in the nvmf2 branch includes a userland library (lib/libnvmf) +which contains an abstraction for transports and an implementation of +a TCP transport. +It also includes changes to man:nvmecontrol[8] to add 'discover', +'connect', and 'disconnect' commands to manage connections to a remote +controller. + +The branch also contains an in-kernel Fabrics implementation. +[.filename]#nvmf_transport.ko# contains a transport abstraction that +sits in between the nvmf host (initiator in SCSI terms) and the +individual transports. +[.filename]#nvmf_tcp.ko# contains an implementation of the TCP +transport layer. +[.filename]#nvmf.ko# contains an NVMe over Fabrics host (initiator) +which connects to a remote controller and exports remote namespaces as +disk devices. +Similar to the man:nvme[4] driver for NVMe over PCI-express, +namespaces are exported via [.filename]#/dev/nvmeXnsY# devices which +only support simple operations, but are also exported as ndaX disk +devices via CAM. +Unlike man:nvme[4], man:nvmf[4] does not support the man:nvd[4] disk +driver. +nvmecontrol can be used with remote namespaces and remote controllers, +for example to fetch log pages, display identify info, etc. + +Note that man:nvmf[4] is currently a bit simple and some error cases +are still a TODO. +If an error occurs, the queues (and backing network connections) are +dropped, but the devices stay around, but with I/O requests paused. +'nvmecontrol reconnect' can be used to connect a new set of network +connections to resume operation. +Unlike iSCSI which uses a persistent daemon (man:iscsid[8]) to +reconnect after an error, reconnections must be done manually. + +The current code is very new and likely not robust. +It is certainly not ready for production use. +Experienced users who do not mind all their data vanishing in a puff +of smoke after a kernel panic and who have an interest in NVMe over +Fabrics can start testing it at their own risk. + +The next main task is to implement a Fabrics controller (target in +SCSI language). +Probably a simple one in userland first followed by a "real" one that +offloads the data handling to the kernel but is somewhat integrated +with man:ctld[8] so that individual disk devices can be exported +either via iSCSI or NVMe or both using a single config file and daemon +to manage all of that. +This may require a fair bit of refactoring in ctld to make it less +iSCSI-specific. +Working on the controller side will also validate some of the +currently under-tested API design decisions in the +transport-independent layer. +I think it probably does not make sense to merge any of the NVMe over +Fabrics changes into the tree until after this step. + +Sponsored by: Chelsio Communications