From owner-freebsd-virtualization@freebsd.org Mon May 18 21:13:07 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2014D2DD974; Mon, 18 May 2020 21:13:07 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49QsDf2Hmfz4Nmg; Mon, 18 May 2020 21:13:06 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: by mail-pl1-f170.google.com with SMTP id t7so4756378plr.0; Mon, 18 May 2020 14:13:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=9EXpGAr8AtJBewCN/0kKauPjulQUQgJAAADngV5FrwQ=; b=FxRtNCxFc2zGB+CY4d8PgYxGdoG7ozBrZ7L7dDT3TQr4x7N3haa7Z3JqM65WQt2NwH hiqNKv+7w859vTrcEhY0wqW1TXlHeDAg6wBA6GZGmF2k5GtY/alNxEu6e0exZ1nkUPRD sn5hWLjX5rU3CX2+J16qDM6AEeNEofKc9vsp3Gw0M3uqKSHfy1A73mfyoq1MKyiyWkHd ukmDefk753tE7KpaJ30UQjc5dx+4KEXHn2ynUV9PrHVcj+Fqu+abidR3xy51NVSS+Kl4 9PdUprVC6hLQiY6Tgfp5a0DLZjqGD/i72H4fgrW9Sb5Hxxe4adKcBn6cmKt0mnlPiwCK BReA== X-Gm-Message-State: AOAM531pT9DiKzUbzNS9t9XP+MPsbc5tO6WVZg7c3ujaNZgQuZnFsS6/ YDCYfV6yg8uwhx9xY07Vv/3IJEVo X-Google-Smtp-Source: ABdhPJzcJe084jUH5gY+86M2JSsZl6icl+CJhSBIunbEW3Ul3VIlyUsXl5W2eKOfPRh1pLveIXthXQ== X-Received: by 2002:a17:90a:8c8e:: with SMTP id b14mr1307288pjo.222.1589836384366; Mon, 18 May 2020 14:13:04 -0700 (PDT) Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com. [209.85.214.176]) by smtp.gmail.com with ESMTPSA id g9sm8349825pgj.89.2020.05.18.14.13.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 May 2020 14:13:04 -0700 (PDT) Received: by mail-pl1-f176.google.com with SMTP id b12so4724175plz.13; Mon, 18 May 2020 14:13:04 -0700 (PDT) X-Received: by 2002:a17:90a:9292:: with SMTP id n18mr1386473pjo.3.1589836383769; Mon, 18 May 2020 14:13:03 -0700 (PDT) MIME-Version: 1.0 From: Chuck Tuffli Date: Mon, 18 May 2020 14:12:52 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Updates to bhyve NVMe emulation To: freebsd-virtualization@freebsd.org, freebsd-scsi@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 49QsDf2Hmfz4Nmg X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ctuffli@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ctuffli@gmail.com X-Spamd-Result: default: False [0.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.214.170:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_SHORT(0.29)[0.287]; RCVD_COUNT_THREE(0.00)[4]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.214.170:from]; FORGED_SENDER(0.30)[chuck@freebsd.org,ctuffli@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[chuck@freebsd.org,ctuffli@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 May 2020 21:13:07 -0000 In an effort to provide better NVMe emulation in bhyve, I've started running the NVM Express Compliance tests against the bhyve emulation and fixing issues the tests find. The tests are part of the University of New Hampshire (UNH) Interoperability Lab (IOL) suite. I have posted a series of patches in Phabricator and would appreciate any feedback people might have. And because there are a few changes, the plan is to leave them up for review longer than I normally do. Note that I've cross-posted to the SCSI list to possibly pickup NVMe expertise. Summary of the patch series with links to differential reviews: bhyve: convert NVMe logging statements usr.sbin/bhyve/pci_nvme.c 1 files changed, 157 insertions(+), 157 deletions(-) Differential Revision: https://reviews.freebsd.org/D24878 bhyve: refactor NVMe IO command handling usr.sbin/bhyve/pci_nvme.c 1 files changed, 175 insertions(+), 180 deletions(-) Differential Revision: https://reviews.freebsd.org/D24879 bhyve: implement NVMe Flush command usr.sbin/bhyve/pci_nvme.c 1 files changed, 43 insertions(+), 1 deletions(-) Differential Revision: https://reviews.freebsd.org/D24880 bhyve: add a comment explaining NVME dsm option usr.sbin/bhyve/pci_nvme.c 1 files changed, 2 insertions(+), 1 deletions(-) Differential Revision: https://reviews.freebsd.org/D24881 bhyve: add locks around NVMe queue accesses usr.sbin/bhyve/pci_nvme.c 1 files changed, 104 insertions(+), 61 deletions(-) Differential Revision: https://reviews.freebsd.org/D19841 bhyve: Consolidate NVMe CQ update usr.sbin/bhyve/pci_nvme.c 1 files changed, 51 insertions(+), 49 deletions(-) Differential Revision: https://reviews.freebsd.org/D24882 bhyve: implement NVMe Namespace Identification Descriptor usr.sbin/bhyve/pci_nvme.c 1 files changed, 20 insertions(+), 11 deletions(-) Differential Revision: https://reviews.freebsd.org/D24884 bhyve: fix NVMe Get Log Page command usr.sbin/bhyve/pci_nvme.c 1 files changed, 15 insertions(+), 5 deletions(-) Differential Revision: https://reviews.freebsd.org/D24885 bhyve: fix NVMe queue creation and deletion usr.sbin/bhyve/pci_nvme.c 1 files changed, 89 insertions(+), 34 deletions(-) Differential Revision: https://reviews.freebsd.org/D24886 bhyve: add more compliant NVMe Get/Set Features usr.sbin/bhyve/pci_nvme.c 1 files changed, 214 insertions(+), 152 deletions(-) Differential Revision: https://reviews.freebsd.org/D24887 bhyve: make unsupported NVMe commands a debug message usr.sbin/bhyve/pci_nvme.c 1 files changed, 1 insertions(+), 1 deletions(-) Differential Revision: https://reviews.freebsd.org/D24888 bhyve: implement NVMe Format NVM command usr.sbin/bhyve/pci_nvme.c 1 files changed, 78 insertions(+), 0 deletions(-) Differential Revision: https://reviews.freebsd.org/D24889 bhyve: refactor NVMe I/O read/write usr.sbin/bhyve/pci_nvme.c 1 files changed, 108 insertions(+), 78 deletions(-) Differential Revision: https://reviews.freebsd.org/D24890 bhyve: base pci_nvme_ioreq size on advertised MDTS usr.sbin/bhyve/pci_nvme.c 1 files changed, 60 insertions(+), 92 deletions(-) Differential Revision: https://reviews.freebsd.org/D24891 bhyve: validate NVMe deallocate range values usr.sbin/bhyve/pci_nvme.c 1 files changed, 9 insertions(+), 0 deletions(-) Differential Revision: https://reviews.freebsd.org/D24893 bhyve: implement NVMe SMART data I/O statistics usr.sbin/bhyve/pci_nvme.c 1 files changed, 71 insertions(+), 1 deletions(-) Differential Revision: https://reviews.freebsd.org/D24894 bhyve: validate the NVMe LBA start and count usr.sbin/bhyve/pci_nvme.c 1 files changed, 54 insertions(+), 15 deletions(-) Differential Revision: https://reviews.freebsd.org/D24895 bhyve: Add AER support to NVMe emulation usr.sbin/bhyve/pci_nvme.c 1 files changed, 115 insertions(+), 6 deletions(-) Differential Revision: https://reviews.freebsd.org/D24896 bhyve: add basic NVMe Firmware Commit support usr.sbin/bhyve/pci_nvme.c 1 files changed, 9 insertions(+), 0 deletions(-) Differential Revision: https://reviews.freebsd.org/D24897 bhyve: add NVMe Feature Interrupt Vector Config usr.sbin/bhyve/pci_nvme.c 1 files changed, 39 insertions(+), 0 deletions(-) Differential Revision: https://reviews.freebsd.org/D24898 bhyve: fix NVMe Get Features, Predictable Latency usr.sbin/bhyve/pci_nvme.c 1 files changed, 4 insertions(+), 0 deletions(-) Differential Revision: https://reviews.freebsd.org/D24899 bhyve: NVMe handle zero length DSM ranges usr.sbin/bhyve/pci_nvme.c 1 files changed, 45 insertions(+), 33 deletions(-) Differential Revision: https://reviews.freebsd.org/D24900 bhyve: fix NVMe Active Namespace list usr.sbin/bhyve/pci_nvme.c 1 files changed, 2 insertions(+), 1 deletions(-) Differential Revision: https://reviews.freebsd.org/D24901 --chuck