From nobody Sat Apr 9 22:50:46 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E180F1A9DA8D; Sat, 9 Apr 2022 22:50:50 +0000 (UTC) (envelope-from rpokala@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KbVhV66D3z4sdC; Sat, 9 Apr 2022 22:50:50 +0000 (UTC) (envelope-from rpokala@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649544650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QYb6dH6gJQMYDfvRgqRmCKgE+msJEq6ECvDAKmR4u4U=; b=RahaVPpUxE1xlg5viOKWVHlQP6Aiw1s2cjw7DdfeUfhvVR/bzAQFYYu8IO7sxvIkht9laD QoyS0MpltqS3As64Ot0vMWAQCcbYN6+2MHJxWStFT7yloqjugmTVHL38g055iHTX/PTcus 81hRkt4Rm9hY8MOcXNHfnijHryt4AQJQHzvDfOMaGkHuCi/zQXpsC7cX7i9qSPS6mh2mCs t7vUlDeaOq+i1BQ6fUuV0Q2BoG7LEo8n9+qqr+7tmo+JacgVy+5mMD54kIJdnjUhJtVwf1 ovOyzXgB1wy9QaiDMg4NDUcBNAbrXIed7csGGpSJ+ZkYv9aDuNZzh2BhAq2+3w== Received: from [192.168.1.10] (unknown [IPv6:2601:641:700:5284:3dba:62a2:3333:eee8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: rpokala) by smtp.freebsd.org (Postfix) with ESMTPSA id 301D6D201; Sat, 9 Apr 2022 22:50:50 +0000 (UTC) (envelope-from rpokala@freebsd.org) User-Agent: Microsoft-MacOutlook/16.59.22031300 Date: Sat, 09 Apr 2022 15:50:46 -0700 Subject: Re: 214df80a9cb3 - main - nvme: new define for size of host memory buffer sizes From: Ravi Pokala To: Warner Losh , , , Message-ID: <3BC728DB-39BB-43BC-BECE-720FECB5B20D@panasas.com> Thread-Topic: 214df80a9cb3 - main - nvme: new define for size of host memory buffer sizes References: <202204090506.239567Ag038413@gitrepo.freebsd.org> In-Reply-To: <202204090506.239567Ag038413@gitrepo.freebsd.org> List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649544650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QYb6dH6gJQMYDfvRgqRmCKgE+msJEq6ECvDAKmR4u4U=; b=QZbXG5M9YDHWLDfu5AXmKK5pO1TbqOJefhJXpEuz6shjh/KX9b1UFQX3ARsSCY0T/NdkYC IzVbY0RVAPrXfxKCeGLxd58/OpZ2Wh1FSjKGI+MFna0MSleExo7sg71UWZsguhpAlgwLej WpFIvfyR+6kqVX76wCOCMO8qcXrYSbFuV7Zrx04qIt1shZs61JKaUlySrrdcVbeMSyJJ2u wJGs+MSceVkfRx7DdwYTWMegwMQVHxYmvSH5JW3OaTzb4bhpdbKzHSJl3lbhMok3cOvhAB 9VrCs0RWVwbebgA/DR1Sp5ol9JJ2AF7XH9QCBlLHp2UTcz1heyshEvpDcdQhEQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1649544650; a=rsa-sha256; cv=none; b=UVRmm22OanwhHPB/NPbJpUUPzcZ1DHcKJiiY0pEXwJaVlhWNeYNOA2m6GFehE6s6ZuthAp /YGNOGYVOA7yFAHF2xVDsztB39h22Uj5xSINyH3EtRpewdPLiLVf41QcZlxcLlrQEn1Cvw whF+SRf59DJ74kUSKjBbvZH+wbeUufugEFHtaAGGBzk8azl/XoZmoIHn3b75mApiKHQYJT 2lyIahuy3qhyyUpGN8gQUGf5lZ6m2fyc6s995Jh+AoJBdoIiKfBl5n6iJiKZgmWMP4NhqY A9cvtBIsbPrK1UU0pPv2g2XKR65GA+99mdKdsKP2EhbIsNsCzj4RB+a+gga0KQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N -----Original Message----- From: on behalf of Warner Losh Date: 2022-04-08, Friday at 22:06 To: , , Subject: git: 214df80a9cb3 - main - nvme: new define for size of host memor= y buffer sizes The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=3D214df80a9cb3e95a140b13af7d= 19deec2bbfae76 commit 214df80a9cb3e95a140b13af7d19deec2bbfae76 Author: Warner Losh AuthorDate: 2022-04-09 05:01:06 +0000 Commit: Warner Losh CommitDate: 2022-04-09 05:05:25 +0000 nvme: new define for size of host memory buffer sizes The nvme spec defines the various fields that specify sizes for hos= t memory buffers in terms of 4096 chunks. So, rather than use a bare = 4096 here, use NVME_HMB_UNITS. This is explicitly not the host page size= of 4096, nor the default memory page size (mps) of the NVMe drive, but= its own thing and needs its own define. Hi Warner, Are you sure about that?=20 NVMe-1.4, Figure 297: Host Memory Buffer =E2=80=93 Host Memory Buffer Descriptor = Entry | Buffer Size (BSIZE): Indicates the number of contiguous | memory page size (CC.MPS) units for this descriptor. |=20 | Buffer Address (BADD): Indicates the host memory address for | this descriptor aligned to the memory page size (CC.MPS).=20 | The lower bits (n:0) of this field indicate the offset | within the memory page is 0h (e.g., if the memory page size | is 4 KiB, then bits 11:00 shall be 0h; if the memory page | size is 8 KiB, then bits 12:00 shall be 0h). They both reference mps, not 4096 bytes. -Ravi (rpokala@) No functional change is intended, only the logical spelling of 4k. Sponsored by: Netflix --- sys/dev/nvme/nvme.h | 3 +++ sys/dev/nvme/nvme_ctrlr.c | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/sys/dev/nvme/nvme.h b/sys/dev/nvme/nvme.h index f2ef2467c9b1..55e94c40dd2c 100644 --- a/sys/dev/nvme/nvme.h +++ b/sys/dev/nvme/nvme.h @@ -62,6 +62,9 @@ /* Cap transfers by the maximum addressable by page-sized PRP (4KB -> = 2MB). */ #define NVME_MAX_XFER_SIZE MIN(maxphys, (PAGE_SIZE/8*PAGE_SIZE)) +/* Host memory buffer sizes are always in 4096 byte chunks */ +#define NVME_HMB_UNITS 4096 + /* Register field definitions */ #define NVME_CAP_LO_REG_MQES_SHIFT (0) #define NVME_CAP_LO_REG_MQES_MASK (0xFFFF) diff --git a/sys/dev/nvme/nvme_ctrlr.c b/sys/dev/nvme/nvme_ctrlr.c index 95a2b5c4285d..6996b3151b0d 100644 --- a/sys/dev/nvme/nvme_ctrlr.c +++ b/sys/dev/nvme/nvme_ctrlr.c @@ -936,11 +936,11 @@ nvme_ctrlr_hmb_alloc(struct nvme_controller *ctrl= r) max =3D (uint64_t)physmem * PAGE_SIZE / 20; TUNABLE_UINT64_FETCH("hw.nvme.hmb_max", &max); - min =3D (long long unsigned)ctrlr->cdata.hmmin * 4096; + min =3D (long long unsigned)ctrlr->cdata.hmmin * NVME_HMB_UNITS; if (max =3D=3D 0 || max < min) return; - pref =3D MIN((long long unsigned)ctrlr->cdata.hmpre * 4096, max); - minc =3D MAX(ctrlr->cdata.hmminds * 4096, PAGE_SIZE); + pref =3D MIN((long long unsigned)ctrlr->cdata.hmpre * NVME_HMB_UNITS, m= ax); + minc =3D MAX(ctrlr->cdata.hmminds * NVME_HMB_UNITS, PAGE_SIZE); if (min > 0 && ctrlr->cdata.hmmaxd > 0) minc =3D MAX(minc, min / ctrlr->cdata.hmmaxd); ctrlr->hmb_chunk =3D pref; @@ -1023,7 +1023,7 @@ again: for (i =3D 0; i < ctrlr->hmb_nchunks; i++) { ctrlr->hmb_desc_vaddr[i].addr =3D htole64(ctrlr->hmb_chunks[i].hmbc_paddr); - ctrlr->hmb_desc_vaddr[i].size =3D htole32(ctrlr->hmb_chunk / 4096); + ctrlr->hmb_desc_vaddr[i].size =3D htole32(ctrlr->hmb_chunk / NVME_HMB_= UNITS); } bus_dmamap_sync(ctrlr->hmb_desc_tag, ctrlr->hmb_desc_map, BUS_DMASYNC_PREWRITE);