From owner-freebsd-fs@freebsd.org  Thu Oct  4 18:43:26 2018
Return-Path: <owner-freebsd-fs@freebsd.org>
Delivered-To: freebsd-fs@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBE3F10B7121
 for <freebsd-fs@mailman.ysv.freebsd.org>; Thu,  4 Oct 2018 18:43:26 +0000 (UTC)
 (envelope-from wollman@khavrinen.csail.mit.edu)
Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu
 [IPv6:2603:400a:0:7ec::801e:1c14])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "khavrinen.csail.mit.edu", Issuer "Client CA" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 8084E7F6E8
 for <freebsd-fs@freebsd.org>; Thu,  4 Oct 2018 18:43:26 +0000 (UTC)
 (envelope-from wollman@khavrinen.csail.mit.edu)
Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1])
 by khavrinen.csail.mit.edu (8.15.2/8.15.2) with ESMTPS id w94IhPS2080217
 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256
 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA)
 for <freebsd-fs@freebsd.org>; Thu, 4 Oct 2018 14:43:25 -0400 (EDT)
 (envelope-from wollman@khavrinen.csail.mit.edu)
Received: (from wollman@localhost)
 by khavrinen.csail.mit.edu (8.15.2/8.15.2/Submit) id w94IhPIb080216;
 Thu, 4 Oct 2018 14:43:25 -0400 (EDT) (envelope-from wollman)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <23478.24397.495369.226706@khavrinen.csail.mit.edu>
Date: Thu, 4 Oct 2018 14:43:25 -0400
From: Garrett Wollman <wollman@csail.mit.edu>
To: freebsd-fs@freebsd.org
Subject: ZFS/NVMe layout puzzle
X-Mailer: VM 8.2.0b under 26.1 (amd64-portbld-freebsd11.2)
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2
 (khavrinen.csail.mit.edu [127.0.0.1]); Thu, 04 Oct 2018 14:43:25 -0400 (EDT)
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.27
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Oct 2018 18:43:27 -0000

Say you're using an all-NVMe zpool with PCIe switches to multiplex
drives (e.g., 12 4-lane NVMe drives on one side, 1 PCIe x8 slot on the
other).  Does it make more sense to spread each vdev across switches
(and thus CPU sockets) or to have all of the drives in a vdev on the
same switch?  I have no intuition about this at all, and it may not
even matter.  (You can be sure I'll be doing some benchmarking.)

I'm assuming the ZFS code doesn't have any sort of CPU affinity that
would allow it to take account of the PCIe topology even if that
information were made available to it.

-GAWollman