From owner-freebsd-hardware@freebsd.org Mon Jun 11 21:37:17 2018 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDE2D1021011 for ; Mon, 11 Jun 2018 21:37:17 +0000 (UTC) (envelope-from lidl@pix.net) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.pix.net", Issuer "Pix.Com Technologies LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 537257FD46 for ; Mon, 11 Jun 2018 21:37:17 +0000 (UTC) (envelope-from lidl@pix.net) Received: from torb.pix.net (torb.pix.net [192.168.16.32]) (authenticated bits=0) by hydra.pix.net (8.15.2/8.15.2) with ESMTPA id w5BLbGlC008348; Mon, 11 Jun 2018 17:37:16 -0400 (EDT) (envelope-from lidl@pix.net) To: freebsd-hardware@freebsd.org References: Subject: Re: PCIe multipliers, how do they work? From: Kurt Lidl Message-ID: Date: Mon, 11 Jun 2018 17:37:16 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jun 2018 21:37:17 -0000 > I'm wondering how these things work. The wikipedia PCIe page [1] says: > "PCI Express switches can create multiple endpoints out of one endpoint > to allow sharing one endpoint with multiple devices." So maybe they > use a PCIe switch? Poking around wikipedia and google has thus far > uncovered very little info about PCIe switches. Wikipedia is less > helpful than usual, and they keep making google less and less useful > for no apparent reason. I don't see any other obvious keywords > to google for. > > It isn't obvious how slot id/address is handled. How do commands and > data get routed to/from the correct card? > > Is any firmware or OS support required? It's worth reading the following two pages closely, as they talk about PCI Express endpoints, and how PCI-Express switches have to work to enable multiple cards. http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1 http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2 One such family of PCI-express switch parts is from PLX Technologies, now owned by Broadcom. One of typical functions of such as switch is to provide "non transparent bridging" - basically mapping the address space from one machine's PCI address into the address space of another PCI system. This can be used a communication path between devices, for example, using a block and slot type software architecture. There is some support for NTB in FreeBSD's master branch, under sys/dev/ntb - with drivers for the PLX/Avago/Broadcom parts, as well as a driver for the Intel parts found on certain Xeon and Atom CPUs. -Kurt