From owner-freebsd-hardware@freebsd.org Fri May 11 02:04:38 2018 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A4F2FAFDF3; Fri, 11 May 2018 02:04:38 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: from mail-it0-x242.google.com (mail-it0-x242.google.com [IPv6:2607:f8b0:4001:c0b::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 994368703E; Fri, 11 May 2018 02:04:37 +0000 (UTC) (envelope-from dieterbsd@gmail.com) Received: by mail-it0-x242.google.com with SMTP id n202-v6so371264ita.1; Thu, 10 May 2018 19:04:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=/uJFpaEp2Oq1CBUINjp9tKROg/rZgE1i304fj5C5rtI=; b=Jj6WsTot233bJglP7FtA6mmkksUg9yX994QUvA6nFFPHN/TLlRkEm7R1oj9xPCW3BF P3aQdu52NTnoqIKQ717K535+iKQb3QrGup3qsUsdJ2zesdz5dueIeNfTuiJMU9Tavzb5 YuFOxFrrM1cFv97CyQiNj9P5vRkcS2tT91zLzo045w7ULlL9KhW6/RQKr6sjAiGxAgNm Sppj8etJKxfcbM2y5Lyt3oADUe17UaOF7OvdrgTZIK9as1qE22fgRdMD021gwzCaKNWu cJNVvZaBgXBPhvI549Vxa2KwP49XWiQ7VYHmPtz8bensdtPURIdH6hJSbsXzTy3AVWfe Lseg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=/uJFpaEp2Oq1CBUINjp9tKROg/rZgE1i304fj5C5rtI=; b=O+oGqSLNoEkIxXc3tQw9w27jCiSeHbkqsULU95Bz08aj/lUKhdYrAHeOWeBM1n9BUw wYWSJQxSuLSpos0MNTZ7ywDiHIsPOKyKhGqgYdoDRLJfIHtkltMkGKJBkNldVTyx5jbM Hs4iVnhYg3YbZCf3gMq8cKH8sGmYZgOfRRtdb9l/MQTZxQH43SC6Q8fd/iJoTZyvnszC e+TvsrxlwT3ymYo9KJhYl0yztl47yp2RH+9Lx9UXG7fZizcJva36Oe31or3NQyaBn5dH W+WpDPkrKIhzrArp3YJmAMMonHcPzzXt041YKUppOoJGvG5gB72lXFHsv80dF47rFgwz GSfQ== X-Gm-Message-State: ALKqPweBZYG2/zVXfYRc2FyQzM5WQUsdEE8c9BtASaGiaJePDfuComxC lKfZi1Ek8QfjJvBzAEtEclWt3iYk84nBscPSdyc= X-Google-Smtp-Source: AB8JxZqZM3Xv5cazBaCVh2DJ81oWObgqLxZ+rm51z6rqmvSqHxL/U1kFdB27kB8mCk5e/xiL3VhZyyMr2ucnKzzNHIw= X-Received: by 2002:a24:4c55:: with SMTP id a82-v6mr1336086itb.1.1526004277018; Thu, 10 May 2018 19:04:37 -0700 (PDT) MIME-Version: 1.0 Received: by 10.192.132.228 with HTTP; Thu, 10 May 2018 19:04:36 -0700 (PDT) From: Dieter BSD Date: Thu, 10 May 2018 19:04:36 -0700 Message-ID: Subject: PCIe multipliers, how do they work? To: freebsd-hardware@freebsd.org Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2018 02:04:38 -0000 Looking into ways to get additional expansion slots, since board designers cannot count past 7 and often not even that high. :-( There are PCIe riser cards that split a wide slot into 2 or more narrower slots, for example 1 x8 slot becomes 2 x4 slots. These would be very useful, except it appears that they require "bifurcation" support in the mainboard's firmware. Which most boards do not provide. And most boards are not supported by FLOSS firmware, so adding bifurcation support would be rather difficult. There are also PCIe cards which provide multiple slots, typically connected with a usb cable. These tend to convert 1 PCIe_x1 slot into multiple PCIe_x1 slots. I get the impression that these do not require bifurcation support. They seem to be aimed at "miners" for attaching multiple gpu cards. I'm not interested in mining or in attaching multiple gpu cards. I'm interesting in adding additional sata cards, Ethernet cards, and such. Unlike the bifurcation type riser splitter cards, which seem to conserve PCIe lanes, these are more like a sata port multiplier, with the same type of bandwidth limitation. I'm wondering how these things work. The wikipedia PCIe page [1] says: "PCI Express switches can create multiple endpoints out of one endpoint to allow sharing one endpoint with multiple devices." So maybe they use a PCIe switch? Poking around wikipedia and google has thus far uncovered very little info about PCIe switches. Wikipedia is less helpful than usual, and they keep making google less and less useful for no apparent reason. I don't see any other obvious keywords to google for. It isn't obvious how slot id/address is handled. How do commands and data get routed to/from the correct card? Is any firmware or OS support required? Is there some other solution that I haven't stumbled across? I'd really like to split an x8 slot into 4 x2 slots, which doesn't seem to be an off-the-shelf option either way. [1] en.wikipedia.org/wiki/PCI_Express From owner-freebsd-hardware@freebsd.org Fri May 11 19:03:31 2018 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDC99FC66F5; Fri, 11 May 2018 19:03:30 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "puchar.net", Issuer "puchar.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 629CF75840; Fri, 11 May 2018 19:03:30 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id w4BJ3a1M049989 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 11 May 2018 21:03:36 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id w4BJ3VUC049986; Fri, 11 May 2018 21:03:31 +0200 (CEST) (envelope-from puchar-wojtek@puchar.net) Date: Fri, 11 May 2018 21:03:31 +0200 (CEST) From: Wojciech Puchar To: Dieter BSD cc: freebsd-hardware@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: PCIe multipliers, how do they work? In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2018 19:03:31 -0000 > There are also PCIe cards which provide multiple slots, typically > connected with a usb cable. These tend to convert 1 PCIe_x1 slot > into multiple PCIe_x1 slots. I get the impression that these do > not require bifurcation support. They seem to be aimed at "miners" these cards consist of PCIe switch which is supported out of the box. former ones require BIOS to reconfigure CPUs PCI lanes so instead of eg one 16x lane there will be 4 4x lanes. From owner-freebsd-hardware@freebsd.org Fri May 11 20:11:24 2018 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 089D7FC9F5A for ; Fri, 11 May 2018 20:11:24 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x22d.google.com (mail-io0-x22d.google.com [IPv6:2607:f8b0:4001:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81FFF842DD for ; Fri, 11 May 2018 20:11:23 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x22d.google.com with SMTP id d73-v6so8393620iog.3 for ; Fri, 11 May 2018 13:11:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=U8no60hfNY+daL4DSJBxkV3lr8BfXzTHbm4NBjiMHkk=; b=uE3cWzegEDj6jiuI9FQP0ehglCtTFb1vWxfXRKdhMEJ/AszI5cR/SNV63RzYQaHXNH wL+H+Smlo22f2O2LvDbhqNsGPX7Y22b/4w6eVBJz+yQ7P+pIs0+TxsJi+Exk7zFgxh9Z vYPqKdTDd5ASGKjM1RHPJMtzu0tW9x3WxiuQvfvuHkISed9X06lwuRIPOUBSk2b+uDP5 a4q0h4VbPNRtbR/Rq+iPtvNxf8REA7MM7AEqhz+nuQgRWGWldmPlz0upx+URkUvzRwKi sTIfwxHuKLGSg+XLkqMkIWQSJ+/Mm1eKEUJ2g/IvQ3zFb/E1VNyqOcAI3v04nrztaEZp PzkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=U8no60hfNY+daL4DSJBxkV3lr8BfXzTHbm4NBjiMHkk=; b=MpJLmom0ScNDhjatcov++4Yeyw/8ZRlXfwIgLuhaR/7bvUIS1whz303ROdeFjyrX9l AMd/Lbobz30qgI/15ItH7QF+Sbb23J4BKSJ67Cuawpdc7xIkWDYHTToYH4NcSq5i25S0 4fdgqhxojW221LpFKVNpoC5G7E6/V0O1E/4FLAxP+CHZ46V+cKR5jpaiA6ccIhKMmXjl kxO8vwkPUeQEk9V1ikYX2LsQyrdTAJGLd5q9nlKXkQXDhdponbJJIJmFhfWTSdP8hEHl 8Dev4ZcPGgK+sGtrI4l5AHt2ZJ3LLyS9noBaVLjwSHy6HQvSUo5fMhLTP2jLsuDlJ59T yeOA== X-Gm-Message-State: ALKqPwf17hlehnwHgy1oSX3owv1SvWRpKMraAO2Z9bFguxE/P62dTMBL 8jFtz12KD4+8itJSpcVOIAZH387Rv0diLEVEfU7I/g== X-Google-Smtp-Source: AB8JxZrjYzQ1dKVAHx3CmIUGhHF2zKnKwgpkUk8xeWTDL49/11hsK9J7vA1npMM6Xo9N9ykpJcC2UZ168bmbZSSkEh8= X-Received: by 2002:a6b:12a3:: with SMTP id 35-v6mr7424683ios.168.1526069482761; Fri, 11 May 2018 13:11:22 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:a649:0:0:0:0:0 with HTTP; Fri, 11 May 2018 13:11:22 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: References: From: Warner Losh Date: Fri, 11 May 2018 14:11:22 -0600 X-Google-Sender-Auth: SIwOxazy3NP0l9wFHuzCOWHBDvU Message-ID: Subject: Re: PCIe multipliers, how do they work? To: Wojciech Puchar Cc: Dieter BSD , "freebsd-hackers@freebsd.org" , freebsd-hardware@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 May 2018 20:11:24 -0000 On Fri, May 11, 2018 at 1:03 PM, Wojciech Puchar wrote: > There are also PCIe cards which provide multiple slots, typically >> connected with a usb cable. These tend to convert 1 PCIe_x1 slot >> into multiple PCIe_x1 slots. I get the impression that these do >> not require bifurcation support. They seem to be aimed at "miners" >> > > these cards consist of PCIe switch which is supported out of the box. > former ones require BIOS to reconfigure CPUs PCI lanes so instead of eg > one 16x lane there will be 4 4x lanes. > Usually they require a driver for management functions, like if you want to turn off one of the ports (because you know there's a bad card in it, for example). But for normal probe / attach they are usually drop in. Some mobos can require some BIOS adjustment, though to properly setup lane bifurcation and such... The ones that are true switches, and not just signal retimers, usually don't: they take the full 8 or 16 lanes and multiplex it amongst the 16-32 downstream lanes they typically provide. We're looking at one that does x4 lanes and expands to 4 cards at x4 lanes. The devices on the other side fill just over a lane each, but we'd run out of lanes if we did x2 fan-out. This the cards we're looking at, we can get x4 combined rates form the 4 cards that on their own are kinda crappy. Warner