Date: Fri, 20 Dec 2024 17:04:20 +0100 From: Mohammad Noureldin <mohammad@thelightbird.com> To: Joseph Mingrone <jrm@freebsd.org> Cc: hackers@freebsd.org, scottl@freebsd.org, Ed Maste <emaste@freebsd.org> Subject: Re: USB4/TBT3 support Message-ID: <CAAQ96DyTnbkj4M2cVrHgvcTtPhvz-mKs%2B8bn_NuAqrkkYW5goQ@mail.gmail.com> In-Reply-To: <864j8lmaub.fsf@phe.ftfl.ca> References: <864j8lmaub.fsf@phe.ftfl.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000053f2d80629b5cd46 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Joe, Sorry for taking a long while responding to your email. Between some changes on the personal level, really bad luck securing the main TBT3/USB4 hosts and saving up to secure a proper set of HW for the task, it took me that long while to come back on the beginning of the track= . I've compiled in [1]: - All the details you've shared in your email message which includes the notes from scottl@ - The notes left behind by hps@ (may he RIP) - What I have in mind of how to pick up the work from here and going FWD Any comments and/or feedback or even shouting is more than welcome And for the rest, I wish. you and all of the FreeBSD community happy and peaceful holidays [1] https://wiki.freebsd.org/MohammadNoureldin/FreeBSDUSB4TBT3Support On Fri, Jul 19, 2024 at 9:38=E2=80=AFPM Joseph Mingrone <jrm@freebsd.org> w= rote: > Hello, > > Is anyone working on USB4 / Thunderbolt 3 (TBT3) support? > > Scott Long did a lot of work on this a few years ago, but he had to move > on to other things, so he passed things on to Hans Petter Selasky. > Fortunately, Hans Petter dropped the code in a public repository. > > https://github.com/hselasky/usb4 > > https://github.com/hselasky/usb4/commit/dd85c216a2a6bee5361c7166595ba6ca4= 61578b5 > > Here is an overview of what Scott shared with me. > > Mostly completed work: > > - Debug/Trace framework > - NHI controller driver > - PCIe bridge driver > - WMI driver > - Integrated Connection Manager handshake and authentication handling > - Router and Config Space layer handling (in progress, almost complete) > > Remaining work: > > - tbtconfig (userland tool) > - man pages > - DMAR/IOMMU integration, PCIe tunnelling control > - Support for resetting and firmware flashing on the NHI via out-of-band > control > - Host Connection Manager > - Cross-domain login > - ThunderboltIP > > Here are the details that Scott shared. > > The driver originally targeted the Thunderbolt 3 controllers that > were sold under the names =E2=80=9CAlpineRidge" and =E2=80=9CIcelake= =E2=80=9D, in the late > 2010's, before the USB standards group publicly released the USB4 > spec. The driver set I wrote was complete enough to activate > Thunderbolt3 peripherals that otherwise would be disabled by > default when plugged in. The driver also attempted to make it > easier to identify things like PCIe tunnels in the topology, but > that was mostly cosmetic. Unfortunately, the AlpineRidge chips > proved to be extremely hard to work with despite their wide > availability, and I spent way too much time fighting them and not > enough time developing more useful functionality. The WMI driver > was written to work around vexing problems with the Alpine Ridge > controller that I never figured out. > > Much of the infrastructure from the TBT3 support extends to modern > USB4 controllers, but there are still a lot of missing pieces. The > NHI driver doesn't know how to probe a USB4 controller yet, but > that should be easy to fix. Even more important, though, is that > the code lacks a functional USB4 Connection Manager. Most of the > pieces required to traverse the topology, discover routers and > adapters, read and write their properties, and build routes between > endpoints exists now, but there's no state machine yet that > integrates those pieces together into a real Connection Manager. > Without that, no attached peripherals will actually run. The TBT3 > controllers like AlpineRidge and IceLake have a connection manager > in firmware, so that's why those controllers function even with > minimal host OS support. This isn't especially hard code to write, > but it's missing nonetheless. > > Once the connection manager is written, it'll need to configure > connections with the USB3, PCIe, and DisplayPort devices that > operate over tunnels, and it'll need a cross domain handler for > connecting to another host. USB3 tunnel support might require > significant changes in the USB3 stack in order to work with USB4. > > Additionally you might need to write a USB-PD driver. Without it, > negotiation on USB-C connectors for power delivery advertisements, > cable orientation, alt mode configuration, and USB3 vs USB4 lane > assignment might not work. If those negotiations are not handled > then nothing that you plug into the port will even be seen by the > controller. > > I'd totally stay away from spending time on supporting Falcon > Ridge, Alpine Ridge, and Titan Ridge controllers. They're old, > they're extremely difficult to work with, and they're not worth the > headache. In fact, just ignore all TBT3 controllers, and remove > the Internal Connection Manager code. The ICM module isn't code > that I'm all that proud of anyways =3D-). Focus on writing an HCM, > supporting PCIe and DP tunneling, and integrating IOMMU protections > into both the NHI driver and the PCIe tunnel drivers. > > One thing that would be pretty awesome is ThunderboltIP support. > It turns out that these controllers are really cheap 40Gbps > devices, and have the potential to perform pretty well at line rate > as a replacement for traditional 40Gb ethernet controllers, at > least in a point-to-point configuration. > > In short, a lot of work has been done, but a lot of work remains to be > done. The USB4 spec is complicated, and requires intimate knowledge > of the USB-PD, USB-C, and USB3 specs. There's also a spec for > writing a Host Connection Manager that you=E2=80=99ll need to get familia= r with. > > If anyone has already started or would like to continue Scott's work, > could you please let me know? > > Joe > --=20 Thanks - Mohammad Noureldin -- "Life is like riding a bicycle. To keep your balance you must keep moving" - Albert Einstein --00000000000053f2d80629b5cd46 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi Joe,</div><div><br></div><div>Sorry for taking a l= ong while responding to your email.</div><div><br></div><div>Between some c= hanges on the personal level, really bad luck securing the main TBT3/USB4 h= osts and saving up to secure a proper set of HW for the task, it took me th= at long while to come back on the beginning of the track.</div><div><br></d= iv><div>I've compiled in [1]:</div><div>- All the details you've sh= ared in your email message which includes the notes from scottl@</div><div>= - The notes left behind by hps@ (may he RIP)=C2=A0</div><div>- What I have = in mind of how to pick up the work from here and going FWD</div><div><br></= div><div>Any comments and/or feedback or even shouting is more than welcome= </div><div><br></div><div>And for the rest, I wish. you and all of the Free= BSD community happy and peaceful holidays</div><div><br></div><div>[1]=C2= =A0<a href=3D"https://wiki.freebsd.org/MohammadNoureldin/FreeBSDUSB4TBT3Sup= port">https://wiki.freebsd.org/MohammadNoureldin/FreeBSDUSB4TBT3Support</a>= </div><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr"= class=3D"gmail_attr">On Fri, Jul 19, 2024 at 9:38=E2=80=AFPM Joseph Mingro= ne <<a href=3D"mailto:jrm@freebsd.org">jrm@freebsd.org</a>> wrote:<br= ></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;= border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br> <br> Is anyone working on USB4 / Thunderbolt 3 (TBT3) support?<br> <br> Scott Long did a lot of work on this a few years ago, but he had to move<br= > on to other things, so he passed things on to Hans Petter Selasky.<br> Fortunately, Hans Petter dropped the code in a public repository.<br> <br> <a href=3D"https://github.com/hselasky/usb4" rel=3D"noreferrer" target=3D"_= blank">https://github.com/hselasky/usb4</a><br>; <a href=3D"https://github.com/hselasky/usb4/commit/dd85c216a2a6bee5361c7166= 595ba6ca461578b5" rel=3D"noreferrer" target=3D"_blank">https://github.com/h= selasky/usb4/commit/dd85c216a2a6bee5361c7166595ba6ca461578b5</a><br> <br> Here is an overview of what Scott shared with me.<br> <br> Mostly completed work:<br> <br> - Debug/Trace framework<br> - NHI controller driver<br> - PCIe bridge driver<br> - WMI driver<br> - Integrated Connection Manager handshake and authentication handling<br> - Router and Config Space layer handling (in progress, almost complete)<br> <br> Remaining work:<br> <br> - tbtconfig (userland tool)<br> - man pages<br> - DMAR/IOMMU integration, PCIe tunnelling control<br> - Support for resetting and firmware flashing on the NHI via out-of-band co= ntrol<br> - Host Connection Manager<br> - Cross-domain login<br> - ThunderboltIP<br> <br> Here are the details that Scott shared.<br> <br> =C2=A0 =C2=A0 =C2=A0The driver originally targeted the Thunderbolt 3 contro= llers that<br> =C2=A0 =C2=A0 =C2=A0were sold under the names =E2=80=9CAlpineRidge" an= d =E2=80=9CIcelake=E2=80=9D, in the late<br> =C2=A0 =C2=A0 =C2=A02010's, before the USB standards group publicly rel= eased the USB4<br> =C2=A0 =C2=A0 =C2=A0spec.=C2=A0 The driver set I wrote was complete enough = to activate<br> =C2=A0 =C2=A0 =C2=A0Thunderbolt3 peripherals that otherwise would be disabl= ed by<br> =C2=A0 =C2=A0 =C2=A0default when plugged in.=C2=A0 The driver also attempte= d to make it<br> =C2=A0 =C2=A0 =C2=A0easier to identify things like PCIe tunnels in the topo= logy, but<br> =C2=A0 =C2=A0 =C2=A0that was mostly cosmetic.=C2=A0 Unfortunately, the Alpi= neRidge chips<br> =C2=A0 =C2=A0 =C2=A0proved to be extremely hard to work with despite their = wide<br> =C2=A0 =C2=A0 =C2=A0availability, and I spent way too much time fighting th= em and not<br> =C2=A0 =C2=A0 =C2=A0enough time developing more useful functionality.=C2=A0= The WMI driver<br> =C2=A0 =C2=A0 =C2=A0was written to work around vexing problems with the Alp= ine Ridge<br> =C2=A0 =C2=A0 =C2=A0controller that I never figured out.<br> <br> =C2=A0 =C2=A0 =C2=A0Much of the infrastructure from the TBT3 support extend= s to modern<br> =C2=A0 =C2=A0 =C2=A0USB4 controllers, but there are still a lot of missing = pieces.=C2=A0 The<br> =C2=A0 =C2=A0 =C2=A0NHI driver doesn't know how to probe a USB4 control= ler yet, but<br> =C2=A0 =C2=A0 =C2=A0that should be easy to fix.=C2=A0 Even more important, = though, is that<br> =C2=A0 =C2=A0 =C2=A0the code lacks a functional USB4 Connection Manager.=C2= =A0 Most of the<br> =C2=A0 =C2=A0 =C2=A0pieces required to traverse the topology, discover rout= ers and<br> =C2=A0 =C2=A0 =C2=A0adapters, read and write their properties, and build ro= utes between<br> =C2=A0 =C2=A0 =C2=A0endpoints exists now, but there's no state machine = yet that<br> =C2=A0 =C2=A0 =C2=A0integrates those pieces together into a real Connection= Manager.<br> =C2=A0 =C2=A0 =C2=A0Without that, no attached peripherals will actually run= .=C2=A0 The TBT3<br> =C2=A0 =C2=A0 =C2=A0controllers like AlpineRidge and IceLake have a connect= ion manager<br> =C2=A0 =C2=A0 =C2=A0in firmware, so that's why those controllers functi= on even with<br> =C2=A0 =C2=A0 =C2=A0minimal host OS support.=C2=A0 This isn't especiall= y hard code to write,<br> =C2=A0 =C2=A0 =C2=A0but it's missing nonetheless.<br> <br> =C2=A0 =C2=A0 =C2=A0Once the connection manager is written, it'll need = to configure<br> =C2=A0 =C2=A0 =C2=A0connections with the USB3, PCIe, and DisplayPort device= s that<br> =C2=A0 =C2=A0 =C2=A0operate over tunnels, and it'll need a cross domain= handler for<br> =C2=A0 =C2=A0 =C2=A0connecting to another host.=C2=A0 USB3 tunnel support m= ight require<br> =C2=A0 =C2=A0 =C2=A0significant changes in the USB3 stack in order to work = with USB4.<br> <br> =C2=A0 =C2=A0 =C2=A0Additionally you might need to write a USB-PD driver.= =C2=A0 Without it,<br> =C2=A0 =C2=A0 =C2=A0negotiation on USB-C connectors for power delivery adve= rtisements,<br> =C2=A0 =C2=A0 =C2=A0cable orientation, alt mode configuration, and USB3 vs = USB4 lane<br> =C2=A0 =C2=A0 =C2=A0assignment might not work.=C2=A0 If those negotiations = are not handled<br> =C2=A0 =C2=A0 =C2=A0then nothing that you plug into the port will even be s= een by the<br> =C2=A0 =C2=A0 =C2=A0controller.<br> <br> =C2=A0 =C2=A0 =C2=A0I'd totally stay away from spending time on support= ing Falcon<br> =C2=A0 =C2=A0 =C2=A0Ridge, Alpine Ridge, and Titan Ridge controllers.=C2=A0= They're old,<br> =C2=A0 =C2=A0 =C2=A0they're extremely difficult to work with, and they&= #39;re not worth the<br> =C2=A0 =C2=A0 =C2=A0headache.=C2=A0 In fact, just ignore all TBT3 controlle= rs, and remove<br> =C2=A0 =C2=A0 =C2=A0the Internal Connection Manager code.=C2=A0 The ICM mod= ule isn't code<br> =C2=A0 =C2=A0 =C2=A0that I'm all that proud of anyways =3D-). Focus on = writing an HCM,<br> =C2=A0 =C2=A0 =C2=A0supporting PCIe and DP tunneling, and integrating IOMMU= protections<br> =C2=A0 =C2=A0 =C2=A0into both the NHI driver and the PCIe tunnel drivers.<b= r> <br> =C2=A0 =C2=A0 =C2=A0One thing that would be pretty awesome is ThunderboltIP= support.<br> =C2=A0 =C2=A0 =C2=A0It turns out that these controllers are really cheap 40= Gbps<br> =C2=A0 =C2=A0 =C2=A0devices, and have the potential to perform pretty well = at line rate<br> =C2=A0 =C2=A0 =C2=A0as a replacement for traditional 40Gb ethernet controll= ers, at<br> =C2=A0 =C2=A0 =C2=A0least in a point-to-point configuration.<br> <br> In short, a lot of work has been done, but a lot of work remains to be<br> done.=C2=A0 The USB4 spec is complicated, and requires intimate knowledge<b= r> of the USB-PD, USB-C, and USB3 specs.=C2=A0 There's also a spec for<br> writing a Host Connection Manager that you=E2=80=99ll need to get familiar = with.<br> <br> If anyone has already started or would like to continue Scott's work,<b= r> could you please let me know?<br> <br> Joe<br> </blockquote></div><div><br clear=3D"all"></div><div><br></div><span class= =3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_s= ignature"><div dir=3D"ltr"><div><div dir=3D"ltr" style=3D"color:rgb(136,136= ,136)">Thanks<br>- Mohammad Noureldin<br>--<br>"Life is like riding a = bicycle. To keep your balance you must keep moving"<br>- Albert Einste= in</div></div><div dir=3D"ltr" style=3D"color:rgb(136,136,136)"><br></div><= /div></div></div> --00000000000053f2d80629b5cd46--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAQ96DyTnbkj4M2cVrHgvcTtPhvz-mKs%2B8bn_NuAqrkkYW5goQ>