Date: Tue, 4 Mar 2025 11:53:50 -0800 From: Warner Losh <imp@bsdimp.com> To: Wei Hu <weh@microsoft.com> Cc: Li-Wen Hsu <lwhsu@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: What is scbus and how it is numbered on FreeBSD? Message-ID: <CANCZdfrM=MBPvSq_D8_qZKeaW8imZJJxgmessU9FRnFLQKvfuw@mail.gmail.com> In-Reply-To: <OSQP153MB13074BDF30B5828E7ACB4C72BBCD2@OSQP153MB1307.APCP153.PROD.OUTLOOK.COM> References: <OSQP153MB13074BDF30B5828E7ACB4C72BBCD2@OSQP153MB1307.APCP153.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000084cdad062f89a250 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 26, 2025 at 9:01=E2=80=AFPM Wei Hu <weh@microsoft.com> wrote: > Hi Warner, > > > > I have some questions regarding the FreeBSD scbus and its relationship to > the SCSI concept of bus (of bus:target:lun). Multiple people inside > Microsoft raised same questions recently and I am not exactly sure. For > example, my FreeBSD VM has 3 SCSI disks and 2 Nvme disks. Dmesg shows: > > > > # dmesg | grep "da[0-9] at" > > da0 at blkvsc0 bus 0 scbus2 target 0 lun 0 > > da1 at blkvsc1 bus 0 scbus3 target 1 lun 0 > > da2 at storvsc3 bus 0 scbus5 target 0 lun 0 > > nda0 at nvme0 bus 0 scbus6 target 0 lun 1 > > nda1 at nvme1 bus 0 scbus7 target 0 lun 1 > > > > # camcontral devlist > > <Msft Virtual Disk 1.0> at scbus2 target 0 lun 0 (da0,pass0) > > <Msft Virtual Disk 1.0> at scbus3 target 1 lun 0 (da1,pass1) > > <Msft Virtual Disk 1.0> at scbus5 target 0 lun 0 (da2,pass2) > > <Microsoft NVMe Direct Disk NVMDV001> at scbus6 target 0 lun 1 > (nda0,pass3) > > <Microsoft NVMe Direct Disk NVMDV001> at scbus7 target 0 lun 1 > (nda1,pass4) > > > > - Are the scbus numbers showing above referring to the host adapters? > The scbus number is the bus of bus:target:lun. They are assigned sequentially as the different SIMs register with CAM. We never try to reuse numbers if a device departs and a new device arrives. These are allocated in xpt_bus_register() in the sim. The sim has no control over these numbers. Internally to CAM, these are called 'path_id' or similar. These are always unique. > - Are scbus umbered after the discovering order at boot time or based on > any rules? Why are scbus0, 1 and 4 missing from dmesg? > One: if a SIM registers and unregisters. This would cause gaps like this. Registering multiple buses can cause this (we only report the periphs present, camcontrol devlist -v will report all the buses). Wired entries can also cause this. I think we'll need to see camcontrol devlist -v to know which of these. > - Is "bus 0" showed in the dmesg output the SCSI concept of bus? Does it > have any relationship to the scbus numbers? > 'bus 0' here is an extra layer. Years ago, the ahc and ahd devices could support multiple parallel scsi buses on on card as one sim. So you'd see 'bus 0' and 'bus 1' on those systems. Each of those buses would also have a scbus as well. All the traffic for both of these buses would go through the same sim/sim_action() instance. So there's a weak relationship to scbus numbers since usually bus 0 as scbusN will have a bus 1 as scbusN+1 because we serialize registration and these drivers register 0 and 1. There's a surprising number of SIMs that do this (most often they call this parameter a 'channel' which is a better term imho). These could all be better documented. Warner > Thanks so much, > > Wei > --00000000000084cdad062f89a250 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Feb 26,= 2025 at 9:01=E2=80=AFPM Wei Hu <<a href=3D"mailto:weh@microsoft.com">we= h@microsoft.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= ding-left:1ex"><div class=3D"msg7068221580809636609"> <div lang=3D"EN-US" style=3D"overflow-wrap: break-word;"> <div class=3D"m_7068221580809636609WordSection1"> <p class=3D"m_7068221580809636609MsoPlainText">Hi Warner,<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><u></u>=C2=A0<u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">I have some questions regard= ing the FreeBSD scbus and its relationship to the SCSI concept of bus (of b= us:target:lun). Multiple people inside Microsoft raised same questions rece= ntly and I am not exactly sure. For example, my FreeBSD VM has 3 SCSI disks and 2 Nvme disks. Dmesg shows:<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><u></u>=C2=A0<u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"># dmesg | grep "da[0-9]= at"<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">da0 at blkvsc0 <span style= =3D"background:yellow"> bus 0</span> scbus2 target 0 lun 0<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">da1 at blkvsc1 bus 0 scbus3 = target 1 lun 0<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">da2 at storvsc3 bus 0 scbus5= target 0 lun 0<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">nda0 at nvme0 bus 0 scbus6 t= arget 0 lun 1<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">nda1 at nvme1 bus 0 scbus7 t= arget 0 lun 1<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><u></u>=C2=A0<u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"># camcontral devlist<u></u><= u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><Msft Virtual Disk 1.0>= ;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scbu= s2 target 0 lun 0 (da0,pass0)<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><Msft Virtual Disk 1.0>= ;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scbu= s3 target 1 lun 0 (da1,pass1)<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><Msft Virtual Disk 1.0>= ;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scbu= s5 target 0 lun 0 (da2,pass2)<u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText"><Microsoft NVMe Direct Di= sk NVMDV001>=C2=A0 at scbus6 target 0 lun 1 (nda0,pass3)<u></u><u></u></= p> <p class=3D"m_7068221580809636609MsoPlainText"><Microsoft NVMe Direct Di= sk NVMDV001>=C2=A0 at scbus7 target 0 lun 1 (nda1,pass4)<u></u><u></u></= p> <p class=3D"m_7068221580809636609MsoPlainText"><u></u>=C2=A0<u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">- Are the scbus numbers show= ing above referring to the host adapters?</p></div></div></div></blockquote= ><div><br></div><div>The scbus number is the bus of bus:target:lun. They ar= e assigned sequentially as the different SIMs register with CAM. We never t= ry to reuse numbers if a device departs and a new device arrives. These are= allocated in xpt_bus_register() in the sim. The sim has no control over th= ese numbers. Internally to CAM, these are called 'path_id' or simil= ar. These are always unique.</div><div>=C2=A0</div><blockquote class=3D"gma= il_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,2= 04,204);padding-left:1ex"><div class=3D"msg7068221580809636609"><div lang= =3D"EN-US" style=3D"overflow-wrap: break-word;"><div class=3D"m_70682215808= 09636609WordSection1"><p class=3D"m_7068221580809636609MsoPlainText"> <u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">- Are scbus umbered after th= e discovering order at boot time or based on any rules? Why are scbus0, 1 a= nd 4 missing from dmesg?</p></div></div></div></blockquote><div><br></div><= div>One: if a SIM registers and unregisters. This would cause gaps like thi= s. Registering multiple buses can cause this (we only report the periphs pr= esent, camcontrol devlist -v will report all the buses). Wired entries can = also cause this. I think we'll need to see camcontrol devlist -v to kno= w which of these.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);pad= ding-left:1ex"><div class=3D"msg7068221580809636609"><div lang=3D"EN-US" st= yle=3D"overflow-wrap: break-word;"><div class=3D"m_7068221580809636609WordS= ection1"><p class=3D"m_7068221580809636609MsoPlainText"><u></u><u></u></p> <p class=3D"m_7068221580809636609MsoPlainText">- Is "<span style=3D"ba= ckground:yellow">bus 0</span>" showed in the dmesg output the SCSI con= cept of bus? Does it have any relationship to the scbus numbers?</p></div><= /div></div></blockquote><div><br></div><div>'bus 0' here is an extr= a layer. Years ago, the ahc and ahd devices could support multiple parallel= scsi buses on on card as one sim. So you'd see 'bus 0' and = 9;bus 1' on those systems. Each of those buses would also have a scbus = as well. All the traffic for both of these buses would go through the same = sim/sim_action() instance.=C2=A0 So there's a weak relationship to scbu= s numbers since usually bus 0 as scbusN will have a bus 1 as scbusN+1 becau= se we serialize registration and these drivers register 0 and 1. There'= s a surprising number of SIMs that do this (most often they call this param= eter a 'channel' which is a better term imho).</div><div><br></div>= <div>These could all be better documented.</div><div><br></div><div>Warner<= /div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><di= v class=3D"msg7068221580809636609"><div lang=3D"EN-US" style=3D"overflow-wr= ap: break-word;"><div class=3D"m_7068221580809636609WordSection1"><p class= =3D"m_7068221580809636609MsoPlainText"></p> <p class=3D"m_7068221580809636609MsoPlainText">Thanks so much,<u></u><u></u= ></p> <p class=3D"m_7068221580809636609MsoPlainText">Wei =C2=A0<u></u><u></u></p> </div> </div> </div></blockquote></div></div> --00000000000084cdad062f89a250--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrM=MBPvSq_D8_qZKeaW8imZJJxgmessU9FRnFLQKvfuw>