Date: Sun, 6 Feb 2022 23:05:14 +0300 From: Mehmet Erol Sanliturk <m.e.sanliturk@gmail.com> To: Warner Losh <imp@bsdimp.com> Cc: Sean Bruno <sbruno@freebsd.org>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: USB Disk Stalls on -current Message-ID: <CAOgwaMs0sKDU95FD=iXHCDmUABzJL%2BrPvPrgcx_Oc%2B-c9Z=HOQ@mail.gmail.com> In-Reply-To: <CANCZdfpWwP5oSh8ktgj9hBnhpn%2BRR1HaEAY7sLQSTtQOw-AHGA@mail.gmail.com> References: <7e8459e4-d708-7750-402c-cda2adf6199f@freebsd.org> <CANCZdfqG-%2B9dfFz-%2BeezZaqbPQN5-mQpw%2B214CkiKC%2B_kmW2ig@mail.gmail.com> <60ebd011-c2b8-3524-1476-123f11128ffe@freebsd.org> <CANCZdfpWwP5oSh8ktgj9hBnhpn%2BRR1HaEAY7sLQSTtQOw-AHGA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000af1a4d05d75f0223 Content-Type: text/plain; charset="UTF-8" On Sun, Feb 6, 2022 at 10:11 PM Warner Losh <imp@bsdimp.com> wrote: > > > On Sun, Feb 6, 2022 at 12:02 PM Sean Bruno <sbruno@freebsd.org> wrote: > >> >> >> > >> > >> > So there's some tools you can use. For usb, there's usbdump that can >> > get you the USB transactions. I've not used it enough to give more >> details >> > here. This will let you know what's going on, and when, on the USB >> endpoint. >> > >> > You can also enable the CAM_IOSCHED stuff. This will allow you to get >> > latency >> > measurements for 'requests in the sim' which basically will tell you >> > what your >> > latency spread is for the drives. This will tell you if things are >> > getting caught >> > up in the USB layer, or after CAM's da driver completes the I/O request >> > (granted, that's almost certainly not happening, but it will help you >> > figure out >> > what's going on and put numbers to the oddities you are seeing). >> > >> > Also, make sure you have good cables. I've had lots of hicups over the >> > years from dodgy USB cables. Also make sure you have good, high quality >> > enclosures. Many from the USB2 time-period are sketchy at best and I >> > went through several at one point trying to find a good one. I'd be >> > tempted to >> > get USB 3 enclosures. I've had better luck with USB3 gear than USB2 gear >> > here, but you need a USB-3 controller to get USB-3 speeds which might >> not >> > be compatible with the NUC's built-in stuff (though my NUC has one USB3 >> > port, there's lots of different models). >> > >> > Usually, though, I see weirdness associated with dmesg messages from >> > usb, cam, etc when the hardware is on the sketch end. >> > >> > Warner >> >> I'm assuming that I have a fairly dodgy USB device, as the pauses seem >> to correspond to this from CAM being emitted: >> >> Feb 6 11:56:43 alice kernel: (da0:umass-sim1:1:0:0): READ(10). CDB: 28 >> 00 36 69 02 6e 00 00 80 00 >> Feb 6 11:56:43 alice kernel: (da0:umass-sim1:1:0:0): CAM status: CCB >> request completed with an error >> Feb 6 11:56:43 alice kernel: (da0:umass-sim1:1:0:0): Retrying command, >> 2 more tries remain >> >> >> Things resume after this is emitted, but there is a substantial >> (multiple minutes) pause here. I would assume that timeouts would fire >> much quicker. >> > > The default timeout is 60s. > > You can reduce that substantially by setting kern.cam.da.default_timeout > to a smaller level. Disk operations completed within 5s these days, > except spin ups. Heck, nearly all complete within 500ms. You > might try setting this value to maybe 3 or 5 or 10 to see if that helps the > hiccups without introducing extra retries when the load is heavy. The > smaller values give a faster recovery, but too small a number may result > in timeouts and errors under load. I think you need to set this as a > tuneable. > > Warner > Are your external disks "GREEN" , i.e. , "energy saver" kind . If the external disks are energy saver kind , they will start to sleep when they are not used for a while , and waking them up will take time which causes significant distress , because to use them requires waiting every such wake up . At that point another important trouble is slowness of USB external disks with respect to internal ( non-energy saver ) SATA disks . When response time is important , it is necessary to avoid such "GREEN" disks . Mehmet Erol Sanliturk --000000000000af1a4d05d75f0223 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:tahoma,sans-serif;font-size:large"><br></div></div><br><div class= =3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sun, Feb 6, 2022 = at 10:11 PM Warner Losh <<a href=3D"mailto:imp@bsdimp.com">imp@bsdimp.co= m</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"= ><div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote"= ><div dir=3D"ltr" class=3D"gmail_attr">On Sun, Feb 6, 2022 at 12:02 PM Sean= Bruno <<a href=3D"mailto:sbruno@freebsd.org" target=3D"_blank">sbruno@f= reebsd.org</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"><br> <br> > <br> > <br> > So there's some tools you can use. For usb, there's usbdump th= at can<br> > get you the USB transactions. I've not used it enough to give more= details<br> > here. This will let you know what's going on, and when, on the USB= endpoint.<br> > <br> > You can also enable the CAM_IOSCHED stuff. This will allow you to get = <br> > latency<br> > measurements for 'requests in the sim' which basically will te= ll you <br> > what your<br> > latency spread is for the drives. This will tell you if things are <br= > > getting caught<br> > up in the USB layer, or after CAM's da driver completes the I/O re= quest<br> > (granted, that's almost certainly not happening, but it will help = you <br> > figure out<br> > what's going on and put numbers to the oddities you are seeing).<b= r> > <br> > Also, make sure you have good cables. I've had lots of hicups=C2= =A0over the<br> > years from dodgy USB cables. Also make sure you have good, high qualit= y<br> > enclosures. Many from the USB2 time-period are sketchy at best and I<b= r> > went through several at one point trying to find a good one. I'd b= e <br> > tempted to<br> > get USB 3 enclosures. I've had better luck with USB3 gear than USB= 2 gear<br> > here, but you need a USB-3 controller to get USB-3 speeds which might = not<br> > be compatible with the NUC's built-in stuff (though my NUC has one= USB3<br> > port, there's lots of different models).<br> > <br> > Usually, though, I see weirdness associated with dmesg messages from<b= r> > usb, cam, etc when the hardware is on the sketch end.<br> > <br> > Warner<br> <br> I'm assuming that I have a fairly dodgy USB device, as the pauses seem = <br> to correspond to this from CAM being emitted:<br> <br> Feb=C2=A0 6 11:56:43 alice kernel: (da0:umass-sim1:1:0:0): READ(10). CDB: 2= 8 <br> 00 36 69 02 6e 00 00 80 00<br> Feb=C2=A0 6 11:56:43 alice kernel: (da0:umass-sim1:1:0:0): CAM status: CCB = <br> request completed with an error<br> Feb=C2=A0 6 11:56:43 alice kernel: (da0:umass-sim1:1:0:0): Retrying command= , <br> 2 more tries remain<br> <br> <br> Things resume after this is emitted, but there is a substantial <br> (multiple minutes) pause here.=C2=A0 I would assume that timeouts would fir= e <br> much quicker.<br></blockquote><div><br></div><div>The default timeout is 60= s.</div><div><br></div><div>You can reduce that substantially by setting ke= rn.cam.da.default_timeout</div><div>to a smaller level. Disk operations com= pleted within 5s these days,</div><div>except spin ups. Heck, nearly all co= mplete within 500ms. You</div><div>might try setting this value to maybe 3 = or 5 or 10 to see if that helps the</div><div>hiccups without introducing e= xtra retries when the load is heavy. The</div><div>smaller values give a fa= ster recovery, but too small a number may result</div><div>in timeouts and = errors under load. I think you need to set this as a tuneable.<br></div><di= v><br></div><div>Warner<br></div></div></div></blockquote><div><br></div><d= iv><br></div><div><br></div><div><div style=3D"font-family:tahoma,sans-seri= f;font-size:large" class=3D"gmail_default">Are your external disks=C2=A0 &q= uot;GREEN" , i.e. ,=C2=A0 "energy saver" kind .</div><div st= yle=3D"font-family:tahoma,sans-serif;font-size:large" class=3D"gmail_defaul= t"><br></div><div style=3D"font-family:tahoma,sans-serif;font-size:large" c= lass=3D"gmail_default">If the external disks are energy saver kind , they w= ill start to sleep when they are not</div><div style=3D"font-family:tahoma,= sans-serif;font-size:large" class=3D"gmail_default">used for a while , and = waking them up will take time which causes significant distress ,</div><div= style=3D"font-family:tahoma,sans-serif;font-size:large" class=3D"gmail_def= ault">because to use them requires waiting every such wake up=C2=A0 .</div>= <div style=3D"font-family:tahoma,sans-serif;font-size:large" class=3D"gmail= _default"><br></div><div style=3D"font-family:tahoma,sans-serif;font-size:l= arge" class=3D"gmail_default">At that point another important trouble is sl= owness of USB external disks <br></div><div style=3D"font-family:tahoma,san= s-serif;font-size:large" class=3D"gmail_default">with respect to internal (= non-energy saver ) SATA disks .<br></div><div style=3D"font-family:tahoma,= sans-serif;font-size:large" class=3D"gmail_default"><br></div><div style=3D= "font-family:tahoma,sans-serif;font-size:large" class=3D"gmail_default">Whe= n response time is important , it is necessary to avoid such "GREEN&qu= ot; disks .</div><div style=3D"font-family:tahoma,sans-serif;font-size:larg= e" class=3D"gmail_default"></div><br><div style=3D"font-family:tahoma,sans-= serif;font-size:large" class=3D"gmail_default"><br></div><div style=3D"font= -family:tahoma,sans-serif;font-size:large" class=3D"gmail_default"><br></di= v><div style=3D"font-family:tahoma,sans-serif;font-size:large" class=3D"gma= il_default">Mehmet Erol Sanliturk<br></div><div style=3D"font-family:tahoma= ,sans-serif;font-size:large" class=3D"gmail_default"><br></div><div style= =3D"font-family:tahoma,sans-serif;font-size:large" class=3D"gmail_default">= <br></div><div style=3D"font-family:tahoma,sans-serif;font-size:large" clas= s=3D"gmail_default"><br></div><div style=3D"font-family:tahoma,sans-serif;f= ont-size:large" class=3D"gmail_default"></div><br></div><div>=C2=A0</div></= div></div> --000000000000af1a4d05d75f0223--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOgwaMs0sKDU95FD=iXHCDmUABzJL%2BrPvPrgcx_Oc%2B-c9Z=HOQ>