Date: Tue, 12 May 2020 21:45:43 +0000 From: bugzilla-noreply@freebsd.org To: usb@FreeBSD.org Subject: [Bug 244356] Writing to a USB 3.0 stick is very slow Message-ID: <bug-244356-19105-PgOPdIdMl7@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-244356-19105@https.bugs.freebsd.org/bugzilla/> References: <bug-244356-19105@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D244356 --- Comment #26 from Olivier Certner <olivier.freebsd@free.fr> --- I did some tests yesterday on a very recent machine (MB Asus X570) with 3 different USB sticks: - SanDisk Extreme GO USB 3.1 64GB (abbrev: SD_64G) - SanDisk Extreme GO USB 3.1 128GB (abbrev: SD_128G) - Kingston DataTraveler 100 G3 32GB (abbrev: KT_32G) They confirm and extend what S=C3=A9bastien has already reported. ********************* * Executive Summary * ********************* A. For tests performed yesterday and whose reports are below: - Transfers to USB 3.0 mass storage appears to stall very frequently, for at least 2 or 3s, and frequently for several runs of that. Duration of periods= of non-zero bandwidth may become as low as 2 or 3s. - There is an initial grace period where bandwidth is steady and at arguably normal numbers. It lasted between 10 and 20min. But once the stalls start, bandwidth is never steady again, even when changing USB hubs, USB sticks, filesystems and method of writing (for the precise extent of the variations, please see the reports below). The exact conditions for the grace period to manifest are not clear yet; I suspect a reboot should reenable it, but the machine was never rebooted during the tests. - Using FreeBSD 12.1-STABLE at r355888, unmodified. B. For related experience during the last 2 years: - I observed similar overall behavior on very old laptops (> 10 years) that only have USB 2.0 hubs, with some set of USB 3.0 sticks (including KT_32G f= or sure). - Same on another old desktop machine with USB 3.0 hubs (MB Asus P5P43TD). - Same also with other USB 3.0 sticks (from yet another manufacturer, Corsa= ir). - Using continuously updated FreeBSD versions (10 to 12; but never CURRENT = up to now). - No such behavior is exhibited on Linux (USB key is ejected properly, and = the data on it is checked after replug and found correct). C. Wrap-up and follow-up: Currently, I thus have 5 USB 3.0 sticks, none of which can sustain acceptab= le steady bandwidth when transferring data to filesystems, except probably for some limited time after machine boot-up. Unless I'm particularly unlucky, I= 'm suspecting the extent of the problem is such that a significant number of U= SB 3.0 sticks may actually not work with decent bandwidth for a non-trivial duration at all. Having positive reports might help in narrowing the problem down. I really want this problem to be fixed, because I don't see myself buying m= ore and more USB sticks in the hope that one is finally going to work, or havin= g to change motherboards or laptops just because of this, especially when it see= ms the hardware really can work as expected (which might need quirks, workarou= nds, etc.). If knowledgeable people could kindly look into the reports and attachments,= it would be much appreciated. I'm more than willing to run more tests based on feedbacks. I'm probably yet too new to USB and CAM to make fast debugging progress on my own, and I may not have the proper hardware background, so I would appreciate any guidance as to what to test and which code to try tinkering with. ***************************************** * Hardware/software, common test traits * ***************************************** All sticks are MBR-partitioned, with a single 4K-aligned partition occupying all available space. FS type varies and is mentioned in the reports below. FreeBSD version is 12.1-STABLE at r355888. For most tests, I grabbed iostat statistics every second (iostat -w 1). Ins= ide the iostat output, you'll want to look for da0 (last columns) for the curre= nt USB stick numbers. /usr partition is on a M.2 NVMe SSD, appearing as nvd0 (first disk columns). Other active disks during tests were ada3 and ada4, w= ith unrelated traffic (none of their FSes were used for the tests below). All h= ard disks on these machines are ZFS pools. Tests are reported on chronological order. The machine was never rebooted in the course of these tests. *********** * Reports * *********** I. Initial tests with SD_64G For these initial tests, I did not put in place instrumentation as advised earlier in this PR, but I wish I had, since I observed a different behavior (the grace period), which is why I'm reporting about them. First, I used a freshly created exFAT partition. exFAT was created with pac= kage `exfat-utils` and then mounted thanks to package `fusefs-exfat`; both are at version 1.3.0. I copied /usr/local/bin (~661M) using `cp -a`. This completed without error= s, at a rate of ~66MB/s, rate that was steady according to iostat. Then, I replaced the exFAT partition by a standard UFS+SU one (default, so = 32K blocks, 8K frags), and did the same test. Got same result. Finally, I reformatted the partition with UFS (without SU) and did the test again. Got steady albeit lower performance (around 50/55MB/s if I recall correctly). So far so good. II. Iostat-instrumented tests with SD_64G Done on a freshly created UFS+SU partition. This time I launched the copy of whole `usr` (reported to be 46GB by `du`). I stopped it after about 35GB we= re copied. The corresponding attachment is: SD_64G_ufs_SU_usr_cp.txz. Just to repeat some information given in introduction in order for the cont= ext to be very clear: `usr`'s files are read from the VM cache (previous tests)= or a ZFS partition on the NVMe SSD. As can be seen from the iostat file, transfer bandwidth to the stick varies, being below 10MB/s at start, and then most of the time being at several ten= ths of MB/s, with ranges of up to 2 minutes where the bandwidth is mostly betwe= en 30 and 60 MB/s. The transfer proceeds well for approximately 10min and a ha= lf (again, one line per second in the iostat file). But then, it starts to exhibit a very curious pattern, where bandwidth alternatively goes to 0 during 2 to 10s (except for one (small) transaction every 2 or 3s), and then rises to reasonable values (10s of MB/s) with very high TPS. Traffic never goes back to the first phase's pattern. I stopped iostat collection relatively quickly afterwards, but let the copy proceed more, and stopped it after ~35GB were transferred. In total, the transfer took a little more than 3 hours (see trace in attachment), which g= ives little more than 3MB/s for the whole run. I relaunched this test but this time replacing the UFS+SU partition by an e= xFAT one, hoping that the initial correct behavior would return (see I.). I did = this without unplugging/replugging the stick after the UFS+SU test. The corresponding attachment is: SD_64G_exFAT_usr_cp.txz. It contains only the iostat trace. Behavior is similar to the second phase of the previous run. There was no "normal" phase (with decent steady bandwidth) first. Finally, I reran the test on a fresh UFS+SU partition, with again the same result. I did collect only a couple of seconds of iostat. Transfer stopped after 8 hours with a device full, meaning an overall transfer rate of around ~2.25MB/s. The corresponding attachment is: SD_64G_no_unplug_ufs_SU_usr_cp.= txz. III. USB and umass instrumented tests, with variations >From this point, I systematically set sysctl `hw.usb.umass.debug` to -1, and run usbdump as prescribed by HPS earlier in the PR. I also grabbed iostat output, except for A. A. Using a different stick: SD_128G Corresponding attachment: SD_128G_ufs_SU_usr_cp.txz. This is a bigger stick from the same manufacturer. Test is the "same" as for SD_64G: I ran the `cp -a` test to a fresh UFS+SU MBR partition covering the whole stick. Result was essentially the same as previous tests, with stalls= and writes alternating. B. Using `dd if=3D/dev/zero` instead of `cp -a /usr` Corresponding attachment: SD_64G_ufs_SU_dd.txz. I used this to rule out any influence of the source filesystem (ZFS on NVMe SSD), as well as some metadata operations on the destination FS. `dd` write= s to a fresh new partition, into a single file. For this one, I collected only 2 minutes of iostat, but stalls are clearly visible, and the problem seems worse: There are several runs of 2 to 3s blo= cks delimited by a single transaction before bandwidth can really take off, and= for no more than 2 or 3s. The full test ran for 6 minutes, and kernel logs as w= ell as usbdump output are provided in the attachment. C. Using `dd if=3D/dev/zero`, yet another stick: KT_23G, plugged on rear US= B hub Corresponding attachment: KT_32G_ufs_SU_dd.txz. The plugs I had used up to now were facade ones on the case, and using them KT_32G stick is not recognized as plugged. So I switched to using plugs sol= ded to the motherboard. This is a smaller stick from another manufacturer. Test was done as for B.,= and ran 5 minutes. Overall same results. D. Using `dd if=3D/dev/zero`, original stick, plugged on rear USB hub Corresponding attachment: SD_64G_2nd_hub_ufs_SU_dd.txz. This test was done just to ensure that results were not influenced by the h= ub. Indeed, KT_32G not recognized on first hub could leave the impression that something was wrong on these plugs. But results were again the same. Traces inside the attachment as usual. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-244356-19105-PgOPdIdMl7>