Date: Fri, 5 Oct 2007 12:55:00 -0700 From: Mark Day <mday@apple.com> To: Dmitry Marakasov <amdmi3@amdmi3.ru> Cc: freebsd-fs@freebsd.org Subject: Re: Very slow writes on flash + msdosfs Message-ID: <5F743816-CE50-489E-AB48-C278212791C5@apple.com> In-Reply-To: <20071005004820.GA29814@hades.panopticon> References: <20071005004820.GA29814@hades.panopticon>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mac OS X (where msdosfs is derived from FreeBSD), I fixed it by
just buffering up the I/O into 128KB chunks. Simple but effective.
The diffs are below, based on the Mac OS X sources. But I'll bet it's
not too hard to apply to FreeBSD; I don't think the newfs_msdos
sources have diverged too much. I hope this helps.
-Mark
Index: newfs_msdos.tproj/newfs_msdos.c
===================================================================
--- newfs_msdos.tproj/newfs_msdos.c (revision 26754)
+++ newfs_msdos.tproj/newfs_msdos.c (revision 26759)
@@ -73,6 +73,7 @@
#define DOSMAGIC 0xaa55 /* DOS magic number */
#define MINBPS 128 /* minimum bytes per sector */
+#define MAXBPS 4096 /* maximum bytes per sector */
#define MAXSPC 128 /* maximum sectors per cluster */
#define MAXNFT 16 /* maximum number of FATs */
#define DEFBLK 4096 /* default block size */
@@ -81,6 +82,16 @@
#define RESFTE 2 /* reserved FAT entries */
/*
+ * The size of our in-memory I/O buffer. This is the size of the
writes we
+ * do to the device (except perhaps a few odd sectors at the end).
+ *
+ * This must be a multiple of the sector size. Larger is generally
faster,
+ * but some old devices have bugs if you ask them to do more than 128KB
+ * per I/O.
+ */
+#define IO_BUFFER_SIZE (128*1024)
+
+/*
* [2873845] FAT12 volumes can have 1..4084 clusters. FAT16 can have
* 4085..65524 clusters. FAT32 is 65525 clusters or more.
* Since many other implementations are off by 1, 2, 4, 8, 10, or 16,
@@ -312,7 +323,8 @@
struct bsxbpb *bsxbpb;
struct bsx *bsx;
struct de *de;
- u_int8_t *img;
+ u_int8_t *io_buffer; /* The buffer for sectors being
constructed/written */
+ u_int8_t *img; /* Current sector within io_buffer */
const char *fname, *dtype, *bname;
ssize_t n;
time_t now;
@@ -445,6 +457,9 @@
if (bpb.bps < MINBPS)
errx(1, "bytes/sector (%u) is too small; minimum is %u",
bpb.bps, MINBPS);
+ if (bpb.bps > MAXBPS)
+ errx(1, "bytes/sector (%u) is too large; maximum is %u",
+ bpb.bps, MAXBPS);
if (!(fat = opt_F)) {
if (opt_f)
fat = 12;
@@ -702,12 +717,13 @@
bpb.sec = 0;
}
print_bpb(&bpb);
- if (!opt_N) {
+ if (!opt_N) {
gettimeofday(&tv, NULL);
now = tv.tv_sec;
tm = localtime(&now);
- if (!(img = malloc(bpb.bps)))
+ if (!(io_buffer = malloc(IO_BUFFER_SIZE)))
err(1, NULL);
+ img = io_buffer;
dir = bpb.res + (bpb.spf ? bpb.spf : bpb.bspf) * bpb.nft;
for (lsn = 0; lsn < dir + (fat == 32 ? bpb.spc : rds); lsn++) {
x = lsn;
@@ -808,9 +824,21 @@
(u_int)tm->tm_mday;
mk2(de->date, x);
}
- if ((n = write(fd, img, bpb.bps)) == -1)
+ img += bpb.bps;
+ if (img >= (io_buffer + IO_BUFFER_SIZE)) {
+ /* We filled the I/O buffer, so write it out now */
+ if ((n = write(fd, io_buffer, IO_BUFFER_SIZE)) == -1)
+ err(1, "%s", fname);
+ if (n != IO_BUFFER_SIZE)
+ errx(1, "%s: can't write sector %u", fname, lsn);
+ img = io_buffer;
+ }
+ }
+ if (img != io_buffer) {
+ /* The I/O buffer was partially full; write it out before
exit */
+ if ((n = write(fd, io_buffer, img-io_buffer)) == -1)
err(1, "%s", fname);
- if (n != bpb.bps)
+ if (n != (img-io_buffer))
errx(1, "%s: can't write sector %u", fname, lsn);
}
}
On Oct 4, 2007, at 5:48 PM, Dmitry Marakasov wrote:
> Hi!
>
> I have USB flash:
>
> da0 at umass-sim0 bus 0 target 0 lun 0
> da0: <JetFlash TS2GJF150 8.07> Removable Direct Access SCSI-2 device
> da0: 40.000MB/s transfers
> da0: 1999MB (4095998 512 byte sectors: 255H 63S/T 254C)
>
> It attaches to ehci and I get pretty good read/write speed:
>
> % dd if=/dev/zero of=/dev/da0 bs=1m
> dd: /dev/da0: short write on character device
> dd: /dev/da0: end of device
> 2000+0 records in
> 1999+1 records out
> 2097150976 bytes transferred in 178.132363 secs (11772993 bytes/sec)
>
> % dd if=/dev/da0 of=/dev/zero bs=1m count=100
> 100+0 records in
> 100+0 records out
> 104857600 bytes transferred in 7.240738 secs (14481618 bytes/sec)
>
> But newfs_msdos ran on it is very slow:
>
> % time newfs_msdos -F32 -LAMDmi3 -k 0xffff /dev/da0s1
> /dev/da0s1: 4072456 sectors in 509057 FAT32 clusters (4096 bytes/
> cluster)
> bps=512 spc=8 res=32 nft=2 mid=0xf0 spt=63 hds=255 hid=0
> bsec=4080447 bspf=3978 rdcl=2 infs=1 bkbs=0xffff
> newfs_msdos -F32 -LAMDmi3 -k 0xffff /dev/da0s1 0,02s user 0,21s
> system 0% cpu 2:54,37 total
>
> when it runs, gkrellm shows 23kb/s writes.
>
> Writing a 1.0Mb directory with 340 files takes 1.5 minutes (up to
> 300k/sec
> writes => seems like much more data is actually written that it's
> needed).
>
> Larger files behave somewhat better (up to 3 MB/s).
>
> Btw, dd if=/dev/zero of=/dev/da0 bs=512 show the same 23k/s speed as
> newfs_msdosfs.
>
> So where is the problem? Why's there no caching and why's there 1
> sector
> writes?
>
> PS. I use 6.1, has the situation changed in -CURRENT?
>
> --
> Best regards,
> Dmitry Marakasov mailto:amdmi3@amdmi3.ru
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5F743816-CE50-489E-AB48-C278212791C5>