Date: Fri, 23 Apr 2004 13:52:56 -0700 From: Tim Kientzle <tim@kientzle.com> To: Julian Elischer <julian@elischer.org> Cc: cvs-all@FreeBSD.org Subject: Re: bad news for bsdtar.. Message-ID: <40898228.8040502@kientzle.com> In-Reply-To: <Pine.BSF.4.21.0404231145150.6894-100000@InterJet.elischer.org> References: <Pine.BSF.4.21.0404231145150.6894-100000@InterJet.elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Julian Elischer wrote: > done on a small subdirectory of our data... > (only accounts starting with 849xxx) > total data as reported by 'du': 9394486 KB (9GB) > (note du takes links into account and doesn't count the same data twice) > number of files with > 1 link: > # find 849* -type f -links +1 | wc -l > 240395 > # > total number of file directory entries.. > # find 849* -type f | wc -l > 598018 > # > total number of directories: > %find 849* -type d | wc -l > 51245 > > average filesize 23k > but major file types are: jpeg images.. 100kb > jpeg thumbnails, 3kb > transaction descriptors 1kb > ######################################################## > the bad news: > > old (FreeBSD 2.x) tar with symlink hack > %time ~/tar cf /dev/null 849* > 5.309u 73.584s 21:15.44 6.1% 329+5150k 786697+0io 0pf+0w > > freebsd 4.8 tar > %time tar cf /dev/null 849 > 5.159u 17.724s 4:08.97 9.1% 364+5863k 66144+0io 1pf+0w > > bsdtar > %time ~/bsdtar cf /dev/null 849* > 16.423u 68.141s 31:40.82 4.4% 402+1639k 784651+0io 0pf+0w > > It looks to me that the tar in 4.8 is doing something very > sneeky with it's IOs.. > maybe using mmap? > maybe detecting /dev/null and instead discarding the data? > maybe reading the entire file in a single read? Very interesting. bsdtar, I note, used about 1/4 as much memory as either of the others. I'd be interested to know what happens if you edit write.c to change the buffer used in write_file_data from the current miserly 8k to something more reasonable, like 128k. I love a challenge... ;-) Tim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?40898228.8040502>