From owner-svn-src-all@freebsd.org Sat Jan 2 10:07:27 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0C05A5E6C8; Sat, 2 Jan 2016 10:07:27 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail107.syd.optusnet.com.au (mail107.syd.optusnet.com.au [211.29.132.53]) by mx1.freebsd.org (Postfix) with ESMTP id 5BC221843; Sat, 2 Jan 2016 10:07:26 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail107.syd.optusnet.com.au (Postfix) with ESMTPS id A5EE8D42B23; Sat, 2 Jan 2016 21:07:16 +1100 (AEDT) Date: Sat, 2 Jan 2016 21:07:15 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Allan Jude cc: "Jonathan T. Looney" , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r292955 - head/lib/libmd In-Reply-To: <56876DF1.4030807@freebsd.org> Message-ID: <20160102210313.M934@besplex.bde.org> References: <201512301804.tBUI4oGp065466@repo.freebsd.org> <20151231115651.R995@besplex.bde.org> <20151231143314.Y1520@besplex.bde.org> <5684D606.3080609@freebsd.org> <56857911.5010205@freebsd.org> <56876DF1.4030807@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=PfoC/XVd c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=6I5d2MoRAAAA:8 a=JKkEluhCpGZgfhysuqUA:9 a=CjuIK1q_8ugA:10 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jan 2016 10:07:27 -0000 On Sat, 2 Jan 2016, Allan Jude wrote: > On 2015-12-31 13:50, Allan Jude wrote: >> On 2015-12-31 13:32, Jonathan T. Looney wrote: >>> On 12/31/15, 2:15 AM, "Allan Jude" wrote: >>> >>>> It seems these problems also slow things down, a lot: >>>> >>>> # time md5 /media/md5test/bigdata >>>> MD5 (/media/md5test/bigdata) = 6afad0bf5d8318093e943229be05be67 >>>> 4.310u 3.476s 0:07.79 99.8% 20+167k 0+0io 0pf+0w >>>> # time env LD_PRELOAD=/usr/obj/media/svn/md5/head/tmp/lib/libmd.so >>>> /usr/obj/media/svn/md5/head/sbin/md5/md5 /media/md5test/bigdata >>>> MD5 (/media/md5test/bigdata) = 6afad0bf5d8318093e943229be05be67 >>>> 4.133u 0.354s 0:04.49 99.7% 20+167k 1+0io 0pf+0w >>>> >>>> (file is fully cached in ZFS ARC, dd reads it at 11GB/s) >>>> >>>> Will investigate more tomorrow. >>> >>> md5 will be slower than dd due to the extra processing it needs to do to >>> generate the hash. I suspect that explains the difference you're seeing >>> between those utilities. >> >> Sorry, you missed my point here. >> >> I replaced MDXFile() with the implementation included in my earlier >> email. Using the newer libmd with that code, cut the time to md5 the >> SAME data down a lot. I need to do a more scientific test on a box that >> isn't doing other stuff still though. >> >> The comment about dd doing 11GB/s, was just to clarify that I wasn't >> reading the file from disk, which would introduce other variables. > > I found the cause of my bogus benchmark, the world on my test machine > was just old enough to be missing jmg@'s bufsize patch. > > Now the difference is about 1 second on a 2GB file, so ignore my > foolishness. That patch is surprisingly new. The main slowness that I complained about was for the other path in md5 that must be used for special files. That uses stdio so it suffers from stdio trusting st_blksize. But st_blksize is rarely as small as the old size BUFSIZ in MDXFile. Bruce