From owner-freebsd-questions Tue Nov 16 6:41:52 1999 Delivered-To: freebsd-questions@freebsd.org Received: from kot.ne.mediaone.net (kot.ne.mediaone.net [24.218.15.190]) by hub.freebsd.org (Postfix) with ESMTP id 194F814D37 for ; Tue, 16 Nov 1999 06:41:41 -0800 (PST) (envelope-from mi@kot.ne.mediaone.net) Received: from rtfm.newton (rtfm.newton [10.10.0.1]) by kot.ne.mediaone.net (8.9.3/8.9.1) with ESMTP id JAA50133; Tue, 16 Nov 1999 09:41:41 -0500 (EST) From: Mikhail Teterin X-Relay-IP: 10.10.0.1 Received: (from mi@localhost) by rtfm.newton (8.9.3/8.9.3) id JAA27403; Tue, 16 Nov 1999 09:41:40 -0500 (EST) (envelope-from mi@kot.ne.mediaone.net) Message-Id: <199911161441.JAA27403@rtfm.newton> Subject: Re: -lmd's MD5File() In-Reply-To: <23589.942733382@critter.freebsd.dk> from Poul-Henning Kamp at "Nov 16, 1999 07:23:02 am" To: Poul-Henning Kamp Date: Tue, 16 Nov 1999 09:41:40 -0500 (EST) Cc: questions@freebsd.org X-Face: %UW#n0|w>ydeGt/b@1-.UFP=K^~-:0f#O:D7w hJ5G_<5143Bb3kOIs9XpX+"V+~$adGP:J|SLieM31VIhqXeLBli", Mikhail Teterin writes: =>Why does not the function in subject use mmap? In my tests, using mmap =>instead of reading the file block at a time gave 17-26% performance =>improvement: => => The new method: 135735 microseconds per iteration => The old method: 164525 microseconds per iteration => => The new method: 387283 microseconds per iteration => The old method: 529743 microseconds per iteration => =>Should I submit patches, or is there a good reason? Thanks! = =How would it work for a 5GB file ? It will fall back to the old method. This will make the code uglier, but quite a bit more efficient -- most of the time, files are smaller then 2Gb... And, I hope, eventually FreeBSD will allow bigger files to be mmaped... This is how my testings were done (I like TCL). After `make', run tclsh8.0 test.tcl -mi # This is a shell archive. Save it in a file, remove anything before # this line, and then unpack it by entering "sh file". Note, it may # create directories; files and directories will be owned by you and # have default permissions. # # This archive contains: # # /tmp/md5 # /tmp/md5/test.tcl # /tmp/md5/mitest.c # /tmp/md5/Makefile # echo c - /tmp/md5 mkdir -p /tmp/md5 > /dev/null 2>&1 echo x - /tmp/md5/test.tcl sed 's/^X//' >/tmp/md5/test.tcl << 'END-of-/tmp/md5/test.tcl' Xload libmitest.so X Xif {[md5file1 /kernel] != [md5file2 /kernel]} { X puts stderr "Bad -- the results differ!" X} X Xputs "The new method: [time {md5file1 /kernel} 30]" Xputs "The old method: [time {md5file2 /kernel} 30]" END-of-/tmp/md5/test.tcl echo x - /tmp/md5/mitest.c sed 's/^X//' >/tmp/md5/mitest.c << 'END-of-/tmp/md5/mitest.c' X#include X#include X#include X#include X#include X#include X#include X X#define MAX_MMAP_SIZE 2147483648 /* 2 Gb */ X Xstatic char * my_MD5File(const char *filename, char *buf) X{ X MD5_CTX ctx; X int f; X void *p; X struct stat stats; X X if(stat(filename, &stats) == -1) err(filename); X X if(stats.st_size > MAX_MMAP_SIZE) { X fprintf(stderr, "%d is above %d -- falling back", X stats.st_size, MAX_MMAP_SIZE); X return MD5File(filename, buf); X } X X f = open(filename,O_RDONLY); X X p = mmap(NULL, stats.st_size, PROT_READ, 0, f, 0); X if(!p) err("mmap"); X X MD5Init(&ctx); X MD5Update(&ctx, p, stats.st_size); X X munmap(p, stats.st_size); X close(f); X X return MD5End(&ctx, buf); X} X Xstatic int Xmd5file(ClientData cd, Tcl_Interp *I, int objc, Tcl_Obj * CONST objv[]) X{ X char *result; X X result = Tcl_GetStringFromObj(objv[1], NULL); X result = cd ? MD5File(result, NULL) : my_MD5File(result, NULL); X X if(!result) { X Tcl_SetResult(I, "(null)", TCL_STATIC); X return TCL_ERROR; X } X X Tcl_SetResult(I, result, TCL_DYNAMIC); X return TCL_OK; X} X Xint Mitest_Init(Tcl_Interp *I) { X Tcl_CreateObjCommand(I, "md5file1", md5file, NULL, NULL); X Tcl_CreateObjCommand(I, "md5file2", md5file, NULL+1, NULL); X return Tcl_PkgProvide(I, "mitest", "1.0"); X X} END-of-/tmp/md5/mitest.c echo x - /tmp/md5/Makefile sed 's/^X//' >/tmp/md5/Makefile << 'END-of-/tmp/md5/Makefile' XLIB=mitest XSHLIB_MAJOR=1 X XSRCS=mitest.c X XCFLAGS+=-I/usr/local/include/tcl8.0 X XLDADD=-lmd X X.include END-of-/tmp/md5/Makefile exit To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message