From owner-svn-src-all@FreeBSD.ORG Fri Oct 31 06:48:22 2008 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 134481065673; Fri, 31 Oct 2008 06:48:22 +0000 (UTC) (envelope-from keramida@freebsd.org) Received: from igloo.linux.gr (igloo.linux.gr [62.1.205.36]) by mx1.freebsd.org (Postfix) with ESMTP id 996138FC12; Fri, 31 Oct 2008 06:48:21 +0000 (UTC) (envelope-from keramida@freebsd.org) Received: from kobe.laptop (adsl25-227.kln.forthnet.gr [77.49.152.227]) (authenticated bits=128) by igloo.linux.gr (8.14.3/8.14.3/Debian-5) with ESMTP id m9V6mA0g004474 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 31 Oct 2008 08:48:16 +0200 Received: from kobe.laptop (kobe.laptop [127.0.0.1]) by kobe.laptop (8.14.3/8.14.3) with ESMTP id m9V6m90S000453; Fri, 31 Oct 2008 08:48:09 +0200 (EET) (envelope-from keramida@freebsd.org) Received: (from keramida@localhost) by kobe.laptop (8.14.3/8.14.3/Submit) id m9V6m9Vw000452; Fri, 31 Oct 2008 08:48:09 +0200 (EET) (envelope-from keramida@freebsd.org) From: Giorgos Keramidas To: Ivan Voras References: <200810301405.m9UE5wwZ053250@svn.freebsd.org> Date: Fri, 31 Oct 2008 08:48:09 +0200 In-Reply-To: <200810301405.m9UE5wwZ053250@svn.freebsd.org> (Ivan Voras's message of "Thu, 30 Oct 2008 14:05:58 +0000 (UTC)") Message-ID: <877i7pdzva.fsf@kobe.laptop> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-MailScanner-ID: m9V6mA0g004474 X-Hellug-MailScanner: Found to be clean X-Hellug-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-4.303, required 5, autolearn=not spam, ALL_TRUSTED -1.80, AWL 0.10, BAYES_00 -2.60) X-Hellug-MailScanner-From: keramida@freebsd.org X-Spam-Status: No Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r184471 - in head/bin: cat cp X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Oct 2008 06:48:22 -0000 On Thu, 30 Oct 2008 14:05:58 +0000 (UTC), Ivan Voras wrote: > Author: ivoras > Date: Thu Oct 30 14:05:57 2008 > New Revision: 184471 > URL: http://svn.freebsd.org/changeset/base/184471 > > Log: > Teach cat(1) and cp(1) to use a larger buffer if enough memory is present > in the system. A simple heuristics is used to detect what is "enough" > memory: if number of physmem pages is greater than 32k (equalling 128 MB > on machines with 4 kB pages). > > Typical immediate result of these changes is reduction in context switches > and the goal is to increase efficiency by using large buffers: > before: /usr/bin/time -hlp cat file1 > file2 > ... > 163 voluntary context switches > 11194 involuntary context switches > after: /usr/bin/time -hlp ./cat file1 > file2 > ... > 417 voluntary context switches > 272 involuntary context switches Nice :) I didn't notice this in -hackers, but it may be worth letting the buffer size grow above 1 MB when we have more than, say, 512 MB of memory. By running dd(1) with buffer sizes of 1, 2, 4, 6, 8, 10, 12, 14, and 16 MB there are larger buffer sizes that reduce involuntary context switches even more. My laptop (with more than 2 GB or memory), with powerd disabled, running in single user mode, and copying a file of 200 MB using the script: ,----------------------------------------------------------------------- | #!/bin/sh | | mblist="1 2 4 6 8 10 12 14 16" | | for bufsize in ${mblist} ; do | cmd="dd if=testfile bs=${bufsize}m | dd of=testfile.out bs=${bufsize}m" | logname=`printf '%02d.dd-times' "${bufsize}"` | true > "${logname}" | for cnt in `jot 30 1 30` ; do | echo "Overwriting testfile" | cat datafile > testfile | printf '%2d MB buffer, run %2d, logfile=%s\n' \ | "${bufsize}" "${cnt}" "${logname}" | /usr/bin/time -l sh -c "${cmd}" 2>> "${logname}" | tail -1 "${logname}" | done | done `----------------------------------------------------------------------- got the following results for sizes < 10 MB: : tcsh% ministat -w 78 0? : x 01 : + 02 : * 04 : % 06 : # 08 : +------------------------------------------------------------------------------+ : | # % #% * * + x | : | ## %#*#* %%% # O%*O% ***+* *+ + + x x x x xx x | : |## %### %%O#O#O##OOO#OOOOOOO *OO**@*O**++*xx x x xxx x * xx x x x x x| : | |___|____AM|MA____||A|___A_M_|__| |__________MA__________| | : +------------------------------------------------------------------------------+ : N Min Max Median Avg Stddev : x 30 1383 1653 1493 1496.1 69.360801 : + 30 1286 1434 1381 1371.0667 44.393952 : Difference at 95.0% confidence : -125.033 +/- 30.1005 : -8.35728% +/- 2.01193% : (Student's t, pooled s = 58.2312) : * 30 1242 1523 1342 1339.3667 56.059379 : Difference at 95.0% confidence : -156.733 +/- 32.5975 : -10.4761% +/- 2.17883% : (Student's t, pooled s = 63.0618) : % 30 1199 1401 1292 1293.1667 50.603996 : Difference at 95.0% confidence : -202.933 +/- 31.3824 : -13.5642% +/- 2.09762% : (Student's t, pooled s = 60.7111) : # 30 1179 1402 1275 1273.9667 57.161889 : Difference at 95.0% confidence : -222.133 +/- 32.8523 : -14.8475% +/- 2.19586% : (Student's t, pooled s = 63.5547) Buffer sizes even larger didn't yield as good results as 6 MB or 8 MB, but there seems to be an improvement in 16 MB too: : tcsh% ministat -w 78 01 1? : x 01 : + 10 : * 12 : % 14 : # 16 : +------------------------------------------------------------------------------+ : | # # % % % | : | # # # ##* ## O#+% OO* %+%* + %*+ x*x x x xx x | : |# ## # #OO*#O OO* *OO%OOOOOOOO*OOOOOOO**@ xOO*%x*#%* * x xx x x x x x| : | |________A|_____||_|_AM___AAMM___|_||________A_________| | : +------------------------------------------------------------------------------+ : N Min Max Median Avg Stddev : x 30 1383 1653 1493 1496.1 69.360801 : + 30 1200 1511 1383 1363.5667 69.974716 : Difference at 95.0% confidence : -132.533 +/- 36.0126 : -8.85859% +/- 2.4071% : (Student's t, pooled s = 69.6684) : * 30 1208 1496 1336 1333.1667 79.330858 : Difference at 95.0% confidence : -162.933 +/- 38.5167 : -10.8905% +/- 2.57447% : (Student's t, pooled s = 74.5128) : % 30 1208 1488 1377 1374.1333 59.381138 : Difference at 95.0% confidence : -121.967 +/- 33.3741 : -8.15231% +/- 2.23074% : (Student's t, pooled s = 64.5641) : # 30 1124 1363 1246 1245.7667 59.806979 : Difference at 95.0% confidence : -250.333 +/- 33.4755 : -16.7324% +/- 2.23752% : (Student's t, pooled s = 64.7603) It looks like it's probably worth pushing the limit up to 8 MB when we have lots of memory :)