From owner-svn-src-head@FreeBSD.ORG Mon May 30 17:52:55 2011 Return-Path: Delivered-To: svn-src-head@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D6A51065670; Mon, 30 May 2011 17:52:55 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail08.syd.optusnet.com.au (mail08.syd.optusnet.com.au [211.29.132.189]) by mx1.freebsd.org (Postfix) with ESMTP id 2421F8FC16; Mon, 30 May 2011 17:52:54 +0000 (UTC) Received: from c122-106-165-191.carlnfd1.nsw.optusnet.com.au (c122-106-165-191.carlnfd1.nsw.optusnet.com.au [122.106.165.191]) by mail08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p4UHqkkY005457 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 31 May 2011 03:52:47 +1000 Date: Tue, 31 May 2011 03:52:46 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: mdf@FreeBSD.org In-Reply-To: Message-ID: <20110531032658.N5049@besplex.bde.org> References: <201105131848.p4DIm1j7079495@svn.freebsd.org> <201105282103.43370.pieter@degoeje.nl> <20110531004247.C4034@besplex.bde.org> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-993397294-1306777966=:5049" Cc: svn-src-head@FreeBSD.org, Pieter de Goeje , svn-src-all@FreeBSD.org, src-committers@FreeBSD.org, Bruce Evans Subject: Re: svn commit: r221853 - in head/sys: dev/md dev/null sys vm X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2011 17:52:55 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-993397294-1306777966=:5049 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Mon, 30 May 2011 mdf@FreeBSD.org wrote: > On Mon, May 30, 2011 at 8:25 AM, Bruce Evans wrote= : >> On Sat, 28 May 2011 mdf@FreeBSD.org wrote: >>> ... >>> Meanwhile you could try setting ZERO_REGION_SIZE to PAGE_SIZE and I >>> think that will restore things to the original performance. >> >> Using /dev/zero always thrashes caches by the amount > size> + (unless the arch uses nontemporal memory >> accesses for uiomove, which none do AFAIK). =A0So a large source buffer >> is always just a pessimization. =A0A large target buffer size is also a >> pessimization, but for the target buffer a fairly large size is needed >> to amortize the large syscall costs. =A0In this PR, the target buffer >> size is 64K. =A0ZERO_REGION_SIZE is 64K on i386 and 2M on amd64. =A064K+= 64K >> on i386 is good for thrashing the L1 cache. > > That depends -- is the cache virtually or physically addressed? The > zero_region only has 4k (PAGE_SIZE) of unique physical addresses. So > most of the cache thrashing is due to the user-space buffer, if the > cache is physically addressed. Oops. I now remember thinking that the much larger source buffer would be OK since it only uses 1 physical page. But it is apparently virtually addressed. > =A0It will only have a >> noticeable impact on a current L2 cache in competition with other >> threads. =A0It is hard to fit everything in the L1 cache even with >> non-bloated buffer sizes and 1 thread (16 for the source (I)cache, 0 >> for the source (D)cache and 4K for the target cache might work). =A0On >> amd64, 2M+2M is good for thrashing most L2 caches. =A0In this PR, the >> thrashing is limited by the target buffer size to about 64K+64K, up >> from 4K+64K, and it is marginal whether the extra thrashing from the >> larger source buffer makes much difference. >> >> The old zbuf source buffer size of PAGE_SIZE was already too large. > > Wouldn't this depend on how far down from the use of the buffer the > actual copy happens? Another advantage to a large virtual buffer is > that it reduces the number of times the copy loop in uiomove has to > return up to the device layer that initiated the copy. This is all > pretty fast, but again assuming a physical cache fewer trips is > better. Yes, I had forgotten that I have to keep going back to the uiomove() level for each iteration. That's a lot of overhead although not nearly as much as going back to the user level. If this is actually important to optimize, then I might add a repeat count to uiomove() and copyout() (actually a different function for the latter). linux-2.6.10 uses a mmapped /dev/zero and has had this since Y2K according to its comment. Sigh. You will never beat that by copying, but I think mmapping /dev/zero is only much more optimal for silly benchmarks. linux-2.6.10 also has a seekable /dev/zero. Seeks don't really work, but some of them "succeed" and keep the offset at 0 . ISTR remember a FreeBSD PR about the file offset for /dev/zero not "working" because it is garbage instead of 0. It is clearly a Linuxism to depend on it being nonzero. IIRC, the file offset for device files is at best implementation-defined in POSIX. Bruce --0-993397294-1306777966=:5049--