From owner-freebsd-stable@FreeBSD.ORG Sun Jan 2 08:46:00 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6354F106566B; Sun, 2 Jan 2011 08:46:00 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 599718FC14; Sun, 2 Jan 2011 08:45:58 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 220416B4901; Sun, 2 Jan 2011 09:45:57 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 19.9304] X-CRM114-CacheID: sfid-20110102_09455_46FAFBEF X-CRM114-Status: Good ( pR: 19.9304 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D203B43.7070607@fsn.hu> Date: Sun, 02 Jan 2011 09:45:55 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: "J. Hellenthal" References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D1FF9AC.60200@DataIX.net> In-Reply-To: <4D1FF9AC.60200@DataIX.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, Martin Matuska Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jan 2011 08:46:00 -0000 On 01/02/2011 05:06 AM, J. Hellenthal wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 01/01/2011 13:18, Attila Nagy wrote: >> On 12/16/2010 01:44 PM, Martin Matuska wrote: >>> Link to the patch: >>> >>> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz >>> >>> >>> >> I've used this: >> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101223-nopython.patch.xz >> >> on a server with amd64, 8 G RAM, acting as a file server on >> ftp/http/rsync, the content being read only mounted with nullfs in >> jails, and the daemons use sendfile (ftp and http). >> >> The effects can be seen here: >> http://people.fsn.hu/~bra/freebsd/20110101-zfsv28-fbsd/ >> the exact moment of the switch can be seen on zfs_mem-week.png, where >> the L2 ARC has been discarded. >> >> What I see: >> - increased CPU load >> - decreased L2 ARC hit rate, decreased SSD (ad[46]), therefore increased >> hard disk load (IOPS graph) >> >> Maybe I could accept the higher system load as normal, because there >> were a lot of things changed between v15 and v28 (but I was hoping if I >> use the same feature set, it will require less CPU), but dropping the >> L2ARC hit rate so radically seems to be a major issue somewhere. >> As you can see from the memory stats, I have enough kernel memory to >> hold the L2 headers, so the L2 devices got filled up to their maximum >> capacity. >> >> Any ideas on what could cause these? I haven't upgraded the pool version >> and nothing was changed in the pool or in the file system. >> > Running arc_summary.pl[1] -p4 should print a summary about your l2arc > and you should also notice in that section that there is a high number > of "SPA Mismatch" mine usually grew to around 172k before I would notice > a crash and I could reliably trigger this while in scrub. > > What ever is causing this needs desperate attention! > > I emailed mm@ privately off-list when I noticed this going on but have > not received any feedback as of yet. It's at zero currently (2 days of uptime): kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0