From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 20:03:12 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D5761065695; Mon, 3 Jan 2011 20:03:12 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 50EBE8FC24; Mon, 3 Jan 2011 20:03:10 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 0C5D56B624F; Mon, 3 Jan 2011 21:03:09 +0100 (CET) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000041, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 14.1274] X-CRM114-CacheID: sfid-20110103_21030_C28A4C88 X-CRM114-Status: Good ( pR: 14.1274 ) X-Spambayes-Classification: ham; 0.00 Message-ID: <4D222B7B.1090902@fsn.hu> Date: Mon, 03 Jan 2011 21:03:07 +0100 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.23) Gecko/20090817 Thunderbird/2.0.0.23 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Artem Belevich References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2011 20:03:12 -0000 On 01/01/2011 08:09 PM, Artem Belevich wrote: > On Sat, Jan 1, 2011 at 10:18 AM, Attila Nagy wrote: >> What I see: >> - increased CPU load >> - decreased L2 ARC hit rate, decreased SSD (ad[46]), therefore increased >> hard disk load (IOPS graph) >> > ... >> Any ideas on what could cause these? I haven't upgraded the pool version and >> nothing was changed in the pool or in the file system. > The fact that L2 ARC is full does not mean that it contains the right > data. Initial L2ARC warm up happens at a much higher rate than the > rate L2ARC is updated after it's been filled initially. Even > accelerated warm-up took almost a day in your case. In order for L2ARC > to warm up properly you may have to wait quite a bit longer. My guess > is that it should slowly improve over the next few days as data goes > through L2ARC and those bits that are hit more often take residence > there. The larger your data set, the longer it will take for L2ARC to > catch the right data. > > Do you have similar graphs from pre-patch system just after reboot? I > suspect that it may show similarly abysmal L2ARC hit rates initially, > too. > > After four days, the L2 hit rate is still hovering around 10-20 percents (was between 60-90), so I think it's clearly a regression in the ZFSv28 patch... And the massive growth in CPU usage can also very nicely be seen... I've updated the graphs at (switch time can be checked on the zfs-mem graph): http://people.fsn.hu/~bra/freebsd/20110101-zfsv28-fbsd/ There is a new phenomenom: the large IOPS peaks. I use this munin script on a lot of machines and never seen anything like this... I'm not sure whether it's related or not.