From owner-freebsd-stable@FreeBSD.ORG Sun Jan 2 22:38:58 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89104106564A; Sun, 2 Jan 2011 22:38:58 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3B7B28FC13; Sun, 2 Jan 2011 22:38:58 +0000 (UTC) Received: by iyb26 with SMTP id 26so12163674iyb.13 for ; Sun, 02 Jan 2011 14:38:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :organization:user-agent:mime-version:to:cc:subject:references :in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; bh=DlS+VaP+DxZGRHWUSqBfykgdwbniHs8vw0f/JeOI5q0=; b=AxqWk9WWEDNdPW++/1jlRKBvF/N968XA7tvSYfSE+7SX1eWqWix0x5tq0H+owYF5lV jMVygvFGau9wgwZO0wKfNZJA1PBHNj1IKIW8YUCSHn68vdoOLmGjMoASUP20DVjxe4Bn Hn/AA0BqUFVOqeElyxbKeMuWm4vdwx022W0Fg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:organization:user-agent:mime-version:to :cc:subject:references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=YmURWDdFUedpRN0hrLamu/Deu2teI7pjjZk/Hrt5XDDdwmnWy4oTwB7s3CmNsoK8A+ qYtFwB3SqMc+9LW1IQmyuCHDWqVm8B76zI1mFcdMFAmG4qw8Bt/LhbMnl1AF7ywQ+VLT CR0t1mLSoO0OCbXKwhXSDdy7I/Tm7NsS7E1Nc= Received: by 10.231.171.197 with SMTP id i5mr3820668ibz.54.1294007937440; Sun, 02 Jan 2011 14:38:57 -0800 (PST) Received: from centel.dataix.local (adsl-99-181-155-97.dsl.klmzmi.sbcglobal.net [99.181.155.97]) by mx.google.com with ESMTPS id z4sm18133772ibg.13.2011.01.02.14.38.54 (version=SSLv3 cipher=RC4-MD5); Sun, 02 Jan 2011 14:38:55 -0800 (PST) Sender: "J. Hellenthal" Message-ID: <4D20FE7D.6030803@DataIX.net> Date: Sun, 02 Jan 2011 17:38:53 -0500 From: "J. Hellenthal" Organization: http://www.DataIX.net User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.13) Gecko/20101230 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Attila Nagy References: <4D0A09AF.3040005@FreeBSD.org> <4D1F7008.3050506@fsn.hu> <4D1FF9AC.60200@DataIX.net> <4D203B43.7070607@fsn.hu> In-Reply-To: <4D203B43.7070607@fsn.hu> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: New ZFSv28 patchset for 8-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Jan 2011 22:38:58 -0000 On 01/02/2011 03:45, Attila Nagy wrote: > On 01/02/2011 05:06 AM, J. Hellenthal wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 01/01/2011 13:18, Attila Nagy wrote: >>> On 12/16/2010 01:44 PM, Martin Matuska wrote: >>>> Link to the patch: >>>> >>>> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101215.patch.xz >>>> >>>> >>>> >>>> >>> I've used this: >>> http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101223-nopython.patch.xz >>> >>> >>> on a server with amd64, 8 G RAM, acting as a file server on >>> ftp/http/rsync, the content being read only mounted with nullfs in >>> jails, and the daemons use sendfile (ftp and http). >>> >>> The effects can be seen here: >>> http://people.fsn.hu/~bra/freebsd/20110101-zfsv28-fbsd/ >>> the exact moment of the switch can be seen on zfs_mem-week.png, where >>> the L2 ARC has been discarded. >>> >>> What I see: >>> - increased CPU load >>> - decreased L2 ARC hit rate, decreased SSD (ad[46]), therefore increased >>> hard disk load (IOPS graph) >>> >>> Maybe I could accept the higher system load as normal, because there >>> were a lot of things changed between v15 and v28 (but I was hoping if I >>> use the same feature set, it will require less CPU), but dropping the >>> L2ARC hit rate so radically seems to be a major issue somewhere. >>> As you can see from the memory stats, I have enough kernel memory to >>> hold the L2 headers, so the L2 devices got filled up to their maximum >>> capacity. >>> >>> Any ideas on what could cause these? I haven't upgraded the pool version >>> and nothing was changed in the pool or in the file system. >>> >> Running arc_summary.pl[1] -p4 should print a summary about your l2arc >> and you should also notice in that section that there is a high number >> of "SPA Mismatch" mine usually grew to around 172k before I would notice >> a crash and I could reliably trigger this while in scrub. >> >> What ever is causing this needs desperate attention! >> >> I emailed mm@ privately off-list when I noticed this going on but have >> not received any feedback as of yet. > It's at zero currently (2 days of uptime): > kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 > Right but do you have a 'cache' 'l2arc' vdev attached to any pool in the system ? This suggests to me that you do not at this time. If not can you attach a cache vdev and run a scrub on it and monitor the value of that MIB ? -- Regards, jhell,v JJH48-ARIN