From owner-freebsd-fs@FreeBSD.ORG  Thu Dec 19 11:31:34 2013
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id B429AB12;
 Thu, 19 Dec 2013 11:31:34 +0000 (UTC)
Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140])
 by mx1.freebsd.org (Postfix) with ESMTP id D57071628;
 Thu, 19 Dec 2013 11:31:33 +0000 (UTC)
Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua
 [212.40.38.100])
 by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA09033;
 Thu, 19 Dec 2013 13:31:32 +0200 (EET) (envelope-from avg@FreeBSD.org)
Received: from localhost ([127.0.0.1])
 by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD))
 id 1VtbpD-00007h-OC; Thu, 19 Dec 2013 13:31:31 +0200
Message-ID: <52B2D8D6.8090306@FreeBSD.org>
Date: Thu, 19 Dec 2013 13:30:30 +0200
From: Andriy Gapon <avg@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:24.0) Gecko/20100101 Thunderbird/24.2.0
MIME-Version: 1.0
To: freebsd-fs <freebsd-fs@FreeBSD.org>
Subject: l2arc_feed_thread cpu utlization
X-Enigmail-Version: 1.6
Content-Type: text/plain; charset=X-VIET-VPS
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Dec 2013 11:31:34 -0000


This is just a heads up, no patch yet.

l2arc_feed_thread periodically wakes up and scans certain amount of ARC buffers
and writes eligible buffers to a cache device.
Number of scanned buffers is limited by a threshold on the amount of data in the
buffers seen.  The threshold is applied on a per buffer list basis.  In upstream
there are 4 relevant lists: (data, metadata) X (MFU, MRU).  In FreeBSD each of
the lists was subdivided into 16 lists.  This was done to reduce contention on
the locks that protect the lists.  But as a side effect l2arc_feed_thread can
scan 16 times more data (~ buffers).

So, if you have a rather large ARC and L2ARC and your buffers tend to be
sufficiently small, then you could observe l2arc_feed_thread burning a
noticeable amount of CPU.  On some of our systems I observed it using up to 40%
of a single core.  Scaling back the threshold by factor of 16 makes CPU
utilization go down by the same factor.

I plan to commit this change to FreeBSD ZFS code.
Any comments are welcome.
-- 
Andriy Gapon