From owner-freebsd-fs@FreeBSD.ORG Fri Feb 14 20:23:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AFA71983 for ; Fri, 14 Feb 2014 20:23:27 +0000 (UTC) Received: from mail-pa0-f46.google.com (mail-pa0-f46.google.com [209.85.220.46]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8442F12E2 for ; Fri, 14 Feb 2014 20:23:27 +0000 (UTC) Received: by mail-pa0-f46.google.com with SMTP id rd3so12646602pab.5 for ; Fri, 14 Feb 2014 12:23:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=llOyIgKR6Wr/RDZCzBTBVNVgq0HJ8OpWDTWzMm/pXHA=; b=aHJm7+Jjojkhk4peoVGuXuUgJ3iw7Cs/gNyO2SbCDHKcriHV30xVEFyAjn0pE7Mql+ dLZqxLvFKffzeth9vKRecZDZKHE/4gZiZpYFP9J2QpxcUJN3gxn1/gWIsHXhCP0UG0yO LqFJqPtX33DyNkV0IzDyCHsDo31RKMZh1Sq/pkeZu+HlZylcmOtrXYUKWoxS9xOZd0me S3j2wl/7inOYyMG9Gq//4kVtIuJ7yciFfEtNGMN07esuiXPz9MEpQF/tfDIPEmvmYRIT 9GxnJ+Es9IwPIE3jqvbcfUEZOHh81SE+vI/iKJXJcXwb7kNEFyYV5eWqoe02Gr2/zmt7 53vA== X-Gm-Message-State: ALoCoQkjsW4fz+xUSenJnma/zmoMTyQGZFTmJp4rl1RKoy3jTmMdihUHupukXP8Y5mg1VA85sl66 MIME-Version: 1.0 X-Received: by 10.69.1.2 with SMTP id bc2mr11355305pbd.102.1392409400845; Fri, 14 Feb 2014 12:23:20 -0800 (PST) Received: by 10.70.44.165 with HTTP; Fri, 14 Feb 2014 12:23:20 -0800 (PST) In-Reply-To: <52FE0378.7070608@FreeBSD.org> References: <52B2D8D6.8090306@FreeBSD.org> <52FE0378.7070608@FreeBSD.org> Date: Fri, 14 Feb 2014 12:23:20 -0800 Message-ID: Subject: Re: l2arc_feed_thread cpu utlization From: Brendan Gregg To: Andriy Gapon Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 20:23:27 -0000 G'Day Andriy, Thanks for the patch. If most of the data is in one list (anyone have statistics to confirm such a likelyhood? I know this happened a lot pre-list-split), then I think this means we only scan that at 1/32nd of the previous rate. It should solve the CPU issue, but could make warmup very slow. I think the feed algorithm needs to be rethought, although that can be done as future work. I'm trying to think of what simple that can be done right now to solve CPU usage and warmup rate. Lets say we keep this change, but in l2arc_write_buffers we maintain an extra copy of write_sz, say, list_write_sz, that is reset to zero for each list. Then, when we reach headroom and choose to abort, we can check list_write_sz and determine how fruitful the scanning has been so far. If that's greater than a threshold, then keep scanning, up to the full L2ARC_WRITE_SIZE for that list. That way, we've scanned only 1/32nd of the previous length as a test, and only if that is fruitful enough do we keep scanning. Again, it probably needs to be rethought, but something like that may work fine in the meantime. Brendan On Fri, Feb 14, 2014 at 3:52 AM, Andriy Gapon wrote: > on 19/12/2013 13:30 Andriy Gapon said the following: > > > > This is just a heads up, no patch yet. > > > > l2arc_feed_thread periodically wakes up and scans certain amount of ARC > buffers > > and writes eligible buffers to a cache device. > > Number of scanned buffers is limited by a threshold on the amount of > data in the > > buffers seen. The threshold is applied on a per buffer list basis. In > upstream > > there are 4 relevant lists: (data, metadata) X (MFU, MRU). In FreeBSD > each of > > the lists was subdivided into 16 lists. This was done to reduce > contention on > > the locks that protect the lists. But as a side effect > l2arc_feed_thread can > > scan 16 times more data (~ buffers). > > > > So, if you have a rather large ARC and L2ARC and your buffers tend to be > > sufficiently small, then you could observe l2arc_feed_thread burning a > > noticeable amount of CPU. On some of our systems I observed it using up > to 40% > > of a single core. Scaling back the threshold by factor of 16 makes CPU > > utilization go down by the same factor. > > > > I plan to commit this change to FreeBSD ZFS code. > > Any comments are welcome. > > Here is what I have in mind: > https://github.com/avg-I/freebsd/compare/wip;hc;l2arc_feed_thread_scan_rate > > The calculations in the macro look somewhat ugly, but they should be > correct :-) > > -- > Andriy Gapon > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Brendan Gregg, Joyent http://dtrace.org/blogs/brendan