Date: Fri, 14 Feb 2014 12:23:20 -0800 From: Brendan Gregg <brendan.gregg@joyent.com> To: Andriy Gapon <avg@freebsd.org> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: l2arc_feed_thread cpu utlization Message-ID: <CA%2BXzFFgN1Mq94ydeA=F1uy3oA=OPiF0w4yrWr4smXY9ovR%2Bqdg@mail.gmail.com> In-Reply-To: <52FE0378.7070608@FreeBSD.org> References: <52B2D8D6.8090306@FreeBSD.org> <52FE0378.7070608@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
G'Day Andriy, Thanks for the patch. If most of the data is in one list (anyone have statistics to confirm such a likelyhood? I know this happened a lot pre-list-split), then I think this means we only scan that at 1/32nd of the previous rate. It should solve the CPU issue, but could make warmup very slow. I think the feed algorithm needs to be rethought, although that can be done as future work. I'm trying to think of what simple that can be done right now to solve CPU usage and warmup rate. Lets say we keep this change, but in l2arc_write_buffers we maintain an extra copy of write_sz, say, list_write_sz, that is reset to zero for each list. Then, when we reach headroom and choose to abort, we can check list_write_sz and determine how fruitful the scanning has been so far. If that's greater than a threshold, then keep scanning, up to the full L2ARC_WRITE_SIZE for that list. That way, we've scanned only 1/32nd of the previous length as a test, and only if that is fruitful enough do we keep scanning. Again, it probably needs to be rethought, but something like that may work fine in the meantime. Brendan On Fri, Feb 14, 2014 at 3:52 AM, Andriy Gapon <avg@freebsd.org> wrote: > on 19/12/2013 13:30 Andriy Gapon said the following: > > > > This is just a heads up, no patch yet. > > > > l2arc_feed_thread periodically wakes up and scans certain amount of ARC > buffers > > and writes eligible buffers to a cache device. > > Number of scanned buffers is limited by a threshold on the amount of > data in the > > buffers seen. The threshold is applied on a per buffer list basis. In > upstream > > there are 4 relevant lists: (data, metadata) X (MFU, MRU). In FreeBSD > each of > > the lists was subdivided into 16 lists. This was done to reduce > contention on > > the locks that protect the lists. But as a side effect > l2arc_feed_thread can > > scan 16 times more data (~ buffers). > > > > So, if you have a rather large ARC and L2ARC and your buffers tend to be > > sufficiently small, then you could observe l2arc_feed_thread burning a > > noticeable amount of CPU. On some of our systems I observed it using up > to 40% > > of a single core. Scaling back the threshold by factor of 16 makes CPU > > utilization go down by the same factor. > > > > I plan to commit this change to FreeBSD ZFS code. > > Any comments are welcome. > > Here is what I have in mind: > https://github.com/avg-I/freebsd/compare/wip;hc;l2arc_feed_thread_scan_rate > > The calculations in the macro look somewhat ugly, but they should be > correct :-) > > -- > Andriy Gapon > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- Brendan Gregg, Joyent http://dtrace.org/blogs/brendan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BXzFFgN1Mq94ydeA=F1uy3oA=OPiF0w4yrWr4smXY9ovR%2Bqdg>