Date: Sat, 15 Feb 2014 13:58:37 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: Brendan Gregg <brendan.gregg@joyent.com> Cc: freebsd-fs <freebsd-fs@FreeBSD.org> Subject: Re: l2arc_feed_thread cpu utlization Message-ID: <52FF566D.3060601@FreeBSD.org> In-Reply-To: <CA%2BXzFFgN1Mq94ydeA=F1uy3oA=OPiF0w4yrWr4smXY9ovR%2Bqdg@mail.gmail.com> References: <52B2D8D6.8090306@FreeBSD.org> <52FE0378.7070608@FreeBSD.org> <CA%2BXzFFgN1Mq94ydeA=F1uy3oA=OPiF0w4yrWr4smXY9ovR%2Bqdg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
on 14/02/2014 22:23 Brendan Gregg said the following: > G'Day Andriy, > > Thanks for the patch. If most of the data is in one list (anyone have statistics > to confirm such a likelyhood? I know this happened a lot pre-list-split), then I > think this means we only scan that at 1/32nd of the previous rate. It should > solve the CPU issue, but could make warmup very slow. Brendan, I do not have any stats, but I think that the data should be spread more or less evenly between the lists. I mean the 16 sub-lists for data and 16 sub-lists for metadata. First, a list is picked up based on hash and that _should_ produce more or less even distribution. Second, if the hash funciton is not good enough then whole list splitting is pointless. In either case this was just a quick hack on my part. > I think the feed algorithm needs to be rethought, although that can be done as > future work. I'm trying to think of what simple that can be done right now to > solve CPU usage and warmup rate. I completely agree with you. I do not particularly like the fact that the threshold is per sub-list in FreeBSD. I would prefer a more "wholisitic" threshold. > Lets say we keep this change, but in l2arc_write_buffers we maintain an extra > copy of write_sz, say, list_write_sz, that is reset to zero for each list. Then, > when we reach headroom and choose to abort, we can check list_write_sz and > determine how fruitful the scanning has been so far. If that's greater than a > threshold, then keep scanning, up to the full L2ARC_WRITE_SIZE for that list. > That way, we've scanned only 1/32nd of the previous length as a test, and only > if that is fruitful enough do we keep scanning. > > Again, it probably needs to be rethought, but something like that may work fine > in the meantime. This sounds interesting. I will think more about this. Thanks! > > On Fri, Feb 14, 2014 at 3:52 AM, Andriy Gapon <avg@freebsd.org > <mailto:avg@freebsd.org>> wrote: > > on 19/12/2013 13:30 Andriy Gapon said the following: > > > > This is just a heads up, no patch yet. > > > > l2arc_feed_thread periodically wakes up and scans certain amount of ARC > buffers > > and writes eligible buffers to a cache device. > > Number of scanned buffers is limited by a threshold on the amount of data > in the > > buffers seen. The threshold is applied on a per buffer list basis. In > upstream > > there are 4 relevant lists: (data, metadata) X (MFU, MRU). In FreeBSD each of > > the lists was subdivided into 16 lists. This was done to reduce contention on > > the locks that protect the lists. But as a side effect l2arc_feed_thread can > > scan 16 times more data (~ buffers). > > > > So, if you have a rather large ARC and L2ARC and your buffers tend to be > > sufficiently small, then you could observe l2arc_feed_thread burning a > > noticeable amount of CPU. On some of our systems I observed it using up > to 40% > > of a single core. Scaling back the threshold by factor of 16 makes CPU > > utilization go down by the same factor. > > > > I plan to commit this change to FreeBSD ZFS code. > > Any comments are welcome. > > Here is what I have in mind: > https://github.com/avg-I/freebsd/compare/wip;hc;l2arc_feed_thread_scan_rate > > The calculations in the macro look somewhat ugly, but they should be correct :-) > > -- > Andriy Gapon > > _______________________________________________ > freebsd-fs@freebsd.org <mailto:freebsd-fs@freebsd.org> mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > <mailto:freebsd-fs-unsubscribe@freebsd.org>" > > > > > -- > Brendan Gregg, Joyent http://dtrace.org/blogs/brendan -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52FF566D.3060601>