Date: Wed, 7 Apr 2010 20:05:56 +0200 From: Peter Schuller <peter.schuller@infidyne.com> To: freebsd-fs@FreeBSD.org Cc: rercola@acm.jhu.edu Subject: ZFS arc sizing (maybe related to kern/145229) Message-ID: <z2o5a1151761004071105kb129ca4q7dfd002270d53561@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello, I found this PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=145229 Part of the behavior matches mine, which is that I would see the ARC sit at around 200 MB in size and stay there - even though the minimum arc size was set to 800 MB. While I cannot say anything about 8.0-RC1 vs. 8.0 as talked about in the PR, I'm definitely seeing some unexpected ARC sizing. For various anecdotal reasons I thought that use of mmap() could be the triggering factor, so I wrote a small test program: http://distfiles.scode.org/mlref/alloc.c (compile with: gcc -g -std=c99 -o alloc alloc.c). It can be used to allocate a certain number of MB and fill it with data. Allocation by either malloc() or mmap(): ./alloc malloc 30000 ./alloc mmap 30000 # will write to an mkstemp generated file in pwd I have monitored memory use as reported by top, and the ARC size according to arcstats.size, while running this. Some observations (a timeline follows with points marked using "== Tn =="): * In general behavior is great after a fresh reboot; the ARC is at its maximum size (after filling it). * More generall, so far I have never seen the ARC shrink too much as long as the 'free' memory ('free' in top terms) is high. * After a fresh boot I can successfully force the ARC to shrink by allocating memory - malloc() or mmap(), both work. * After killing alloc, I can get the ARC to grow again by reading some large files. So it's not as simple as "it'll shrink but never grow", for both malloc() and mmap() based contention. * A day or so later the ARC had become a bit smaller - now at ~ 630 MB (ktorrent running in the background doing some reads, otherwise normal desktop). This is T1 on the timeline below. * By running alloc I was successfully able to make the ARC shrink a bit more, and then grow back again by reading data. This is T1-T2. Note how it fluctuates and seems to want to stay at around 600 MB even though I'm reading in data, and even though there was extra 'free' as a result of the alloc run. * Around T2 I re-ran alloc again and let it eat a significant amount of memory (4 GB). * Then I megan reading big files again. At first it upped a bit and landed on almost exactly 800, which is suspicously correlated with the 800 MB minimum arc size. * After further reading, it finally (T3) decides to grow further up to the maximum. In general my anecdotal feeling is that memory tends to end up in the active or inactive categories in top (instead of free), resulting in a shrunk ARC. By pushing whatever is in active/inactive away using the allocation program, I can then, upon killing it, convert that memory to 'free' again and allowing the ARC to size itself properly. I have not yet checked any code, I'm just anecdotally observing that the ARC sizing seems to be a function of mostly the 'free' memory. Related is that I only run ZFS on this machine, and it is unclear to me *what* it is that is being counted as inactive/active to any great extent (is there a good way to figure this out?). Worth mentioning is that "unexplained" inactive/active memory reminds me of an old thread I started: http://freebsd.monkey.org/freebsd-current/200709/msg00201.html That thread dealt mostly with a discrepancy in total memory use, which was apparantly a bug, but I distinctly remember the same behavior back then. I saw non-wired but active/inactive (but non-free) memory and I was never sure *what* it was, and the only way to make it go back to free was to do allocations. Back to today and ktorrent, note that stopping ktorrent (thus implying munmap()) does *not* instantly produce free memory. In the abscence of non-ZFS file systems that might cause caching to be visible as active/inactive, I'm not sure why I am seeing such memory accounted for. Time line follows; these are just dates and ARC sizes in MB resulting from a little for loop looking at arcstats.size: Wed Apr 7 19:36:17 CEST 2010: 627.361 Wed Apr 7 19:36:20 CEST 2010: 626.334 Wed Apr 7 19:36:23 CEST 2010: 626.196 Wed Apr 7 19:36:27 CEST 2010: 623.965 Wed Apr 7 19:36:30 CEST 2010: 626.42 Wed Apr 7 19:36:33 CEST 2010: 626.963 Wed Apr 7 19:36:36 CEST 2010: 626.947 Wed Apr 7 19:36:39 CEST 2010: 628.621 Wed Apr 7 19:36:43 CEST 2010: 628.937 Wed Apr 7 19:36:46 CEST 2010: 628.984 Wed Apr 7 19:36:49 CEST 2010: 629.015 == T1 == Wed Apr 7 19:36:52 CEST 2010: 406.518 Wed Apr 7 19:36:55 CEST 2010: 411.562 Wed Apr 7 19:36:59 CEST 2010: 414.874 Wed Apr 7 19:37:02 CEST 2010: 435.895 Wed Apr 7 19:37:05 CEST 2010: 517.359 Wed Apr 7 19:37:08 CEST 2010: 616.899 Wed Apr 7 19:37:11 CEST 2010: 718.44 Wed Apr 7 19:37:14 CEST 2010: 502.596 Wed Apr 7 19:37:17 CEST 2010: 587.716 Wed Apr 7 19:37:20 CEST 2010: 682.35 Wed Apr 7 19:37:23 CEST 2010: 541.922 Wed Apr 7 19:37:26 CEST 2010: 631.701 Wed Apr 7 19:37:29 CEST 2010: 529.565 Wed Apr 7 19:37:33 CEST 2010: 534.683 Wed Apr 7 19:37:36 CEST 2010: 539.814 Wed Apr 7 19:37:39 CEST 2010: 521.79 Wed Apr 7 19:37:42 CEST 2010: 517.15 Wed Apr 7 19:37:45 CEST 2010: 519.398 Wed Apr 7 19:37:48 CEST 2010: 524.562 Wed Apr 7 19:37:51 CEST 2010: 553.67 == T2 == Wed Apr 7 19:37:54 CEST 2010: 649.677 Wed Apr 7 19:37:57 CEST 2010: 737.425 Wed Apr 7 19:38:00 CEST 2010: 800.162 Wed Apr 7 19:38:03 CEST 2010: 800.324 Wed Apr 7 19:38:07 CEST 2010: 800.628 Wed Apr 7 19:38:10 CEST 2010: 801.115 Wed Apr 7 19:38:13 CEST 2010: 802.433 Wed Apr 7 19:38:17 CEST 2010: 800.622 Wed Apr 7 19:38:21 CEST 2010: 800.585 Wed Apr 7 19:38:24 CEST 2010: 800.913 Wed Apr 7 19:38:27 CEST 2010: 800.773 Wed Apr 7 19:38:31 CEST 2010: 800.905 Wed Apr 7 19:38:34 CEST 2010: 800.891 Wed Apr 7 19:38:37 CEST 2010: 800.592 Wed Apr 7 19:38:40 CEST 2010: 801.034 Wed Apr 7 19:38:44 CEST 2010: 800.535 == T3 == Wed Apr 7 19:38:47 CEST 2010: 877.722 Wed Apr 7 19:38:51 CEST 2010: 990.442 Wed Apr 7 19:38:54 CEST 2010: 1085.77 Wed Apr 7 19:38:57 CEST 2010: 1168.08 Wed Apr 7 19:39:00 CEST 2010: 1276.54 Wed Apr 7 19:39:03 CEST 2010: 1340.78 Wed Apr 7 19:39:06 CEST 2010: 1443.13 Wed Apr 7 19:39:09 CEST 2010: 1561.67 Wed Apr 7 19:39:12 CEST 2010: 1623.57 Wed Apr 7 19:39:15 CEST 2010: 1661.85 Wed Apr 7 19:39:18 CEST 2010: 1663.32 Wed Apr 7 19:39:21 CEST 2010: 1670.45 Wed Apr 7 19:39:24 CEST 2010: 1671.15 Wed Apr 7 19:39:28 CEST 2010: 1674.46 Wed Apr 7 19:39:31 CEST 2010: 1678.91 Wed Apr 7 19:39:34 CEST 2010: 1681.35 Wed Apr 7 19:39:37 CEST 2010: 1683.25 Wed Apr 7 19:39:40 CEST 2010: 1685.97 Wed Apr 7 19:39:43 CEST 2010: 1688.21 Wed Apr 7 19:39:46 CEST 2010: 1696.55 Wed Apr 7 19:39:49 CEST 2010: 1698.85 Wed Apr 7 19:39:52 CEST 2010: 1700 Wed Apr 7 19:39:55 CEST 2010: 1700.01 Wed Apr 7 19:39:58 CEST 2010: 1700.02 Wed Apr 7 19:40:01 CEST 2010: 1700.03 -- / Peter Schuller
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?z2o5a1151761004071105kb129ca4q7dfd002270d53561>