Date: Thu, 26 Nov 2009 10:14:20 -0500 From: Linda Messerschmidt <linda.messerschmidt@gmail.com> To: freebsd-hackers@freebsd.org Subject: Superpages on amd64 FreeBSD 7.2-STABLE Message-ID: <237c27100911260714x2fcb194ew1e6ce11e764efd08@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
We have a squid proxy process with very large memory requirements (10 - 20 GB) on a machine with 24GB of RAM. Unfortunately, we have to rotate the logs of this process once per day. When we do, it fork()s and exec()s about 16-20 child processes as helpers. Since it's got this multi-million-entry page table, that's a disaster, because it has to copy all those page table entries for each child, then throw them out. This takes a couple of minutes of 100% CPU usage, during which time the machine is pretty much unresponsive. Someone on the squid list suggested we try the new superpages feature (vm.pmap.pg_ps_enabled) in 7.2. We did, and after some tuning, we got it to work. Here's some "sysctl vm.pmap" for a similar machine with 16GB of RAM that does NOT have this setting enabled: vm.pmap.pv_entry_count: 2307899 vm.pmap.pde.promotions: 0 vm.pmap.pde.p_failures: 0 vm.pmap.pde.mappings: 0 vm.pmap.pde.demotions: 0 vm.pmap.pv_entry_max: 4276871 vm.pmap.pg_ps_enabled: 0 Now, here is the machine that does have it, just prior to the daily rotation mentioned above: vm.pmap.pv_entry_count: 61361 vm.pmap.pde.promotions: 23123 vm.pmap.pde.p_failures: 327946 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 17848 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 So it obviously this feature makes a huge difference and is a brilliant idea. :-) My (limited) understanding is that one of the primary benefits of this feature is to help situations like ours... a page table that's 512x smaller can be copied 512x faster. However, in practice this doesn't happen. It's like fork() breaks up the squid process into 4kb pages again. Here's the same machine's entries just after rotation: vm.pmap.pv_entry_count: 1908056 vm.pmap.pde.promotions: 23212 vm.pmap.pde.p_failures: 413171 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 21470 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 So some 3,600 superpages spontaneously turned into 1,850,000 4k pages. Once this happens, squid seems reluctant to use more superpages until its restarted. We get a lot of p_failures and a slow-but-steady stream of demotions. Here's the same machine just now: vm.pmap.pv_entry_count: 2022786 vm.pmap.pde.promotions: 25281 vm.pmap.pde.p_failures: 996027 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 21683 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 And a few minutes later: vm.pmap.pv_entry_count: 2021556 vm.pmap.pde.promotions: 25331 vm.pmap.pde.p_failures: 1001773 vm.pmap.pde.mappings: 1641 vm.pmap.pde.demotions: 21684 vm.pmap.pv_entry_max: 7330186 vm.pmap.pg_ps_enabled: 1 (There were *no* p_failures or demotions in the several hours prior to rotation.) This trend continues... the pv_entry_count bounces up and down even though memory usage is increasing, so it's like it's trying to recover and convert things back (promotions), but it's having a lot of trouble (p_failures). It's not clear to me if this might be a problem with the superpages implementation, or if squid does something particularly horrible to its memory when it forks to cause this, but I wanted to ask about it on the list in case somebody who understands it better might know whats going on. :-) This is on FreeBSD-STABLE 7.2 amd64 r198976M. Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?237c27100911260714x2fcb194ew1e6ce11e764efd08>