Date: Mon, 24 Mar 2003 01:57:32 -0800 From: "Andrew Kinney" <andykinney@advantagecom.net> To: freebsd-hackers@freebsd.org Subject: shared mem and panics when out of PV Entries Message-ID: <3E7E660C.10419.ECBF5C7@localhost>
next in thread | raw e-mail | index | archive | help
Hello,
(very long message with background information on the issue
follows)
The machine discussed in this email is tracking RELENG_4_7.
The machine is a well loaded web/mail/ftp server with dual Athlon
MP 2000+ CPUs, 4GB of RAM, and 8GB of available swap space.
The most it has ever swapped was 292KB (yes, KB not MB) and
the CPUs are about 60% idle most of the time. I don't believe we're
pushing the hardware all that hard given the specifications of the
system.
We're getting panics ("page fault" panic traced back to
pmap_insert_entry) because we're running out of PV Entries. I've
bumped up PMAP_SHPGPERPROC to 400, then 800, and then
1500 over the last several months in an attempt to solve this
problem by increasing the PV Entries limit (per the warning in
pmap_collect). Each time, we still bumped the limit and got
panics due to running out of PV Entries. I've verified that maxed
PV Entries are indeed the cause of the panics by logging sysctl
vm.zone. Our PV Entry limit is now 11113502 (from sysctl
vm.zone) and we're still running into this limit.
With 1GB KVA space, I really shouldn't take
PMAP_SHPGPERPROC much higher since each PV Entry takes
28 bytes of KVA space and with 11113502 PV Entries, that is
nearly 300MB of KVA space used just for PV Entries. Also, from
past experience, if you set PMAP_SHPGPERPROC too high, the
system will not boot. I'm not 100% sure, but I believe that would
happen right around PMAP_SHPGPERPROC=1600 from my
calculations.
Now, I could increase KVA space by way of increasing
KVA_PAGES and presumably continue increasing
PMAP_SHPGPERPROC until the problem goes away, but per a
previous discussion, there are problems (related to pthreads) with
increasing KVA_PAGES in FreeBSD 4.7. This has apparently
been fixed in FreeBSD 4.8.
Since moving to FreeBSD 4.8 isn't an option I can exercise in the
near term, I'd like to approach this problem from a different angle
and possibly solve it without needing to increase KVA_PAGES. In
my opinion, increasing KVA_PAGES would only be a short term
bandaid for the PV Entries issue. I'll eventually need to increase
KVA_PAGES for a different reason, though, because sysctl
vm.kvm_free does hit "0" sometimes after the system has been
running for awhile, but I'll tackle that issue if/when it becomes a
stability issue.
The source of the PV Entry usage is shared memory in Apache
(PHP and/or mod_perl). We know it's Apache memory sharing
that causes the problem because nearly all PV Entries are freed
when Apache is stopped and the problem is worst right after
Apache starts when you have a lot of "clean" memory segments
getting shared (10 to 11 million PV Entries used). After Apache
has run for awhile and the shared memory segments become
unshared due to them being "dirtied", the number of PV Entries
used declines and settles in at a much lower number (3 to 5 million
PV Entries used most days). Apache on this machine likes to
hover right around 256 children processes, so you could see how
requiring duplicate PV Entries per shared memory segment per
Apache process could add up quite quickly.
Under ideal circumstances, we'd correct our PHP/PERL code to
avoid running into the issue at all, but this is customer code that
we don't have much control over since we allow users to select and
run their own CGI scripts without our intervention. So, for this
instance, we have to solve it at the system level by bullet-proofing
our system as much as possible.
I've read that setting sysctl kern.ipc.shm_use_phys to "1" will
cause shared memory segments not to use PV Entries at all.
(See
http://www.geocrawler.com/archives/3/159/2002/6/50/9031353/ )
However, in the real world with FreeBSD 4.7, setting
kern.ipc.shm_use_phys to "1" seems to have no visible effect on
the number of PV Entries consumed by Apache memory sharing.
We're currently running sysctl kern.ipc.shm_use_phys=1 and are
still seeing the heavy PV Entry usage after rebooting (sysctl set on
boot) with no apparent difference in usage.
My thoughts at this point are leaning towards setting some limits
on shared memory (kern.ipc.shmmax, kern.ipc.shmseg,
kern.ipc.shmall, and kern.ipc.shmmax sysctl's) to limit the number
of PV Entries that Apache can consume by way of shared
memory. Is that a viable approach to this problem? What kind of
gotchas and caveats should I watch out for in taking this approach?
Any recommendations on which one to tweak first?
If I'm headed in completely the wrong direction to solve this PV
Entries limit issue, I'd appreciate any alternative approaches to
solving the problem that anyone is willing to offer.
If you need any further information about our settings or usage,
please let me know. I've been as thorough as possible in this
email without getting overly verbose, but in complex issues like this
I recognize that I may not have included all the information needed
for the experts to make a fair assessment of the issue and suggest
work arounds. Also, though I believe that the highly technical
nature of this message made it suitable for posting to freebsd-
hackers, if it would be better suited to a different list, please point
me in the right direction.
Thanks in advance for any advice or assistance you can offer.
Sincerely,
Andrew Kinney
President and
Chief Technology Officer
Advantagecom Networks, Inc.
http://www.advantagecom.net
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3E7E660C.10419.ECBF5C7>