Date: Wed, 19 Mar 2014 05:59:49 -0500 From: Karl Denninger <karl@denninger.net> To: Adrian Chadd <adrian@freebsd.org> Cc: Alan Cox <alc@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Tracking down what has inact pages locked up Message-ID: <532978A5.3050707@denninger.net> In-Reply-To: <CAJ-VmomFVv_xUwhgcnEpaHX%2BZesXdmR4GVr4rKSjA2dJK687NQ@mail.gmail.com> References: <53260B36.2070409@denninger.net> <201403181505.47349.jhb@freebsd.org> <5328A024.6050901@denninger.net> <201403181730.02471.jhb@freebsd.org> <532910D1.3010704@denninger.net> <CAJ-VmomFVv_xUwhgcnEpaHX%2BZesXdmR4GVr4rKSjA2dJK687NQ@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Yes, but I will have to be strategic on this -- probably have to wait for the weekend as this particular box is one that will generate a lot of screaming if I reboot it to turn in-kernel NAT back on (say much less play on the console in single-user mode) during the work week :-) I did that and didn't see anything sucking up the RAM -- which was what got my attention. On 3/18/2014 11:53 PM, Adrian Chadd wrote: > Can you dump out vmstat -z during the leak and once you've unloaded it? > > > -a > > > On 18 March 2014 20:36, Karl Denninger <karl@denninger.net> wrote: >> On 3/18/2014 4:30 PM, John Baldwin wrote: >>> On Tuesday, March 18, 2014 3:36:04 pm Karl Denninger wrote: >>>> On 3/18/2014 2:05 PM, John Baldwin wrote: >>>>> On Sunday, March 16, 2014 4:36:06 pm Karl Denninger wrote: >>>>>> Is there a reasonable way to determine who or what has that memory >>>>>> locked up -- and thus why the vm system is not demoting that space into >>>>>> the cache bucket so it can be freed (which, if my understanding is >>>>>> correct, should be happening long before now!) >>>>> I have a hackish thing (for 8.x, might work on 10.x) to let you figure >>>>> out >>>>> what is using up RAM. This should perhaps go into the base system at >>>>> some >>>>> point. >>>>> >>>>> Grab the bits at http://people.freebsd.org/~jhb/vm_objects/ >>>>> >>>>> You will want to build the kld first and use 'make load' to load it. It >>>>> adds >>>>> a new sysctl that dumps info about all the VM objects in the system. >>>>> You can >>>>> then build the 'vm_objects' tool and run it. It can take a while to run >>>>> if >>>>> you have NFS mounts, so I typically save its output to a file first and >>>>> then >>>>> use sort on the results. sort -n will show you the largest consumer of >>>>> RAM, >>>>> sort -n -k 3 will show you the largest consumer of inactive pages. Note >>>>> that 'df' and 'ph' objects are anonymous, and that filename paths aren't >>>>> always reliable, but this can still be useful. >>>>> >>>> Thanks. >>>> >>>> I suspect the cause of the huge inact consumption is a RAM leak in the >>>> NAT code in IPFW. It was not occurring in 9.2-STABLE, but is on >>>> 10.0-STABLE, and reverting to natd in userland stops it -- which >>>> pretty-well isolates where it's coming from. >>> Memory for in-kernel NAT should be wired pages, not inactive. >> Yeah, should be. :-) >> >> But..... it managed to lock up 19GB of the 24GB the system has in inact >> pages over 12 hours, and dropping the system to single user and unloading >> the modules did not release the RAM...... which is why the question (on how >> to track down what the hell is going on.) >> >> Changing the config back to natd as opposed to in-kernel NAT, however, made >> the problem disappear. >> >> -- >> -- Karl >> karl@denninger.net >> >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > %SPAMBLOCK-SYS: Matched [@freebsd.org+], message ok -- -- Karl karl@denninger.net [-- Attachment #2 --] 0 *H 01 0 + 0 *H O0K030 *H 01 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 130824190344Z 180823190344Z0[1 0 UUS10UFlorida10UKarl Denninger1!0 *H karl@denninger.net0"0 *H 0 bi՞] MNԿawx?`)'ҴcWgR@BlWh+ u}ApdCF JVй~FOL }EW^bچYp3K& ׂ(R lxڝ.xz?6 &nsJ +1v9 v/( kqĪp[vjcK%fϻe?iq]z lyzFO'ppdX//Lw(3 JI A*S#՟H[f|CGqJKoo y.oEuOw$ /섀$삻J9b|A P~8]D1YI<"""Y^T2iQ2 b yH)] Ƶ0y$ _N 6XqMC 9 XgώjGTP"#nˋ"Bk1 00 U 0 0 `HB0 U 0, `HB OpenSSL Generated Certificate0 U |8 ˴d[20 U #0]Af4 U3x&^"408 `HB+)https://cudasystems.net:11443/revoked.crl0 *H gBwH] j\x`( &gW32"Uf^. ^Iϱ k!DQA g{(w /)\N'[ oRW @C HO>)XrTNɘ!u`xt5(=f\-l3<@C6mnhv##1ŃbH͍_Nq aʷ?rk$^9TIa!kh,D -ct1 0001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 + ;0 *H 1 *H 0 *H 1 140319105949Z0# *H 18Pii|o0l *H 1_0]0 `He*0 `He0 *H 0*H 0 *H @0+0 *H (0 +71001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H 101 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H yŰ921)榥9 DBkaMņRXtZ %q%Fi W`+)کčJܗt- b|&-''/LxtVS#B &+\d?)Wn7 p6XO_&w<YN䁑8C_T<\HN1 ]jw2l褠=wHD ire^n"'}DwID YTDmu%fARS|'XhYqZQ&>F[m&pA].9$O0l> c +J- ݍ k:2c&%|I4EKWMEA_Vd[0, pt:;#]F1 -R Iű > *ƿwNbe dUfǏ wBe'DvzQ @,RL**Q>*T{f?-help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?532978A5.3050707>