Date: Wed, 18 Jul 2012 10:08:52 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Harm Weites <harm@weites.com> Cc: freebsd-embedded@freebsd.org, Bernhard Schmidt <bschmidt@freebsd.org> Subject: Re: TP-Link wr1043nd out of swap space Message-ID: <CAJ-Vmo=L6QnJ4QwkbVo6gHu2dOB5aDez1u5DA3mnGm99xuphvw@mail.gmail.com> In-Reply-To: <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <CAOfEmZhPeEiiwUfn=5dt=vnkARV6551r%2BR_pSZrU4=07NiA8ow@mail.gmail.com> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <CAOfEmZjLbqD941=wg7ReruhgY4u%2Bu0R7KkeDexFMUVQVssKdCQ@mail.gmail.com> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <CAJ-Vmo=YTkVsn1eiGPpXewi_48BwDj=A72DwAWsUY_9X7=xC1w@mail.gmail.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net>
next in thread | previous in thread | raw e-mail | index | archive | help
.. christ, has this really broken so significantly? I haven't updated my 1043nd in a couple months (as I have other test devices); are you sure you're correctly defining MALLOC_PRODUCTION? I'll see if I can/should just add NBUF=128 to the kernel configuration files, to save a little extra RAM. Thanks for that pointer. FWIW, I'm running this: FreeBSD home-11bg-ap 10.0-CURRENT FreeBSD 10.0-CURRENT #2 r234855:234941M: Wed Dec 31 16:00:00 PST 1969 adrian@dummy:/home/adrian/work/freebsd/svn/obj/mipseb/mips.mips/usr/home/adrian/work/freebsd/svn/src/sys/TP-WN1043ND mips .. so maybe try updating to that revision and see if you still see good/bad memory usage? I'd really appreciate it if both of you could build some kernel/world revisions and help me track down where this memory usage went up. I need the help. :) Thanks! Adrian On 16 July 2012 14:02, Harm Weites <harm@weites.com> wrote: > Hi, > > setting NBUF to 128 didn't bring any noticable change. > > I've changed /etc/rc to just start /bin/sh to make it easier to run some > diagnostics right after kernel boot, here are some of my findings. > > r238194 > this is quite interesting since there are no (user) processes running, > apart from /bin/sh. > ---------------- > rtl8366rb0port0: link state changed to UP > *** Start /bin/sh > pid 18 (sh), uid 0, was killed: out of swap space > Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to > single user mode^M > Enter full pathname of shell or RETURN for /bin/sh: > # vmstat > pid 20 (sh), uid 0, was killed: out of swap space > Jul 16 11:59:41 init: single user shell terminated > > > r235767 > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 > 83 2 18 80 > > r228268 > Right after kernel boot (so without active networking/services): > ---------------- > # vmstat > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr fl0 in sy cs us > sy id > 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 > 9 91 > > And after initializing networking (and starting hostapd): > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 > 1 5 94 > > Furthermore, after manually starting all scripts and observing vmstat > after each step, I noticed a decrease from 13M to 10M after starting the > wifi script (this starting hostapd). > > r231714 with the following processes: > -hostapd > -dropbear > -dhcprelay > -syslogd > -rtadvd > -dhclient > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 > 1 2 97 > > Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in > the 'out of swap space' error. Hopefully I can at least tweak those > services a little, or perhaps there is something with a smaller > footprint already in ports :) > > I can only hope ~ 3000k is enough to route traffic... > > r228256:228258 > This is from the image Adrian put online, where hostapd isn't running; > just inetd. > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 > 2 18 80 > > I am by no means a kernel adept, so I can't do much but show my > observations upon different kernel/userland configurations. > > Any tips/pointers to aid in the dig are greatly appreciated. > > Perhaps someone else with a 1043ND can offer his/her findings with any > particular kernel revision. > > regards > > Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: >> On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: >> > Hi, >> > >> > I would really appreciate it if people (read; not me) would be able to >> > do the digging needed to get to the bottom of user/kernel memory >> > usage. >> > >> > I really need to focus on just the net80211/wifi stack side of things. >> > I'm going to focus on getting the ath(4) memory usage down over the >> > next few months so it remains feasible to run on 32MB platforms, as >> > those still ship. But I can't keep the rest of the kernel and userland >> > in check. >> > >> > Thanks, >> > >> > >> > Adrian >> >> I had to chase down "out of swap space" aborts on an ARM platform with >> 64MB not long ago, and I discovered that the kernel by default allocates >> 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 >> after that). I added "option NBUF=128" to our kernel config and that >> limited wired vfs buffer space to about 2MB, which seems much more >> reasonable for an embedded platform that does relatively little disk IO. >> >> I suspect the NBUF value could go even lower, but I'm also afraid that >> making it too low will lead to other problems; I don't really know >> enough to make an informed decision. So far the 128 value is working >> well in testing, but we haven't actually put any units in the field with >> that setting (I think we will pretty soon). >> >> -- Ian >> >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=L6QnJ4QwkbVo6gHu2dOB5aDez1u5DA3mnGm99xuphvw>