From owner-freebsd-embedded@FreeBSD.ORG Wed Jul 18 17:08:54 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EB0A31065675; Wed, 18 Jul 2012 17:08:54 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3301F8FC1B; Wed, 18 Jul 2012 17:08:53 +0000 (UTC) Received: by lbon10 with SMTP id n10so3005056lbo.13 for ; Wed, 18 Jul 2012 10:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=q+QXcfYUSvJNHVhYTlJR6Xm7ci+Hf1f5aoOWe4YNiMM=; b=UoFkv3EclI7l5flZzu3rQmaT84jtU47iWkfVtnG3JFZKBmPtsaWqq5FKVrdW8pkLpe W67u/YqTNVoGpsvxGMhRlh2WJIO70sU3DnBDbqwKM8CcvKXg6b3eTlluR2G0/5LA/KND IiN4gIGLu6PryzM/ctCd2hNxU5MuQtx/nXX1vEfTFzo6GAcmsz5VxpgMAL9FPn3o00tn AZQ3qxy07+t4VwlqvcLit3Q2qR22z/4uOi56xA6F2qCPz19bSygzjHvJPiwXV6Xp1D+D eus4RXILK2/RqaYfdKcdj/j3vBp8C6fh4ZCi7lZGPOREb+WBN+Vz1ulAwpE316Nvjfnm Wkfw== MIME-Version: 1.0 Received: by 10.112.47.104 with SMTP id c8mr2109853lbn.101.1342631332622; Wed, 18 Jul 2012 10:08:52 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.112.20.197 with HTTP; Wed, 18 Jul 2012 10:08:52 -0700 (PDT) In-Reply-To: <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> Date: Wed, 18 Jul 2012 10:08:52 -0700 X-Google-Sender-Auth: RIIN43yvbnlkalGOkAtx-1w8yAk Message-ID: From: Adrian Chadd To: Harm Weites Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-embedded@freebsd.org, Bernhard Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 17:08:55 -0000 .. christ, has this really broken so significantly? I haven't updated my 1043nd in a couple months (as I have other test devices); are you sure you're correctly defining MALLOC_PRODUCTION? I'll see if I can/should just add NBUF=128 to the kernel configuration files, to save a little extra RAM. Thanks for that pointer. FWIW, I'm running this: FreeBSD home-11bg-ap 10.0-CURRENT FreeBSD 10.0-CURRENT #2 r234855:234941M: Wed Dec 31 16:00:00 PST 1969 adrian@dummy:/home/adrian/work/freebsd/svn/obj/mipseb/mips.mips/usr/home/adrian/work/freebsd/svn/src/sys/TP-WN1043ND mips .. so maybe try updating to that revision and see if you still see good/bad memory usage? I'd really appreciate it if both of you could build some kernel/world revisions and help me track down where this memory usage went up. I need the help. :) Thanks! Adrian On 16 July 2012 14:02, Harm Weites wrote: > Hi, > > setting NBUF to 128 didn't bring any noticable change. > > I've changed /etc/rc to just start /bin/sh to make it easier to run some > diagnostics right after kernel boot, here are some of my findings. > > r238194 > this is quite interesting since there are no (user) processes running, > apart from /bin/sh. > ---------------- > rtl8366rb0port0: link state changed to UP > *** Start /bin/sh > pid 18 (sh), uid 0, was killed: out of swap space > Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to > single user mode^M > Enter full pathname of shell or RETURN for /bin/sh: > # vmstat > pid 20 (sh), uid 0, was killed: out of swap space > Jul 16 11:59:41 init: single user shell terminated > > > r235767 > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 > 83 2 18 80 > > r228268 > Right after kernel boot (so without active networking/services): > ---------------- > # vmstat > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr fl0 in sy cs us > sy id > 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 > 9 91 > > And after initializing networking (and starting hostapd): > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 > 1 5 94 > > Furthermore, after manually starting all scripts and observing vmstat > after each step, I noticed a decrease from 13M to 10M after starting the > wifi script (this starting hostapd). > > r231714 with the following processes: > -hostapd > -dropbear > -dhcprelay > -syslogd > -rtadvd > -dhclient > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 > 1 2 97 > > Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in > the 'out of swap space' error. Hopefully I can at least tweak those > services a little, or perhaps there is something with a smaller > footprint already in ports :) > > I can only hope ~ 3000k is enough to route traffic... > > r228256:228258 > This is from the image Adrian put online, where hostapd isn't running; > just inetd. > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 > 2 18 80 > > I am by no means a kernel adept, so I can't do much but show my > observations upon different kernel/userland configurations. > > Any tips/pointers to aid in the dig are greatly appreciated. > > Perhaps someone else with a 1043ND can offer his/her findings with any > particular kernel revision. > > regards > > Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: >> On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: >> > Hi, >> > >> > I would really appreciate it if people (read; not me) would be able to >> > do the digging needed to get to the bottom of user/kernel memory >> > usage. >> > >> > I really need to focus on just the net80211/wifi stack side of things. >> > I'm going to focus on getting the ath(4) memory usage down over the >> > next few months so it remains feasible to run on 32MB platforms, as >> > those still ship. But I can't keep the rest of the kernel and userland >> > in check. >> > >> > Thanks, >> > >> > >> > Adrian >> >> I had to chase down "out of swap space" aborts on an ARM platform with >> 64MB not long ago, and I discovered that the kernel by default allocates >> 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 >> after that). I added "option NBUF=128" to our kernel config and that >> limited wired vfs buffer space to about 2MB, which seems much more >> reasonable for an embedded platform that does relatively little disk IO. >> >> I suspect the NBUF value could go even lower, but I'm also afraid that >> making it too low will lead to other problems; I don't really know >> enough to make an informed decision. So far the 128 value is working >> well in testing, but we haven't actually put any units in the field with >> that setting (I think we will pretty soon). >> >> -- Ian >> >> > >