From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 12 00:58:10 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9AFE2106566B for ; Mon, 12 Mar 2012 00:58:10 +0000 (UTC) (envelope-from freebsd-hackers@herveybayaustralia.com.au) Received: from mail.unitedinsong.com.au (mail.unitedinsong.com.au [150.101.178.33]) by mx1.freebsd.org (Postfix) with ESMTP id DE4818FC0C for ; Mon, 12 Mar 2012 00:58:09 +0000 (UTC) Received: from mail.unitedinsong.com.au (bell.herveybayaustralia.com.au [192.168.0.40]) by mail.unitedinsong.com.au (Postfix) with ESMTP id 235655C28 for ; Mon, 12 Mar 2012 11:11:40 +1000 (EST) Received: from laptop1.herveybayaustralia.com.au (laptop1.herveybayaustralia.com.au [192.168.0.177]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.unitedinsong.com.au (Postfix) with ESMTPSA id E8FCA5C22 for ; Mon, 12 Mar 2012 11:11:39 +1000 (EST) Message-ID: <4F5D48E6.9010802@herveybayaustralia.com.au> Date: Mon, 12 Mar 2012 10:52:54 +1000 From: Da Rock <9Phackers@herveybayaustralia.com.au> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111109 Thunderbird/7.0.1 MIME-Version: 1.0 To: "hackers@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 12 Mar 2012 02:33:11 +0000 Cc: Subject: Kernel memory usage X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "hackers@freebsd.org" List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Mar 2012 00:58:10 -0000 I may be required to move this to embedded, but I am only looking for generalisation. Recently a thread came up on questions regarding memory usage, and a post was made regarding wired memory being nearly all kernel- something I was ready to dispute, but then I thought I'd better make sure. So I tested a few theories first off: 1. Comparing memory usage across machines I checked servers and desktops as well as vm's for memory usage, and I found some interesting results. On a firewall with no apps installed only 35M wired is used, on a desktop up to 700M~ can be used. Even as a dedicated server with a few services used it remains around 35M. Surely this means that the wired memory used is not just kernel? But I held off my assumptions as it was still plausible that the structures used by the kernel could balloon that far, too. 2. Stripped down, lean mean, kernel machine I then (using a vm I was building a kernel for anyway) stripped down a kernel in a VBox VM using le drivers for network to see what could be achieved. This is my kernel conf: include GENERIC ident VPN options IPSEC options IPSEC_DEBUG options IPSEC_NAT_T device crypto device enc # minimise kernel nooptions UFS_GJOURNAL nooptions MD_ROOT nooptions NFSCL nooptions NFSD nooptions NFSLOCKD nooptions NFS_ROOT nooptions MSDOSFS nooptions CD9660 nooptions PROCFS nooptions PSEUDOFS nodevice fdc nodevice mvs nodevice siis nodevice ahc nodevice ahd nodevice esp nodevice hptiop nodevice isp nodevice mpt nodevice mps nodevice sym nodevice trm nodevice adv nodevice adw nodevice aic nodevice bt nodevice ses nodevice amr nodevice arcmsr nodevice ciss nodevice dpt nodevice hptmv nodevice hptrr nodevice irr nodevice ips nodevice mly nodevice twa nodevice aac nodevice aacp nodevice ida nodevice mfi nodevice mlx nodevice twe nodevice tws nodevice splash nodevice cbb nodevice pccard nodevice cardbus nodevice uart nodevice ppc nodevice ppbus nodevice lpt nodevice plip nodevice ppi nodevice puc nodevice bxe nodevice de nodevice em nodevice igb nodevice ixgbe nodevice ti nodevice txp nodevice vx nodevice miibus nodevice ae nodevice age nodevice alc nodevice ale nodevice bce nodevice bfe nodevice bge nodevice dc nodevice et nodevice fxp nodevice jme nodevice lge nodevice msk nodevice nfe nodevice nge nodevice pcn nodevice re nodevice rl nodevice sf nodevice sge nodevice sis nodevice sk nodevice ste nodevice stge nodevice tl nodevice tx nodevice vge nodevice vr nodevice wb nodevice xl nodevice cs nodevice ed nodevice ex nodevice ep nodevice fe nodevice sn nodevice xe nodevice wlan nooptions IEEE80211_DEBUG nooptions IEEE80211_AMPDU_AGE nooptions IEEE80211_SUPPORT_MESH nodevice wlan_wep nodevice wlan_ccmp nodevice wlan_tkip nodevice wlan_amrr nodevice an nodevice ath nodevice ath_pci nodevice ath_hal nooptions AH_SUPPORT_AR5416 nodevice ath_rate_sample nodevice ipw nodevice iwi nodevice iwn nodevice malo nodevice mwl nodevice ral nodevice wi nodevice wpi nodevice md nooption USB_DEBUG nodevice uhci nodevice ohci nodevice ehci nodevice xhci nodevice usb nodevice uhid nodevice ukbd nodevice ulpt nodevice umass nodevice ums nodevice urio nodevice u3g nodevice uark nodevice ubsa nodevice uftdi nodevice uipaq nodevice uplcom nodevice uslcom nodevice uvisor nodevice uvscom nodevice aue nodevice axe nodevice cdce nodevice cue nodevice kue nodevice rue nodevice udav nodevice rum nodevice run nodevice uath nodevice upgt nodevice ural nodevice urtw nodevice zyd #nodevice firewire nodevice fwe nodevice fwip #nodevice dcons #nodevice dcons_rom nodevice sound nodevice snd_es137x nodevice snd_hda nodevice snd_ich nodevice snd_uaudio nodevice snd_via8233 World was also rebuilt as recommended by the handbook. As you can see I was rebuilding for IPSEC (for testing purposes only note). I couldn't remove dcons (not sure why- missing defined references), and that pulled in firewire too. The result was surprising: a 14M kernel became 6.3M, and when running I found it only used 19M used for wired (whoopee!) as opposed to 35M. No services were running per se, only the usual suspects in base: vpn-test# ps ax PID TT STAT TIME COMMAND 0 ?? DLs 0:00.11 [kernel] 1 ?? ILs 0:00.01 /sbin/init -- 2 ?? DL 0:00.00 [crypto] 3 ?? DL 0:00.00 [crypto returns] 4 ?? DL 0:00.00 [sctp_iterator] 5 ?? DL 0:00.00 [xpt_thrd] 6 ?? DL 0:00.10 [pagedaemon] 7 ?? DL 0:00.00 [vmdaemon] 8 ?? DL 0:00.01 [pagezero] 9 ?? DL 0:00.30 [bufdaemon] 10 ?? DL 0:00.00 [audit] 11 ?? RL 589:34.00 [idle] 12 ?? WL 0:38.64 [intr] 13 ?? DL 0:12.21 [geom] 14 ?? DL 0:03.30 [yarrow] 15 ?? DL 0:04.40 [syncer] 16 ?? DL 0:00.63 [vnlru] 17 ?? DL 0:00.53 [softdepflush] 104 ?? Is 0:00.00 adjkerntz -i 582 ?? Is 0:00.00 /sbin/devd 737 ?? Ss 0:00.09 /usr/sbin/syslogd -s 912 ?? Ss 0:02.68 /usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid 977 ?? Is 0:00.00 /usr/sbin/sshd 984 ?? Ss 0:00.81 sendmail: accepting connections (sendmail) 988 ?? Is 0:00.01 sendmail: Queue runner@00:30:00 for /var/spool/client 994 ?? Ss 0:00.12 /usr/sbin/cron -s 2061 ?? Is 0:00.03 sshd: admin [priv] (sshd) 2064 ?? S 0:00.04 sshd: admin@pts/0 (sshd) 1052 v0 Is 0:00.01 login [pam] (login) 1060 v0 I+ 0:00.02 -csh (csh) 1053 v1 Is+ 0:00.00 /usr/libexec/getty Pc ttyv1 1054 v2 Is+ 0:00.00 /usr/libexec/getty Pc ttyv2 1055 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3 1056 v4 Is+ 0:00.00 /usr/libexec/getty Pc ttyv4 1057 v5 Is+ 0:00.00 /usr/libexec/getty Pc ttyv5 1058 v6 Is+ 0:00.00 /usr/libexec/getty Pc ttyv6 1059 v7 Is+ 0:00.00 /usr/libexec/getty Pc ttyv7 2065 0 Is 0:00.02 -csh (csh) 2068 0 I 0:00.01 su 2069 0 S 0:00.04 _su (csh) 2414 0 R+ 0:00.00 ps ax After some minutes though, the usage went up to 39M. WTF? So I'm now thoroughly confused. Google didn't really show up any real answers either. Aside from discovering the true nature of what was discussed in the posts, my hypothetical theorising is for my own academics as well; and I'm trying to slim down my kernels to see if I can improve performance with low memory systems. Can process structures cause this much ballooning? I find that hard to believe given not much is running here. My only other thought would be pipe or network structures? Anything I'm missing? FWIW, my trimmed down, slim and slender, aerodynamic kernel can boot in seconds as opposed to minutes. Sweet! Although compiling clang took years... and gcc was built as well :/ Cheers