From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 19 14:12:04 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AB88716A4B3 for ; Fri, 19 Sep 2003 14:12:04 -0700 (PDT) Received: from visp.engelschall.com (visp.engelschall.com [195.27.176.148]) by mx1.FreeBSD.org (Postfix) with ESMTP id 317E143FE0 for ; Fri, 19 Sep 2003 14:12:01 -0700 (PDT) (envelope-from rse@engelschall.com) Received: by visp.engelschall.com (Postfix, from userid 1005) id 0DF1F4CE574; Fri, 19 Sep 2003 23:12:01 +0200 (CEST) Received: by en1.engelschall.com (Postfix, from userid 10000) id 20A6D2877C; Fri, 19 Sep 2003 23:11:39 +0200 (CEST) Date: Fri, 19 Sep 2003 23:11:39 +0200 From: "Ralf S. Engelschall" To: freebsd-hackers@freebsd.org Message-ID: <20030919211138.GA67987@engelschall.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="a8Wt8u1KmwUX3Y2C" Content-Disposition: inline User-Agent: Mutt/1.4.1i Organization: Engelschall, Germany. X-Web-Homepage: http://www.engelschall.com/ X-PGP-Public-Key: http://www.engelschall.com/ho/rse/pgprse.asc X-PGP-Fingerprint: 00 C9 21 8E D1 AB 70 37 DD 67 A2 3A 0A 6F 8D A5 Subject: Gaps in memory usage summary!? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: rse@engelschall.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Sep 2003 21:12:04 -0000 --a8Wt8u1KmwUX3Y2C Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I'm trying to automate the summarization of memory usage on our FreeBSD boxes with the attached little Perl program and have some remaining questions I was not able to answer myself. When run on three representing boxes (all -STABLE and -CURRENT as of yesterday) I get this output: ------------------------------------------------------------------------------ FreeBSD-4.8-STABLE/i386 SYSTEM MEMORY INFORMATION: mem_wire: 162615296 ( 155MB) [ 7%] Wired: disabled for paging out mem_active: + 159956992 ( 152MB) [ 7%] Active: recently referenced mem_inactive:+ 950210560 ( 906MB) [ 45%] Inactive: recently not referenced mem_cache: + 75137024 ( 71MB) [ 3%] Cached: almost avail. for allocation mem_free: + 759664640 ( 724MB) [ 36%] Free: fully available for allocation mem_gap_vm: + 364544 ( 0MB) [ 0%] Memory gap: UNKNOWN -------------- ------------ ----------- ------ mem_all: = 2107949056 ( 2010MB) [100%] Total real memory managed mem_gap_sys: + 35684352 ( 34MB) Memory gap: Kernel?! -------------- ------------ ----------- mem_phys: = 2143633408 ( 2044MB) Total real memory available mem_gap_hw: + 3850240 ( 3MB) Memory gap: Segment Mappings?! -------------- ------------ ----------- mem_hw: = -2147483648 ( 2048MB) Total real memory installed SYSTEM MEMORY SUMMARY: mem_used: 322936832 ( 307MB) [ 15%] Logically unused memory mem_avail: + 1785012224 ( 1702MB) [ 84%] Logically available memory -------------- ------------ ----------- ------ mem_total: = 2107949056 ( 2010MB) [100%] Logically total memory ------------------------------------------------------------------------------ FreeBSD-5.1-CURRENT/i386 SYSTEM MEMORY INFORMATION: mem_wire: 136101888 ( 129MB) [ 12%] Wired: disabled for paging out mem_active: + 187867136 ( 179MB) [ 17%] Active: recently referenced mem_inactive:+ 648417280 ( 618MB) [ 61%] Inactive: recently not referenced mem_cache: + 9150464 ( 8MB) [ 0%] Cached: almost avail. for allocation mem_free: + 68550656 ( 65MB) [ 6%] Free: fully available for allocation mem_gap_vm: + 0 ( 0MB) [ 0%] Memory gap: UNKNOWN -------------- ------------ ----------- ------ mem_all: = 1050087424 ( 1001MB) [100%] Total real memory managed mem_gap_sys: + 19030016 ( 18MB) Memory gap: Kernel?! -------------- ------------ ----------- mem_phys: = 1069117440 ( 1019MB) Total real memory available mem_gap_hw: + 4624384 ( 4MB) Memory gap: Segment Mappings?! -------------- ------------ ----------- mem_hw: = 1073741824 ( 1024MB) Total real memory installed SYSTEM MEMORY SUMMARY: mem_used: 323969024 ( 308MB) [ 30%] Logically unused memory mem_avail: + 726118400 ( 692MB) [ 69%] Logically available memory -------------- ------------ ----------- ------ mem_total: = 1050087424 ( 1001MB) [100%] Logically total memory ------------------------------------------------------------------------------ FreeBSD-5.1-CURRENT/alpha SYSTEM MEMORY INFORMATION: mem_wire: 32555008 ( 31MB) [ 12%] Wired: disabled for paging out mem_active: + 57344000 ( 54MB) [ 22%] Active: recently referenced mem_inactive:+ 116170752 ( 110MB) [ 45%] Inactive: recently not referenced mem_cache: + 4620288 ( 4MB) [ 1%] Cached: almost avail. for allocation mem_free: + 12484608 ( 11MB) [ 4%] Free: fully available for allocation mem_gap_vm: + 31113216 ( 29MB) [ 12%] Memory gap: UNKNOWN -------------- ------------ ----------- ------ mem_all: = 254287872 ( 242MB) [100%] Total real memory managed mem_gap_sys: + 11116544 ( 10MB) Memory gap: Kernel?! -------------- ------------ ----------- mem_phys: = 265404416 ( 253MB) Total real memory available mem_gap_hw: + 3031040 ( 2MB) Memory gap: Segment Mappings?! -------------- ------------ ----------- mem_hw: = 268435456 ( 256MB) Total real memory installed SYSTEM MEMORY SUMMARY: mem_used: 121012224 ( 115MB) [ 47%] Logically unused memory mem_avail: + 133275648 ( 127MB) [ 52%] Logically available memory -------------- ------------ ----------- ------ mem_total: = 254287872 ( 242MB) [100%] Logically total memory ------------------------------------------------------------------------------ My remaining questions now are: 1. What memory is "map_gap_vm" and why is there no such gap on Intel boxes but on Alpha boxes? The number I arithmetically calculate is vm.stats.vm.v_page_count - ( vm.stats.vm.v_wire_count + vm.stats.vm.v_active_count + vm.stats.vm.v_inactive_count + vm.stats.vm.v_cache_count + vm.stats.vm.v_free_count ) and I always would expect 0 here, but as you can see it is not at least on Alpha. Is this a bug in our VM statistic code? Any clues? 2. What memory is "mem_gap_sys", i.e. (hw.physmem - (vm.stats.vm.v_page_count * hw.pagesize)) ? Is this the loaded kernel code? 3. What memory is "mem_gap_hw", i.e. (rounded(hw.physmem) - hw.physmem) ? Is this caused by some special hardware memory segmenting? I would be happy if someone could shed some light on this. Thanks. Ralf S. Engelschall rse@engelschall.com www.engelschall.com --a8Wt8u1KmwUX3Y2C Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=freebsd-memory #!/usr/bin/perl # query the system through the generic sysctl(8) interface # (this does not require special priviledges) my $sysctl = {}; my $sysctl_output = `/sbin/sysctl -a`; foreach my $line (split(/\n/, $sysctl_output)) { if ($line =~ m/^([^:]+):\s+(.+)\s*$/s) { $sysctl->{$1} = $2; } } # round the physical memory size to the next power of two which is # reasonable for memory cards. We do this by first determining the # guessed memory card size under the assumption that usual computer # hardware has an average of a maximally eight memory cards installed # and those are usually of equal size. sub mem_rounded { my ($mem_size) = @_; my $chip_size = 1; my $chip_guess = ($mem_size / 8) - 1; while ($chip_guess != 0) { $chip_guess >>= 1; $chip_size <<= 1; } my $mem_round = (int($mem_size / $chip_size) + 1) * $chip_size; return $mem_round; } # determine the individual known information # NOTICE: forget hw.usermem, it is just (hw.physmem - vm.stats.vm.v_wire_count). # NOTICE: forget vm.stats.misc.zero_page_count, it is just the subset of # vm.stats.vm.v_free_count which is already pre-zeroed. my $mem_hw = &mem_rounded($sysctl->{"hw.physmem"}); my $mem_phys = $sysctl->{"hw.physmem"}; my $mem_all = $sysctl->{"vm.stats.vm.v_page_count"} * $sysctl->{"hw.pagesize"}; my $mem_wire = $sysctl->{"vm.stats.vm.v_wire_count"} * $sysctl->{"hw.pagesize"}; my $mem_active = $sysctl->{"vm.stats.vm.v_active_count"} * $sysctl->{"hw.pagesize"}; my $mem_inactive = $sysctl->{"vm.stats.vm.v_inactive_count"} * $sysctl->{"hw.pagesize"}; my $mem_cache = $sysctl->{"vm.stats.vm.v_cache_count"} * $sysctl->{"hw.pagesize"}; my $mem_free = $sysctl->{"vm.stats.vm.v_free_count"} * $sysctl->{"hw.pagesize"}; # determine the individual unknown information my $mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache + $mem_free); my $mem_gap_sys = $mem_phys - $mem_all; my $mem_gap_hw = $mem_hw - $mem_phys; # determine logical summary information my $mem_used = $mem_wire + $mem_active + $mem_gap_vm; my $mem_avail = $mem_inactive + $mem_cache + $mem_free; my $mem_total = $mem_used + $mem_avail; # information annotations my $info = { "mem_wire" => 'Wired: disabled for paging out', "mem_active" => 'Active: recently referenced', "mem_inactive" => 'Inactive: recently not referenced', "mem_cache" => 'Cached: almost avail. for allocation', "mem_free" => 'Free: fully available for allocation', "mem_gap_vm" => 'Memory gap: UNKNOWN', "mem_all" => 'Total real memory managed', "mem_gap_sys" => 'Memory gap: Kernel?!', "mem_phys" => 'Total real memory available', "mem_gap_hw" => 'Memory gap: Segment Mappings?!', "mem_hw" => 'Total real memory installed', "mem_used" => 'Logically unused memory', "mem_avail" => 'Logically available memory', "mem_total" => 'Logically total memory', }; # print system results printf("SYSTEM MEMORY INFORMATION:\n"); printf("mem_wire: %12d (%7dMB) [%3d%%] %s\n", $mem_wire, $mem_wire / (1024*1024), ($mem_wire / $mem_all) * 100, $info->{"mem_wire"}); printf("mem_active: + %12d (%7dMB) [%3d%%] %s\n", $mem_active, $mem_active / (1024*1024), ($mem_active / $mem_all) * 100, $info->{"mem_active"}); printf("mem_inactive:+ %12d (%7dMB) [%3d%%] %s\n", $mem_inactive, $mem_inactive / (1024*1024), ($mem_inactive / $mem_all) * 100, $info->{"mem_inactive"}); printf("mem_cache: + %12d (%7dMB) [%3d%%] %s\n", $mem_cache, $mem_cache / (1024*1024), ($mem_cache / $mem_all) * 100, $info->{"mem_cache"}); printf("mem_free: + %12d (%7dMB) [%3d%%] %s\n", $mem_free, $mem_free / (1024*1024), ($mem_free / $mem_all) * 100, $info->{"mem_free"}); printf("mem_gap_vm: + %12d (%7dMB) [%3d%%] %s\n", $mem_gap_vm, $mem_gap_vm / (1024*1024), ($mem_gap_vm / $mem_all) * 100, $info->{"mem_gap_vm"}); printf("-------------- ------------ ----------- ------\n"); printf("mem_all: = %12d (%7dMB) [100%%] %s\n", $mem_all, $mem_all / (1024*1024), $info->{"mem_all"}); printf("mem_gap_sys: + %12d (%7dMB) %s\n", $mem_gap_sys, $mem_gap_sys / (1024*1024), $info->{"mem_gap_sys"}); printf("-------------- ------------ -----------\n"); printf("mem_phys: = %12d (%7dMB) %s\n", $mem_phys, $mem_phys / (1024*1024), $info->{"mem_phys"}); printf("mem_gap_hw: + %12d (%7dMB) %s\n", $mem_gap_hw, $mem_gap_hw / (1024*1024), $info->{"mem_gap_hw"}); printf("-------------- ------------ -----------\n"); printf("mem_hw: = %12d (%7dMB) %s\n", $mem_hw, $mem_hw / (1024*1024), $info->{"mem_hw"}); # print logical results printf("\n"); printf("SYSTEM MEMORY SUMMARY:\n"); printf("mem_used: %12d (%7dMB) [%3d%%] %s\n", $mem_used, $mem_used / (1024*1024), ($mem_used / $mem_total) * 100, $info->{"mem_used"}); printf("mem_avail: + %12d (%7dMB) [%3d%%] %s\n", $mem_avail, $mem_avail / (1024*1024), ($mem_avail / $mem_total) * 100, $info->{"mem_avail"}); printf("-------------- ------------ ----------- ------\n"); printf("mem_total: = %12d (%7dMB) [100%%] %s\n", $mem_total, $mem_total / (1024*1024), $info->{"mem_total"}); --a8Wt8u1KmwUX3Y2C--