From owner-freebsd-hackers@FreeBSD.ORG Mon Nov 28 03:47:41 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC5FC1065670 for ; Mon, 28 Nov 2011 03:47:41 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id 24F6F8FC08 for ; Mon, 28 Nov 2011 03:47:40 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.4/8.14.4) with ESMTP id pAS36gUU049540 for ; Sun, 27 Nov 2011 21:06:42 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.4/8.14.4/Submit) id pAS36gpR049539 for freebsd-hackers@freebsd.org; Sun, 27 Nov 2011 21:06:42 -0600 (CST) (envelope-from brooks) Date: Sun, 27 Nov 2011 21:06:42 -0600 From: Brooks Davis To: freebsd-hackers@freebsd.org Message-ID: <20111128030642.GA39998@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: FOSDEM 2012 - BSD Licensed Operating System Developers Room X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: fosdem@freebsd.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Nov 2011 03:47:41 -0000 We have a Devroom at FOSDEM 2012, which takes place on February 4-5 in Brussels, Belgium. The Devroom will mainly cover topics for BSD Licensed Operating System users, developers and contributors. For more information about the Devroom, please see: http://wiki.freebsd.org/201202DevRoom For more information about FOSDEM, please see the website at http://www.fosdem.org/2012/. We are now accepting proposals for talks. Each session will last 30 to 60 minutes (including discussion). Please note that we only accept talks in English. Talks about developments in particular operating systems are welcome as are talks or forums on ways we can work together. We also welcome presentations on foundational technologies which are of interest to developers of BSD licensed operating systems and presentations on derivative operating systems such as Debian GNU/kFreeBSD. From owner-freebsd-hackers@FreeBSD.ORG Tue Nov 29 22:50:43 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA2EF106566C for ; Tue, 29 Nov 2011 22:50:43 +0000 (UTC) (envelope-from kline@thought.org) Received: from thought.org (plato.thought.org [209.180.213.209]) by mx1.freebsd.org (Postfix) with ESMTP id BF57F8FC0C for ; Tue, 29 Nov 2011 22:50:43 +0000 (UTC) Received: by thought.org (Postfix, from userid 1001) id CEC98E8098E; Tue, 29 Nov 2011 14:34:02 -0800 (PST) Date: Tue, 29 Nov 2011 14:34:02 -0800 From: Gary Kline To: Hackers Mailing List Message-ID: <20111129223359.GA5279@thought.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Organization: Thought Unlimited. Public service Unix since 1986. Of_Interest: With 25 years of service to the Unix community. User-Agent: Mutt/1.5.21 (2010-09-15) Subject: maybe a gtk wizark here will know... X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Nov 2011 22:50:43 -0000 does anybody on -hackers know how to put a boarder around a string on a gtk widget? i have figured out how to create a default-sized gtk widget and two buttons: "inc" and "dec". inbetween the buttons is my starting point of [a string] "0". i want to put a boarder around the "0" but after looking for three solid days cannot figure out how to do this. if anybody know---or knows of a gtk tutorial with this info, will you please let me know? tia, gary -- Gary Kline kline@thought.org http://www.thought.org Public Service Unix Journey Toward the Dawn, E-Book: http://www.thought.org The 8.57a release of Jottings: http://jottings.thought.org Twenty-five years of service to the Unix community. From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 12:49:59 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A15A1106564A for ; Wed, 30 Nov 2011 12:49:59 +0000 (UTC) (envelope-from prvs=1315c564f5=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 16FD88FC0A for ; Wed, 30 Nov 2011 12:49:58 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 30 Nov 2011 12:39:16 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 30 Nov 2011 12:39:16 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016804107.msg for ; Wed, 30 Nov 2011 12:39:16 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1315c564f5=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-hackers@freebsd.org Message-ID: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> From: "Steven Hartland" To: Date: Wed, 30 Nov 2011 12:39:10 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Subject: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 12:49:59 -0000 We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avm fre flt re pi po fr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 The raw output is:- vmstat -c 2 -w 1 -n 0 -H procs memory page faults cpu r b w avm fre flt re pi po fr sr in sy cs us sy id 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 Top shows:- last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 1893 processes:1 running, 1892 sleeping CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free Swap: 4096M Total, 4096M Free sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) Virtual Memory: (Total: 1106403532K Active: 32540260K) Real Memory: (Total: 4563648K Active: 3921644K) Shared Virtual Memory: (Total: 19976K Active: 16396K) Shared Real Memory: (Total: 9040K Active: 8436K) Free Memory Pages: 2161740K As mentioned this machine has 8GB of ram and according to both top and swapinfo is using no swap at all >From dmesg:- real memory = 8589934592 (8192 MB) avail memory = 8255553536 (7873 MB) swapinfo Device 1K-blocks Used Avail Capacity /dev/gptid/09f211f7-39ce-11e0-8 4194304 0 4194304 0% uname -a FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 sysctl hw.pagesize hw.pagesize: 4096 It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- memory Information about the usage of virtual and real memory. Virtual pages (reported in units of 1024 bytes) are considered active if they belong to processes which are running or have run in the last 20 seconds. avm active virtual pages fre size of the free list Totalling up RSS from ps axo "rss" gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 16:56:02 2011 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AF82106566B; Wed, 30 Nov 2011 16:56:02 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 739FE8FC08; Wed, 30 Nov 2011 16:56:02 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 14F6D46B0A; Wed, 30 Nov 2011 11:56:02 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 8396AB921; Wed, 30 Nov 2011 11:56:01 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Wed, 30 Nov 2011 11:52:46 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <20111125073630.GC7915@DataIX.net> In-Reply-To: <20111125073630.GC7915@DataIX.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201111301152.47002.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 30 Nov 2011 11:56:01 -0500 (EST) Cc: hackers@freebsd.org Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 16:56:02 -0000 On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: > > Found a troubling result of the following and figured someone might want to take a look. > > Pay close attention to the output and behavior. > > sysctl net.inet.udp.blackhole=0 > sysctl net.inet.udp.blackhole > sysctl -d net.inet.udp.blackhole=1 > sysctl net.inet.udp.blackhole > > > Is this expected ? should it not just display the description instead of adjusting ? as well not display the description like it is adjusting the description too ? Hah, cute. It should probably fail with an error if you do something like that, yes. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 16:56:02 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AF82106566B; Wed, 30 Nov 2011 16:56:02 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 739FE8FC08; Wed, 30 Nov 2011 16:56:02 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 14F6D46B0A; Wed, 30 Nov 2011 11:56:02 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 8396AB921; Wed, 30 Nov 2011 11:56:01 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Wed, 30 Nov 2011 11:52:46 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <20111125073630.GC7915@DataIX.net> In-Reply-To: <20111125073630.GC7915@DataIX.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201111301152.47002.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 30 Nov 2011 11:56:01 -0500 (EST) Cc: hackers@freebsd.org Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 16:56:02 -0000 On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: > > Found a troubling result of the following and figured someone might want to take a look. > > Pay close attention to the output and behavior. > > sysctl net.inet.udp.blackhole=0 > sysctl net.inet.udp.blackhole > sysctl -d net.inet.udp.blackhole=1 > sysctl net.inet.udp.blackhole > > > Is this expected ? should it not just display the description instead of adjusting ? as well not display the description like it is adjusting the description too ? Hah, cute. It should probably fail with an error if you do something like that, yes. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 18:48:32 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DBE5106564A; Wed, 30 Nov 2011 18:48:32 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 00CDF8FC13; Wed, 30 Nov 2011 18:48:31 +0000 (UTC) Received: by eaai12 with SMTP id i12so1522009eaa.13 for ; Wed, 30 Nov 2011 10:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=wkuWDVyd3STQTFimlQ7dFpBMKb0x1ecNwpz2kuk5wwU=; b=lxwMHLpiyxcbDlE8363EV1Yu+uLMmtjQR4/QHl3KrYjiiGW5Em5PEZ3s7KMVQ8oEcx efaFm+EKbnLRTf0zib+RMPbidvXSRXqo29ny4SE1ra7hxAEsC9XuJS69kDOCIvyOlqh3 BZc7b/aZe6s9/O5VJr8WglhnTA6yPgvcNC+WY= Received: by 10.227.205.197 with SMTP id fr5mr1451010wbb.3.1322678910977; Wed, 30 Nov 2011 10:48:30 -0800 (PST) Received: from DataIX.net (ppp-21.135.dialinfree.com. [209.172.21.135]) by mx.google.com with ESMTPS id m13sm3038234wbh.0.2011.11.30.10.48.26 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 30 Nov 2011 10:48:30 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pAUImKDj070451 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Nov 2011 13:48:21 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pAUImFC5070450; Wed, 30 Nov 2011 13:48:15 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Wed, 30 Nov 2011 13:48:15 -0500 From: Jason Hellenthal To: John Baldwin Message-ID: <20111130184815.GA70174@DataIX.net> References: <20111125073630.GC7915@DataIX.net> <201111301152.47002.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201111301152.47002.jhb@freebsd.org> Cc: freebsd-hackers@freebsd.org, hackers@freebsd.org Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 18:48:32 -0000 On Wed, Nov 30, 2011 at 11:52:46AM -0500, John Baldwin wrote: > On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: > > > > Found a troubling result of the following and figured someone might want to > take a look. > > > > Pay close attention to the output and behavior. > > > > sysctl net.inet.udp.blackhole=0 > > sysctl net.inet.udp.blackhole > > sysctl -d net.inet.udp.blackhole=1 > > sysctl net.inet.udp.blackhole > > > > > > Is this expected ? should it not just display the description instead of > adjusting ? as well not display the description like it is adjusting the > description too ? > > Hah, cute. It should probably fail with an error if you do something like > that, yes. > Yeah thats what I thought about it to but the more I thought about it, if it just displayed the values changing instead of the description when =N is supplied I think that would be acceptable to. 0 -> 1 in this case. Or possibly sys.oid: 0 -> 1 # since sysctl.conf(5) also takes comments like that. Not really thats something at the top of the list for fixes though. Low fruit. Food for thought. From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 18:48:32 2011 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9DBE5106564A; Wed, 30 Nov 2011 18:48:32 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 00CDF8FC13; Wed, 30 Nov 2011 18:48:31 +0000 (UTC) Received: by eaai12 with SMTP id i12so1522009eaa.13 for ; Wed, 30 Nov 2011 10:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=wkuWDVyd3STQTFimlQ7dFpBMKb0x1ecNwpz2kuk5wwU=; b=lxwMHLpiyxcbDlE8363EV1Yu+uLMmtjQR4/QHl3KrYjiiGW5Em5PEZ3s7KMVQ8oEcx efaFm+EKbnLRTf0zib+RMPbidvXSRXqo29ny4SE1ra7hxAEsC9XuJS69kDOCIvyOlqh3 BZc7b/aZe6s9/O5VJr8WglhnTA6yPgvcNC+WY= Received: by 10.227.205.197 with SMTP id fr5mr1451010wbb.3.1322678910977; Wed, 30 Nov 2011 10:48:30 -0800 (PST) Received: from DataIX.net (ppp-21.135.dialinfree.com. [209.172.21.135]) by mx.google.com with ESMTPS id m13sm3038234wbh.0.2011.11.30.10.48.26 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 30 Nov 2011 10:48:30 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pAUImKDj070451 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Nov 2011 13:48:21 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pAUImFC5070450; Wed, 30 Nov 2011 13:48:15 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Wed, 30 Nov 2011 13:48:15 -0500 From: Jason Hellenthal To: John Baldwin Message-ID: <20111130184815.GA70174@DataIX.net> References: <20111125073630.GC7915@DataIX.net> <201111301152.47002.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201111301152.47002.jhb@freebsd.org> Cc: freebsd-hackers@freebsd.org, hackers@freebsd.org Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 18:48:32 -0000 On Wed, Nov 30, 2011 at 11:52:46AM -0500, John Baldwin wrote: > On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: > > > > Found a troubling result of the following and figured someone might want to > take a look. > > > > Pay close attention to the output and behavior. > > > > sysctl net.inet.udp.blackhole=0 > > sysctl net.inet.udp.blackhole > > sysctl -d net.inet.udp.blackhole=1 > > sysctl net.inet.udp.blackhole > > > > > > Is this expected ? should it not just display the description instead of > adjusting ? as well not display the description like it is adjusting the > description too ? > > Hah, cute. It should probably fail with an error if you do something like > that, yes. > Yeah thats what I thought about it to but the more I thought about it, if it just displayed the values changing instead of the description when =N is supplied I think that would be acceptable to. 0 -> 1 in this case. Or possibly sys.oid: 0 -> 1 # since sysctl.conf(5) also takes comments like that. Not really thats something at the top of the list for fixes though. Low fruit. Food for thought. From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 21:01:57 2011 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6ECC41065672; Wed, 30 Nov 2011 21:01:57 +0000 (UTC) (envelope-from jlaffaye.freebsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9D9AF8FC17; Wed, 30 Nov 2011 21:01:56 +0000 (UTC) Received: by eekc13 with SMTP id c13so916259eek.13 for ; Wed, 30 Nov 2011 13:01:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :content-type:content-transfer-encoding; bh=iUxmg1gjKbtwEzIuS42SXpKR+dWUJslWKbxCzqb2ay8=; b=WwGQbr+7sDiWl77lwC0KhH2/yTvDrm7aBFABxWFVcUNVs2YKCziDSYZDqjZjgyqaNK jZwOI/xAjyeAIA/OIpjfaF++/RXmQhpfLs0eXVsOHxzXLhE+Yb9npSh7DhgXdqevqWor 171q8ZtoWVEivCtAz85MaECjwwxkzt097CQNM= Received: by 10.216.55.137 with SMTP id k9mr69296wec.32.1322685137940; Wed, 30 Nov 2011 12:32:17 -0800 (PST) Received: from chulak.jlaffaye.net (lantea.jlaffaye.net. [109.190.125.169]) by mx.google.com with ESMTPS id em4sm3297525wbb.20.2011.11.30.12.32.16 (version=SSLv3 cipher=OTHER); Wed, 30 Nov 2011 12:32:17 -0800 (PST) Sender: Julien Laffaye Message-ID: <4ED692D2.5060407@FreeBSD.org> Date: Wed, 30 Nov 2011 21:32:18 +0100 From: Julien Laffaye User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.2) Gecko/20110922 Thunderbird/6.0.2 MIME-Version: 1.0 To: ports@freebsd.org, current@freebsd.org, hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Baptiste Daroussin Subject: [CFT] pkgng alpha2 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 21:01:57 -0000 Hi all, We are releasing pkgng (the next pkg_install) alpha2 to the world and we want you to test it! There is no "good" method to test it: use it as you would in the real world. Of course, you are encouraged to backup your data or test it in some kind of virtualized environment. After using it for some time, you will certainly find bugs. You can report them on the issues tracker [1]. If you find missing features, that is things you can't do with pkgng but can with pkg_install, you can also report them. New features are not the expected outcome of this call, as we want to release a final version ASAP. FYI, an alpha3 should follow shortly to fix issues in alpha3 and test additional features. After that, there will be a feature freeze with beta1. Getting started: You can download or git clone the source code of pkgng on the github page [2]. Then, a boring `make' followed by `make install' will do it. If you have some packages installed by pkg_add, you can convert the old database to the pkgng database with the 'pkg2ng' shell script in the ports/ folder. You can also add packages from the ports tree (with bsd.pkgng.mk) or with a pkgng repository. All is documented in the README and manpages. If you are a newcomer to pkgng, this doc reading step is also valuable to us. Indeed, if you fight to get the right infos, or if some things feel counter intuitive, we should improve it! Which brings me to the topic of contributing to pkgng. The best thing you can do is to write down the documentation you would have loved to read while testing pkgng! And of course, if you have a patch with your bug report, it is much appreciated. If you read this entire mail and wonder what is this pkgng thing, you can read the wiki page [3], bapt's presentation from BSDCan [4], EuroBSDCon [5] [6] and browse the source code. Regards, Julien, on behalf of the pkgng team. And remember, we _do_ want to hear back from you! Please also note that it is still alpha code and it can kill kitten and puppies. You are warned ;-) [1] : https://github.com/pkgng/pkgng/issues [2] : https://github.com/pkgng/pkgng [3] : http://wiki.freebsd.org/pkgng [4] : http://people.freebsd.org/~bapt/pkgng-bsdcan2011.pdf [5] : http://wiki.freebsd.org/201110DevSummit/Ports?action=AttachFile&do=get&target=pkgng-devsummit.pdf [6] : http://wiki.freebsd.org/201110DevSummit?action=AttachFile&do=get&target=pkgng-devsummit-track.pdf From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 21:55:55 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3644106566C for ; Wed, 30 Nov 2011 21:55:55 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 75FFD8FC19 for ; Wed, 30 Nov 2011 21:55:55 +0000 (UTC) Received: by bkat2 with SMTP id t2so1333599bka.13 for ; Wed, 30 Nov 2011 13:55:54 -0800 (PST) Received: by 10.205.127.77 with SMTP id gz13mr4320751bkc.76.1322688521723; Wed, 30 Nov 2011 13:28:41 -0800 (PST) Received: from dfleuriot.local ([2a01:e35:2f1b:d6f0:e6ce:8fff:fe15:741a]) by mx.google.com with ESMTPS id hw14sm6651857bkc.16.2011.11.30.13.28.37 (version=SSLv3 cipher=OTHER); Wed, 30 Nov 2011 13:28:38 -0800 (PST) Message-ID: <4ED6A005.60300@my.gd> Date: Wed, 30 Nov 2011 22:28:37 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <20111125073630.GC7915@DataIX.net> <201111301152.47002.jhb@freebsd.org> <20111130184815.GA70174@DataIX.net> In-Reply-To: <20111130184815.GA70174@DataIX.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 21:55:56 -0000 On 11/30/11 7:48 PM, Jason Hellenthal wrote: > > > On Wed, Nov 30, 2011 at 11:52:46AM -0500, John Baldwin wrote: >> On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: >>> >>> Found a troubling result of the following and figured someone might want to >> take a look. >>> >>> Pay close attention to the output and behavior. >>> >>> sysctl net.inet.udp.blackhole=0 >>> sysctl net.inet.udp.blackhole >>> sysctl -d net.inet.udp.blackhole=1 >>> sysctl net.inet.udp.blackhole >>> >>> >>> Is this expected ? should it not just display the description instead of >> adjusting ? as well not display the description like it is adjusting the >> description too ? >> >> Hah, cute. It should probably fail with an error if you do something like >> that, yes. >> > > Yeah thats what I thought about it to but the more I thought about it, if it just displayed the values changing instead of the description when =N is supplied I think that would be acceptable to. 0 -> 1 in this case. Or possibly sys.oid: 0 -> 1 # since sysctl.conf(5) also takes comments like that. > > Not really thats something at the top of the list for fixes though. Low fruit. Food for thought. > Perhaps you would kindly provide a patch for this ? I might have a look at it if nobody does, sounds trivial enough that I might be able to handle it ;) From owner-freebsd-hackers@FreeBSD.ORG Wed Nov 30 22:07:26 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01F82106566C for ; Wed, 30 Nov 2011 22:07:26 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8D3518FC13 for ; Wed, 30 Nov 2011 22:07:25 +0000 (UTC) Received: by faak28 with SMTP id k28so1435248faa.13 for ; Wed, 30 Nov 2011 14:07:24 -0800 (PST) Received: by 10.180.81.73 with SMTP id y9mr3004145wix.37.1322690844484; Wed, 30 Nov 2011 14:07:24 -0800 (PST) Received: from dfleuriot.local (angel.c-mal.com. [82.241.189.111]) by mx.google.com with ESMTPS id hq5sm981454wib.7.2011.11.30.14.07.20 (version=SSLv3 cipher=OTHER); Wed, 30 Nov 2011 14:07:20 -0800 (PST) Message-ID: <4ED6A916.9030409@my.gd> Date: Wed, 30 Nov 2011 23:07:18 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <20111125073630.GC7915@DataIX.net> <201111301152.47002.jhb@freebsd.org> <20111130184815.GA70174@DataIX.net> In-Reply-To: <20111130184815.GA70174@DataIX.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2011 22:07:26 -0000 On 11/30/11 7:48 PM, Jason Hellenthal wrote: > > > On Wed, Nov 30, 2011 at 11:52:46AM -0500, John Baldwin wrote: >> On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: >>> >>> Found a troubling result of the following and figured someone might want to >> take a look. >>> >>> Pay close attention to the output and behavior. >>> >>> sysctl net.inet.udp.blackhole=0 >>> sysctl net.inet.udp.blackhole >>> sysctl -d net.inet.udp.blackhole=1 >>> sysctl net.inet.udp.blackhole >>> >>> >>> Is this expected ? should it not just display the description instead of >> adjusting ? as well not display the description like it is adjusting the >> description too ? >> >> Hah, cute. It should probably fail with an error if you do something like >> that, yes. >> > > Yeah thats what I thought about it to but the more I thought about it, if it just displayed the values changing instead of the description when =N is supplied I think that would be acceptable to. 0 -> 1 in this case. Or possibly sys.oid: 0 -> 1 # since sysctl.conf(5) also takes comments like that. > > Not really thats something at the top of the list for fixes though. Low fruit. Food for thought. > Ok so I'm totally not a dev, don't take my findings for granted, but: (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /usr/src/sbin/sysctl/sysctl -d net.inet.udp.blackhole=1 Breakpoint 5, show_var (oid=0x7fffffffe1d0, nlen=4) at /usr/src/sbin/sysctl/sysctl.c:549 549 { (gdb) s 562 bzero(buf, BUFSIZ); (gdb) s 563 bzero(name, BUFSIZ); (gdb) s 564 qoid[0] = 0; (gdb) s 565 memcpy(qoid + 2, oid, nlen * sizeof(int)); (gdb) s 567 qoid[1] = 1; (gdb) s 568 j = sizeof(name); (gdb) s 569 i = sysctl(qoid, nlen + 2, name, &j, 0, 0); (gdb) s 570 if (i || !j) (gdb) s 573 if (Nflag) { (gdb) s 578 if (eflag) (gdb) s 581 sep = ": "; (gdb) s 583 if (dflag) { /* just print description */ (gdb) s 584 qoid[1] = 5; (gdb) s 585 j = sizeof(buf); (gdb) s 586 i = sysctl(qoid, nlen + 2, buf, &j, 0, 0); (gdb) s 587 if (!nflag) (gdb) s 588 printf("%s%s", name, sep); (gdb) s net.inet.udp.blackhole: 589 printf("%s", buf); (gdb) s Do not send port unreachables for refused connects 590 return (0); (gdb) s 719 } At this point, we have shown the description and should exit. Instead, we're calling parse() again ? (gdb) s parse (string=0x7fffffffee07 "net.inet.udp.blackhole") at /usr/src/sbin/sysctl/sysctl.c:299 299 if (sysctl(mib, len, 0, 0, newval, newsize) == -1) { (gdb) s 318 if (!bflag) (gdb) s 319 printf(" -> "); (gdb) s -> 320 i = nflag; (gdb) s 321 nflag = 1; (gdb) s 322 j = show_var(mib, len); (gdb) s Breakpoint 5, show_var (oid=0x7fffffffe1d0, nlen=4) at /usr/src/sbin/sysctl/sysctl.c:549 549 { (gdb) s 562 bzero(buf, BUFSIZ); (gdb) s 563 bzero(name, BUFSIZ); (gdb) s 564 qoid[0] = 0; (gdb) s 565 memcpy(qoid + 2, oid, nlen * sizeof(int)); (gdb) s 567 qoid[1] = 1; (gdb) s 568 j = sizeof(name); (gdb) s 569 i = sysctl(qoid, nlen + 2, name, &j, 0, 0); (gdb) s 570 if (i || !j) (gdb) s 573 if (Nflag) { (gdb) s 578 if (eflag) (gdb) s 581 sep = ": "; (gdb) s 583 if (dflag) { /* just print description */ (gdb) s 584 qoid[1] = 5; (gdb) s 585 j = sizeof(buf); (gdb) s 586 i = sysctl(qoid, nlen + 2, buf, &j, 0, 0); (gdb) s 587 if (!nflag) (gdb) s 589 printf("%s", buf); (gdb) s Do not send port unreachables for refused connects 590 return (0); (gdb) s 719 } (gdb) s parse (string=0x7fffffffee07 "net.inet.udp.blackhole") at /usr/src/sbin/sysctl/sysctl.c:323 323 if (!j && !bflag) (gdb) s 324 putchar('\n'); (gdb) s __sputc (_c=10, _p=0x80085a810) at stdio.h:457 457 if (--_p->_w >= 0 || (_p->_w >= _p->_lbfsize && (char)_c != '\n')) (gdb) s 460 return (__swbuf(_c, _p)); (gdb) s 461 } (gdb) s parse (string=0x7fffffffee07 "net.inet.udp.blackhole") at /usr/src/sbin/sysctl/sysctl.c:325 325 nflag = i; (gdb) s 327 } (gdb) s main (argc=0, argv=0x7fffffffeb48) at /usr/src/sbin/sysctl/sysctl.c:154 154 while (argc-- > 0) (gdb) s 156 exit(warncount); (gdb) s Program exited normally. I think that the issue here is that sysctl continues being run after displaying the description at line 588. I would assume that upon reaching the closing hyphen at line 719 , sysctl would want to stop. It was asked to show a description, it has, end of the story. Instead, at line 590, we simply return 0, then continue running. Perhaps we would want sysctl to exit, instead of returning 0 ? From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 06:57:50 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C47ED106564A for ; Thu, 1 Dec 2011 06:57:50 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 477428FC13 for ; Thu, 1 Dec 2011 06:57:50 +0000 (UTC) Received: by bkat2 with SMTP id t2so2021281bka.13 for ; Wed, 30 Nov 2011 22:57:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=qsKAs2jegDX7JLuSq643YVsHy4A/o1bwjHjehzxcMvU=; b=kd7ZGK8lvvNXwfJ787cztndTmdeT2bCZxBQIoC9LWb601QQW8FS83DP/OhBQj4iEx3 wJylwHCK3j7E42FPWn6T1yvTiQZo1HbSlEUWzVDhRmW13BOQ4T4KPhk0Os/WLpAIaL+e cevWDqfzXD9tvz8efomchEXfE8nZW8chyDytU= Received: by 10.204.7.81 with SMTP id c17mr3213417bkc.63.1322722669131; Wed, 30 Nov 2011 22:57:49 -0800 (PST) Received: from DataIX.net (ppp-21.177.dialinfree.com. [209.172.21.177]) by mx.google.com with ESMTPS id x14sm9283185bkf.10.2011.11.30.22.57.42 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 30 Nov 2011 22:57:48 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pB16vbhJ005726 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 1 Dec 2011 01:57:37 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pB16vMoO005690; Thu, 1 Dec 2011 01:57:22 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Thu, 1 Dec 2011 01:57:22 -0500 From: Jason Hellenthal To: Steven Hartland Message-ID: <20111201065722.GA97051@DataIX.net> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> Cc: freebsd-hackers@freebsd.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 06:57:50 -0000 On Wed, Nov 30, 2011 at 12:39:10PM -0000, Steven Hartland wrote: > We're seeing some impossible memory usage stats reported on machines > here from vmstat and sysctl vm.vmtotal. > > We have machines reporting to be using 31GB total when they only have > 8GB physical and are not using any swap. > > Here's an output from one of our machines:- > vmstat -c 2 -w 1 -n 0 > procs memory page faults cpu > r b w avm fre flt re pi po fr sr in sy cs us sy id > 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 > 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 > > The raw output is:- > vmstat -c 2 -w 1 -n 0 -H > procs memory page faults cpu > r b w avm fre flt re pi po fr sr in sy cs us sy id > 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 > 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 > > Top shows:- > last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 > 1893 processes:1 running, 1892 sleeping > CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle > Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free > Swap: 4096M Total, 4096M Free > > sysctl vm.vmtotal > vm.vmtotal: > System wide totals computed every five seconds: (values in kilobytes) > =============================================== > Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) > Virtual Memory: (Total: 1106403532K Active: 32540260K) > Real Memory: (Total: 4563648K Active: 3921644K) > Shared Virtual Memory: (Total: 19976K Active: 16396K) > Shared Real Memory: (Total: 9040K Active: 8436K) > Free Memory Pages: 2161740K > > As mentioned this machine has 8GB of ram and according to both top and > swapinfo is using no swap at all > > >From dmesg:- > real memory = 8589934592 (8192 MB) > avail memory = 8255553536 (7873 MB) > > swapinfo > Device 1K-blocks Used Avail Capacity > /dev/gptid/09f211f7-39ce-11e0-8 4194304 0 4194304 0% > > uname -a > FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 > > sysctl hw.pagesize > hw.pagesize: 4096 > > It looks like it may be out by a factor of 4, possibly due to the fact > the its a 4k page size not 1k as indicated by the vmstat man page:- > > memory Information about the usage of virtual and real memory. Virtual > pages (reported in units of 1024 bytes) are considered active if > they belong to processes which are running or have run in the > last 20 seconds. > > avm active virtual pages > fre size of the free list > > > Totalling up RSS from ps axo "rss" gives a total in the region of that if > the vm stats are out by a factor of 4, in this case it should be: 8132557 > which is 7.75GB a much more realistic value. > > Am I totally missing something or is there problem here? > > Regards > Steve > This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 09:01:24 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE725106564A for ; Thu, 1 Dec 2011 09:01:24 +0000 (UTC) (envelope-from lichray@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4AE738FC1D for ; Thu, 1 Dec 2011 09:01:23 +0000 (UTC) Received: by eaai12 with SMTP id i12so2619096eaa.13 for ; Thu, 01 Dec 2011 01:01:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=3kFzZRw3bjVwyyd28njhEW4C3NXCdtg7lI8cV7XSyQU=; b=sIMqafScIQ26bXVWckWF6e/VjumIjsMWRt0NxI3laLn/0/ERqTvHJPPo8eEDdpkvHx TFzPtMsUyVrt4lu41yf2ooIsG9Hh33ks1G6mihaX/8PKSML1cMyLhudjiXe3p21ceJf8 3bAecfGpSzkC4UPVxwkjm1Zhr0B2bSK+TeG04= MIME-Version: 1.0 Received: by 10.213.13.68 with SMTP id b4mr378601eba.49.1322730083190; Thu, 01 Dec 2011 01:01:23 -0800 (PST) Received: by 10.14.47.194 with HTTP; Thu, 1 Dec 2011 01:01:23 -0800 (PST) Date: Thu, 1 Dec 2011 03:01:23 -0600 Message-ID: From: Zhihao Yuan To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: To implement RFC 5848 (Signed Syslog Messages)? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 09:01:24 -0000 Hi, hackers: Red Hat's "star" developer, Lennart Poettering, is porting Windows Event Log to GNU/Linux :) https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs&pli=1 Regardless of his stupid arguments, let's talk about something trivial. How about to implement RFC 5848 in our syslogd? It adds the encryption to the existing syslog message layer, and increase the security in transferring. http://tools.ietf.org/html/rfc5848 Albert Mietus made a nice presentation in 2002 http://www.slideshare.net/SoftwareBeterMaken.nl/securing-syslog-on-freebsd Not sure whether his code is accessible or not. -- Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___________________________________________________ 4BSD -- http://4bsd.biz/ From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 09:46:46 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B3CB106566C for ; Thu, 1 Dec 2011 09:46:46 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 565218FC18 for ; Thu, 1 Dec 2011 09:46:46 +0000 (UTC) Received: by dafa1 with SMTP id a1so233727daf.13 for ; Thu, 01 Dec 2011 01:46:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Gq8xh23f48lPdwAnxNLPyM4QZ3FOZO2B4I6TDvLQFIE=; b=vWQz3xxPTqiHktSDChMTXRuN0jkdVk9cgwT/yYJVekjhkqe3Rfp/wTW/42Yn+nJzGV bUke/1OFhvVS3eSoVQ5fxtL6pF+6qdQsUMRU1ySmAEotZlrRZivXYP2yheZBy7dQwTXT WVXq24X3Ooq8nuTG4xlKtLwtTO6PQfIMaA2VY= MIME-Version: 1.0 Received: by 10.68.39.37 with SMTP id m5mr3516460pbk.91.1322732805770; Thu, 01 Dec 2011 01:46:45 -0800 (PST) Received: by 10.68.42.132 with HTTP; Thu, 1 Dec 2011 01:46:45 -0800 (PST) In-Reply-To: References: Date: Thu, 1 Dec 2011 01:46:45 -0800 Message-ID: From: Garrett Cooper To: Zhihao Yuan Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org Subject: Re: To implement RFC 5848 (Signed Syslog Messages)? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 09:46:46 -0000 On Thu, Dec 1, 2011 at 1:01 AM, Zhihao Yuan wrote: > Hi, hackers: > > Red Hat's "star" developer, Lennart Poettering, is porting Windows > Event Log to GNU/Linux :) > https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs&pli=1 > > Regardless of his stupid arguments, let's talk about something > trivial. How about to implement RFC 5848 in our syslogd? It adds the > encryption to the existing syslog message layer, and increase the > security in transferring. > http://tools.ietf.org/html/rfc5848 > > Albert Mietus made a nice presentation in 2002 > http://www.slideshare.net/SoftwareBeterMaken.nl/securing-syslog-on-freebsd > > Not sure whether his code is accessible or not. I agree that encryption and tcp (reliable) transport of logs should be a must for syslogd in FreeBSD. It's going to be interesting how things with Lennart's 'journald' play out -- without defining an industry standard for how messages are presented and categorized, I predict that things will turn into a mess (I could be proved wrong, but given past experience, this is how things evolve unless framework adoption lags standardization). Thanks :), -Garrett From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 10:56:00 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10209106564A for ; Thu, 1 Dec 2011 10:56:00 +0000 (UTC) (envelope-from prvs=1316828ddc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 8F2408FC13 for ; Thu, 1 Dec 2011 10:55:59 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 01 Dec 2011 10:45:56 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 01 Dec 2011 10:45:56 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016822410.msg for ; Thu, 01 Dec 2011 10:45:55 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1316828ddc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-hackers@freebsd.org Message-ID: <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> From: "Steven Hartland" To: "Jason Hellenthal" References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> Date: Thu, 1 Dec 2011 10:44:58 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-hackers@freebsd.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 10:56:00 -0000 ----- Original Message ----- From: "Jason Hellenthal" > This goes along with the thoughts I had about 4 months ago tending to some > zfs statistics as well top showing greater than 100% actual CPU usage. This > is a big pet peave of mine. Its like saying you ate 134% of a bannanna when > in all reallity it is impossible. You can never have more than 100% usage of > anything and when seen is a clear notice that some math is considerably > incorrect leading to other such miscalculations to be performed. Things like > the above already have checks in place that ensure no boundries are being > crossed/overflowed or underrun but it surely makes processing results building > future products a bitch. One instance is the calculation of threads for example > firefox can be seen using upto or more 338% of the CPU. Thats impossible its > like saying anyones CPU grew by 400%. I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 12:43:02 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC4471065670 for ; Thu, 1 Dec 2011 12:43:02 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 624628FC08 for ; Thu, 1 Dec 2011 12:43:01 +0000 (UTC) Received: by lahv2 with SMTP id v2so980415lah.13 for ; Thu, 01 Dec 2011 04:43:00 -0800 (PST) Received: by 10.152.135.179 with SMTP id pt19mr4373126lab.47.1322743380548; Thu, 01 Dec 2011 04:43:00 -0800 (PST) Received: from dfleuriot.local (angel.c-mal.com. [82.241.189.111]) by mx.google.com with ESMTPS id mo6sm4999129lab.11.2011.12.01.04.42.57 (version=SSLv3 cipher=OTHER); Thu, 01 Dec 2011 04:42:58 -0800 (PST) Message-ID: <4ED77650.6050409@my.gd> Date: Thu, 01 Dec 2011 13:42:56 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> In-Reply-To: <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 12:43:02 -0000 On 12/1/11 11:44 AM, Steven Hartland wrote: > ----- Original Message ----- From: "Jason Hellenthal" > >> This goes along with the thoughts I had about 4 months ago tending to >> some >> zfs statistics as well top showing greater than 100% actual CPU usage. >> This >> is a big pet peave of mine. Its like saying you ate 134% of a bannanna >> when >> in all reallity it is impossible. You can never have more than 100% >> usage of >> anything and when seen is a clear notice that some math is considerably >> incorrect leading to other such miscalculations to be performed. >> Things like >> the above already have checks in place that ensure no boundries are being >> crossed/overflowed or underrun but it surely makes processing results >> building >> future products a bitch. One instance is the calculation of threads >> for example >> firefox can be seen using upto or more 338% of the CPU. Thats >> impossible its >> like saying anyones CPU grew by 400%. > > I could understand a bit of overflow as stats are snapshots which may not > be instuntanious, but 31GB instead of under 8GB is hardly a rounding > issue / > overflow. > > With respect to top showing greater than 100% by how much are you talking? > Do your realise that each core = 100%? So if you have a quad core your > system > total will be 400% not 100%? > That's his point, you cannot use 400% of a system as a whole, his point is that top should report 100% where each core accounts for 25% From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 13:23:35 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 870641065675 for ; Thu, 1 Dec 2011 13:23:35 +0000 (UTC) (envelope-from prvs=1316828ddc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 0E0D78FC17 for ; Thu, 1 Dec 2011 13:23:34 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 01 Dec 2011 13:23:32 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 01 Dec 2011 13:23:32 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016825590.msg for ; Thu, 01 Dec 2011 13:23:30 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1316828ddc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-hackers@freebsd.org Message-ID: From: "Steven Hartland" To: "Damien Fleuriot" , References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk><20111201065722.GA97051@DataIX.net><1A338C6C470940B386C307641E350A3E@multiplay.co.uk> <4ED77650.6050409@my.gd> Date: Thu, 1 Dec 2011 13:23:35 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 13:23:35 -0000 ----- Original Message ----- From: "Damien Fleuriot" >> I could understand a bit of overflow as stats are snapshots which may not >> be instuntanious, but 31GB instead of under 8GB is hardly a rounding >> issue / >> overflow. >> >> With respect to top showing greater than 100% by how much are you talking? >> Do your realise that each core = 100%? So if you have a quad core your >> system >> total will be 400% not 100%? >> > > That's his point, you cannot use 400% of a system as a whole, his point > is that top should report 100% where each core accounts for 25% Then I would have to disagree, keeping 100% to mean 100% of a single core is much easier to manage than 100% of a machines total capacity. If you went to 100% = the machine total capacity processes could be using a lot of cpu without even registering 1% on today's machines where 24 cores is common place. It also makes detecting single process / thread bottlenecks easier as if your seeing 100% you know its maxing a core, instead of having to calculate it once you know how many cores the machine has. If your looking for total machine usage then that's also their in the summary at the top of the screen e.g. CPU states: 13.6% user, 0.0% nice, 1.3% system, 0.0% interrupt, 85.1% idle Anyway this is quite off topic, and I don't want to loose sight of the threads goal which is to determine why we can see 31GB of usage on an 8GB machine with very little shared memory usage an no swap usage. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 14:24:12 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CA2F106564A for ; Thu, 1 Dec 2011 14:24:12 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 23AEB8FC15 for ; Thu, 1 Dec 2011 14:24:11 +0000 (UTC) Received: by eekc13 with SMTP id c13so1747079eek.13 for ; Thu, 01 Dec 2011 06:24:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding; bh=c0W6w3w8RxSm8+5FQ2/JtzLz16awyQEUX5jGQVqgdZQ=; b=mehfKb0hrMxF+mGzFZsGgrvhspeq0VyBqeitzevuRFBYfG1428B56CPWn6E5tgp3cz 9djeXOUBE2ohf/ySOwcrW7g6onXwHm+WKi+RF39i85IUr2iC6aOCYKAeghXgrnW9bbiG oK6WPUnCyHu6fTbsV/gyBegr0BkXi+sacMgPk= Received: by 10.14.7.211 with SMTP id 59mr933367eep.146.1322747902835; Thu, 01 Dec 2011 05:58:22 -0800 (PST) Received: from gumby.homeunix.com (87-194-105-247.bethere.co.uk. [87.194.105.247]) by mx.google.com with ESMTPS id 17sm6619485eej.3.2011.12.01.05.58.20 (version=SSLv3 cipher=OTHER); Thu, 01 Dec 2011 05:58:21 -0800 (PST) Date: Thu, 1 Dec 2011 13:58:18 +0000 From: RW To: freebsd-hackers@freebsd.org Message-ID: <20111201135818.641625c5@gumby.homeunix.com> In-Reply-To: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd8.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 14:24:12 -0000 On Wed, 30 Nov 2011 12:39:10 -0000 Steven Hartland wrote: > We're seeing some impossible memory usage stats reported on machines > here from vmstat and sysctl vm.vmtotal. > > We have machines reporting to be using 31GB total when they only have > 8GB physical and are not using any swap. > > Here's an output from one of our machines:- > vmstat -c 2 -w 1 -n 0 > procs memory page faults cpu > r b w avm fre flt re pi po fr sr in sy cs us > sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 > 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 > 8139 3996 0 1 99 > > ... > It looks like it may be out by a factor of 4, possibly due to the fact > the its a 4k page size not 1k as indicated by the vmstat man page:- > I don't think so, I have 16GB and tops shows: Mem: 817M Active, 396M Inact, 1364M Wired, 82M Cache, 1282M Buf, 13G Free but vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avm fre flt re pi po fr sr in sy cs us sy id 0 0 0 9750M 13G 450 5 3 1 560 0 234 50201 5206 2 2 96 1 0 0 9742M 13G 79 4 0 0 573 0 239 51886 4700 0 1 99 From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 15:06:28 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E7A61065677 for ; Thu, 1 Dec 2011 15:06:28 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 165338FC12 for ; Thu, 1 Dec 2011 15:06:28 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id BF4FB46B0A; Thu, 1 Dec 2011 10:06:27 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 51268B914; Thu, 1 Dec 2011 10:06:27 -0500 (EST) From: John Baldwin To: Jason Hellenthal Date: Thu, 1 Dec 2011 10:06:26 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <20111125073630.GC7915@DataIX.net> <201111301152.47002.jhb@freebsd.org> <20111130184815.GA70174@DataIX.net> In-Reply-To: <20111130184815.GA70174@DataIX.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112011006.26833.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 01 Dec 2011 10:06:27 -0500 (EST) Cc: freebsd-hackers@freebsd.org Subject: Re: sysctl description spillover and also setting the sysctl ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 15:06:28 -0000 On Wednesday, November 30, 2011 1:48:15 pm Jason Hellenthal wrote: > > On Wed, Nov 30, 2011 at 11:52:46AM -0500, John Baldwin wrote: > > On Friday, November 25, 2011 2:36:30 am Jason Hellenthal wrote: > > > > > > Found a troubling result of the following and figured someone might want to > > take a look. > > > > > > Pay close attention to the output and behavior. > > > > > > sysctl net.inet.udp.blackhole=0 > > > sysctl net.inet.udp.blackhole > > > sysctl -d net.inet.udp.blackhole=1 > > > sysctl net.inet.udp.blackhole > > > > > > > > > Is this expected ? should it not just display the description instead of > > adjusting ? as well not display the description like it is adjusting the > > description too ? > > > > Hah, cute. It should probably fail with an error if you do something like > > that, yes. > > > > Yeah thats what I thought about it to but the more I thought about it, if > it just displayed the values changing instead of the description when =N > is supplied I think that would be acceptable to. 0 -> 1 in this case. Or > possibly sys.oid: 0 -> 1 # since sysctl.conf(5) also takes > comments like that. > > Not really thats something at the top of the list for fixes though. Low > fruit. Food for thought. I think it's simplest to just make -d force descriptions only and ignore settings. I've committed a one-line fix to sysctl(8) for that. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 15:16:18 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0EF9106566B for ; Thu, 1 Dec 2011 15:16:18 +0000 (UTC) (envelope-from prvs=1316828ddc=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 4BCE38FC17 for ; Thu, 1 Dec 2011 15:16:17 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 01 Dec 2011 15:16:15 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 01 Dec 2011 15:16:15 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016826493.msg for ; Thu, 01 Dec 2011 15:16:14 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1316828ddc=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-hackers@freebsd.org Message-ID: <8F2B03B67CBD4E83BB84761D8EF331B6@multiplay.co.uk> From: "Steven Hartland" To: "RW" , References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201135818.641625c5@gumby.homeunix.com> Date: Thu, 1 Dec 2011 15:16:18 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 15:16:19 -0000 ----- Original Message ----- From: "RW" To: Sent: Thursday, December 01, 2011 1:58 PM Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? > On Wed, 30 Nov 2011 12:39:10 -0000 > Steven Hartland wrote: > >> We're seeing some impossible memory usage stats reported on machines >> here from vmstat and sysctl vm.vmtotal. >> >> We have machines reporting to be using 31GB total when they only have >> 8GB physical and are not using any swap. >> >> Here's an output from one of our machines:- >> vmstat -c 2 -w 1 -n 0 >> procs memory page faults cpu >> r b w avm fre flt re pi po fr sr in sy cs us >> sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 >> 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 >> 8139 3996 0 1 99 >> >> ... >> It looks like it may be out by a factor of 4, possibly due to the fact >> the its a 4k page size not 1k as indicated by the vmstat man page:- >> > > I don't think so, I have 16GB and tops shows: > > Mem: 817M Active, 396M Inact, 1364M Wired, 82M Cache, 1282M Buf, 13G > Free > > but > > vmstat -c 2 -w 1 -n 0 > procs memory page faults cpu > r b w avm fre flt re pi po fr sr in sy cs us sy > id 0 0 0 9750M 13G 450 5 3 1 560 0 234 50201 5206 2 > 2 96 1 0 0 9742M 13G 79 4 0 0 573 0 239 51886 4700 > 0 1 99 Hmm, yes on another machine same OS with 16GB memory we see:- dmesg | grep memory real memory = 17179869184 (16384 MB) avail memory = 16536604672 (15770 MB) vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avm fre flt re pi po fr sr in sy cs us sy id 0 0 0 1948M 666M 395 14 0 0 398 1966 3237 5980 14085 2 4 93 1 0 0 1948M 636M 4 0 0 0 16 0 8215 18660 35322 2 7 92 sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 194) Virtual Memory: (Total: 1075975332K Active: 1979056K) Real Memory: (Total: 309048K Active: 160804K) Shared Virtual Memory: (Total: 75112K Active: 18860K) Shared Real Memory: (Total: 15656K Active: 10572K) Free Memory Pages: 731960K but with top:- last pid: 38187; load averages: 0.15, 0.16, 0.16 up 2+23:33:55 15:13:02 195 processes: 1 running, 194 sleeping CPU: 0.0% user, 0.0% nice, 4.1% system, 0.0% interrupt, 95.9% idle Mem: 886M Active, 12G Inact, 2194M Wired, 599M Cache, 1630M Buf, 138M Free Swap: 8192M Total, 1512K Used, 8190M Free So I've no idea whats going on? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 18:05:38 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 615B51065675 for ; Thu, 1 Dec 2011 18:05:38 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id E5EF08FC14 for ; Thu, 1 Dec 2011 18:05:37 +0000 (UTC) Received: by eekc13 with SMTP id c13so1974066eek.13 for ; Thu, 01 Dec 2011 10:05:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=G+AVjakzmviBjzPceSyMY0pEQ7S+OKbXBS/CIsZdy1E=; b=jexqxHIOYGzAVh3IKaRm13gcBKPBnFJ8sasIn2tgaMA18FKNd9+RRGImnlQ8c9bAjE sHckpCWNdXV/ObKWj4Rii6qHNtgkSeY7/yHvqITfiIaUw7sf8hlOYGkzqTef3Ih51oEF /f1Mug1U+4DxDP5wdAJyB43lsjFhSM5uRHmeM= Received: by 10.227.198.142 with SMTP id eo14mr3335279wbb.28.1322762736810; Thu, 01 Dec 2011 10:05:36 -0800 (PST) Received: from DataIX.net (ppp-21.214.dialinfree.com. [209.172.21.214]) by mx.google.com with ESMTPS id fk3sm6644839wbb.10.2011.12.01.10.05.32 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 01 Dec 2011 10:05:35 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pB1I5RZs068228 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 1 Dec 2011 13:05:27 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pB1I5MLQ068227; Thu, 1 Dec 2011 13:05:22 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Thu, 1 Dec 2011 13:05:22 -0500 From: Jason Hellenthal To: Steven Hartland Message-ID: <20111201180522.GA67513@DataIX.net> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> Cc: freebsd-hackers@freebsd.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 18:05:38 -0000 On Thu, Dec 01, 2011 at 10:44:58AM -0000, Steven Hartland wrote: > ----- Original Message ----- > From: "Jason Hellenthal" > > > This goes along with the thoughts I had about 4 months ago tending to some > > zfs statistics as well top showing greater than 100% actual CPU usage. This > > is a big pet peave of mine. Its like saying you ate 134% of a bannanna when > > in all reallity it is impossible. You can never have more than 100% usage of > > anything and when seen is a clear notice that some math is considerably > > incorrect leading to other such miscalculations to be performed. Things like > > the above already have checks in place that ensure no boundries are being > > crossed/overflowed or underrun but it surely makes processing results building > > future products a bitch. One instance is the calculation of threads for example > > firefox can be seen using upto or more 338% of the CPU. Thats impossible its > > like saying anyones CPU grew by 400%. > > I could understand a bit of overflow as stats are snapshots which may not > be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / > overflow. I agree > > With respect to top showing greater than 100% by how much are you talking? > Do your realise that each core = 100%? So if you have a quad core your system > total will be 400% not 100%? > Yeah I realize that but it still would lead you to believe that if a proccessor has 4 cores on the same die then total for each core could only be 25% usage. And the usage for a proccess only consuming full usage of 1 core is 100%. But you can start firefox on a single uniproccessor and like stated above see large usage percents near 338% or greater which is impossible and leads me to believe were forcing calculation for the entire proccess of threads onto tthread 0. This makes accounting pretty difficult. From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 1 21:58:00 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AFD6106566C; Thu, 1 Dec 2011 21:58:00 +0000 (UTC) (envelope-from alancyang@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id B494E8FC13; Thu, 1 Dec 2011 21:57:59 +0000 (UTC) Received: by eaai12 with SMTP id i12so3829313eaa.13 for ; Thu, 01 Dec 2011 13:57:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=1i0czBtwW0+ccsfehjCsH8PGvKEOzRZhb1tkCrTi2rM=; b=Lg4oKq66TnX19sVHRdAGKJE2IOJiEYYxq1/P40ALUUzYN43x58+DpDhnoylYAyQ2// h3EdAgM1pEBD9n4yKRGU8IptDAXAtvr0xQ/gUDj3EUhf8hgxZBEG02He+pX2x2aA0WVp htc8e7i5uF7eNfwvPzbKsekYrkgCjljDKAuYE= MIME-Version: 1.0 Received: by 10.180.105.232 with SMTP id gp8mr597338wib.65.1322775342594; Thu, 01 Dec 2011 13:35:42 -0800 (PST) Received: by 10.216.168.195 with HTTP; Thu, 1 Dec 2011 13:35:42 -0800 (PST) Date: Thu, 1 Dec 2011 13:35:42 -0800 Message-ID: From: alan yang To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org Subject: DiffServ SNMP agent X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2011 21:58:00 -0000 Hello, I wonder people would know if there is diffserv SNMP agent implementation for FreeBSD that is available? Googled a bit, but was not too successful in finding. Thanks in advance! Alan From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 07:01:16 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 990E4106567E for ; Fri, 2 Dec 2011 07:01:16 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 1BA088FC18 for ; Fri, 2 Dec 2011 07:01:15 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so491140wgb.31 for ; Thu, 01 Dec 2011 23:01:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=bHVhO7koc00lU6qYalmHZo3FGUC7/uqU6fmcqPJies0=; b=ZgiZGIjXa1eWO7FBT8w0BgKpAPAoZB5Te2MVhGfLJqLs3bI+kS/G058H/dkwSg24us Ke5btG+Xq/sjpxtLQR5lXA677C4nDzVUtiOc8Qr26nLtgJEhPFjMCxgw82bl8Up5pFhT anHVAnMqjTMoqgzaQR/PncWGSpX91cK0HUx/Y= Received: by 10.227.207.67 with SMTP id fx3mr4620949wbb.0.1322809274759; Thu, 01 Dec 2011 23:01:14 -0800 (PST) Received: from DataIX.net (ppp-21.70.dialinfree.com. [209.172.21.70]) by mx.google.com with ESMTPS id m5sm2397062wie.2.2011.12.01.23.01.09 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 01 Dec 2011 23:01:13 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pB2715Ip000904 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 2 Dec 2011 02:01:05 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pB270t9Z000903; Fri, 2 Dec 2011 02:00:55 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Fri, 2 Dec 2011 02:00:55 -0500 From: Jason Hellenthal To: Steven Hartland Message-ID: <20111202070055.GA98731@DataIX.net> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> <4ED77650.6050409@my.gd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5vNYLRcllDrimb99" Content-Disposition: inline In-Reply-To: Cc: freebsd-hackers@freebsd.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 07:01:16 -0000 --5vNYLRcllDrimb99 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 01, 2011 at 01:23:35PM -0000, Steven Hartland wrote: > ----- Original Message -----=20 > From: "Damien Fleuriot" >=20 > >> I could understand a bit of overflow as stats are snapshots which may = not > >> be instuntanious, but 31GB instead of under 8GB is hardly a rounding > >> issue / > >> overflow. > >>=20 > >> With respect to top showing greater than 100% by how much are you talk= ing? > >> Do your realise that each core =3D 100%? So if you have a quad core yo= ur > >> system > >> total will be 400% not 100%? > >>=20 > >=20 > > That's his point, you cannot use 400% of a system as a whole, his point > > is that top should report 100% where each core accounts for 25% >=20 > Then I would have to disagree, keeping 100% to mean 100% of a single core > is much easier to manage than 100% of a machines total capacity. >=20 > If you went to 100% =3D the machine total capacity processes could be usi= ng > a lot of cpu without even registering 1% on today's machines where 24 cor= es > is common place. >=20 > It also makes detecting single process / thread bottlenecks easier as if > your seeing 100% you know its maxing a core, instead of having to calcula= te > it once you know how many cores the machine has. >=20 > If your looking for total machine usage then that's also their in the sum= mary > at the top of the screen e.g. > CPU states: 13.6% user, 0.0% nice, 1.3% system, 0.0% interrupt, 85.1% = idle >=20 > Anyway this is quite off topic, and I don't want to loose sight of the th= reads > goal which is to determine why we can see 31GB of usage on an 8GB machine > with very little shared memory usage an no swap usage. >=20 Just to put some visuals to this... =2E `-- DIE |-- Core1 [Idle] |-- Core2 [35% ] | `-- thread127 |-- Core3 [40% ] | `-- thread127 `-- Core4 [100%] `-- thread127 In this case you would say the DIE should be at a total of 175% ? $(((25*0)+(25*0.35)+(25*0.40)+(25*1.00))) Out of sanity and each core only being 25% of the total DIE it should be re= porting. It is using all together 43.75% of the total DIE. But thats not wh= at I see even on SP machines. 1 DIE 1 core and a report of 338% usage for 8= threads of firefox. If someone was attempting to write a scheduler to laun= ch processes & yield back to the scheduler to launch more based on processo= r usage either by core or by die totals that scheduler would be ineffective= at best without alot of kludges put in place to handle all the misinterpre= tation. Somewhere along the line our math has been distorted or the move fr= om SP -> MP, cores, hyperthreading etc.. has just not completed yet and sho= uld not be ignored. --5vNYLRcllDrimb99 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJO2HenAAoJEJBXh4mJ2FR+KtMH/jdx+xhAgNTyDOAUce/PtOHI zREdl3XEnhFfi868XLXYbx98/QSMokqzjXC6IqiP8XFtPG5UwB05KERe4U/MnxKN Ord/wg57gHZuBWROmSgnbDgCKmXMom605ZOT78DAZNfFKmnPg4tAqWISdb+ukzFg K6xyNjlQcnyhiIAN0q4wClSVkYVT+Aqk0L4X5+C3NfJBhqZBXqiZ0tr9Ppkw9+HZ BXIHka1FyJvUW9+Gr74/KWYGsAFfqh5fBLDdtAO0PvhJRjp4N16V4IE9nn6RK2H5 05sdzjal5vfx7pKy96A5z8gNsjy6Yjyflrg7eISpAy+rMpicllw5RLak02JP118= =wqDh -----END PGP SIGNATURE----- --5vNYLRcllDrimb99-- From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 07:46:07 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04D53106564A for ; Fri, 2 Dec 2011 07:46:07 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id ACEB28FC17 for ; Fri, 2 Dec 2011 07:46:06 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so3375569vbb.13 for ; Thu, 01 Dec 2011 23:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=QNnK63iMzpVtir+fzbbgfB1Xg6SV/nfys5cdBKI42Uo=; b=vLxpmWmEVTeJZ8TWlPz114iRQj9WV5PLTM8DTZwg1Ry4QUOKdfHVS8KCVltmZxxoyk xX6NXNKqMQojvcl/r1ZDWPeADVyc3QsNvkHE9KNdmYyV4gmPYYij/WScGl16rE+13WCl Mcjr8otSZaTUyqx/HFjscaZvdTcz3D2ZMf2nU= MIME-Version: 1.0 Received: by 10.52.94.227 with SMTP id df3mr9065414vdb.51.1322810393264; Thu, 01 Dec 2011 23:19:53 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.109.10 with HTTP; Thu, 1 Dec 2011 23:19:53 -0800 (PST) In-Reply-To: <20111202070055.GA98731@DataIX.net> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> <4ED77650.6050409@my.gd> <20111202070055.GA98731@DataIX.net> Date: Fri, 2 Dec 2011 15:19:53 +0800 X-Google-Sender-Auth: 1uVWmlHAIbM9b1t5YRIfMWcFAN0 Message-ID: From: Adrian Chadd To: Jason Hellenthal Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, Steven Hartland Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 07:46:07 -0000 .. where are these statistics coming from? top? Adrian From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 09:56:44 2011 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05069106564A; Fri, 2 Dec 2011 09:56:44 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (cl-414.sto-01.se.sixxs.net [IPv6:2001:16d8:ff00:19d::2]) by mx1.freebsd.org (Postfix) with ESMTP id 217CA8FC12; Fri, 2 Dec 2011 09:56:43 +0000 (UTC) Received: from orion.SpringDaemons.com (c-98-234-219-100.hsd1.ca.comcast.net [98.234.219.100]) by mx0.deglitch.com (Postfix) with ESMTPA id AAB6B8FC3A; Fri, 2 Dec 2011 13:56:40 +0400 (MSK) Received: from orion (localhost [127.0.0.1]) by orion.SpringDaemons.com (Postfix) with SMTP id 19FFD3A15B; Fri, 2 Dec 2011 01:56:58 -0800 (PST) Date: Fri, 2 Dec 2011 01:56:52 -0800 From: Stanislav Sedov To: Gleb Kurtsou Message-Id: <20111202015652.475ee54e.stas@FreeBSD.org> In-Reply-To: <20111119100150.GA1560@reks> References: <20111119100150.GA1560@reks> Organization: The FreeBSD Project X-Mailer: carrier-pigeon Mime-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg="PGP-SHA1"; boundary="Signature=_Fri__2_Dec_2011_01_56_52_-0800_VCl/=6NAmU9YiIfo" Cc: freebsd-hackers@FreeBSD.org, mdf@freebsd.org Subject: Re: gcc 4.2 miscompilation with -O2 -fno-omit-frame-pointer on amd64 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 09:56:44 -0000 --Signature=_Fri__2_Dec_2011_01_56_52_-0800_VCl/=6NAmU9YiIfo Content-Type: text/plain; charset=US-ASCII Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, 19 Nov 2011 12:01:50 +0200 Gleb Kurtsou mentioned: > Hi, >=20 > I was lucky to write a bit of code which gcc 4.2 fails to compile > correctly with -O2. Too keep long story short the code fails for gcc > from base system and last gcc 4.2 snapshot from ports. It works with gcc > 4.3, gcc 4.4 on FreeBSD and Linux. Clang from base is also good. -O and > -Os optimization levels are fine (I've tried with all -f* flags > mentioned in documentation) >=20 > -O2 -fno-omit-frame-pointer combination is troublesome on amd64. I > presume i386 should be fine. These options are also used for > compilation of kernel (with debugging enabled) and modules. >=20 > I'm not able to share the code, but have a test case reproducing the > bug. I've encountered the issue over a week ago and tried narrowing it do= wn > to a simple test I could share but without much success. >=20 > The code itself is very common: initialize two structs on stack, call a > function with pointers to those stucts as arguments. A number of inlined > assertion functions. gcc fails to correctly optimize struct assignments > with -fno-omit-frame-pointer, I have a number of small structs assigned, > gcc decides not to use data coping but to assign fields directly. I've > tried disabling sra, tweaking sra parameters -- no luck in forcing it > to copy data. Replacing one particular assignment with memcpy produces > correct code, but that's not a solution. >=20 > -O2 -fno-omit-frame-pointer -fno-inline is buggy > -O2 -fno-omit-frame-pointer -frename-registers is buggy >=20 > I found similar issue with gcc 4.6, but I'm not able to reproduce it > with gcc test case: > https://bugzilla.redhat.com/show_bug.cgi?id=3D679924 > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=3D47893 >=20 > I'll be glad to help debugging it and will be hanging on #bsddev during > weekend as glk. >=20 Hi! I'm not sure this is relevant to your case, but our base gcc used to have a bug with strict aliasing, which was fixed only in a GPLv3 version of it. That's why we have -fno-strict-aliasing in default CFALGS. So you might try to build using -fno-strict-aliasing. --=20 Stanislav Sedov ST4096-RIPE () ascii ribbon campaign - against html e-mail=20 /\ www.asciiribbon.org - against proprietary attachments --Signature=_Fri__2_Dec_2011_01_56_52_-0800_VCl/=6NAmU9YiIfo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJO2KDpAAoJEL8lojEJL9nw0HIP/38ws0TRzJQEXXiY2oLz/uKd r7nJ+UZ87kDI6EIMs2zYcb19lZw3vhRDLfmHxX3XTqhhQvfDhTu6fcAuDNl1VVOi qoYv+aowD7Hl7aLBUi4L28NOc8mz1hZosCfoJGzxetewYm9r1NRPXPPVQFVkR3rw yJuPF10NFTjnZJS7K/t/NQ6ibcl8Pr54VoaynfNxx6KO725wYoCBKAItpL/Ymrbl fthkbYOW2ZnFxDrfUdJiVh9ACRUyeGaZMlL3tYDvo/QpsiWnTqsjZUfD9Wa2L+vu 8Tq7oo5wWY/hM76oe+abn0lnM58Jc8iWg0GZpvNBu00LsI0yNMiCc7mbRBY6e691 dhJ3vKdGIJiHGws7mO0RnTEx3/F22rWBLqbta61a5lW5J4Znwipfhxic2B/sXXbR r3ztDCw5CS1+VJyYvdLLTkHSqgpBnr4pETGP/oil4uY5a3Gq3MsPlYaOJMBRcHFN 0nyLFGzqoMsOvOBn3tbkQZoJGqZx/qgQwbwsEl5smkL+7gsbfCKX6lktCkoY4LrY +wptXTS1vR5sqqPQuMWuJiOmje0qiFOvXQ7WyfGF6ikdMYmtZIw4945iA3+LtITZ AknNdgxWM/sjEOyacdNHdIRzPoHy8fg1Nrc69W36G1pQkIHV4q2lhmXv2ZWJKDj4 Rvcvw3KDgDBG4ExEpbfS =wi30 -----END PGP SIGNATURE----- --Signature=_Fri__2_Dec_2011_01_56_52_-0800_VCl/=6NAmU9YiIfo-- From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 11:48:22 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D30D6106564A; Fri, 2 Dec 2011 11:48:22 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-lpp01m010-f54.google.com (mail-lpp01m010-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id D375F8FC16; Fri, 2 Dec 2011 11:48:21 +0000 (UTC) Received: by lahv2 with SMTP id v2so1686189lah.13 for ; Fri, 02 Dec 2011 03:48:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=hNf7MTv9dOlftqE3GftkUmYgMAaXTJlpTxtYS/ExuZY=; b=mpf1knwNikuEL8nAQ6Grj2yQWWRqoM5eHFh2aEBnWhFcRtBXHENUAyCQCt5soag/U3 l451/YPfrdjvcWJ76CNydGVgfKRvWx142ZC6+bUINKdqoV2YARkhxNu4ms8/N7DaOWp+ 2J85oF0COCaBeg9OUlIyxx7oAEeKpYSDQQ+0I= Received: by 10.152.103.165 with SMTP id fx5mr6974230lab.38.1322826500786; Fri, 02 Dec 2011 03:48:20 -0800 (PST) Received: from localhost ([78.157.92.5]) by mx.google.com with ESMTPS id pw12sm8198079lab.13.2011.12.02.03.48.19 (version=SSLv3 cipher=OTHER); Fri, 02 Dec 2011 03:48:19 -0800 (PST) Date: Fri, 2 Dec 2011 13:48:22 +0200 From: Gleb Kurtsou To: Stanislav Sedov Message-ID: <20111202114822.GA1511@reks> References: <20111119100150.GA1560@reks> <20111202015652.475ee54e.stas@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20111202015652.475ee54e.stas@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@FreeBSD.org, mdf@freebsd.org Subject: Re: gcc 4.2 miscompilation with -O2 -fno-omit-frame-pointer on amd64 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 11:48:23 -0000 On (02/12/2011 01:56), Stanislav Sedov wrote: > On Sat, 19 Nov 2011 12:01:50 +0200 > Gleb Kurtsou mentioned: > > > Hi, > > > > I was lucky to write a bit of code which gcc 4.2 fails to compile > > correctly with -O2. Too keep long story short the code fails for gcc > > from base system and last gcc 4.2 snapshot from ports. It works with gcc > > 4.3, gcc 4.4 on FreeBSD and Linux. Clang from base is also good. -O and > > -Os optimization levels are fine (I've tried with all -f* flags > > mentioned in documentation) > > > > -O2 -fno-omit-frame-pointer combination is troublesome on amd64. I > > presume i386 should be fine. These options are also used for > > compilation of kernel (with debugging enabled) and modules. > > > > I'm not able to share the code, but have a test case reproducing the > > bug. I've encountered the issue over a week ago and tried narrowing it down > > to a simple test I could share but without much success. > > > > The code itself is very common: initialize two structs on stack, call a > > function with pointers to those stucts as arguments. A number of inlined > > assertion functions. gcc fails to correctly optimize struct assignments > > with -fno-omit-frame-pointer, I have a number of small structs assigned, > > gcc decides not to use data coping but to assign fields directly. I've > > tried disabling sra, tweaking sra parameters -- no luck in forcing it > > to copy data. Replacing one particular assignment with memcpy produces > > correct code, but that's not a solution. > > > > -O2 -fno-omit-frame-pointer -fno-inline is buggy > > -O2 -fno-omit-frame-pointer -frename-registers is buggy > > > > I found similar issue with gcc 4.6, but I'm not able to reproduce it > > with gcc test case: > > https://bugzilla.redhat.com/show_bug.cgi?id=679924 > > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47893 > > > > I'll be glad to help debugging it and will be hanging on #bsddev during > > weekend as glk. > > > > Hi! > > I'm not sure this is relevant to your case, but our base gcc used to have > a bug with strict aliasing, which was fixed only in a GPLv3 version of > it. That's why we have -fno-strict-aliasing in default CFALGS. So you > might try to build using -fno-strict-aliasing. I always have -fno-strict-aliasing, the whole idea of misusing undefined behaviour to perform optimization is crazy. I guess it seemed evident to me so I've skipped the flag above. Besides gcc was barking with aliasing warnings on 3rd party party code in my case, I had to change warnings flags to run tests without -fno-strict-aliasing. I've dropped -fno-omit-frame-pointer, async unwind tables are ok for userland. Another work around was adding -finline-functions. Kernel and modules won't build with -finline-functions. So we are just lucky not to catch it. Thanks, Gleb. > -- > Stanislav Sedov > ST4096-RIPE > > () ascii ribbon campaign - against html e-mail > /\ www.asciiribbon.org - against proprietary attachments From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 14:54:17 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AD9C106566B for ; Fri, 2 Dec 2011 14:54:17 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id CA48D8FC1A for ; Fri, 2 Dec 2011 14:54:16 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so1308730wgb.31 for ; Fri, 02 Dec 2011 06:54:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=BnN2UWr+EUtgdvJGbQGsv7qIdLuvEZmAjGRzGYJEqV8=; b=SVHdquNFor207XI97riUTeIO27HmALx9JgIX2Y1gKgzPYVYRDFPyChgSFNMFm3KKj7 QmbM/voHLy7bnwwK3ctS9Osu9yFoiLQA3udVmBJ36EnSzdoJ7nRne0q5pHkoUBj7DUo4 peKlTrEB5Fzvc932vSeGtfjXqelwz+/V/pE/Q= Received: by 10.227.58.17 with SMTP id e17mr1437468wbh.12.1322837655768; Fri, 02 Dec 2011 06:54:15 -0800 (PST) Received: from DataIX.net (ppp-21.173.dialinfree.com. [209.172.21.173]) by mx.google.com with ESMTPS id w8sm3301949wiz.4.2011.12.02.06.54.11 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 02 Dec 2011 06:54:13 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pB2Es6WH025753 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 2 Dec 2011 09:54:06 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pB2Es1Tm025752; Fri, 2 Dec 2011 09:54:01 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Fri, 2 Dec 2011 09:54:00 -0500 From: Jason Hellenthal To: Adrian Chadd Message-ID: <20111202145400.GA25736@DataIX.net> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> <4ED77650.6050409@my.gd> <20111202070055.GA98731@DataIX.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Cc: freebsd-hackers@freebsd.org, Steven Hartland Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 14:54:17 -0000 Yeah On Fri, Dec 02, 2011 at 03:19:53PM +0800, Adrian Chadd wrote: > .. where are these statistics coming from? top? > > > Adrian From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 15:08:05 2011 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93595106564A for ; Fri, 2 Dec 2011 15:08:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id EF1778FC15 for ; Fri, 2 Dec 2011 15:08:04 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA03424; Fri, 02 Dec 2011 17:08:01 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <4ED8E9D0.4070006@FreeBSD.org> Date: Fri, 02 Dec 2011 17:08:00 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steven Hartland References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> In-Reply-To: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> X-Enigmail-Version: undefined Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 15:08:05 -0000 on 30/11/2011 14:39 Steven Hartland said the following: > We're seeing some impossible memory usage stats reported on machines > here from vmstat and sysctl vm.vmtotal. > > We have machines reporting to be using 31GB total when they only have > 8GB physical and are not using any swap. > > Here's an output from one of our machines:- > vmstat -c 2 -w 1 -n 0 procs memory page > faults cpu > r b w avm fre flt re pi po fr sr in sy cs us sy id > 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 > 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 > > The raw output is:- > vmstat -c 2 -w 1 -n 0 -H > procs memory page faults cpu > r b w avm fre flt re pi po fr sr in sy cs us sy id > 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 > 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 > > Top shows:- > last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 > 1893 processes:1 running, 1892 sleeping > CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle > Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free > Swap: 4096M Total, 4096M Free > > sysctl vm.vmtotal > vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) > =============================================== > Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) > Virtual Memory: (Total: 1106403532K Active: 32540260K) > Real Memory: (Total: 4563648K Active: 3921644K) > Shared Virtual Memory: (Total: 19976K Active: 16396K) > Shared Real Memory: (Total: 9040K Active: 8436K) > Free Memory Pages: 2161740K > > As mentioned this machine has 8GB of ram and according to both top and > swapinfo is using no swap at all > >> From dmesg:- > real memory = 8589934592 (8192 MB) > avail memory = 8255553536 (7873 MB) > > swapinfo > Device 1K-blocks Used Avail Capacity > /dev/gptid/09f211f7-39ce-11e0-8 4194304 0 4194304 0% > > uname -a > FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 > root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 > > sysctl hw.pagesize > hw.pagesize: 4096 > > It looks like it may be out by a factor of 4, possibly due to the fact > the its a 4k page size not 1k as indicated by the vmstat man page:- > > memory Information about the usage of virtual and real memory. Virtual > pages (reported in units of 1024 bytes) are considered active if > they belong to processes which are running or have run in the > last 20 seconds. > > avm active virtual pages > fre size of the free list > I think that the description of avm is not sufficiently complete as it doesn't explain meaning of the "virtual memory" it refers too. Reference to units of 1024 bytes when talking about pages also sounds a bit vague. The values reported are actually byte sizes, non-humanized numbers are in KB ("units of 1024 bytes" in the poetic speech), humanized numbers carry an appropriate suffix. > Totalling up RSS from ps axo "rss" gives a total in the region of that if > the vm stats are out by a factor of 4, in this case it should be: 8132557 > which is 7.75GB a much more realistic value. > > Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 15:33:46 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E1DE106566C for ; Fri, 2 Dec 2011 15:33:46 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 2DF0E8FC16 for ; Fri, 2 Dec 2011 15:33:46 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id BC5B646B0C; Fri, 2 Dec 2011 10:33:45 -0500 (EST) Received: from John-Baldwins-MacBook-Air.local (c-68-36-150-83.hsd1.nj.comcast.net [68.36.150.83]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3EF5DB91E; Fri, 2 Dec 2011 10:33:45 -0500 (EST) Message-ID: <4ED8EFD9.5010805@FreeBSD.org> Date: Fri, 02 Dec 2011 10:33:45 -0500 From: John Baldwin User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Jason Hellenthal References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> <20111201180522.GA67513@DataIX.net> In-Reply-To: <20111201180522.GA67513@DataIX.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 02 Dec 2011 10:33:45 -0500 (EST) Cc: freebsd-hackers@freebsd.org, Steven Hartland Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 15:33:46 -0000 On 12/1/11 1:05 PM, Jason Hellenthal wrote: > > On Thu, Dec 01, 2011 at 10:44:58AM -0000, Steven Hartland wrote: >> ----- Original Message ----- >> From: "Jason Hellenthal" >> >>> This goes along with the thoughts I had about 4 months ago tending to some >>> zfs statistics as well top showing greater than 100% actual CPU usage. This >>> is a big pet peave of mine. Its like saying you ate 134% of a bannanna when >>> in all reallity it is impossible. You can never have more than 100% usage of >>> anything and when seen is a clear notice that some math is considerably >>> incorrect leading to other such miscalculations to be performed. Things like >>> the above already have checks in place that ensure no boundries are being >>> crossed/overflowed or underrun but it surely makes processing results building >>> future products a bitch. One instance is the calculation of threads for example >>> firefox can be seen using upto or more 338% of the CPU. Thats impossible its >>> like saying anyones CPU grew by 400%. >> >> I could understand a bit of overflow as stats are snapshots which may not >> be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / >> overflow. > > I agree > >> >> With respect to top showing greater than 100% by how much are you talking? >> Do your realise that each core = 100%? So if you have a quad core your system >> total will be 400% not 100%? >> > > Yeah I realize that but it still would lead you to believe that if a proccessor has 4 cores on the same die then total for each core could only be 25% usage. And the usage for a proccess only consuming full usage of 1 core is 100%. But you can start firefox on a single uniproccessor and like stated above see large usage percents near 338% or greater which is impossible and leads me to believe were forcing calculation for the entire proccess of threads onto tthread 0. This makes accounting pretty difficult. A single-package machine with 4 cores on the die is not a uniprocessor machine. It is an SMP machine. Try booting a kernel without SMP, _that_ will give you a UP machine. The %CPU usage is actually very simple if you stop trying to make it so complicated. Each potential concurrent thread of execution is mapped to 100%. Thus, if you have a system with 16 potential threads (either due to 4 quad-core packages, or 2 quad-core package where each core has 2 threads via HTT, etc.), then you have a total CPU usage of 1600%. Why does this make sense? Because a machine with 16 concurrent threads can (theoretically) do 16x the work of an otherwise identical machine with a single thread of execution within a given unit of wall time. Another way to look at it is that in FreeBSD, each thread of execution is treated as a "CPU" in top, etc. Thus, 100% CPU means that a given thread is using all of the available cycles on a CPU. If you have a multithreaded app (like Firefox) that is using all of the available cycles on 4 CPUs, then that would be 400% CPU (it's using 4 CPUs). It may only be using 25% of the available system-wide CPU cycles, but that is not what %CPU measures. Anyways, pretty much everyone I've ever talked to about this gets it right away, so the current arrangement is fine for the majority of folks. It has also been that way since FreeBSD first added SMP over a decade ago. It is also true on other OS's such as OS X, so the current arrangement is here to stay. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 16:05:08 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B1FD106566B for ; Fri, 2 Dec 2011 16:05:08 +0000 (UTC) (envelope-from prvs=1317f2055d=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 8766D8FC08 for ; Fri, 2 Dec 2011 16:05:07 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 02 Dec 2011 15:54:19 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 02 Dec 2011 15:54:19 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016838240.msg for ; Fri, 02 Dec 2011 15:54:19 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1317f2055d=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-hackers@freebsd.org Message-ID: <4EEA489A0AE34B0A98D9A99664551555@multiplay.co.uk> From: "Steven Hartland" To: "Jason Hellenthal" References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <20111201065722.GA97051@DataIX.net> <1A338C6C470940B386C307641E350A3E@multiplay.co.uk> <4ED77650.6050409@my.gd> <20111202070055.GA98731@DataIX.net> Date: Fri, 2 Dec 2011 15:54:14 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-hackers@freebsd.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 16:05:08 -0000 ----- Original Message ----- From: "Jason Hellenthal" > Just to put some visuals to this... > > . > `-- DIE > |-- Core1 [Idle] > |-- Core2 [35% ] > | `-- thread127 > |-- Core3 [40% ] > | `-- thread127 > `-- Core4 [100%] > `-- thread127 > > In this case you would say the DIE should be at a total of 175% ? I think your getting confused there; it sounds like your referring to a single CPU capable of multiple tasks via either multiple cores or HTT as an UP machine? If so that's your problem this isn't UP it SMP. Have a look on your machine in /var/run/dmesg.boot if your see it reporting more than one core then your SMP not UP hence the confusion as each of these cores be they real, virtual or physical represents a possible thread of 100% so even if you have a single physical CPU with 4 cores that still represents a possible total of 400% e.g the following shows a machine capable of 1600% if all cores are busy. FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 16 cpu9 (AP): APIC ID: 17 cpu10 (AP): APIC ID: 18 cpu11 (AP): APIC ID: 19 cpu12 (AP): APIC ID: 20 cpu13 (AP): APIC ID: 21 cpu14 (AP): APIC ID: 22 cpu15 (AP): APIC ID: 23 If you want proper UP which will total 100% you could remove SMP from your kernel but I wouldnt advise that ;-) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 17:13:13 2011 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 486B0106564A for ; Fri, 2 Dec 2011 17:13:13 +0000 (UTC) (envelope-from prvs=1317f2055d=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id B23068FC0A for ; Fri, 2 Dec 2011 17:13:12 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 02 Dec 2011 17:13:10 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 02 Dec 2011 17:13:10 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016839211.msg; Fri, 02 Dec 2011 17:13:10 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1317f2055d=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <2537A2099932494E9E7019B53EBDAA34@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <4ED8E9D0.4070006@FreeBSD.org> Date: Fri, 2 Dec 2011 17:13:05 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-hackers@FreeBSD.org Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 17:13:13 -0000 ----- Original Message ----- From: "Andriy Gapon" >> Totalling up RSS from ps axo "rss" gives a total in the region of that if >> the vm stats are out by a factor of 4, in this case it should be: 8132557 >> which is 7.75GB a much more realistic value. >> >> Am I totally missing something or is there problem here? > > Likely more of the former than of latter. Those virtual sizes are not > sufficiently explained, but you have been warned that those are not physical > sizes, so I am not sure why you try to compare the virtual figures with the > physical figures. My miss-understanding was due to what "virtual" actually meant. > Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, > here you immediately increased your virtual size counts by 1GB, even if you hadn't > accessed any bytes in the file yet and so none of them were in physical memory. > The same applies to anonymous memory. > > P.S. the above is reveled by a cursory look through the code (which is publicly > available btw) :-) Yer I did have a dig around before posting and ended up the code for vm.vmtotal, which is where vmstat gets its info from but that's just a summation of each object's size from vm_object_list. Thats where I got lost without an insight into what a vm_object.size actually represents. Your info about mmap'ed files helped point me in the right direction as it identified space that shows as virtual but doesn't show in swap or real ram, which is what I was missing. Given this starting point the following links provided me with addtional information:- http://www.freebsd.org/doc/en/books/arch-handbook/vm.html http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf I was under the incorrect impression that Virtual Memory (VM) was so named as it was a unified physical memory and swap (virtual memory), but its not that simple, as other items such as file-backed objects also count to this total which would never show in physical or swap allocation of other tools such as top and swapinfo. So what I believe is now the big cause of virtual memory uplift vs the memory totals shown by ps / top is that the vm totals include things like file backed memory mapped process binaries, shared libs etc many multiple times. This would explain why this specific machine shows the applification more than others here as it runs thousands of very small lightweight processes. Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) What do people think about expanding that entry in the man page of vmstat to clarify just what active "virtual pages" really means? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 17:49:50 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33D48106566C for ; Fri, 2 Dec 2011 17:49:50 +0000 (UTC) (envelope-from joris.dedieu@gmail.com) Received: from mail-qw0-f47.google.com (mail-qw0-f47.google.com [209.85.216.47]) by mx1.freebsd.org (Postfix) with ESMTP id ED2438FC08 for ; Fri, 2 Dec 2011 17:49:49 +0000 (UTC) Received: by qadb17 with SMTP id b17so367711qad.13 for ; Fri, 02 Dec 2011 09:49:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=Z6Is3mPH3CXFP2XHb04HfCQPhp/RdDaJNe7jHnp7gF8=; b=Exu9k/cus657RYs2hhenKVKbv47TGkHK9CyLrQbdOVdHPcdYn8hoJpBYdXLLlw2BaS 4zRqiwCjFhijmDGhbHLZAadvpq6OHFJs+AgPB9pDFEYQfaZRVn0HYlc9I73YvbtBSCHd fN6emmiAAfCMDnv80Z/zQrRRdonmXQ5skaByY= MIME-Version: 1.0 Received: by 10.224.31.69 with SMTP id x5mr3909614qac.8.1322846577567; Fri, 02 Dec 2011 09:22:57 -0800 (PST) Received: by 10.229.226.133 with HTTP; Fri, 2 Dec 2011 09:22:57 -0800 (PST) Date: Fri, 2 Dec 2011 18:22:57 +0100 Message-ID: From: joris dedieu To: freebsd-hackers Content-Type: text/plain; charset=ISO-8859-1 Subject: rtld and noexec X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 17:49:50 -0000 Hi, Here is a patch I use to prevent loading a shared object from a noexec mountpoint. It's an easy way, I found, after the last root exploit ((http://seclists.org/fulldisclosure/2011/Nov/452), to enhance the security of my web servers (with /home, /tmp and /var/tmp mounted with noexec). - the last ftpd/porftpd (libc ?) exploit does not work (indirect use of rtld via nsswitch) - the previous rtld security issue should have been more difficult to use in a noexec context. - It may help to prevent some miscellaneous usage of common softwares using dlopen like apache or php. I think it also makes sens because loading a shared object sounds like a kind of "execution". What do you think about this patch and the opportunity to open a PR on this subject? Cheers Joris --- libexec/rtld-elf/rtld.c.orig 2011-12-02 12:09:40.000000000 +0100 +++ libexec/rtld-elf/rtld.c 2011-12-02 13:45:18.000000000 +0100 @@ -1123,32 +1123,50 @@ { char *pathname; char *name; + struct statfs mnt; if (strchr(xname, '/') != NULL) { /* Hard coded pathname */ + name = NULL; if (xname[0] != '/' && !trust) { _rtld_error("Absolute pathname required for shared object \"%s\"", xname); return NULL; } if (refobj != NULL && refobj->z_origin) - return origin_subst(xname, refobj->origin_path); + pathname = origin_subst(xname, refobj->origin_path); else - return xstrdup(xname); + pathname = xstrdup(xname); + } + else { /* xname is not a path */ + if (libmap_disable || (refobj == NULL) || + (name = lm_find(refobj->path, xname)) == NULL) + name = (char *)xname; + + dbg(" Searching for \"%s\"", name); + + pathname = search_library_path(name, ld_library_path); + if (pathname == NULL && refobj != NULL) + pathname = search_library_path(name, refobj->rpath); + if (pathname == NULL) + pathname = search_library_path(name, gethints()); + if (pathname == NULL) + pathname = search_library_path(name, STANDARD_LIBRARY_PATH); + } + + if (pathname != NULL) { /* noexec mountpoint in pathname */ + if (statfs(pathname, &mnt) != 0) + free(pathname); + else { + if (mnt.f_flags & MNT_NOEXEC) { + _rtld_error("noexec violation for shared object \"%s\"", pathname); + free(pathname); + return NULL; + } + else + return pathname; + } } - if (libmap_disable || (refobj == NULL) || - (name = lm_find(refobj->path, xname)) == NULL) - name = (char *)xname; - - dbg(" Searching for \"%s\"", name); - - if ((pathname = search_library_path(name, ld_library_path)) != NULL || - (refobj != NULL && - (pathname = search_library_path(name, refobj->rpath)) != NULL) || - (pathname = search_library_path(name, gethints())) != NULL || - (pathname = search_library_path(name, STANDARD_LIBRARY_PATH)) != NULL) - return pathname; - if(refobj != NULL && refobj->path != NULL) { _rtld_error("Shared object \"%s\" not found, required by \"%s\"", name, basename(refobj->path)); From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 22:03:54 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E44531065672 for ; Fri, 2 Dec 2011 22:03:54 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 995498FC12 for ; Fri, 2 Dec 2011 22:03:54 +0000 (UTC) Received: by qaea17 with SMTP id a17so92864qae.13 for ; Fri, 02 Dec 2011 14:03:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type; bh=T4LLuI7qmdm6oa4+dy2PA31lP/6OqSD826aYzXHIW/w=; b=W4a5ui6giOWDu7FtHFCoL3nhHWoyVcDICAuUostdVPc88mX+d5gx9fKoffzRb6Cxll JNb+bZDVVuOqLoYY6o6TSPnXBANDrWjV7thkoJoR2udXSkb2WI1c6NH4cAAguRv17gHF fr2jWHfqkgNKLDDvL8Sg1Qx77SKKIUsEDHtT8= Received: by 10.224.197.202 with SMTP id el10mr67444qab.39.1322862125289; Fri, 02 Dec 2011 13:42:05 -0800 (PST) Received: from kan.dyndns.org (c-24-63-226-98.hsd1.ma.comcast.net. [24.63.226.98]) by mx.google.com with ESMTPS id ha3sm7937693qab.2.2011.12.02.13.42.03 (version=SSLv3 cipher=OTHER); Fri, 02 Dec 2011 13:42:03 -0800 (PST) Date: Fri, 2 Dec 2011 16:41:57 -0500 From: Alexander Kabaev To: joris dedieu Message-ID: <20111202164157.3058d91d@kan.dyndns.org> In-Reply-To: References: X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/AnY462gKKJBURQGWFx8zbXI"; protocol="application/pgp-signature" Cc: freebsd-hackers Subject: Re: rtld and noexec X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 22:03:55 -0000 --Sig_/AnY462gKKJBURQGWFx8zbXI Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Fri, 2 Dec 2011 18:22:57 +0100 joris dedieu wrote: > Hi, >=20 > Here is a patch I use to prevent loading a shared object from a noexec > mountpoint. It's an easy way, I found, after the last root exploit > ((http://seclists.org/fulldisclosure/2011/Nov/452), to enhance the > security of my web servers (with /home, /tmp and /var/tmp mounted with > noexec). >=20 > - the last ftpd/porftpd (libc ?) exploit does not work (indirect use > of rtld via nsswitch) > - the previous rtld security issue should have been more difficult to > use in a noexec context. > - It may help to prevent some miscellaneous usage of common softwares > using dlopen like apache or php. >=20 > I think it also makes sens because loading a shared object sounds like > a kind of "execution". >=20 > What do you think about this patch and the opportunity to open a PR on > this subject? >=20 > Cheers > Joris >=20 >=20 > --- libexec/rtld-elf/rtld.c.orig 2011-12-02 12:09:40.000000000 > +0100 +++ libexec/rtld-elf/rtld.c 2011-12-02 13:45:18.000000000 > +0100 @@ -1123,32 +1123,50 @@ > { > char *pathname; > char *name; > + struct statfs mnt; >=20 > if (strchr(xname, '/') !=3D NULL) { /* Hard coded pathname */ > + name =3D NULL; > if (xname[0] !=3D '/' && !trust) { > _rtld_error("Absolute pathname required for shared object > \"%s\"", xname); > return NULL; > } > if (refobj !=3D NULL && refobj->z_origin) > - return origin_subst(xname, refobj->origin_path); > + pathname =3D origin_subst(xname, refobj->origin_path); > else > - return xstrdup(xname); > + pathname =3D xstrdup(xname); > + } > + else { /* xname is not a path */ > + if (libmap_disable || (refobj =3D=3D NULL) || > + (name =3D lm_find(refobj->path, xname)) =3D=3D NULL) > + name =3D (char *)xname; > + > + dbg(" Searching for \"%s\"", name); > + > + pathname =3D search_library_path(name, ld_library_path); > + if (pathname =3D=3D NULL && refobj !=3D NULL) > + pathname =3D search_library_path(name, refobj->rpath); > + if (pathname =3D=3D NULL) > + pathname =3D search_library_path(name, gethints()); > + if (pathname =3D=3D NULL) > + pathname =3D search_library_path(name, > STANDARD_LIBRARY_PATH); > + } > + > + if (pathname !=3D NULL) { /* noexec mountpoint in pathname */ > + if (statfs(pathname, &mnt) !=3D 0) > + free(pathname); > + else { > + if (mnt.f_flags & MNT_NOEXEC) { > + _rtld_error("noexec violation for shared object > \"%s\"", pathname); > + free(pathname); > + return NULL; > + } > + else > + return pathname; > + } > } >=20 > - if (libmap_disable || (refobj =3D=3D NULL) || > - (name =3D lm_find(refobj->path, xname)) =3D=3D NULL) > - name =3D (char *)xname; > - > - dbg(" Searching for \"%s\"", name); > - > - if ((pathname =3D search_library_path(name, ld_library_path)) !=3D > NULL || > - (refobj !=3D NULL && > - (pathname =3D search_library_path(name, refobj->rpath)) !=3D NULL) > || > - (pathname =3D search_library_path(name, gethints())) !=3D NULL || > - (pathname =3D search_library_path(name, > STANDARD_LIBRARY_PATH)) !=3D NULL) > - return pathname; > - > if(refobj !=3D NULL && refobj->path !=3D NULL) { > _rtld_error("Shared object \"%s\" not found, required by > \"%s\"", name, basename(refobj->path)); > _______________________________________________ 1. There is a race using statfs and then loading the file. 2. We already have the check in do_load_object --=20 Alexander Kabaev --Sig_/AnY462gKKJBURQGWFx8zbXI Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iD8DBQFO2UYqQ6z1jMm+XZYRAiW3AJ9cEXng9NgR8lO/tWakLY8lqLSK4gCeKys0 lDv9CfGN3HZloh/QXW9szNU= =E56d -----END PGP SIGNATURE----- --Sig_/AnY462gKKJBURQGWFx8zbXI-- From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 22:26:19 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 428FB1065670 for ; Fri, 2 Dec 2011 22:26:19 +0000 (UTC) (envelope-from lists@mschuette.name) Received: from mail.asta.uni-potsdam.de (mail.asta.uni-potsdam.de [IPv6:2001:638:807:3a:20d:56ff:fefd:1183]) by mx1.freebsd.org (Postfix) with ESMTP id BB3D88FC0C for ; Fri, 2 Dec 2011 22:26:18 +0000 (UTC) Received: from mail.asta.uni-potsdam.de (mail.asta.uni-potsdam.de [IPv6:2001:638:807:3a:20d:56ff:fefd:1183]) by mail.asta.uni-potsdam.de (Postfix) with ESMTP id B20311534DC for ; Fri, 2 Dec 2011 23:26:17 +0100 (CET) X-Virus-Scanned: on mail at asta.uni-potsdam.de Received: from mail.asta.uni-potsdam.de ([141.89.58.198]) by mail.asta.uni-potsdam.de (mail.asta.uni-potsdam.de [141.89.58.198]) (amavisd-new, port 10024) with ESMTP id 0Wdsh7VtT-BA for ; Fri, 2 Dec 2011 23:26:15 +0100 (CET) Received: from dagny.mschuette.name (cl-485.dus-01.de.sixxs.net [IPv6:2a01:198:200:1e4::2]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "Martin Schuette", Issuer "AStA-CA" (verified OK)) (Authenticated sender: mschuett) by mail.asta.uni-potsdam.de (Postfix) with ESMTPSA id BDF151534D3 for ; Fri, 2 Dec 2011 23:26:15 +0100 (CET) Message-ID: <4ED95086.3040107@mschuette.name> Date: Fri, 02 Dec 2011 23:26:14 +0100 From: =?UTF-8?B?TWFydGluIFNjaMO8dHRl?= User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:7.0.1) Gecko/20111110 Thunderbird/7.0.1 MIME-Version: 1.0 CC: freebsd-hackers@freebsd.org References: In-Reply-To: X-Enigmail-Version: undefined Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: To implement RFC 5848 (Signed Syslog Messages)? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 22:26:19 -0000 On 12/01/11 10:01, Zhihao Yuan wrote: > trivial. How about to implement RFC 5848 in our syslogd? In 2008 I implemented the syslog RFCs for NetBSD's syslogd, so if you are interested please take a look at the syslog code in NetBSD-current and at my report, linked under http://mschuette.name/wp/gsoc-syslogd/ I've always wanted to create a FreeBSD port for it, but never found enough time to continue the development :( > Albert Mietus made a nice presentation in 2002 > http://www.slideshare.net/SoftwareBeterMaken.nl/securing-syslog-on-freebsd > > Not sure whether his code is accessible or not. Albert Mietus' code is available at http://sourceforge.net/projects/syslog-sec/ -- Martin From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 22:45:03 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC53B106566C for ; Fri, 2 Dec 2011 22:45:03 +0000 (UTC) (envelope-from lichray@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4B9FE8FC15 for ; Fri, 2 Dec 2011 22:45:02 +0000 (UTC) Received: by eaai12 with SMTP id i12so4638537eaa.13 for ; Fri, 02 Dec 2011 14:45:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=jX/qy+Gym+ByWM1OLnD8J1/j9pKPinrXAbsqmCYcwwA=; b=qAqrSgvFVVn0x+5bGElkA8Lj5vkxVo1qgpW5qBFfqzwBmKcv04Y81CX8T44k/kszzL R3Lj5oBh5mu2bg/Oa7poif/enmq32qI9ARJ344CPxVpcDAUlw8BvfSVoQRKKdbcoEl6a DBA4frUSjVYP9/4OJkSWvg1WbQuPbFXTrN1Ns= MIME-Version: 1.0 Received: by 10.213.35.140 with SMTP id p12mr274637ebd.125.1322865901714; Fri, 02 Dec 2011 14:45:01 -0800 (PST) Received: by 10.14.47.194 with HTTP; Fri, 2 Dec 2011 14:45:01 -0800 (PST) In-Reply-To: <4ED95086.3040107@mschuette.name> References: <4ED95086.3040107@mschuette.name> Date: Fri, 2 Dec 2011 16:45:01 -0600 Message-ID: From: Zhihao Yuan To: =?UTF-8?Q?Martin_Sch=C3=BCtte?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: To implement RFC 5848 (Signed Syslog Messages)? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 22:45:03 -0000 2011/12/2 Martin Sch=C3=BCtte : > On 12/01/11 10:01, Zhihao Yuan wrote: >> trivial. How about to implement RFC 5848 in our syslogd? > > In 2008 I implemented the syslog RFCs for NetBSD's syslogd, so if you > are interested please take a look at the syslog code in NetBSD-current > and at my report, linked under http://mschuette.name/wp/gsoc-syslogd/ That's an amazing work. Did you compared those documents (they were drafts in 08') with the final versions? Any differences? > > I've always wanted to create a FreeBSD port for it, but never found > enough time to continue the development =C2=A0:( > >> Albert Mietus made a nice presentation in 2002 >> http://www.slideshare.net/SoftwareBeterMaken.nl/securing-syslog-on-freeb= sd >> >> Not sure whether his code is accessible or not. > > Albert Mietus' code is available at > http://sourceforge.net/projects/syslog-sec/ > > -- > Martin > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " --=20 Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___________________________________________________ 4BSD -- http://4bsd.biz/ From owner-freebsd-hackers@FreeBSD.ORG Fri Dec 2 23:26:29 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89486106564A for ; Fri, 2 Dec 2011 23:26:29 +0000 (UTC) (envelope-from lists@mschuette.name) Received: from mail.asta.uni-potsdam.de (mail.asta.uni-potsdam.de [IPv6:2001:638:807:3a:20d:56ff:fefd:1183]) by mx1.freebsd.org (Postfix) with ESMTP id 3CAC28FC14 for ; Fri, 2 Dec 2011 23:26:29 +0000 (UTC) Received: from mail.asta.uni-potsdam.de (mail.asta.uni-potsdam.de [IPv6:2001:638:807:3a:20d:56ff:fefd:1183]) by mail.asta.uni-potsdam.de (Postfix) with ESMTP id 7D7131534DD for ; Sat, 3 Dec 2011 00:26:27 +0100 (CET) X-Virus-Scanned: on mail at asta.uni-potsdam.de Received: from mail.asta.uni-potsdam.de ([141.89.58.198]) by mail.asta.uni-potsdam.de (mail.asta.uni-potsdam.de [141.89.58.198]) (amavisd-new, port 10024) with ESMTP id sqM1xRAYby6q for ; Sat, 3 Dec 2011 00:26:26 +0100 (CET) Received: from dagny.mschuette.name (cl-485.dus-01.de.sixxs.net [IPv6:2a01:198:200:1e4::2]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "Martin Schuette", Issuer "AStA-CA" (verified OK)) (Authenticated sender: mschuett) by mail.asta.uni-potsdam.de (Postfix) with ESMTPSA id ECA2A1534DC for ; Sat, 3 Dec 2011 00:26:25 +0100 (CET) Message-ID: <4ED95EA0.4000709@mschuette.name> Date: Sat, 03 Dec 2011 00:26:24 +0100 From: =?UTF-8?B?TWFydGluIFNjaMO8dHRl?= User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:7.0.1) Gecko/20111110 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <4ED95086.3040107@mschuette.name> In-Reply-To: X-Enigmail-Version: undefined Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: To implement RFC 5848 (Signed Syslog Messages)? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Dec 2011 23:26:29 -0000 On 12/02/11 23:45, Zhihao Yuan wrote: >> In 2008 I implemented the syslog RFCs for NetBSD's syslogd, so if you > That's an amazing work. Did you compared those documents (they were > drafts in 08') with the final versions? Any differences? I followed the IETF process and as far as I know there are two major differences: a) For syslog-sign I encoded signatures with a PEM format in the way of X.509/OpenSSL, but the final RFC specifies an OpenPGP-like encoding. b) For TLS transport the rules for peer certificate verification (always a very confusing problem) were discussed and modified in the later drafts. Most notably the RFC requires support for wildcards in DNS names, which is not implemented. -- Martin From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 04:19:08 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11D06106566C for ; Sat, 3 Dec 2011 04:19:08 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 94E9A8FC14 for ; Sat, 3 Dec 2011 04:19:07 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so2592512wgb.31 for ; Fri, 02 Dec 2011 20:19:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=CbYNQLTAU5WcrC4mb12R/Klz+U1+rHxx9hBkSPcDvtM=; b=HnMWvPx3IJdFO5BSxDCrTm4hfLctPoe3f5n+WpgmxbHWdcj2tsrcQZNMFnapya9bPr MLuJpPlczdyei/CyTAlMPjK8fZcAo7NIl9bUV/NpfhfjqK3dgwnsvBLO+XnCV9glGpoF jXuXZAgtSTe9l0y5hQr6F+j7F/upIZqEhXUZo= Received: by 10.180.24.65 with SMTP id s1mr1443315wif.59.1322885944722; Fri, 02 Dec 2011 20:19:04 -0800 (PST) Received: from DataIX.net (ppp-21.171.dialinfree.com. [209.172.21.171]) by mx.google.com with ESMTPS id fg15sm13840380wbb.7.2011.12.02.20.19.00 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 02 Dec 2011 20:19:03 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pB34ItYQ059772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 2 Dec 2011 23:18:55 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pB34Il2u059771; Fri, 2 Dec 2011 23:18:47 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Fri, 2 Dec 2011 23:18:47 -0500 From: Jason Hellenthal To: Steven Hartland Message-ID: <20111203041847.GC38979@DataIX.net> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <4ED8E9D0.4070006@FreeBSD.org> <2537A2099932494E9E7019B53EBDAA34@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2537A2099932494E9E7019B53EBDAA34@multiplay.co.uk> Cc: freebsd-hackers@freebsd.org, Andriy Gapon Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 04:19:08 -0000 On Fri, Dec 02, 2011 at 05:13:05PM -0000, Steven Hartland wrote: > ----- Original Message ----- > From: "Andriy Gapon" > > >> Totalling up RSS from ps axo "rss" gives a total in the region of that if > >> the vm stats are out by a factor of 4, in this case it should be: 8132557 > >> which is 7.75GB a much more realistic value. > >> > >> Am I totally missing something or is there problem here? > > > > Likely more of the former than of latter. Those virtual sizes are not > > sufficiently explained, but you have been warned that those are not physical > > sizes, so I am not sure why you try to compare the virtual figures with the > > physical figures. > > My miss-understanding was due to what "virtual" actually meant. > > > Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, > > here you immediately increased your virtual size counts by 1GB, even if you hadn't > > accessed any bytes in the file yet and so none of them were in physical memory. > > The same applies to anonymous memory. > > > > P.S. the above is reveled by a cursory look through the code (which is publicly > > available btw) :-) > > Yer I did have a dig around before posting and ended up the code for vm.vmtotal, > which is where vmstat gets its info from but that's just a summation of each object's > size from vm_object_list. Thats where I got lost without an insight into what > a vm_object.size actually represents. > > Your info about mmap'ed files helped point me in the right direction as it > identified space that shows as virtual but doesn't show in swap or real ram, > which is what I was missing. > > Given this starting point the following links provided me with addtional > information:- > http://www.freebsd.org/doc/en/books/arch-handbook/vm.html > http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html > http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ > http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf > > I was under the incorrect impression that Virtual Memory (VM) was so named as it > was a unified physical memory and swap (virtual memory), but its not that simple, > as other items such as file-backed objects also count to this total which would > never show in physical or swap allocation of other tools such as top and swapinfo. > > So what I believe is now the big cause of virtual memory uplift vs the memory > totals shown by ps / top is that the vm totals include things like file backed > memory mapped process binaries, shared libs etc many multiple times. > > This would explain why this specific machine shows the applification more than > others here as it runs thousands of very small lightweight processes. > > Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) > > What do people think about expanding that entry in the man page of vmstat to > clarify just what active "virtual pages" really means? > > Regards > Steve > Thanks for your research Steve. That makes perfect sense and additions to the documentation are surely needed. From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 09:06:17 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C0E5106564A for ; Sat, 3 Dec 2011 09:06:17 +0000 (UTC) (envelope-from filippo.sironi@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8CCBA8FC08 for ; Sat, 3 Dec 2011 09:06:16 +0000 (UTC) Received: by eekc13 with SMTP id c13so2847777eek.13 for ; Sat, 03 Dec 2011 01:06:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:subject:date:message-id :cc:to:mime-version:x-mailer; bh=LndU+Wd85oQh5x0UhoOWwAPSPBCBn/l0UiqreQQFtag=; b=c2UDduKCbWK/glj80Y8hKsr2oRQJ0T4EWjOEdgCvHtty8y8bIz/CUuFVazMPuWfM2f tVTDpJvkNpYOlplojqYxyMK02Yz9QHuksjoAp3jmHXIgVTI5DrwaBQQ80YBEjEQBYsBB Bmpd9tO0gnFiwB6t+oKV4N7jQ1gOZnNUgzw1g= Received: by 10.14.9.84 with SMTP id 60mr122846ees.60.1322901837884; Sat, 03 Dec 2011 00:43:57 -0800 (PST) Received: from filippo.sironi.dynamic.micro.elet.polimi.it (micro.elet.polimi.it. [131.175.127.118]) by mx.google.com with ESMTPS id z54sm35429315eeh.5.2011.12.03.00.43.56 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 03 Dec 2011 00:43:56 -0800 (PST) From: Filippo Sironi Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Sat, 3 Dec 2011 09:42:44 +0100 Message-Id: To: freebsd-hackers@freebsd.org Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) Cc: Sironi Filippo Subject: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 09:06:17 -0000 Dear all, I've a strange problem I cannot understand... I implemented a piece of code within the FreeBSD 7.2 kernel that = basically allocate a per-thread memory page to store a certain amount of = information that must be read and written without crossing user to = kernel and kernel to user boundaries. To read and write the memory page I decided to allocate it in kernel = space and then mmap'ing it to user space using a "virtual cdev" as an = entry point for user space threads. The implementation of the mmap I = came up with is really trivial, I basically do: *paddr =3D vtophys(curthread->private_info); Now for the problem. I enter the mmap and private_info is the correct kernel space virtual = address, each thread as its own address and the physical address - = returned by vtophys - is different too. The problem is that when I come = back to user space all the threads write in memory page mapped by the = first thread that called the mmap leaving their memory pages untouched. This problem bugs me but I cannot find a solution. I don't really = understand the behavior. Any ideas? Thanks a lot, Filippo From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 09:20:57 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1760106566C for ; Sat, 3 Dec 2011 09:20:57 +0000 (UTC) (envelope-from filippo.sironi@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6CA128FC08 for ; Sat, 3 Dec 2011 09:20:56 +0000 (UTC) Received: by bkat2 with SMTP id t2so5602820bka.13 for ; Sat, 03 Dec 2011 01:20:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=1J3Q61eyt2JBJVYwbL/gGqykcexQokpK9I38yB2cYX4=; b=cqVgAAcJ7N0Hpk4jBD18pYO6EcyjhgkV2CeddQYveMBHRazXIn1LzJ5V0yniedxwDq xhFrLLUT4sQiRleLBq7a4ShHEFQCRZsDpj/Z4rXjkJb3eCDqgwPuj6iSE7qCHNUYLTtF 1DuyS4pPJdxujesQxs1wuES5tb5lhXadAhprc= Received: by 10.204.141.2 with SMTP id k2mr773433bku.81.1322904056132; Sat, 03 Dec 2011 01:20:56 -0800 (PST) Received: from filippo.sironi.dynamic.micro.elet.polimi.it (micro.elet.polimi.it. [131.175.127.118]) by mx.google.com with ESMTPS id q6sm20828586bka.6.2011.12.03.01.20.54 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 03 Dec 2011 01:20:55 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Filippo Sironi In-Reply-To: <10964.1322903784@critter.freebsd.dk> Date: Sat, 3 Dec 2011 10:19:43 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <10964.1322903784@critter.freebsd.dk> To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.1084) Cc: Sironi Filippo , Poul-Henning Kamp Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 09:20:58 -0000 Ok, that's what I was suspecting. Do you have any ideas on how to get the behavior I described in the = previous mail? On 03/dic/2011, at 10.16, Poul-Henning Kamp wrote: > In message , Filippo = Sironi wri > tes: >=20 >> The problem is that when I come =3D >> back to user space all the threads write in memory page mapped by the = =3D >> first thread that called the mmap leaving their memory pages = untouched. >=20 > That's how cdev->mmap works, it is global rather than per-thread. >=20 >=20 > --=20 > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe =20 > Never attribute to malice what can adequately be explained by = incompetence. From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 10:38:16 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1FAED106564A for ; Sat, 3 Dec 2011 10:38:16 +0000 (UTC) (envelope-from filippo.sironi@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9F3D28FC0A for ; Sat, 3 Dec 2011 10:38:15 +0000 (UTC) Received: by bkat2 with SMTP id t2so5653432bka.13 for ; Sat, 03 Dec 2011 02:38:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=+kp61R6MShXXcktAvj2iBntQwy50XDd9v5Z4LWuzlVY=; b=VT80ecOkULMP6BHyghry6q5t7Ctcr5nm5SWMDOU0pYy0Qv1rEAmIN9hEbD+70U3E8D F6fEUVbKBO0N50ZxN+I25ZO3DGhrkZXz12UQBoSbut1DByaCJGIvsDPPxpP2RtSyOYFy 6Ret8kxUd7HcyUIpCPKUqmKmElblXB+oZt+r4= Received: by 10.204.155.141 with SMTP id s13mr895087bkw.40.1322908694373; Sat, 03 Dec 2011 02:38:14 -0800 (PST) Received: from filippo.sironi.dynamic.micro.elet.polimi.it (micro.elet.polimi.it. [131.175.127.118]) by mx.google.com with ESMTPS id p13sm21095883bkd.4.2011.12.03.02.38.13 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 03 Dec 2011 02:38:13 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Filippo Sironi In-Reply-To: <11076.1322906459@critter.freebsd.dk> Date: Sat, 3 Dec 2011 11:37:01 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <0011FD6A-E29D-4F67-913C-897BA1B2F56F@gmail.com> References: <11076.1322906459@critter.freebsd.dk> To: freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.1084) Cc: Poul-Henning Kamp Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 10:38:16 -0000 I need to access the memory both from user space and kernel space, I = cannot do that (simply) with an mmap or thread-specific storage if I = recall correctly. On 03/dic/2011, at 11.00, Poul-Henning Kamp wrote: > In message , Filippo = Sironi wri > tes: >=20 > Why don't you just use mmap(2) ? I couldn't see anything you > couldn't do with it. >=20 > There's also support in pthread for thread specific storage, which > should be your first choice. >=20 >=20 > --=20 > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe =20 > Never attribute to malice what can adequately be explained by = incompetence. From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 09:31:59 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32879106566C for ; Sat, 3 Dec 2011 09:31:59 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id E64328FC0C for ; Sat, 3 Dec 2011 09:31:58 +0000 (UTC) Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id CCB8F5DAA; Sat, 3 Dec 2011 09:16:24 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id pB39GOZb010965; Sat, 3 Dec 2011 09:16:24 GMT (envelope-from phk@phk.freebsd.dk) To: Filippo Sironi From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sat, 03 Dec 2011 09:42:44 +0100." Content-Type: text/plain; charset=ISO-8859-1 Date: Sat, 03 Dec 2011 09:16:24 +0000 Message-ID: <10964.1322903784@critter.freebsd.dk> X-Mailman-Approved-At: Sat, 03 Dec 2011 12:14:16 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 09:31:59 -0000 In message , Filippo Sironi wri tes: >The problem is that when I come = >back to user space all the threads write in memory page mapped by the = >first thread that called the mmap leaving their memory pages untouched. That's how cdev->mmap works, it is global rather than per-thread. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 10:01:00 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBAA5106564A for ; Sat, 3 Dec 2011 10:01:00 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 9B5C88FC0A for ; Sat, 3 Dec 2011 10:01:00 +0000 (UTC) Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 67AEE5DC0; Sat, 3 Dec 2011 10:00:59 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id pB3A0xXg011077; Sat, 3 Dec 2011 10:00:59 GMT (envelope-from phk@phk.freebsd.dk) To: Filippo Sironi From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sat, 03 Dec 2011 10:19:43 +0100." Content-Type: text/plain; charset=ISO-8859-1 Date: Sat, 03 Dec 2011 10:00:59 +0000 Message-ID: <11076.1322906459@critter.freebsd.dk> X-Mailman-Approved-At: Sat, 03 Dec 2011 12:23:24 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 10:01:00 -0000 In message , Filippo Sironi wri tes: >Ok, that's what I was suspecting. >Do you have any ideas on how to get the behavior I described in the = >previous mail? Why don't you just use mmap(2) ? I couldn't see anything you couldn't do with it. There's also support in pthread for thread specific storage, which should be your first choice. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 11:03:37 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89C90106566C for ; Sat, 3 Dec 2011 11:03:37 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 485488FC08 for ; Sat, 3 Dec 2011 11:03:37 +0000 (UTC) Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 13A665DC9; Sat, 3 Dec 2011 11:03:36 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.5/8.14.5) with ESMTP id pB3B3ZVX011432; Sat, 3 Dec 2011 11:03:35 GMT (envelope-from phk@phk.freebsd.dk) To: Filippo Sironi From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sat, 03 Dec 2011 11:37:01 +0100." <0011FD6A-E29D-4F67-913C-897BA1B2F56F@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Date: Sat, 03 Dec 2011 11:03:35 +0000 Message-ID: <11431.1322910215@critter.freebsd.dk> X-Mailman-Approved-At: Sat, 03 Dec 2011 12:23:33 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 11:03:37 -0000 In message <0011FD6A-E29D-4F67-913C-897BA1B2F56F@gmail.com>, Filippo Sironi wri tes: >I need to access the memory both from user space and kernel space, I = >cannot do that (simply) with an mmap or thread-specific storage if I = >recall correctly. Have you looked how shm_open(2) and friends are implemented ? That might allow you to. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 17:38:02 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0DCDD1065678 for ; Sat, 3 Dec 2011 17:38:02 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id D65AE8FC14 for ; Sat, 3 Dec 2011 17:38:01 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id pB3HbwUW085687 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 3 Dec 2011 09:38:00 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4EDA5E84.1030605@freebsd.org> Date: Sat, 03 Dec 2011 09:38:12 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.24) Gecko/20111103 Thunderbird/3.1.16 MIME-Version: 1.0 To: Filippo Sironi References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 17:38:02 -0000 On 12/3/11 12:42 AM, Filippo Sironi wrote: > Dear all, > > I've a strange problem I cannot understand... > I implemented a piece of code within the FreeBSD 7.2 kernel that basically allocate a per-thread memory page to store a certain amount of information that must be read and written without crossing user to kernel and kernel to user boundaries. > To read and write the memory page I decided to allocate it in kernel space and then mmap'ing it to user space using a "virtual cdev" as an entry point for user space threads. The implementation of the mmap I came up with is really trivial, I basically do: > > *paddr = vtophys(curthread->private_info); > > Now for the problem. > I enter the mmap and private_info is the correct kernel space virtual address, each thread as its own address and the physical address - returned by vtophys - is different too. The problem is that when I come back to user space all the threads write in memory page mapped by the first thread that called the mmap leaving their memory pages untouched. > > This problem bugs me but I cannot find a solution. I don't really understand the behavior. > each thread needs to store a different address as the base of it's memory.. the memory map of a process is just that.. the memory map of the PROCESS All threads see the same map. > Any ideas? > Thanks a lot, > Filippo > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 3 17:43:40 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DC87106566C for ; Sat, 3 Dec 2011 17:43:40 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 214858FC15 for ; Sat, 3 Dec 2011 17:43:39 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id pB3Hhb70085706 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 3 Dec 2011 09:43:38 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4EDA5FD7.9030903@freebsd.org> Date: Sat, 03 Dec 2011 09:43:51 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.24) Gecko/20111103 Thunderbird/3.1.16 MIME-Version: 1.0 To: Filippo Sironi References: <11076.1322906459@critter.freebsd.dk> <0011FD6A-E29D-4F67-913C-897BA1B2F56F@gmail.com> In-Reply-To: <0011FD6A-E29D-4F67-913C-897BA1B2F56F@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Poul-Henning Kamp Subject: Re: mmap implementation for cdev X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2011 17:43:40 -0000 On 12/3/11 2:37 AM, Filippo Sironi wrote: > I need to access the memory both from user space and kernel space, I cannot do that (simply) with an mmap or thread-specific storage if I recall correctly. that's how mmap works.. if you give it pages to expose to the user, you can still access them from in the kernel. you just have to make the threads in user land access different addresses. > On 03/dic/2011, at 11.00, Poul-Henning Kamp wrote: > >> In message, Filippo Sironi wri >> tes: >> >> Why don't you just use mmap(2) ? I couldn't see anything you >> couldn't do with it. >> >> There's also support in pthread for thread specific storage, which >> should be your first choice. >> >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk@FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > >