From owner-freebsd-hackers@FreeBSD.ORG Sun Apr 7 14:49:51 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 16B1CF14 for ; Sun, 7 Apr 2013 14:49:51 +0000 (UTC) (envelope-from sebastian.n.feld@gmail.com) Received: from mail-ie0-x242.google.com (mail-ie0-x242.google.com [IPv6:2607:f8b0:4001:c03::242]) by mx1.freebsd.org (Postfix) with ESMTP id E676EFBC for ; Sun, 7 Apr 2013 14:49:50 +0000 (UTC) Received: by mail-ie0-f194.google.com with SMTP id qd14so1798920ieb.9 for ; Sun, 07 Apr 2013 07:49:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=mLpE5dh2JRgmvvdTMse3fmVqZEu0wt93WOHbJUsv6Ys=; b=kwPo0UxZ0i6Z+3gAVwLtCuyhju6EKURgMXGMVkxmz0m15WdxCbQT8p2MPdWBpiDp37 U90J/d3Rp0oPftJU6KrKlMsz9VaorR2wkkyf1ZFJCHdrmMMFFlMpyOOrkClzRkNpsC9l xa6sav+6Ub1ojh5KatV7pbU9zrgxY45oE9P2qH03x2rQVc+v8rBiw9gRVif2kdYvzh/k 2TVvTg4W6oEAr8h2mvXT5XcmBG7y49jqhi2zGwkOclxteQr3d1p3z07Ze4IDbgWAMyXV ji0ir47nFjRcbCQSyh3HlSUPtqlUlnvXbQln37gY3A7sVDFJspIVTgqbJmKX5B0h2eOR FOFw== MIME-Version: 1.0 X-Received: by 10.50.155.134 with SMTP id vw6mr4352861igb.34.1365346190728; Sun, 07 Apr 2013 07:49:50 -0700 (PDT) Received: by 10.42.101.20 with HTTP; Sun, 7 Apr 2013 07:49:50 -0700 (PDT) Date: Sun, 7 Apr 2013 16:49:50 +0200 Message-ID: Subject: Multiple page size support on FreeBSD? From: Sebastian Feld To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2013 14:49:51 -0000 Does FreeBSD support multiple page sizes in a single userland process like Solaris 11 does? From owner-freebsd-hackers@FreeBSD.ORG Sun Apr 7 19:12:59 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 152A491A; Sun, 7 Apr 2013 19:12:59 +0000 (UTC) (envelope-from cedric.blancher@gmail.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) by mx1.freebsd.org (Postfix) with ESMTP id D811AAD8; Sun, 7 Apr 2013 19:12:58 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id 17so6035613iea.26 for ; Sun, 07 Apr 2013 12:12:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=rjfbGtvoYwaJdEDHzDy0t2A8GueIK7clV/XT6dnYpKI=; b=tmSPxJm5MLK+RzhGBoF2ZS6CCwEIfBy48XngFtzbldDGLGY/RYhZWzzogCuRwnqmAk 0XuQAv1wtkDisiVhk6gaVRG37gjhrW46hmorOsnKNxNmlyluYD317L6WOhZ8vC+C+hbJ HvzKNOYLjFmEjnJVJTXyxEQrJANTabaWYTi0ufkt5LL3DCTybw+SSqCuaIyFytgy15Iy SnYZ/YVSfWn0NccSAVle73o6epFIytcic61TTR1MOb2NrjpxOvffnwcA+73C6YIg+X/o X7ZvnaLKYE5bYdJK6NruzKb5H3xVVLor7q53cyvuyeXG7bDgjwqA8azsU8IXgFLWXx/7 O3hg== MIME-Version: 1.0 X-Received: by 10.50.153.81 with SMTP id ve17mr4934010igb.24.1365361977954; Sun, 07 Apr 2013 12:12:57 -0700 (PDT) Sender: cedric.blancher@gmail.com Received: by 10.50.40.36 with HTTP; Sun, 7 Apr 2013 12:12:57 -0700 (PDT) In-Reply-To: References: Date: Sun, 7 Apr 2013 21:12:57 +0200 X-Google-Sender-Auth: HnVli7FsgViY2Y6OnjPE8jF2ewY Message-ID: Subject: Fwd: Where does FreeBSD tr -C differ from tr -c? From: Cedric Blancher To: freebsd-hackers@freebsd.org, freebsd-standards@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2013 19:12:59 -0000 Forwarding to freebsd-hackers@/freebsd-standards@freebsd.org The question remain open and I need help. tr -C is implemented by FreeBSD tr -C but I can't find examples (or a testcase) where tr -c and tr -C differ. Ced PS: Who wrote tr -C and how can I contact the author? ---------- Forwarded message ---------- From: Cedric Blancher Date: 5 April 2013 16:04 Subject: Where does FreeBSD tr -C differ from tr -c? To: freebsd-i18n@freebsd.org Has anyone discovered examples for FreeBSD tr -c producing different output than tr -C? I tried this (ksh93) test script but it NEVER produces a difference in the en_US.utf8 and fr_FR.utf-8 locales: ------ snip ------ builtin rm typeset string typeset -li16 n_ch ; for (( n_ch=1 ; n_ch < 0x5000 ; n_ch++ )) ; do ch="$(printf "\u[${n_ch/~(El)16#/}]")" string+="$ch" done typeset -li16 m1 m2 for (( m1=0x32 ; m1 < 0x3000 ; m1+=7 )) ; do (( m2=m1+1500 )) range="$(printf "\u[${m1/~(El)16#/}]-\u[${m2/~(El)16#/}]")" tr -Cd "$range" <<<"$string" >'res_C' & tr -cd "$range" <<<"$string" >'res_c' & wait res_c="$( <'res_c' )" res_C="$( <'res_C' )" rm 'res_c' 'res_C' if [[ "$res_c" != "$res_C" ]] ; then printf 'DIFFER range=%q\n' "${range}" fi done exit 0 ------ snip ------ So when does tr -C differ from tr -c? I need examples, please... Ced -- Cedric Blancher Institute Pasteur -- Cedric Blancher Institute Pasteur From owner-freebsd-hackers@FreeBSD.ORG Sun Apr 7 20:32:06 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B5424A83; Sun, 7 Apr 2013 20:32:06 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (unknown [IPv6:2001:610:1108:5012::107]) by mx1.freebsd.org (Postfix) with ESMTP id 803E5EF0; Sun, 7 Apr 2013 20:32:06 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id AF4C6120207; Sun, 7 Apr 2013 22:31:51 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 6A0322848C; Sun, 7 Apr 2013 22:31:51 +0200 (CEST) Date: Sun, 7 Apr 2013 22:31:51 +0200 From: Jilles Tjoelker To: Cedric Blancher Subject: Re: Fwd: Where does FreeBSD tr -C differ from tr -c? Message-ID: <20130407203151.GA47134@stack.nl> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org, freebsd-standards@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2013 20:32:06 -0000 On Sun, Apr 07, 2013 at 09:12:57PM +0200, Cedric Blancher wrote: > Forwarding to freebsd-hackers@/freebsd-standards@freebsd.org > The question remain open and I need help. tr -C is implemented by > FreeBSD tr -C but I can't find examples (or a testcase) where tr -c > and tr -C differ. Reading the rationale of POSIX, here is an example of a difference: % printf 'a\200'|LC_ALL=en_US.US-ASCII tr -cd '\000-\177'|hd 00000000 61 |a| 00000001 % printf 'a\200'|LC_ALL=en_US.US-ASCII tr -Cd '\000-\177'|hd 00000000 61 80 |a.| 00000002 Because the bytes 128..255 are not characters in us-ascii, they cannot be removed with -Cd, only with -cd. Here is another difference (using LC_CTYPE=en_US.UTF-8, rest C): % echo $'\U0001a000'|tr -cd '\U0001a000'|hd % echo $'\U0001a000'|tr -Cd '\U0001a000'|hd 00000000 f0 9a 80 80 |....| 00000004 The cause is that iswrune(3) returns false for the unassigned code point U+0001A000. This may well contain bugs because Unicode adds new characters from time to time and our tables seem to be updated very rarely. POSIX also says things about collation order. You may not have detected this because FreeBSD does not implement LC_COLLATE for multibyte locales yet. > PS: Who wrote tr -C and how can I contact the author? You can read the Subversion logs but people may no longer be around. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 03:22:46 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7213B763 for ; Mon, 8 Apr 2013 03:22:46 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-5.mit.edu (DMZ-MAILSEC-SCANNER-5.MIT.EDU [18.7.68.34]) by mx1.freebsd.org (Postfix) with ESMTP id 1CE2FE1B for ; Mon, 8 Apr 2013 03:22:45 +0000 (UTC) X-AuditID: 12074422-b7f5c6d000000545-76-516237ffe7b1 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP id CA.C2.01349.FF732615; Sun, 7 Apr 2013 23:22:39 -0400 (EDT) Received: from outgoing.mit.edu (OUTGOING-AUTH-1.MIT.EDU [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id r383McRf005861; Sun, 7 Apr 2013 23:22:39 -0400 Received: from multics.mit.edu (SYSTEM-LOW-SIPB.MIT.EDU [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r383MbLw008552 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sun, 7 Apr 2013 23:22:38 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id r383Manu027948; Sun, 7 Apr 2013 23:22:36 -0400 (EDT) Date: Sun, 7 Apr 2013 23:22:36 -0400 (EDT) From: Benjamin Kaduk To: Sebastian Feld Subject: Re: Multiple page size support on FreeBSD? In-Reply-To: Message-ID: References: User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrDIsWRmVeSWpSXmKPExsUixCmqrPvfPCnQ4PUjMYvtm/8xWnTdbmV1 YPKY8Wk+i8fOWXfZA5iiuGxSUnMyy1KL9O0SuDImN79iKjjOUvH+4GaWBsarzF2MnBwSAiYS N9csYIOwxSQu3FsPZHNxCAnsY5SY9v4yO4SzgVHi7MsjYB1CAgeZJB4c0oCw6yWuXj7ECmKz CGhJPD/xixHEZhNQkZj5ZiPYVBEBfYlHLxeygNjMAvISFzYfAqsRFjCW6Jj/Amwmp0CgxN/z bWBxXgEHiZ4Jn9gg5gdI3P+wAqxGVEBHYvX+KSwQNYISJ2c+gZppKXHuz3W2CYyCs5CkZiFJ LWBkWsUom5JbpZubmJlTnJqsW5ycmJeXWqRrqpebWaKXmlK6iREUqOwuSjsYfx5UOsQowMGo xMN74GtioBBrYllxZe4hRkkOJiVR3hyTpEAhvqT8lMqMxOKM+KLSnNTiQ4wSHMxKIryBykA5 3pTEyqrUonyYlDQHi5I477WUm/5CAumJJanZqakFqUUwWRkODiUJ3gNmQI2CRanpqRVpmTkl CGkmDk6Q4TxAw+eA1PAWFyTmFmemQ+RPMSpKifNmgCQEQBIZpXlwvbBE8opRHOgVYd7NIFU8 wCQE1/0KaDAT0GD+C/Egg0sSEVJSDYyVgvnxCn9iGY+x/9odV7qu0CVwrUadTMJRgQr9acdW /uVzObE/bPJ1B/spadtvPp544ve6v5s9N89Nm5jvUeglnLgktvZr94LfqhvfahxfO73guJWY p9XCL6/nfc7KNNvkynem4o9UdG/2rQ05l96skGl9eMvg+PxnaQqTg/dx3TN+5PBJt/mfEktx RqKhFnNRcSIAx+hm3P8CAAA= Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 03:22:46 -0000 On Sun, 7 Apr 2013, Sebastian Feld wrote: > Does FreeBSD support multiple page sizes in a single userland process > like Solaris 11 does? Superpage promotion happens automatically when consecutive data are accessed according to the proper heuristic. As promotion is dynamic, necessarily a mixture of page sizes will be in use. I'm not familiar with the Solaris 11 feature, so I can't say whether there is a direct parallel. You may be interested in this talk by Kirk McKusick at BSDCan 2011 on the topic: http://www.youtube.com/watch?v=0wIxny-n_Mg -Ben Kaduk From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 10:39:47 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E073A922 for ; Mon, 8 Apr 2013 10:39:47 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id 6A2411E3 for ; Mon, 8 Apr 2013 10:39:47 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5) with ESMTP id r38AdVJq006023; Mon, 8 Apr 2013 12:39:31 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5/Submit) with ESMTP id r38AdVb7006020; Mon, 8 Apr 2013 12:39:31 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Mon, 8 Apr 2013 12:39:31 +0200 (CEST) From: Wojciech Puchar To: Benjamin Kaduk Subject: Re: Multiple page size support on FreeBSD? In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Mon, 08 Apr 2013 12:39:31 +0200 (CEST) Cc: freebsd-hackers@freebsd.org, Sebastian Feld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 10:39:47 -0000 > Superpage promotion happens automatically when consecutive data are accessed > according to the proper heuristic. and in practice - unless there are only few processes, never really works. this is a result of my own tests. From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 13:29:57 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DF03CAB2 for ; Mon, 8 Apr 2013 13:29:57 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id A78CFE09 for ; Mon, 8 Apr 2013 13:29:57 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 383444AC57 for ; Mon, 8 Apr 2013 17:29:56 +0400 (MSK) Date: Mon, 8 Apr 2013 17:29:53 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <1654361502.20130408172953@serebryakov.spb.ru> To: freebsd-hackers@freebsd.org Subject: building world and kernel without ebuilding ("bootstrap"?) clang? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 13:29:57 -0000 Hello, Freebsd-hackers. I need to rebuild world and kernel often for experiments with NanoBSD, and I want to use clang. But clang bootstrap is drastically slow. And second clang build is not any faster. Whole "buildworld" with lots of parts, like gcc toolchain, games, examples, etc., switched off, takes about 90 (!) minutes in my case. And I don't need "in world" clang too, as it will not be installed on target device. My build host is very closely related to sources which I build -- both are amd64 CURRENT, build host is not as fresh as sources for NanoBSD, but clang is always the same (I rebuild host on every clang change). Is it possible to build NanoBSD faster? Use system compiler, and don't build bootstrap compiler at all? -- // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 20:28:06 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 31F553AC for ; Mon, 8 Apr 2013 20:28:06 +0000 (UTC) (envelope-from aamitr4@gmail.com) Received: from mail-la0-x243.google.com (mail-la0-x243.google.com [IPv6:2a00:1450:4010:c03::243]) by mx1.freebsd.org (Postfix) with ESMTP id B50CB984 for ; Mon, 8 Apr 2013 20:28:05 +0000 (UTC) Received: by mail-la0-f67.google.com with SMTP id fp12so789185lab.10 for ; Mon, 08 Apr 2013 13:28:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=R72kNwDqp18w3IeV6vzI8hRqH+ae6wSwzXd9TbPIOCs=; b=CAlXfUR3OeIQTkiAgCAJneDvnuHunJmHpA599nrNuFT1zFMQlkXhQIpCOwdBpQ34lE ammQyukWqWz/aESsjPn+qE2EPzLof4GUD2q8p7hLONl+oUgvE4LeioaEmyE8WOaJp0gg wplp7ESbIYDBG9Iw3NiTCI9Ew2yV1blDKVQMlkGOQWtW5BaiOHpVWqyDLRaYjrOga5Le 891qzIIsw14XpKSSQ6m8LQeaYUNsd3vAgZpkkqx5eKHMFUMLEc1/Ou7EK+TO+yzRwBUM LLZ5HXrdwmXx/Aopee24ZKNimBrT0uzzwH2sHIUEXSD17DjAwIcrS9YzOxtUy2uZs+WL LHtg== MIME-Version: 1.0 X-Received: by 10.112.199.230 with SMTP id jn6mr1909022lbc.131.1365452884587; Mon, 08 Apr 2013 13:28:04 -0700 (PDT) Received: by 10.152.5.42 with HTTP; Mon, 8 Apr 2013 13:28:04 -0700 (PDT) Date: Mon, 8 Apr 2013 20:28:04 +0000 Message-ID: Subject: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Amit Rawat To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 20:28:06 -0000 GSOC posted the list of selected organization for GSOC 2013 and I am highly happy that FreeBSD is among the selected organization. I am a third year student interested to work in the field of embedded system. I applied last year and the title of my project was " Kernel Size Reduction for Embedded System". The link to my last year proposal is " https://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/amitrawat10/1#c8001" But due to some flaws it doesn't get selected. I am looking to improve my proposal for this year and apply again. I explain some portion of my project pictorially on my blog " https://amit10rawat.wordpress.com/2013/02/26/kernel-size-reduction-for-embedded-system/ " I am looking for suggestion and new ideas by which I can reduce the size of kernel. Amit Rawat(amraw) From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 22:07:51 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 21D9DD08 for ; Mon, 8 Apr 2013 22:07:51 +0000 (UTC) (envelope-from deeptech71@gmail.com) Received: from mail-ea0-x234.google.com (mail-ea0-x234.google.com [IPv6:2a00:1450:4013:c01::234]) by mx1.freebsd.org (Postfix) with ESMTP id AFC3BEFE for ; Mon, 8 Apr 2013 22:07:50 +0000 (UTC) Received: by mail-ea0-f180.google.com with SMTP id d10so2375850eaj.25 for ; Mon, 08 Apr 2013 15:07:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=u8f7sSeVGxYs2KSuXm2DjYcCnUuN6Fn08qhv3rG4SYI=; b=oq3muQTnZ1g5uONEuHGGmaVNXbhxu7K4wtHSiJHrOgmekOM5DNko+fOjbHbsAlteAn tH8abDN5FjA1uLSy3uIIJz3CUcnzTcmdIOUH1Bbxy2K0BOOPK0gJNMt5J683CrGXFHca t55c7EfVdHLkE+udJ/fSbzNHew71ve3bGBfC1O2ZInB8WkqjpRjGBvhm1RSmbYlhL7f6 wYMYTpSWEBHHTinFGEtzezaoDtVU/BVf9FHaUrIg23biDR/IuqRLU7cicTAfIDAMqOOM uUxLROoi9HBxzjRs/y3f/HxZFJ4RrbLQ1tDssh63P3ZkVsFZbOx4zcoKRHUxOQEI/Uj+ MRAQ== X-Received: by 10.15.98.141 with SMTP id bj13mr24193895eeb.29.1365458869670; Mon, 08 Apr 2013 15:07:49 -0700 (PDT) Received: from [192.168.1.80] (1F2E5BE5.dsl.pool.telekom.hu. [31.46.91.229]) by mx.google.com with ESMTPS id a41sm281032eei.4.2013.04.08.15.07.47 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 08 Apr 2013 15:07:48 -0700 (PDT) Message-ID: <51634D82.9060901@gmail.com> Date: Tue, 09 Apr 2013 01:06:42 +0200 From: deeptech71 User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:18.0) Gecko/20100101 SeaMonkey/2.15 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 08 Apr 2013 22:15:16 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 22:07:51 -0000 Lev Serebryakov wrote: > Is it possible to build NanoBSD faster? Use system compiler, and don't build bootstrap compiler at all? There are some issues when building "updated" sources [1]. There was a thread about adding support for an "external" compiler [2], but that yielded a non-working solution [3]. 1. Add the following or similar to /etc/make.conf: CC=/full/path/to/clang CPP=/full/path/to/clang-cpp CXX=/full/path/to/clang++ Note: make sure clang-cpp or similar exists. 2. Add the following to /etc/src.conf: WITHOUT_GCC=1 WITHOUT_CLANG=1 Note: ``make delete-old'' will prompt you to remove the compilers. 3. In case of an "external", modern version of Clang, remove its header files that are already present in the system at /usr/include (eg., stdio.h). 4. Selectively apply to the source tree (all are required for my purposes): Index: usr.bin/xlint/llib/Makefile =================================================================== --- usr.bin/xlint/llib/Makefile (revision 249254) +++ usr.bin/xlint/llib/Makefile (working copy) @@ -9,9 +9,9 @@ CLEANFILES+= ${LIBS} llib-lposix.ln: llib-lposix - ${LINT} ${LINTFLAGS} -Cposix ${.ALLSRC} + CC=${CC:Q} ${LINT} ${LINTFLAGS} -Cposix ${.ALLSRC} llib-lstdc.ln: llib-lstdc - ${LINT} ${LINTFLAGS} -Cstdc ${.ALLSRC} + CC=${CC:Q} ${LINT} ${LINTFLAGS} -Cstdc ${.ALLSRC} .include Index: Makefile.inc1 =================================================================== --- Makefile.inc1 (revision 249254) +++ Makefile.inc1 (working copy) @@ -680,7 +680,7 @@ ITOOLS= [ awk cap_mkdb cat chflags chmod chown \ date echo egrep find grep id install ${_install-info} \ ln lockf make mkdir mtree ${_nmtree_itools} mv pwd_mkdb \ - rm sed sh sysctl test true uname wc ${_zoneinfo} + rm sed sh sysctl test true uname wc ${_zoneinfo} cp btxld dd ls # # distributeworld Index: sys/conf/kern.mk =================================================================== --- sys/conf/kern.mk (revision 249254) +++ sys/conf/kern.mk (working copy) @@ -5,7 +5,7 @@ # CWARNFLAGS?= -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes \ -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual \ - -Wundef -Wno-pointer-sign -fformat-extensions \ + -Wundef -Wno-pointer-sign -Wno-format \ -Wmissing-include-dirs -fdiagnostics-show-option \ ${CWARNEXTRA} # 5. If need be, use the following fake script as Clang (required for my purposes): #!/bin/sh /path/to/real/clang --sysroot=/usr/obj/tmp "$@" || /path/to/real/clang "$@" 6. If there are still build errors, yell that FreeBSD sux. And let these seemingly-retarded instructions be a warning to all active developers. (What a nice day for a flame war!) [1] http://lists.freebsd.org/pipermail/freebsd-current/2013-February/040160.html [2] http://lists.freebsd.org/pipermail/freebsd-arch/2013-February/014055.html [3] http://people.freebsd.org/~brooks/patches/xcc3.diff From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 22:44:26 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 503EA561 for ; Mon, 8 Apr 2013 22:44:26 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) by mx1.freebsd.org (Postfix) with ESMTP id 191B911B for ; Mon, 8 Apr 2013 22:44:26 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 9AE6335930C; Tue, 9 Apr 2013 00:44:23 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 6C8E62848C; Tue, 9 Apr 2013 00:44:23 +0200 (CEST) Date: Tue, 9 Apr 2013 00:44:23 +0200 From: Jilles Tjoelker To: Amit Rawat Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " Message-ID: <20130408224423.GA64696@stack.nl> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 22:44:26 -0000 On Mon, Apr 08, 2013 at 08:28:04PM +0000, Amit Rawat wrote: > GSOC posted the list of selected organization for GSOC 2013 and I am > highly happy that FreeBSD is among the selected organization. > I am a third year student interested to work in the field of embedded > system. I applied last year and the title of my project was " Kernel > Size Reduction for Embedded System". The link to my last year proposal > is > https://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/amitrawat10/1#c8001 > But due to some flaws it doesn't get selected. I am looking to improve my > proposal for this year and apply again. I explain some portion of my > project pictorially on my blog > https://amit10rawat.wordpress.com/2013/02/26/kernel-size-reduction-for-embedded-system/ > I am looking for suggestion and new ideas by which I can reduce the > size of kernel. It looks like the overlay idea could be implemented more simply by taking advantage of the VM system: make part of the kernel code pageable. Memory formerly occupied by rarely used kernel code can then be used by userland applications. You will need some sort of backing store where the kernel code can be read after booting; this is not normally available. However, almost no kernel code is safe in a situation where an instruction fetch may fault. Reading or writing the secondary storage can easily cause a deadlock. It causes the thread to sleep, which is not allowed while holding a mutex. It would help if you could wire down pieces which will need to be used in the near future from a place where a fault is safe, but this may also be very slow even if the code is already in memory. Some other ideas for kernel size reduction: * Find pieces of code that are required but seem big for what they do for you, and try to make them smaller. The proposal should list concrete parts. * Find variables and functions that are required only during kernel initialization, place them in a special section and add this section to the free memory pool after kernel initialization. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 8 23:10:05 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7FB42223 for ; Mon, 8 Apr 2013 23:10:05 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) by mx1.freebsd.org (Postfix) with ESMTP id 1BA2A275 for ; Mon, 8 Apr 2013 23:10:04 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id hn17so3093189wib.12 for ; Mon, 08 Apr 2013 16:10:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=P6UtKKEboLfWIT4YP6tVpIt9CJJwWVnA9aynpEa3qjI=; b=z0uxgrKLfVgV/K9Ezmc1BkLI8eqtFUp55xOFiMgnqifod5MpqU+izZGAJ0TbQSStcR 8BgICmtVR7wnb0RX5Y64Qj4Ku0g++gTLFnXPPJTrf1zZJ8zx9/h+18l+he0+ZH+Pafqf nx3Cs2QjXX6OegvKcscetGXkm0LJQpVercaooZcPwZGMoh6OPNZdO7+v+GJSr94ou8j9 uckWX0dKGSdkF3jGuWsr/Cr9FrLeN8LNXYWnGZFNZGf2PrIll5/yTaTZ60hFMeaeRZGA LFJRp+jOEOrLhn1jTZ5nrklTl7nrvdZsYSZg8suGmCVtE1TJrgrzPvfXzKCVJjpUpwrQ hdGg== MIME-Version: 1.0 X-Received: by 10.181.11.164 with SMTP id ej4mr15945995wid.29.1365462604176; Mon, 08 Apr 2013 16:10:04 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.121.136 with HTTP; Mon, 8 Apr 2013 16:10:04 -0700 (PDT) In-Reply-To: <20130408224423.GA64696@stack.nl> References: <20130408224423.GA64696@stack.nl> Date: Mon, 8 Apr 2013 16:10:04 -0700 X-Google-Sender-Auth: J9q5tX4yNGbvTvTrGO4oVySJxsM Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Adrian Chadd To: Jilles Tjoelker Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 23:10:05 -0000 Hi, Your idea is interesting, but it doesn't fix the underlying problem - there's just too much code. :( Adrian From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 00:19:10 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 018F4926 for ; Tue, 9 Apr 2013 00:19:09 +0000 (UTC) (envelope-from erichsfreebsdlist@alogt.com) Received: from alogt.com (alogt.com [69.36.191.58]) by mx1.freebsd.org (Postfix) with ESMTP id C3CCD5EB for ; Tue, 9 Apr 2013 00:19:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=alogt.com; s=default; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=5UsScINwOjb/R9WDjb7MD0rJOouaaEwArrhhePIkx8M=; b=IPKBP3Fr+Q/JqwYfRiXTAW9UPRvPNF9QiQPqvhTYhyR5Q6+4u5Pic4pQB7OSH2+TS6BX5pzFhGTkDvYbHcNdPBb2VmlaD+Lc2k3U5Z/pd3EqLmpxjhwYUSCXhcBGE1voqcsQ/bM8sJiRHfbueelmI2V5p2r1QtvXvCvfYQiCbPU=; Received: from [122.129.203.50] (port=25688 helo=X220.ovitrap.com) by sl-508-2.slc.westdc.net with esmtpsa (SSLv3:DHE-RSA-AES128-SHA:128) (Exim 4.80) (envelope-from ) id 1UPMHE-0039fG-60; Mon, 08 Apr 2013 18:19:08 -0600 Date: Tue, 9 Apr 2013 07:19:04 +0700 From: Erich Dollansky To: Amit Rawat Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " Message-ID: <20130409071904.5c4dc684@X220.ovitrap.com> In-Reply-To: References: X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - sl-508-2.slc.westdc.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - alogt.com X-Get-Message-Sender-Via: sl-508-2.slc.westdc.net: authenticated_id: erichsfreebsdlist@alogt.com X-Source: X-Source-Args: X-Source-Dir: Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 00:19:10 -0000 Hi, On Mon, 8 Apr 2013 20:28:04 +0000 Amit Rawat wrote: > GSOC posted the list of selected organization for GSOC 2013 and I am > highly happy that FreeBSD is among the selected organization. > > I am a third year student interested to work in the field of embedded > system. I applied last year and the title of my project was " Kernel > Size Reduction for Embedded System". The link to my last year > proposal is " > https://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/amitrawat10/1#c8001" > But due to some flaws it doesn't get selected. I am looking to > improve my proposal for this year and apply again. I explain some > portion of my project pictorially on my blog > > " > https://amit10rawat.wordpress.com/2013/02/26/kernel-size-reduction-for-embedded-system/ > " > > I am looking for suggestion and new ideas by which I can reduce the > size of kernel. > did you look at historic operating systems and how they did it? When I was a student, we simply loaded a module into memory and then wrote it to an external memory when not needed. It was a very basic swapping algorithm with fixed size. It did not differentiate between code and data etc. All calls where done through a wrapper which was always in memory. So, the module was loaded by the OS. The OS did not notice that we removed the code from memory. Only the wrapper knew of what we did. We selected functions/modules which have to be in memory and which could be on disk. There was a fixed number of memory segments we could use for loading external modules. The trick was to select the modules so that the system did not lock up. We did not use a tree structure what you might could. The advantage of this was that the actual code was not modified. Yes, I know that this is hard core. My problem would be that I do not know how much effort it would be to implement this in a modern operating system. Erich > Amit Rawat(amraw) > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 00:35:10 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 91258B53; Tue, 9 Apr 2013 00:35:10 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 80D4A666; Tue, 9 Apr 2013 00:35:10 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id AFFD51A3C19; Mon, 8 Apr 2013 17:35:04 -0700 (PDT) Message-ID: <5163622F.60604@mu.org> Date: Mon, 08 Apr 2013 17:34:55 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " References: <20130408224423.GA64696@stack.nl> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Amit Rawat , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 00:35:10 -0000 On 4/8/13 4:10 PM, Adrian Chadd wrote: > Hi, > > Your idea is interesting, but it doesn't fix the underlying problem - > there's just too much code. :( > If you were to API'ify some of the more basic things such as fget, fdrop, filedesc stuff you could potentially swap out the systems for simpler (albeit less efficient) algorithms, the cost there may be slow smp performance, or maybe not allowing threads? What we really need is someone to pin down those parts of code that smaller systems may not need and provide compromise when we remove them. Other ideas are simple like for instance removing certain syscalls (for example, more recent ones such as openat) and features such as unix descriptor passing. However, until a bunch of embedded folks come forward and state what they are really willing to sacrifice, then we won't really have anything to go on, and it will be guessing at what will work for a space that not all of us are familiar with. So I'm hoping some people can make the tough call to give direction here, otherwise nothing good will come of it. Has anyone actually done this? Or maybe compared against another embedded OS? -Alfred From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 01:15:57 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6EF3BFF5 for ; Tue, 9 Apr 2013 01:15:57 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: from mail-ee0-f47.google.com (mail-ee0-f47.google.com [74.125.83.47]) by mx1.freebsd.org (Postfix) with ESMTP id 0D5567E3 for ; Tue, 9 Apr 2013 01:15:55 +0000 (UTC) Received: by mail-ee0-f47.google.com with SMTP id t10so2589730eei.20 for ; Mon, 08 Apr 2013 18:15:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=sLA3lgDI63LcKW7SFgnRDi+5GisV9q5PyncaLsEq9BU=; b=OpZr9WNrwpb483Nx0OL2OuaS0OP7guC5kkkVIeu6L8fn24UYSAJw+AHy8VNyYDVYLf NxtEqILK3jA882hGcyz2yWcoGjt3IGfKVMG6UMvkTWjg3JIdyY05gtrXUmQQJAKq1iF5 7/g0UqUCAz36nMPT771ZPlAShscyUZjshLMyAwZuvqEpWzLlNH2aPEuZdD2Wu32qt6Af KBCVFeLVgP+ybA3iATnX6hokMx1Asp5LrTtTzNMnlC6wk3ANGlSwgjE5QNpIAqISRD+S 7DpycUCtwwYIp1BLMtmJWLjeV2l+hHDxYuqivqUexx+lQUYJZeplmA1CdJgU83aOzyR5 Welg== MIME-Version: 1.0 X-Received: by 10.14.87.199 with SMTP id y47mr41164098eee.17.1365470149248; Mon, 08 Apr 2013 18:15:49 -0700 (PDT) Received: by 10.223.61.17 with HTTP; Mon, 8 Apr 2013 18:15:49 -0700 (PDT) Date: Mon, 8 Apr 2013 18:15:49 -0700 Message-ID: Subject: copyinstr() From: Vijay Singh To: hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 01:15:57 -0000 Hi, I was looking for some help with copyinstr() on an amd64 platform. My from address happens to be in the kernel (stack). I am getting an EFAULT, and I am wondering how to fix that. Would using memory from malloc() make a difference? -vijay From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 01:42:57 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BCB953F0 for ; Tue, 9 Apr 2013 01:42:57 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) by mx1.freebsd.org (Postfix) with ESMTP id 5706F8A3 for ; Tue, 9 Apr 2013 01:42:57 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id c10so4472307wiw.4 for ; Mon, 08 Apr 2013 18:42:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=RSMK+j4R3s9DJ0vTYHFxXHtupMGXcKSiCCoRkedkW4U=; b=m9fA7e0S+E5W0GXPwE2CNevdvI4hZ3Y4vEFkRAvAkQhAyrgzjZZJDHzopyC9cU9FRN UwuOaoEGFOJLJvk8qb0en8DPOb21Z91r44DtAFoqcVHQYxOWY7hIW3MAqPhG1G68K0/L Xk7RFmb+3dOGz/TpdyiKUJmcJXIxkS1ndSD93UkZs2fSKh691Y5aiiqTB9DchcgPYjkd nmZX+TGhIuMQQn8lFzAak2glCemGk4XMY8NmnGwdafEJL9OpfsQRP3QIR4Q/aWBzv38x hNizATxEONADUOxto09osb9ZjLpqENqxA8wvCKnO1hCz66BavZy+w+S+5K+rtCms247K IfXg== MIME-Version: 1.0 X-Received: by 10.180.105.99 with SMTP id gl3mr16409408wib.22.1365471774404; Mon, 08 Apr 2013 18:42:54 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.121.136 with HTTP; Mon, 8 Apr 2013 18:42:54 -0700 (PDT) In-Reply-To: <5163622F.60604@mu.org> References: <20130408224423.GA64696@stack.nl> <5163622F.60604@mu.org> Date: Mon, 8 Apr 2013 18:42:54 -0700 X-Google-Sender-Auth: DAyJyJoocvyncX_j8oTJjBWbD6c Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Adrian Chadd To: Alfred Perlstein Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, Amit Rawat , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 01:42:57 -0000 Well, it's relatively easy to experience what it's like. Reboot your machine with 32mb. Try to do things like bring up network interfaces. Snark when stupid stuff occurs, like you can't allocate enough mbufs for the driver RX path _and_ run the ifconfig command to completion to bring said interface up. There's just a lot of code. You can start by cross-building one of the MIPS kernels targeting a small system (eg AP121) and look at the text/data sections of the resulting .o's. Group them together into subsystems and take a look. Now, as for what we can get away with - I'm still going through another round of review. Yes, there's likely a bunch of syscalls or syscall behaviours that we just don't need in the embedded world. Things like all the POSIX compatible fine grained locking? Likely don't need. But there's some reasonably big areas of bloat that we could easy hit right now. I've chopped out some of the more silly abuses in the past (posix acl code that only gets used by ZFS, always being compiled in? Sigh.) Eg: text data bss dec hex filename 59772 160 49184 109116 1aa3c kern_umtx.o That's a lot of both code and bss just for mutex handling, don't you think? Do we really need 59KiB of code and 48KiB of BSS just for mutex handling? text data bss dec hex filename 184 0 12160 12344 3038 sc_machdep.o .. 8 consoles? 12k of BSS? again, not much, but .. adrian@lucy:~/work/freebsd/svn/src/sys/cam]> cat /tmp/AP121-nodebug.txt | egrep 'ata' text data bss dec hex filename 11536 0 0 11536 2d10 ata_all.o 17624 1504 16 19144 4ac8 ata_da.o 6388 448 16 6852 1ac4 ata_pmp.o 18960 304 0 19264 4b40 ata_xpt.o .. 52 odd KiB tied up in CAM ATA transport, which we don't use unless the ATA code is compiled in. It's just sitting there, waiting for an ATA device to come along. lucy# cat /tmp/AP121-nodebug.txt | grep "vfs_" | grep -v devfs | sort -k3 4160 48 0 4208 1070 vfs_acl.o 4752 48 0 4800 12c0 vfs_export.o 5464 0 0 5464 1558 vfs_extattr.o 8128 288 0 8416 20e0 vfs_default.o 11020 160 0 11180 2bac vfs_cluster.o 7916 96 16 8028 1f5c vfs_lookup.o 19908 144 16 20068 4e64 vfs_vnops.o 34504 208 16 34728 87a8 vfs_syscalls.o 3068 64 32 3164 c5c vfs_hash.o 22700 208 32 22940 599c vfs_mount.o 1760 144 160 2064 810 vfs_init.o 14520 16 160 14696 3968 vfs_mountroot.o 13996 1568 176 15740 3d7c vfs_cache.o 64852 1680 256 66788 104e4 vfs_subr.o 52188 2000 304 54492 d4dc vfs_bio.o .. 260KiB just for VFS handling. etc, etc. I'd love to fix this, but I have to make a choice right now between porting to more of the Atheros wifi/soc platforms, or tackling this particular issue. I'd love for others to help out here. I'm sure that reducing code size in general is going be beneficial on the lower end platforms, even just in cache savings. Thanks, Adrian From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 02:29:03 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F25D6CD3 for ; Tue, 9 Apr 2013 02:29:03 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) by mx1.freebsd.org (Postfix) with ESMTP id C0802A5B for ; Tue, 9 Apr 2013 02:29:03 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id e14so7926281iej.2 for ; Mon, 08 Apr 2013 19:29:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dragondata.com; s=google; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer; bh=/dpaBv+Z7rR62XR/uaP3gj26Su6/Xxj+OhO/kehA3ho=; b=CJnxkK+Uf9xlhKF8UBWJqvqfxd7A8UQb/NfgH0/uV/ZDaHklBDiX+3JkhZcEmStX+W Dfbz/dnVeVmF0+MFXgRMTRqZUGm60naDnjvR6nn72eaw4kNGNWx7fvBXJPucwOxS3O3U s3gmCVL2ZL3W96o0q6gbPhadEpyfRpVD1n3hE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=/dpaBv+Z7rR62XR/uaP3gj26Su6/Xxj+OhO/kehA3ho=; b=E5v/ufpbfjBjO80vcscwhTr6fxO5Ev5P1VjfjoQ8wkuwtLl87jYATXIqlXobjKkAlK zWiof25hCasYnkfXiplJyhf7icwUtspjGWqUeTA4fA6/lMAm/iGOt+CRNHweJgNmjz3y h2NSJ9dW1ToTqNSBGI5BvZfOApUSGne+H8Q4oWNHqQGgV0wnqzCE28l25F4YgcS8NrYr tDWrX1gWbqsynh9xOosvg9xAF1w6WTvBpaw8BE3zh3AfosINTw5sSEgh5XhEIQyQcsMW 6XYNUE667J4JpFhCeXyhcts0Nt0pjQWA8ElX6hg1mcH62hiUD37z46YIeD+xrvp395Nl tQiQ== X-Received: by 10.50.135.105 with SMTP id pr9mr9184327igb.6.1365474542347; Mon, 08 Apr 2013 19:29:02 -0700 (PDT) Received: from vpn132.rw1.your.org (vpn132.rw1.your.org. [204.9.51.132]) by mx.google.com with ESMTPS id ih1sm20897127igc.3.2013.04.08.19.29.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 08 Apr 2013 19:29:01 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Kevin Day In-Reply-To: <5163622F.60604@mu.org> Date: Mon, 8 Apr 2013 21:28:58 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20130408224423.GA64696@stack.nl> <5163622F.60604@mu.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQlZFOd2QDNsmzZHHQrB+RVKIJKQ6RyhZ2+tIjdGS5UmxlmcE34GxcRDsGvQlqgiKJ+8HiNK Cc: freebsd-hackers@freebsd.org, Adrian Chadd , Amit Rawat , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 02:29:04 -0000 On Apr 8, 2013, at 7:34 PM, Alfred Perlstein wrote: > However, until a bunch of embedded folks come forward and state what = they are really willing to sacrifice, then we won't really have anything = to go on, and it will be guessing at what will work for a space that not = all of us are familiar with. >=20 > So I'm hoping some people can make the tough call to give direction = here, otherwise nothing good will come of it. >=20 > Has anyone actually done this? Or maybe compared against another = embedded OS? Ages ago we had to make things work in 16 or 32MB of total system memory = on i386.=20 For the most part, disabling every compiled-in option/driver we didn't = need was 90% of the effort. Which options/drivers is going to be totally = application dependent, so that really can't be done for you. As for the rest, there isn't any large low hanging fruit that can get = culled from the kernel easily. The base kernel isn't modular enough to = trim out individual syscalls or anything, and doing so wouldn't have = made a huge dent. There are a lot of ways FreeBSD could be more embedded friendly (being = able turn on/off parts of userland depending on licenses is a huge one), = but producing a trimmed kernel isn't something I'd rank very highly. If = building a kernel with everything modularized as possible isn't small = enough, FreeBSD probably isn't going to work for you for other reasons. From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 05:05:35 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 26E379B for ; Tue, 9 Apr 2013 05:05:35 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x236.google.com (mail-we0-x236.google.com [IPv6:2a00:1450:400c:c03::236]) by mx1.freebsd.org (Postfix) with ESMTP id B39FD9B for ; Tue, 9 Apr 2013 05:05:34 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id k14so5135097wer.13 for ; Mon, 08 Apr 2013 22:05:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=NvkdZtWLPOBfKiKxn+w6o3rrZtWDdyRrtFgVDofKsQ8=; b=Qkd9FTjhEwC8/kKtlfYLBSyoLqR+A5sURmYc/I07WaEacpuN4gIUF3lQD79JK+qbX7 unD3Kv1k5lcSwwcrA9D3wpVmSbhoVcDGP4OAP6QVimw4yWO4MLtkLBPpDOIgGoXCdxuZ MsRIXyD7Q246sBy7Jd/a7JZXIhlpYw8XOEjiEi5kBXjJQpYtWdQYLK3gZPmKfs4y3kcF VSwK5sYQ0bqK+M5sppARhyRCtXybdXOyHEfnU+DouUDT/JXwJpZEjFv3Y+9VAF4/Afo0 2vbb0ts6Z+poj8SZbSbOcqYTX5vefb3B/EN8Ei8oigiSXTXU55FogIRmdineZfpogR9U BX0Q== MIME-Version: 1.0 X-Received: by 10.194.143.50 with SMTP id sb18mr5886412wjb.44.1365483933844; Mon, 08 Apr 2013 22:05:33 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.121.136 with HTTP; Mon, 8 Apr 2013 22:05:33 -0700 (PDT) In-Reply-To: References: <20130408224423.GA64696@stack.nl> <5163622F.60604@mu.org> Date: Mon, 8 Apr 2013 22:05:33 -0700 X-Google-Sender-Auth: G5ZJvtF3QDZYawU3C3_PEnuI1Nw Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Adrian Chadd To: Kevin Day Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Alfred Perlstein , Amit Rawat , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 05:05:35 -0000 On 8 April 2013 19:28, Kevin Day wrote: > Ages ago we had to make things work in 16 or 32MB of total system memory = on i386. > > For the most part, disabling every compiled-in option/driver we didn't ne= ed was 90% of the effort. Which options/drivers is going to be totally appl= ication dependent, so that really can't be done for you. > > As for the rest, there isn't any large low hanging fruit that can get cul= led from the kernel easily. The base kernel isn't modular enough to trim ou= t individual syscalls or anything, and doing so wouldn't have made a huge d= ent. > > There are a lot of ways FreeBSD could be more embedded friendly (being ab= le turn on/off parts of userland depending on licenses is a huge one), but = producing a trimmed kernel isn't something I'd rank very highly. If buildin= g a kernel with everything modularized as possible isn't small enough, Free= BSD probably isn't going to work for you for other reasons. The MIPS kernels I'm producing are pretty bare. There's not a lot of options to disable at this point.. :( Adrian From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 06:19:00 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6865A943 for ; Tue, 9 Apr 2013 06:19:00 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) by mx1.freebsd.org (Postfix) with ESMTP id 568232BC for ; Tue, 9 Apr 2013 06:19:00 +0000 (UTC) Received: from delphij-macbook.local (unknown [IPv6:2001:470:83bf:0:10ba:ce65:5048:454a]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by anubis.delphij.net (Postfix) with ESMTPSA id DE4B666E8; Mon, 8 Apr 2013 23:18:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=delphij.net; s=anubis; t=1365488340; bh=tHiuNGtOom7UgyFiuKUPJod2oCVeoKTwF9BsGnPeCIo=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=IBFekZizB6x0QpRn1IxhxC0YLK5m2YKvJ5fi9ok7VBMmEPBL8JFhGXCzfHYfMGTan eJGUmJBoAZKHupLGwMOsxI4qs+z2+Z2I8fPNTHAQ5sPOfaFwfex85F8n2Ag3RwuU9u yD/5bKop6V80aKtJHKhHj6YduiId7GLJYQhxhY7c= Message-ID: <5163B2CC.8080004@delphij.net> Date: Mon, 08 Apr 2013 23:18:52 -0700 From: Xin Li Organization: The FreeBSD Project MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: copyinstr() References: In-Reply-To: X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: d@delphij.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 06:19:00 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 4/8/13 6:15 PM, Vijay Singh wrote: > Hi, I was looking for some help with copyinstr() on an amd64 > platform. > > My from address happens to be in the kernel (stack). I am getting > an EFAULT, and I am wondering how to fix that. Since you are doing a copy*in*str, I think "from" address would be the uaddr? In that case, it should be a userland address... > Would using memory from malloc() make a difference? Maybe not... EFAULT means the address is not valid. Cheers, -----BEGIN PGP SIGNATURE----- iQEcBAEBCAAGBQJRY7LMAAoJEG80Jeu8UPuzXRcH+gOoKAIelHbF2tp78HHgp0+b zuRF+Cj8mBOxJEyS3zpKEVKdq8HgYEJRq39Tkp2c60xwYOUcU0HRU6ixTJ0whZFo Gbx7tBbp4+89TtkPVm/u/JlqeToQNuQSFJBxNGi1qOjPpJQfuClPQ9EI4N4LDesh g8B7D5N4YoIUhLkg2FEix7c3XrzTeDRCfXYsfHna4f3VMrlNze0R61TpRqh6qx8/ eJDBA25m6+Y6129qo8wdkOZWLT6ZSIPrc6WgQuCP3jTYJemhiM1RdTFLqM87PNBd EGuL1+FGgDUzhieJoOx/FhD01Cypc7/Qs6pxaF1BGxpCaL0SPFyBJ+WBnv9A7DA= =iMeL -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 10:03:58 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 27562CF3 for ; Tue, 9 Apr 2013 10:03:58 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id E2D3BE69 for ; Tue, 9 Apr 2013 10:03:57 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 773104AC57; Tue, 9 Apr 2013 14:03:48 +0400 (MSK) Date: Tue, 9 Apr 2013 14:03:45 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <48101083.20130409140345@serebryakov.spb.ru> To: deeptech71 Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? In-Reply-To: <51634D82.9060901@gmail.com> References: <51634D82.9060901@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 10:03:58 -0000 Hello, deeptech71. You wrote 9 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2013 =D0=B3., 3:06:42: d> 1. Add the following or similar to /etc/make.conf: d> CC=3D/full/path/to/clang d> CPP=3D/full/path/to/clang-cpp d> CXX=3D/full/path/to/clang++ d> Note: make sure clang-cpp or similar exists. d> 2. Add the following to /etc/src.conf: d> WITHOUT_GCC=3D1 d> WITHOUT_CLANG=3D1 This is NanoBSD build, so it doesn't use system /etc/make.conf and /etc/src.conf. I'll try to add it into NanoBSD configs. d> Note: ``make delete-old'' will prompt you to remove the compilers. NanoBSD builds world into empty object dir every time. What should I `delete-old'? d> 3. In case of an "external", modern version of Clang, remove its d> header files that are already present in the system at /usr/include (eg.= , stdio.h). Is "system" version (system is snapshot from Mar 30) is "external" and "m= odern"? d> 4. Selectively apply to the source tree (all are required for my purpose= s): I'll give it a try. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 15:43:32 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F20A6440; Tue, 9 Apr 2013 15:43:31 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id E177A2A2; Tue, 9 Apr 2013 15:43:31 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 85A061A3C44; Tue, 9 Apr 2013 08:43:25 -0700 (PDT) Message-ID: <51643712.6010609@mu.org> Date: Tue, 09 Apr 2013 08:43:14 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " References: <20130408224423.GA64696@stack.nl> <5163622F.60604@mu.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Amit Rawat , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 15:43:32 -0000 On 4/8/13 6:42 PM, Adrian Chadd wrote: > Well, it's relatively easy to experience what it's like. No it's not. We all have jobs that demand different things from us. Taking the time to guess at the problem, only to be told "you're doing it wrong" by someone actually in the position to build the list of requirements is not only a big honking waste of time but not fun nor interesting. Either gather some acceptable feature/performance regressions together that "small" can live with or stop evangelizing. Looking at .o files and guessing what to trim isn't going to work. It sounds like what you want is some magic where you get all the features and your small image while not having to compromise on features/speed. Cool, maybe someone will invent something amazing that gives you less from more, but until then it makes sense to actually be pragmatic and put together a list of things to trim based on sacrifice, not just "because they are big". -Alfred > > Reboot your machine with 32mb. Try to do things like bring up network > interfaces. Snark when stupid stuff occurs, like you can't allocate > enough mbufs for the driver RX path _and_ run the ifconfig command to > completion to bring said interface up. > > There's just a lot of code. You can start by cross-building one of the > MIPS kernels targeting a small system (eg AP121) and look at the > text/data sections of the resulting .o's. Group them together into > subsystems and take a look. > > Now, as for what we can get away with - I'm still going through > another round of review. Yes, there's likely a bunch of syscalls or > syscall behaviours that we just don't need in the embedded world. > Things like all the POSIX compatible fine grained locking? Likely > don't need. But there's some reasonably big areas of bloat that we > could easy hit right now. I've chopped out some of the more silly > abuses in the past (posix acl code that only gets used by ZFS, always > being compiled in? Sigh.) > > Eg: > > text data bss dec hex filename > 59772 160 49184 109116 1aa3c kern_umtx.o > > That's a lot of both code and bss just for mutex handling, don't you > think? Do we really need 59KiB of code and 48KiB of BSS just for mutex > handling? > > text data bss dec hex filename > 184 0 12160 12344 3038 sc_machdep.o > > .. 8 consoles? 12k of BSS? again, not much, but .. > > adrian@lucy:~/work/freebsd/svn/src/sys/cam]> cat > /tmp/AP121-nodebug.txt | egrep 'ata' > text data bss dec hex filename > 11536 0 0 11536 2d10 ata_all.o > 17624 1504 16 19144 4ac8 ata_da.o > 6388 448 16 6852 1ac4 ata_pmp.o > 18960 304 0 19264 4b40 ata_xpt.o > > .. 52 odd KiB tied up in CAM ATA transport, which we don't use unless > the ATA code is compiled in. It's just sitting there, waiting for an > ATA device to come along. > > lucy# cat /tmp/AP121-nodebug.txt | grep "vfs_" | grep -v devfs | sort -k3 > 4160 48 0 4208 1070 vfs_acl.o > 4752 48 0 4800 12c0 vfs_export.o > 5464 0 0 5464 1558 vfs_extattr.o > 8128 288 0 8416 20e0 vfs_default.o > 11020 160 0 11180 2bac vfs_cluster.o > 7916 96 16 8028 1f5c vfs_lookup.o > 19908 144 16 20068 4e64 vfs_vnops.o > 34504 208 16 34728 87a8 vfs_syscalls.o > 3068 64 32 3164 c5c vfs_hash.o > 22700 208 32 22940 599c vfs_mount.o > 1760 144 160 2064 810 vfs_init.o > 14520 16 160 14696 3968 vfs_mountroot.o > 13996 1568 176 15740 3d7c vfs_cache.o > 64852 1680 256 66788 104e4 vfs_subr.o > 52188 2000 304 54492 d4dc vfs_bio.o > > .. 260KiB just for VFS handling. > > etc, etc. > > I'd love to fix this, but I have to make a choice right now between > porting to more of the Atheros wifi/soc platforms, or tackling this > particular issue. I'd love for others to help out here. I'm sure that > reducing code size in general is going be beneficial on the lower end > platforms, even just in cache savings. > > Thanks, > > > Adrian > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 17:15:59 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BF1AACF2 for ; Tue, 9 Apr 2013 17:15:59 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-ob0-x22f.google.com (mail-ob0-x22f.google.com [IPv6:2607:f8b0:4003:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 91C7FA6C for ; Tue, 9 Apr 2013 17:15:59 +0000 (UTC) Received: by mail-ob0-f175.google.com with SMTP id va7so7101202obc.6 for ; Tue, 09 Apr 2013 10:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=u4le+lY+D2TY/A1vx5OmjSyfQJ1RWVOpzuuYIwQaSoE=; b=dOLU7PzMxdw1iV9W0COFWXa/szSMimaE1sK3/+L/mPnw6/YKDV4r5CuwbFJriEuJcN D45f0qwxSQlGiKyNiAseAA28VMRuxCyS8fkOBIc223ntDUY8FDfaAs7hT1vfsJXN7/45 mOlmx8K0RFN9VAKDdQCuead5mHcovn0YEP9jgBfoBK+Vl+Ylx/l8ylPnre2lCD0dAO3d eXLSjQQ6wycFBpwG+OYj3zpohVDVJovqLgeDO73OQYo1EWcQ2k3ijEqc0MdIaTqpHRV7 d+Aq/YauiLIsEj2zvvSWEVPGfpYUixm26O4ZzIftNbjB8/zcEG3/tbwevI0RchE4i+d3 Gp7Q== MIME-Version: 1.0 X-Received: by 10.60.132.237 with SMTP id ox13mr1280151oeb.33.1365527759166; Tue, 09 Apr 2013 10:15:59 -0700 (PDT) Received: by 10.182.161.100 with HTTP; Tue, 9 Apr 2013 10:15:59 -0700 (PDT) In-Reply-To: References: Date: Tue, 9 Apr 2013 19:15:59 +0200 Message-ID: Subject: Re: copyinstr() From: Oliver Pinter To: Vijay Singh Content-Type: text/plain; charset=ISO-8859-1 Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 17:15:59 -0000 On 4/9/13, Vijay Singh wrote: > Hi, I was looking for some help with copyinstr() on an amd64 platform. > > My from address happens to be in the kernel (stack). I am getting an > EFAULT, and I am wondering how to fix that. > > Would using memory from malloc() make a difference? The copyinstr check the address before do anything. amd64/support.S: /* * copyinstr(from, to, maxlen, int *lencopied) - MP SAFE * %rdi, %rsi, %rdx, %rcx * * copy a string from from to to, stop when a 0 character is reached. * return ENAMETOOLONG if string is longer than maxlen, and * EFAULT on protection violations. If lencopied is non-zero, * return the actual length in *lencopied. */ ENTRY(copyinstr) movq %rdx,%r8 /* %r8 = maxlen */ movq %rcx,%r9 /* %r9 = *len */ xchgq %rdi,%rsi /* %rdi = from, %rsi = to */ movq PCPU(CURPCB),%rcx movq $cpystrflt,PCB_ONFAULT(%rcx) movq $VM_MAXUSER_ADDRESS,%rax /* make sure 'from' is within bounds */ subq %rsi,%rax jbe cpystrfl [...] cpystrflt: movq $EFAULT,%rax [...] Try copyout() instead of copyinstr(), as there in amd64 are no copyoutstr(). > > -vijay > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 17:24:58 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 886C5F0B for ; Tue, 9 Apr 2013 17:24:58 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id 4DB11B09 for ; Tue, 9 Apr 2013 17:24:58 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id D88964AC58; Tue, 9 Apr 2013 21:24:46 +0400 (MSK) Date: Tue, 9 Apr 2013 21:24:43 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <810652246.20130409212443@serebryakov.spb.ru> To: deeptech71 Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? In-Reply-To: <51634D82.9060901@gmail.com> References: <51634D82.9060901@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 17:24:58 -0000 Hello, deeptech71. You wrote 9 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2013 =D0=B3., 3:06:42: d> There are some issues when building "updated" sources [1]. There d> was a thread about adding support for an "external" compiler [2], d> but that yielded a non-working solution [3]. Ok, World build time was reduced from 1:27 to just 0:27. building clang and binutils twice takes one hour (!!!!!)? or 2/3 of buildworld time!. But, of course, headers should be equivalent to ones in sources. Also, disabled binutils lead to inability to install worils (install -s doesn't work). Returning binutils to build (to allow `install -s' works) yelds only 1 or 2 minutes, Ok. It is very sad, that external compiler cannot be used with "fresh" sources (when headers are different). We need some solution for this, for sure! When compilers are identical (same clang revision), but sources has updated headers. It does proper trick with libraries, but not with headers! Also, it should be possible to use system strip in such case. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 17:36:50 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2D6115FA for ; Tue, 9 Apr 2013 17:36:50 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id AD960CB7 for ; Tue, 9 Apr 2013 17:36:49 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5) with ESMTP id r39HaXbo013351; Tue, 9 Apr 2013 19:36:33 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5/Submit) with ESMTP id r39HaX0n013348; Tue, 9 Apr 2013 19:36:33 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Tue, 9 Apr 2013 19:36:33 +0200 (CEST) From: Wojciech Puchar To: Amit Rawat Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Tue, 09 Apr 2013 19:36:33 +0200 (CEST) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 17:36:50 -0000 > happy that FreeBSD is among the selected organization. > > I am a third year student interested to work in the field of embedded > system. I applied last year and the title of my project was " Kernel Size why only in embedded system. smaller programs are always good :) And yes FreeBSD kernel is huge. doesn't really matter with 1GB or more RAM but yes - it is huge even relative to linux. From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 17:41:39 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5EBA3792 for ; Tue, 9 Apr 2013 17:41:39 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 4F522D18 for ; Tue, 9 Apr 2013 17:41:39 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 15A601A3C1C; Tue, 9 Apr 2013 10:41:38 -0700 (PDT) Message-ID: <516452C7.7040607@mu.org> Date: Tue, 09 Apr 2013 10:41:27 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Wojciech Puchar Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 17:41:39 -0000 On 4/9/13 10:36 AM, Wojciech Puchar wrote: >> happy that FreeBSD is among the selected organization. >> >> I am a third year student interested to work in the field of embedded >> system. I applied last year and the title of my project was " Kernel >> Size > why only in embedded system. smaller programs are always good :) > > And yes FreeBSD kernel is huge. doesn't really matter with 1GB or more > RAM but yes - it is huge even relative to linux. Ah, any insight as to why? -Alfred From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 17:53:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CD489BA5 for ; Tue, 9 Apr 2013 17:53:49 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id 57167DFA for ; Tue, 9 Apr 2013 17:53:49 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5) with ESMTP id r39Hrd5r013443; Tue, 9 Apr 2013 19:53:39 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5/Submit) with ESMTP id r39HrdCD013440; Tue, 9 Apr 2013 19:53:39 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Tue, 9 Apr 2013 19:53:39 +0200 (CEST) From: Wojciech Puchar To: Alfred Perlstein Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " In-Reply-To: <516452C7.7040607@mu.org> Message-ID: References: <516452C7.7040607@mu.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Tue, 09 Apr 2013 19:53:39 +0200 (CEST) Cc: freebsd-hackers@freebsd.org, Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 17:53:49 -0000 >> And yes FreeBSD kernel is huge. doesn't really matter with 1GB or more RAM >> but yes - it is huge even relative to linux. > > Ah, any insight as to why? my custom compiled kernel: -r-xr-xr-x 1 root wheel 8791402 6 kwi 22:08 /boot//kernel/kernel only with features i need. linux is AFAIK like 3-4MB uncompressed. From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 17:59:38 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 87C02E95 for ; Tue, 9 Apr 2013 17:59:38 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) by mx1.freebsd.org (Postfix) with ESMTP id 229E1E4D for ; Tue, 9 Apr 2013 17:59:37 +0000 (UTC) Received: by mail-wi0-f177.google.com with SMTP id hm14so3964114wib.16 for ; Tue, 09 Apr 2013 10:59:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=nacq5hWbG+F5gAVs21AC4KFc7a/U5gHc28/KRnOO0Ts=; b=eoypYT9+uZAiA+KY1pI6TykDI9XLj+8OI/OEGA59UXndR/FxzsR+/sWMxR/MYAnV3e PUHAA5FjQONry8Ez4tdMrxBa/sGgTnFLmGGx1KT7G0NwkNozSGSSMBreQPbdQFTRygHj mpUvTWZmmgrVloOj69IcWtStffj9ncb/ZSSDFfahZyUReIQRfvIKeHZzzcHELx3El36B hk+Kjl6O1f4Aqi70Qa2Vq5Br0wfzBExgpnYUMXTBPw2garN+HTV40caewoR9ABsj856K jqJcEnKG7htTJHJj8Jixx5Q9yAYu4IZ41UJkEdM4B5iz3h3Ar9VgnjPNi49gYz5lJn+J cBlA== MIME-Version: 1.0 X-Received: by 10.180.73.6 with SMTP id h6mr21348325wiv.27.1365530377258; Tue, 09 Apr 2013 10:59:37 -0700 (PDT) Received: by 10.216.139.72 with HTTP; Tue, 9 Apr 2013 10:59:37 -0700 (PDT) In-Reply-To: References: <516452C7.7040607@mu.org> Date: Tue, 9 Apr 2013 20:59:37 +0300 Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Kimmo Paasiala To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 Cc: freebsd-hackers@freebsd.org, Alfred Perlstein , Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 17:59:38 -0000 On Tue, Apr 9, 2013 at 8:53 PM, Wojciech Puchar wrote: >>> And yes FreeBSD kernel is huge. doesn't really matter with 1GB or more >>> RAM but yes - it is huge even relative to linux. >> >> >> Ah, any insight as to why? > > my custom compiled kernel: > > -r-xr-xr-x 1 root wheel 8791402 6 kwi 22:08 /boot//kernel/kernel > > only with features i need. linux is AFAIK like 3-4MB uncompressed. > Your comparison is far from accurate, include the memory taken by loaded kernel modules on both systems and then you might get some proper numbers. -Kimmo From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 18:06:03 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D74361D9 for ; Tue, 9 Apr 2013 18:06:03 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qa0-f43.google.com (mail-qa0-f43.google.com [209.85.216.43]) by mx1.freebsd.org (Postfix) with ESMTP id 9C23DEB5 for ; Tue, 9 Apr 2013 18:06:03 +0000 (UTC) Received: by mail-qa0-f43.google.com with SMTP id i13so1524961qae.16 for ; Tue, 09 Apr 2013 11:05:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=7X6ltFf0gFYysSugsF5Iv/EDeNV3pJK7/IKGUxMWM+U=; b=oCJP2JP0gsXqKFp0+senxaBVBiFszAp6PT+JV8C11f69Ye674+cpofq4RMg+AS1v8b fQjr09YM1kMf24GOxsFTZkklIH2DFsDlzerRhOlt8WCBYagBDm73eJr8uqNnRIR+jXop dwuzRyADnoVSlyQoeUHyLgHsPqnb/j7/iYB7gOs/XY8loN46cYxtiO/sZLzkvemBLx+3 Vaoz97YN+/vT4FmwGYCAUXdJp1eiZenTYB7jNUBQNaHQKTQqqVSTyDLicRuLeRGu8Y5x i2Go9/LvpJKrLyVoD44+aqa6L3SpxnN3+LaLT0ymh02TJfHENVzdzHAHL2rzuGAOghvG HA5w== MIME-Version: 1.0 X-Received: by 10.224.23.135 with SMTP id r7mr11050661qab.80.1365530756901; Tue, 09 Apr 2013 11:05:56 -0700 (PDT) Received: by 10.49.51.9 with HTTP; Tue, 9 Apr 2013 11:05:56 -0700 (PDT) In-Reply-To: References: <516452C7.7040607@mu.org> Date: Tue, 9 Apr 2013 11:05:56 -0700 Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Freddie Cash To: Wojciech Puchar Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Hackers , Alfred Perlstein , Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 18:06:03 -0000 You have to look at the in-memory sizes, not the on-disk sizes. Linux kernels are very barebones when it comes to what is compiled directly into the kernel image on disk. Everything else is loaded from modules at boot time. Especially if using distro-provided kernels. They even use ram disks / initrds to get around the "can't boot without drivers for Y, but Y is a module and not loaded at boot", adding extra memory pressure that's not shown in the on-disk size of the kernel image file. FreeBSD kernels tend to be the opposite, with everything compiled directly into the kernel image on-disk, and very little actually being loaded via modules. At least GENERIC, anyway. You would need to manually compile kernels with the same sets of drivers on each system, in order to do a proper comparison of on-disk sizes. Or, look at in-memory stats for the two, once the systems are booted, all modules are loaded, and the system is ready for use. Just comparing ls output of default FreeBSD/Linux installs isn't useful in any way. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 18:20:10 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 35B986E7 for ; Tue, 9 Apr 2013 18:20:10 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id ED56DF7F for ; Tue, 9 Apr 2013 18:20:09 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 32CF24AC58; Tue, 9 Apr 2013 22:20:08 +0400 (MSK) Date: Tue, 9 Apr 2013 22:20:03 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <154649784.20130409222003@serebryakov.spb.ru> To: Kimmo Paasiala Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " In-Reply-To: References: <516452C7.7040607@mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Wojciech Puchar , Alfred Perlstein , Amit Rawat , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 18:20:10 -0000 Hello, Kimmo. You wrote 9 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2013 =D0=B3., 21:59:37: KP> Your comparison is far from accurate, include the memory taken by KP> loaded kernel modules on both systems and then you might get some KP> proper numbers. Linux is known to _work_ on SOHO MIPS boxes, with 4MiB of flash and 16MiB of RAM. You could say about ``loaded kernel modules,'' but when whole firmware, with all needed utils, like PPPoE client, Web-based UI, DHCP server, etc, is only 3.5MiB (Ok, it is compressed, but anyway it should work on 16MiB of RAM), it looks like functional Linux kernel could be about 1MiB without any modules. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 18:47:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EB1C2F56 for ; Tue, 9 Apr 2013 18:47:30 +0000 (UTC) (envelope-from etnapierala@gmail.com) Received: from mail-ee0-f46.google.com (mail-ee0-f46.google.com [74.125.83.46]) by mx1.freebsd.org (Postfix) with ESMTP id 7CCF922B for ; Tue, 9 Apr 2013 18:47:30 +0000 (UTC) Received: by mail-ee0-f46.google.com with SMTP id d49so2071762eek.5 for ; Tue, 09 Apr 2013 11:47:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=x1nAEwftGcUxzQZNYzVwjYh1kxm82G5lF1n7/jHp1Pc=; b=hPsx5We8AXRN59G0/+DS/3k1RXWqMjpkxOnkIaRxENUNKAW7Chvd5TNBdPvuMSlZb/ DOKD5J9788yhOUXLwcfZdOL7ZO3HWeamA7BVDPdDsT/kci0+U3r+CTXcaXQcDKsQjV/v B2QRRc/FMzN32Db7LNPYK0uhjuby6lvc5NHdUx3YH+zu2iBWfXmn/s+1Bl7yMJldEGXH EK9yRFwe179kCJW9hJZ06HYbg520/Ygk2n+5QhnYK4h76qxqHhEf0ZfLC/zxSE8kVwiq TrqlVSOgh4mCi2AO1EFcNpLOdxK0x8hXCoHG22T8OhMVTVf/TmVeF2DtW0sWEGL6zYWx 4J6w== X-Received: by 10.14.107.69 with SMTP id n45mr40864720eeg.23.1365533249456; Tue, 09 Apr 2013 11:47:29 -0700 (PDT) Received: from [192.168.1.104] (45.81.datacomsa.pl. [195.34.81.45]) by mx.google.com with ESMTPS id a41sm5358699eei.4.2013.04.09.11.47.27 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 09 Apr 2013 11:47:28 -0700 (PDT) Sender: =?UTF-8?Q?Edward_Tomasz_Napiera=C5=82a?= Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: =?iso-8859-2?Q?Edward_Tomasz_Napiera=B3a?= In-Reply-To: Date: Tue, 9 Apr 2013 20:47:26 +0200 Content-Transfer-Encoding: 7bit Message-Id: <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> References: <516452C7.7040607@mu.org> To: Freddie Cash X-Mailer: Apple Mail (2.1283) Cc: Wojciech Puchar , Alfred Perlstein , Amit Rawat , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 18:47:31 -0000 In order to optimize - in this case for size - we need a way to measure what should we focus on, and it looks like we don't have it yet. Would it be possible to write a tool - e.g. by instrumenting LLVM - that would make it possible to calculate, for every function in the call graph, the amount of code in that function and everything it "pulls in", i.e. all the code paths that it might call. When we have that, clustering the graph should give us some idea what to focus on. Or perhaps such a tool already exists? -- If you cut off my head, what would I say? Me and my head, or me and my body? From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 20:21:28 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C130EE28; Tue, 9 Apr 2013 20:21:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-we0-x233.google.com (mail-we0-x233.google.com [IPv6:2a00:1450:400c:c03::233]) by mx1.freebsd.org (Postfix) with ESMTP id 30478ABE; Tue, 9 Apr 2013 20:21:28 +0000 (UTC) Received: by mail-we0-f179.google.com with SMTP id p43so5556859wea.38 for ; Tue, 09 Apr 2013 13:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ZqRuGYxpQggRxy02yYe7egzcr36n/CoeC4xLpNLNB80=; b=F7g/rt57lyQfwK3y1vrzdmFBX4X9ieWgy4si7caTpoavQRK5XBFNWxc76vTgUgCkTl cPt0aKsn9iWtSqKFWC6f4dt78XhnjvKK+sBOLD/6aemTjP0M9MCTwEADBlj/KN+3V5tm IRkzIu83e11BO/xHP1obT9K+PAs4Kp3GAhO6d3SPS0DgRZd/pVcDtku4j9SJfPghwD49 daVZXiYw95d5fBK/4fyK8ToqLjJFntlgHmueQs8Gpn3SJotuoEZCTfbfZgLu3JP7ptzA smU1xrIsWu88EtX2lslFOXig/riuFVz/Iw/QhpV/248DEGvSFnXUdWwIlloJOlOkPQ5/ xS0Q== MIME-Version: 1.0 X-Received: by 10.194.88.138 with SMTP id bg10mr41317120wjb.13.1365538887220; Tue, 09 Apr 2013 13:21:27 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.121.136 with HTTP; Tue, 9 Apr 2013 13:21:27 -0700 (PDT) In-Reply-To: <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> References: <516452C7.7040607@mu.org> <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> Date: Tue, 9 Apr 2013 13:21:27 -0700 X-Google-Sender-Auth: nWm8-7aygPy7Hb3A0WN8dEvyM_s Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Adrian Chadd To: =?ISO-8859-2?Q?Edward_Tomasz_Napiera=B3a?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Wojciech Puchar , Alfred Perlstein , FreeBSD Hackers , Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 20:21:28 -0000 On 9 April 2013 11:47, Edward Tomasz Napiera=C5=82a wro= te: > In order to optimize - in this case for size - we need a way to measure > what should we focus on, and it looks like we don't have it yet. We have a good starting point. We can look at the code/data/bss from each .o file that's included in the build. You can build a bare-bones kernel and modules, and use that to see how big things are. You can group those by subsystem to get an idea of how big each subsystem is. Whether or not it's loaded is (mostly) irrelevant - if we compile out USB but then include it as a module, the underlying size is almost the same anyway. Thanks, Adrian From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 20:01:19 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 053549AF for ; Tue, 9 Apr 2013 20:01:19 +0000 (UTC) (envelope-from deeptech71@gmail.com) Received: from mail-ee0-f47.google.com (mail-ee0-f47.google.com [74.125.83.47]) by mx1.freebsd.org (Postfix) with ESMTP id 90D569C3 for ; Tue, 9 Apr 2013 20:01:18 +0000 (UTC) Received: by mail-ee0-f47.google.com with SMTP id t10so3062750eei.34 for ; Tue, 09 Apr 2013 13:01:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=mKDmi3hC7I1vliSKEBPrv96wKhXnbxfyx0S92k+b2iY=; b=mAxfGdziXogjlT5YQ91vkn8k5zWpkuBpuoRCdvekW4XGq+7MpbD+1xF6o/hBEak5Yn +bSUuUuclH2RjqNZjdl1CTTJ4KLQF9OQnwJ/64NrS0DiSyhzv6lUv20gOH9wAebIDNnR ONRYm/sHg8sVTLMNB/wRR+YKz+nDvhQt35unCoTnyiqucaL4csH6RKUH4s/sbt6kdVZj ARKG1ktugR44Zo4mlkK1oobje4mMJrSCH7Pn9U+uRWUPcskxInih+twuD133GnHblMrL HTlmPym0JPnK56AaIatn9B6RCpk2Vj0iZGvBrdDNOq0l10vs7IVX1/zIsRkdOsO6MN9w 7rwQ== X-Received: by 10.14.183.198 with SMTP id q46mr63670290eem.1.1365537677460; Tue, 09 Apr 2013 13:01:17 -0700 (PDT) Received: from [192.168.1.80] (91EC42B1.dsl.pool.telekom.hu. [145.236.66.177]) by mx.google.com with ESMTPS id q5sm39891887eeo.17.2013.04.09.13.01.15 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 09 Apr 2013 13:01:16 -0700 (PDT) Message-ID: <5164815A.6040908@gmail.com> Date: Tue, 09 Apr 2013 23:00:10 +0200 From: deeptech71 User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:18.0) Gecko/20100101 SeaMonkey/2.15 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 09 Apr 2013 21:03:21 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 20:01:19 -0000 Lev Serebryakov wrote: > Is "system" version (system is snapshot from Mar 30) is "external" and "modern"? No. The base version of Clang has been patched to work well with the base system, at least regarding the discussed issue of standard header files. Lev Serebryakov wrote: >It is very sad, that external compiler cannot be used with "fresh" > sources (when headers are different). We need some solution for this, > for sure! When compilers are identical (same clang revision), but > sources has updated headers. It does proper trick with libraries, but > not with headers! Compiler headers are not the main problem. The base system is generally compilable with any compiler, modulo the main problem. Source files in /usr/src should include (ie., #include <...>) headers from /usr/src/include, but in case of an external compiler, they don't, they actually include the headers from /usr/include (don't think about stdio.h, but rather about things like net80211/ieee80211_mesh.h)! The main problem occurs when library headers change (eg., an import of a newer version of /usr/src/contrib/somelib). It happens that the source files need a new definition (eg., #define IEEE80211_MESHRT_FLAGS_GATE 0x08), but such is not found in /usr/include, only in /usr/src/include. However, during ``make installworld'', headers get copied from /usr/src/include to /usr/include, so after that, /usr/include is equivalently usable. After that, compilation won't fail just due to this issue... until /usr/src is significantly updated again. Again, this has nothing to do with compiler headers. If the latest (trunk) version of Clang/LLVM is used -- let's say it's in /home/me/llvm --, then stdio.h (etc.) needs to be removed from /home/me/llvm/include/... (at least I have no other solution), even if /usr/include and /usr/src/include are equivalent. When Clang or GCC is built as part of "make buildworld", it is also built specially once: a bootstrapped compiler uses /usr/obj/tmp/usr/include (a copy of /usr/src/include) instead of /usr/include. From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 9 21:18:09 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D25E585B for ; Tue, 9 Apr 2013 21:18:09 +0000 (UTC) (envelope-from jrisom@gmail.com) Received: from mail-ia0-x230.google.com (mail-ia0-x230.google.com [IPv6:2607:f8b0:4001:c02::230]) by mx1.freebsd.org (Postfix) with ESMTP id A3A97EB1 for ; Tue, 9 Apr 2013 21:18:09 +0000 (UTC) Received: by mail-ia0-f176.google.com with SMTP id i1so6602715iaa.7 for ; Tue, 09 Apr 2013 14:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=nQQUXZThu7JHKO2QU65NUIodzQzZXj52KG6WzYwwxgA=; b=nBd8lyAE9GMNHR2ETvR4DL/+Wk74Vrz+ohEZ8R8RdBjsxTGZAXny4pFahA3x+RB2qa PoX4WJxG+0s5muEHJtwO7zC1SMWT62ydLG66Rjxpy3nwq2OPuqCa79c1T2EPXq3K4chH 006t0TYAQKBx2/N/WWbG6TAc+WzM27lGuiUfgDEJ2toTTo8ymeUzxmldpeejglYrHCrH 8TZ8r4icKHFL2+rhbwe33Cu+h4fQMJFo6vsv+4yHXLSNtCLLEvVLqtghqIZqr1BaF4kT DAxgGq7nmxcQ8SWaE3LWlEPouqf+j3Gv/jzeiofc6IArN5QHvDV0OinomB6wKPvqF06O 568g== X-Received: by 10.50.181.201 with SMTP id dy9mr11427492igc.18.1365542289344; Tue, 09 Apr 2013 14:18:09 -0700 (PDT) Received: from [192.168.1.34] (c-98-212-197-211.hsd1.il.comcast.net. [98.212.197.211]) by mx.google.com with ESMTPS id hi4sm26992702igc.6.2013.04.09.14.18.08 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 09 Apr 2013 14:18:08 -0700 (PDT) Message-ID: <51648588.7010209@gmail.com> Date: Tue, 09 Apr 2013 16:18:00 -0500 From: Joshua Isom User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " References: <516452C7.7040607@mu.org> <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> In-Reply-To: <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2013 21:18:09 -0000 On 4/9/2013 1:47 PM, Edward Tomasz Napierała wrote: > In order to optimize - in this case for size - we need a way to measure > what should we focus on, and it looks like we don't have it yet. > > Would it be possible to write a tool - e.g. by instrumenting LLVM - that > would make it possible to calculate, for every function in the call graph, > the amount of code in that function and everything it "pulls in", i.e. all > the code paths that it might call. When we have that, clustering the graph > should give us some idea what to focus on. > > Or perhaps such a tool already exists? > Would clang's LTO help for size? I know work's starting on the bsd elftools ld, but I doubt it has any LTO support yet. Running -Os on the kernel as a whole instead of object files could probably help a lot also. I might try to set it up and see a size comparision. Also, what about the userland? Linux got popular for embedded partly for busybox and uclibc. If Linux didn't exist, someone would have ported minix instead. From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 06:53:44 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6519A157 for ; Wed, 10 Apr 2013 06:53:44 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.16.84]) by mx1.freebsd.org (Postfix) with ESMTP id 1BAC780B for ; Wed, 10 Apr 2013 06:53:43 +0000 (UTC) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by kabab.cs.huji.ac.il with esmtp id 1UPoub-000PiJ-DI; Wed, 10 Apr 2013 09:53:41 +0300 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.3 To: Wojciech Puchar Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " In-reply-to: References: Comments: In-reply-to Wojciech Puchar message dated "Tue, 09 Apr 2013 19:36:33 +0200." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 10 Apr 2013 09:53:41 +0300 From: Daniel Braniss Message-ID: Cc: freebsd-hackers@freebsd.org, Amit Rawat X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 06:53:44 -0000 > > happy that FreeBSD is among the selected organization. > > > > I am a third year student interested to work in the field of embedded > > system. I applied last year and the title of my project was " Kernel Size > why only in embedded system. smaller programs are always good :) > > And yes FreeBSD kernel is huge. doesn't really matter with 1GB or more > RAM but yes - it is huge even relative to linux. > hum, Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-PRERELEASE #2: Wed Dec 12 13:15:53 IST 2012 danny@rnd:/home/obj/rnd/alix/i386.i386/r+d/stable/9/sys/ALIX i386 CPU: Geode(TM) Integrated Processor by AMD PCS (498.06-MHz 586-class CPU) Origin = "AuthenticAMD" Id = 0x5a2 Family = 0x5 Model = 0xa Stepping = 2 Features=0x88a93d AMD Features=0xc0400000 real memory = 268435456 (256 MB) avail memory = 253120512 (241 MB) pnpbios: Bad PnP BIOS data checksum K6-family MTRR support enabled (2 registers) cryptosoft0: on motherboard pcib0 pcibus 0 on motherboard pci0: on pcib0 Geode LX: PC Engines ALIX.3 v0.99h tinyBIOS V1.4a (C)1997-2007 and top: last pid: 31829; load averages: 0.00, 0.00, 0.00 up 60+17:07:18 09:52:20 24 processes: 1 running, 23 sleeping CPU: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle Mem: 17M Active, 56M Inact, 33M Wired, 34M Buf, 136M Free Swap: 512M Total, 512M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 31829 danny 1 21 0 9784K 2004K RUN 0:00 1.86% top 4288 danny 1 21 0 13160K 5128K select 0:01 1.37% xterm 21604 root 1 20 0 21940K 12796K select 16:51 0.00% python 687 root 1 20 0 11160K 2660K select 5:55 0.00% ntpd 3967 danny 1 20 0 13024K 4364K select 1:55 0.00% elockd 537 root 1 20 0 9592K 9612K select 1:54 0.00% amd-6.2a3 ... this host can run x11 apps! so 'Huge' is a relative matter, my first PDP11/45 has 64K :-) danny From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 13:28:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D70B753F for ; Wed, 10 Apr 2013 13:28:49 +0000 (UTC) (envelope-from mj@feral.com) Received: from virtual.feral.com (virtual.feral.com [216.224.170.83]) by mx1.freebsd.org (Postfix) with ESMTP id A3E23F19 for ; Wed, 10 Apr 2013 13:28:49 +0000 (UTC) Received: from [192.168.135.7] (76-14-49-207.sf-cable.astound.net [76.14.49.207]) by virtual.feral.com (8.14.4/8.14.4) with ESMTP id r3ADRedu013435 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 10 Apr 2013 06:27:41 -0700 Message-ID: <516568CD.80104@feral.com> Date: Wed, 10 Apr 2013 06:27:41 -0700 From: Matthew Jacob User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (virtual.feral.com [216.224.170.83]); Wed, 10 Apr 2013 06:27:42 -0700 (PDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 13:28:49 -0000 On 4/9/2013 11:53 PM, Daniel Braniss wrote: > this host can run x11 apps! so 'Huge' is a relative matter, my first > PDP11/45 has 64K :-) danny Bah. Real old farts ran munix on a 32k PDP 11/03- shell and apps in the low 16k and the kernel in the upper. Or was it the other way around? At Tektronix, a PDP 11/70 supported 64 users runing vi and compiling simultaneously, although starting a link job meant going out for coffee. As a point of comparison with huge and speed: in 1987 my Sun 3/50 with a 15MHz 68020 and 4MB of memory could open the mailtool and I could be reading email within a second. My current desktop with 8GB of memory and running 8 cores @ 2.2GHz and Thunderbird running almost entirely memory before being un-iconified still takes a couple of seconds to be usable. Progress! From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 14:09:46 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 577D6A3B for ; Wed, 10 Apr 2013 14:09:46 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1EDF310B for ; Wed, 10 Apr 2013 14:09:46 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id D41404AC57; Wed, 10 Apr 2013 18:09:39 +0400 (MSK) Date: Wed, 10 Apr 2013 18:09:35 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <1952710103.20130410180935@serebryakov.spb.ru> To: deeptech71 Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? In-Reply-To: <5164815A.6040908@gmail.com> References: <5164815A.6040908@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 14:09:46 -0000 Hello, deeptech71. You wrote 10 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2013 =D0=B3., 1:00:10: d> Lev Serebryakov wrote: >> Is "system" version (system is snapshot from Mar 30) is "external" and "= modern"? d> Lev Serebryakov wrote: >>It is very sad, that external compiler cannot be used with "fresh" >> sources (when headers are different). We need some solution for this, >> for sure! When compilers are identical (same clang revision), but >> sources has updated headers. It does proper trick with libraries, but >> not with headers! d> Compiler headers are not the main problem. The base system is d> generally compilable with any compiler, modulo the main problem. d> Source files in /usr/src should include (ie., #include <...>) d> headers from /usr/src/include, but in case of an external compiler, d> they don't, they actually include the headers from /usr/include d> (don't think about stdio.h, but rather about things like d> net80211/ieee80211_mesh.h)! The main problem occurs when library d> headers change (eg., an import of a newer version of d> /usr/src/contrib/somelib). It happens that the source files need a d> new definition (eg., #define IEEE80211_MESHRT_FLAGS_GATE 0x08), but d> such is not found in /usr/include, only in /usr/src/include. I speak exactly about this situation. d> However, during ``make installworld'', headers get copied from d> /usr/src/include to /usr/include, so after that, /usr/include is d> equivalently usable. After that, compilation won't fail just due to d> this issue... until /usr/src is significantly updated again. Yep. And it happens rather often for -CURRENT. d> When Clang or GCC is built as part of "make buildworld", it is d> also built specially once: a bootstrapped compiler uses d> /usr/obj/tmp/usr/include (a copy of /usr/src/include) instead of /usr/in= clude. I wonder, is it possible to add some options to base compiler to make it use /usr/obj/tmp/usr/include when it build world... Reduction of build time from 1:27 to just 0:27 is very impressive and useful. Also, it looks like clang is cross-compiler always, i.e base clang could build binaries for any supported platform, not only for platform which is the same as host one... It expands possibility of using base compiler to fast build experimental builds for embedded devices even more. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 14:43:12 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 19F2AFD5 for ; Wed, 10 Apr 2013 14:43:12 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id EAF11258 for ; Wed, 10 Apr 2013 14:43:11 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C6286B941; Wed, 10 Apr 2013 10:43:10 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Subject: Re: Multiple page size support on FreeBSD? Date: Wed, 10 Apr 2013 10:06:13 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201304101006.13960.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 10 Apr 2013 10:43:10 -0400 (EDT) Cc: Wojciech Puchar , Benjamin Kaduk , Sebastian Feld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 14:43:12 -0000 On Monday, April 08, 2013 6:39:31 am Wojciech Puchar wrote: > > Superpage promotion happens automatically when consecutive data are accessed > > according to the proper heuristic. > > and in practice - unless there are only few processes, never really works. > > this is a result of my own tests. How do your tests work? Do you examine PTEs directly to check for superpages or are you relying on the vm.pmap.pde sysctls? -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 14:50:06 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 30FC95FE; Wed, 10 Apr 2013 14:50:06 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id 71B332CE; Wed, 10 Apr 2013 14:50:05 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id r3AEo4L4066912; Wed, 10 Apr 2013 09:50:04 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id r3AEo4ST066911; Wed, 10 Apr 2013 09:50:04 -0500 (CDT) (envelope-from brooks) Date: Wed, 10 Apr 2013 09:50:04 -0500 From: Brooks Davis To: Lev Serebryakov Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? Message-ID: <20130410145004.GA66560@lor.one-eyed-alien.net> References: <5164815A.6040908@gmail.com> <1952710103.20130410180935@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dDRMvlgZJXvWKvBx" Content-Disposition: inline In-Reply-To: <1952710103.20130410180935@serebryakov.spb.ru> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: deeptech71 , freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 14:50:06 -0000 --dDRMvlgZJXvWKvBx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Apr 10, 2013 at 06:09:35PM +0400, Lev Serebryakov wrote: > Hello, deeptech71. > You wrote 10 ???????????? 2013 ??., 1:00:10: >=20 > d> Lev Serebryakov wrote: > >> Is "system" version (system is snapshot from Mar 30) is "external" and= "modern"? >=20 > d> Lev Serebryakov wrote: > >>It is very sad, that external compiler cannot be used with "fresh" > >> sources (when headers are different). We need some solution for this, > >> for sure! When compilers are identical (same clang revision), but > >> sources has updated headers. It does proper trick with libraries, but > >> not with headers! >=20 > d> Compiler headers are not the main problem. The base system is > d> generally compilable with any compiler, modulo the main problem. >=20 > d> Source files in /usr/src should include (ie., #include <...>) > d> headers from /usr/src/include, but in case of an external compiler, > d> they don't, they actually include the headers from /usr/include > d> (don't think about stdio.h, but rather about things like > d> net80211/ieee80211_mesh.h)! The main problem occurs when library > d> headers change (eg., an import of a newer version of > d> /usr/src/contrib/somelib). It happens that the source files need a > d> new definition (eg., #define IEEE80211_MESHRT_FLAGS_GATE 0x08), but > d> such is not found in /usr/include, only in /usr/src/include. > I speak exactly about this situation. >=20 > d> However, during ``make installworld'', headers get copied from > d> /usr/src/include to /usr/include, so after that, /usr/include is > d> equivalently usable. After that, compilation won't fail just due to > d> this issue... until /usr/src is significantly updated again. > Yep. And it happens rather often for -CURRENT. >=20 > d> When Clang or GCC is built as part of "make buildworld", it is > d> also built specially once: a bootstrapped compiler uses > d> /usr/obj/tmp/usr/include (a copy of /usr/src/include) instead of /usr/= include. > I wonder, is it possible to add some options to base compiler to > make it use /usr/obj/tmp/usr/include when it build world... The key is to pass the --sysroot option to the compiler. I've got a not quite finished (mostly due to a complete lack of documentation) set of patches to Makefile.inc1 to do this: http://people.freebsd.org/~brooks/patches/xcc3.diff If you set XCC=3D/path/to/clang XCXX=3D/path/to/clang++ XCPP=3Dclang-cpp th= en when building world and kernel you will use those compilers and not build a cross compiler. In that mode you will still build and use a cross binutils > Reduction of build time from 1:27 to just 0:27 is very impressive > and useful. >=20 > Also, it looks like clang is cross-compiler always, i.e base clang > could build binaries for any supported platform, not only for > platform which is the same as host one... It expands possibility of > using base compiler to fast build experimental builds for embedded > devices even more. The above patches support cross build using clang. -- Brooks --dDRMvlgZJXvWKvBx Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFRZXwbXY6L6fI4GtQRAp5hAKDSq/f1gMSNCsfcr9VONAF0mY4T9QCfWHYj gLu3uPo8ssbvc7rnJtzouvA= =nfJG -----END PGP SIGNATURE----- --dDRMvlgZJXvWKvBx-- From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 14:43:42 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5DAD640C for ; Wed, 10 Apr 2013 14:43:42 +0000 (UTC) (envelope-from jra40@hermes.cam.ac.uk) Received: from ppsw-50.csi.cam.ac.uk (ppsw-50.csi.cam.ac.uk [131.111.8.150]) by mx1.freebsd.org (Postfix) with ESMTP id 27463280 for ; Wed, 10 Apr 2013 14:43:41 +0000 (UTC) X-Cam-AntiVirus: no malware found X-Cam-SpamDetails: not scanned X-Cam-ScannerInfo: http://www.ucs.cam.ac.uk/email/scanner/ Received: from cpc2-cmbg15-2-0-cust323.5-4.cable.virginmedia.com ([86.26.13.68]:52366 helo=[192.168.0.5]) by ppsw-50.csi.cam.ac.uk (smtp.hermes.cam.ac.uk [131.111.8.157]:465) with esmtpsa (PLAIN:jra40) (TLSv1:DHE-RSA-AES256-SHA:256) id 1UPwFQ-0007ug-sG (Exim 4.72) (return-path ); Wed, 10 Apr 2013 15:43:40 +0100 Date: Wed, 10 Apr 2013 15:43:39 +0100 From: Jonathan Anderson To: Joshua Isom Message-ID: In-Reply-To: <51648588.7010209@gmail.com> References: <516452C7.7040607@mu.org> <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> <51648588.7010209@gmail.com> Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " X-Mailer: sparrow 1.6.4 (build 1176) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Sender: Jonathan Anderson X-Mailman-Approved-At: Wed, 10 Apr 2013 15:33:48 +0000 Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 14:43:42 -0000 On Tuesday, 9 April 2013 at 22:18, Joshua Isom wrote: > Would clang's LTO help for size? I know work's starting on the bsd > elftools ld, but I doubt it has any LTO support yet. Running -Os on the > kernel as a whole instead of object files could probably help a lot > also. I might try to set it up and see a size comparision. The last I heard, LTO on the kernel required something like 16 GB of RAM and produced a not-quite-working image. Jon -- Jonathan Anderson Research Associate Computer Laboratory University of Cambridge jonathan.anderson@cl.cam.ac.uk +44 1223 763 747 From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 15:56:49 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B0E193A5; Wed, 10 Apr 2013 15:56:49 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 430537E3; Wed, 10 Apr 2013 15:56:49 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 7E9884AC57; Wed, 10 Apr 2013 19:56:47 +0400 (MSK) Date: Wed, 10 Apr 2013 19:56:43 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <814732856.20130410195643@serebryakov.spb.ru> To: Brooks Davis Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? In-Reply-To: <20130410145004.GA66560@lor.one-eyed-alien.net> References: <5164815A.6040908@gmail.com> <1952710103.20130410180935@serebryakov.spb.ru> <20130410145004.GA66560@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: deeptech71 , freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 15:56:49 -0000 Hello, Brooks. You wrote 10 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2013 =D0=B3., 18:50:04: BD> The key is to pass the --sysroot option to the compiler. I've got a not BD> quite finished (mostly due to a complete lack of documentation) set of BD> patches to Makefile.inc1 to do this: BD> http://people.freebsd.org/~brooks/patches/xcc3.diff BD> If you set XCC=3D/path/to/clang XCXX=3D/path/to/clang++ XCPP=3Dclang-cp= p then BD> when building world and kernel you will use those compilers and not BD> build a cross compiler. In that mode you will still build and use a BD> cross binutils I've checked this patch right now, it works for me for "buildworld" and "buildkernel" but not for "installworld": mkdir -p /tmp/install.FxqSvHpP progs=3D$(for prog in [ awk cap_mkdb cat chflags chmod chown date echo egr= ep find grep id install ln lockf make mkdir mtree nmtree mv pwd_mkdb rm = sed sh sysctl test true uname wc zic tzsetup; do if progpath=3D`which $pro= g`; then echo $progpath; else echo "Required tool $prog not found in PAT= H." >&2; exit 1; fi; done); libs=3D$(ldd -f "%o %p\n" -f "%o %p\n" $pro= gs 2>/dev/null | sort -u | while read line; do $line; if [ "$2 $3" !=3D = "not found" ]; then echo $2; else echo "Required library $1 not found." = >&2; exit 1; fi; done); cp $libs $progs /tmp/install.FxqSvHpP cp -R ${PATH_LOCALE:-"/usr/share/locale"} /tmp/install.FxqSvHpP/locale cd /data/src; MAKEOBJDIRPREFIX=3D/data/obj.nano/gateway.v2 MACHINE_ARCH=3Da= md64 MACHINE=3Damd64 CPUTYPE=3D PATH=3D/data/obj.nano/gateway.v2/data/src/t= mp/legacy/usr/sbin:/data/obj.nano/gateway.v2/data/src/tmp/legacy/usr/bin:/d= ata/obj.nano/gateway.v2/data/src/tmp/legacy/usr/games:/data/obj.nano/gatewa= y.v2/data/src/tmp/legacy/bin:/data/obj.nano/gateway.v2/data/src/tmp/usr/sbi= n:/data/obj.nano/gateway.v2/data/src/tmp/usr/bin:/data/obj.nano/gateway.v2/= data/src/tmp/usr/games:/tmp/install.FxqSvHpP LD_LIBRARY_PATH=3D/tmp/instal= l.FxqSvHpP PATH_LOCALE=3D/tmp/install.FxqSvHpP/locale /data/obj.nano/gatew= ay.v2/data/src/make.amd64/make -f Makefile.inc1 __MAKE_SHELL=3D/tmp/inst= all.FxqSvHpP/sh reinstall; MAKEOBJDIRPREFIX=3D/data/obj.nano/gateway.v2 MA= CHINE_ARCH=3Damd64 MACHINE=3Damd64 CPUTYPE=3D PATH=3D/data/obj.nano/gateway= .v2/data/src/tmp/legacy/usr/sbin:/data/obj.nano/gateway.v2/data/src/tmp/leg= acy/usr/bin:/data/obj.nano/gateway.v2/data/src/tmp/legacy/usr/games:/data/o= bj.nano/gateway.v2/data/src/tmp/legacy/bin:/data/obj.nano/gateway.v2/data/s= rc/tmp/usr/sbin:/data/obj.nano/gateway.v2/data/src/tmp/usr/bin:/data/obj.na= no/gateway.v2/data/src/tmp/usr/games:/tmp/install.FxqSvHpP LD_LIBRARY_PATH= =3D/tmp/install.FxqSvHpP PATH_LOCALE=3D/tmp/install.FxqSvHpP/locale rm -rf= /tmp/install.FxqSvHpP cc: not found "/data/src/share/mk/bsd.compiler.mk", line 9: warning: "cc --version" retur= ned non-zero status "/data/src/share/mk/bsd.compiler.mk", line 17: Unable to determine compiler= type for cc. Consider setting COMPILER_TYPE. *** [installworld] Error code 1 1 error *** [installworld] Error code 2 1 error (sources is /data/src, OBJDIRPREFIX is /data/obj.nano/gateway.v2) --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 16:02:05 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 023826C7; Wed, 10 Apr 2013 16:02:05 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [46.4.40.135]) by mx1.freebsd.org (Postfix) with ESMTP id AFEDD824; Wed, 10 Apr 2013 16:02:04 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:900d:c887:884e:713b]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 0255F4AC57; Wed, 10 Apr 2013 20:02:01 +0400 (MSK) Date: Wed, 10 Apr 2013 20:01:57 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <504187318.20130410200157@serebryakov.spb.ru> To: Brooks Davis Subject: Re: building world and kernel without ebuilding ("bootstrap"?) clang? In-Reply-To: <20130410145004.GA66560@lor.one-eyed-alien.net> References: <5164815A.6040908@gmail.com> <1952710103.20130410180935@serebryakov.spb.ru> <20130410145004.GA66560@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: deeptech71 , freebsd-hackers@FreeBSD.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 16:02:05 -0000 Hello, Brooks. You wrote 10 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2013 =D0=B3., 18:50:04: BD> The key is to pass the --sysroot option to the compiler. I've got a not BD> quite finished (mostly due to a complete lack of documentation) set of BD> patches to Makefile.inc1 to do this: BD> http://people.freebsd.org/~brooks/patches/xcc3.diff BD> If you set XCC=3D/path/to/clang XCXX=3D/path/to/clang++ XCPP=3Dclang-cp= p then BD> when building world and kernel you will use those compilers and not BD> build a cross compiler. In that mode you will still build and use a BD> cross binutils BTW, building binutils is required by `install -s' :) --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 17:49:29 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2D36ACF3; Wed, 10 Apr 2013 17:49:29 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id AA6DEDEF; Wed, 10 Apr 2013 17:49:28 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5) with ESMTP id r3AHnAbO019445; Wed, 10 Apr 2013 19:49:10 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.5/Submit) with ESMTP id r3AHn9gY019442; Wed, 10 Apr 2013 19:49:09 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Wed, 10 Apr 2013 19:49:09 +0200 (CEST) From: Wojciech Puchar To: John Baldwin Subject: Re: Multiple page size support on FreeBSD? In-Reply-To: <201304101006.13960.jhb@freebsd.org> Message-ID: References: <201304101006.13960.jhb@freebsd.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Wed, 10 Apr 2013 19:49:10 +0200 (CEST) Cc: freebsd-hackers@freebsd.org, Benjamin Kaduk , Sebastian Feld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 17:49:29 -0000 > How do your tests work? Do you examine PTEs directly to check for superpages > or are you relying on the vm.pmap.pde sysctls? the later. anyway - algorithm described on list - that heuristics detects consecutive page access doesn't really help the urgent case - RANDOM access to large amount of memory. sequential access will get minimal improvement. IMHO the only way that really make sens is to add options to madvise to give kernel information about usage. From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 18:47:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DDC4C33C; Wed, 10 Apr 2013 18:47:30 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from dmz-mailsec-scanner-7.mit.edu (DMZ-MAILSEC-SCANNER-7.MIT.EDU [18.7.68.36]) by mx1.freebsd.org (Postfix) with ESMTP id 735D91A3; Wed, 10 Apr 2013 18:47:30 +0000 (UTC) X-AuditID: 12074424-b7f8c6d0000028c4-1d-5165b28fb71b Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) by dmz-mailsec-scanner-7.mit.edu (Symantec Messaging Gateway) with SMTP id 70.D7.10436.F82B5615; Wed, 10 Apr 2013 14:42:23 -0400 (EDT) Received: from outgoing.mit.edu (OUTGOING-AUTH-1.MIT.EDU [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id r3AIgLDk023351; Wed, 10 Apr 2013 14:42:22 -0400 Received: from multics.mit.edu (SYSTEM-LOW-SIPB.MIT.EDU [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id r3AIgIDX006700 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 10 Apr 2013 14:42:20 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id r3AIgIaA013116; Wed, 10 Apr 2013 14:42:18 -0400 (EDT) Date: Wed, 10 Apr 2013 14:42:18 -0400 (EDT) From: Benjamin Kaduk To: Wojciech Puchar Subject: Re: Multiple page size support on FreeBSD? In-Reply-To: Message-ID: References: <201304101006.13960.jhb@freebsd.org> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrNIsWRmVeSWpSXmKPExsUixCmqrNu/KTXQoGc+v8X2zf8YLfYeuc5s 0XW7ldVi0q73bA4sHjM+zWfx2DnrLrvHg5thAcxRXDYpqTmZZalF+nYJXBlPth5mLPjDXnHz VxdrA+Miti5GTg4JAROJrwt7oWwxiQv31gPZXBxCAvsYJXZfPskC4WxklNhyYTIThHOISeLq 4/vMIC1CAg2MEktvgtksAtoSN2/NYQSx2QRUJGa+2Qg0ioNDBGhFz8FokDCzQKnE8emrmEBs YQFjiY75L8BaOQW8JBpXHWcFsXkFHCTOLJvADLFrOZPEngUzwM4TFdCRWL1/CgtEkaDEyZlP WCCGWkqc+3OdbQKj4CwkqVlIUgsYmVYxyqbkVunmJmbmFKcm6xYnJ+blpRbpmuvlZpbopaaU bmIEhTG7i8oOxuZDSocYBTgYlXh4D1amBgqxJpYVV+YeYpTkYFIS5bXcCBTiS8pPqcxILM6I LyrNSS0+xCjBwawkwmu9DCjHm5JYWZValA+TkuZgURLnvZ5y019IID2xJDU7NbUgtQgmK8PB oSTB6w4yVLAoNT21Ii0zpwQhzcTBCTKcB2i4C0gNb3FBYm5xZjpE/hSjopQ4ryFIQgAkkVGa B9cLSzOvGMWBXhHmzQSp4gGmKLjuV0CDmYAGG/angAwuSURISTUwytzcs8nNlDU2N/Sc/dr5 F1P+JR6Nm3KKjb1LuXerWPWPSyazS2rWqoUZmv7Ou2rYYfb4XpjCLf5as54y6RIBCzOTHR1s NxvCDFTXT+AVzOGPzBN9Kub04671rssfbn94/dRuy+fHU49M0rTquH5LSVdmDdO72ZNOmZre kBG5YnHjn5zsHcv1SizFGYmGWsxFxYkAICxmTQ4DAAA= Cc: freebsd-hackers@freebsd.org, Sebastian Feld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 18:47:30 -0000 On Wed, 10 Apr 2013, Wojciech Puchar wrote: >> How do your tests work? Do you examine PTEs directly to check for >> superpages >> or are you relying on the vm.pmap.pde sysctls? > > the later. > > anyway - algorithm described on list - that heuristics detects consecutive > page access doesn't really help the urgent case - RANDOM access to large > amount of memory. The algorithm is not a heuristic based on consecutive accesses, promotion occurs when the entire superpage's worth of memory has actually been accessed. If I remember correctly, the performance gain from superpages was only a few percent, so spending more time trying to decide when to use them would make the algorithm a net wash. You should really watch the talk I linked to if you're interested, it was quite interesting. > sequential access will get minimal improvement. > > IMHO the only way that really make sens is to add options to madvise to give > kernel information about usage. Maybe. -Ben Kaduk From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 20:00:35 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 81E575EC for ; Wed, 10 Apr 2013 20:00:35 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 701936EE for ; Wed, 10 Apr 2013 20:00:35 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 4AEB61A3C19; Wed, 10 Apr 2013 13:00:33 -0700 (PDT) Message-ID: <5165C4D7.3050308@mu.org> Date: Wed, 10 Apr 2013 13:00:23 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Benjamin Kaduk Subject: Re: Multiple page size support on FreeBSD? References: <201304101006.13960.jhb@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Wojciech Puchar , Sebastian Feld , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 20:00:35 -0000 On 4/10/13 11:42 AM, Benjamin Kaduk wrote: > On Wed, 10 Apr 2013, Wojciech Puchar wrote: > >>> How do your tests work? Do you examine PTEs directly to check for >>> superpages >>> or are you relying on the vm.pmap.pde sysctls? >> >> the later. >> >> anyway - algorithm described on list - that heuristics detects >> consecutive page access doesn't really help the urgent case - RANDOM >> access to large amount of memory. > > The algorithm is not a heuristic based on consecutive accesses, > promotion occurs when the entire superpage's worth of memory has > actually been accessed. If I remember correctly, the performance gain > from superpages was only a few percent, so spending more time trying > to decide when to use them would make the algorithm a net wash. > > You should really watch the talk I linked to if you're interested, it > was quite interesting. > >> sequential access will get minimal improvement. >> >> IMHO the only way that really make sens is to add options to madvise >> to give kernel information about usage. > > Maybe. It is cool that FreeBSD got this work via Alan Cox and the others that contributed. I am wondering if it makes sense to have an explicit model. At one place, for a platform with high performance but a very small TLB, we made it possible to explicitly request a large TLB for our process and it made a BIG difference for performance. Sometimes being "general purpose" means that you can expose such low level things for the user to tune instead of requiring them to fit the app to a heuristic that may change. -Alfred > > -Ben Kaduk > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 20:06:28 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0703D8C6 for ; Wed, 10 Apr 2013 20:06:28 +0000 (UTC) (envelope-from jrisom@gmail.com) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id CCA2F7A4 for ; Wed, 10 Apr 2013 20:06:27 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id ar20so671673iec.14 for ; Wed, 10 Apr 2013 13:06:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=LG9oL42RhBb6RQn5EjZ/ygClMHiFgulGmNA2e1mJmT0=; b=po3/RFsjgTvR8NbyUolKqyrQpkWO6dz+O9i1pm11a+RF9ILIuXrX06b+oGZZdlCkXj bR/WhmZ+lqKOzmDJoY/3E0TTLgC5DT24B3SK6E9oJ8RBf4aINGvthVvSt3lIVfVLeJZ3 tZUMNurnm+ebjT/x7vf85LVaFioCQufstfqBW6LYIFmew9NhXMmFMzJm3/cCHlkkeYds pP3yjIaqyvL4/8AppPOBT/3TpsjEyU3niKv9MDL+c5d+ebaK61rGvqIy5S8vTUprKNGu lo/Hg8yglSJ94Xv92AUc2n8xOi4Nhx2O/Z5crgDwy0fnXoIRCBxVoTXGTelXgqACRRQC BplQ== X-Received: by 10.50.191.228 with SMTP id hb4mr2434018igc.37.1365624387532; Wed, 10 Apr 2013 13:06:27 -0700 (PDT) Received: from [192.168.1.34] (c-98-212-197-211.hsd1.il.comcast.net. [98.212.197.211]) by mx.google.com with ESMTPS id xc3sm28688008igb.10.2013.04.10.13.06.26 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 10 Apr 2013 13:06:26 -0700 (PDT) Message-ID: <5165C639.7040300@gmail.com> Date: Wed, 10 Apr 2013 15:06:17 -0500 From: Joshua Isom User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Jonathan Anderson Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " References: <516452C7.7040607@mu.org> <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> <51648588.7010209@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 20:06:28 -0000 On 4/10/2013 9:43 AM, Jonathan Anderson wrote: > > The last I heard, LTO on the kernel required something like 16 GB of RAM and produced a not-quite-working image. > > > Jon > I upgraded my system with 32Gb for a reason. From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 20:12:46 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1D3EBA59; Wed, 10 Apr 2013 20:12:46 +0000 (UTC) (envelope-from sebastian.n.feld@gmail.com) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id DE3E77EE; Wed, 10 Apr 2013 20:12:45 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id ar20so678968iec.14 for ; Wed, 10 Apr 2013 13:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=yMGzvgZ/Hc8ITmCKeDFsLwFhjVMtFX2GTqGS77h2ifw=; b=jCN6X9CXQSU3lTxPBmkDtlE64M4RcNevDfkWMoKzzIMZmT0/1pGwm9TiFEvXtSK0mq Ghl2gj+ujDJC8pdlzZ6QuJWmIKq60yg581pNLAqOg89KjFB3OY7utR0ZroEOk/f5WOmB rb8UqsIhuLhi4JwaQMhlzmlJB1TLxVpUiYIEYOvr+chrly+CgDkxWjUdZCfvQZLZJWfc odULnP7vqDTbvuAzzz7BkpAFBLzlrrsGrFlau/KfqqKj2V+cFnC3GesXe1WDmRLQoteK dE/Nhat491zumlVCZDHX26Z2f8ezHGjj3c9iEudzV6kvDTVsPxOflNq7fLwKlpwdQvb1 ezsA== MIME-Version: 1.0 X-Received: by 10.50.155.134 with SMTP id vw6mr14413561igb.34.1365624765637; Wed, 10 Apr 2013 13:12:45 -0700 (PDT) Received: by 10.42.101.20 with HTTP; Wed, 10 Apr 2013 13:12:45 -0700 (PDT) In-Reply-To: References: <201304101006.13960.jhb@freebsd.org> Date: Wed, 10 Apr 2013 22:12:45 +0200 Message-ID: Subject: Re: Multiple page size support on FreeBSD? From: Sebastian Feld To: Wojciech Puchar Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, Benjamin Kaduk X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 20:12:46 -0000 On Wed, Apr 10, 2013 at 7:49 PM, Wojciech Puchar wrote: >> How do your tests work? Do you examine PTEs directly to check for >> superpages >> or are you relying on the vm.pmap.pde sysctls? > > > the later. > > anyway - algorithm described on list - that heuristics detects consecutive > page access doesn't really help the urgent case - RANDOM access to large > amount of memory. > > sequential access will get minimal improvement. > > IMHO the only way that really make sens is to add options to madvise to give > kernel information about usage. Solaris has an API to control the usage for large pages, search for MPSS - multiple page size support. It consists of APIs in madvise, and /usr/bin/pagesize -a to list all available page sizes, and other stuff. Since this has been established since at least Solaris 9 (and 10) and is used by database and HPC software, it might be a good starting point. From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 20:22:28 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DB5C1C9E for ; Wed, 10 Apr 2013 20:22:28 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from exprod7og119.obsmtp.com (exprod7og119.obsmtp.com [64.18.2.16]) by mx1.freebsd.org (Postfix) with ESMTP id 8CC3F859 for ; Wed, 10 Apr 2013 20:22:28 +0000 (UTC) Received: from P-EMHUB03-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob119.postini.com ([64.18.6.12]) with SMTP ID DSNKUWXJ/i49CeLFkCDVhS3q4k8jr073xh7l@postini.com; Wed, 10 Apr 2013 13:22:28 PDT Received: from P-CLDFE01-HQ.jnpr.net (172.24.192.59) by P-EMHUB03-HQ.jnpr.net (172.24.192.37) with Microsoft SMTP Server (TLS) id 8.3.213.0; Wed, 10 Apr 2013 13:09:38 -0700 Received: from o365mail.juniper.net (207.17.137.149) by o365mail.juniper.net (172.24.192.59) with Microsoft SMTP Server id 14.1.355.2; Wed, 10 Apr 2013 13:09:38 -0700 Received: from va3outboundpool.messaging.microsoft.com (216.32.180.16) by o365mail.juniper.net (207.17.137.149) with Microsoft SMTP Server (TLS) id 14.1.355.2; Wed, 10 Apr 2013 13:12:07 -0700 Received: from mail214-va3-R.bigfish.com (10.7.14.230) by VA3EHSOBE009.bigfish.com (10.7.40.29) with Microsoft SMTP Server id 14.1.225.23; Wed, 10 Apr 2013 20:09:37 +0000 Received: from mail214-va3 (localhost [127.0.0.1]) by mail214-va3-R.bigfish.com (Postfix) with ESMTP id 1BAC888010E for ; Wed, 10 Apr 2013 20:09:37 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.244.213; KIP:(null); UIP:(null); (null); H:CH1PRD0510HT003.namprd05.prod.outlook.com; R:internal; EFV:INT X-SpamScore: -5 X-BigFish: PS-5(zzbb2dI98dI9371I542I1432Izz1f42h1fc6h1ee6h1de0h1fdah1202h1e76h1d1ah1d2ahzz17326ah8275dh8275chz2dh2a8h668h839h947hd25hf0ah1288h12a5h12a9h12bdh137ah13b6h1441h1504h1537h153bh15d0h162dh1631h1758h18e1h1946h19b5h19ceh1ad9h1b0ah1155h) Received: from mail214-va3 (localhost.localdomain [127.0.0.1]) by mail214-va3 (MessageSwitch) id 136562457555068_27568; Wed, 10 Apr 2013 20:09:35 +0000 (UTC) Received: from VA3EHSMHS004.bigfish.com (unknown [10.7.14.230]) by mail214-va3.bigfish.com (Postfix) with ESMTP id F3C96D40061; Wed, 10 Apr 2013 20:09:34 +0000 (UTC) Received: from CH1PRD0510HT003.namprd05.prod.outlook.com (157.56.244.213) by VA3EHSMHS004.bigfish.com (10.7.99.14) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 10 Apr 2013 20:09:33 +0000 Received: from CH1PRD0510MB392.namprd05.prod.outlook.com ([169.254.6.36]) by CH1PRD0510HT003.namprd05.prod.outlook.com ([10.255.150.38]) with mapi id 14.16.0293.003; Wed, 10 Apr 2013 20:09:33 +0000 From: Andrew Duane To: Alfred Perlstein , Benjamin Kaduk Subject: RE: Multiple page size support on FreeBSD? Thread-Topic: Multiple page size support on FreeBSD? Thread-Index: AQHONiYhR4l/DNF6FkudeEdNuuszSpjP4RAw Date: Wed, 10 Apr 2013 20:09:32 +0000 Message-ID: <477C1270D3E5484DA2303CEBE274C9E1250B3DF7@CH1PRD0510MB392.namprd05.prod.outlook.com> References: <201304101006.13960.jhb@freebsd.org> <5165C4D7.3050308@mu.org> In-Reply-To: <5165C4D7.3050308@mu.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [66.129.232.2] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-FOPE-CONNECTOR: Id%12219$Dn%MU.ORG$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net X-FOPE-CONNECTOR: Id%12219$Dn%MIT.EDU$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net X-FOPE-CONNECTOR: Id%12219$Dn%WOJTEK.TENSOR.GDYNIA.PL$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net X-FOPE-CONNECTOR: Id%12219$Dn%GMAIL.COM$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net X-FOPE-CONNECTOR: Id%12219$Dn%FREEBSD.ORG$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net Cc: Wojciech Puchar , Sebastian Feld , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 20:22:28 -0000 Like all "performance" items (especially VM), it depends on the hardware an= d the load. On systems with small TLBs it helps more than with large TLBs. = With software that needs access to lots of different areas the TLB gets mor= e traffic so large ones help more. The answer for your firefox browser box = with i386 is probably different from my compilation engine running MIPS, or= his web server running AMD. Back at Digital, we spent a lot of time trying to find the "one true answer= " to superpages, only to discover there wasn't one. We ended up with a comb= ination of automatic use from big allocations, a rarely used API to advise = for big TLBs, and some background work that coalesced when possible. =A0.................................... Andrew L. Duane Resident Architect - AT&T Technical Lead m=A0=A0=A0+1 603.770.7088 o +1 408.933.6944 (2-6944) skype: andrewlduane aduane@juniper.net -----Original Message----- From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd-hackers@freeb= sd.org] On Behalf Of Alfred Perlstein Sent: Wednesday, April 10, 2013 4:00 PM To: Benjamin Kaduk Cc: Wojciech Puchar; Sebastian Feld; freebsd-hackers@freebsd.org Subject: Re: Multiple page size support on FreeBSD? On 4/10/13 11:42 AM, Benjamin Kaduk wrote: > On Wed, 10 Apr 2013, Wojciech Puchar wrote: > >>> How do your tests work? Do you examine PTEs directly to check for=20 >>> superpages or are you relying on the vm.pmap.pde sysctls? >> >> the later. >> >> anyway - algorithm described on list - that heuristics detects=20 >> consecutive page access doesn't really help the urgent case - RANDOM=20 >> access to large amount of memory. > > The algorithm is not a heuristic based on consecutive accesses,=20 > promotion occurs when the entire superpage's worth of memory has=20 > actually been accessed. If I remember correctly, the performance gain=20 > from superpages was only a few percent, so spending more time trying=20 > to decide when to use them would make the algorithm a net wash. > > You should really watch the talk I linked to if you're interested, it=20 > was quite interesting. > >> sequential access will get minimal improvement. >> >> IMHO the only way that really make sens is to add options to madvise=20 >> to give kernel information about usage. > > Maybe. It is cool that FreeBSD got this work via Alan Cox and the others that cont= ributed. I am wondering if it makes sense to have an explicit model. At one place, for a platform with high performance but a very small TLB, we= made it possible to explicitly request a large TLB for our process and it = made a BIG difference for performance. Sometimes being "general purpose" means that you can expose such low level = things for the user to tune instead of requiring them to fit the app to a h= euristic that may change. -Alfred > > -Ben Kaduk > _______________________________________________ > freebsd-hackers@freebsd.org mailing list=20 > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/l= istinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 20:28:06 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 86195F82 for ; Wed, 10 Apr 2013 20:28:06 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 67B6D89B for ; Wed, 10 Apr 2013 20:28:06 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id CB59C1A3CD2; Wed, 10 Apr 2013 13:28:05 -0700 (PDT) Message-ID: <5165CB4C.4010608@mu.org> Date: Wed, 10 Apr 2013 13:27:56 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: Andrew Duane Subject: Re: Multiple page size support on FreeBSD? References: <201304101006.13960.jhb@freebsd.org> <5165C4D7.3050308@mu.org> <477C1270D3E5484DA2303CEBE274C9E1250B3DF7@CH1PRD0510MB392.namprd05.prod.outlook.com> In-Reply-To: <477C1270D3E5484DA2303CEBE274C9E1250B3DF7@CH1PRD0510MB392.namprd05.prod.outlook.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Wojciech Puchar , Benjamin Kaduk , Sebastian Feld , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 20:28:06 -0000 On 4/10/13 1:09 PM, Andrew Duane wrote: > Like all "performance" items (especially VM), it depends on the hardware and the load. On systems with small TLBs it helps more than with large TLBs. With software that needs access to lots of different areas the TLB gets more traffic so large ones help more. The answer for your firefox browser box with i386 is probably different from my compilation engine running MIPS, or his web server running AMD. > > Back at Digital, we spent a lot of time trying to find the "one true answer" to superpages, only to discover there wasn't one. We ended up with a combination of automatic use from big allocations, a rarely used API to advise for big TLBs, and some background work that coalesced when possible. Thank you Andrew. I agree. A good heuristic is great, but sometimes exposing the API unlocks some really awesome performance capabilities. It seems like both Digital and Sun went this route. I'm hoping we can do that as well. -Alfred > .................................... > Andrew L. Duane > Resident Architect - AT&T Technical Lead > m +1 603.770.7088 > o +1 408.933.6944 (2-6944) > skype: andrewlduane > aduane@juniper.net > > > > -----Original Message----- > From: owner-freebsd-hackers@freebsd.org [mailto:owner-freebsd-hackers@freebsd.org] On Behalf Of Alfred Perlstein > Sent: Wednesday, April 10, 2013 4:00 PM > To: Benjamin Kaduk > Cc: Wojciech Puchar; Sebastian Feld; freebsd-hackers@freebsd.org > Subject: Re: Multiple page size support on FreeBSD? > > On 4/10/13 11:42 AM, Benjamin Kaduk wrote: >> On Wed, 10 Apr 2013, Wojciech Puchar wrote: >> >>>> How do your tests work? Do you examine PTEs directly to check for >>>> superpages or are you relying on the vm.pmap.pde sysctls? >>> the later. >>> >>> anyway - algorithm described on list - that heuristics detects >>> consecutive page access doesn't really help the urgent case - RANDOM >>> access to large amount of memory. >> The algorithm is not a heuristic based on consecutive accesses, >> promotion occurs when the entire superpage's worth of memory has >> actually been accessed. If I remember correctly, the performance gain >> from superpages was only a few percent, so spending more time trying >> to decide when to use them would make the algorithm a net wash. >> >> You should really watch the talk I linked to if you're interested, it >> was quite interesting. >> >>> sequential access will get minimal improvement. >>> >>> IMHO the only way that really make sens is to add options to madvise >>> to give kernel information about usage. >> Maybe. > It is cool that FreeBSD got this work via Alan Cox and the others that contributed. > > I am wondering if it makes sense to have an explicit model. > > At one place, for a platform with high performance but a very small TLB, we made it possible to explicitly request a large TLB for our process and it made a BIG difference for performance. > > Sometimes being "general purpose" means that you can expose such low level things for the user to tune instead of requiring them to fit the app to a heuristic that may change. > > -Alfred > > >> -Ben Kaduk >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 10 23:25:39 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 23BF99A8 for ; Wed, 10 Apr 2013 23:25:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) by mx1.freebsd.org (Postfix) with ESMTP id B14C412A for ; Wed, 10 Apr 2013 23:25:38 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id ez12so1008864wid.17 for ; Wed, 10 Apr 2013 16:25:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=tA1aO2zJtRD5hs0ABNzhlcAICgBZF4rLoSRUmuBABU8=; b=LEgWhvzhfbl7Qfr3es1YpEB7zFY1GzFTnD1qunHlwFNjwcz6V2L3FSJERx/7E98XZL kzf++vPJTL9oI4M1PG42Ho0gvRxoZZlWarzHL1O1KFGT9R1pPRc+J3ipmRzKYkpW7WAd U37WnMFOXdLot09vTJOP1zI+M+CoQUs5VbjAQkj7xCg9VWeZBZfjOhUDHztYr5vFRtJt JRamJxt3Xm2m/srrDlOp2fnebKV7M5hpe73kRAUNOA3J772lCCJMMh5ywwYyS2VKwKNH ELt4FytmD3Y83TJ3UVOR5Eoh50lNeDRGSVVSABNiwu4mZV73SbyC48ruJ8gu3SFvxk/e sqDg== MIME-Version: 1.0 X-Received: by 10.180.19.39 with SMTP id b7mr30174846wie.15.1365636337806; Wed, 10 Apr 2013 16:25:37 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.121.136 with HTTP; Wed, 10 Apr 2013 16:25:37 -0700 (PDT) In-Reply-To: <5165C639.7040300@gmail.com> References: <516452C7.7040607@mu.org> <92799D4C-797C-4304-B299-DD1DBA49CFFC@FreeBSD.org> <51648588.7010209@gmail.com> <5165C639.7040300@gmail.com> Date: Wed, 10 Apr 2013 16:25:37 -0700 X-Google-Sender-Auth: CyvJqvQfILvARbnPL7MhBR1_92Q Message-ID: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " From: Adrian Chadd To: Joshua Isom Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@freebsd.org, Jonathan Anderson X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 23:25:39 -0000 On 10 April 2013 13:06, Joshua Isom wrote: > > I upgraded my system with 32Gb for a reason. Yes, yes you did. TO force me to fix ath(4) and busdma. ;-) Adrian From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 11 18:44:54 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 86F7258F for ; Thu, 11 Apr 2013 18:44:54 +0000 (UTC) (envelope-from cedric.blancher@gmail.com) Received: from mail-ia0-x236.google.com (mail-ia0-x236.google.com [IPv6:2607:f8b0:4001:c02::236]) by mx1.freebsd.org (Postfix) with ESMTP id 585E6128F for ; Thu, 11 Apr 2013 18:44:54 +0000 (UTC) Received: by mail-ia0-f182.google.com with SMTP id u20so1697593iag.41 for ; Thu, 11 Apr 2013 11:44:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=+Fx83oLcqacrtGXJEeDvTYwX0dl7rxxwPirV7pCksZQ=; b=Xtvp12mapAhevBg8gndjf02k7/IV2vaOIoe/S3rnYepazDrbpcVdoKnJTqdYWCWFIh Z4JZB1sq4mSR57mkO+fGlrC3A9KMkCDFVOVPAb5G5HtwfQ2FbIbU1UDh4gTbXYY+ld0t /OhDgUO3HrfCEg+hKRda6zjsPXnNRGJsZ64eexJ4eVSK47ZBPvEuqvopZj86KHOTrJXY 0LYsHIw/Ql8ypcapQm9B/hJcPJ9cFsMoJnbbAAydOfbz5B24zBzJzE+cACKHhUPlEkLf +IKUAlWng4LQt+hGcp3bB7UUPybOue6DrrGxfQHOoxmb6eRXzy8Vz5c+loHLs8L5iKi7 KF6Q== MIME-Version: 1.0 X-Received: by 10.50.22.3 with SMTP id z3mr5323275ige.80.1365705893903; Thu, 11 Apr 2013 11:44:53 -0700 (PDT) Sender: cedric.blancher@gmail.com Received: by 10.50.126.8 with HTTP; Thu, 11 Apr 2013 11:44:53 -0700 (PDT) In-Reply-To: <5165CB4C.4010608@mu.org> References: <201304101006.13960.jhb@freebsd.org> <5165C4D7.3050308@mu.org> <477C1270D3E5484DA2303CEBE274C9E1250B3DF7@CH1PRD0510MB392.namprd05.prod.outlook.com> <5165CB4C.4010608@mu.org> Date: Thu, 11 Apr 2013 20:44:53 +0200 X-Google-Sender-Auth: UtkCAzHHxkdgAjBh5U01c0n8OvM Message-ID: Subject: Re: Multiple page size support on FreeBSD? From: Cedric Blancher To: Alfred Perlstein Content-Type: text/plain; charset=ISO-8859-1 Cc: Wojciech Puchar , Sebastian Feld , Benjamin Kaduk , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2013 18:44:54 -0000 On 10 April 2013 22:27, Alfred Perlstein wrote: > On 4/10/13 1:09 PM, Andrew Duane wrote: >> >> Like all "performance" items (especially VM), it depends on the hardware >> and the load. On systems with small TLBs it helps more than with large TLBs. >> With software that needs access to lots of different areas the TLB gets more >> traffic so large ones help more. The answer for your firefox browser box >> with i386 is probably different from my compilation engine running MIPS, or >> his web server running AMD. >> >> Back at Digital, we spent a lot of time trying to find the "one true >> answer" to superpages, only to discover there wasn't one. We ended up with a >> combination of automatic use from big allocations, a rarely used API to >> advise for big TLBs, and some background work that coalesced when possible. > > > Thank you Andrew. I agree. A good heuristic is great, but sometimes > exposing the API unlocks some really awesome performance capabilities. > > It seems like both Digital and Sun went this route. > > I'm hoping we can do that as well. > > -Alfred > > >> .................................... >> Andrew L. Duane >> Resident Architect - AT&T Technical Lead >> m +1 603.770.7088 >> o +1 408.933.6944 (2-6944) >> skype: andrewlduane >> aduane@juniper.net >> >> >> >> -----Original Message----- >> From: owner-freebsd-hackers@freebsd.org >> [mailto:owner-freebsd-hackers@freebsd.org] On Behalf Of Alfred Perlstein >> Sent: Wednesday, April 10, 2013 4:00 PM >> To: Benjamin Kaduk >> Cc: Wojciech Puchar; Sebastian Feld; freebsd-hackers@freebsd.org >> Subject: Re: Multiple page size support on FreeBSD? >> >> On 4/10/13 11:42 AM, Benjamin Kaduk wrote: >>> >>> On Wed, 10 Apr 2013, Wojciech Puchar wrote: >>> >>>>> How do your tests work? Do you examine PTEs directly to check for >>>>> superpages or are you relying on the vm.pmap.pde sysctls? >>>> >>>> the later. >>>> >>>> anyway - algorithm described on list - that heuristics detects >>>> consecutive page access doesn't really help the urgent case - RANDOM >>>> access to large amount of memory. >>> >>> The algorithm is not a heuristic based on consecutive accesses, >>> promotion occurs when the entire superpage's worth of memory has >>> actually been accessed. If I remember correctly, the performance gain >>> from superpages was only a few percent, so spending more time trying >>> to decide when to use them would make the algorithm a net wash. >>> >>> You should really watch the talk I linked to if you're interested, it >>> was quite interesting. >>> >>>> sequential access will get minimal improvement. >>>> >>>> IMHO the only way that really make sens is to add options to madvise >>>> to give kernel information about usage. >>> >>> Maybe. >> >> It is cool that FreeBSD got this work via Alan Cox and the others that >> contributed. >> >> I am wondering if it makes sense to have an explicit model. >> >> At one place, for a platform with high performance but a very small TLB, >> we made it possible to explicitly request a large TLB for our process and it >> made a BIG difference for performance. >> >> Sometimes being "general purpose" means that you can expose such low level >> things for the user to tune instead of requiring them to fit the app to a >> heuristic that may change. As Sebastian said it might be worth (well, very worth) to implement the Solaris userland API since there is software which uses it (since Solaris >=9 ). Hey, even ksh93 uses MHA_MAPSIZE_STACK (to force 64k pages when available; http://www.opensource.apple.com/source/ksh/ksh-18/ksh/src/cmd/ksh93/sh/pmain.c?txt) and the X11 xorg server does it, too. Ced -- Cedric Blancher Institute Pasteur From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 11 19:30:25 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 78EE142B for ; Thu, 11 Apr 2013 19:30:25 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 12287152F for ; Thu, 11 Apr 2013 19:30:24 +0000 (UTC) Received: from server.rulingia.com (c220-239-237-213.belrs5.nsw.optusnet.com.au [220.239.237.213]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id r3BJUG5R072600 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 12 Apr 2013 05:30:16 +1000 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id r3BJUAnF024159 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 12 Apr 2013 05:30:10 +1000 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id r3BJUA0t024158; Fri, 12 Apr 2013 05:30:10 +1000 (EST) (envelope-from peter) Date: Fri, 12 Apr 2013 05:30:10 +1000 From: Peter Jeremy To: Freddie Cash Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " Message-ID: <20130411193010.GH31958@server.rulingia.com> References: <516452C7.7040607@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JWEK1jqKZ6MHAcjA" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Wojciech Puchar , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2013 19:30:25 -0000 --JWEK1jqKZ6MHAcjA Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2013-Apr-09 11:05:56 -0700, Freddie Cash wrote: >You have to look at the in-memory sizes, not the on-disk sizes. Or, even better, look at the difference between installed physical RAM and how much RAM is available to userland processes. --=20 Peter Jeremy --JWEK1jqKZ6MHAcjA Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlFnD0IACgkQ/opHv/APuIduHgCgq/CosfGxFANJeMiB4ecSQNW3 8O4AoJnrpCwTWNWWPk6b3mYZhZzJ4lEI =adil -----END PGP SIGNATURE----- --JWEK1jqKZ6MHAcjA-- From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 12 12:30:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 385B7372 for ; Fri, 12 Apr 2013 12:30:30 +0000 (UTC) (envelope-from lars.engels@0x20.net) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) by mx1.freebsd.org (Postfix) with ESMTP id 031A9386 for ; Fri, 12 Apr 2013 12:30:29 +0000 (UTC) Received: from 0x20.net (0x20.net [217.69.76.212]) (Authenticated sender: lala) by mail.0x20.net (Postfix) with ESMTPA id 07AC16A6001 for ; Fri, 12 Apr 2013 14:30:29 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 12 Apr 2013 14:30:28 +0200 From: Lars Engels To: Subject: Re: GSOC 2013 project " Kernel Size Reduction for Embedded System " In-Reply-To: <516568CD.80104@feral.com> References: <516568CD.80104@feral.com> Message-ID: <86c97844c3a21dce5922552694c3eeb5@mail.0x20.net> X-Sender: lars.engels@0x20.net User-Agent: Roundcube Webmail/0.7 X-Mailman-Approved-At: Fri, 12 Apr 2013 12:43:27 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2013 12:30:30 -0000 Am 10.04.2013 15:27, schrieb Matthew Jacob: > On 4/9/2013 11:53 PM, Daniel Braniss wrote: >> this host can run x11 apps! so 'Huge' is a relative matter, my first >> PDP11/45 has 64K :-) danny > Bah. Real old farts ran munix on a 32k PDP 11/03- shell and apps in > the low 16k and the kernel in the upper. Or was it the other way > around? At Tektronix, a PDP 11/70 supported 64 users runing vi and > compiling simultaneously, although starting a link job meant going out > for coffee. > > As a point of comparison with huge and speed: in 1987 my Sun 3/50 > with a 15MHz 68020 and 4MB of memory could open the mailtool and I > could be reading email within a second. > > My current desktop with 8GB of memory and running 8 cores @ 2.2GHz > and Thunderbird running almost entirely memory before being > un-iconified still takes a couple of seconds to be usable. That's why I use mutt. :-) From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 13 10:00:00 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D3CE819E; Sat, 13 Apr 2013 10:00:00 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f44.google.com (mail-ee0-f44.google.com [74.125.83.44]) by mx1.freebsd.org (Postfix) with ESMTP id 4736BA99; Sat, 13 Apr 2013 09:59:59 +0000 (UTC) Received: by mail-ee0-f44.google.com with SMTP id c41so1617765eek.3 for ; Sat, 13 Apr 2013 02:59:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=RhCqstBPNV+kOI0U6HDfLAY5u503SAlnFetgdPL7HnM=; b=pI2/N+Cvttfi7SxaSdWsTP3f6hKQGQ557SWJAEb0avZ0Dk16+kB2mLpUtxkKTI5Zgr A6F6XYP0z8vOmWdTBvBpMzBBI52+3jH8XWgC3in+8g1zDWAAab/OmC2e7CDBi5K3rs9R 5Wfw+8V4m2dVMFXp6za2V/kB0n8aHav/R44Dx2rStAXyJH0Lat5GgR9wxmoPlIn6zvXB 6u6IxyrwWqQQHGu6O3o5BFRWrVsIsQBmb2b7elV3/XgWv7W0wJzA8tD1dpg9VaAn5UIm joIuLipJc3W4bE2/U1kol0JfjrboHxz0qc5dlTurCOeyrn0QKlR5EHXtOimHwhQkgLuO chrQ== X-Received: by 10.15.102.3 with SMTP id bq3mr37751103eeb.42.1365847193119; Sat, 13 Apr 2013 02:59:53 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id cd3sm15594347eeb.6.2013.04.13.02.59.51 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 13 Apr 2013 02:59:52 -0700 (PDT) Sender: Alexander Motin Message-ID: <51692C95.3010901@FreeBSD.org> Date: Sat, 13 Apr 2013 12:59:49 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130326 Thunderbird/17.0.4 MIME-Version: 1.0 To: freebsd-geom@FreeBSD.org, freebsd-hackers@FreeBSD.org Subject: devstat overhead VS precision Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2013 10:00:00 -0000 Hi. It is long known that collecting disk and GEOM statistics may cause significant processing overhead under high IOPS. On my recent high-IOPS benchmarks performance difference was reaching three times! Last time situation improved a lot by more active use of TSC, but there are still many systems where TSCs are not synchronized. I propose to switch that statistics from using binuptime() to getbinuptime() to solve the problem globally. From one side getbinuptime() resolution is limited by 1ms, but since time is usually averaged over the many I/Os, additional sub-millisecond precision will come from sampling. Since most of tools now show request processing times up to 0.1ms, that precision should be sufficient. I believe real disk performance is more important that n-th digit in some statistics. The following patch does the change and makes disk performance irrelevant to the timecounter performance: http://people.freebsd.org/~mav/devstat_time.patch Are there any objections against it? -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 13 22:45:19 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 214FCDC9 for ; Sat, 13 Apr 2013 22:45:19 +0000 (UTC) (envelope-from jrisom@gmail.com) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) by mx1.freebsd.org (Postfix) with ESMTP id E93D31691 for ; Sat, 13 Apr 2013 22:45:18 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id e14so4623267iej.16 for ; Sat, 13 Apr 2013 15:45:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=RXyxtPhZAkCICdXocsQlF1pFl65O8EhErSty8UOH0XQ=; b=jfFoY+horg9eIvBtkNB/G7S9Jgbp9oUSXUJedn4Ob3JwC6hm2H0bg+HHnAkIcCEiel hr+BcW6Um94v1R5f5/3SY1XdMh2W3z8osf2yiTbQ5siLJ1+1RhI3gzpczXodjmNVDnE6 03M3TZP9MRywZ5kE3L7KiKTCT6Pmo/VcbxgETjFe2rfTBcEhRNkbYOWdHR4HakbPCJX1 EBebxEwCCKQkP5Q+gKD7egIc1JiaT6JlHFfdn6KUCFfiXHXo5SGKgip2UO7NUz5CzHsS pY/wE7CNrxwcwPMsIMPLhHxPiei3N8Wu+y0O/j+wscdLqxBvZVKazu/9a+JIIFEALnqV 6H/w== X-Received: by 10.50.78.37 with SMTP id y5mr2240430igw.43.1365893118506; Sat, 13 Apr 2013 15:45:18 -0700 (PDT) Received: from [192.168.1.34] (c-98-212-197-211.hsd1.il.comcast.net. [98.212.197.211]) by mx.google.com with ESMTPS id ih1sm4558798igc.3.2013.04.13.15.45.17 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 13 Apr 2013 15:45:17 -0700 (PDT) Message-ID: <5169DFFB.4030101@gmail.com> Date: Sat, 13 Apr 2013 17:45:15 -0500 From: Joshua Isom User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130328 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-hackers@FreeBSD.org Subject: USB Printer quirks Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2013 22:45:19 -0000 I've got my printer to finally work, now that I got around to hooking it up again and trying. It's a Kodak AiO, and there's a driver that's been around for a few years and works under Linux. It doesn't work properly under FreeBSD, or a Debian/kFreeBSD jail, but does work under a Linux jail. With FreeBSD, the best I'd get is one document and then need to power cycle the printer to get it to do anything. From what I remember of the logs, the communication wasn't coming back properly. Cups is reporting it's not getting bidirectional communication, but it is mostly printing properly. Since it works under the Linux jail, and not the Debian jail, I'm assuming it has to be a FreeBSD kernel issue that's masked by the Linux overhead. But where would I begin to troubleshoot the underlying issue? Right now I'm running a -CURRENT kernel: FreeBSD jri.homeunix.com 10.0-CURRENT FreeBSD 10.0-CURRENT #15 r249169M: Fri Apr 5 16:12:04 CDT 2013 root@jri.homeunix.com:/usr/obj/root/ATH/head/sys/ATH amd64 Any help would be appreciated.