From owner-freebsd-arch@FreeBSD.ORG Sun Oct 21 02:32:48 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D69D177C; Sun, 21 Oct 2012 02:32:47 +0000 (UTC) (envelope-from marcel@xcllnt.net) Received: from mail.xcllnt.net (mail.xcllnt.net [70.36.220.4]) by mx1.freebsd.org (Postfix) with ESMTP id 984CC8FC0C; Sun, 21 Oct 2012 02:32:46 +0000 (UTC) Received: from marcelm-sslvpn-nc.jnpr.net (natint3.juniper.net [66.129.224.36]) (authenticated bits=0) by mail.xcllnt.net (8.14.5/8.14.5) with ESMTP id q9L2Wi1w035388 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 20 Oct 2012 19:32:45 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Behavior of madvise(MADV_FREE) From: Marcel Moolenaar In-Reply-To: <5082F0F3.1070102@rice.edu> Date: Sat, 20 Oct 2012 19:33:12 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <8D2E1B5A-6DD1-49E3-8F55-B3B816449FFB@xcllnt.net> References: <9FEBC10C-C453-41BE-8829-34E830585E90@xcllnt.net> <4835.1350062021@critter.freebsd.dk> <5082F0F3.1070102@rice.edu> To: Alan Cox X-Mailer: Apple Mail (2.1499) Cc: Poul-Henning Kamp , "freebsd-arch@freebsd.org Arch" , Tim LaBerge , Jason Evans X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 02:32:48 -0000 On Oct 20, 2012, at 11:44 AM, Alan Cox wrote: >=20 >> Also, moving the complexity of exactly which hint to give the >> kernel under different scenarios isn't that appealing at all. >> It just doesn't scale. >=20 >=20 > I think that you're being a bit too pessimistic here. If your use = case really corresponds to "this memory is free and will not be reused = (or reallocated for a very long time)", then that is qualitatively very = different from the way malloc(3) uses MADV_FREE. malloc(3)'s use of = MADV_FREE is highly speculative. It doesn't really know what the = application is going to do in the future. I don't think that having two = distinct hints that distinguish between "speculative" and = "non-speculative" uses would be problematic. The distinction is real = and also easy to explain. The only danger is that application writers = really don't understand their application and use the wrong hint. Maybe. I need to think about this. On the surface it's hard to belief that any allocator can reliably predict the future, so all hints are speculative in that sense. I do buy into the fact that malloc(3) has no a priori knowledge of the behavior of an application and an application with a special-purpose allocator has an allocator with more knowledge of the behavior. I'm just not sure this warrants different hints. I agree that the more complicated the hints the more likely they are not being used at all or they are used the wrong way. >=20 >> ... If some VM changes warrant a new hint >> to madvise(), you may end up changing multiple daemons. It >> seems better to have just 1 hint (i.e. MADV_FREE) and have the >> kernel change its behaviour depending on the situation. When >> there's plenty of memory, you may even ignore the hint. Under >> severe memory pressure you may want to free up the page right >> away so that you can give it to some thread that's waiting >> for a page. >=20 >=20 > How is this really different from the existing behavior? If a thread = is waiting for a page, then the page daemon is running. In particular, = it is moving pages from the head of the inactive queue, where they were = placed by MADV_FREE, to the cache/free queue and waking up the waiting = thread when the aggregate cache/free target is met. What we see with FreeBSD 6.1 is that memory remains inactive indefinitely. If the behaviour has changed in more recent versions, then we'll reap the benefits soon. If not, then we (=3D Juniper) may want to look into this. >> At the edge of needing to swap, complex algorithms >> may be worthwhile -- or maybe not. I don't know. >>=20 >> This leads to: >> 1. Keep MADV_FREE as it behaves in FreeBSD right now or make >> it even more sloppy. >=20 >=20 > I'm not sure that I understand what you mean by "sloppy" here. Can = you elaborate? It's just a sloppy way of saying that the hint can be ignored altogether or that we simply mark the page as clean and not do anything else. The point was mostly that the performance argument is more important. > 2. Have an idle thread that moves inactive pages to the cache >> or free queue if they've been inactive for X minutes, for >> some tunable X. Have it back off when the pageout daemon >> kicks in. >=20 >=20 > The existing page daemon already wakes up periodically and looks = around for something to do. In particular, have a look at = vm_pageout_page_stats(). That function tries to do something analogous = to what you propose. In part, it tries to prevent munmap(2)ed = file-backed pages from getting stuck in the active queue. I'll take a look. That's good to know. \begin{disclaimer} Juniper's problem is being stuck with an obsolete version of FreeBSD and we're likely to look for solutions to problems that don't exist anymore in recent versions. Just bear with us for a while :-) \end{disclaimer} --=20 Marcel Moolenaar marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Sun Oct 21 02:33:06 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5A67281C; Sun, 21 Oct 2012 02:33:06 +0000 (UTC) (envelope-from marcel@xcllnt.net) Received: from mail.xcllnt.net (mail.xcllnt.net [70.36.220.4]) by mx1.freebsd.org (Postfix) with ESMTP id 2EB048FC0A; Sun, 21 Oct 2012 02:33:06 +0000 (UTC) Received: from marcelm-sslvpn-nc.jnpr.net (natint3.juniper.net [66.129.224.36]) (authenticated bits=0) by mail.xcllnt.net (8.14.5/8.14.5) with ESMTP id q9L2Wi1x035388 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 20 Oct 2012 19:33:05 -0700 (PDT) (envelope-from marcel@xcllnt.net) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Behavior of madvise(MADV_FREE) From: Marcel Moolenaar In-Reply-To: <508305A6.4090106@rice.edu> Date: Sat, 20 Oct 2012 19:33:39 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <9FEBC10C-C453-41BE-8829-34E830585E90@xcllnt.net> <4835.1350062021@critter.freebsd.dk> <5082F0F3.1070102@rice.edu> <92742.1350761696@critter.freebsd.dk> <508305A6.4090106@rice.edu> To: Alan Cox X-Mailer: Apple Mail (2.1499) Cc: Jason Evans , Poul-Henning Kamp , Tim LaBerge , "freebsd-arch@freebsd.org Arch" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 02:33:06 -0000 On Oct 20, 2012, at 1:12 PM, Alan Cox wrote: > On 10/20/2012 14:34, Poul-Henning Kamp wrote: >> -------- >> In message<5082F0F3.1070102@rice.edu>, Alan Cox writes: >>=20 >>> I'm sympathetic. Once upon a time, I was often called upon to = explain >>> to network administrators why their idle web cache didn't have = oodles of >>> "free" memory and how this wasn't a problem. >> You too ? :-) >>=20 >>> I think that you're being a bit too pessimistic here. If your use = case >>> really corresponds to "this memory is free and will not be reused = (or >>> reallocated for a very long time)" >> Which brings me to a question I have wondered: Why not simply >> munmap(2) it until you need it again ? >>=20 >=20 > My recollection is that Marcel said that the memory was acquired via = sbrk(2). Correct. --=20 Marcel Moolenaar marcel@xcllnt.net From owner-freebsd-arch@FreeBSD.ORG Sun Oct 21 02:47:33 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6422FE13 for ; Sun, 21 Oct 2012 02:47:33 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 142D28FC0C for ; Sun, 21 Oct 2012 02:47:32 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id q9L2lQZP005178 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 20 Oct 2012 19:47:26 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id q9L2lQrb005177; Sat, 20 Oct 2012 19:47:26 -0700 (PDT) (envelope-from jmg) Date: Sat, 20 Oct 2012 19:47:26 -0700 From: John-Mark Gurney To: Peter Wemm Subject: Re: using SSE2 in kernel C code (improving AES-NI module) Message-ID: <20121021024726.GA1563@funkthat.com> Mail-Followup-To: Peter Wemm , Konstantin Belousov , freebsd-arch@freebsd.org References: <20121019233833.GS1967@funkthat.com> <20121020054847.GB35915@deviant.kiev.zoral.com.ua> <20121020171124.GU1967@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sat, 20 Oct 2012 19:47:26 -0700 (PDT) Cc: Konstantin Belousov , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 02:47:33 -0000 Peter Wemm wrote this message on Sat, Oct 20, 2012 at 11:10 -0700: > Or, another option.. do something like genassym or the many other > kernel build tools. aicasm builds and runs a userland tool to > generate something to build into the kernel. With sufficient > cross-contamination safeguards I wonder if something similar might be > able to be done here. Well, looks like I may this working... Turns out I can't name the file .s otherwise config puts it in SFILES which causes all sorts of problems.. So, I went w/ .nos, does any one else have any suggestions? how does this look to people: aesni_wrap2.nos optional aesni \ dependency "$S/crypto/aesni/aesni_wrap2.c" \ compile-with "${CC} -O3 -fPIC -S -o aesni_wrap2.nos $S/crypto/aesni/aesni_wrap2.c" \ no-obj no-implicit-rule before-depend \ clean "aesni_wrap2.nos" aesni_wrap2.o optional aesni \ dependency "aesni_wrap2.nos" \ compile-with "${NORMAL_S} aesni_wrap2.nos" \ no-implicit-rule \ clean "aesni_wrap2.o" We'll have to do something similar in the module Makefile, but that is easier... Also, I thought we had a better way to note that some devices depend upon others than just throwing a depend error... If you include aesni w/o crypto, you get error about missing cryptodev_if.h... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Sun Oct 21 04:43:07 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8EF466DB for ; Sun, 21 Oct 2012 04:43:07 +0000 (UTC) (envelope-from meowthink@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 6A6EA8FC16 for ; Sun, 21 Oct 2012 04:43:07 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1TPnNM-0003fm-Rn for freebsd-arch@freebsd.org; Sat, 20 Oct 2012 21:43:00 -0700 Date: Sat, 20 Oct 2012 21:43:00 -0700 (PDT) From: meowthink To: freebsd-arch@freebsd.org Message-ID: <1350794580507-5753746.post@n5.nabble.com> In-Reply-To: <20121008175206.5961858094@chaos.jnpr.net> References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121008175206.5961858094@chaos.jnpr.net> Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 04:43:07 -0000 I encouraged the same issue when build with dtrace-enabled kernconf. I do think it's a problem of Makefile.inc1 which manually defined these NO_CTFs. By Simon's hack on bsd.own.mk, the build process goes on, but not as before. For instance, /libexec/ld-elf.so.1 contains CTF data but /libexec/ld-elf32.so.1 not (it contains CTF data before MFC r228158). At least all 32-bit libraries on amd64 are built without CTF. Though didn't use userland dtrace(esp. with 32-bit compat), I doubt this somehow breaks such functionalities. Also cc'ed jhb & fjoe who brought us this change. Regards, Meowthink -- View this message in context: http://freebsd.1045724.n5.nabble.com/bsd-own-mk-just-let-WITHOUT-take-precedence-tp5749766p5753746.html Sent from the freebsd-arch mailing list archive at Nabble.com. From owner-freebsd-arch@FreeBSD.ORG Sun Oct 21 06:10:24 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 71E5A2D7 for ; Sun, 21 Oct 2012 06:10:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 088308FC12 for ; Sun, 21 Oct 2012 06:10:22 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q9L6AOgh064105; Sun, 21 Oct 2012 09:10:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q9L6AC5a062979; Sun, 21 Oct 2012 09:10:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q9L6ABd7062973; Sun, 21 Oct 2012 09:10:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 21 Oct 2012 09:10:11 +0300 From: Konstantin Belousov To: Peter Wemm , Konstantin Belousov , freebsd-arch@freebsd.org Subject: Re: using SSE2 in kernel C code (improving AES-NI module) Message-ID: <20121021061011.GG35915@deviant.kiev.zoral.com.ua> References: <20121019233833.GS1967@funkthat.com> <20121020054847.GB35915@deviant.kiev.zoral.com.ua> <20121020171124.GU1967@funkthat.com> <20121021024726.GA1563@funkthat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="m8pCv11XKdWQJ6wE" Content-Disposition: inline In-Reply-To: <20121021024726.GA1563@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 06:10:24 -0000 --m8pCv11XKdWQJ6wE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Oct 20, 2012 at 07:47:26PM -0700, John-Mark Gurney wrote: > Peter Wemm wrote this message on Sat, Oct 20, 2012 at 11:10 -0700: > > Or, another option.. do something like genassym or the many other > > kernel build tools. aicasm builds and runs a userland tool to > > generate something to build into the kernel. With sufficient > > cross-contamination safeguards I wonder if something similar might be > > able to be done here. >=20 > Well, looks like I may this working... Turns out I can't name the file > .s otherwise config puts it in SFILES which causes all sorts of problems.. > So, I went w/ .nos, does any one else have any suggestions? >=20 > how does this look to people: > aesni_wrap2.nos optional aesni = \ > dependency "$S/crypto/aesni/aesni_wrap2.c" = \ > compile-with "${CC} -O3 -fPIC -S -o aesni_wrap2.nos $S/crypto/= aesni/aesni_wrap2.c" \ =20 > no-obj no-implicit-rule before-depend = \ > clean "aesni_wrap2.nos" > aesni_wrap2.o optional aesni = \ > dependency "aesni_wrap2.nos" = \ > compile-with "${NORMAL_S} aesni_wrap2.nos" = \ > no-implicit-rule = \ > clean "aesni_wrap2.o" >=20 > We'll have to do something similar in the module Makefile, but that is > easier... >=20 > Also, I thought we had a better way to note that some devices depend > upon others than just throwing a depend error... If you include aesni > w/o crypto, you get error about missing cryptodev_if.h... >=20 Hm, if such thing is possible, why do you need to compile through the =2ES at all ? All you need is to specify the special compiling flags, including -msse and -msse2. Note, you shall not need -fPIC, at least for amd64. I would suggest to use -O2, as well as to try to honour the -g settings. Most likely, you can put the ${CFLAGS} on the command line, followed by -msse -msse2. --m8pCv11XKdWQJ6wE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlCDkcMACgkQC3+MBN1Mb4ijFQCgutJynPglzNMBaAWsGeHQlQ9D o8AAn1bRvECRuO0K4Gvrs1rmpCd4COda =OA4o -----END PGP SIGNATURE----- --m8pCv11XKdWQJ6wE-- From owner-freebsd-arch@FreeBSD.ORG Sun Oct 21 09:41:58 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E5DB39E for ; Sun, 21 Oct 2012 09:41:58 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id AABC98FC0C for ; Sun, 21 Oct 2012 09:41:57 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id q9L9fpC0070293; Sun, 21 Oct 2012 04:41:51 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id q9L9fprq070292; Sun, 21 Oct 2012 04:41:51 -0500 (CDT) (envelope-from brooks) Date: Sun, 21 Oct 2012 04:41:51 -0500 From: Brooks Davis To: meowthink Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Message-ID: <20121021094151.GC27212@lor.one-eyed-alien.net> References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121008175206.5961858094@chaos.jnpr.net> <1350794580507-5753746.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="R+My9LyyhiUvIEro" Content-Disposition: inline In-Reply-To: <1350794580507-5753746.post@n5.nabble.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Oct 2012 09:41:58 -0000 --R+My9LyyhiUvIEro Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Oct 20, 2012 at 09:43:00PM -0700, meowthink wrote: > I encouraged the same issue when build with dtrace-enabled kernconf. > I do think it's a problem of Makefile.inc1 which manually defined these > NO_CTFs. By Simon's hack on bsd.own.mk, the build process goes on, but not > as before. For instance, /libexec/ld-elf.so.1 contains CTF data but > /libexec/ld-elf32.so.1 not (it contains CTF data before MFC r228158). At > least all 32-bit libraries on amd64 are built without CTF. > Though didn't use userland dtrace(esp. with 32-bit compat), I doubt this > somehow breaks such functionalities. >=20 > Also cc'ed jhb & fjoe who brought us this change. I've MFCd r228120 which allows WITH_CTF to be set in make.conf. It looks like there may be another bug effecting the setting of CTFCONVERT_CMD. -- Brooks --R+My9LyyhiUvIEro Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQg8NeXY6L6fI4GtQRAu+2AKDDtyEGfA2ljw6KNlJ1rB4pXekjrQCeKFSJ YC+05Rn74jH0shjKBfhx18w= =LGc9 -----END PGP SIGNATURE----- --R+My9LyyhiUvIEro-- From owner-freebsd-arch@FreeBSD.ORG Mon Oct 22 14:37:42 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 15510EF6; Mon, 22 Oct 2012 14:37:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id DBBBF8FC17; Mon, 22 Oct 2012 14:37:41 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3EDE3B924; Mon, 22 Oct 2012 10:37:41 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Date: Mon, 22 Oct 2012 09:40:39 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <20121007001423.9878F58094@chaos.jnpr.net> <1350794580507-5753746.post@n5.nabble.com> <20121021094151.GC27212@lor.one-eyed-alien.net> In-Reply-To: <20121021094151.GC27212@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201210220940.39281.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 22 Oct 2012 10:37:41 -0400 (EDT) Cc: meowthink , Brooks Davis X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 14:37:42 -0000 On Sunday, October 21, 2012 5:41:51 am Brooks Davis wrote: > On Sat, Oct 20, 2012 at 09:43:00PM -0700, meowthink wrote: > > I encouraged the same issue when build with dtrace-enabled kernconf. > > I do think it's a problem of Makefile.inc1 which manually defined these > > NO_CTFs. By Simon's hack on bsd.own.mk, the build process goes on, but not > > as before. For instance, /libexec/ld-elf.so.1 contains CTF data but > > /libexec/ld-elf32.so.1 not (it contains CTF data before MFC r228158). At > > least all 32-bit libraries on amd64 are built without CTF. > > Though didn't use userland dtrace(esp. with 32-bit compat), I doubt this > > somehow breaks such functionalities. > > > > Also cc'ed jhb & fjoe who brought us this change. > > I've MFCd r228120 which allows WITH_CTF to be set in make.conf. > It looks like there may be another bug effecting the setting of > CTFCONVERT_CMD. I had this change as a question mark to merge in my list, thanks. Can you also merge it to 8? -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Mon Oct 22 15:20:53 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D75D9AD0; Mon, 22 Oct 2012 15:20:53 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id 9FC378FC0A; Mon, 22 Oct 2012 15:20:53 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 888204C03EA; Mon, 22 Oct 2012 10:20:52 -0500 (CDT) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 86E564C03AC; Mon, 22 Oct 2012 10:20:52 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id Qml3ogMGiWEC; Mon, 22 Oct 2012 10:20:52 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id E9D854C0274; Mon, 22 Oct 2012 10:20:51 -0500 (CDT) Message-ID: <50856452.40902@rice.edu> Date: Mon, 22 Oct 2012 10:20:50 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: Marcel Moolenaar Subject: Re: Behavior of madvise(MADV_FREE) References: <9FEBC10C-C453-41BE-8829-34E830585E90@xcllnt.net> <4835.1350062021@critter.freebsd.dk> <5082F0F3.1070102@rice.edu> <8D2E1B5A-6DD1-49E3-8F55-B3B816449FFB@xcllnt.net> In-Reply-To: <8D2E1B5A-6DD1-49E3-8F55-B3B816449FFB@xcllnt.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Poul-Henning Kamp , "freebsd-arch@freebsd.org Arch" , Tim LaBerge , Jason Evans X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 15:20:53 -0000 On 10/20/2012 21:33, Marcel Moolenaar wrote: > On Oct 20, 2012, at 11:44 AM, Alan Cox wrote: >>> Also, moving the complexity of exactly which hint to give the >>> kernel under different scenarios isn't that appealing at all. >>> It just doesn't scale. >> >> I think that you're being a bit too pessimistic here. If your use case really corresponds to "this memory is free and will not be reused (or reallocated for a very long time)", then that is qualitatively very different from the way malloc(3) uses MADV_FREE. malloc(3)'s use of MADV_FREE is highly speculative. It doesn't really know what the application is going to do in the future. I don't think that having two distinct hints that distinguish between "speculative" and "non-speculative" uses would be problematic. The distinction is real and also easy to explain. The only danger is that application writers really don't understand their application and use the wrong hint. > Maybe. I need to think about this. On the surface it's hard to > belief that any allocator can reliably predict the future, so > all hints are speculative in that sense. I do buy into the fact > that malloc(3) has no a priori knowledge of the behavior of an > application and an application with a special-purpose allocator > has an allocator with more knowledge of the behavior. I'm just > not sure this warrants different hints. > > I agree that the more complicated the hints the more likely they > are not being used at all or they are used the wrong way. > >>> ... If some VM changes warrant a new hint >>> to madvise(), you may end up changing multiple daemons. It >>> seems better to have just 1 hint (i.e. MADV_FREE) and have the >>> kernel change its behaviour depending on the situation. When >>> there's plenty of memory, you may even ignore the hint. Under >>> severe memory pressure you may want to free up the page right >>> away so that you can give it to some thread that's waiting >>> for a page. >> >> How is this really different from the existing behavior? If a thread is waiting for a page, then the page daemon is running. In particular, it is moving pages from the head of the inactive queue, where they were placed by MADV_FREE, to the cache/free queue and waking up the waiting thread when the aggregate cache/free target is met. > What we see with FreeBSD 6.1 is that memory remains inactive > indefinitely. If the behaviour has changed in more recent > versions, then we'll reap the benefits soon. If not, then we > (= Juniper) may want to look into this. You might have a look at the commit message for svn revision 172317. It describes a problem that existed in FreeBSD from 5.x up until 7.0, when the physical memory allocator was replaced. Effectively, the page daemon could wind up spinning its wheels. However, one thing hasn't changed we only move pages from the inactive queue to the cache queue as required to meet the aggregate target on cache and free pages. If the demand for memory is static, we won't move pages from the inactive queue to the cache queue. > >>> At the edge of needing to swap, complex algorithms >>> may be worthwhile -- or maybe not. I don't know. >>> >>> This leads to: >>> 1. Keep MADV_FREE as it behaves in FreeBSD right now or make >>> it even more sloppy. >> >> I'm not sure that I understand what you mean by "sloppy" here. Can you elaborate? > It's just a sloppy way of saying that the hint can be ignored > altogether or that we simply mark the page as clean and not > do anything else. The point was mostly that the performance > argument is more important. > > >> 2. Have an idle thread that moves inactive pages to the cache >>> or free queue if they've been inactive for X minutes, for >>> some tunable X. Have it back off when the pageout daemon >>> kicks in. >> >> The existing page daemon already wakes up periodically and looks around for something to do. In particular, have a look at vm_pageout_page_stats(). That function tries to do something analogous to what you propose. In part, it tries to prevent munmap(2)ed file-backed pages from getting stuck in the active queue. > I'll take a look. That's good to know. > > \begin{disclaimer} > Juniper's problem is being stuck with an obsolete version of > FreeBSD and we're likely to look for solutions to problems that > don't exist anymore in recent versions. Just bear with us for > a while :-) > \end{disclaimer} > From owner-freebsd-arch@FreeBSD.ORG Mon Oct 22 19:39:04 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 664) id 87ADEAE5; Mon, 22 Oct 2012 19:39:04 +0000 (UTC) Date: Mon, 22 Oct 2012 12:39:04 -0700 From: David O'Brien To: Brooks Davis Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Message-ID: <20121022193903.GA88336@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, Brooks Davis , Simon Gerraty , freebsd-arch@freebsd.org References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121008154853.GC23400@lor.one-eyed-alien.net> X-Operating-System: FreeBSD 10.0-CURRENT X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-arch@freebsd.org, Simon Gerraty X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: obrien@freebsd.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 19:39:04 -0000 On Mon, Oct 08, 2012 at 10:48:53AM -0500, Brooks Davis wrote: > On Sat, Oct 06, 2012 at 05:14:23PM -0700, Simon Gerraty wrote: > > After being bitten by: > > make: "/b/sjg/work/fbsd-head/src/share/mk/bsd.own.mk" line 490: WITH_CTF > > and WITHOUT_CTF can't both be set. ... > I'm not sure if I agree or not, I'll have to think more. This sort of > thing that leads to me yelling at my computer "but I @#%$@# set > WITH_FOO you ^@$@! machine." :) Brooks, Isn't that what some folks are currently doing trying to build a fully DTrace ready system? Have you had a chance to review Simon's latest diff? This is the only build knob I'm aware of where setting it manually in the environment doesn't work the same as setting it in /etc/{src,make}.conf. Its been too hard to build a fully DTrace-ready FreeBSD for a long time. We really need closure on this -- DTrace is too useful. Note that our Handbook () still has: &prompt.root; cd /usr/src &prompt.root; make WITH_CTF=1 kernel We really need to make this as easy as possible for users and get the docs matching reality. -- -- David (obrien@FreeBSD.org) From owner-freebsd-arch@FreeBSD.ORG Mon Oct 22 23:49:41 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F05C1DBF for ; Mon, 22 Oct 2012 23:49:40 +0000 (UTC) (envelope-from nancy@nm-led.com) Received: from m14-111.vip.163.com (m14-111.vip.163.com [220.181.14.111]) by mx1.freebsd.org (Postfix) with ESMTP id D3E7C8FC16 for ; Mon, 22 Oct 2012 23:49:38 +0000 (UTC) Received: from lenovo-o7fsbapc (unknown [183.14.72.223]) by smtp5 (Coremail) with SMTP id huCowEDJ70OP24VQdJIBAA--.1901S2; Tue, 23 Oct 2012 07:49:35 +0800 (CST) Date: Tue, 23 Oct 2012 07:49:25 +0800 (CST) From: Nancy from Norming To: freebsd-arch@FreeBSD.org Message-ID: <12966890.19157.1350949765936.JavaMail.SYSTEM@lenovo-o7fsbapc> Subject: Dear customer , unique led lighting is coming ! Low Carbon , From here ...... MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_19155_14958674.1350949765905" X-Priority: 3 X-mailer: javamail@entsoft X-CM-TRANSID: huCowEDJ70OP24VQdJIBAA--.1901S2 X-Coremail-Antispam: 1Uf129KBjvJXoWrKF1rKrW3Kw18Wr1xWryrCrg_yoW8JrW7pa yrXw48KrZrC3yag34qyw4jgr1Fv34ktayUWr95GrZIyFs0gFyavF13Kw4UXrykXrWkAw1v qw4Yy34Ska4qk3DanT9S1TB71UUUjFJqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07bpjgxUUUUU= X-CM-SenderInfo: 5qdqu5o6qpgz1hgou0bp/1tbiDhBQ+FCA+n8QRQAAs2 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 23:49:41 -0000 This is a multi-part message in MIME format. ------=_Part_19155_14958674.1350949765905 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: base64 RGVhciBFeGNlbGxlbnQgY3VzdG9tZXIsDQoNClRoZSB1cGdyYWRlZCA1VyBsZWQgc3BvdCBmZWF0 dXJlcyBhcmUgYXMgZm9sbG93czogMS4gVGhlIGFwcGVhcmFuY2UgaXMgdGhlIHNhbWUgYXMgdGhl IGhhbG9nZW4gbGFtcC4gMi4gQWRvcHQgdGhlIGxhdGVzdCBuZXcgdGVjaG5vbG9neSwgaXQgaW5j bHVkZXMgYQ0KcmVmbGVjdG9yIHdoaWNoIGluY3JlYXNlIHRoZSBicmlnaHRuZXNzIGJ5IDMwJS4g My4gTWFjaGluZWQgcHVyZSBhbHVtaW51bSB3aXRoIHdoaXRlLWNvYXRlZCB0aGVybWFsIHJhZGlh dGlvbiBmb3IgYmV0dGVyIGhlYXQgZGlzc2lwYXRpb24sDQp0aGVybWFsIHBlcmZvcm1hbmNlIGlu Y3JlYXNlZCA1MCUuIDQuIDEwMCUgcmVwbGFjZSAzNVcgdHJhZGl0aW9uYWwgYnVsYiBkaXJlY3Rs eS4gV2FybVdoaXRlIDI3MDBLLCBDb29sIFdoaXRlIDUwMDBLLiA1LiBMb3ctdm9sdGFnZSBkaW1t aW5nLA0KcGVyZmVjdCBkaW1taW5nIHBlcmZvcm1hbmNlOiBmcm9tIDAtMTAwJS4NCg0KQ0UgYW5k IFJvSFMgYXBwcm92ZWQgLDIgeWVhcnMgcXVhbGl0eSB3YXJyYW50eS5HVTEwL01SMTYvRTI3L0Iy MiBhdmFpbGFibGUgLkZvbGxvd3MgYXJlIHJlbGV2YW50IHBob3RvcyBmb3Igc3R1ZHk6DQoNCg0K DQoNCg0KVmVyeSBuaWNlIGFuZCBoaWdoIHF1YWxpdHkgUGF0ZW50IHByb2R1Y3QNCg0KDQpJZiB5 b3Ugd2FudCB0byBvcmRlciBzb21lIHNhbXBsZXMgZm9yIGV2YWx1YXRpb24gLGtpbmRseSBwbGVh c2UgbGV0IG1lIGtub3cuIEkgYW0gbG9va2luZyBmb3J3YXJkIHRvIHlvdXIgZWFybHkgcmVwbHkg Lg0KDQpCZXN0IHJlZ2FyZHMgdG8geW91IGFuZCB5b3VyIGZhbWlseSAuDQoNCllvdXJzIE5hbmN5 IA0KDQoNClNoZW56aGVuIE5vcm1pbmcgTGlnaHRpbmcgQ28uLCBMdGQgDQoNCk5hbmN5IFpoYW5n DQpNb2I6ICs4Ni03NTUtMTU4MjA3NzA0ODMgICAgICAgICAgICAgICAgICAgICBUZWw6ICs4Ni03 NTUtMjc2MTUwNTQgICAgICAgICAgICAgICAgICAgICAgRmF4OiArODYtNzU1LTI3NjE1ODcwDQpF LW1haWw6bmFuY3lAbm9ybWluZ2xpZ2h0aW5nLmNvbSAgICAgICAgTVNOOiB6aGFuZy04NDhAaG90 bWFpbC5jb20gICAgICAgIFNreXBlOiBuYW5jeTE4ODk5MDgNCldlYjogd3d3Lm5vcm1pbmdsaWdo dGluZy5jb20gICAgICAgICAgICAgICBPbmxpbmUgU2hvd3Jvb206IHd3dy5ub3JtaW5nLmVuLmFs aWJhYmEuY29tDQpBREQuOiA0IEZsb29yLCBCdWlsZGluZyBCLCBIZW5na2VuZyAxc3QgSW5kdXN0 cmlhbCBwYXJrLCBTaGl5YW4gVG93biwgQmFvwrQgQW4gRGlzdHJpY3QsIA0KU2hlbnpoZW4sIFAu Ui4gQ2hpbmE= ------=_Part_19155_14958674.1350949765905-- From owner-freebsd-arch@FreeBSD.ORG Tue Oct 23 07:04:25 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4DC016C8 for ; Tue, 23 Oct 2012 07:04:25 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 1B82F8FC0C for ; Tue, 23 Oct 2012 07:04:24 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id q9N74Irg067870 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Oct 2012 00:04:18 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id q9N74HcS067869; Tue, 23 Oct 2012 00:04:17 -0700 (PDT) (envelope-from jmg) Date: Tue, 23 Oct 2012 00:04:17 -0700 From: John-Mark Gurney To: Konstantin Belousov Subject: Re: using SSE2 in kernel C code (improving AES-NI module) Message-ID: <20121023070417.GD1563@funkthat.com> Mail-Followup-To: Konstantin Belousov , Peter Wemm , freebsd-arch@freebsd.org References: <20121019233833.GS1967@funkthat.com> <20121020054847.GB35915@deviant.kiev.zoral.com.ua> <20121020171124.GU1967@funkthat.com> <20121021024726.GA1563@funkthat.com> <20121021061011.GG35915@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121021061011.GG35915@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Tue, 23 Oct 2012 00:04:18 -0700 (PDT) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 07:04:25 -0000 Konstantin Belousov wrote this message on Sun, Oct 21, 2012 at 09:10 +0300: > On Sat, Oct 20, 2012 at 07:47:26PM -0700, John-Mark Gurney wrote: > > Peter Wemm wrote this message on Sat, Oct 20, 2012 at 11:10 -0700: > > > Or, another option.. do something like genassym or the many other > > > kernel build tools. aicasm builds and runs a userland tool to > > > generate something to build into the kernel. With sufficient > > > cross-contamination safeguards I wonder if something similar might be > > > able to be done here. > > > > Well, looks like I may this working... Turns out I can't name the file > > .s otherwise config puts it in SFILES which causes all sorts of problems.. > > So, I went w/ .nos, does any one else have any suggestions? > > > > how does this look to people: > > aesni_wrap2.nos optional aesni \ > > dependency "$S/crypto/aesni/aesni_wrap2.c" \ > > compile-with "${CC} -O3 -fPIC -S -o aesni_wrap2.nos $S/crypto/aesni/aesni_wrap2.c" \ > > no-obj no-implicit-rule before-depend \ > > clean "aesni_wrap2.nos" > > aesni_wrap2.o optional aesni \ > > dependency "aesni_wrap2.nos" \ > > compile-with "${NORMAL_S} aesni_wrap2.nos" \ > > no-implicit-rule \ > > clean "aesni_wrap2.o" > > > > We'll have to do something similar in the module Makefile, but that is > > easier... > > > > Also, I thought we had a better way to note that some devices depend > > upon others than just throwing a depend error... If you include aesni > > w/o crypto, you get error about missing cryptodev_if.h... > > > Hm, if such thing is possible, why do you need to compile through the > .S at all ? All you need is to specify the special compiling flags, > including -msse and -msse2. Thanks, I managed to get it down to one... > Note, you shall not need -fPIC, at least for amd64. I would suggest to use > -O2, as well as to try to honour the -g settings. If I don't do -fpic I get: aesni_wrap2.o:(.eh_frame+0x20): relocation truncated to fit: R_X86_64_32 against `.text' when linking the kernel... If you can explain to me how to get rid of this error, I'll do it.. > Most likely, you can put the ${CFLAGS} on the command line, followed > by -msse -msse2. I can't use CFLAGS because it removes access to the xmmintrin.h header file... It looks like an option is to use: -fpic ${OPTFLAGS:C/^-O2$/-O3/} ${DEBUG} In my testing, -O2 is significantly slower, hence the bump to -O3: x O2.txt + O3.txt N Min Max Median Avg Stddev x 20 1741.3491 1754.987 1752.9267 1751.5602 3.5616947 + 20 2223.217 2244.4501 2242.7028 2240.3183 5.7020691 Difference at 95.0% confidence 488.758 +/- 3.04271 27.9042% +/- 0.173715% (Student's t, pooled s = 4.75391) Those are MB/sec... Index: files.amd64 =================================================================== --- files.amd64 (revision 241041) +++ files.amd64 (working copy) @@ -137,6 +137,11 @@ crypto/aesni/aeskeys_amd64.S optional aesni crypto/aesni/aesni.c optional aesni crypto/aesni/aesni_wrap.c optional aesni +aesni_wrap2.o optional aesni \ + dependency "$S/crypto/aesni/aesni_wrap2.c" \ + compile-with "${CC} -c -fpic ${COPTFLAGS:C/^-O2$/-O3/} ${DEBUG} -o aesni_wrap2.o $S/crypto/aesni/aesni_wrap2.c" \ + no-implicit-rule \ + clean "aesni_wrap2.o" crypto/blowfish/bf_enc.c optional crypto | ipsec crypto/des/des_enc.c optional crypto | ipsec | netsmb crypto/via/padlock.c optional padlock I still need to fix up i386, and will let people review a full patch to address both arches before committing... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Tue Oct 23 08:47:53 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BEB39741 for ; Tue, 23 Oct 2012 08:47:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 041718FC0A for ; Tue, 23 Oct 2012 08:47:51 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q9N8ltEl021628; Tue, 23 Oct 2012 11:47:55 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q9N8lhxP021891; Tue, 23 Oct 2012 11:47:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q9N8lh4T021890; Tue, 23 Oct 2012 11:47:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 23 Oct 2012 11:47:43 +0300 From: Konstantin Belousov To: Peter Wemm , freebsd-arch@freebsd.org Subject: Re: using SSE2 in kernel C code (improving AES-NI module) Message-ID: <20121023084743.GQ35915@deviant.kiev.zoral.com.ua> References: <20121019233833.GS1967@funkthat.com> <20121020054847.GB35915@deviant.kiev.zoral.com.ua> <20121020171124.GU1967@funkthat.com> <20121021024726.GA1563@funkthat.com> <20121021061011.GG35915@deviant.kiev.zoral.com.ua> <20121023070417.GD1563@funkthat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1D9gOFySlrGasJlk" Content-Disposition: inline In-Reply-To: <20121023070417.GD1563@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 08:47:54 -0000 --1D9gOFySlrGasJlk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 23, 2012 at 12:04:17AM -0700, John-Mark Gurney wrote: > Konstantin Belousov wrote this message on Sun, Oct 21, 2012 at 09:10 +030= 0: > > On Sat, Oct 20, 2012 at 07:47:26PM -0700, John-Mark Gurney wrote: > > > Peter Wemm wrote this message on Sat, Oct 20, 2012 at 11:10 -0700: > > > > Or, another option.. do something like genassym or the many other > > > > kernel build tools. aicasm builds and runs a userland tool to > > > > generate something to build into the kernel. With sufficient > > > > cross-contamination safeguards I wonder if something similar might = be > > > > able to be done here. > > >=20 > > > Well, looks like I may this working... Turns out I can't name the fi= le > > > .s otherwise config puts it in SFILES which causes all sorts of probl= ems.. > > > So, I went w/ .nos, does any one else have any suggestions? > > >=20 > > > how does this look to people: > > > aesni_wrap2.nos optional aesni = \ > > > dependency "$S/crypto/aesni/aesni_wrap2.c" = \ > > > compile-with "${CC} -O3 -fPIC -S -o aesni_wrap2.nos $S/cry= pto/aesni/aesni_wrap2.c" \ =20 > > > no-obj no-implicit-rule before-depend = \ > > > clean "aesni_wrap2.nos" > > > aesni_wrap2.o optional aesni = \ > > > dependency "aesni_wrap2.nos" = \ > > > compile-with "${NORMAL_S} aesni_wrap2.nos" = \ > > > no-implicit-rule = \ > > > clean "aesni_wrap2.o" > > >=20 > > > We'll have to do something similar in the module Makefile, but that is > > > easier... > > >=20 > > > Also, I thought we had a better way to note that some devices depend > > > upon others than just throwing a depend error... If you include aesni > > > w/o crypto, you get error about missing cryptodev_if.h... > > >=20 > > Hm, if such thing is possible, why do you need to compile through the > > .S at all ? All you need is to specify the special compiling flags, > > including -msse and -msse2. >=20 > Thanks, I managed to get it down to one... >=20 > > Note, you shall not need -fPIC, at least for amd64. I would suggest to = use > > -O2, as well as to try to honour the -g settings. >=20 > If I don't do -fpic I get: > aesni_wrap2.o:(.eh_frame+0x20): relocation truncated to fit: R_X86_64_32 = against `.text' >=20 > when linking the kernel... If you can explain to me how to get rid of > this error, I'll do it.. Yes, because you need -mcmodel=3Dkernel on amd64, but -fPIC on i386. This is why I suggested to use CFLAGS, which takes care of it in single place. It would be huge PITA to duplicate the kernel compilation flag for arch in some obscure place. The best would be to edit the CFLAGS in place, if possible (I do not know make to judge). Second possible way is to add some var like CFLAGS_SSE to centralized place. >=20 > > Most likely, you can put the ${CFLAGS} on the command line, followed > > by -msse -msse2. >=20 > I can't use CFLAGS because it removes access to the xmmintrin.h header > file... It looks like an option is to use: > -fpic ${OPTFLAGS:C/^-O2$/-O3/} ${DEBUG} >=20 > In my testing, -O2 is significantly slower, hence the bump to -O3: > x O2.txt > + O3.txt > N Min Max Median Avg Stdd= ev > x 20 1741.3491 1754.987 1752.9267 1751.5602 3.56169= 47 > + 20 2223.217 2244.4501 2242.7028 2240.3183 5.70206= 91 > Difference at 95.0% confidence > 488.758 +/- 3.04271 > 27.9042% +/- 0.173715% > (Student's t, pooled s =3D 4.75391) >=20 > Those are MB/sec... I think that -O3 compile output have to be validated manually, due to high-risk optimizations. Anyway, if it works there, great. >=20 > Index: files.amd64 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- files.amd64 (revision 241041) > +++ files.amd64 (working copy) > @@ -137,6 +137,11 @@ > crypto/aesni/aeskeys_amd64.S optional aesni > crypto/aesni/aesni.c optional aesni > crypto/aesni/aesni_wrap.c optional aesni > +aesni_wrap2.o optional aesni \ > + dependency "$S/crypto/aesni/aesni_wrap2.c" \ > + compile-with "${CC} -c -fpic ${COPTFLAGS:C/^-O2$/-O3/} ${DEBUG} -o a= esni_wrap2.o $S/crypto/aesni/aesni_wrap2.c" \ > + no-implicit-rule \ > + clean "aesni_wrap2.o" > crypto/blowfish/bf_enc.c optional crypto | ipsec=20 > crypto/des/des_enc.c optional crypto | ipsec | netsmb > crypto/via/padlock.c optional padlock >=20 >=20 > I still need to fix up i386, and will let people review a full patch > to address both arches before committing... >=20 > --=20 > John-Mark Gurney Voice: +1 415 225 5579 >=20 > "All that I will do, has been done, All that I have, has not." --1D9gOFySlrGasJlk Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlCGWa8ACgkQC3+MBN1Mb4jdlQCgp+rejsSxhHgcrXYHOtXtYXEs FzIAoO3Tar4TX1dsw4xfZaYhsVwlTuqZ =e4Jo -----END PGP SIGNATURE----- --1D9gOFySlrGasJlk-- From owner-freebsd-arch@FreeBSD.ORG Tue Oct 23 14:18:50 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 28520D71 for ; Tue, 23 Oct 2012 14:18:50 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id EB3B48FC0C for ; Tue, 23 Oct 2012 14:18:49 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 5FD13B91A; Tue, 23 Oct 2012 10:18:49 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: using SSE2 in kernel C code (improving AES-NI module) Date: Tue, 23 Oct 2012 08:34:20 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <20121019233833.GS1967@funkthat.com> <20121021061011.GG35915@deviant.kiev.zoral.com.ua> <20121023070417.GD1563@funkthat.com> In-Reply-To: <20121023070417.GD1563@funkthat.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201210230834.20167.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 23 Oct 2012 10:18:49 -0400 (EDT) Cc: Konstantin Belousov , John-Mark Gurney X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 14:18:50 -0000 On Tuesday, October 23, 2012 3:04:17 am John-Mark Gurney wrote: > Konstantin Belousov wrote this message on Sun, Oct 21, 2012 at 09:10 +0300: > > Most likely, you can put the ${CFLAGS} on the command line, followed > > by -msse -msse2. > > I can't use CFLAGS because it removes access to the xmmintrin.h header > file... It looks like an option is to use: > -fpic ${OPTFLAGS:C/^-O2$/-O3/} ${DEBUG} Err, shouldn't later options override new ones? If you have "${CFLAGS} -msse -msse2 -maes" does that not work? -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Tue Oct 23 14:54:26 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 968C7F8E; Tue, 23 Oct 2012 14:54:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 0947A8FC0C; Tue, 23 Oct 2012 14:54:25 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q9NEsURT054770; Tue, 23 Oct 2012 17:54:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q9NEsIG9024259; Tue, 23 Oct 2012 17:54:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q9NEsIcm024258; Tue, 23 Oct 2012 17:54:18 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 23 Oct 2012 17:54:18 +0300 From: Konstantin Belousov To: John Baldwin Subject: Re: using SSE2 in kernel C code (improving AES-NI module) Message-ID: <20121023145418.GV35915@deviant.kiev.zoral.com.ua> References: <20121019233833.GS1967@funkthat.com> <20121021061011.GG35915@deviant.kiev.zoral.com.ua> <20121023070417.GD1563@funkthat.com> <201210230834.20167.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2yGx+1UeQj0dgFI1" Content-Disposition: inline In-Reply-To: <201210230834.20167.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: John-Mark Gurney , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 14:54:26 -0000 --2yGx+1UeQj0dgFI1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 23, 2012 at 08:34:20AM -0400, John Baldwin wrote: > On Tuesday, October 23, 2012 3:04:17 am John-Mark Gurney wrote: > > Konstantin Belousov wrote this message on Sun, Oct 21, 2012 at 09:10 +0= 300: > > > Most likely, you can put the ${CFLAGS} on the command line, followed > > > by -msse -msse2. > >=20 > > I can't use CFLAGS because it removes access to the xmmintrin.h header > > file... It looks like an option is to use: > > -fpic ${OPTFLAGS:C/^-O2$/-O3/} ${DEBUG} >=20 > Err, shouldn't later options override new ones? If you have > "${CFLAGS} -msse -msse2 -maes" does that not work? =20 It is -nostdinc which cause issue for John-Mark. The header probably still shall not be used. --2yGx+1UeQj0dgFI1 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlCGr5oACgkQC3+MBN1Mb4gpcgCePjBgXPozAQaQ7YQ8Tx1Rb6SD wiQAoMV6x0Jri+jl97mnWX39mrYlN5sh =+FBy -----END PGP SIGNATURE----- --2yGx+1UeQj0dgFI1-- From owner-freebsd-arch@FreeBSD.ORG Tue Oct 23 20:35:19 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2F181958; Tue, 23 Oct 2012 20:35:19 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id 7BCFB8FC12; Tue, 23 Oct 2012 20:35:18 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id q9NKZCT2093204; Tue, 23 Oct 2012 15:35:12 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id q9NKZCws093203; Tue, 23 Oct 2012 15:35:12 -0500 (CDT) (envelope-from brooks) Date: Tue, 23 Oct 2012 15:35:12 -0500 From: Brooks Davis To: John Baldwin Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Message-ID: <20121023203512.GA85624@lor.one-eyed-alien.net> References: <20121007001423.9878F58094@chaos.jnpr.net> <1350794580507-5753746.post@n5.nabble.com> <20121021094151.GC27212@lor.one-eyed-alien.net> <201210220940.39281.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3V7upXqbjpZ4EhLz" Content-Disposition: inline In-Reply-To: <201210220940.39281.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: meowthink , Brooks Davis , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 20:35:19 -0000 --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 22, 2012 at 09:40:39AM -0400, John Baldwin wrote: > On Sunday, October 21, 2012 5:41:51 am Brooks Davis wrote: > > On Sat, Oct 20, 2012 at 09:43:00PM -0700, meowthink wrote: > > > I encouraged the same issue when build with dtrace-enabled kernconf. > > > I do think it's a problem of Makefile.inc1 which manually defined the= se > > > NO_CTFs. By Simon's hack on bsd.own.mk, the build process goes on, bu= t not > > > as before. For instance, /libexec/ld-elf.so.1 contains CTF data but > > > /libexec/ld-elf32.so.1 not (it contains CTF data before MFC r228158).= At > > > least all 32-bit libraries on amd64 are built without CTF. > > > Though didn't use userland dtrace(esp. with 32-bit compat), I doubt t= his > > > somehow breaks such functionalities. > > >=20 > > > Also cc'ed jhb & fjoe who brought us this change. > >=20 > > I've MFCd r228120 which allows WITH_CTF to be set in make.conf. > > It looks like there may be another bug effecting the setting of > > CTFCONVERT_CMD. >=20 > I had this change as a question mark to merge in my list, thanks. > Can you also merge it to 8? Will do. -- Brooks --3V7upXqbjpZ4EhLz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQhv9/XY6L6fI4GtQRAn3eAKCtIRX5Nx23mbhOQamo7nIK25PSaQCdGazQ DOSeHYSgVn6+0PBjf/TYmNs= =JUXB -----END PGP SIGNATURE----- --3V7upXqbjpZ4EhLz-- From owner-freebsd-arch@FreeBSD.ORG Wed Oct 24 15:45:10 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 538351DD; Wed, 24 Oct 2012 15:45:10 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id DDB4C8FC16; Wed, 24 Oct 2012 15:45:08 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id q9OFj82N000734; Wed, 24 Oct 2012 10:45:08 -0500 (CDT) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id q9OFj8Xk000733; Wed, 24 Oct 2012 10:45:08 -0500 (CDT) (envelope-from brooks) Date: Wed, 24 Oct 2012 10:45:08 -0500 From: Brooks Davis To: obrien@freebsd.org, Brooks Davis , Simon Gerraty , freebsd-arch@freebsd.org Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Message-ID: <20121024154508.GA93546@lor.one-eyed-alien.net> References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121022193903.GA88336@dragon.NUXI.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="J/dobhs11T7y2rNN" Content-Disposition: inline In-Reply-To: <20121022193903.GA88336@dragon.NUXI.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 15:45:10 -0000 --J/dobhs11T7y2rNN Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 22, 2012 at 12:39:04PM -0700, David O'Brien wrote: > On Mon, Oct 08, 2012 at 10:48:53AM -0500, Brooks Davis wrote: > > On Sat, Oct 06, 2012 at 05:14:23PM -0700, Simon Gerraty wrote: > > > After being bitten by: > > > make: "/b/sjg/work/fbsd-head/src/share/mk/bsd.own.mk" line 490: WITH_= CTF > > > and WITHOUT_CTF can't both be set. > ... > > I'm not sure if I agree or not, I'll have to think more. This sort of > > thing that leads to me yelling at my computer "but I @#%$@# set > > WITH_FOO you ^@$@! machine." :) >=20 > Brooks, > Isn't that what some folks are currently doing trying to build a fully > DTrace ready system? Yes, though I've partially fixed this so you can actually follow the instructions in the wiki and set WITH_CTF though make.conf. > Have you had a chance to review Simon's latest diff? Yes it's fine if the problem we want to solve is being able to set WITH_FOO and WITHOUT_FOO. I'm not sure we don't really just want to let WITH_FOO be overridden by NO_FOO more reliably. > This is the only build knob I'm aware of where setting it manually in the > environment doesn't work the same as setting it in /etc/{src,make}.conf. There is also INSTALLLIB, MAN, and PROFILE, but I agree it's not ideal. I think I'd rather have NO_* override WITH_* in the later checks than always having WITHOUT_FOO override WITH_FOO. > Its been too hard to build a fully DTrace-ready FreeBSD for a long time. > We really need closure on this -- DTrace is too useful. >=20 >=20 > Note that our Handbook () still has: >=20 > >=20 > &prompt.root; cd /usr/src > > &prompt.root; make WITH_CTF=3D1 kernel > >=20 >=20 > We really need to make this as easy as possible for users and get the > docs matching reality. I believe this is largely wrong and outdated. -- Brooks >=20 > --=20 > -- David (obrien@FreeBSD.org) >=20 --J/dobhs11T7y2rNN Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQiA0DXY6L6fI4GtQRAsGtAKCJF9/dzuL3bQyzqjO3vXn97UvINgCfcdPW vsPWGd6G3/P2Tg0X/U/87yI= =zM+X -----END PGP SIGNATURE----- --J/dobhs11T7y2rNN-- From owner-freebsd-arch@FreeBSD.ORG Wed Oct 24 16:19:49 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 50A4940F; Wed, 24 Oct 2012 16:19:49 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from exprod7og101.obsmtp.com (exprod7og101.obsmtp.com [64.18.2.155]) by mx1.freebsd.org (Postfix) with ESMTP id 7B0428FC16; Wed, 24 Oct 2012 16:19:48 +0000 (UTC) Received: from P-EMHUB03-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob101.postini.com ([64.18.6.12]) with SMTP ID DSNKUIgVI856+ujjww3jF8Gp/XtI61N/CjIa@postini.com; Wed, 24 Oct 2012 09:19:48 PDT Received: from magenta.juniper.net (172.17.27.123) by P-EMHUB03-HQ.jnpr.net (172.24.192.33) with Microsoft SMTP Server (TLS) id 8.3.213.0; Wed, 24 Oct 2012 09:18:45 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id q9OGIjh60974; Wed, 24 Oct 2012 09:18:45 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id E8E5658094; Wed, 24 Oct 2012 09:18:44 -0700 (PDT) To: Brooks Davis Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence In-Reply-To: <20121024154508.GA93546@lor.one-eyed-alien.net> References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121022193903.GA88336@dragon.NUXI.org> <20121024154508.GA93546@lor.one-eyed-alien.net> Comments: In-reply-to: Brooks Davis message dated "Wed, 24 Oct 2012 10:45:08 -0500." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 24 Oct 2012 09:18:44 -0700 Message-ID: <20121024161844.E8E5658094@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 16:19:49 -0000 On Wed, 24 Oct 2012 10:45:08 -0500, Brooks Davis writes: >> Have you had a chance to review Simon's latest diff? > >Yes it's fine if the problem we want to solve is being able to set >WITH_FOO and WITHOUT_FOO. I'm not sure we don't really just want to let >WITH_FOO be overridden by NO_FOO more reliably. That can work too, except the comments in bsd.own.mk indicated a desire to deprecate NO_* ? Since WITH_FOO could be in the environment you cannot simply .undef it and set WITHOUT_FOO when NO_* is seen - which is the cause of the errors. To be consistent, you need to test for NO_* pretty much everywhere that I was checking for WITHOUT_* - the two basically become synonymous. Thus all the logic for setting WITHOUT_* based on NO_* should be removed? Would that be a step forwards or backwards? From owner-freebsd-arch@FreeBSD.ORG Wed Oct 24 17:07:57 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1F0EBAA8 for ; Wed, 24 Oct 2012 17:07:57 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id D8D678FC18 for ; Wed, 24 Oct 2012 17:07:56 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id rp8so1449684pbb.13 for ; Wed, 24 Oct 2012 10:07:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=jlAcRXVk5I0bYuLXV4wl7tL/0eaSQr39YDCdJRuitEQ=; b=cX+LCIBUKSCPXMDHCuY87ESXNi0jAML2BREztegHQ3yVDTLnVhvI4ih0/UU5ePnN7H F074pK+JchJsy6lAPqKgExXlgbo73eu4Pq7lx0QpIbJzlnFJaDWeXu1Zei4ziQGf9e+v aVzG8Sxslx+q5tuJ3UfrB+sdEsN0pcD/k1TVkO/VjTnziuzVZKECpUpjYhLmV4cTf4sL mpgdC85CprNvCo29r5/x7HAVERlhaXKqSarlD+R1zKM7Fs+QNB1tOt/qi8ckP0Wy8j/a 2xUaT0A1Aomcb3yJJxrtUszD0ss7mBoPsFWOdaQhjfoAgXS19zxkjDdakjqmnBVCZLd3 Ej0A== Received: by 10.66.74.40 with SMTP id q8mr45688863pav.29.1351098476212; Wed, 24 Oct 2012 10:07:56 -0700 (PDT) Received: from [10.0.0.63] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id ve6sm9712132pbc.58.2012.10.24.10.07.41 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 24 Oct 2012 10:07:52 -0700 (PDT) Sender: Warner Losh Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <20121024161844.E8E5658094@chaos.jnpr.net> Date: Wed, 24 Oct 2012 11:07:38 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <18C71853-5C11-4979-BE0D-37E8E4535031@bsdimp.com> References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121022193903.GA88336@dragon.NUXI.org> <20121024154508.GA93546@lor.one-eyed-alien.net> <20121024161844.E8E5658094@chaos.jnpr.net> To: Simon J. Gerraty X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQktnqq3I/CTGHb0TqPo0sTV8M2a0R6FAaaV8UgielSt3JEnt4ZTBTIArl7spSH+VIc2+jZK Cc: Brooks Davis , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 17:07:57 -0000 On Oct 24, 2012, at 10:18 AM, Simon J. Gerraty wrote: >=20 > On Wed, 24 Oct 2012 10:45:08 -0500, Brooks Davis writes: >>> Have you had a chance to review Simon's latest diff? >>=20 >> Yes it's fine if the problem we want to solve is being able to set >> WITH_FOO and WITHOUT_FOO. I'm not sure we don't really just want to = let >> WITH_FOO be overridden by NO_FOO more reliably. >=20 > That can work too, except the comments in bsd.own.mk indicated a = desire > to deprecate NO_* ? NO_* is on its way out, so we should likely just fail for the NO_ = options... In FreeBSD 6.x they were deprecated, so it isn't like there = hasn't been warning. > Since WITH_FOO could be in the environment you cannot simply .undef it > and set WITHOUT_FOO when NO_* is seen - which is the cause of the = errors. > To be consistent, you need to test for NO_* pretty much everywhere = that > I was checking for WITHOUT_* - the two basically become synonymous. > Thus all the logic for setting WITHOUT_* based on NO_* should be = removed? All of the WITH_FOO and WITHOUT_FOO ultimately make it to MK_FOO. = Anything that's still using NO_FOO should be hastened out of the tree = quickly... > Would that be a step forwards or backwards? I think it would be a step forwards. Warner= From owner-freebsd-arch@FreeBSD.ORG Wed Oct 24 17:31:26 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E535C2A0; Wed, 24 Oct 2012 17:31:26 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from exprod7og125.obsmtp.com (exprod7og125.obsmtp.com [64.18.2.28]) by mx1.freebsd.org (Postfix) with ESMTP id 5C84A8FC16; Wed, 24 Oct 2012 17:31:26 +0000 (UTC) Received: from P-EMHUB03-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob125.postini.com ([64.18.6.12]) with SMTP ID DSNKUIgl50Ir/TI5QP5MNzMgBBcdZueXZndj@postini.com; Wed, 24 Oct 2012 10:31:26 PDT Received: from magenta.juniper.net (172.17.27.123) by P-EMHUB03-HQ.jnpr.net (172.24.192.33) with Microsoft SMTP Server (TLS) id 8.3.213.0; Wed, 24 Oct 2012 10:30:00 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id q9OHU0h07490; Wed, 24 Oct 2012 10:30:00 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 141CB58094; Wed, 24 Oct 2012 10:30:00 -0700 (PDT) To: Warner Losh Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence In-Reply-To: <18C71853-5C11-4979-BE0D-37E8E4535031@bsdimp.com> References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121022193903.GA88336@dragon.NUXI.org> <20121024154508.GA93546@lor.one-eyed-alien.net> <20121024161844.E8E5658094@chaos.jnpr.net> <18C71853-5C11-4979-BE0D-37E8E4535031@bsdimp.com> Comments: In-reply-to: Warner Losh message dated "Wed, 24 Oct 2012 11:07:38 -0600." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Wed, 24 Oct 2012 10:30:00 -0700 Message-ID: <20121024173000.141CB58094@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain Cc: Brooks Davis , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 17:31:27 -0000 >NO_* is on its way out, so we should likely just fail for the NO_ = >options... In FreeBSD 6.x they were deprecated, so it isn't like there = >hasn't been warning. That's what I thought. >All of the WITH_FOO and WITHOUT_FOO ultimately make it to MK_FOO. = >Anything that's still using NO_FOO should be hastened out of the tree = >quickly... The issue is that buildworld uses -DNO_CTF etc (-DWITHOUT_CTF wouldn't make any difference btw) which causes WITHOUT_CTF to be set, which then causes an error if WITH_CTF is set in the environment. My original proposal was to deal with this by simply letting WITHOUT_* take precedence over WITH_*. The counter proposal was to instead leverage NO_*. >> Would that be a step forwards or backwards? > >I think it would be a step forwards. The question was about undeprecating NO_*, based on your other comments, I guess you mean backwards? Mind you, it can make sense for WITH_, WITHOUT_ and NO_ to coexist. WITH* represent user preferences, whereas NO_* represents inability to support something. Thanks --sjg From owner-freebsd-arch@FreeBSD.ORG Wed Oct 24 19:13:40 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F39BCBEF for ; Wed, 24 Oct 2012 19:13:39 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-qa0-f47.google.com (mail-qa0-f47.google.com [209.85.216.47]) by mx1.freebsd.org (Postfix) with ESMTP id 9D4048FC08 for ; Wed, 24 Oct 2012 19:13:39 +0000 (UTC) Received: by mail-qa0-f47.google.com with SMTP id i29so3378439qaf.13 for ; Wed, 24 Oct 2012 12:13:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=0tR+de0BkYK59HvWAcO2UELAwzU0nYIehjrnTlLw94k=; b=AA2lNsglcaTqREsKH9FYTNZfArUW717TphSwoEhV6wXSEkcfarKI9stbFf01S2dfDL NmGp1KxjRAw87bWk01RBty14YXbVJDCbKZwYN2798EANbkQiA5bWNszX9Azgxh1yUmBg 1iDFrPYxiwqxLP2pnmk7m05QEvj+73ADRDXYkFIJAWOTVlzeInHu/bBPMFYnDZl7PeXy MiDYTWTSHtwOZTIQiRH2IF5cQxmhje7OP8RvOH99ZbvIqRcXmk91igriPbsUIJYV3ewI a/gH4qwcCchXKe2zBDrD3HKTMF7KVQJgahUXLCjxQQvozrhws0MM2P26eoPeuImquaPB F/rQ== MIME-Version: 1.0 Received: by 10.49.28.231 with SMTP id e7mr9585952qeh.49.1351106018855; Wed, 24 Oct 2012 12:13:38 -0700 (PDT) Sender: jim.harris@gmail.com Received: by 10.49.35.37 with HTTP; Wed, 24 Oct 2012 12:13:38 -0700 (PDT) Date: Wed, 24 Oct 2012 12:13:38 -0700 X-Google-Sender-Auth: EQ8gRcYZSuFlGk_DJZK1cDn7jng Message-ID: Subject: CACHE_LINE_SIZE on x86 From: Jim Harris To: freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 19:13:40 -0000 While investigating padding of the ULE scheduler locks (r242014), I recently discovered that CACHE_LINE_SIZE on x86 is defined as 128 (not 64). From what I can tell from svn logs, this was to account for 128 byte cache "sectors" that existed on the NetBurst micro architecture CPUs. I'm curious if there's been consideration in changing this back to 64? With maybe a kernel config option to modify it? On 2S systems (but not on 1S systems), I see a benefit using CACHE_LINE_SIZE=128 for the scheduler locks. I suspect this is related to data prefetching but am still running experiments to verify. Thanks, -Jim From owner-freebsd-arch@FreeBSD.ORG Wed Oct 24 21:27:31 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ED45A2CA; Wed, 24 Oct 2012 21:27:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id B4C558FC08; Wed, 24 Oct 2012 21:27:31 +0000 (UTC) Received: by mail-pa0-f54.google.com with SMTP id bi1so703964pad.13 for ; Wed, 24 Oct 2012 14:27:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=6h1eC2WLLc3huxWNqjJRk9dRBvc5FWFvIRtHa7CAdPk=; b=K94hBL7gahgKxTeWWVtHqacvig4D67cNJjGZbEUQLimDhRMaisINtZizI+/EY9g8xi tyaIcZoAYNBgCuLRTMQ2HaAzLPAfDTo7flkunA6V4J4IAj3FIQZ/pfCs2r7Y7h/OctKV a0dl73LrlnbirpHFHJoXF7J/VqGqBOteD2/lXppDt0xiY8hYcVc+KpCUDySRpErS+OfK V5rVPwCPWKQV2jhKvfy6cTGYM5eZAfNp/hjli0p7EKbN5fHGLXqe17IdjAGX2D8YXqWr XrlgPSP2yqjeHVxOsMcgN865cDH7zOuHjhv4Z1bPmVAJZL1yiRJfNJG9eOsY8Y8U8NNG be8w== MIME-Version: 1.0 Received: by 10.68.197.9 with SMTP id iq9mr52876439pbc.130.1351114051291; Wed, 24 Oct 2012 14:27:31 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.146.233 with HTTP; Wed, 24 Oct 2012 14:27:31 -0700 (PDT) In-Reply-To: References: Date: Wed, 24 Oct 2012 14:27:31 -0700 X-Google-Sender-Auth: eS3fqXP1WQGWCsZ-55lPIdDtLq0 Message-ID: Subject: Re: CACHE_LINE_SIZE on x86 From: Adrian Chadd To: Jim Harris Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 21:27:32 -0000 On 24 October 2012 12:13, Jim Harris wrote: > While investigating padding of the ULE scheduler locks (r242014), I > recently discovered that CACHE_LINE_SIZE on x86 is defined as 128 (not > 64). From what I can tell from svn logs, this was to account for 128 > byte cache "sectors" that existed on the NetBurst micro architecture > CPUs. > > I'm curious if there's been consideration in changing this back to 64? > With maybe a kernel config option to modify it? On 2S systems (but > not on 1S systems), I see a benefit using CACHE_LINE_SIZE=128 for the > scheduler locks. I suspect this is related to data prefetching but am > still running experiments to verify. Well, is it worth maintaining multiple alignment options, for aligning different things? eg cache alignment for memory allocations, larger alignment for compile-time structure alignment, etc? Adrian From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 02:12:11 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AA2F8C6E for ; Thu, 25 Oct 2012 02:12:11 +0000 (UTC) (envelope-from oppermann@networx.ch) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 114E98FC12 for ; Thu, 25 Oct 2012 02:12:10 +0000 (UTC) Received: (qmail 37917 invoked from network); 25 Oct 2012 03:49:54 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 25 Oct 2012 03:49:54 -0000 Message-ID: <50889FEE.50702@networx.ch> Date: Thu, 25 Oct 2012 04:11:58 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: CACHE_LINE_SIZE on x86 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Jim Harris , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 02:12:11 -0000 On 24.10.2012 23:27, Adrian Chadd wrote: > On 24 October 2012 12:13, Jim Harris wrote: >> While investigating padding of the ULE scheduler locks (r242014), I >> recently discovered that CACHE_LINE_SIZE on x86 is defined as 128 (not >> 64). From what I can tell from svn logs, this was to account for 128 >> byte cache "sectors" that existed on the NetBurst micro architecture >> CPUs. >> >> I'm curious if there's been consideration in changing this back to 64? >> With maybe a kernel config option to modify it? On 2S systems (but >> not on 1S systems), I see a benefit using CACHE_LINE_SIZE=128 for the >> scheduler locks. I suspect this is related to data prefetching but am >> still running experiments to verify. > > Well, is it worth maintaining multiple alignment options, for aligning > different things? Unlikely. That's going to be far too specific on a particular micro- architecture. > eg cache alignment for memory allocations, larger alignment for > compile-time structure alignment, etc? That's handled in malloc() and UMA zones. Completely irrelevant here. -- Andre From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 09:12:31 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 57B6252E; Thu, 25 Oct 2012 09:12:31 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from csmtp2.one.com (csmtp2.one.com [91.198.169.22]) by mx1.freebsd.org (Postfix) with ESMTP id 0E83B8FC17; Thu, 25 Oct 2012 09:12:30 +0000 (UTC) Received: from [192.168.1.18] (unknown [217.157.7.221]) by csmtp2.one.com (Postfix) with ESMTPA id 0F45F3081E3A; Thu, 25 Oct 2012 09:12:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: How about finally replacing GNATS? From: Erik Cederstrand In-Reply-To: Date: Thu, 25 Oct 2012 11:12:29 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> To: freebsd-arch@freebsd.org X-Mailer: Apple Mail (2.1499) Cc: Eitan Adler , giffunip@tutopia.com, Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 09:12:31 -0000 Den 27/08/2011 kl. 21.21 skrev Eitan Adler : >> We need someone to do the job. And it is not an "easy" migration in = the sene >> that we need to keep old PRs, and if you take more time than you = thought, >> everyone would yell at you (we need a bugtracker for releases = management, >> ...). >=20 > Multiple people have stepped up and offered to help (including myself) > once a proposal is accepted. Getting people to help for certain tasks > is easier than one might think. As phk@ said we need to make this a > group effort. Reviving an old thread. Is anyone actively working on replacing GNATS? I = have an itch to scratch. If not, I'd like to help gather requirements from committers, bug = busters and bug reporters so we can get the ball rolling. Thanks, Erik= From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 13:51:00 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D58A7A2B; Thu, 25 Oct 2012 13:51:00 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id A93488FC1B; Thu, 25 Oct 2012 13:51:00 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 2011BB94B; Thu, 25 Oct 2012 09:51:00 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: CACHE_LINE_SIZE on x86 Date: Thu, 25 Oct 2012 09:18:00 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201210250918.00602.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 25 Oct 2012 09:51:00 -0400 (EDT) Cc: Jim Harris X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 13:51:00 -0000 On Wednesday, October 24, 2012 3:13:38 pm Jim Harris wrote: > While investigating padding of the ULE scheduler locks (r242014), I > recently discovered that CACHE_LINE_SIZE on x86 is defined as 128 (not > 64). From what I can tell from svn logs, this was to account for 128 > byte cache "sectors" that existed on the NetBurst micro architecture > CPUs. > > I'm curious if there's been consideration in changing this back to 64? > With maybe a kernel config option to modify it? On 2S systems (but > not on 1S systems), I see a benefit using CACHE_LINE_SIZE=128 for the > scheduler locks. I suspect this is related to data prefetching but am > still running experiments to verify. All the i7 and later systems I've seen (maybe even Penryn?) have a BIOS option (typically enabled by default) to enable adjacent cache line prefetching (my understanding is that this only affects the LLC, and it seems to always fetch an aligned 128 bytes, so if your miss is in the "second" line it fetches N-1 and N, not always fetching N and N+1). That is why I thought we still use 128 bytes on x86. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 14:39:33 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 914BF15F; Thu, 25 Oct 2012 14:39:33 +0000 (UTC) (envelope-from des@des.no) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id 452E18FC14; Thu, 25 Oct 2012 14:39:33 +0000 (UTC) Received: from ds4.des.no (smtp.des.no [194.63.250.102]) by smtp.des.no (Postfix) with ESMTP id 854986A01; Thu, 25 Oct 2012 16:39:32 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 480B5941D; Thu, 25 Oct 2012 16:39:32 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Erik Cederstrand Subject: Re: How about finally replacing GNATS? References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> Date: Thu, 25 Oct 2012 16:39:31 +0200 In-Reply-To: (Erik Cederstrand's message of "Thu, 25 Oct 2012 11:12:29 +0200") Message-ID: <86sj928vks.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Julien Laffaye , Eitan Adler , giffunip@tutopia.com, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 14:39:33 -0000 Erik Cederstrand writes: > Reviving an old thread. Is anyone actively working on replacing GNATS? > I have an itch to scratch. > > If not, I'd like to help gather requirements from committers, bug > busters and bug reporters so we can get the ball rolling. We've already decided to switch to Bugzilla, and I believe there is a test instance running somewhere, but we need some sort of compatibility shim for handling reports submitted with send-pr(1). DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 14:56:39 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D6E9868; Thu, 25 Oct 2012 14:56:39 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from csmtp2.one.com (csmtp2.one.com [91.198.169.22]) by mx1.freebsd.org (Postfix) with ESMTP id 553D78FC0C; Thu, 25 Oct 2012 14:56:38 +0000 (UTC) Received: from [172.20.10.3] (109.56.86.183.mobile.3.dk [109.56.86.183]) by csmtp2.one.com (Postfix) with ESMTPA id 481053018E37; Thu, 25 Oct 2012 14:56:31 +0000 (UTC) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: How about finally replacing GNATS? From: Erik Cederstrand In-Reply-To: <86sj928vks.fsf@ds4.des.no> Date: Thu, 25 Oct 2012 16:56:27 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> To: =?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?= X-Mailer: Apple Mail (2.1499) Cc: Eitan Adler , freebsd-arch@freebsd.org, giffunip@tutopia.com, Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 14:56:39 -0000 Den 25/10/2012 kl. 16.39 skrev Dag-Erling Sm=F8rgrav : > Erik Cederstrand writes: >> Reviving an old thread. Is anyone actively working on replacing = GNATS? >> I have an itch to scratch. >>=20 >> If not, I'd like to help gather requirements from committers, bug >> busters and bug reporters so we can get the ball rolling. >=20 > We've already decided to switch to Bugzilla, and I believe there is a > test instance running somewhere, but we need some sort of = compatibility > shim for handling reports submitted with send-pr(1). Would a Python script called "send-pr" using the Bugzilla JSONRPC = webservice and a dummy Bugzilla user (to retain the possibility to = report anonymously) be acceptable? Thanks, Erik= From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 15:11:28 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 80D5CAE7; Thu, 25 Oct 2012 15:11:28 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id C3A488FC17; Thu, 25 Oct 2012 15:11:27 +0000 (UTC) Received: by mail-bk0-f54.google.com with SMTP id jf20so863600bkc.13 for ; Thu, 25 Oct 2012 08:11:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=3P02FoMy+J0d7KXzFIJuJWqOSHKlf3tPy3USfMsc2u0=; b=DuB2N4l7ZVd96/EH/dWcyYSB0I40NFujVNhtq+dLlEf1Nd9UyxMc2+WhOxP+dv3MQh CYm8F/NxzBG6TgUm6+C3Ulw37LGApKTevt1OvakWH/eHDMPmJtLfJvzFsDYkpNYgWCUJ ByS9MmgYgFrh+xh9eCHm1Pt39CCzt3D50xU1GdaFqRpGOQUKVemGSjAnApcA5yxNnDWz m1MTqwRNMaWhf2EN2ADVibYLe205qcD5Z5/NUSiKt9sXlLK2hLPD2lz5lCC0kHkloP0u M62Ytx1gdu1bN7eEVS3ov/jZnYzkji2QhqRyg6fOkO+tmTUphOXLXi5E9VATBCmX68/D 1W+Q== MIME-Version: 1.0 Received: by 10.204.128.201 with SMTP id l9mr6042205bks.66.1351177886419; Thu, 25 Oct 2012 08:11:26 -0700 (PDT) Received: by 10.204.50.197 with HTTP; Thu, 25 Oct 2012 08:11:25 -0700 (PDT) Received: by 10.204.50.197 with HTTP; Thu, 25 Oct 2012 08:11:25 -0700 (PDT) In-Reply-To: <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> Date: Thu, 25 Oct 2012 16:11:25 +0100 Message-ID: Subject: Re: How about finally replacing GNATS? From: Chris Rees To: Erik Cederstrand Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-arch@freebsd.org, =?ISO-8859-1?Q?Dag=2DErling_Sm=F8rgrav?= , giffunip@tutopia.com, Eitan Adler , Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 15:11:28 -0000 On 25 Oct 2012 15:57, "Erik Cederstrand" wrote: > > Den 25/10/2012 kl. 16.39 skrev Dag-Erling Sm=F8rgrav : > > > Erik Cederstrand writes: > >> Reviving an old thread. Is anyone actively working on replacing GNATS? > >> I have an itch to scratch. > >> > >> If not, I'd like to help gather requirements from committers, bug > >> busters and bug reporters so we can get the ball rolling. > > > > We've already decided to switch to Bugzilla, and I believe there is a > > test instance running somewhere, but we need some sort of compatibility > > shim for handling reports submitted with send-pr(1). > > Would a Python script called "send-pr" using the Bugzilla JSONRPC webservice and a dummy Bugzilla user (to retain the possibility to report anonymously) be acceptable? It'd be better in sh, because we would like to put it in base. I think there are other issues, but I'll get back to you with more info when I'm home. Chris From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 15:53:18 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0876248C; Thu, 25 Oct 2012 15:53:18 +0000 (UTC) (envelope-from des@des.no) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id AEA358FC08; Thu, 25 Oct 2012 15:53:17 +0000 (UTC) Received: from ds4.des.no (smtp.des.no [194.63.250.102]) by smtp.des.no (Postfix) with ESMTP id 9C1426A36; Thu, 25 Oct 2012 17:53:16 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 66F59942A; Thu, 25 Oct 2012 17:53:16 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Erik Cederstrand Subject: Re: How about finally replacing GNATS? References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> Date: Thu, 25 Oct 2012 17:53:16 +0200 In-Reply-To: <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> (Erik Cederstrand's message of "Thu, 25 Oct 2012 16:56:27 +0200") Message-ID: <86ehkm8s5v.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Eitan Adler , freebsd-arch@freebsd.org, giffunip@tutopia.com, Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 15:53:18 -0000 Erik Cederstrand writes: > Would a Python script called "send-pr" using the Bugzilla JSONRPC > webservice and a dummy Bugzilla user (to retain the possibility to > report anonymously) be acceptable? No. 1) We don't have Python in base. 2) The solution you propose requires an internet connection. 3) We need to be able to handle PRs submitted from existing FreeBSD installations, so the conversion has to happen at the receiving end. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 16:30:16 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A0207307 for ; Thu, 25 Oct 2012 16:30:16 +0000 (UTC) (envelope-from oppermann@networx.ch) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 007E68FC16 for ; Thu, 25 Oct 2012 16:30:15 +0000 (UTC) Received: (qmail 42256 invoked from network); 25 Oct 2012 18:07:52 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 25 Oct 2012 18:07:52 -0000 Message-ID: <5089690A.8070503@networx.ch> Date: Thu, 25 Oct 2012 18:30:02 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: John Baldwin Subject: Re: CACHE_LINE_SIZE on x86 References: <201210250918.00602.jhb@freebsd.org> In-Reply-To: <201210250918.00602.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Jim Harris , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 16:30:16 -0000 On 25.10.2012 15:18, John Baldwin wrote: > On Wednesday, October 24, 2012 3:13:38 pm Jim Harris wrote: >> While investigating padding of the ULE scheduler locks (r242014), I >> recently discovered that CACHE_LINE_SIZE on x86 is defined as 128 (not >> 64). From what I can tell from svn logs, this was to account for 128 >> byte cache "sectors" that existed on the NetBurst micro architecture >> CPUs. >> >> I'm curious if there's been consideration in changing this back to 64? >> With maybe a kernel config option to modify it? On 2S systems (but >> not on 1S systems), I see a benefit using CACHE_LINE_SIZE=128 for the >> scheduler locks. I suspect this is related to data prefetching but am >> still running experiments to verify. > > All the i7 and later systems I've seen (maybe even Penryn?) have a BIOS option > (typically enabled by default) to enable adjacent cache line prefetching (my > understanding is that this only affects the LLC, and it seems to always fetch > an aligned 128 bytes, so if your miss is in the "second" line it fetches N-1 > and N, not always fetching N and N+1). That is why I thought we still use 128 > bytes on x86. As long as the additionally prefetched cache line has its own MOESI state and gets marked as shared there is not problem with using only 64B alignment and padding. -- Andre From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 18:08:58 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0C471C01 for ; Thu, 25 Oct 2012 18:08:58 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id C36118FC0C for ; Thu, 25 Oct 2012 18:08:57 +0000 (UTC) Received: by mail-pa0-f54.google.com with SMTP id bi1so1445776pad.13 for ; Thu, 25 Oct 2012 11:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=iCxRnTyfLrybEM7XdcBxI/Y2yEWjDF7sL6bcmPaxW7Q=; b=rS6jVynuz6jf8X6OMrtQzbxIYbuZbB6MmLMJ6n2NzZaM00RnrI2WSqPs1JIqVCtm4k EQ5yGVcYlvqsc4PiQLkYJtfhhSNpalyJJRRtcpr/W012wueOQi7U3qr9hZvajiLBUjtR DrXXdgK97LwlXDN0mkyV1tW7eDSpZ2XDWclps= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding:x-gm-message-state; bh=iCxRnTyfLrybEM7XdcBxI/Y2yEWjDF7sL6bcmPaxW7Q=; b=erdtTrFTaOvxvFESfWF+Jxcu2/JT2QXDDa4Vyksn4bs3tHDO8X0vFKTcW0N5xTp04V EhSptBBSSnBogI9XK8sGTtkboPups1FKpBAM3y8lw4mOxLwvvJARxej2d4JJeXEejNwa oDzHYeO3qxSViXfQ8ry5pKm4mCMx9IbK6r3isB1ujUJi4x3+wU5EbL2rBIMHh+o/fPzv Yx9R1Rhf98tnbm6Ot21R/hrlv641ud0aPs3LR/0ge+x5ooqy2JdK7g5kvh8h1GODU24Y cDg/h5SRLrO8CSk5VJr+VOQTsyj2wqLtOonjoRBPswd7bU8PdtwFjWi4mFXHehVi1hYr NCOg== Received: by 10.68.129.72 with SMTP id nu8mr62046863pbb.29.1351188537216; Thu, 25 Oct 2012 11:08:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.66.161.163 with HTTP; Thu, 25 Oct 2012 11:08:26 -0700 (PDT) In-Reply-To: <86ehkm8s5v.fsf@ds4.des.no> References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> <86ehkm8s5v.fsf@ds4.des.no> From: Eitan Adler Date: Thu, 25 Oct 2012 14:08:26 -0400 Message-ID: Subject: Re: How about finally replacing GNATS? To: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQm4ufz8r/O4MeIxPwup15E2vPc0pPEjPmVRA24LtzfpV0kGykSwzbb9q6EN0WUWGGfIY9fP Cc: freebsd-arch@freebsd.org, giffunip@tutopia.com, Erik Cederstrand , Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 18:08:58 -0000 On 25 October 2012 11:53, Dag-Erling Sm=C3=B8rgrav wrote: > Erik Cederstrand writes: >> Would a Python script called "send-pr" using the Bugzilla JSONRPC >> webservice and a dummy Bugzilla user (to retain the possibility to >> report anonymously) be acceptable? We don't have python is base, but a shell utility would be very useful. > 2) The solution you propose requires an internet connection. This doesn't matter. > 3) We need to be able to handle PRs submitted from existing FreeBSD > installations, so the conversion has to happen at the receiving end. Maybe. We have been considering whether it would be sufficient to reject old send-pr messages but we are *not yet ready* to have this conversation. In either case having tools like the one proposed can be helpful. --=20 Eitan Adler From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 18:53:01 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DE74372B; Thu, 25 Oct 2012 18:53:01 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from csmtp3.one.com (csmtp3.one.com [91.198.169.23]) by mx1.freebsd.org (Postfix) with ESMTP id 9128A8FC12; Thu, 25 Oct 2012 18:53:01 +0000 (UTC) Received: from [192.168.124.181] (unknown [176.222.238.90]) by csmtp3.one.com (Postfix) with ESMTPA id AB8B92406353; Thu, 25 Oct 2012 18:52:59 +0000 (UTC) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: How about finally replacing GNATS? From: Erik Cederstrand In-Reply-To: <86ehkm8s5v.fsf@ds4.des.no> Date: Thu, 25 Oct 2012 20:52:59 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <0EF669C7-0ED4-4BD4-9170-C49A7CA91603@cederstrand.dk> References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> <86ehkm8s5v.fsf@ds4.des.no> To: =?iso-8859-1?Q?Dag-Erling_Sm=F8rgrav?= X-Mailer: Apple Mail (2.1499) Cc: Julien Laffaye , Eitan Adler , giffunip@tutopia.com, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 18:53:01 -0000 Den 25/10/2012 kl. 17.53 skrev Dag-Erling Sm=F8rgrav : > Erik Cederstrand writes: >> Would a Python script called "send-pr" using the Bugzilla JSONRPC >> webservice and a dummy Bugzilla user (to retain the possibility to >> report anonymously) be acceptable? >=20 > No. >=20 > 1) We don't have Python in base. Is this really a problem? Casual users could just use the Bugzilla = interface, and experienced users spitting out bug reports should be able = to install a port. > 2) The solution you propose requires an internet connection. send-pr does too, right? It may be able to cache the report (I haven't = checked), but a Python send-pr could do that, too. > 3) We need to be able to handle PRs submitted from existing FreeBSD > installations Does send-pr really need to stay in base? It could be a port, and users = could just install that port. Implementing a JSON HTTPS client in bourne shell seems, well, painful. = But maybe I could write a Python script to process emails received at = FreeBSD-gnats-submit@freebsd.org and shepherd them into Bugzilla? Then = only there MX would need Python. Erik= From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 19:03:33 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1DD9388B; Thu, 25 Oct 2012 19:03:33 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6C1028FC17; Thu, 25 Oct 2012 19:03:32 +0000 (UTC) Received: by mail-bk0-f54.google.com with SMTP id jf20so1009829bkc.13 for ; Thu, 25 Oct 2012 12:03:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=KUbsCP4jENA5w4HeI9XuIbs1fEB6SqJqNHbH3L/ekZs=; b=gUVvSoI/oxxi2H0JJ2TzdYveclKQ6HBOy5MIqQKW3o3OmPAHxNsgYSZ4zxstwPgc5m GtsqsYwTMsHR4Vf5Wk45rI0wRUEp6Gik0oUDMMh7RTds27i5aAzuN8Xeft/FVqrhmW1M Kg5XC+Ozk7nl4P2Xnp6npGZr1LO7OfT7W7ocdKCNpS9JGh1BmaAbt7vAJTJhL30WFEhV StzamPhOCzogZ3Ha03Ou34HoJ6kvBhEBT67mgfRP4PX4UYtyFceD8MOsyGNMFzP24omL /xKfir+NGqdnqe02Js9kQWPiKJWQuMLwYwQikklbQo64ifEOSSHWh05nuZBsI/AjMAKJ YIKg== Received: by 10.204.4.200 with SMTP id 8mr6658344bks.81.1351191805307; Thu, 25 Oct 2012 12:03:25 -0700 (PDT) MIME-Version: 1.0 Sender: utisoft@gmail.com Received: by 10.204.50.197 with HTTP; Thu, 25 Oct 2012 12:02:55 -0700 (PDT) In-Reply-To: <0EF669C7-0ED4-4BD4-9170-C49A7CA91603@cederstrand.dk> References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> <86ehkm8s5v.fsf@ds4.des.no> <0EF669C7-0ED4-4BD4-9170-C49A7CA91603@cederstrand.dk> From: Chris Rees Date: Thu, 25 Oct 2012 20:02:55 +0100 X-Google-Sender-Auth: 9lqfKd_fPaqRoPZV8glIbrAetKk Message-ID: Subject: Re: How about finally replacing GNATS? To: Erik Cederstrand Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: =?ISO-8859-1?Q?Dag=2DErling_Sm=F8rgrav?= , freebsd-arch@freebsd.org, giffunip@tutopia.com, Eitan Adler , Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 19:03:33 -0000 On 25 October 2012 19:52, Erik Cederstrand wrote: > Den 25/10/2012 kl. 17.53 skrev Dag-Erling Sm=F8rgrav : > >> Erik Cederstrand writes: >>> Would a Python script called "send-pr" using the Bugzilla JSONRPC >>> webservice and a dummy Bugzilla user (to retain the possibility to >>> report anonymously) be acceptable? >> >> No. >> >> 1) We don't have Python in base. > > Is this really a problem? Casual users could just use the Bugzilla interf= ace, and experienced users spitting out bug reports should be able to insta= ll a port. > >> 2) The solution you propose requires an internet connection. > > send-pr does too, right? It may be able to cache the report (I haven't ch= ecked), but a Python send-pr could do that, too. > >> 3) We need to be able to handle PRs submitted from existing FreeBSD >> installations > > Does send-pr really need to stay in base? It could be a port, and users c= ould just install that port. > > Implementing a JSON HTTPS client in bourne shell seems, well, painful. Bu= t maybe I could write a Python script to process emails received at FreeBSD= -gnats-submit@freebsd.org and shepherd them into Bugzilla? Then only there = MX would need Python. As Dag-Erling said, this is the only acceptable solution for retaining backwards-compatibility-- users still need to be able to use send-pr for the forseeable future. Luckily GNATS is easy to parse. Unfortunately I lack the cycles currently to get this done :( Chris From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 19:10:08 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D1331ADA for ; Thu, 25 Oct 2012 19:10:08 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id 930038FC1B for ; Thu, 25 Oct 2012 19:10:08 +0000 (UTC) Received: by mail-pa0-f54.google.com with SMTP id bi1so1485559pad.13 for ; Thu, 25 Oct 2012 12:10:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=CncO9OCJKtnF24rSBQK8mx629N93FxBd2p72PuiIG4E=; b=mHbuyaVjs1zwEcyGM/5Y7edauSt9huMBBqLTyfAC+qkHRGmsQgjSYQaOfMXiEkghc5 X9pQvh8lOR9Xq/YERjS9BD+imDF4arAmlUXIGvQIGWt1mySsz4W2RZMc2s4FxJYbvqBj +DrcH5YiawrzMifUixSOmxZ/r6BO+U9HFOUSQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:x-gm-message-state; bh=CncO9OCJKtnF24rSBQK8mx629N93FxBd2p72PuiIG4E=; b=iJcP42AV+4oPsCWPavXrDiUqUmPDUlcPnSB41KnpA4gk99OIFpYONFq6cJWbHi3+ic FzFzv8ZCA/2j1JkZBxy+Dmxso69XsiEAigXlrnG53oIULLJHdWIRE0AOTf/ddwo4uclS CozcncxqT8UvCzCLmL0DnQcama7uiqsgkfvD+gDKJAEP1lQXQj+n48xC/FMy72E8ZELh sGW1e8w2orhvOPioWc1DkAbDmo4DR75ZoVjfQtOhKV39j8yEcpYy12kMIn0RMEiDqee3 qMBh/0O8UbTyQfm6Sflis95yby22I3gC7Ffh+h4M5TTnqcvcFVBjMw7HtzMXyurKm3nr 5+VA== Received: by 10.68.253.102 with SMTP id zz6mr62249041pbc.99.1351192207989; Thu, 25 Oct 2012 12:10:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.66.161.163 with HTTP; Thu, 25 Oct 2012 12:09:37 -0700 (PDT) In-Reply-To: References: <1314468512.87152.YahooMailClassic@web113511.mail.gq1.yahoo.com> <4E593932.8060303@FreeBSD.org> <86sj928vks.fsf@ds4.des.no> <4FB39F66-0434-43F1-9705-3C81CDF2A25F@cederstrand.dk> <86ehkm8s5v.fsf@ds4.des.no> <0EF669C7-0ED4-4BD4-9170-C49A7CA91603@cederstrand.dk> From: Eitan Adler Date: Thu, 25 Oct 2012 15:09:37 -0400 Message-ID: Subject: Re: How about finally replacing GNATS? To: Chris Rees Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQmaGVAtB0Qd1U5dw8QOgTSPAFzDvIVij1h8XdxsPozWX7vpk+jXgExC7DsrxYylHlFd8IAB Cc: =?UTF-8?Q?Dag=2DErling_Sm=C3=B8rgrav?= , freebsd-arch@freebsd.org, giffunip@tutopia.com, Erik Cederstrand , Julien Laffaye X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 19:10:08 -0000 On 25 October 2012 15:02, Chris Rees wrote: > As Dag-Erling said, this is the only acceptable solution for retaining > backwards-compatibility-- users still need to be able to use send-pr > for the forseeable future. Maybe. Lets not discuss this now as some of the relevant questions don't have answers yet. > Luckily GNATS is easy to parse. Unfortunately I lack the cycles > currently to get this done :( Luckily we don't have to worry about this for now. The next step is to make the converter from GNATS -> Bugzilla work incrementally so we can run them both at once with GNATS acting as the master. -- Eitan Adler From owner-freebsd-arch@FreeBSD.ORG Thu Oct 25 19:27:16 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 664) id AC91F6D; Thu, 25 Oct 2012 19:27:16 +0000 (UTC) Date: Thu, 25 Oct 2012 12:27:15 -0700 From: David O'Brien To: Brooks Davis Subject: Re: bsd.own.mk - just let WITHOUT_* take precedence Message-ID: <20121025192715.GA31501@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, Brooks Davis , Simon Gerraty , freebsd-arch@freebsd.org References: <20121007001423.9878F58094@chaos.jnpr.net> <20121008154853.GC23400@lor.one-eyed-alien.net> <20121022193903.GA88336@dragon.NUXI.org> <20121024154508.GA93546@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121024154508.GA93546@lor.one-eyed-alien.net> X-Operating-System: FreeBSD 10.0-CURRENT X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-arch@freebsd.org, Simon Gerraty X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: obrien@freebsd.org List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2012 19:27:16 -0000 On Wed, Oct 24, 2012 at 10:45:08AM -0500, Brooks Davis wrote: > > Note that our Handbook () still has: > >