From owner-freebsd-arch@FreeBSD.ORG Sun Jul 20 17:49:35 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4F3C75A; Sun, 20 Jul 2014 17:49:35 +0000 (UTC) Received: from mail-oa0-x22b.google.com (mail-oa0-x22b.google.com [IPv6:2607:f8b0:4003:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 883482A82; Sun, 20 Jul 2014 17:49:35 +0000 (UTC) Received: by mail-oa0-f43.google.com with SMTP id i7so6234671oag.2 for ; Sun, 20 Jul 2014 10:49:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ptlbTxhVcJl4CjSOrkYHPELsmQY+2ROLuRIZzk6cyTE=; b=mrgPZvVef9ykhFmS/cJlUq1hgdLJWiwoKuior+IW0MCJ5JB0YZj4O3gX4/PO3WBb2c GLL0BaqhBR73cJW0lQrXnC40R6JAE80NQ8LpO70UJ4sugcsJu0LVPq4D22ZGV8Bg/Rmn sUohppA00Nhr94TJ61aLOUMArAUHgHPl1ka2bBrOc/cNAYN/p3CwjAhEyGSGSuje107o m+vtcNmxXx+nLK3MV4bEQTbVGQz9Uh0sd3aho2hM66+wsVgTiv5UJUiwKT+2ieS/RolK uV1/CCtnwfYe8qbhpG62XL6N1feN6tskDI9zvW/hP8g23PnpMkQiiG1Wltza3CzkwP8i 6Mcw== MIME-Version: 1.0 X-Received: by 10.60.123.103 with SMTP id lz7mr30152761oeb.18.1405878574657; Sun, 20 Jul 2014 10:49:34 -0700 (PDT) Received: by 10.182.216.197 with HTTP; Sun, 20 Jul 2014 10:49:34 -0700 (PDT) In-Reply-To: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> Date: Sun, 20 Jul 2014 19:49:34 +0200 Message-ID: Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: Oliver Pinter To: Pedro Giffuni Content-Type: text/plain; charset=ISO-8859-1 Cc: PaX Team , freebsd-arch@freebsd.org, Shawn Webb , Bryan Drewery X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 17:49:35 -0000 On 7/20/14, Pedro Giffuni wrote: > (Assuming @FreeBSD addresses are subscribed to arch, or check the archives) > > FWIW, > > The issues I pointed out are still standing: > > - It is yet undetermined what the performance effect will be, and it is not > clear (but seems likely from past measurements) if there will be a > performance hit even when ASLR is off. > -Apparently there are applications that will segfault (?). > > I wouldn't object to see it in the tree though: it has obviously been the > result of a lot of work and it is configurable and well integrated. It will > certainly have to be some time in the tree and undergo extensive testing > before turning it on by default though so it sounds reasonable to bring it > in but leave it initially inactive. > > Pedro. Probably pho@ has free time, to test ASLR changes? From owner-freebsd-arch@FreeBSD.ORG Sun Jul 20 20:19:02 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C14DBED4; Sun, 20 Jul 2014 20:19:02 +0000 (UTC) Received: from mail-qa0-x22d.google.com (mail-qa0-x22d.google.com [IPv6:2607:f8b0:400d:c00::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B50927BD; Sun, 20 Jul 2014 20:19:02 +0000 (UTC) Received: by mail-qa0-f45.google.com with SMTP id cm18so4412336qab.4 for ; Sun, 20 Jul 2014 13:19:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Q/3cahY3EhOrJM4XMnzJhiubmem7jQD2zo2lxjN1Zmg=; b=BJlNjoQqwV2ImmKwpW0kj4X1vtA88OkiyIT4/l/BtkZ6AzIQvKft1mVv2VGrvbHhGz MbU8AVDWinFKPOfuoTR7aj33EwMzi0LBC5aDoZDmHdxcHAMs5aqwNqusjD9MHUEgr3kj mbQ09mJq3w2HxfC4ljrNMHQv18JgFKYEYhLOy26tHERwuyAI/RpSjMggFeS4SmmSMqBp wP6dxUFprijuzSbqJ8koNFvTW63gnYueODwzFGxusnVc+qxGjLs2FUV0kZbRKgu/ghk3 UphJnjR9IypmFP7XLBWpMZKRisgStIDirXcwbF01wiNDM7gNdHwn5MV1CLzzWH/qimBt ZNXQ== X-Received: by 10.140.51.235 with SMTP id u98mr31141307qga.69.1405887541486; Sun, 20 Jul 2014 13:19:01 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id s9sm13773435qge.42.2014.07.20.13.19.00 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 20 Jul 2014 13:19:00 -0700 (PDT) Date: Sun, 20 Jul 2014 16:18:58 -0400 From: Shawn Webb To: Pedro Giffuni Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140720201858.GB29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uZ3hkaAS1mZxFaxD" Content-Disposition: inline In-Reply-To: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , freebsd-arch@freebsd.org, Oliver Pinter , Bryan Drewery X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 20:19:02 -0000 --uZ3hkaAS1mZxFaxD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 19, 2014 06:35 PM -0500, Pedro Giffuni wrote: > (Assuming @FreeBSD addresses are subscribed to arch, or check the archive= s) >=20 > FWIW, >=20 > The issues I pointed out are still standing: >=20 > - It is yet undetermined what the performance effect will be, and it is n= ot clear (but seems likely from past measurements) if there will be a perfo= rmance hit even when ASLR is off. > -Apparently there are applications that will segfault (?). So I have an old Dell Latitude E6500 that I bought at Defcon a year or so ago that I'm doing testing on. Even though it's quite an underpowered laptop, I'm running ZFS on it for BE support (in case one of our changes kills it). I'll run unixbench on it a few times to benchmark the ASLR patch. I'll test these three scenarios: 1) ASLR compiled in and enabled; 2) ASLR compiled in and disabled; 3) ASLR compiled out (GENERIC kernel). In each of these three scenarios, I'll have the kernel debugging features (WITNESS, INVARIANTS, etc.) turned off to better simulate a production system and to remove just one more variable in the tests. I'll run unixbench ten times under each scenario and I'll compute averages. Since this is an older laptop (and it's running ZFS), these tests will take a couple days. I'll have an answer for you soon. >=20 > I wouldn?t object to see it in the tree though: it has obviously been the= result of a lot of work and it is configurable and well integrated. It wil= l certainly have to be some time in the tree and undergo extensive testing = before turning it on by default though so it sounds reasonable to bring it = in but leave it initially inactive. That's great to hear. Oliver and I didn't make the PAX_ASLR option default in the GENERIC kernel, so there really isn't anything that needs to happen to make ASLR disabled by default. It's up to the user to add the PAX_ASLR option to their kernel config. The same goes for the WITH_PIE {src,make}.conf tunable. Thanks, Shawn --uZ3hkaAS1mZxFaxD Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTzCQxAAoJEGqEZY9SRW7u+8MP/jCJ+0DvnNNuDn61qzFI77cl lNJm4ZA1nZAhtNsa6spKn8obRs+woh5IgG+isKDsa2T6TEjJA3QbhQg8n9EWgstp mt29kZ8V51dMpN/QiGiVLBP6Jz3JtEFIf5vVuXWrAxkqozYqNHdJmPdE56fXRqjd 2jVao+Vms3M8aB1wCi9j1APhi01NdmgZNMxA/Z+X/yUN3FMJ67IwxuXBbzlwV9Kk 73LCXBpebaauRuMXblS+ZizNg2Qqzo29NUVDjkru3tos2sN63meFlK/UvwhXPMwe aHY9h0Q9NE6mecXILbAkB2NwaWFNBXZ1cOUyHPXxy/bv5Fhq4sk4TO5SvsTO/RqW AxVCqe1qEf1FfAIg/cRIOSc2NpV2fePQ48kB3R+yd7soy3RX7Qivyt/fPJNAdVzM b/5C+EYDz6BGJcyNzUhdAB/IxrXLhT+0nck8l59A6Xzklh1xvq2NdK9LBa1GW/AH H3OZ5DEmDk/Y6boULbphMi3YlxqDR17N/NN8nxubJIqBQ7o2zHtUlXNKP1OeqDMa jTh23A2AiD5jl6plWVxdTxZ/kNx0WiPlqcYOuN9r3H37iuSx/XHkBwVNf8W6w2me RZaHkxzrn65QkVCOk4+LeVKAe7mADspv9x+L8snsHfM3/uH+nQ9H0mHZuKwUgD1V sSXnZ1E5qkU9PtrBER2h =w2Cz -----END PGP SIGNATURE----- --uZ3hkaAS1mZxFaxD-- From owner-freebsd-arch@FreeBSD.ORG Sun Jul 20 22:05:16 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50F959E3; Sun, 20 Jul 2014 22:05:16 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C42B2152; Sun, 20 Jul 2014 22:05:15 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6KM5Eui025717 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 20 Jul 2014 15:05:15 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6KM5ETE025716; Sun, 20 Jul 2014 15:05:14 -0700 (PDT) (envelope-from jmg) Date: Sun, 20 Jul 2014 15:05:14 -0700 From: John-Mark Gurney To: Ian Lepore Subject: Re: [CFR] mge driver / elf reloc Message-ID: <20140720220514.GP45513@funkthat.com> Mail-Followup-To: Ian Lepore , Fabien Thomas , freebsd-arm@freebsd.org, arch@FreeBSD.org References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1405810447.85788.41.camel@revolution.hippie.lan> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sun, 20 Jul 2014 15:05:15 -0700 (PDT) Cc: arch@freebsd.org, freebsd-arm@freebsd.org, Fabien Thomas X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 22:05:16 -0000 Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: > Sorry to take so long to reply to this, I'm trying to get caught up. I > see you've already committed the mge fixes. I think the ELF alignment > fix looks good and should also be committed. So, re the elf alignment... I think we should get a set of macros that handle load/stores to/from unaligned addresses that are transparent to the caller.... I need these for some other code I'm writing... I thought Open/Net had these available, but I can't seem to find them right now... Comments? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Sun Jul 20 22:44:13 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 33469C71; Sun, 20 Jul 2014 22:44:13 +0000 (UTC) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0E44124A8; Sun, 20 Jul 2014 22:44:12 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id s6KMPYm2034654; Sun, 20 Jul 2014 22:25:35 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.101] (192.168.1.66 [192.168.1.66]) by kientzle.com with SMTP id vdaw6hpf82wki98ffnynjfr39i; Sun, 20 Jul 2014 22:25:34 +0000 (UTC) (envelope-from tim@kientzle.com) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [CFR] mge driver / elf reloc From: Tim Kientzle In-Reply-To: <20140720220514.GP45513@funkthat.com> Date: Sun, 20 Jul 2014 15:25:34 -0700 Content-Transfer-Encoding: 7bit Message-Id: References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1878.6) Cc: arch@freebsd.org, freebsd-arm@freebsd.org, Fabien Thomas , Ian Lepore X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 22:44:13 -0000 On Jul 20, 2014, at 3:05 PM, John-Mark Gurney wrote: > Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: >> Sorry to take so long to reply to this, I'm trying to get caught up. I >> see you've already committed the mge fixes. I think the ELF alignment >> fix looks good and should also be committed. > > So, re the elf alignment... > > I think we should get a set of macros that handle load/stores to/from > unaligned addresses that are transparent to the caller.... I need > these for some other code I'm writing... > > I thought Open/Net had these available, but I can't seem to find them > right now... $ man 9 byteorder is most of what you want, lacking only some aliases to pick the correct macro for native byte order. Tim From owner-freebsd-arch@FreeBSD.ORG Sun Jul 20 23:10:59 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 36AA3CC2; Sun, 20 Jul 2014 23:10:59 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E6AFB2766; Sun, 20 Jul 2014 23:10:58 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6KNAubX026526 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 20 Jul 2014 16:10:57 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6KNAu2a026525; Sun, 20 Jul 2014 16:10:56 -0700 (PDT) (envelope-from jmg) Date: Sun, 20 Jul 2014 16:10:56 -0700 From: John-Mark Gurney To: Tim Kientzle Subject: Re: [CFR] mge driver / elf reloc Message-ID: <20140720231056.GQ45513@funkthat.com> Mail-Followup-To: Tim Kientzle , Ian Lepore , arch@freebsd.org, freebsd-arm@freebsd.org, Fabien Thomas References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Sun, 20 Jul 2014 16:10:57 -0700 (PDT) Cc: arch@freebsd.org, freebsd-arm@freebsd.org, Fabien Thomas , Ian Lepore X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 23:10:59 -0000 Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: > > On Jul 20, 2014, at 3:05 PM, John-Mark Gurney wrote: > > > Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: > >> Sorry to take so long to reply to this, I'm trying to get caught up. I > >> see you've already committed the mge fixes. I think the ELF alignment > >> fix looks good and should also be committed. > > > > So, re the elf alignment... > > > > I think we should get a set of macros that handle load/stores to/from > > unaligned addresses that are transparent to the caller.... I need > > these for some other code I'm writing... > > > > I thought Open/Net had these available, but I can't seem to find them > > right now... > > $ man 9 byteorder > > is most of what you want, lacking only some aliases to pick > the correct macro for native byte order. Um, those doesn't help if you want native endian order... Also, only the enc/dec functions are documented to work on non-aligned address, so that doesn't help in most cases... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 14:46:41 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7DED4119 for ; Mon, 21 Jul 2014 14:46:41 +0000 (UTC) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 40A5F2DE8 for ; Mon, 21 Jul 2014 14:46:40 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id y20so6918395ier.27 for ; Mon, 21 Jul 2014 07:46:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=A5erOhAY4+GeP9su/j+jYVNuXCwQ35/Ztm9Cc2Qi4mk=; b=F4qOwBfkllfOCNHcXUEIP+60FqvfFbva0WofIyb1vqDyD2pf6ZQx2dsvDpjqM4GdC/ U5nqoD4tvY9FPfguMQy3vGpt+1NcP/kxuyLhicb7Pm8FXhZTysqSO5T7C2ZNYVN2yde3 W1BeEeqr/DKOmagLKyae+W8AVoYCY8n7I6FZE/Ia5dl+2Agx9w20RKnvNi9qHjYMrNCz j2KbGfm6CvcId+HRxeDHEkV4M0ZabxczNCo1mnP4C0BPGfFbpARW/UaCBz71qCrayvyg jTYNZTv29xmx/jLIjITmPg3rbUaFu863NwHwfjDaUzp1JF5PLyEGJplHMTNTy3urR3IG bmPQ== X-Gm-Message-State: ALoCoQmvAqbanXNEYCdLPOqAl3itO6NNivbpBY85rLs88TySgiTb1MT+hS9O5JQSIn7U95tsOAKi X-Received: by 10.50.43.202 with SMTP id y10mr5984003igl.10.1405953999771; Mon, 21 Jul 2014 07:46:39 -0700 (PDT) Received: from bsdimp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id hu5sm18932790igb.16.2014.07.21.07.46.38 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 21 Jul 2014 07:46:39 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_034E84E0-2852-49EA-AC44-CF120A14ECC5"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [CFR] mge driver / elf reloc From: Warner Losh In-Reply-To: <20140720231056.GQ45513@funkthat.com> Date: Mon, 21 Jul 2014 08:46:49 -0600 Message-Id: <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1878.6) Cc: Tim Kientzle , freebsd-arm , Fabien Thomas , Ian Lepore , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 14:46:41 -0000 --Apple-Mail=_034E84E0-2852-49EA-AC44-CF120A14ECC5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Jul 20, 2014, at 5:10 PM, John-Mark Gurney wrote: > Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: >>=20 >> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney = wrote: >>=20 >>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: >>>> Sorry to take so long to reply to this, I'm trying to get caught = up. I >>>> see you've already committed the mge fixes. I think the ELF = alignment >>>> fix looks good and should also be committed. >>>=20 >>> So, re the elf alignment... >>>=20 >>> I think we should get a set of macros that handle load/stores = to/from >>> unaligned addresses that are transparent to the caller.... I need >>> these for some other code I'm writing...=20 >>>=20 >>> I thought Open/Net had these available, but I can't seem to find = them >>> right now... >>=20 >> $ man 9 byteorder >>=20 >> is most of what you want, lacking only some aliases to pick >> the correct macro for native byte order. >=20 > Um, those doesn't help if you want native endian order=85 Ummm, yes they do. enc converts from native order. dec decodes to native = byte order. They are more general cases than the ntoh* functions that are = more traditional since they also work on byte streams that may not be completely aligned = when sitting in memory. Which is what you are asking for. > Also, only the enc/dec functions are documented to work on non-aligned > address, so that doesn't help in most cases=85 They work on all addresses. They are even documented to work on any = address: The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), = be64dec(), le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and = le64dec() functions encode and decode integers to/from byte strings on any = align- ment in big/little endian format. So they are quite useful in general. Peeking under the covers at the = implementation also shows they will work for any alignment, so I=92m having trouble = understanding where this objection is really coming from. Warner --Apple-Mail=_034E84E0-2852-49EA-AC44-CF120A14ECC5 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTzSfZAAoJEGwc0Sh9sBEAtwQQANWF//KOUKKWQ4IVrQUwsOVf BAGJz6ulwAR/lAt22Wx8jXyxGcl/uKql6nxmsakPSgOB/9Pb+iqkkYUmsAjlrK9J iXYQ/QtkJmdht8VZUkabIovz/KRpP2r1qLwszVoo/lSUb5OBM9bI8zVmNaFCsvIS SvUPzrz1VLjYlGSTVXvSXN+Qd46J750cs9Hb19CnbRZ5pZ0pRzUIufY/G+XdzJqX TLAxasKDdYtXIUbOydNpucD9wA5tiiMajQRJyFaZuBv5SkfPH6BTac/XYrJz8uc7 bW1rkANW0THGVPQFjETh5VvkK/TlDDUIgJ/lFhwf/ZvtsAgT0mKTUK6twHc0N4Sn EuJer92IuDfqCIdmlQ0eyzvgLCsGRM3Pr6OCZwYUmwQ4k8JQEF9hmM4o/RzUocue HiG1NPwLZNVx84xr0+5wr9TC71hHBcG7goIAdoxY0cinJFiwK3E1zg4Dr8pRicf0 HJpsTq9lGHy7x6fEN25akJM0I3I3Eph8DgazazaSi0vMDSRqMOPsvjUgQ3e9Bzo2 /QGjI8miyXR8US341cS2VxugI0y2PSmwUYrFymbMKgSlTvJYXV2yKhPPoT7nVcpO NDpnfDf5EHT2q9eYr1iKs5y09Ed1QyBnOA/6VLKDPVIjptYpcxZk/xGd7Vm61cOX Ula38Jy3qNhbDLxuHzSQ =IL3U -----END PGP SIGNATURE----- --Apple-Mail=_034E84E0-2852-49EA-AC44-CF120A14ECC5-- From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 15:04:18 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AACAA840; Mon, 21 Jul 2014 15:04:18 +0000 (UTC) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A1822FCD; Mon, 21 Jul 2014 15:04:18 +0000 (UTC) Received: from c-50-155-136-3.hsd1.co.comcast.net ([50.155.136.3] helo=ilsoft.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1X9F8M-000NPd-MH; Mon, 21 Jul 2014 15:04:10 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s6LF49ek032726; Mon, 21 Jul 2014 09:04:09 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 50.155.136.3 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX19AoiW6fTllgrCw/54MlGPe X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: [CFR] mge driver / elf reloc From: Ian Lepore To: Warner Losh In-Reply-To: <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> Content-Type: text/plain; charset="windows-1251" Date: Mon, 21 Jul 2014 09:04:08 -0600 Message-ID: <1405955048.85788.74.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ilsoft.org id s6LF49ek032726 Cc: arch@freebsd.org, freebsd-arm X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 15:04:18 -0000 On Mon, 2014-07-21 at 08:46 -0600, Warner Losh wrote: > On Jul 20, 2014, at 5:10 PM, John-Mark Gurney wrote: >=20 > > Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: > >>=20 > >> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney wro= te: > >>=20 > >>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: > >>>> Sorry to take so long to reply to this, I'm trying to get caught u= p. I > >>>> see you've already committed the mge fixes. I think the ELF align= ment > >>>> fix looks good and should also be committed. > >>>=20 > >>> So, re the elf alignment... > >>>=20 > >>> I think we should get a set of macros that handle load/stores to/fr= om > >>> unaligned addresses that are transparent to the caller.... I need > >>> these for some other code I'm writing...=20 > >>>=20 > >>> I thought Open/Net had these available, but I can't seem to find th= em > >>> right now... > >>=20 > >> $ man 9 byteorder > >>=20 > >> is most of what you want, lacking only some aliases to pick > >> the correct macro for native byte order. > >=20 > > Um, those doesn't help if you want native endian order=85 >=20 > Ummm, yes they do. enc converts from native order. dec decodes to nativ= e byte > order. They are more general cases than the ntoh* functions that are mo= re traditional > since they also work on byte streams that may not be completely aligned= when > sitting in memory. Which is what you are asking for. >=20 > > Also, only the enc/dec functions are documented to work on non-aligne= d > > address, so that doesn't help in most cases=85 >=20 > They work on all addresses. They are even documented to work on any add= ress: >=20 > The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64dec= (), > le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64dec= () > functions encode and decode integers to/from byte strings on any a= lign- > ment in big/little endian format. >=20 > So they are quite useful in general. Peeking under the covers at the im= plementation > also shows they will work for any alignment, so I=92m having trouble un= derstanding > where this objection is really coming from. >=20 > Warner >=20 The functionality requested was alignment-safe copy/assign without any endian change. The code in question was conceptually something like =20 if (pointer & 0x03) do-alignment-safe-thing else directly-deref-the-pointer -- Ian From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 16:26:01 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4B966BA; Mon, 21 Jul 2014 16:26:01 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AC4E827DC; Mon, 21 Jul 2014 16:26:01 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6LGPxp8041488 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 21 Jul 2014 09:26:00 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6LGPxia041487; Mon, 21 Jul 2014 09:25:59 -0700 (PDT) (envelope-from jmg) Date: Mon, 21 Jul 2014 09:25:59 -0700 From: John-Mark Gurney To: Warner Losh Subject: Re: [CFR] mge driver / elf reloc Message-ID: <20140721162559.GS45513@funkthat.com> Mail-Followup-To: Warner Losh , Tim Kientzle , arch@freebsd.org, freebsd-arm , Fabien Thomas , Ian Lepore References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Mon, 21 Jul 2014 09:26:00 -0700 (PDT) Cc: Tim Kientzle , freebsd-arm , Fabien Thomas , Ian Lepore , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 16:26:02 -0000 Warner Losh wrote this message on Mon, Jul 21, 2014 at 08:46 -0600: > > On Jul 20, 2014, at 5:10 PM, John-Mark Gurney wrote: > > > Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: > >> > >> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney wrote: > >> > >>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: > >>>> Sorry to take so long to reply to this, I'm trying to get caught up. I > >>>> see you've already committed the mge fixes. I think the ELF alignment > >>>> fix looks good and should also be committed. > >>> > >>> So, re the elf alignment... > >>> > >>> I think we should get a set of macros that handle load/stores to/from > >>> unaligned addresses that are transparent to the caller.... I need > >>> these for some other code I'm writing... > >>> > >>> I thought Open/Net had these available, but I can't seem to find them > >>> right now... > >> > >> $ man 9 byteorder > >> > >> is most of what you want, lacking only some aliases to pick > >> the correct macro for native byte order. > > > > Um, those doesn't help if you want native endian order? > > Ummm, yes they do. enc converts from native order. dec decodes to native byte No they don't.. If you want to read a value in memory that is native endian order to native endian order (no conversion), they cannot be used w/o using something like below... > order. They are more general cases than the ntoh* functions that are more traditional > since they also work on byte streams that may not be completely aligned when > sitting in memory. Which is what you are asking for. So, you're saying that I now need to write code like: #if LITTLE_ENDIAN /* or how ever this is spelled*/ var = le32enc(foo); #else var = be32enc(foo); #endif If I want to read a arch native endian value? No thank you... > > Also, only the enc/dec functions are documented to work on non-aligned > > address, so that doesn't help in most cases? > > They work on all addresses. They are even documented to work on any address: > > The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64dec(), > le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64dec() > functions encode and decode integers to/from byte strings on any align- > ment in big/little endian format. > > So they are quite useful in general. Peeking under the covers at the implementation > also shows they will work for any alignment, so I?m having trouble understanding > where this objection is really coming from. There are places where you write code such as: int i; memcpy(&i, inp, sizeof i); /* use i */ In order to avoid alignment faults... None of the functions in byteorder do NO conversion of endian, or you have to know which endian you are but that doesn't work on MI code... Did you read what the commited code did? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 16:31:50 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40B77823 for ; Mon, 21 Jul 2014 16:31:50 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 025D32823 for ; Mon, 21 Jul 2014 16:31:49 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id l13so3005888iga.10 for ; Mon, 21 Jul 2014 09:31:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=wDeDoNSXcf7CdkYzdv5YdOLpCzIvdO6tHRVSoCK1RKA=; b=UEoi+9TNIERYap1GXqJSrIV8Hna6wptOpzOP8LqLnJlJwLj4gaQwQSSFI4S2MBckTR Slo3EX/f7Tf9JRtXorLBYomvOgAZTVmHgHXxk2YTbZ2/FuoEOgpGjl1cjv4ru+TQ9cXK 5Ee6V+Ny9kHK4NQkqXl8h9eAb9RtIEsS8OWAIQ87Bt2YEKg88yyKPLF4BC18jessiwsJ bQnt60UnMgrBoc3TTiaFqP34POo6sZwbhdakftHs+tz5sOUOice5zvTcEV7vIbVOJ03z 4/PCyijHmnE4EuwfhlsUN0ynAMMfX93oC37q8AI6xNdP7mHOHZey/+p7Pcyp8IDK43mi INyQ== X-Gm-Message-State: ALoCoQmpMeWhx0uX0dQ1GehnZCvjdoXpNqmGmrIoXcnTjjTHqs2tpNcl4TyjiSfNxhke+ne3pxuG X-Received: by 10.50.153.83 with SMTP id ve19mr7156700igb.4.1405960303422; Mon, 21 Jul 2014 09:31:43 -0700 (PDT) Received: from bsdimp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id o9sm2673343igv.18.2014.07.21.09.31.42 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 21 Jul 2014 09:31:42 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_20074026-A1BD-4AA6-916F-13D253CB6830"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [CFR] mge driver / elf reloc From: Warner Losh In-Reply-To: <20140721162559.GS45513@funkthat.com> Date: Mon, 21 Jul 2014 10:31:40 -0600 Message-Id: <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1878.6) Cc: Tim Kientzle , freebsd-arm , Fabien Thomas , Ian Lepore , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 16:31:50 -0000 --Apple-Mail=_20074026-A1BD-4AA6-916F-13D253CB6830 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Jul 21, 2014, at 10:25 AM, John-Mark Gurney wrote: > Warner Losh wrote this message on Mon, Jul 21, 2014 at 08:46 -0600: >>=20 >> On Jul 20, 2014, at 5:10 PM, John-Mark Gurney = wrote: >>=20 >>> Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: >>>>=20 >>>> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney = wrote: >>>>=20 >>>>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: >>>>>> Sorry to take so long to reply to this, I'm trying to get caught = up. I >>>>>> see you've already committed the mge fixes. I think the ELF = alignment >>>>>> fix looks good and should also be committed. >>>>>=20 >>>>> So, re the elf alignment... >>>>>=20 >>>>> I think we should get a set of macros that handle load/stores = to/from >>>>> unaligned addresses that are transparent to the caller.... I need >>>>> these for some other code I'm writing...=20 >>>>>=20 >>>>> I thought Open/Net had these available, but I can't seem to find = them >>>>> right now... >>>>=20 >>>> $ man 9 byteorder >>>>=20 >>>> is most of what you want, lacking only some aliases to pick >>>> the correct macro for native byte order. >>>=20 >>> Um, those doesn't help if you want native endian order? >>=20 >> Ummm, yes they do. enc converts from native order. dec decodes to = native byte >=20 > No they don't.. If you want to read a value in memory that is native > endian order to native endian order (no conversion), they cannot be > used w/o using something like below=85 Missed the native to native. bcopy works, but is ugly, as you note = below. >> order. They are more general cases than the ntoh* functions that are = more traditional >> since they also work on byte streams that may not be completely = aligned when >> sitting in memory. Which is what you are asking for. >=20 > So, you're saying that I now need to write code like: > #if LITTLE_ENDIAN /* or how ever this is spelled*/ > var =3D le32enc(foo); > #else > var =3D be32enc(foo); > #endif >=20 > If I want to read a arch native endian value? No thank you=85 I=92m not saying that at all. >>> Also, only the enc/dec functions are documented to work on = non-aligned >>> address, so that doesn't help in most cases? >>=20 >> They work on all addresses. They are even documented to work on any = address: >>=20 >> The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), = be64dec(), >> le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and = le64dec() >> functions encode and decode integers to/from byte strings on any = align- >> ment in big/little endian format. >>=20 >> So they are quite useful in general. Peeking under the covers at the = implementation >> also shows they will work for any alignment, so I?m having trouble = understanding >> where this objection is really coming from. >=20 > There are places where you write code such as: > int i; > memcpy(&i, inp, sizeof i); > /* use i */ >=20 > In order to avoid alignment faults... None of the functions in = byteorder > do NO conversion of endian, or you have to know which endian you are = but > that doesn't work on MI code... >=20 > Did you read what the commited code did? No, I missed that bit beaded on your reply (which seemed to imply you = needed endian conversion) which implied the enc/dec are only documented to work = on non-aligned which is what I was correcting. But maybe the more basic question is why do you even have packed structures that are native endian that you want to access as naturally aligned structures? How did they become native endian and why weren=92t they converted to a more natural layout at that time? Warner --Apple-Mail=_20074026-A1BD-4AA6-916F-13D253CB6830 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTzUBtAAoJEGwc0Sh9sBEAlpEP/0YBNPGJbxbgWHd3OwBOQEeG gH9ZQ4OJ9NDxUo0frRuLb6LRtZPBJ+iNS6GHnygnnSrdDqcTV6sV5+u1h0SGhQ8H oy3tC10rHTj3IRnqzTP+mdUb2wL4ztBWnRImkK16rAr7/mUY2BF06SLZLu9gkK4A FTq18IQS4jTG+NLLiTaBuGnv7jJj8m7LbkN3mtkylSIFecTRPFDjIr8x+S7dOTjJ hn45Z07Js62fiw8l+Wm1lSBNOgc1t9bVfSZvN19o05WGsDuIKwCZQQykEgiGKOFR B9jcgLhSxJyvwU+G7MYOVRZAe3uCBQ86AhzONoYcjWRTPsSRoYCgcfhbN69q/Cbo fssySrfTSqf+jE6JGDLyBCRg8SAPtY+INCHPvC3lOwcOy0ZwVBujBkttTxjOJhHE +QPqwNSlbuL0KGT4ybdROlBOyLdTLhfgPfH4J/PZqaj71Bal+O0ZHDDoDG7yA9Q2 CsRf3FEtkTJ2GlefkjlTQF5eMlKI7ebfFFenMdD5WDUMU6bfFGlUQ9sos0kahORK D0OqvKN09+aBNZ4Z3NDmG72TzGEoexnHBNPoShMf4t7Gq8Qq+Z3mZY3TON3e4+ba Jm2ZjANSO/G1WJXHHh1wY+Ji+G8jHTA9UkMFl6E/NVPYVP29JtJcu8i65QT4hZXS X4qVhGagEZic5iaTeSp3 =4kUJ -----END PGP SIGNATURE----- --Apple-Mail=_20074026-A1BD-4AA6-916F-13D253CB6830-- From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 16:56:22 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19697494; Mon, 21 Jul 2014 16:56:22 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DD1EE2C4A; Mon, 21 Jul 2014 16:56:21 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s6LGuJfZ041955 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 21 Jul 2014 09:56:20 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s6LGuJhm041954; Mon, 21 Jul 2014 09:56:19 -0700 (PDT) (envelope-from jmg) Date: Mon, 21 Jul 2014 09:56:19 -0700 From: John-Mark Gurney To: Warner Losh Subject: Re: [CFR] mge driver / elf reloc Message-ID: <20140721165619.GT45513@funkthat.com> Mail-Followup-To: Warner Losh , Tim Kientzle , arch@freebsd.org, freebsd-arm , Fabien Thomas , Ian Lepore References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Mon, 21 Jul 2014 09:56:20 -0700 (PDT) Cc: Tim Kientzle , freebsd-arm , Fabien Thomas , Ian Lepore , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 16:56:22 -0000 Warner Losh wrote this message on Mon, Jul 21, 2014 at 10:31 -0600: > > On Jul 21, 2014, at 10:25 AM, John-Mark Gurney wrote: > > > Warner Losh wrote this message on Mon, Jul 21, 2014 at 08:46 -0600: > >> > >> On Jul 20, 2014, at 5:10 PM, John-Mark Gurney wrote: > >> > >>> Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: > >>>> > >>>> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney wrote: > >>>> > >>>>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: > >>>>>> Sorry to take so long to reply to this, I'm trying to get caught up. I > >>>>>> see you've already committed the mge fixes. I think the ELF alignment > >>>>>> fix looks good and should also be committed. > >>>>> > >>>>> So, re the elf alignment... > >>>>> > >>>>> I think we should get a set of macros that handle load/stores to/from > >>>>> unaligned addresses that are transparent to the caller.... I need > >>>>> these for some other code I'm writing... > >>>>> > >>>>> I thought Open/Net had these available, but I can't seem to find them > >>>>> right now... > >>>> > >>>> $ man 9 byteorder > >>>> > >>>> is most of what you want, lacking only some aliases to pick > >>>> the correct macro for native byte order. > >>> > >>> Um, those doesn't help if you want native endian order? > >> > >> Ummm, yes they do. enc converts from native order. dec decodes to native byte > > > > No they don't.. If you want to read a value in memory that is native > > endian order to native endian order (no conversion), they cannot be > > used w/o using something like below? > > Missed the native to native. bcopy works, but is ugly, as you note below. > > >> order. They are more general cases than the ntoh* functions that are more traditional > >> since they also work on byte streams that may not be completely aligned when > >> sitting in memory. Which is what you are asking for. > > > > So, you're saying that I now need to write code like: > > #if LITTLE_ENDIAN /* or how ever this is spelled*/ > > var = le32enc(foo); > > #else > > var = be32enc(foo); > > #endif > > > > If I want to read a arch native endian value? No thank you? > > I?m not saying that at all. > > >>> Also, only the enc/dec functions are documented to work on non-aligned > >>> address, so that doesn't help in most cases? > >> > >> They work on all addresses. They are even documented to work on any address: > >> > >> The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64dec(), > >> le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64dec() > >> functions encode and decode integers to/from byte strings on any align- > >> ment in big/little endian format. > >> > >> So they are quite useful in general. Peeking under the covers at the implementation > >> also shows they will work for any alignment, so I?m having trouble understanding > >> where this objection is really coming from. > > > > There are places where you write code such as: > > int i; > > memcpy(&i, inp, sizeof i); > > /* use i */ > > > > In order to avoid alignment faults... None of the functions in byteorder > > do NO conversion of endian, or you have to know which endian you are but > > that doesn't work on MI code... > > > > Did you read what the commited code did? > > No, I missed that bit beaded on your reply (which seemed to imply you needed > endian conversion) which implied the enc/dec are only documented to work on non-aligned > which is what I was correcting. Hmm... It appears that byteorder(9) leaks to userland though isn't documented to be available in userland... Should we fix this and make it an offical userland API? How to document it? In my case, the enc/dec version would have been enough for what I need, but not "part of the userland API"... There is an xref from byteorder(3), but no comment on either that they will work.. > But maybe the more basic question is why do you even have packed > structures that are native endian that you want to access as naturally > aligned structures? How did they become native endian and why weren?t > they converted to a more natural layout at that time? The original message said why... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 17:16:47 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4915DBB9 for ; Mon, 21 Jul 2014 17:16:47 +0000 (UTC) Received: from mail-ig0-f174.google.com (mail-ig0-f174.google.com [209.85.213.174]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA4882E43 for ; Mon, 21 Jul 2014 17:16:46 +0000 (UTC) Received: by mail-ig0-f174.google.com with SMTP id c1so3059670igq.1 for ; Mon, 21 Jul 2014 10:16:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=2vnYV1OwPZU48U6RQgSDbsqe/S1b9OH1hRT5ZR2Sr6Q=; b=DewDNN7WNPo3b+JdmPTz+QGMQaT0m+4m14pK9YCdWlxBDASqD9Ycwf5aEexKN0XRED /1SSYO6AP4ZTkPMxMboEiLpfj+JZVIaSZZjY7gsCaHvCzEmgQX9gqRrKG8b/EepU80bE I8pKooMpzX1ZXfGV7w/RaKXiUA8AAn5VDSNmHvselmbZbKlm5hQIayZBoLVhSF690t75 GeT5xDZwtzWVXKIqOZ6I/WbP3DD4ckAB7R09J8J66FFnHnhYE8OsS08eX3SzCANfkVH8 Byu2ibOS35vrn1XLbxBZY4/9tmJjGMwth7+/hK23xjO1hEAV1XMtVJ6xlkbnSeq6uFbn RvXA== X-Gm-Message-State: ALoCoQkuL/zt4RTBr40b+WgGLPD9CidRjsyAFUWCAXDVRUOHsvCEGtBoiCSswvbFaNpqItcMHmZS X-Received: by 10.50.80.116 with SMTP id q20mr7508438igx.22.1405962999843; Mon, 21 Jul 2014 10:16:39 -0700 (PDT) Received: from bsdimp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id dx6sm40361495igb.15.2014.07.21.10.16.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 21 Jul 2014 10:16:38 -0700 (PDT) Sender: Warner Losh Content-Type: multipart/signed; boundary="Apple-Mail=_0BBD206C-C6E0-434F-B718-ABB5CFD259E3"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [CFR] mge driver / elf reloc From: Warner Losh In-Reply-To: <20140721165619.GT45513@funkthat.com> Date: Mon, 21 Jul 2014 11:16:37 -0600 Message-Id: References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> <20140721165619.GT45513@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1878.6) Cc: Tim Kientzle , freebsd-arm , Fabien Thomas , Ian Lepore , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 17:16:47 -0000 --Apple-Mail=_0BBD206C-C6E0-434F-B718-ABB5CFD259E3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Jul 21, 2014, at 10:56 AM, John-Mark Gurney wrote: > Warner Losh wrote this message on Mon, Jul 21, 2014 at 10:31 -0600: >>=20 >> On Jul 21, 2014, at 10:25 AM, John-Mark Gurney = wrote: >>=20 >>> Warner Losh wrote this message on Mon, Jul 21, 2014 at 08:46 -0600: >>>>=20 >>>> On Jul 20, 2014, at 5:10 PM, John-Mark Gurney = wrote: >>>>=20 >>>>> Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 = -0700: >>>>>>=20 >>>>>> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney = wrote: >>>>>>=20 >>>>>>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 = -0600: >>>>>>>> Sorry to take so long to reply to this, I'm trying to get = caught up. I >>>>>>>> see you've already committed the mge fixes. I think the ELF = alignment >>>>>>>> fix looks good and should also be committed. >>>>>>>=20 >>>>>>> So, re the elf alignment... >>>>>>>=20 >>>>>>> I think we should get a set of macros that handle load/stores = to/from >>>>>>> unaligned addresses that are transparent to the caller.... I = need >>>>>>> these for some other code I'm writing...=20 >>>>>>>=20 >>>>>>> I thought Open/Net had these available, but I can't seem to find = them >>>>>>> right now... >>>>>>=20 >>>>>> $ man 9 byteorder >>>>>>=20 >>>>>> is most of what you want, lacking only some aliases to pick >>>>>> the correct macro for native byte order. >>>>>=20 >>>>> Um, those doesn't help if you want native endian order? >>>>=20 >>>> Ummm, yes they do. enc converts from native order. dec decodes to = native byte >>>=20 >>> No they don't.. If you want to read a value in memory that is native >>> endian order to native endian order (no conversion), they cannot be >>> used w/o using something like below? >>=20 >> Missed the native to native. bcopy works, but is ugly, as you note = below. >>=20 >>>> order. They are more general cases than the ntoh* functions that = are more traditional >>>> since they also work on byte streams that may not be completely = aligned when >>>> sitting in memory. Which is what you are asking for. >>>=20 >>> So, you're saying that I now need to write code like: >>> #if LITTLE_ENDIAN /* or how ever this is spelled*/ >>> var =3D le32enc(foo); >>> #else >>> var =3D be32enc(foo); >>> #endif >>>=20 >>> If I want to read a arch native endian value? No thank you? >>=20 >> I?m not saying that at all. >>=20 >>>>> Also, only the enc/dec functions are documented to work on = non-aligned >>>>> address, so that doesn't help in most cases? >>>>=20 >>>> They work on all addresses. They are even documented to work on any = address: >>>>=20 >>>> The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), = be64dec(), >>>> le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and = le64dec() >>>> functions encode and decode integers to/from byte strings on any = align- >>>> ment in big/little endian format. >>>>=20 >>>> So they are quite useful in general. Peeking under the covers at = the implementation >>>> also shows they will work for any alignment, so I?m having trouble = understanding >>>> where this objection is really coming from. >>>=20 >>> There are places where you write code such as: >>> int i; >>> memcpy(&i, inp, sizeof i); >>> /* use i */ >>>=20 >>> In order to avoid alignment faults... None of the functions in = byteorder >>> do NO conversion of endian, or you have to know which endian you are = but >>> that doesn't work on MI code... >>>=20 >>> Did you read what the commited code did? >>=20 >> No, I missed that bit beaded on your reply (which seemed to imply you = needed >> endian conversion) which implied the enc/dec are only documented to = work on non-aligned >> which is what I was correcting. >=20 > Hmm... It appears that byteorder(9) leaks to userland though isn't > documented to be available in userland... Should we fix this and make > it an offical userland API? How to document it? In my case, the > enc/dec version would have been enough for what I need, but not "part = of > the userland API"... There is an xref from byteorder(3), but no > comment on either that they will work.. Yes. We should fix this now that it isn=92t confined to the kernel. >> But maybe the more basic question is why do you even have packed >> structures that are native endian that you want to access as = naturally >> aligned structures? How did they become native endian and why weren?t >> they converted to a more natural layout at that time? >=20 > The original message said why=85 I personally think the original code should unconditionally call = load_ptr() and store_ptr() and if the optimization for aligned access is actually worth doing, the = test for it should be in those functions rather than inline throughout the code. The code will = be clearer, and it would be easier to optimize those cases that actually matter. I=92m frankly surprised that these relocations are being generated = unaligned. Perhaps that=92s the real bug here that should be fixed. While I=92m OK with the original = patch (subject to the above), I=92d be curious what other cases there are for this = functionality. You had said that you had additional use cases in the network stack, but I=92m having = trouble groking the use cases. If this is a huge deal, then defining functions to do this is trivial. = I=92m just not sure it is common enough to need a special macro/function call in the base. Warner --Apple-Mail=_0BBD206C-C6E0-434F-B718-ABB5CFD259E3 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTzUr1AAoJEGwc0Sh9sBEAl6MP+QGfnLxyKTIOrOuCgJ8ps7dW SewJMNsTNebtICWMUBe+SWZ7NDnKEr6M7T61o9wpdckc3n7K2XtrPrppiUHmDF79 Y+nXzJWAMAsDxyJ9Y8axXupV1WsA0xqQ4+VnHWx5wLCAjVkfejw3O5CqF9ljMVr2 jnyge07Kdrfd31Jz/WguuHuBp9E32wv4UCvYWUweRDdwKUmMW2oLMtdbo1beD3pH RLV4lKDMrjpoaiTc9VkaRBT97e4AJFHZQYvUqfb6T0PGQQb3I3cCAir96WbOwpBP NQR/kTg4zRdwsjSos99bwyEkLu1k9JHduEdCMpfB+Y7itff1aybnqFMZzRlNT9X1 ILYpOvvQBlI9QtjbtiEQV/ABGQtLr+R1yMHYqMk5SafAvrR3yQ48FpYFIimnGd2d IIoDZuJxt/yTiM/sXsHhxSDx3qWyfSspcYwuGwdRWiMeG+XDrXBoeoiC48eGMuwG 9jbb3FLNa8cZ3eMJlQn2GEIBBK9qttVBbZxLWK09V/e+fqL0htUzA8XadLrCZUSR uUYoZsgGefPDfty/H51PPSCiOSYABdBkYuUuUZzAsWTp8SXbD7apbyK1TztgkJ6I 7mLv7R0s7I1gWH2qqmn9dwDuEW8Auhku85xTc0XMm06NyQ7RqRcx4f4mX0fxRmOI jpMqP2vyWDtX4xNOsfUD =QDdl -----END PGP SIGNATURE----- --Apple-Mail=_0BBD206C-C6E0-434F-B718-ABB5CFD259E3-- From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 17:53:21 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D66EE8EF; Mon, 21 Jul 2014 17:53:21 +0000 (UTC) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 8158B21F3; Mon, 21 Jul 2014 17:53:21 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id D809D10525BD; Tue, 22 Jul 2014 03:53:11 +1000 (EST) Date: Tue, 22 Jul 2014 03:53:10 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Ian Lepore Subject: Re: [CFR] mge driver / elf reloc In-Reply-To: <1405955048.85788.74.camel@revolution.hippie.lan> Message-ID: <20140722022100.S2586@besplex.bde.org> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <1405955048.85788.74.camel@revolution.hippie.lan> MIME-Version: 1.0 X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=eojmkOZX c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=5zrNXb6StqYA:10 a=JzwRw_2MAAAA:8 a=nlC_4_pT8q9DhB4Ho9EA:9 a=cz2ZRIgtxKwA:10 a=wJWlkF7cXJYA:10 a=4FFUHJPBAAAA:8 a=SwFuXEZpslA4kWn8nBEA:9 a=nDTWHVfmmjYssBZ0:21 a=zWtDtkundl5VsItk:21 a=45ClL6m2LaAA:10 a=WglDhLyllLQA:10 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: arch@freebsd.org, freebsd-arm X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 17:53:22 -0000 On Mon, 21 Jul 2014, Ian Lepore wrote: > On Mon, 2014-07-21 at 08:46 -0600, Warner Losh wrote: >> On Jul 20, 2014, at 5:10 PM, John-Mark Gurney wrote: >> >>> Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: >>>> >>>> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney wrote= : >>>> >>>>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -0600: >>>>>> Sorry to take so long to reply to this, I'm trying to get caught up.= I >>>>>> see you've already committed the mge fixes. I think the ELF alignme= nt >>>>>> fix looks good and should also be committed. >>>>> >>>>> So, re the elf alignment... >>>>> >>>>> I think we should get a set of macros that handle load/stores to/from >>>>> unaligned addresses that are transparent to the caller.... I need >>>>> these for some other code I'm writing... >>>>> >>>>> I thought Open/Net had these available, but I can't seem to find them >>>>> right now... >>>> >>>> $ man 9 byteorder >>>> >>>> is most of what you want, lacking only some aliases to pick >>>> the correct macro for native byte order. >>> >>> Um, those doesn't help if you want native endian order=85 >> >> Ummm, yes they do. enc converts from native order. dec decodes to native= byte >> order. They are more general cases than the ntoh* functions that are mor= e traditional >> since they also work on byte streams that may not be completely aligned = when >> sitting in memory. Which is what you are asking for. >> >>> Also, only the enc/dec functions are documented to work on non-aligned >>> address, so that doesn't help in most cases=85 >> >> They work on all addresses. They are even documented to work on any addr= ess: >> >> The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64dec(= ), >> le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64dec(= ) >> functions encode and decode integers to/from byte strings on any al= ign- >> ment in big/little endian format. >> >> So they are quite useful in general. Peeking under the covers at the imp= lementation >> also shows they will work for any alignment, so I=92m having trouble und= erstanding >> where this objection is really coming from. > > The functionality requested was alignment-safe copy/assign without any > endian change. The code in question was conceptually something like > > if (pointer & 0x03) > do-alignment-safe-thing > else > directly-deref-the-pointer The enc/dec functions could be pessimized like that, but are actually pessimized in other ways. Pessimizations in the above include conditional branches for the check and large code. Everything has to be inlined else it is much slower than direct dereference, but then it has scattered branches that may exhaust the branch prediction cache (if any) and large code that may exhaust other caches. The enc/dec functions don't have any branches, but they have large code like: % static __inline void % le32enc(void *pp, uint32_t u) % { % =09uint8_t *p =3D (uint8_t *)pp; %=20 % =09p[0] =3D u & 0xff; % =09p[1] =3D (u >> 8) & 0xff; % =09p[2] =3D (u >> 16) & 0xff; % =09p[3] =3D (u >> 24) & 0xff; % } Fastest on x86 (unless alignment check exceptions are enabled in CR0) is to just do the access and let the hardware combine the bytes. The accesses should probably be written someything like le32enc() above, but more carefully. Compilers should convert the above to a single 32-bit access on x86 (unless alignment check exceptions...). The pessimizations are that compilers aren't that smart, and/or the enc/dec functions are written in such a way that compilers are constrained from doing this. IIRC, clang does this for one direction only and gcc-4.2.1 is not smart enough to do this for either direction. IIRC, the problematic direction is the above one, where the value is returned indirectly. Sprinkling __restrict might help. When there is endianness conversion, there must be both an access (hopefully 32 bits) and swapping operation on a register, unless all the bytes are copied from memory to memory 1 or 2 at a time. clang is smart about converting the large expressions in __bswapNN_gen() into single bswap instructions if the original rvalue is in a register, but IIRC it is not so smart for the equivalent conversions written with bytes and indirections. (x86 and IIRC some other arches have the __bswapNN_gen() macros. These macros are just as MI as the enc/dec inlines and more carefully written but they have not been deduplicated due to namespace problems inhibiting a complete cleanup of the MD endian.h files.) Accesses are often pessimized using the __packed mistake. This bug is still uncommon in ipv4 headers, but is used with minimal damage in struct ip. struct ip is declared as __packed and __aligned(4). Here __aligned(4) tells that the struct aligned normally although it is packed. ipv6 headers ask for full pessimizations by declaring almost everything as __packed without __aligned(N). This tells the compiler that the struct might only be 1-byte aligned, so all accesses to it must be 1 byte at a time except on arches like x86 where the above optimization applies (the optimization is much easier to do when the accesses are not expressed bytewise in the code). Bytewise accesses are also less inherently atomic. I think there used to be a problem with __packed and __aligned() attributes not being inherited by function pointers, but can't fdind any problem now. ia64 code for loading an int from a __packed struct (p.x): % addl r16 =3D @ltoffx(p#), r1 % ;; % ld8.mov r16 =3D [r16], p# % ;; % mov r14 =3D r16 % ;; % ld1 r15 =3D [r14], 1 % ;; % ld1 r14 =3D [r14] % ;; % shl r14 =3D r14, 8 % ;; % or r14 =3D r15, r14 % adds r15 =3D 2, r16 % ;; % ld1 r15 =3D [r15] % ;; % shl r15 =3D r15, 16 % ;; % or r15 =3D r14, r15 % adds r16 =3D 3, r16 % ;; % ld1 r8 =3D [r16] % ;; % shl r8 =3D r8, 24 % ;; % or r8 =3D r15, r8 It seems to have 5 memory references. The enc/dec32 functions produce a similar mess on x86 when the compiler can't optimize them. This is with gcc. clang doesn't work on ia64 and/or pluto. ia64 code for loading an int from a __packed __aligned(4) struct (p.x): addl r14 =3D @ltoffx(p#), r1 ;; ld8.mov r14 =3D [r14], p# ;; ld4 r8 =3D [r14] This behaviour can probable be exploited in the enc/dec functions. When there is no endianness conversion, write 32-bit accesses as p->x where p is a pointer to a __packed __aligned(1) struct containing x. The compiler will then produce the above mess on ia64 and a single 32-bit access if possible. No ifdefs required. When there is an endianess conversion, do it on a register with value p->x. Lightly tested implementation: % struct _le32 { % =09uint32_t _x; % } __packed; %=20 % #define=09le32enc(p, u)=09(((struct _le32 *)(p))->_x =3D (u)) This is simpler than the inline version, and fixes some namespace errors. If you want avoid the mess on ia64, this can be done at compile time if the alignment is known to be more than 1 then. I think if can often be known, as for struct ip (don't generate misaligned struct ip, and if you start with one then copy to an aligned one). The enc/dec inline functions could handle this by being converted to macros that take the alignment parameter. If the alignment is not known until runtime...don't do that. To add an alignment parameter to the above, use something like: #define=09_sle32(a)=09struct _le32 { uint32_t _x; } __packed __aligned(a) #define=09le32enc(p, u, a)=09(((_sle32(a) *)(p))->_x =3D (u)) Macroizing the struct declaration to add the alignment parameter to it and avoiding backslashes made the code less readable but even shorter. This was tested on i386 and ia64 with alignments 1 and 4 and and gave the expected output in asm. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 18:28:22 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 58A3F2AB; Mon, 21 Jul 2014 18:28:22 +0000 (UTC) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 140C624C6; Mon, 21 Jul 2014 18:28:21 +0000 (UTC) Received: from c-50-155-136-3.hsd1.co.comcast.net ([50.155.136.3] helo=ilsoft.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1X9IJv-000FsY-QT; Mon, 21 Jul 2014 18:28:20 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s6LISIeP033020; Mon, 21 Jul 2014 12:28:18 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 50.155.136.3 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1/IbHu2Ip3HrQmS1MVVZC3X X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: [CFR] mge driver / elf reloc From: Ian Lepore To: Warner Losh In-Reply-To: References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> <20140721165619.GT45513@funkthat.com> Content-Type: text/plain; charset="windows-1251" Date: Mon, 21 Jul 2014 12:28:17 -0600 Message-ID: <1405967297.85788.78.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by ilsoft.org id s6LISIeP033020 Cc: arch@freebsd.org, freebsd-arm X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 18:28:22 -0000 On Mon, 2014-07-21 at 11:16 -0600, Warner Losh wrote: > On Jul 21, 2014, at 10:56 AM, John-Mark Gurney wrote= : >=20 > > Warner Losh wrote this message on Mon, Jul 21, 2014 at 10:31 -0600: > >>=20 > >> On Jul 21, 2014, at 10:25 AM, John-Mark Gurney wr= ote: > >>=20 > >>> Warner Losh wrote this message on Mon, Jul 21, 2014 at 08:46 -0600: > >>>>=20 > >>>> On Jul 20, 2014, at 5:10 PM, John-Mark Gurney w= rote: > >>>>=20 > >>>>> Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -07= 00: > >>>>>>=20 > >>>>>> On Jul 20, 2014, at 3:05 PM, John-Mark Gurney = wrote: > >>>>>>=20 > >>>>>>> Ian Lepore wrote this message on Sat, Jul 19, 2014 at 16:54 -06= 00: > >>>>>>>> Sorry to take so long to reply to this, I'm trying to get caug= ht up. I > >>>>>>>> see you've already committed the mge fixes. I think the ELF a= lignment > >>>>>>>> fix looks good and should also be committed. > >>>>>>>=20 > >>>>>>> So, re the elf alignment... > >>>>>>>=20 > >>>>>>> I think we should get a set of macros that handle load/stores t= o/from > >>>>>>> unaligned addresses that are transparent to the caller.... I n= eed > >>>>>>> these for some other code I'm writing...=20 > >>>>>>>=20 > >>>>>>> I thought Open/Net had these available, but I can't seem to fin= d them > >>>>>>> right now... > >>>>>>=20 > >>>>>> $ man 9 byteorder > >>>>>>=20 > >>>>>> is most of what you want, lacking only some aliases to pick > >>>>>> the correct macro for native byte order. > >>>>>=20 > >>>>> Um, those doesn't help if you want native endian order? > >>>>=20 > >>>> Ummm, yes they do. enc converts from native order. dec decodes to = native byte > >>>=20 > >>> No they don't.. If you want to read a value in memory that is nativ= e > >>> endian order to native endian order (no conversion), they cannot be > >>> used w/o using something like below? > >>=20 > >> Missed the native to native. bcopy works, but is ugly, as you note b= elow. > >>=20 > >>>> order. They are more general cases than the ntoh* functions that a= re more traditional > >>>> since they also work on byte streams that may not be completely al= igned when > >>>> sitting in memory. Which is what you are asking for. > >>>=20 > >>> So, you're saying that I now need to write code like: > >>> #if LITTLE_ENDIAN /* or how ever this is spelled*/ > >>> var =3D le32enc(foo); > >>> #else > >>> var =3D be32enc(foo); > >>> #endif > >>>=20 > >>> If I want to read a arch native endian value? No thank you? > >>=20 > >> I?m not saying that at all. > >>=20 > >>>>> Also, only the enc/dec functions are documented to work on non-al= igned > >>>>> address, so that doesn't help in most cases? > >>>>=20 > >>>> They work on all addresses. They are even documented to work on an= y address: > >>>>=20 > >>>> The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64= dec(), > >>>> le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64= dec() > >>>> functions encode and decode integers to/from byte strings on an= y align- > >>>> ment in big/little endian format. > >>>>=20 > >>>> So they are quite useful in general. Peeking under the covers at t= he implementation > >>>> also shows they will work for any alignment, so I?m having trouble= understanding > >>>> where this objection is really coming from. > >>>=20 > >>> There are places where you write code such as: > >>> int i; > >>> memcpy(&i, inp, sizeof i); > >>> /* use i */ > >>>=20 > >>> In order to avoid alignment faults... None of the functions in byt= eorder > >>> do NO conversion of endian, or you have to know which endian you ar= e but > >>> that doesn't work on MI code... > >>>=20 > >>> Did you read what the commited code did? > >>=20 > >> No, I missed that bit beaded on your reply (which seemed to imply yo= u needed > >> endian conversion) which implied the enc/dec are only documented to = work on non-aligned > >> which is what I was correcting. > >=20 > > Hmm... It appears that byteorder(9) leaks to userland though isn't > > documented to be available in userland... Should we fix this and mak= e > > it an offical userland API? How to document it? In my case, the > > enc/dec version would have been enough for what I need, but not "part= of > > the userland API"... There is an xref from byteorder(3), but no > > comment on either that they will work.. >=20 > Yes. We should fix this now that it isn=92t confined to the kernel. >=20 > >> But maybe the more basic question is why do you even have packed > >> structures that are native endian that you want to access as natural= ly > >> aligned structures? How did they become native endian and why weren?= t > >> they converted to a more natural layout at that time? > >=20 > > The original message said why=85 >=20 > I personally think the original code should unconditionally call load_p= tr() and store_ptr() > and if the optimization for aligned access is actually worth doing, the= test for it should be > in those functions rather than inline throughout the code. The code wil= l be clearer, and > it would be easier to optimize those cases that actually matter. >=20 > I=92m frankly surprised that these relocations are being generated unal= igned. Perhaps that=92s > the real bug here that should be fixed. While I=92m OK with the origina= l patch (subject to the > above), I=92d be curious what other cases there are for this functional= ity. You had said that > you had additional use cases in the network stack, but I=92m having tro= uble groking the > use cases. >=20 > If this is a huge deal, then defining functions to do this is trivial. = I=92m just not sure it is common > enough to need a special macro/function call in the base. >=20 > Warner >=20 I was trying to figure out how unaligned relocs even happen, and I think one possibility would be something like this... char thing[] =3D "x"; struct { char x; char *something; } __packed foo =3D {'a', thing}; Which seems contrived enough that I wonder how it happens in real life. -- Ian From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 19:27:41 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF3F7857; Mon, 21 Jul 2014 19:27:41 +0000 (UTC) Received: from mail110.syd.optusnet.com.au (mail110.syd.optusnet.com.au [211.29.132.97]) by mx1.freebsd.org (Postfix) with ESMTP id 89B1A2A79; Mon, 21 Jul 2014 19:27:40 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail110.syd.optusnet.com.au (Postfix) with ESMTPS id 3D6A1789EA1; Tue, 22 Jul 2014 05:27:38 +1000 (EST) Date: Tue, 22 Jul 2014 05:27:37 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Warner Losh Subject: Re: [CFR] mge driver / elf reloc In-Reply-To: <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> Message-ID: <20140722041517.M3229@besplex.bde.org> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> MIME-Version: 1.0 X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=dZS5gxne c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=5zrNXb6StqYA:10 a=JzwRw_2MAAAA:8 a=nlC_4_pT8q9DhB4Ho9EA:9 a=cz2ZRIgtxKwA:10 a=wJWlkF7cXJYA:10 a=4FFUHJPBAAAA:8 a=YqLNQ_TDFebG-c2cgr8A:9 a=vXiuCQx8rN4fGCkZ:21 a=tLJyy6OA8_LDPdHi:21 a=45ClL6m2LaAA:10 a=WglDhLyllLQA:10 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: Tim Kientzle , Ian Lepore , John-Mark Gurney , arch@freebsd.org, freebsd-arm , Fabien Thomas X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 19:27:42 -0000 On Mon, 21 Jul 2014, Warner Losh wrote: > On Jul 21, 2014, at 10:25 AM, John-Mark Gurney wrote: > >> Warner Losh wrote this message on Mon, Jul 21, 2014 at 08:46 -0600: >>> >>> On Jul 20, 2014, at 5:10 PM, John-Mark Gurney wrote: >>> >>>> Tim Kientzle wrote this message on Sun, Jul 20, 2014 at 15:25 -0700: >>>>> $ man 9 byteorder >>>>> >>>>> is most of what you want, lacking only some aliases to pick >>>>> the correct macro for native byte order. >>>> >>>> Um, those doesn't help if you want native endian order? >>> >>> Ummm, yes they do. enc converts from native order. dec decodes to nativ= e byte >> >> No they don't.. If you want to read a value in memory that is native >> endian order to native endian order (no conversion), they cannot be >> used w/o using something like below=85 > > Missed the native to native. bcopy works, but is ugly, as you note below. Indeed, the API is missing support for the easy case of host load/store. But this is a feature. He used memcpy(), not bcopy(). This is a case where using memcpy() instead of bcopy() is not a style bug. Using memcpy() is pretty. >>> order. They are more general cases than the ntoh* functions that are mo= re traditional >>> since they also work on byte streams that may not be completely aligned= when >>> sitting in memory. Which is what you are asking for. >> >> So, you're saying that I now need to write code like: >> #if LITTLE_ENDIAN /* or how ever this is spelled*/ >> =09var =3D le32enc(foo); >> #else >> =09var =3D be32enc(foo); >> #endif >> >> If I want to read a arch native endian value? No thank you=85 > > I=92m not saying that at all. > >>>> Also, only the enc/dec functions are documented to work on non-aligned >>>> address, so that doesn't help in most cases? >>> >>> They work on all addresses. They are even documented to work on any add= ress: >>> >>> The be16enc(), be16dec(), be32enc(), be32dec(), be64enc(), be64dec(= ), >>> le16enc(), le16dec(), le32enc(), le32dec(), le64enc(), and le64dec(= ) >>> functions encode and decode integers to/from byte strings on any al= ign- >>> ment in big/little endian format. >>> >>> So they are quite useful in general. Peeking under the covers at the im= plementation >>> also shows they will work for any alignment, so I?m having trouble unde= rstanding >>> where this objection is really coming from. >> >> There are places where you write code such as: >> =09int i; >> =09memcpy(&i, inp, sizeof i); >> =09/* use i */ >> >> In order to avoid alignment faults... None of the functions in byteorde= r >> do NO conversion of endian, or you have to know which endian you are but >> that doesn't work on MI code... This is good code. memcpy() is likely to be optimized better than anything in . Even with the optimizations in my previous reply. Compilers do magic things with memcpy(), like turning it into a no-op if the memory is already in a register and the register doesn't need to be moved. Sometimes the register needs to be moved but the move can be, and is, to another register (perhaps in a different register set, like XMM instead of a general register on x86). Doing nothing much for memcpy()s that are just for a alignment is a special case of this. For the above, on all arches, the compiler should load the int-sized memory at address inp into an int-sized register and then rename the register to i (i should never hit memory unless it is changed and the change needs to be stored). The instructions that are used for the load depend on what the compiler knows about the alignment of inp, and the alignment requirements of the arch. On arches without strict alignment, like x86 without CR0_AC, there is nothing better than doing a mislaligned load *(int *)inp the same as you could do by lieing to the compiler. On arches with strict alignment, there is nothing better than loading 1 byte at a time and combining if the alignment of inp is 1 or unknown. This seems to make most of byteorder(9) a mistake. Just about everything can be done better using memcpy() and standard functions in byteorder(3), except the (3) functions only go up to 32 bits and have 1980's spellings [sl] for the integer sizes. All alignment stuff can be done in host order using memcpy(). Most byte swapping stuff can be done using ntohl(). The direction for n/h can be confusing but might need fewer ifdefs than be/le. le32enc done using more-standard APIs: first htole32() in a register. Only in byteorder(9) then store to an aligned uint32_t temporary variable (optimized away) then memcpy() to final place le32dec done using more-standard APIs: first memcpy() to aligned uint32_t temporary variable (optimized away) then load to a register then le32toh on a register. Only in byteorder(9) These take 1 line extra each (for the memcpy()). You already have to do this for h/l conversions. E.g.: htonl to from register to misaligned memory (like le32enc): first htonl() in the register. Standard in byteorder(3) and POSIX. then store to an aligned uint32_t temporary variable (optimized away) then memcpy() to final place -fbuiltin is essential for optimizing memcpy(), so memcpy must be spelled __builtin_memcpy in time-critical code if -fbuiltin might be turned off. -fbuiltin used to be only turned off in the kernel for LINT, but this was broken 13 years ago by using -ffreestanding. 13 years is not long enough for LINT (conf/NOTES) to have caught up with the change. clang breaks this even more. With gcc, you can use -fbuiltin after -ffreestanding to turn builtins back on, but with clang -ffreestanding has apparently has precedence so -fbuiltin never turns builtins back on. Also, gcc's man page actually documents these options, and somewhere in the gcc documentation there is a recommendation to keep -fbuiltin off and only use selected builtins via __builtin_foo. Turning all builtins back on originally only caused minor problems with a few builtins like the one for printf. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 19:55:35 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 870E12E3; Mon, 21 Jul 2014 19:55:35 +0000 (UTC) Received: from mail110.syd.optusnet.com.au (mail110.syd.optusnet.com.au [211.29.132.97]) by mx1.freebsd.org (Postfix) with ESMTP id 184C52D2D; Mon, 21 Jul 2014 19:55:34 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail110.syd.optusnet.com.au (Postfix) with ESMTPS id 874907857FB; Tue, 22 Jul 2014 05:55:31 +1000 (EST) Date: Tue, 22 Jul 2014 05:55:30 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans Subject: Re: [CFR] mge driver / elf reloc In-Reply-To: <20140722022100.S2586@besplex.bde.org> Message-ID: <20140722053651.V3446@besplex.bde.org> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <1405955048.85788.74.camel@revolution.hippie.lan> <20140722022100.S2586@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=dZS5gxne c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=5zrNXb6StqYA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=VRqNOJ3i96_1dxBxEUkA:9 a=CjuIK1q_8ugA:10 Cc: arch@freebsd.org, freebsd-arm , Ian Lepore X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 19:55:35 -0000 On Tue, 22 Jul 2014, Bruce Evans wrote: > ... > This behaviour can probable be exploited in the enc/dec functions. > When there is no endianness conversion, write 32-bit accesses as > p->x where p is a pointer to a __packed __aligned(1) struct containing > x. The compiler will then produce the above mess on ia64 and a single > 32-bit access if possible. No ifdefs required. When there is an > endianess conversion, do it on a register with value p->x. > > Lightly tested implementation: > > % struct _le32 { > % uint32_t _x; > % } __packed; > % % #define le32enc(p, u) (((struct _le32 *)(p))->_x = (u)) > > This is simpler than the inline version, and fixes some namespace errors. Oops, that is not so good. I just rememembed another optimization that at least clang does in at least some cases for the inline version: if the the compiler can see that the pointer is to aligned memory (which it can sometimes do since the function is inline), then the compiler generates 1 32-bit access. The __packed attribute in the above defeats this by forcing to 1-byte alignment. > If you want avoid the mess on ia64, this can be done at compile time if > the alignment is known to be more than 1 then. I think if can often > be known, as for struct ip (don't generate misaligned struct ip, and > if you start with one then copy to an aligned one). The enc/dec > inline functions could handle this by being converted to macros that > take the alignment parameter. If the alignment is not known until > runtime...don't do that. > > To add an alignment parameter to the above, use something like: > > #define _sle32(a) struct _le32 { uint32_t _x; } __packed __aligned(a) > #define le32enc(p, u, a) (((_sle32(a) *)(p))->_x = (u)) > > Macroizing the struct declaration to add the alignment parameter to it > and avoiding backslashes made the code less readable but even shorter. > > This was tested on i386 and ia64 with alignments 1 and 4 and and gave > the expected output in asm. Also not so good. If the pointer was 'a'-byte aligned, then the 'a' parameter recovers by tells the compiler this in a worse way after __packed clobbers the original alignment. It is better for the caller to do any alignment. In other replies, we discussed using memcpy() to force alignment. The above macros could use it to copy to aligned memory and load as if from that memory less hackishly. This would work in the inline functions, but doesn't do a lot that the compiler can't already do. It is still better for callers to do any alignment. They can use memcpy(), or if they know that the pointer is n-byte aligned then they can cast the pointer an __aligned(n) one where the inline fnction can see the cast. The original pointer type might be void * or char * so memcpy() for alignment would have to copy memory, but _aligned(n) would be virtual. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 20:06:07 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C08770C; Mon, 21 Jul 2014 20:06:07 +0000 (UTC) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "tensor.andric.com", Issuer "CAcert Class 3 Root" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9DB532E1D; Mon, 21 Jul 2014 20:06:06 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7::a5aa:5821:70ec:fa81] (unknown [IPv6:2001:7b8:3a7:0:a5aa:5821:70ec:fa81]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id D84F25C44; Mon, 21 Jul 2014 22:06:01 +0200 (CEST) Content-Type: multipart/signed; boundary="Apple-Mail=_C32C06B5-EAAB-4B90-B807-E0690CCC4FBF"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Total confusion over toolchain/xdev behavior From: Dimitry Andric In-Reply-To: <1405706147.19254.17.camel@bruno> Date: Mon, 21 Jul 2014 22:05:55 +0200 Message-Id: References: <1404688077.1059.115.camel@bruno> <1405706147.19254.17.camel@bruno> To: Sean Bruno X-Mailer: Apple Mail (2.1878.6) Cc: Baptiste Daroussin , freebsd-arch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 20:06:07 -0000 --Apple-Mail=_C32C06B5-EAAB-4B90-B807-E0690CCC4FBF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On 18 Jul 2014, at 19:55, Sean Bruno wrote: > On Sun, 2014-07-06 at 16:07 -0700, Sean Bruno wrote: >> Objective: install an xcompile toolchain into a jail for use by >> poudriere during arm/mips/sparc/power ports pkgs builds. The build >> should be possible from a non-root user. >>=20 >> As far as I can tell, the xdev target is completely busted for >> non-clang >> arch's right now as it tries to build clang no matter what I do. Its >> missing some pretty key documentation to making it work correctly, so >> a >> lot of my attempts have been "guess and check" with verbose make. >>=20 >>=20 >=20 > Quite a bit of success with one blocking failure. Thanks to Warner, > Baptiste and Dimitry for plugging along through some of my ranting (on > bcc here). >=20 > The xdev target can be used to produce a compiler toolchain that can = be > used to build ports. However, final linking seems to fail and makes = it > impossible to use for building many, many ports. >=20 > Regardless of the cross compile bits, simply using the xdev target for > building ports natively on amd64 for amd64 manifests this issue, this > leads me to believe that xdev is *not* building the tool chain > correctly. There is no chroot/jail/emualtion involved in this test > case. >=20 > My test case: >=20 > 1. build amd64 xdev tool chain: > make -s -j4 xdev XDEV=3Damd64 XDEV_ARCH=3Damd64 For me, this failed due to a problem with xdev-install trying to install include files into /usr/amd64-freebsd/usr/include/atf-c, which did not exist. I had to add the following part to Makefile.inc1 to make it = work: Index: Makefile.inc1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Makefile.inc1 (revision 268920) +++ Makefile.inc1 (working copy) @@ -1942,6 +1942,10 @@ -p ${XDDESTDIR}/usr >/dev/null mtree -deU -f ${.CURDIR}/etc/mtree/BSD.include.dist \ -p ${XDDESTDIR}/usr/include >/dev/null +.if ${MK_TESTS} !=3D "no" + mtree -deU -f ${.CURDIR}/etc/mtree/BSD.tests.dist \ + -p ${XDDESTDIR}/usr >/dev/null +.endif .ORDER: xdev-build _xi-mtree _xi-cross-tools _xi-includes _xi-libraries xdev-install: xdev-build _xi-mtree _xi-cross-tools _xi-includes = _xi-libraries > 2. modify make.conf to use this toolchain: > CC=3D/usr/amd64-freebsd/usr/bin/cc > CPP=3D/usr/amd64-freebsd/usr/bin/cpp > CXX=3D/usr/amd64-freebsd/usr/bin/cc++ > AS=3D/usr/amd64-freebsd/usr/bin/as > NM=3D/usr/amd64-freebsd/usr/bin/nm > RANLIB=3D/usr/amd64-freebsd/usr/bin/ranlib > LD=3D/usr/amd64-freebsd/usr/bin/ld > OBJCOPY=3D/usr/amd64-freebsd/usr/bin/objcopy > SIZE=3D/usr/amd64-freebsd/usr/bin/llvm-size > STRIPBIN=3D/usr/amd64-freebsd/usr/bin/strip >=20 > 3. attempt to build ports-mgmt/pkg In my case, this would fail much earlier, because it would try to use the native nm and ar, not the ones from /usr/amd64-freebsd/usr/bin. I had to specify X_BUILD_FOR=3Damd64-freebsd in make.conf, and modify bsd.port.mk as follows: Index: Mk/bsd.port.mk =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Mk/bsd.port.mk (revision 361353) +++ Mk/bsd.port.mk (working copy) @@ -1157,13 +1157,16 @@ .else X_SYSROOT=3D /usr/${X_BUILD_FOR} .endif -CC=3D ${X_SYSROOT}/usr/bin/cc -CXX=3D ${X_SYSROOT}/usr/bin/c++ -PKG_ENV+=3D ABI_FILE=3D${X_SYSROOT}/usr/lib/crt1.o -NM=3D ${X_BUILD_FOR}-nm -STRIP_CMD=3D ${X_BUILD_FOR}-strip -MAKE_ENV+=3D NM=3D${NM} STRIPBIN=3D${X_BUILD_FOR}-strip = PKG_CONFIG_SYSROOT_DIR=3D"${X_SYSROOT}" -CONFIGURE_ENV+=3D PKG_CONFIG_SYSROOT_DIR=3D"${X_SYSROOT}" +.for t in ADDR2LINE AR AS CC CPP CPPFILT GPROF LD NM OBJCOPY OBJDUMP = RANLIB READELF SIZE STRINGS +${t}=3D ${X_SYSROOT}/usr/bin/${t:C/PP/++/:tl} +CONFIGURE_ENV+=3D ${t}=3D"${${t}}" +MAKE_ENV+=3D ${t}=3D"${${t}}" +.endfor +CXX=3D ${X_SYSROOT}/usr/bin/c++ +STRIP_CMD=3D ${X_SYSROOT}/usr/bin/strip +CONFIGURE_ENV+=3D CXX=3D${CXX} STRIPBIN=3D${STRIP_CMD} = PKG_CONFIG_SYSROOT_DIR=3D"${X_SYSROOT}" +MAKE_ENV+=3D CXX=3D${CXX} STRIPBIN=3D${STRIP_CMD} = PKG_CONFIG_SYSROOT_DIR=3D"${X_SYSROOT}" +PKG_ENV+=3D ABI_FILE=3D${X_SYSROOT}/usr/lib/crt1.o # only bmake support the below STRIPBIN=3D ${STRIP_CMD} .export.env STRIPBIN This to ensure that all cross-tools are passed as environment variables to configure and make. > *NOTE* if you add -lbsdxml to CFLAGS this will not happen. Other > packages manfiest similar issues, with different libs. After quite a bit of debugging and instrumenting ld, it turns out this is because the "native" ld has access to a populated hints file (/var/run/ld-elf.so.hints), while the "cross" ld in /usr/amd64-freebsd does not. This can be seen as a bug, or at least inconsistent behavior on the part of ld. The code path followed in each case is subtly different. I'm still investigating what upstream did with it. For now, the way of least resistance would seem to be building ldconfig as part of xdev, and running it post-install. However, it turns out the hints file path is rather hardcoded in ldconfig, so to have that work, it should be made configurable at build time first. That said, the behavior of the "cross" ld seems *more* correct though, in my opinion: our choice (or at least the intention of it) was to stop pulling in DT_NEEDED libraries automatically, for compatibility with newer upstream binutils. E.g. adding just -larchive to a program that uses it should fail with several missing symbols, and you should manually specify the full dependency list: -larchive -lz -bz2 -llzma -lbsdxml -lcrypto This is also what the latest ld 2.24 requires. -Dimitry --Apple-Mail=_C32C06B5-EAAB-4B90-B807-E0690CCC4FBF Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iEYEARECAAYFAlPNcqgACgkQsF6jCi4glqOtjwCfTMz8IXNNRs3cSAZjYmecEc7Z F7gAn0fDCUpIsGjR+rSZ0s4625mgX0ts =Inrj -----END PGP SIGNATURE----- --Apple-Mail=_C32C06B5-EAAB-4B90-B807-E0690CCC4FBF-- From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 20:11:47 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF2D9A4D; Mon, 21 Jul 2014 20:11:47 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 9E5312EC9; Mon, 21 Jul 2014 20:11:47 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.48.2]) by phk.freebsd.dk (Postfix) with ESMTP id 8F4331578; Mon, 21 Jul 2014 20:04:46 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s6LK4hPp038241; Mon, 21 Jul 2014 20:04:44 GMT (envelope-from phk@phk.freebsd.dk) To: Bruce Evans Subject: Re: [CFR] mge driver / elf reloc In-reply-to: <20140722041517.M3229@besplex.bde.org> From: "Poul-Henning Kamp" References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> <20140722041517.M3229@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <38239.1405973083.1@critter.freebsd.dk> Date: Mon, 21 Jul 2014 20:04:43 +0000 Message-ID: <38240.1405973083@critter.freebsd.dk> Cc: Tim Kientzle , Ian Lepore , John-Mark Gurney , arch@freebsd.org, freebsd-arm , Fabien Thomas X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 20:11:48 -0000 In message <20140722041517.M3229@besplex.bde.org>, Bruce Evans writes: >This seems to make most of byteorder(9) a mistake. Just about everything >can be done better using memcpy() and standard functions in byteorder(3), Covered by Bruce' "just about" is that code analysers like Coverity and FlexeLint cannot see through his proposed version of the macros on little-endian architectures because of the inline assembler used to access the exchange instructions which the C-compiler do not expose. Analysers have no such problems with the pure-C versions there today. The real fault is with the ISO-C people who still lives in a world where endianess doesn't exist. If they had done their job sensibly, variables, including members of structs, could be declared as having a specific storage-layout. My theory is that they confident the PDP-10 and CRAY-1 are just about to make a comeback... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 22:18:30 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB697BFC; Mon, 21 Jul 2014 22:18:29 +0000 (UTC) Received: from mail108.syd.optusnet.com.au (mail108.syd.optusnet.com.au [211.29.132.59]) by mx1.freebsd.org (Postfix) with ESMTP id 6C5732AC0; Mon, 21 Jul 2014 22:18:28 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail108.syd.optusnet.com.au (Postfix) with ESMTPS id 04CF61A2E38; Tue, 22 Jul 2014 08:18:19 +1000 (EST) Date: Tue, 22 Jul 2014 08:18:18 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Poul-Henning Kamp Subject: Re: [CFR] mge driver / elf reloc In-Reply-To: <38240.1405973083@critter.freebsd.dk> Message-ID: <20140722065334.D3681@besplex.bde.org> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> <20140722041517.M3229@besplex.bde.org> <38240.1405973083@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=eojmkOZX c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=5zrNXb6StqYA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=wuhp96_2kGOEJxqg69kA:9 a=CjuIK1q_8ugA:10 Cc: Tim Kientzle , Ian Lepore , John-Mark Gurney , arch@freebsd.org, freebsd-arm , Fabien Thomas X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 22:18:30 -0000 On Mon, 21 Jul 2014, Poul-Henning Kamp wrote: > In message <20140722041517.M3229@besplex.bde.org>, Bruce Evans writes: > >> This seems to make most of byteorder(9) a mistake. Just about everything >> can be done better using memcpy() and standard functions in byteorder(3), > > Covered by Bruce' "just about" is that code analysers like Coverity > and FlexeLint cannot see through his proposed version of the macros > on little-endian architectures because of the inline assembler used > to access the exchange instructions which the C-compiler do not expose. Not a problem: - byteorder(9) already uses the bswap macros in many cases (including for all conversions to and from big-endian on little-endian hosts) - we removed the pure exchange instructions from the x86 bswap16() macro some time ago - we still use inline asm and bswap instructions in it for the x86 bswap32() and bswap64(), but this is just to avoid pessimizing for gcc or ifdefs to do it only for gcc. clang doesn't need any asms. - maybe code analyzers don't understand the __packed that I used. That would be either a bug in the analysers or just another unportability in __packed. > Analysers have no such problems with the pure-C versions there today. Further checking on pluto shows that gcc pessimizes all cases in le32dec() and le32enc(). It forgets the alignment of variables in the caller when pointers to them are passed as void *, even though the function are inline. On little-endian arches like ia64, these functions do no byte swapping so they are equivalent to memcpy() (after loading or storing the non-memory parameter to memory) Here they are using _builtin_memcpy(), with namespace bugs fixed: static __inline uint32_t le32dec(const void *_p) { uint32_t _t; __builtin_memcpy(&_t, _p, sizeof(_t)); return (_t); } static __inline void le32enc(void *_p, uint32_t _u) { __builtin_memcpy(_p, &_u, sizeof(_u)); } gcc optimizes these as well as possible. It now sees alignment in the caller. The above assumes a little-endian host. I think adding htole32()'s makes it work for all hosts. This involves ifdefs, but the ifdefs are already there to select htole32(). Hopefully this remains optimal when htole32() is nontrivial. It is probably just a missing optimization that old gcc doesn't see the alignment in the caller for non-builtins too, but the above is also simpler. It moves most expressions to the nontrivial htole32(), etc. For other sizes, s/32/64/, etc. No need for shifts and masks. These are now in htole32(), etc., or implicit in memcpy(). For other endiannesses, s/le/be/. No need to reverse the expressions. The reversal is in htobe32(), etc. Bruce From owner-freebsd-arch@FreeBSD.ORG Mon Jul 21 23:31:08 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ACDBD8CE; Mon, 21 Jul 2014 23:31:08 +0000 (UTC) Received: from mail.soaustin.net (pancho.soaustin.net [76.74.250.40]) by mx1.freebsd.org (Postfix) with ESMTP id 8B3DF211B; Mon, 21 Jul 2014 23:31:07 +0000 (UTC) Received: by mail.soaustin.net (Postfix, from userid 502) id 93D155607F; Mon, 21 Jul 2014 18:31:06 -0500 (CDT) Date: Mon, 21 Jul 2014 18:31:06 -0500 From: Mark Linimon To: Bruce Evans Subject: Re: [CFR] mge driver / elf reloc Message-ID: <20140721233106.GA17346@lonesome.com> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <1405955048.85788.74.camel@revolution.hippie.lan> <20140722022100.S2586@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140722022100.S2586@besplex.bde.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: arch@freebsd.org, freebsd-arm , Ian Lepore X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 23:31:08 -0000 On Tue, Jul 22, 2014 at 03:53:10AM +1000, Bruce Evans wrote: > This is with gcc. clang doesn't work on ia64 and/or pluto. Since Marcel has dropped support for ia64, and in fact removed ia64- specific code from -HEAD, I'm not sure how much good this analysis will accomplish :-) mcl From owner-freebsd-arch@FreeBSD.ORG Tue Jul 22 07:16:27 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8C282774; Tue, 22 Jul 2014 07:16:27 +0000 (UTC) Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by mx1.freebsd.org (Postfix) with ESMTP id 2075D24DE; Tue, 22 Jul 2014 07:16:26 +0000 (UTC) Received: from c122-106-147-133.carlnfd1.nsw.optusnet.com.au (c122-106-147-133.carlnfd1.nsw.optusnet.com.au [122.106.147.133]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id A59C410E2951; Tue, 22 Jul 2014 17:16:09 +1000 (EST) Date: Tue, 22 Jul 2014 17:16:08 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Mark Linimon Subject: Re: [CFR] mge driver / elf reloc In-Reply-To: <20140721233106.GA17346@lonesome.com> Message-ID: <20140722161501.K865@besplex.bde.org> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <1405955048.85788.74.camel@revolution.hippie.lan> <20140722022100.S2586@besplex.bde.org> <20140721233106.GA17346@lonesome.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=eojmkOZX c=1 sm=1 tr=0 a=7NqvjVvQucbO2RlWB8PEog==:117 a=PO7r1zJSAAAA:8 a=5zrNXb6StqYA:10 a=kj9zAlcOel0A:10 a=JzwRw_2MAAAA:8 a=hmZ8BYfCJ5Q0rm1RuK8A:9 a=CjuIK1q_8ugA:10 Cc: arch@freebsd.org, freebsd-arm , Ian Lepore X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 07:16:27 -0000 On Mon, 21 Jul 2014, Mark Linimon wrote: > On Tue, Jul 22, 2014 at 03:53:10AM +1000, Bruce Evans wrote: >> This is with gcc. clang doesn't work on ia64 and/or pluto. > > Since Marcel has dropped support for ia64, and in fact removed ia64- > specific code from -HEAD, I'm not sure how much good this analysis > will accomplish :-) ia64/pluto is just an example of an arch with strict alignment requirements. clang is broken for it so I could only test with gcc. This analysis applies to all non-x86 arches in FreeBSD cluster machines, since there aren't many others and the only other one (sparc64/flame) also has strict alignment requirements. clang is broken on it too, so I could only test with gcc: % #include % % struct foo { % int x; % } __packed; % % struct foo x; % % static __inline uint32_t % xle32dec(const void *_p) % { % uint32_t _t; % % __builtin_memcpy(&_t, _p, sizeof(_t)); % return (_t); % } % % static __inline void % xle32enc(void *_p, uint32_t _u) % { % % __builtin_memcpy(_p, &_u, sizeof(_u)); % } % % uint32_t % q(void) % { % return xle32dec(&x); % } % % void % r(void) % { % return xle32enc(&x, 1); % } This tests the memcpy versions. __packed gives the expected mess: % % .file "z.c" % .section ".text" % .align 4 % .align 32 % .global r % .type r, #function % .proc 020 % r: % .register %g2, #scratch % .register %g3, #scratch % add %sp, -208, %sp % mov 1, %g1 % st %g1, [%sp+2235] % sethi %hi(x), %g2 % or %g2, %lo(x), %g3 % ldub [%sp+2235], %g1 % stb %g1, [%g2+%lo(x)] % ldub [%sp+2236], %g1 % stb %g1, [%g3+1] % ldub [%sp+2237], %g1 % stb %g1, [%g3+2] % ldub [%sp+2238], %g1 % stb %g1, [%g3+3] % jmp %o7+8 % sub %sp, -208, %sp % .size r, .-r % .align 4 % .align 32 % .global q % .type q, #function % .proc 016 % q: % add %sp, -208, %sp % sethi %hi(x), %g1 % or %g1, %lo(x), %g2 % ldub [%g1+%lo(x)], %g1 % stb %g1, [%sp+2235] % ldub [%g2+1], %g1 % stb %g1, [%sp+2236] % ldub [%g2+2], %g1 % stb %g1, [%sp+2237] % ldub [%g2+3], %g1 % stb %g1, [%sp+2238] % lduw [%sp+2235], %o0 % jmp %o7+8 % sub %sp, -208, %sp % .size q, .-q % .common x,4,1 % .ident "GCC: (GNU) 4.2.1 20070831 patched [FreeBSD]" I think both functions copy the memory bytewise (4+4 memory references) and do 1 load of the final copy or 1 store to the temporary copy. So the memcpy is not virtual, and the memcpy versions might be worse than the -current versions which should use 4+1 memory references plus lots of shifts and masks on a registers. Register operations are faster but there are many more of them. Removing __packed gives the expected direct accesses: % .file "z.c" % .section ".text" % .align 4 % .align 32 % .global r % .type r, #function % .proc 020 % r: % .register %g2, #scratch % add %sp, -208, %sp % mov 1, %g2 % sethi %hi(x), %g1 % st %g2, [%g1+%lo(x)] % jmp %o7+8 % sub %sp, -208, %sp % .size r, .-r % .align 4 % .align 32 % .global q % .type q, #function % .proc 016 % q: % add %sp, -208, %sp % sethi %hi(x), %g1 % lduw [%g1+%lo(x)], %o0 % jmp %o7+8 % sub %sp, -208, %sp % .size q, .-q % .common x,4,4 % .ident "GCC: (GNU) 4.2.1 20070831 patched [FreeBSD]" The memcpy's seem to be virtual now. Maybe the compiler is avoiding the shifts and masks for the packed case intentionally. Timing tests on flame and pluto showed some problems. The memcpy versions are mostly faster in the non-__packed case and slower in the __packed case. This is as expected. The above case where the compiler virtulalize the memcpy is especially slow, as expected, but there are some other slow cases, an lots of differences between flame and pluto. The __packed case is 4-20 times slower. Bruce From owner-freebsd-arch@FreeBSD.ORG Tue Jul 22 07:38:00 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB5F9B52; Tue, 22 Jul 2014 07:38:00 +0000 (UTC) Received: from work.netasq.com (gwlille.netasq.com [91.212.116.1]) by mx1.freebsd.org (Postfix) with ESMTP id 606A62679; Tue, 22 Jul 2014 07:37:59 +0000 (UTC) Received: from work.netasq.com (localhost [127.0.0.1]) by work.netasq.com (Postfix) with ESMTP id BBE962700B44; Tue, 22 Jul 2014 09:37:57 +0200 (CEST) Received: from work.netasq.com (localhost [127.0.0.1]) by work.netasq.com (Postfix) with ESMTP id A136D2700A68; Tue, 22 Jul 2014 09:37:57 +0200 (CEST) Received: from [10.2.1.1] (unknown [91.212.116.2]) by work.netasq.com (Postfix) with ESMTPSA id 43C282700B44; Tue, 22 Jul 2014 09:37:57 +0200 (CEST) Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [CFR] mge driver / elf reloc From: Fabien Thomas In-Reply-To: Date: Tue, 22 Jul 2014 09:37:56 +0200 Message-Id: <3AE875C4-0259-4741-9433-BA0A9C1DCC40@freebsd.org> References: <14D22EA6-B73C-47BA-9A86-A957D24F23B8@freebsd.org> <1405810447.85788.41.camel@revolution.hippie.lan> <20140720220514.GP45513@funkthat.com> <20140720231056.GQ45513@funkthat.com> <9464C309-B390-4A27-981A-E854921B1C98@bsdimp.com> <20140721162559.GS45513@funkthat.com> <467619B1-F530-49AF-91BF-14CA3A31908B@bsdimp.com> <20140721165619.GT45513@funkthat.com> To: Warner Losh X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: Tim Kientzle , John-Mark Gurney , freebsd-arm , Ian Lepore , arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 07:38:00 -0000 >>=20 >> The original message said why=85 >=20 > I personally think the original code should unconditionally call = load_ptr() and store_ptr() > and if the optimization for aligned access is actually worth doing, = the test for it should be > in those functions rather than inline throughout the code. The code = will be clearer, and > it would be easier to optimize those cases that actually matter. >=20 > I=92m frankly surprised that these relocations are being generated = unaligned. Perhaps that=92s > the real bug here that should be fixed. While I=92m OK with the = original patch (subject to the > above), I=92d be curious what other cases there are for this = functionality. You had said that > you had additional use cases in the network stack, but I=92m having = trouble groking the > use cases. The problem was two years ago but if I remember correctly the module = used a packed structure with function pointer inside (at least). The fix came from NetBSD: = http://ftp.netbsd.org/pub/NetBSD/NetBSD-release-5/src/libexec/ld.elf_so/ar= ch/arm/mdreloc.c = http://ftp.netbsd.org/pub/NetBSD/NetBSD-release-5/src/libexec/ld.elf_so/ar= ch/mips/mips_reloc.c >=20 > If this is a huge deal, then defining functions to do this is trivial. = I=92m just not sure it is common > enough to need a special macro/function call in the base. >=20 > Warner From owner-freebsd-arch@FreeBSD.ORG Tue Jul 22 08:02:41 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DEB29F10; Tue, 22 Jul 2014 08:02:41 +0000 (UTC) Received: from mail-qg0-x22b.google.com (mail-qg0-x22b.google.com [IPv6:2607:f8b0:400d:c04::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FFFA28D1; Tue, 22 Jul 2014 08:02:41 +0000 (UTC) Received: by mail-qg0-f43.google.com with SMTP id a108so6648514qge.30 for ; Tue, 22 Jul 2014 01:02:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=mPZNMhTTGxlm9J/V9d02a7W59eW07vX2g5HuQosyiMk=; b=sR7KVaSvwaTDgivftK9IUXkyQZOCXYLCdC66ta0s78MD8BntDwTkMAzhLhGF5VTtMI MJM/Yp/jhAPcpuBRYro94Z5plgTDdeRytdf/oi1nwRy/buOc3AKkqg5mgrcru4c9AzlD p2blMeeqxJu3ZQ0dzRUPPb4sDpALciKOq79O++97oIo159wK4WElUlGHx2v9fda8o6eX cU9fxuP/Oy3TuXYPyZUgzlGxkfh/b+Hx2fuXRpkDXWl/BZBAJkKIY+Z3zd+IQwTjJ7W/ p94V4sq9lzoEzQ6MUW2I7w4PRUcJo5wXhW1dOXiVmaaudGCCH2Se2K5SnwCQc9thSOLV Qseg== MIME-Version: 1.0 X-Received: by 10.224.156.145 with SMTP id x17mr34137783qaw.49.1406016160188; Tue, 22 Jul 2014 01:02:40 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Tue, 22 Jul 2014 01:02:40 -0700 (PDT) Date: Tue, 22 Jul 2014 01:02:40 -0700 X-Google-Sender-Auth: JvCqFbRyxP-Yi9KyslREGjFtYKU Message-ID: Subject: [rfc] UDP RSS transmit/receive message options; ip_output() inp flowid changes From: Adrian Chadd To: FreeBSD Net , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 08:02:42 -0000 Hi! I'd appreciate a review of this: http://people.freebsd.org/~adrian/rss/20140722-rss-udp-1.diff The overview: * Add a new flag to ip_output() to instruct it to not override the flowid with the inp cached value. Some forms of UDP transmit will break this. * Add new IP socket options to receive flowid and rss bucket information with the UDP datagram (via recvmsg.) * Add awareness in the send path of those same messages so the transmit path can avoid doing a hash calculation. The software hash calculations aren't done yet in the UDP transmit path - I'll work on those next. There are some comments which won't make it into the final committed patch. Bonus points if you get this far and see what they are. -a From owner-freebsd-arch@FreeBSD.ORG Tue Jul 22 16:29:04 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7FBB159C for ; Tue, 22 Jul 2014 16:29:04 +0000 (UTC) Received: from mail.ignoranthack.me (ignoranthack.me [199.102.79.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5E2BC2931 for ; Tue, 22 Jul 2014 16:29:04 +0000 (UTC) Received: from [192.168.200.205] (c-50-131-5-126.hsd1.ca.comcast.net [50.131.5.126]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: sbruno@ignoranthack.me) by mail.ignoranthack.me (Postfix) with ESMTPSA id 6B85A193A3A for ; Tue, 22 Jul 2014 16:29:03 +0000 (UTC) Subject: Re: Total confusion over toolchain/xdev behavior, follow-up discussions From: Sean Bruno Reply-To: sbruno@freebsd.org To: freebsd-arch In-Reply-To: References: <1404688077.1059.115.camel@bruno> <1405706147.19254.17.camel@bruno> Content-Type: text/plain; charset="us-ascii" Date: Tue, 22 Jul 2014 09:29:02 -0700 Message-ID: <1406046542.1063.31.camel@bruno> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 16:29:04 -0000 On Mon, 2014-07-21 at 22:05 +0200, Dimitry Andric wrote: > On 18 Jul 2014, at 19:55, Sean Bruno wrote: > > On Sun, 2014-07-06 at 16:07 -0700, Sean Bruno wrote: > >> Objective: install an xcompile toolchain into a jail for use by > >> poudriere during arm/mips/sparc/power ports pkgs builds. The build > >> should be possible from a non-root user. > >> > > E.g. adding just -larchive to a program that uses it should fail with > several missing symbols, and you should manually specify the full > dependency list: > > -larchive -lz -bz2 -llzma -lbsdxml -lcrypto > > This is also what the latest ld 2.24 requires. > > -Dimitry > We've had some side channel discussions about this, and we came to the conclusion that its time to modify ports to explicitly link required libs. Therefore, post pkg 1.3, I'll start doing xdev builds to weed out the ports that need attention and start getting things into shape for future builds. sean From owner-freebsd-arch@FreeBSD.ORG Tue Jul 22 23:28:31 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C244F00; Tue, 22 Jul 2014 23:28:31 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id D722723CA; Tue, 22 Jul 2014 23:28:30 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 876B146B38; Tue, 22 Jul 2014 19:28:29 -0400 (EDT) Date: Wed, 23 Jul 2014 00:28:29 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Shawn Webb Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch In-Reply-To: <20140720201858.GB29618@pwnie.vrt.sourcefire.com> Message-ID: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> User-Agent: Alpine 2.11 (BSF 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: PaX Team , Pedro Giffuni , Oliver Pinter , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 23:28:31 -0000 On Sun, 20 Jul 2014, Shawn Webb wrote: >> - It is yet undetermined what the performance effect will be, and it is not >> clear (but seems likely from past measurements) if there will be a >> performance hit even when ASLR is off. -Apparently there are applications >> that will segfault (?). > > So I have an old Dell Latitude E6500 that I bought at Defcon a year or > so ago that I'm doing testing on. Even though it's quite an underpowered > laptop, I'm running ZFS on it for BE support (in case one of our changes > kills it). I'll run unixbench on it a few times to benchmark the ASLR > patch. I'll test these three scenarios: > 1) ASLR compiled in and enabled; > 2) ASLR compiled in and disabled; > 3) ASLR compiled out (GENERIC kernel). > > In each of these three scenarios, I'll have the kernel debugging features > (WITNESS, INVARIANTS, etc.) turned off to better simulate a production > system and to remove just one more variable in the tests. > > I'll run unixbench ten times under each scenario and I'll compute averages. > > Since this is an older laptop (and it's running ZFS), these tests will take > a couple days. I'll have an answer for you soon. Hi Shawn: Great news that this work is coming to fruition -- ASLR is long overdue. Are you having any luck with performance measurements? Unixbench seems like a good starting point, but I wonder if it would be useful to look, in particular, at memory-mapping intensive workloads that might be affected as a result of changes in kernel VM data-structure use, or greater fragmentation of the address space. I'm not sure I have a specific application here in mind -- in the past I might have pointed out tools such as ElectricFence that tend to increase fragmentation themselves. Also, could you say a little more about the effects that the change might have on transparent superpage use -- other than suitable alignment of large mappings, it's not clear to me what effect it might have. I wonder if some equipment in the FreeBSD Netperf cluster might be used to help with performance characterisation -- in particular, very recent high-end server hardware, and also, lower-end embedded-style systems that have markedly different virtual-memory implementations in hardware and software. Often those two classes of systems see markedly different performance-change characteristics as a result of greater cache-centrism and instruction-level parallelism in the higher-end designs that can mask increases in instruction count. I think someone has already commented that Peter Holm's help might be enlisted; you have have seen his 'stress2' suite, which could help with stability testing. Thanks, Robert From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 00:45:47 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55A7123E; Wed, 23 Jul 2014 00:45:47 +0000 (UTC) Received: from mail-qa0-x22c.google.com (mail-qa0-x22c.google.com [IPv6:2607:f8b0:400d:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D034029F6; Wed, 23 Jul 2014 00:45:46 +0000 (UTC) Received: by mail-qa0-f44.google.com with SMTP id f12so517952qad.31 for ; Tue, 22 Jul 2014 17:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=pCzA7sivE+EYhnyF8qO9LSNwLFeabMVSDhkDypDj888=; b=mXh4OERqFg+SGxARg4NmbEiBJeD8IRVwe6/DknBJFU4je0y41Exmzb6jMhZD/siOmc lJ55xTLwoeWqtVgQeZWN9BGf09AaUoRAq6IxrUP90Jzx0qQCfO9DiWLSZnrc7sJg565N G3DrN6R/XG/K/G89OTV5BIAskYo906GkFqAtgXnG1D2ZZsuCETDCYcNfMmnmShA9WK9A 7SyIyUXNJSsjQOFP4FKBX/RLKGV7MzZynSn0uzRQQB9VcRTmM11sIbKinu3ZyEp/Pv7P zTqOJsEKNMhwjC7iP09p3InmQovCHceg/+nV2GKl55699eMvhDnb30QNbPzNV5hxscsd LHNg== X-Received: by 10.140.27.144 with SMTP id 16mr691632qgx.18.1406076345663; Tue, 22 Jul 2014 17:45:45 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id w15sm1219200qay.34.2014.07.22.17.45.44 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Jul 2014 17:45:44 -0700 (PDT) Date: Tue, 22 Jul 2014 20:45:43 -0400 From: Shawn Webb To: Robert Watson Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140723004543.GH29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6lXr1rPCNTf1w0X8" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , Pedro Giffuni , Oliver Pinter , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 00:45:47 -0000 --6lXr1rPCNTf1w0X8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 23, 2014 12:28 AM +0100, Robert Watson wrote: > On Sun, 20 Jul 2014, Shawn Webb wrote: >=20 > >> - It is yet undetermined what the performance effect will be, and it i= s not=20 > >> clear (but seems likely from past measurements) if there will be a=20 > >> performance hit even when ASLR is off. -Apparently there are applicati= ons=20 > >> that will segfault (?). > > > > So I have an old Dell Latitude E6500 that I bought at Defcon a year or > > so ago that I'm doing testing on. Even though it's quite an underpowered > > laptop, I'm running ZFS on it for BE support (in case one of our changes > > kills it). I'll run unixbench on it a few times to benchmark the ASLR > > patch. I'll test these three scenarios: > > 1) ASLR compiled in and enabled; > > 2) ASLR compiled in and disabled; > > 3) ASLR compiled out (GENERIC kernel). > > > > In each of these three scenarios, I'll have the kernel debugging featur= es=20 > > (WITNESS, INVARIANTS, etc.) turned off to better simulate a production= =20 > > system and to remove just one more variable in the tests. > > > > I'll run unixbench ten times under each scenario and I'll compute avera= ges. > > > > Since this is an older laptop (and it's running ZFS), these tests will = take=20 > > a couple days. I'll have an answer for you soon. >=20 > Hi Shawn: >=20 > Great news that this work is coming to fruition -- ASLR is long overdue. >=20 > Are you having any luck with performance measurements? Unixbench seems l= ike a=20 > good starting point, but I wonder if it would be useful to look, in=20 > particular, at memory-mapping intensive workloads that might be affected = as a=20 > result of changes in kernel VM data-structure use, or greater fragmentati= on of=20 > the address space. I'm not sure I have a specific application here in mi= nd --=20 > in the past I might have pointed out tools such as ElectricFence that ten= d to=20 > increase fragmentation themselves. The unixbench tests on that laptop have finished. However, I've been fighting a pesky migraine these last couple days, so I haven't had the opportunity to aggregate the results into a nice little spreadsheet. I'm hoping to finish it up by the end of the week. I'll take a look at ElectricFence this weekend. Additionally, I have a netbook somewhere. Once I find it and its power cord, I'll install FreeBSD/x86 and re-run the same tests on that. >=20 > Also, could you say a little more about the effects that the change might= have=20 > on transparent superpage use -- other than suitable alignment of large=20 > mappings, it's not clear to me what effect it might have. Since we're just modifying the hint passed to the underlying VM system, superpage support works as it should with ASLR enabled. The VM system will modify the hint in order to be able to use superpages. In those cases, we might lose a little bit of entropy. However, due to superpages (on amd64, at least) requring 2MB alignment, you'd lose some entropy no matter how ASLR was implemented--at the end of the day, you need that alignment for superpages to work. >=20 > I wonder if some equipment in the FreeBSD Netperf cluster might be used t= o=20 > help with performance characterisation -- in particular, very recent high= -end=20 > server hardware, and also, lower-end embedded-style systems that have mar= kedly=20 > different virtual-memory implementations in hardware and software. Often= =20 > those two classes of systems see markedly different performance-change=20 > characteristics as a result of greater cache-centrism and instruction-lev= el=20 > parallelism in the higher-end designs that can mask increases in instruct= ion=20 > count. Any additional testing would be very much welcome. Our ASLR implementation misbehaves on ARM, so testing on ARM-based embedded devices is pretty limited. My next goal is to figure out why it bugs out on ARM. Essentially, when a child process exits/dies and the parent process gets sent SIGCHLD, the parent process' pc register somehow gets set to 0xc0000000 and segfaults. Here's a screenshot of the process: https://twitter.com/lattera/status/490529645997998080 FreeBSD 11-CURRENT hasn't been stable at all on sparc64, even without the ASLR patches. I have an SunFire 280R box that I've attempted to test ASLR our on, but I couldn't get a stable enough installation of vanilla FreeBSD to work long enough to recompile world/kernel. And generating an installation ISO from my amd64 box doesn't work as the VTOC8 bootloader isn't recognized by the BIOS (not sure if that's what it's called in sparc land). >=20 > I think someone has already commented that Peter Holm's help might be=20 > enlisted; you have have seen his 'stress2' suite, which could help with= =20 > stability testing. I'll take a look at that, too. Thanks a lot for your suggestions and feedback. Thanks, Shawn --6lXr1rPCNTf1w0X8 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTzwW2AAoJEGqEZY9SRW7u/ZAP/3eiNNfWqYY5R/ZL+j98/amy GNeADbDO8OaPQRQaTYhQADU//XIbbf4mUFsj9SO9FPJxOka9h8UsBfNMF6jspnnw 2RAXvbnelXWlfLl1p+U0KE3umZUw1Ukm7IPs+KWwaaAlgaKcGEyOk9RQpzcNiqbD qDW5ubq8AGBvgSNWOOQGnc7G5IqegNScZxv78ZWwDV/c9I8g51sJySDEJjSE05bk QHSBDeNlg3+lJQH7NqRerH6GM532QueILCvVr5ARbiSvFufsYtvuHY3nI+eTnEko alNQVQ/ITmYZ/WWH0KP9sF8itS2+jcfuIo+LueETB11TBiRCtuneRtYBDNb/UaTs LDl1WcJ5RB0RoQpgUpPNSCkndhuilT7wARXKOYX3o9hWoEfu+xxkpmzo5aVVEs4t tjUfF3SuqiDTXXf6LvbQafpW1cX8gt1a6selBO7r417ANBdXpm1UN99kQrCfCBJj g4IDKBotb78xG4o+ES/YR6Je65O6CIRVt4tGZPSM5Ej4mdVIGjElTR05Wq7l5WmU utwJnWHSAwHnlFOSb8aR6TTF3OSxCdILS5gPeGbK6Jym9jcbjAv+PD6aeLMANUqn czC2gRLQ+Qpuu41o8Ti6AyN4toKJDFvP4RBQWK0xnGCaAtYU6SnW32TVpbhe0GrB 40+zyhUfD0Zy3phhq4vm =ErzN -----END PGP SIGNATURE----- --6lXr1rPCNTf1w0X8-- From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 19:15:46 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B92A5D58 for ; Wed, 23 Jul 2014 19:15:46 +0000 (UTC) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2lp0237.outbound.protection.outlook.com [207.46.163.237]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A8462FC8 for ; Wed, 23 Jul 2014 19:15:45 +0000 (UTC) Received: from BL2PR05CA0036.namprd05.prod.outlook.com (10.255.226.36) by BLUPR05MB721.namprd05.prod.outlook.com (10.141.207.144) with Microsoft SMTP Server (TLS) id 15.0.995.14; Wed, 23 Jul 2014 17:42:15 +0000 Received: from BY2FFO11FD054.protection.gbl (2a01:111:f400:7c0c::104) by BL2PR05CA0036.outlook.office365.com (2a01:111:e400:c04::36) with Microsoft SMTP Server (TLS) id 15.0.990.7 via Frontend Transport; Wed, 23 Jul 2014 17:42:14 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD054.mail.protection.outlook.com (10.1.15.191) with Microsoft SMTP Server (TLS) id 15.0.980.11 via Frontend Transport; Wed, 23 Jul 2014 17:42:14 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Wed, 23 Jul 2014 10:42:13 -0700 Received: from juniper.net (eta.jnpr.net [172.21.19.189]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6NHgBn97255 for ; Wed, 23 Jul 2014 10:42:12 -0700 (PDT) (envelope-from amesh@juniper.net) Date: Wed, 23 Jul 2014 10:42:11 -0700 From: Arthur Mesh To: Subject: pam_lastlog Message-ID: <20140723174211.GQ57013@juniper.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="tgGnixv3tJWXBxdL" Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(44976005)(84676001)(99396002)(19580395003)(97736001)(83322001)(46102001)(551544002)(84326002)(85306003)(71186001)(92566001)(81342001)(92726001)(77982001)(21056001)(54356999)(87936001)(50986999)(79102001)(19580405001)(31966008)(16796002)(76482001)(74502001)(36756003)(74662001)(64706001)(102836001)(4396001)(107046002)(20776003)(2351001)(80022001)(229853001)(107886001)(86362001)(33656002)(85852003)(83072002)(106466001)(81156004)(69596002)(512954002)(6806004)(68736004)(81542001)(83506001)(105596002)(95666004)(110136001); DIR:OUT; SFP:; SCL:1; SRVR:BLUPR05MB721; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 028166BF91 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=amesh@juniper.net; X-OriginatorOrg: juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 19:15:46 -0000 --tgGnixv3tJWXBxdL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Greetings, At Juniper Networks, we have a concept of "template" system users, where actual users of the system are defined on remote authentication servers such as Radius or Tacacs+. These users are mapped to a single locally defined user (called template user). Such mapping makes it easier for sysadmins to manage large amount of deployed systems, etc using readily available RADIUS/TACACS+ deployments. Most of the glue to make this 1:N mapping work is done via various changes to various PAM modules. While reading some existing PAM modules used by FreeBSD, we came across pam_lastlog.so (session management module responsible for updating accounting database (utmpx)) that does something curious. Prior to doing its thing, pam_lastlog always ensures that the username in question exists in the password database. Given that session management happens only after authentication (pam_authenticate(3)) has succeeded and account has been validated (pam_acct_mgmt(3)), this seems like a layering violation. Thoughts? Here is a proposed change where that adds a knob to disable this lookup: Index: lib/libpam/modules/pam_lastlog/pam_lastlog.8 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- lib/libpam/modules/pam_lastlog/pam_lastlog.8 (revision 282460) +++ lib/libpam/modules/pam_lastlog/pam_lastlog.8 (working copy) @@ -81,6 +81,8 @@ suppress warning messages to the user. .It Cm no_fail Ignore I/O failures. +.It Cm no_user_lookup +Skip looking up user account. .El .Sh SEE ALSO .Xr last 1 , Index: lib/libpam/modules/pam_lastlog/pam_lastlog.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- lib/libpam/modules/pam_lastlog/pam_lastlog.c (revision 282460) +++ lib/libpam/modules/pam_lastlog/pam_lastlog.c (working copy) @@ -68,7 +68,6 @@ pam_sm_open_session(pam_handle_t *pamh, int flags, int argc __unused, const char *argv[] __unused) { - struct passwd *pwd; struct utmpx *utx, utl; time_t t; const char *user; @@ -79,8 +78,11 @@ pam_err =3D pam_get_user(pamh, &user, NULL); if (pam_err !=3D PAM_SUCCESS) return (pam_err); - if (user =3D=3D NULL || (pwd =3D getpwnam(user)) =3D=3D NULL) + if (user =3D=3D NULL) return (PAM_SERVICE_ERR); + if (openpam_get_option(pamh, "no_user_lookup") =3D=3D NULL && + getpwnam(user) =3D=3D NULL) + return (PAM_SERVICE_ERR); PAM_LOG("Got user: %s", user); =20 pam_err =3D pam_get_item(pamh, PAM_RHOST, &rhost); --=20 Arthur Mesh Juniper Networks +1 408 936-4968 --tgGnixv3tJWXBxdL Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAEBCAAGBQJTz/PzAAoJEO/ZUtudxDntv/4L/jOA4h8u9jPhpKvLt5+FWFGu bts+CiYiwu3SUc5Dsd2y7pLfHTch4xQHXM1kC6MPbNC1rH6E+k6Ma7WNvanfomdq mgZy+dLsjqyYVaSKTpvyVGEo/9jIpNUK/Y+vbBsJzqXBAqSmwY65sLPZVSjVs67u EgCSIqS/B789tZuvDj43pRui2LYWKAy2eDhy6mU91EvFmIhGCiW3Bw8WkqWAIzuZ 0KAOO1fBMs2c1tZE7Gcy7FdEDwjhX8pX4WzWsLWMiRZ/eEYAMC+tVkx9+xIZegAc p6PjIdDEkOe5ncm1OSUyFnC8qcGWHh0LL5jTjc8Dx02Jy0zlU630Dy82Pu4DgtC/ AKV0P9FdRmoc0iGFB9Ms6DG5GwOji/8mjGDttHlBL5S5RdUwCyGZKF9AFHDmceE3 ceNDrWFxqck9GwvBhqYRIQETK14p2El6RUaNnt9lpVty7S/ECklutcQ82STZ9+I6 r7i827rOSJxufhYQz9ujD6/e5Zka32KMDCZexRC21A== =0l45 -----END PGP SIGNATURE----- --tgGnixv3tJWXBxdL-- From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 21:47:12 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 68F15EE2; Wed, 23 Jul 2014 21:47:12 +0000 (UTC) Received: from mail-qg0-x22d.google.com (mail-qg0-x22d.google.com [IPv6:2607:f8b0:400d:c04::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E02C12E05; Wed, 23 Jul 2014 21:47:11 +0000 (UTC) Received: by mail-qg0-f45.google.com with SMTP id f51so2207364qge.4 for ; Wed, 23 Jul 2014 14:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=jH0Y1nxmk7HSnJXrKKhKv9OOXAGqOUSnJINt5KfbAbw=; b=TSF6U+Q2K6Qst3MU2eoeAYUijk3KarPjm+ceXUr2ilayM+6iYIHhiigvx5z6ThlJyf YplseFNMjPS4PRCpo6GHxuWCZX+uJQYD1eI/KpnBY4U618TAr1CtRskp2a1/Z/EX1hJk MV5ztQNm2edfNTyKKqHnw5NIgY1NQ23h0g3BjObcgQ7O50BxpjESZrmHaGyGa2TVDc1c WZ6aMB4cWOPuNMWQPFmrTwXZxoAmIiuen9usDQmBsoX1qM4liOSjZV6YJIdFzGLOs3fF o9Fy3mgSEKcjzfC6Cw7mHCQoJswa2ZwNHb1z864pVC5P7CMtlHqaHzD+RucU/i3jrxVo Qfdw== X-Received: by 10.140.21.137 with SMTP id 9mr6605327qgl.37.1406152030952; Wed, 23 Jul 2014 14:47:10 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id z11sm5919773qaz.45.2014.07.23.14.47.09 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jul 2014 14:47:10 -0700 (PDT) Date: Wed, 23 Jul 2014 17:47:08 -0400 From: Shawn Webb To: Robert Watson Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140723214708.GO29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KM+e2hnYAO+MCJ5e" Content-Disposition: inline In-Reply-To: <20140723004543.GH29618@pwnie.vrt.sourcefire.com> X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , Pedro Giffuni , Oliver Pinter , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 21:47:12 -0000 --KM+e2hnYAO+MCJ5e Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 22, 2014 08:45 PM -0400, Shawn Webb wrote: > On Jul 23, 2014 12:28 AM +0100, Robert Watson wrote: > > On Sun, 20 Jul 2014, Shawn Webb wrote: > >=20 > > >> - It is yet undetermined what the performance effect will be, and it= is not=20 > > >> clear (but seems likely from past measurements) if there will be a= =20 > > >> performance hit even when ASLR is off. -Apparently there are applica= tions=20 > > >> that will segfault (?). > > > > > > So I have an old Dell Latitude E6500 that I bought at Defcon a year or > > > so ago that I'm doing testing on. Even though it's quite an underpowe= red > > > laptop, I'm running ZFS on it for BE support (in case one of our chan= ges > > > kills it). I'll run unixbench on it a few times to benchmark the ASLR > > > patch. I'll test these three scenarios: > > > 1) ASLR compiled in and enabled; > > > 2) ASLR compiled in and disabled; > > > 3) ASLR compiled out (GENERIC kernel). > > > > > > In each of these three scenarios, I'll have the kernel debugging feat= ures=20 > > > (WITNESS, INVARIANTS, etc.) turned off to better simulate a productio= n=20 > > > system and to remove just one more variable in the tests. > > > > > > I'll run unixbench ten times under each scenario and I'll compute ave= rages. > > > > > > Since this is an older laptop (and it's running ZFS), these tests wil= l take=20 > > > a couple days. I'll have an answer for you soon. > >=20 > > Hi Shawn: > >=20 > > Great news that this work is coming to fruition -- ASLR is long overdue. > >=20 > > Are you having any luck with performance measurements? Unixbench seems= like a=20 > > good starting point, but I wonder if it would be useful to look, in=20 > > particular, at memory-mapping intensive workloads that might be affecte= d as a=20 > > result of changes in kernel VM data-structure use, or greater fragmenta= tion of=20 > > the address space. I'm not sure I have a specific application here in = mind --=20 > > in the past I might have pointed out tools such as ElectricFence that t= end to=20 > > increase fragmentation themselves. >=20 > The unixbench tests on that laptop have finished. However, I've been > fighting a pesky migraine these last couple days, so I haven't had the > opportunity to aggregate the results into a nice little spreadsheet. I'm > hoping to finish it up by the end of the week. >=20 > I'll take a look at ElectricFence this weekend. Additionally, I have a > netbook somewhere. Once I find it and its power cord, I'll install > FreeBSD/x86 and re-run the same tests on that. >=20 > >=20 > > Also, could you say a little more about the effects that the change mig= ht have=20 > > on transparent superpage use -- other than suitable alignment of large= =20 > > mappings, it's not clear to me what effect it might have. >=20 > Since we're just modifying the hint passed to the underlying VM system, > superpage support works as it should with ASLR enabled. The VM system > will modify the hint in order to be able to use superpages. In those > cases, we might lose a little bit of entropy. However, due to superpages > (on amd64, at least) requring 2MB alignment, you'd lose some entropy no > matter how ASLR was implemented--at the end of the day, you need that > alignment for superpages to work. >=20 > >=20 > > I wonder if some equipment in the FreeBSD Netperf cluster might be used= to=20 > > help with performance characterisation -- in particular, very recent hi= gh-end=20 > > server hardware, and also, lower-end embedded-style systems that have m= arkedly=20 > > different virtual-memory implementations in hardware and software. Oft= en=20 > > those two classes of systems see markedly different performance-change= =20 > > characteristics as a result of greater cache-centrism and instruction-l= evel=20 > > parallelism in the higher-end designs that can mask increases in instru= ction=20 > > count. >=20 > Any additional testing would be very much welcome. Our ASLR > implementation misbehaves on ARM, so testing on ARM-based embedded > devices is pretty limited. My next goal is to figure out why it bugs out > on ARM. Essentially, when a child process exits/dies and the parent > process gets sent SIGCHLD, the parent process' pc register somehow gets > set to 0xc0000000 and segfaults. Here's a screenshot of the process: > https://twitter.com/lattera/status/490529645997998080 >=20 > FreeBSD 11-CURRENT hasn't been stable at all on sparc64, even without > the ASLR patches. I have an SunFire 280R box that I've attempted to test > ASLR our on, but I couldn't get a stable enough installation of vanilla > FreeBSD to work long enough to recompile world/kernel. And generating an > installation ISO from my amd64 box doesn't work as the VTOC8 bootloader > isn't recognized by the BIOS (not sure if that's what it's called in > sparc land). >=20 > >=20 > > I think someone has already commented that Peter Holm's help might be= =20 > > enlisted; you have have seen his 'stress2' suite, which could help with= =20 > > stability testing. >=20 > I'll take a look at that, too. Thanks a lot for your suggestions and > feedback. >=20 > Thanks, >=20 > Shawn Update: I've uploaded the patch for review to phabricator: https://phabric.freebsd.org/D473 The uploaded patch does NOT contain the mac_bsdextended(4)/ugidfw(8) integration. With that said, though, one change to one of the mac_bsdextended(4) headers has been left in this patch as it is the main integration point for per-binary ASLR toggles. If we decide not to use mac_bsdextended(4)/ugidfw(8) at all, that change can be moved to the main sys/pax.h header rather than the mac_bsdextended(4) header. I'm going to work later this evening on aggregating and preparing the benchmark results. Hopefully I'll have the results ready for you all tomorrow. Thanks, Shawn --KM+e2hnYAO+MCJ5e Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0C1bAAoJEGqEZY9SRW7uXEwP/3QNaq1S54Rp3oXLVMdgHPsf 4HDqw9UiaETqgkpSt7UJ47dvzqnnziZOZXbAqUi66eGAd24ef+n0sU+D2FbwNnnG J8ITh+1morKaHvpgzC4R+kC3yDsc5H6nfQMZGeqgWOT5J64GdqjaEunocOK1c+gU tzLtRLaasaja9IeOLN+2xNM1oP/T/DiGVJZrD8aYTC4snoEo3NM9l0S5vHw0YbP9 j7Jv1NLHjX7EngDEVZiTrEDTp/wZdowFghNZBtARllzG5/927g5BxB5WAlUv3hx6 n56E4ONW4t3GZHHCxldub1IzbOLiMnLTc80Mv+oP7MlW0XQIvAtRkU6sgcQNFmXm 5kwBXBR9CiA8RAz5yZJagNKmMHKbz4SQdIo5lg/shh0ZKedJTAFLPTufN/t8MNKg EV3HBd447yhjD+ApHSm3fl1+zw1oi6sTUBEYiYv6RT7a9ZBcyV61TBPQSjiSV25t 08m/UBfVCtxzABEBnFb5e98/VQ9Drm55Qk7fA6y8Cj3MjcMku5J8gGAdn10SYMRt 9Ak8YprhVk3PQgkFJnS0Rhbjc4uoV70TnpOkVRDhfzPW0SoNWK1QjIpQHgefAtEF vx9d789/KutXtE9YV24QDzxYnor6mbC96EzNK1oeiFaGeB9nhkVrNFE8QWT7VCWl 5wvDbjRE9e/jO6S7zSob =DAdJ -----END PGP SIGNATURE----- --KM+e2hnYAO+MCJ5e-- From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 23:38:05 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AF58A87B for ; Wed, 23 Jul 2014 23:38:05 +0000 (UTC) Received: from nm23-vm1.bullet.mail.bf1.yahoo.com (nm23-vm1.bullet.mail.bf1.yahoo.com [98.139.213.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D85527DC for ; Wed, 23 Jul 2014 23:38:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1406158683; bh=OrE8zvFiB+ShNHteaNeu+gIxKMqC/B928hgbIhwPEjw=; h=Received:Received:Received:X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Message-Id:References:To:X-Mailer; b=lD0Ul6vV+S/VNDfE1LuDJUpl0doZUyzdrew9JzDE2EBlAJDibZSB5pEy/DVB8Hcl/m9wW8yCBJCMjIPLo+1hgTwRj1scSP+NdzWqwG+7Z0B/RSU8mzEwAP4C1s617mJXJ6vXfR651hslnWYf0cToTzTQruS467FtjAGLZhg3szZ/8z6piD4aj1XoWR3ED+h2VRvOLZZcpGallr/WBy9b2TDMce3ygN6/JC/lQgLKsXxFWdY3v1VqpS8nVPB64ZaIAep8QmG2YrNwDinnzzB/3/ON+1wBOVdgte3qWlyRDgRLaiHseTbK5CCM+NCW/ZzhwkiFsYYHiFJg5bMJZP8+9A== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048; d=yahoo.com; b=Zl7e2q0Wbw+Uee9XvsTUUG6A0fcJmMW3oyTDIxrh9KV1dFfE/YIdeswyen9MQO/NANrs+NJsbGm2C1RgL/8J4/nFuhCm+y9GZCupKt3jwzg2rleMxWcXDL0BE+RDyi09jC+YzleW0u5KmKsBS0y/m3DM1ESdB2fzFcDKKBDN71IXGNvzJvvDGT3u9sNGv6UEu9t2GjsAP9+dYaCMTfVig7RXXFsT4Ko5xI77Ho0KNSF3It0ifGgk22AxtUrE8CisZidHB2U9kNwkeO+YitP/QlqXIhfvwzQ5MTEi7l5yn3FCoUcN0AT7SItoZmugSSPrPQW0QKuWzkH2M2eSANCI9Q==; Received: from [66.196.81.174] by nm23.bullet.mail.bf1.yahoo.com with NNFMP; 23 Jul 2014 23:38:03 -0000 Received: from [68.142.230.64] by tm20.bullet.mail.bf1.yahoo.com with NNFMP; 23 Jul 2014 23:38:02 -0000 Received: from [127.0.0.1] by smtp221.mail.bf1.yahoo.com with NNFMP; 23 Jul 2014 23:38:02 -0000 X-Yahoo-Newman-Id: 639167.85326.bm@smtp221.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: CdyOcVYVM1lynDnkPIcmc5aEQu3Kr4Fm6g.rqEQ7uYcSbPG nUv5m.avfECfF4PJ31xdk5iVFfwDU9FS1t0fBL3jG37sHLyJrzSGZcBB8.7L 6rr_.9DqCGcmn_s2A2d5PR5mpy5KVffAd1mu45ZxNXR7bSmaDWg3SbwHBeAy HpUOB.qex1otsrPwjAVU1yhlg3OfVw5HW4M4MnzgjuPfjsxW1bYfs1zdfLR3 33DqKEgxC0cZ9CdJDq.K2Ls5CZBTcK8XhN8jqQ6UCUS3Per6uo4qdZerNhgN ResIL1dHMHUE3B9cjNvmH5s6ck3uDRcZSBXk1Phshqz59DLgupJwZq7JRUxa Jt9iCvkfL5oakn2TpOF.cRlxFIVgp0nKj7YMnEMUd8b8kej9pZPb_FkTsPxf fiNaVDdMWhj3I.0TOQsV4aLhmN5MaPZ83jh9DahMsYawCKbbo0JjYclWRDXJ gPd3f7Eu4qFr0oUgG5wz3wtRIwWAERhpi6gI8XdG__nG7Be.OCP9cYA86vFX dlVL4aDM_KVJc5FtpvYyZzJHcEKGTAQ8Mg54v2CoTgtKT.SSzUK_zw4vWpaN 4n5DAICg3rZF9Hp9xLtaoFpSahTwg4D6KH5nE.NMAvwf5kMplnJ8RQiXQNQF FKrzPFk9tD2Avk5GvQwI_cL8ki97UVyI- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: Pedro Giffuni In-Reply-To: <20140723004543.GH29618@pwnie.vrt.sourcefire.com> Date: Wed, 23 Jul 2014 18:37:57 -0500 Message-Id: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> To: Shawn Webb X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: PaX Team , Oliver Pinter , Robert Watson , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 23:38:05 -0000 Hi; Il giorno 22/lug/2014, alle ore 19:45, Shawn Webb ha = scritto: >>> ... >>=20 >> Hi Shawn: >>=20 >> Great news that this work is coming to fruition -- ASLR is long = overdue. >>=20 >> Are you having any luck with performance measurements? Unixbench = seems like a=20 >> good starting point, but I wonder if it would be useful to look, in=20= >> particular, at memory-mapping intensive workloads that might be = affected as a=20 >> result of changes in kernel VM data-structure use, or greater = fragmentation of=20 >> the address space. I'm not sure I have a specific application here = in mind --=20 >> in the past I might have pointed out tools such as ElectricFence that = tend to=20 >> increase fragmentation themselves. >=20 > The unixbench tests on that laptop have finished. However, I've been > fighting a pesky migraine these last couple days, so I haven't had the > opportunity to aggregate the results into a nice little spreadsheet. = I'm > hoping to finish it up by the end of the week. >=20 > I'll take a look at ElectricFence this weekend. Additionally, I have a > netbook somewhere. Once I find it and its power cord, I'll install > FreeBSD/x86 and re-run the same tests on that. >=20 Somewhat related to ElectricFence=85 will ASLR have an adverse effect on = debuggers? I googled around and got to this: http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ So I guess we may have to patch gdb (and lldb)? Pedro. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 23:44:59 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82CE9BEF; Wed, 23 Jul 2014 23:44:59 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 086AF287E; Wed, 23 Jul 2014 23:44:58 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so2310300qgf.20 for ; Wed, 23 Jul 2014 16:44:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=uJdUGdvtC2HHJL4WeRfFTS8JKREzxprLIim+BOwtCdk=; b=lMtc5DLCbEQrzPlri1r+FqvpzVnnpEiZakh8P1HPKgpIR/PjdCfxmSmI+gPVEjDX3W J5SjZG3RVxNUVUZ6vH47vq3hBfP2/NLkbtshrjn9luFA7NEeQMVsFHknK9iGf0bEZU00 SPsRFnizPrMxduz8FHrgORZxUo1s+8BBji3975bS3B8xrn36UgGg07G+s8w+IULNiAXy rWrhuW2WARlXTrrHZUDF6prcvYP/CsY9O4gXgXFhy8dAO3oOymMY+yMl7a9qfDs1O00R D9Icc9Dws716V2ylY9+TjBme4tu4gXT+vssMbH54Tsn6MmLwHE6fuRSopxIEH58cDeRG w+uw== X-Received: by 10.224.54.136 with SMTP id q8mr7852349qag.79.1406159098101; Wed, 23 Jul 2014 16:44:58 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id j97sm5133625qgd.37.2014.07.23.16.44.56 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jul 2014 16:44:57 -0700 (PDT) Date: Wed, 23 Jul 2014 19:44:55 -0400 From: Shawn Webb To: Pedro Giffuni Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140723234455.GP29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="o41d8xLWOaLD8vYh" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , Oliver Pinter , Robert Watson , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 23:44:59 -0000 --o41d8xLWOaLD8vYh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 23, 2014 06:37 PM -0500, Pedro Giffuni wrote: > Hi; >=20 > Il giorno 22/lug/2014, alle ore 19:45, Shawn Webb ha = scritto: >=20 > >>> ... > >>=20 > >> Hi Shawn: > >>=20 > >> Great news that this work is coming to fruition -- ASLR is long overdu= e. > >>=20 > >> Are you having any luck with performance measurements? Unixbench seem= s like a=20 > >> good starting point, but I wonder if it would be useful to look, in=20 > >> particular, at memory-mapping intensive workloads that might be affect= ed as a=20 > >> result of changes in kernel VM data-structure use, or greater fragment= ation of=20 > >> the address space. I'm not sure I have a specific application here in= mind --=20 > >> in the past I might have pointed out tools such as ElectricFence that = tend to=20 > >> increase fragmentation themselves. > >=20 > > The unixbench tests on that laptop have finished. However, I've been > > fighting a pesky migraine these last couple days, so I haven't had the > > opportunity to aggregate the results into a nice little spreadsheet. I'm > > hoping to finish it up by the end of the week. > >=20 > > I'll take a look at ElectricFence this weekend. Additionally, I have a > > netbook somewhere. Once I find it and its power cord, I'll install > > FreeBSD/x86 and re-run the same tests on that. > >=20 >=20 > Somewhat related to ElectricFence? will ASLR have an adverse effect on de= buggers? >=20 > I googled around and got to this: >=20 > http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ I've been doing all my ClamAV development on my FreeBSD box with ASLR enabled. Development tools like gdb and valgrind work great, even with corefiles. I have not, however, tried lldb. >=20 > So I guess we may have to patch gdb (and lldb)? =46rom my experience with ClamAV development, I don't think so. gdb uses ptrace to get and set the registers, so ASLR doesn't matter. >=20 > Pedro. >=20 --o41d8xLWOaLD8vYh Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0Ej1AAoJEGqEZY9SRW7uBDEP/ixd5suvrTzdlY+bKBRkfEJ4 SFRQi/UViQSU+6NtbnlvyJnbF6KfgKaM24jOo6O0so8Mwkm7CUtZVcidHITF2X5e wJz98Rpq7iaocrsLMoYhoSI0YYMcyI2IdsFquS0MOacxmEePBTSKBlJN1R60CbpB uZxhtpoSpD1o3rqWsG+JAMsrTDdOagrrtdO6zTf9VHLAG+Tjr4nFNCQJiXwnIs/H KbeqPYKp7Sj9Yu9E5rjkctYDYdkVRcDaFTHylQ8i19Uo1b+ThV+yvDr4LvHUTREc MqlumWz7a3PvWBhEyogUzBIJHXUnIsxyUREC4L2Q8iyH09U51uwKWLN4LumtXweL fG6fRv+0fZHYbMVsCDdrUhKgtcWG6zpkgG0bpD8sAIS7a/n7/LmGbkbhAHEBhHMQ m90WcKeGHiL0XDJJ35iKRip7E4zNN6DTmh1TP52feIbK6agCPLcT+DkEBdFypnFh ZFTHvDV4Fu6DBgIePcdSuPnlwslDPSK8lAGxWyKH77hEYXngvgbqfcHAsoLdtq2x XIa+56Ynq1ksZC/NxrDCG+TJ8kIFTqi6yJgDZlU6pWPlpUv/tIQKaAhSsfodLsYk nsAGkoYMXjrtZL+Bn4jakvaKR1mKLcs/dMTEb82X3Ioq9sCFptBjSJ43pooAlpuE V2IZJGRI2Q6PUibHOBc1 =JlWH -----END PGP SIGNATURE----- --o41d8xLWOaLD8vYh-- From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 23:50:28 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 36744D49 for ; Wed, 23 Jul 2014 23:50:28 +0000 (UTC) Received: from nm3-vm0.bullet.mail.bf1.yahoo.com (nm3-vm0.bullet.mail.bf1.yahoo.com [98.139.212.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D60E028CA for ; Wed, 23 Jul 2014 23:50:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1406159419; bh=Y6YFsgnpNlewKZEL2WsE6jFv6JfzuNw2awmdusDPFhs=; h=Received:Received:Received:X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Message-Id:References:To:X-Mailer; b=QxSmFQRxcnXh3ChMXoBeG0edj+uEtimoCnvVGTZrS0bRTxlKsRmek32fGxQ+ZRwjhLJ5KNu1gnDjQ6J/D0Nlw3XYo0XFZ4B56DstI2wOGypXfsNU5BwGt7gzcGozrfM6T+Jx1vthKoXWQ4ejLMewBXWPDj0PT880GobCmuPYdFhtznRxWL7ZPSXRUkZ7w3snggzFSj/iehzH4s+Hpxz32W+QPzDxIPGjcQQQJ5U1P0ENK3eJuMwFMT7IFlgTCjpD4HPzEhhr8TBkAoSbuPkPV1Xo20mhlAYlPc/aAhMMj3Xj/aOdu5sEIgglpKRIAD9xCiulL72HQpq7TCRwRTZU+Q== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048; d=yahoo.com; b=mXE8NSrCAVtG1Sh9XWPmCA8NPhgcJm8PCgwn3OkyKqjnBdicew+iFdp7ow+CoX570tPJNA591I+MKYWG1sDJDM9xi25+UWa9yK9S9nFut8FyYoVvNl40VzC/GGXIUVMFlY5sCPXGPYhlwDzenzbDI4nqcM7EHJHnkzuRKXg8h+QbyNxkr0joeSD+5Hpt//znkEeJg5u3lQRD5uw+aEug7k5lYjRdtOmzCXqIbOlb8Q8MQmgn+sWVBfjKo2DlK/oWQqKX/eTVpQM9YrVDgQunCQyi3RrwbiiDzYKfnTxj7VpkXRC7y1uuipX/1qYJ+Zmy6SnrjIMaJzqLeNvbkHIB9w==; Received: from [66.196.81.172] by nm3.bullet.mail.bf1.yahoo.com with NNFMP; 23 Jul 2014 23:50:19 -0000 Received: from [98.139.211.206] by tm18.bullet.mail.bf1.yahoo.com with NNFMP; 23 Jul 2014 23:50:19 -0000 Received: from [127.0.0.1] by smtp215.mail.bf1.yahoo.com with NNFMP; 23 Jul 2014 23:50:19 -0000 X-Yahoo-Newman-Id: 865338.57364.bm@smtp215.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 9Xs4wCcVM1kvA1cqCVR_918nUe1xw1soGCBf_PTraCguNAP uO5r8AsqQZ9NG5lIB5ITgK4qQTvNxppPSWzOJ._beWHYvUOjRSWV6S7HETOY pV8z.adift1Bpbp8hetBkp4Ly3amNE8dW.SubSNPWEfvInZDT5skQW3GaW.t bvhXh2HB5MqDSFaNDDtX8qdUlA.wlKPkpG0EnxusWQB5MRWE7oqFnt1OZm4M xSIxaSmrpM4GAcHlKiMLgqsC8JJC24F75UaeaSZUn.fxAEA80AXqxefm9Iah _yVE95NNiFS6cBwINvjSg8VsxlnuA5vdHhBopAAeEwEVzjfHV_sLR4XOuzQV xHXvMbF2.tRpQZmmKTYiv2w8XVoIeKXVKY9LFoeYpsSKC8V5Q6W82ygrOauL FzkJ3hurvTomJCpaMKTbbzEDwJmpmbTw1rKrlul_VCVdlS3adTABnXVIlI3X rVs2aPYGblGb0Mc6lLjtInp6btwmKWNuws5gFBz1K47eNCd06e9ULS882Z1a myoYH_5NiwDVBvS36gGc1JZomklD6y1ISl6KNL5lKkDCejac7qQE0E7RNKwF 5pJdlxFexHiUPV8of.fIz6z4Zrwnt94.nXjE5OajBFPszZf.eYxT4IWowc0i WjyxEVTRLHNiNjKDIwHtgWggyIDs5rsC06AAn6BiZ.XsVHQ3OZYXckXtrJJq DEjnI_MX03JL0rRAPF5LhQYSkQl5R5Vpqv1Twk.AKvh9MWDnLF0B.35YT3MM qPRTalSXxMslwgZvpmg-- X-Yahoo-SMTP: xcjD0guswBAZaPPIbxpWwLcp9Unf Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: Pedro Giffuni In-Reply-To: <20140723234455.GP29618@pwnie.vrt.sourcefire.com> Date: Wed, 23 Jul 2014 18:50:15 -0500 Message-Id: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140723234455.GP29618@pwnie.vrt.sourcefire.com> To: Shawn Webb X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: PaX Team , Oliver Pinter , Robert Watson , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 23:50:28 -0000 Il giorno 23/lug/2014, alle ore 18:44, Shawn Webb ha = scritto: > On Jul 23, 2014 06:37 PM -0500, Pedro Giffuni wrote: >> Hi; >>=20 >> Il giorno 22/lug/2014, alle ore 19:45, Shawn Webb = ha scritto: >>=20 >>>>> ... >>>>=20 >>>> Hi Shawn: >>>>=20 >>>> Great news that this work is coming to fruition -- ASLR is long = overdue. >>>>=20 >>>> Are you having any luck with performance measurements? Unixbench = seems like a=20 >>>> good starting point, but I wonder if it would be useful to look, in=20= >>>> particular, at memory-mapping intensive workloads that might be = affected as a=20 >>>> result of changes in kernel VM data-structure use, or greater = fragmentation of >>>> the address space. I'm not sure I have a specific application here = in mind --=20 >>>> in the past I might have pointed out tools such as ElectricFence = that tend to=20 >>>> increase fragmentation themselves. >>>=20 >>> The unixbench tests on that laptop have finished. However, I've been >>> fighting a pesky migraine these last couple days, so I haven't had = the >>> opportunity to aggregate the results into a nice little spreadsheet. = I'm >>> hoping to finish it up by the end of the week. >>>=20 >>> I'll take a look at ElectricFence this weekend. Additionally, I have = a >>> netbook somewhere. Once I find it and its power cord, I'll install >>> FreeBSD/x86 and re-run the same tests on that. >>>=20 >>=20 >> Somewhat related to ElectricFence? will ASLR have an adverse effect = on debuggers? >>=20 >> I googled around and got to this: >>=20 >> http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ >=20 > I've been doing all my ClamAV development on my FreeBSD box with ASLR > enabled. Development tools like gdb and valgrind work great, even with > corefiles. I have not, however, tried lldb. >=20 OK, but it=92s worth to take a look if we need to support something to = turn it off. Apparently gdb disables ASLR on MacOSX too: http://reverse.put.as/2011/08/11/how-gdb-disables-aslr-in-mac-os-x-lion/ Pedro. From owner-freebsd-arch@FreeBSD.ORG Wed Jul 23 23:53:03 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2454EE14; Wed, 23 Jul 2014 23:53:03 +0000 (UTC) Received: from mail-qg0-x229.google.com (mail-qg0-x229.google.com [IPv6:2607:f8b0:400d:c04::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9C7BD2950; Wed, 23 Jul 2014 23:53:02 +0000 (UTC) Received: by mail-qg0-f41.google.com with SMTP id q107so2378767qgd.28 for ; Wed, 23 Jul 2014 16:53:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=T4HffjYVT0XBLWJlB0QtAlkTwxEQ2GCdi+i7rAsRClQ=; b=W+BOTE1wkwJTfCZvqLtjftIv+EH/xvUOZ8pYY+x180gHAx76Lwh0aKtHxe/iOJopWR O7StjXl4iWPv58a5ELefoWcAIYOfPRZD+fUecERyspTvqr2gbZBqV+o4AQQN/uScg3Vl dKS7EJlLy5uknkMomFW57PYv/K0A+xgoXzgy0+Iz4MQ7WLbKxXUKBF9e+XCgV28mliK5 D0LxwvuPk/skF+DOIUeQvhnje24HCXFr59mfi4rOZ+6soNkVfpD/135evvI/CYl5rvnb wO1niA04X8BM2YrgYXVoymsAf8Hr9MHvSro6PORmJ2dCNqxL6RSe+l/ExIC5C58KwwFP 6Uig== X-Received: by 10.224.63.194 with SMTP id c2mr7914983qai.21.1406159581589; Wed, 23 Jul 2014 16:53:01 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id b6sm6332703qak.42.2014.07.23.16.53.00 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Jul 2014 16:53:00 -0700 (PDT) Date: Wed, 23 Jul 2014 19:52:58 -0400 From: Shawn Webb To: Pedro Giffuni Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140723235258.GQ29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140723234455.GP29618@pwnie.vrt.sourcefire.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jigfid2yHjNFZUTO" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , Oliver Pinter , Robert Watson , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2014 23:53:03 -0000 --jigfid2yHjNFZUTO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 23, 2014 06:50 PM -0500, Pedro Giffuni wrote: >=20 > Il giorno 23/lug/2014, alle ore 18:44, Shawn Webb ha = scritto: >=20 > > On Jul 23, 2014 06:37 PM -0500, Pedro Giffuni wrote: > >> Hi; > >>=20 > >> Il giorno 22/lug/2014, alle ore 19:45, Shawn Webb = ha scritto: > >>=20 > >>>>> ... > >>>>=20 > >>>> Hi Shawn: > >>>>=20 > >>>> Great news that this work is coming to fruition -- ASLR is long over= due. > >>>>=20 > >>>> Are you having any luck with performance measurements? Unixbench se= ems like a=20 > >>>> good starting point, but I wonder if it would be useful to look, in= =20 > >>>> particular, at memory-mapping intensive workloads that might be affe= cted as a=20 > >>>> result of changes in kernel VM data-structure use, or greater fragme= ntation of > >>>> the address space. I'm not sure I have a specific application here = in mind --=20 > >>>> in the past I might have pointed out tools such as ElectricFence tha= t tend to=20 > >>>> increase fragmentation themselves. > >>>=20 > >>> The unixbench tests on that laptop have finished. However, I've been > >>> fighting a pesky migraine these last couple days, so I haven't had the > >>> opportunity to aggregate the results into a nice little spreadsheet. = I'm > >>> hoping to finish it up by the end of the week. > >>>=20 > >>> I'll take a look at ElectricFence this weekend. Additionally, I have a > >>> netbook somewhere. Once I find it and its power cord, I'll install > >>> FreeBSD/x86 and re-run the same tests on that. > >>>=20 > >>=20 > >> Somewhat related to ElectricFence? will ASLR have an adverse effect on= debuggers? > >>=20 > >> I googled around and got to this: > >>=20 > >> http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ > >=20 > > I've been doing all my ClamAV development on my FreeBSD box with ASLR > > enabled. Development tools like gdb and valgrind work great, even with > > corefiles. I have not, however, tried lldb. > >=20 >=20 > OK, but it?s worth to take a look if we need to support something to turn= it off. > Apparently gdb disables ASLR on MacOSX too: >=20 > http://reverse.put.as/2011/08/11/how-gdb-disables-aslr-in-mac-os-x-lion/ Completely agreed. It's easily possible my use case is different than others'. The more eyes on this project and the more testers we have, the better. Thanks, Shawn --jigfid2yHjNFZUTO Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0ErZAAoJEGqEZY9SRW7uKI4P/1rIOKJkikWx23zxDB5E2Sdt 5Cfl9kyd5P+ON5vZjdQADYnPlltgpppR1BCFyYlZNN59aWybfYQcg1sFbIAE2Vdv 243oyVs8d2clUpm6KGEaOTG2qLvfJ6aTQkWduo+vAE0KF7srQFb6px+Ms3b04GTA YW9fBj8uVn2Qrh0TqssLOkTj5DNFLTyVz8GinyJj7jNR0XPDN657mC167madlqS8 +0r0lvAtER+NRXPJCgykB9WMBo6JLL7MnQkq9kgsVLOGvLHOqB9iXASc4ha2gJGb UIsMJopuCe1cilJPiSw2ba3eCm4d61bgPnD9ZjBC/Mae8xinrS6lQbV4vNRhW/kT ZtHbTqrp+Mw99k1dnbaMh1Pf7SgLReucU8Ql2dOUkUC0FPtc6QVyf8iPCiL7c+hO PNtoL2kSQzH+Vu3n4ovUFXhA11bUInb5bGcWn4Wehdn2ncJ/MuO5xcJD2ehfV98v RaiVH4qkZgOmhWrVhvSx8v2IrvppJhPzu8tA7o3vYvIQwo+qffITaZ9a7TQy0Kt1 KRwXjl4y45XqeToLXMuigNG/GfY33wiKo7+poKpS/Z1c6N4z/75WhJXnskkrdzPx WwF7ZteELrz90VQX8xeqVSiHbGPdLulFn3yPmUFBTpvVb1Alk6X+6yMCuKks1NQR 51tsO+GH0U+rA8Iu2YGU =yHNJ -----END PGP SIGNATURE----- --jigfid2yHjNFZUTO-- From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 00:46:01 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00878C69 for ; Thu, 24 Jul 2014 00:46:00 +0000 (UTC) Received: from mail-pa0-f52.google.com (mail-pa0-f52.google.com [209.85.220.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C140D2D8A for ; Thu, 24 Jul 2014 00:46:00 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id bj1so2727403pad.25 for ; Wed, 23 Jul 2014 17:45:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=cKMJ3dIowaAUo5lF9A2yKKo+4d5kpTn6KqSBsCHXy2E=; b=hl2o8f6ZmpjPrb9WpjyUBGc9xzg370IiLT5DNOVWE9RqvvALlEDerkeRekCkTVScit uRNhCrYQX87DvAV4AU0M2t/T7jZZSAwZODXbw5WN5z9rvpYCkJGbV/UluEPf4HIzORF5 K0P3okKqgKY6YrnKklqjsQbAvpKESb0MKQooz2UIzMIJor6aPUMttM/X+4IKWq+bzJxb cDLNpiR5Q8z2FuLxsfnKOd9gcB8LEpAxl10oJnDZ4WKyaDTxG5VMGbQFbYJc+WP7pec7 fThANXIR8pHav0dzwMP8Hu44Ssj/SFTDWapvpvXj9B1egmswpHw7lMBdIJYdc9TPrgEA PGjQ== X-Gm-Message-State: ALoCoQlLAl2/J/aSK3w5v/1h3SirRFRd21Zw4++hGYJpmaaCKSYS4zGEfG0Pz3oA5jOzmQcMMHql X-Received: by 10.66.236.35 with SMTP id ur3mr6252957pac.35.1406162753554; Wed, 23 Jul 2014 17:45:53 -0700 (PDT) Received: from [192.168.1.103] (c-24-6-220-224.hsd1.ca.comcast.net. [24.6.220.224]) by mx.google.com with ESMTPSA id vu7sm14208334pab.34.2014.07.23.17.45.52 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 23 Jul 2014 17:45:52 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: Tim Kientzle In-Reply-To: Date: Wed, 23 Jul 2014 17:45:51 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <796EDB88-3768-48AA-B909-8A7FFBED0C1E@kientzle.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> To: Pedro Giffuni X-Mailer: Apple Mail (2.1878.6) Cc: Shawn Webb , Oliver Pinter , Robert Watson , freebsd-arch@freebsd.org, PaX Team , Bryan Drewery X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 00:46:01 -0000 On Jul 23, 2014, at 4:37 PM, Pedro Giffuni wrote: > Hi; >=20 > Il giorno 22/lug/2014, alle ore 19:45, Shawn Webb = ha scritto: >=20 >>>> ... >>>=20 >>> Hi Shawn: >>>=20 >>> Great news that this work is coming to fruition -- ASLR is long = overdue. >>>=20 >>> Are you having any luck with performance measurements? Unixbench = seems like a=20 >>> good starting point, but I wonder if it would be useful to look, in=20= >>> particular, at memory-mapping intensive workloads that might be = affected as a=20 >>> result of changes in kernel VM data-structure use, or greater = fragmentation of=20 >>> the address space. I'm not sure I have a specific application here = in mind --=20 >>> in the past I might have pointed out tools such as ElectricFence = that tend to=20 >>> increase fragmentation themselves. >>=20 >> The unixbench tests on that laptop have finished. However, I've been >> fighting a pesky migraine these last couple days, so I haven't had = the >> opportunity to aggregate the results into a nice little spreadsheet. = I'm >> hoping to finish it up by the end of the week. >>=20 >> I'll take a look at ElectricFence this weekend. Additionally, I have = a >> netbook somewhere. Once I find it and its power cord, I'll install >> FreeBSD/x86 and re-run the same tests on that. >>=20 >=20 > Somewhat related to ElectricFence=85 will ASLR have an adverse effect = on debuggers? >=20 > I googled around and got to this: >=20 > http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ >=20 > So I guess we may have to patch gdb (and lldb)? I suspect the issue here is that debugging often requires multiple runs of a program with repeatable behavior between runs. Consider: * I run the program under GDB, it crashes at a certain PC address * I restart the program, set a breakpoint at that PC address I want to be confident that the PC address where I=92m setting the breakpoint will have the same meaning between runs. Tim From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 07:03:24 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD9281AE for ; Thu, 24 Jul 2014 07:03:24 +0000 (UTC) Received: from mail-qg0-x230.google.com (mail-qg0-x230.google.com [IPv6:2607:f8b0:400d:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 830D02C99 for ; Thu, 24 Jul 2014 07:03:24 +0000 (UTC) Received: by mail-qg0-f48.google.com with SMTP id i50so2743422qgf.7 for ; Thu, 24 Jul 2014 00:03:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=BK1NtdXRbbvxbVyHBhCyKbFZH8zMGbUGNaj3sMGQfHw=; b=TwlwIOkvbL17JWrKijLQT0Y2oRjpQLH7xk+9b4dlt8gusgtst0Jqi2jullPlfimFwP Wj+oogbpe48OwsrPZsKKpJ7OrHlD2lnkiN78TciTkCIWGkUV2AriJm7qV5OF8dmJzojL m8+FNUwzsnz7Q9F4/+mPLfUMyAa6LOi4+VD5AicRy/tFpmeL6SRx06BiaGHt5AiHnA3E NrK5aYSIppCElEB/L2LbtLo1cXb2Jd5K2P5+FgAAXl2DssuZPamG5tZuoSYum0azuHhd fw0QfHlNk0L6VvQ8gNZtv6rNTt0fliBXhVCD89RTq0WjT0dboYh0Sy8pklI7/a645rml T0/Q== MIME-Version: 1.0 X-Received: by 10.140.102.142 with SMTP id w14mr10787007qge.101.1406185403460; Thu, 24 Jul 2014 00:03:23 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.1.6 with HTTP; Thu, 24 Jul 2014 00:03:23 -0700 (PDT) Date: Thu, 24 Jul 2014 00:03:23 -0700 X-Google-Sender-Auth: 3ZWo4bP1HzRG2RaDdZXiMWxoaVo Message-ID: Subject: [rfc] where to put cpuid_t ? From: Adrian Chadd To: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 07:03:24 -0000 Hi! I've been chipping away at a cpuid_t in a local branch for a while and I think I have the very basics working well enough. However, the most annoying thing that's crept up is the most bikeshed topic of them all - where should it live. I'd like to avoid having to include sys/pcpu.h or sys/_cpuset.h just for the id type - it seems a bit overkill. So - suggestions? Otherwise I'm going to leave it in sys/pcpu.h and just polluate appropriately. I'd like to try and get cpuid_t and a handful of KBI-changing things into -HEAD before 11 is branched so we at least have a hope of trying to support > 128 CPUs out of the box in the immediate future and > 253 CPUs in the later future. -a From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 07:51:53 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F395B63; Thu, 24 Jul 2014 07:51:53 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 6DF412069; Thu, 24 Jul 2014 07:51:53 +0000 (UTC) Received: from [10.218.87.102] (unknown [85.255.233.71]) by cyrus.watson.org (Postfix) with ESMTPSA id 6A4FD46B38; Thu, 24 Jul 2014 03:51:51 -0400 (EDT) References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140723234455.GP29618@pwnie.vrt.sourcefire.com> Mime-Version: 1.0 (1.0) In-Reply-To: Message-Id: <860889F3-EA75-4C08-A1A4-904CE7A94899@FreeBSD.org> X-Mailer: iPad Mail (11D257) From: "Robert N. M. Watson" Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Date: Thu, 24 Jul 2014 08:51:47 +0100 To: Pedro Giffuni Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: PaX Team , "freebsd-arch@freebsd.org" , Oliver Pinter , Bryan Drewery , Shawn Webb X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 07:51:53 -0000 >>>>> Great news that this work is coming to fruition -- ASLR is long overdu= e. >>>>>=20 >>>>> Are you having any luck with performance measurements? Unixbench seem= s like a=20 >>>>> good starting point, but I wonder if it would be useful to look, in=20= >>>>> particular, at memory-mapping intensive workloads that might be affect= ed as a=20 >>>>> result of changes in kernel VM data-structure use, or greater fragment= ation of >>>>> the address space. I'm not sure I have a specific application here in= mind --=20 >>>>> in the past I might have pointed out tools such as ElectricFence that t= end to=20 >>>>> increase fragmentation themselves. >>>>=20 >>>> The unixbench tests on that laptop have finished. However, I've been >>>> fighting a pesky migraine these last couple days, so I haven't had the >>>> opportunity to aggregate the results into a nice little spreadsheet. I'= m >>>> hoping to finish it up by the end of the week. >>>>=20 >>>> I'll take a look at ElectricFence this weekend. Additionally, I have a >>>> netbook somewhere. Once I find it and its power cord, I'll install >>>> FreeBSD/x86 and re-run the same tests on that. >>>=20 >>> Somewhat related to ElectricFence? will ASLR have an adverse effect on d= ebuggers? >>>=20 >>> I googled around and got to this: >>>=20 >>> http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ >>=20 >> I've been doing all my ClamAV development on my FreeBSD box with ASLR >> enabled. Development tools like gdb and valgrind work great, even with >> corefiles. I have not, however, tried lldb. >=20 > OK, but it=E2=80=99s worth to take a look if we need to support something t= o turn it off. > Apparently gdb disables ASLR on MacOSX too: >=20 > http://reverse.put.as/2011/08/11/how-gdb-disables-aslr-in-mac-os-x-lion/ >=20 > Pedro. >=20 From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 09:43:32 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 28334BAE; Thu, 24 Jul 2014 09:43:32 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id D70C72B77; Thu, 24 Jul 2014 09:43:31 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 708F546B42; Thu, 24 Jul 2014 05:43:31 -0400 (EDT) Date: Thu, 24 Jul 2014 10:43:31 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Tim Kientzle Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch In-Reply-To: <796EDB88-3768-48AA-B909-8A7FFBED0C1E@kientzle.com> Message-ID: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <796EDB88-3768-48AA-B909-8A7FFBED0C1E@kientzle.com> User-Agent: Alpine 2.11 (BSF 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: Shawn Webb , Oliver Pinter , Pedro Giffuni , freebsd-arch@freebsd.org, PaX Team , Bryan Drewery X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 09:43:32 -0000 On Wed, 23 Jul 2014, Tim Kientzle wrote: >>> I'll take a look at ElectricFence this weekend. Additionally, I have a >>> netbook somewhere. Once I find it and its power cord, I'll install >>> FreeBSD/x86 and re-run the same tests on that. >> >> Somewhat related to ElectricFence… will ASLR have an adverse effect on >> debuggers? >> >> I googled around and got to this: >> >> http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ >> >> So I guess we may have to patch gdb (and lldb)? > > I suspect the issue here is that debugging often requires multiple runs of a > program with repeatable behavior between runs. > > Consider: > > * I run the program under GDB, it crashes at a certain PC address > > * I restart the program, set a breakpoint at that PC address > > I want to be confident that the PC address where I’m setting the breakpoint > will have the same meaning between runs. Non-determism in debugging is a big issue with diversification/randomisation-based mitigation techniques. There are a number of aspects to the problem, but the most clear implication is that it should be possible to create deterministic and reproducible debugging environments in the local development context. This means, I think, being able to create a hierarchy of processes in which the randomisation features are by policy turned off. The contexts in which that property is set are interesting -- do you want a "no randomisation subshell" in which every program you run has ASLR turned off? Or do you just want gdb to turn it off? What if, this time around, you want gdb to have it turned on? And how do you deal with setuid/setgid/transitioning binaries -- we don't want a regular user to say "turn off ASLR for this process subtree" and have it prevent ASLR from protecting a setuid binary from the user. I think the natural conclusion is that you need multiple means to disable ASLR that operate at different granularities, and that have different control mechanisms. Off-hand, I see a few: (1) A global enable/disable flag that sets the default policy. (2) A new inherited process property, or perhaps credential property, enabling ASLR. This can be changed to 'disabled' using a system call -- perhaps prctl() or similar. If we hit a transitioning binary (e.g., setuid) then in the same way that we manipulate other properties, we'd reset to the global default. It would be easy to imagine this being CR_ASLR on the cr_flags field of the credential. This could be set in various ways by userspace applications -- by gdb, as a login-shell property, perhaps via 'su' or something simular? (3) As suggested by Kostik, an ELF note on binaries indicating that the binary is not ASLR-compatible, which would override (I guess) the global policy flag and process/credential flag. We could then set this with NO_ASLR=true in Makefiles, during package creation, etc. (4) It sounds like a jail-scope policy would also be useful -- I guess this might actually be the same as (1) in the sense that (1) could be represented in terms of a jail-scope policy. I'm not opposed to MAC policy modules being able to manipulate ASLR behaviour, but I think I'd prefer that the core policy controls (e.g., the above) be MAC-independent. Part of the reason is that you may want ASLR on very low-end systems where the additional cost of MAC interposition is more measurable. How does this interact with features like the Linuxulator? Do we also want ABI emulations to be able to disable ASLR as ABIs might not support it [well]? Robert From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 09:49:55 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2A4FDE2; Thu, 24 Jul 2014 09:49:54 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id CBFBD2BC8; Thu, 24 Jul 2014 09:49:54 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 87D8146B0D; Thu, 24 Jul 2014 05:49:53 -0400 (EDT) Date: Thu, 24 Jul 2014 10:49:53 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Adrian Chadd Subject: Re: [rfc] where to put cpuid_t ? In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (BSF 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 09:49:55 -0000 On Thu, 24 Jul 2014, Adrian Chadd wrote: > I've been chipping away at a cpuid_t in a local branch for a while and I > think I have the very basics working well enough. > > However, the most annoying thing that's crept up is the most bikeshed topic > of them all - where should it live. > > I'd like to avoid having to include sys/pcpu.h or sys/_cpuset.h just for the > id type - it seems a bit overkill. > > So - suggestions? Otherwise I'm going to leave it in sys/pcpu.h and just > polluate appropriately. > > I'd like to try and get cpuid_t and a handful of KBI-changing things into > -HEAD before 11 is branched so we at least have a hope of trying to support > > 128 CPUs out of the box in the immediate future and > 253 CPUs in the > later future. Sounds like bikeshed fodder. Usually you end up with widely used core types in _types.h or _foo.h, with some notion of publiciness being defined in types.h, param.h, or foo.h. I'd leave it where it is until you discover you need the header everywhere or nested all over the place, in which case I'd then shift it to a centralised header. At least for SRI and Cambridge, the ability to name a few hundred cores/threads is concern that will arise later this year. I'd like to see a CPU ID type that can represent >128 threads almost immediately -- ideally no less than 512 in the beginning of next year as hardware starts to come together. (In case anyone is wondering where we find such hardware: we use FPGA-based soft-core processors in our research, and are about to move from a single-FPGA platform which typically hosts a smallish numbers of cores and threads to a multi-FPGA platform which could lead to quite fast growth in core count, and also quite NUMAish properties. We can time-division multiplex core implementations on FPGAs leading to many more virtual than physical cores). Robert From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 11:33:57 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 12CF55B7 for ; Thu, 24 Jul 2014 11:33:57 +0000 (UTC) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id CB32825CD for ; Thu, 24 Jul 2014 11:33:56 +0000 (UTC) Received: from nine.des.no (smtp.des.no [194.63.250.102]) by smtp-int.des.no (Postfix) with ESMTP id B19A67A23; Thu, 24 Jul 2014 11:33:54 +0000 (UTC) Received: by nine.des.no (Postfix, from userid 1001) id 0C950954; Thu, 24 Jul 2014 13:33:37 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Arthur Mesh Subject: Re: pam_lastlog References: <20140723174211.GQ57013@juniper.net> Date: Thu, 24 Jul 2014 13:33:37 +0200 In-Reply-To: <20140723174211.GQ57013@juniper.net> (Arthur Mesh's message of "Wed, 23 Jul 2014 10:42:11 -0700") Message-ID: <86iomnt2i6.fsf@nine.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 11:33:57 -0000 Arthur Mesh writes: > Here is a proposed change where that adds a knob to disable this lookup: Why not just remove it altogether? DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 13:52:05 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8E47ABE9; Thu, 24 Jul 2014 13:52:05 +0000 (UTC) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 111D52248; Thu, 24 Jul 2014 13:52:04 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id q107so3287855qgd.12 for ; Thu, 24 Jul 2014 06:52:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=+SWXkQhjhHOAnFj/8i2Ybw8I2B9FpiFRf4zv9yzC3Ac=; b=UnEO++cDQT765oYCL+oqS6EsG5zqTr2zYDczRsEb/4KzM7zud3a2CLiAExCp02rfJo 0LUL9MDmc0BMM+oiauo3U6NDzX9s1e+hN7eCUmg7DjFRFNEjMAyrxI8pB1PVP2A1+sRl GVmESSbk2GULOarkaa5d+JDG2+Ghmofu5MK9sz371c73GVCJZhBG7h9ZE483hlJF0qud Mk5Ok7fAm0khNhYLtcgKamS8OJOBtSNWGsOalO7YMqhbkDgjF9uU4GJpMiaiAaC+FV6+ eIu+d0p1GnyOhIh9Frw3ZXD+OYUIbC1rcTddgMA3mUFb4DLGfQsmOHGvunGoiqwBh2H4 K5bw== X-Received: by 10.224.134.201 with SMTP id k9mr15475750qat.59.1406209923994; Thu, 24 Jul 2014 06:52:03 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id b6sm8923789qak.42.2014.07.24.06.52.02 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jul 2014 06:52:02 -0700 (PDT) Date: Thu, 24 Jul 2014 09:52:00 -0400 From: Shawn Webb To: Robert Watson Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140724135200.GR29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <796EDB88-3768-48AA-B909-8A7FFBED0C1E@kientzle.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="CPn8Wy5ME997YUMW" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Tim Kientzle , Bryan Drewery , Pedro Giffuni , freebsd-arch@freebsd.org, PaX Team , Oliver Pinter X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 13:52:05 -0000 --CPn8Wy5ME997YUMW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 24, 2014 10:43 AM +0100, Robert Watson wrote: > On Wed, 23 Jul 2014, Tim Kientzle wrote: >=20 > >>> I'll take a look at ElectricFence this weekend. Additionally, I have = a=20 > >>> netbook somewhere. Once I find it and its power cord, I'll install=20 > >>> FreeBSD/x86 and re-run the same tests on that. > >> > >> Somewhat related to ElectricFence? will ASLR have an adverse effect on= =20 > >> debuggers? > >> > >> I googled around and got to this: > >> > >> http://www.outflux.net/blog/archives/2010/07/03/gdb-turns-off-aslr/ > >> > >> So I guess we may have to patch gdb (and lldb)? > > > > I suspect the issue here is that debugging often requires multiple runs= of a=20 > > program with repeatable behavior between runs. > > > > Consider: > > > > * I run the program under GDB, it crashes at a certain PC address > > > > * I restart the program, set a breakpoint at that PC address > > > > I want to be confident that the PC address where I?m setting the breakp= oint=20 > > will have the same meaning between runs. >=20 > Non-determism in debugging is a big issue with=20 > diversification/randomisation-based mitigation techniques. There are a n= umber=20 > of aspects to the problem, but the most clear implication is that it shou= ld be=20 > possible to create deterministic and reproducible debugging environments = in=20 > the local development context. This means, I think, being able to create= a=20 > hierarchy of processes in which the randomisation features are by policy= =20 > turned off. The contexts in which that property is set are interesting -= - do=20 > you want a "no randomisation subshell" in which every program you run has= ASLR=20 > turned off? Or do you just want gdb to turn it off? What if, this time= =20 > around, you want gdb to have it turned on? And how do you deal with=20 > setuid/setgid/transitioning binaries -- we don't want a regular user to s= ay=20 > "turn off ASLR for this process subtree" and have it prevent ASLR from=20 > protecting a setuid binary from the user. Great points. I agree completely. This is why I've added two facilities for toggling ASLR. I modified mac_bsdextended(4)/ugidfw(8) to provide toggling ASLR on a per-binary basis. I also made the sysctl tunables settable per-jail. This allows one to place the executable in question in a jail where ASLR is turned off just for that jail. I came across this on Tuesday of this week when doing ClamAV development. Since I do all my development in a jail, I came across a bug in ClamAV and I needed deterministic behavior, so I turned off ASLR just for that jail and was able to find the cause of the bug and fix it. >=20 > I think the natural conclusion is that you need multiple means to disable= ASLR=20 > that operate at different granularities, and that have different control= =20 > mechanisms. Off-hand, I see a few: >=20 > (1) A global enable/disable flag that sets the default policy. We have that via the sysctl security.pax.aslr.status, which is settable at boot-time via /boot/loader.conf or at runtime if the kernel has been compiled with PAX_SYSCTLS option. >=20 > (2) A new inherited process property, or perhaps credential property, ena= bling > ASLR. This can be changed to 'disabled' using a system call -- perh= aps > prctl() or similar. If we hit a transitioning binary (e.g., setuid)= then > in the same way that we manipulate other properties, we'd reset to t= he > global default. It would be easy to imagine this being CR_ASLR on t= he > cr_flags field of the credential. This could be set in various ways= by > userspace applications -- by gdb, as a login-shell property, perhaps= via > 'su' or something simular? This is interesting. I didn't think of using the prctl facility. I'll look into that. >=20 > (3) As suggested by Kostik, an ELF note on binaries indicating that the b= inary > is not ASLR-compatible, which would override (I guess) the global po= licy > flag and process/credential flag. We could then set this with > NO_ASLR=3Dtrue in Makefiles, during package creation, etc. I really don't like ELF flags. It's really heavy-handed. It can trip file-based IDSs (the hash of the file changes when you change an ELF flag). It requires that the user, sysadmin, or team of sysadmins keep a database of binaries for which the ELF flag needs to be changed when the binary gets updated or replaced (think: freebsd-update, pkgng/ports). This is why I went with mac_bsdextended(4)/ugidfw(8). In the end, the user is still maintaining a database (via ugidfw(8) rules). But the user doesn't need to go back and modify anything if that binary were to ever be modified. With that said, we are planning on also adding FS extended attribute support as well. Though I'm unsure if extended attributes work over network-mounted filesystems like NFS. >=20 > (4) It sounds like a jail-scope policy would also be useful -- I guess th= is > might actually be the same as (1) in the sense that (1) could be > represented in terms of a jail-scope policy. We have already implemented this as per-jail sysctl tunables. >=20 > I'm not opposed to MAC policy modules being able to manipulate ASLR behav= iour,=20 > but I think I'd prefer that the core policy controls (e.g., the above) be= =20 > MAC-independent. Part of the reason is that you may want ASLR on very lo= w-end=20 > systems where the additional cost of MAC interposition is more measurable. How much overhead does mac_bsdextended(4) have? I'm assuming a small ruleset pertaining to ASLR. >=20 > How does this interact with features like the Linuxulator? Do we also wa= nt=20 > ABI emulations to be able to disable ASLR as ABIs might not support it [w= ell]? Oliver has done testing on the linuxulator and has found it to be stable. He can share more info about this. Thanks, Shawn --CPn8Wy5ME997YUMW Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0Q9/AAoJEGqEZY9SRW7uwNQP/ieOC7enFVBkD0V5LgXn3Txi xF2MZQM3JMPIpwv6+J7fhnKezt2YSC2lrpuqWBNwD3XOTXXV0aAA9JwsB5bjTLYo /4H+2db28XhYA48oNZ3LVODDHiF4gEc3xOh026j/wqvtvsSYavbS9c9ete5CvWqn Y2cFhUSMeVwNprqanRZ07yCdPCYPbgdU3loNwgEYVJYhXyozapq3U7+Y4fQJrYDd 8O8v0LfODkJeTVYdE2qJFAnYd3UEm6gEKHLnP1dJfxzBQ0FUXmJqly+zOsC8rljE ibxaZQjceBtKSlSKsoftymN1hN/FmDEH+ldSL3CKRnJ+NMgKQpiL71r8E3cON4R3 XUpivGUbAQ7kS57IXTXjjJI/ikyawDUacBctSzfFNaFth6RfB7c0bcCj7mlr22pA ig7uOr8qsjVuXZovBphGkcu2BLNKs1C7MO7nmZjDQ47VnrkzE7TERzN0tMxOdT4Z ZcX1y/Z5NN/FjxYs4xXAKVQz+frl866GcTTVo44ex24LH21uwREgRYpU3mxFSHJi a8SHmHvnxwRP+Md2qXpeA7QL1h274Lytdnshytv89nrvZ2AJQOd7I2GE0LTcjQEc Fp9L9IXKv6dXxMn/M4iw5ZK+y9x2MNMSQP3oAQehK7NJUZ8ZxcDeHzfAs4cwLvH0 4vB84c7u71GUWPFqBs88 =+6yX -----END PGP SIGNATURE----- --CPn8Wy5ME997YUMW-- From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 15:23:57 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AB73DE3B for ; Thu, 24 Jul 2014 15:23:57 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0144.outbound.protection.outlook.com [207.46.163.144]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F9AA2B53 for ; Thu, 24 Jul 2014 15:23:56 +0000 (UTC) Received: from BL2PR05CA0027.namprd05.prod.outlook.com (10.255.226.27) by BY2PR05MB728.namprd05.prod.outlook.com (10.141.223.25) with Microsoft SMTP Server (TLS) id 15.0.995.14; Thu, 24 Jul 2014 15:23:47 +0000 Received: from BY2FFO11FD014.protection.gbl (2a01:111:f400:7c0c::145) by BL2PR05CA0027.outlook.office365.com (2a01:111:e400:c04::27) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 24 Jul 2014 15:23:46 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BY2FFO11FD014.mail.protection.outlook.com (10.1.14.76) with Microsoft SMTP Server (TLS) id 15.0.980.11 via Frontend Transport; Thu, 24 Jul 2014 15:23:46 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 24 Jul 2014 08:23:16 -0700 Received: from juniper.net (eta.jnpr.net [172.21.19.189]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6OFNCn96679; Thu, 24 Jul 2014 08:23:14 -0700 (PDT) (envelope-from amesh@juniper.net) Date: Thu, 24 Jul 2014 08:23:12 -0700 From: Arthur Mesh To: Dag-Erling Smorgrav Subject: Re: pam_lastlog Message-ID: <20140724152312.GT57013@juniper.net> References: <20140723174211.GQ57013@juniper.net> <86iomnt2i6.fsf@nine.des.no> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="iytlPraCFSCqrfnM" Content-Disposition: inline In-Reply-To: <86iomnt2i6.fsf@nine.des.no> User-Agent: Mutt/1.5.23 (2014-03-12) X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(24454002)(51704005)(189002)(199002)(84326002)(81542001)(85852003)(50986999)(76176999)(86362001)(64706001)(102836001)(6806004)(81342001)(74502001)(79102001)(83322001)(20776003)(21056001)(44976005)(69596002)(83506001)(84676001)(54356999)(19580405001)(87936001)(92726001)(99396002)(77982001)(16796002)(31966008)(83072002)(68736004)(36756003)(97736001)(4396001)(107046002)(19580395003)(85306003)(110136001)(46102001)(71186001)(105596002)(76482001)(512954002)(80022001)(74662001)(92566001)(81156004)(95666004)(106466001)(33656002); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB728; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 028256169F Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=amesh@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 15:23:57 -0000 --iytlPraCFSCqrfnM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 24, 2014 at 01:33:37PM +0200, Dag-Erling Smorgrav wrote: > Arthur Mesh writes: > > Here is a proposed change where that adds a knob to disable this lookup: >=20 > Why not just remove it altogether? Works for me; hope nobody depends on this little quirk to be there. --=20 Arthur Mesh Juniper Networks +1 408 936-4968 --iytlPraCFSCqrfnM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAEBCAAGBQJT0STgAAoJEO/ZUtudxDntm/QL/18Gdkz6KZwMXaWUIVn7dpKp JQgBIcAJFcGgVCb+8VQ70DNgXozLfFZLIZei1h9+0mKGSMd0frZVHk9+peTjNz8d VmY6igEPKMFjPxnGDA7zPLsQJI4HaIhAAordv9a/4wxQQsOW4nMjkv4Qucpvf7OY tyQp6WiI33KK4sQFQK9VnkSYSgVjuDFWJwvo/biM47aSNYyJVqr5EhwLQYb1GWEH GbhRNTOruKlBRIQ5dqc23ye4RKCA3JIQs07PibpH6ImBRiO3Lr5oiqZ1wzgGFXd3 /qlUukl2bJNhuNmdREAXW/BC6DkRFLFhVM25hJgzptA/RU6eHcno9WO5CaW9YJ/6 BSEO9mggybmEeTBfzLeLoOpOsPsLn/+pU94rF93jSYBDwnyxR9li2hiB/fX6B41G 8PXgfYHSKn6l+mtRuj/qz1QEBurZazrWa8IttXZAdXnh/JcT6eF9kXgtv12vV1FV Ls6JQHoBOsNotdYfQx+QQ5Jujsmk7pQ7W6v3bT7hYg== =c9Ng -----END PGP SIGNATURE----- --iytlPraCFSCqrfnM-- From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 15:27:38 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DAAFFF48 for ; Thu, 24 Jul 2014 15:27:38 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1blp0187.outbound.protection.outlook.com [207.46.163.187]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 832432B80 for ; Thu, 24 Jul 2014 15:27:37 +0000 (UTC) Received: from BL2PR05CA0030.namprd05.prod.outlook.com (10.255.226.30) by DM2PR05MB736.namprd05.prod.outlook.com (10.141.178.25) with Microsoft SMTP Server (TLS) id 15.0.990.7; Thu, 24 Jul 2014 15:27:35 +0000 Received: from BN1AFFO11FD030.protection.gbl (2a01:111:f400:7c10::195) by BL2PR05CA0030.outlook.office365.com (2a01:111:e400:c04::30) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Thu, 24 Jul 2014 15:27:34 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1AFFO11FD030.mail.protection.outlook.com (10.58.52.168) with Microsoft SMTP Server (TLS) id 15.0.980.11 via Frontend Transport; Thu, 24 Jul 2014 15:27:34 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 24 Jul 2014 08:27:26 -0700 Received: from juniper.net (eta.jnpr.net [172.21.19.189]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6OFRKn00205; Thu, 24 Jul 2014 08:27:20 -0700 (PDT) (envelope-from amesh@juniper.net) Date: Thu, 24 Jul 2014 08:27:20 -0700 From: Arthur Mesh To: Dag-Erling Smorgrav Subject: Re: pam_lastlog Message-ID: <20140724152720.GU57013@juniper.net> References: <20140723174211.GQ57013@juniper.net> <86iomnt2i6.fsf@nine.des.no> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="SvF6CGw9fzJC4Rcx" Content-Disposition: inline In-Reply-To: <86iomnt2i6.fsf@nine.des.no> User-Agent: Mutt/1.5.23 (2014-03-12) X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(189002)(199002)(24454002)(51704005)(36756003)(81542001)(105596002)(19580405001)(81156004)(71186001)(77982001)(84676001)(81342001)(74502001)(16796002)(102836001)(50986999)(31966008)(85306003)(74662001)(20776003)(80022001)(64706001)(86362001)(83322001)(19580395003)(68736004)(76482001)(83506001)(99396002)(84326002)(44976005)(46102001)(4396001)(79102001)(69596002)(6806004)(76176999)(21056001)(54356999)(83072002)(92726001)(92566001)(85852003)(512954002)(110136001)(95666004)(33656002)(87936001)(97736001)(551544002)(107046002)(106466001); DIR:OUT; SFP:; SCL:1; SRVR:DM2PR05MB736; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 028256169F Received-SPF: SoftFail (: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=amesh@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 15:27:39 -0000 --SvF6CGw9fzJC4Rcx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 24, 2014 at 01:33:37PM +0200, Dag-Erling Smorgrav wrote: > Arthur Mesh writes: > > Here is a proposed change where that adds a knob to disable this lookup: >=20 > Why not just remove it altogether? Here is a diff to remove it altogether: Index: lib/libpam/modules/pam_lastlog/pam_lastlog.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- lib/libpam/modules/pam_lastlog/pam_lastlog.c (revision 269064) +++ lib/libpam/modules/pam_lastlog/pam_lastlog.c (working copy) @@ -49,7 +49,6 @@ #include =20 #include -#include #include #include #include @@ -68,7 +67,6 @@ pam_sm_open_session(pam_handle_t *pamh, int flags, int argc __unused, const char *argv[] __unused) { - struct passwd *pwd; struct utmpx *utx, utl; time_t t; const char *user; @@ -79,7 +77,7 @@ pam_err =3D pam_get_user(pamh, &user, NULL); if (pam_err !=3D PAM_SUCCESS) return (pam_err); - if (user =3D=3D NULL || (pwd =3D getpwnam(user)) =3D=3D NULL) + if (user =3D=3D NULL) return (PAM_SERVICE_ERR); PAM_LOG("Got user: %s", user); =20 --=20 Arthur Mesh Juniper Networks +1 408 936-4968 --SvF6CGw9fzJC4Rcx Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAEBCAAGBQJT0SXYAAoJEO/ZUtudxDntCq0L/ia4j8A2Lv4vIKotyCho2YJK 0ouJ3UiOhHalp/dAYPFfwRFBsSJ+w/Vk0C4eoCG39G9l9fyKBAxIipZCfgh7GL96 /eSMCBWiI1JopePfKeAeX+cEjHNGQwBeGkvzHau22Ev+wP2Sb1ByDVVs1ONhhUBU 2vUp9X0Q90sFbWXziEJ7GdYm5LHrwENaycXktDYIJRjDPP4sCFlI+FtvJl/IuJ0a tQB1dvrDLi5N7UwG7k8DPiZCk7ApT9LXBOJO4cR49BIkowUFZ1IHfZqaIEi4spci fmMLuy+2LF/40j7yhjST7cqN5i0BhAd5Vj9K8Eani9Z12Uoowd0ugkD7RSYHOH/S d/PV3/A1MXexju1WUDBVgqR5NVGkgsoc38bqQgoxxaXxniirctTQbA0w3KnNNvVR CyiHv0xD9h8Jk8P5c4OaswOobW+4FZdXeMKdMrIyfy0crawXbqgwqE2P7VQO2hbv 2gcU6qbcckjvpkuvPnXvy4dPzMsFa+/6P8otb13ofw== =V+kI -----END PGP SIGNATURE----- --SvF6CGw9fzJC4Rcx-- From owner-freebsd-arch@FreeBSD.ORG Thu Jul 24 17:57:09 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81D4D9E1; Thu, 24 Jul 2014 17:57:09 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06EF02A78; Thu, 24 Jul 2014 17:57:08 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so3645048qgf.20 for ; Thu, 24 Jul 2014 10:57:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=vEkm3/b35/OUY+ULic3Ga1IfJTbbwXtZg7hX7+eFV/o=; b=br4sMxa9COp0qIUqm2Z+nNa4ef92eaRmTfdu2vjk+SHSjDZwEdn6TmglQxVeCLduTj aubmdnuH1Cfv6qzMDBiqxkMTt6i2lear8+5jXhX6x0V8uvUdTVHHbs2Z9poflM+gbjhn GaA980yuHQZ1uLpRbVdWZwWPd56QWnjUuwU073OkEL+1ABjMEY+ssAQCGlyUZg7XUaie 9RebxuKMUKNk8W2+YrYtoCnZE1EoY8APJ92eGVMXg/m7DKGLK9a3yuOpy9ZeNBEB+Sdh uQvcqtglruF/kdPyZ/yaIR7TRgQL3Zp1HGMP/6CP3jGK33fzaum5ZI4ocxscoEfUoDpi ZUwA== X-Received: by 10.140.81.51 with SMTP id e48mr17127437qgd.31.1406224627909; Thu, 24 Jul 2014 10:57:07 -0700 (PDT) Received: from pwnie.vrt.sourcefire.com (moist.vrt.sourcefire.com. [198.148.79.134]) by mx.google.com with ESMTPSA id j3sm7890182qgj.48.2014.07.24.10.57.06 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Jul 2014 10:57:07 -0700 (PDT) Date: Thu, 24 Jul 2014 13:57:04 -0400 From: Shawn Webb To: Robert Watson Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140724175704.GT29618@pwnie.vrt.sourcefire.com> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iiZKCn1f/U0ES2iY" Content-Disposition: inline In-Reply-To: <20140723004543.GH29618@pwnie.vrt.sourcefire.com> X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.23 (2014-03-12) Cc: PaX Team , Pedro Giffuni , Oliver Pinter , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 17:57:09 -0000 --iiZKCn1f/U0ES2iY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Jul 22, 2014 08:45 PM -0400, Shawn Webb wrote: > On Jul 23, 2014 12:28 AM +0100, Robert Watson wrote: > > On Sun, 20 Jul 2014, Shawn Webb wrote: > >=20 > > >> - It is yet undetermined what the performance effect will be, and it= is not=20 > > >> clear (but seems likely from past measurements) if there will be a= =20 > > >> performance hit even when ASLR is off. -Apparently there are applica= tions=20 > > >> that will segfault (?). > > > > > > So I have an old Dell Latitude E6500 that I bought at Defcon a year or > > > so ago that I'm doing testing on. Even though it's quite an underpowe= red > > > laptop, I'm running ZFS on it for BE support (in case one of our chan= ges > > > kills it). I'll run unixbench on it a few times to benchmark the ASLR > > > patch. I'll test these three scenarios: > > > 1) ASLR compiled in and enabled; > > > 2) ASLR compiled in and disabled; > > > 3) ASLR compiled out (GENERIC kernel). > > > > > > In each of these three scenarios, I'll have the kernel debugging feat= ures=20 > > > (WITNESS, INVARIANTS, etc.) turned off to better simulate a productio= n=20 > > > system and to remove just one more variable in the tests. > > > > > > I'll run unixbench ten times under each scenario and I'll compute ave= rages. > > > > > > Since this is an older laptop (and it's running ZFS), these tests wil= l take=20 > > > a couple days. I'll have an answer for you soon. > >=20 > > Hi Shawn: > >=20 > > Great news that this work is coming to fruition -- ASLR is long overdue. > >=20 > > Are you having any luck with performance measurements? Unixbench seems= like a=20 > > good starting point, but I wonder if it would be useful to look, in=20 > > particular, at memory-mapping intensive workloads that might be affecte= d as a=20 > > result of changes in kernel VM data-structure use, or greater fragmenta= tion of=20 > > the address space. I'm not sure I have a specific application here in = mind --=20 > > in the past I might have pointed out tools such as ElectricFence that t= end to=20 > > increase fragmentation themselves. >=20 > The unixbench tests on that laptop have finished. However, I've been > fighting a pesky migraine these last couple days, so I haven't had the > opportunity to aggregate the results into a nice little spreadsheet. I'm > hoping to finish it up by the end of the week. >=20 > I'll take a look at ElectricFence this weekend. Additionally, I have a > netbook somewhere. Once I find it and its power cord, I'll install > FreeBSD/x86 and re-run the same tests on that. >=20 > >=20 > > Also, could you say a little more about the effects that the change mig= ht have=20 > > on transparent superpage use -- other than suitable alignment of large= =20 > > mappings, it's not clear to me what effect it might have. >=20 > Since we're just modifying the hint passed to the underlying VM system, > superpage support works as it should with ASLR enabled. The VM system > will modify the hint in order to be able to use superpages. In those > cases, we might lose a little bit of entropy. However, due to superpages > (on amd64, at least) requring 2MB alignment, you'd lose some entropy no > matter how ASLR was implemented--at the end of the day, you need that > alignment for superpages to work. >=20 > >=20 > > I wonder if some equipment in the FreeBSD Netperf cluster might be used= to=20 > > help with performance characterisation -- in particular, very recent hi= gh-end=20 > > server hardware, and also, lower-end embedded-style systems that have m= arkedly=20 > > different virtual-memory implementations in hardware and software. Oft= en=20 > > those two classes of systems see markedly different performance-change= =20 > > characteristics as a result of greater cache-centrism and instruction-l= evel=20 > > parallelism in the higher-end designs that can mask increases in instru= ction=20 > > count. >=20 > Any additional testing would be very much welcome. Our ASLR > implementation misbehaves on ARM, so testing on ARM-based embedded > devices is pretty limited. My next goal is to figure out why it bugs out > on ARM. Essentially, when a child process exits/dies and the parent > process gets sent SIGCHLD, the parent process' pc register somehow gets > set to 0xc0000000 and segfaults. Here's a screenshot of the process: > https://twitter.com/lattera/status/490529645997998080 >=20 > FreeBSD 11-CURRENT hasn't been stable at all on sparc64, even without > the ASLR patches. I have an SunFire 280R box that I've attempted to test > ASLR our on, but I couldn't get a stable enough installation of vanilla > FreeBSD to work long enough to recompile world/kernel. And generating an > installation ISO from my amd64 box doesn't work as the VTOC8 bootloader > isn't recognized by the BIOS (not sure if that's what it's called in > sparc land). >=20 > >=20 > > I think someone has already commented that Peter Holm's help might be= =20 > > enlisted; you have have seen his 'stress2' suite, which could help with= =20 > > stability testing. >=20 > I'll take a look at that, too. Thanks a lot for your suggestions and > feedback. The unixbench results are in. The overall scores are below. ASLR Disabled: 456.33 ASLR Enabled: 357.05 No ASLR: 474.03 I've uploaded the raw results to http://0xfeedface.org/~shawn/aslr/2014-07-24_benchmark.tar.gz Take these results with a grain of salt, given that some of unixbench's test are filesystem-related and I'm running ZFS on an old laptop with little RAM. It does show that there is a performance impact when ASLR is enabled. Within the last day, I have made some changes to clean up the code backing our ASLR implementation that would enhance the performance when ASLR is enabled. I'll re-run the tests when ASLR is enabled starting tonight and I'll have a new set of results tomorrow. I'll also give ElectricFence a try. Those results will come later. Thanks, Shawn --iiZKCn1f/U0ES2iY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0UjwAAoJEGqEZY9SRW7upXQP+gMuiSjxgS1WzW+c9rTRPG4J ocLpIDYd0HabUwx/q5VC0hPJFF/svikEn4VGa+/Avm47WMA9s00MHcW7B5cvz35R h6hXZHXom3bEd4FDUAqx0keMbQTPdHe4OUNcbF1EvX1hhslcEfPk5U8xIaMd23H1 tXeM91ADGO9gl3HY1KizKRSFtX9u8GLpUmVBEhkLTVpgYHxnuDTstZC7Yekrkmsd MXePHEMUeXJXIrwt/TEgFepIIY7eSTsL40dMtRZHXEtsP1J+wb/aA5rawfwd5kU3 1mrRrs5Gp7YN4YvFG/rtGrIbBVH0Yg9SeXW54GdImS6wJsn8dWcgFYkQ0tVQr7Ee EndVVYZMeZDfTZseloxHMeELa+/93VqVyPKKvsUTxncC2yelKARGSqT9DbrUKiWS KxqwpG3vL9m4tNmDy9ikOEJvwSnMzKz+4jP0bKAQYPcf3oSgQHsJTenSKWJ46r7U FCuS6WfP3GSt2bCT2WrxeQptAtR6L19aMdO+s+G4V+vkJ351G2SapUhewgzJPsKD x6eJCpwVymOWPxfWU2a46Kd7MJybIvJCgctet8BUnN3v9pFO/frQTjmi6PydOhwX RYkNrVgiZhrF09/BmxS0BCQcBEfME2sw6xbxZeDXyx6plmbFOPVuVoUBJ/3uLkXO tmz8E55e/GDBgPKdkPd4 =FgZY -----END PGP SIGNATURE----- --iiZKCn1f/U0ES2iY-- From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 04:49:29 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E6F5C10; Fri, 25 Jul 2014 04:49:29 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0141.outbound.protection.outlook.com [207.46.163.141]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 884FF200B; Fri, 25 Jul 2014 04:49:27 +0000 (UTC) Received: from BY2PR05CA008.namprd05.prod.outlook.com (10.242.32.38) by CO2PR05MB730.namprd05.prod.outlook.com (10.141.228.15) with Microsoft SMTP Server (TLS) id 15.0.990.7; Fri, 25 Jul 2014 04:49:25 +0000 Received: from BN1BFFO11FD003.protection.gbl (2a01:111:f400:7c10::1:172) by BY2PR05CA008.outlook.office365.com (2a01:111:e400:2c2a::38) with Microsoft SMTP Server (TLS) id 15.0.990.7 via Frontend Transport; Fri, 25 Jul 2014 04:49:24 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BN1BFFO11FD003.mail.protection.outlook.com (10.58.144.66) with Microsoft SMTP Server (TLS) id 15.0.980.11 via Frontend Transport; Fri, 25 Jul 2014 04:49:24 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Thu, 24 Jul 2014 21:49:23 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6P4nLn65071; Thu, 24 Jul 2014 21:49:22 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 9F0D3580A2; Thu, 24 Jul 2014 21:49:21 -0700 (PDT) To: Subject: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Thu, 24 Jul 2014 21:49:21 -0700 From: Simon Gerraty Message-ID: <20140725044921.9F0D3580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(377424004)(51704005)(189002)(164054003)(80022001)(87936001)(81542001)(68736004)(74662001)(106466001)(46102001)(64706001)(4396001)(62966002)(21056001)(79102001)(104166001)(6806004)(50466002)(97736001)(2351001)(86362001)(48376002)(93546004)(101356003)(84676001)(92566001)(50226001)(33656002)(88136002)(77156001)(105596002)(90896003)(69596002)(95666004)(57986006)(229853001)(83322001)(99396002)(87286001)(110136001)(47776003)(102836001)(85852003)(76482001)(92726001)(50986999)(81342001)(83072002)(20776003)(81156004)(107046002)(89996001)(74502001)(44976005)(93916002)(70486001)(77982001)(19580395003)(31966008)(85306003)(76506005)(102176002)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:CO2PR05MB730; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 02830F0362 Received-SPF: SoftFail (: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: sjg@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 04:49:29 -0000 Hi, At a vendor summit a few years ago I asked about whether anyone but us (Juniper) would be interested in the ablity to have standard BSD apps output XML. I was actually surprised by the amount of interest expressed. I've occasionally nagged our UI team ever since for a clean and simple API that we could contribute to address this. We now have a what I think is a viable candidate and we'd like to take the next steps towards contributing it and converting at least a few apps. Not only does it handle TXT and XML output but JSON and HTML as well, and very rich HTML at that. With some slick javascript - you can do amazing things with the level of detail you can get out of this sort of thing. The API is of necessity a bit more complex than just printf(3). Considering the level of functionality available though it is a good tradeoff. The main open issue (assuming this functionality is still desired) is support of wide charachters. We figure the worst case solution is a sed(1) script to generate the wide version of the API from the normal one, but perhaps simply always using UTF8 would be a better solution? Thanks --sjg The following from Phil provides some idea of the functionality available and the API. The one shown here uses the default output handle (stdout), but there are variants that allow multiple output handles. >Here's some sample output (from my libxo-ified "w"): > >(This uses an environment variable to trigger the mode and options: > T X J and H are the modes (text, xml, json, html); > P means pretty print (indent, newlines) > I means print help (datatype, description), if provided (there aren't for "w") > x means print xpath to the data) > >% foreach i ( T XP JP HP HPIx ) > echo === $i === > env LIBXO_OPTIONS=$i ./xtest -n | head -10 > end >=== T === > 6:47PM up 18 days, 2:01, 9 user%s, load averages: 0.00, 0.00, 0.00 >USER TTY FROM LOGIN@ IDLE WHAT >phil pts/0 76.182.32.73 5:09PM 33 /bin/sh >phil pts/1 76.182.32.73 05Jul14 2 /usr/bin/perl /u/phil/bin/plum ( >phil pts/2 76.182.32.73 05Jul14 1 /bin/tcsh >phil pts/3 76.182.32.73 05Jul14 2days ssh dent >phil pts/4 76.182.32.73 Tue02PM 2days ssh svl-junos-d026.juniper.net >phil pts/5 76.182.32.73 Wed01AM 2days telnet man-o-war 2006 >phil pts/6 76.182.32.73 Fri10PM 2days ssh 198.85.229.65 >phil pts/7 76.182.32.73 Fri10PM 2days ssh zap >=== XP === > > 6:47PM > 18 days > 2:01 > 9 > 0.00 > 0.00 > 0.00 > > >=== JP === >"uptime-information": { > "time-of-day": " 6:47PM", > "uptime": "18 days", > "uptime": 2:01, > "users": 9, > "load-average-1": 0.00, > "load-average-5": 0.00, > "load-average-15": 0.00, > "user-table": { > "user-entry": [ >=== HP === >
>
6:47PM
>
>
up
>
>
18 days
>
,
>
>
2:01
>
,
>=== HPIx === >
>
6:47PM
>
>
up
>
>
18 days
>
,
>
>
2:01
>
,
> >Thanks, > Phil > >----- > >FWIW: here's the diff for "w". I don't have "wchar_t" support >yet, so I just undid it for now. > > >diff -rbu /usr/src/usr.bin/w/pr_time.c ./pr_time.c >--- /usr/src/usr.bin/w/pr_time.c 2010-12-21 12:09:25.000000000 -0500 >+++ ./pr_time.c 2014-07-21 17:12:19.000000000 -0400 >@@ -55,10 +55,10 @@ > int > pr_attime(time_t *started, time_t *now) > { >- static wchar_t buf[256]; >+ static char buf[256]; > struct tm tp, tm; > time_t diff; >- wchar_t *fmt; >+ char *fmt; > int len, width, offset = 0; > > tp = *localtime(started); >@@ -67,7 +67,7 @@ > > /* If more than a week, use day-month-year. */ > if (diff > 86400 * 7) >- fmt = L"%d%b%y"; >+ fmt = "%d%b%y"; > > /* If not today, use day-hour-am/pm. */ > else if (tm.tm_mday != tp.tm_mday || >@@ -75,23 +75,23 @@ > tm.tm_year != tp.tm_year) { > /* The line below does not take DST into consideration */ > /* else if (*now / 86400 != *started / 86400) { */ >- fmt = use_ampm ? L"%a%I%p" : L"%a%H"; >+ fmt = use_ampm ? "%a%I%p" : "%a%H"; > } > > /* Default is hh:mm{am,pm}. */ > else { >- fmt = use_ampm ? L"%l:%M%p" : L"%k:%M"; >+ fmt = use_ampm ? "%l:%M%p" : "%k:%M"; > } > >- (void)wcsftime(buf, sizeof(buf), fmt, &tp); >- len = wcslen(buf); >- width = wcswidth(buf, len); >+ (void)strftime(buf, sizeof(buf), fmt, &tp); >+ len = strlen(buf); >+ width = len; > if (len == width) >- (void)wprintf(L"%-7.7ls", buf); >+ xo_emit("{:login-time/%-7.7s}", buf); > else if (width < 7) >- (void)wprintf(L"%ls%.*s", buf, 7 - width, " "); >+ xo_emit("{:login-time/%s}%.*s", buf, 7 - width, " "); > else { >- (void)wprintf(L"%ls", buf); >+ xo_emit("{:login-time/%s}", buf); > offset = width - 7; > } > return (offset); >@@ -108,7 +108,7 @@ > /* If idle more than 36 hours, print as a number of days. */ > if (idle >= 36 * 3600) { > int days = idle / 86400; >- (void)printf(" %dday%s ", days, days > 1 ? "s" : " " ); >+ xo_emit(" {:idle/%dday%s} ", days, days > 1 ? "s" : " " ); > if (days >= 100) > return (2); > if (days >= 10) >@@ -117,15 +117,15 @@ > > /* If idle more than an hour, print as HH:MM. */ > else if (idle >= 3600) >- (void)printf(" %2d:%02d ", >+ xo_emit(" {:idle/%2d:%02d/} ", > (int)(idle / 3600), (int)((idle % 3600) / 60)); > > else if (idle / 60 == 0) >- (void)printf(" - "); >+ xo_emit(" - "); > > /* Else print the minutes idle. */ > else >- (void)printf(" %2d ", (int)(idle / 60)); >+ xo_emit(" {:idle/%2d} ", (int)(idle / 60)); > > return (0); /* not idle longer than 9 days */ > } >diff -rbu /usr/src/usr.bin/w/w.c ./w.c >--- /usr/src/usr.bin/w/w.c 2010-12-21 12:09:25.000000000 -0500 >+++ ./w.c 2014-07-21 18:13:50.000000000 -0400 >@@ -86,6 +86,7 @@ > #include > #include > #include >+#include > > #include "extern.h" > >@@ -260,9 +261,12 @@ > } > (void)fclose(ut); > >+ xo_open_container("uptime-information"); >+ > if (header || wcmd == 0) { > pr_header(&now, nusers); > if (wcmd == 0) { >+ xo_close_container("uptime-information"); > (void)kvm_close(kd); > exit(0); > } >@@ -274,11 +278,11 @@ > #define HEADER_WHAT "WHAT\n" > #define WUSED (UT_NAMESIZE + UT_LINESIZE + W_DISPHOSTSIZE + \ > sizeof(HEADER_LOGIN_IDLE) + 3) /* header width incl. spaces */ >- (void)printf("%-*.*s %-*.*s %-*.*s %s", >+ xo_emit("{T:/%-*.*s} {T:/%-*.*s} " >+ "{T:/%-*.*s} {T:LOGIN@} {T:IDLE} {T:WHAT}\n", > UT_NAMESIZE, UT_NAMESIZE, HEADER_USER, > UT_LINESIZE, UT_LINESIZE, HEADER_TTY, >- W_DISPHOSTSIZE, W_DISPHOSTSIZE, HEADER_FROM, >- HEADER_LOGIN_IDLE HEADER_WHAT); >+ W_DISPHOSTSIZE, W_DISPHOSTSIZE, HEADER_FROM); > } > > if ((kp = kvm_getprocs(kd, KERN_PROC_ALL, 0, &nentries)) == NULL) >@@ -347,6 +351,9 @@ > } > } > >+ xo_open_container("user-table"); >+ xo_open_list("user-entry"); >+ > for (ep = ehead; ep != NULL; ep = ep->next) { > char host_buf[UT_HOSTSIZE + 1]; > struct sockaddr_storage ss; >@@ -356,6 +363,8 @@ > time_t t; > int isaddr; > >+ xo_open_instance("user-entry"); >+ > host_buf[UT_HOSTSIZE] = '\0'; > strncpy(host_buf, ep->utmp.ut_host, UT_HOSTSIZE); > p = *host_buf ? host_buf : "-"; >@@ -388,6 +397,9 @@ > p = buf; > } > if (dflag) { >+ xo_open_container("process-table"); >+ xo_open_list("process-entry"); >+ > for (dkp = ep->dkp; dkp != NULL; dkp = debugproc(dkp)) { > const char *ptr; > >@@ -395,23 +407,37 @@ > dkp->ki_comm, MAXCOMLEN); > if (ptr == NULL) > ptr = "-"; >- (void)printf("\t\t%-9d %s\n", >+ xo_open_instance("process-entry"); >+ xo_emit("\t\t{:process-id/%-9d/%d} " >+ "{:command/%s}\n", > dkp->ki_pid, ptr); >+ xo_close_instance("process-entry"); > } >+ xo_close_list("process-entry"); >+ xo_close_container("process-table"); > } >- (void)printf("%-*.*s %-*.*s %-*.*s ", >+ xo_emit("{:user/%-*.*s/%@**@s} {:tty/%-*.*s/%@**@s} " >+ "{:from/%-*.*s/%@**@s} ", > UT_NAMESIZE, UT_NAMESIZE, ep->utmp.ut_name, > UT_LINESIZE, UT_LINESIZE, > strncmp(ep->utmp.ut_line, "tty", 3) && > strncmp(ep->utmp.ut_line, "cua", 3) ? > ep->utmp.ut_line : ep->utmp.ut_line + 3, > W_DISPHOSTSIZE, W_DISPHOSTSIZE, *p ? p : "-"); >+ > t = _time_to_time32(ep->utmp.ut_time); > longattime = pr_attime(&t, &now); > longidle = pr_idle(ep->idle); >- (void)printf("%.*s\n", argwidth - longidle - longattime, >- ep->args); >+ xo_emit("{:command/%.*s/%@*@s}\n", >+ argwidth - longidle - longattime, ep->args); >+ >+ xo_close_instance("user-entry"); > } >+ >+ xo_close_list("user-entry"); >+ xo_close_container("user-table"); >+ xo_close_container("uptime-information"); >+ > (void)kvm_close(kd); > exit(0); > } >@@ -430,7 +456,7 @@ > */ > if (strftime(buf, sizeof(buf), > use_ampm ? "%l:%M%p" : "%k:%M", localtime(nowp)) != 0) >- (void)printf("%s ", buf); >+ xo_emit("{:time-of-day/%s} ", buf); > /* > * Print how long system has been up. > */ >@@ -444,35 +470,45 @@ > uptime %= 3600; > mins = uptime / 60; > secs = uptime % 60; >- (void)printf(" up"); >+ xo_emit(" up"); >+ xo_attr("seconds", "%lu", (unsigned long) tp.tv_sec); > if (days > 0) >- (void)printf(" %d day%s,", days, days > 1 ? "s" : ""); >+ xo_emit(" {:uptime/%d day%s},", >+ days, days > 1 ? "s" : ""); > if (hrs > 0 && mins > 0) >- (void)printf(" %2d:%02d,", hrs, mins); >+ xo_emit(" {:uptime/%2d:%02d},", hrs, mins); > else if (hrs > 0) >- (void)printf(" %d hr%s,", hrs, hrs > 1 ? "s" : ""); >+ xo_emit(" {:uptime/%d hr%s},", >+ hrs, hrs > 1 ? "s" : ""); > else if (mins > 0) >- (void)printf(" %d min%s,", mins, mins > 1 ? "s" : ""); >+ xo_emit(" {:uptime/%d min%s},", >+ mins, mins > 1 ? "s" : ""); > else >- (void)printf(" %d sec%s,", secs, secs > 1 ? "s" : ""); >+ xo_emit(" {:uptime/%d sec%s},", >+ secs, secs > 1 ? "s" : ""); > } > > /* Print number of users logged in to system */ >- (void)printf(" %d user%s", nusers, nusers == 1 ? "" : "s"); >+ xo_emit(" {:users/%d} user%s", nusers, nusers == 1 ? "" : "s"); > > /* > * Print 1, 5, and 15 minute load averages. > */ > if (getloadavg(avenrun, sizeof(avenrun) / sizeof(avenrun[0])) == -1) >- (void)printf(", no load average information available\n"); >+ xo_emit(", no load average information available\n"); > else { >- (void)printf(", load averages:"); >+ static const char *format[] = { >+ " {:load-average-1/%.2f}", >+ " {:load-average-5/%.2f}", >+ " {:load-average-15/%.2f}", >+ }; >+ xo_emit(", load averages:"); > for (i = 0; i < (int)(sizeof(avenrun) / sizeof(avenrun[0])); i++) { > if (use_comma && i > 0) >- (void)printf(","); >- (void)printf(" %.2f", avenrun[i]); >+ xo_emit(","); >+ xo_emit(format[i], avenrun[i]); > } >- (void)printf("\n"); >+ xo_emit("\n"); > } > } > >@@ -493,10 +529,9 @@ > usage(int wcmd) > { > if (wcmd) >- (void)fprintf(stderr, >- "usage: w [-dhin] [-M core] [-N system] [user ...]\n"); >+ xo_error("usage: w [-dhin] [-M core] [-N system] [user ...]\n"); > else >- (void)fprintf(stderr, "usage: uptime\n"); >+ xo_error("usage: uptime\n"); > exit(1); > } > > From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 06:54:12 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 958C6A5E; Fri, 25 Jul 2014 06:54:12 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4EF9C2B36; Fri, 25 Jul 2014 06:54:12 +0000 (UTC) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 830DC6A6005; Fri, 25 Jul 2014 08:54:10 +0200 (CEST) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.7/8.14.7) with ESMTP id s6P6sAwU049006; Fri, 25 Jul 2014 08:54:10 +0200 (CEST) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.7/8.14.7/Submit) id s6P6s8CF048014; Fri, 25 Jul 2014 08:54:08 +0200 (CEST) (envelope-from lars) Date: Fri, 25 Jul 2014 08:54:08 +0200 From: Lars Engels To: Simon Gerraty Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML Message-ID: <20140725065408.GB10844@e-new.0x20.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="jho1yZJdad60DJr+" Content-Disposition: inline In-Reply-To: <20140725044921.9F0D3580A2@chaos.jnpr.net> X-Editor: VIM - Vi IMproved 7.4 X-Operation-System: FreeBSD 8.4-RELEASE-p4 User-Agent: Mutt/1.5.23 (2014-03-12) Cc: sjg@freebsd.org, arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 06:54:12 -0000 --jho1yZJdad60DJr+ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 24, 2014 at 09:49:21PM -0700, Simon Gerraty wrote: > Hi, >=20 > At a vendor summit a few years ago I asked about whether anyone but us > (Juniper) would be interested in the ablity to have standard BSD apps > output XML. >=20 > I was actually surprised by the amount of interest expressed. > I've occasionally nagged our UI team ever since for a clean and simple > API that we could contribute to address this. >=20 > We now have a what I think is a viable candidate > and we'd like to take the next steps towards contributing it and > converting at least a few apps. >=20 > Not only does it handle TXT and XML output but JSON and HTML as well, > and very rich HTML at that. > With some slick javascript - you can do amazing things with the level of > detail you can get out of this sort of thing. >=20 > The API is of necessity a bit more complex than just printf(3). > Considering the level of functionality available though it is a good > tradeoff. >=20 > The main open issue (assuming this functionality is still desired) is > support of wide charachters. >=20 > We figure the worst case solution is a sed(1) script to generate the wide > version of the API from the normal one, but perhaps simply always using > UTF8 would be a better solution? >=20 > Thanks > --sjg FYI: There's also a Summer of Code project to handle the same: https://wiki.freebsd.org/SummerOfCode2014/MachineReadableFromUserlandUtils But I think that one mainly handles JSON output. Lars --jho1yZJdad60DJr+ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQF8BAEBCgBmBQJT0f8QXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RjQwMDE3RTRERjUzMTI1N0FGRTUxNDlF NTRDQjM3RDNBMDg5RDZEAAoJEOVMs306CJ1tMfQH/1wGzxwO9YENOq7UDGkDxQ5T yFKk5M66FIc+5H1XfXxUvKm7KSNTwToAaQvrXDzNJQzUhlBOkYQIXLLIm6cqnS21 T6YIzY+iAwlzmSgzzJjZmvZYs+8po42VJDyXbCyj/vcmE2hO67m18mjZumROwhTm Ss/IDuecISS39KcWizxP4anhC+wGxUnBSKU6+slLtqInIxNrUhn6bL4rRkgVhazA CrVFFYCAO7MZK0Isx68BuNvcvInpQJUwN+JlGG5yhCMrYTicCVlQpPj10tDCCd9r v8pfjj+8FvruiY1OpcPUOqJ0N2i/gIGv4NE4tnnD7NuczLyBglXyDF+MhVoswyI= =QktG -----END PGP SIGNATURE----- --jho1yZJdad60DJr+-- From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 07:17:47 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 60D3C43D; Fri, 25 Jul 2014 07:17:47 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 35A092D81; Fri, 25 Jul 2014 07:17:47 +0000 (UTC) Received: from [10.0.1.9] (host86-132-108-36.range86-132.btcentralplus.com [86.132.108.36]) by cyrus.watson.org (Postfix) with ESMTPSA id 38BAC46B43; Fri, 25 Jul 2014 03:17:43 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: "Robert N. M. Watson" In-Reply-To: <20140724175704.GT29618@pwnie.vrt.sourcefire.com> Date: Fri, 25 Jul 2014 08:17:36 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140724175704.GT29618@pwnie.vrt.sourcefire.com> To: Shawn Webb X-Mailer: Apple Mail (2.1878.6) Cc: PaX Team , Pedro Giffuni , Oliver Pinter , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 07:17:47 -0000 On 24 Jul 2014, at 18:57, Shawn Webb wrote: >>> I think someone has already commented that Peter Holm's help might = be=20 >>> enlisted; you have have seen his 'stress2' suite, which could help = with=20 >>> stability testing. >>=20 >> I'll take a look at that, too. Thanks a lot for your suggestions and >> feedback. >=20 > The unixbench results are in. The overall scores are below. >=20 > ASLR Disabled: 456.33 > ASLR Enabled: 357.05 > No ASLR: 474.03 >=20 > I've uploaded the raw results to > http://0xfeedface.org/~shawn/aslr/2014-07-24_benchmark.tar.gz >=20 > Take these results with a grain of salt, given that some of = unixbench's > test are filesystem-related and I'm running ZFS on an old laptop with > little RAM. It does show that there is a performance impact when ASLR = is > enabled. Just in case you've not spotted it, there's some useful benchmarking = advice here: https://wiki.freebsd.org/BenchmarkAdvice Unfortunately, the numbers above are a bit opaque, as it's not clear = whether the differences/non-differences are statistically significant. = Likewise, we'd expect that ASLR might impact some types of behaviour = more than others, and so reduction to a single number can overlook = problems or overemphasise differences. For now, the key thing is really = that there not be any measurable performance difference when ASLR is = disabled, and the numbers above make it a bit unclear if that is the = case. The numbers are definitely difference above -- but perhaps this is = a result of non-essential code generation differences, noise in the run, = etc. Typically, you would want to use a technique such as a t-test to = compare runs and decide if the difference is significant. Tools such as = ministat are very useful here, although you have to be a bit careful as = most performance measurements are already arithmetic means due to the = need to run individual instances of the operation of interest many = times, and comparing means of means is a messy business. The next direction will be to dig more into areas where there are = statistically significant changes to decide whether they are caused by = ASLR, or perhaps are just non-essential differences in code generation. = It may be useful to consider using a suite like 'libmicro' that can = drill into individual system-call behaviour more, as well as = larger-scale benchmarks that consider the behaviour of applications with = realisticish workloads -- Postgres has been of particular interest = lately. Robert= From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 07:33:06 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 58F857E6; Fri, 25 Jul 2014 07:33:06 +0000 (UTC) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 2A8872F41; Fri, 25 Jul 2014 07:33:06 +0000 (UTC) Received: from [10.0.1.9] (host86-132-108-36.range86-132.btcentralplus.com [86.132.108.36]) by cyrus.watson.org (Postfix) with ESMTPSA id 7E2D846B1A; Fri, 25 Jul 2014 03:33:03 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch From: "Robert N. M. Watson" In-Reply-To: <20140724135200.GR29618@pwnie.vrt.sourcefire.com> Date: Fri, 25 Jul 2014 08:32:58 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <796EDB88-3768-48AA-B909-8A7FFBED0C1E@kientzle.com> <20140724135200.GR29618@pwnie.vrt.sourcefire.com> To: Shawn Webb X-Mailer: Apple Mail (2.1878.6) Cc: Tim Kientzle , Bryan Drewery , Pedro Giffuni , freebsd-arch@freebsd.org, PaX Team , Oliver Pinter X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 07:33:06 -0000 On 24 Jul 2014, at 14:52, Shawn Webb wrote: > Great points. I agree completely. This is why I've added two = facilities > for toggling ASLR. I modified mac_bsdextended(4)/ugidfw(8) to provide > toggling ASLR on a per-binary basis. I also made the sysctl tunables > settable per-jail. This allows one to place the executable in question > in a jail where ASLR is turned off just for that jail. I came across > this on Tuesday of this week when doing ClamAV development. Since I do > all my development in a jail, I came across a bug in ClamAV and I = needed > deterministic behavior, so I turned off ASLR just for that jail and = was > able to find the cause of the bug and fix it. Could you say a bit more about the ugidfw use case: is it common to want = to not use ASLR for particular users or groups, rather than it being = focused on applications or installed system instances? One question I have is whether it is sometimes the case that code in = libraries is ASLR-unfriendly, as opposed to code in applications? = Tagging binaries is easy, but as run-time linking of some specific = library within a process might happen after quite a few layout choices = have been influenced by ASLR. Or is it really just applications that = have problems? >> (2) A new inherited process property, or perhaps credential property, = enabling >> ASLR. This can be changed to 'disabled' using a system call -- = perhaps >> prctl() or similar. If we hit a transitioning binary (e.g., = setuid) then >> in the same way that we manipulate other properties, we'd reset = to the >> global default. It would be easy to imagine this being CR_ASLR = on the >> cr_flags field of the credential. This could be set in various = ways by >> userspace applications -- by gdb, as a login-shell property, = perhaps via >> 'su' or something simular? >=20 > This is interesting. I didn't think of using the prctl facility. I'll > look into that. The main use case I have in mind for this is repeatable application = development: I'm running test suites against one or more in-development = programs and would like to get deterministic test-suite runs. This might = well not be under the debugger, so automatic disabling under the = debugger doesn't help, but it is a debugger-like environment in which = determinism is important. >> (3) As suggested by Kostik, an ELF note on binaries indicating that = the binary >> is not ASLR-compatible, which would override (I guess) the global = policy >> flag and process/credential flag. We could then set this with >> NO_ASLR=3Dtrue in Makefiles, during package creation, etc. >=20 > I really don't like ELF flags. It's really heavy-handed. It can trip > file-based IDSs (the hash of the file changes when you change an ELF > flag). It requires that the user, sysadmin, or team of sysadmins keep = a > database of binaries for which the ELF flag needs to be changed when = the > binary gets updated or replaced (think: freebsd-update, pkgng/ports). > This is why I went with mac_bsdextended(4)/ugidfw(8). In the end, the > user is still maintaining a database (via ugidfw(8) rules). But the = user > doesn't need to go back and modify anything if that binary were to = ever > be modified. >=20 > With that said, we are planning on also adding FS extended attribute > support as well. Though I'm unsure if extended attributes work over > network-mounted filesystems like NFS. I'm not imagining that the ELF note is something you change frequently = or dynamically: rather, that it's effectively a part of application = linking by the system build or packaging system. That means that the ELF = note will be in place before any IDS tools generate initial signatures, = etc. E.g., NO_ASLR in a base-system Makefile would cause us to always = generate the binary with a suitable ELF note in place. Similarly, as = part of the port definition, we might identify that some or all binaries = generated by the port require the ELF note. This will be much more = robust than using techniques like ugidfw or extended attributes, where = changes in file ownership, storing on filesystems without extended = attributes, etc, will prove extremely fragile. With that in mind: is there a use case in which you see binaries being = distributed by the FreeBSD Project that an end user will want to always = run without ASLR despite the fact that we believe the program to be = ASLR-friendly? Other than debugging environments, where tools such as = gdb can dynamically disable ASLR for specific processes, or we're not in = a 'debug shell' per above? >> I'm not opposed to MAC policy modules being able to manipulate ASLR = behaviour,=20 >> but I think I'd prefer that the core policy controls (e.g., the = above) be=20 >> MAC-independent. Part of the reason is that you may want ASLR on = very low-end=20 >> systems where the additional cost of MAC interposition is more = measurable. >=20 > How much overhead does mac_bsdextended(4) have? I'm assuming a small > ruleset pertaining to ASLR. Depends a lot on the system and configuration. The MAC Framework has = certain baseline costs when enabled but inactive -- which we have = attempted to minimise to the greatest extent possible. These are = typically masked pretty well on cache-centred, superscalar = architectures, but may be more visible on systems with less ILP and = slower clock rates (e.g., our FPGA-based BERI processors). Turning the = MAC Framework on, however, introduces additional costs -- I've not = benchmarked ugidfw lately, but my guess would be that it has a >0% and = <5% overhead on cached filesystem operations -- something to confirm if = we anticipate it entering universal use rather than being selectively = configured as part of a local tradeoff. I have other concerns about ugidfw as well: we don't have a useful = scheme to 'merge' different sets of ugidfw rule sets -- e.g., ones = provided by the vendor (make the pgsql user always turn off ASLR) vs the = end administrator, as it's intended to be an administrator-facing tool = rather than a vendor-facing tool. More generally, I'd like to learn more = about use cases in which users and groups, rather than application = installs or virtual-system instances, become a desirable granularity for = ASLR policy. >> How does this interact with features like the Linuxulator? Do we = also want=20 >> ABI emulations to be able to disable ASLR as ABIs might not support = it [well]? >=20 > Oliver has done testing on the linuxulator and has found it to be > stable. He can share more info about this. The question I had in mind was: are their ABIs that we emulate in which = introducing ASLR might violate assumptions of the ABI, or disrupt more = subtle functions of the emulator? For example, Kostik has pointed out = that mechanisms such as FreeBSD's ps_strings (part of our ABI) are = sensitive to randomised layout of the initial process stack, and changes = could cause certain ps(1) and procstat(1) functions to fail (e.g., ps = -e, procstat -x). A careful analysis of each ABI would be required to = convince ourselves that this isn't a problem. With ASLR use on many = other systems, it could well be that only the FreeBSD ABI remains = sensitive to surprising randomness, but that's not an assumption I would = reach naturally as well-known addresses have long been part of ABIs. Robert= From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 08:50:20 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2DBCE7FE; Fri, 25 Jul 2014 08:50:20 +0000 (UTC) Received: from r00tworld.com (r00tworld.com [212.85.137.150]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AC5992536; Fri, 25 Jul 2014 08:50:18 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by r00tworld.com (8.13.1/8.13.1) with ESMTP id s6P8XnPr029277; Fri, 25 Jul 2014 10:33:49 +0200 Received: from r00tworld.com ([127.0.0.1]) by localhost (r00tworld.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 13915-09; Fri, 25 Jul 2014 10:33:43 +0200 (CEST) Received: from [192.168.111.1] (x.r00tworld.com [212.85.137.150]) by r00tworld.com (8.13.1/8.13.1) with ESMTP id s6P8Xbd5029260 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 25 Jul 2014 10:33:39 +0200 From: "PaX Team" To: Shawn Webb , "Robert N. M. Watson" Date: Fri, 25 Jul 2014 10:33:34 +0200 MIME-Version: 1.0 Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Reply-to: pageexec@freemail.hu Message-ID: <53D2165E.6871.5524D050@pageexec.freemail.hu> Priority: normal In-reply-to: References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org>, <20140724175704.GT29618@pwnie.vrt.sourcefire.com>, X-mailer: Pegasus Mail for Windows (4.70) Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7BIT Content-description: Mail message body X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.1.12 (r00tworld.com [212.85.137.150]); Fri, 25 Jul 2014 10:33:39 +0200 (CEST) X-Virus-Scanned: r00tworld Anti-Virus System Cc: Pedro Giffuni , Oliver Pinter , Bryan Drewery , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 08:50:20 -0000 On 25 Jul 2014 at 8:17, Robert N. M. Watson wrote: > > The unixbench results are in. The overall scores are below. > > > > ASLR Disabled: 456.33 > > ASLR Enabled: 357.05 > > No ASLR: 474.03 > > > > I've uploaded the raw results to > > http://0xfeedface.org/~shawn/aslr/2014-07-24_benchmark.tar.gz > > > > Take these results with a grain of salt, given that some of unixbench's > > test are filesystem-related and I'm running ZFS on an old laptop with > > little RAM. It does show that there is a performance impact when ASLR is > > enabled. > > Just in case you've not spotted it, there's some useful benchmarking advice here: > > https://wiki.freebsd.org/BenchmarkAdvice > > Unfortunately, the numbers above are a bit opaque, as it's not clear > whether the differences/non-differences are statistically significant. I'm also wondering how stuff like power management was taken into account. Unixbench seems to run various programs for a fixed period of time but that doesn't mean much if thermal throttling, turbo modes, etc kick on and off at random points in the meantime. My suggestion would be to benchmark something that does a fixed amount of work instead (say compile a smaller package) *and* use the CPU's own performance counters (i.e., something like 'perf' on linux). In my experience a good ASLR implementation would not have a measurable impact at all, if there's anything then it's usually due to the too heavyweight entropy extraction method during execve on execve dominated loads (e.g., compiling something or apache forking for each request, etc). cheers, PaX Team From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 15:38:04 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7A5BC5D0; Fri, 25 Jul 2014 15:38:04 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 182632EA1; Fri, 25 Jul 2014 15:38:03 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s6PFbrLf010848 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 25 Jul 2014 18:37:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s6PFbrLf010848 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s6PFbr7o010847; Fri, 25 Jul 2014 18:37:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 25 Jul 2014 18:37:53 +0300 From: Konstantin Belousov To: "Robert N. M. Watson" Subject: Re: [RFC] ASLR Whitepaper and Candidate Final Patch Message-ID: <20140725153753.GB93733@kib.kiev.ua> References: <96C72773-3239-427E-A90B-D05FF0F5B782@freebsd.org> <20140720201858.GB29618@pwnie.vrt.sourcefire.com> <20140723004543.GH29618@pwnie.vrt.sourcefire.com> <20140724175704.GT29618@pwnie.vrt.sourcefire.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZZtaxrRwL/ezAVYu" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: Shawn Webb , Oliver Pinter , Pedro Giffuni , freebsd-arch@freebsd.org, PaX Team , Bryan Drewery X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 15:38:04 -0000 --ZZtaxrRwL/ezAVYu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 25, 2014 at 08:17:36AM +0100, Robert N. M. Watson wrote: >=20 > On 24 Jul 2014, at 18:57, Shawn Webb wrote: >=20 > >>> I think someone has already commented that Peter Holm's help might be= =20 > >>> enlisted; you have have seen his 'stress2' suite, which could help wi= th=20 > >>> stability testing. > >>=20 > >> I'll take a look at that, too. Thanks a lot for your suggestions and > >> feedback. > >=20 > > The unixbench results are in. The overall scores are below. > >=20 > > ASLR Disabled: 456.33 > > ASLR Enabled: 357.05 > > No ASLR: 474.03 > >=20 > > I've uploaded the raw results to > > http://0xfeedface.org/~shawn/aslr/2014-07-24_benchmark.tar.gz > >=20 > > Take these results with a grain of salt, given that some of unixbench's > > test are filesystem-related and I'm running ZFS on an old laptop with > > little RAM. It does show that there is a performance impact when ASLR is > > enabled. >=20 > Just in case you've not spotted it, there's some useful benchmarking > advice here: > > https://wiki.freebsd.org/BenchmarkAdvice > > Unfortunately, the numbers above are a bit opaque, as it's not clear > whether the differences/non-differences are statistically significant. > Likewise, we'd expect that ASLR might impact some types of behaviour > more than others, and so reduction to a single number can overlook > problems or overemphasise differences. For now, the key thing is > really that there not be any measurable performance difference when > ASLR is disabled, and the numbers above make it a bit unclear if > that is the case. The numbers are definitely difference above -- but > perhaps this is a result of non-essential code generation differences, > noise in the run, etc. Typically, you would want to use a technique > such as a t-test to compare runs and decide if the difference is > significant. Tools such as ministat are very useful here, although you > have to be a bit careful as most performance measurements are already > arithmetic means due to the need to run individual instances of the > operation of interest many times, and comp aring means of means is a > messy business. > > The next direction will be to dig more into areas where there are > statistically significant changes to decide whether they are caused > by ASLR, or perhaps are just non-essential differences in code > generation. It may be useful to consider using a suite like 'libmicro' > that can drill into individual system-call behaviour more, as well as > larger-scale benchmarks that consider the behaviour of applications > with realisticish workloads -- Postgres has been of particular > interest lately. The unixbench includes the execve(2) speed test, AFAIR. It is probably the only relevant test from the whole suite. Benchmarking the proposed bug is hard, because it affects mostly the loads which are typically excluded from the normal benchmark's measurement phases. The setup stage, where the image is activated and faulting of the pages needed for the later steady stages is the most vulnerable. The benchmarks itself would typically execute in the mode where only larger page tables, and correlated higher TLB miss frequency could be observed. I expect that the later is hidden by L2 cache if the test working set fits into L2. --ZZtaxrRwL/ezAVYu Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJT0nnQAAoJEJDCuSvBvK1Bh5sP/0ntjqKcSWQP3j3PteT7LHdB BW8y032uIi6FDJywDS4uG8k8lCm7OC/J4uFEnA7z5DMEicLUO02jOhy5cmUH4+a0 m4HkkVjvKZW2FRVcEggrf/cD+doSC2SZ2GqYaKCrYEq4WCuZe18z+TcMH0u8WosD hamBMcECXaYlkTG3sesW2kZd7tYVhj35c77avzw8m9cl33f5Iuo6maUSGIOBFcR+ r40ubtE5fuBshlA+emnTlaNE0qWsxZSK7dlP2LHtrXActI9hAGY0ZeAcO5tHhusr dHp0vc4oaw6f2awaLcU7yul0RfFVMrpZxkXEyOxi3UVn/lMzN2+MUZmI2PjHzkP8 6hA4yNKCyGWKg+zBTcoCERq14Os6lKkViR+0PPeybVanoPxGyYW0UGWPNPNSRhEI LdS+YYH55zXxPrePShhIk4m0hZf2Gzj5AxQU7zIoB20tnrLbtSF2RmrA4pURf4l4 Zrm9yV1+du3mv/Eogd98FUH273bIprZSGhMK0zna3M3cIIdzWShBI7cKWpztBHfD 2JR+5R1BUqDdflhCQOQCFLSAVCgvzTdS7IPkR7AhVH2djfxJYXCImetb+ahXboJu lgBRF4ryPu+m7qoSnymNRdpsT6APi6xjCEbB9ihb/e/iAEjGJzjBZ5JDHalGAbFF LDDihhNu3JWAl3YSM8PV =Llzq -----END PGP SIGNATURE----- --ZZtaxrRwL/ezAVYu-- From owner-freebsd-arch@FreeBSD.ORG Fri Jul 25 16:29:49 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2D403213; Fri, 25 Jul 2014 16:29:49 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1lp0140.outbound.protection.outlook.com [207.46.163.140]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 60F022376; Fri, 25 Jul 2014 16:29:47 +0000 (UTC) Received: from BN1PR05CA004.namprd05.prod.outlook.com (10.255.197.24) by BY2PR05MB726.namprd05.prod.outlook.com (10.141.223.19) with Microsoft SMTP Server (TLS) id 15.0.990.7; Fri, 25 Jul 2014 16:29:44 +0000 Received: from BL2FFO11FD019.protection.gbl (2a01:111:f400:7c09::156) by BN1PR05CA004.outlook.office365.com (2a01:111:e400:400::24) with Microsoft SMTP Server (TLS) id 15.0.995.14 via Frontend Transport; Fri, 25 Jul 2014 16:29:43 +0000 Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by BL2FFO11FD019.mail.protection.outlook.com (10.173.161.37) with Microsoft SMTP Server (TLS) id 15.0.980.11 via Frontend Transport; Fri, 25 Jul 2014 16:29:43 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 25 Jul 2014 09:29:18 -0700 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id s6PGTGn47008; Fri, 25 Jul 2014 09:29:17 -0700 (PDT) (envelope-from sjg@juniper.net) Received: from chaos (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id BCC4F580A2; Fri, 25 Jul 2014 09:29:16 -0700 (PDT) To: Lars Engels Subject: Re: XML Output: libxo - provide single API to output TXT, XML, JSON and HTML In-Reply-To: <20140725065408.GB10844@e-new.0x20.net> References: <20140725044921.9F0D3580A2@chaos.jnpr.net> <20140725065408.GB10844@e-new.0x20.net> Comments: In-reply-to: Lars Engels message dated "Fri, 25 Jul 2014 08:54:08 +0200." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Fri, 25 Jul 2014 09:29:16 -0700 Message-ID: <20140725162916.BCC4F580A2@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:66.129.239.16; CTRY:US; IPV:NLI; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(6009001)(199002)(189002)(69234005)(77982001)(102176002)(87286001)(64706001)(83072002)(6806004)(88136002)(85852003)(105596002)(74662001)(20776003)(79102001)(4396001)(106466001)(46102001)(81156004)(70486001)(50466002)(69596002)(44976005)(95666004)(48376002)(74502001)(87936001)(21056001)(76482001)(62966002)(47776003)(84676001)(50986999)(57986006)(81542001)(83322001)(90896003)(31966008)(97736001)(99396002)(76506005)(102836001)(89996001)(558084003)(93546004)(93916002)(77156001)(92566001)(85306003)(104166001)(101356003)(86362001)(80022001)(50226001)(81342001)(76176999)(110136001)(92726001)(107046002)(33656002)(42262001); DIR:OUT; SFP:; SCL:1; SRVR:BY2PR05MB726; H:P-EMF02-SAC.jnpr.net; FPR:; MLV:sfv; PTR:InfoDomainNonexistent; MX:1; LANG:en; X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-Forefront-PRVS: 02830F0362 Received-SPF: SoftFail (: domain of transitioning juniper.net discourages use of 66.129.239.16 as permitted sender) Authentication-Results: spf=softfail (sender IP is 66.129.239.16) smtp.mailfrom=sjg@juniper.net; X-OriginatorOrg: juniper.net Cc: arch@freebsd.org, marcel@freebsd.org, phil@juniper.net X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 16:29:49 -0000 On Fri, 25 Jul 2014 08:54:08 +0200, Lars Engels writes: >FYI: There's also a Summer of Code project to handle the same: Yes I know. Juniper (as noted in that page) have been doing this for many years (phil infact), and obviously have a strong vested interest in how it is done in FreeBSD. Thanks --sjg From owner-freebsd-arch@FreeBSD.ORG Sat Jul 26 19:25:14 2014 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F3E51DBC for ; Sat, 26 Jul 2014 19:25:13 +0000 (UTC) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBE65296E for ; Sat, 26 Jul 2014 19:25:13 +0000 (UTC) Received: from a6.norwich.yourspac.nsdsl.net ([94.229.131.101]:55023 helo=[172.16.19.1]) by vps.hungerhost.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) (envelope-from ) id 1XB7ah-0008Bq-F8 for arch@freebsd.org; Sat, 26 Jul 2014 15:25:11 -0400 From: "George Neville-Neil" To: arch@freebsd.org Subject: Updating of DTraceToolkit for FreeBSD Date: Sat, 26 Jul 2014 20:25:09 +0100 Message-ID: <40272D9D-3CA1-4092-9CBB-D035D10C0407@neville-neil.com> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.7.2r3905) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 19:25:14 -0000 Hi, Note that I'll be updating bits of the DTraceToolkit in our cddl/contrib directory to work with FreeBSD. Changes will include removing the dependency on ksh in all scripts that I touch along with the conversion of probes to match those on FreeBSD. I am working exclusively in HEAD for now. Please try the scripts on your own systems and let me know if they work or if you have problems. The changes can be tracked via the commits mailing list. To kick off the fun I modified rwsnoop to run on FreeBSD. Note that until I or someone else adds a few more things (like the fds array) some of the scripts will not work exactly the same on Illumos, MacOS and FreeBSD, but that is the eventual goal. Best, George