From owner-freebsd-arch@FreeBSD.ORG Sun Dec 30 23:13:58 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2FD5D9D9; Sun, 30 Dec 2012 23:13:58 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 3C1C58FC0A; Sun, 30 Dec 2012 23:13:57 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id qBUNDuIG079079; Sun, 30 Dec 2012 16:13:56 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id qBUNDhtn084905; Sun, 30 Dec 2012 16:13:43 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Subject: Re: [RFC/RFT] calloutng From: Ian Lepore To: Alexander Motin In-Reply-To: <50DB4EFE.2020600@FreeBSD.org> References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> Content-Type: multipart/mixed; boundary="=-0QeMdI0U6ePkF5/otAWO" Date: Sun, 30 Dec 2012 16:13:43 -0700 Message-ID: <1356909223.54953.74.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: Davide Italiano , freebsd-arch@freebsd.org, FreeBSD Current , Marius Strobl X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2012 23:13:58 -0000 --=-0QeMdI0U6ePkF5/otAWO Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Wed, 2012-12-26 at 21:24 +0200, Alexander Motin wrote: > On 26.12.2012 01:21, Marius Strobl wrote: > > On Tue, Dec 18, 2012 at 11:03:47AM +0200, Alexander Motin wrote: > >> Experiments with dummynet shown ineffective support for very short > >> tick-based callouts. New version fixes that, allowing to get as many > >> tick-based callout events as hz value permits, while still be able to > >> aggregate events and generating minimum of interrupts. > >> > >> Also this version modifies system load average calculation to fix some > >> cases existing in HEAD and 9 branches, that could be fixed with new > >> direct callout functionality. > >> > >> http://people.freebsd.org/~mav/calloutng_12_17.patch > >> > >> With several important changes made last time I am going to delay commit > >> to HEAD for another week to do more testing. Comments and new test cases > >> are welcome. Thanks for staying tuned and commenting. > > > > FYI, I gave both calloutng_12_15_1.patch and calloutng_12_17.patch a > > try on sparc64 and it at least survives a buildworld there. However, > > with the patched kernels, buildworld times seem to increase slightly but > > reproducible by 1-2% (I only did four runs but typically buildworld > > times are rather stable and don't vary more than a minute for the > > same kernel and source here). Is this an expected trade-off (system > > time as such doesn't seem to increase)? > > I don't think build process uses significant number of callouts to > affect results directly. I think this additional time could be result of > the deeper next event look up, done by the new code, that is practically > useless for sparc64, which effectively has no cpu_idle() routine. It > wouldn't affect system time and wouldn't show up in any statistics > (except PMC or something alike) because it is executed inside timer > hardware interrupt handler. If my guess is right, that is a part that > probably still could be optimized. I'll look on it. Thanks. > > > Is there anything specific to test? > > Since the most of code is MI, for sparc64 I would mostly look on related > MD parts (eventtimers and timecounters) to make sure they are working > reliably in more stressful conditions. I still have some worries about > possible deadlock on hardware where IPIs are used to fetch present time > from other CPU. > > Here is small tool we are using for test correctness and performance of > different user-level APIs: http://people.freebsd.org/~mav/testsleep.c > I grabbed testsleep.c to test an arm event timer implementation, and had to fix a couple nits... kqueueto was missing from the names[] array, and I had to add a "* 1000" to a couple places where usec was stuffed into a timespec's tv_nsec. I also tested the calloutng_12_17 patches and the kqueue stuff behaved very strangely. Then I noticed you had a 12_26 patchset so I tested that (after crudely fixing a couple uninitialized var warnings), and it all looks good on this arm (Raspberry Pi). I'll attach the results. It's so sweet to be able to do precision sleeps. -- Ian --=-0QeMdI0U6ePkF5/otAWO Content-Disposition: inline; filename="calloutng_test.txt" Content-Type: text/plain; name="calloutng_test.txt"; charset="us-ascii" Content-Transfer-Encoding: 7bit for t in 1 300 3000 30000 300000 ; do for m in select poll usleep nanosleep kqueue kqueueto syscall ; do ./testsleep $t $m done done With calloutng_12_26.patch... HZ=100 HZ=250 HZ=1000 ---------- ---------------- ---------------- ---------------- select 1 55.79 1 50.96 1 61.32 poll 1 1109.46 1 1107.86 1 1114.38 usleep 1 56.33 1 72.90 1 62.78 nanosleep 1 52.66 1 55.23 1 64.23 kqueue 1 1114.23 1 1113.81 1 1121.21 kqueueto 1 65.44 1 71.00 1 75.01 syscall 1 4.70 1 4.45 1 4.55 select 300 355.79 300 357.76 300 362.35 poll 300 1107.85 300 1122.55 300 1115.62 usleep 300 355.28 300 357.28 300 360.79 nanosleep 300 354.49 300 355.82 300 360.62 kqueue 300 1112.57 300 1118.13 300 1117.16 kqueueto 300 375.98 300 378.62 300 395.61 syscall 300 4.41 300 4.45 300 4.54 select 3000 3246.75 3000 3246.74 3000 3252.72 poll 3000 3238.10 3000 3229.12 3000 3250.10 usleep 3000 3242.47 3000 3237.06 3000 3249.61 nanosleep 3000 3238.79 3000 3231.55 3000 3248.11 kqueue 3000 3240.01 3000 3236.07 3000 3247.60 kqueueto 3000 3265.36 3000 3267.22 3000 3274.96 syscall 3000 4.69 3000 4.44 3000 4.50 select 30000 31714.60 30000 31941.17 30000 32467.69 poll 30000 31522.76 30000 31983.00 30000 32497.81 usleep 30000 31459.67 30000 31980.76 30000 32458.71 nanosleep 30000 31431.02 30000 31982.22 30000 32525.20 kqueue 30000 31466.75 30000 31873.90 30000 31973.54 kqueueto 30000 31564.67 30000 32522.35 30000 32475.59 syscall 30000 4.70 30000 4.73 30000 4.89 select 300000 319133.02 300000 311562.33 300000 309918.62 poll 300000 319604.27 300000 311422.94 300000 310000.76 usleep 300000 319314.60 300000 311269.69 300000 309996.34 nanosleep 300000 319497.58 300000 311425.40 300000 309997.13 kqueue 300000 309995.55 300000 303980.27 300000 309908.82 kqueueto 300000 319505.88 300000 311424.97 300000 309996.16 syscall 300000 4.41 300000 4.45 300000 4.89 With no patches... HZ=100 HZ=250 HZ=1000 ---------- ---------------- ---------------- ---------------- select 1 19941.70 1 7989.10 1 1999.16 poll 1 19904.61 1 7987.32 1 1999.78 usleep 1 19904.95 1 7993.30 1 1999.96 nanosleep 1 19905.64 1 7993.71 1 1999.72 kqueue 1 10001.61 1 4004.00 1 1000.27 kqueueto 1 19904.00 1 7993.03 1 1999.54 syscall 1 4.04 1 4.05 1 4.75 select 300 19904.66 300 7998.39 300 2000.27 poll 300 19904.35 300 7993.47 300 1999.86 usleep 300 19903.96 300 7994.11 300 1999.81 nanosleep 300 19904.48 300 7993.77 300 1999.80 kqueue 300 10001.68 300 4004.18 300 1000.31 kqueueto 300 19997.86 300 7993.37 300 1999.59 syscall 300 4.01 300 4.00 300 4.32 select 3000 19904.80 3000 7998.85 3000 3998.43 poll 3000 19904.92 3000 8005.93 3000 3999.39 usleep 3000 19904.50 3000 7992.88 3000 3999.44 nanosleep 3000 19904.84 3000 7993.34 3000 3999.36 kqueue 3000 10001.58 3000 4003.97 3000 3000.72 kqueueto 3000 19903.56 3000 7993.24 3000 3999.34 syscall 3000 4.02 3000 4.37 3000 4.29 select 30000 39905.02 30000 35991.79 30000 31051.77 poll 30000 39905.49 30000 35980.35 30000 30995.64 usleep 30000 39903.78 30000 35979.48 30000 30995.23 nanosleep 30000 39904.55 30000 35981.61 30000 30995.87 kqueue 30000 30002.73 30000 32019.54 30000 30004.83 kqueueto 30000 39903.59 30000 35979.64 30000 30996.05 syscall 30000 4.44 30000 4.04 30000 4.31 select 300000 310001.23 300000 303995.86 300000 300994.30 poll 300000 309902.73 300000 303981.58 300000 300996.17 usleep 300000 309903.64 300000 303980.17 300000 300997.42 nanosleep 300000 309903.32 300000 303980.36 300000 300993.64 kqueue 300000 300002.77 300000 300019.46 300000 300006.90 kqueueto 300000 309903.31 300000 303978.10 300000 300996.84 syscall 300000 4.01 300000 4.04 300000 4.29 --=-0QeMdI0U6ePkF5/otAWO-- From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 06:18:44 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D496F382; Mon, 31 Dec 2012 06:18:44 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 37D5F8FC08; Mon, 31 Dec 2012 06:18:43 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 025DD7300A; Mon, 31 Dec 2012 07:17:36 +0100 (CET) Date: Mon, 31 Dec 2012 07:17:35 +0100 From: Luigi Rizzo To: Ian Lepore Subject: Re: [RFC/RFT] calloutng Message-ID: <20121231061735.GA5866@onelab2.iet.unipi.it> References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> <1356909223.54953.74.camel@revolution.hippie.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1356909223.54953.74.camel@revolution.hippie.lan> User-Agent: Mutt/1.4.2.3i Cc: Davide Italiano , Marius Strobl , Alexander Motin , FreeBSD Current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 06:18:44 -0000 On Sun, Dec 30, 2012 at 04:13:43PM -0700, Ian Lepore wrote: ... > I grabbed testsleep.c to test an arm event timer implementation, and had > to fix a couple nits... kqueueto was missing from the names[] array, and > I had to add a "* 1000" to a couple places where usec was stuffed into a > timespec's tv_nsec. > > I also tested the calloutng_12_17 patches and the kqueue stuff behaved > very strangely. Then I noticed you had a 12_26 patchset so I tested > that (after crudely fixing a couple uninitialized var warnings), and it > all looks good on this arm (Raspberry Pi). I'll attach the results. > > It's so sweet to be able to do precision sleeps. interesting numbers, but there seems to be some problem in computing the exact interval; delays are much larger than expected. In this test, the original timer code used to round to the next multiple of 1 tick and then add another tick (except for the kqueue case), which is exactly what you see in the second set of measurements. The calloutng code however seems to do something odd: in addition to fixed overhead (some 50us, which you can see in the tests for 1us and 300us), all delay seem to be ~10% larger than what is requested, upper bounded to 10ms (note, the numbers are averages so i cannot tell whether all samples are the same or there is some distribution of values). I am not sure if this error is peculiar of the ARM version or also appears on x86/amd64 but I believe it should be fixed. If you look at the results below: 1us possily ok: for very short intervals i would expect some kind of 'reschedule' without actually firing a timer; maybe 50us are what it takes to do a round through the scheduler ? 300us probably ok i guess the extra 50-90us are what it takes to do a round through the scheduler 1000us borderline (this is the case for poll and kqueue, which are rounded to 1ms) here intervals seem to be increased by 10%, and i cannot see a good reason for this (more below). 3000us and above: wrong here again, the intervals seem to be 10% larger than what is requested, perhaps limiting the error to 10-20ms. Maybe the 10% extension results from creating a default 'precision' for legacy calls, but i do not think this is done correctly. First of all, if users do not specify a precision themselves, the automatically generated value should never exceed one tick. Second, the only point of a 'precision' parameter is to merge requests that may be close in time, so if there is already a timer scheduled within [Treq, Treq+precision] i will get it; but if there no pending timer, then one should schedule it for the requested interval. Davide/Alexander, any ideas ? cheers luigi > for t in 1 300 3000 30000 300000 ; do > for m in select poll usleep nanosleep kqueue kqueueto syscall ; do > ./testsleep $t $m > done > done > > > With calloutng_12_26.patch... > > HZ=100 HZ=250 HZ=1000 > ---------- ---------------- ---------------- ---------------- > select 1 55.79 1 50.96 1 61.32 > poll 1 1109.46 1 1107.86 1 1114.38 > usleep 1 56.33 1 72.90 1 62.78 > nanosleep 1 52.66 1 55.23 1 64.23 > kqueue 1 1114.23 1 1113.81 1 1121.21 > kqueueto 1 65.44 1 71.00 1 75.01 > syscall 1 4.70 1 4.45 1 4.55 > select 300 355.79 300 357.76 300 362.35 > poll 300 1107.85 300 1122.55 300 1115.62 > usleep 300 355.28 300 357.28 300 360.79 > nanosleep 300 354.49 300 355.82 300 360.62 > kqueue 300 1112.57 300 1118.13 300 1117.16 > kqueueto 300 375.98 300 378.62 300 395.61 > syscall 300 4.41 300 4.45 300 4.54 > select 3000 3246.75 3000 3246.74 3000 3252.72 > poll 3000 3238.10 3000 3229.12 3000 3250.10 > usleep 3000 3242.47 3000 3237.06 3000 3249.61 > nanosleep 3000 3238.79 3000 3231.55 3000 3248.11 > kqueue 3000 3240.01 3000 3236.07 3000 3247.60 > kqueueto 3000 3265.36 3000 3267.22 3000 3274.96 > syscall 3000 4.69 3000 4.44 3000 4.50 > select 30000 31714.60 30000 31941.17 30000 32467.69 > poll 30000 31522.76 30000 31983.00 30000 32497.81 > usleep 30000 31459.67 30000 31980.76 30000 32458.71 > nanosleep 30000 31431.02 30000 31982.22 30000 32525.20 > kqueue 30000 31466.75 30000 31873.90 30000 31973.54 > kqueueto 30000 31564.67 30000 32522.35 30000 32475.59 > syscall 30000 4.70 30000 4.73 30000 4.89 > select 300000 319133.02 300000 311562.33 300000 309918.62 > poll 300000 319604.27 300000 311422.94 300000 310000.76 > usleep 300000 319314.60 300000 311269.69 300000 309996.34 > nanosleep 300000 319497.58 300000 311425.40 300000 309997.13 > kqueue 300000 309995.55 300000 303980.27 300000 309908.82 > kqueueto 300000 319505.88 300000 311424.97 300000 309996.16 > syscall 300000 4.41 300000 4.45 300000 4.89 > > > With no patches... > > HZ=100 HZ=250 HZ=1000 > ---------- ---------------- ---------------- ---------------- > select 1 19941.70 1 7989.10 1 1999.16 > poll 1 19904.61 1 7987.32 1 1999.78 > usleep 1 19904.95 1 7993.30 1 1999.96 > nanosleep 1 19905.64 1 7993.71 1 1999.72 > kqueue 1 10001.61 1 4004.00 1 1000.27 > kqueueto 1 19904.00 1 7993.03 1 1999.54 > syscall 1 4.04 1 4.05 1 4.75 > select 300 19904.66 300 7998.39 300 2000.27 > poll 300 19904.35 300 7993.47 300 1999.86 > usleep 300 19903.96 300 7994.11 300 1999.81 > nanosleep 300 19904.48 300 7993.77 300 1999.80 > kqueue 300 10001.68 300 4004.18 300 1000.31 > kqueueto 300 19997.86 300 7993.37 300 1999.59 > syscall 300 4.01 300 4.00 300 4.32 > select 3000 19904.80 3000 7998.85 3000 3998.43 > poll 3000 19904.92 3000 8005.93 3000 3999.39 > usleep 3000 19904.50 3000 7992.88 3000 3999.44 > nanosleep 3000 19904.84 3000 7993.34 3000 3999.36 > kqueue 3000 10001.58 3000 4003.97 3000 3000.72 > kqueueto 3000 19903.56 3000 7993.24 3000 3999.34 > syscall 3000 4.02 3000 4.37 3000 4.29 > select 30000 39905.02 30000 35991.79 30000 31051.77 > poll 30000 39905.49 30000 35980.35 30000 30995.64 > usleep 30000 39903.78 30000 35979.48 30000 30995.23 > nanosleep 30000 39904.55 30000 35981.61 30000 30995.87 > kqueue 30000 30002.73 30000 32019.54 30000 30004.83 > kqueueto 30000 39903.59 30000 35979.64 30000 30996.05 > syscall 30000 4.44 30000 4.04 30000 4.31 > select 300000 310001.23 300000 303995.86 300000 300994.30 > poll 300000 309902.73 300000 303981.58 300000 300996.17 > usleep 300000 309903.64 300000 303980.17 300000 300997.42 > nanosleep 300000 309903.32 300000 303980.36 300000 300993.64 > kqueue 300000 300002.77 300000 300019.46 300000 300006.90 > kqueueto 300000 309903.31 300000 303978.10 300000 300996.84 > syscall 300000 4.01 300000 4.04 300000 4.29 From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 08:00:29 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6AC51140; Mon, 31 Dec 2012 08:00:29 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-f176.google.com (mail-vc0-f176.google.com [209.85.220.176]) by mx1.freebsd.org (Postfix) with ESMTP id D7BC88FC08; Mon, 31 Dec 2012 08:00:28 +0000 (UTC) Received: by mail-vc0-f176.google.com with SMTP id fo13so12569151vcb.35 for ; Mon, 31 Dec 2012 00:00:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=SPs38VQcmnu+caTyQwu/Z/8HaDdk5iZEn2W1EkN9GTY=; b=NmodQung/gEwb88SxllsH2ZvEVxRoSOfm0S0ecCqtwIPYzltchxjyEPlprnzL3S12k rtr2tTcSSneJl1pHbHOp6eQncY5piw9F1gNp/SCfFHBvjHwcgY78LTzNZQKx5vcdr3Hh w6arJyGm8YDd7E0BG0rdMHavwEvX+NWSg5l2P1IXjorAIcdrGMb3B4gaKnWLCsDMPU9L IfgY0BZ+Sq+RUaJ2jgZwXcBb2dIyqOfxGJVq51W/t8LwcUxW3A/o6pJPhsGLhgy1c720 yj54Q31umHsmJRO9wHiVa6TEmMpiJQvY289ZuoHsHyVpmCp4RqfnXe1XczQuwsdcelvb WWhQ== MIME-Version: 1.0 Received: by 10.58.222.40 with SMTP id qj8mr64611180vec.36.1356940822078; Mon, 31 Dec 2012 00:00:22 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.229.136 with HTTP; Mon, 31 Dec 2012 00:00:21 -0800 (PST) In-Reply-To: <20121231061735.GA5866@onelab2.iet.unipi.it> References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> <1356909223.54953.74.camel@revolution.hippie.lan> <20121231061735.GA5866@onelab2.iet.unipi.it> Date: Mon, 31 Dec 2012 00:00:21 -0800 X-Google-Sender-Auth: u3rbG-o2zq2jDv9iiGJpHWs8HMg Message-ID: Subject: Re: [RFC/RFT] calloutng From: Davide Italiano To: Luigi Rizzo Content-Type: text/plain; charset=ISO-8859-1 Cc: Ian Lepore , Alexander Motin , FreeBSD Current , Marius Strobl , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 08:00:29 -0000 On Sun, Dec 30, 2012 at 10:17 PM, Luigi Rizzo wrote: > On Sun, Dec 30, 2012 at 04:13:43PM -0700, Ian Lepore wrote: > ... >> I grabbed testsleep.c to test an arm event timer implementation, and had >> to fix a couple nits... kqueueto was missing from the names[] array, and >> I had to add a "* 1000" to a couple places where usec was stuffed into a >> timespec's tv_nsec. >> >> I also tested the calloutng_12_17 patches and the kqueue stuff behaved >> very strangely. Then I noticed you had a 12_26 patchset so I tested >> that (after crudely fixing a couple uninitialized var warnings), and it >> all looks good on this arm (Raspberry Pi). I'll attach the results. >> >> It's so sweet to be able to do precision sleeps. > > interesting numbers, but there seems to be some problem in computing > the exact interval; delays are much larger than expected. > > In this test, the original timer code used to round to the next multiple > of 1 tick and then add another tick (except for the kqueue case), > which is exactly what you see in the second set of measurements. > > The calloutng code however seems to do something odd: > in addition to fixed overhead (some 50us, which you can see in > the tests for 1us and 300us), all delay seem to be ~10% larger > than what is requested, upper bounded to 10ms (note, the > numbers are averages so i cannot tell whether all samples are > the same or there is some distribution of values). > > I am not sure if this error is peculiar of the ARM version or also > appears on x86/amd64 but I believe it should be fixed. > > If you look at the results below: > > 1us possily ok: > for very short intervals i would expect some kind > of 'reschedule' without actually firing a timer; maybe > 50us are what it takes to do a round through the scheduler ? > > 300us probably ok > i guess the extra 50-90us are what it takes to do a round > through the scheduler > > 1000us borderline (this is the case for poll and kqueue, which are > rounded to 1ms) > here intervals seem to be increased by 10%, and i cannot see > a good reason for this (more below). > > 3000us and above: wrong > here again, the intervals seem to be 10% larger than what is > requested, perhaps limiting the error to 10-20ms. > > > Maybe the 10% extension results from creating a default 'precision' > for legacy calls, but i do not think this is done correctly. > > First of all, if users do not specify a precision themselves, the > automatically generated value should never exceed one tick. > > Second, the only point of a 'precision' parameter is to merge > requests that may be close in time, so if there is already a > timer scheduled within [Treq, Treq+precision] i will get it; > but if there no pending timer, then one should schedule it > for the requested interval. > > Davide/Alexander, any ideas ? > The first question is what couple timecounter/eventtimer is used. Some hardware is slower than other, and this may affect numbers. Thanks for testing, BTW. Davide > cheers > luigi > >> for t in 1 300 3000 30000 300000 ; do >> for m in select poll usleep nanosleep kqueue kqueueto syscall ; do >> ./testsleep $t $m >> done >> done >> >> >> With calloutng_12_26.patch... >> >> HZ=100 HZ=250 HZ=1000 >> ---------- ---------------- ---------------- ---------------- >> select 1 55.79 1 50.96 1 61.32 >> poll 1 1109.46 1 1107.86 1 1114.38 >> usleep 1 56.33 1 72.90 1 62.78 >> nanosleep 1 52.66 1 55.23 1 64.23 >> kqueue 1 1114.23 1 1113.81 1 1121.21 >> kqueueto 1 65.44 1 71.00 1 75.01 >> syscall 1 4.70 1 4.45 1 4.55 >> select 300 355.79 300 357.76 300 362.35 >> poll 300 1107.85 300 1122.55 300 1115.62 >> usleep 300 355.28 300 357.28 300 360.79 >> nanosleep 300 354.49 300 355.82 300 360.62 >> kqueue 300 1112.57 300 1118.13 300 1117.16 >> kqueueto 300 375.98 300 378.62 300 395.61 >> syscall 300 4.41 300 4.45 300 4.54 >> select 3000 3246.75 3000 3246.74 3000 3252.72 >> poll 3000 3238.10 3000 3229.12 3000 3250.10 >> usleep 3000 3242.47 3000 3237.06 3000 3249.61 >> nanosleep 3000 3238.79 3000 3231.55 3000 3248.11 >> kqueue 3000 3240.01 3000 3236.07 3000 3247.60 >> kqueueto 3000 3265.36 3000 3267.22 3000 3274.96 >> syscall 3000 4.69 3000 4.44 3000 4.50 >> select 30000 31714.60 30000 31941.17 30000 32467.69 >> poll 30000 31522.76 30000 31983.00 30000 32497.81 >> usleep 30000 31459.67 30000 31980.76 30000 32458.71 >> nanosleep 30000 31431.02 30000 31982.22 30000 32525.20 >> kqueue 30000 31466.75 30000 31873.90 30000 31973.54 >> kqueueto 30000 31564.67 30000 32522.35 30000 32475.59 >> syscall 30000 4.70 30000 4.73 30000 4.89 >> select 300000 319133.02 300000 311562.33 300000 309918.62 >> poll 300000 319604.27 300000 311422.94 300000 310000.76 >> usleep 300000 319314.60 300000 311269.69 300000 309996.34 >> nanosleep 300000 319497.58 300000 311425.40 300000 309997.13 >> kqueue 300000 309995.55 300000 303980.27 300000 309908.82 >> kqueueto 300000 319505.88 300000 311424.97 300000 309996.16 >> syscall 300000 4.41 300000 4.45 300000 4.89 >> >> >> With no patches... >> >> HZ=100 HZ=250 HZ=1000 >> ---------- ---------------- ---------------- ---------------- >> select 1 19941.70 1 7989.10 1 1999.16 >> poll 1 19904.61 1 7987.32 1 1999.78 >> usleep 1 19904.95 1 7993.30 1 1999.96 >> nanosleep 1 19905.64 1 7993.71 1 1999.72 >> kqueue 1 10001.61 1 4004.00 1 1000.27 >> kqueueto 1 19904.00 1 7993.03 1 1999.54 >> syscall 1 4.04 1 4.05 1 4.75 >> select 300 19904.66 300 7998.39 300 2000.27 >> poll 300 19904.35 300 7993.47 300 1999.86 >> usleep 300 19903.96 300 7994.11 300 1999.81 >> nanosleep 300 19904.48 300 7993.77 300 1999.80 >> kqueue 300 10001.68 300 4004.18 300 1000.31 >> kqueueto 300 19997.86 300 7993.37 300 1999.59 >> syscall 300 4.01 300 4.00 300 4.32 >> select 3000 19904.80 3000 7998.85 3000 3998.43 >> poll 3000 19904.92 3000 8005.93 3000 3999.39 >> usleep 3000 19904.50 3000 7992.88 3000 3999.44 >> nanosleep 3000 19904.84 3000 7993.34 3000 3999.36 >> kqueue 3000 10001.58 3000 4003.97 3000 3000.72 >> kqueueto 3000 19903.56 3000 7993.24 3000 3999.34 >> syscall 3000 4.02 3000 4.37 3000 4.29 >> select 30000 39905.02 30000 35991.79 30000 31051.77 >> poll 30000 39905.49 30000 35980.35 30000 30995.64 >> usleep 30000 39903.78 30000 35979.48 30000 30995.23 >> nanosleep 30000 39904.55 30000 35981.61 30000 30995.87 >> kqueue 30000 30002.73 30000 32019.54 30000 30004.83 >> kqueueto 30000 39903.59 30000 35979.64 30000 30996.05 >> syscall 30000 4.44 30000 4.04 30000 4.31 >> select 300000 310001.23 300000 303995.86 300000 300994.30 >> poll 300000 309902.73 300000 303981.58 300000 300996.17 >> usleep 300000 309903.64 300000 303980.17 300000 300997.42 >> nanosleep 300000 309903.32 300000 303980.36 300000 300993.64 >> kqueue 300000 300002.77 300000 300019.46 300000 300006.90 >> kqueueto 300000 309903.31 300000 303978.10 300000 300996.84 >> syscall 300000 4.01 300000 4.04 300000 4.29 From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 10:17:40 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DC49B2A0; Mon, 31 Dec 2012 10:17:40 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f41.google.com (mail-ee0-f41.google.com [74.125.83.41]) by mx1.freebsd.org (Postfix) with ESMTP id 0AD238FC12; Mon, 31 Dec 2012 10:17:39 +0000 (UTC) Received: by mail-ee0-f41.google.com with SMTP id d41so6247476eek.28 for ; Mon, 31 Dec 2012 02:17:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=FSL5cERnn0KcJG32xw3ftGzxZpAT3VlCzYXtzysnkNQ=; b=tmu98rReHUMAyUT/PEKZbqH9Rjw2jrzosdtFs53O/nrOy6+l9OzQx6ki32nTazeqDC 67rrzw3wqPEhmPcW9BWgwequQn0CPNY2fknpBstnaLCiqVcCXDLYb7v+b+jEALCoB6Ri ob27Gyo2KNGa82b8sHChTuhgFqA2iq5kjpHloM8Rr/HQLKkRnSlTCrmrvL+dLbyxfliQ qu6T7eoXiou3lf+nirBjVNSzJGn+3W0xz5MJj36M6se4zHUNLBsINs1bi6TVuvZgFumx Hhyns86l45Jj7wb7LKhN5EMLzMU4MJ/AlhCNzCf0MLe/B7Q+gwqOB/tsBQygtwiNegBz P1/Q== X-Received: by 10.14.194.195 with SMTP id m43mr110059168een.44.1356949052812; Mon, 31 Dec 2012 02:17:32 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([91.198.175.1]) by mx.google.com with ESMTPS id 6sm84105691eea.3.2012.12.31.02.17.30 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 31 Dec 2012 02:17:31 -0800 (PST) Sender: Alexander Motin Message-ID: <50E16637.9070501@FreeBSD.org> Date: Mon, 31 Dec 2012 12:17:27 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120628 Thunderbird/13.0.1 MIME-Version: 1.0 To: Luigi Rizzo Subject: Re: [RFC/RFT] calloutng References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> <1356909223.54953.74.camel@revolution.hippie.lan> <20121231061735.GA5866@onelab2.iet.unipi.it> In-Reply-To: <20121231061735.GA5866@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Davide Italiano , Ian Lepore , FreeBSD Current , Marius Strobl , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 10:17:40 -0000 On 31.12.2012 08:17, Luigi Rizzo wrote: > On Sun, Dec 30, 2012 at 04:13:43PM -0700, Ian Lepore wrote: > ... >> I grabbed testsleep.c to test an arm event timer implementation, and had >> to fix a couple nits... kqueueto was missing from the names[] array, and >> I had to add a "* 1000" to a couple places where usec was stuffed into a >> timespec's tv_nsec. >> >> I also tested the calloutng_12_17 patches and the kqueue stuff behaved >> very strangely. I've rewritten kqueue timeouts at the calloutng_12_26.patch. >> Then I noticed you had a 12_26 patchset so I tested >> that (after crudely fixing a couple uninitialized var warnings), and it >> all looks good on this arm (Raspberry Pi). I'll attach the results. >> >> It's so sweet to be able to do precision sleeps. Thank you for testing, Ian. > interesting numbers, but there seems to be some problem in computing > the exact interval; delays are much larger than expected. > > In this test, the original timer code used to round to the next multiple > of 1 tick and then add another tick (except for the kqueue case), > which is exactly what you see in the second set of measurements. > > The calloutng code however seems to do something odd: > in addition to fixed overhead (some 50us, which you can see in > the tests for 1us and 300us), all delay seem to be ~10% larger > than what is requested, upper bounded to 10ms (note, the > numbers are averages so i cannot tell whether all samples are > the same or there is some distribution of values). > > I am not sure if this error is peculiar of the ARM version or also > appears on x86/amd64 but I believe it should be fixed. > > If you look at the results below: > > 1us possily ok: > for very short intervals i would expect some kind > of 'reschedule' without actually firing a timer; maybe > 50us are what it takes to do a round through the scheduler ? > > 300us probably ok > i guess the extra 50-90us are what it takes to do a round > through the scheduler > > 1000us borderline (this is the case for poll and kqueue, which are > rounded to 1ms) > here intervals seem to be increased by 10%, and i cannot see > a good reason for this (more below). > > 3000us and above: wrong > here again, the intervals seem to be 10% larger than what is > requested, perhaps limiting the error to 10-20ms. > > > Maybe the 10% extension results from creating a default 'precision' > for legacy calls, but i do not think this is done correctly. > > First of all, if users do not specify a precision themselves, the > automatically generated value should never exceed one tick. > > Second, the only point of a 'precision' parameter is to merge > requests that may be close in time, so if there is already a > timer scheduled within [Treq, Treq+precision] i will get it; > but if there no pending timer, then one should schedule it > for the requested interval. > > Davide/Alexander, any ideas ? All mentioned effects could be explained with implemented logic. 50us at 1us is probably sum of minimal latency of the hardware eventtimer on the specific platform and some software processing overhead (syscall, callout, timecouters, scheduler, etc). At later points system starts to noticeably use precision specified by kern.timecounter.alloweddeviation sysctl. It affects results from two sides: 1) extending intervals for specified percent of time to allow event aggregation, and 2) choosing time base between fast getbinuptime() and precise binuptime(). Extending interval is needed to aggregate not only callouts with each other, but also callouts with other system events, which are impossible to schedule in advance. It gives specified relative error, but no more then one CPU wakeup period in absolute: for busy CPU (not skipping hardclock() ticks) it is 1/hz, for completely idle one it can be up to 0.5s. Second point allows to reduce processing overhead by the cost of error up to 1/hz for long periods (>(100/allowed)*(1/hz)), when it is used. To get best possible precision kern.timecounter.alloweddeviation sysctl can be set to smaller value. Setting it to 0 will effectively disable all optimizations, but should give 50us precision in all cases. >> for t in 1 300 3000 30000 300000 ; do >> for m in select poll usleep nanosleep kqueue kqueueto syscall ; do >> ./testsleep $t $m >> done >> done >> >> >> With calloutng_12_26.patch... >> >> HZ=100 HZ=250 HZ=1000 >> ---------- ---------------- ---------------- ---------------- >> select 1 55.79 1 50.96 1 61.32 >> poll 1 1109.46 1 1107.86 1 1114.38 >> usleep 1 56.33 1 72.90 1 62.78 >> nanosleep 1 52.66 1 55.23 1 64.23 >> kqueue 1 1114.23 1 1113.81 1 1121.21 >> kqueueto 1 65.44 1 71.00 1 75.01 >> syscall 1 4.70 1 4.45 1 4.55 >> select 300 355.79 300 357.76 300 362.35 >> poll 300 1107.85 300 1122.55 300 1115.62 >> usleep 300 355.28 300 357.28 300 360.79 >> nanosleep 300 354.49 300 355.82 300 360.62 >> kqueue 300 1112.57 300 1118.13 300 1117.16 >> kqueueto 300 375.98 300 378.62 300 395.61 >> syscall 300 4.41 300 4.45 300 4.54 >> select 3000 3246.75 3000 3246.74 3000 3252.72 >> poll 3000 3238.10 3000 3229.12 3000 3250.10 >> usleep 3000 3242.47 3000 3237.06 3000 3249.61 >> nanosleep 3000 3238.79 3000 3231.55 3000 3248.11 >> kqueue 3000 3240.01 3000 3236.07 3000 3247.60 >> kqueueto 3000 3265.36 3000 3267.22 3000 3274.96 >> syscall 3000 4.69 3000 4.44 3000 4.50 >> select 30000 31714.60 30000 31941.17 30000 32467.69 >> poll 30000 31522.76 30000 31983.00 30000 32497.81 >> usleep 30000 31459.67 30000 31980.76 30000 32458.71 >> nanosleep 30000 31431.02 30000 31982.22 30000 32525.20 >> kqueue 30000 31466.75 30000 31873.90 30000 31973.54 >> kqueueto 30000 31564.67 30000 32522.35 30000 32475.59 >> syscall 30000 4.70 30000 4.73 30000 4.89 >> select 300000 319133.02 300000 311562.33 300000 309918.62 >> poll 300000 319604.27 300000 311422.94 300000 310000.76 >> usleep 300000 319314.60 300000 311269.69 300000 309996.34 >> nanosleep 300000 319497.58 300000 311425.40 300000 309997.13 >> kqueue 300000 309995.55 300000 303980.27 300000 309908.82 >> kqueueto 300000 319505.88 300000 311424.97 300000 309996.16 >> syscall 300000 4.41 300000 4.45 300000 4.89 >> >> >> With no patches... >> >> HZ=100 HZ=250 HZ=1000 >> ---------- ---------------- ---------------- ---------------- >> select 1 19941.70 1 7989.10 1 1999.16 >> poll 1 19904.61 1 7987.32 1 1999.78 >> usleep 1 19904.95 1 7993.30 1 1999.96 >> nanosleep 1 19905.64 1 7993.71 1 1999.72 >> kqueue 1 10001.61 1 4004.00 1 1000.27 >> kqueueto 1 19904.00 1 7993.03 1 1999.54 >> syscall 1 4.04 1 4.05 1 4.75 >> select 300 19904.66 300 7998.39 300 2000.27 >> poll 300 19904.35 300 7993.47 300 1999.86 >> usleep 300 19903.96 300 7994.11 300 1999.81 >> nanosleep 300 19904.48 300 7993.77 300 1999.80 >> kqueue 300 10001.68 300 4004.18 300 1000.31 >> kqueueto 300 19997.86 300 7993.37 300 1999.59 >> syscall 300 4.01 300 4.00 300 4.32 >> select 3000 19904.80 3000 7998.85 3000 3998.43 >> poll 3000 19904.92 3000 8005.93 3000 3999.39 >> usleep 3000 19904.50 3000 7992.88 3000 3999.44 >> nanosleep 3000 19904.84 3000 7993.34 3000 3999.36 >> kqueue 3000 10001.58 3000 4003.97 3000 3000.72 >> kqueueto 3000 19903.56 3000 7993.24 3000 3999.34 >> syscall 3000 4.02 3000 4.37 3000 4.29 >> select 30000 39905.02 30000 35991.79 30000 31051.77 >> poll 30000 39905.49 30000 35980.35 30000 30995.64 >> usleep 30000 39903.78 30000 35979.48 30000 30995.23 >> nanosleep 30000 39904.55 30000 35981.61 30000 30995.87 >> kqueue 30000 30002.73 30000 32019.54 30000 30004.83 >> kqueueto 30000 39903.59 30000 35979.64 30000 30996.05 >> syscall 30000 4.44 30000 4.04 30000 4.31 >> select 300000 310001.23 300000 303995.86 300000 300994.30 >> poll 300000 309902.73 300000 303981.58 300000 300996.17 >> usleep 300000 309903.64 300000 303980.17 300000 300997.42 >> nanosleep 300000 309903.32 300000 303980.36 300000 300993.64 >> kqueue 300000 300002.77 300000 300019.46 300000 300006.90 >> kqueueto 300000 309903.31 300000 303978.10 300000 300996.84 >> syscall 300000 4.01 300000 4.04 300000 4.29 -- Alexander Motin From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 14:33:48 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 37D45ADC; Mon, 31 Dec 2012 14:33:48 +0000 (UTC) (envelope-from masked@internode.on.net) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [IPv6:2001:44b8:8060:ff02:300:1:6:5]) by mx1.freebsd.org (Postfix) with ESMTP id 32A2C8FC13; Mon, 31 Dec 2012 14:33:46 +0000 (UTC) Received: from ppp221-140.static.internode.on.net (HELO forexamplePC) ([150.101.221.140]) by ipmail05.adl6.internode.on.net with SMTP; 01 Jan 2013 01:03:45 +1030 Message-ID: <641F54E990B14A0AB0A287AE7F810192@forexamplePC> From: "Michael Vale" To: "Simon J. Gerraty" References: <20121227180044.7947F58094@chaos.jnpr.net> In-Reply-To: <20121227180044.7947F58094@chaos.jnpr.net> Subject: Re: Cross Compiling of ports Makefiles. Date: Tue, 1 Jan 2013 01:33:44 +1100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 16.4.3505.912 X-MimeOLE: Produced By Microsoft MimeOLE V16.4.3505.912 Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 14:33:48 -0000 -----Original Message----- From: Simon J. Gerraty Sent: Friday, December 28, 2012 5:00 AM To: Michael Vale Cc: freebsd-hackers@freebsd.org ; freebsd-arch@freebsd.org ; freebsd-ports@freebsd.org Subject: Re: Cross Compiling of ports Makefiles. >Doing the same thing could also prevent the need for a DESTDIR JAIL >install at all and just use the real build machine’s build env, rather >than a jail. Regardless. We still have to install these targets and >their DESTDIR is skewed. There is a few options, I think I know what you mean, but not clear on the "their DESTDIR is skewed" bit. I'm not sure what I meant here either. Thank-you for taking the time to read the entire e-mail! :) >One is to have a MAKEOBJDIRPREFIX like option, and redefine every >target’s DESTDIR ${makeobjDESTDIR} before running do-install. Now i’ve >yet to complete this stage, but I believe this is the way to do it. Would it be sufficient to have an INSTALL_PREFIX and/or INSTALL_DESTDIR so that DESTDIR can be different during install ? [I was recently experimenting with something similar...] So how would that work? pre-install: INSTALL_DESTDIR=/usr/obj/crossoutroot/ DESTDIR=${INSTALL_DESTDIR} do-install: ? I will try something like that. Thank-you for taking the time to reply Simon, Michael Vale. ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5998 - Release Date: 12/30/12 ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5998 - Release Date: 12/30/12 From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 15:02:56 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2890F3B5; Mon, 31 Dec 2012 15:02:56 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 63D648FC0A; Mon, 31 Dec 2012 15:02:50 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id qBVF2nlH089212; Mon, 31 Dec 2012 08:02:49 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id qBVF2iQ3085747; Mon, 31 Dec 2012 08:02:44 -0700 (MST) (envelope-from freebsd@damnhippie.dyndns.org) Subject: Re: [RFC/RFT] calloutng From: Ian Lepore To: Alexander Motin In-Reply-To: <50E16637.9070501@FreeBSD.org> References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> <1356909223.54953.74.camel@revolution.hippie.lan> <20121231061735.GA5866@onelab2.iet.unipi.it> <50E16637.9070501@FreeBSD.org> Content-Type: text/plain; charset="us-ascii" Date: Mon, 31 Dec 2012 08:02:44 -0700 Message-ID: <1356966164.54953.121.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Davide Italiano , freebsd-arch@freebsd.org, FreeBSD Current , Marius Strobl X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 15:02:56 -0000 On Mon, 2012-12-31 at 12:17 +0200, Alexander Motin wrote: > On 31.12.2012 08:17, Luigi Rizzo wrote: > > On Sun, Dec 30, 2012 at 04:13:43PM -0700, Ian Lepore wrote: > > ... > >> I grabbed testsleep.c to test an arm event timer implementation, and had > >> to fix a couple nits... kqueueto was missing from the names[] array, and > >> I had to add a "* 1000" to a couple places where usec was stuffed into a > >> timespec's tv_nsec. > >> > >> I also tested the calloutng_12_17 patches and the kqueue stuff behaved > >> very strangely. > > I've rewritten kqueue timeouts at the calloutng_12_26.patch. > > >> Then I noticed you had a 12_26 patchset so I tested > >> that (after crudely fixing a couple uninitialized var warnings), and it > >> all looks good on this arm (Raspberry Pi). I'll attach the results. > >> > >> It's so sweet to be able to do precision sleeps. > > Thank you for testing, Ian. > > > interesting numbers, but there seems to be some problem in computing > > the exact interval; delays are much larger than expected. > > > > In this test, the original timer code used to round to the next multiple > > of 1 tick and then add another tick (except for the kqueue case), > > which is exactly what you see in the second set of measurements. > > > > The calloutng code however seems to do something odd: > > in addition to fixed overhead (some 50us, which you can see in > > the tests for 1us and 300us), all delay seem to be ~10% larger > > than what is requested, upper bounded to 10ms (note, the > > numbers are averages so i cannot tell whether all samples are > > the same or there is some distribution of values). > > > > I am not sure if this error is peculiar of the ARM version or also > > appears on x86/amd64 but I believe it should be fixed. > > > > If you look at the results below: > > > > 1us possily ok: > > for very short intervals i would expect some kind > > of 'reschedule' without actually firing a timer; maybe > > 50us are what it takes to do a round through the scheduler ? > > > > 300us probably ok > > i guess the extra 50-90us are what it takes to do a round > > through the scheduler > > > > 1000us borderline (this is the case for poll and kqueue, which are > > rounded to 1ms) > > here intervals seem to be increased by 10%, and i cannot see > > a good reason for this (more below). > > > > 3000us and above: wrong > > here again, the intervals seem to be 10% larger than what is > > requested, perhaps limiting the error to 10-20ms. > > > > > > Maybe the 10% extension results from creating a default 'precision' > > for legacy calls, but i do not think this is done correctly. > > > > First of all, if users do not specify a precision themselves, the > > automatically generated value should never exceed one tick. > > > > Second, the only point of a 'precision' parameter is to merge > > requests that may be close in time, so if there is already a > > timer scheduled within [Treq, Treq+precision] i will get it; > > but if there no pending timer, then one should schedule it > > for the requested interval. > > > > Davide/Alexander, any ideas ? > > All mentioned effects could be explained with implemented logic. 50us at > 1us is probably sum of minimal latency of the hardware eventtimer on the > specific platform and some software processing overhead (syscall, > callout, timecouters, scheduler, etc). At later points system starts to > noticeably use precision specified by kern.timecounter.alloweddeviation > sysctl. It affects results from two sides: 1) extending intervals for > specified percent of time to allow event aggregation, and 2) choosing > time base between fast getbinuptime() and precise binuptime(). Extending > interval is needed to aggregate not only callouts with each other, but > also callouts with other system events, which are impossible to schedule > in advance. It gives specified relative error, but no more then one CPU > wakeup period in absolute: for busy CPU (not skipping hardclock() ticks) > it is 1/hz, for completely idle one it can be up to 0.5s. Second point > allows to reduce processing overhead by the cost of error up to 1/hz for > long periods (>(100/allowed)*(1/hz)), when it is used. > > To get best possible precision kern.timecounter.alloweddeviation sysctl > can be set to smaller value. Setting it to 0 will effectively disable > all optimizations, but should give 50us precision in all cases. > > >> for t in 1 300 3000 30000 300000 ; do > >> for m in select poll usleep nanosleep kqueue kqueueto syscall ; do > >> ./testsleep $t $m > >> done > >> done > >> > >> [test results snipped] > I should have posted some information about the test platform... It's a single-processor 700mhz arm chip. There was essentially nothing else running during the tests other than mostly-idle daemons (sshd, ntpd, the usual suspects). Kernel debugging options off (INVARIANTS[_SUPPORT], DIAGNOSTIC, and WITNESS). Some sysctl values of interest... rpi# sysctl kern.timecounter kern.timecounter.fast_gettime: 1 kern.timecounter.tick: 1 kern.timecounter.choice: BCM2835 Timecounter(1000) dummy(-1000000) kern.timecounter.hardware: BCM2835 Timecounter kern.timecounter.alloweddeviation: 5 kern.timecounter.stepwarnings: 1 kern.timecounter.tc.BCM2835 Timecounter.mask: 4294967295 kern.timecounter.tc.BCM2835 Timecounter.counter: 734706756 kern.timecounter.tc.BCM2835 Timecounter.frequency: 1000000 kern.timecounter.tc.BCM2835 Timecounter.quality: 1000 rpi# sysctl kern.eventtimer kern.eventtimer.choice: BCM2835 Event Timer 3(1000) kern.eventtimer.et.BCM2835 Event Timer 3.flags: 2 kern.eventtimer.et.BCM2835 Event Timer 3.frequency: 1000000 kern.eventtimer.et.BCM2835 Event Timer 3.quality: 1000 kern.eventtimer.periodic: 0 kern.eventtimer.timer: BCM2835 Event Timer 3 kern.eventtimer.activetick: 1 kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 4 BTW, is there any advantage to implementing periodic mode for an eventtimer? It would be easy enough to do for this hardware, but it looks like all this new event timer code is pretty much a stake through the heart of periodic timer ticks. -- Ian From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 15:37:20 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F32EF881; Mon, 31 Dec 2012 15:37:19 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f176.google.com (mail-ob0-f176.google.com [209.85.214.176]) by mx1.freebsd.org (Postfix) with ESMTP id 8F2218FC0A; Mon, 31 Dec 2012 15:37:18 +0000 (UTC) Received: by mail-ob0-f176.google.com with SMTP id un3so11366041obb.7 for ; Mon, 31 Dec 2012 07:37:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=uBXqLFDZbyxzE9qw79sdCK3jopldso1qmOhvyiDRBzo=; b=wZxq/fglN06yy5V8n+1wENbLTlElZnXnvoIPmltdcgrGtyQp6RTaa1w13EPolvGaBx 3jFR5IDSKztJbbZtzcPc9A20WnDrim/qy8jXJOcQ6X2thCMN6Ru+XPx1zzeyWRiFBllf lpoazWJg8nRhLn3ZwpHukzdZSP6WX9c9cxYSLs8oS/EUkz9qHdUHJYbaxaDwY6eYtluK u13oJF+bb/9k+z9RDx5ERJt8IgziK6ughRoCzU2jJP6/UtETu6J+RYJjT+4zI72RbIbV rZe3Jx5bewctsj0q0nQ74H6TC2Ri7BpvRtZ+GHpHbVicMs9mVNBFWmf0ymip6bokwe4V w+Pg== MIME-Version: 1.0 Received: by 10.60.10.133 with SMTP id i5mr21777668oeb.24.1356968237743; Mon, 31 Dec 2012 07:37:17 -0800 (PST) Received: by 10.76.143.33 with HTTP; Mon, 31 Dec 2012 07:37:17 -0800 (PST) In-Reply-To: <641F54E990B14A0AB0A287AE7F810192@forexamplePC> References: <20121227180044.7947F58094@chaos.jnpr.net> <641F54E990B14A0AB0A287AE7F810192@forexamplePC> Date: Mon, 31 Dec 2012 07:37:17 -0800 Message-ID: Subject: Re: Cross Compiling of ports Makefiles. From: Garrett Cooper To: Michael Vale Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, freebsd-arch@freebsd.org, freebsd-ports@freebsd.org, "Simon J. Gerraty" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 15:37:20 -0000 On Mon, Dec 31, 2012 at 6:33 AM, Michael Vale wro= te: > -----Original Message----- From: Simon J. Gerraty > Sent: Friday, December 28, 2012 5:00 AM > To: Michael Vale > Cc: freebsd-hackers@freebsd.org ; freebsd-arch@freebsd.org ; > freebsd-ports@freebsd.org > Subject: Re: Cross Compiling of ports Makefiles. > > >> Doing the same thing could also prevent the need for a DESTDIR JAIL >> install at all and just use the real build machine=92s build env, rather >> than a jail. Regardless. We still have to install these targets and >> their DESTDIR is skewed. There is a few options, > > > I think I know what you mean, but not clear on the "their DESTDIR is > skewed" bit. You were probably thinking of PREFIX, not DESTDIR. DESTDIR in ports-land should be an install directory wherein if I do... ./configure --prefix=3D/usr/local make all make install DESTDIR=3D/chroot/ It would install all (well, ok most if it needs to touch more of the base system) of the package files into /chroot/usr/local and pathing would be setup appropriately that way. > I'm not sure what I meant here either. Thank-you for taking the time to > read the entire e-mail! :) > >> One is to have a MAKEOBJDIRPREFIX like option, and redefine every >> target=92s DESTDIR ${makeobjDESTDIR} before running do-install. Now i= =92ve >> yet to complete this stage, but I believe this is the way to do it. > > Would it be sufficient to have an INSTALL_PREFIX and/or INSTALL_DESTDIR > so that DESTDIR can be different during install ? > [I was recently experimenting with something similar...] > > > So how would that work? > > pre-install: > INSTALL_DESTDIR=3D/usr/obj/crossoutroot/ > DESTDIR=3D${INSTALL_DESTDIR} > > do-install: > > ? I will try something like that. > > Thank-you for taking the time to reply Simon, I've thought about this item quite a bit and while I might not have hashed out all of the internal details, I think that it should go something like this: 0. Run standard build/install targets which will go and create necessary binaries into a predefined world dir. Once the installation is complete (installworld / distribution)... 1. Mount the necessary mountpoints. 2. Install the "host-compiled" tools into a predefined set of directories. Example: /chroot/h/bin /chroot/h/sbin /chroot/h/usr/bin /chroot/h/usr/libexec /chroot/h/usr/sbin 3. Do a nullfs overlay of the "host-compiled" tools on top of the target system's equivalent directories in order to provide the needed bits for executing the build. 4. Verify sanity for the install base (just in case the new binaries don't run on the host system due to KPI differences) with a basic check like we use for make in `upgrade_checks`. 5. Mock up the build environment to look like the target system, like what's described . 6. Jump into the world dir. 7. Start building/installing packages. Notes: I'm suggesting steps 2. and 3. because while fixing hardcoding in ports packages is a noble effort, there's just way too much work to be done in order to accomplish the job (we have other problems to contend with around the ports tree) and it's an ongoing battle trying to get everyone to use sane build methodologies (I'm making an assumption based on several projects I've used, but many devs aren't capable build system engineers because it takes a bit of mental skewing, so hardcoding abounds in a number of places). Doing this will allow us to have a working prototype quicker, so in the event that others wish to make the process more streamlined in the future, they could do so. Whether or not this handled in the FreeBSD build system or outside it is an implementation detail, but for the sake of modularity (and to keep KISS principle with the FreeBSD build system, which I would argue is complicated enough) I would say make them separate processes. Besides, we already have canned methods for doing this in NanoBSD, PicoBSD, etc already -- and they really could deserve some consolidation (speaking of which, have you looked at using these in zrouter?). You might want to look at FreeNAS 8.x's build architecture and use that as a starting point for how to do things. imp@ designed the initial system, I (gcooper@) modified it heavily, and jhixson@/jpaetzel@ modified it more after I left iXsystems in order to work with their plugin architecture. I can send you a copy of the system I was starting to architect ("Avatar") if you wish. Overall, I like the direction you're going in and I'm glad someone is picking this up. I greatly appreciate it because it's been on my laundry list of items that need to get done for some time :). HTH, -Garrett From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 15:41:55 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3FEDBBC; Mon, 31 Dec 2012 15:41:55 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-f180.google.com (mail-ea0-f180.google.com [209.85.215.180]) by mx1.freebsd.org (Postfix) with ESMTP id BC9F58FC0A; Mon, 31 Dec 2012 15:41:54 +0000 (UTC) Received: by mail-ea0-f180.google.com with SMTP id f13so4898631eai.25 for ; Mon, 31 Dec 2012 07:41:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=ytQzY76ftkiIspE7Hu+QwblRBnbzQQJtZbgSK0k+nSI=; b=iDk2W4a1kJQjwisGyA8v+RYJ18RWJl/ww+eWkSCRMY0ZszbUIgKpiQ3MJSVE3yUPm9 2PjnYgr2q4oKBd4prkUxxRPGUOvwfMpEpcGpFKeu5QIcxgAjafk1uXNkXotiBUl4IKxW eWmeDsq+lSwv1TcUJwBrB0LTI4fxqQKlPtar3V7mtmDCy2Q+LlUzQEB/hIKbKcmK89qU xazJuPNR0TBBf/DXVjriCYOZrwmrp2I+UW6W5dHb6bZ+sh4NIESCEjSVZOHhcNoqRl6l dDjOLBV494WfVNZ+6qOq6Ld0ebNqy1M3AgZHUnygD//B3hDbanAcdDhn9OknRuGpertr XkIg== X-Received: by 10.14.173.65 with SMTP id u41mr110983401eel.13.1356968513616; Mon, 31 Dec 2012 07:41:53 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([91.198.175.1]) by mx.google.com with ESMTPS id f49sm85749799eep.12.2012.12.31.07.41.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 31 Dec 2012 07:41:52 -0800 (PST) Sender: Alexander Motin Message-ID: <50E1B23C.6060804@FreeBSD.org> Date: Mon, 31 Dec 2012 17:41:48 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120628 Thunderbird/13.0.1 MIME-Version: 1.0 To: Ian Lepore Subject: Re: [RFC/RFT] calloutng References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> <1356909223.54953.74.camel@revolution.hippie.lan> <20121231061735.GA5866@onelab2.iet.unipi.it> <50E16637.9070501@FreeBSD.org> <1356966164.54953.121.camel@revolution.hippie.lan> In-Reply-To: <1356966164.54953.121.camel@revolution.hippie.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Davide Italiano , freebsd-arch@freebsd.org, FreeBSD Current , Marius Strobl X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 15:41:55 -0000 On 31.12.2012 17:02, Ian Lepore wrote: > On Mon, 2012-12-31 at 12:17 +0200, Alexander Motin wrote: >> On 31.12.2012 08:17, Luigi Rizzo wrote: >>> On Sun, Dec 30, 2012 at 04:13:43PM -0700, Ian Lepore wrote: >>> ... >>>> I grabbed testsleep.c to test an arm event timer implementation, and had >>>> to fix a couple nits... kqueueto was missing from the names[] array, and >>>> I had to add a "* 1000" to a couple places where usec was stuffed into a >>>> timespec's tv_nsec. >>>> >>>> I also tested the calloutng_12_17 patches and the kqueue stuff behaved >>>> very strangely. >> >> I've rewritten kqueue timeouts at the calloutng_12_26.patch. >> >>>> Then I noticed you had a 12_26 patchset so I tested >>>> that (after crudely fixing a couple uninitialized var warnings), and it >>>> all looks good on this arm (Raspberry Pi). I'll attach the results. >>>> >>>> It's so sweet to be able to do precision sleeps. >> >> Thank you for testing, Ian. >> >>> interesting numbers, but there seems to be some problem in computing >>> the exact interval; delays are much larger than expected. >>> >>> In this test, the original timer code used to round to the next multiple >>> of 1 tick and then add another tick (except for the kqueue case), >>> which is exactly what you see in the second set of measurements. >>> >>> The calloutng code however seems to do something odd: >>> in addition to fixed overhead (some 50us, which you can see in >>> the tests for 1us and 300us), all delay seem to be ~10% larger >>> than what is requested, upper bounded to 10ms (note, the >>> numbers are averages so i cannot tell whether all samples are >>> the same or there is some distribution of values). >>> >>> I am not sure if this error is peculiar of the ARM version or also >>> appears on x86/amd64 but I believe it should be fixed. >>> >>> If you look at the results below: >>> >>> 1us possily ok: >>> for very short intervals i would expect some kind >>> of 'reschedule' without actually firing a timer; maybe >>> 50us are what it takes to do a round through the scheduler ? >>> >>> 300us probably ok >>> i guess the extra 50-90us are what it takes to do a round >>> through the scheduler >>> >>> 1000us borderline (this is the case for poll and kqueue, which are >>> rounded to 1ms) >>> here intervals seem to be increased by 10%, and i cannot see >>> a good reason for this (more below). >>> >>> 3000us and above: wrong >>> here again, the intervals seem to be 10% larger than what is >>> requested, perhaps limiting the error to 10-20ms. >>> >>> >>> Maybe the 10% extension results from creating a default 'precision' >>> for legacy calls, but i do not think this is done correctly. >>> >>> First of all, if users do not specify a precision themselves, the >>> automatically generated value should never exceed one tick. >>> >>> Second, the only point of a 'precision' parameter is to merge >>> requests that may be close in time, so if there is already a >>> timer scheduled within [Treq, Treq+precision] i will get it; >>> but if there no pending timer, then one should schedule it >>> for the requested interval. >>> >>> Davide/Alexander, any ideas ? >> >> All mentioned effects could be explained with implemented logic. 50us at >> 1us is probably sum of minimal latency of the hardware eventtimer on the >> specific platform and some software processing overhead (syscall, >> callout, timecouters, scheduler, etc). At later points system starts to >> noticeably use precision specified by kern.timecounter.alloweddeviation >> sysctl. It affects results from two sides: 1) extending intervals for >> specified percent of time to allow event aggregation, and 2) choosing >> time base between fast getbinuptime() and precise binuptime(). Extending >> interval is needed to aggregate not only callouts with each other, but >> also callouts with other system events, which are impossible to schedule >> in advance. It gives specified relative error, but no more then one CPU >> wakeup period in absolute: for busy CPU (not skipping hardclock() ticks) >> it is 1/hz, for completely idle one it can be up to 0.5s. Second point >> allows to reduce processing overhead by the cost of error up to 1/hz for >> long periods (>(100/allowed)*(1/hz)), when it is used. >> >> To get best possible precision kern.timecounter.alloweddeviation sysctl >> can be set to smaller value. Setting it to 0 will effectively disable >> all optimizations, but should give 50us precision in all cases. >> >>>> for t in 1 300 3000 30000 300000 ; do >>>> for m in select poll usleep nanosleep kqueue kqueueto syscall ; do >>>> ./testsleep $t $m >>>> done >>>> done >>>> >>>> [test results snipped] >> > > I should have posted some information about the test platform... It's a > single-processor 700mhz arm chip. There was essentially nothing else > running during the tests other than mostly-idle daemons (sshd, ntpd, the > usual suspects). Kernel debugging options off (INVARIANTS[_SUPPORT], > DIAGNOSTIC, and WITNESS). > > Some sysctl values of interest... > > rpi# sysctl kern.timecounter > kern.timecounter.fast_gettime: 1 > kern.timecounter.tick: 1 > kern.timecounter.choice: BCM2835 Timecounter(1000) dummy(-1000000) > kern.timecounter.hardware: BCM2835 Timecounter > kern.timecounter.alloweddeviation: 5 > kern.timecounter.stepwarnings: 1 > kern.timecounter.tc.BCM2835 Timecounter.mask: 4294967295 > kern.timecounter.tc.BCM2835 Timecounter.counter: 734706756 > kern.timecounter.tc.BCM2835 Timecounter.frequency: 1000000 > kern.timecounter.tc.BCM2835 Timecounter.quality: 1000 > rpi# sysctl kern.eventtimer > kern.eventtimer.choice: BCM2835 Event Timer 3(1000) > kern.eventtimer.et.BCM2835 Event Timer 3.flags: 2 > kern.eventtimer.et.BCM2835 Event Timer 3.frequency: 1000000 > kern.eventtimer.et.BCM2835 Event Timer 3.quality: 1000 > kern.eventtimer.periodic: 0 > kern.eventtimer.timer: BCM2835 Event Timer 3 > kern.eventtimer.activetick: 1 > kern.eventtimer.idletick: 0 > kern.eventtimer.singlemul: 4 > > BTW, is there any advantage to implementing periodic mode for an > eventtimer? It would be easy enough to do for this hardware, but it > looks like all this new event timer code is pretty much a stake through > the heart of periodic timer ticks. Periodic-mode-only hardware is still supported, but present code takes almost no advantage from periodic mode if one-shot mode is supported. It can't use interrupts as time source to run events (as legacy code did) because of possible drift from system timecounter that makes impossible to specify absolute event time. The only benefit is that timer hardware is not reprogrammed each time, and I don't think that this economy worth result. But for all hardware supporting periodic mode I've implemented respective support at least for completeness and testing purposes. -- Alexander Motin From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 16:32:00 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9AEBE961; Mon, 31 Dec 2012 16:32:00 +0000 (UTC) (envelope-from masked@internode.on.net) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [IPv6:2001:44b8:8060:ff02:300:1:6:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7A80E8FC08; Mon, 31 Dec 2012 16:31:59 +0000 (UTC) Received: from ppp221-140.static.internode.on.net (HELO forexamplePC) ([150.101.221.140]) by ipmail05.adl6.internode.on.net with SMTP; 01 Jan 2013 03:01:54 +1030 Message-ID: <62E9D311E3F4410C93F39BF78047F4C4@forexamplePC> From: "Michael Vale" To: "Garrett Cooper" References: <20121227180044.7947F58094@chaos.jnpr.net> <641F54E990B14A0AB0A287AE7F810192@forexamplePC> In-Reply-To: Subject: Re: Cross Compiling of ports Makefiles. Date: Tue, 1 Jan 2013 03:31:53 +1100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="Windows-1252"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 16.4.3505.912 X-MimeOLE: Produced By Microsoft MimeOLE V16.4.3505.912 Cc: freebsd-hackers@freebsd.org, "Simon J. Gerraty" , freebsd-ports@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 16:32:00 -0000 -----Original Message----- From: Garrett Cooper Sent: Tuesday, January 01, 2013 2:37 AM To: Michael Vale Cc: freebsd-hackers@freebsd.org ; freebsd-arch@freebsd.org ; freebsd-ports@freebsd.org ; Simon J. Gerraty Subject: Re: Cross Compiling of ports Makefiles. On Mon, Dec 31, 2012 at 6:33 AM, Michael Vale wrote: > -----Original Message----- From: Simon J. Gerraty > Sent: Friday, December 28, 2012 5:00 AM > To: Michael Vale > Cc: freebsd-hackers@freebsd.org ; freebsd-arch@freebsd.org ; > freebsd-ports@freebsd.org > Subject: Re: Cross Compiling of ports Makefiles. > > >> Doing the same thing could also prevent the need for a DESTDIR JAIL >> install at all and just use the real build machine’s build env, rather >> than a jail. Regardless. We still have to install these targets and >> their DESTDIR is skewed. There is a few options, > > > I think I know what you mean, but not clear on the "their DESTDIR is > skewed" bit. You were probably thinking of PREFIX, not DESTDIR. DESTDIR in ports-land should be an install directory wherein if I do... ./configure --prefix=/usr/local make all make install DESTDIR=/chroot/ It would install all (well, ok most if it needs to touch more of the base system) of the package files into /chroot/usr/local and pathing would be setup appropriately that way. Ahh, What I think I meant was their DESTDIR is skewed because it installs into ${JAILDIR}, not ${DESTDIR}${ROOTFS}, which could just == ${DESTDIR}. ${PREFIX} cannot be used as it is required to be correct for pkg register, pkgdb, ldconfig etc. > I'm not sure what I meant here either. Thank-you for taking the time to > read the entire e-mail! :) > >> One is to have a MAKEOBJDIRPREFIX like option, and redefine every >> target’s DESTDIR ${makeobjDESTDIR} before running do-install. Now i’ve >> yet to complete this stage, but I believe this is the way to do it. > > Would it be sufficient to have an INSTALL_PREFIX and/or INSTALL_DESTDIR > so that DESTDIR can be different during install ? > [I was recently experimenting with something similar...] > > > So how would that work? > > pre-install: > INSTALL_DESTDIR=/usr/obj/crossoutroot/ > DESTDIR=${INSTALL_DESTDIR} > > do-install: > > ? I will try something like that. > > Thank-you for taking the time to reply Simon, I've thought about this item quite a bit and while I might not have hashed out all of the internal details, I think that it should go something like this: 0. Run standard build/install targets which will go and create necessary binaries into a predefined world dir. Once the installation is complete (installworld / distribution)... 1. Mount the necessary mountpoints. 2. Install the "host-compiled" tools into a predefined set of directories. Example: /chroot/h/bin /chroot/h/sbin /chroot/h/usr/bin /chroot/h/usr/libexec /chroot/h/usr/sbin 3. Do a nullfs overlay of the "host-compiled" tools on top of the target system's equivalent directories in order to provide the needed bits for executing the build. 4. Verify sanity for the install base (just in case the new binaries don't run on the host system due to KPI differences) with a basic check like we use for make in `upgrade_checks`. 5. Mock up the build environment to look like the target system, like what's described . 6. Jump into the world dir. 7. Start building/installing packages. Notes: I'm suggesting steps 2. and 3. because while fixing hardcoding in ports packages is a noble effort, there's just way too much work to be done in order to accomplish the job (we have other problems to contend with around the ports tree) and it's an ongoing battle trying to get everyone to use sane build methodologies (I'm making an assumption based on several projects I've used, but many devs aren't capable build system engineers because it takes a bit of mental skewing, so hardcoding abounds in a number of places). Doing this will allow us to have a working prototype quicker, so in the event that others wish to make the process more streamlined in the future, they could do so. Whether or not this handled in the FreeBSD build system or outside it is an implementation detail, but for the sake of modularity (and to keep KISS principle with the FreeBSD build system, which I would argue is complicated enough) I would say make them separate processes. Besides, we already have canned methods for doing this in NanoBSD, PicoBSD, etc already -- and they really could deserve some consolidation (speaking of which, have you looked at using these in zrouter?). You might want to look at FreeNAS 8.x's build architecture and use that as a starting point for how to do things. imp@ designed the initial system, I (gcooper@) modified it heavily, and jhixson@/jpaetzel@ modified it more after I left iXsystems in order to work with their plugin architecture. I can send you a copy of the system I was starting to architect ("Avatar") if you wish. Overall, I like the direction you're going in and I'm glad someone is picking this up. I greatly appreciate it because it's been on my laundry list of items that need to get done for some time :). HTH, -Garrett _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5998 - Release Date: 12/30/12 ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5998 - Release Date: 12/30/12 From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 17:45:53 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 786D322B; Mon, 31 Dec 2012 17:45:53 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from exprod7og103.obsmtp.com (exprod7og103.obsmtp.com [64.18.2.159]) by mx1.freebsd.org (Postfix) with ESMTP id 275598FC08; Mon, 31 Dec 2012 17:45:52 +0000 (UTC) Received: from P-EMHUB03-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob103.postini.com ([64.18.6.12]) with SMTP ID DSNKUOHPR4X1WDfWEaj4M3ZMhmiCWqlGiFnZ@postini.com; Mon, 31 Dec 2012 09:45:53 PST Received: from magenta.juniper.net (172.17.27.123) by P-EMHUB03-HQ.jnpr.net (172.24.192.33) with Microsoft SMTP Server (TLS) id 8.3.213.0; Mon, 31 Dec 2012 09:43:08 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id qBVHh8338885; Mon, 31 Dec 2012 09:43:08 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 05BDE58094; Mon, 31 Dec 2012 09:43:08 -0800 (PST) To: Michael Vale Subject: Re: Cross Compiling of ports Makefiles. In-Reply-To: <641F54E990B14A0AB0A287AE7F810192@forexamplePC> References: <20121227180044.7947F58094@chaos.jnpr.net> <641F54E990B14A0AB0A287AE7F810192@forexamplePC> Comments: In-reply-to: "Michael Vale" message dated "Tue, 01 Jan 2013 01:33:44 +1100." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Mon, 31 Dec 2012 09:43:07 -0800 Message-ID: <20121231174308.05BDE58094@chaos.jnpr.net> Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 17:45:53 -0000 >>One is to have a MAKEOBJDIRPREFIX like option, and redefine every >>target=E2=80=99s DESTDIR ${makeobjDESTDIR} before running do-install. No= w i=E2=80=99ve >>yet to complete this stage, but I believe this is the way to do it. > >Would it be sufficient to have an INSTALL_PREFIX and/or INSTALL_DESTDIR >so that DESTDIR can be different during install ? >[I was recently experimenting with something similar...] > >So how would that work? I was assuming a sub-make involved: install: ${MAKE} DESTDIR=3D${INSTALL_DESTDIR} realinstall From owner-freebsd-arch@FreeBSD.ORG Mon Dec 31 18:42:26 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 46356D13; Mon, 31 Dec 2012 18:42:26 +0000 (UTC) (envelope-from masked@internode.on.net) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [IPv6:2001:44b8:8060:ff02:300:1:6:5]) by mx1.freebsd.org (Postfix) with ESMTP id 40A8D8FC08; Mon, 31 Dec 2012 18:42:24 +0000 (UTC) Received: from ppp221-140.static.internode.on.net (HELO forexamplePC) ([150.101.221.140]) by ipmail05.adl6.internode.on.net with SMTP; 01 Jan 2013 05:12:23 +1030 Message-ID: <674B18C07D96418681C34AD34370F5B6@forexamplePC> From: "Michael Vale" To: "Simon J. Gerraty" References: <20121227180044.7947F58094@chaos.jnpr.net> <641F54E990B14A0AB0A287AE7F810192@forexamplePC> <20121231174308.05BDE58094@chaos.jnpr.net> In-Reply-To: <20121231174308.05BDE58094@chaos.jnpr.net> Subject: Re: Cross Compiling of ports Makefiles. Date: Tue, 1 Jan 2013 05:42:19 +1100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 16.4.3505.912 X-MimeOLE: Produced By Microsoft MimeOLE V16.4.3505.912 Cc: freebsd-hackers@freebsd.org, freebsd-ports@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 18:42:26 -0000 -----Original Message----- From: Simon J. Gerraty Sent: Tuesday, January 01, 2013 4:43 AM To: Michael Vale Cc: freebsd-hackers@freebsd.org ; freebsd-arch@freebsd.org ; freebsd-ports@freebsd.org Subject: Re: Cross Compiling of ports Makefiles. >>One is to have a MAKEOBJDIRPREFIX like option, and redefine every >>target’s DESTDIR ${makeobjDESTDIR} before running do-install. Now i’ve >>yet to complete this stage, but I believe this is the way to do it. > >Would it be sufficient to have an INSTALL_PREFIX and/or INSTALL_DESTDIR >so that DESTDIR can be different during install ? >[I was recently experimenting with something similar...] > >So how would that work? I was assuming a sub-make involved: install: ${MAKE} DESTDIR=${INSTALL_DESTDIR} realinstall Thank-you Simon, that seems to have done the trick. I'll finish up my 'initial' work and submit a patch! ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5998 - Release Date: 12/30/12 ----- No virus found in this message. Checked by AVG - www.avg.com Version: 2013.0.2805 / Virus Database: 2637/5998 - Release Date: 12/30/12