From owner-freebsd-arch@FreeBSD.ORG Sun Dec 9 15:13:04 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5D7B74B9; Sun, 9 Dec 2012 15:13:04 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 161B38FC08; Sun, 9 Dec 2012 15:13:03 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.17]) by ltcfislmsgpa03.fnfis.com (8.14.5/8.14.5) with ESMTP id qB9FD2ve028417 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 9 Dec 2012 09:13:02 -0600 Received: from [10.0.0.102] (10.14.152.61) by smtp.fisglobal.com (10.132.206.17) with Microsoft SMTP Server (TLS) id 14.2.309.2; Sun, 9 Dec 2012 09:13:01 -0600 Subject: Re: PR conf/121064 -- [patch] sys/boot/forth -- Use ASCII characters for box/line characters in frames.4th MIME-Version: 1.0 (Apple Message framework v1283) From: Devin Teske In-Reply-To: Date: Sun, 9 Dec 2012 07:12:59 -0800 Message-ID: <2599CA42-6700-4451-A67A-9610863D829F@fisglobal.com> References: To: X-Mailer: Apple Mail (2.1283) X-Originating-IP: [10.14.152.61] X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.9.8185, 1.0.431, 0.0.0000 definitions=2012-12-09_05:2012-12-07,2012-12-09,1970-01-01 signatures=0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Adrian Chadd , Devin Teske , Garrett Cooper X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Dec 2012 15:13:04 -0000 Appears to be no objections. As I move forward with reviewed/tested commit, would like to document what = I tested (for posterity): 1. boot_serial=3D" " 2. boot_serial=3D"abc" 3. boot_serial=3D"YES" All produce the new serial-compatible frames. 4. console=3D"comconsole" 5. console=3D"vidconsole comconsole" 6. console=3D" vidconsole comconsole2 comconsole " Also produce the new serial-compatible frames. 7. console=3D"vidconsole comconsole2" 8. console=3D"spinconsole" 9. console=3D"vidconsole" 10. boot_serial=3D"" 11. boot_multicons=3D"" Produce the old serial UN-friendly frames (as expected) 12. boot_multicons=3D"1" 13. boot_multicons=3D"YES" Both of which produce serial friendly frames. In-addition, I also unit-tested the "contains?" function for several hours,= making sure it works as-expected. --=20 Devin On Dec 7, 2012, at 4:43 PM, Devin Teske wrote: > Hi -arch, >=20 > Eitan brought PR conf/121064 to my attention. >=20 > This particular PR appears to be ~5 years old and addresses an ~10 year o= ld issue introduced by SVN r115410 >=20 > Below is the commit message and patch that I'm proposing to commit for th= is PR (also, I amended said patch to the PR audit trail; see conf/121064). >=20 > I asked gcooper to review the working/tested patch. >=20 > Here's a before/after shot (courtesy of jh) -- against 9.0-R: >=20 > http://twitpic.com/bjxfcz >=20 > Please let me know if you have any objections, feature-requests, nagging = feelings, etc. > --=20 > Cheers, > Devin >=20 > =3D=3D=3D >=20 > Use ASCII characters for box/line characters in frames.4th >=20 > Committed with changes to support the following from loader.conf(5): > + console=3D"vidconsole comconsole" (not just console=3D"comconsole") > + boot_serial=3D"anything" (not just boot_serial=3D"YES") > + boot_multicons=3D"anything" (unsupported in originally-submitted patch) >=20 > PR: conf/121064 > Submitted by: koitsu > Reviewed by: gcooper, adrian (co-mentor) [pending your review] > Approved by: adrian (co-mentor) [pending your approval] > Obtained from: > MFC after: > Security: > --This line, and those below, will be ignored-- >> Description of fields to fill in above: 76 columns -= -| >> PR: If a GNATS PR is affected by the change. >> Submitted by: If someone else sent in the change. >> Reviewed by: If someone else reviewed your modification. >> Approved by: If you needed approval for this commit. >> Obtained from: If the change is from a third party. >> MFC after: N [day[s]|week[s]|month[s]]. Request a reminder email. >> Security: Vulnerability reference (one per line) or description. >> Empty fields above will be automatically removed. >=20 > M forth/support.4th > M forth/frames.4th >=20 > _____________ > The information contained in this message is proprietary and/or confident= ial. If you are not the intended recipient, please: (i) delete the message = and all copies; (ii) do not disclose, distribute or use the message in any = manner; and (iii) notify the sender immediately. In addition, please be awa= re that any message addressed to our domain is subject to archiving and rev= iew by persons other than the intended recipient. Thank you. > _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-arch@FreeBSD.ORG Mon Dec 10 08:51:20 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0C82E603; Mon, 10 Dec 2012 08:51:20 +0000 (UTC) (envelope-from des@des.no) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id AB1938FC08; Mon, 10 Dec 2012 08:51:19 +0000 (UTC) Received: from ds4.des.no (smtp.des.no [194.63.250.102]) by smtp.des.no (Postfix) with ESMTP id C1615661B; Mon, 10 Dec 2012 09:51:18 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 8C371A6E1; Mon, 10 Dec 2012 09:51:18 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Tim Kientzle Subject: Re: svn commit: r243594 - head/sys/netinet References: <201211270304.qAR34PfD056691@svn.freebsd.org> <50B53ABC.1010304@freebsd.org> <50B57F46.1060207@mu.org> <50C205DC.1040701@freebsd.org> <50C23B5E.3020509@freebsd.org> <50C26BF9.1050106@mu.org> <50C278B0.3040107@mu.org> <52564974-563C-499F-9860-ADACA0DD22CE@bsdimp.com> <50C2839B.30709@mu.org> <50C3A8E4.3000401@freebsd.org> <2305DE0C-D8D0-49F0-8D69-B5091AA9D2CD@kientzle.com> Date: Mon, 10 Dec 2012 09:51:18 +0100 In-Reply-To: <2305DE0C-D8D0-49F0-8D69-B5091AA9D2CD@kientzle.com> (Tim Kientzle's message of "Sat, 8 Dec 2012 13:49:24 -0800") Message-ID: <86k3sq9to9.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.4 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Adrian Chadd , Alfred Perlstein , Andre Oppermann , Alfred Perlstein , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Dec 2012 08:51:20 -0000 Tim Kientzle writes: > Andre Oppermann writes: > > Adrian Chadd writes: > > > [...] I wonder what platform Oleksandr is using where an ARM board > > > has 1GB of RAM. > > Your favorite smartphone likely has 512MB-1GB of RAM. > PandaBoard has 1GB RAM. RaspberryPi has 512MB. Not to mention DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-arch@FreeBSD.ORG Tue Dec 11 20:29:47 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D0770B3A for ; Tue, 11 Dec 2012 20:29:47 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id 240098FC12 for ; Tue, 11 Dec 2012 20:29:25 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id qBBKTP3I040966 for ; Tue, 11 Dec 2012 14:29:25 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id qBBKTP1t040965 for freebsd-arch@freebsd.org; Tue, 11 Dec 2012 14:29:25 -0600 (CST) (envelope-from brooks) Date: Tue, 11 Dec 2012 14:29:25 -0600 From: Brooks Davis To: freebsd-arch@freebsd.org Subject: [CFT] Importing NetBSD's vis/unvis(3) Message-ID: <20121211202925.GA40927@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XsQoSWH+UP9D9v3l" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Dec 2012 20:29:47 -0000 --XsQoSWH+UP9D9v3l Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable As part of importing NetBSD's mtree I need to add more vis(3) API functions from NetBSD. The easiest path seems to be a wholesale import of their code with the addition of VIS_GLOB support for compatibility. The attached patch accomplishes this. Please review or test. The ABI of unvis changes slightly so I've added a compatibility shim for it. Note that old files must be removed in addition to applying the patch so Make finds the right files. Thanks, Brooks http://people.freebsd.org/~brooks/patches/vis.diff rm lib/libc/gen/unvis.c \ lib/libc/gen/vis.3 \ lib/libc/gen/vis.c \ lib/libc/gen/unvis.3 \ include/vis.h Index: lib/libc/gen/Makefile.inc =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- lib/libc/gen/Makefile.inc (revision 244091) +++ lib/libc/gen/Makefile.inc (working copy) @@ -32,13 +32,16 @@ sigsetops.c sleep.c srand48.c statvfs.c stringlist.c strtofflags.c \ sysconf.c sysctl.c sysctlbyname.c sysctlnametomib.c \ syslog.c telldir.c termios.c time.c times.c timezone.c tls.c \ - ttyname.c ttyslot.c ualarm.c ulimit.c uname.c unvis.c \ - usleep.c utime.c utxdb.c valloc.c vis.c wait.c wait3.c waitpid.c \ + ttyname.c ttyslot.c ualarm.c ulimit.c uname.c unvis-compat.c \ + usleep.c utime.c utxdb.c valloc.c wait.c wait3.c waitpid.c \ waitid.c wordexp.c =20 .PATH: ${.CURDIR}/../../contrib/libc-pwcache SRCS+=3D pwcache.c pwcache.h =20 +.PATH: ${.CURDIR}/../../contrib/libc-vis +SRCS+=3D unvis.c vis.c + MISRCS+=3Dmodf.c =20 CANCELPOINTS_SRCS=3Dsem.c sem_new.c Index: lib/libc/gen/unvis-compat.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- lib/libc/gen/unvis-compat.c (revision 0) +++ lib/libc/gen/unvis-compat.c (working copy) @@ -0,0 +1,46 @@ +/*- + * Copyright (c) 2012 SRI International + * All rights reserved. + * + * This software was developed by SRI International and the University of + * Cambridge Computer Laboratory under DARPA/AFRL contract (FA8750-10-C-02= 37) + * ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURP= OSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENT= IAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STR= ICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY W= AY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include + +#define _UNVIS_END 1 + +int +__unvis_44bsd(char *cp, int c, int *astate, int flag) +{ + + if (flag & _UNVIS_END) + flag =3D (flag & ~_UNVIS_END) ^ UNVIS_END; + return unvis(cp, c, astate, flag); +} + +__sym_compat(unvis, __vis_44bsd, FBSD_1.0); Property changes on: lib/libc/gen/unvis-compat.c ___________________________________________________________________ Added: svn:mime-type ## -0,0 +1 ## +text/plain Added: svn:keywords ## -0,0 +1 ## +FreeBSD=3D%H Added: svn:eol-style ## -0,0 +1 ## +native Index: lib/libc/gen/Symbol.map =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- lib/libc/gen/Symbol.map (revision 244091) +++ lib/libc/gen/Symbol.map (working copy) @@ -298,7 +298,6 @@ ualarm; ulimit; uname; - unvis; strunvis; strunvisx; usleep; @@ -388,9 +387,21 @@ __FreeBSD_libc_enter_restricted_mode; getcontextx; gid_from_group; + nvis; pwcache_userdb; pwcache_groupdb; + snvis; + strnunvis; + strnunvisx; + strnvis; + strnvisx; + strsnvis; + strsnvisx; + strsvis; + strsvisx; + svis; uid_from_user; + unvis; waitid; }; =20 Index: include/Makefile =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- include/Makefile (revision 244091) +++ include/Makefile (working copy) @@ -23,9 +23,12 @@ stdnoreturn.h stdio.h stdlib.h string.h stringlist.h \ strings.h sysexits.h tar.h termios.h tgmath.h \ time.h timeconv.h timers.h ttyent.h \ - ulimit.h unistd.h utime.h utmpx.h uuid.h varargs.h vis.h \ + ulimit.h unistd.h utime.h utmpx.h uuid.h varargs.h \ wchar.h wctype.h wordexp.h xlocale.h =20 +.PATH: ${.CURDIR}/../contrib/libc-vis +INCS+=3D vis.h + MHDRS=3D float.h floatingpoint.h stdarg.h =20 PHDRS=3D sched.h _semaphore.h diff -ruN contrib/libc-vis/unvis.3 contrib/libc-vis/unvis.3 --- contrib/libc-vis/unvis.3 1969-12-31 18:00:00.000000000 -0600 +++ contrib/libc-vis/unvis.3 2012-10-20 09:09:04.000000000 -0500 @@ -0,0 +1,247 @@ +.\" $NetBSD: unvis.3,v 1.23 2011/03/17 14:06:29 wiz Exp $ +.\" $FreeBSD$ +.\" +.\" Copyright (c) 1989, 1991, 1993 +.\" The Regents of the University of California. All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" 3. Neither the name of the University nor the names of its contributors +.\" may be used to endorse or promote products derived from this softwa= re +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PUR= POSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIAB= LE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUEN= TIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, ST= RICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY = WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" @(#)unvis.3 8.2 (Berkeley) 12/11/93 +.\" +.Dd October 20, 2012 +.Dt UNVIS 3 +.Os +.Sh NAME +.Nm unvis , +.Nm strunvis +.Nd decode a visual representation of characters +.Sh LIBRARY +.Lb libc +.Sh SYNOPSIS +.In vis.h +.Ft int +.Fn unvis "char *cp" "int c" "int *astate" "int flag" +.Ft int +.Fn strunvis "char *dst" "const char *src" +.Ft int +.Fn strnunvis "char *dst" "size_t dlen" "const char *src" +.Ft int +.Fn strunvisx "char *dst" "const char *src" "int flag" +.Ft int +.Fn strnunvisx "char *dst" "size_t dlen" "const char *src" "int flag" +.Sh DESCRIPTION +The +.Fn unvis , +.Fn strunvis +and +.Fn strunvisx +functions +are used to decode a visual representation of characters, as produced +by the +.Xr vis 3 +function, back into +the original form. +.Pp +The +.Fn unvis +function is called with successive characters in +.Ar c +until a valid sequence is recognized, at which time the decoded +character is available at the character pointed to by +.Ar cp . +.Pp +The +.Fn strunvis +function decodes the characters pointed to by +.Ar src +into the buffer pointed to by +.Ar dst . +The +.Fn strunvis +function simply copies +.Ar src +to +.Ar dst , +decoding any escape sequences along the way, +and returns the number of characters placed into +.Ar dst , +or \-1 if an +invalid escape sequence was detected. +The size of +.Ar dst +should be equal to the size of +.Ar src +(that is, no expansion takes place during decoding). +.Pp +The +.Fn strunvisx +function does the same as the +.Fn strunvis +function, +but it allows you to add a flag that specifies the style the string +.Ar src +is encoded with. +Currently, the supported flags are: +.Dv VIS_HTTPSTYLE +and +.Dv VIS_MIMESTYLE . +.Pp +The +.Fn unvis +function implements a state machine that can be used to decode an +arbitrary stream of bytes. +All state associated with the bytes being decoded is stored outside the +.Fn unvis +function (that is, a pointer to the state is passed in), so +calls decoding different streams can be freely intermixed. +To start decoding a stream of bytes, first initialize an integer to zero. +Call +.Fn unvis +with each successive byte, along with a pointer +to this integer, and a pointer to a destination character. +The +.Fn unvis +function has several return codes that must be handled properly. +They are: +.Bl -tag -width UNVIS_VALIDPUSH +.It Li \&0 (zero) +Another character is necessary; nothing has been recognized yet. +.It Dv UNVIS_VALID +A valid character has been recognized and is available at the location +pointed to by cp. +.It Dv UNVIS_VALIDPUSH +A valid character has been recognized and is available at the location +pointed to by cp; however, the character currently passed in should +be passed in again. +.It Dv UNVIS_NOCHAR +A valid sequence was detected, but no character was produced. +This return code is necessary to indicate a logical break between characte= rs. +.It Dv UNVIS_SYNBAD +An invalid escape sequence was detected, or the decoder is in an unknown s= tate. +The decoder is placed into the starting state. +.El +.Pp +When all bytes in the stream have been processed, call +.Fn unvis +one more time with flag set to +.Dv UNVIS_END +to extract any remaining character (the character passed in is ignored). +.Pp +The +.Ar flag +argument is also used to specify the encoding style of the source. +If set to +.Dv VIS_HTTPSTYLE +or +.Dv VIS_HTTP1808 , +.Fn unvis +will decode URI strings as specified in RFC 1808. +If set to +.Dv VIS_HTTP1866 , +.Fn unvis +will decode URI strings as specified in RFC 1866. +If set to +.Dv VIS_MIMESTYLE , +.Fn unvis +will decode MIME Quoted-Printable strings as specified in RFC 2045. +If set to +.Dv VIS_NOESCAPE , +.Fn unvis +will not decode \e quoted characters. +.Pp +The following code fragment illustrates a proper use of +.Fn unvis . +.Bd -literal -offset indent +int state =3D 0; +char out; + +while ((ch =3D getchar()) !=3D EOF) { +again: + switch(unvis(\*[Am]out, ch, \*[Am]state, 0)) { + case 0: + case UNVIS_NOCHAR: + break; + case UNVIS_VALID: + (void)putchar(out); + break; + case UNVIS_VALIDPUSH: + (void)putchar(out); + goto again; + case UNVIS_SYNBAD: + errx(EXIT_FAILURE, "Bad character sequence!"); + } +} +if (unvis(\*[Am]out, '\e0', \*[Am]state, UNVIS_END) =3D=3D UNVIS_VALID) + (void)putchar(out); +.Ed +.Sh ERRORS +The functions +.Fn strunvis , +.Fn strnunvis , +.Fn strunvisx , +and +.Fn strnunvisx +will return \-1 on error and set +.Va errno=20 +to: +.Bl -tag -width Er +.It Bq Er EINVAL +An invalid escape sequence was detected, or the decoder is in an unknown s= tate. +.El +.Pp +In addition the functions +.Fn strnunvis=20 +and +.Fn strnunvisx +will can also set +.Va errno +on error to: +.Bl -tag -width Er +.It Bq Er ENOSPC +Not enough space to perform the conversion. +.El +.Sh SEE ALSO +.Xr unvis 1 , +.Xr vis 1 , +.Xr vis 3 +.Rs +.%A R. Fielding +.%T Relative Uniform Resource Locators +.%O RFC1808 +.Re +.Sh HISTORY +The +.Fn unvis +function +first appeared in +.Bx 4.4 . +The +.Fn strnunvis +and +.Fn strnunvisx +functions appeared in +.Nx 6.0 +and +.Fx 10.0 . diff -ruN contrib/libc-vis/unvis.c contrib/libc-vis/unvis.c --- contrib/libc-vis/unvis.c 1969-12-31 18:00:00.000000000 -0600 +++ contrib/libc-vis/unvis.c 2012-10-20 09:22:09.000000000 -0500 @@ -0,0 +1,562 @@ +/* $NetBSD: unvis.c,v 1.39 2012/03/13 21:13:37 christos Exp $ */ + +/*- + * Copyright (c) 1989, 1993 + * The Regents of the University of California. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURP= OSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENT= IAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STR= ICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY W= AY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +#if defined(LIBC_SCCS) && !defined(lint) +#if 0 +static char sccsid[] =3D "@(#)unvis.c 8.1 (Berkeley) 6/4/93"; +#else +__RCSID("$NetBSD: unvis.c,v 1.39 2012/03/13 21:13:37 christos Exp $"); +#endif +#endif /* LIBC_SCCS and not lint */ +__FBSDID("$FreeBSD$"); + +#include "namespace.h" +#include + +#include +#include +#include +#include +#include +#include + +#define _DIAGASSERT(x) assert(x) + +/* + * Return the number of elements in a statically-allocated array, + * __x. + */ +#define __arraycount(__x) (sizeof(__x) / sizeof(__x[0])) + +#ifdef __weak_alias +__weak_alias(strnunvisx,_strnunvisx) +#endif + +#if !HAVE_VIS +/* + * decode driven by state machine + */ +#define S_GROUND 0 /* haven't seen escape char */ +#define S_START 1 /* start decoding special sequence */ +#define S_META 2 /* metachar started (M) */ +#define S_META1 3 /* metachar more, regular char (-) */ +#define S_CTRL 4 /* control char started (^) */ +#define S_OCTAL2 5 /* octal digit 2 */ +#define S_OCTAL3 6 /* octal digit 3 */ +#define S_HEX 7 /* mandatory hex digit */ +#define S_HEX1 8 /* http hex digit */ +#define S_HEX2 9 /* http hex digit 2 */ +#define S_MIME1 10 /* mime hex digit 1 */ +#define S_MIME2 11 /* mime hex digit 2 */ +#define S_EATCRNL 12 /* mime eating CRNL */ +#define S_AMP 13 /* seen & */ +#define S_NUMBER 14 /* collecting number */ +#define S_STRING 15 /* collecting string */ + +#define isoctal(c) (((u_char)(c)) >=3D '0' && ((u_char)(c)) <=3D '7') +#define xtod(c) (isdigit(c) ? (c - '0') : ((tolower(c) - 'a') + 10)) +#define XTOD(c) (isdigit(c) ? (c - '0') : ((c - 'A') + 10)) + +/* + * RFC 1866 + */ +static const struct nv { + const char *name; + uint8_t value; +} nv[] =3D { + { "AElig", 198 }, /* capital AE diphthong (ligature) */ + { "Aacute", 193 }, /* capital A, acute accent */ + { "Acirc", 194 }, /* capital A, circumflex accent */ + { "Agrave", 192 }, /* capital A, grave accent */ + { "Aring", 197 }, /* capital A, ring */ + { "Atilde", 195 }, /* capital A, tilde */ + { "Auml", 196 }, /* capital A, dieresis or umlaut mark */ + { "Ccedil", 199 }, /* capital C, cedilla */ + { "ETH", 208 }, /* capital Eth, Icelandic */ + { "Eacute", 201 }, /* capital E, acute accent */ + { "Ecirc", 202 }, /* capital E, circumflex accent */ + { "Egrave", 200 }, /* capital E, grave accent */ + { "Euml", 203 }, /* capital E, dieresis or umlaut mark */ + { "Iacute", 205 }, /* capital I, acute accent */ + { "Icirc", 206 }, /* capital I, circumflex accent */ + { "Igrave", 204 }, /* capital I, grave accent */ + { "Iuml", 207 }, /* capital I, dieresis or umlaut mark */ + { "Ntilde", 209 }, /* capital N, tilde */ + { "Oacute", 211 }, /* capital O, acute accent */ + { "Ocirc", 212 }, /* capital O, circumflex accent */ + { "Ograve", 210 }, /* capital O, grave accent */ + { "Oslash", 216 }, /* capital O, slash */ + { "Otilde", 213 }, /* capital O, tilde */ + { "Ouml", 214 }, /* capital O, dieresis or umlaut mark */ + { "THORN", 222 }, /* capital THORN, Icelandic */ + { "Uacute", 218 }, /* capital U, acute accent */ + { "Ucirc", 219 }, /* capital U, circumflex accent */ + { "Ugrave", 217 }, /* capital U, grave accent */ + { "Uuml", 220 }, /* capital U, dieresis or umlaut mark */ + { "Yacute", 221 }, /* capital Y, acute accent */ + { "aacute", 225 }, /* small a, acute accent */ + { "acirc", 226 }, /* small a, circumflex accent */ + { "acute", 180 }, /* acute accent */ + { "aelig", 230 }, /* small ae diphthong (ligature) */ + { "agrave", 224 }, /* small a, grave accent */ + { "amp", 38 }, /* ampersand */ + { "aring", 229 }, /* small a, ring */ + { "atilde", 227 }, /* small a, tilde */ + { "auml", 228 }, /* small a, dieresis or umlaut mark */ + { "brvbar", 166 }, /* broken (vertical) bar */ + { "ccedil", 231 }, /* small c, cedilla */ + { "cedil", 184 }, /* cedilla */ + { "cent", 162 }, /* cent sign */ + { "copy", 169 }, /* copyright sign */ + { "curren", 164 }, /* general currency sign */ + { "deg", 176 }, /* degree sign */ + { "divide", 247 }, /* divide sign */ + { "eacute", 233 }, /* small e, acute accent */ + { "ecirc", 234 }, /* small e, circumflex accent */ + { "egrave", 232 }, /* small e, grave accent */ + { "eth", 240 }, /* small eth, Icelandic */ + { "euml", 235 }, /* small e, dieresis or umlaut mark */ + { "frac12", 189 }, /* fraction one-half */ + { "frac14", 188 }, /* fraction one-quarter */ + { "frac34", 190 }, /* fraction three-quarters */ + { "gt", 62 }, /* greater than */ + { "iacute", 237 }, /* small i, acute accent */ + { "icirc", 238 }, /* small i, circumflex accent */ + { "iexcl", 161 }, /* inverted exclamation mark */ + { "igrave", 236 }, /* small i, grave accent */ + { "iquest", 191 }, /* inverted question mark */ + { "iuml", 239 }, /* small i, dieresis or umlaut mark */ + { "laquo", 171 }, /* angle quotation mark, left */ + { "lt", 60 }, /* less than */ + { "macr", 175 }, /* macron */ + { "micro", 181 }, /* micro sign */ + { "middot", 183 }, /* middle dot */ + { "nbsp", 160 }, /* no-break space */ + { "not", 172 }, /* not sign */ + { "ntilde", 241 }, /* small n, tilde */ + { "oacute", 243 }, /* small o, acute accent */ + { "ocirc", 244 }, /* small o, circumflex accent */ + { "ograve", 242 }, /* small o, grave accent */ + { "ordf", 170 }, /* ordinal indicator, feminine */ + { "ordm", 186 }, /* ordinal indicator, masculine */ + { "oslash", 248 }, /* small o, slash */ + { "otilde", 245 }, /* small o, tilde */ + { "ouml", 246 }, /* small o, dieresis or umlaut mark */ + { "para", 182 }, /* pilcrow (paragraph sign) */ + { "plusmn", 177 }, /* plus-or-minus sign */ + { "pound", 163 }, /* pound sterling sign */ + { "quot", 34 }, /* double quote */ + { "raquo", 187 }, /* angle quotation mark, right */ + { "reg", 174 }, /* registered sign */ + { "sect", 167 }, /* section sign */ + { "shy", 173 }, /* soft hyphen */ + { "sup1", 185 }, /* superscript one */ + { "sup2", 178 }, /* superscript two */ + { "sup3", 179 }, /* superscript three */ + { "szlig", 223 }, /* small sharp s, German (sz ligature) */ + { "thorn", 254 }, /* small thorn, Icelandic */ + { "times", 215 }, /* multiply sign */ + { "uacute", 250 }, /* small u, acute accent */ + { "ucirc", 251 }, /* small u, circumflex accent */ + { "ugrave", 249 }, /* small u, grave accent */ + { "uml", 168 }, /* umlaut (dieresis) */ + { "uuml", 252 }, /* small u, dieresis or umlaut mark */ + { "yacute", 253 }, /* small y, acute accent */ + { "yen", 165 }, /* yen sign */ + { "yuml", 255 }, /* small y, dieresis or umlaut mark */ +}; + +/* + * unvis - decode characters previously encoded by vis + */ +int +unvis(char *cp, int c, int *astate, int flag) +{ + unsigned char uc =3D (unsigned char)c; + unsigned char st, ia, is, lc; + +/* + * Bottom 8 bits of astate hold the state machine state. + * Top 8 bits hold the current character in the http 1866 nv string decodi= ng + */ +#define GS(a) ((a) & 0xff) +#define SS(a, b) (((uint32_t)(a) << 24) | (b)) +#define GI(a) ((uint32_t)(a) >> 24) + + _DIAGASSERT(cp !=3D NULL); + _DIAGASSERT(astate !=3D NULL); + st =3D GS(*astate); + + if (flag & UNVIS_END) { + switch (st) { + case S_OCTAL2: + case S_OCTAL3: + case S_HEX2: + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case S_GROUND: + return UNVIS_NOCHAR; + default: + return UNVIS_SYNBAD; + } + } + + switch (st) { + + case S_GROUND: + *cp =3D 0; + if ((flag & VIS_NOESCAPE) =3D=3D 0 && c =3D=3D '\\') { + *astate =3D SS(0, S_START); + return UNVIS_NOCHAR; + } + if ((flag & VIS_HTTP1808) && c =3D=3D '%') { + *astate =3D SS(0, S_HEX1); + return UNVIS_NOCHAR; + } + if ((flag & VIS_HTTP1866) && c =3D=3D '&') { + *astate =3D SS(0, S_AMP); + return UNVIS_NOCHAR; + } + if ((flag & VIS_MIMESTYLE) && c =3D=3D '=3D') { + *astate =3D SS(0, S_MIME1); + return UNVIS_NOCHAR; + } + *cp =3D c; + return UNVIS_VALID; + + case S_START: + switch(c) { + case '\\': + *cp =3D c; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case '0': case '1': case '2': case '3': + case '4': case '5': case '6': case '7': + *cp =3D (c - '0'); + *astate =3D SS(0, S_OCTAL2); + return UNVIS_NOCHAR; + case 'M': + *cp =3D (char)0200; + *astate =3D SS(0, S_META); + return UNVIS_NOCHAR; + case '^': + *astate =3D SS(0, S_CTRL); + return UNVIS_NOCHAR; + case 'n': + *cp =3D '\n'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'r': + *cp =3D '\r'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'b': + *cp =3D '\b'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'a': + *cp =3D '\007'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'v': + *cp =3D '\v'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 't': + *cp =3D '\t'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'f': + *cp =3D '\f'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 's': + *cp =3D ' '; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'E': + *cp =3D '\033'; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + case 'x': + *astate =3D SS(0, S_HEX); + return UNVIS_NOCHAR; + case '\n': + /* + * hidden newline + */ + *astate =3D SS(0, S_GROUND); + return UNVIS_NOCHAR; + case '$': + /* + * hidden marker + */ + *astate =3D SS(0, S_GROUND); + return UNVIS_NOCHAR; + } + goto bad; + + case S_META: + if (c =3D=3D '-') + *astate =3D SS(0, S_META1); + else if (c =3D=3D '^') + *astate =3D SS(0, S_CTRL); + else=20 + goto bad; + return UNVIS_NOCHAR; + + case S_META1: + *astate =3D SS(0, S_GROUND); + *cp |=3D c; + return UNVIS_VALID; + + case S_CTRL: + if (c =3D=3D '?') + *cp |=3D 0177; + else + *cp |=3D c & 037; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + + case S_OCTAL2: /* second possible octal digit */ + if (isoctal(uc)) { + /* + * yes - and maybe a third + */ + *cp =3D (*cp << 3) + (c - '0'); + *astate =3D SS(0, S_OCTAL3); + return UNVIS_NOCHAR; + } + /* + * no - done with current sequence, push back passed char + */ + *astate =3D SS(0, S_GROUND); + return UNVIS_VALIDPUSH; + + case S_OCTAL3: /* third possible octal digit */ + *astate =3D SS(0, S_GROUND); + if (isoctal(uc)) { + *cp =3D (*cp << 3) + (c - '0'); + return UNVIS_VALID; + } + /* + * we were done, push back passed char + */ + return UNVIS_VALIDPUSH; + + case S_HEX: + if (!isxdigit(uc)) + goto bad; + /*FALLTHROUGH*/ + case S_HEX1: + if (isxdigit(uc)) { + *cp =3D xtod(uc); + *astate =3D SS(0, S_HEX2); + return UNVIS_NOCHAR; + } + /* + * no - done with current sequence, push back passed char + */ + *astate =3D SS(0, S_GROUND); + return UNVIS_VALIDPUSH; + + case S_HEX2: + *astate =3D S_GROUND; + if (isxdigit(uc)) { + *cp =3D xtod(uc) | (*cp << 4); + return UNVIS_VALID; + } + return UNVIS_VALIDPUSH; + + case S_MIME1: + if (uc =3D=3D '\n' || uc =3D=3D '\r') { + *astate =3D SS(0, S_EATCRNL); + return UNVIS_NOCHAR; + } + if (isxdigit(uc) && (isdigit(uc) || isupper(uc))) { + *cp =3D XTOD(uc); + *astate =3D SS(0, S_MIME2); + return UNVIS_NOCHAR; + } + goto bad; + + case S_MIME2: + if (isxdigit(uc) && (isdigit(uc) || isupper(uc))) { + *astate =3D SS(0, S_GROUND); + *cp =3D XTOD(uc) | (*cp << 4); + return UNVIS_VALID; + } + goto bad; + + case S_EATCRNL: + switch (uc) { + case '\r': + case '\n': + return UNVIS_NOCHAR; + case '=3D': + *astate =3D SS(0, S_MIME1); + return UNVIS_NOCHAR; + default: + *cp =3D uc; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + } + + case S_AMP: + *cp =3D 0; + if (uc =3D=3D '#') { + *astate =3D SS(0, S_NUMBER); + return UNVIS_NOCHAR; + } + *astate =3D SS(0, S_STRING); + /*FALLTHROUGH*/ + + case S_STRING: + ia =3D *cp; /* index in the array */ + is =3D GI(*astate); /* index in the string */ + lc =3D is =3D=3D 0 ? 0 : nv[ia].name[is - 1]; /* last character */ + + if (uc =3D=3D ';') + uc =3D '\0'; + + for (; ia < __arraycount(nv); ia++) { + if (is !=3D 0 && nv[ia].name[is - 1] !=3D lc) + goto bad; + if (nv[ia].name[is] =3D=3D uc) + break; + } + + if (ia =3D=3D __arraycount(nv)) + goto bad; + + if (uc !=3D 0) { + *cp =3D ia; + *astate =3D SS(is + 1, S_STRING); + return UNVIS_NOCHAR; + } + + *cp =3D nv[ia].value; + *astate =3D SS(0, S_GROUND); + return UNVIS_VALID; + + case S_NUMBER: + if (uc =3D=3D ';') + return UNVIS_VALID; + if (!isdigit(uc)) + goto bad; + *cp +=3D (*cp * 10) + uc - '0'; + return UNVIS_NOCHAR; + + default: + bad: + /* + * decoder in unknown state - (probably uninitialized) + */ + *astate =3D SS(0, S_GROUND); + return UNVIS_SYNBAD; + } +} + +/* + * strnunvisx - decode src into dst + * + * Number of chars decoded into dst is returned, -1 on error. + * Dst is null terminated. + */ + +int +strnunvisx(char *dst, size_t dlen, const char *src, int flag) +{ + char c; + char t =3D '\0', *start =3D dst; + int state =3D 0; + + _DIAGASSERT(src !=3D NULL); + _DIAGASSERT(dst !=3D NULL); +#define CHECKSPACE() \ + do { \ + if (dlen-- =3D=3D 0) { \ + errno =3D ENOSPC; \ + return -1; \ + } \ + } while (/*CONSTCOND*/0) + + while ((c =3D *src++) !=3D '\0') { + again: + switch (unvis(&t, c, &state, flag)) { + case UNVIS_VALID: + CHECKSPACE(); + *dst++ =3D t; + break; + case UNVIS_VALIDPUSH: + CHECKSPACE(); + *dst++ =3D t; + goto again; + case 0: + case UNVIS_NOCHAR: + break; + case UNVIS_SYNBAD: + errno =3D EINVAL; + return -1; + default: + _DIAGASSERT(/*CONSTCOND*/0); + errno =3D EINVAL; + return -1; + } + } + if (unvis(&t, c, &state, UNVIS_END) =3D=3D UNVIS_VALID) { + CHECKSPACE(); + *dst++ =3D t; + } + CHECKSPACE(); + *dst =3D '\0'; + return (int)(dst - start); +} + +int +strunvisx(char *dst, const char *src, int flag) +{ + return strnunvisx(dst, (size_t)~0, src, flag); +} + +int +strunvis(char *dst, const char *src) +{ + return strnunvisx(dst, (size_t)~0, src, 0); +} + +int +strnunvis(char *dst, size_t dlen, const char *src) +{ + return strnunvisx(dst, dlen, src, 0); +} +#endif diff -ruN contrib/libc-vis/vis.3 contrib/libc-vis/vis.3 --- contrib/libc-vis/vis.3 1969-12-31 18:00:00.000000000 -0600 +++ contrib/libc-vis/vis.3 2012-10-20 09:15:06.000000000 -0500 @@ -0,0 +1,456 @@ +.\" $NetBSD: vis.3,v 1.27 2011/05/17 07:10:39 joerg Exp $ +.\" $FreeBSD$ +.\" +.\" Copyright (c) 1989, 1991, 1993 +.\" The Regents of the University of California. All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" 3. Neither the name of the University nor the names of its contributors +.\" may be used to endorse or promote products derived from this softwa= re +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PUR= POSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIAB= LE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUEN= TIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, ST= RICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY = WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" @(#)vis.3 8.1 (Berkeley) 6/9/93 +.\" +.Dd October 20, 2012 +.Dt VIS 3 +.Os +.Sh NAME +.Nm vis , +.Nm nvis , +.Nm strvis , +.Nm strnvis , +.Nm strvisx , +.Nm strnvisx , +.Nm svis , +.Nm snvis , +.Nm strsvis , +.Nm strsnvis , +.Nm strsvisx +.Nm strsnvisx +.Nd visually encode characters +.Sh LIBRARY +.Lb libc +.Sh SYNOPSIS +.In vis.h +.Ft char * +.Fn vis "char *dst" "int c" "int flag" "int nextc" +.Ft char * +.Fn nvis "char *dst" "size_t dlen" "int c" "int flag" "int nextc" +.Ft int +.Fn strvis "char *dst" "const char *src" "int flag" +.Ft int +.Fn strnvis "char *dst" "size_t dlen" "const char *src" "int flag" +.Ft int +.Fn strvisx "char *dst" "const char *src" "size_t len" "int flag" +.Ft int +.Fn strnvisx "char *dst" "size_t dlen" "const char *src" "size_t len" "int= flag" +.Ft char * +.Fn svis "char *dst" "int c" "int flag" "int nextc" "const char *extra" +.Ft char * +.Fn snvis "char *dst" "size_t dlen" "int c" "int flag" "int nextc" "const = char *extra" +.Ft int +.Fn strsvis "char *dst" "const char *src" "int flag" "const char *extra" +.Ft int +.Fn strsnvis "char *dst" "size_t dlen" "const char *src" "int flag" "const= char *extra" +.Ft int +.Fn strsvisx "char *dst" "const char *src" "size_t len" "int flag" "const = char *extra" +.Ft int +.Fn strsnvisx "char *dst" "size_t dlen" "const char *src" "size_t len" "in= t flag" "const char *extra" +.Sh DESCRIPTION +The +.Fn vis +function +copies into +.Fa dst +a string which represents the character +.Fa c . +If +.Fa c +needs no encoding, it is copied in unaltered. +The string is null terminated, and a pointer to the end of the string is +returned. +The maximum length of any encoding is four +characters (not including the trailing +.Dv NUL ) ; +thus, when +encoding a set of characters into a buffer, the size of the buffer should +be four times the number of characters encoded, plus one for the trailing +.Dv NUL . +The flag parameter is used for altering the default range of +characters considered for encoding and for altering the visual +representation. +The additional character, +.Fa nextc , +is only used when selecting the +.Dv VIS_CSTYLE +encoding format (explained below). +.Pp +The +.Fn strvis , +.Fn strnvis , +.Fn strvisx , +and +.Fn strnvisx +functions copy into +.Fa dst +a visual representation of +the string +.Fa src . +The +.Fn strvis +and +.Fn strnvis +functions encode characters from +.Fa src +up to the +first +.Dv NUL . +The +.Fn strvisx +and +.Fn strnvisx +functions encode exactly +.Fa len +characters from +.Fa src +(this +is useful for encoding a block of data that may contain +.Dv NUL Ns 's ) . +Both forms +.Dv NUL +terminate +.Fa dst . +The size of +.Fa dst +must be four times the number +of characters encoded from +.Fa src +(plus one for the +.Dv NUL ) . +Both +forms return the number of characters in dst (not including +the trailing +.Dv NUL ) . +The +.Dq n +versions of the functions also take an additional argument +.Fa dlen +that indicates the length of the +.Fa dst +buffer. +If +.Fa dlen +is not large enough to fix the converted string then the +.Fn strnvis +and +.Fn strnvisx +functions return \-1 and set +.Va errno +to +.Dv ENOSPC . +.Pp +The functions +.Fn svis , +.Fn snvis , +.Fn strsvis , +.Fn strsnvis , +.Fn strsvisx , +and +.Fn strsnvisx +correspond to +.Fn vis , +.Fn nvis , +.Fn strvis , +.Fn strnvis , +.Fn strvisx , +and +.Fn strnvisx +but have an additional argument +.Fa extra , +pointing to a +.Dv NUL +terminated list of characters. +These characters will be copied encoded or backslash-escaped into +.Fa dst . +These functions are useful e.g. to remove the special meaning +of certain characters to shells. +.Pp +The encoding is a unique, invertible representation composed entirely of +graphic characters; it can be decoded back into the original form using +the +.Xr unvis 3 , +.Xr strunvis 3 +or +.Xr strnunvis 3 +functions. +.Pp +There are two parameters that can be controlled: the range of +characters that are encoded (applies only to +.Fn vis , +.Fn nvis , +.Fn strvis , +.Fn strnvis , +.Fn strvisx , +and +.Fn strnvisx ) , +and the type of representation used. +By default, all non-graphic characters, +except space, tab, and newline are encoded. +(See +.Xr isgraph 3 . ) +The following flags +alter this: +.Bl -tag -width VIS_WHITEX +.It Dv VIS_GLOB +Also encode magic characters +.Ql ( * , +.Ql \&? , +.Ql \&[ +and +.Ql # ) +recognized by +.Xr glob 3 . +.It Dv VIS_SP +Also encode space. +.It Dv VIS_TAB +Also encode tab. +.It Dv VIS_NL +Also encode newline. +.It Dv VIS_WHITE +Synonym for +.Dv VIS_SP +\&| +.Dv VIS_TAB +\&| +.Dv VIS_NL . +.It Dv VIS_SAFE +Only encode "unsafe" characters. +Unsafe means control characters which may cause common terminals to perform +unexpected functions. +Currently this form allows space, tab, newline, backspace, bell, and +return - in addition to all graphic characters - unencoded. +.El +.Pp +(The above flags have no effect for +.Fn svis , +.Fn snvis , +.Fn strsvis , +.Fn strsnvis , +.Fn strsvisx , +and +.Fn strsnvisx . +When using these functions, place all graphic characters to be +encoded in an array pointed to by +.Fa extra . +In general, the backslash character should be included in this array, see = the +warning on the use of the +.Dv VIS_NOSLASH +flag below). +.Pp +There are four forms of encoding. +All forms use the backslash character +.Ql \e +to introduce a special +sequence; two backslashes are used to represent a real backslash, +except +.Dv VIS_HTTPSTYLE +that uses +.Ql % , +or +.Dv VIS_MIMESTYLE +that uses +.Ql =3D . +These are the visual formats: +.Bl -tag -width VIS_CSTYLE +.It (default) +Use an +.Ql M +to represent meta characters (characters with the 8th +bit set), and use caret +.Ql ^ +to represent control characters see +.Pf ( Xr iscntrl 3 ) . +The following formats are used: +.Bl -tag -width xxxxx +.It Dv \e^C +Represents the control character +.Ql C . +Spans characters +.Ql \e000 +through +.Ql \e037 , +and +.Ql \e177 +(as +.Ql \e^? ) . +.It Dv \eM-C +Represents character +.Ql C +with the 8th bit set. +Spans characters +.Ql \e241 +through +.Ql \e376 . +.It Dv \eM^C +Represents control character +.Ql C +with the 8th bit set. +Spans characters +.Ql \e200 +through +.Ql \e237 , +and +.Ql \e377 +(as +.Ql \eM^? ) . +.It Dv \e040 +Represents +.Tn ASCII +space. +.It Dv \e240 +Represents Meta-space. +.El +.Pp +.It Dv VIS_CSTYLE +Use C-style backslash sequences to represent standard non-printable +characters. +The following sequences are used to represent the indicated characters: +.Bd -unfilled -offset indent +.Li \ea Tn - BEL No (007) +.Li \eb Tn - BS No (010) +.Li \ef Tn - NP No (014) +.Li \en Tn - NL No (012) +.Li \er Tn - CR No (015) +.Li \es Tn - SP No (040) +.Li \et Tn - HT No (011) +.Li \ev Tn - VT No (013) +.Li \e0 Tn - NUL No (000) +.Ed +.Pp +When using this format, the nextc parameter is looked at to determine +if a +.Dv NUL +character can be encoded as +.Ql \e0 +instead of +.Ql \e000 . +If +.Fa nextc +is an octal digit, the latter representation is used to +avoid ambiguity. +.It Dv VIS_OCTAL +Use a three digit octal sequence. +The form is +.Ql \eddd +where +.Em d +represents an octal digit. +.It Dv VIS_HTTPSTYLE +Use URI encoding as described in RFC 1738. +The form is +.Ql %xx +where +.Em x +represents a lower case hexadecimal digit. +.It Dv VIS_MIMESTYLE +Use MIME Quoted-Printable encoding as described in RFC 2045, only don't +break lines and don't handle CRLF. +The form is: +.Ql %XX +where +.Em X +represents an upper case hexadecimal digit. +.El +.Pp +There is one additional flag, +.Dv VIS_NOSLASH , +which inhibits the +doubling of backslashes and the backslash before the default +format (that is, control characters are represented by +.Ql ^C +and +meta characters as +.Ql M-C ) . +With this flag set, the encoding is +ambiguous and non-invertible. +.Sh ERRORS +The functions +.Fn nvis +and +.Fn snvis +will return +.Dv NULL +and the functions +.Fn strnvis , +.Fn strnvisx , +.Fn strsnvis , +and +.Fn strsnvisx , +will return \-1 when the +.Fa dlen +destination buffer length size is not enough to perform the conversion whi= le +setting +.Va errno +to: +.Bl -tag -width Er +.It Bq Er ENOSPC +The destination buffer size is not large enough to perform the conversion. +.El +.Sh SEE ALSO +.Xr unvis 1 , +.Xr vis 1 , +.Xr glob 3 , +.Xr unvis 3 +.Rs +.%A T. Berners-Lee +.%T Uniform Resource Locators (URL) +.%O RFC1738 +.Re +.Sh HISTORY +The +.Fn vis , +.Fn strvis , +and +.Fa strvisx +functions first appeared in +.Bx 4.4 . +The +.Fn svis , +.Fn strsvis , +and +.Fn strsvisx +functions appeared in +.Nx 1.5 +and +.Fx 10.0 . +The buffer size limited versions of the functions +.Po Fn nvis , +.Fn strnvis , +.Fn strnvisx , +.Fn snvis , +.Fn strsnvis , +and +.Fn strsnvisx Pc +appeared in +.Nx 6.0 +and +.Fx 10.0 . diff -ruN contrib/libc-vis/vis.c contrib/libc-vis/vis.c --- contrib/libc-vis/vis.c 1969-12-31 18:00:00.000000000 -0600 +++ contrib/libc-vis/vis.c 2012-10-20 09:18:00.000000000 -0500 @@ -0,0 +1,583 @@ +/* $NetBSD: vis.c,v 1.44 2011/03/12 19:52:48 christos Exp $ */ + +/*- + * Copyright (c) 1989, 1993 + * The Regents of the University of California. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURP= OSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENT= IAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STR= ICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY W= AY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/*- + * Copyright (c) 1999, 2005 The NetBSD Foundation, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTO= RS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIM= ITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICU= LAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTO= RS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF = THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#if defined(LIBC_SCCS) && !defined(lint) +__RCSID("$NetBSD: vis.c,v 1.44 2011/03/12 19:52:48 christos Exp $"); +#endif /* LIBC_SCCS and not lint */ +__FBSDID("$FreeBSD$"); + +#include "namespace.h" +#include + +#include +#include +#include +#include + +#ifdef __weak_alias +__weak_alias(strvisx,_strvisx) +#endif + +#if !HAVE_VIS || !HAVE_SVIS +#include +#include +#include +#include + +#define _DIAGASSERT(x) assert(x) + +static char *do_svis(char *, size_t *, int, int, int, const char *); + +#undef BELL +#define BELL '\a' + +#define isoctal(c) (((u_char)(c)) >=3D '0' && ((u_char)(c)) <=3D '7') +#define iswhite(c) (c =3D=3D ' ' || c =3D=3D '\t' || c =3D=3D '\n') +#define issafe(c) (c =3D=3D '\b' || c =3D=3D BELL || c =3D=3D '\r') +#define xtoa(c) "0123456789abcdef"[c] +#define XTOA(c) "0123456789ABCDEF"[c] + +#define MAXEXTRAS 9 + +#define MAKEEXTRALIST(flag, extra, orig_str) \ +do { \ + const char *orig =3D orig_str; \ + const char *o =3D orig; \ + char *e; \ + while (*o++) \ + continue; \ + extra =3D malloc((size_t)((o - orig) + MAXEXTRAS)); \ + if (!extra) break; \ + for (o =3D orig, e =3D extra; (*e++ =3D *o++) !=3D '\0';) \ + continue; \ + e--; \ + if (flag & VIS_GLOB) { \ + *e++ =3D '*'; \ + *e++ =3D '?'; \ + *e++ =3D '['; \ + *e++ =3D '#'; \ + } \ + if (flag & VIS_SP) *e++ =3D ' '; \ + if (flag & VIS_TAB) *e++ =3D '\t'; \ + if (flag & VIS_NL) *e++ =3D '\n'; \ + if ((flag & VIS_NOSLASH) =3D=3D 0) *e++ =3D '\\'; \ + *e =3D '\0'; \ +} while (/*CONSTCOND*/0) + +/* + * This is do_hvis, for HTTP style (RFC 1808) + */ +static char * +do_hvis(char *dst, size_t *dlen, int c, int flag, int nextc, const char *e= xtra) +{ + + if ((isascii(c) && isalnum(c)) + /* safe */ + || c =3D=3D '$' || c =3D=3D '-' || c =3D=3D '_' || c =3D=3D '.' || c = =3D=3D '+' + /* extra */ + || c =3D=3D '!' || c =3D=3D '*' || c =3D=3D '\'' || c =3D=3D '(' || c= =3D=3D ')' + || c =3D=3D ',') { + dst =3D do_svis(dst, dlen, c, flag, nextc, extra); + } else { + if (dlen) { + if (*dlen < 3) + return NULL; + *dlen -=3D 3; + } + *dst++ =3D '%'; + *dst++ =3D xtoa(((unsigned int)c >> 4) & 0xf); + *dst++ =3D xtoa((unsigned int)c & 0xf); + } + + return dst; +} + +/* + * This is do_mvis, for Quoted-Printable MIME (RFC 2045) + * NB: No handling of long lines or CRLF. + */ +static char * +do_mvis(char *dst, size_t *dlen, int c, int flag, int nextc, const char *e= xtra) +{ + if ((c !=3D '\n') && + /* Space at the end of the line */ + ((isspace(c) && (nextc =3D=3D '\r' || nextc =3D=3D '\n')) || + /* Out of range */ + (!isspace(c) && (c < 33 || (c > 60 && c < 62) || c > 126)) || + /* Specific char to be escaped */=20 + strchr("#$@[\\]^`{|}~", c) !=3D NULL)) { + if (dlen) { + if (*dlen < 3) + return NULL; + *dlen -=3D 3; + } + *dst++ =3D '=3D'; + *dst++ =3D XTOA(((unsigned int)c >> 4) & 0xf); + *dst++ =3D XTOA((unsigned int)c & 0xf); + } else { + dst =3D do_svis(dst, dlen, c, flag, nextc, extra); + } + return dst; +} + +/* + * This is do_vis, the central code of vis. + * dst: Pointer to the destination buffer + * c: Character to encode + * flag: Flag word + * nextc: The character following 'c' + * extra: Pointer to the list of extra characters to be + * backslash-protected. + */ +static char * +do_svis(char *dst, size_t *dlen, int c, int flag, int nextc, const char *e= xtra) +{ + int isextra; + size_t odlen =3D dlen ? *dlen : 0; + + isextra =3D strchr(extra, c) !=3D NULL; +#define HAVE(x) \ + do { \ + if (dlen) { \ + if (*dlen < (x)) \ + goto out; \ + *dlen -=3D (x); \ + } \ + } while (/*CONSTCOND*/0) + if (!isextra && isascii(c) && (isgraph(c) || iswhite(c) || + ((flag & VIS_SAFE) && issafe(c)))) { + HAVE(1); + *dst++ =3D c; + return dst; + } + if (flag & VIS_CSTYLE) { + HAVE(2); + switch (c) { + case '\n': + *dst++ =3D '\\'; *dst++ =3D 'n'; + return dst; + case '\r': + *dst++ =3D '\\'; *dst++ =3D 'r'; + return dst; + case '\b': + *dst++ =3D '\\'; *dst++ =3D 'b'; + return dst; + case BELL: + *dst++ =3D '\\'; *dst++ =3D 'a'; + return dst; + case '\v': + *dst++ =3D '\\'; *dst++ =3D 'v'; + return dst; + case '\t': + *dst++ =3D '\\'; *dst++ =3D 't'; + return dst; + case '\f': + *dst++ =3D '\\'; *dst++ =3D 'f'; + return dst; + case ' ': + *dst++ =3D '\\'; *dst++ =3D 's'; + return dst; + case '\0': + *dst++ =3D '\\'; *dst++ =3D '0'; + if (isoctal(nextc)) { + HAVE(2); + *dst++ =3D '0'; + *dst++ =3D '0'; + } + return dst; + default: + if (isgraph(c)) { + *dst++ =3D '\\'; *dst++ =3D c; + return dst; + } + if (dlen) + *dlen =3D odlen; + } + } + if (isextra || ((c & 0177) =3D=3D ' ') || (flag & VIS_OCTAL)) { + HAVE(4); + *dst++ =3D '\\'; + *dst++ =3D (u_char)(((u_int32_t)(u_char)c >> 6) & 03) + '0'; + *dst++ =3D (u_char)(((u_int32_t)(u_char)c >> 3) & 07) + '0'; + *dst++ =3D (c & 07) + '0'; + } else { + if ((flag & VIS_NOSLASH) =3D=3D 0) { + HAVE(1); + *dst++ =3D '\\'; + } + + if (c & 0200) { + HAVE(1); + c &=3D 0177; *dst++ =3D 'M'; + } + + if (iscntrl(c)) { + HAVE(2); + *dst++ =3D '^'; + if (c =3D=3D 0177) + *dst++ =3D '?'; + else + *dst++ =3D c + '@'; + } else { + HAVE(2); + *dst++ =3D '-'; *dst++ =3D c; + } + } + return dst; +out: + *dlen =3D odlen; + return NULL; +} + +typedef char *(*visfun_t)(char *, size_t *, int, int, int, const char *); + +/* + * Return the appropriate encoding function depending on the flags given. + */ +static visfun_t +getvisfun(int flag) +{ + if (flag & VIS_HTTPSTYLE) + return do_hvis; + if (flag & VIS_MIMESTYLE) + return do_mvis; + return do_svis; +} + +/* + * isnvis - visually encode characters, also encoding the characters + * pointed to by `extra' + */ +static char * +isnvis(char *dst, size_t *dlen, int c, int flag, int nextc, const char *ex= tra) +{ + char *nextra =3D NULL; + visfun_t f; + + _DIAGASSERT(dst !=3D NULL); + _DIAGASSERT(extra !=3D NULL); + MAKEEXTRALIST(flag, nextra, extra); + if (!nextra) { + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return NULL; + } + *dst =3D '\0'; /* can't create nextra, return "" */ + return dst; + } + f =3D getvisfun(flag); + dst =3D (*f)(dst, dlen, c, flag, nextc, nextra); + free(nextra); + if (dst =3D=3D NULL || (dlen && *dlen =3D=3D 0)) { + errno =3D ENOSPC; + return NULL; + } + *dst =3D '\0'; + return dst; +} + +char * +svis(char *dst, int c, int flag, int nextc, const char *extra) +{ + return isnvis(dst, NULL, c, flag, nextc, extra); +} + +char * +snvis(char *dst, size_t dlen, int c, int flag, int nextc, const char *extr= a) +{ + return isnvis(dst, &dlen, c, flag, nextc, extra); +} + + +/* + * strsvis, strsvisx - visually encode characters from src into dst + * + * Extra is a pointer to a \0-terminated list of characters to + * be encoded, too. These functions are useful e. g. to + * encode strings in such a way so that they are not interpreted + * by a shell. + * + * Dst must be 4 times the size of src to account for possible + * expansion. The length of dst, not including the trailing NULL, + * is returned. + * + * Strsvisx encodes exactly len bytes from src into dst. + * This is useful for encoding a block of data. + */ +static int +istrsnvis(char *dst, size_t *dlen, const char *csrc, int flag, const char = *extra) +{ + int c; + char *start; + char *nextra =3D NULL; + const unsigned char *src =3D (const unsigned char *)csrc; + visfun_t f; + + _DIAGASSERT(dst !=3D NULL); + _DIAGASSERT(src !=3D NULL); + _DIAGASSERT(extra !=3D NULL); + MAKEEXTRALIST(flag, nextra, extra); + if (!nextra) { + *dst =3D '\0'; /* can't create nextra, return "" */ + return 0; + } + f =3D getvisfun(flag); + for (start =3D dst; (c =3D *src++) !=3D '\0'; /* empty */) { + dst =3D (*f)(dst, dlen, c, flag, *src, nextra); + if (dst =3D=3D NULL) { + errno =3D ENOSPC; + return -1; + } + } + free(nextra); + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return -1; + } + *dst =3D '\0'; + return (int)(dst - start); +} + +int +strsvis(char *dst, const char *csrc, int flag, const char *extra) +{ + return istrsnvis(dst, NULL, csrc, flag, extra); +} + +int +strsnvis(char *dst, size_t dlen, const char *csrc, int flag, const char *e= xtra) +{ + return istrsnvis(dst, &dlen, csrc, flag, extra); +} + +static int +istrsnvisx(char *dst, size_t *dlen, const char *csrc, size_t len, int flag, + const char *extra) +{ + unsigned char c; + char *start; + char *nextra =3D NULL; + const unsigned char *src =3D (const unsigned char *)csrc; + visfun_t f; + + _DIAGASSERT(dst !=3D NULL); + _DIAGASSERT(src !=3D NULL); + _DIAGASSERT(extra !=3D NULL); + MAKEEXTRALIST(flag, nextra, extra); + if (! nextra) { + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return -1; + } + *dst =3D '\0'; /* can't create nextra, return "" */ + return 0; + } + + f =3D getvisfun(flag); + for (start =3D dst; len > 0; len--) { + c =3D *src++; + dst =3D (*f)(dst, dlen, c, flag, len > 1 ? *src : '\0', nextra); + if (dst =3D=3D NULL) { + errno =3D ENOSPC; + return -1; + } + } + free(nextra); + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return -1; + } + *dst =3D '\0'; + return (int)(dst - start); +} + +int +strsvisx(char *dst, const char *csrc, size_t len, int flag, const char *ex= tra) +{ + return istrsnvisx(dst, NULL, csrc, len, flag, extra); +} + +int +strsnvisx(char *dst, size_t dlen, const char *csrc, size_t len, int flag, + const char *extra) +{ + return istrsnvisx(dst, &dlen, csrc, len, flag, extra); +} +#endif + +#if !HAVE_VIS +/* + * vis - visually encode characters + */ +static char * +invis(char *dst, size_t *dlen, int c, int flag, int nextc) +{ + char *extra =3D NULL; + unsigned char uc =3D (unsigned char)c; + visfun_t f; + + _DIAGASSERT(dst !=3D NULL); + + MAKEEXTRALIST(flag, extra, ""); + if (! extra) { + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return NULL; + } + *dst =3D '\0'; /* can't create extra, return "" */ + return dst; + } + f =3D getvisfun(flag); + dst =3D (*f)(dst, dlen, uc, flag, nextc, extra); + free(extra); + if (dst =3D=3D NULL || (dlen && *dlen =3D=3D 0)) { + errno =3D ENOSPC; + return NULL; + } + *dst =3D '\0'; + return dst; +} + +char * +vis(char *dst, int c, int flag, int nextc) +{ + return invis(dst, NULL, c, flag, nextc); +} + +char * +nvis(char *dst, size_t dlen, int c, int flag, int nextc) +{ + return invis(dst, &dlen, c, flag, nextc); +} + + +/* + * strvis, strvisx - visually encode characters from src into dst + * + * Dst must be 4 times the size of src to account for possible + * expansion. The length of dst, not including the trailing NULL, + * is returned. + * + * Strvisx encodes exactly len bytes from src into dst. + * This is useful for encoding a block of data. + */ +static int +istrnvis(char *dst, size_t *dlen, const char *src, int flag) +{ + char *extra =3D NULL; + int rv; + + MAKEEXTRALIST(flag, extra, ""); + if (!extra) { + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return -1; + } + *dst =3D '\0'; /* can't create extra, return "" */ + return 0; + } + rv =3D istrsnvis(dst, dlen, src, flag, extra); + free(extra); + return rv; +} + +int +strvis(char *dst, const char *src, int flag) +{ + return istrnvis(dst, NULL, src, flag); +} + +int +strnvis(char *dst, size_t dlen, const char *src, int flag) +{ + return istrnvis(dst, &dlen, src, flag); +} + +static int +istrnvisx(char *dst, size_t *dlen, const char *src, size_t len, int flag) +{ + char *extra =3D NULL; + int rv; + + MAKEEXTRALIST(flag, extra, ""); + if (!extra) { + if (dlen && *dlen =3D=3D 0) { + errno =3D ENOSPC; + return -1; + } + *dst =3D '\0'; /* can't create extra, return "" */ + return 0; + } + rv =3D istrsnvisx(dst, dlen, src, len, flag, extra); + free(extra); + return rv; +} + +int +strvisx(char *dst, const char *src, size_t len, int flag) +{ + return istrnvisx(dst, NULL, src, len, flag); +} + +int +strnvisx(char *dst, size_t dlen, const char *src, size_t len, int flag) +{ + return istrnvisx(dst, &dlen, src, len, flag); +} + +#endif diff -ruN contrib/libc-vis/vis.h contrib/libc-vis/vis.h --- contrib/libc-vis/vis.h 1969-12-31 18:00:00.000000000 -0600 +++ contrib/libc-vis/vis.h 2012-10-20 09:20:17.000000000 -0500 @@ -0,0 +1,114 @@ +/* $NetBSD: vis.h,v 1.19 2011/03/12 19:52:45 christos Exp $ */ +/* $FreeBSD$ */ + +/*- + * Copyright (c) 1990, 1993 + * The Regents of the University of California. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURP= OSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENT= IAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STR= ICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY W= AY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)vis.h 8.1 (Berkeley) 6/2/93 + */ + +#ifndef _VIS_H_ +#define _VIS_H_ + +#include + +/* + * to select alternate encoding format + */ +#define VIS_OCTAL 0x001 /* use octal \ddd format */ +#define VIS_CSTYLE 0x002 /* use \[nrft0..] where appropiate */ + +/* + * to alter set of characters encoded (default is to encode all + * non-graphic except space, tab, and newline). + */ +#define VIS_SP 0x004 /* also encode space */ +#define VIS_TAB 0x008 /* also encode tab */ +#define VIS_NL 0x010 /* also encode newline */ +#define VIS_WHITE (VIS_SP | VIS_TAB | VIS_NL) +#define VIS_SAFE 0x020 /* only encode "unsafe" characters */ + +/* + * other + */ +#define VIS_NOSLASH 0x0040 /* inhibit printing '\' */ +#define VIS_HTTP1808 0x0080 /* http-style escape % hex hex */ +#define VIS_HTTPSTYLE 0x0080 /* http-style escape % hex hex */ +#define VIS_GLOB 0x0100 /* encode glob(3) magics */ +#define VIS_MIMESTYLE 0x0200 /* mime-style escape =3D HEX HEX */ +#define VIS_HTTP1866 0x0400 /* http-style &#num; or &string; */ +#define VIS_NOESCAPE 0x0800 /* don't decode `\' */ +#define _VIS_END 0x1000 /* for unvis */ + +/* + * unvis return codes + */ +#define UNVIS_VALID 1 /* character valid */ +#define UNVIS_VALIDPUSH 2 /* character valid, push back passed char */ +#define UNVIS_NOCHAR 3 /* valid sequence, no character produced */ +#define UNVIS_SYNBAD -1 /* unrecognized escape sequence */ +#define UNVIS_ERROR -2 /* decoder in unknown state (unrecoverable) */ + +/* + * unvis flags + */ +#define UNVIS_END _VIS_END /* no more characters */ + +#include + +__BEGIN_DECLS +char *vis(char *, int, int, int); +char *nvis(char *, size_t, int, int, int); + +char *svis(char *, int, int, int, const char *); +char *snvis(char *, size_t, int, int, int, const char *); + +int strvis(char *, const char *, int); +int strnvis(char *, size_t, const char *, int); + +int strsvis(char *, const char *, int, const char *); +int strsnvis(char *, size_t, const char *, int, const char *); + +int strvisx(char *, const char *, size_t, int); +int strnvisx(char *, size_t, const char *, size_t, int); + +int strsvisx(char *, const char *, size_t, int, const char *); +int strsnvisx(char *, size_t, const char *, size_t, int, const char *); + +int strunvis(char *, const char *); +int strnunvis(char *, size_t, const char *); + +int strunvisx(char *, const char *, int); +int strnunvisx(char *, size_t, const char *, int); + +#ifndef __LIBC12_SOURCE__ +int unvis(char *, int, int *, int); +#endif +__END_DECLS + +#endif /* !_VIS_H_ */ --XsQoSWH+UP9D9v3l Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQx5ekXY6L6fI4GtQRAv37AKCd4HDxx4DS8YkqWYuZGE2pA0sKNgCeKFEO 9nPfU793+g/zKLWkh+PE5ZI= =nMVb -----END PGP SIGNATURE----- --XsQoSWH+UP9D9v3l-- From owner-freebsd-arch@FreeBSD.ORG Thu Dec 13 23:12:49 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2BD92D13; Thu, 13 Dec 2012 23:12:49 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id B163C8FC08; Thu, 13 Dec 2012 23:12:48 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so3164676vba.13 for ; Thu, 13 Dec 2012 15:12:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type:content-transfer-encoding; bh=xQGObPxuA86pQu5j39tFpy4e/mQsJVE8nxbrZsItQGg=; b=nNTy79GvhA9M9wW+6vNcT/sdjsVHcm2kMjAK7MlFuTxFmZCuKomXMnil9ao+UMIeyQ +np99VymQus95d0zWRhQ2TuM6INKPJeAWADbFQEI0FLyx6ZhdP3Pm/lAr4v7q60LRY9c QiA31SHpjywwCOtKzaVuVIxSjPZ35+Oid7wK+uR0QDaMBULiKbH1sgXiP29Wt3tLA07b 5uTMiCZUWsDBTYmwO6XTw4ShhEQkgWLmEdTdG+K+hV5JUbgdla652RH3XcNa5tTE/Jam ZMaH44olf1re92kE01KLVSOC22wUybW/3I5rwkNDX5PJVhN+k2Uf+VX3QMugNhinQ6Ts 6t9A== MIME-Version: 1.0 Received: by 10.52.70.46 with SMTP id j14mr5334048vdu.99.1355440367354; Thu, 13 Dec 2012 15:12:47 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.245.130 with HTTP; Thu, 13 Dec 2012 15:12:47 -0800 (PST) Date: Fri, 14 Dec 2012 00:12:47 +0100 X-Google-Sender-Auth: DHruZbByEWtmhyCCkhKsl9v0H58 Message-ID: Subject: [RFC/RFT] calloutng From: Davide Italiano To: freebsd-current , freebsd-arch@freebsd.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Dec 2012 23:12:49 -0000 Hi. This patch takes callout(9) and redesign the KPI and the implementation. The main objective of this work is making the subsystem tickless. In the last several years, this possibility has been discussed widely (http://markmail.org/message/q3xmr2ttlzpqkmae), but until now noone really implemented that. If you want a complete history of what has been done in the last months you can check the calloutng project repository http://svnweb.freebsd.org/base/projects/calloutng/ For lazy people, here's a summary: 1) callout(9) is not anymore constrained to the resolution a periodic "hz" clock can give. In order to do that, the eventtimers(4) subsystem is used as backend. 2) Conversely from what discussed in past, we maintained the callwheel as underlying data structure for keeping track of the outstading timeouts. This choice has a couple of advantages, in particular we can still take benefits from the O(1) average complexity of the wheel for all the operations. Also, we thought the code duplication that would arise from the use of a two-staged backend for callout (e.g. use wheel for coarse resolution event and another data structure, such as an heap for high resolution events), is unacceptable. In fact, as long as callout gained the ability to migrate from a cpu to another having a double backend would mean doubling the code for the migration path. 3) A way to dispatch interrupts from hardware interrupt context has been implemented, using special callout flag. This has limited applicability, but avoid the dispatching of a SWI thread for handling specific callouts, avoiding the wake up of another CPU for processing and a (relatively useless) context switch 4) As long as new callout mechanism deals with bintime and not anymore with ticks, time is specified as absolute and not relative anymore. In order to get current time binuptime() or getbinuptime() is used, and a sysctl is introduced to selectively choose the function to use, based on a precision threshold. 5) A mechanism for specifying precision tolerance has been implemented. The callout processing mechanism has been adapted and the callout data structure augmented so that the codepath can take advantage and aggregate events which overlap in time. The new proposed KPI for callout is the following: callout_reset_bt_on(..., struct bintime time, struct bintime pr, ..., int f= lags) where =91time=92 argument represets the time at which the callout should fire, =91pr=92 represents the precision tolerance expressed as an absolute value, and =91flags=92, which could be used to specify new features, i.e. for now, the possibility to run the callout from fast interrupt context. The old KPI has been extended introducing the callout_reset_flags() function, which is the same of callout_reset*(), but takes an additional argument =91int flags=92 that can be used in the same fashion of the =91flags=92 argument for the new KPI. Using the =91flags=92 consumer= s can also specify relative precision tolerance in terms of power-of-two portion of the timeout passed as ticks. Using this strategy, the new precision mechanism can be used for the existing services without major modifications. Some consumers have been ported to the new KPI, in particular nanosleep(), poll(), select(), because they take immediate advantage from the arbitrary precision offered by the new infrastructure. For some statistics about the outcome of the conversion to the new service, please refer to the end of this e-mail: http://lists.freebsd.org/pipermail/freebsd-arch/2012-July/012756.html We didn't measure any significant performance regressions with hwmpc(4), using some benckmarks programs: http://people.freebsd.org/~davide/poll_test/poll_test.c http://people.freebsd.org/~mav/testsleep.c http://people.freebsd.org/~mav/testidle.c We tested the code on amd64, MIPS and arm. Any kind of testing or comment would be really appreciated. The full diff of the work against HEAD can be found at: http://people.freebsd.org/~davide/calloutng.diff If noone have objections, we plan to merge the repository to HEAD in a week or so. Thanks, Davide From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 06:41:15 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D08F1180; Fri, 14 Dec 2012 06:41:15 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 053D28FC08; Fri, 14 Dec 2012 06:41:14 +0000 (UTC) Received: by mail-ee0-f54.google.com with SMTP id c13so1716189eek.13 for ; Thu, 13 Dec 2012 22:41:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=/JEv1mLcg622mmBZ07HvWGdevo9oyH95hSzu69md+fM=; b=PpspHUbC72i4CAoVI0AfJBnkmXu/UclgpYLq5XPM3Nl1FL/AaWIe6gI3oOM0yAdsQT Wos++AARCh5F9dK+e4UXt0cai3lwgwvFcLT7gQfbKxktoBxrmzvSD2lzECi9t/FdZXKq C4bfc/B123HUIJR6TESnXg2gmAg4Cau9vde696CTFrt9QVM9HMdtX2XCica6k9ZB+d2U n30P6BPqmH8UHvczjxXwAAHRtbApT4y1gUrVgFv3nUfev8r+IFSr1pu9aTZNlgP5wCR5 wE4dUnmZrbbKpdHSL6oK1vdQTr2pKF5kfWlGStuJ2yiw4JsFABW1PdbULqXzQWOeZpuV 76kg== MIME-Version: 1.0 Received: by 10.14.209.193 with SMTP id s41mr12493191eeo.9.1355467267853; Thu, 13 Dec 2012 22:41:07 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.14.0.2 with HTTP; Thu, 13 Dec 2012 22:41:07 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Dec 2012 07:41:07 +0100 X-Google-Sender-Auth: avhdjzvO5ecoEtcfvzAV7u6e5DM Message-ID: Subject: Re: [RFC/RFT] calloutng From: Luigi Rizzo To: Davide Italiano Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 06:41:16 -0000 On Fri, Dec 14, 2012 at 12:12 AM, Davide Italiano wrote= : > Hi. > This patch takes callout(9) and redesign the KPI and the > implementation. The main objective of this work is making the > subsystem tickless. In the last several years, this possibility has > been discussed widely (http://markmail.org/message/q3xmr2ttlzpqkmae), > but until now noone really implemented that. > If you want a complete history of what has been done in the last > months you can check the calloutng project repository > http://svnweb.freebsd.org/base/projects/calloutng/ > For lazy people, here's a summary: > thanks for the work and the detailed summary. Perhaps it would be useful if you could provide a few high level details on the use and performance of the new scheme, such as: - is the old callout KPI still available ? (i am asking because it would help maintaining third party kernel modules that are expected to work on different FreeBSD releases) - do you have numbers on what is the fastest rate at which callouts can be fired (e.g. say you have a callout which increments a counter and schedules the next callout in (struct bintime){0,1} ) ? - is there a possibility that if callout requests are too close to each other (e.g. the above test) the thread dispatching callouts will run forever ? if so, is there a way to make such thread yield after a while ? - since you mentioned nanosleep() poll() and select() have been ported to the new callout, is there a way to guarantee that user using these functions with a very short timeout are actually descheduled as opposed to "interval too short, don't bother" ? - do you have numbers on how many calls per second we can have for a process that does for (;;) { nanosleep(min_value_that_causes_descheduling); I also have some comments on the diff: - can you provide a diff -p ? - for several functions the only change is the name of an argument from "busy" to "us". Can you elaborate the reason for the change, and whether "us" means microseconds or the pronoun ?) Finally, a more substantial comment: - a lot of functions which formerly had only a "timo" argument now have "timo, bt, precision, flags". Take seltdwait() as an example. It seems that you have been undecided between two approaches: for some of these functions you have preserved the original function that deals with ticks and introduced a new one that deals with the bintime, whereas in other cases you have modified the original function to add "bt, precision, flags". I would suggest a more uniform approach, namely: - preserve all the existing functions (T) that take a timeout in ticks; - add a new set of corresponding functions (BT) that take bt, precision, flags _instead_ of the ticks - the functions (T) make immediately the conversion from ticks to bintime(s), using macros or inline - optionally, convert kernel components to the new (BT) functions where this makes sense (e.g. we can exploit the finer-granularity of the new calls, etc.) cheers luigi 1) callout(9) is not anymore constrained to the resolution a periodic > "hz" clock can give. In order to do that, the eventtimers(4) subsystem > is used as backend. > 2) Conversely from what discussed in past, we maintained the callwheel > as underlying data structure for keeping track of the outstading > timeouts. This choice has a couple of advantages, in particular we can > still take benefits from the O(1) average complexity of the wheel for > all the operations. Also, we thought the code duplication that would > arise from the use of a two-staged backend for callout (e.g. use wheel > for coarse resolution event and another data structure, such as an > heap for high resolution events), is unacceptable. In fact, as long as > callout gained the ability to migrate from a cpu to another having a > double backend would mean doubling the code for the migration path. > 3) A way to dispatch interrupts from hardware interrupt context has > been implemented, using special callout flag. This has limited > applicability, but avoid the dispatching of a SWI thread for handling > specific callouts, avoiding the wake up of another CPU for processing > and a (relatively useless) context switch > 4) As long as new callout mechanism deals with bintime and not anymore > with ticks, time is specified as absolute and not relative anymore. In > order to get current time binuptime() or getbinuptime() is used, and a > sysctl is introduced to selectively choose the function to use, based > on a precision threshold. > 5) A mechanism for specifying precision tolerance has been > implemented. The callout processing mechanism has been adapted and the > callout data structure augmented so that the codepath can take > advantage and aggregate events which overlap in time. > > > The new proposed KPI for callout is the following: > callout_reset_bt_on(..., struct bintime time, struct bintime pr, ..., int > flags) > where =91time=92 argument represets the time at which the callout should > fire, =91pr=92 represents the precision tolerance expressed as an absolut= e > value, and =91flags=92, which could be used to specify new features, i.e. > for now, the possibility to run the callout from fast interrupt > context. > The old KPI has been extended introducing the callout_reset_flags() > function, which is the same of callout_reset*(), but takes an > additional argument =91int flags=92 that can be used in the same fashion > of the =91flags=92 argument for the new KPI. Using the =91flags=92 consum= ers > can also specify relative precision tolerance in terms of power-of-two > portion of the timeout passed as ticks. > Using this strategy, the new precision mechanism can be used for the > existing services without major modifications. > > Some consumers have been ported to the new KPI, in particular > nanosleep(), poll(), select(), because they take immediate advantage > from the arbitrary precision offered by the new infrastructure. > For some statistics about the outcome of the conversion to the new > service, please refer to the end of this e-mail: > http://lists.freebsd.org/pipermail/freebsd-arch/2012-July/012756.html > We didn't measure any significant performance regressions with > hwmpc(4), using some benckmarks programs: > http://people.freebsd.org/~davide/poll_test/poll_test.c > http://people.freebsd.org/~mav/testsleep.c > http://people.freebsd.org/~mav/testidle.c > > We tested the code on amd64, MIPS and arm. Any kind of testing or > comment would be really appreciated. The full diff of the work against > HEAD can be found at: http://people.freebsd.org/~davide/calloutng.diff > If noone have objections, we plan to merge the repository to HEAD in a > week or so. > > Thanks, > > Davide > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " > --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2211611 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+------------------------------- From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 12:57:44 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A24C9E42; Fri, 14 Dec 2012 12:57:44 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 285DB8FC08; Fri, 14 Dec 2012 12:57:43 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fy27so1650624vcb.13 for ; Fri, 14 Dec 2012 04:57:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=8PJB2ecllx1iDqVgIRkKoBl1zftOU9L4PS3dIO1jDlk=; b=HQu4Wz4olQxj5ZoZ5m0AnHXJw99h5xeCcI1PsImakr8PBgnlgDerL9USEPuJe2yXZ/ E7NrpsV4UGGNts38KMtPpA9DCtTIPsCIzsCl5lhlU5N/0Ud4/ffnh1Ga9gpkrvFwJznh 0ZCztZwtqbU7eeV8smtFkR1B4TF3itsMszPLgF78bqxoJVJa1ByZw9TbGDeNqUohSN06 Ff+aAdbfNJuk0jfKOj3+L+iphF3p7Elsd7HPbzTkeEg0vFjgpYz1qG1ZGxfZI5QUIi7+ FywBy2pSGOYD3ggAoeyPfyKmbb3IwLEyaFzdaIyxZyDlmHGw3/Ms3lWJtVyqJcQyDSPt bbdw== MIME-Version: 1.0 Received: by 10.220.154.148 with SMTP id o20mr8871880vcw.54.1355489857090; Fri, 14 Dec 2012 04:57:37 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.245.130 with HTTP; Fri, 14 Dec 2012 04:57:36 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Dec 2012 13:57:36 +0100 X-Google-Sender-Auth: Ifv5VTCJMyv6JvwmdGGnT1itmkw Message-ID: Subject: Re: [RFC/RFT] calloutng From: Davide Italiano To: Luigi Rizzo Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 12:57:44 -0000 On Fri, Dec 14, 2012 at 7:41 AM, Luigi Rizzo wrote: > > On Fri, Dec 14, 2012 at 12:12 AM, Davide Italiano > wrote: >> >> Hi. >> This patch takes callout(9) and redesign the KPI and the >> implementation. The main objective of this work is making the >> subsystem tickless. In the last several years, this possibility has >> been discussed widely (http://markmail.org/message/q3xmr2ttlzpqkmae), >> but until now noone really implemented that. >> If you want a complete history of what has been done in the last >> months you can check the calloutng project repository >> http://svnweb.freebsd.org/base/projects/calloutng/ >> For lazy people, here's a summary: > > > thanks for the work and the detailed summary. > Perhaps it would be useful if you could provide a few high level > details on the use and performance of the new scheme, such as: > > - is the old callout KPI still available ? (i am asking because it would > help maintaining third party kernel modules that are expected to > work on different FreeBSD releases) > Obviously the old KPI is still available. callout(9) is a very popular interface and I don't think removing the old interface is a good idea, because could make unhappy some vendor when its code doesn't build anymore on FreeBSD. > - do you have numbers on what is the fastest rate at which callouts > can be fired (e.g. say you have a callout which increments a > counter and schedules the next callout in (struct bintime){0,1} ) ? > > > - is there a possibility that if callout requests are too close to each > other (e.g. the above test) the thread dispatching callouts will > run forever ? if so, is there a way to make such thread yield > after a while ? > > - since you mentioned nanosleep() poll() and select() have been > ported to the new callout, is there a way to guarantee that user > using these functions with a very short timeout are actually > descheduled as opposed to "interval too short, don't bother" ? > > - do you have numbers on how many calls per second we can > have for a process that does > for (;;) { nanosleep(min_value_that_causes_descheduling); > > I also have some comments on the diff: > - can you provide a diff -p ? > > - for several functions the only change is the name of an argument > from "busy" to "us". Can you elaborate the reason for the change, > and whether "us" means microseconds or the pronoun ?) > Please see r242905 by mav@. > Finally, a more substantial comment: > - a lot of functions which formerly had only a "timo" argument > now have "timo, bt, precision, flags". Take seltdwait() as an example. > seltdwait() is not part of the public KPI. It has been modified to avoid code duplication. Having seltdwait() and seltdwait_bt(), i.e. two separate functions, even though we could share most of the code is not a clever approach, IMHO. As I told before, seltdwait() is not exposed so we can modify its argument without breaking anything. > It seems that you have been undecided between two approaches: > for some of these functions you have preserved the original function > that deals with ticks and introduced a new one that deals with the > bintime, > whereas in other cases you have modified the original function to add > "bt, precision, flags". > I'm not. All the functions which are part of the public KPI (e.g. condvar(9), sleepq(9), *) are still available. *_flags variants have been introduced so that consumers can take advantage of the new 'precision tolerance mechanism' implemented. Also, *_bt variants have been introduced. I don't see any "undecision" between the two approaches. Please note that now the callout backend deals with bintime, so every time callout_reset_on() is called, the 'tick' argument passed is silently converted to bintime. > I would suggest a more uniform approach, namely: > - preserve all the existing functions (T) that take a timeout in ticks; > - add a new set of corresponding functions (BT) that take > bt, precision, flags _instead_ of the ticks > - the functions (T) make immediately the conversion from ticks to > bintime(s), using macros or inline > - optionally, convert kernel components to the new (BT) functions > where this makes sense (e.g. we can exploit the finer-granularity > of the new calls, etc.) > > cheers > luigi > > 1) callout(9) is not anymore constrained to the resolution a periodic >> >> "hz" clock can give. In order to do that, the eventtimers(4) subsystem >> is used as backend. >> 2) Conversely from what discussed in past, we maintained the callwheel >> as underlying data structure for keeping track of the outstading >> timeouts. This choice has a couple of advantages, in particular we can >> still take benefits from the O(1) average complexity of the wheel for >> all the operations. Also, we thought the code duplication that would >> arise from the use of a two-staged backend for callout (e.g. use wheel >> for coarse resolution event and another data structure, such as an >> heap for high resolution events), is unacceptable. In fact, as long as >> callout gained the ability to migrate from a cpu to another having a >> double backend would mean doubling the code for the migration path. >> 3) A way to dispatch interrupts from hardware interrupt context has >> been implemented, using special callout flag. This has limited >> applicability, but avoid the dispatching of a SWI thread for handling >> specific callouts, avoiding the wake up of another CPU for processing >> and a (relatively useless) context switch >> 4) As long as new callout mechanism deals with bintime and not anymore >> with ticks, time is specified as absolute and not relative anymore. In >> order to get current time binuptime() or getbinuptime() is used, and a >> sysctl is introduced to selectively choose the function to use, based >> on a precision threshold. >> 5) A mechanism for specifying precision tolerance has been >> implemented. The callout processing mechanism has been adapted and the >> callout data structure augmented so that the codepath can take >> advantage and aggregate events which overlap in time. >> >> >> The new proposed KPI for callout is the following: >> callout_reset_bt_on(..., struct bintime time, struct bintime pr, ..., in= t >> flags) >> where =91time=92 argument represets the time at which the callout should >> fire, =91pr=92 represents the precision tolerance expressed as an absolu= te >> value, and =91flags=92, which could be used to specify new features, i.e= . >> for now, the possibility to run the callout from fast interrupt >> context. >> The old KPI has been extended introducing the callout_reset_flags() >> function, which is the same of callout_reset*(), but takes an >> additional argument =91int flags=92 that can be used in the same fashion >> of the =91flags=92 argument for the new KPI. Using the =91flags=92 consu= mers >> can also specify relative precision tolerance in terms of power-of-two >> portion of the timeout passed as ticks. >> Using this strategy, the new precision mechanism can be used for the >> existing services without major modifications. >> >> Some consumers have been ported to the new KPI, in particular >> nanosleep(), poll(), select(), because they take immediate advantage >> from the arbitrary precision offered by the new infrastructure. >> For some statistics about the outcome of the conversion to the new >> service, please refer to the end of this e-mail: >> http://lists.freebsd.org/pipermail/freebsd-arch/2012-July/012756.html >> We didn't measure any significant performance regressions with >> hwmpc(4), using some benckmarks programs: >> http://people.freebsd.org/~davide/poll_test/poll_test.c >> http://people.freebsd.org/~mav/testsleep.c >> http://people.freebsd.org/~mav/testidle.c >> >> We tested the code on amd64, MIPS and arm. Any kind of testing or >> comment would be really appreciated. The full diff of the work against >> HEAD can be found at: http://people.freebsd.org/~davide/calloutng.diff >> If noone have objections, we plan to merge the repository to HEAD in a >> week or so. >> >> Thanks, >> >> Davide >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.or= g" > > > > > -- > -----------------------------------------+------------------------------- > Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione > http://www.iet.unipi.it/~luigi/ . Universita` di Pisa > TEL +39-050-2211611 . via Diotisalvi 2 > Mobile +39-338-6809875 . 56122 PISA (Italy) > -----------------------------------------+------------------------------- > From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 13:13:07 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 651F862E; Fri, 14 Dec 2012 13:13:07 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id E054C8FC17; Fri, 14 Dec 2012 13:13:06 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fy27so1677467vcb.13 for ; Fri, 14 Dec 2012 05:13:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=2q+HuqqGX8DRlnglV+Ns18Qpoent+zbqkLMHF+3sPJ0=; b=hSlMOjCkOy3XLqSatRzGY93GCVux7B46Fogzjjvv7cCs+1UE5WGYTMCMDEXfoYQjgQ 8L0O+lXEYr/FcGtlPkiBcFtbtUiTCaw6Spju3IZRRWcdcwRB84pGrCrz4VgM41wwiidb HeaU509NMARDqIIF1s+l1tBYTHytaAN5CePFu9RsDQuPJp+FKHnJYTb+FL65RhMiX24p me2EaStiHZ0t0REaiQVhlxhv1Dnvz5ppEkKfhFEEPx6qgEnuq/Lc/7FxN9Prh+kX+dHB vAd3PkunhpNOO4MxLS6Zc0sLFJDPIFOR1/tXyqKbgHpdOhaHmnyi9NgvIPfBuPKT9tzO sRLQ== MIME-Version: 1.0 Received: by 10.52.92.139 with SMTP id cm11mr7744830vdb.85.1355490785961; Fri, 14 Dec 2012 05:13:05 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.245.130 with HTTP; Fri, 14 Dec 2012 05:13:05 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Dec 2012 14:13:05 +0100 X-Google-Sender-Auth: fVarY5sYh8B5WuY61x-2-2K7Uvo Message-ID: Subject: Re: [RFC/RFT] calloutng From: Davide Italiano To: Luigi Rizzo Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 13:13:07 -0000 On Fri, Dec 14, 2012 at 1:57 PM, Davide Italiano wrote= : > On Fri, Dec 14, 2012 at 7:41 AM, Luigi Rizzo wrote: >> >> On Fri, Dec 14, 2012 at 12:12 AM, Davide Italiano >> wrote: >>> >>> Hi. >>> This patch takes callout(9) and redesign the KPI and the >>> implementation. The main objective of this work is making the >>> subsystem tickless. In the last several years, this possibility has >>> been discussed widely (http://markmail.org/message/q3xmr2ttlzpqkmae), >>> but until now noone really implemented that. >>> If you want a complete history of what has been done in the last >>> months you can check the calloutng project repository >>> http://svnweb.freebsd.org/base/projects/calloutng/ >>> For lazy people, here's a summary: >> >> >> thanks for the work and the detailed summary. >> Perhaps it would be useful if you could provide a few high level >> details on the use and performance of the new scheme, such as: >> >> - is the old callout KPI still available ? (i am asking because it would >> help maintaining third party kernel modules that are expected to >> work on different FreeBSD releases) >> > > Obviously the old KPI is still available. callout(9) is a very popular > interface and I don't think removing the old interface is a good idea, > because could make unhappy some vendor when its code doesn't build > anymore on FreeBSD. > >> - do you have numbers on what is the fastest rate at which callouts >> can be fired (e.g. say you have a callout which increments a >> counter and schedules the next callout in (struct bintime){0,1} ) ? >> Right now, all the services rely on the old interface. This means they cannot be fired before 1 tick has elapsed, e.g. considering hz =3D 1000 on most of the machines, 1 millisecond. Now that nanosleep() relies on the new interface, we measured 4-5 microseconds latency for the processing before the callout is actually fired. I can't say if we can still lower this value, but I cannot imagine, for now, a consumer that actually request a shorter timeout. >> >> - is there a possibility that if callout requests are too close to each >> other (e.g. the above test) the thread dispatching callouts will >> run forever ? if so, is there a way to make such thread yield >> after a while ? >> Most of the processing is still done in a SWI thread, "at a later time", so I don't think this is a problem. >> - since you mentioned nanosleep() poll() and select() have been >> ported to the new callout, is there a way to guarantee that user >> using these functions with a very short timeout are actually >> descheduled as opposed to "interval too short, don't bother" ? >> >> - do you have numbers on how many calls per second we can >> have for a process that does >> for (;;) { nanosleep(min_value_that_causes_descheduling); >> I don't follow you here. >> I also have some comments on the diff: >> - can you provide a diff -p ? >> >> - for several functions the only change is the name of an argument >> from "busy" to "us". Can you elaborate the reason for the change, >> and whether "us" means microseconds or the pronoun ?) >> > > Please see r242905 by mav@. > >> Finally, a more substantial comment: >> - a lot of functions which formerly had only a "timo" argument >> now have "timo, bt, precision, flags". Take seltdwait() as an example. >> > > seltdwait() is not part of the public KPI. It has been modified to > avoid code duplication. Having seltdwait() and seltdwait_bt(), i.e. > two separate functions, even though we could share most of the code is > not a clever approach, IMHO. > As I told before, seltdwait() is not exposed so we can modify its > argument without breaking anything. > >> It seems that you have been undecided between two approaches: >> for some of these functions you have preserved the original function >> that deals with ticks and introduced a new one that deals with the >> bintime, >> whereas in other cases you have modified the original function to add >> "bt, precision, flags". >> > > I'm not. All the functions which are part of the public KPI (e.g. > condvar(9), sleepq(9), *) are still available. *_flags variants have > been introduced so that consumers can take advantage of the new > 'precision tolerance mechanism' implemented. Also, *_bt variants have > been introduced. I don't see any "undecision" between the two > approaches. > Please note that now the callout backend deals with bintime, so every > time callout_reset_on() is called, the 'tick' argument passed is > silently converted to bintime. > >> I would suggest a more uniform approach, namely: >> - preserve all the existing functions (T) that take a timeout in ticks= ; >> - add a new set of corresponding functions (BT) that take >> bt, precision, flags _instead_ of the ticks >> - the functions (T) make immediately the conversion from ticks to >> bintime(s), using macros or inline >> - optionally, convert kernel components to the new (BT) functions >> where this makes sense (e.g. we can exploit the finer-granularity >> of the new calls, etc.) >> > This is the strategy we followed. > > >> cheers >> luigi >> >> 1) callout(9) is not anymore constrained to the resolution a periodic >>> >>> "hz" clock can give. In order to do that, the eventtimers(4) subsystem >>> is used as backend. >>> 2) Conversely from what discussed in past, we maintained the callwheel >>> as underlying data structure for keeping track of the outstading >>> timeouts. This choice has a couple of advantages, in particular we can >>> still take benefits from the O(1) average complexity of the wheel for >>> all the operations. Also, we thought the code duplication that would >>> arise from the use of a two-staged backend for callout (e.g. use wheel >>> for coarse resolution event and another data structure, such as an >>> heap for high resolution events), is unacceptable. In fact, as long as >>> callout gained the ability to migrate from a cpu to another having a >>> double backend would mean doubling the code for the migration path. >>> 3) A way to dispatch interrupts from hardware interrupt context has >>> been implemented, using special callout flag. This has limited >>> applicability, but avoid the dispatching of a SWI thread for handling >>> specific callouts, avoiding the wake up of another CPU for processing >>> and a (relatively useless) context switch >>> 4) As long as new callout mechanism deals with bintime and not anymore >>> with ticks, time is specified as absolute and not relative anymore. In >>> order to get current time binuptime() or getbinuptime() is used, and a >>> sysctl is introduced to selectively choose the function to use, based >>> on a precision threshold. >>> 5) A mechanism for specifying precision tolerance has been >>> implemented. The callout processing mechanism has been adapted and the >>> callout data structure augmented so that the codepath can take >>> advantage and aggregate events which overlap in time. >>> >>> >>> The new proposed KPI for callout is the following: >>> callout_reset_bt_on(..., struct bintime time, struct bintime pr, ..., i= nt >>> flags) >>> where =91time=92 argument represets the time at which the callout shoul= d >>> fire, =91pr=92 represents the precision tolerance expressed as an absol= ute >>> value, and =91flags=92, which could be used to specify new features, i.= e. >>> for now, the possibility to run the callout from fast interrupt >>> context. >>> The old KPI has been extended introducing the callout_reset_flags() >>> function, which is the same of callout_reset*(), but takes an >>> additional argument =91int flags=92 that can be used in the same fashio= n >>> of the =91flags=92 argument for the new KPI. Using the =91flags=92 cons= umers >>> can also specify relative precision tolerance in terms of power-of-two >>> portion of the timeout passed as ticks. >>> Using this strategy, the new precision mechanism can be used for the >>> existing services without major modifications. >>> >>> Some consumers have been ported to the new KPI, in particular >>> nanosleep(), poll(), select(), because they take immediate advantage >>> from the arbitrary precision offered by the new infrastructure. >>> For some statistics about the outcome of the conversion to the new >>> service, please refer to the end of this e-mail: >>> http://lists.freebsd.org/pipermail/freebsd-arch/2012-July/012756.html >>> We didn't measure any significant performance regressions with >>> hwmpc(4), using some benckmarks programs: >>> http://people.freebsd.org/~davide/poll_test/poll_test.c >>> http://people.freebsd.org/~mav/testsleep.c >>> http://people.freebsd.org/~mav/testidle.c >>> >>> We tested the code on amd64, MIPS and arm. Any kind of testing or >>> comment would be really appreciated. The full diff of the work against >>> HEAD can be found at: http://people.freebsd.org/~davide/calloutng.diff >>> If noone have objections, we plan to merge the repository to HEAD in a >>> week or so. >>> >>> Thanks, >>> >>> Davide >>> _______________________________________________ >>> freebsd-current@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg" >> >> >> >> >> -- >> -----------------------------------------+------------------------------= - >> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione >> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >> TEL +39-050-2211611 . via Diotisalvi 2 >> Mobile +39-338-6809875 . 56122 PISA (Italy) >> -----------------------------------------+------------------------------= - >> From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 13:46:54 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3FCA55AC; Fri, 14 Dec 2012 13:46:54 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) by mx1.freebsd.org (Postfix) with ESMTP id CD8CB8FC14; Fri, 14 Dec 2012 13:46:53 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id D0446358C5A; Fri, 14 Dec 2012 14:46:52 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id B88452848C; Fri, 14 Dec 2012 14:46:52 +0100 (CET) Date: Fri, 14 Dec 2012 14:46:52 +0100 From: Jilles Tjoelker To: Brooks Davis Subject: Re: [CFT] Importing NetBSD's vis/unvis(3) Message-ID: <20121214134652.GB89880@stack.nl> References: <20121211202925.GA40927@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121211202925.GA40927@lor.one-eyed-alien.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 13:46:54 -0000 On Tue, Dec 11, 2012 at 02:29:25PM -0600, Brooks Davis wrote: > As part of importing NetBSD's mtree I need to add more vis(3) API > functions from NetBSD. The easiest path seems to be a wholesale import > of their code with the addition of VIS_GLOB support for compatibility. > The attached patch accomplishes this. Please review or test. > The ABI of unvis changes slightly so I've added a compatibility shim for > it. Looks like NetBSD changed it such that it cannot be kept compatible this time, by using vis(3) flags for unvis(3) which must collide with our old unvis(3) flags. > Note that old files must be removed in addition to applying the patch so > Make finds the right files. > http://people.freebsd.org/~brooks/patches/vis.diff > [snip] > diff -ruN contrib/libc-vis/unvis.c contrib/libc-vis/unvis.c > --- contrib/libc-vis/unvis.c 1969-12-31 18:00:00.000000000 -0600 > +++ contrib/libc-vis/unvis.c 2012-10-20 09:22:09.000000000 -0500 > @@ -0,0 +1,562 @@ > [snip] > +/* > + * RFC 1866 > + */ > +static const struct nv { > + const char *name; > + uint8_t value; > +} nv[] = { > + { "AElig", 198 }, /* capital AE diphthong (ligature) */ > + { "Aacute", 193 }, /* capital A, acute accent */ > + { "Acirc", 194 }, /* capital A, circumflex accent */ > [snip] > +}; Please avoid adding 100 relative relocations, for example by changing const char *name to char name[7]. RTLD will have to adjust 100 pointers for the load address of libc.so.7, and even in a static library the pointers take up a disproportionate amount of space. > [snip] > +#define CHECKSPACE() \ > + do { \ > + if (dlen-- == 0) { \ > + errno = ENOSPC; \ > + return -1; \ > + } \ > + } while (/*CONSTCOND*/0) A creative use of [ENOSPC], but NetBSD does it too... > [snip] -- Jilles Tjoelker From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 14:21:56 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D48CD376; Fri, 14 Dec 2012 14:21:56 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6EA958FC17; Fri, 14 Dec 2012 14:21:56 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so3581454oag.13 for ; Fri, 14 Dec 2012 06:21:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=97x28J9YvCya+h3C2TFfcQ+VeL2iGwlkcR8md7mgS3s=; b=W3g+WsXcYheGDMTGnOG2aUZhxjOInZABdVHMFN9+6ZEn9RDoMKb6tjfTUGFtgAochc Wcx3Tu6hqK5Bkki+r8G9mOYeS2quX4UFZiliMgNuUihDIgbV+JY5C6KFLb+ux5HnrJmC 5bAueKBfzYI0TP+bKUP/PMAMvOrAyvtrpFd/afp/Oe3rp5wIY/wAdzuwrqYwTouTX1uv iJhtnMGwIIflSmCJ7e6XbvUbekfVusmpXIYMPhBREV0kGUJNAZwKBoloPRiPhdk+ApiB KClC3CIW3+JZMjIj33plrkKr4NlsVGIKgGKd0HByBkK6SgmF1ZZIktHRIfgFxFBeTxkd XKXA== MIME-Version: 1.0 Received: by 10.182.17.72 with SMTP id m8mr4614135obd.55.1355494915778; Fri, 14 Dec 2012 06:21:55 -0800 (PST) Received: by 10.76.34.227 with HTTP; Fri, 14 Dec 2012 06:21:55 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Dec 2012 15:21:55 +0100 Message-ID: Subject: Re: [RFC/RFT] calloutng From: Oliver Pinter To: Davide Italiano Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 14:21:57 -0000 Hi! 635 - return tticks; 636 + getbinuptime(&pbt); 637 + bt.sec =3D data / 1000; 638 + bt.frac =3D (data % 1000) * (uint64_t)1844674407309000LL; 639 + bintime_add(&bt, &pbt); 640 + return bt; 641 } What is this 1844674407309000LL constant? 783 @@ -275,7 +288,7 @@ 784 do { 785 th =3D timehands; 786 gen =3D th->th_generation; 787 - bintime2timeval(&th->th_offset, tvp); 788 + Bintime2timeval(&th->th_offset, tvp); 789 } while (gen =3D=3D 0 || gen !=3D th->th_generation); 790 } 791 Capital B is there possible a typo? On 12/14/12, Davide Italiano wrote: > On Fri, Dec 14, 2012 at 1:57 PM, Davide Italiano > wrote: >> On Fri, Dec 14, 2012 at 7:41 AM, Luigi Rizzo wrote: >>> >>> On Fri, Dec 14, 2012 at 12:12 AM, Davide Italiano >>> wrote: >>>> >>>> Hi. >>>> This patch takes callout(9) and redesign the KPI and the >>>> implementation. The main objective of this work is making the >>>> subsystem tickless. In the last several years, this possibility has >>>> been discussed widely (http://markmail.org/message/q3xmr2ttlzpqkmae), >>>> but until now noone really implemented that. >>>> If you want a complete history of what has been done in the last >>>> months you can check the calloutng project repository >>>> http://svnweb.freebsd.org/base/projects/calloutng/ >>>> For lazy people, here's a summary: >>> >>> >>> thanks for the work and the detailed summary. >>> Perhaps it would be useful if you could provide a few high level >>> details on the use and performance of the new scheme, such as: >>> >>> - is the old callout KPI still available ? (i am asking because it woul= d >>> help maintaining third party kernel modules that are expected to >>> work on different FreeBSD releases) >>> >> >> Obviously the old KPI is still available. callout(9) is a very popular >> interface and I don't think removing the old interface is a good idea, >> because could make unhappy some vendor when its code doesn't build >> anymore on FreeBSD. >> >>> - do you have numbers on what is the fastest rate at which callouts >>> can be fired (e.g. say you have a callout which increments a >>> counter and schedules the next callout in (struct bintime){0,1} ) ? >>> > > Right now, all the services rely on the old interface. This means they > cannot be fired before 1 tick has elapsed, e.g. considering hz =3D 1000 > on most of the machines, 1 millisecond. > Now that nanosleep() relies on the new interface, we measured 4-5 > microseconds latency for the processing before the callout is actually > fired. I can't say if we can still lower this value, but I cannot > imagine, for now, a consumer that actually request a shorter timeout. > >>> >>> - is there a possibility that if callout requests are too close to each >>> other (e.g. the above test) the thread dispatching callouts will >>> run forever ? if so, is there a way to make such thread yield >>> after a while ? >>> > > Most of the processing is still done in a SWI thread, "at a later > time", so I don't think this is a problem. > >>> - since you mentioned nanosleep() poll() and select() have been >>> ported to the new callout, is there a way to guarantee that user >>> using these functions with a very short timeout are actually >>> descheduled as opposed to "interval too short, don't bother" ? >>> >>> - do you have numbers on how many calls per second we can >>> have for a process that does >>> for (;;) { nanosleep(min_value_that_causes_descheduling); >>> > > I don't follow you here. > >>> I also have some comments on the diff: >>> - can you provide a diff -p ? >>> >>> - for several functions the only change is the name of an argument >>> from "busy" to "us". Can you elaborate the reason for the change, >>> and whether "us" means microseconds or the pronoun ?) >>> >> >> Please see r242905 by mav@. >> >>> Finally, a more substantial comment: >>> - a lot of functions which formerly had only a "timo" argument >>> now have "timo, bt, precision, flags". Take seltdwait() as an example= . >>> >> >> seltdwait() is not part of the public KPI. It has been modified to >> avoid code duplication. Having seltdwait() and seltdwait_bt(), i.e. >> two separate functions, even though we could share most of the code is >> not a clever approach, IMHO. >> As I told before, seltdwait() is not exposed so we can modify its >> argument without breaking anything. >> >>> It seems that you have been undecided between two approaches: >>> for some of these functions you have preserved the original function >>> that deals with ticks and introduced a new one that deals with the >>> bintime, >>> whereas in other cases you have modified the original function to add >>> "bt, precision, flags". >>> >> >> I'm not. All the functions which are part of the public KPI (e.g. >> condvar(9), sleepq(9), *) are still available. *_flags variants have >> been introduced so that consumers can take advantage of the new >> 'precision tolerance mechanism' implemented. Also, *_bt variants have >> been introduced. I don't see any "undecision" between the two >> approaches. >> Please note that now the callout backend deals with bintime, so every >> time callout_reset_on() is called, the 'tick' argument passed is >> silently converted to bintime. >> >>> I would suggest a more uniform approach, namely: >>> - preserve all the existing functions (T) that take a timeout in >>> ticks; >>> - add a new set of corresponding functions (BT) that take >>> bt, precision, flags _instead_ of the ticks >>> - the functions (T) make immediately the conversion from ticks to >>> bintime(s), using macros or inline >>> - optionally, convert kernel components to the new (BT) functions >>> where this makes sense (e.g. we can exploit the finer-granularity >>> of the new calls, etc.) >>> >> > > This is the strategy we followed. > >> >> >>> cheers >>> luigi >>> >>> 1) callout(9) is not anymore constrained to the resolution a periodic >>>> >>>> "hz" clock can give. In order to do that, the eventtimers(4) subsystem >>>> is used as backend. >>>> 2) Conversely from what discussed in past, we maintained the callwheel >>>> as underlying data structure for keeping track of the outstading >>>> timeouts. This choice has a couple of advantages, in particular we can >>>> still take benefits from the O(1) average complexity of the wheel for >>>> all the operations. Also, we thought the code duplication that would >>>> arise from the use of a two-staged backend for callout (e.g. use wheel >>>> for coarse resolution event and another data structure, such as an >>>> heap for high resolution events), is unacceptable. In fact, as long as >>>> callout gained the ability to migrate from a cpu to another having a >>>> double backend would mean doubling the code for the migration path. >>>> 3) A way to dispatch interrupts from hardware interrupt context has >>>> been implemented, using special callout flag. This has limited >>>> applicability, but avoid the dispatching of a SWI thread for handling >>>> specific callouts, avoiding the wake up of another CPU for processing >>>> and a (relatively useless) context switch >>>> 4) As long as new callout mechanism deals with bintime and not anymore >>>> with ticks, time is specified as absolute and not relative anymore. In >>>> order to get current time binuptime() or getbinuptime() is used, and a >>>> sysctl is introduced to selectively choose the function to use, based >>>> on a precision threshold. >>>> 5) A mechanism for specifying precision tolerance has been >>>> implemented. The callout processing mechanism has been adapted and the >>>> callout data structure augmented so that the codepath can take >>>> advantage and aggregate events which overlap in time. >>>> >>>> >>>> The new proposed KPI for callout is the following: >>>> callout_reset_bt_on(..., struct bintime time, struct bintime pr, ..., >>>> int >>>> flags) >>>> where =91time=92 argument represets the time at which the callout shou= ld >>>> fire, =91pr=92 represents the precision tolerance expressed as an abso= lute >>>> value, and =91flags=92, which could be used to specify new features, i= .e. >>>> for now, the possibility to run the callout from fast interrupt >>>> context. >>>> The old KPI has been extended introducing the callout_reset_flags() >>>> function, which is the same of callout_reset*(), but takes an >>>> additional argument =91int flags=92 that can be used in the same fashi= on >>>> of the =91flags=92 argument for the new KPI. Using the =91flags=92 con= sumers >>>> can also specify relative precision tolerance in terms of power-of-two >>>> portion of the timeout passed as ticks. >>>> Using this strategy, the new precision mechanism can be used for the >>>> existing services without major modifications. >>>> >>>> Some consumers have been ported to the new KPI, in particular >>>> nanosleep(), poll(), select(), because they take immediate advantage >>>> from the arbitrary precision offered by the new infrastructure. >>>> For some statistics about the outcome of the conversion to the new >>>> service, please refer to the end of this e-mail: >>>> http://lists.freebsd.org/pipermail/freebsd-arch/2012-July/012756.html >>>> We didn't measure any significant performance regressions with >>>> hwmpc(4), using some benckmarks programs: >>>> http://people.freebsd.org/~davide/poll_test/poll_test.c >>>> http://people.freebsd.org/~mav/testsleep.c >>>> http://people.freebsd.org/~mav/testidle.c >>>> >>>> We tested the code on amd64, MIPS and arm. Any kind of testing or >>>> comment would be really appreciated. The full diff of the work against >>>> HEAD can be found at: http://people.freebsd.org/~davide/calloutng.diff >>>> If noone have objections, we plan to merge the repository to HEAD in a >>>> week or so. >>>> >>>> Thanks, >>>> >>>> Davide >>>> _______________________________________________ >>>> freebsd-current@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>>> To unsubscribe, send any mail to >>>> "freebsd-current-unsubscribe@freebsd.org" >>> >>> >>> >>> >>> -- >>> -----------------------------------------+-----------------------------= -- >>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazion= e >>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >>> TEL +39-050-2211611 . via Diotisalvi 2 >>> Mobile +39-338-6809875 . 56122 PISA (Italy) >>> -----------------------------------------+-----------------------------= -- >>> > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " > From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 14:42:20 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B26F2CCF; Fri, 14 Dec 2012 14:42:20 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1E1968FC15; Fri, 14 Dec 2012 14:42:19 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fy27so1849222vcb.13 for ; Fri, 14 Dec 2012 06:42:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=qOhXhYMsRcEo6Ji3MGDvErU9Ml4I9pNNXrR6HdVSpY8=; b=OB0aie8Mp5rAliaIqP98zB7P1+wtZKeU4v5nE5eKoHhUPqkHYOHyJGJ+NVVMeAt9+6 qxwsBl5/oxzCgFPL1+Rs5XdZgfqCebm4OEVr9Mifv0aGPcD+u2UMque2rWm0qgC8I34q yXeW6ND9C2zuyIP720k8M2V1oWQvVLZE2km/tVtpWieU1xKNYMU7kk4uhiXFbIkIYzvl o8RhWqgIWol4uLVn6TOSbtC/OhzR2XKnkgwwGX8GMF1IDnclvEG6zPw9a349YZfLK7eC QsjDwehyf5HZ2GL82egM/guwDrQfiPTxzKyERQ4uzTrScs84ziYmZ9zXF7JSCl/p8vbF i/1A== MIME-Version: 1.0 Received: by 10.52.70.46 with SMTP id j14mr8057207vdu.99.1355496138797; Fri, 14 Dec 2012 06:42:18 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.245.130 with HTTP; Fri, 14 Dec 2012 06:42:18 -0800 (PST) In-Reply-To: References: Date: Fri, 14 Dec 2012 15:42:18 +0100 X-Google-Sender-Auth: e1H7Zj66rHMppm5oDrHiD1IuL1A Message-ID: Subject: Re: [RFC/RFT] calloutng From: Davide Italiano To: Oliver Pinter Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 14:42:20 -0000 On Fri, Dec 14, 2012 at 3:21 PM, Oliver Pinter wrot= e: > Hi! > > 635 - return tticks; > 636 + getbinuptime(&pbt); > 637 + bt.sec =3D data / 1000; > 638 + bt.frac =3D (data % 1000) * (uint64_t)1844674407309000LL; > 639 + bintime_add(&bt, &pbt); > 640 + return bt; > 641 } > > What is this 1844674407309000LL constant? > > > 783 @@ -275,7 +288,7 @@ > 784 do { > 785 th =3D timehands; > 786 gen =3D th->th_generation; > 787 - bintime2timeval(&th->th_offset, tvp); > 788 + Bintime2timeval(&th->th_offset, tvp); > 789 } while (gen =3D=3D 0 || gen !=3D th->th_generation); > 790 } > 791 > > Capital B is there possible a typo? > Hi Oliver, thanks for reporting. Yes, both are typos. The costant is /* 18446744073709 =3D int(2^64 / 1000000) */ used to convert from timeval to bintime. > On 12/14/12, Davide Italiano wrote: >> On Fri, Dec 14, 2012 at 1:57 PM, Davide Italiano >> wrote: >>> On Fri, Dec 14, 2012 at 7:41 AM, Luigi Rizzo wrote= : >>>> >>>> On Fri, Dec 14, 2012 at 12:12 AM, Davide Italiano >>>> wrote: >>>>> >>>>> Hi. >>>>> This patch takes callout(9) and redesign the KPI and the >>>>> implementation. The main objective of this work is making the >>>>> subsystem tickless. In the last several years, this possibility has >>>>> been discussed widely (http://markmail.org/message/q3xmr2ttlzpqkmae), >>>>> but until now noone really implemented that. >>>>> If you want a complete history of what has been done in the last >>>>> months you can check the calloutng project repository >>>>> http://svnweb.freebsd.org/base/projects/calloutng/ >>>>> For lazy people, here's a summary: >>>> >>>> >>>> thanks for the work and the detailed summary. >>>> Perhaps it would be useful if you could provide a few high level >>>> details on the use and performance of the new scheme, such as: >>>> >>>> - is the old callout KPI still available ? (i am asking because it wou= ld >>>> help maintaining third party kernel modules that are expected to >>>> work on different FreeBSD releases) >>>> >>> >>> Obviously the old KPI is still available. callout(9) is a very popular >>> interface and I don't think removing the old interface is a good idea, >>> because could make unhappy some vendor when its code doesn't build >>> anymore on FreeBSD. >>> >>>> - do you have numbers on what is the fastest rate at which callouts >>>> can be fired (e.g. say you have a callout which increments a >>>> counter and schedules the next callout in (struct bintime){0,1} ) ? >>>> >> >> Right now, all the services rely on the old interface. This means they >> cannot be fired before 1 tick has elapsed, e.g. considering hz =3D 1000 >> on most of the machines, 1 millisecond. >> Now that nanosleep() relies on the new interface, we measured 4-5 >> microseconds latency for the processing before the callout is actually >> fired. I can't say if we can still lower this value, but I cannot >> imagine, for now, a consumer that actually request a shorter timeout. >> >>>> >>>> - is there a possibility that if callout requests are too close to eac= h >>>> other (e.g. the above test) the thread dispatching callouts will >>>> run forever ? if so, is there a way to make such thread yield >>>> after a while ? >>>> >> >> Most of the processing is still done in a SWI thread, "at a later >> time", so I don't think this is a problem. >> >>>> - since you mentioned nanosleep() poll() and select() have been >>>> ported to the new callout, is there a way to guarantee that user >>>> using these functions with a very short timeout are actually >>>> descheduled as opposed to "interval too short, don't bother" ? >>>> >>>> - do you have numbers on how many calls per second we can >>>> have for a process that does >>>> for (;;) { nanosleep(min_value_that_causes_descheduling); >>>> >> >> I don't follow you here. >> >>>> I also have some comments on the diff: >>>> - can you provide a diff -p ? >>>> >>>> - for several functions the only change is the name of an argument >>>> from "busy" to "us". Can you elaborate the reason for the change, >>>> and whether "us" means microseconds or the pronoun ?) >>>> >>> >>> Please see r242905 by mav@. >>> >>>> Finally, a more substantial comment: >>>> - a lot of functions which formerly had only a "timo" argument >>>> now have "timo, bt, precision, flags". Take seltdwait() as an exampl= e. >>>> >>> >>> seltdwait() is not part of the public KPI. It has been modified to >>> avoid code duplication. Having seltdwait() and seltdwait_bt(), i.e. >>> two separate functions, even though we could share most of the code is >>> not a clever approach, IMHO. >>> As I told before, seltdwait() is not exposed so we can modify its >>> argument without breaking anything. >>> >>>> It seems that you have been undecided between two approaches: >>>> for some of these functions you have preserved the original function >>>> that deals with ticks and introduced a new one that deals with the >>>> bintime, >>>> whereas in other cases you have modified the original function to ad= d >>>> "bt, precision, flags". >>>> >>> >>> I'm not. All the functions which are part of the public KPI (e.g. >>> condvar(9), sleepq(9), *) are still available. *_flags variants have >>> been introduced so that consumers can take advantage of the new >>> 'precision tolerance mechanism' implemented. Also, *_bt variants have >>> been introduced. I don't see any "undecision" between the two >>> approaches. >>> Please note that now the callout backend deals with bintime, so every >>> time callout_reset_on() is called, the 'tick' argument passed is >>> silently converted to bintime. >>> >>>> I would suggest a more uniform approach, namely: >>>> - preserve all the existing functions (T) that take a timeout in >>>> ticks; >>>> - add a new set of corresponding functions (BT) that take >>>> bt, precision, flags _instead_ of the ticks >>>> - the functions (T) make immediately the conversion from ticks to >>>> bintime(s), using macros or inline >>>> - optionally, convert kernel components to the new (BT) functions >>>> where this makes sense (e.g. we can exploit the finer-granularity >>>> of the new calls, etc.) >>>> >>> >> >> This is the strategy we followed. >> >>> >>> >>>> cheers >>>> luigi >>>> >>>> 1) callout(9) is not anymore constrained to the resolution a periodic >>>>> >>>>> "hz" clock can give. In order to do that, the eventtimers(4) subsyste= m >>>>> is used as backend. >>>>> 2) Conversely from what discussed in past, we maintained the callwhee= l >>>>> as underlying data structure for keeping track of the outstading >>>>> timeouts. This choice has a couple of advantages, in particular we ca= n >>>>> still take benefits from the O(1) average complexity of the wheel for >>>>> all the operations. Also, we thought the code duplication that would >>>>> arise from the use of a two-staged backend for callout (e.g. use whee= l >>>>> for coarse resolution event and another data structure, such as an >>>>> heap for high resolution events), is unacceptable. In fact, as long a= s >>>>> callout gained the ability to migrate from a cpu to another having a >>>>> double backend would mean doubling the code for the migration path. >>>>> 3) A way to dispatch interrupts from hardware interrupt context has >>>>> been implemented, using special callout flag. This has limited >>>>> applicability, but avoid the dispatching of a SWI thread for handling >>>>> specific callouts, avoiding the wake up of another CPU for processing >>>>> and a (relatively useless) context switch >>>>> 4) As long as new callout mechanism deals with bintime and not anymor= e >>>>> with ticks, time is specified as absolute and not relative anymore. I= n >>>>> order to get current time binuptime() or getbinuptime() is used, and = a >>>>> sysctl is introduced to selectively choose the function to use, based >>>>> on a precision threshold. >>>>> 5) A mechanism for specifying precision tolerance has been >>>>> implemented. The callout processing mechanism has been adapted and th= e >>>>> callout data structure augmented so that the codepath can take >>>>> advantage and aggregate events which overlap in time. >>>>> >>>>> >>>>> The new proposed KPI for callout is the following: >>>>> callout_reset_bt_on(..., struct bintime time, struct bintime pr, ..., >>>>> int >>>>> flags) >>>>> where =91time=92 argument represets the time at which the callout sho= uld >>>>> fire, =91pr=92 represents the precision tolerance expressed as an abs= olute >>>>> value, and =91flags=92, which could be used to specify new features, = i.e. >>>>> for now, the possibility to run the callout from fast interrupt >>>>> context. >>>>> The old KPI has been extended introducing the callout_reset_flags() >>>>> function, which is the same of callout_reset*(), but takes an >>>>> additional argument =91int flags=92 that can be used in the same fash= ion >>>>> of the =91flags=92 argument for the new KPI. Using the =91flags=92 co= nsumers >>>>> can also specify relative precision tolerance in terms of power-of-tw= o >>>>> portion of the timeout passed as ticks. >>>>> Using this strategy, the new precision mechanism can be used for the >>>>> existing services without major modifications. >>>>> >>>>> Some consumers have been ported to the new KPI, in particular >>>>> nanosleep(), poll(), select(), because they take immediate advantage >>>>> from the arbitrary precision offered by the new infrastructure. >>>>> For some statistics about the outcome of the conversion to the new >>>>> service, please refer to the end of this e-mail: >>>>> http://lists.freebsd.org/pipermail/freebsd-arch/2012-July/012756.html >>>>> We didn't measure any significant performance regressions with >>>>> hwmpc(4), using some benckmarks programs: >>>>> http://people.freebsd.org/~davide/poll_test/poll_test.c >>>>> http://people.freebsd.org/~mav/testsleep.c >>>>> http://people.freebsd.org/~mav/testidle.c >>>>> >>>>> We tested the code on amd64, MIPS and arm. Any kind of testing or >>>>> comment would be really appreciated. The full diff of the work agains= t >>>>> HEAD can be found at: http://people.freebsd.org/~davide/calloutng.dif= f >>>>> If noone have objections, we plan to merge the repository to HEAD in = a >>>>> week or so. >>>>> >>>>> Thanks, >>>>> >>>>> Davide >>>>> _______________________________________________ >>>>> freebsd-current@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>>>> To unsubscribe, send any mail to >>>>> "freebsd-current-unsubscribe@freebsd.org" >>>> >>>> >>>> >>>> >>>> -- >>>> -----------------------------------------+----------------------------= --- >>>> Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazio= ne >>>> http://www.iet.unipi.it/~luigi/ . Universita` di Pisa >>>> TEL +39-050-2211611 . via Diotisalvi 2 >>>> Mobile +39-338-6809875 . 56122 PISA (Italy) >>>> -----------------------------------------+----------------------------= --- >>>> >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.or= g" >> From owner-freebsd-arch@FreeBSD.ORG Fri Dec 14 16:45:44 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B1D92C2B for ; Fri, 14 Dec 2012 16:45:44 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (lor.one-eyed-alien.net [69.66.77.232]) by mx1.freebsd.org (Postfix) with ESMTP id 109628FC13 for ; Fri, 14 Dec 2012 16:45:42 +0000 (UTC) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.5/8.14.5) with ESMTP id qBEGjaDW072341; Fri, 14 Dec 2012 10:45:36 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.5/8.14.5/Submit) id qBEGja3j072340; Fri, 14 Dec 2012 10:45:36 -0600 (CST) (envelope-from brooks) Date: Fri, 14 Dec 2012 10:45:36 -0600 From: Brooks Davis To: Jilles Tjoelker Subject: Re: [CFT] Importing NetBSD's vis/unvis(3) Message-ID: <20121214164536.GH40927@lor.one-eyed-alien.net> References: <20121211202925.GA40927@lor.one-eyed-alien.net> <20121214134652.GB89880@stack.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZYOWEO2dMm2Af3e3" Content-Disposition: inline In-Reply-To: <20121214134652.GB89880@stack.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Dec 2012 16:45:44 -0000 --ZYOWEO2dMm2Af3e3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Dec 14, 2012 at 02:46:52PM +0100, Jilles Tjoelker wrote: > On Tue, Dec 11, 2012 at 02:29:25PM -0600, Brooks Davis wrote: > > As part of importing NetBSD's mtree I need to add more vis(3) API > > functions from NetBSD. The easiest path seems to be a wholesale import > > of their code with the addition of VIS_GLOB support for compatibility. > > The attached patch accomplishes this. Please review or test. >=20 > > The ABI of unvis changes slightly so I've added a compatibility shim for > > it. >=20 > Looks like NetBSD changed it such that it cannot be kept compatible this > time, by using vis(3) flags for unvis(3) which must collide with our old > unvis(3) flags. Are you agreeing that the shim is required or hinting that I got it wrong? > > Note that old files must be removed in addition to applying the patch so > > Make finds the right files. >=20 > > http://people.freebsd.org/~brooks/patches/vis.diff >=20 > > [snip] > > diff -ruN contrib/libc-vis/unvis.c contrib/libc-vis/unvis.c > > --- contrib/libc-vis/unvis.c 1969-12-31 18:00:00.000000000 -0600 > > +++ contrib/libc-vis/unvis.c 2012-10-20 09:22:09.000000000 -0500 > > @@ -0,0 +1,562 @@ > > [snip] > > +/* > > + * RFC 1866 > > + */ > > +static const struct nv { > > + const char *name; > > + uint8_t value; > > +} nv[] =3D { > > + { "AElig", 198 }, /* capital AE diphthong (ligature) */ > > + { "Aacute", 193 }, /* capital A, acute accent */ > > + { "Acirc", 194 }, /* capital A, circumflex accent */ > > [snip] > > +}; >=20 > Please avoid adding 100 relative relocations, for example by changing > const char *name to char name[7]. >=20 > RTLD will have to adjust 100 pointers for the load address of libc.so.7, > and even in a static library the pointers take up a disproportionate > amount of space. I'll submit a patch for this upstream. Thanks, Brooks --ZYOWEO2dMm2Af3e3 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iD8DBQFQy1evXY6L6fI4GtQRAs2NAJ95PXUm41riZGNdx+0FMqiu3d4O1gCg43QC SS12Ad9L9TVhjc+OE0mw0gY= =Hf3D -----END PGP SIGNATURE----- --ZYOWEO2dMm2Af3e3-- From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 08:44:53 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F33C81C5; Sat, 15 Dec 2012 08:44:52 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail12.syd.optusnet.com.au (mail12.syd.optusnet.com.au [211.29.132.193]) by mx1.freebsd.org (Postfix) with ESMTP id 7ED5C8FC0A; Sat, 15 Dec 2012 08:44:52 +0000 (UTC) Received: from c122-106-175-26.carlnfd1.nsw.optusnet.com.au (c122-106-175-26.carlnfd1.nsw.optusnet.com.au [122.106.175.26]) by mail12.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id qBF8ig3G022984 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 15 Dec 2012 19:44:43 +1100 Date: Sat, 15 Dec 2012 19:44:42 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Oliver Pinter Subject: Re: [RFC/RFT] calloutng In-Reply-To: Message-ID: <20121215183409.U1603@besplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=fev1UDsF c=1 sm=1 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=mUAV9h2nInsA:10 a=w_hYTCC3XhR2lAYixAoA:9 a=CjuIK1q_8ugA:10 a=bxQHXO5Py4tHmhUgaywp5w==:117 Cc: Davide Italiano , freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 08:44:53 -0000 On Fri, 14 Dec 2012, Oliver Pinter wrote: > 635 - return tticks; > 636 + getbinuptime(&pbt); > 637 + bt.sec = data / 1000; > 638 + bt.frac = (data % 1000) * (uint64_t)1844674407309000LL; > 639 + bintime_add(&bt, &pbt); > 640 + return bt; Style bugs: missing spaces around return value in new and old code. > 641 } > > What is this 1844674407309000LL constant? This is 2**64 / 10**6 * 10**3 obfuscated by printing it in hex and doing the scaling by powers of 10 manually, and then giving it a bogus type using the abominable long long misfeature. I try to kill this obfuscation and the abimination whenever I see them. In sys/time.h, this resulted in a related binary conversion using a scale factor of ((uint64_t)1 << 63) / (1000000000 >> 1). Here the power of 2 term is 2**63. 2**64 cannot be used since it exceeds uintmax_t. The power of 10 term is 10**9. This is divided by 2 to compensate for dividing 2**64 by 2. The abomination is avoided by using smaller literal values and expandling them to 64-bit values using shifts. Long long suffixes on literal constants are only needed to support C90 compilers with the long long extension on 32-bit systems anyway. Otherwise, C90+extension compilers will warn about literal constants larger than ULONG_MAX (which can only occur on 32-bit systems). Since C99 is now the default, the warnings would only without LL in the above if you use nonstandard CFLAGS. The above has to convert from the bad units of milliseconds to the bloated units of bintimes, and it is less refined than most other bintime conversions. I think that since it doesn't try to be optimal, it should just use the standard bintime conversions after first converting milliseconds to a timeval. It already does essentially that with its divisions by 1000: struct timeval tv; tv.tv_sec = data / 1000; tv.tv_usec = data % 1000 * 1000; timeval2bintime(&tv, &bt); The compliler will probably optimize /1000 and %1000 to shifts in both this and the above. Then timeval2bintime() does the almost the same multiplication as above, but spelled differently accuracy. Both give unnecessary inaccuracy in the conversion to weenieseconds: the first gives: bt.frac = data % 1000 * (2**64 / 10**6 * 10**3); the second gives: bt.frac = data % 1000 * 1000 * (2**64 / 10**6); Because of the different grouping of the multiplications, the second is unfortunately slower (1 more multiplication that cannot be done at compile time). The second also gives unnecessary (but findamental to the method) inaccuracy by pulling out the factor of 1000. The first gives the same inaccuracy, and now it is because the constant is not correctly rounded. It should be 2.0**64 / 10**3 = 1844674407309551.616 (exactly) = 1844674407309552 (rounded to nearest int) but is actually rounded down to a multiple of 1000. It would be better to round the scale factors so that the conversions are inverses of each other and tticks can be recovered from bt, but this is impossible. I tried to make the bintime conversions invert most values correctly by rounding to nearest, but phk didn't like this and the result is the bogus comment about always rounding down in time.h. So when you start with 999 msec in tticks, the resulting bt will be rounded down a little and converting back will give 998 msec; the next round of conversions will reduce 1 more, and so on until you reach a value that is exactly representable in both milliseconds and weenieseconds (875?). This despite weenieseconds providing vastly more accuracy than can be measured and vastly more accuracy than needed to represent all other time values in the kernel in a unique way. Just not in a unique way that is expressible using simple scaling conversions. The conversions that give uniqueness can still be monotonic, but can't be nonlinear in the same way that simple scaling gives. Bruce From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 09:01:16 2012 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A542588; Sat, 15 Dec 2012 09:01:16 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail06.syd.optusnet.com.au (mail06.syd.optusnet.com.au [211.29.132.187]) by mx1.freebsd.org (Postfix) with ESMTP id 8D23C8FC12; Sat, 15 Dec 2012 09:01:14 +0000 (UTC) Received: from c122-106-175-26.carlnfd1.nsw.optusnet.com.au (c122-106-175-26.carlnfd1.nsw.optusnet.com.au [122.106.175.26]) by mail06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id qBF915ck029165 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 15 Dec 2012 20:01:06 +1100 Date: Sat, 15 Dec 2012 20:01:05 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans Subject: Re: [RFC/RFT] calloutng In-Reply-To: <20121215183409.U1603@besplex.bde.org> Message-ID: <20121215194819.C1778@besplex.bde.org> References: <20121215183409.U1603@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=e5de0tV/ c=1 sm=1 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=mUAV9h2nInsA:10 a=fliXX_qwswpPfyGDAOYA:9 a=CjuIK1q_8ugA:10 a=bxQHXO5Py4tHmhUgaywp5w==:117 Cc: Davide Italiano , freebsd-arch@FreeBSD.org, freebsd-current , Oliver Pinter X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 09:01:16 -0000 On Sat, 15 Dec 2012, Bruce Evans wrote: > On Fri, 14 Dec 2012, Oliver Pinter wrote: >> >> What is this 1844674407309000LL constant? > > This is > > 2**64 / 10**6 * 10**3 > > obfuscated by printing it in hex and doing the scaling by powers of > 10 manually, and then giving it a bogus type using the abominable long > long misfeature. I try to kill this obfuscation and the abimination > whenever I see them. In sys/time.h, this resulted in a related binary > conversion using a scale factor of > > ((uint64_t)1 << 63) / (1000000000 >> 1). > > Here the power of 2 term is 2**63. 2**64 cannot be used since it exceeds > uintmax_t. The power of 10 term is 10**9. This is divided by 2 to > compensate for dividing 2**64 by 2. The abomination is avoided by using > smaller literal values and expandling them to 64-bit values using shifts. Bah, this is only de-obfuscated and de-abominated in my version: % Index: time.h % =================================================================== % RCS file: /home/ncvs/src/sys/sys/time.h,v % retrieving revision 1.65 % diff -u -2 -r1.65 time.h % --- time.h 7 Apr 2004 04:19:49 -0000 1.65 % +++ time.h 7 Apr 2004 11:28:54 -0000 % @@ -118,6 +118,5 @@ % % bt->sec = ts->tv_sec; % - /* 18446744073 = int(2^64 / 1000000000) */ % - bt->frac = ts->tv_nsec * (uint64_t)18446744073LL; % + bt->frac = ts->tv_nsec * (((uint64_t)1 << 63) / (1000000000 >> 1)); % } % The magic 1844... in time.h is at least commented on. This makes it less obscure, but takes twice as many source lines and risks the comment getting out of date with the code. The comment is also sloppy with types and uses the '^' operator without saying that it is exponentiation and nothing like the C '^' operator. The types are especially critical in the shift exprression. I like to use the Fortran '**' operator in C comments without saying what it is instead. In another reply to this thread, the value in the explanation is off by a factor of 1000 and the rounding to a multiple of 1000 is not explained. It is easy to have such errors in comments, while the code tends to be more correct since it gets checked by running it. Bruce From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 10:52:40 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2C79A386; Sat, 15 Dec 2012 10:52:40 +0000 (UTC) (envelope-from oliver.pntr@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id BC6398FC13; Sat, 15 Dec 2012 10:52:39 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so4451911oag.13 for ; Sat, 15 Dec 2012 02:52:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZFke9YRlzBdmV18enKt3D8DATotVbthF0mjlZT34+H4=; b=COoSVBhpDrOJ6AZWL33kjfrRNTVqks/Un1syw2cshwBDPI9GMWsx+3KJBN5RBfOOt1 3A0dA95wh+MJlqQRQ8S41pfa6bqtDGyx+1uI4CEv39gzmImbofEjznJlo2YYJPVagJWV ZYgwMm4brw5ImuUsMbi5XyFnQ5S7J0DMMafD8pQnFqKovwXV2X5kH/iWycZVRNT2YXCW b4LxVJkwNieQNOtCFJMr5nVquazYU8uysSRcVSAF7K+nlHeUOLKT38dV5sw/0q7IYyxR 6JB8QdZ23BrSX66XTO7koARNpfad2DMCtzRId44bRnJSKJwxC1zpKmVh/MX4o2DrX3nF XkBA== MIME-Version: 1.0 Received: by 10.182.53.3 with SMTP id x3mr6972987obo.87.1355568758888; Sat, 15 Dec 2012 02:52:38 -0800 (PST) Received: by 10.76.34.227 with HTTP; Sat, 15 Dec 2012 02:52:38 -0800 (PST) In-Reply-To: <20121215183409.U1603@besplex.bde.org> References: <20121215183409.U1603@besplex.bde.org> Date: Sat, 15 Dec 2012 11:52:38 +0100 Message-ID: Subject: Re: [RFC/RFT] calloutng From: Oliver Pinter To: Bruce Evans Content-Type: text/plain; charset=ISO-8859-1 Cc: Davide Italiano , freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 10:52:40 -0000 On 12/15/12, Bruce Evans wrote: > On Fri, 14 Dec 2012, Oliver Pinter wrote: > >> 635 - return tticks; >> 636 + getbinuptime(&pbt); >> 637 + bt.sec = data / 1000; >> 638 + bt.frac = (data % 1000) * (uint64_t)1844674407309000LL; >> 639 + bintime_add(&bt, &pbt); >> 640 + return bt; > > Style bugs: missing spaces around return value in new and old code. > >> 641 } >> >> What is this 1844674407309000LL constant? > > This is > > 2**64 / 10**6 * 10**3 > > obfuscated by printing it in hex and doing the scaling by powers of > 10 manually, and then giving it a bogus type using the abominable long > long misfeature. I try to kill this obfuscation and the abimination > whenever I see them. In sys/time.h, this resulted in a related binary > conversion using a scale factor of > > ((uint64_t)1 << 63) / (1000000000 >> 1). > > Here the power of 2 term is 2**63. 2**64 cannot be used since it exceeds > uintmax_t. The power of 10 term is 10**9. This is divided by 2 to > compensate for dividing 2**64 by 2. The abomination is avoided by using > smaller literal values and expandling them to 64-bit values using shifts. > > Long long suffixes on literal constants are only needed to support C90 > compilers with the long long extension on 32-bit systems anyway. > Otherwise, C90+extension compilers will warn about literal constants > larger than ULONG_MAX (which can only occur on 32-bit systems). Since > C99 is now the default, the warnings would only without LL in the above > if you use nonstandard CFLAGS. > > The above has to convert from the bad units of milliseconds to the bloated > units of bintimes, and it is less refined than most other bintime > conversions. I think that since it doesn't try to be optimal, it should > just use the standard bintime conversions after first converting > milliseconds to a timeval. It already does essentially that with its > divisions by 1000: > > struct timeval tv; > > tv.tv_sec = data / 1000; > tv.tv_usec = data % 1000 * 1000; > timeval2bintime(&tv, &bt); > > The compliler will probably optimize /1000 and %1000 to shifts in both > this and the above. Then timeval2bintime() does the almost the same > multiplication as above, but spelled differently accuracy. Both give > unnecessary inaccuracy in the conversion to weenieseconds: the first > gives: > > bt.frac = data % 1000 * (2**64 / 10**6 * 10**3); > > the second gives: > > bt.frac = data % 1000 * 1000 * (2**64 / 10**6); > > Because of the different grouping of the multiplications, the second > is unfortunately slower (1 more multiplication that cannot be done at > compile time). The second also gives unnecessary (but findamental to > the method) inaccuracy by pulling out the factor of 1000. The first > gives the same inaccuracy, and now it is because the constant is not > correctly rounded. It should be > > 2.0**64 / 10**3 = 1844674407309551.616 (exactly) > = 1844674407309552 (rounded to nearest int) > > but is actually rounded down to a multiple of 1000. > > It would be better to round the scale factors so that the conversions > are inverses of each other and tticks can be recovered from bt, but > this is impossible. I tried to make the bintime conversions invert > most values correctly by rounding to nearest, but phk didn't like this > and the result is the bogus comment about always rounding down in time.h. > So when you start with 999 msec in tticks, the resulting bt will be > rounded down a little and converting back will give 998 msec; the > next round of conversions will reduce 1 more, and so on until you > reach a value that is exactly representable in both milliseconds and > weenieseconds (875?). This despite weenieseconds providing vastly > more accuracy than can be measured and vastly more accuracy than needed > to represent all other time values in the kernel in a unique way. Just > not in a unique way that is expressible using simple scaling conversions. > The conversions that give uniqueness can still be monotonic, but can't > be nonlinear in the same way that simple scaling gives. Thanks for the detailed answer. :) > > Bruce > From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 11:24:54 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 86E44C0D; Sat, 15 Dec 2012 11:24:54 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail12.syd.optusnet.com.au (mail12.syd.optusnet.com.au [211.29.132.193]) by mx1.freebsd.org (Postfix) with ESMTP id 142DB8FC0A; Sat, 15 Dec 2012 11:24:53 +0000 (UTC) Received: from c122-106-175-26.carlnfd1.nsw.optusnet.com.au (c122-106-175-26.carlnfd1.nsw.optusnet.com.au [122.106.175.26]) by mail12.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id qBFBOnEA019642 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 15 Dec 2012 22:24:51 +1100 Date: Sat, 15 Dec 2012 22:24:49 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Oliver Pinter Subject: Re: [RFC/RFT] calloutng In-Reply-To: Message-ID: <20121215222156.H2309@besplex.bde.org> References: <20121215183409.U1603@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=Zvi1sKHG c=1 sm=1 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=mUAV9h2nInsA:10 a=NQ0KR7aI5Mbd9Gopmc4A:9 a=CjuIK1q_8ugA:10 a=oZ2IPoelaH8A:10 a=bxQHXO5Py4tHmhUgaywp5w==:117 Cc: Davide Italiano , freebsd-current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 11:24:54 -0000 On Sat, 15 Dec 2012, Oliver Pinter wrote: > On 12/15/12, Bruce Evans wrote: >> ... >> Because of the different grouping of the multiplications, the second >> is unfortunately slower (1 more multiplication that cannot be done at >> compile time). The second also gives unnecessary (but findamental to >> the method) inaccuracy by pulling out the factor of 1000. The first >> gives the same inaccuracy, and now it is because the constant is not >> correctly rounded. It should be >> >> 2.0**64 / 10**3 = 1844674407309551.616 (exactly) >> = 1844674407309552 (rounded to nearest int) >> >> but is actually rounded down to a multiple of 1000. >> ... mav@ already fixed the rounding before I wrote that :-). He also changed some (uint64_t)1's to use the long long abomination :-(. > Thanks for the detailed answer. :) Bruce From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 16:55:58 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 94BE0B43; Sat, 15 Dec 2012 16:55:58 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) by mx1.freebsd.org (Postfix) with ESMTP id C63868FC12; Sat, 15 Dec 2012 16:55:57 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id 12so1892022wgh.31 for ; Sat, 15 Dec 2012 08:55:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :content-type:content-transfer-encoding; bh=eoauxAZqa62RhIFNayvyKCrYwtGy06dZzfaaGxohpwM=; b=NRZVmX0JUw8fnTMsl8B3ofM4BBz4tyWl6sJv8DzcGtAmYGHF5ic+yOkLLvx/or3rUE b3Bn7lt3J3ZsJZTTWheUyQeW6XJWi43Jretz/XLj68I79EU+D5A+6/p3YpxZkDe8cfrJ PecX3se7d5G5KJB+CZdnnkk41B9UhNEpaRZdm0XgyTSRULHVKtyxTpkdnjBMAthOWF9Q wVWAPQueyU0wSHKdN41Q74XPNb39cECbDWrcQJ8OyP/rTP4lywp+MheWJF16aIeu2ILP boVFaBNmNs2kkBvebAVUWQGtljQMNE+8ITc7Fx0NJhrmJ6pVUKd0hImsWa7OqQL37Vyf IaNA== Received: by 10.181.13.75 with SMTP id ew11mr8133537wid.9.1355590556876; Sat, 15 Dec 2012 08:55:56 -0800 (PST) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id t17sm3281287wiv.6.2012.12.15.08.55.54 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 15 Dec 2012 08:55:56 -0800 (PST) Sender: Alexander Motin Message-ID: <50CCAB99.4040308@FreeBSD.org> Date: Sat, 15 Dec 2012 18:55:53 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120628 Thunderbird/13.0.1 MIME-Version: 1.0 To: FreeBSD Current , freebsd-arch@freebsd.org Subject: Re: [RFC/RFT] calloutng Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: Davide Italiano X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 16:55:58 -0000 Hi. I'm sorry to interrupt review, but as usual good ideas came during the final testing, causing another round. :) Here is updated patch for HEAD, that includes several new changes: http://people.freebsd.org/~mav/calloutng_12_15.patch The new changes are: -- Precision and event aggregation code was reworked. Instead of previous -prec/+prec representation, precision is now single-sided -- -0/+prec. It allowed to significantly improve precision on long time intervals for APIs which imply that event should not happen before the specified time. Depending on CPU activity, mistake for long time intervals now will never be more then 1-500ms, even if specified precision allows more. -- Some minor optimizations were made to reduce callout overhead and latency by 1.5-2us. Now on Core2Duo amd64 system with LAPIC eventtimer and TSC timecounter usleep(1) call from user-level executes in just 5-6us, instead of 7-8us before. Now it can do 180K cycles per second on single CPU with only partial CPU load. -- Number of kernel subsystems (dcons, syscons, yarrow, led, atkbd, setrlimit) were modified to reduce number of interrupts, also with event aggregation by explicit specification of the acceptable events precision. Now my Core2Duo test system has only 30 interrupts per second in idle. If not remaining syscons events, it could easily be 15. My IvyBridge ultrabook first time in its history shown 5.5 hours of battery time with full screen brightness and 10 hours with lid closed. -- Some kernel functions were added to make KPIs more complete. I've successfully tested this patch on amd64 and arm. -- Alexander Motin From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 18:05:37 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E2C21256; Sat, 15 Dec 2012 18:05:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) by mx1.freebsd.org (Postfix) with ESMTP id E3A8F8FC12; Sat, 15 Dec 2012 18:05:36 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id hj13so1151423wib.13 for ; Sat, 15 Dec 2012 10:05:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=yI89ZhXmodb2CDrycrx11CGIJo58aoQ0396euQN7jd4=; b=Zrrej/HM2rPuyEwwvqhf1igRhK60nQSuJ/DPvErHN8krQkaDUiA0cznkFYtdGrx9t6 C1Mm3mx66Rz4otinllUioCLAWN5hwrmnonFpn0J6JeS2p+ZwSvJRGFYAxyIhMY6saPPQ lA8KzTJlA76WJ91YFJYKI1ptnIB7RpaBPsqErq4lfZX43b98yOUj/LvgoTtPrjHxyWLN XvJtoiqpx79hgFO4C21wbPjvyzA4K5Yqo1hryCxw8ddGQiOwjc9EGombsjqKqzEODqMe v9J3zIC/Q1NMHhPeVYh6bRA+6BBaJRp4M/5CN9jv2KqgoG/tzs7kjXzxXRdyRCoIrcd7 PIJg== MIME-Version: 1.0 Received: by 10.180.24.4 with SMTP id q4mr8289579wif.19.1355594735846; Sat, 15 Dec 2012 10:05:35 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.217.57.9 with HTTP; Sat, 15 Dec 2012 10:05:35 -0800 (PST) In-Reply-To: <50CCAB99.4040308@FreeBSD.org> References: <50CCAB99.4040308@FreeBSD.org> Date: Sat, 15 Dec 2012 10:05:35 -0800 X-Google-Sender-Auth: cbYU2j5_y0TgoBYMR87_RQx4cRI Message-ID: Subject: Re: [RFC/RFT] calloutng From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 Cc: Davide Italiano , FreeBSD Current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 18:05:38 -0000 Hi, Can you please test with MIPS? David has a MIPS board now. Can you also test that various performance tests haven't been affected? Eg, do iperf tests through that MIPS board, configured up as an AP. Please test with a bunch of disk IO activity too. I know this is a lot to ask for, but I'd hate to see some driver / subsystem behaviour change because you didn't quite see the evil way the callout mechanism is used, or how the timer stuff is affecting driver pre-emption. Thanks, Adrian On 15 December 2012 08:55, Alexander Motin wrote: > Hi. > > I'm sorry to interrupt review, but as usual good ideas came during the final > testing, causing another round. :) Here is updated patch for HEAD, that > includes several new changes: > http://people.freebsd.org/~mav/calloutng_12_15.patch > > The new changes are: > -- Precision and event aggregation code was reworked. Instead of previous > -prec/+prec representation, precision is now single-sided -- -0/+prec. It > allowed to significantly improve precision on long time intervals for APIs > which imply that event should not happen before the specified time. > Depending on CPU activity, mistake for long time intervals now will never be > more then 1-500ms, even if specified precision allows more. > -- Some minor optimizations were made to reduce callout overhead and > latency by 1.5-2us. Now on Core2Duo amd64 system with LAPIC eventtimer and > TSC timecounter usleep(1) call from user-level executes in just 5-6us, > instead of 7-8us before. Now it can do 180K cycles per second on single CPU > with only partial CPU load. > -- Number of kernel subsystems (dcons, syscons, yarrow, led, atkbd, > setrlimit) were modified to reduce number of interrupts, also with event > aggregation by explicit specification of the acceptable events precision. > Now my Core2Duo test system has only 30 interrupts per second in idle. If > not remaining syscons events, it could easily be 15. My IvyBridge ultrabook > first time in its history shown 5.5 hours of battery time with full screen > brightness and 10 hours with lid closed. > -- Some kernel functions were added to make KPIs more complete. > > I've successfully tested this patch on amd64 and arm. > > -- > Alexander Motin > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 18:14:39 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 71F0E885; Sat, 15 Dec 2012 18:14:39 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id B9FAA8FC0C; Sat, 15 Dec 2012 18:14:38 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fy27so3498890vcb.13 for ; Sat, 15 Dec 2012 10:14:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=XAaxIesgeYTRBfjwUk8HCNywdiBP33u/wuBmbbbHCY0=; b=aklRXlzDbV+pFLuFHpLGZMxi0SCfsPsntQFWwE1y6Xo2jJgDwQqc7Dw0eCuwhZaZCF 0XJeiuiNjoZ9nWuBthnmAi3/50XgUP3qamt80Nd+y+1N3bOxtrd3XxvC9ha1akmrMEcA 6v5gqf4M2RWn0Yi8mn+nMCHfpEgJb/hTEj2cPMpTuvCz6cnvgqasF5Qg02w4MznxH3vC Wzey3delqwljHfID8eeI2x9XbNAkOaFaL9/FUzTtWsq82RyA00ifZlHhUNFQ1wZUGCdg DGLsxMTOkIC/WDqu1I7iOVM4mYglAZXJVJuxyelgWOL+SgYSb8O94u/FeW3H08cS0X5d wQ0A== MIME-Version: 1.0 Received: by 10.220.154.148 with SMTP id o20mr15374194vcw.54.1355595277325; Sat, 15 Dec 2012 10:14:37 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.245.130 with HTTP; Sat, 15 Dec 2012 10:14:37 -0800 (PST) In-Reply-To: References: <50CCAB99.4040308@FreeBSD.org> Date: Sat, 15 Dec 2012 10:14:37 -0800 X-Google-Sender-Auth: WTlB-VC05qzxBuYU_Dgiq7YU3mY Message-ID: Subject: Re: [RFC/RFT] calloutng From: Davide Italiano To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: Alexander Motin , FreeBSD Current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 18:14:39 -0000 On Sat, Dec 15, 2012 at 10:05 AM, Adrian Chadd wrote: > Hi, > > Can you please test with MIPS? David has a MIPS board now. > Quoting from my first mail -- "We tested the code on amd64, MIPS and arm." I used the board you gave to me. I run only some "basic" tests, but I can look forward running something more complicated in order to track regressions (if any). > Can you also test that various performance tests haven't been > affected? Eg, do iperf tests through that MIPS board, configured up as > an AP. > Can you be more specific? Do we need it configured as AP or this situation can be in some way simulated? > Please test with a bunch of disk IO activity too. > What benckmark/tool do you suggest? iozone? Do you think is better attaching some external drive and run test on that rather than on the flash present on the board? > I know this is a lot to ask for, but I'd hate to see some driver / > subsystem behaviour change because you didn't quite see the evil way > the callout mechanism is used, or how the timer stuff is affecting > driver pre-emption. > I understand your concerns. I'll try to do my best in order to heavily test. Any kind of suggestion is obviously appreciated. > Thanks, > > > Adrian > > On 15 December 2012 08:55, Alexander Motin wrote: >> Hi. >> >> I'm sorry to interrupt review, but as usual good ideas came during the final >> testing, causing another round. :) Here is updated patch for HEAD, that >> includes several new changes: >> http://people.freebsd.org/~mav/calloutng_12_15.patch >> >> The new changes are: >> -- Precision and event aggregation code was reworked. Instead of previous >> -prec/+prec representation, precision is now single-sided -- -0/+prec. It >> allowed to significantly improve precision on long time intervals for APIs >> which imply that event should not happen before the specified time. >> Depending on CPU activity, mistake for long time intervals now will never be >> more then 1-500ms, even if specified precision allows more. >> -- Some minor optimizations were made to reduce callout overhead and >> latency by 1.5-2us. Now on Core2Duo amd64 system with LAPIC eventtimer and >> TSC timecounter usleep(1) call from user-level executes in just 5-6us, >> instead of 7-8us before. Now it can do 180K cycles per second on single CPU >> with only partial CPU load. >> -- Number of kernel subsystems (dcons, syscons, yarrow, led, atkbd, >> setrlimit) were modified to reduce number of interrupts, also with event >> aggregation by explicit specification of the acceptable events precision. >> Now my Core2Duo test system has only 30 interrupts per second in idle. If >> not remaining syscons events, it could easily be 15. My IvyBridge ultrabook >> first time in its history shown 5.5 hours of battery time with full screen >> brightness and 10 hours with lid closed. >> -- Some kernel functions were added to make KPIs more complete. >> >> I've successfully tested this patch on amd64 and arm. >> >> -- >> Alexander Motin >> >> _______________________________________________ >> freebsd-arch@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-arch >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 20:35:16 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 75EE8BBE; Sat, 15 Dec 2012 20:35:16 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0C44E8FC0A; Sat, 15 Dec 2012 20:35:15 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id s9so7586098iec.13 for ; Sat, 15 Dec 2012 12:35:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=J2rqCHuEssg/RRlH1bpsFC4yRbJUO9RlxBCqtMW2kCk=; b=ELpVsTItFCJwhWDbWRxp/RviTZ2fnlIZSfLjw5XcuEArjiw+XpsCnHqsB7mNvh721N U+wvD4WE2w1CIGTpPZbsogM6inVUP06H/x9Sj3lzX6zv8Ll90SP8HJxU+b5EZAu1x3es eUlIM5wFzLTxy3wpXGD5Ykb4PqHipg1XX9leAHY19gaM/H0oRqkfhnhjBswDyd/et5TS ZgEhQeD5YoXdxVqtmjTIHQeE0TKeSR5kmZdVKbZNf5InCJHDzTt8AsLEDBQX57oq4k2o McgBDRaAEfZ0nKKtBHa6Rsh0TXOcbcRP+HbW5Jk8BUhV9FIQkfkgHTaflD8EhOFBzoSW 10xQ== Received: by 10.42.75.6 with SMTP id y6mr7722823icj.30.1355603715468; Sat, 15 Dec 2012 12:35:15 -0800 (PST) Received: from oddish ([66.11.160.25]) by mx.google.com with ESMTPS id x7sm2003527igk.8.2012.12.15.12.35.14 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 15 Dec 2012 12:35:14 -0800 (PST) Date: Sat, 15 Dec 2012 15:34:59 -0500 From: Mark Johnston To: Alexander Motin Subject: Re: [RFC/RFT] calloutng Message-ID: <20121215203458.GA22361@oddish> References: <50CCAB99.4040308@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50CCAB99.4040308@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Davide Italiano , FreeBSD Current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 20:35:16 -0000 On Sat, Dec 15, 2012 at 06:55:53PM +0200, Alexander Motin wrote: > Hi. > > I'm sorry to interrupt review, but as usual good ideas came during the > final testing, causing another round. :) Here is updated patch for > HEAD, that includes several new changes: > http://people.freebsd.org/~mav/calloutng_12_15.patch This patch breaks the libprocstat build. Specifically, the OpenSolaris sys/time.h defines the preprocessor symbols gethrestime and gethrestime_sec. These symbols are also defined in cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h. libprocstat:zfs.c is compiled using include paths that pick up the OpenSolaris time.h, and with this patch _callout.h includes sys/time.h. zfs.c includes taskqueue.h (with _KERNEL defined), which includes _callout.h, so both time.h and zfs_context.h are included in zfs.c, and the symbols are thus defined twice. The patch below fixes the build for me. Another approach might be to include sys/_task.h instead of taskqueue.h at the beginning of zfs.c. Thanks, -Mark diff --git a/lib/libprocstat/zfs.c b/lib/libprocstat/zfs.c index aa6d78e..f8844bf 100644 --- a/lib/libprocstat/zfs.c +++ b/lib/libprocstat/zfs.c @@ -35,6 +35,7 @@ #undef lbolt #undef lbolt64 +#undef gethrestime #undef gethrestime_sec #include #include From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 20:46:13 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DE6CCE8D; Sat, 15 Dec 2012 20:46:13 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5D4C08FC14; Sat, 15 Dec 2012 20:46:13 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id wz12so2820047pbc.13 for ; Sat, 15 Dec 2012 12:46:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=rDDLRFOudkIXCVcGGOqERmHSD2Yu2R5ZRzxPpsD5hcY=; b=R/D7uscuPt+jIvhMHUZdfZW2/Nj/JeIQSPNhie1gHce6AuCDVZIvuCrWLBO9XhpZjn dgbwxjthzc6mqkAKwLMr60MLBpuNG/pWudv+kZivW6eNoYxtIVj0Oi+d8O8ex0WTA24J ybVSvcZXAG92zGNQQt1dgcr/FSZarOjC0iIZ8B3v7bXSdaxrnFUOKs9ZX3dJvPoNhLfe adYd9mYXfDq2sWLrOFQGe7wah+eSArwEUMdwMal58SeKUYDa9oYxl6gsshhUA7pZRkwL XOOuN2w39VFnayDMofPkM/e7AFlT2PZz0LDu35kCySc/MjfRKdxLiZMPsis0oqa9GK6Q 5s6A== Received: by 10.66.88.133 with SMTP id bg5mr27607992pab.21.1355604367814; Sat, 15 Dec 2012 12:46:07 -0800 (PST) Received: from [192.168.20.11] (c-24-19-191-56.hsd1.wa.comcast.net. [24.19.191.56]) by mx.google.com with ESMTPS id rk17sm5226225pbb.3.2012.12.15.12.46.04 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 15 Dec 2012 12:46:06 -0800 (PST) Subject: Re: [RFC/RFT] calloutng Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Garrett Cooper In-Reply-To: <20121215203458.GA22361@oddish> Date: Sat, 15 Dec 2012 12:46:02 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <35705A81-690A-4993-B0C3-C8BC0BC89C67@gmail.com> References: <50CCAB99.4040308@FreeBSD.org> <20121215203458.GA22361@oddish> To: Mark Johnston X-Mailer: Apple Mail (2.1283) Cc: Davide Italiano , Alexander Motin , FreeBSD Current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 20:46:14 -0000 On Dec 15, 2012, at 12:34 PM, Mark Johnston wrote: > On Sat, Dec 15, 2012 at 06:55:53PM +0200, Alexander Motin wrote: >> Hi. >>=20 >> I'm sorry to interrupt review, but as usual good ideas came during = the=20 >> final testing, causing another round. :) Here is updated patch for=20= >> HEAD, that includes several new changes: >> http://people.freebsd.org/~mav/calloutng_12_15.patch >=20 > This patch breaks the libprocstat build. >=20 > Specifically, the OpenSolaris sys/time.h defines the preprocessor > symbols gethrestime and gethrestime_sec. These symbols are also = defined > in cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h. > libprocstat:zfs.c is compiled using include paths that pick up the > OpenSolaris time.h, and with this patch _callout.h includes = sys/time.h. >=20 > zfs.c includes taskqueue.h (with _KERNEL defined), which includes > _callout.h, so both time.h and zfs_context.h are included in zfs.c, = and > the symbols are thus defined twice. >=20 > The patch below fixes the build for me. Another approach might be to > include sys/_task.h instead of taskqueue.h at the beginning of zfs.c. I had a patch open once upon a time to cleanup inclusion of = sys/time.h all over the tree and deal with the sys/time.h <-> time.h = pollution issue, but it got dropped due to lack of interest (20~30 = apps/libs were affected IIRC and I only really got assistance in fixing = the UFS and bsnmpd pieces, and gave up due to lack of response from = maintainers). dtrace/zfs is a definite instigator in this pollution (I = remember nasty cddl/... pollution with the compat sys/time.h header). Bottom line: make sure anything new you're defining isn't = already defined via POSIX or other OSes, and if so please try to make = the implementations match (so that eventual POSIX inclusion might be = possible) and when in doubt I suggest consulting standards@ / brde@. Cheers, -Garrett= From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 20:50:47 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6DA4E351; Sat, 15 Dec 2012 20:50:47 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id D9C948FC0A; Sat, 15 Dec 2012 20:50:46 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fy27so3578976vcb.13 for ; Sat, 15 Dec 2012 12:50:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=J7IG0GsmlLlh6Q10fkGbRhKCrRcIVQfRIM9SxQL8dpg=; b=Fx1g79EIl6LPTrl7O4IotE/ljZ8or4h6gGFZTujxUBWQhT7XazkLqvDZ4RKD9OPA5c N0YZu5M4Cdopxq/zHbKaGGgzWmUJxgYPdlUSeXkWS4QuSM1CoeGlHu9BmkzNyjsFiIa5 5WNh/TOIUsZ3Spx5ZkTxGkuIwplbNy0bDJ9D0HBr/kR7RbTdu65eUGGhruYZywrDbTmH 9S6izpHGhdhVyutURYuVnrMS1DEbXYpBtNzmZYiz/uNDlzYdAzRAUcOtC9BSPRhJQ3lH kyyzTk57uLONpaf06OGucbnPsWcNRoshedHHB8C92Te/Oai2qwIokEc7roXjuz/ash0G 6ZRA== MIME-Version: 1.0 Received: by 10.52.70.46 with SMTP id j14mr13583598vdu.99.1355604645889; Sat, 15 Dec 2012 12:50:45 -0800 (PST) Sender: davide.italiano@gmail.com Received: by 10.58.245.130 with HTTP; Sat, 15 Dec 2012 12:50:45 -0800 (PST) In-Reply-To: <20121215203458.GA22361@oddish> References: <50CCAB99.4040308@FreeBSD.org> <20121215203458.GA22361@oddish> Date: Sat, 15 Dec 2012 12:50:45 -0800 X-Google-Sender-Auth: DcVtRg3QFEp51CzTIX97kavb0O4 Message-ID: Subject: Re: [RFC/RFT] calloutng From: Davide Italiano To: Mark Johnston Content-Type: text/plain; charset=ISO-8859-1 Cc: Alexander Motin , FreeBSD Current , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 20:50:47 -0000 On Sat, Dec 15, 2012 at 12:34 PM, Mark Johnston wrote: > On Sat, Dec 15, 2012 at 06:55:53PM +0200, Alexander Motin wrote: >> Hi. >> >> I'm sorry to interrupt review, but as usual good ideas came during the >> final testing, causing another round. :) Here is updated patch for >> HEAD, that includes several new changes: >> http://people.freebsd.org/~mav/calloutng_12_15.patch > > This patch breaks the libprocstat build. > > Specifically, the OpenSolaris sys/time.h defines the preprocessor > symbols gethrestime and gethrestime_sec. These symbols are also defined > in cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h. > libprocstat:zfs.c is compiled using include paths that pick up the > OpenSolaris time.h, and with this patch _callout.h includes sys/time.h. > > zfs.c includes taskqueue.h (with _KERNEL defined), which includes > _callout.h, so both time.h and zfs_context.h are included in zfs.c, and > the symbols are thus defined twice. > > The patch below fixes the build for me. Another approach might be to > include sys/_task.h instead of taskqueue.h at the beginning of zfs.c. > > Thanks, > -Mark > > diff --git a/lib/libprocstat/zfs.c b/lib/libprocstat/zfs.c > index aa6d78e..f8844bf 100644 > --- a/lib/libprocstat/zfs.c > +++ b/lib/libprocstat/zfs.c > @@ -35,6 +35,7 @@ > > #undef lbolt > #undef lbolt64 > +#undef gethrestime > #undef gethrestime_sec > #include > #include I fixed (or at least workarounded) that issue during the summer. http://svnweb.freebsd.org/base?view=revision&revision=237068 Probably that was lost somewhere. We're going to regenerate a patch, but for now I suggest to patch that manually or to checkout the calloutng project repository. From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 21:03:33 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7ADA2A9F; Sat, 15 Dec 2012 21:03:33 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id AD1C68FC19; Sat, 15 Dec 2012 21:03:32 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id u54so2129547wey.13 for ; Sat, 15 Dec 2012 13:03:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=uVNWanL7oPU4mSpbgzQpla/m8Id0omxxLsJEieKqPhg=; b=p+ZLMK8hO6kLuGhgS/qEkJ0SffXPxe/CuzN8ZUgb+ON8H8dtzGz7SA+AIhb/hQKoyY oQOc+9UrXPo48ZG0MLOqYi6W3W8I5hvlbM22T/5J1gIjbodm6zw5jQM3B74SuI/4GyiM oV4nT2cM73/UGxPdzZ4RI/Hhh+j7zA38Zw14ROldVdNIrGdeXy1yGAe+KwOtDN3KGd7+ ugiyGTRGoAkNh8UIQOE7BfrBnY59i4vhYOFFDZ/W1dO1l42raqKA0VnLOO/vaQEmiQxW qnp90iyghSkAWN9wXvc6EhNFNL4uYHA/cAUR2Fv8ug9C6k6D8uOFZ4UofTf/auvjvJK8 KApw== Received: by 10.180.88.138 with SMTP id bg10mr8703142wib.13.1355605411525; Sat, 15 Dec 2012 13:03:31 -0800 (PST) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id cf6sm4316001wib.3.2012.12.15.13.03.29 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 15 Dec 2012 13:03:30 -0800 (PST) Sender: Alexander Motin Message-ID: <50CCE59F.1080107@FreeBSD.org> Date: Sat, 15 Dec 2012 23:03:27 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120628 Thunderbird/13.0.1 MIME-Version: 1.0 To: Davide Italiano Subject: Re: [RFC/RFT] calloutng References: <50CCAB99.4040308@FreeBSD.org> <20121215203458.GA22361@oddish> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org, FreeBSD Current , Mark Johnston X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 21:03:33 -0000 On 15.12.2012 22:50, Davide Italiano wrote: > On Sat, Dec 15, 2012 at 12:34 PM, Mark Johnston wrote: >> On Sat, Dec 15, 2012 at 06:55:53PM +0200, Alexander Motin wrote: >>> I'm sorry to interrupt review, but as usual good ideas came during the >>> final testing, causing another round. :) Here is updated patch for >>> HEAD, that includes several new changes: >>> http://people.freebsd.org/~mav/calloutng_12_15.patch >> >> This patch breaks the libprocstat build. >> >> Specifically, the OpenSolaris sys/time.h defines the preprocessor >> symbols gethrestime and gethrestime_sec. These symbols are also defined >> in cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h. >> libprocstat:zfs.c is compiled using include paths that pick up the >> OpenSolaris time.h, and with this patch _callout.h includes sys/time.h. >> >> zfs.c includes taskqueue.h (with _KERNEL defined), which includes >> _callout.h, so both time.h and zfs_context.h are included in zfs.c, and >> the symbols are thus defined twice. >> >> The patch below fixes the build for me. Another approach might be to >> include sys/_task.h instead of taskqueue.h at the beginning of zfs.c. > > I fixed (or at least workarounded) that issue during the summer. > http://svnweb.freebsd.org/base?view=revision&revision=237068 > Probably that was lost somewhere. We're going to regenerate a patch, > but for now I suggest to patch that manually or to checkout the > calloutng project repository. Sorry, it's my fault. I've tried to save some time on patch generation and forgot about that change in lib/. We haven't touched user-level in our work except that file. Here is patch with that chunk added: http://people.freebsd.org/~mav/calloutng_12_15_1.patch -- Alexander Motin From owner-freebsd-arch@FreeBSD.ORG Sat Dec 15 21:13:31 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B3C241D0; Sat, 15 Dec 2012 21:13:31 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id D5A298FC13; Sat, 15 Dec 2012 21:13:30 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id u54so2131715wey.13 for ; Sat, 15 Dec 2012 13:13:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=zlGgyR5dR9vhntoD3Pa0ZBhySFyS8xxNaaXEj9GspIc=; b=gpWImdjc6HcvMWu+SRVYI3jfgZxT6C3x/5K3C/Bo3cUvD62UUefxWLe0NIJ1gCQZBq EAj/kPL6EEZmj2ZY6TzVvdAmPVzOi6lxJ47wrFfilVygYE6CxLr2bTt+DbSCuMGGOUXV 96FfaqXlqFHiQViF+BuJQhbL3QQARTx1u8lbH+7lOAnj1uwsxPQDyWbPlt++dqULb5Zq IWVPAynLeDvNh0I4u6YNqG3QjyOKlV/ty2bbfW+mxV9eX7LDYEbPmsqCQkMdm2KCoEmJ ItOeP4w4pfCBnumTQlSNv9XCL2gEV7sd4fmPRQPGhJuEjihurZjy1L/oE7WXeiDorl+3 J07Q== X-Received: by 10.180.87.39 with SMTP id u7mr8719439wiz.6.1355606009701; Sat, 15 Dec 2012 13:13:29 -0800 (PST) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id i2sm3895838wiw.3.2012.12.15.13.13.27 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 15 Dec 2012 13:13:28 -0800 (PST) Sender: Alexander Motin Message-ID: <50CCE7F6.2090505@FreeBSD.org> Date: Sat, 15 Dec 2012 23:13:26 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120628 Thunderbird/13.0.1 MIME-Version: 1.0 To: Davide Italiano Subject: Re: [RFC/RFT] calloutng References: <50CCAB99.4040308@FreeBSD.org> <20121215203458.GA22361@oddish> <50CCE59F.1080107@FreeBSD.org> In-Reply-To: <50CCE59F.1080107@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org, FreeBSD Current , Mark Johnston X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Dec 2012 21:13:31 -0000 On 15.12.2012 23:03, Alexander Motin wrote: > On 15.12.2012 22:50, Davide Italiano wrote: >> On Sat, Dec 15, 2012 at 12:34 PM, Mark Johnston >> wrote: >>> On Sat, Dec 15, 2012 at 06:55:53PM +0200, Alexander Motin wrote: >>>> I'm sorry to interrupt review, but as usual good ideas came during the >>>> final testing, causing another round. :) Here is updated patch for >>>> HEAD, that includes several new changes: >>>> http://people.freebsd.org/~mav/calloutng_12_15.patch >>> >>> This patch breaks the libprocstat build. >>> >>> Specifically, the OpenSolaris sys/time.h defines the preprocessor >>> symbols gethrestime and gethrestime_sec. These symbols are also defined >>> in cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h. >>> libprocstat:zfs.c is compiled using include paths that pick up the >>> OpenSolaris time.h, and with this patch _callout.h includes sys/time.h. >>> >>> zfs.c includes taskqueue.h (with _KERNEL defined), which includes >>> _callout.h, so both time.h and zfs_context.h are included in zfs.c, and >>> the symbols are thus defined twice. >>> >>> The patch below fixes the build for me. Another approach might be to >>> include sys/_task.h instead of taskqueue.h at the beginning of zfs.c. >> >> I fixed (or at least workarounded) that issue during the summer. >> http://svnweb.freebsd.org/base?view=revision&revision=237068 >> Probably that was lost somewhere. We're going to regenerate a patch, >> but for now I suggest to patch that manually or to checkout the >> calloutng project repository. > > Sorry, it's my fault. I've tried to save some time on patch generation > and forgot about that change in lib/. We haven't touched user-level in > our work except that file. Here is patch with that chunk added: > http://people.freebsd.org/~mav/calloutng_12_15_1.patch And one more part I've missed is manual pages update, that probably needs more improvements: http://people.freebsd.org/~mav/calloutng_12_15.man.patch -- Alexander Motin