From owner-freebsd-arch@freebsd.org Wed Jan 9 21:26:28 2019 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4DAD1490DFB for ; Wed, 9 Jan 2019 21:26:27 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from shxd.cx (mail.shxd.cx [64.201.244.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 769DB7726A; Wed, 9 Jan 2019 21:26:26 +0000 (UTC) (envelope-from devin@shxd.cx) Received: from [74.217.198.10] (port=51585 helo=[10.1.7.18]) by shxd.cx with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1ghLMm-00017p-GI; Wed, 09 Jan 2019 13:26:24 -0800 From: Devin Teske Message-Id: Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: GNU-compatible, BSD-licensed bc Date: Wed, 9 Jan 2019 13:26:23 -0800 In-Reply-To: Cc: Devin Teske , "freebsd-arch@freebsd.org" To: Gavin Howard References: <8FFA4578-0BAE-4F9F-8A06-AE83283BDEA4@FreeBSD.org> <61F802DC-2E59-4E0A-955D-899EBD7874A1@FreeBSD.org> X-Mailer: Apple Mail (2.3445.9.1) Sender: devin@shxd.cx X-Rspamd-Queue-Id: 769DB7726A X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [4.27 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MX_GOOD(-0.01)[cached: mail.shxd.cx]; FREEMAIL_TO(0.00)[gmail.com]; FORGED_SENDER(0.30)[dteske@FreeBSD.org,devin@shxd.cx]; IP_SCORE(0.23)[ip: (0.65), ipnet: 64.201.240.0/20(0.32), asn: 36734(0.26), country: US(-0.08)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:36734, ipnet:64.201.240.0/20, country:US]; MIME_TRACE(0.00)[0:+,1:+]; FROM_NEQ_ENVFROM(0.00)[dteske@FreeBSD.org,devin@shxd.cx]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(0.89)[0.889,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[FreeBSD.org]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.47)[0.465,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.99)[0.994,0]; R_SPF_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jan 2019 21:26:28 -0000 > On Jan 9, 2019, at 12:47 PM, Gavin Howard = wrote: >=20 > On Wed, Jan 9, 2019 at 12:58 PM Devin Teske > wrote: >>=20 >>=20 >>=20 >> On Jan 7, 2019, at 11:45 PM, Gavin Howard = wrote: >>=20 >> On Mon, Jan 7, 2019 at 10:57 PM Devin Teske = wrote: >>=20 >>=20 >>=20 >>=20 >> On Jan 7, 2019, at 4:42 PM, Gavin Howard = wrote: >>=20 >> On Mon, Jan 7, 2019 at 5:38 PM Conrad Meyer wrote: >>=20 >>=20 >> On Mon, Jan 7, 2019 at 4:00 PM Devin Teske = wrote: >>=20 >> How do you handle arbitrary arithmetic precision? >>=20 >>=20 >> Looks like https://github.com/gavinhoward/bc/blob/master/src/num.c . >>=20 >>=20 >> That is correct. Because this bc is meant to help bootstrap the Linux >> kernel and have no dependencies other than POSIX 2008, I wrote my = own. >>=20 >>=20 >> Impressive. It might be worth turning this into a library itself. >>=20 >>=20 >> Eh...it is completely tuned for bc. And it won't be fast, >> unfortunately. See below. >>=20 >> Also, the POSIX bc standard mandates doing math in decimal. OpenSSL >> would not be smart if they did that. >>=20 >>=20 >> Not sure I understand what you mean here. >>=20 >>=20 >> Well, for starters, OpenSSL's BIGNUM is integer only. Yes, those >> integers can be any size, but they are only integers. That is not = good >> enough for bc; it has to allow arbitrary precision, including >> non-integers, and its fractional part can be any size, up to a = certain >> limit, which you can get from my bc by typing "limits" at the prompt >> and looking for the value of BC_NUM_MAX (which is actually the = maximum >> number of decimal digits, period). >>=20 >>=20 >> Thanks for explaining that further. >>=20 >> [snip] >>=20 >> There are also a few >> peculiarities with the POSIX bc standard that (more or less) require = a >> standalone implementation. >>=20 >>=20 >> How hard would it be to convert a bn(3)-based library to use your = code? >> Would there be any performance impact -- I've benchmarked bn(3) to >> be pretty fast. >>=20 >>=20 >> It would not be terribly hard, but as I said above, it would not be >> fast at all, at least compared to a well-written hardware-based, >> binary bignum implementation. But if you want to, go ahead; I would >> appreciate the credit (though the license does not even require = that). >>=20 >>=20 >> Well, unfortunately, my needs are purely whole-integer arithmetic but >> speed is paramount. >>=20 >> My application of OpenSSL bn(3) is here: >> https://reviews.freebsd.org/D16132 >>=20 >> Though worth noting that I haven't updated the review since November. >> Since then, I have made many changes which can be seen on GitHub: >> https://FrauBSD.org/libcmb >>=20 >>=20 >>=20 >> Also, right now I am working on getting a release candidate out that >> will enable me to make a quick port that Stefan could use as a = jumping >> off point. My build system changed between 1.0 and now, and I would >> like to be able to test it. >>=20 >>=20 >> Cool. Looking forward to it. >>=20 >>=20 >> It's out. It works great. The Makefile that I sent to the mailing = list >> a few messages back does the job well enough, though I was told in a >> private message not to use the GNU bc port's pkg-descr file, since it >> might be under the GPL. >>=20 >> However, note that this is not the final 1.1 release; it is just for >> testing, even though it is high quality. >>=20 >>=20 >> I'm wondering why you chose to write your own configure.sh instead of >> leveraging autotools. >=20 > I had a goal of absolutely zero dependencies (that is one reason why I > imported and adapted an existing history implementation instead of > just making readline or editline an optional dependency). That was > also why I wrote a custom parser, even though it is complicated, > rather than using Lex/Flex and YACC/Bison. Autotools would be a > dependency, so I wrote a custom one. >=20 >> Also, it looks like you have a high number of build-time options. I = also >> notice that you're into writing tests for your software. It might be = interesting >> to apply my tool for combining all possible combinations of build = options. >>=20 >> Seen here: >> = https://github.com/FrauBSD/pkgcenter/blob/master/depend/libcmb/release/Mak= efile = >>=20 >> It's a great way to make sure all the various build options work = together. >> -- >> Cheers, >> Devin >=20 > Thank you. It looks interesting, though unfortunately, some of my > build options are exclusive (for example, the `-b` and `-d` flags to > the `configure.sh` script cannot both be used at the same time), and > from a cursory glance, I can't tell if libcmb can handle them. >=20 You likely would not use libcmb directly in the build process but rather the cmb utility to provide your combinations. For example, taking into account that -b and -d are exclusive: cmb -e -- -E -g -G -H "-k 64" "-O3" -S Produces 128 lines, each line representing a unique combination of the = above options. You could, for each unique combination, then perform one as-is, one with = "-d", one with "-B", etc. Latest version of cmb is here: https://FrauBSD.org/cmb Which depends on libcmb: https://FrauBSD.org/libcmb > I do have a way of testing all of the (valid) build option > combinations in `release.sh`, which is a script I run before every > release. It tests that it compiles without error and/or warnings on > gcc and clang, in debug, release, reldebug, and min size modes, > running the test suite on every build. release.sh could be enhanced to support more than 16 variants in = runtestseries() I have a similar release process for libcmb/cmb (previously linked) = except instead of just -Wall, I am a glutton for punishment by using with = clang: -Weverything -Werror > It also runs the test suite > under ASan, UBSan, MSan, and Valgrind. (I think it runs 532 builds and > does the test suite for all of them.) It takes 5 hours 20 minutes to > run on my fast, custom-built machine. >=20 Impressive! > And then on top of that, I have a script to generate random math > problems (`tests/randmath.py`), which I run for about 10 million > iterations. And then I run the afl fuzzer for at least 100 cycles on > both calculators, fixing every single crash it finds. I am doing the > latter two right now, and if it passes, the release will be out right > after that. >=20 Sweet, looking forward. Excellent work. --=20 Devin=