From owner-freebsd-numerics@freebsd.org Wed Nov 20 20:06:32 2019 Return-Path: Delivered-To: freebsd-numerics@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 237B41C0AEF for ; Wed, 20 Nov 2019 20:06:32 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47JDGs14PPz4HQP; Wed, 20 Nov 2019 20:06:28 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.15.2/8.15.2) with ESMTPS id xAKK6K07061279 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 20 Nov 2019 12:06:20 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.15.2/8.15.2/Submit) id xAKK6KUc061278; Wed, 20 Nov 2019 12:06:20 -0800 (PST) (envelope-from sgk) Date: Wed, 20 Nov 2019 12:06:20 -0800 From: Steve Kargl To: Jeff Walden Cc: Dimitry Andric , freebsd-numerics@freebsd.org Subject: Re: UB in various hypot() implementations (left-shifting a negative, number) Message-ID: <20191120200620.GA61176@troutmask.apl.washington.edu> Reply-To: sgk@troutmask.apl.washington.edu References: <20191116211419.GA40056@troutmask.apl.washington.edu> <11f139aa-708a-4799-dfee-1a5b988a3c0b@mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <11f139aa-708a-4799-dfee-1a5b988a3c0b@mit.edu> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47JDGs14PPz4HQP X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of sgk@troutmask.apl.washington.edu has no SPF policy when checking 128.95.76.21) smtp.mailfrom=sgk@troutmask.apl.washington.edu X-Spamd-Result: default: False [-1.03 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[sgk@troutmask.apl.washington.edu]; NEURAL_HAM_MEDIUM(-0.85)[-0.854,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[washington.edu]; REPLYTO_ADDR_EQ_FROM(0.00)[]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-0.86)[-0.859,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.22)[ip: (0.06), ipnet: 128.95.0.0/16(-0.28), asn: 73(-0.83), country: US(-0.05)]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-numerics@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussions of high quality implementation of libm functions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Nov 2019 20:06:32 -0000 On Mon, Nov 18, 2019 at 09:55:03PM -0800, Jeff Walden wrote: > On 11/16/19 1:14 PM, Steve Kargl wrote: > > Well, clearly, the patch to e_hypotl.c is wrong. It > > clear the significand when t1 = 0 whereas t1 = 1 leaves > > one bit set in the significand. Simply looking at the > > value of t1 under a poor man's debugger shows the > > difference. Adding "printf("%Le %La\n", t1, t1);" after > > the SET_HIGH_WORD gives > > > > 2.962347e-2493 0x1p-8280 <-- t1 = 1 > > 0.000000e+00 0x1p-8280 <-- t1 = 0 > > > > for hypotl(ldexpl(1.1,-16000), ldexpl(2.1, -16000)). > > Hrm. My understanding based on an earlier line in e_hypotl.c > was that that change should be identical, but something mildly > subtle is going on and my understanding of |long double| is not > quite as good as I thought it was. I've removed that file's > changes and left just the original two in, then resubmitted the diff. > > Jeff In general, float, double, and long double all use the same general approach where a value is scaled if required. The details of unraveling the scaling do differ. float operates on the 32-bit integer representation of the float value. double operates on the 32-bit integer portion of the double that contains the exponent. For at least ld80 long double, the macro SET_HIGH_WORD assigns directly to the bitfield of struct within a union. The bitfield represents the sign and exponent. So, setting t1=0 clears the significand, which has an undesirable effect. -- Steve