From owner-freebsd-numerics@freebsd.org Tue Feb 13 06:25:31 2018 Return-Path: Delivered-To: freebsd-numerics@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E7B6F02165 for ; Tue, 13 Feb 2018 06:25:31 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: from mail-yw0-x231.google.com (mail-yw0-x231.google.com [IPv6:2607:f8b0:4002:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E834F86C57 for ; Tue, 13 Feb 2018 06:25:27 +0000 (UTC) (envelope-from lists@eitanadler.com) Received: by mail-yw0-x231.google.com with SMTP id z82so5212418ywb.1 for ; Mon, 12 Feb 2018 22:25:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:from:date:message-id:subject:to; bh=7wYj1+TjJxOGS9ia+D9Hu6zbNR2MnAAIeIymhia0lAE=; b=D9NgudLJxmFFI+MIlhVGfdH86NXFZYKt/VnPZLMXPmCrFbcJY+HTOviUx3qX2IaZFW OAHUlfGehFLAeHVD5eSo8C2Ma03GaTPvlR0/kY8app4RT3Ht7Y8P58iVmUdDKo20lxFM iouieul6DJPy87qsm0LaBHaJMD8Kj9P4h/+eY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=7wYj1+TjJxOGS9ia+D9Hu6zbNR2MnAAIeIymhia0lAE=; b=YCxMH4iuyvEH4yYG1YqHKz4Kf82MYnbBpDVRBB+9xM8CC4t/9RR9mEfv+K3GqBggwq Zl5xctVAeXNXKrnT2VLZWmTfSJFfceDWKFTdPyJYfh6zuGlRqnaGDsEcdxBPLJLSU1MZ Tz3Zi728rSauU1zmfEtCpP/BL9jaKNZzlP2Rhvy7h3K0+LwUIIeABkHM+lzvc3SziA2E CCui/pAjERliH99safPHCNk++VRbNRKVpatgdmz3c3CEo54n+H05MqT7+l6LgBtv9DDA 8hj+YfrLAWCPxjTIgY76vJm1w4+e8gnwd/O+R1PU2sgVlPgNuHsqOKapfxSUkDoQ5O/7 qf3w== X-Gm-Message-State: APf1xPAYbsCXgEHkau1lMGX6q328v2/VqvMPlzpNNUJprZHn7N95jBeq uVWk8NAaGiRbPcZOXK18WaKTaQGmtutTOfObzlK7+YZn X-Google-Smtp-Source: AH8x227V05yNM2MAultd/PMFOMch2/x9z+C5Ry1TzyA38uK6uVnWADP3Bm5zTDu7rYeN0yWBgQ2oeFENGmlejfyC8ao= X-Received: by 10.37.38.8 with SMTP id m8mr83783ybm.97.1518503126819; Mon, 12 Feb 2018 22:25:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.113.7 with HTTP; Mon, 12 Feb 2018 22:24:56 -0800 (PST) From: Eitan Adler Date: Mon, 12 Feb 2018 22:24:56 -0800 Message-ID: Subject: signed overflow in atan2 To: freebsd-numerics@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-numerics@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussions of high quality implementation of libm functions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2018 06:25:31 -0000 source: https://github.com/freebsd/freebsd/pull/130 `atan2(y, x)`'s special case for `x == 1.0` (in which case the result must be `atan(y)`) is unnecessarily complicated #130 As a component of atan2(y, x), the case of x == 1.0 is farmed out to atan(y). The current implementation of this comparison is vulnerable to signed integer underflow (that is, undefined behavior), and it's performed in a somewhat more complicated way than it need be. Change it to not be quite so cute, rather directly comparing the high/low bits of x to the specific IEEE-754 bit pattern that encodes 1.0. Note that while there are three different e_atan* files in the relevant directory, only this one needs fixing. e_atan2f.c already compares against the full bit pattern encoding 1.0f, while e_atan2l.cuses bitwise-ands/ors/nots and so doesn't require a change. (I ran into this in Mozilla's SpiderMonkey embedding of fdlibm, where a test of the behavior of Math.atan2(1, -0) triggered clang's signed-overflow-sanitizer runtime error.) ----- patch: if(((ix|((lx|-lx)>>31))>0x7ff00000)|| ((iy|((ly|-ly)>>31))>0x7ff00000)) /* x or y is NaN */ return x+y; - if((hx-0x3ff00000|lx)==0) return atan(y); /* x=1.0 */ + if(hx==0x3ff00000&&lx==0) return atan(y); /* x=1.0 */ m = ((hy>>31)&1)|((hx>>30)&2); /* 2*sign(x)+sign(y) */ /* when y = 0 */ ---- Is this correct? Any objections to be committing ? -- Eitan Adler