From owner-svn-src-head@freebsd.org  Wed Feb 21 15:54:23 2018
Return-Path: <owner-svn-src-head@freebsd.org>
Delivered-To: svn-src-head@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE95FF0FA9D;
 Wed, 21 Feb 2018 15:54:23 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id 715F682CB9;
 Wed, 21 Feb 2018 15:54:23 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 685301A98A;
 Wed, 21 Feb 2018 15:54:23 +0000 (UTC)
 (envelope-from emaste@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w1LFsNZC043129;
 Wed, 21 Feb 2018 15:54:23 GMT (envelope-from emaste@FreeBSD.org)
Received: (from emaste@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id w1LFsN3m043128;
 Wed, 21 Feb 2018 15:54:23 GMT (envelope-from emaste@FreeBSD.org)
Message-Id: <201802211554.w1LFsN3m043128@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: emaste set sender to
 emaste@FreeBSD.org using -f
From: Ed Maste <emaste@FreeBSD.org>
Date: Wed, 21 Feb 2018 15:54:23 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-head@freebsd.org
Subject: svn commit: r329723 - head/usr.bin/ministat
X-SVN-Group: head
X-SVN-Commit-Author: emaste
X-SVN-Commit-Paths: head/usr.bin/ministat
X-SVN-Commit-Revision: 329723
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-head@freebsd.org
X-Mailman-Version: 2.1.25
Precedence: list
List-Id: SVN commit messages for the src tree for head/-current
 <svn-src-head.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/svn-src-head>,
 <mailto:svn-src-head-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-head/>
List-Post: <mailto:svn-src-head@freebsd.org>
List-Help: <mailto:svn-src-head-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/svn-src-head>,
 <mailto:svn-src-head-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Feb 2018 15:54:23 -0000

Author: emaste
Date: Wed Feb 21 15:54:23 2018
New Revision: 329723
URL: https://svnweb.freebsd.org/changeset/base/329723

Log:
  ministat: disallow negative variance / nan Stddev
  
  With all values identical it was possible for Var() to return a negative
  value due to limited floating point precision, resulting in "nan"
  reported as Stddev.
  
  Variance cannot actually be negative, so just return 0.  We can later
  investigate alternate algorithms for calculating variance to reduce the
  effect of catastrophic cancellation here.
  
  Reported by:	Arshan Khanifar <arshankhanifar_gmail.com>
  Approved by:	phk
  Sponsored by:	The FreeBSD Foundation

Modified:
  head/usr.bin/ministat/ministat.c

Modified: head/usr.bin/ministat/ministat.c
==============================================================================
--- head/usr.bin/ministat/ministat.c	Wed Feb 21 15:51:48 2018	(r329722)
+++ head/usr.bin/ministat/ministat.c	Wed Feb 21 15:54:23 2018	(r329723)
@@ -208,6 +208,12 @@ static double
 Var(struct dataset *ds)
 {
 
+	/*
+	 * Due to limited precision it is possible that sy^2/n > syy,
+	 * but variance cannot actually be negative.
+	 */
+	if (ds->syy <= ds->sy * ds->sy / ds->n)
+		return (0);
 	return (ds->syy - ds->sy * ds->sy / ds->n) / (ds->n - 1.0);
 }