Date: Wed, 21 Feb 2018 15:54:23 +0000 (UTC) From: Ed Maste <emaste@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r329723 - head/usr.bin/ministat Message-ID: <201802211554.w1LFsN3m043128@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: emaste Date: Wed Feb 21 15:54:23 2018 New Revision: 329723 URL: https://svnweb.freebsd.org/changeset/base/329723 Log: ministat: disallow negative variance / nan Stddev With all values identical it was possible for Var() to return a negative value due to limited floating point precision, resulting in "nan" reported as Stddev. Variance cannot actually be negative, so just return 0. We can later investigate alternate algorithms for calculating variance to reduce the effect of catastrophic cancellation here. Reported by: Arshan Khanifar <arshankhanifar_gmail.com> Approved by: phk Sponsored by: The FreeBSD Foundation Modified: head/usr.bin/ministat/ministat.c Modified: head/usr.bin/ministat/ministat.c ============================================================================== --- head/usr.bin/ministat/ministat.c Wed Feb 21 15:51:48 2018 (r329722) +++ head/usr.bin/ministat/ministat.c Wed Feb 21 15:54:23 2018 (r329723) @@ -208,6 +208,12 @@ static double Var(struct dataset *ds) { + /* + * Due to limited precision it is possible that sy^2/n > syy, + * but variance cannot actually be negative. + */ + if (ds->syy <= ds->sy * ds->sy / ds->n) + return (0); return (ds->syy - ds->sy * ds->sy / ds->n) / (ds->n - 1.0); }
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201802211554.w1LFsN3m043128>