From owner-svn-src-all@freebsd.org Thu Dec 31 07:15:05 2015 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70502A56061; Thu, 31 Dec 2015 07:15:05 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 323321AF2; Thu, 31 Dec 2015 07:15:04 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id B2FBCD2AA; Thu, 31 Dec 2015 07:15:03 +0000 (UTC) Subject: Re: svn commit: r292955 - head/lib/libmd To: Bruce Evans References: <201512301804.tBUI4oGp065466@repo.freebsd.org> <20151231115651.R995@besplex.bde.org> <20151231143314.Y1520@besplex.bde.org> Cc: "Jonathan T. Looney" , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org From: Allan Jude X-Enigmail-Draft-Status: N1110 Message-ID: <5684D606.3080609@freebsd.org> Date: Thu, 31 Dec 2015 02:15:18 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <20151231143314.Y1520@besplex.bde.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cefVxOIBD2vm4RApBE5O4tQ0lfigJF2Qm" X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2015 07:15:05 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --cefVxOIBD2vm4RApBE5O4tQ0lfigJF2Qm Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-12-30 23:10, Bruce Evans wrote: > On Thu, 31 Dec 2015, Bruce Evans wrote: >=20 >> >> wc /proc/0/* works. md5 works like wc using a hack to avoid calling t= his >> broken function. E.g., >> >> for i in $(ls /proc/0/*); do echo -n "$i: "; md5 <$i; done # gives= >> >> /proc/0/cmdline: 3c5896b1ac441f4998f052e2126e8d20 >> /proc/0/ctl: d41d8cd98f00b204e9800998ecf8427e >> /proc/0/etype: 674441960ca1ba2de08ad4e50c9fde98 >> /proc/0/rlimit: 67d6ad67b412e1bceb7cb508a3492197 >> /proc/0/status: 3ccc3067b97c3232ea2dbcb64c458fd4 >=20 > Further examples: >=20 > md5 # on terminal input >=20 > works correctly by not using MDXFileChunk(). >=20 > md5 /dev/stdin # on the same terminal input >=20 > produces the d41d8cd98f00b204e9800998ecf8427e garbage using > MDXFileChunk(). truss shows that lseek() is broken too -- MDXFileChunk= () > tries it and it succeeds on the unseekable file /dev/stdin. Bugs in th= is > area are common. E.g., lseek() on named pipes was broken so that it > succeeded, by rearranging the plumbing use fileops more or less and not= > attaching lseek right. >=20 > cat | md5 # on the same terminal input >=20 > works correctly by not using MDXFileChunk(). >=20 > cat | md5 /dev/stdin # on the same terminal input >=20 > doesn't work correctly, but it fails better than without the pipe. Now= > a seek error occurs and is reported as "md5: /dev/stdin: Illegal seek" > (I can't see where it is reported). Then md5 exits. Then cat waits to= > read input. Then cat fails to write output and is killed by SIGPIPE. > So md5 handled the seek error in a fail-safe though incorrect way. One= > correct way is to fall back to the working code, but it is better to > just use that without an lseek check. >=20 > It is a bug that [l]stat() on /dev/stdin sees the device file and not t= he > actual stdin file. md5 can't work around this except by not using stat= (). >=20 > Bruce >=20 It seems these problems also slow things down, a lot: # time md5 /media/md5test/bigdata MD5 (/media/md5test/bigdata) =3D 6afad0bf5d8318093e943229be05be67 4.310u 3.476s 0:07.79 99.8% 20+167k 0+0io 0pf+0w # time env LD_PRELOAD=3D/usr/obj/media/svn/md5/head/tmp/lib/libmd.so /usr/obj/media/svn/md5/head/sbin/md5/md5 /media/md5test/bigdata MD5 (/media/md5test/bigdata) =3D 6afad0bf5d8318093e943229be05be67 4.133u 0.354s 0:04.49 99.7% 20+167k 1+0io 0pf+0w (file is fully cached in ZFS ARC, dd reads it at 11GB/s) Will investigate more tomorrow. char * MDXFile(const char *filename, char *buf) { unsigned char buffer[16*1024]; MDX_CTX ctx; int f, i; MDXInit(&ctx); f =3D open(filename, O_RDONLY); if (f < 0) return 0; i =3D 0; while (1) { i =3D read(f, buffer, sizeof(buffer)); if (i <=3D 0) break; MDXUpdate(&ctx, buffer, i); } close(f); return (MDXEnd(&ctx, buf)); } --=20 Allan Jude --cefVxOIBD2vm4RApBE5O4tQ0lfigJF2Qm Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJWhNYKAAoJEBmVNT4SmAt+E8cP/3R383YXkUdn1jV4haIADU/r SiVvOgU6eNOlrPv7koeL7A7qrkb23/Hs+D+ucXChaTx5cu8EBMdLgRrL4dKnu/tW CyXMpq8k1i2h9PLdFiAM+SybVJ5FFkZdMwYvc17DxWkIGRyGuYteOE6QTB4IE2IW V8FM7I372IpDGWzTyBzPhg7kO83WYrN3MfisC9PYuwQN2JoEg5TY5sjp84qiIJzl OQjf6ZAQ2wBXqhF90L97Wz2dzP+IqJ8RovfP6yajGjC432aLQMoEEd/lb8OS6ftd Fr68jJhmjGSCtSU0Ix+SFhofDe8p3AJuodU+XAaXIpMl/m0jNqW3TDsjyopHBY2L +W4QjIbx1IZ4a9v8q33hPnFmY+WMhrhD7I/g6cqAFK81Ahfi9NfKhidDosjvHJVl rFn/hxyv+qy7mvGhYFd/FkyNmXFKTfv3ZQC+C6x9hVE2jg4gC6zA4yCsNq1tSN40 50Nt2wCo2m5Zr+QyFx6o9JyRP0Z2FNW0DBy9hloUASg3+LgJ7dpCFKVvwsJR/h7h /Odz1hYe3kHR9Zo2Zh+6PE21FMA2OzcrT6XLPbiLbRqLHhuGKBSt08XZEa2pm50m yMG5SwYLKfB1kkqHKAScvZB65Cnq11t60xBHB1Al1HySGawwVib9tZSc5ehu8Ss9 yKFYIitpyPSRnpBV3WIh =S8Jb -----END PGP SIGNATURE----- --cefVxOIBD2vm4RApBE5O4tQ0lfigJF2Qm--